laboratory in Kénitra and CIAD one in Montbéliard. This collaboration helped my thesis to be supervised by many professors leading to more interactions and scientific exchanges.

First and foremost, I would like to express my special appreciation and thanks to the defense committee for accepting to judge my work. I thank Professors Hichem Snoussi and Abdellatif Elafia for their considered time to read and evaluate this dissertation. Also, I thank Professors Ludovic Macaire and Habib Benlahmar for being examiners of my thesis defense.

I would like to thank my supervisor at UTBM, Prof Yassine Ruichek, for all the support, encouragement, and guidance he granted me. With his guidance and constant feedback, this Ph.D. achieved many contributions and publications. He gives interest and time to each scientific work with his prominent recommendations and numerous corrections. I thank him for his efforts to correct my writings and for providing the necessary computation machines to perform the experiments. He is endowed with perfection, dynamism, openness, and fascinating scientific standards. I gratefully thank Prof Rochdi Messoussi, my supervisor at Ibn Tofail University and director of LASTID laboratory and with whom I started this Ph.D. I thank him for believing in me, taking me as one of his Ph.D. students, and helping me to establish this collaboration with UTBM. Also, he demonstrated excellent listening and communication capacities during my research stays in Morocco, providing all the requested materials and supports.

Prof Messoussi always tries to build a research environment that promotes the interaction between the labmates and produces more scientific interactions.

In addition to my directors, I sincerely thank Prof Youssef El Merabet for his continuous guidance and support as my Co-advisor. This thesis has started thanks to him as he advised me to start a Ph.D. with the LASTID laboratory right after defending my engineer degree. He introduced to me the field of this thesis, the laboratories, and the possible collaborations that can be established. I thank him for all the scientific exchanges we had throughout this thesis and the corrections he made to my journal and conference papers. I gratefully thank Prof Raja Touahni for all her efforts for all the administrative procedures iii iv related to my thesis and its defense, scholarships, and for her high availability and mental support. She gave many valuable pieces of advice that will help me in my professional and personal life. I would like to thank all the members of the two laboratories for the lively and friendly atmosphere in the office and our generous and pleasant exchanges, inside and outside the laboratory. Their support was invaluable throughout my thesis, making my time both memorable and enjoyable. I really enjoyed the daily coffee breaks I had with my labmates and professors.

Last but not least, I would like to extend my sincere gratitude to my parents for their continued support, which has helped me stay strong and focused on the thesis work. I also extend my thanks to my sisters, my brother, and all my relatives and friends for their affection and encouragement.

INTRODUCTION 1.1/ FACE PERCEPTION

The face of a human being carries enough information to extract cognitive classification about his identity, facial expression, gender, and estimating his age. The face is considered as the primary communication channel rich in information observed by the environment and interpreted with high precision. Moreover, facial media is the most universal across all communications channels. It can be understood regardless the social, geographical, gender, and age belonging. The face particularities highlighted in the distinguishing of specific visual features related to the person's identity, emotional state, and other biometrics led to the computer vision-based facial analysis. The strategy consists of emulating the human perception skills on a computer through machine learning techniques, which proved that computers could perform perception tasks by learning and not programming similarly to humankind. The motivation also comes from the nature of the facial features, which are visual and not intrinsic or to be concluded as verbal language.

The computer vision discipline cares about how computers can process the outputs of digital images and videos and understand them to make decisions. It covers all analysis performed by our biological vision pipeline, including seeing a visual stimulus, processing it, and then extracting semantic information that can be used to make a decision or exploited in other processes.

In our social life, we spend more time interpreting faces than any other single stimulus through face perception. It is classified as the human brain's most sophisticated perceptual skill [1]. Since their birth, infants demonstrate fundamental facial processing capacities and show keen interest in faces [2]. Babies (1-3 days) can detect faces even with rotations reaching 45 degrees [3]. However, the studies found that this interest in faces is not sustained in early childhood cycles as the child grows. The interest is reduced in children aged 1 to 4 months. It re-emerges and seems to reach the peak lately on the first year, but it decreases slightly over the next two years of growth [4]. The observed re-emergence may be motivated by the child's self motor abilities and experiences [4].

1
Infants of two days old can mimic adult facial expressions, demonstrate the ability to recognize details such as the shape of the mouth and eyes, and move their muscles to create similar patterns on their faces [5]. Despite this ability, newborns so far are not conscious of the emotional content represented through facial expressions. Five-month-old infants pay equal attention to a person's image-making an anxious facial expression and making a happy facial expression, and showing similar Event-Related Potentials (ERPs) for both expressions. Nevertheless, seven-month-olds pay more attention to the fearful face, and their ERPs-based response for the scarred face is much stronger than the one for the happy face. Hence, this statement highlights an increased attentional and cognitive focus toward fear that reflects the emotion's threat-salient nature [START_REF] Peltola | Emergence of enhanced attention to fearful faces between 5 and 7 months of age[END_REF]. Face perception is a highly complex function for the human visual system (Figure 1.1). Some non-faces categories share visual properties with faces and potentially resemble faces (such as fruits or animals). Faces represent a very homogeneous visual category and are visually similar (surface, facial elements, general structure). Finally, the perception of faces must remain effective in a natural environment and, therefore, despite many physical conditions that modify faces' visual appearance (orientation of the head, relative size, level of lighting, partial vision, expression of emotions, etc.). Face perception allows at a glance to (1) detect faces in the scene and differentiate them from other categories (objects, animals) (face detection); (2) discriminate between several identities (individual face discrimination); (3) recognize familiar characters (recognition of faces). These processes are carried out efficiently by the human brain despite significant variations in relative size, the head's orientation, facial expression. Indeed, humans are performing well in perceiving faces.

A face can be detected in a visual scene in 100ms [START_REF] Crouzet | Fast saccades toward faces: face detection in just 100 ms[END_REF], and a face can be perceived as familiar in 200ms [START_REF] Caharel | Face familiarity decisions take 200 msec in the human brain: Electrophysiological evidence from a go/no-go speeded task[END_REF]. This efficiency makes it very difficult to understand the mechanisms involved. Understanding how the human brain perceives faces is, therefore, an essential issue for neuroscience. To this end, identifying the brain structures involved is a crucial step. The search for these brain structures began with neuropsychology and the description of patients with brain damage and then developed strongly 20-30 years ago to develop functional neuroimaging. Our ability to recognize a face seems so easy, fast, and automatic that it is difficult for us to imagine the mechanisms involved. As early as the 19th century, despite the absence of scientific evidence, some scientists wondered. For example, Francis Galton, who was interested in the physiognomy of faces, wrote in 1883:

" The general expression of a face is the sum of a multitude of small details, which are viewed in such rapid succession that we seem to perceive them all at a single glance."

Later, psychology made the same hypothesis, suggesting that faces are perceived as a whole, as a global unit [START_REF] Wagemans | A century of gestalt psychology in visual perception: I. perceptual grouping and figure-ground organization[END_REF]. This ability to perceive the multiple elements of a face simultaneously within a single, global representation is called a holistic process [10; 11]. Several shreds of evidence support a holistic mechanism of the perception of the faces. One compelling evidence for a holistic process of facial perception comes from the composite illusion of faces [START_REF] Rossion | The composite face illusion: A whole window into our understanding of holistic face perception[END_REF]. This illusion is based on the fact that one part of a face cannot be perceived without being combined with the rest parts. For example, two identical upper halves will be perceived as different if they are associated with two different lower halves (Figure 1.2). Another proof comes from the face inversion effect, which is a phenomenon where identifying inverted (upside-down) faces compared to upright faces is much more complicated than doing the same for non-facial objects. Behavioral studies have shown that a face's perception is disturbed when faces are inverted [13; 14; 15]. Thus, faces' perception is not based on the individual elements (nose, mouth, and eyes, are the same upright and inverted) but on a global representation only when the face is seen upright. From a theoretical point of view, the privileged theory concerning the organization of the visual system is a hierarchical organization with initially a perception "element by element" then a perception of these elements in a global representation [16; 17]. This view is almost incompatible with a holistic perception of faces and the results presented above in which global representation plays a significant role (composite illusion, inversion effect).

Another model for organizing facial perception called "coarse-to-fine" has been proposed [START_REF] Rossion | Picture-plane inversion leads to qualitative changes of face perception[END_REF], where faces would always be seen as a whole, first in a rough representation only to categorize a face as a face. Then, this global representation would be enriched in detail until reaching enough information to individualize the face and determine its identity.

In 1997, the authors in [START_REF] Kanwisher | The fusiform face area: a module in human extrastriate cortex specialized for face perception[END_REF] published an functional Magnetic Resonance Imaging (fMRI) study introducing the so-called "functional localizer" approach. This approach consists of presenting participants with faces and control stimuli (familiar objects) and asking them to complete a simple task on one of the two categories. A brain area is defined as selective to faces if it responds more to faces than to objects. This area is defined as a region of interest, and the role of this area is then further investigated using other fMRI experiments. Using this approach, the authors identified the most particular face area and localized it in the right spindle-shaped gyrus, in its posterior and middle parts (Figure 1.3). This area was named FFA (Fusiform Face Area). The authors considered this area to be the only important modulus for facial perception, ignoring the other selective facial areas highlighted in their study. The faces' perception would no longer be based on a vast network, but mainly on a single region, the most selective to faces. Subsequent studies in the 2000s were greatly influenced by this view and often focused their analyzes on FFA. With the rise of the internet and modern technologies, we can now confidently confirm that we live in a society of images. Nowadays, anyone can use their smartphone's camera to take a photo or video and share it on the web and social media. Large amounts of video are uploaded to platforms like YouTube. All these images that flood the internet are full of data that could be valuable for businesses. However, to be able to collect and analyze this data, computers need to be able to "see" an image and understand its content. This is the main objective of Computer Vision. The field of Computer Vision brings together multiple techniques from various fields of engineering or computer science. In general, the different methods aim to reproduce human vision. In order to understand the content of images, machines must extract a description: an object, a description, a 3D model, etc.

Specific computer vision systems may also require image processing, namely simplification or an increase in its content. Examples include normalizing the image's photometric properties, cropping its edges, or removing "noise" such as digital artifacts induced by low light. Allowing a computer to emulate biological vision is not straightforward. To this day, computer vision still fails to match the human vision. Part of the reason is that we still do not know how human vision works. We need to understand how perceptual organs such as the eyes work, but also how the brain interprets this perception. Although research in this area is advancing rapidly, we are still far from unraveling all the mysteries of vision. Another challenge is linked to the complexity of the visual world. An object can be perceived from multiple angles, partially hidden by other objects in various lighting conditions. However, a proper computer vision system must perceive the content in any situation and extract information from it. In fact, computer vision represents a real scientific challenge.

Despite the difficulties associated with computer vision development, significant advances have been made over the years that have already enabled Computer Vision to perform many tasks. It is effective for optical character recognition, also called "OCR." It is also used in automatic checkouts. The photogrammetry for generating 3D models is also based on computer vision. It is also used for machine inspection for medical imaging analysis. In the field of automobile safety, computer vision is used for the detection of dangers. With the emergence of self-driving cars, Computer Vision nowadays takes center stage in the automotive industry as it is what will allow vehicles to "see" on the road.

The facial recognition technologies of the most recent smartphones, such as the famous Face ID of the latest Apple iPhones, are also based on Computer Vision. Automatic surveillance cameras also exploit this technology.

On the other hand, the pattern recognition field concerns the automatic discovery of patterns in data through computer algorithms and the use of these patterns to carry out actions such as classifying data into different categories. Machine learning , which originates from artificial intelligence, is commonly used to refer to supervised learning methods. In contrast, data mining places more emphasis on unsupervised methods and a stronger connection to professional use. Pattern recognition has its functions in engineering. The term is popular in computer vision: one of the major conferences on computer vision is called the Computer Vision and Pattern Recognition Conference. In pattern recognition, it may be more interesting to formalize, explain, and visualize the pattern, while machine learning has traditionally focused on maximizing recognition rates. However, all of these fields have evolved considerably from their roots in artificial intelligence, engineering, and statistics. They have become more and more similar by integrating each other's developments and ideas.

In machine learning, pattern recognition works on labeling a given input, and it was referred to as discriminant analysis in statistics back to 1963. The most general application of pattern recognition is classification, where a classifier learns to predict a label to the processed input according to the training classes. However, pattern recognition is a more general issue that encompasses other types of output as well. Other examples are regression, which predicts a true-valued output to each input; sequence tagging gives a class to each pattern of a sequence of values (e.g., part of speech markup). Pattern recognition techniques generally aim to learn a logical response for all possible inputs and perform the "most probable" match of the inputs, considering their statistical variations. This is opposed to pattern matching, which looks to exact matches in the input with pre-existing patterns stored in a database. A typical example of a pattern-matching algorithm is regular expression matching. It searches for patterns of a given kind in text data, which is included in many text editors and word processors' search capabilities.

Unlike pattern recognition, pattern matching is generally not a type of machine learning.

However, pattern-matching algorithms (especially with reasonably general and carefully tailored shapes) sometimes provide output similar in quality to pattern recognition algorithms.

The flexibility of pattern recognition algorithms and their application alongside computer vision motivated human face analysis through machine learning. The facial analysis relies on processing the visual features by extracting them first and then finding their patterns.

The visual components are the combination of three features: textural, color, and shape.

The computation of relevant facial characteristics that help machine learning frameworks to predict the face identity or the dominant emotion according to the input face image relies on the discriminative power of the applied textural, color, and shape feature extraction methods.

The textural analysis is widely present in our natural world and plays a significant role in various critical applications. Any object or pattern's visual appearance can be represented in a texture form at a certain level by its size, shape, organization, and proportions of its parts. The texture is detected on both artificial and natural objects such as on wood, plants, materials, and skin. Texture analysis becomes a fundamental branch of image processing and computer vision by exploring how objects can be textured. Hence, many applications can be redefined as texture classification tasks, including face recognition and content-based image retrieval [20]. Moreover, texture classification has been adopted in medical image analysis and helps to achieve good results in congestive heart failure [START_REF] Raghavendra | Automated screening of congestive heart failure using variational mode decomposition and texture features extracted from ultrasound images[END_REF], human skin analysis [START_REF] Oghaz | An optimized skin texture model using gray-level co-occurrence matrix[END_REF], brain degenerative diseases [START_REF] Moraru | Texture anisotropy technique in brain degenerative diseases[END_REF], etc. During past decades, texture classification gained too much attention due to its difficulties in terms of variability and inhomogeneity, such as scale changes, variable illumination, surface shape variability, and imaging conditions.

The description or representation of shapes is an essential topic in image analysis for object recognition and classification. Descriptions are given in terms of properties of the objects contained in the images and relationships between them. These properties correspond to the characteristics of the position, size, and shape of the objects. Each shape or image to be stored in the database is processed to obtain the shape features. The shape features are then used by the various shape representation techniques to organize and efficiently retrieve the useful shape information into index structures. Shape-based feature extraction has been widely used to develop face detectors that demonstrate outstanding performance even when the face is affected by affine transformations. Also, the recognition of emotions from facial expression relies on computing shape information on the regions that are believed to present emotion-related visual components (eyes, mouth, nouse).

On the other hand, color is also important and the most straightforward feature that humans perceive when looking at an image. The human visual system, by nature, is more sensitive to color information than to grayscale, so color is the first candidate used for feature extraction. However, the facial analysis relies on dealing with the color features as three separated plans RGB, where each one of them is fed to extract the textural and shape-based characteristics. Then, a strategy is to gather the three extracted features and form the final one describing the color input face image.

After all, the discussed theoretical advances in computer vision and pattern recognition depend mainly on computation power. Nowadays, many calculation devices are available for the research community with more capabilities and lower prices than in the early '80s.

Also, the appearance of ready machine learning environments based on cloud computing helps the researchers to focus on coding their design ideas rather than solving hardware and software issues that were used in face analysis earlier. Moreover, the machine learning dedicated environment like Python and MATLAB managed to offer standard programming platforms so the researchers can share their source codes, which accelerated machine learning progress based on reusing, enhancing, and sharing pipelines. This task serves diverse applications that interest many markets. It can be helpful to children diagnosed with autism to understand their social environment better. Moreover, such technology would evaluate E-learning contents and public services more efficiently, accurately, and in real-time. Furthermore, the industry of Human Support Robot would develop robots qualified to adapt their interactions according to the emotional atmosphere.

In real-world scenarios, the desired system is expected to recognize unseen individuals' emotions in real-time, which makes this task among the toughest ones in computer vision.

In the literature, the recognition of facial expression presents four different levels, as can be seen in Figure 1.4, which shows the difficulty levels regarding the way the emotion is expressed (Spontaneous vs. Posed) and the person expressing it (the same person as the train or a different one). Spontaneous emotions are hard to be classified since each individual expresses a given emotion differently compared to another person.

Furthermore, this fact often leads to interclass samples interferences meaning that two emotion classes are represented over two images with the same overall appearance.

Moreover, the recording of spontaneous emotions must be performed. Simultaneously, the subjects are not aware of it, which is very hard to establish since the subjects should deliver authorization to record and use their images/videos. Therefore, few works were interested in spontaneous emotion recognition and focused only on verifying the The second challenge of facial expression recognition relies on correctly decoding individuals' observations not taking part in the train session. In this thesis, we deal with person-independent posed FER.

The developed framework for face recognition and facial expression classification, within the context of this thesis, should fulfill the following criteria:

• The developed systems must be reliable in terms of recognition/identification results by achieving very high recognition rates.

• Provide a stable performance when dealing with an unconstrained environment such as the lighting conditions, background inhomogeneity, the face's position, facial expressions, and, most importantly, the recognition of obscured faces.

• As with recognition rate, execution time is also an essential metric for evaluating facial recognition architecture, knowing that most future applications will be realtime.

• Compatibility with low-end devices such as robot systems, prototyping platforms (Raspberry Pi, Arduino), and mobile devices.

• Re-usability of the developed source codes by sharing them with the research community, which will offer opportunities for more enhancements and further implementations.

1.4. OUTLINE OF THE PHD THESIS DISSERTATION

1.4/ OUTLINE OF THE PHD THESIS DISSERTATION

In order to give the readers and community a comprehensive presentation of the contributions introduced in this thesis and also preparing their background to a fluent reading experience, the rest of the thesis dissertation is organized as follows:

• Machine learning algorithms are trained following one of two strategies, namely supervised and non-supervised learning. The difference between them is the way the machine reaches the knowledge. The unsupervised algorithm relies on the algorithm to autonomously discover a pattern linking the inputs between them based on their similarities.

It works mainly on clustering the inputs as subcategories representing a classification space. On the other hand, the supervised manner requires labeling all the observations for the training. Therefore, the learning is much easier than the unsupervised case, but it depends on the labeling quantity and quality. Facial analysis frameworks are achieved through supervised learning with full annotations regarding the challenging nature of such applications.

Despite the progress made in face recognition in terms of computation speed and performance accuracy, the identification process is not guaranteed all the time, and it is vulnerable when the exterior environment is wild. Thus, the computational abilities to correctly classify a facial image depends on many parameters related to the environment such as lighting and contrast, face position, facial expressions, image quality, and also

2.2/ FACIAL IMAGE CLASSIFICATION FRAMEWORK

Image-based classification systems share a typical architecture regardless of the targeted application. Despite more than 20 years of extensive research and the large number of papers published in journals and conferences dedicated to this area, we still can not claim that artificial systems can measure to human performance regarding the discussed facerelated challenges: difficult light imaging conditions, head pose, aging, facial expressions, and occlusions. However, the literature works share a standard structure and lead to a generic facial analysis system, which involves three basic steps as illustrated in Figure 2.1. This system's input can be an image or a video stream of the face to be recognized, while its output is the person identification label and/or its emotional state.

Detection Feature Extraction Classification

2.3/ FEATURE EXTRACTION

Feature extraction aims to encode the image space into another that motivates the person-related visual features than the common ones and performs dimensionality re- as a high dimensional signal vector by listing all the pixels as one vector and not a matrix.

The face recognition literature was enriched by the proposal of several global methods, including approaches based on different transforms such as wavelet sub-bands [START_REF] Huang | Non-uniform patch based face recognition via 2d-dwt[END_REF], Gabor filters [START_REF] Abhishree | Face recognition using gabor filter based feature extraction with anisotropic diffusion as a pre-processing technique[END_REF], optimal matrix factorization [START_REF] Guan | Nenmf: An optimal gradient method for nonnegative matrix factorization[END_REF] and steerable pyramid transform [START_REF] Aroussi | Local appearance based face recognition method using block based steerable pyramid transform[END_REF], independent component analysis (ICA) method [START_REF] Secchi | Hierarchical independent component analysis: a multi-resolution non-orthogonal data-driven basis[END_REF], Zernike moments method [START_REF] Bereta | Local descriptors and similarity measures for frontal face recognition: a comparative analysis[END_REF], global

Gabor-Zernike feature descriptor [START_REF] Fathi | A new global-gabor-zernike feature descriptor and its application to face recognition[END_REF], principal components analysis (PCA) method [START_REF] Cavalcanti | Weighted modular image principal component analysis for face recognition[END_REF], linear discriminant analysis (LDA) based Fisherface method [START_REF] Lu | Incremental complete lda for face recognition[END_REF], etc. It is stated by the literature researches that the global techniques are fast in extracting and computing the similarities between the features. However, they demonstrated many weaknesses, which can be found in [START_REF] Yang | Face recognition using adaptive local ternary patterns method[END_REF]. Figure 2 If the coordinates of the center pixel are (x c , y c ) then the coordinates of his P neighbors (x p , y p ) on the edge of the circle with radius R can be calculated with the sinus and cosines:

Ξ(x, y) =          1 , x ≥ y 0 , x < y (2.2)
x p = x c + R.cos( 2πp P ) (2.
3)

y p = y c + R.sin( 2πp P ) (2.4)
In Figure 2.5 the neighborhood was expanded to capture dominant feature with largescale structures. The neighborhood can be denoted by a pair (P, R) where P is the sampling points on a circle of radius of R. Therefore, there are 2 P different output values.

Figure 2.5: The circular (8,1),(16,2) and (8,2) neighborhoods.

2.3.2.2/ LBP-LIKE DESCRIPTORS

The LBP descriptor proved an outstanding performance in many applications, leading to an important expansion of LBP-like methods and inspiring the researchers to develop new ones.. Indeed, after Ojala's proposal [35] and thanks to the demonstrated flexibility and efficiency, the overall LBP-like concept has proven very prominent, and a great variety of LBP variants have been proposed in the literature to overcome the weaknesses and enhance the strengths of the original LBP method covering discriminative power, robustness, and applicability of LBP. Tan and Triggs introduced a three-level operator referred to as local ternary patterns (LTP), which is a generalization of LBP proving more effective than the original operator in face recognition application [START_REF] Tan | Enhanced local texture feature sets for face recognition under difficult lighting conditions[END_REF]. The concept behind is based on three values (1, 0, or -1) encoding the difference between the center pixel and its neighbor pixels considering a ∆ threshold. Motivated by the basic LBP operator, Zhang et al. [START_REF] Jabid | Local directional pattern (ldp) for face recognition[END_REF] proposed a higher-order local derivative pattern (LDP) for face recognition, which provides more detailed descriptions of facial images. However, the LDP presents more noise sensitivity than LBP. CS-LBP [START_REF] Heikkilä | Description of interest regions with centersymmetric local binary patterns[END_REF] was developed for image matching based on SIFT framework. CS-LBP could extract the feature of a given region with less dimensionality than SIFT and LBP. Based on the CS-LBP descriptor, Fu and Wei [START_REF] Fu | Centralized binary patterns embedded with image euclidean distance for facial expression recognition[END_REF] proposed centralized binary pattern (CBP) for facial expression recognition. CBP considers the information in the center pixel by comparing its value to the average of all eight pixels in the neighborhood and giving it the largest weight. However, it is not easy to manually set a suitable CS-LBP threshold in all methods using thresholding like LTP. Also for face recognition, Rivera et al. [START_REF] Rivera | Local directional number pattern for face analysis: Face and expression recognition[END_REF] The outstanding performance of success of LBP method in many applications related to computer vision, has motivated much new researches to propose enhanced LBP variants.

Regarding its flexibility and ease of implementation, the LBP operator can be worked out to meet the requirements of various applications, which include non traditional texture problems such as dynamic texture and scene classification, medical image analysis, face recognition, etc. The new extensions of LBP try to enhance the following aspects of the basic LBP descriptor:

1. Neighborhood topology and sampling: This extension focuses on defining more prominent neighborhood area and sampling the pixels in an effective way to cover most of the textural transitions.

2.

Thresholding and quantization: Thresholding process is one of the key elements of the LBP philosophy. The researchers develop sophisticated thresholding kernel functions to perform the binarization and quantization of gray level transitions to multiple levels based on thresholding settings.

3.

Encoding and regrouping: This category relys on spliting the pixels of the neghborhood area into groups of patterns, to be combined, to improve distinctiveness.

4.

Combining complementary features: Current trend in local image and video descriptors is to combine multiple complementary LBP-like descriptors, or LBP-like with non-LBP descriptors in the objective of exploring the advantages of different concepts.

More details about the taxonomy of existing LBP variants can be found in [START_REF] Liu | Local binary features for texture classification: Taxonomy and experimental study[END_REF] where their merits and demerits and their underlying connections were analyzed. A wide variety of LBP-like methods have been proposed in pattern recognition literature, usually developed to extract features to achieve specific applications (see Table 2.1). Unlike global approaches, local descriptors, which overcome the limitations mentioned above, divide the whole face into smaller image patches, from which local features are extracted and combined to build one face descriptor. The authors in [START_REF] Fathi | A new global-gabor-zernike feature descriptor and its application to face recognition[END_REF] reported that local description is non-sensitive to nonessential and irrelevant patterns of facial and textural images .

Heisele et al. [START_REF] Heisele | A component-based framework for face detection and identification[END_REF] evaluated global and local descriptors to disclose that the local descriptors outperform global ones. Note that most of LBP variants have been proposed usually for one specific application, which are intended by the researchers and then each variant is tuned to reach it best performance according to the specific application. Indeed, as we will see later on this paper, majority of LBP variants have been proposed for texture analysis and, to a lesser degree, to fulfill the specifications and deal with the challenges of different applications including dynamic texture and scene classification, medical image analysis, etc . However, the performance of most of these LBP extensions have not been yet evaluated on face recognition problem. Hence, the state-of-the-art lacks of an extensive review of handcrafted descriptors for face recognition, which should be done by performing a large scale empirical study on challenging face datasets to investigate the weakness and strengths of LBP-like methods. We then proceed to the transfer learning technique to extract the features of the subject database that belongs to the same application as the database used for the initial training (same classes). Once the features are obtained, we train the classifier and evaluate the performance of each deep feature.

2.4/ CLASSIFICATION

The supervised classification objective is mainly to define rules making it possible to classify objects in classes from qualitative or quantitative variables characterizing these objects. The methods often extend to quantitative Y variables (regression). In the following, we introduce the widely used algorithms for face analysis. Table 2.2 lists the literature metrics used to compute the pairwise distance.

(r, s) = (x r -x s )(x r -x s ) Standardized Euclidean distance d(r, s) = (x r -x s )trace(Σ) -1 (x r -x s ) Mahalanobis distance d(r, s) = (x r -x s )Σ -1 (x r -x s ) City Block metric d(r, s) = n j=1 x r j -x s j Minkowski metric d(r, s) = p n j=1 x r j -x s j p Cosine distance d(r, s) = 1 - x r x s √ x r x r √ x s x s) Correlation distance d(r, s) = 1 - (x r -xr )(x s -xs ) √ (x r -xr )(x r -xr ) √ (x s -xs )(x s -xs ) Hamming distance d(r, s) = #(x r j x s j ) n Jaccard distance d(r, s) = #[(x r j x s j )∧((x r j 0)∨(x s j 0))] #[(x r j 0)∨(x s j 0)]
x and x denote a column vector and its transpose respectively.

x r and x s indicate the r th and s th samples in the data set, respectively.

x r j indicates the j th feature of the r th sample in the data set.

xr indicates the mean of all features in the r th sample in the data set. Σ is the sample covariance matrix. The symbol # denotes counts; the number of instances satisfying the associated property.

2.4.2/ SUPPORT VECTOR MACHINES

The Support Vector Machine (SVM) is a discriminative classifier formally based on defining separating hyperplanes. In other words, given labeled training data (supervised learning), the algorithm outputs optimal hyperplanes dividing the two-dimensional space generating an SVM model based on the representation of the training data as points in space, mapped so that the examples of the separate categories or classes are divided by a dividing plane that maximizes the margin between different classes which allows to classify the query points and predict their classes.

Figure 3.3 illustrates the concept behind Support Vector Machines classification. On the left side, we have the input feature vectors calculated earlier using the handcrafted descriptors mapped by a complex curve. The classifier will rearrange these input points to reach a linear separation using a set of mathematical functions called kernels.

                         Linear : u v Polynomial : (γu v + C) d RBF : e (-γ * |u-v| 2 )
S igmoid : tanh (γu v + C)

(2.5)

2.4.3/ NEURAL NETWORKS

Neural Networks are extensively used for classification purposes as well as regressionbased tasks. Like SVM, a neural network-based classifier requires to be trained to generate a model that predicts the labels of probe images. The Neural classifier has the shape of an autoencoder organized in successive layers: an input layer, an output layer, and between the two one or more intermediate layers, also called hidden layers. There is no connection between neurons in the same layer, but every neuron in one layer is connected to all neurons in the next layer. The input layer has the same dimension as the feature vector extracted from the image, while the last layer dedicates one neuron to each class label. A hidden layer helps the autoencoder classifier model nonlinear relationships between the inputs and their labels. A single hidden layer is sufficient in theory, but having more hidden layers makes it easier to model a non-continuous discrimination function.

The autoencoder classifier training relies on finding the configuration of the connection weights between neurons to associate with each input feature vector its corresponding 1 https://www.csie.ntu.edu.tw/ cjlin/libsvm/ label. Therefore, increasing the hidden layers extends the weight configurations to be checked.

Input Layer

MxN neurons them to form the feature vector fed to an auto-encoder classifier to predict the image label. The evaluation was performed on LFW. However, the results were not outperforming the state-of-the-art regarding the modalities considered and the computational complexity. [START_REF] Sharma | A new pose invariant face recognition system using pca and anfis[END_REF] introduced a pose-invariant face recognition framework based on PCA for feature extraction and adaptive neuro-fuzzy inference system (ANFIS) classifier. However, this work is not complete in terms of evaluation that was limited to the ORL database. [START_REF] Moussa | A novel face recognition approach based on genetic algorithm optimization[END_REF] develop a fast face recognition system based on DCT and PCA techniques for feature extraction and a Genetic Algorithm (GA) to compute discriminant features from DCT-PCA ones and remove the irrelevant bins. The minimum Euclidian distance (ED) is used to calculate the similarities and then classify the images. [START_REF] Mian | An efficient multimodal 2d-3d hybrid approach to automatic face recognition[END_REF] proposed 2D and 3D-based modalities for face recognition through a hybrid transform to correct the pose of a 3D face using its texture and achieving efficiency and robustness to facial expressions. The SIFT descriptor is used to compute the features from corrected 3D views and fed to the iterative closest point (ICP) algorithm for the decision. This system is less sensitive and robust to facial expressions, which achieved a 98.6% verification rate and 96.1% identification rate on the complete FRGC v2 database. [START_REF] Karanwal | Od-lbp: Orthogonal difference-local binary pattern for face recognition[END_REF] proposed a new LBP-like face descriptor referred to as Orthogonal difference-local binary pattern (OD-LBP). Their scheme relies on computing three orthogonal transformations and then computing the differences between them, leading to 3 feature maps. Each feature map is divided into nine non-overlapping sub-regions to perform local histogram count. Then the local histogram vectors are concatenated and fed to the SVM algorithm to perform training and evaluation. The authors adopted five benchmarks but with a low number of individuals (less than 50), which is not challenging compared to other large datasets. To extract the local features, the whole image is divided into sub-regions, while the global features are extracted directly from the whole image. After that, PCA and Fisher linear discriminant (FLD) techniques are introduced on the fused feature vector to reduce the dimensionality. The CMU-PIE, FERET, and AR face databases are used for the evaluation. [START_REF] Kamencay | A novel approach to face recognition using image segmentation based on spca-knn method[END_REF] developed a new face recognition method based on SIFT features and PCA for space reduction and Nearest Neighbor as the classifier. The framework is referred to as SPCA-KNN and was evaluated on a dataset of 100 subjects with 1000 images for training and 500 for testing. However, this work lacks a comprehensive comparison with the state-of-the-art, and the evaluation should be on more benchmarks.

MxN input

2.6/ FACIAL EXPRESSION RECOGNITION STATE-OF-THE-ART

The computer vision community has conducted many studies devoted to facial expression recognition (FER) by applying machine learning techniques. In this section, we briefly present some state-of-the-art FER frameworks to highlight some of the proposed architectures that rely on either handcrafted descriptors or deep-learning methods. Shan et al. [START_REF] Shan | Facial expression recognition based on local binary patterns: A comprehensive study[END_REF] proposed an approach, referred to as Boosted-LBP, based on combining a basic LBP descriptor with the Adaboost algorithm to enhance the classification performance. They conducted experiments on CK+, MMI, and JAFFE databases, and found that the Boosted-LBP outperforms the basic LBP combined with a multi-class SVM classifier. Moreover, they reported that local methods (LBP) perform better than global methods (Gabor filters). Zhang et al. [START_REF] Zhang | Facial expression recognition using lbp and lpq based on gabor wavelet transform[END_REF] proposed a novel facial expression recognition method using a local binary pattern (LBP) and local phase quantization (LPQ) based on a Gabor face image. First, Gabor wavelets are applied to capture the prominent visual attributes, which are separable and robust to illumination changes, by extracting multi-scale and multi-direction spatial frequency features from the face image. Then, the LBP and LPQ features based on the Gabor wavelet transform are fused for face representation. Considering that the dimensions of a fused feature are too large, the PCA-LDA algorithm is used to extract compressed features. Finally, the method is tested and verified using multiclass SVM classifiers. Lekdioui et al. [START_REF] Lekdioui | Facial decomposition for expression recognition using texture/shape descriptors and svm classifier[END_REF] proposed an automatic FER framework based on a local appearance approach, extracting the features from seven regions of interest (ROIs) covering the left eyebrow, right eyebrow, left eye, right eye, eyebrows, nose, and mouth. They evaluated the LBP, LTP, and CLBP texture descriptors and their combination with the HOG operator cascading with a linear SVM classifier. They found that the concatenation of LTP and HOG leads to the best FER performance on three datasets (CK, FEED, and KDEF). Their framework strengths rely on extracting the appearance features from seven sub-images defined from landmarks carrying information about the facial expression class, in addition to combining the LBP-Like descriptor with the HOG operator.

However, this architecture presents certain drawbacks that we can point out. The seven extracted sub-images have different sizes and orientations, but their computed features have the same length. We found that the nose region of interest is vertically oriented compared to the eyebrow regions, which are horizontal, and the eye regions, which are almost square. Therefore, different amounts of information on different locations are represented over feature vectors of the same length. Furthermore, this study did not cover an important number of handcrafted methods, and no deep-learning method was evaluated. The method proposed by Makhmudkhujaev et al. [START_REF] Makhmudkhujaev | Facial expression recognition with local prominent directional pattern[END_REF] uses a new handcrafted LBP descriptor referred to as local prominent directional pattern (LPDP) for FER application. It is also an appearance-based approach exploring the benefits of extracting features from three patches: edge, curved edge, and corner-like texture maps. Their study focuses only on the handcrafted descriptor LPDP and its scheme to extract textural features. The authors also used a thresholding parameter to discriminate significant features from insignificant patterns in featureless/smooth regions of a face. Afterwards, a feature selection method is applied to reduce the dimensionality of the final feature vector because they use the spatial division on the input image. However, this system takes as input the entire face image, which makes it inconvenient for person-independent FER applications. In addition, no shape descriptor has been adopted in the overall framework, relying only on LPDP extracted features. Minchul et al. • The performance of the proposed MNTCDP operator and its stability are evaluated on four benchmark databases. To make this investigation more meaningful, we record the average classification rate over ten random splits and different training images.

• We conducted a fair and systematic comparison between the proposed MNTCDP descriptor and, on the one hand, a large number of state-of-the-art LBP variants, which have been rarely evaluated in the face recognition field, and on the other hand, several recent state-of-the-art face recognition systems.

• The performance of the proposed MNTCDP descriptor is compared to the one achieved by two deep learning methods adopting the same experimental protocols.

3.2/ MIXED NEIGHBORHOOD TOPOLOGY CROSS DECODED PAT-TERNS

The proposed MNTCDP face descriptor relies on two essential aspects: neighborhood topology and pattern encoding, which are the core components of a face image descriptor.

3.2.1/ NEIGHBORHOOD TOPOLOGY

The essence of MNTCDP descriptor is to perform neighborhood topology and pattern encoding in the most informative directions contained within face image. For face recognition, useful face image information consists of two parts [START_REF] Ding | Multi-directional multi-level dual-cross patterns for robust face recognition[END_REF]: the configuration of facial components and the shape of each facial component. The shape of facial components is, in fact, rather regular. After geometric normalization of the face image, the central parts of several facial components, i.e., the eyebrows, eyes, nose, and mouth, extend either horizontally or vertically, while their ends converge in approximately diagonal directions (π/4 and 3π/4). In addition, wrinkles in the forehead lie flat, while those in the cheeks are either raised or inclined.

Based on the above observations, neighborhood topology of MNTCDP is conducted as shown in Figure 3.1. In order to capture more discriminant information and without in-creasing computational complexity, we adopted 5 × 5 block in MNTCDP descriptor, which allows combining radii (2) and angles (4), compared to 3 × 3 block supporting only angle variation. The pixels of such block cover at the same time, the four orientations which are We define, as shown in Figure 3.2, two sampling groups S G 1 and S G 2 . S G 1 contains the pixels in blue color, i.e., the even pixels of level A located in the vertical and horizontal directions {A 0 ,A 2 ,A 4 ,A 6 } and the odd ones of level B located in the two diagonal directions {B 1 ,B 3 ,B 5 ,B 7 }, while S G 2 contains the pixels in green color, i.e., the odd pixels of level A located in the two diagonal directions {A 1 ,A 3 ,A 5 ,A 7 } and the even ones of level A located in the vertical and horizontal directions {B 0 ,B 2 ,B 4 ,B 6 }. In a more manageable way, the sampling groups S G 1 and S G 2 are given as follows (cf. Eq. 3.1 and Eq. 3.2):

S G 1 = {A 2i , B 2i+1 }; i = 0, 1..., 3 (3.1)
S G 2 = {A 2i+1 , B 2i }; i = 0, 1..., 3 (3.2)
It is known that the average gray level is a widely accepted statistical parameter for texture analysis. The rest of the pixels within the 5×5 neighborhood, labeled as C i;i=0,..,7 in Fig-

M A 2i = p∈ω 2i A p + q∈υ 2i C q + B 2i + I c P + 1 ; (3.3) M A 2i+1 = p∈ω 2i+1 A p + q∈υ 2i+1 C q + B 2i+1 + I c P + 1 ; (3.

3.2.2/ PATTERN ENCODING

After defining the neighborhood topology, which permits to define pixels that will be used in the modeling, we present, as a second step, the pattern encoding scheme of the proposed MNTCDP descriptor. The local information is encoded using two encoders named and Ec 2 associated to the two sampling groups S G 1 and S G 2 are computed as:

Ec 1 (χ) = 3 i=0 Λ(B 2i+1 ,M A 2i+1 ) × 2 i + P-1 i=4 Λ(A 2(i-4) ,M A 2(i-4) ) × 2 i + Λ(I c , M Ec 1 ) × 2 P (3.5) Ec 2 (χ) = 3 i=0 Λ(B 2i ,M A 2i ) × 2 i + P-1 i=4 Λ(A 2(i-4)+1 ,M A 2(i-4)+1 ) × 2 i + Λ(I c , M Ec 2 ) × 2 P (3.6)
where

M Ec 1 = 3 i=0 A 2i P/2 (3.7) M Ec 2 = 3 i=0 A 2i+1 P/2 (3.8)
In equations 3.2.2 and 3.2.2, χ is the set of gray-scale values of a 5×5 square neighborhood.

To make the representation more robust, the coarse and fine information can be captured by multi-scale which can be made through a linear combination of different features obtained by several operators. In this paper, the final MNTCDP code obtained for each pixel of the image, is the concatenation of the two features generated by the two cross encoders Ec 1 and Ec 2 : After encoding each pixel in the face image using the two cross encoders Ec 1 and Ec 2 , two code maps are produced. To incorporate more spatial information into the final MNTCDP descriptor, the obtained code maps are spatially divided into small spatial w × w nonoverlapping portions referred to as blocks histograms of MNTCDP codes are extracted from each block. All these regional sub-histograms of dimensionality m are concatenated through Eq. 5.12 to form the holistic face representation of dimensionality m × w 2 . The overall framework of the MNTCDP based face representation approach is illustrated in Figure 3.3. This face representation can be directly used to measure the similarity be- tween a pair of face images using metrics such as City Block and chi-squared distances or histogram intersection.

MNTCDP = Ec 1 , Ec 2 (3.9)
H = w 2 i=0 H i (3.10)
Where H is the final descriptor, is the concatenation operation, and H i is the histogram of the MNTCDP codes for block B i calculated using Eq. 5.9.

H i = H Ec 1 i , H Ec 2 i (3.11)
where

H Ec 1 i (k) = χ⊂B i δ(Ec 1 (χ)), k) (3.12) H Ec 2 i (k) = χ⊂B i δ(Ec 2 (χ)), k) (3.13)
In Equations 5.10 and 3.13, k ∈ [0, N bins ] is a pattern to compare to Ec 1 or Ec 2 patterns, N bins =2 9 is the number of bins, χ is the set of gray-scale values of a 5×5 square neighborhood and the delta function δ(•) is defined as below (cf. Eq. 5.11):

δ(a, b) =        1, if a = b; 0, otherwise (3.14
D L1 (h i , h j ) = k |h k i -h k j | (3.15)
where h i ={h 1 i ,h 2 i ,...,h k i } is the query feature vector and h j ={h 1 j ,h 2 j ,...,h k j } is the target feature vector.

This kind of parameter-free classifiers is particularly suitable for feature comparison pur-poses as they can handle a large number of classes, avoids parameter overfitting, and requires no learning/training ( [START_REF] Fernández | Texture description through histograms of equivalent patterns[END_REF], [110]). The procedure is repeated ten times, each time with new subdivision into training and validation sets, and the accuracy obtained with each subdivision is recorded.

3.4/ EXPERIMENTAL ANALYSIS

This section evaluates the proposed MNTCDP descriptor for face recognition to validate its performance and stability. Experiments are conducted through the framework presented in Figure 3.5 using five challenging and widely used benchmarks. A comparison is performed against the 71 LBP-variants listed in Table 2.1 and PCNANet2 and CBFD deep face features within our framework. Moreover, this section highlights the superiority of our MNTCDP-based recognition system according to the state-of-the-art ones.

3.4.1.4/ FERET

The FERET3 face image database [START_REF] Phillips | The feret evaluation methodology for face-recognition algorithms[END_REF] is a large benchmark, widely used database.

The FERET database was collected in 15 sessions in 6 years. The database contains 1564 sets of images for a total of 14126 images, including 1199 individuals. We used a subset that contains 1400 images of 200 persons (7 images per person) as adopted in [START_REF] Huang | Fuzzy linear regression discriminant projection for face recognition[END_REF] and many other state-of-the-art recent works: [START_REF] Huang | Fuzzy linear regression discriminant projection for face recognition[END_REF], [START_REF] Yang | Face recognition using adaptive local ternary patterns method[END_REF], [START_REF] Karczmarek | An application of chain codebased local descriptor and its extension to face recognition[END_REF]. Figure 3.9 presents the images of one class illustrating the various orientations.

3.4.1.5/ AR

The AR Face Database4 was created by [START_REF] Ding | Features versus context: An approach for precise and detailed detection and delineation of faces and facial features[END_REF], containing over 4000 face images of 70 men and 56 women. In addition to many images, sunglass and scarf occlusions make this database more challenging the classification task and evaluate the robustness and effectiveness of the described methods. A subset that contains 100 individuals (50 men and 50 women) is selected from the AR database. Each subject has 26 samples covering expression changes, illumination variations, and different disguises (sunglass and scarf).

3.4.2/ EXPERIMENTS CONFIGURATION

To better illustrate the advantages of MNTCDP against the evaluated state-of-the-art descriptors according to the particularities of selected datasets, six experiments are conducted:

• Experiment #1: This experiment is performed on the ORL database, considering all possible numbers of training images. For each tested descriptor, the accuracies are recorded over ten random splits. After performing a series of extensive experiments on the ORL database, we found that the best number of blocks required to obtain the best accuracy is nine blocks for all evaluated descriptors.

• Experiment #2: Like the previous one, this experiment is carried out on the adopted subset of the FERET database. It consists of investigating the performance of MNTCDP descriptor according to the multi-orientation of facial images, which is a sheer fact in face recognition. As on the ORL dataset, we calculate the accuracy over ten subdivisions in the train and test sets, and the best number of blocks to compute the histogram feature is nine blocks for all the tested descriptors.

• Experiment #3: In the objective of evaluating the performance of the proposed descriptor against noisy conditions, this experiment is performed on the Yale dataset, The histogram computation for this experiment was performed on 324 blocks.

• Experiment #5: This is a second experiment carried out on the Extended Yale B database using the evaluation protocol adopted in [START_REF] Mehta | Dominant rotated local binary patterns (drlbp) for texture classification[END_REF] with five subsets, where Subset 1 is used as the train set, and the remaining four subsets are considered as the validation sets. The recognition rate is calculated on each of the four validation sets without cross-validation. In this experiment, we adopted the same number of blocks as in the previous one.

• Experiment #6: The objective of this experiment is to evaluate the performance of the proposed method against sunglass and scarf occlusion configurations, which was not evaluated in the previous experiments. To achieve that, we used the adopted subset of the AR database. Like the other used datasets, we adopted the and AR (Exp#6) databases, respectively. Table 3.5 summarizes the obtained recognition rates on Extended Yale B database adopting the subsets based experimental protocol adopted in [START_REF] Mehta | Dominant rotated local binary patterns (drlbp) for texture classification[END_REF].

3.4.3.1/ PERFORMANCE ANALYSIS ON ORL: EXPERIMENT #1

The ORL database is widely used in the literature to test new descriptors and approaches thanks to frontal images taken in a controlled environment, equilibrated number of subjects, and the samples per each subject. The challenge herein is to achieve 100% average accuracy with a number of reference images as minimum as possible. Table 3 To test the robustness of the proposed descriptor on FERET, a real-world challenge database, we set up a subset containing the frontal images and also samples with various orientations taken at different sessions. Unlike existing state-of-the-art approaches, we used facial images with the whole original background (not cropped images), negatively influencing the description process. Table 3.2 summarizes the obtained average face recognition accuracies recorded over ten splits under all possible numbers of training images. It can be seen that MNTCDP shows the best performance as it outperforms all the evaluated methods. MNTCDP achieves a higher recognition rate of 99.7% with six images in the train set over ten random splits, vs. 99.1% with DDBP, 99.05% with DCLBP and AECLBP-S, 98.99% with CRLBP-S-MxC, which are considered as the top performing descriptors (following the proposed one) in this experiment. To underline that some descriptors like LDBP and QBP which achieved good results on the ORL database, could not keep the same performance on this subset of FERET. In contrast, DDBP, which was not among the top-ranked descriptors on the ORL database, performs well on the FERET database. It would be of interest to note that facial images with orientations between 67.5°and 90°are hard to recognize, especially when the train and test sets are composed of samples with different angles, which is the case of the selected subset. This fact has prevented the proposed descriptor from reaching perfect accuracy of 100% over ten subdivisions. Moreover, achieving an average accuracy above 95% in a challenging database like FERET is considered a very satisfactory result. This is the case of the proposed descriptor, which achieves higher accuracies (above 96%) starting from 4 images in the train set. From Figure 3.12, we find that the accuracy recorded by the proposed MNTCDP method surpasses all the tested state-of-the-art descriptors and this under all numbers of training images. Furthermore, MNTCDP realizes a clear advantage against the other descriptors between the two train/test configurations 2/5 and 5/2. In this experiment, the objective is to demonstrate the performance and the effectiveness of the proposed MNTCDP operator and the state-of-the-art evaluated methods against massive illumination changes and variable background of the subjects. Table 3.3 shows the classification performance of the evaluated descriptors. It is apparent from Table 3.3 and Figure 3.13 which illustrates the performance evolution of the five top-ranked descriptors according to the different number of training images that the proposed MNTCDP descriptor proves a great success in the Yale database. Indeed, it achieves, for almost all the train/test configurations, accuracy above 91%, and it is the only descriptor able to reach 100% over ten random splits using seven or more images in the train. Note that, oppositely to all tested descriptors, MNTCDP provided rising scores from Train/Test configuration to the following one. Moreover, we remark that MNTCDP is more stable than the evaluated descriptors according to the train/test configurations. The second-best performing descriptor is dLBPα with its high score of 99.67% achieved only with two images in the test set. XCS-LBP method occupies the third rank by reaching 99.3% recognition rate average over ten splits at 7/4 configuration. The remarkable performance of the proposed MNTCDP operator for the classification of face samples of the Extended YALE B database using the two evaluation protocols (Exp#4 and Exp#5) demonstrates its strength and effectiveness against massive illumination changes and the significant number of samples per subject. The experiment on the AR dataset was designed to evaluate the performance of the proposed MNTCDP method and the state-of-the-art descriptors against occlusions. The selected subset contains 2600 images of 50 men and 50 women subjects with 26 face images per individual, including sunglass and scarf occlusion situations. In light of the previous statements, we conclude that MNTCDP outperformed PCANet2 and CBFD methods in performance and stability in all performed experiments. PCANet2 method suffered of major performance drops and as we saw it performed badly at various configuration of training and testing sets. CBFD presented better stability than the PCANet2 method but did not outperform the proposed method on all the datasets. Moreover, the evaluated deep methods have many parameters to adjust, which require many experiments in order to reach the optimal ones. Meanwhile, the proposed descriptor has no parameter to set and gave excellent and competitive accuracies on all the tested datasets. Deep learning methods are computationally expensive and require pre-trained models and feature learning phase in case of deep feature methods.

3.5/ COMPARISON WITH STATE-OF-THE-ART SYSTEMS

This section is dedicated to comparing the proposed MNTCDP face recognition system performance recorded on each dataset against those reported in recent state-of-the-art works. It should be pointed out that each published paper considered for comparison purpose is reviewed carefully to determine the top recognition rate achieved on each dataset along with the used experimental setting, i.e., the used configuration into training/validation sets and number of splits. The extracted results from reviewed state-of-theart papers are arranged in Table 3.8 for Experiments #1 to #4 and Experiment #6, and in Table 3.9 for Experiment #5. Based on these results, we can readily make the following statements:

• ORL: It can be inferred that MNTCDP based system significantly outperforms all reported papers by achieving 100% average accuracy over ten random splits using only 5 images in the train set, while the best performing state-of-the-art work [START_REF] Karczmarek | An application of chain codebased local descriptor and its extension to face recognition[END_REF] {1 recorded only 98% at the same configuration over only one split. Note that the top accuracy reported in the literature was 100% reached by [START_REF] Yang | Face recognition using adaptive local ternary patterns method[END_REF] but with eight images in the train and 2 for the test, over only 1 split.

• FERET: The results recorded on the FERET database listed in Table 3.8 show that MNTCDP outperforms again the best performing existing approaches, especially with the two most used 5-2 configuration into training/testing sets where it reached a score of 98.52% over ten random splits. As can be seen, many works suffer on the 5/2 configuration using only 1 split and are unable to achieve a competitive recognition rate to that obtained by the proposed MNTCDP based system. The most competitive existing recognition rate was 98% at 3/4 configuration over only 1 split, which is achieved by [START_REF] Gaidhane | An efficient approach for face recognition based on common eigenvalues[END_REF]. Note that the performance of the MNTCDP based system over 1 split achieved a score of 100% after feeding the train set with 5 images.

• YALE: It can be observed clearly from Table 3.8, that the best achieved state-of-theart recognition rate was 96% at 5/6 configuration over only 1 split [START_REF] Zhou | Face recognition based on an improved center symmetric local binary pattern[END_REF] vs 99% with MNTCDP based system over 10 splits (see Table 3.3). To our best knowledge, we are the first to achieve 100% average accuracy with the configuration of 7 images for the train set and 4 images as the probe set.

• Extended YALE B: The recognition accuracy reported in the literature and the one obtained by our system according to the training images (Exp#4) are listed in Table 3.8. It is easily found that the MNTCDP based system outperformed all existing state-of-the-art systems, by achieving 98.91% average accuracy over 10 splits using half/half configuration, exceeding [START_REF] Wang | Correntropy matching pursuit with application to robust digit and face recognition[END_REF], [START_REF] Tian | Coupled learning for facial deblur[END_REF] and [START_REF] Zhang | Nuclear norm-based 2-dpca for extracting features from images[END_REF] by 12.36%, 5.51% and 5.88%, respectively. Table 3.9 summarizes the face recognition results on Extended YALE B using illumination subsets evaluation protocol. In light of these results, it is clear that many state-of-the-art approaches as well as the proposed MNTCDP based system manage to differentiate all classes perfectly (average accuracy equal to 100%) on Subsets 2 and 3. Furthermore, our system achieved the top accuracy of 99.81% on Subset 4 and ranked second on Subset 5 by achieving a score of 98.52% surpassed by [START_REF] Zhu | Illumination invariant single face image recognition under heterogeneous lighting condition[END_REF] (only with only 0.37%) which reached 98.89%. Note that the method presented in [START_REF] Zhu | Illumination invariant single face image recognition under heterogeneous lighting condition[END_REF] could not reach 100% on Subset 4 .

• AR Database: On AR dataset, the MNTCDP based system managed to outperform all reported state-of-the-art systems. It reached 99.28% average accuracy over 10 random permutations into train/test subdivisions using 20 images in the train set and 6 samples for testing. At the same time, the best result of the literature works is 99% accuracy achieved by [START_REF] Chen | Discriminative dictionary pair learning based on differentiable support vector function for visual recognition[END_REF] at the same configuration but considering only one split. At 20/6 train/test configuration and evaluating the performance over 10 splits, [START_REF] Wu | Multi-feature kernel discriminant dictionary learning for face recognition[END_REF] reached 98.97% average accuracy. Hence, the proposed hand-crafted MNTCDP based system demonstrated its effectiveness and stability according to facial images with sunglass and scarf occlusions. The UTBM takes part of the HSR Developers Community with an HSR named Eighty-Eight, worth a value of 60,000 ethrough the partnership with Toyota. To promote the autonomy of the elderly or disabled through home assistance, Toyota has joined forces with research institutes to create the HSR Developers Community -a cooperation that will accelerate the development and commissioning of the HSR robot. Thanks to its light, compact, and very maneuverable cylindrical body with a folding arm, it can grab objects on the floor or placed them on shelves and perform various tasks. In addition to its local control, the robot can be remotely controlled by family or friends, the operator's face and voice being relayed in real-time to offer fundamental human interactions while helping with daily tasks. Since its first presentation in 2012, the HSR has received several improvements based on feedback from patients and caregivers.

One of the objectives of this thesis, which we discussed in the first chapter, is embedding the developed facial analysis frameworks within end devices such as HSR. Indeed, we programed Eighty-Eight to host the MNTCDP-based face recognition system so it will be capable of recognizing our lab members. To do so, we collected our lab database of 10 subjects with 50 images each to use as references. The 50 samples are covering different backgrounds and lighting conditions. The collection was made using the HSR integrated wide camera, and the framework was developed in a python3 environment. Figure 3.17 • The poses considered by our framework are up to 90 • (full profile), which makes our system more robust and generic. Our system detects the pose angle of the input image then selects the appropriate GAN generator according to the estimated angle. Afterward, the generated frontal image is fed to the face recognition network based on ResNet architecture. We adopted two GANs denoted as HP-GAN and FP-GAN to deal with the poses around 45 • (half-profile) and the ones close to 90 • (full-profile), respectively. This makes the frontalization more fast and efficient since each GAN will be trained on fewer images but correlated ones. Also, only one generator is loaded each time depending on the estimated angle, which keeps the same computational cost of a framework with a generator for all the poses. We adopted two paired GANs (one for each pose) to maintain the person related features Each GAN is selected to serve the frontalization of a given input image within its range.

To do so, a pose angle detector is estimates the angle, we used the Dlib [START_REF] King | Dlib-ml: A machine learning toolkit[END_REF] package that proved to be highly fast and accurate. The input images with pose angle less than of produced frontal image detected as fake by the discriminator, which maximizes its performance to differentiate real from fake.

G * = arg min G max D ([log(D(Im F )) + log(1 -D(G(Im P )))]) + λ||Im F -G(Im P )|| 1 (4.1)
λ is a regularization parameter assigned to the L1 loss included in the objective function G * to help the generator produce less blurry images. The L1 loss is used to update the generator parameters through the adopted optimizer. In our work, we investigated four generator networks and two discriminator networks widely used in the state-of-the-art.

Paired Input Target

Fake / Real blocks can be adjusted according to the application needs. However, [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] reported that 6 and 9 blocks provide good generation. Also, a high number of blocks risks the gradient vanishing in addition to high computational cost.

∇ D 1 m m i=1 [log(D(Im F i )) + log(1 -D(G(Im P i )))] such that D(Im F i ) =

4.4.1/ DATABASE

The performance of deep architecture highly depends on the dataset used for training.

However, the proposed PIFR datasets are very few, and most of them are no longer available such as the MultiPIE dataset, which is the widely used one. Therefore, we were faced with finding another alternative to train and evaluation our PIFR framework. We proposed to combine 4 state-of-the-art databases so we can build a consistent and challenging benchmark referred to as the Combined-PIFR database. We adopted ColorFERET, FEI, RaFD, and KDEF databases, which provide frontal, Half-Profile, and Full-Profile samples.

In the following, we present the properties of each database.

• ColorFERET database [START_REF] Phillips | The feret evaluation methodology for face-recognition algorithms[END_REF]: The Facial Recognition Technology (FERET)

database is known as the first benchmark used to evaluate face recognition frameworks. Regarding the resources and materials devoted to design this database, the creators managed to simulate illumination, head pose, and aging challenges.

The images were collected in 15 sessions between August 1993 and July 1996, accumulating a total of 14,126 images representing 1199 individuals.

• FEI Face database [START_REF] Thomaz | A new ranking method for principal components analysis and its application to face image analysis[END_REF]: The Brazilian FEI face database has been collected be- • Mean Squared Error (MSE): As 2D-image is a digital signal, we can assess the quality based on representing the average of the square errors between an image and its target (frontalized image and frontal ground-truth).

MS E = 1 m n c c k=1 m i=1 n j=1 ||G(Im P (i, j, k)) -Im F (i, j, k)|| 2 (4.2)
Where m, n, and c serve as the number of rows, columns, and channels, respectively. G(Im P (i, j, k)) is the pixel value of the frontalized profile input that is compared to the ground-truth one Im F (i, j, k).

• Structure Similarity (SSIM): This measure is a perceptual metric that quantifies image quality degradation caused by processing such as data compression or by losses in data reproduction such as our PIFR proposed solution. SSIM incorporates important structural information (luminance and contrast), meaning that the nearby pixels have strong inter-dependencies and carry information about the structure of the objects in the visual scene. Luminance tends to be less visible in bright regions, while contrast becomes less visible where there is significant activity in the image.

SSIM ranges from 0 to 1, the higher the better.

S S I M = (2µ x µ y + c 1 )(2σ x,y + c 2 ) (µ 2 x + µ 2 y + c 1 )(σ 2 x + σ 2 y + c 2 ) (4.3)
Where {µ x ,σ x }, {µ y ,σ y } are the {mean, variance} values of the frontalized profile input G(Im P ) and ground-truth Im F , respectively while σ x,y is the covariance. c 1 and c 2 are two constants serving to stabilize the fraction as c 1 = (0.01 × L) 2 and c 2 = (0.03 × L) 2 , where L is the range of pixels that typically takes 255. From Figure 4.9, which illustrates the frontalization quality of the 8 evaluated GAN combinations, it appears that U-Net 256 served more clear generation compared to ResNet-9B

and ResNet-6B, however, their MSE and SSIM metrics were close. Moreover, the Patch-GAN discriminator struggled and could not help each of the generators to produce a complete shape of the face. Therefore, the U-Net 256 with Pixel discriminator managed to be the best performing combination. did not exceed 50% on 1000 probe images. In addition, there still room to improve the frontalization solutions as our proposed PIFR framework reached around 83%. We point out that the handcrafted face recognition framework based on the MNTCDP descriptor did not deliver good results on the frontalized images by the GAN. This fact is due to the convolutional generation of the GAN as the pixels variations are less compared to the original image, which can be stated through the blur effect. On the other hand, LBP-like methods rely the high variability of the pixels within small neighborhoods to compute relevant features.

4.4.4/ IMPLEMENTATION AND EXECUTION TIME ANALYSIS

The execution time is one of the key performance indicators considered for evaluating face recognition frameworks as it should respect the real-time constraint. The training of the frontalziation and classification s ub-systems i s p erformed o ffline, so on ly th e execution time to identify one probe image is enough to assess the efficiency of our proposed PIFR framework. Therefore, we run an experiment to calculate the elapsed time for each stage:

Angle detection, frontalization, and classification. T he r ecorded t imes a re i llustrated in Table 4.3. Note that the angle detection time is not included in this table since it is common to all the possible combinations of frontalization and classification architectures proposed and discussed in this paper. Dlib takes around 0.48 ms to estimate the pose angle.

Moreover, the discriminator networks are not used in the evaluation stage, and only the generators are used. Therefore, we have only four architectures for the frontalization stage along with the four face classifiers. I t c an b e i nferred t hat t he c omputation time of the top-performing configuration, w hich i s t he U -Net 2 56 g enerator w ith ResNet101 classifier, took only 9.72 ms to predict the identity of the probe i mage. Hence, by adding did not exceed 50% on 1000 probe images. In addition, there still room to improve the frontalization solutions as our proposed PIFR framework reached around 83%. We point out that the handcrafted face recognition framework based on the MNTCDP descriptor (see Chapter 3) did not deliver good results on the frontalized images by the GAN. This fact is due to the convolutional generation of the GAN as the pixels variations are less compared to the original image, which can be stated through the blur effect.

On the other hand, LBP-like methods rely the high variability of the pixels within small neighborhoods to compute relevant features.

4.4.4/ IMPLEMENTATION AND EXECUTION TIME ANALYSIS

The execution time is one of the key performance indicators considered for evaluating face recognition frameworks as it should respect the real-time constraint. The training of the frontalziation and classification sub-systems is performed offline, so only the execution time to identify one probe image is enough to assess the efficiency of our proposed PIFR framework. Therefore, we run an experiment to calculate the elapsed time for each stage:

Angle detection, frontalization, and classification. The recorded times are illustrated in Table 4.3. Note that the angle detection time is not included in this table since it is common to all the possible combinations of frontalization and classification architectures proposed and discussed in this paper. Dlib takes around 0.48 ms to estimate the pose angle.

Moreover, the discriminator networks are not used in the evaluation stage, and only the generators are used. Therefore, we have only four architectures for the frontalization stage along with the four face classifiers. It can be inferred that the computation time of the top-performing configuration, which is the U-Net 256 generator with ResNet101 classifier, took only 9.72 ms to predict the identity of the probe image. Hence, by adding keeping a real-time prediction in less than 10ms. On the other hand, the Combined-PIFR database proved to be challenging and presents big room for improvement.

FACIAL EXPRESSION RECOGNITION

5.1/ INTRODUCTION

This chapter presents our contributions to the facial expression recognition task. Two popular approaches have been proposed in the literature for decoding facial expressions.

The first is geometric-based feature extraction. This approach relies on encoding geometric information such as the position, distance, and angle on the facial landmark points that a landmark detector should first identify and then extracts the feature vectors. The second approach is the appearance-based technique, which characterizes the appearance textural information resulting from the emotion classes' facial movements. Therefore, a set of features is extracted and is expected to contain relevant discriminating information to classify the different classes. The appearance-based approach utilizes many techniques for feature extraction. Moreover, the automatic FER task is further categorized into static and dynamic approaches depending on the input configuration. Static FER relies on using only one image to detect the dominant emotions. On the other hand, a dynamic framework requires many observations (samples) of the same person representing the evolution of the facial expression. We contributed to both cases by two independent frameworks. We proposed a hybrid framework for static person-independent FER that combines geometric and appearance concepts by extracting the textural and shape features from facial landmarks. The proposed combination is expected to promote an enhanced performance for person-independent FER because we consider geometric and appearance information that carries sufficient relevant features to describe the emotional classes. The dynamic person-independent FER is based on a Deep LSTM-CNN network to compute discriminant filter responses and encode the temporal information through the LSTM gates. Our network includes a dedicated residual sub-network "R-SubNet" to each input image for extracting the features. An LSTM block joins the R-SubNets and predicts the dominant emotion on the input images. The evaluation of both contributions is performed on widely used benchmarks with the leave-one-subject-out (LOSO) protocol, which is adopted to ensure the person-independent constraint. The achieved results are compared to much state-of-the-art work to prove the superiority and improvement of our proposed frameworks.

5.2/ STATIC PERSON-INDEPENDENT FER

We propose an entirely new framework for person-independent FER based on combining textural and shape features from 49 detected landmarks in an input facial image. The geometric representation is obtained by interpolating 49 keypoints (landmarks) detected in the input image generating a binary patch, which is exploited to compute the shape features using the HOG method. By contrast, the appearance description is also extracted based on the detected landmarks instead of the whole face image, making our proposed FER framework able to fulfill the person-independent constraint. The appearance features are extracted from 32 pixel ×32 pixel sub-images centered on each landmark using a brand new handcrafted descriptor, referred to as orthogonal and parallel-based directions-generic query map binary patterns (OPD-GQMBP). The OPD-GQMBP handcrafted descriptor is based on orthogonality and parallelism geometries for selecting the most prominent neighbors. It adopts an n × n neighborhood region to extract four feature maps based on four defined thresholding structures for each central pixel. The four feature maps are then decoded into one histogram. Afterward, the 49 feature vectors are concatenated to form the final appearance feature. During the classification step, we use the SVC library preprocessed by the PCA technique to reduce the dimensionality of the feature vectors. To highlight the contributions of our study and describe the workflow of our system in detail, we first describe the new textural handcrafted descriptor referred to as OPD-GQMBP. We then present how it is combined with the HOG shape descriptor to build the overall workflow.

5.2.1/ OPD-GQMBP: NEW HANDCARFTED DESCRIPTOR FOR FER

As discussed in the earlier chapter, the LBP operator is extremely flexible, and many of its aspects can be employed to develop enhanced descriptors for specific tasks. In our case, we propose a new LBP variant, referred to as OPD-GQMBP, which is based on new neighborhood topologies leading to four discriminant feature maps and adopts the LBP original kernel function that outputs low computational codes. The motivation behind the OPD-GQMBP descriptor relies on selecting orthogonal and parallel neighboring pixels that are believed to present the most information within a sub-block. In mathematics, orthogonality is defined as the generalization of the perpendicularity notion adopted by [START_REF] Barkan | Fast high dimensional vector multiplication face recognition[END_REF], who proposed a reduced LBP version referred to as OC-LBP, which considers two sets of four pixels located on the orthogonal lines. Thus, it produces only a feature Because all pixels within the n × n neighborhood are identified, we can proceed to the thresholding process. In this step, we generate for each sampling group S G k t its feature map k t , and obtain four feature maps:

S G 2 Ort (I c ) = L Ort 2,1 (I c ) = {(x, x)/x T } L Ort 2,2 (I c ) = {(x, -x)/x T } (5.2) S G 1 Par (I c ) = L Par 1,1 (I c ) = {(x, 1)/x T } L Par 1,2 (I c ) = {(x, -1)/x T, } (5.3
(I c ) =              1 Ort (I c ) = Γ(S G 1 Ort (I c )) 2 Ort (I c ) = Γ(S G 2 Ort (I c )) 1
Par (I c ) = Γ(S G 1 Par (I c )) 2
Par (I c ) = Γ(S G 2 Par (I c ))              (5.6) with Γ(S G k t )(I c ) = ∆(L t k,1 (I c ), L t k,2 (I c )) (5.7)
Where ∆ is the Heaviside function, which was initially used in the LBP operator defined in Eq 5.8, and applied the two lines of the same group to the threshold element by element.

Thus, the length of the generated binary code for each feature map is the size (n) of the neighborhood, and the number of possible produced patterns is 2 n . Thus, by concatenating the patterns produced by all feature maps, we generate 4 × 2 n possible patterns. After encoding each pixel in the input image and obtaining the four feature maps, we transform them into a histogram vector as the final descriptor for the image, as defined in

∆(x, y) =          1 , x ≥ y 0 , x < y (5.8) H(F) = H 1 Ort , H 2 Ort , H 1 Par , H 2 Par (5.9)
where

H k t (p) = χ∈F δ( k t (χ), p) (5.10)
In Eq 5.10, p ∈ [0, 2 n -1] is a pattern used to compare to the patterns k t (χ), χ is the gray-scale value of the computed feature image F, and the delta function δ(•), which is defined as follows (see. Eq. 5.11):

δ(a, b) = 1, if a = b; 0, otherwise (5.11)
To include more spatial information into the OPD-GQMBP descriptor, the feature image is spatially divided into w× w small non-overlapping blocks B i . All corresponding histograms H(B i ) extracted from all blocks are concatenated to form the final holistic image representation through Eq. 5.12.

H = w 2 i=1 H(B i ) (5.12)
where H is the final descriptor, is the concatenation operation, and H(B i ) is the histogram of the OPD-GQMBP descriptor computed on the i th block. Note that each elementary histogram H(B i ) has a length of 4 × 2 n , whereas the dimensionality of H is

4 × 2 n × w 2 .

5.2.2/ OVERALL FER FRAMEWORK

After defining the neighborhood topology and thresholding kernel of the OPD-GQMBP descriptor, we now present the overall view of our proposed system for the FER task.

The idea behind this framework is to combine the shape and appearance information to provide a more accurate FER, whereas most of the state-of-the-art proposed FER systems rely only on one piece of information, either geometric or appearance-based. To do so, we computed the textural and shape features based on the location of 49 detected key points (landmarks) on the input face image. The shape representation is obtained by interpolating the 49 landmarks to form curves to be further analyzed using the HOG operator, whereas the appearance representation is based on texture analysis by applying the proposed OPD-GQMBP operator on specific sub-images of the input image. To make the FER system more able to fulfill the person-independent constraint, the appearance features are extracted from sub-images with a pixel resolution of 32 × 32 and centered on each landmark carrying sufficient and relevant information about the expressed emotion and less irrelevant information of the person's face. Figure 5.4 illustrates the overall pipeline of the proposed FER system. First, the input image is fed to the dlib landmarks extractor to locate the 49 points (green color). These locations are then interpolated to generate a binary patch of the expressed emotion, upon which the HOG operator is applied to compute the shape feature vector. Meanwhile, the OPD-GQMBP descriptor (or state-of-the-art descriptors for comparison) was used to extract the textural features from each 32 × 32 sized sub-image centered on one landmark leading to a set of 49 histograms (49 landmarks) that are further concatenated together to construct the appearance feature vector. Note that spatial division was adopted to compute the OPD-GQMBP feature vector by dividing each sub-image into non-overlapping blocks of size w × w, as illus- This comprehensive analysis aims to compare the performance of the proposed OPD-GQMBP descriptor to the ones recorded in the literature of feature extraction methods, including handcrafted and deep-based approaches. We evaluated the top 10 deep learning models proposed so far in the state-of-the-art. These models were initially trained on the FER2013 database, and each of them reached a validation accuracy above 60% on 25000 images, which can be considered very significant. Then, we use transfer learning to extract the features of the five databases adopted in this experiment. We respected the same evaluation protocol (LOSO) to provide a fair and systematic analysis. Table 5.2 lists the performance reached by each method or model. We provide two metrics, the first one is the average of the accuracies recorded for all the runs of each database depending on the number of individuals, where JAFFE has 10 runs, CK+ 106, KDEF 70, OuluCasia 80, and 67 runs for the RaFD database. The second metric is the maximum accuracy reached overall the runs per database. We highlight with the green color the top 3 average values.

It can be seen from the average accuracies that the proposed OPD-GQMBP descriptor managed to score the top performance on all tested datasets. On the CK+ database, the OPD-GQMBP descriptor achieved 96.48% on 106 runs with a maximum of 100% recognition accuracy, keeping more than a 2% gap to the following top method, which is the handcrafted MNTCDP descriptor that managed to secure an average accuracy of The deep learning networks did not perform well on the five benchmarks despite reaching a validation accuracy above 60% on 25,000 images of the FER2013 database. The problem is that the deep learning methods should be fine-tuned on each dataset before extracting the features to get satisfying results. The subject application for this contribution is person-independent, and to ensure that the probe images of a given person are unseen by the framework, we should perform fine-tuning on each run and for each deep method. Hence, since we have 333 persons on the five datasets and considered 10 deep learning models, we need 3330 fine-tunes to expect satisfying results from these models, which is time and resources consuming. Moreover, such frameworks are intended for real implementations and deployments, and fine-tuning is not always a possible option;

besides, the probe images will have generally different characteristics than the train ones.

5.2.3.4/ COMPARISON AGAINST STATE-OF-THE-ART FER SYSTEMS

Now, we compare the results obtained by our proposed FER framework to those achieved by previous works in the field of facial expression recognition. Tables 5.3, 5.4, 5.5, 5.6 and 5.7 list the highest accuracies on the five datasets reported in well-indexed journals and conferences of the literature. We tried to collect the maximum of the works that followed the same adopted evaluation protocol (person independent). [START_REF] Bashar | Robust facial expression recognition based on median ternary pattern (mtp)[END_REF] Handcrafted 67.14 LPQ+SLPM+NN [START_REF] Turan | Histogram-based local descriptors for facial expression recognition (fer): A comprehensive study[END_REF] Handcrafted 67.61 EDR-PCANet [START_REF] Sun | Automatically query active features based on pixel-level for facial expression recognition[END_REF] Deep 69.4 C-classLDA-NN [START_REF] Kyperountas | Salient feature and reliable classifier selection for facial expression classification[END_REF] Deep 74.73 LBP based LDA [START_REF] Shan | Facial expression recognition based on local binary patterns: A comprehensive study[END_REF] Handcrafted 73.4 BDBN [START_REF] Liu | Facial expression recognition via a boosted deep belief network[END_REF] Deep 68 CFER [START_REF] Sun | Cognitive facial expression recognition with constrained dimensionality reduction[END_REF] Handcrafted 76.46 Features fusion [START_REF] Poursaberi | Gauss-laguerre wavelet textural feature fusion with geometrical information for facial expression identification[END_REF] Handcrafted 70 OuluCasia, the misclassification errors were very high. The Happy emotion is the only one to be highly recognized with an accuracy of 96.3%, followed by Surprise with 91.3%.

The rest accuracies were between 66.3% and 80%. Also, Angry and Neutral dramatically influenced the other emotions with 34.5% and 36.6%, respectively. On the RaFD database, all the rates were high with the perfect recognition for three classes (Contempt, Fear, and Sad) in addition to Angry and Disgust that were misclassified only once. However, there is a mutual confusion between Happy and Surprise since 3 Happy samples were identified as Surprise and 4 times in the opposite direction (Surprise to Happy).

Overall the databases, we disclose that Happy and Neutral emotions are the most recognized ones. In addition to that, Neutral is very perturbing the framework as it presents high false-negative rates on many benchmarks. The computational cost is one of the key performance indicators considered in such machine learning applications. Therefore, we run an experiment that calculates the elapsed time to predict the label of a given input image that has a resolution of 762 by 562 pixels, highlighting the execution time of each step of the proposed framework:

• Dlib landmarks detection.

• Shape feature extraction based on HOG descriptor.

• Appearance feature extraction using the proposed OPD-GQMBP as well as the state-of-the-art handcrafted methods denoted as "getFeatures".

• PCA dimensionality reduction.

• Label prediction by the SVM library.

We excluded the deep-learning methods from this evaluation since they require GPU units to perform the feature extraction. Thus, it will be unfair to compare CPU-based methods with the GPU ones regarding the computation power of GPUs. 15.9 15.9 15.9 15.9 15.9 15.9 15.9 15.9 15.9 15.9 15.9 15.9 15.9 The LSTM network is composed of special memory units in the recurrent hidden layer.

These memory blocks contain neurons with self-connections to save the current cells state in addition to special activation units called gates to control the flow of inputs to consider. Each memory unit is controlled by input and output gates, where the input manages the input activations connected to the memory cell input and the output gate controls the flow of cell activations for the rest of the LSTM network. Later, a forget gate was added to the memory block to prevent them from processing continuous input streams that are not segmented into subsequences, which was a collapsing mode of traditional LSTM. The forget gate scales the current state of the cell before adding it as input to the cell through the self-recurrent connection of the cell, therefore adaptively forgetting or resetting the cell's memory. Figure 5.16 shows the internal architecture of an LSTM unit with forget gate.

Based on Figure 5.16, the kernel function linking the input and output of an LSTM memory block can be expressed as follows:

f t = σ W f h h t-1 + W f x x t + b f i t = σ (W ih h t-1 + W ix x t + b i ) ct = tanh (W ch h t-1 + W cx x t + b c) , c t = f t • c t-1 + i t • ct , o t = σ (W oh h t-1 + W ox x t + b o ) , h t = o t • tanh (c t ) .
(5.13) In our contribution, we used Deep-LSTM architecture that is simply stacking multiple LSTM layers. Moreover, we defined the LSTM to also serve features compression by a factor of 2 that helps to smoothly predict the emotion by the final fully connected layer.

5.3.0.2/ DEEP CNN-LSTM FOR DYNAMIC FER

In this thesis, we proposed an End-to-End deep architecture for dynamic personindependent FER from four sequence samples, where the fourth represents the peak of the facial expression. As shown in Figure 5.17, the proposed architecture incorporates different techniques divided into three main steps.

The preprocessing step of our proposed framework is to keep only the visual features related to the facial expression and ignore those describing the person's identity, which will help fulfill the person-independent constraint. The Dlib package, as used in the previous contribution, detects with high accuracy 49 landmarks on the human face. These less person-related features. We developed a method to make the CNN encoders focus mainly on these regions by building the attention map of each input image. The attention map is achieved by inserting the landmarks on the input image then computing an Edge Glow filter that suppresses the textural information. Therefore, the CNN dedicated streams are extracting only the facial expression filters. We adopted dedicated CNN streams for the feature extraction step to guarantee an efficient extraction of the deep features from each input attention map. Moreover, dedicated streams offer flexible training by adapting the weights to each input rather than finding one suitable configuration for all of them. Also, the extraction is performed in parallel and not in sequence. Sequence computing makes the network unable to track the patterns between the inputs and will be influenced by the last input fed. Each stream outputs N-filter 3×3 bank that is averaged and pooled into an N sized vector. The four vectors are concatenated to form the input of the classification sub-network, which is the third step of our network. Our network architecture is flexible to all models of deep CNN that can be used for extracting the features.

We will present a comprehensive analysis of the CNN models presented earlier (Section 2.11). The third step classifies the extracted features through deep LSTM cells and fully connected layers as their configurations are mentioned in Figure5.17. The flatten filters from the dedicated CNN are firstly scaled by a first fully connected layer FC1 that has the exact size of the pooled feature vector. The need for feature adaption comes from the LSTM cell, which will process the outputs of FC1 as the values should be normalized, representing a homogeneous sequence. Also, FC1 is motivating the non-linearity, hence discovering more patterns. The main corps of our architecture is the Deep-LSTM, which is learning the temporal information across the four concatenated deep features.

An LSTM gate processes its corresponding bin from the feature vector, then its output is fed to the next one. In addition, the first LSTM cell takes 256 activations as input performs feature reduction with a factor of two, leading to 128 activations that are further processed with a second LSTM cell. The final output computed from the Deep-LSTM is 64 activation neurons that represent the temporal patterns. The activations compression makes the prediction more efficient and avoids gradient loss when the number of classes is low, like in FER that has six or seven classes. Therefore, we used two LSTM cells to compress the activations of FC1 efficiently in addition to temporal processing. Moreover, Deep-LSTMs are more stable than one LSTM layer and increase the temporal correlation between the input activations. The final layer of our network is fully connected for predicting the emotion corresponding to the four input time samples. FC2 incorporates Nclasses neurons with softmax classification layer. The 64 activations computed by Deep-LSTMs are discriminant enough to detect the correct emotion, which will be proved through an in-depth experimental analysis.

5.3.1/ EXPERIMENTAL ANALYSIS

This subsection is devoted to evaluating our proposed CNN-LSTM for dynamic personindependent facial expressions recognition. We adopted three benchmarks from the state-the-art that include sequence-based posed facial expressions. We set up a hard LOSO evaluation protocol by selecting one sequence of four samples per emotion, and we divided the individuals into ten folds, which guarantees the person-independent constraint. The conducted experiment considers different CNN deep models as feature extractors to find the ultimate configuration for our framework on the three benchmarks. This subsection presents the setup of the three benchmarks, the recorded results of the deep CNN, and comparative analysis against state-of-the-art works.

5.3.1.1/ DYNAMIC FER DATASETS

We adopted CK+, OuluCasia, and MMI benchmarks for evaluating our developed CNN-LSTM network. These datasets are sequence-based facial expressions. Hence, we present in the following the configuration adopted for each dataset.

• MMI: The MMI database includes a total of 213 sequences from 32 subjects labeled with six basic expressions (excluding "contempt"), where 205 sequences are recorded in frontal view. The sequence starts with a neutral expression and reaches the peak in the middle before returning to the neutral. In addition, MMI presents challenging conditions, i.e., there is large interpersonal variation as subjects do not perform the same expression uniformly, and many of them wear accessories (e.g., glasses, mustache). For experiments, the first frame (neutral face) and the three peak frames in each frontal sequence are usually selected to perform personindependent 10-fold cross-validation.

• CK+ and OuluCasia: The two benchmarks were already introduced in the earlier contribution related to static person-independent FER. We selected the first sample representing the neutral and the last three frames of the peak expression for the dynamic use case. The LOSO 10 folds cross-validation protocol is adopted similar to the MMI benchmark by dividing the subjects into 10 folds.

For the three benchmarks, the 10 folds do not share the same amount of images (subjects); also, we selected only one instance per subject for hard training of our network. It can be inferred from the recorded results that the achieved overall performance on the three datasets is very satisfying as the lowest average accuracy is above 81%. This fact proves that the proposed CNN-LSTM architecture is robust to person-independent constraints and grants sustainable recognition performance across the three benchmarks.

Moreover, it also highlights that the architecture is generic and does not rely on any specific CNN feature encoder for the achieved minimum performance. Therefore, it is concluded that this performance is mainly thanks to the architecture that we proposed.

On the CK+ dataset, the best-performing configuration is ResNet18, as it reaches 99.41% average accuracy, while the 50 layers ResNet version comes second with 98.72% and the third best-performing model is Inception.v3. The worst-performing model is Inception.v2 with 96.19%, and the rest of the CNN models achieved close results averaged around 97%. According to the max accuracy, all the models reached a 100% rate on at least one fold. The CK+ dataset particularity relies on the uniformity of expressed emotion across all the individuals justifying the high average accuracies. On the other hand, the use of dynamic four samples as inputs improved the performance compared to the static case from 97.53% to 99.41%.

On the OuluCasia dataset, ResNet18 and 50 models managed to be the two top- The MMI benchmarks recorded rates were higher than the OuluCasia ones. The ResNet models successfully ran to be ranked as the three top-performing models and were the only ones that reached above 93% average accuracy. The fourth-ranked model is Inception.v3 with 92.75% average accuracy, followed by the AlexNet model scored with 92.60%. InceptionResnet version 1 and 2 achieved below 90% as the lowest average accuracies (90.75%) on the MMI benchmark, while the rest of the models performed competitively, and their rates ranged are around 91% and 92%. In terms of maximum accuracy, only the ResNet and AlexNet architectures reached 96.3%. This fact proves that the MMI benchmark is challenging due to the large interpersonal variation as the facial expressions for the same emotion varies from a subject to another.

The ResNet models demonstrated outstanding performance compared to the rest of evaluated models. Indeed, the ResNet models are efficient feature extractors thanks to their CK+ as the top state-of-the-art average accuracy, concerning 10 folds cross-validation, is 91.46% reached by ExpNet [START_REF] Chang | Expnet: Landmark-free, deep, 3d facial expressions[END_REF]. Our ResNet18-LSTM managed to outperform it and scored 93.46% with a clear improvement of 2%. The rest literature works were stuck and could not bypass 82% average accuracy. The OuluCasia records highlight a similar performance of many methods, including ours. We underline that only our proposed model, in addition to the PPDN [START_REF] Zhao | Peak-piloted deep network for facial expression recognition[END_REF] and its deeper version [START_REF] Yu | Deeper cascaded peak-piloted network for weak expression recognition[END_REF] architecture, managed to guarantee an average accuracy above 86%. The Deeper Cascaded Peak-piloted Network [START_REF] Yu | Deeper cascaded peak-piloted network for weak expression recognition[END_REF] slightly outperforms PPDN with 0.02%.

The Peak-Piloted Deep Network (PPDN) [START_REF] Zhao | Peak-piloted deep network for facial expression recognition[END_REF] is one of the competitive works to our approach on the CK+ and OuluCasia benchmarks. The PPDN is based on a specialpurpose back-propagation procedure referred to as peak gradient suppression (PGS) for network training. It considers two inputs representing the peak and non-peak samples encoded throw one CNN, and then the extracted features are fed to different classification layers to compare the cross-entropy of the peak and non-peak. ExpNet [START_REF] Chang | Expnet: Landmark-free, deep, 3d facial expressions[END_REF], which is the most competitive model on the MMI dataset, is a 3D approach for FER relying on computing 29D vectors characterizing the facial expression. The classification is done using the nearest neighbor rule with k = 5.

5.4/ CONCLUSION

In this chapter, we presented our contributions related to person-independent facial expressions recognition covering both static and dynamic scenarios. We proposed a frame- On the other hand, we proposed two systems for facial expression recognition covering both static and dynamic-based analysis. We considered the person-independent evaluation constraint, which is the most challenging scenario for FER and motivates the generic implementations within end devices. In the static analysis, only one sample is considered to output the emotion label. We built a system based on combining textural and shape features extracted from 49 emotion-related landmarks detected by the Dlib. To

6.2/ FUTURE WORKS AND PERSPECTIVES

We are working on the first perspective related to performing more ablation studies on the dynamic person-independent FER. We will run experiments to investigate the effect of the number of input samples and other attention maps that can enhance more the personindependent performance. Also, we will extend the adopted benchmarks by considering the wild ones collected from the internet. The wild datasets present more challenges to FER frameworks as the expressions are not uniform between the individuals, and the background also presents massive changes from a scene to another. Moreover, the processing time should be analyzed to find the optimal configuration leading to the best performance and response time. Finally, we started writing a paper about this contribution to send it to a high-impact journal for publication.

The accuracy recorded by the developed FER system combined with the proposed OPD-GQMBP descriptor was the highest compared to those reported by the existing stateof-the-art FER systems that adopted the same protocol (person-independent LOSO).

Although our system indeed managed to outperform many state-of-the-art systems, it needs improvement on databases containing Asian individuals (e.g., JAFFE and Oulu CASIA) because they tend to present similar facial features, which confuse the classifier in the case of the LOSO protocol. We believe that the ultimate solution is to reduce the number of extracted appearance features and focus on the binary patch calculated based on the detected landmarks. This patch should incorporate more information, not only the landmark location, and handcraft a shape descriptor to obtain the most prominent information. Moreover, this proposal will help develop generic person-independent FER systems because the input images will be coded into a standard patch. Even though our framework, along with the proposed descriptor, performs efficiently in computational time, we think there is room for improvement using other dimension reduction strategies and/or landmark selections. We are also investigating an enhancement of the performance of our framework by utilizing other sophisticated classifiers than an SVM and combining learnable features with the proposed OPD-GQMBP descriptor. In addition, we intend to extend the set of the studied emotion classes with the compound classes, reaching 20 different classes. We also considered creating a mixed database from existing databases to gather all challenges in a unique benchmark, offering more challenging testing and evaluation.

Similar to implementing the MNTCDP-based face recognition system within the Human Support Robot, we also intend to embed the static facial expression framework on the HSR. Therefore, the robot will recognize the person and its emotional state through two independent frameworks. To increase the FER performance, we will change the setup of the training database, making it person-dependent since the environment and the subjects interacting with the robot will not change. Hence, only one reference database is enough for the two tasks that also will optimize the computational resources. Collecting this database is the only step to perform, we imagine a standard recording protocol similar to the state-of-the-art benchmarks simulating various lighting conditions and backgrounds.

Moreover, we intend to improve the PIFR performance on the Combined-PIFR database by enhancing the quality of the frontalized profile images. This could be achieved through new GAN architectures and/or adversarial loss functions by including perceptual and qualitative losses. Furthermore, we aim to apply image translation based on GAN to generate 3D models conditioned by the profile image so the classification can be more accurate. Furthermore, we plan to generalize the frontalization process to be generic, and then it can be evaluated on cross databases and real-life probe images. Also, the classification sub-system could be addressed to get the maximum from the frontalized images. We can rely on an ablation study to conduct an efficient training of the CNNs.

In addition, the frontalization technique can be included in other facial applications suffering from pose variations, such as facial expression recognition, gender classification, and age estimation. This technique reduces the computational complexity by building an efficient system dealing with the standard view (frontal) and translate any other view before processing it.

• "New framework for person-independent facial expression recognition combining textural and shape analysis through new feature extraction approach". Kas, Mo-

Figure 1 . 1 :

 11 Figure 1.1: Examples of face-like objects

Figure 1 . 2 :

 12 Figure 1.2: Lower and Upper parts missalignement effect on face percetion

Figure 1 . 3 :

 13 Figure 1.3: Left: fMRI image of the FFA, Right: ERPs responses of the FFA toFace/Object[START_REF] Rossion | Erp evidence for the speed of face categorization in the human brain: Disentangling the contribution of low-level visual cues from face perception[END_REF]

1. 3 /

 3 photo or video by comparing their face with those stored in a database. This technology relies on capturing the visual features, that are converted into data (pixels) and processed to calculate a low dimensionality feature space. The obtained space is compared and matched with the reference database to find the most similar identity to the detected face. Typically, traditional facial recognition frameworks analyze around 80 facial feature regions referred to as nodal points. These features include geometric measurements as the distance between the eyes and the length of the nose. These visual and geometric characteristics differ from one person to another, helping facial recognition be accurate in identifying technology. The most recent new technologies are based on the texture, shape, and color analysis of the skin that is unique to each individual, leading to more precise results. In the early '90s, facial recognition began to gain popularity when the US Department of Defense searched for a technology that could detect people. In early 2001, facial recognition made an impression when it was first used in a public space during Super Bowl XXXV in Tampa. The authorities then used it to detect possible criminals and terrorists among the spectators. Systems were then deployed to other at-risk areas of the United States to monitor potential criminal activities. Currently, face recognition is regarded as one of the top three biometric technologies for identifying a person and the fastest-growing biometric technology. Its market could reach a value of 7.7 billion dollars by 2022. This technology is gradually used for surveillance and security purposes, especially by governments and authorities who incorporate it into video surveillance systems. Indeed, facial recognition has been used in France, the United Kingdom, and the United States. For example, Nice's city is experimenting with facial recognition for surveillance purposes as part of its Carnival, while the US government is using it at airports to identify individuals whose visas have expired. Companies in various industries are also more and more exploiting facial recognition, such as health, marketing, or tourism. It is also found on many services and products intended for the general public. For example, since the iPhone X exhibition in 2017, Apple smartphones feature Face ID technology allowing users to unlock them by showing their faces to the front camera. A 3D scanner compares more than 30,000 characteristics to verify user identity accurately. Face ID also allows validating purchases with Apple Pay. For its part, Facebook is developing Deep-

F a c i a l E x p r e s s i o n s R e c o g n i t i o n 2 Figure 1 . 4 :

 214 Figure 1.4: Facial Expression Recognition difficulty levels

  Chapter 2 reviews the typical facial analysis configuration. It discusses in-depth the key steps composing such a framework, namely feature extraction and classification. This chapter also highlights some existing works from the literature devoted to face recognition and facial expression analysis. • Chapter 3 presents our proposed framework for face recognition based on a new local features descriptor referred to as Mixed Neighborhood Topology Cross Decoded Patterns (MNTCDP). It explains the overall architecture and the considered benchmark for a comprehensive evaluation. It also discusses a real implementation of the proposed framework within a Human Support Robot of Toyota.• Chapter 4 is devoted to introducing a brand new solution to deal with the recognition of profile images. It is based on Generative Adversarial Network-based image translation (GANs). We trained a GAN to generate a profile input's frontal face and then processed it with existing frontal recognition systems. Also, Chapter 4 includes a comprehensive explanation of the concept of GAN for image translation purposes.• Chapter 5 exhibits our contributions to the facial expression recognition task that cover both static and dynamic-based scenarios. The static-based FER framework relies on extracting textural and shape features from specific face landmarks that carry enough information to detect the dominant emotion through the SVM classifier. On the other hand, dynamic FER contribution incorporates Long Term Short Memory (LSTM) deep network to encode the temporal information efficiently with a guiding attention map to focus on the emotion-related landmarks and guarantee the person-independent constraint. This chapter includes comprehensive comparisons with the state-of-the-art works.2 FACIAL ANALYSIS LITERATURE 2.1/ INTRODUCTIONFacial analysis frameworks are based on a standard image classification pipeline, aiming to develop a system capable of automatically assigning a label to an image. Such frameworks rely on machine learning techniques to develop a computational capacity emulating human skills, and their life cycle includes two phases. The first phase is to train the computer to construct a model representing the relationship between the input space, which is the image pixels, and the decision to be made represented as labels or classes. The second evaluates the framework by calculating performance indicators such as precision, accuracy, recall, and computation complexity on unseen inputs that simulate a real use case of the developed framework. If the reported metrics satisfy and fulfill the requirements, we proceed to implement the considered image classification system for general use.

Figure 2 . 1 :

 21 Figure 2.1: Generic architecture of an image-based facial analysis framework

Figure 2 . 2 :

 22 Figure 2.2: CNN-based architecture for image facial analysis

2. 3

 3 .1/ HOLISTIC DESCRIPTION Holistic face recognition uses global information of faces to perform face recognition. The global information of faces is essentially represented by a small number of features derived directly from face images' pixel information. This small number of features captures the variance between different individual faces and identifies individuals uniquely. Holistic methods use the entire face as input. In these methods, each face image is represented

Figure 2 . 3 :

 23 Figure 2.3: Example of extracting global features using wavelet transformations

Figure 2 . 4 :

 24 Figure 2.4: The LBP pixel transformation process.

•

  DenseNet: Densely connected convolutional networks were inspired by the ResNet topology. They incorporate dense residual blocks composed of batch normalization, ReLU activation, and a 3 × 3 convolution. The ResNet models use the sum function as a skip connection, whereas DenseNet integrates the concatenation. Therefore, each input layer receives all outputs of the earlier versions. The concatenation process generates an output with a large number of channels, which makes DenseNet models computationally heavy. • Inception: Google proposed its own deep learning inspired by LeNet, referred to as Inception, stacking more convolutional layers deeper to achieve a better performance, which comes at the cost of heavy computations and a complex design. The philosophy of inception relies on concatenating the responses of different convolution filters at the same layer, forming the input of the next layer. Moreover, they used a 1 × 1 convolution filter as a feature reduction technique before jumping to the next layer. Google introduced four versions of the Inception architecture, Inception.v1 known as GoogLeNet with 27 layers, Inception.v2, Inception.v3, and Inception.v4 tackling batch normalization, factorization, and grid size control problems, respectively. Google proposed two versions of a residual network inspired by the performance of ResNet, known as InceptionResNet.v1 and InceptionResNet.v2 based on creating the skip connections on the previous Inception models. Inception-ResNet.v1 and Inception-ResNet.v2 networks have the same computational cost of Inception.v3 and Inception.v4, respectively. To employ deep learning architectures for deep feature extraction in solving the facial analysis problems, we follow the basic procedure shown in Figure 2.11. Initially, the model was trained end-to-end on a big dataset, mainly the LFW database. Afterwards, the model is expected to achieve a good training performance using the validation set.

2. 4 . 1 /Figure 2 . 8 :

 4128 Figure 2.8: Deep-feature based transfer learning approach for face and facial expression recognition.

Figure 2 . 9 :

 29 Figure 2.9: Example of Nearest Neighbor matching with K = 1 and K = 3

Figure 2 . 10 :

 210 Figure 2.10: Feature vectors representation both in the input feature spaces.

Figure 2 . 11 :

 211 Figure 2.11: Neural Networks-based classification architecture.

[ 99 ]

 99 proposed a computationally efficient hybrid face recognition system that relies on both local and holistic description. The Local Gabor Binary Pattern Histogram Sequence (LGBPHS) method is employed to realize the feature extraction on the whole facial image. After that, the PCA technique is used for dimensionality reduction. The experimental evaluations on Extended Yale Face Database B demonstrated an improved recognition rate as compared to the basic PCA and Gabor wavelet techniques under illumination variations. [100] proposed a novel hybrid technique for face representation and recognition, which exploits both local and subspace features.

35 •

 35 [START_REF] Shin | Baseline cnn structure analysis for facial expression recognition[END_REF] used a convolutional neural network model to achieve facial expression recognition. They adopted and aligned cropped faces from FER-2013, SFEW2.0, CK+, KDEF, and Jaffe with respect to the landmark position of the eyes. The training data were augmented 10 times by flipping them. Five types of data input (raw, histogram equalization, isotropic smoothing, diffusion-based normalization, and difference of Gaussian) were tested. They then selected the one that showed the highest accuracy as a target structure for fine-parameter tuning. Finally, the CNN network with histogram equalization images was chosen as the baseline CNN model for further research. Yu et al.[START_REF] Yu | Image based static facial expression recognition with multiple deep network learning[END_REF] proposed a method that contains a face detection module based on an ensemble of three state-of-the-art face detectors, JDA, DCNN, and MoT. Subsequently, a classification module composed of an ensemble of deep convolutional neural networks (CNNs) was adopted based on averaging the output responses. Each CNN model is initialized randomly and pretrained on the Facial Expression Recognition (FER) Challenge 2013 database. The pretrained models were then fine-tuned on the training set of SFEW 2.0. To combine multiple CNN models, they presented two schemes for learning the ensemble weights of the network responses: minimizing the log-likelihood loss and minimizing the hinge losses. According to the results reported in their study, the hinge loss performs slightly better than the log-like and single CNN models on the validation and test sets of the FER2013 and SEFW databases. Therefore, their framework is computationally heavy, and the outcomes are not very promising. Jung et al.[START_REF] Jung | Joint fine-tuning in deep neural networks for facial expression recognition[END_REF] proposed a new CNN framework based on combining the temporal appearance and temporal ge-ometry extracted from two CNN models. The faces in the input image sequences are detected, cropped, and rescaled to a pixel resolution of 64 × 64, and 49 landmark points are then extracted using the IntraFace algorithm. Finally, these two models are combined using an element-wise sum of the outputs of the last fully connected layers from the two temporal CNN models. Through several experiments conducted on the CK+, MMI, and Oulu-CASIA databases as well as numerous data from various data augmentation techniques, the framework built showed that the two models cooperate with each other.However, the joint model did not improve the recognition of all of the facial expressions and achieved the same performance as the temporal appearance and temporal geometry models of the Disgusted, Fear, Happy, and Surprised classes. In addition, the temporal appearance CNN model outperformed the geometry model on all tested databases. Most of the previous methods have considered the entire facial region as the input information, and have paid less attention to the sub-regions of human faces, which may lead to a large difference between the extracted and expected representations. Indeed, when the extracted information obtained from the entire face image is irrelevant, the final recognition result will be affected.2.7/ CONCLUSIONThis chapter was dedicated to present a comprehensive background on image-based facial analysis tasks. We presented the generic configuration of the classification framework, then presenting an overview of the different feature extraction and classification techniques. We focused on local-based face description since it is the topic of this thesis. We highlighted in-depth the concept of the Local Binary Patterns method and its flexibility to develop a wide variety of descriptors intended for different applications. Moreover, we discussed the state-of-the-art of face facial expression recognition underlining the limitations of existing methods. LOCAL DESCRIPTION-BASED FACE RECOGNITION 3.1/ INTRODUCTION This chapter is dedicated to introducing our contribution related to developing a new local descriptor while keeping the effectiveness and simplicity of the traditional LBP and addressing its weakness. We propose a conceptually simple, high-quality, and yet robust framework of LBP, referred to as Mixed Neighborhood Topology Cross Decoded Patterns (MNTCDP), for face recognition. The proposed MNTCDP descriptor is a ten bits code assigned to each sub-region of size 5×5, which aims at achieving both simplicity and efficiency at the same time. It computes and describes the relationship between the referenced pixel and its neighbors on a 5 × 5 pixel block by encoding gray-level difference based on two-level radius (R = 1 & R = 2) and multi-direction angles: 0 • , 45 • , 90 • and 135 •. The idea behind is to combine the radius with the angle to extract a more detailed and discriminating description. To make the thresholding process more accurate, each pixel is compared to the average gray level of its 3×3 neighbor pixels within its 5×5 neighborhood encompassing. Unlike most of the existing hand-crafted descriptors, which encode the pixels considering simple neighborhood topology and pattern encoding, the MNTCDP descriptor proposes an advanced encoding way exploiting multi-radial and multi-orientation information simultaneously. This particularity gives the ability to the proposed descriptor to extract more relevant information than the existing descriptors, as shown later. The extracted feature vector is then fed to the simplest K-Nearest Neighbor classifier configured with City Block distance measure for classification. MNTCDP has the following outstanding advantages: As shown further, it allows considerably enhancing the discriminative power of LBP variants and their robustness to small variations (due to image noise) and has low computational complexity. At the feature extraction stage, there is no pre-learning process and no additional parameters to be learned. The main contributions can be summarized and briefed in the following points: We propose a novel LBP-like descriptor referred to as Mixed Neighborhood Topology Cross Decoded Patterns (MNTCDP), which combines two topological dimensions to describe a given pixel keeping a low computational complexity and simple conception. MNTCDP extracts the local features from 5×5 neighboring pixels that fulfill the lack of information required for a local descriptor.

[ 0 •Figure 3 . 1 :

 031 Figure 3.1: Local sampling topology of MNTCDP.

ures 3 .

 3 1 and 3.2, are considered in order to calculate the average local gray level of the whole 3×3 neighborhood around each pixel in level A, which will be used in the pattern encoding phase. Based on the above neighborhood topology, eight average local gray levels labeled as {M A0 ,. . . . . . M A7 }, are computed as follows (cf. Eqs 3.3 and 3.4).

Figure 3 . 2 :

 32 Figure 3.2: The two groups of sampling adopted in MNTCDP descriptor.

Ec 1 and

 1 Ec 2 based, in addition to the central pixel, on the two groups of samples S G 1 and S G 2 and the eight average local gray levels {M A0 ,. . . . . . M A7 } defined previously. The two encoders Ec 1 and Ec 2 adopt the basic thresholding function where each pixel, except the central pixel, is compared to the average local gray level of the 3 × 3 image patch to which the pixel belongs to. The central pixel is compared locally to the average local gray level M Ec 1 (cf. Eq. 3.7) of the even pixels A 2i;i∈[0-3] and the average local gray level M Ec 2 (cf. Eq. 3.8) of the odd pixels A 2i+1;i∈[0-3] of level A, and incorporated in the modeling of the two encoders Ec 1 and Ec 2 , respectively. The codes produced by the two encoders Ec 1

3. 2 . 3 /Figure 3 . 3 :

 2333 Figure 3.3: The overall framework of the MNTCDP feature vector calculation.

Figure 3 . 4 :

 34 Figure 3.4: Comparing the obtained feature histograms of two images of the same person using MNTCDP, LBP, LTP, nLBPd and dLBPα descriptors.

  ) 3.3/ FACE RECOGNITION SYSTEM USING MNTCDP Similar to most state-of-the-art face recognition systems, the proposed system, as shown in Figure 3.5, involves several steps. First, the images of each dataset are preliminarily divided into ten random splits generated for each number of training images. The number l of training images is varied from 1 to N pc -1 (N pc is the number of images per class) and the rest (N pcl) is taken as the testing set. Each split contains two sub-sets, one for the training and the other for testing. The images of the training and testing sets are fed to the next stage without submitting them to any kind of preprocessing technique.Secondly, the feature images are obtained using the proposed MNTCDP operator. After that, each feature image is further divided into w × w non-overlapping sub-image blocks, and a histogram of patterns is generated for each block. The histogram bins of all blocks are chained to form the final MNTCDP descriptor of the whole image. Finally, the images of the test set are classified through a supervised image classification task. Our focus was on evaluating the discrimination power of the proposed MNTCDP descriptor, and therefore we tried to make as few assumptions as possible and chose the simple nonparametric nearest-neighbor rule (1-NN) to compute the minimum L1 distance between the probe image and the gallery images (cf. Eq. 3.15).

Figure 3 . 5 :

 35 Figure 3.5: Overview of the proposed face recognition system.

3. 4

 4 Figures 3.6, 3.7, 3.8, 3.9 and 3.10 illustrate images of subjects from ORL, YALE, Ex-

  Figures 3.6, 3.7, 3.8, 3.9 and 3.10 illustrate images of subjects from ORL, YALE, Extended YALE B, FERET and AR databases, respectively, used in our experiments. The main characteristics of each database are described in the following subsections.

3. 4

 4 .1.1/ ORL The ORL 1 Database of Face [111] is composed of 400 images, covering 40 individuals with ten images per person. The size of each image is 92 × 112.

  Figure 3.6 illustrates ten images of one person in the ORL database. This database is characterized by a homogeneous dark background, slightly varying lighting conditions, and various facial expressions. The faces are in an upright position in frontal view, with a slight left-right rotation.

Figure 3 . 6 :

 36 Figure 3.6: Images of subject from ORL database.

Figure 3 . 7 :

 37 Figure 3.7: Images of subject from YALE database.

Figure 3 . 8 :

 38 Figure 3.8: Images of subject from Extended YALE B database.

Figure 3 . 9 :

 39 Figure 3.9: Images of subject from the used subset of FERET database.

Figure 3 .

 3 Figure 3.10 shows the samples of one class from the used subset.

Figure 3 . 10 :

 310 Figure 3.10: Images of subject from the used subset of AR database.

  composed of images with illumination changes. In this dataset, unlike the previous datasets where the evaluated descriptors requested only nine blocks to achieve their highest performance, empirical results show that dividing the encoded image into 100 non-overlapped blocks represents the best adequate block number results in high recognition accuracy. Indeed, histogram feature calculation on 100 blocks permits the feature to precisely describe the feature face images of the YALE database by identifying the difference between the face and the background and overcoming low contrast issues. The stability of the descriptor is again evaluated under all possible numbers of training images. • Experiments #4: This experiment was designed to test the robustness of the MNTCDP descriptor on a dataset with an important number of images and massive lighting and illumination changes, which are offered by the Extended Yale B database. This experiment aims to investigate the performance according to the number of training images we recorded, for each tested descriptor, the accuracies over ten random splits and under five different train/test configurations (i.e., Train= 5/ Test= 59 images, 10/54 images, 20/44 images, 30/34 images, and 32/32 images).

  evaluation protocol based on various training images. The classification accuracy is recorded over 10 random subdivisions, under 5 Train/Test division configurations (ie. Train= 5/ Test= 21 images, 10/16 images, 13/13 images, 15/11 images and 20/6 images). The histogram feature computation is performed on 324 non-overlapping blocks. • Implementation and Execution: The face recognition experiments have been performed on an HP ProDesk with Core i7 Processor 4.0 GHz with turbo boost technology and 16GB of RAM, running with Ubuntu 16.04 LTS (Xenial Xerus) operating system. The descriptors and the recognition system have been implemented in the MATLAB® R2016b environment.

3. 4 . 3 /

 43 EXPERIMENTAL EVALUATION AGAINST LBP-LIKE DESCRIPTORS Tables 3.1, 3.2, 3.3, 3.4 and 3.6 report the average accuracies (i.e., over 10 random splits) achieved by the proposed MNTCDP descriptor and the top 50 evaluated state-of-the-art descriptors on ORL (Exp#1), FERET (Exp#2), YALE (Exp#3), Extended Yale B (Exp#4)

  .1 reports the average accuracy of each evaluated descriptor. The results in this Tableclearlydepict that MNTCDP is the top 1 under all numbers of training images, as it reaches the highest accuracies compared to the tested state-of-the-art descriptors. It is to underline that the proposed MNTCDP descriptor achieves a score of 98.64% at 3/7 configuration (training/testing sets), outperforming the rest of the descriptors. Note that realizing this recognition rate over ten random splits with few images in the train set (3 images only) is a promising result for real-time applications requiring minimum computational resources.The most important thing that can be noticed is the 100% average accuracy recorded over the half/half (Train/Test) configuration and over the ten random splits. Achieving a 100% recognition rate over ten subdivisions with only five images in the train set demonstrates both the stability and the powerful description of MNTCDP. The AECLBP-S-MxC and NI-CI-LBP descriptors came in second place and reached an encouraging result in the ORL database where they realized 100% average accuracy at 8/2 configuration while other tested descriptors recorded 100% average accuracy only for the 9/1 setup. Note that many descriptors like XCS-LBP, WLD, and RDLBP suffer from the drop and decrease in their average recognition rates when the number of training images increases. Figure3.11 illustrates the performance evolution of the top 5 descriptors on the ORL dataset. It can be found from Figure3.11 that the performance of the MNTCDP descriptor increased rapidly to reach 100% average accuracy compared to all the evaluated state-of-the-art descriptors. The second remarkable statement, which can be made from this Figure, is the performance gap between the MNTCDP descriptor and its competitors, mainly when the test set contains more images than in the train one.

Figure 3 . 11 :

 311 Figure 3.11: Performance evolution of top 5 descriptors on ORL dataset.

Figure 3 . 12 :

 312 Figure 3.12: Performance evolution of the top 5 descriptors on FERET dataset.

3. 4 . 3 . 4 /Figure 3 . 13 :

 434313 Figure 3.13: Performance evolution of the top 5 descriptors on YALE dataset.

Figure 3 . 14 :

 314 Figure 3.14: Performance evolution of top 5 descriptors on EYB dataset.

Figure 3 . 15 :

 315 Figure 3.15: Performance evolution of top 5 descriptors on AR dataset.

Figure 3 . 16 :

 316 Figure 3.16: Performance evaluation of the tested deep learning methods.

Figure 3 . 17 :

 317 Figure 3.17: Camera system of Toyota HSR.

Figure 3 . 18 :

 318 Figure 3.18: Images of subject from the collected database .

69

 69

4. 3

 3 .1/ OVERALL FRAMEWORK

20 •Algorithm 1

 201 are considered as frontal images since they present all the visual features revealing the person identity, angles between 20 • and 60 • are processed as half-profile while the remaining (from 60 • to 90 • ) are considered as full-profile. Once the input image is frontalized, we feed it to the face classifier based on ResNet architecture that is trained only on frontal samples. The overall pipeline is illustrated in Figure 4.2. As can be seen, the system contains frontalization and classification blocks. The first one includes the Dlib based pose estimation and the two generators (HP-GAN and FP-GAN), while the second block is charged to identify the label (identity) of the person based on the generated image. The frontalization and classification steps are summarized in Algorithm 1. The process of the proposed PIFR framework Input: Input Image Im Output: Subject ID Functions: Dlib: estimate the pose angle; Gen: trained Generators HP and FP; Classifier: trained classifier read the input image; calculate the pose angle α = Dlib(Im); if α ≤ 20 • then the image is detected as frontal classify the input image without frontalization ID =Classifier(Im) if 20 • < α ≤ 60 • then the image is detected as half-profile half-profile frontalization Im =Gen HP (Im) classification of the frontalized image ID =Classifier(Im ) if 60 • < α ≤ 90 • then the image is detected as full-profile full-profile frontalization Im =Gen FP (Im) classification of the frontalized image ID =Classifier(Im )

Figure 4 . 3 :

 43 Figure 4.3: Paired GAN used for frontalization of profile image

  true and D(G(Im P i )) = f ake update the discriminator parameters D=optim(D) calculate the descending L1 loss gradient of the generator: ∇ G λ m m i=1 ||Im F i -G(Im P i )|| 1 update the generator parameters G=optim(G) supported size of the images controlled by the amount of GPU memory and most of the works adopt 256 and 128 resolutions.

Figure 4 .

 4 4 illustrates the architecture of U-Net256 and U-Net128 generators as they share the same u-shape with different filter sizes and numbers. On the other hand, ResNet-based generator adopts residual blocks to compute relevant features from the input image, exploring the distinguishing power of the original ResNet[START_REF] He | Deep residual learning for image recognition[END_REF]. Hence, the input image is downsampled n times, where n is generally set to 2 or 3, then the resulted from convoluted feature maps are further processed by residual sub-network referred to as ResNet-Block. Afterward, the output of the ResNet-Blocks is upsampled to reach the specified size of the generator output.

Figure 4 .

 4 5 shows the architecture of ResNet-based generator with 6 and 9 blocks. The number of intermediate

Figure 4 . 4 :Figure 4 . 5 :

 4445 Figure 4.4: The architectures of U-Net generators as reported in[START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]

Figure 4 . 6 :

 46 Figure 4.6: The layers used to build the discriminator

Figure 4 .

 4 Figure 4.7 shows the difference between ResNet and DenseNet in terms of skip connection, where the DenseNet network concatenates the outputs of the convolution layers all together before proceeding to the classification layer. This fact could lead to a heavy amount of data to deal with and hence more computational cost. On the other hand, ResNet sums these outputs that may reduce the feature extraction power since many sum combinations can have close values.

Figure 4 . 7 :

 47 Figure 4.7: Comparison of ResNet and DenseNet residual connections

tween•

  June 2005 and March 2006 at the Artificial Intelligence Laboratory of FEI in Brazil. It includes a set of 2800 images covering 200 individuals, each represented over 14 images. All images, which have a common white homogeneous background, are colorful and have a resolution of 640 × 480 with a variable scale of 10%. The main challenge of this database is the profile rotation, which varies from 0 to 180 • . Moreover, this database offers age and gender variability by including students and staff at FEI lab with equal female/male partition, between 19 and 40 years old with a distinct appearance, hairstyle, and adorns. • KDEF Face database [179]: The Karolinska Directed Emotional Faces (KDEF) database contains 70 individuals, each is displaying 7 different emotional expressions. The changes in the facial landmarks can affect enough the appearance of the face, hence the person will not be correctly recognized, which challenges the face recognition systems and assesses their robustness to the appearance changes. Also, the database provides frontal, half-profile, and full-profile samples for all the subjects. RaFD [180]: The Radboud Faces Database (RaFD) is composed of 67 individuals, including Caucasian, Moroccan, Dutch adults, and Caucasian children, both boys and girls, and displaying 8 emotional expressions across multiple head poses (frontal to full profile). The challenge of this database relies on the variety of the subjects and their ages.

Figure 4 . 9 :

 49 Figure 4.9: Comparison of the produced frontal images by the evaluated GAN's architectures

histogram with a 2×2 3

 3 feature histogram. The OPD-GQMBP descriptor is generic and adjustable depending on the needs of the considered application. It adopts a n × n sub-block neighborhood, where n is an odd integer (3,5,7,9,...), to maintain symmetric neighborhoods. The concept behind this is the selection of prominent pixels within this neighborhood. Given a central pixel I c , as shown in Figure 5.1, we define four-pixel groups, each of which contains n × 2 pixels forming two lines. Two sampling groups are based on orthogonality {S G 1 Ort , S G 2 Ort }, whereas the two others are based on the concept of parallelism, i.e., {S G 1 Par , S G 2 Par }. Therefore, each sampling group S G is defined on two lines S G k t (I c ) = {L t k,1 (I c ), L t k,2 (I c )} where t stands for the type (Ort/Par) of the sampling group and k the group number (1/2).

Figure 5 .

 5 2 shows a Cartesian coordinate system centered on the central pixel I c to encode the position of each pixel considered in each sampling group within a 7 × 7 neighborhood. The sampling groups are defined as follows:

FFigure 5 . 1 :

 51 Figure 5.1: OPD-GQMBP neighborhood topologies.

) S G 2 ( 5

 25 Par (I c ) = L Par 2,1 (I c ) = {(1, y)/y T } L Par 2,2 (I c ) = {(-1, y)/y T }

Figure 5 . 2 : 2 ( 5 . 5 )

 52255 Figure 5.2: Cartesian system used to identify the pixel coordinates for each line and group.

Figure 5 . 3 :

 53 Figure 5.3: Texture feature extraction workflow based on the proposed OPD-GQMBP operator.

trated in Figure 5 . 3 .

 53 The number of spatial blocks that divide the sub-image depends on the considered dataset and is related to the camera resolution and image blur. Indeed, blurred images require fewer blocks than clear images, which present more details to be detected. At the end of the feature extraction stage, the HOG and OPD-GQMBP computed histogram vectors are concatenated to compose the final image descriptor that is further fed to a dimensionality reduction using the PCA method before proceeding to the 5.2.3.1/ EXPERIMENTAL DATASETS • The Japanese Female Facial Expression (JAFFE) dataset has 213 facial expression images, representing the seven basic emotions: Anger (30 images), Disgust (29 images), Fear (32 images), Happiness (31 images), Sadness (31 images), Surprise (30 images), and Neutral (30 images).

Figure 5 .

 5 5 illustrates some examples of the facial expressions. This database is very challenging regarding the particularity of Japanese females that have similar face features generating more inter-classes visual features.

Figure 5 . 5 :Figure 5 . 6 :Figure 5 . 7 :Figure 5 . 8 :

 55565758 Figure 5.5: Samples of two subjects from JAFFE database

Figures 5 .

 5 Figures 5.10, 5.11, 5.12, 5.13 and 5.14 illustrate the confusion charts generated from the results of our proposed FER framework for CK+, JAFFE, KDEF, OuluCasia and RaFD, respectively.

Figure 5 . 10 :Figure 5 . 11 :

 510511 Figure 5.10: Confusion matrix of the seven emotions of CK+

Figure 5 . 12 :

 512 Figure 5.12: Confusion matrix of the seven emotions of KDEF

(

  2048 patterns) descriptor was performed in 83.5 ms. Overall, we can disclose that the proposed framework, along with the OPD-GQMBP method, managed to predict the label of an image with a resolution of 762 by 562 pixels in less than 150 ms, allowing to process 7 frames per second, which is considered real-time feedback according to the specifications of person-independent FER systems[218; 219].

Figure 5 . 15 :

 515 Figure 5.15: Comparison of the overall execution time elapsed to predict the label of an image

Figure 5 . 16 :

 516 Figure 5.16: Internal configuration of the LSTM Cell

Figure 5 .Figure 5 . 18 :

 5518 Figure 5.18 shows an example of three individuals performing the same facial expression from the three benchmarks.

6 CONCLUSION 6 . 1

 661 work for static person-independent FER based on a new textural features extractor referred to as referred to as orthogonal and parallel-based directions-generic query map binary patterns (OPD-GQMBP). OPD-GQMBP is applied on 49 patchs of 32 × 32 pixel size and the computed 32 vectors are concatenated to form the textural feature. The latter is further combined with shape-based one computed by the HOG method on a binary map representing the landmarks of the emotion's regions of interest. Hence, these landmarks helped our framework to fulfill the person-independent constraints. The proposed architecture was evaluated on 5 widely used benchmarks with respect the LOSO evaluation protocol to guarantee the person-independent use case. The evaluation demonstrated the outstanding performance of our framework against state-of-the-art deep and handcrafted methods. For dynamic person-independent FER, we proposed a deep CNN-LSTM network that considers 4 temporal samples leading to the peak of the emotion. We inserted 49 emotion-related landmarks to highlight the region of interest, then computing the egde filter to remove person-related visual features. Then, we fed each input sample to a dedicated CNN encoder to compute deep features. The 4 calculated features are concatenanted and forwarded to an LSTM block of two cells encoding the tempo-ral patterns. The achieved 10 folds cross-validation metrics proved the superiority of the proposed CNN-LSTM network against the state-of-the-art, especially when used with ResNet18 model as features extractor. / THESIS SUMMARY This thesis was devoted to developing new and efficient handcrafted and deep-learningbased frameworks for face and facial expression recognition through image analysis covering all the steps of an image-based classification pipeline, starting with preprocessing, feature extraction, and classification. It has focused mainly on enhancing the recognition performance under delicate situations and strict evaluation protocols. For face recognition, we treated the severe lighting condition changes between the reference and probe samples. Also, face occlusion situations like wearing scarves and sunglasses have been addressed. On the other hand, we considered facial expression recognition with respect to the person-independent evaluation protocol with fewer training samples. The proposed architectures within the context of this thesis fulfilled important constraints related to performance stability, real-time inference, compatibility with end devices such as robots, and flexibility to other implementations and applications.Through the first two chapters, we discussed the challenges related to face and emotion image-based analysis and how computer vision and machine learning managed to push this analysis forward similar to human perception. Also, we presented the typical configuration adopted to develop such frameworks and the methods used for feature extraction step and classification one. Moreover, we pointed out that the state-of-the-art still lacks efficient systems that reply to the mentioned constraints. Our main contributions, as presented in this thesis, covered the proposal of a new face descriptor referred to as Mixed Neighborhood Topology Cross Decoded Patterns (MNTCDP); a Pose Invariant Face Recognition (PIFR) system exploring the strength of Generative Adversarial Networks (GAN) in image translation; and two person-independent facial expression recognition systems to cover static and dynamic use cases with emotion-related landmarks attention maps. The proposed MNTCDP-based face recognition framework incorporates the Nearest-Neighbor classification rule with the city-block distance, which kept a low computational complexity and high matching power. The MNTCDP face descriptor incorporates a new neighborhood topology of 5 × 5 block size and selects only the prominent pixels instead of all the 25 ones. The selected pixels are further encoded through a discriminant kernel function that compares the gray-level values to well-defined local means. The conducted experiments on five benchmarks proved an outstanding and stable performance as compared to handcrafted and deep-based methods and models. The five benchmarks were chosen to cover all the discussed face recognition challenging conditions. The results highlighted the robustness of the MNTCDP descriptor to lighting changes, especially on the Extended Yale B database and face occlusions simulated over the AR face database. The MNTCDP contribution was further validated by an actual implementation within the UTBM Human Support Robot (Toyota HSR 88), allowing this robot to recognize the members of our laboratory according to a reference base that we collected containing few samples per individual.The second contribution presented in this manuscript handled the challenge of Pose Invariant Face Recognition (PIFR), which is the most complex problem that a face classification framework can face. We presented how the state-of-the-art is dealing with it where the frameworks are supposed to include profile samples of the individuals so the probe ones could be recognized, which is hard to be satisfied in real-world implementations. What we proposed is based on image translation through Generative Adversarial Networks (GAN). Our PIFR framework takes the profile probe input and generates its frontal one based on the GAN person-independent learning, then classifies it through residual-CNN module trained only on frontal images. The experiment we conducted on a benchmark that we created from the literature ones, by assuring no person-overlapping between the train and test sets, proved the improvement bought by the GAN frontalization technique. The experiment highlighted an improvement of nearly 40% and concluded that the ResNet101 for classification and the U-Net generator offer the highest frontalization quality, keeping an inference time less than 10ms for the overall PIFR framework.

  efficiently encode the textural information, we proposed a new local descriptor referred to as Orthogonal and Parallel-based Directions-Generic Query map Binary Patterns (OPD-GQMBP). It supports user-specified neighborhood size(3,5,7,9..) and considers the pixels in linear distributions that captures more discriminant features. The textural features extracted from local regions of 32 × 32 pixels centered on the 49 landmarks are combined with shape-based one computed using HOG descriptor. We interpolated the locations of the 49 landmarks forming a binary map describing the dominant emotion. All the features are fed to the PCA dimensionality reduction and SVM classifier predicting the facial expression. The best kernel size configuration for OPD-GQMBP as facial expression descriptor is seven, leading to the highest recognition rates on five widely used benchmarks for person-independent FER. The recorded results proved the superiority of the proposed FER framework, as it outperformed the handcrafted descriptors and deep models of the literature and reached state-of-the-art accuracy on the five benchmarks. As a second contribution in facial expression recognition, we proposed a deep model for dynamic-based FER that considers four temporal samples finishing with the peak of the expression. We applied a landmark-based edge mask on each sample to promote the person-independent analysis and reduce the person-related visual features. Our deep model includes four CNN dedicated streams to extract the features from the four samples; then, it concatenates the four features before feeding them to an LSTM block that manages to encode the temporal patterns across the four samples. The recorded results on three state-of-the-art benchmarks for dynamic facial expression recognition proved that the ResNet18-LSTM configuration outperformed the evaluated CNN configurations and surpassed the accuracies reported in the literature.

  13 those related to the framework itself in terms of the overall recognition framework, including pre-processing steps, extraction of the image feature, and the decision/classification algorithm. These two last issues, i.e., feature extraction and classifier designation, constitute the two critical sub-problems in facial analysis. Many literature approaches have achieved good face recognition performance thanks to the extraction of discriminant features, which can be considered as the critical stage in the overall recognition framework.

	Indeed, extracting non-discriminating features that may ignore important details and de-
	scriptions severely affects recognition performance even if we adopt a sophisticated clas-
	sifier. Consequently, most facial and image classification research gives more interest
	to extracting the characteristics, giving birth to various face descriptors. Face recogni-
	tion systems typically rely on three categories of feature extractors: global (or holistic
	methods), local descriptors, and deep features.

Table 2 .

 2 1: summary of texture descriptors tested. The computation time of this deep learning method and the performance dependence to the pre-training database remain the major drawbacks of this architecture. The architecture of this deep feature method is illustrated in Figure 2.6.

	Category	Complete name	Abbreviation	Application	Year Ref
	Combining with complementary features Local Extreme Complete Trio Pattern	LECTP	Image retrieval	[50]
		Rotation-invariant features based on directional coding	DC	Texture classification	[51]
		Statistical binary patterns (2)	SBP2	Texture classification	[45]
		Statistical binary patterns (3)	SBP3	Texture classification	[45]
		Statistical binary patterns (4)	SBP4	Texture classification	[45]
		Quad Binary Pattern	QBP	Target tracking	[41]
		Dominant Rotated Local Binary Patterns	DRLBP	Texture classification	[52]
		Adaptive Local Ternary Pattern	ALTP	Face recognition	[34]
		Center-Symmetric Cdaptive LTP	CSALTP	Face recognition	[34]
		Magnitude Maximum Edge Position Octal Pattern	MMEPOP	Image retrieval	[53]
		Sign Maximum Edge Position Octal Pattern	SMEPOP	Image retrieval	[53]
	Encoding and regrouping	Adjacent Evaluation Completed LBP (S)	AECLBP-S	Texture classification	[54]
		Adjacent Evaluation Completed LBP (M)	AECLBP-M	Texture classification	[54]
		Adjacent Evaluation Completed LBP (S-MxC)	AECLBP-S-MxC Texture classification	[54]
		Adjacent Evaluation LTP	AELTP	Texture classification	[54]
		Orthogonal Combination Of Local Ternary Patterns	OC-LTP	Infrared imagery recognition [55]
		Complete Robust Local Binary Pattern (M)	CRLBP-M	Texture classification	[56]
		Complete Robust Local Binary Pattern (S)	CRLBP-S	Texture classification	[56]
		Complete Robust Local Binary Pattern (S-MxC)	CRLBP-S-MxC	Texture classification	[56]
		Local Gray Code Pattern	LGCP	Face expression analysis	[57]
		Local Maximum Edge Binary Patterns	LMEBP	Image retrieval	[58]
		Center-Symmetric Local Ternary Pattern	CS-LTP	Feature description	[59]
		extended Center-Symmetric Local Ternary Patterns	eCS-LTP	Image retrieval	[60]
		Center-symmetric Local Binary Patterns	CS-LBP	Texture classification	[38]
		Gradient Texture Unit Coding	GTUC	Texture classification	[61]
		Attractive-and-Repulsive Center-Symmetric Local Binary Patterns ARCS-LBP	Texture classification	[62]
		Local Optimal Oriented Pattern	LOOP	Species recognition	[63]
		Repulsive-and-attractive local binary gradient contours	RALBGC	Texture classification	[43]
		Local concave-and-convex micro structure patterns	LCCMSP	Texture classification	[42]
		Local directional ternary pattern	LDTP	Texture classification	[44]
		Local neighborhood difference pattern	LNDP	Texture image retrieval	[47]
		Local quadruple pattern	LQPAT	Recognition and retrieval	[46]
		Extended Local Graph Structure	ELGS	Texture classification	[64]
		Diagonal Direction Binary Pattern	DDBP	Face recognition	[65]
		Linear Directional Binary Pattern	LDBP	Face recognition	[65]
		Directional Local Binary Patterns	dLBPα	Texture classification	[66]
		Difference Symmetric Local Graph Structure	DSLGS	Finger vein recognition	[67]
		eXtended Center-Symmetric Local Binary Pattern	XCS-LBP	Texture classification	[68]
	Neighborhood topology and sampling	Multi-Orientation Weighted Symmetric Local Graph Structure Local Binary Patterns by neighborhoods	MOW-SLGS nLBPd	Finger vein recognition Texture classification	[69] [66]
		Symmetric Local Graph Structure	SLGS	Finger vein recognition	[70]
		Local Extreme Sign Trio Pattern	LESTP	Image retrieval	[50]
		Local Directional Number Pattern	LDN	Face expression analysis	[40]
		Improved Binary Gradient Contours (1)	IBGC1	Texture classification	[71]
		Local Graph Structure	LGS	Face recognition	[72]
		Binary Gradient Contours (1)	BGC-1	Texture classification	[73]
		Binary Gradient Contours (2)	BGC-2	Texture classification	[73]
		Binary Gradient Contours (3)	BGC-3	Texture classification	[73]
		Improved Center-Symmetric Texture Spectrum	ICSTS	Texture classification	[74]
		Directional Binary Code	DBC	Face recognition	[75]
		Local Derivative Pattern	LDP	Face recognition	[37]
		Center-Symmetric Texture Spectrum	CSTS	Texture classification	[74]
		Simplified Texture Spectrum	STS	Texture classification	[76]
		Simplified Texture unit *	STU*	Texture classification	[77]
		Simplified Texture Unit +	STU+	Texture classification	[77]
		Local Quantization Code Histogram	LQCH	Texture classification	[78]
		Complete Robust Local Binary Pattern (C)	CRLBP-C	Texture classification	[56]
		Robust Local Binary Pattern	RLBP	Texture classification	[79]
		Local Binary Count	LBC	Texture classification	[80]
	Thresholding and quantization	Completed Local Binary Count Improved Local Ternary Pattern	CLBC ILTP	Texture classification Medical image analysis	[80] [81]
		Centralized Local Binary Pattern	CLBP	Face expression analysis	[39]
		Local Ternary Pattern	LTP	Face recognition	[36]
		Improved Local Binary Patterns	ILBP	Face detection	[82]
		Local Phase Quantization	LPQ	Texture classification	[83]
	Other Methods	Texture spectrum (∆)	TS(∆)	Texture classification	[84]
		Texture spectrum (0)	TS(0)	Texture classification	[85]
	2.3.3/ LEARNABLE FEATURES
	During the last years, computer vision field is experiencing the birth of deep learning
	and deep features based methods, which improve the performance of recognition and
	classification process in many applications including, among others, face recognition,
	medical image analysis, content-based image retrieval.

Table 2 .

 2 2: Examples of Distance Metrics for Classification

	Metric

Table 3

 3 LBP and CI-LBP among the top 5 ranked descriptors, thanks to their higher recognition rates recorded with fewer images in the train set. They recorded, when using the configuration 5/59, 92.39%, and 91.26%, respectively. It is clear that MLDCBBP is the descriptor that achieved the highest recognition rate average, CHAPTER 3. LOCAL DESCRIPTION-BASED FACE RECOGNITION almost all the evaluated descriptors achieved a 100% recognition rate. We can express the same remark for subset three, where many descriptors and the proposed MNTCDP descriptor manage to differentiate all classes perfectly (average accuracy equal to 100%). Note that MNTCDP succeeded to record the top 1 accuracy on Subsets 4 and 5, by reaching 99.81% and 98.52% recognition rate, respectively. ELGS is the second-best performing descriptor on Subset 4 with an accuracy of 98.67%. Meanwhile, the LPQ descriptor secured the second rank on Subset 5 by recording 98.07% accuracy. MNTCDP descriptor demonstrated, unlike all the evaluated state-of-the-art descriptors, consistent stability across the four subsets despite the challenging conditions of Subsets 4 and 5.

	conserving sustainable stability over Train/Test configurations.
	• Experiment #5 (Subset evaluation protocol): Extended YALE B database can be
	divided into five subsets from slight to extreme lighting variations. Subset 1 is con-
	sidered a training set; it contains images taken under nominal lighting conditions,

.4 reports the obtained classification results (average accuracies over ten random splits). The results in Table

3

.4 indicate that the proposed MNTCDP operator has obtained the maximum recognition rates for 20/44, 30/34, and 32/32 Train/Test configurations. MNTCDP obtains the maximum score of 98.91% at half/half configuration, followed by NI-LBP and DDBP, which can be considered as the second and third top-performing descriptors on the Extended YALE B dataset, where they reached 98.08% and 98.04% average accuracies, respectively, at half/half configuration. Figure

3

.14 illustrates the evolution of the top 5 descriptors' performance according to the Train/Test partitions. We considered NI-CIwhile the other subsets are used as test sets. Subsets 2 and 3 are characterized by slight-to-moderate illumination variations, while subsets 4 and 5 depict severe illumination changes. The obtained recognition rates of the four subsets are listed in Table

3

.5. As can be seen, we disclose that subset two is the easiest one where

Table 3

 3 satisfying rate on images with occlusions, outperforming all the other evaluated descriptors. The maximum average recognition rate of 99.28% is obtained again by the MNTCDP descriptor at 20/6 train/test configuration. LDN descriptor, which was not among the top five descriptors on the previous datasets, managed to be the second-best performing descriptor. It achieved an average accuracy of 99.1% by adopting 20 samples per individual in the train set. LQP descriptor, as on Yale dataset, reached the thirdhighest score of 99.03% at 20/6 configuration on this subset of AR database, followed by SLGS and DSLGS descriptors, which recorded 99.05% and 99.01% average accuracies at the same configuration, respectively. On the other side, WLD and LESTP handcrafted descriptors are the worst-performing methods in the experiment, and the first one attained a maximum average accuracy of 88.55% exploiting 20 samples (in the train) and testing 6 images, while the second one reached 88.9% average accuracy for the same number of testing images. Figure3.15 illustrates the evolution of recognition rates according to the number of training images. It's clear that MNTCDP realizes the highest accuracy at each Train/Test configuration, presenting sustainable stability of its performance.

	.6 lists the

ment #6. MNTCDP achieved an average accuracy of 82.33% over ten random splits using only 5 images in the train set, surpassing all the evaluated state-of-the-art handcrafted descriptors 4%. By enlarging the train set to 13 images per subject, which represents half of the subject set, the proposed descriptor attained 96.18% average accuracy, which is a very

  rate over ten splits against PCANet2 and CBFD methods. The proposed descriptor consistently increased the recognition rate from a train/test configuration to the next one and CBFD method, while PCANet2 deep learning architecture suffered from performance drop at 5/5 and 6/4 configurations. We mention that PCANet2 achieved 100% accuracy using four images in the train set but only over one split.• FERET Database (Exp#2): As we can see, the CBFD method surpassed MNTCDP and PCANet2 methods from 1/6 to 4/3 train/test configurations, but MNTCDP achieved the top average accuracy at the two last configurations. The maximum accuracy obtained at 6/1 configuration by all the methods is close: MNTCDP attained 99.7% over ten splits, PCANet2 method recorded the second top accuracy of 99.5% over one split, while CBFD gave the 3rd accuracy of 99% over one split.PCANet2 could not overcome the drawback of a performance drop, and it was again unstable as on ORL database.

	creased to 71.11% on Subset 5. CBFD method could not maintain its effectiveness
	on Subsets 4 and 5 as it recorded only 70.34% and 54.03%, respectively.
	• AR Database (Exp#6): On AR Database, the maximum accuracy of all the tested
	methods was arround 99%. MNTCDP reached 99.28% average accuracy over 10
	splits at 20/6 configuration. by PCANet2 method attained 99.08% one split accuracy
	at 13/13 configuration, while CBFD achieved 99% over one split at 15/11 train/test
	configuration. Deep learning methods demonstrated their effectiveness and perfor-
	mance at low numbers of training images, however their stability cannot be guar-
	anteed. Evaluating its performance over 10 random splits, MNTCDP was capable
	of reaching higher accuracies and providing performance stability from train/test
	configuration to another.
	ages (Exp#4), experienced some drops at some configurations, while the proposed
	MNTCDP showed stability. The maximum accuracy achieved by PCANet2 was lim-
	ited to 94.55% over one split at 30/34 configuration and CBFD method reached
	97.5% accuracy at 32/32 configuration, while MNTCDP managed to record 98.91
	over 10 random splits at 32/32 configuration. In Exp#5(based on dataset parti-
	tion into subsets according to the illumination conditions from slight to severe), as
	we can see, all the tested methods did successfully achieve 100% accuracy on
	Subset 2. The performance of PCANet2 and CBFD methods on Subset 3 de-

• ORL Database (Exp#1): MNTCDP achieved at each train/test configuration the highest recognition

• YALE Database (Exp#3): The effectiveness and stability of MNTCDP are again proved on YALE Database. It reached 100% average accuracy at 7/4 configuration and showed stability, while the PCANet2 deep learning method was not stable at all on this dataset, which suffered a significant performance drop of 83% (from 90.48% at 4/7 configuration to 07.78% at the following one). CBFD method presented promising performance, it achieved 99.12% using only one image in the train set and recorded 100% using 6 images in the train instead of 7 samples in the case of our proposed descriptor. However, CBFD method experienced minor performance drop several times and could not keep the 100% accuracy even over 1 split.

• Extended YALE B Database (Exp#4 and Exp#5): The performance of deep learning methods on Extended Yale B dataset according to the number of training imcreased slightly (below 100%) as they achieved 99.47% and 98%, respectively, while MNTCDP kept its performance at 100%. MNTCDP reached good accuracies on Subset 4 (99.81% accuarcy) and Subset 5 (98.52% accuarcy) outperforming the two other methods. PCANet2 achieved 92.21% on Subset 4 but its performance de-

Table 3 .

 3 1: Average recognition accuracy (%) over 10 splits on ORL dataset (Exp#1)

					ORL Database Train/Test
	Descriptor	1/9	2/8	3/7	4/6	5/5	6/4	7/3	8/2	9/1
	MNTCDP	85.89	95.59	98.64	99.67	100	100	100	100	100
	AECLBP-M	75.61	87.34	93.54	95.21	97.25	98.31	98.25	99	99.25
	AECLBP-S	77.06	89.72	94.39	97.42	98.4	99.44	98.83	99.5	99.5
	AECLBP-S-MxC	81.92	92.06	96.71	98.75	99.05	99.44	99.5	100	100
	AELTP	79.72	91.38	95.43	97.87	98.65	99.5	99.25	99.5	99.75
	ALTP	79.64	91.59	96.07	97.75	98.65	99.5	99.42	99.63	99.5
	BGC-1	79.44	91.5	95.89	98.17	99.05	99.25	99.25	99.63	99.75
	BGC-2	77.06	90.06	95.36	97.79	98.6	99	99.25	99.5	99.75
	BGC-3	76.42	89.34	94.25	97.21	98.65	99.13	98.92	99.63	99.75
	CI-LBP	39.69	54.69	60.43	65.33	68.45	73.31	74.58	75.25	74.75
	CLBC-M	65.58	79.84	86.21	88.88	92.55	94.94	94.83	95.75	95.5
	CRLBP-M	76.83	87.38	93.86	95.04	96.7	98.31	97.92	98.88	99
	CRLBP-S	79.06	91.22	95.32	97.67	98.95	98.94	98.83	99.63	99.75
	CRLBP-S-MxC	81.89	91.69	96.64	98.38	98.9	99.5	99.25	99.88	99.75
	CSALTP	71.94	86.69	91.57	95.79	97.65	98.69	98.67	99.63	99.75
	DBC	72.03	85.44	91.86	94.29	97	98.13	98.17	99.38	99.25
	DCLBP	80.47	91.44	96.79	98.04	98.7	99.31	99.33	99.63	99.75
	DDBP	80.39	90.53	95.29	97.83	98.85	99.19	99.17	99.5	99.75
	dLBPa	73.22	85.66	91.36	94.5	96.4	97.5	98	98.88	99.25
	DRLBP	65.94	80.16	87.07	90.92	92.75	95.5	95.58	96.75	97.5
	DSLGS	79.5	91.06	95.96	98.13	98.6	99.44	99.08	99.5	99
	ELGS	78.5	90.94	95.96	98.21	98.9	99.69	99.25	99.63	100
	IBGC-1	81.03	92.78	97.21	98.42	99.15	99.5	99.58	99.75	100
	ILTP	81.67	92.31	97.14	98.88	98.8	99.69	99.5	99.75	99.5
	LBP	77	89.69	95.11	97.04	98.1	99	99	99.5	99.25
	LCCMSP	74.58	86.25	93.93	96	97.8	98.88	99.17	99.63	99.75
	LDBP	80.86	92.22	96.61	98.42	99.45	99.69	99.75	99.75	100
	LDN	65.89	80.53	86.82	90.83	93.95	94.56	94.83	97.5	96
	LECTP	65.39	77.22	83.61	88.42	90.25	92.75	92.67	96.25	93.75
	LETRIST	80.81	91.25	95.21	97.96	98.2	99.25	99.67	99.75	99.5
	LESTP	71.33	84.16	90.14	92.58	94.3	96.56	96.83	97.88	97.75
	LGCP	68.97	83.78	89.93	94.04	96.15	98.19	98.08	99.25	99.5
	LGS	77.78	90.47	95.25	97.29	98.4	99.31	98.83	99.25	98.75
	LMEBP	60.22	72.97	81.61	84.58	87	90	89.83	93.13	93
	LNDP	72.53	85.56	91.64	95.58	96.9	98.06	99.17	99.25	99.5
	LQPAT	76.69	88.31	93.5	96.17	97.8	98.63	99.5	99.75	100
	LPQ	79.86	92.75	96.29	98.96	99.15	99.56	99.5	99.75	99.75
	LTP	81.03	92.47	96.36	98.42	98.95	99.56	99.42	99.75	100
	MMEPOP	69.97	85.34	91.25	94.04	95.8	97.38	97.75	98.63	98
	MOW-SLGS	78.33	91.66	95.89	97.88	99.1	99.75	99.5	99.88	100
	LBP	78.83	91.16	95.5	97.63	98.45	99.13	98.67	99.5	99
	NI-CI-LBP	75.28	87.91	93.82	96.62	98.3	98.63	99	100	99.5
	NI-LBP	75.64	88.66	94.36	97	98.3	99.06	99	99.38	99.75
	NI-RD-LBP	76.89	89.16	94.71	97.58	98.5	99.25	99.25	99.63	100
	QBP	74.78	87.47	92.54	95.58	97.2	98.56	98.25	98.75	98.75
	RD-CI-LBP	75.47	88	93.54	96.42	98.35	98.88	98.58	99.88	98.75
	RDLBP	75.75	88.28	93.46	96.25	97.9	99	98.67	99.38	99.25
	SLGS	79.5	91.06	95.96	98.13	98.6	99.44	99.08	99.5	99
	SMEPOP	77.39	89.72	95.07	97.33	98.45	98.88	99.25	99.38	99.5
	WLD	80.25	91.88	96.36	97.46	98.25	99.25	98.92	99.5	98.5
	XCS-LBP	77.11	86.97	92.25	95.04	96.15	98	97.83	99.13	99.25

Table 3 .

 3 2: Average recognition accuracy (%) over 10 splits on FERET dataset (Exp#2)

			FERET Database Train/Test
	Descriptor	1/6	2/5	3/4	4/3	5/2	6/1
	MNTCDP	61.57	83.56	92.21	96.48	98.52	99.7
	AECLBP-M	53.51	74.13	84.51	90.4	94.02	96.43
	AECLBP-S	60.76	81.72	90.33	94.91	97.54	99.05
	AECLBP-S-MxC	51.42	73.08	83.59	90.44	93.99	97.19
	AELTP	55.57	76.57	86.86	92.33	95.8	98.14
	ALTP	55.9	76.84	87.34	92.71	96.11	97.89
	BGC-1	57.48	78.17	88.2	93.35	96.26	98.29
	BGC-2	55.7	76.71	87.12	92.63	95.88	97.99
	BGC-3	54.37	75.9	86.17	91.91	95.35	97.54
	CI-LBP	31.36	44.64	52.63	58.39	64.4	67.14
	CLBC-M	45.94	64.72	74.92	81.01	85.95	90.05
	CRLBP-M	54.07	75.21	84.77	90.03	93.87	96.38
	CRLBP-S	56.28	76.94	86.96	92.35	95.4	97.24
	CRLBP-S-MxC	61.51	82.02	90.5	94.87	97.09	98.99
	CSALTP	49.31	70	81.16	87.44	91.93	95.03
	DBC	49.1	69.92	81.71	88.78	93.29	96.13
	DCLBP	58.24	79.58	89.16	94.44	96.91	99.05
	DDBP	58.69	80.21	89.28	94.89	97.41	99.1
	dLBPa	57.54	77.28	86.49	91.93	94.77	96.68
	DRLBP	54.92	72.58	82.5	88.11	92.14	94.42
	DSLGS	57.04	78.09	88.24	93.74	96.66	98.64
	ELGS	57.24	78.43	88.63	93.53	96.66	98.49
	IBGC-1	58.61	79.73	89.79	94.71	97.36	99.05
	ILTP	57.22	78.59	88.18	93.25	96.63	98.69
	LBP	55.44	76.39	87.24	92.71	95.8	97.59
	LCCMSP	63.92	82.17	91.19	94.64	96.81	97.79
	LDBP	58.83	79.54	89.52	94.74	96.88	98.69
	LDN	45.38	66.15	77.61	85.39	90.05	93.42
	LECTP	53.14	70.25	80.24	85.73	89.57	92.51
	LESTP	52.77	71.03	80.8	85.9	90.2	93.17
	LETRIST	65.86	83.15	90.82	95.09	97.01	98.19
	LGCP	41.48	61.78	73.48	80.6	86.38	90.05
	LGS	57.5	78.08	88.27	93.25	96.26	98.44
	LMEBP	51.31	68.54	77.66	84	87.86	90.5
	LNDP	52.91	73.26	83.22	90.75	94.22	97.14
	LPQ	55.6	76.87	86.56	92.41	95.83	98.14
	LQPAT	52.48	74.14	83.94	91.96	94.97	97.24
	LTP	55.6	76.87	86.56	92.41	95.83	98.14
	MMEPOP	49.21	70.89	83.13	89.95	94.22	96.63
	MOW-SLGS	57.19	77.89	88.13	93.57	96.58	98.54
	nLBPd	55.64	76.69	87.19	92.48	95.8	98.14
	NI-CI-LBP	53.23	74.03	83.92	89.53	93.39	95.33
	NI-LBP	51.54	72.55	83.42	90.08	93.97	96.93
	NI-RD-LBP	55.8	76.31	86.76	92.35	95.5	97.49
	QBP	50.93	70.45	81.03	87.47	91.33	94.17
	RD-CI-LBP	56.06	76.55	85.58	90.79	93.87	95.18
	RDLBP	57.35	76.74	87.27	92.35	95.23	97.39
	SLGS	57.04	78.09	88.24	93.74	96.66	98.64
	SMEPOP	52.82	74.12	85.63	91.56	95.03	97.49
	WLD	62.64	80.15	88.39	92.91	95.5	98.04
	XCS-LBP	52.94	72.47	82.55	88.64	93.54	96.63

Table 3 .

 3 3: Average recognition accuracy (%) over 10 splits on YALE dataset (Exp#3)

					YALE Database Train/Test
	Descriptor	1/10	2/9	3/8	4/7	5/6	6/5	7/4	8/3	9/2	10/1
	MNTCDP	91.53	96.59	97.58	98.29	99	98.93	100	100	100	100
	AECLBP-M	84.07	91.7	92	92	92.89	92.8	95.5	93.78	91	93.33
	AECLBP-S	89.47	94.96	95	95.52	96.33	95.07	96.83	96	95.67	97.33
	AECLBP-S-MxC	86.67	91.7	90.92	91.24	90.89	90.13	92	90.22	89	90.67
	AELTP	88.73	94.3	93.83	94.67	94.33	93.2	96.17	94.89	93.67	96.67
	ALTP	89	95.41	96.25	95.9	96.56	95.47	96.83	95.56	94.67	97.33
	BGC-1	87	90.22	92	92.1	92.78	92.67	93.17	92.22	91.67	92.67
	BGC-2	87.4	91.26	92.92	92.95	94.78	94.13	95.33	93.33	93.67	94.67
	BGC-3	87.6	92.59	93.33	93.24	93.56	93.33	94.17	93.11	92.67	92.67
	CI-LBP	70.73	81.56	85.33	85.81	86.22	86.93	89.83	88.44	90.67	88.67
	CLBC-M	80.93	89.56	90.58	90.76	91.56	91.2	93	90.67	89.33	93.33
	CRLBP-M	82.33	90.22	91	89.9	91.44	91.6	94.17	92	90	93.33
	CRLBP-S	88.4	93.7	94.58	94.38	94.89	94.27	94.83	94.22	93.33	94.67
	CRLBP-S-MxC	84.6	90.37	90.33	89.9	90.22	89.2	91	88.44	87.33	88.67
	CSALTP	88.07	92.81	92.42	92.48	91.89	90.93	92.83	91.33	91.67	92.67
	DBC	86.67	90.22	93.25	92.57	93.56	92.93	93.5	92	91.67	90.67
	DCLBP	89.6	95.78	95.75	96.48	96.56	95.6	97.83	96.22	95.33	96
	DDBP	87.87	93.41	94.58	93.81	95	93.6	94.67	92.89	93.67	96
	dLBPa	86.73	94	96.58	96.67	98.11	99.07	98.5	99.56	99.67	99.33
	DRLBP	82.6	87.56	90.83	91.52	92.22	91.6	92.17	91.56	91.33	90.67
	DSLGS	87.33	92.59	92.92	93.05	93.67	93.2	94	92.22	93	92
	ELGS	87.07	92.44	93.42	93.52	94.22	93.07	94.33	93.33	93.67	94.67
	IBGC-1	87.47	91.48	93.33	93.14	94.67	93.47	94.17	92	93	94
	ILTP	87.13	91.93	93.17	92.95	93.89	92.93	95	94.44	93.33	96.67
	LBP	89.53	93.85	95.25	95.33	96.78	95.6	96.33	95.56	95	97.33
	LCCMSP	73.93	80.74	84.42	82.48	84	82.93	83.83	84.44	83	84
	LDBP	88.13	92.44	94.33	94.38	95.67	94.53	94.67	93.33	93	92.67
	LDN	85.73	90.81	92.83	93.71	94.33	94.53	95.17	94.44	95	96.67
	LECTP	70.87	80.3	82.67	84.29	84.22	85.2	86.83	85.78	85	82.67
	LETRIST	71.53	83.63	89.42	89.24	89.44	90.27	90.5	92.67	92	93.33
	LESTP	75	83.48	85.42	86.29	86.56	87.87	89.83	88.22	86.33	87.33
	LGCP	87	92.15	91.67	92.29	92.22	91.2	92.17	91.33	91.67	92.67
	LGS	86.87	92.52	93.25	93.24	93.67	92.8	93.67	92.67	92.67	93.33
	LMEBP	72.67	83.56	85.33	87.71	87.56	88.4	89.83	88.89	88.67	86
	LNDP	75.13	84.67	89	89.24	89.22	89.87	90.67	88.89	90	94
	LPQ	87.93	94.52	95.92	96.95	97.56	97.87	97.5	98	97	98
	LQPAT	80	87.41	90.83	90.57	89.67	90.8	91.33	91.56	93	92.67
	LTP	87.87	91.63	92.75	92	92.89	92.67	94.5	93.78	92.33	94
	MMEPOP	84.2	91.85	95.25	96.19	96.44	98	97.67	97.56	97.33	97.33
	MOW-SLGS	88.2	93.26	93.5	93.71	94.22	92.93	93.83	92.44	92.33	94
	nLBPd	87.53	90.96	93.08	92.86	94.11	92.93	93.83	92.67	92.67	94.67
	NI-CI-LBP	78.4	87.78	89.92	90.1	90.22	90.53	92.5	91.33	92.67	90.67
	NI-LBP	79.73	87.48	90.25	90.19	90.56	90.4	91.67	90.89	93	90
	NI-RD-LBP	80.53	88.07	90.33	90.76	90.78	90.4	91.67	89.78	92	88.67
	QBP	82.4	89.85	90.92	90.48	91.22	91.47	91.83	90.44	89	89.33
	RD-CI-LBP	79.47	88.3	90.08	91.14	91.56	91.6	93	92	93.33	90.67
	RDLBP	80.73	88.52	90.17	91.05	92.11	91.07	92.83	90.89	94	90.67
	SLGS	87.33	92.59	92.92	93.05	93.67	93.2	94	92.22	93	92
	SMEPOP	85.93	90.81	92.08	92.38	93	93.6	93.83	92.67	93	93.33
	WLD	66.47	75.7	79.58	81.33	83.67	82.8	85.83	84.89	84	82.67
	XCS-LBP	90.33	96.52	97	97.9	98.56	98.4	99.33	98.44	97.33	98

Table 3 .

 3 4: Average recognition accuracy (%) over 10 splits on Extended Yale B dataset (Exp#4)

			Extended YALE B Train/Test
	Descriptor	5/59	10/54	20/44	30/34	32/32
	MNTCDP	87.78	94.42	97.87	98.76	98.91
	AECLBP-M	49.62	60.8	71.72	76.67	79.07
	AECLBP-S	78.96	90.41	96.24	97.41	97.84
	AECLBP-S-MxC	60.7	75.99	88.49	93.16	94.57
	AELTP	57.77	69.9	82.1	85.89	87.49
	ALTP	77.06	89.2	96.01	97.43	97.88
	BGC-1	75.04	88.67	95.9	97.5	97.84
	BGC-2	76.43	89.78	96.21	97.6	97.84
	BGC-3	76.92	89.66	96.21	97.41	97.78
	CI-LBP	91.26	95.05	97.1	97.37	97.68
	CLBC-M	47.26	57.7	65.36	69.87	72.3
	CRLBP-M	50.34	60.93	73.08	78	80.65
	CRLBP-S	69.09	82.85	92.7	95.68	96.59
	CRLBP-S-MxC	57.34	72.62	85.42	91.24	92.42
	CSALTP	76.68	89.52	96.24	97.52	97.74
	DBC	79.59	88.93	96.16	97.64	97.8
	DCLBP	66.13	79.36	91.02	94.4	95.86
	DDBP	79.16	91.41	96.77	97.64	98.04
	dLBPa	82.99	92.72	96.96	97.54	97.9
	DRLBP	72.08	85.39	93.61	96.21	97.35
	DSLGS	76.81	90.06	96.19	97.54	97.94
	ELGS	75.88	89.65	96.16	97.49	97.88
	IBGC-1	75.57	89.28	95.99	97.43	97.84
	ILTP	54.49	65.73	76.5	81.03	83.13
	LBP	75.74	89.05	95.69	97.49	97.8
	LCCMSP	57.87	75.43	89.76	94.15	94.81
	LDBP	79.16	91.41	96.77	97.64	98.04
	LDN	78.87	89.71	95.9	97.22	97.76
	LECTP	67.74	80.63	91.69	94.9	95.82
	LETRIST	58.36	73.71	87.47	92.59	93.78
	LESTP	63.04	75.9	87.63	91.64	92.93
	LGCP	77.17	89.98	96.16	97.47	97.84
	LGS	75.8	89.73	95.99	97.49	97.82
	LMEBP	68.3	81.04	91.7	94.95	95.7
	LNDP	71.63	85.24	94.31	96.84	97.62
	LPQ	84.11	92.53	96.7	97.66	97.92
	LQPAT	79.3	91.22	96.71	97.73	97.96
	LTP	58.21	70.23	82.1	86.02	87.78
	MMEPOP	75.98	87.44	94.8	96.86	97.49
	MOW-SLGS	76.08	89.53	96.07	97.5	97.92
	nLBPd	75.93	89.68	96.04	97.49	97.76
	NI-CI-LBP	92.39	95.36	96.59	96.61	96.38
	NI-LBP	86.68	94.76	97.47	97.9	98.08
	NI-RD-LBP	84.84	92.41	95.41	96.25	96.04
	QBP	59.04	73.27	85.69	90.13	91.56
	RD-CI-LBP	90.04	92.96	95.39	95.9	95.8
	RDLBP	80.5	85.39	92.06	93.62	93.76
	SLGS	76.81	90.06	96.19	97.54	97.94
	SMEPOP	73.41	87.62	95.24	97.09	97.58
	WLD	36.86	26.48	34.34	39.24	37.76
	XCS-LBP	58.58	74.07	87.63	91.14	92.59

Table 3 .

 3

	5: Recognition accuracy (%) on Extended Yale B dataset using subsets
	evaluation protocol (Exp#5)
		Extended YALE B Subsets
	Descriptor	Sub2	Sub3	Sub4	Sub5
	MNTCDP	100	100	99.8	98.5
	AECLBP-M	100	88.92	12.74	10.67
	AECLBP-S	100	100	95.63	93.04
	AECLBP-S-MxC	100	93.67	52.66	48.59
	AELTP	100	95.51	23.57	9.93
	ALTP	100	100	94.87	89.48
	BGC-1	100	100	96.01	92.44
	BGC-2	100	100	97.53	96.15
	BGC-3	100	99.74	98.48	97.19
	CI-LBP	100	98.94	96.01	82.52
	CLBC-M	100	85.22	29.28	10.07
	CRLBP-M	100	86.81	12.74	11.85
	CRLBP-S	100	99.21	79.28	61.04
	CRLBP-S-MxC	100	91.29	43.92	33.93
	CSALTP	100	100	95.82	95.41
	DBC	100	100	94.49	83.11
	DCLBP	100	95.25	79.85	66.07
	DDBP	100	100	97.15	94.81
	dLBPa	100	98.42	92.02	88.59
	DRLBP	100	98.68	82.89	68.74
	DSLGS	100	100	97.72	95.85
	ELGS	100	100	98.67	96.15
	IBGC-1	100	100	97.91	94.22
	ILTP	100	92.35	23.95	7.41
	LBP	100	99.47	93.16	87.7
	LCCMSP	100	97.25	87.36	76.44
	LDBP	100	100	94.11	90.37
	LDN	100	99.74	96.77	91.11
	LECTP	100	97.89	46.96	35.7
	LETRIST	100	93.40	58.37	35.56
	LESTP	100	96.83	27.57	20.74
	LGCP	100	99.74	97.15	96
	LGS	100	100	98.29	96
	LMEBP	100	99.21	44.49	35.26
	LNDP	100	99.74	86.50	73.48
	LPQ	100	100	99.05	98.07
	LQPAT	100	100	97.34	97.48
	LTP	100	94.72	28.71	11.56
	MMEPOP	100	99.47	92.59	83.7
	MOW-SLGS	100	100	97.15	94.22
	nLBPd	100	100	97.53	96.59
	NI-CI-LBP	100	99.74	98.67	94.81
	NI-LBP	100	99.74	97.34	96.44
	NI-RD-LBP	100	99.47	96.01	82.96
	QBP	100	94.46	62.55	34.67
	RD-CI-LBP	100	99.47	96.58	85.93
	RDLBP	100	99.21	87.26	72.89
	SLGS	100	100	97.72	95.85
	SMEPOP	100	100	95.63	90.96
	WLD	87.17	45.12	18.63	22.22
	XCS-LBP	100	97.36	46.96	36.89

Table 3 .

 3 6: Average recognition accuracy (%) over 10 splits on AR dataset (Exp#6)

			AR Database Train/Test
	Descriptor	5/21	10/16	13/13	15/11	20/6
	MNTCDP	82.33	93.53	96.18	97.37	99.28
	AECLBP-M	69.12	84.63	90.13	92.33	96.33
	AECLBP-M	69.12	84.63	90.13	92.33	96.33
	AECLBP-S	77.56	91.69	95.08	97.02	99.23
	AECLBP-S-MxC	62.99	81.26	88.04	91.24	96.63
	AELTP	71.03	86.69	91.65	94.15	97.87
	ALTP	69.58	86.04	91.17	93.91	97.77
	BGC-1	75.87	90.41	94.45	96.15	98.75
	BGC-2	76.26	90.74	94.69	96.47	98.98
	BGC-3	74.18	89.73	93.65	95.88	98.63
	CI-LBP	57.9	73.66	77.5	80.93	85.28
	CLBC-M	64.03	80.56	86.48	89.45	94.28
	CRLBP-M	65.46	81.39	87.05	89.89	94.15
	CRLBP-S	71.78	87.91	92.68	94.8	98.32
	CRLBP-S-MxC	57.98	76.83	84.27	88.23	94.25
	CSALTP	67.34	84.45	89.88	92.77	97.15
	DBC	72.63	86.88	91.75	94.11	97.17
	DCLBP	75.35	89.96	94.19	96.14	98.75
	DDBP	78.33	91.78	95.33	96.78	99.07
	dLBPa	77.49	90.2	93.95	95.45	98.4
	DRLBP	74.86	88.94	93.7	95.38	97.82
	DSLGS	78.3	91.95	95.43	96.97	99.15
	ELGS	76.99	91.28	94.92	96.75	99.1
	IBGC-1	77.02	90.65	94.55	96.27	98.72
	ILTP	69.91	85.49	90.92	93.32	97.1
	LBP	75.58	90.17	94.32	96.15	98.75
	nLBPd	76.27	90.67	94.61	96.49	98.97
	LCCMSP	55.97	76.03	83.42	87.21	94.18
	LDBP	77.72	91.52	95.15	96.66	99.02
	LDN	79.29	91.99	95.62	97	99.1
	LECTP	61.66	77.54	83.76	86.84	92.08
	LESTP	56.48	72.96	79.29	82.51	88.9
	LETRIST	59.28	77.19	84.08	87.57	92.77
	LGCP	73.46	89.14	93.51	95.51	98.48
	LGS	76.97	91.32	94.92	96.73	99.08
	LMEBP	61.38	77.46	83.8	86.6	92.12
	LNDP	77.2	90.81	94.3	96.17	98.73
	LPQ	79.57	91.79	95.22	96.49	99.03
	LQPAT	77.07	91.24	94.7	96.44	98.97
	LTP	69.15	84.92	90.36	93.13	97.02
	MMEPOP	77.33	90.09	94.02	95.43	98.43
	MOW-SLGS	77.56	91.48	95.2	96.89	99.08
	NI-CI-LBP	78.32	91.11	94.32	96.01	98.47
	NI-LBP	78.2	91.49	94.65	96.35	98.67
	NI-RD-LBP	78.34	91.8	94.81	96.57	98.92
	QBP	63.52	81.45	87.06	90.32	95.47
	RDLBP	77.85	91.27	94.7	96.34	98.75
	RD-CI-LBP	78.04	91.04	94.18	95.99	98.43
	SLGS	78.3	91.95	95.43	96.97	99.15
	SMEPOP	73.08	88.77	93.16	95.2	98.55
	WLD	57.73	73.42	79.84	83.25	88.55
	XCS-LBP	67.25	82.79	88.12	90.84	95.33

Table 3 .

 3 7: Comparison with PCANet2 and CBFD deep learning methods

	Database		MNTCDP 10 Splits avg acc	PCANet2 1 Split acc	CBFD 1 Split acc
		1/9	85.89	77.5	85
		2/8	95.59	88.13	87
		3/7	98.64	94.29	95
	ORL Database Exp#1 configuration	4/6	99.67	100	97
		5/5	100	99	100
		6/4	100	98.75	100
		7/3	100	100	100
		8/2	100	100	100
		9/1	100	100	100
		1/6	61.57	62.4	67
		2/5	83.56	81.61	85
	FERET Database Exp#2 configuration	3/4	92.21	25	93
		4/3	96.48	53	97
		5/2	98.52	97.99	97
		6/1	99.7	99.5	99
		1/10	91.53	80.67	99.12
		2/9	96.59	94.07	99.45
		3/8	97.58	93.33	99.33
	YALE Database Exp#3 configuration	4/7	98.29	90.48	98.1
		5/6	99	7.78	99.36
		6/5	98.93	8	100
		7/4	100	11.67	98.74
		8/3	100	6.67	98.82
		9/2	100	10	98.97
		10/1	100	13.33	100
		5/59	87.78	82.91	88.36
	Extended YALE B Exp#4 configuration	10/54 20/44	94.42 97.87	92.89 94.29	95.12 81.85
		30/34	98.76	94.55	97.26
		32/32	98.91	71.11	97.5
		5/21	82.33	96.29	94
	AR Database Exp#6 configuration	10/16 13/13	93.53 96.18	98.69 99.08	96 98
		15/11	97.37	98.45	99
		20/6	99.28	98.5	97
	Accuracy over the test subsets, Subset 1 is used for training phase
		Sub2	100	100	100
	Extended YALE B Subsets Exp#5 [52]	Sub3	100	99.47	98
		Sub4	99.81	92.21	70.34
		Sub5	98.52	63.11	54.03

Table 3 .

 3 9: Comparison with recent state-of-the-art systems on EYB database using subset evaluation protocol.

		Extended YALE B Subsets
	Ref. (Publication \& Year)	Sub2	Sub3	Sub4	Sub5
	[52] (ESWA 2016)	100	100	94	95
	[40] (IEEE TIP 2012)	100	100	-	-
	[153] (Info.Sc 2016)	99.8	99.6	99.4	97.8
	[154] (PR 2017)	-	98	94	93
	[155] (PR 2017)	100	99.12	96.99	97.74
	[156] (IEEE TIF&S 2016)	100	99.54	93.89	93.17
	[157] (DSP 2015)	100	91.42	83.65	85.71
	[152] (PR 2017)	100	99.12	99.44	98.89
	[158] (IEEE SPL 2015)	100	100	94.55	91.14
	[130] ( FGCS 2017)	100	100	95	95
	Proposed MNTCDP based system	100	100	99.81	98.52
	3.6/ IMPLEMENTATION IN THE HUMAN SUPPORT ROBOT (HSR)

OF UTBM

  Algorithm 2 GAN training process for the PIFR proposed framework Input: Paired Input Images {Im F , Im P }

	Output: Trained Generator G
	Functions: optim: optimizer to update the networks; D: discriminator network;
	for N epochs do
	for N/m steps do
	sample a minibatch of m frontal samples {Im F 1 , Im F 2 ,.....,Im F m } sample a minibatch of m profile samples {Im P 1 , Im P 2 ,.....,Im P m } translate the m profile images to frontal ones {G(Im P 1 ), G(Im P 2 ),.....,G(Im P m )}
	calculate the ascending loss gradient of the discriminator:

Table 4 .

 4 1 lists the calculated MSE and SSIM values. It appears that the Pixel discriminator allowed the majority of the generators to achieve the highest results and then good generated images compared to the 70×70 PatchGAN, which is quite straightforward since the Pixel discriminator is more precise than the PatchGAN by checking the generation quality of each pixel. Moreover, the full-profile (90 • ) challenged the GAN more as the metrics are bit lower than the ones of 45 • with 62.41% SSIM and 4.187 e4 MSE vs. 71.19% SSIM and 3 e3 MSE.

Table 4 .

 4 1: Similarity-based comparison of the evaluated GAN models

	Pose Angle GAN	G e n e r a to r	Discriminator MSE PatchGAN SSIM SSIM	Pixel	MSE
		U-Net 256	0.678	3.19 e3	0.7109	3.1634 e3
	45	U-Net 128	0.6353	3.1417 e3	0.6442	3.114 e3
		ResNet 9B	0.7104	2.7748 e3	0.7119	3.0229 e3
		ResNet 6B	0.6956	2.9238 e3	0.7114	3.0006 e3
		U-Net 256	0.6057	4.385 e3	0.6144	4.4067 e3
	90	U-Net 128	0.5594	4.3302 e3	0.5714	4.3097 e3
		ResNet 9B	0.6032	4.2446 e3	0.6241	4.2144 e3
		ResNet 6B	0.6069	4.2372 e3	0.6039	4.187 e4

Table 4 .

 4

	2: Recognition rate of the proposed framework on the Combined-PIFR database
			test set
	Discriminator	Generator	Face Classifier ResNet18 ResNet50 ResNet101 DenseNet
		ResNet_6B	23.505	56.47	66.835	49.3525
	P a c t	ResNet_9B	19.955	38.845	51.895	40.5875
	h G A	U-Net 128	30.48	71.81	77.29	56.77
	N	U-Net 256	34.065	70.615	78.785	67.3825
		ResNet_6B	48.805	48.705	62.05	52.04
	P i x e l	ResNet_9B U-Net 128	44.625 52.39	44.425 74.305	60.655 78.685	47.5575 61.6525
		U-Net 256	71.015	74.1	82.97	73.955
	Baseline	31.08	41.61	49.4	39.05
	Frontalization Improvement	39.935	32.695	33.57	34.905

Table 4 .

 4

	2: Recognition rate of the proposed framework on the Combined-PIFR database
			test set
	Discriminator	Generator	Face Classifier ResNet18 ResNet50 ResNet101 DenseNet
		ResNet_6B	23.505	56.47	66.835	49.3525
	P a t	ResNet_9B	19.955	38.845	51.895	40.5875
	c h G A	U-Net 128	30.48	71.81	77.29	56.77
	N	U-Net 256	34.065	70.615	78.785	67.3825
		ResNet_6B	48.805	48.705	62.05	52.04
	P i x e l	ResNet_9B U-Net 128	44.625 52.39	44.425 74.305	60.655 78.685	47.5575 61.6525
		U-Net 256	71.015	74.1	82.97	73.955
	Baseline	31.08	41.61	49.4	39.05
	Frontalization Improvement	39.935	32.695	33.57	34.905

Table 5 .

 5 1: Average FER rate of each OPD-GQMBP configuration (neighborhood size),

		overall databases
	N S ize Config	CK+ JAFFE KDEF OuluCasia RaFD
	OPD -GQMBP 3 95.74 73.33 88.57	73.93	96.08
	OPD -GQMBP 5 96.01 73.33 87.96	74.82	95.9
	OPD -GQMBP 7 96.48 78.57	90.2	77.32	97.39
	OPD -GQMBP 9 97.53	71.9	87.96	75.95	96.08
	5.2.3.3/ COMPARATIVE ANALYSIS AGAINST STATE-OF-THE-ART HANDCRAFTED AND
	DEEP FEATURE METHODS

  AlexNet) as minimum average on the CK+. Based on Table5.1, we found that n = 9 configuration of the OPD-GQMBP method scored 97.53% on CK+, which improves by 1% compared to OPD -GQMBP with n = 7 adopted in this evaluation. The OPD-GQMBP reached 77.62% on JAFFE with a lead of 3% to the rest of the methods since the second highest avg accuracy is 74.76% recorded by the DCP Table5.2: Average and Maximum accuracies recorded on the five datasets by each method on this database was around 2% for the favor of OPD-GQMBP descriptor, which proves its discriminative power for FER application. Despite the quality of the recorded images in terms of resolution, lighting conditions, and uniform background, the KDEF database is also challenging in face to the CK+. It presents fewer individ-uals (fewer runs), who express the seven emotions in different manners making more intraclass similarities and approaching the spontaneous facial expressions. OuluCasia can be considered the toughest among the adopted benchmarks due to the blur images On the other hand, ResNet50 was the best among deep feature methods with an average accuracy of 84.51%. In terms of stability, the OPD-GQMBP descriptor performed well on the five datasets, always reaching the top average accuracies and 100% max all the time. ELGS method also presented stable performance across all databases. On the other hand, DCP suffered a performance drop on KDEF and Oulu-

	JAFFE	KDEF	CK+	OuluCasia	RaFD
	94.36%. All the evaluated methods were capable of reaching 100% Max accuracy on the CK+ dataset. Moreover, most handcrafted methods performed above 87.81%, reached by DCP descriptor, while the deep features reached a maximum average of 88.76% with VGG19 only 54.18% (Method Avg Max Avg Max Avg Max Avg Max Avg Max VGG16 56.67 76.19 76.53 100 86.34 100 56.96 100 82.84 VGG19 53.81 76.19 76.73 100 88.76 100 58.57 100 81.53 ResNet18 35.24 57.14 72.24 100 72.67 100 52.32 100 83.02 ResNet50 44.76 61.9 75.71 100 77.32 100 56.96 100 84.51 ResNet101 52.86 71.43 70.82 100 76.79 100 53.57 100 78.17 AlexNet 43.81 66.67 66.94 100 54.18 100 29.64 71.43 67.54 DenseNet 48.1 80.95 70.2 100 76.15 100 50.36 100 77.8 GoogLeNet 47.14 66.67 71.63 100 69.98 100 44.64 85.71 79.66 Inceptionv3 39.52 57.14 65.31 100 71.22 100 44.46 85.71 70.52 InceptionResNetv2 47.62 66.67 76.33 100 80.17 100 55 100 81.72 ELGS 73.81 95.24 85.71 100 92.59 100 70 100 95.34 DSLGS 60.95 85.71 78.57 100 91.2 100 64.64 100 91.23 MNTCDP 69.05 85.71 85.51 100 94.36 100 55.36 85.71 95.15 QBP 62.38 85.71 79.39 100 91.99 100 66.43 100 91.79 DRLBP 70 85.71 84.49 100 90.35 100 65.36 100 96.08 LNDP 69.52 95.24 86.94 100 92.86 100 68.75 100 96.08 DCP 74.76 100 69.18 100 87.81 100 45.89 100 87.31 DC 66.67 95.24 84.69 100 89.87 100 74.64 100 95.52 LCCMSP 71.43 95.24 86.12 100 92.28 100 69.29 100 97.01 LOOP 64.76 85.71 85.92 100 91.7 100 69.64 100 96.27 LDTP 70 90.48 88.37 100 93.3 100 67.5 100 97.01 ARCS-LBP 65.24 95.24 85.51 100 91.45 100 75.5 100 96.64 OPD-GQMBP 77.62 100 90.2 100 96.48 100 77.32 100 97.2 descriptor followed by ELGS as the third best performing method (73.81%). According Deep Features Handcrafted LBP Variants to the maximum accuracy on the JAFFE database, only OPD-GQMBP and DCP meth-ods managed to score 100%. The Japanese females' high facial similarities make the JAFFE a very tough benchmark to the FER systems. The results demonstrated that the JAFFE database is very challenging, especially for deep features where their top aver-age accuracy was limited to 56.67% obtained by VGG16, while the handcrafted ones granted an average accuracy above 64.76% reached by the LOOP descriptor. For the KDEF database, the first remark to be concluded is that the OPD-GQMBP descriptor is the only method that breaks the 90% average performance ceiling and the best state-of-the-art method reached 88.37% (LDTP). The handcrafted methods' performance and deep ones vary from 78.57% to 88.37% and from 65.31% to 76.73%, respectively. Again, scored 77.32% as the overall best performing method, followed by ARCS-LBP (75.5%) and DC (74.64%) descriptors. The lowest accuracy was recorded by AlexNet deep net-work reaching only 29.64%. All the methods performed 100% Max accuracy except the MNTCDP, AlexNet, GoogLeNet, and Inceptionv3. Moreover, only ELGS, DC, ARCS-LBP, and OPD-GQMBP methods exceeded 70% average accuracy, while the DCP descriptor was the second top-ranked on JAFFE, found its performance limited to 45.89% as the lowest average accuracy over all the handcrafted LBP-like methods. The RaFD database is collected by a set of well-trained individuals who clearly express eight emotions (basic emotions + Contempt). Indeed, the peak average accuracy exceeded 97% by the pro-posed OPD-GQMBP descriptor with a minor lead (0.19%) against LDTP and LCCMSP methods (97.01%). Moreover, all the handcrafted descriptors reached above 90% ex-cept DCP (87.31%). Casia datasets. For the deep feature methods, VGG16 can be considered as the best the leadership gap and the south Asian persons composing this database. The OPD-GQMBP descriptor performing deep feature method.

Table 5 .

 5 As can be found in Table5.3, the proposed FER framework outperforms all the listed systems, including both handcrafted and deep based ones on the CK+ database. We reached 96.48% (97.53% with the OPD-GQMBP neighborhood size n = 9), while the best accuracy of the state-of-the-art is 94.96% achieved using LPQ with SVM classifier.

	3: State-of-the-art person independent FER accuracies on CK+ database
	Methods	Type	Avg accuracy
	LPDP [105]	Handcrafted	94.5
	DCNN [182] DNN [183]	Deep Deep	94.44 93.52
	CNN+AFM [184]	Deep	89.84
	AlexNet+SVM [184]	Deep	86.83
	GoogLeNet [184]	Deep	85.71
	STM-ExpLet [185]	Deep	94.13
	CER [186]	Handcrafted	92.34
	SRC+ICV [187]	Handcrafted	90.5
	MSR [188]	Handcrafted	91.4
	Gabor+SRC [186]	Handcrafted	82.82
	3DCNN-DAP [189]	Deep	92.4
	LTeP+SVM [190]	Handcrafted	94.93
	LPQ+SLPM+NN [191] Handcrafted	94.61
	WPLBP [192]	Handcrafted	91.72
	Proposed	Handcrafted	97.53
	Moreover, the majority of the published works performed between 90% and 94%. On
	the JAFFE database, the state-of-the-art accuracies are low compared to CK+, where
	the maximum reported is 76.46% scored by CFER based framework outperformed by
	our FER framework (77.62%). The proposed framework managed to surpass with signif-
	icant leads all the works on the KDEF dataset, except for WCFN (89.55%) and AlexNet
	(89.33%) based systems where the margin is small (0.65% and 0.87% to our frame-
	work performance, respectively). On the OuluCasia dataset, the proposed framework
	(77.32%) outperformed all the state-of-the-art methods, where the top accuracy was lim-
	ited to 75.52% reached by Atlases. On the RaFD database, our proposed framework
	obtained 97.2%. Many works of the literature performed nearly to 97%. However, the
	majority were applied only on 7 emotion classes. Overall, we conclude that the proposed
	facial expression recognition framework managed to outperform all the tested state-of-
	the-art ones.
	5.2.3.5/ CONFUSION MATRIX-BASED ANALYSIS FOR THE FER

The confusion matrix allows analyzing the performance of the recognition according to each label (each emotion in our case). Through this chart, we are capable of analyzing the recognition rate of each emotion and what are the easiest and hardest ones to be recognized. Also, this analysis allows identifying which emotion is affecting the others.

Table 5 .

 5 4: State-of-the-art person independent FER accuracies on JAFFE database

	Methods	Type	Avg accuracy
	LTeP+SVM

Table 5 .

 5 5: State-of-the-art person independent FER accuracies on KDEF database

	Methods	Type	Avg accuracy
	AlexNet+FC6+LDA [199]	Deep	89.33
	HOG+SRC [200]	Handcrafted	78
	VGG-Face Deep [201]	Deep	72.55
	SCAE [202]	Deep	86.73
	CNN Caffe-ImageNet [203] Deep	59.15
	Geometric Features [204]	Handcrafted	79.69
	DFD [205]	Handcrafted	82.24
	HPBSVM [206]	Handcrafted	81.84
	WCFN [207] MobileNet [208]	Deep Deep	89.55 73.74
	EDR-PCANet [193]	Deep	80.61
	Proposed	Handcrafted	90.2

Table 5 .

 5 6: State-of-the-art person independent FER accuracies on OuluCasia database

	Methods	Type	Avg accuracy
	STM-ExpLet [185]	Deep	74.59
	LBP+Gabor+SVM [209] Handcrafted	74.37
	HOG 3D [210]	Handcrafted	70.63
	AdaLBP [211] Atlases [208]	Handcrafted Deep	73.54 75.52
	Proposed	Handcrafted	77.32

Table 5 .

 5 7: State-of-the-art person independent FER accuracies on RaFD database Angry samples were identified as Sad. On KDEF, the errors were less compared to JAFFE. Happy and Neutral once again were the highly recognized emotions (98.6%), but Angry is the challenging class with only a 78% rate. Our FER framework confused Disgust 7 times with Fear and 6 times Fear with Sad. Happy and Disgust emotions were not affecting any other emotion except 1 time for each with Angry only. As expected on

	Methods	Type	Avg accuracy
	Visual Attention CNN [212] Deep	95.2
	DS+FE+GEM+SVM [213]	Handcrafted	90.8
	LPQ+FE+GEM+SVM [213] Handcrafted	94.4
	LBP+FE+GEM+SVM [213] Handcrafted	94.5
	DS+SVM [214] Metric Learning [215]	Handcrafted Deep	79 95.95
	BAE-BNN-3 [216]	Deep	96.93
	W-CR-AFM [184] Net1-Net2 [217]	Deep Deep	96.27 93.41
	Proposed	Handcrafted	97.2

Figure 5.14: Confusion matrix of the eight emotions of RaFD Neutral were the easiest to identify with an accuracy of 86.7% and the hardest is Fear (60%), which was confused with all the emotions: 3 times with Disgust, Neutral, Surprise, and once with the rest. Sad class presented the highest false-negative rate (34.4%). The females composing this database express the Angry and Sad emotion in similar ways since 5

  The CPU can be used to calculate the feature vector by a deep model, but it takes around 4 seconds to compute it for an input image with a size of only 224 by 224 (3 times smaller than the size of the original image), which is very high compared to the handcrafted ones. The obtained computation times are illustrated graphically in Figure5.15. As can be seen, the elapsed times for the landmarks detection, HOG shape feature extraction, and SVM prediction did not change across the evaluated methods, where they recorded 15.9 ms, 19.58 ms, and 2.306 ms respectively. The process of extracting the appearance features demonstrated to be the most time taking one within our framework (more than 50% of the total time). The fastest handcrafted method is DRLBP that extracted the features from the 49 landmarks in 31.5 ms, while the LDTP took 311.6 ms to extract these features judged thus as the heaviest one. However, the DRLBP did not perform well in terms of classification accuracy, whereas the proposed OPD-GQMBP descriptor managed to offer an execution time of 90.34 ms only, which is so beneficial regarding its high performance, as demonstrated earlier. Moreover, all the best-performing descriptors took more than 100 ms to compute the appearance features. Although the PCA stage is common to all the descriptors for dimension reduction, its execution time was variable and affected by the number of generated patterns of each handcrafted method as a high number of patterns leads to more computation. Nevertheless, it can be remarked that the PCA computation times are similar for the methods sharing the same size of generated patterns. The methods producing 256 patterns such as LOOP, QBP, DC, ARCS-LBP, and DRLBP recorded a PCA computation time around 11 ms. The PCA process of the proposed OPD-GQMBP descriptor as well as those producing 512 patterns, took around 20 ms, while LDTP and LNDP methods generating 1024 patterns took about 36 ms. On the other hand, the PCA-based dimensionality reduction process related to the LCCMSP

19.58 19.58 19.58 19.58 19.58 19.58 19.58 19.58 19.58 19.58 19.58 19.58 19.58 61.4 167.3 62.37 31.6 30.8 65.22 145.72 311.6 34.22 293.2 101.8 90.34

Table 5 .

 5 8: Evaluation of the proposed CNN-LSTM according to different CNN models on the three benchmarks DenseNet200 96.85 100 84.33 85.94 92.17 95.96 GoogLeNet 97.85 100 84.48 85.94 92.32 95.96 Inception.v2 96.19 100 85.02 85.94 91.75 95.96 Inception.v3 98.52 100 84.88 85.94 92.75 95.96 InceptionResNet.v1 96.63 100 81.48 82.81 89.32 92.71 InceptionResNet.v2 97.30 100 81.91 82.81 89.75 92.71 ResNet101 98.36 100 84.91 87.5 93.03 96.3 ResNet18 99.41 100 86.62 89.06 93.46 96.3 ResNet50 98.72 100 85.19 87.5 93.03 96.3 VGG16 97.30 100 82.91 85.94 90.75 92.71 VGG19 97.74 100 84.05 85.94 91.89 92.71 performing by reaching 86.62% and 85.19%, respectively. The third best-performing model is Inception.v2, that controversy outperformed Inception.v3 on the OuluCasia as they scored 85.02% and 84.88%, respectively. The fourth-ranked model is the ResNet101 model by 84.91%. The lowest recorded average accuracy is 81.48% by InceptionRes-Net.v1, which was also among the worst-models on the CK+. Unlike on the CK+ benchmark, none of the evaluated models could reach 100% as max accuracy. Indeed, only the ResNet18 reached 89.06% and 87.5% for ResNet50, while the rest did not exceed 85.94%. The advantage of using four temporal samples is confirmed once again on the OuluCasia as the person-independent recognition rate increased from 77.32% in the static use case to 85.62%.

	Network CNN-LSTM	CK+ Avg Max	OuluCasia Avg Max	Avg	MMI Max
	AlexNet	97.74 100 84.76 85.94 92.60 96.3

  hamed; El-merabet, Youssef; Ruichek, Yassine; Messoussi, Rochdi;; Information Sciences-Elsevier; 2021 Impact Factor: 5.524 • "Saliency Heat-Map as Visual Attention for Autonomous Driving Using Generative Adversarial Network (GAN)". Lateef, Fahad; Kas, Mohamed; Ruichek, Yassine;; IEEE Transactions on Intelligent Transportation Systems-IEEE; 2021 Impact Factor: 7.42 • "Generative Adversarial Networks for 2D-based CNN Pose-Invariant Face Recognition". Kas, Mohamed; El-merabet, Youssef; Ruichek, Yassine; Messoussi, Rochdi;; Expert Systems with Applications-Elsevier; Under Review Impact Factor: 4.292 • "Coarse-to-Fine SVD-GAN based Framework for Enhanced Frame Synthesis". Kas, Mohamed; Kajo, Ibrahim; Ruichek, Yassine; Engineering Applications of Artificial Intelligence-Elsevier; Under Review Impact Factor: 3.526 International Conferences: • Kas, Mohamed, et al. "Survey on Local Binary Pattern Descriptors for Face Recognition." Proceedings of the New Challenges in Data Sciences: Acts of the Second Conference of the Moroccan Classification Society. ACM, 2019. • Kas, Mohamed, et al. "Comprehensive Experimental Analysis of Handcrafted Descriptors for Face Recognition." 2018 International Symposium on Advanced Electrical and Communication Technologies (ISAECT). IEEE, 2018. • Mohamed Kas, Youssef El Merabet, Yassine Ruichek, Rochdi Messoussi (11-13 December 2017) Local Directional Multi Radius Binary Pattern: Novel Descriptor for Face Recognition Application; the 9th International Conference on Soft Computing and Pattern Recognition-Advances in Intelligent Systems and Computing -Springer -MIR Labs USA Marrakech Maroc • Slimani, K., Kas, M., El Merabet, Y., Messoussi, R., Ruichek, Y. (2018, March). Facial emotion recognition: A comparative analysis using 22 LBP variants. In Proceedings of the 2nd Mediterranean Conference on Pattern Recognition and Artificial Intelligence (pp. 88-94). ACM.

http://www.itl.nist.gov/iad/humanid/feret/feret_master.html

http://www2.ece.ohio-state.edu/ aleix/ARdatabase.html

ACKNOWLEDGEMENTS

This Ph.D. journey started in 2016 thanks to a collaboration between the University of Technologies Belfort-Montbéliard France and the Faculty of Science of Ibn Tofail University Morocco. All my researches have been carried out jointly between the LASTID

• Evaluation of multiple GAN architectures and Residual CNN face classifiers.

• Collection of a comprehensive dataset to train and evaluate our framework respecting the person-independent constraint. The database will be shared with the stateof-the-art for future research. However, only a few pre-print works are found in arXiv that are not fully peer-reviewed and are evaluated only for the face verification task. The first work was published in 2017 referred to as TP-GAN [START_REF] Huang | Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis[END_REF], which is adopting two generators: Global Pathway (GP) and Local Pathway (LP), devoted to global and local features that are a quite concept used previously for 2D/3D local texture warping [165; 166]. The LP generator takes as input the sub-images centered on the eyes, nose, and mouth to synthesize their corresponding sub-images in frontal view, then performs position aggregation. Meanwhile, the GP processes the whole face image, and its output is concatenated with LP output to form the final synthesized frontal image that will be fed to the discriminator network.

However, this work lacks many explanations about how the LP generator works since the landmarks cannot be all detected in the case of 90 • . Therefore, the number of extracted sub-images is not consistent. Moreover, there is only one discriminator for the two pathways, which is an issue since the outputs of the two pathways are not correlated, and the local one generates more images than the global one. Also, the weights of the local generator cannot be optimized because it has no direct feedback from the discriminator. Besides, the two pathways are paired generators, and each has different inputs from the other. Hence two independent subsets should be prepared for training, and each generator must be trained independently. Also, for evaluating, the authors did not adopt a person-independent protocol to train their TP-GAN. Hence the model will also be optimized on identities from the test set. Moreover, they adopted LightCNN [START_REF] Wu | A light cnn for deep face representation with noisy labels[END_REF], a state-of-the-art model for face verification, to assess the performance on the baseline images and the synthesized ones, which helps to highlight the improvement of the TP-GAN. This work has inspired other researchers to adopt the same philosophy to serve face frontalization. Both DA-GAN [START_REF] Yin | Dual-attention gan for large-pose face frontalization[END_REF] and M2FPA [START_REF] Li | M2fpa: A multi-yaw multi-pitch high-quality dataset and benchmark for facial pose analysis[END_REF] employed the Attention Guided GAN originally proposed in [START_REF] Tang | Attention-guided generative adversarial networks for unsupervised image-to-image translation[END_REF], which introduced the capability of GANs to focus on particular regions of the input image based on heat or binary maps. This ability has been explored by [START_REF] Yin | Dual-attention gan for large-pose face frontalization[END_REF] and [START_REF] Li | M2fpa: A multi-yaw multi-pitch high-quality dataset and benchmark for facial pose analysis[END_REF] in the same way computing the U-Net generator loss according to three generated masks: hair, landmarks, and skin. These masks are obtained from the generated frontal image by a state-of-the-art face parser [START_REF] Luo | Hierarchical face parsing via deep learning[END_REF]. The bottleneck of this approach is the calculated masks since the generated image it is not clear at the early epochs of the training. Hence the parser will not be capable of detecting the three masks, and as a result, the back-propagation loss would not be precise to optimize the generator, which may derive the system into the collapsing mode, a well-known weakness of GANs.

Moreover, [START_REF] Yin | Dual-attention gan for large-pose face frontalization[END_REF] and [START_REF] Li | M2fpa: A multi-yaw multi-pitch high-quality dataset and benchmark for facial pose analysis[END_REF] adopted the same evaluation protocol and LightCNN model as [START_REF] Huang | Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis[END_REF] trying to demonstrate the improvement of their frontalization techniques in case of face verification. However, the reported results demonstrate that the performance of the LightCNN on the baseline images (original) is similar to the one reached on frontalized poses and the GAN improvement is very slight. For example on MultiPie dataset [START_REF] Gross | Multi-pie[END_REF],

the LightCNN reached 97.71% on 45 • pose probe images, while [START_REF] Huang | Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis[END_REF], [START_REF] Yin | Dual-attention gan for large-pose face frontalization[END_REF], and [START_REF] Li | M2fpa: A multi-yaw multi-pitch high-quality dataset and benchmark for facial pose analysis[END_REF] reached 95.38%, 99.53% and 99.15%, respectively. Hence, the improvement was less than 2%, and also TP-GAN impacted the performance of the LightCNN. Furthermore, all three works lack proper comparison of the generated images with ground truth to evaluate the quality of the proposed GANs based on quantitative metrics such as MSE and SSIM that are widely used to assess the performance of GANs. Another issue observed in these works is the inconsistency of the LightCNN baseline results, however, adopting the same evaluation protocol and model weights. Indeed, on MultiPie 90 • test set, the TP-GAN authors reported only 9.00%, but we found that it reached 66.08% as reported in the DA-GAN [START_REF] Yin | Dual-attention gan for large-pose face frontalization[END_REF]. Also, no execution time evaluation was that is important for such applications imposing real-time constraint.

4.3/ PROPOSED GAN-BASED PIFR FRAMEWORK

We propose to deal with the PIFR challenge by exploring the benefits of GANs to generate identity preserving frontal images of poses reaching full profile (90 • ). This section is arranged into subsections to comprehensively present the overall framework, the GAN architectures, and the ResNet face-classification subsystem.

4.3.2/ GENERATIVE ADVERSARIAL NETWORKS

The proposed IFR framework is based on GAN image translation, which translates an input image to a target space based on a trained generator. The GAN consists of a generator and discriminator convolutional networks. The first one is an auto-encoder that tries to encode the profile input image Im P into a reduced latent space z and then generates an image G(Im P ) that looks like the frontal one Im F . The discriminator network helps the generator to synthesize images similar to Im F . The discriminator is a pixel classification network trained to identify real and fake images and gives the feedback to the generator to optimize its weights to generate real images similar to Im F . It supervises the generator and judges the quality of the generated image G(Im P ). Therefore, at each epoch (batch) the discriminator is the first to be optimized, then the generator network.

We summarized in Algorithm 2 the steps of training a GAN. Note that only the trained generator is needed in the test stage, which requires low computational cost. The GAN is similar to any deep neural network in training based on the backpropagation process and objective loss function G * . Referring to the original work [START_REF] Goodfellow | Generative adversarial nets[END_REF], the generator tries to fool the discriminator by producing fake images looking like the real ones, while the discriminator seeks to differentiate the fake from the real images correctly. This process is a min-max game as Eq 4.1 formulates, where the generator tries to minimize the loss After presenting the properties of the considered datasets for making the Combined-PIFR database, we discuss the evaluation protocol that we followed to train and evaluate the proposed PIFR framework, which is composed of a GAN-frontalization subsystem and a CNN based classifier. These two subsystems must be trained respecting the personindependent constraint so the GAN can fairly generate frontal samples of unseen subjects. Figure 4.8 explains in-depth how we performed the dataset partitioning and the amount of data dedicated to train each subsystem and to evaluate the overall framework.

For training the GANs, we selected only one pair of images per subject for each angle The adopted GAN in our PIFR framework supports 4 generator's models and 2 models for the discriminator that are presented previously. In this subsection, we investigate the performance of all the possible combinations of these networks (8 combinations) for HP-GAN and FP-GAN by calculating the MSE and SSIM values between the generated images and their ground-truths.

5.2.3/ EXPERIMENTAL ANALYSIS AND DISCUSSIONS

To show the effectiveness of our proposed framework for person-independent Facial Ex- The proposed OPD-GQMBP descriptor is a generic method defined by the neighborhood size n, which can be seen as a user-specified parameter depending on the needs of the considered application. To find the best value for FER, we run an experiment evaluating the performance of four configurations: n=3, 5, 7, and 9. For each, we evaluated the FER framework on the five datasets using the LOSO protocol. The smaller neighborhood sizes provide less computational cost but with weak discriminative power, and higher ones enhance the discriminative power, but they require more resources to store and classify the extracted features. For example, a neighborhood size of 3 (OPD -GQMBP 3 ) generates only 4 × 2 3 = 32 possible patterns, while a neighborhood size of 5 (OPD -GQMBP 5 ), 7

(OPD -GQMBP 7 ), and 9 (OPD -GQMBP 9 ) produce 128, 512, and 2048 patterns, respectively. Table 5.1 shows the obtained recognition rates resulting from this experiment.

It can be concluded that with a higher neighborhood size, we obtain more discriminative feature extraction. The most effective configuration is n = 7, which managed to reach the top performance on 4 databases. Thus, the 512 generated patterns proved to be enough for characterizing the seven emotional classes. OPD -GQMBP 9 achieved top accuracy of 97.53% on CK+ database outperforming OPD -GQMBP 7 , but it suffered performance drop on the other datasets. Indeed, in some cases, methods that generate a high number of patterns may cause performance drop due to pattern redundancy. It's clear that OPD -GQMBP 3 configuration could not outperform the other ones. However, the recorded accuracies remain prominent regarding the low computation (32 patterns only). The performance of OPD -GQMBP 3 and OPD -GQMBP 5 are very similar with slight variations. We acknowledge that we could not evaluate neighborhood sizes greater than 9 due to the required computation resources (out of memory). Therefore, we adopt the OPD -GQMBP 7 in the rest of the experiments since it corresponds to the best configuration among the tested ones. sions. Also, DenseNet200 could not be ranked among the top-performing ones on any dataset. The light models performed well thanks to the dedicated streams as few layers are required to find the optimal configuration and extract the relevant features.

5.3.1.3/ COMPARISON AGAINST STATE-OF-THE-ART

Table 5.9 illustrates the state-of-the-art work that we outperformed on the three considered benchmarks for dynamic person-independent FER based on the FER survey published in [START_REF] Li | Deep facial expression recognition: A survey[END_REF]. This involves implementing new descriptors, whether color, texture, or shape, to characterize regions and propose new deep learning architectures for the various applications linked to facial analysis. We restrict our focus on face recognition and person-independent facial expressions classification tasks, which are more challenging, especially in unconstrained environments. Our thesis lead to the proposal of many contributions related to facial analysis based on handcrafted and deep architecture. We contributed to face recognition by an effective local features descriptor referred to as Mixed Neighborhood Topology Cross Decoded Patterns (MNTCDP). Our face descriptor relies on a new neighborhood topology and a sophisticated kernel function that help to effectively encode the personrelated features.

We evaluated the proposed MNTCDP-based face recognition system according to well-known and challenging benchmarks of the state-of-the-art, covering individuals' diversity, uncontrolled environment, variable background and lighting conditions.

The achieved results outperformed several state-of-the-art ones. As a second contribution, we handled the challenge of pose-invariant face recognition (PIFR) by developing a Generative Adversarial Network (GAN) based image translation to generate a frontal image corresponding to a profile one.

Hence, this translation makes the recognition much easier since most reference databases include only frontal face samples.

We made an End-to-End deep architecture that contains the GAN for translating profile samples and a ResNet-based classifier to identify the person from its synthesized frontal image. The experiments, which we conducted on an adequate dataset with respect to personindependent constraints between the training and testing, highlight significant improvement in the PIFR performance. Our contributions to the facial expression recognition task cover both static and dynamic-based scenarios. The static-based FER framework relies on extracting textural and shape features from specific face landmarks that carry enough information to detect the dominant emotion. We proposed a new descriptor referred to as Orthogonal and Parallel-based Directions Generic Query Map Binary Patterns (OPD-GQMBP) to efficiently extract emotion-related textural features from 49 landmarks (regions of 32 by 32 pixels). These features are combined with shape ones computed by using Histogram of Oriented Gradients (HOG) descriptor on a binary mask representing the interpolation of the 49 landmarks.

The classification is done through the SVM classifier. The achieved Person-Independent performance on five benchmarks with respect to Leave One Subject Out protocol demonstrated the effectiveness of the overall proposed framework against deep and handcrafted state-of-the-art ones. On the other hand, dynamic FER contribution incorporates Long Term Short Memory (LSTM) deep network to encode the temporal information efficiently with a guiding attention map to focus on the emotion-related landmarks and guarantee the person-independent constraint. We considered four samples as inputs representing the evolution of the emotion to its peak. Each sample is encoded through a ResNetbased stream, and the four streams are joined by an LSTM block that predicts the dominant emotion. The experiments conducted on three datasets for dynamic FER showed that the proposed deep CNN-LSTM architecture outperforms the state-of-the-art. Notre descripteur de visage repose sur une nouvelle topologie de voisinage et une fonction de noyau avancée permettant en encodage efficace des caractéristiques liées à la personne. Nous avons évalué le système de reconnaissance faciale proposé à base du MNTCDP sur des bases de données connues de l'état de l'art, disposant de challenges, couvrant la diversité des individus, environnement non contrôlé, des conditions de fond et d'éclairage variables. Les résultats obtenus ont dépassé plusieurs résultats de l'état de l'art. Pour la deuxième contribution, nous avons relevé le défi de la reconnaissance faciale invariante aux poses (PIFR) en développant une méthode de génération d'images basée sur le Generative Adversarial Network, afin de générer une image frontale correspondant à une image en profil. Cette transformation rend la reconnaissance beaucoup plus facile puisque la plupart des bases de données de référence n'incluent que des échantillons de face frontale. Nous avons créé une architecture profonde de bout en bout, composé du GAN pour la génération des échantillons de profil et un classificateur basé sur ResNet pour l'identification de la personne à partir de son image frontale synthétisée. Les expériences, que nous avons menées sur une base de données adéquate, notamment en termes de chevauchement des individus entre la base de l'apprentissage et celle de l'évaluation, mettent en évidence une grande amélioration des performances du PIFR grâce à la génération d'images frontales du GAN. Nos contributions à la tâche de reconnaissance des expressions faciales (FER) couvrent à la fois des scénarios statiques (une image) et dynamiques (plusieurs images). Notre architecture pour FER avec le mode statique repose sur l'extraction de caractéristiques de texture et de forme à partir de points de repère du visage spécifiques qui présentent suffisamment d'informations pour détecter l'émotion dominante.

Nous avons proposé un descripteur appelé Orthogonal and Parallel-based Directions Generic Query Map Binary Patterns pour extraire efficacement les caractéristiques texturales liées aux émotions à partir de 49 points de repère. Ces caractéristiques sont combinées avec celles à base de forme calculées à l'aide du descripteur HOG sur un masque binaire représentant l'interpolation des 49 points. La classification est réalisée via le SVM. Less performances obtenues sur cinq bases de données avec le protocole Leave One Subject Out ont démontré l'efficacité de l'architecture proposée par rapport à l'état de l'art. D'autre part, notre contribution relative à la FER avec le mode dynamique intègre un réseau LSTM pour encoder avec précision les informations temporelles avec un masque d'attention permettant de se concentrer sur les repères liés aux émotions et garantir la robustesse de la reconnaissance. Nous avons considéré quatre échantillons comme entrées représentant l'évolution de l'émotion jusqu'à son pic. Chaque échantillon est codé via une branche CNN et les quatre branches sont jointes par un bloc LSTM qui prédit l'émotion dominante. Les expériences menées sur trois bases de données pour FER dynamique ont montré que l'architecture CNN-LSTM profonde proposée dépasse l'état de l'art.

Résumé