Di Huang
email: di.huang@ec-lyon.fr

Caifeng Shan
email: caifeng.shan@philips.com

Mohsen Ardabilian

Yunhong Wang

Liming Chen

Mohsen Ardebilian

Keywords: Local Binary Patterns (LBP), local features, face detection, face recognition, facial expression analysis

I. INTRODUCTION

uring the last few years, Local Binary Patterns (LBP) [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] has aroused increasing interest in image processing and computer vision. As a non-parametric method, LBP summarizes local structures of images efficiently by comparing each pixel with its neighboring pixels. The most important properties of LBP are its tolerance regarding monotonic illumination changes and its computational simplicity. LBP was originally proposed for texture analysis [START_REF] Ojala | A comparative study of texture measures with classification based on featured distribution[END_REF], and has proved a simple yet powerful approach to describe local structures. It has been extensively exploited in many applications, for instance, face image analysis [START_REF] Ahonen | Face recognition with local binary patterns[END_REF]- [START_REF] Hadid | A discriminative feature space for detecting and recognizing faces[END_REF], image and video retrieval [START_REF] Huijsmans | Content-based indexing performance: a class size normalized precision, recall, generality evaluation[END_REF]- [START_REF] Grangier | A discriminative kernel-based approach to rank images from text queries[END_REF], environment modeling [START_REF] Ali | Visual tree detection for autonomous navigation in forest environment[END_REF]- [START_REF] Nanni | Ensemble of multiple pedestrian representations[END_REF], visual inspection [START_REF] Mäenpää | Optimising colour and texture features for real-time visual inspection[END_REF]- [START_REF] Turtinen | Visual characterization of paper using Isomap and local binary patterns[END_REF], motion analysis [START_REF] Heikkilä | A texture-based method for modeling the background and detecting moving objects[END_REF]- [START_REF] Kellokumpu | Human activity recognition using a dynamic texture based method[END_REF], biomedical and aerial image analysis [START_REF] Oliver | False positive reduction in mammographic mass detection using local binary patterns[END_REF]- [START_REF] Kluckner | A 3D teacher for car detection in aerial images[END_REF], remote sensing [START_REF] Lucieer | Multivariate texture-based segmentation of remotely sensed imagery for extraction of objects and their uncertainty[END_REF], so forth (see a comprehensive bibliography of LBP methodology online [16]).

LBP-based facial image analysis has been one of the most popular and successful applications in recent years. Facial image analysis is an active research topic in computer vision, with a wide range of important applications, e.g., human-computer interaction, biometric identification, surveil-lance and security, and computer animation etc. LBP has been exploited for facial representation in different tasks containing face detection [START_REF] Hadid | A discriminative feature space for detecting and recognizing faces[END_REF], [START_REF] Jin | Face detection using improved LBP under Bayesian framework[END_REF]- [START_REF] Zhang | Spatial histogram features for face detection in color images[END_REF], face recognition [START_REF] Ahonen | Face description with local binary patterns: application to face recognition[END_REF]- [START_REF] Zhao | LBP discriminant analysis for face verification[END_REF], facial expression analysis [START_REF] Shan | Facial expression recognition based on local binary patterns: a comprehensive study[END_REF]- [START_REF] Gritti | Local features based facial expression recognition with face registration errors[END_REF], demographic (gender, race, age, etc.) classification [START_REF] Sun | Gender classification based on boosting local binary pattern[END_REF]- [START_REF] Yang | Demographic classification with local binary patterns[END_REF], and other related applications [START_REF] Gao | Standardization of face image sample quality[END_REF]- [START_REF] Ma | Robust head pose estimation using LGBP[END_REF]. The development of LBP methodology can be well illustrated in facial image analysis, and most of its recent variations are proposed in this area.

Some brief surveys on image analysis [START_REF] Pietikäinen | Image analysis with local binary patterns[END_REF] or face analysis [START_REF] Marcel | On the recent use of local binary patterns for face authentication[END_REF]- [START_REF] Hadid | The local binary pattern and its applications to face analysis[END_REF] using LBP were given, but all these works discussed limited papers of the literature, and many new related methods have appeared in more recent years. In this paper, we present a comprehensive survey of the LBP methodology, including its recent variations and LBP-based feature selection, as well as the application to facial image analysis. To the best of our knowledge, this paper is the first survey that extensively reviews LBP methodology and its application to facial image analysis, with more than 100 related literatures reviewed.

The remainder of this paper is organized as follows. The LBP methodology is introduced in Section II. Section III presents the recent variations of LBP. LBP-based feature selection methods are discussed in section IV. Section V describes different facets of its applications on facial image analysis. Finally, Section VI concludes the paper.

II. LOCAL BINARY PATTERNS

The original LBP operator labels the pixels of an image with decimal numbers, called Local Binary Patterns or LBP codes, which encode the local structure around each pixel. It proceeds thus, as illustrated in Fig. 1: Each pixel is compared with its eight neighbors in a 3x3 neighborhood by subtracting the center pixel value; The resulting strictly negative values are encoded with 0 and the others with 1; A binary number is obtained by concatenating all these binary codes in a clockwise direction starting from the top-left one and its corresponding decimal value is used for labeling. The derived binary numbers are referred to as Local Binary Patterns or LBP codes.

Local Binary Patterns and Its Application to

Facial Image Analysis: A Survey Di Huang, Caifeng Shan, Mohsen Ardebilian, Yunhong Wang, and Liming Chen D One limitation of the basic LBP operator is that its small 3x3 neighborhood cannot capture dominant features with large scale structures. To deal with the texture at different scales, the operator was later generalized to use neighborhoods of different sizes [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF]. A local neighborhood is defined as a set of sampling points evenly spaced on a circle which is centered at the pixel to be labeled, and the sampling points that do not fall within the pixels are interpolated using bilinear interpolation, thus allowing for any radius and any number of sampling points in the neighborhood. Fig. 2 shows some examples of the extended LBP operator, where the notation (P, R) denotes a neighborhood of P sampling points on a circle of radius of R. Fig. 2. Examples of the extended LBP operator [START_REF] Ahonen | Face description with local binary patterns: application to face recognition[END_REF]: the circular (8, 1), (16,[START_REF] Ojala | A comparative study of texture measures with classification based on featured distribution[END_REF], and (24, 3) neighborhoods.

Formally, given a pixel at (x c , y c ), the resulting LBP can be expressed in decimal form as:

where i c and i P are respectively gray-level values of the central pixel and P surrounding pixels in the circle neighborhood with a radius R, and function s(x) is defined as:

1 0 ( ) 0 0 if x s x if x ≥ ⎧ = ⎨ < ⎩ (2)
By the definition above, the basic LBP operator is invariant to monotonic gray-scale transformations preserving pixel intensity order in the local neighborhoods. The histogram of LBP labels calculated over a region can be exploited as a texture descriptor.

The operator LBP (P, R) produces 2 p different output values, corresponding to 2 p different binary patterns formed by P pixels in the neighborhood. If the image is rotated, these surrounding pixels in each neighborhood will move correspondingly along the perimeter of the circle, resulting in a different LBP value, except patterns with only 1s and 0s. In order to remove rotation effect, a rotation-invariant LBP is proposed in [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF]:

, , min{ ( , ) | 0, 1, ..., 1}

ri P R P R LBP ROR LBP i i P = = - (3)
where ROR(x, i) performs a circular bit-wise right shift on the P-bit number x i times. The LBP ri (P, R) operator quantifies occurrence statistics of individual rotation invariant patterns corresponding to certain micro-features in the image; hence, the patterns can be considered as a feature detector [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF]. However, in [START_REF] Pietikäinen | Rotation-Invariant Texture Classification using feature distributions[END_REF], it was shown that such a rotation-invariant LBP operator does not necessarily provide discriminative information, since the occurrence frequencies of the individual patterns incorporated in LBP ri (P, R) vary greatly and the crude quantization of the angular spaces at 45° intervals.

It has been shown that certain patterns contain more information than others [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF]. It is possible to use only a subset of 2 p binary patterns to describe the texture of images. Ojala et al. named these patterns uniform patterns, denoted LBP U2 (P, R) . A local binary pattern is called uniform if it contains at most two bitwise transitions from 0 to 1 or vice versa when the corresponding bit string is considered circular. For instance, 00000000 (0 transitions) and 01110000 (2 transitions) are both uniform whereas 11001001 (4 transitions) and 01010011 (6 transitions) are not. It is observed that the uniform patterns account for around 90% of all the patterns in a (8, 1) neighborhood and around 70% in a (16, 2) neighborhood in texture images [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF]. A similar experiment was conducted on the FERET database, and it was found that 90.6% of the patterns in a (8, 1) neighborhood and 85.2% in a (8, 2) neighborhood are uniform [START_REF] Ahonen | Face description with local binary patterns: application to face recognition[END_REF]. More recently, Shan and Gritti [START_REF] Shan | Learning discriminative LBP-histogram bins for facial expression recognition[END_REF] verified validity of uniform patterns for representing faces from the viewpoint of machine learning. Specifically, they applied AdaBoost to select the discriminative patterns for facial expression recognition, and their experiments demonstrated that, using LBP [START_REF] Nanni | Ensemble of multiple pedestrian representations[END_REF][START_REF] Ojala | A comparative study of texture measures with classification based on featured distribution[END_REF] operator, 91.1% of these selected patterns are uniform. Accumulating the non-uniform patterns into a single bin yields an LBP operator with less than 2 p labels. For example, the number of labels with the neighborhood of 8 pixels is 256 for the standard LBP but only 59 for LBP U2 .

It should be noted that, at about the same time that the original LBP operator was proposed, Zabih and Woodfill introduced a Census Transform (CT) method [START_REF] Zabih | Non-parametric local transforms for computing visual correspondence[END_REF] which is very similar to LBP. CT also maps the local neighborhood surrounding a pixel onto a binary string, and the only difference between LBP and CT is the opposite order of bit string. Later, CT and its variations were exploited for facial image analysis [START_REF] Froba | Face detection with the modified census transform[END_REF]- [START_REF] Wang | Robust real-time face detection with skin color detection and the modified census transform[END_REF].

The C/C++ and Matlab implementations of the LBP operator can be found online [46].

III. RECENT VARIATIONS OF LBP

LBP methodology has been developed recently with plenty of variations for improved performance in different applications. These variations focus on different aspects of the original LBP operator: (1) improvement of its discriminative capability;

(2) enhancement of its robustness; (3) selection of its neighborhood; (4) extension to 3D data; (5) combination with other approaches. In this section, we review recent variations of LBP (See Table . 1 for the overview).

A. Enhancing the discriminative capability

The LBP operator defines a certain number of patterns for describing the local structures. To enhance their discriminative capability, more patterns or information could be encoded. In [START_REF] Jin | Face detection using improved LBP under Bayesian framework[END_REF], Jin et al. modified the LBP operator to describe more local structure information under certain circumstances. Specifically, they proposed an Improved LBP (ILBP) which compares all the pixels (including the central pixel) with the mean intensity of all the pixels in the patch (as shown in Fig. 3). For instance, the LBP [START_REF] Nanni | Ensemble of multiple pedestrian representations[END_REF][START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] operator produces only 256 (2 8 ) in a 3x3 neighborhood, while ILBP has 511 patterns (2 9 -1, as all zeros and all ones are the same). Later, ILBP was extended to use the neighborhoods of any size instead of the original 3x3 patch [START_REF] Jin | Learning local descriptors for face detection[END_REF]. Almost at the same time, a similar strategy was introduced to extend CT to Modified CT [START_REF] Froba | Face detection with the modified census transform[END_REF], named Modified LBP in [START_REF] Ruiz-Del-Solar | Illumination compensation and normalization in eigenspace-based face recognition: a comparative study of different pre-processing approaches[END_REF]. A Mean LBP is presented in [START_REF] Bai | A hierarchical face recognition method based on local binary pattern[END_REF], similar to ILBP, but without considering the

Subsection

Variations Properties Year & Reference

A: Enhancing the discriminative ability

Improved LBP (Mean LBP)

Consider the effects of central pixels; present complete structure patterns. 2004 [START_REF] Jin | Face detection using improved LBP under Bayesian framework[END_REF], 2005 [START_REF] Ruiz-Del-Solar | Illumination compensation and normalization in eigenspace-based face recognition: a comparative study of different pre-processing approaches[END_REF] 2008 [START_REF] Bai | A hierarchical face recognition method based on local binary pattern[END_REF] Hamming LBP Incorporate non-uniform patterns into uniform patterns 2007 [START_REF] Yang | A LBP-based face recognition method with Hamming distance constraint[END_REF] Extended LBP Discriminate the same local binary patterns; cause high dimensionality. 2007 [START_REF] Huang | A robust method for near infrared face recognition based on extended local binary pattern[END_REF][START_REF] Huang | Combining statistics of geometrical and correlative features for 3D face recognition[END_REF] Completed LBP Include both the sign and the magnitude information of the given local region 2010 [START_REF] Guo | A completed modeling of local binary pattern operator for texture classification[END_REF] B: Improving the robustness Local Ternary Patterns Bring in new threshold; no longer strictly invariant to gray-level transformation. 2007 [START_REF] Tan | Enhanced local texture feature sets for face recognition under difficult lighting conditions[END_REF] Soft LBP Not invariant to monotonic grayscale changes; cause high computational complexity. 2007 [START_REF] Ahonen | Soft histograms for local binary patterns[END_REF] C: Choosing the neighborhood

Elongated LBP Extract the anisotropic information and lose anisotropic information; not invariant to rotation. 2007 [START_REF] Liao | Face recognition by using elongated local binary patterns with average maximum distance gradient magnitude[END_REF] Multi-Block LBP Capture micro-and macro-structure information 2007 [START_REF] Zhang | Face detection based on Multi-Block LBP representation[END_REF][START_REF] Liao | Learning multi-scale block local binary patterns for face recognition[END_REF] Three/Four Patch LBP Encode patch type of texture information 2008 [START_REF] Wolf | Descriptor Based Methods in the Wild[END_REF] D: Extending to 3D 3D LBP Extend LBP to 3D volume data 2007 [START_REF] Fehr | Rotational invariant uniform local binary patterns for full 3D volume texture analysis[END_REF], 2008 [START_REF] Paulhac | Comparison between 2D and 3D local binary pattern methods for characterization of three-dimensional textures[END_REF] Volume LBP (LBP-TOP) Describe dynamic texture; cause high dimensionality. 2007 [START_REF] Zhao | Experiments with facial expression recognition using spatiotemporal local binary patterns[END_REF][START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF] E: Combining with other features LBP and Gabor Wavelet Combine advantages of Gabor and LBP; increase time cost and cause high dimensionality.

2005 [START_REF] Zhang | Local Gabor Binary Pattern Histogram Sequence (LGBPHS): A novel non-statistical model for face representation and recognition[END_REF][START_REF] He | An enhanced LBP feature based on facial expression recognition[END_REF], 2006 [START_REF] Shan | Ensemble of piecewise FDA based on spatial histograms of local (Gabor) binary patterns for face recognition[END_REF] 2007 [START_REF] Yan | Exploring feature descriptors for face recognition[END_REF][START_REF] Tan | Fusing Gabor and LBP feature sets for kernel-based face recognition[END_REF], 2008 [START_REF] Singh | Integrated multilevel image fusion and match score fusion of visible and infrared face images for robust face recognition[END_REF][START_REF] Lei | Gabor volume based local binary pattern for face representation and recognition[END_REF] LBP and SIFT Bring in the advantages of SIFT; reduce feature vector length 2006 [START_REF] Heikkilä | Description of interest regions with center-symmetric local binary patterns[END_REF], 2009 [START_REF] Heikkilä | Description of interest regions with local binary patterns[END_REF] 2010 [START_REF] Huang | 3D face recognition using distinctiveness enhanced facial representations and local feature hybrid matching[END_REF] LBP Histogram Fourier Obtain rotation invariance globally for the whole region 2009 [START_REF] Ahonen | Rotation invariant image description with local binary pattern histogram fourier features[END_REF] Table .1 List of recent LBP variations.

central pixels.

Fig. 3. An example of ILBP operator [START_REF] Jin | Face detection using improved LBP under Bayesian framework[END_REF].

In [START_REF] Yang | A LBP-based face recognition method with Hamming distance constraint[END_REF], Yang and Wang proposed Hamming LBP to improve the discriminative ability of the original LBP. They reclassified non-uniform patterns based on Hamming distance, instead of collecting them into a single bin as LBP u2 does. In the Hamming LBP, these non-uniform patterns are incorporated into existing uniform patterns by minimizing the Hamming distance between them. E.g., the non-uniform pattern (10001110) 2 is converted into the uniform one (10001111) 2 since their Hamming distance is one. When several uniform patterns have the same Hamming distance with a non-uniform pattern, the one with the minimum Euclidian distance will be selected.

The Extended LBP (ELBP) [START_REF] Huang | A robust method for near infrared face recognition based on extended local binary pattern[END_REF]- [START_REF] Huang | Combining statistics of geometrical and correlative features for 3D face recognition[END_REF] is another approach to improve the discriminative capability of LBP. The ELBP operator not only performs binary comparison between the central pixel and its neighbors, but also encodes their exact gray-value differences using some additional binary units. Specifically, the ELBP feature consists of several LBP codes at multiple layers which encode the gray-value difference (GD) between the central pixel and its neighboring pixels. As shown in Fig. 4, the first layer of ELBP is actually the original LBP code encoding the sign of GD. The following layers of ELBP then encode the absolute value of GD. Basically, each absolute GD value is first encoded in its binary representation, and then all the binary values at a given layer result in an additional local binary pattern. For example, in Fig. 4, the first layer is the original LBP code that encodes the sign of GD, thus yielding a decimal number of 211 from its binary form (11010011) 2 . The absolute values of GD, i.e. 1, 5, 3, 2, 1, 2, 3, 0, are first encoded in their binary numbers: (001) 2 , (101) 2 , (011) 2 , (010) 2 , …, etc. Using a same weight scheme of LBP on all the binary bits, its ELBP code of the corresponding layer can be generated, e.g., L 2 is composed of (01000000) 2 and its decimal value is 64; L 3 is composed of (00110110) 2 and its decimal value is 54; finally L 4 is composed of (11101010) 2 and its decimal value is 234. As a result, when describing similar local textures, although the first layer LBP is not discriminative enough, the information encoded in the other additional layers can be utilized to distin-guish them. Its downside is that ELBP greatly increases feature dimensionality.

More recently, Guo et al. proposed a Complete LBP (CLBP) [START_REF] Guo | A completed modeling of local binary pattern operator for texture classification[END_REF], which, in our opinion, is quite similar with ELBP. CLBP also includes both the sign and the gray-value differences between a given central pixel and its neighbors in order to improve the discriminative power of the original LBP operator. Unlike the binary bit coding strategy used by ELBP, CLBP compares the absolute value of GD with the given central pixel again to generate an LBP liked code.

B. Improving the robustness

LBP is sensitive to noise since the operator thresholds exactly at the value of central pixel. To address this problem, Tan and Triggs [START_REF] Tan | Enhanced local texture feature sets for face recognition under difficult lighting conditions[END_REF] extended the original LBP to a version with 3-value codes, called Local Ternary Patterns (LTP). In LTP, indicator s(x) in (1) is replaced by: ( )

1 , , 0
1 n c n c n c n c i i t s i i t i i t i i t ≥ + ⎧ ⎪ = -< ⎨ ⎪ - ≤ - ⎩ (4)
where t is a user-specified threshold. The LTP codes are more resistant to noise, but no longer strictly invariant to gray-level transformations. A coding scheme is used to split each ternary pattern into two parts: the positive one and the negative one, as illustrated in Fig. 5. One problem of LTP is that threshold t needs to be set, which is not simple. Fig. 5. An example of the LTP operator [START_REF] Tan | Enhanced local texture feature sets for face recognition under difficult lighting conditions[END_REF].

The Soft LBP (SLBP) was introduced in [START_REF] Ahonen | Soft histograms for local binary patterns[END_REF], which em-ploys two fuzzy membership functions instead of (2) for thresholding:

1, 0 ( ) 0.5 0.5

1 d x d x s x d x d d x d < - ⎧ ⎪ ⎪ = + -≤ ≤ ⎨ ⎪ > ⎪ ⎩ (5) 0, 1,
( ) 1 ( )

d d s x s x = - (6)
Parameter d controls the amount of fuzzification that the fuzzy function performs. When the local neighborhood consists of P sampling points, the histogram with a uniform pattern operator has bins numbered 0, 1, …, 2 P -1. The contribution of a single pixel (x c , y c ) to bin h of the histogram is:

SLBP x y h b h s i i b h s i i

- = = -+ - - ∏ (7)
where b p (h)∈{0,1} denotes the numerical value of the p th bit of binary representation of h.

With SLBP, one pixel contributes to more than one bin, but the sum of the contributions of the pixel to all bins is always 1. SLBP enhances the robustness in the sense that a small change in the input image causes only a small change in output. But, it loses the invariance to monotonic variations and also increases the computation complexity. As with LTP, a proper value of d should be set.

C. Choosing the neighborhood

The choice of an appropriate neighborhood for LBP-based techniques has a significant impact on the final performance. It involves the number of sampling points, the distribution of the sampling points, the shape of the neighborhood, and the size of the neighborhood as well. Fig. 6. Two examples of Elongated LBP operator [START_REF] Liao | Face recognition by using elongated local binary patterns with average maximum distance gradient magnitude[END_REF].

Neighboring pixels in the original LBP are defined on a circle. Liao and Chung [START_REF] Liao | Face recognition by using elongated local binary patterns with average maximum distance gradient magnitude[END_REF] argued that the main reason to define the neighborhood in such an isotropic manner is to obtain rotation invariance for texture description. But this is not suitable for all applications; on the contrary, the anisotropic information could also be an important feature. As a result, they proposed Elongated LBP with neighboring pixels lying on an ellipse. Fig. 6 gives two examples of the Elongated LBP, where A, B denote the long axis and short axis respectively, and m is the number of neighboring pixels. Following original LBP, bilinear interpolation technique is adopted for neighboring pixels that do not fall exactly at the pixels. The Elongated LBP operator could be rotated around the central pixel, with a specific angle to characterize elongated local structures in different orientations, to achieve multi-orientation analysis.

In order to capture not only the microstructures but also the macrostructures, Li et al. [18] [55] proposed Multi-Block LBP (MB-LBP) which, instead of comparing pixels, compares average intensities of neighboring sub-regions. The original LBP can be regarded as a special case of the MB-LBP. Fig. 7 shows an example of MB-LBP, where each sub-region consists of six pixels. The sub-region can either be a rectangle or a square. The average intensities over the blocks can be computed efficiently by using summed-area table [START_REF] Simard | Boxlets: a fast convolution algorithm for signal processing and neural networks[END_REF] or integral image. A similar scheme is introduced in [57]: Three-Patch LBP (TP-LBP) and Four-Patch LBP (FP-LBP) are proposed to compare distances between the whole blocks (patches) concerned, instead of single pixel [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] or average intensity in [START_REF] Liao | Learning multi-scale block local binary patterns for face recognition[END_REF], and any distance function can be used (e.g., L2 norm of their gray level differences).

D. Extending to 3D LBP

Several researchers have been trying to extend the LBP from 2D plane to 3D volume [START_REF] Zhao | Experiments with facial expression recognition using spatiotemporal local binary patterns[END_REF] [58]- [START_REF] Paulhac | Comparison between 2D and 3D local binary pattern methods for characterization of three-dimensional textures[END_REF] [60]; however, it is not so straightforward as it appears at first glance. There are two difficulties: first, equidistant sampling on a sphere is a difficult job, and second, it is also difficult to set an order to those sampling points, which is important to achieve rotation invariance. To endow the LBP with the ability to capture dynamic texture information, in [START_REF] Zhao | Experiments with facial expression recognition using spatiotemporal local binary patterns[END_REF] and [START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF], Zhao and Pietikäinen extended the LBP neighborhood from 2D plane to 3D space. The operator is named as Volume LBP (VLBP or 3D-LBP). VLBP combines motion and appearance information, and can thus be used for analyzing image sequences or videos. It should be noted that this approach makes use of dynamic texture analysis of 2D time series instead of full 3D volumetric data. The VLBP features are not only insensitive to translation and rotation (toward rotations around the z axis), but robust to monotonic grayscale changes as well. Compared with LBP (P, R) , VLBP (L, P, R) takes time domain into account, and the parameter L denotes the length of the time interval. From a small local neighborhood in volume, comparing neighboring pixels with the central pixel, a number of binary units are obtained, and the weights for these units are given as a spiral line (see Fig. 8). In order to make VLBP computationally simple and easy to extend, only co-occurrences on three separate planes are considered. The textures are modeled with the concatenated LBP histograms extracted from Three Orthogonal Planes XY, XT, and YT, and thus this simpler version of VLBP is named LBP-TOP. The traditional circular sampling is replaced by an ellipse so that different radius parameters can be set in the space and time domain.

In [START_REF] Fehr | Rotational invariant uniform local binary patterns for full 3D volume texture analysis[END_REF], Fehr exploited the Spherical Harmonic transform to produce an orthogonal basis on the 2-Sphere, and then compute the LBP features in the frequency domain. This method overcomes both problems mentioned above. Paulhac et al. proposed another solution to apply LBP to 3D [START_REF] Paulhac | Comparison between 2D and 3D local binary pattern methods for characterization of three-dimensional textures[END_REF]. They used a number of circles to represent the sphere, adding the parameter S, thus the operator denotes LBP (S, P, R) (see Fig. 8) and they also defined the uniform rule as in 2D. This method causes the problem that different textures could have the same LBP description.

E. Combining with other features

As a method for describing local features, LBP can be combined with other approaches. For example, a set of approaches were proposed to combine Gabor wavelets [START_REF] Marcelja | Mathematical description of the responses of simple cortical cells[END_REF]- [START_REF] Wiskott | Face recognition by elastic bunch graph matching[END_REF] and LBP features using different methods. It was concluded in [START_REF] Yan | Exploring feature descriptors for face recognition[END_REF]- [START_REF] Singh | Integrated multilevel image fusion and match score fusion of visible and infrared face images for robust face recognition[END_REF] that Gabor wavelet and LBP based features are mutually complementary, because LBP captures the local appearance detail; while Gabor-wavelets extract shape information over a broader range of scales. A simple fusion strategy is to first extract Gabor and LBP features in the parallel way, and then fuse two kinds of features on feature level, matching score level, or decision level [START_REF] Tan | Fusing Gabor and LBP feature sets for kernel-based face recognition[END_REF]- [START_REF] Singh | Integrated multilevel image fusion and match score fusion of visible and infrared face images for robust face recognition[END_REF].

Another combination way is the serial strategy which consists in first applying Gabor filters and then LBP to the raw image [24] [67]. The Multiple Gabor feature maps (GFM) are computed by convolving input images with multi-scale and multi-orientation Gabor filters. Each GFM is divided into small non-overlapped regions from which LBP histograms are extracted and finally concatenated into a single feature histogram. Multi-resolution Histograms of Local Variation Patterns (MHLVP) [START_REF] Zhang | Multi-resolution Histograms of Local Variation Patterns (MHLVP) for robust face recognition[END_REF] as well as Local Gabor Binary Pattern Histogram (LGBPH) [START_REF] Zhang | Local Gabor Binary Pattern Histogram Sequence (LGBPHS): A novel non-statistical model for face representation and recognition[END_REF]- [START_REF] Yao | Domain-partitioning rankboost for face recognition[END_REF], have been proposed based on such a procedure. Recently, He et al. [START_REF] He | An enhanced LBP feature based on facial expression recognition[END_REF] proposed a similar serial method using both wavelets and LBP, which firstly uses wavelets to decompose raw images into four frequency images, i.e., low frequency, horizontal high frequency, vertical high frequency and diagonal high frequency, as the inputs of the original LBP.

Motivated by LBP-TOP and LGBPH, Lei et al. proposed to construct a 3rd-order Gabor image based volume and then apply the original LBP to three orthogonal planes, to extract the discriminative information not only in the spatial domain, but also in the Gabor frequency and orientation domains [START_REF] Lei | Gabor volume based local binary pattern for face representation and recognition[END_REF]. To reduce the computation complexity, a GV-LBP operator is introduced to describe the variations in spatial, frequency and orientation domains simultaneously by defining the orientation and scale neighboring points in different Gabor images. A Center-Symmetric LBP (CS-LBP) [START_REF] Heikkilä | Description of interest regions with center-symmetric local binary patterns[END_REF]- [START_REF] Heikkilä | Description of interest regions with local binary patterns[END_REF], was proposed by only comparing pairs of neighboring pixels which are in the same diameter of the circle. This variation combines the LBP operator with the SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] definition and thus produces fewer binary units than the original LBP does. The difference between CS-LBP and LBP with 8 neighboring pixels is given in Fig. 10. Later, in [START_REF] Fu | Centralized binary patterns embedded with image Euclidean distance for facial expression recognition[END_REF], Fu and Wei introduced Centralized Binary Patterns (CBP), making a small modification to this scheme; by comparing the central pixel with the mean of all the pixels in the neighborhood to produce an additional binary unit assigned the largest weight to strengthen its effect. More recently, Huang et al. [START_REF] Huang | 3D face recognition using distinctiveness enhanced facial representations and local feature hybrid matching[END_REF] proposed to combine the LBP and SIFT approach in a serial way: first adopts the original LBP operator with different parameter settings to extract MS-LBP images based on which SIFT is exploited to do matching.

Ahonen et al. proposed an approach, named LBP Histogram Fourier features (LBP-HF) [START_REF] Ahonen | Rotation invariant image description with local binary pattern histogram fourier features[END_REF], to combine LBP and the Discrete Fourier Transform (DFT). Unlike the existing local rotation-invariant LBP features, the LBP-HF descriptor is produced by computing an LBP histogram over the whole region and then constructing rotationally invariant features from the histogram with DFT. That means, rotation invariance is obtained globally, and features are thus invariant to rotations of the whole input signal, but they still retain information about the relative distribution of different orientations of uniform LBP.

IV. LBP FEATURE SELECTION

In most existing work, the input image is divided into small regions, from which LBP histograms are extracted, and the local histograms are further concatenated into a spatially enhanced feature vector of the dimensionality of O ( 103 ). Moreover, some recent variations even increase the feature vector length dramatically, such as Extended LBP, VLBP and Gabor Wavelets based LBP. It is believed that the derived LBP-based feature vector provides an over-complete representation with redundant information [START_REF] Zhang | Boosting Local Binary Pattern-based face recognition[END_REF], which could be reduced to be more compact and discriminative. Furthermore, when building real-time systems, it is also desired to have LBP-based representation with reduced feature length. For all the reasons, the problem of LBP feature selection has recently been addressed in many literatures. We classify these techniques into two categories: the first one is to reduce the feature length based on some rules (like uniform patterns), whilst the other one exploits feature selection techniques to choose the discriminative patterns. Both streams have their own merits and drawbacks: the first one is simple, but has limited feature selection ability; on the contrary, the second has a better feature selection capacity, but usually requires off-line training that could be computationally expensive.

A. Rule-based Strategy

Uniform pattern is an effective rule to select LBP features, and it has been widely adopted in existing work. There are also other rules which could be used. For instance, Lahdenoja et al. [START_REF] Lahdenoja | Reducing the feature vector length in local binary pattern based face recognition[END_REF] proposed a symmetry level scheme for uniform patterns to further reduce the length of LBP feature vectors. The symmetry level L sym of each pattern is defined as the minimum of the total number of ones and zeros in that pattern. For example, L sym of both patterns (00111111) 2 and (00011000) 2 are 2. The symmetry level is rotation invariant according to the definition. The most symmetric pattern contains the same number of ones and zeros, indicating a symmetric edge, while the patterns with the lowest symmetry level are the ones consisting of only ones or zeros. It is claimed that the patterns with high symmetry level occur more frequently in the images with more discriminative power [START_REF] Lahdenoja | Reducing the feature vector length in local binary pattern based face recognition[END_REF]. This conclusion is supported by experiments: the comparative performance was obtained using only the patterns of high symmetry level, but the length of feature vectors was reduced by a quarter.

B. Boosting LBP Features

Boosting learning [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF] provides an effective way for feature selection. In [START_REF] Zhang | Boosting Local Binary Pattern-based face recognition[END_REF], by shifting and scaling a sub-window over face image, many more sub-regions are obtained to extract local LBP histograms; the distance between the corresponding histograms of two images is utilized as the discriminative fea-ture, and AdaBoost is used to learn a few of the most efficient features. Compared with [START_REF] Ahonen | Face recognition with local binary patterns[END_REF], the approach achieves slightly better performance but with fewer histograms computed from the local regions. A similar approach was also adopted in [START_REF] Shan | Facial expression recognition based on local binary patterns: a comprehensive study[END_REF]. In these works, the n-th bin of a local histogram is utilized as a whole for region description, and feature selection is performed at region level. AdaBoost can also be exploited to learn the discriminative bins of an LBP histogram [START_REF] Shan | Learning discriminative LBP-histogram bins for facial expression recognition[END_REF], since not all the bins are necessary to supply useful information. Their experiments illustrate that the selected LBP bins provide a much more compact representation with a greatly reduced length of feature vector, while producing better performance. AdaBoost has been widely adopted for LBP feature selection in various tasks [START_REF] Zhang | Face detection based on Multi-Block LBP representation[END_REF]

C. LBP Subspace Learning

Subspace learning (or Dimensionality Reduction) [START_REF] Shakhnarovich | Face recognition in subspaces[END_REF] maps data set from a high dimensional space to a lower dimensional space, and thus can be applied to LBP-based features to derive a low-dimensional compact representation. For example, Chan et al. introduced Linear Discriminant Analysis (LDA) to project high-dimensional Multi-Scale LBP features into a discriminant space [START_REF] Chan | Multi-scale local binary pattern histograms for face recognition[END_REF], and the same scheme was later exploited with the Multi-Spectral LBP features calculated from color images [START_REF] Chan | Multispectral local binary pattern histogram for component-based color face verification[END_REF]. To deal with the small sample size problem of LDA, Shan et al. [START_REF] Shan | Ensemble of piecewise FDA based on spatial histograms of local (Gabor) binary patterns for face recognition[END_REF] proposed an Ensemble of Piecewise LDA, which partitions the entire LGBP feature vector into segments; then applies LDA to each segment separately. Their approach was verified to be more effective than applying LDA to high-dimensional holistic feature vector. Combining Gabor wavelets and LBP features for face recognition, Tan and Triggs [START_REF] Tan | Fusing Gabor and LBP feature sets for kernel-based face recognition[END_REF] firstly projected original feature vectors into the Principal Component Analysis (PCA) space, and then utilized Kernel Discriminative Common Vectors (KDCV) to extract the discriminative features.

Dual-Space LDA was also adopted to select discriminative LBP features, and proved to be effective [START_REF] Zhao | Contextual distance for data perception[END_REF]. Zhao et al. [START_REF] Zhao | Laplacian PCA and its applications[END_REF] employed Laplacian PCA (LPCA) for LBP feature selection, and their experiments showed that LPCA outperforms PCA and KPCA on selecting LBP-based feature. Wolf and Guttmann [START_REF] Wolf | Artificial complex cells via the tropical semiring[END_REF] adopted max-plus PCA to select LBP feature, and achieved a better performance than traditional PCA. In [START_REF] Shan | Appearance manifold of facial expression[END_REF], Shan et al. applied Locality Preserving Projections for manifold learning. Gao and Wang [START_REF] Gao | Boosting in random subspaces for face recognition[END_REF] proposed to select LBP feature by applying boosting learning in random subspaces. Specifically, multiple low-dimensional subspaces are randomly generated from original high-dimension feature space as the input to boosting.

D. Other Methods

Shan et al. [START_REF] Shan | Conditional mutual information based boosting for facial expression recognition[END_REF] adopted the Conditional Mutual Information (CMI) maximization criterion for LBP feature selection. Their experiments show that selected LBP features perform very well. Raja and Gong [START_REF] Raja | Sparse multi-scale local binary patterns[END_REF] proposed the Multi-Scale Selected Local Binary Feature predicates as an improvement to traditional LBP. A feature selection method, named Binary Histogram Intersection Minimization (BHIM), is introduced to generate the predicates, which comprise individual point features from multiple scales. The experiments illustrate that BHIM establishes less redundant LBP feature sets than CMI and AdaBoost do, and it produced promising performance [START_REF] Raja | Sparse multi-scale local binary patterns[END_REF]. In [START_REF] Nanni | RegionBoost learning for 2D+3D based face recognition[END_REF], Nanni and Lumini adopted Sequential Forward Floating Selection (SFFS) to select the LBP feature extracted from both 2D and 3D images.

V. LBP BASED FACIAL IMAGE ANALYSIS

Machine-based face recognition involves two crucial aspects, i.e. facial representation [3] [63] [99]- [START_REF] Pang | Iterative subspace analysis based on feature line distance[END_REF] and classifier design [START_REF] Cover | Nearest neighbor pattern classification[END_REF]- [START_REF] Cortes | Support-vector networks[END_REF]. Facial representation consists in deriving a set of relevant features from original images for describing faces, in order to facilitate effective machine-based recognition. "Good" facial features are desired to have the following properties [START_REF] Hadid | A discriminative feature space for detecting and recognizing faces[END_REF]: first, they can tolerate within-class variations while discriminate different classes well; second, they can be easily extracted from the raw images to allow fast processing; and finally, they lie in a space with low dimensionality to avoid computationally expensive classifiers. Since it was introduced for face representation [START_REF] Ahonen | Face recognition with local binary patterns[END_REF], LBP has proved to be an efficient descriptor for facial image analysis as it fulfills the above criteria quite well, and recent years have witnessed increasing interest in LBP features for facial representation.

In this section, we first present LBP-based facial description, and then review existing works on different tasks including face detection, face recognition, facial expression analysis, demographic classification, and other applications.

A. LBP-based Face Description

A face image can be considered as a composition of the micro-patterns described by LBP. One can build an LBP histogram computed over the whole face image. However, such a representation only encodes the occurrences of micro-patterns without any indication about their locations. To also consider the shape information of faces, Ahonen et al. [START_REF] Ahonen | Face recognition with local binary patterns[END_REF] proposed to divide face images into m local regions from which local LBP histograms can be extracted, and then to concatenate them into a single, spatially enhanced feature histogram (as shown in Fig. 11). The resulting histogram encodes both the local texture and global shape of face images. Most of the existing works adopt the above scheme to extract LBP features for facial representation. However, dividing face images into a grid of sub-regions is somewhat arbitrary, and the sub-regions are not necessary well aligned with facial features. Moreover, the resulting facial description depends on the chosen size and the positions of these sub-regions. To address this issue, in [START_REF] Zhang | Boosting Local Binary Pattern-based face recognition[END_REF] and [START_REF] Shan | Conditional mutual information based boosting for facial expression recognition[END_REF], many more sub-regions are obtained by shifting and scaling a sub-window over the face images, and boost learning [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF] is adopted to select the most discriminative sub-regions in term of LBP histograms (as shown in Fig. 12). In their experiments, sub-regions of various positions and different sizes were selected. More recently, facial representation based on LBP histograms extracted from overlapped sub-regions was evaluated in [START_REF] Gritti | Local features based facial expression recognition with face registration errors[END_REF]. Furthermore, the sub-regions do not need to be rectangular. For example, they can also be circular [START_REF] Ahonen | Face description with local binary patterns: application to face recognition[END_REF] or triangular [START_REF] Vazquez | A new image division for LBP method to improve face recognition under varying lighting conditions[END_REF] regions.

To achieve a more comprehensive description of local facial patterns, the LBP operators with different numbers of sampling points and various neighborhood radii can be combined. E.g., in [START_REF] Chan | Multi-scale local binary pattern histograms for face recognition[END_REF], [START_REF] Yan | Exploring feature descriptors for face recognition[END_REF] and [START_REF] Huang | Shape localization based on statistical method using extended local binary pattern[END_REF], the Multi-Scale LBP or Multi-Radius LBP were introduced for facial description, to reduce sensitivity of LBP-based face representations to the scale of face images (Fig. 13). More recently, for facial expression recognition, Shan and Gritti [START_REF] Shan | Learning discriminative LBP-histogram bins for facial expression recognition[END_REF] first extracted LBP features of different scales, and then ran AdaBoost to learn the most discriminative features. It proved that a boosted classifier of Multi-Scale LBP consistently outperforms that of single scale LBP, and the selected LBP bins are distributed at all scales on the Cohn-Kanade database.

B. Face Detection

The purpose of face detection is to determine the locations and sizes of human faces in digital images. In [START_REF] Hadid | A discriminative feature space for detecting and recognizing faces[END_REF], Hadid et al. first used LBP for face detection. To describe low-resolution faces, a 4-neighborhood LBP operator, LBP [START_REF] Hadid | A discriminative feature space for detecting and recognizing faces[END_REF][START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] , was applied to overlapping small regions. The Support Vector Machine (SVM) classifier was adopted to discriminate faces from non-faces. To compare with the state-of-the-art methods, they performed their experiments on the MIT-CMU dataset, and the proposed method detected 221 faces without any false positives. Later, they [START_REF] Hadid | A hybrid approach to face detection under unconstrained environments[END_REF] proposed a hybrid method to address face detection under unconstrained environments. Their method first searched for the potential skin regions in an input image to avoid scanning the entire image as was done in [START_REF] Hadid | Color-based face detection using skin locus model and hierarchical filtering[END_REF]. Then a coarse-to-fine strategy is employed to determine whether the scanned regions are faces or not: in the coarse stage, LBP feature vector extracted from the whole region is utilized as the input to a polynomial SVM; patterns which are not rejected by the first SVM classifier are further analyzed by the second finer one whose inputs are extracted from overlapped blocks inside the region. The detection rate reported is 93.4% with 13 false positives.

Motivated by the fact that LBP is invariant to monotonic transformations, Zhang and Zhao [START_REF] Zhang | Spatial histogram features for face detection in color images[END_REF] proposed to compute the spatial histograms on color measurements for face detection in color images. After extracting five measurements, Y, R, G, B and θ, in the RGB and YUV color space from the original images, LBP was utilized to transform the obtained measures to histograms, which are computed as facial description with 23 different spatial templates to preserve the shape information of faces. A hierarchical classifier combining histogram matching and SVM was used to discriminate between faces and non-faces. The experiments were conducted on 251 color images including 356 frontal faces with variations in color, position, size and also expression, and precision of 91.7% was reported. Jin et al. [START_REF] Jin | Face detection using improved LBP under Bayesian framework[END_REF] exploited Improved LBP features for facial representation, and modeled faces and non-faces using the multivariable Gaussian distribution. Given the ILBP based features of an input image, the likelihood of the face class and non-face class are calculated separately; the Bayesian decision rule is then applied to decide whether the image is a face or not. The Yale B and MIT-CMU databases were used for evaluation, and the detection rate was more than 90% with a false positive rate of 2.99x10 -7 . They later extracted ILBP based features from larger neighborhoods [START_REF] Jin | Learning local descriptors for face detection[END_REF], and trained a cascade Ada-Boost detector, which achieved detection result of 93.0% on the MIT-CMU database and 94.6% on the Yale B database. In [START_REF] Zhang | Face detection based on Multi-Block LBP representation[END_REF], Zhang et al. exploited MB-LBP for face detection. Experimented on a dataset containing 13,000 faces and 50,000 non-faces with a false alarm rate set to 0.001, the MB-LBP based approach achieved superior accuracy which was 15% higher than Haar-like features and 8% higher than original LBP features. The experiments on the MIT-CMU database also showed that the approach displayed a comparable performance with that in [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF], but with fewer features.

C. Face Recognition

Face recognition aims to identify or verify a person from a digital image or a video sequence. Ahonen et al [START_REF] Ahonen | Face recognition with local binary patterns[END_REF] introduced LBP in face recognition with Nearest Neighbor (NN) classifier and Chi square distance as the dissimilarity measure. The experimental results showed that their approach outperforms PCA, EBGM and the Bayesian Intra/Extra-personal Classifier on all four probe sets of the FERET database. They later investigated whether these good results are due to the use of local regions or the discriminative capacity of LBP methodology [START_REF] Ahonen | Face recognition based on the appearance of local regions[END_REF]. Based on comparisons with three other texture descriptors extracting features from the same local patches, the strength of LBP for representing faces was clearly confirmed. In [START_REF] Hadid | A discriminative feature space for detecting and recognizing faces[END_REF], face recognition experiments were also carried out on the MoBo database, which is challenging because of the images with low resolution. As mentioned above, AdaBoost later was applied to select a few of the most effective LBP based features for face recognition [START_REF] Zhang | Boosting Local Binary Pattern-based face recognition[END_REF]. Compared to the approach in [START_REF] Ahonen | Face recognition with local binary patterns[END_REF], the boosting-LBP based method achieves a slightly better recognition rate while using fewer LBP features.

Zhang et al. [START_REF] Zhang | Multi-resolution Histograms of Local Variation Patterns (MHLVP) for robust face recognition[END_REF] introduced MHLVP for face recognition based on histogram intersection. Their experiments on FERET database showed that their algorithm provides better accuracy than some milestone approaches, containing the best ones in FERET'97. In particular, they achieved 95.9% accuracy on the fc set with illumination changes. They employed LGBPHS [START_REF] Zhang | Local Gabor Binary Pattern Histogram Sequence (LGBPHS): A novel non-statistical model for face representation and recognition[END_REF], similar to MHLVP but with weighted rules, for the same task. In addition to the FERET dataset, they also ran experiments on the AR database. The results on both databases were very promising. Furthermore, they [START_REF] Shan | Ensemble of piecewise FDA based on spatial histograms of local (Gabor) binary patterns for face recognition[END_REF] exploited LGBP with an ensemble of piecewise LDA, which not only reduces the feature dimension, but also improves the performance on the FERET database. Yao et al. [START_REF] Yao | Domain-partitioning rankboost for face recognition[END_REF] adopted the Domain-Partitioning RankBoost to select LGBPH features for face recognition. The subsets fb and Dup I from the FERET database were used for evaluating the approach, and comparable results were achieved with only 50 selected features. Zhang et al. [START_REF] Zhang | Are Gabor phases really useless for face recognition[END_REF] argued that Gabor phases are also useful for face recognition. By encoding Gabor phases through LBP and forming local feature histograms, impressive recognition rates were obtained on FERET database (99% for fb, 96% for fc, 78% for Dup I, 77% for Dup II).

Zhao et al. [START_REF] Zhao | LBP discriminant analysis for face verification[END_REF] applied Kernel LDA with the LBP features for face recognition where their kernel function was designed using the Chi square distance and Radial Basis Function. Their method has been proved effective on FRGC Exp2.0.1, achieving a verification rate of 97.4% with FAR at 0.001 and 99.2% with FAR at 0.01. Rodriguez and Marcel [START_REF] Rodriguez | Face authentication using adapted local binary pattern histograms[END_REF] proposed an LBP-based generative method for face authentication. Specifically, LBP histogram is interpreted as a probability distribution, and a generic face model is considered as a collection of LBP histograms. A client-specific model is then obtained by Maximum a Posteriori (MAP) from a generic face model. The outcomes on XM2VTS and BANCA, reveals that their approach outperforms the approaches in [START_REF] Ahonen | Face recognition with local binary patterns[END_REF] and [START_REF] Zhang | Boosting Local Binary Pattern-based face recognition[END_REF].

Li et al. [START_REF] Li | Learning to fuse 3D+2D based face recognition at both feature and decision levels[END_REF] designed a framework for fusing 2D and 3D face recognition based on LBP features at both feature and decision levels. AdaBoost was used for LBP feature selection. The experiments on a database containing 252 subjects illustrate the advantages of two-level fusion over decision-level fusion. To our knowledge, this is the first work to apply LBP to 3D domain. Later Huang et al. [START_REF] Huang | Combining statistics of geometrical and correlative features for 3D face recognition[END_REF] extended LBP to 3DLBP, actually similar to Extended LBP as in [START_REF] Tan | Fusing Gabor and LBP feature sets for kernel-based face recognition[END_REF], for 3D face recognition based on range images; their approach achieved the promising result of 9.4% EER on FRGC v2.0 Exp3. ROC I.

Nanni and Lumini [START_REF] Nanni | RegionBoost learning for 2D+3D based face recognition[END_REF] also utilized LBP to extract both 2D and 3D facial features; their experiments were conducted on a subset of 198 persons from the Notre-Dame database collection D; the reported EER was 3.5%.

Li et al. [START_REF] Li | Highly accurate and fast face recognition using near infrared images[END_REF]- [START_REF] Li | Illumination invariant face recognition using near-infrared images[END_REF] later applied LBP with NIR images to obtain robust facial descriptions under illumination variations. The method achieved a verification rate of 90% at FAR=0.001 and 95% at FAR=0.01 on a database with 870 subjects. The same method was utilized with Enhanced NIR images for face verification outdoor, especially in sunlight [START_REF] Yi | Outdoor face recognition using enhanced near infrared imaging[END_REF]. Pan et al. [START_REF] Pan | Part-based face recognition using near infrared images[END_REF] proposed to improve the robustness of this work to variations of pose. NIR face images were decomposed into several parts in accordance with key facial components, and LBP features extracted from these parts were selected by AdaBoost; the outputs of part classifiers were then fused to give the final score. The verification rate of their approach is 96.03% with FAR=0.001. Huang et al. [START_REF] Huang | A robust method for near infrared face recognition based on extended local binary pattern[END_REF] adopted AdaBoost to learn ELBP features for NIR face recognition, and obtained a recognition rate of 95.74% on a database with 60 individuals.

Yan et al. examined Multi-Radius LBP for face recognition [START_REF] Yan | Exploring feature descriptors for face recognition[END_REF]. Their experiments on Purdue (90 subjects) and CMU-PIE (68 subjects) datasets showed that LBP and Gabor features are mutually complementary and a combination of similarity scores can bring performance improvement. In [START_REF] Chan | Multi-scale local binary pattern histograms for face recognition[END_REF], Chan et al. employed the Multi-Scale LBP with LDA for face recognition. The reported performance on the FERET and XM2VTS databases was better than the state-of-the-art approaches. They also projected multi-spectral LBP features extracted from local regions into an LDA subspace as the discriminative regional description. They proved the effectiveness of their method on the FRGC and XM2VTS databases. Hadid et al. [82] [85] introduced VLBP to extract local facial dynamics for spatio-temporal face recognition from video sequences. AdaBoost was applied to learn the specific facial dynamics of each subject from LBP-based features while ignoring intra-personal temporal information such as facial expressions. Their approach achieved superior performances on various databases: MoBo (97.9%), Honda/UCSD (96.0%) and CRIM (98.5%). Lei et al. [START_REF] Lei | Gabor volume based local binary pattern for face representation and recognition[END_REF] used GV-LBP-TOP and E-GV-LBP based features for face recognition, and both methods achieved encouraging results on FERET and FRGC2.0 databases. Yang and Wang [START_REF] Yang | A LBP-based face recognition method with Hamming distance constraint[END_REF] introduced Hamming LBP for face recognition on the FRGC dataset. The experimental results reveal that the Hamming LBP outperforms the original LBP, especially when variations of illumination and facial expression exist. Liao and Chung [START_REF] Liao | Face recognition by using elongated local binary patterns with average maximum distance gradient magnitude[END_REF] exploited Elongated LBP to capture the anisotropic structures of faces. Average Maximum Distance Gradient Magnitude was proposed to embed the information on gray-level difference between the reference and the neighboring pixel in each Elongated LBP pattern. With a subset of 70 persons randomly selected from the FERET database, their method obtained 93.16% accuracy, and 98.50% on the ORL database.

Tan and Triggs [START_REF] Tan | Enhanced local texture feature sets for face recognition under difficult lighting conditions[END_REF] proposed a method for face recognition under illumination variations, which includes preprocessing for reducing sensitivity to illumination changes and LTP for solving the problem caused by LBP's sensitivity to random and quantization noise. A distance transform-based similarity metric was used for decision. The method showed promising performance on three datasets with illumination variations: FRGC Exp 1.0.4 (86.3%); Yale-B (100%); as well as CMU PIE (100%). They also fused Gabor and LBP features to construct heterogeneous features for face recognition in [START_REF] Tan | Fusing Gabor and LBP feature sets for kernel-based face recognition[END_REF]. With the features extracted by KDCV, they achieved satisfying results on the FRGC 1.0.4, FRGC 2.0.4 and FERET databases. Park and Kim [START_REF] Park | Fast adaptive smoothing based on LBP for robust face recognition[END_REF] presented an adaptive smoothing approach for face image normalization under changing lighting. The illumination is estimated by iteratively convolving the input image with a 3x3 averaging kernel weighted by a simple measure of illumination discontinuity at each pixel. In particular, the kernel weights are encoded into a local binary pattern to achieve fast and memory-efficient processing. 633 frontal face images were selected from the Yale B database, and average recognition accuracy was 99.74% with 0.038s time consumed for each image.

Table 2 summarizes the performance of different approaches on the FERET database.

D. Facial Expression Analysis

Machine-based facial expression recognition aims to recognize facial affect states automatically and may rely on both of Table .2 Some performances of LBP-based face recognition on the FERET database. audio and visual clues [START_REF] Zeng | A survey of affect recognition methods: audio, visual, and spontaneous expressions[END_REF]. In this paper, we focus our attention on works purely based on visual information, using facial motion or facial features [START_REF] Fasel | Automatic facial expression analysis: a survey[END_REF]- [START_REF] Sebe | Authentic facial expression analysis[END_REF]. Most of these works only consider the prototypical emotional states which include seven basic universal categories, namely neutral, anger, disgust, fear, happiness, sadness, and surprise Feng et al. [START_REF] Feng | A coarse-to-fine classification scheme for facial expression recognition[END_REF] [120] exploited a coarse-to-fine classification scheme with LBP for facial expression recognition making use of images. More precisely, at the coarse stage, a 7-class problem was first reduced to a 2-class one, while at fine stage; a k-nearest neighbor classifier performed the final decision. Their approach produced 77% average recognition accuracy on JAFFE dataset. In [START_REF] Feng | Facial expression recognition with local binary patterns and linear programming[END_REF]- [START_REF] Feng | Facial expression recognition with local binary patterns and linear programming[END_REF], with the same facial description, a Linear Programming technique was applied for expression classification. A 7-class problem was decomposed into 21 binary classifications using the one-against-one scheme. With this method, they obtained over 90% accuracy on both the JAFFE database and some real videos.

Shan et al. [START_REF] Shan | Robust facial expression recognition using local binary patterns[END_REF] also investigated LBP for facial expression recognition. The template matching with weighted Chi-square statistics and SVM were adopted to classify the basic prototypical facial expressions; and the best performance obtained on the Cohn-Kanade Database reached 88.4% using SVM. In many applications involving facial expression recognition, the input face images are with low-resolution. In [START_REF] Shan | Robust facial expression recognition using local binary patterns[END_REF]- [START_REF] Shan | Recognizing facial expressions at low resolution[END_REF], they further studied this topic. They not only performed evaluation on face images with different resolutions, but also ran experiments on real-world low-resolution video sequences. It was observed that LBP features perform stably and robustly over a useful range of face images with low resolutions. In [START_REF] Shan | Conditional mutual information based boosting for facial expression recognition[END_REF], Shan et al. introduced CMI maximization criterion for LBP feature selection, and the selected features improved recognition accuracy compared with that using AdaBoost. Later, Shan et al. [START_REF] Shan | Appearance manifold of facial expression[END_REF] also studied facial expression manifold learning by embedding image sequences in a high-dimensional LBP space to a low-dimensional manifold. Their experiments on the Cohn-Kanade database illustrated that meaningful projections could be obtained. Shan and Gritti [START_REF] Shan | Learning discriminative LBP-histogram bins for facial expression recognition[END_REF] used AdaBoost to learn a set of discriminative bins of an LBP histogram for facial expression recognition. Their experiments indicated that the selected bins provide a much more compact facial description. It was also evidenced that it is necessary to consider the multi-scale LBP for facial description. By applying SVM to the selected multi-scale LBP bins, they obtained recognition rate of 93.1% on the Cohn-Kanade database, comparable to the best results so far reported on this database.

In [START_REF] He | An enhanced LBP feature based on facial expression recognition[END_REF], He et al. used LBP on four kinds of frequency images decomposed by Gabor wavelets for facial expression recognition. Their approach provided better performance than LBP did on the JAFFE dataset. To consider multiple cues, Liao et al. [START_REF] Liao | Facial expression recognition using advanced local binary patterns, tsallis entropies and global appearance features[END_REF] extracted LBP ri (P, R) features in both intensity and gradient maps, and then computed the Tsallis entropy of the Gabor filter responses as the first feature set and performed null-space LDA for the second feature set. With the SVM classifier, they achieved 94.59% accuracy for images of 64x64 pixels and 84.62% for 16x16 pixels on the JAFFE database. With an Active Appearance Model (AAM), Feng et al. [START_REF] Feng | A novel feature extraction method for facial expression recognition[END_REF] extracted the local texture feature by applying LBP to facial feature points; the direction between each pair of feature points was also considered as shape information. In addition, they used LBP with the entire image to get global texture information. Combining these three types of feature, a NN-based classifier with weighted Chi square statistic was introduced for classification. Subject-independent recognition rate of 72% was reported on the JAFFE dataset. Cao et al. [START_REF] Cao | Facial expression recognition based on LBP-EHMM[END_REF] combined LBP with embedded Hidden Markov Model to recognize facial expressions by using an Active Shape Model (ASM), and achieved 65% accuracy on the JAFFE dataset.

In [START_REF] Zhao | Experiments with facial expression recognition using spatiotemporal local binary patterns[END_REF] and [START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF], Zhao and Pietikäinen employed VLBP and LBP-TOP for facial expression recognition in video sequences. A recognition rate of 96.26% was achieved on the Cohn-Kanade database; the evaluation over a range of image resolutions and frame rates demonstrated that both approaches outperform the state-of-the-art methods. They also compute LBP-TOP at multiple resolutions for describing dynamic events [START_REF] Zhao | Principal appearance and motion from boosted spatiotemporal descriptors[END_REF]. AdaBoost technique was used to learn the principal appearance and motion from the spatiotemporal descriptors. Fu and Wei [START_REF] Fu | Centralized binary patterns embedded with image Euclidean distance for facial expression recognition[END_REF] utilized the CBP instead of LBP for facial expression recognition, and recognition rates of 94.76% and 94.86% were achieved on the JAFFE and Cohn-Kanade databases respectively. The capacity of LBP to describe faces was further demonstrated in [START_REF] Gritti | Local features based facial expression recognition with face registration errors[END_REF], where Gritti et al. compared different local features: LBP, LTP, Histogram of Oriented Gradients (HOG) [128], Gabor wavelets, with various parameter settings for facial expression recognition. Based on their experiments, LBP with an overlapping strategy achieved the best result, 92.9%, on the Cohn-Kanade database. Furthermore, it was indicated that the overlapping LBP is the most robust to deal with registration errors.

E. Demographic Classification and Other Applications

Demographic classification is used to classify by age, gender and ethnicity, based on face images. Sun et al. [START_REF] Sun | Gender classification based on boosting local binary pattern[END_REF] adopted the boosting-LBP based approach [START_REF] Zhang | Boosting Local Binary Pattern-based face recognition[END_REF] for gender recognition, and obtained the performance of 95.75% on the FERET dataset. In [START_REF] Lian | Multi-view gender classification using local binary patterns and support vector machines[END_REF]- [START_REF] Lian | Multi-view gender classification using Multi-resolution local binary patterns and support vector machines[END_REF], Lian and Lu combined the LBP-based facial de-scription with SVM for multi-view gender classification, and reported an average accuracy of 94.08% on the CAS-PEAL face dataset. Yang and Ai [START_REF] Yang | Demographic classification with local binary patterns[END_REF] exploited the LBP based features for a face-based demographic classification which involved gender, ethnicity as well as age classification. Given a local patch, Chi square distance between achieved LBP histograms was utilized as a confidence measure for classification. The positive mean histogram was utilized for initialization, and the steep descent method was applied to find an optimal reference template. They adopted the Real AdaBoost for training a strong classifier. The achieved error rates for gender classification on the FERET, PIE and a snapshot database were 6.7%, 8.9% and 3.68% respectively. Their method also produced promising performance for ethnicity and age classification.

Huang et al. [START_REF] Huang | Shape localization based on statistical method using extended local binary pattern[END_REF] proposed an improved ASM framework, namely ELBP-ASM, in which local appearance patterns of key points are modeled by the Extended LBP. The experiments on a dataset with 250 samples show that ELBP-ASM achieves more accurate results than the original ASM. In order to extend ASM to improve robustness against illumination variations, Marcel et al. [START_REF] Marcel | Robust-to-illumination face localization using active shape models and local binary patterns[END_REF] later presented a divided-square based LBP-ASM to extract histograms from a square region divided into four blocks around each landmark instead of the normal profile. Histograms were then concatenated into a single feature vector representing local appearance. The comparative experiments on XM2VTS dataset showed that this method outperforms ELBP-ASM [START_REF] Huang | Shape localization based on statistical method using extended local binary pattern[END_REF], and requires only raw images for facial keypoint localization.

In [START_REF] Ma | Robust head pose estimation using LGBP[END_REF], Ma et al. introduced the Local Gabor Binary Pattern (LGBP) to encode the local facial characteristics for head pose estimation. With an SVM classifier, estimation rate of 97.14% for 7 poses was gained on a subset of the CAS-PEAL dataset containing 200 subjects. Cao et al. [START_REF] Gao | Standardization of face image sample quality[END_REF] used a facial symmetry-based approach to standardize face image quality. With this method, facial asymmetries caused by non-frontal illumination and improper facial pose can be measured. The effectiveness was evaluated on images of 10 persons of the Yale B dataset.

LBP can also be used as a preprocessing technique on face images. For instance, Heusch et al. [START_REF] Heusch | Local binary patterns as an image preprocessing for face authentication[END_REF] considered LBP as a preprocessing step to remove lighting effects (Fig. 14). Compared with other preprocessing methods, including Histogram Equalization and the technique proposed by Gross & Brajovic [START_REF] Gross | An image preprocessing algorithm for illumination invariant face recognition[END_REF], LBP provided better results on the XM2VTS database. Cardinaux et al. showed that LBP is better combined with feature-based HMM than with appearance-based LDA for face recognition on the BANCA dataset [START_REF] Cardinaux | Face verification using adapted generative models[END_REF]. Poh et al. presented a similar comparative study in [START_REF] Poh | On combination of face authentication experts by a mixture of quality dependent fusion classifiers[END_REF], and their experiments further supported that LBP is effective for face preprocessing, but combining LBP with feature-based Gaussian Mixtures Models did not perform as well as combined with LDA. LBP application for preprocessing was also addressed in [START_REF] Ruiz-Del-Solar | Illumination compensation and normalization in eigenspace-based face recognition: a comparative study of different pre-processing approaches[END_REF]; a comparative study on 5 preprocessing in 16 different Eigenspace-based face recognition systems evidenced that the Modified LBP (MLBP) achieves promising results for illumination compensation and normalization. More recently, in order to highlight the details of facial images, Huang et al. [START_REF] Huang | Asymmetric 3D-2D face recognition based on LBP facial representation and canonical correlation analysis[END_REF]- [START_REF] Huang | Automatic asymmetric 3D-2D face recognition[END_REF] proposed to use LBP to extract the range and texture LBP faces, and Canonical Correlation Analysis (CCA) was then applied to learn the relationship of two types of LBP faces for asymmetric face recognition. The reported result was 95.61% on the FRGC v2.0 dataset [START_REF] Huang | Automatic asymmetric 3D-2D face recognition[END_REF].

F. Face Analysis Systems

Advantages of LBP make it very attractive to build real-time face analysis systems. Furthermore, related hardware designed for high-speed LBP computation [START_REF] Lahdenoja | A massively parallel algorithm for local binary pattern based face recognition[END_REF]- [START_REF] Lahdenoja | Local binary pattern feature vector extraction with CNN[END_REF] also boosts the development of LBP-based real-world applications.

Hadid et al. built an access control system using LBP-based face recognition [START_REF] Hadid | A novel approach to access control based on face recognition[END_REF]. In their system, a camera was set on a door for capturing video frames; LBP features were extracted for both background subtraction and face recognition. The face detection approach in [START_REF] Hadid | Color-based face detection using skin locus model and hierarchical filtering[END_REF] was adopted for face detection in color images, and the face recognition method in [START_REF] Ahonen | Face recognition with local binary patterns[END_REF] was applied for person identification. The face recognition accuracy of 71.6% was obtained on 20 video sequences of 10 subjects.

Trinh et al. [START_REF] Trinh | Multi-face detection system in video sequence[END_REF] presented a system for detecting multiple faces in video sequences where faces are not limited to frontal views. An adaptive selection approach from two skin models in RGB and ratio RGB spaces is used to overcome the illumination problem by automatic focus of the camera. The experimental result of 93% accuracy was reported on the NRC-IIT database, which consists of 23 single-face video sequences of 11 persons with different poses. The system runs at 2.57 fps for image sequences of 320x240 pixels on a standard PC (Pentium 4 2.6GHz, 512MB RAM) in the Visual C++ environment.

Based on LBP features extracted from Near Infrared (NIR) faces, Li et al. [START_REF] Li | Highly accurate and fast face recognition using near infrared images[END_REF]- [START_REF] Li | Illumination invariant face recognition using near-infrared images[END_REF] designed an illumination-invariant face recognition system for cooperative users in an indoor situation. AdaBoost was used to learn Haar features for face detection and eye detection, and to select LBP features for face recognition. All three parts achieve outstanding results with low cost on a large dataset. The system can operate in real time with an EER below 0.3%.

Ekenel et al. [START_REF] Ekenel | Face recognition for smart interactions[END_REF] introduced a portable face recognition system deployed on a laptop using a standard webcam for image acquisition. Based on the relevant regions determined by skin color, the two eyes were first detected with a cascade AdaBoost classifier of Haar features. These were then used to register face images. LBP was used to preprocess facial regions for reducing illumination influences. Their system was evaluated on a small dataset consisting of 42 sequences from 14 subjects, and produced 79% accuracy.

In [START_REF] Hadid | Face and eye detection for person authentication in mobile phones[END_REF], Hadid et al. implemented face detection and authentication on mobile phones equipped with an ARM9 pro-cessor. Cascade AdaBoost with Haar features was applied for face and eye detection, while LBP was exploited for face authentication. Though the CPU and memory capabilities of mobile phones are limited, the experiments showed encouraging performance on face detection and displayed recognition rates of 82% for faces of 40x40 pixels and 96% for faces of 80x80 pixels. The system ran at 2 fps on video sequences with a resolution of 320x240 pixels. Abbo et al. [START_REF] Abbo | Mapping facial expression recognition algorithms on a Low-power smart camera[END_REF] recently mapped an LBP-based facial expression recognition algorithm proposed in [START_REF] Shan | Robust facial expression recognition using local binary patterns[END_REF] on a low-power smart camera, which was assembled with a massively-parallel processor for low level and intermediate vision processing, and an 8051 microcontroller for high level decision making and the camera control tasks.

Hannuksela et al. [START_REF] Hannuksela | Face tracking for spatially aware mobile user interfaces[END_REF] proposed a head tracking system for controlling the user interface on hand-held mobile devices. Face and eye detection were realized using boosting-LBP approach. It worked in real-time on a resource-limited mobile device. In an interactive photo annotation system [START_REF] Cui | EasyAlbum: an interactive photo annotation system based on face clustering and re-ranking[END_REF], LBP was also used to extract facial features for face clustering and re-ranking.

G. Discussion

The techniques developed so far for facial representation can be roughly classified into two main categories: holistic-based ones and local-based ones [START_REF] Zou | A comparative study of local matching approach for face recognition[END_REF]- [START_REF] Zhao | Face recognition: a literature survey[END_REF]. The holistic approaches use the whole facial region to construct a subspace using, e.g. Principal Component Analysis (PCA) [START_REF] Turk | Eigenfaces for recognition[END_REF], Linear Discriminant Analysis (LDA) [START_REF] Belhumeur | Eigenfaces vs. fisherfaces: recognition using class specific linear projection[END_REF], Independent Component Analysis (ICA) [START_REF] Bartlett | Face recognition by independent component analysis[END_REF], or Locally Linear Embedding (LLE) [START_REF] Li | Discriminant locally linear embedding with high-order tensor data[END_REF]. On the other hand, the local-based ones, e.g. [START_REF] Wiskott | Face recognition by elastic bunch graph matching[END_REF] [101] [START_REF] Song | Image Ratio Features for Facial Expression Recognition[END_REF], proceed firstly to locate a number of features or components from a face; then classify them by combining and comparing corresponding local statistics. The local approaches have shown promising performances in recent years. It has been proved by Heisele et al. [START_REF] Heisele | Face recognition: component-based versus global approaches[END_REF] the component-based face recognition techniques (local-based) outperform the global ones (holistic-based), as the global approaches require face images to be accurately normalized according to pose, lighting and scale. In addition, global features are also more sensitive to facial expression variations and occlusions. Since local-based methods extract features from local points or patches, there always remain some invariant features even in the presence of facial expression or occlusion variations, and recognition can still be achieved by matching remaining invariant features. Therefore, local-based methods are potentially more robust than holistic ones to expression variations and occlusions. Moreover, unlike holistic approaches, local-based ones require few samples for enrollment, and can even achieve analysis with a single face image in the gallery set [START_REF] Tan | Face recognition from a single image per person: a survey[END_REF]. From this viewpoint, if LBP can be used in a straightforward manner as a global representation, it is its local-or component-oriented variants that prove to be efficient descriptors for facial image analysis as the previous overview highlights. This motivates the increasing interests in LBP-based features for facial representation since this method was applied for face representation [START_REF] Ahonen | Face recognition with local binary patterns[END_REF].

Compared to other popular local-based features as discussed in [START_REF] Zou | A comparative study of local matching approach for face recognition[END_REF] [155]- [START_REF] Ruiz-Del-Solar | Recognition of faces in unconstrained environments: a comparative study[END_REF], namely Gabor wavelets [START_REF] Wiskott | Face recognition by elastic bunch graph matching[END_REF] [157]- [START_REF] Pang | Effective feature extraction in high dimensional space[END_REF] and SIFT [START_REF] Bicego | On the use of SIFT features for face authentication[END_REF] [155], Luo et al. [START_REF] Luo | Person-specific SIFT features for face recognition[END_REF] showed that SIFT is not as robust as LBP to illumination effects for face recognition on the FERET dataset. Zou et al. [START_REF] Zou | A comparative study of local matching approach for face recognition[END_REF] compared Gabor wavelets and LBP using the same database for the same task, and concluded that Gabor wavelets are more insensitive to illumination changes since they detect amplitude-invariant spatial frequencies of gray values of pixels, whilst LBP is greatly affected by non-monotonic gray value transformations. Ruiz-del-Solar et al. [START_REF] Ruiz-Del-Solar | Recognition of faces in unconstrained environments: a comparative study[END_REF] evaluated these three methods extensively for face recognition not only on controlled datasets, such as the FERET and FRGC, but also on the UCH FaceHRI database, designed for Human-Robot Interaction, and the LFW database, captured in unconstrained environments. Concerning robustness to illumination changes, the study showed that Gabor wavelets achieved the best performance on the FERET database, LBP was not far behind; while SIFT was the last, thus further supporting the previous conclusions. On the UCH FaceHRI database, the LBP approach gained the best results in all the specially-designed experiments for indoor and outdoor lighting, expression, scaling, and rotation, followed by Gabor wavelets and SIFT. On LFW database, LBP and Gabor wavelets obtained a slightly better result than each other with aligned face and non-aligned face respectively, both of which surpassed that of SIFT. On the other hand, in their investigation on computation cost, LBP ran much faster than Gabor wavelet and SIFT.

VI. CONCLUDING REMARKS

LBP is one of the most powerful descriptors for representing local structures. Thanks to its advantages, i.e., its tolerance of monotonic illumination changes and its computational simplicity, LBP has been successfully used for many different image analysis tasks, such as facial image analysis, biomedical image analysis, aerial image analysis, motion analysis, and image and video retrieval.

During the development of LBP methodology, a large number of variations are designed to expand the scope of application, offering better performance as well as improving robustness in one or more aspects of the original LBP. ILBP, Hamming LBP and ELBP enhance the discriminative ability of LBP; LTP and SLBP focus on improving the robustness of LBP on noisy images; MB-LBP, Elongated LBP, TPLBP, and FPLBP, change the scale of LBP to provide other categories of local information; Gabor wavelet-based LBP, CS-LBP as well as LBP-HF combine other methods with LBP to bring in new merits. However, the previous extensions only operate on traditional 2D data, the variant 3DLBP and VLBP should be highlighted, since both of them expand the scope of LBP applications: 3DLBP extends the LBP operator to describe 3D volume data, whilst VLBP endows LBP with the ability to capture dynamic information.

To obtain a small set of the most discriminative LBP-based features for better performance and dimensionality reduction, LBP-based representations are associated with some popular techniques of feature selection schemes for reducing the feature length LBP codes, including rule-based strategy, boosting and subspace learning, etc.

As the most typical and important application of LBP, facial image analysis provides a very good demonstration of the use, development and performance of LBP. From this comprehensive overview, following conclusions can be drawn : (1) localor component-oriented LBP representations are effective representations for facial image analysis as they encode the information of facial configuration while providing local structure patterns; (2) using the local-or component-oriented LBP facial representations, feature selection is particularly important for various tasks in facial image analysis, since this facial description scheme greatly increases feature length.

As the conclusions based on their experiments drawn in [START_REF] Hadid | The local binary pattern and its applications to face analysis[END_REF] revealed, like all texture based methods, LBP seems to be sensitive to severe illumination changes, and to blurred and noisy images. The former one can be seen as non-monotonic lighting variations, as is mostly the case in facial image, since compared with flat texture images, human faces depict 3D volume structures; therefore non-monotonic transformations, e.g., shadows and bright spots can typically occur and change their positions depending on illumination directions. While the latter one is often caused by the bad quality of camera sensors and poor user cooperation of capture condition etc. As a result, in such environments, it is necessary and useful to preprocess the images before applying LBP.

In addition, some open questions for sub-region based LBP description, e.g. facial description, concern the relevant number of components and the corresponding neighborhood of a certain LBP operator for the best analysis result. Although these questions have been discussed in several works and even with machine learning techniques, these conclusions drawn so far have always been dependent on the datasets used and on some given parameters.

Fig. 1 .

 1 Fig. 1. An example of the basic LBP operator.

Fig. 4 .

 4 Fig. 4. An example of the ELBP operator.

Fig. 7 .

 7 Fig. 7. An example of the MB-LBP operator [18].

Fig. 8 .

 8 Fig. 8. Procedure of VLBP 1 ,4, 1 [60].

Fig. 9 .

 9 Fig. 9. Representation of a 3D local binary pattern (S=9, P=16, R=2) [59].

Fig. 10 .

 10 Fig. 10. LBP and CS-LBP features for a neighborhood of 8 pixels [73].

  [START_REF] Li | Learning to fuse 3D+2D based face recognition at both feature and decision levels[END_REF] [START_REF] Shan | Facial expression recognition based on local binary patterns: a comprehensive study[END_REF] [32]-[START_REF] Gao | Standardization of face image sample quality[END_REF] [47][START_REF] Huang | A robust method for near infrared face recognition based on extended local binary pattern[END_REF] [55][START_REF] Li | A near-infrared image based face recognition system[END_REF]-[START_REF] Hannuksela | Face tracking for spatially aware mobile user interfaces[END_REF]. In[START_REF] Yao | Domain-partitioning rankboost for face recognition[END_REF], Yao et al. exploited RankBoost with domain-partitioning weak hypotheses to select the most discriminative LGBPH features.

Fig. 11 .

 11 Fig. 11. LBP based face description [27].

Fig. 12 .

 12 Fig. 12. The top four selected sub-regions [78].

Fig. 13 .

 13 Fig. 13. The evolvement from the LBP to Multi-Radius LBP operator [64].

Fig. 14 .

 14 Fig. 14. The original image (left) processed by the LBP operator (right) [132].