Member, IEEE Guosheng Hu
email: guosheng.hu@inria.fr.

corresponding Xiaojiang Peng
email: xiaojiangp@gmail.com

Yongxin Yang
email: yongxin.yang@qmul.ac.uk

Timothy M Hospedales
email: t.hospedales@ed.ac.uk

Jakob Verbeek
email: jakob.verbeek@inria.fr.

Frankenstein: Learning Deep Face Representations using Small Data

Keywords: face recognition, deep learning, small training data

Deep convolutional neural networks have recently proven extremely effective for difficult face recognition problems in uncontrolled settings. To train such networks, very large training sets are needed with millions of labeled images. For some applications, such as near-infrared (NIR) face recognition, such large training datasets are not publicly available and difficult to collect. In this work, we propose a method to generate very large training datasets of synthetic images by compositing real face images in a given dataset. We show that this method enables to learn models from as few as 10,000 training images, which perform on par with models trained from 500,000 images. Using our approach we also obtain state-of-the-art results on the CASIA NIR-VIS2.0 heterogeneous face recognition dataset.

I. INTRODUCTION

In recent years, deep learning methods, and in particular convolutional neural networks (CNNs), have achieved considerable success in a range of computer vision applications including object recognition [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], object detection [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF], semantic segmentation [START_REF] Pinheiro | From image-level to pixel-level labeling with convolutional networks[END_REF], action recognition [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF], and face recognition [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF]. The recent success of CNNs stems from the following facts: (i) big annotated training datasets are currently available for a variety of recognition problems to learn rich models with millions of free parameters; (ii) massively parallel GPU implementations greatly improve the training efficiency of CNNs; and (iii) new effective CNN architectures are being proposed, such as the VGG-16/19 networks [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], inception networks [START_REF] Szegedy | Going deeper with convolutions[END_REF], and deep residual networks [START_REF] He | Deep residual learning for image recognition[END_REF].

Good features are essential for object recognition, including face recognition. Conventional features include linear functions of the raw pixel values, including Eigenface (Principal Component Analysis) [START_REF] Turk | Face recognition using eigenfaces[END_REF], Fisherface (Linear Discriminant Analysis) [START_REF] Belhumeur | Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection[END_REF], and Laplacianface (Locality Preserving Projection) [START_REF] He | Face recognition using Laplacianfaces[END_REF]. Such linear features were later replaced by hand-crafted local non-linear features, such as Local Binary Patterns [START_REF] Ahonen | Face recognition with local binary patterns[END_REF], Local Phase Quantisation (LPQ) [START_REF] Ahonen | Recognition of blurred faces using local phase quantization[END_REF], and Fisher vectors computed over dense SIFT descriptors [START_REF] Simonyan | Fisher Vector Faces in the Wild[END_REF]. Note that the latter is an example of a feature that also involves unsupervised learning. These traditional features achieve promising face recognition rates in constrained environments, as represented for example in the CMU PIE dataset [START_REF] Sim | The CMU Pose, Illumination, and Expression (PIE) database[END_REF]. However, using these features face recognition performance may degrade dramatically in uncontrolled environments, such as represented in the Labeled Faces in the Wild (LFW) benchmark [START_REF] Huang | Labeled faces in the wild: a database for studying face recognition in unconstrained environments[END_REF]. To improve the performance in such challenging settings, metric learning can be used, see [START_REF] Chen | Bayesian face revisited: A joint formulation[END_REF], [START_REF] Guillaumin | Is that you? Metric learning approaches for face identification[END_REF], [START_REF] Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF]. Metric learning methods learn a (linear) transformation of the features that pulls the objects that have the same label closer together, while pushing the objects that have different labels apart.

Although hand-crafted features and metric learning achieve promising performance for uncontrolled face recognition, it remains cumbersome to improve the design of hand-crafted local features (such as SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]) and their aggregation mechanisms (such as Fisher vectors [START_REF] Sánchez | Image classification with the Fisher vector: Theory and practice[END_REF]). This is because the experimental evaluation results of the features cannot be automatically fed back to improve the robustness to nuisance factors such as pose, illumination and expression. The major advantage of CNNs is that all processing layers, starting from the raw pixel-level input, have configurable parameters that can be learned from data. This obviates the need for manual feature design, and replaces it with supervised data-driven feature learning. Learning the large number of parameters in CNN models (millions of parameters are rather a rule than an exception) requires very large training datasets. For example, the CNNs which achieve state-of-the-art performance on the LFW benchmark are trained using datasets with millions of labeled faces: Facebook's DeepFace [START_REF] Taigman | DeepFace: Closing the gap to human-level performance in face verification[END_REF] and Google's FaceNet [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF] were trained using 4 million and 200 million training samples, respectively.

For some recognition problems large supervised training datasets can be collected relatively easily. For example the CASIA Webface dataset of 500,000 face images was collected semi-automatically from IMDb [START_REF] Yi | Learning face representation from scratch[END_REF]. However, in many other cases collecting large datasets may be costly, and possibly problematic due to privacy regulation. For example, thermal infrared imaging is ideal for low-light nighttime and covert face recognition applications [START_REF] Kong | Recent advances in visual and infrared face recognition -a review[END_REF], but it is not possible to collect millions of labeled training images from the internet for the thermal infrared domain. The lack of large training datasets is an important bottleneck that prevents the use of deep learning methods in such cases, as the models will overfit dramatically when using small training datasets [START_REF] Hu | When face recognition meets with deep learning: an evaluation of convolutional neural networks for face recognition[END_REF].

To address this issue, the use of big synthetic training datasets has been explored by a number of authors [START_REF] Jaderberg | Synthetic data and artificial neural networks for natural scene text recognition[END_REF], [START_REF] Papon | Semantic pose using deep networks trained on synthetic rgb-d[END_REF], [START_REF] Rogez | MoCap-guided data augmentation for 3D pose estimation in the wild[END_REF], [START_REF] Rozantsev | On rendering synthetic images for training an object detector[END_REF]. There are two important advantages of using synthetic data (i) one can generate as many training samples as desired, and (ii) it allows explicit control over the nuisance factors. For instance, we can synthesize face images of all desired viewpoints, whereas data collected from the internet might be mostly limited to near frontal views. Data synthesis has successfully been applied to diverse recognition problems, including text recognition [START_REF] Jaderberg | Synthetic data and artificial neural networks for natural scene text recognition[END_REF], scene understanding [START_REF] Papon | Semantic pose using deep networks trained on synthetic rgb-d[END_REF], and object detection [START_REF] Rozantsev | On rendering synthetic images for training an object detector[END_REF]. Several recent works [START_REF] Zhu | Face alignment across large poses: A 3d solution[END_REF], [START_REF] Feng | Cascaded collaborative regression for robust facial landmark detection trained using a mixture of synthetic and real images with dynamic weighting[END_REF], [START_REF] Masi | Do we really need to collect millions of faces for effective face recognition? ECCV[END_REF] proposed 3D-aided face synthesis techniques for facial landmark detection and face recognition in the wild.

Data augmentation is another technique that is commonly used to reduce the data scarcity problem [START_REF] Paulin | Transformation pursuit for image classification[END_REF], [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. This is similar to data synthesis, but more limited in that existing training images are transformed without affecting the semantic class label, e.g. by applying cropping, rotation, scaling, etc.

The main contribution of this paper is a solution for training deep CNNs using small datasets. To achieve this, we propose a data synthesis technique to expand limited datasets to larger ones that are suitable to train powerful deep CNNs. Specifically, we synthesize images of a 'virtual' subject c by compositing automatically detected face parts (eyes, nose, mouth) of two existing subjects a and b in the dataset in a fixed pattern. Images for the new subject are generated by composing a nose from an image of subject a with a mouth of an image of subject b. This is motivated by the observation that face recognition consists in finding the differences in the appearance and constellation of face parts among people. For a dataset with an equal number of faces per person, this method can increase a dataset of n images to one with n 2 images when using only 2 face parts (we use 5 parts in practice). A dataset like LFW can thus be expanded from a little over 10,000 images to a dataset of 100 million images.

Unlike existing face synthesis methods which use 3D models [START_REF] Zhu | Face alignment across large poses: A 3d solution[END_REF], [START_REF] Feng | Cascaded collaborative regression for robust facial landmark detection trained using a mixture of synthetic and real images with dynamic weighting[END_REF], [START_REF] Masi | Do we really need to collect millions of faces for effective face recognition? ECCV[END_REF], our method is a pure 2D method which is much easier to implement. In addition, our method works on different tasks from [START_REF] Zhu | Face alignment across large poses: A 3d solution[END_REF], [START_REF] Feng | Cascaded collaborative regression for robust facial landmark detection trained using a mixture of synthetic and real images with dynamic weighting[END_REF], [START_REF] Masi | Do we really need to collect millions of faces for effective face recognition? ECCV[END_REF]. Specifically, the methods [START_REF] Zhu | Face alignment across large poses: A 3d solution[END_REF], [START_REF] Feng | Cascaded collaborative regression for robust facial landmark detection trained using a mixture of synthetic and real images with dynamic weighting[END_REF] are used for facial landmark detection, while ours for face recognition. The approach [START_REF] Masi | Do we really need to collect millions of faces for effective face recognition? ECCV[END_REF] assumes a relatively large training data (500,000 images) already exists, while we assume the training data is very small (10,000 images).

We experimentally demonstrate that the synthesized large training datasets indeed significantly improve the generalization capacity of CNNs. In our experiments, we generate a training set of 1.5 million images using an initial labeled dataset of only 10,000 images. Using the synthetic data we improve the face verification rate from 78.97% to 95.77% on LFW. In addition, the proposed face synthesis is also used for NIR-VIS heterogeneous face recognition [START_REF] Ouyang | A survey on heterogeneous face recognition: Sketch, infra-red, 3d and low-resolution[END_REF] and improve the rank-1 face identification rate from 17.41% to 85.05%. With the synthetic data, we achieve state-of-the-art performance on both (1) LFW under the "unrestricted, label-free outside data" protocol and (2) CASIA NIR-VIS 2.0 database under rank-1 face identification protocol.

II. RELATED WORK

Our work relates to three research areas that we briefly review below: face recognition using deep learning methods (Section II-A), face data collection (Section II-B), and data augmentation and synthesis methods (Section II-C).

A. Face recognition using deep learning

Since face recognition is a special case of object recognition, good architectures for general object recognition may carry over to face recognition. Schroff et al. [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF] explored networks that are based on that of Zeiler & Fergus [START_REF] Zeiler | Visualizing and understanding convolutional networks[END_REF] and inception networks [START_REF] Szegedy | Going deeper with convolutions[END_REF]. DeepID3 [START_REF] Sun | Deepid3: Face recognition with very deep neural networks[END_REF] uses aspects of both inception networks and the very deep VGG network [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. Parkhi et al. [START_REF] Parkhi | Deep face recognition[END_REF] use the very deep VGG network, while Yi et al. [START_REF] Yi | Learning face representation from scratch[END_REF] use 3 × 3 filters but fewer layers.

DeepFace [START_REF] Taigman | DeepFace: Closing the gap to human-level performance in face verification[END_REF] uses a 3D model for pose normalization, by which all the faces are rotated to the frontal pose. In this way, pose variations are removed from the faces. Then an 8-layer CNN is trained using four million pose-normalized images.

DeepID [START_REF] Sun | Deep learning face representation from predicting 10,000 classes[END_REF], DeepID2 [START_REF] Sun | Deep learning face representation by joint identification-verification[END_REF], DeepID2+ [START_REF] Sun | Deeply learned face representations are sparse, selective, and robust[END_REF] all train an ensemble of small CNNs. The input of one small CNN is an image patch cropped around a facial part (face, nose, mouth, etc.). The same idea is also used in [START_REF] Liu | Targeting ultimate accuracy: Face recognition via deep embedding[END_REF]. DeepID uses only a classification-based loss to train the CNN, while DeepID2 includes an additional verification-based loss function. To further improve the performance, DeepID2+ adds losses to all the convolutional layers rather than the topmost layer only.

All the above methods train CNNs using large training datasets of 500,000 images or more. To the best of our knowledge, only [START_REF] Hu | When face recognition meets with deep learning: an evaluation of convolutional neural networks for face recognition[END_REF] uses small datasets to train CNNs (only around 10,000 LFW images) and achieves significantly worse performance on the LFW benchmark: 87% vs 97% or higher in [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF], [START_REF] Sun | Deeply learned face representations are sparse, selective, and robust[END_REF], [START_REF] Taigman | DeepFace: Closing the gap to human-level performance in face verification[END_REF]. Clearly, sufficiently large training datasets are extremely important for learning deep face representations.

B. Face dataset collection

Since big data is important for learning a deep face representation, several research groups have collected large datasets with 90,000 up to 2.6 million labeled face images [START_REF] Chen | Bayesian face revisited: A joint formulation[END_REF], [START_REF] Miller | Megaface: A million faces for recognition at scale[END_REF], [START_REF] Parkhi | Deep face recognition[END_REF], [START_REF] Sun | Hybrid deep learning for face verification[END_REF], [START_REF] Yi | Learning face representation from scratch[END_REF]. To achieve this, they collect face images from the internet, by querying for specific websites such as IMDb or general search engines for celebrity names. This data collection process is detailed in [START_REF] Parkhi | Deep face recognition[END_REF], [START_REF] Yi | Learning face representation from scratch[END_REF].

Existing face data collection methods have, however, two main weaknesses. First, and most importantly, internet-based collection of large face datasets is limited to visible spectrum images, and is not applicable to collect e.g. infrared face images. Second, the existing collection methods are expensive and time-consuming. It results from the fact that automatically collected face images are noisy, and manual filtering has to be performed to remove incorrectly labeled images [START_REF] Parkhi | Deep face recognition[END_REF].

The difficulty of collecting large datasets in some domains, e.g. for infrared imaging, motivates the work presented in this paper. To address this issue we propose a data synthesis method that we describe in the next section.

C. Data augmentation and synthesis

The availability of large supervised datasets is the key for machine learning to succeed, and this is true in particular for very powerful deep CNN models with millions of parameters. To alleviate data scarcity in visual recognition tasks, data augmentation has been used to add more examples by applying simple image transformations that do not affect the semanticlevel image label, see e.g. [START_REF] Decoste | Training invariant support vector machines[END_REF]. Examples of such transformations are horizontal mirroring, cropping, small rotations, etc. Since it is not always clear in advance which (combinations of) transformations are the most effective to generate examples that improve the learning the most, Paulin et al. [START_REF] Paulin | Transformation pursuit for image classification[END_REF] proposed to learn which transformations to exploit.

Data augmentation, however, is limited to relatively simple image transformations. Out-of-plane rotations, for example, are hard to accomplish since they would require some degree of 3D scene understanding from a single image. Pose variations of articulated objects are another example of transformations that are non-trivial to obtain, and generally not used in data augmentation methods.

Training models from synthetic data can overcome such difficulties, provided that sufficiently accurate object models are available. Recent examples where visual recognition systems have been trained from synthetic data include the following. Shotton et al. [START_REF] Shotton | Real-time human pose recognition in parts from single depth images[END_REF] train randomized decision forests for human pose estimation from synthesized 3D depth data. Jaderberg et al. [START_REF] Jaderberg | Synthetic data and artificial neural networks for natural scene text recognition[END_REF] use synthetic data to train CNN models for natural scene text recognition. Su et al. [START_REF] Su | Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views[END_REF] use synthetic images of objects to learn a CNN for viewpoint estimation. Papon and Schoeler [START_REF] Papon | Semantic pose using deep networks trained on synthetic rgb-d[END_REF] train a multi-output CNN that predicts class, pose, and location of objects from realistic cluttered room scenes that are synthesized on the fly. Weinmann et al. [START_REF] Weinmann | Material classification based on training data synthesized using a BTF database[END_REF] synthesize material images under different viewing and lighting conditions based on detailed surface geometry measurements, and use these to train a recognition system using a SIFT-VLAD representation [START_REF] Jégou | Aggregating local descriptors into a compact image representation[END_REF]. Ronzantsev et al. [START_REF] Rozantsev | On rendering synthetic images for training an object detector[END_REF] use rough 3D models to synthesize new views of real object category instances. They show that this outperforms more basic data augmentation using crops, flips, rotations, etc.

Data synthesis techniques are also used for face analysis. To improve the accuracy of facial landmark detection in the presence of large pose variations [START_REF] Feng | Cascaded collaborative regression for robust facial landmark detection trained using a mixture of synthetic and real images with dynamic weighting[END_REF], [START_REF] Zhu | Face alignment across large poses: A 3d solution[END_REF], a 3D morphable face models is used to synthesize face images in arbitrary poses. Similar data synthesis techniques are also used for poserobust face recognition [START_REF] Masi | Do we really need to collect millions of faces for effective face recognition? ECCV[END_REF]. Unlike 3D solutions, we propose a 2D data synthesis method to solve the problem of training deep CNNs using very limited training data.

III. SYNTHETIC DATA ENGINE

Human faces are well structured in the sense that they are composed of parts (eyes, nose, mouth, etc.) which are organized in a relatively rigid constellation. Face recognition relies on differences among people in the appearance of facial parts and their constellation. Motivated by this, our synthetic face images are generated by swapping one or more facial parts among existing "parent" images. In our work we use five face parts: right eye (RE), left eye (LE), nose (N), mouth (M) and the rest (R). See Figure 1 for an illustration. For simplicity, we only consider the synthesis using only two parent images in this work. Our synthesis method can easily be extended, however, to the scenario of more than two parent images.

A. Compositing face images

Suppose that we have an original dataset and let S denote the set subjects in the dataset, and let n i denote the number of images of subject i ∈ S. To synthesize an image, we select a tuple (i, j, c, s, t) where i ∈ S, j ∈ S correspond to two subjects that will be mixed, and s ∈ {1, . . . , n i } and t ∈ {1, . . . , n j } are indices of images of i and j that will be used. The bitcode c ∈ {0, 1} 5 defines which parts will be taken from each subject. A zero at a certain position in c means that the corresponding part will be taken from i, otherwise it will be taken from j. There are only 2 5 -2 = 30 valid options for b, since the codes 00000 and 11111 correspond to the original images of s and t respectively, instead of synthetic ones.

To synthesize a new image, we designate one of parent images as the "base" image from which we use the R (the rest) part, and the other as the "injection" image from which one or more parts will be pasted on the base image. Since the size of the facial parts in the two parent images are in general different, we re-size the facial parts of the injection image to that of the base image. The main challenge to implement the proposed synthesis method is to accurately locate the facial parts. Recently, many efficient and accurate landmark detectors have been proposed. We use four landmarks detected by the method of Zhang et al. [START_REF] Zhang | Facial landmark detection by deep multi-task learning[END_REF] to define the rectangular region that corresponds to each face part.

We refer to each choice of (i, j, c) with i = j as a "virtual subject" which consists of a mix of two existing subjects in the dataset. In total we can generate 30|S|(|S|-1)/2 different virtual subjects, and for each of these we can generate n i × n j samples. Note that if we set i = j we can in the same manner synthesize 30n i (n i -1)/2 new images for an existing subject.

Although some works [START_REF] Masi | Do we really need to collect millions of faces for effective face recognition? ECCV[END_REF], [START_REF] Taigman | Deepface: Closing the gap to human-level performance in face verification[END_REF] empirically verified the effectiveness of synthetic data, they did not give much insight into how. In our work, the synthetic data captures a dataset of richer intra-personal variations by generating a large number of images of the same identities, leading to a 'deeper' training set. Also, our engine can synthesize a large number of faces of new identities, generating a 'wider' training set. Thus the synthetic identities interpolate the whole space of pixelidentity mappings. Not surprisingly, a better CNN model can be trained using this deeper and wider training set. The methods of generating our deeper and wider training set are detailed in Section V-A.

B. Compositing artefacts

The synthetic faces present two types of artefacts: (I) hard boundary effects, and (II) inconsistent/unnatural intra-personal variations (lighting, pose, etc.) between facial patches. These are illustrated in Fig. 2. Note that the type I artefacts are generated by not only our method but also 3D synthesis methods such as [START_REF] Masi | Do we really need to collect millions of faces for effective face recognition? ECCV[END_REF], [START_REF] Taigman | Deepface: Closing the gap to human-level performance in face verification[END_REF], [START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF]. As shown in the top-right side of Fig. 2, the artefacts created by 3D methods are due to inaccurate 3D model to 2D image fitting. The inaccurate fitting makes the synthetic faces extract the pixels from background rather than facial areas, leading to bad facial boundaries.

Despite the existence of these artefacts, this synthetic data is still useful for training strong face recognition models. This can be understood from several perspectives: (1) Type I artefacts are common to all the synthetic faces in the training set, therefore the CNN does not learn to rely on artefacts as discriminative features coding for identity, i.e., it learns artefact invariance. This means its performance is not compromised when subsequently presented with artefact-free images at testing-time. Other studies have also shown that synthetic data still improves recognition performance, despite the presence of type I artefacts [START_REF] Masi | Do we really need to collect millions of faces for effective face recognition? ECCV[END_REF]. [START_REF] Ahonen | Recognition of blurred faces using local phase quantization[END_REF] The artefacts can be regarded as noise, which has been shown to improve model generalisation in a variety of settings by increasing robustness and reducing overfitting. For example in the case of CNNs, training with data augmentation in the form of specifically designed deformation noise is important to obtain good recognition performance [START_REF] Yu | Sketch-a-net: A deep neural network that beats humans[END_REF]; and in the case of de-noising auto encoders, training on images with noise, corruption and artefacts has been shown to improve face classification performance [START_REF] Vincent | Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion[END_REF].

(3) As a concrete example to understand how type II artefacts can improve performance, consider two synthetic images with the same identity label, but one a type II artefact on the mouth caused by illumination, e.g., in Fig. 2 (bottom left). Training to predict these images as having the same identity means that the CNN learns an illumination-invariant feature for the mouth. And similarly for other intra-personal variations (such as pose, expression). Thus while some artefact images look strange, they are actually a powerful form of data augmentation that helps the CNN to learn robustness to all these nuisance factors.

IV. FACE RECOGNITION PIPELINE

In this section we describe the different elements of our pipeline for face identification and verification in detail.

Our method 3D synthesis illumination pose Fig. 2: Top row: Type I hard boundary artefacts generated by our method (left) and 3D synthesis methods [START_REF] Masi | Do we really need to collect millions of faces for effective face recognition? ECCV[END_REF], [START_REF] Taigman | Deepface: Closing the gap to human-level performance in face verification[END_REF], [START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF] (right). Bottom row: Type II artefacts due to inconsistencies in illumination (left) and pose (right) generated by our method.

For our method we show the two parent images followed by the composited image.

A. CNN architectures

Face recognition in the wild is a challenging task. As described in Section II-A, the existing deep learning methods highly depend on big training data. Very little research investigates training CNNs using small data. Recently, Hu et al. [START_REF] Hu | When face recognition meets with deep learning: an evaluation of convolutional neural networks for face recognition[END_REF] evaluated CNNs trained using small datasets. Due to the limited training samples, they found the performance of CNNs to be worse than handcrafted features such as highdimensional features [START_REF] Chen | Blessing of dimensionality: Highdimensional feature and its efficient compression for face verification[END_REF] (0.8763 vs 0.9318). In this work, we use a limited training set of around 10,000 images to synthesize a much larger one of around 1.5 million images for CNN training. The synthesized training data captures various deformable facial patterns that are important to improve the generalization capacity of CNNs.

We use two CNN architectures. The first one, from [START_REF] Hu | When face recognition meets with deep learning: an evaluation of convolutional neural networks for face recognition[END_REF] has fewer filters and is referred as CNN-S. The second, from [START_REF] Yi | Learning face representation from scratch[END_REF], is much larger and referred as CNN-L. These two architectures are detailed in Table I. Using the CNN-L model we achieve state-of-the-art performance on the LFW dataset [START_REF] Huang | Labeled faces in the wild: a database for studying face recognition in unconstrained environments[END_REF] under 'unrestricted, label-free outside data' protocol.

B. NIR-VIS heterogeneous face recognition

NIR-VIS (near-infrared to visual) face recognition is important in applications where probe images are captured by NIR cameras that use active lighting which is invisible to the human eye [START_REF] Ouyang | A survey on heterogeneous face recognition: Sketch, infra-red, 3d and low-resolution[END_REF]. Gallery images are, however, generally only available in the visible spectrum. The existing methods for NIR-VIS face recognition include three steps: (i) illumination pre-processing, (ii) feature extraction, and (iii) metric learning. First, the NIR-VIS illumination differences cause the main difficulty of NIR-VIS face recognition. Therefore, illumination normalization methods are usually used to reduce these differences. Second, to reduce the heterogeneities of NIR and VIS images, illumination-robust features such as LBP are usually extracted. Third, metric learning is widely utilized, aiming at removing the differences of modalities and meanwhile keeping the discriminative information of the extracted features.

In this work, we also follow these three steps that are detailed in Section V-D. Unlike the existing work that extracts handcrafted features, we learn face representations using two CNN architectures described above. To our knowledge, we are the first to use deep CNNs for NIR-VIS face recognition. The main difficulty of training CNNs results from lack of NIR training images, which we address via data synthesis.

C. Network Fusion

Fusion of multiple networks is a widely used strategy to improve the performance of deep CNN models. For example, in [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] an ensemble of seven networks is used to improve the object recognition performance due to complementarity of the models trained at different scales. Network fusion is also successfully applied to learn face representations. DeepID and its variants [START_REF] Sun | Deep learning face representation by joint identification-verification[END_REF], [START_REF] Sun | Deep learning face representation from predicting 10,000 classes[END_REF], [START_REF] Sun | Deeply learned face representations are sparse, selective, and robust[END_REF] train multiple CNNs using image patches extracted from different facial parts.

The heterogeneity of NIR and VIS images is intrinsically caused by the different spectral bands from which they are acquired. The images in both modalities, however, are reflective in nature and affected by illumination variations. Illumination normalization can be used to reduce such variability, at the risk of loosing identity-specific characteristics. In this work, we fuse two networks that are trained using the original and illumination-normalized images respectively. This network fusion significantly boosts the recognition rate.

D. Metric Learning

The goal of metric learning is to make different classes more separated, and instances in the same class closer. Most approaches learn a Mahalanobis metric

d 2 A (x i , x j ) = (x i -x j ) T A(x i -x j ), (1)
which maximizes inter-class discrepancy, while minimizing intra-class discrepancy. Other methods, instead learn a generalized dot-product of the form

d 2 B (x i , x j ) = x T i Bx j . (2)
Metric learning methods are widely used for face identification and verification. Because identification and verification are two different tasks, different loss functions should be optimized to learn the metric. Joint Bayesian metric learning (JB) [START_REF] Chen | Bayesian face revisited: A joint formulation[END_REF] and Fisher linear discriminant analysis (LDA) are probably the two most widely used metric learning methods for face verification and identification respectively. In particular, LDA can be seen as a method to learn a metric of the form of Eq. ( 1), while JB learns a verification function that can be written as a weighted sum of Eq. ( 1) and [START_REF] Ahonen | Recognition of blurred faces using local phase quantization[END_REF]. In our work we use JB and LDA to improve the performance of face verification and identification respectively.

V. EXPERIMENTS

A. Data synthesis methods

Given a set of face images and their IDs, we define three strategies for synthesis: Inter-Synthesis, Intra-Synthesis, and Self-Synthesis. Inter-Synthesis synthesizes a new image using two parents from different IDs as shown in Fig. 1. The facial components of an Intra-Synthesized face are from different images with the same ID. Self-synthesis is a special case of Intra-Synthesis. Specifically, one given image synthesizes new images by swapping facial components of itself and its mirrored version. By virtue of Self-Synthesis, one input image can become maximum 32 images which have complementary information. In the view of NIR-VIS cross-modality, we also define 'cross-modality synthesis' which uses images from different modalities to synthesize a new one. Some synthetic images from the CASIA NIR-VIS 2.0 dataset with LSSF [START_REF] Xie | Normalization of face illumination based on large-and small-scale features[END_REF] illumination normalization are shown in Fig. 3. The reasons of using LSSF illumination normalization are detailed in Section V-D. As shown in Fig. 3, the results of Intra-Synthesis method are usually more natural than Inter-Synthesis method since the Intra-Synthesis method uses the same ID. However, as shown in the right of Fig. 3, Intra-Synthesis can also create artefacts due to large pose variations.

B. Implementation details

Before face synthesis, all the raw images are aligned and cropped to size 100×100 as in [START_REF] Yi | Learning face representation from scratch[END_REF] on both datasets. We train our models using images only from LFW and CASIA NIR-VIS2.0 databases. For the CNN-S model on both datasets, we set the learning rate as 0.001, and decrease it by 10 times every 4000 iterations, and stop training after 10K iterations. We find that dropout is not helpful for the small network, and report results obtained without it. For the CNN-L model on the NIR-VIS dataset, we set the learning rate as 0.01, and decrease it by 10 times every 8000 iterations, and stop training after 20K iterations. For the CNN-L model on the LFW dataset, we set the learning rate as 0.01, and decrease it by 10 times every 120K iterations, and stop training after 200K iterations. We set dropout rate as 0.4 for the pool5 layer of the CNN-L model. For both CNN-S and CNN-L models, the batch size is 128, momentum is 0.9, and decay is 0.0005. Softmax loss function is used to guide CNN training. The features used in our recognition experiments with CNN-S and CNN-L are FC-160 (160D) and Pool5 (320D), respectively. C. Face recognition in the wild 1) Database and protocol: Labeled Faces in the Wild (LFW) [START_REF] Huang | Labeled faces in the wild: a database for studying face recognition in unconstrained environments[END_REF] is a public dataset for unconstrained face recognition study. It contains 5,749 unique identities and 13,233 face photographs. The training and test sets are pre-defined in [START_REF] Huang | Labeled faces in the wild: a database for studying face recognition in unconstrained environments[END_REF]. For evaluation, the full dataset is divided into ten splits, and each time nine of them are used for training and the left one for testing. Our work falls in the protocol of 'Unrestricted, Label-Free Outside Data' as we use the identity information to train the neural network (softmax loss). Meanwhile, all face images are aligned using a model trained on unlabeled outside data. As a benchmark for comparison, we report the mean and standard deviation of classification accuracy.

Under LFW protocol, the training set in each fold is different. Therefore, the size of synthetic data and the original raw LFW data in Table II is averaged over 10 folds. We generate 1.5 million training images, including 1 million 'Inter-Syn' ones and 0.5 million 'Intra-Syn' ones, as defined in Section V-A.

2) Analysis of CNN model and synthetic data: We here analyse the trained model by visualising the synthetic images in feature space. We choose the images of two subjects (George W. Bush and Colin Powell), which have the largest number of images in LFW, and the synthetic images derived from the two subjects for analysis. To analyse the effects of replacing different facial components (eyes, nose, mouth), we only replace one patches of a particular facial component of Bush with ones from himself (Intra-Synthesis) or from Powell (Inter-Synthesis), as shown in Fig. 4. For each case, 100 images are synthesised. Therefore, there are 8 groups of images: 2 groups of original images (Bush and Powell), It means that the identity information is kept if Bush's eyes are replaced by those from himself. In contrast, a new identity space is generated by the synthetic images that replace Bush's eyes with Powell's. The same conclusion can be drawn from nose in Fig. 5(b). In Fig. 5(c), however, 'intra-Mouth' and 'inter-Mouth' are not well separated. It means that the mouth component is not very discriminative between people. Fig. 5(d) and (e) contrast the results when training with original and synthetic (Fig. 5(d)) versus original data only (Fig. 5(e)). Fig. 5(d) is relatively more discriminative for identity, particularly the synthetic identities. Thus we can see that training with synthetic data is important to interpolate the identity space, and thus achieve good results for unseen identities -as required at testing time.

3) Impact of synthetic data: Table III analyzes the importance of using the synthetic data. First, CNN-S trained using synthetic data ('Intra-Syn' and 'Inter-Syn') outperforms greatly the model trained using just the original LFW images, showing the importance of data synthesis. Second, 'Inter-

-0.5 0 0.5 1 -3 -2 -1 0 1 2 3 1st PCA component 2nd PCA component Mouth GeorgeWBush ColinPowell intra-Mouth inter-Mouth (c) -4 -3 -2 -1 0 1 -3 -2 -1 0 1 2 3 1st PCA component 2nd PCA component Syn. Model GeorgeWBush ColinPowell intra-Eyes inter-Eyes intra-Mouth inter-Mouth intra-Nose inter-Nose (d) -4 -3 -2 -1 0 1 2 -3 -2 -1 0 1 2 3 1st PCA component 2nd PCA component Org. Model GeorgeWBush ColinPowell intra-Eyes inter-Eyes intra-Mouth inter-Mouth intra-Nose inter-Nose (e)
Fig. 5: Face distributions. 'GeorgeWBush' and 'ColinPowell' denote the original images from these 2 subjects. 'intra-X' and 'inter-X' denote the synthetic images with component replaced by another example from Bush himself and from Powell respectively, as in Fig. 4. Features of (a)-(d) are extracted using a CNN trained using original and synthetic data, while (e) using original data only. One colour represents one (real or synthetic) subject. Syn' works slightly better than 'Intra-Syn', since 'Inter-Syn' can capture richer facial variations. Third, combining 'Inter-Syn' and 'Intra-Syn' works better than either of them because they capture complementary variations. Fourth, averaging the features of 32 'Self-Syn' ('32-Avg' in Table III and defined in Section V-A ) images works consistently better than that of one single test image ('single' in Table III ). Fifth, CNN-L works consistently better than CNN-S using either original LFW or synthetic images because deeper architecture has stronger generalization capacity. Finally, metric learning further enhances the face recognition performance.

4) Impact of Image Blending: In Section III, we speculated the existence of artifacts ('hard boundaries') can improve the robustness of the model. We now experimentally investigate this by reducing such 'hard boundaries' using image blending. In particular, we implemented Poisson image editing [START_REF] Pérez | Poisson image editing[END_REF] to smooth these boundaries. In Fig. 6, we show some results of Poisson image blending. From Fig. 6, as expected, Poisson blending does make the boundaries much smoother compared with our synthesis method. In Table III, we compare the results with and without Poisson image editing. We can see that recognition accuracy based on training data synthesised with 'hard' boundaries is, somewhat, higher than with Poisson blending. This supports the idea that 'hard' boundaries provide a source of noise that is beneficial in making network training more robust detailed in Section III.

5) Comparison with the state-of-the-art: [START_REF] Simonyan | Fisher Vector Faces in the Wild[END_REF] 93.03 ± 1.05 HPEN [START_REF] Zhu | High-fidelity pose and expression normalization for face recognition in the wild[END_REF] 95.25 ± 0.36 MDML-DCPs [START_REF] Ding | Multi-directional multilevel dual-cross patterns for robust face recognition[END_REF] 95.58 ± 0.34 The proposed 95.77 ± 0.38 tion, most of state-of-the-art solutions rely on extremely high dimensional feature vectors derived from densely sampled local features on the face image. In contrast, we just use a 320-dimensional feature vector, which is much more compact. 6) Non-CNN methods using synthetic data: Above we demonstrated the effectiveness of synthetic data to train CNNs. We now consider its usefulness to improving methods based on traditional hand-crafted features. One typical hand-crafted feature used for unconstrained face recognition problem is high dimensional LBP (HD-LBP) [START_REF] Chen | Blessing of dimensionality: Highdimensional feature and its efficient compression for face verification[END_REF]. We extract the HD-LBP feature using the open source code [START_REF] Chen | Review and implementation of high-dimensional local binary patterns and its application to face recognition[END_REF]. From Table V, we see that HD-LBP with JB metric learning trained using original LFW images works much better that without JB (89.02% vs 84.13%), showing the expected effectiveness of metric learning. More interestingly, we see that training JB using both original and synthetic images outperforms that using original images only, 91.03% vs 89.02%. This shows that our synthetic data approach is also useful in combination with conventional hand-crafted features. Fig. 7: Recognition rate (%) on LFW using the original CASIA WEBFACE training data (0.5M) and synthetic data (0-3M).

7) Impact of synthetic images on larger training sets: Our original motivation was to learn effective face representations from small datasets. We demonstrated the effectiveness of generating synthetic data to expand a small training set (LFW, 5K identities and 10K images) in Table III. Recently, some bigger training sets of face images in the wild have been released, such as CASIA WEBFACE [START_REF] Yi | Learning face representation from scratch[END_REF] (10K identities and 0.5M images). We conduct experiments using the latter dataset to assess whether our data synthesis strategy is also useful for such larger datasets. As before, we keep the ratio 2:1 of 'Inter-Syn' and 'Intra-Syn' synthetic images. To investigate the effects of the size of synthetic data, we generate six sets of synthetic images: {0.5M, 1M, 1.5M, 2M, 2.5M, 3M} images. We trained the CNN-L network using the original CASIA WEBFACE images, plus a variable amount of synthetic data. From Fig 7, we can see that the recognition rate on LFW increases with the amount of synthetic data. This demonstrates that our data synthesis strategy is still very effective even with relatively large datasets. D. NIR-VIS face recognition 1) Database and protocol: The largest face database across NIR and VIS spectrum so far is the CASIA NIR-VIS 2.0 face database [START_REF] Li | The casia nir-vis 2.0 face database[END_REF]. It contains 17,580 images of 725 subjects which exhibit intra-personal variations such as pose and expression. This database includes two views: view 1 for parameter tuning and view 2 including 10 folds for performance evaluation. During test, the gallery and probe images are VIS and NIR images respectively, simulating the scenario of face recognition in the dark environment. The rank 1 identification rate including the mean accuracy and standard deviation of 10 folds are reported.

Because the images of CASIA NIR-VIS2.0 are from two modalities (NIR and VIS), we applied 'cross-modality synthesis' to synthesize new images. The size of synthesized data is detailed in Table VI.

2) Illumination normalization and feature extraction: Illumination Normalization (IN) methods are usually used to narrow the gap between NIR and VIS images. To investigate the impact of IN, we preprocessed images using three popular IN methods: illumination normalization based on largeand small-scale features (LSSF) [START_REF] Xie | Normalization of face illumination based on large-and small-scale features[END_REF], Diffence-of-Gaussian filtering-based normalization (DOG), and single-scale retinex (SSR) [START_REF] Jobson | Properties and performance of a center/surround retinex[END_REF]. We train CNN-S and CNN-L using illumination normalized and non-normalized images. For simplicity, only the images from CASIA NIR-VIS2.0 excluding synthetic ones are used. Fig. 8 shows the face recognition rates at different training iterations using different input images for the CNN-S and CNN-L networks. The results show the effectiveness of IN, and LSSF achieves the best performance due to its strong capacity of removing illumination effects without affecting identity information. As for the LFW experiments in Section V-C, CNN-L works better than CNN-S.

We also extracted LBP features from the LSSF-normalized images, and achieve 12. these two problems, we also use the synthetic data generated from LFW images defined in Table II.

Table VII compares the results achieved by these two sources of synthetic data. First, the accuracy achieved by using the synthetic data generated from CASIA NIR-VIS2.0 database is 34.13 ± 2.13, in comparison with 17.41 ± 3.76 without synthetic data. The significant improvement shows the effectiveness of data synthesis. Second, the model trained using raw and synthetic LSSF-normalized LFW images greatly outperforms those synthetic CASIA NIR-VIS2.0 images: 66.37 ± 1.45 vs. 38.45 ± 2.08, although NIR images are completely unseen during training. The reasons are 2-fold:

(1) LFW images contains more subjects which can capture more facial variations as analyzed above. (2) LSSF can greatly reduce the gap between NIR and VIS, therefore, LSSFnormalized LFW synthetic images can generalize well to LSSF-normalized NIR images. To further improve the face recognition performance, we trained the network using both raw and synthetic data from both sources (LFW+CASIA NIR-VIS). The face recognition rate is improved from 66.37 ± 1.45 to 68.97 ± 1.24, showing the value provided by our of bigger synthetic dataset.

4) Comparison with the state-of-the-art: The CNN-L models in Table VIII are all trained using synthetic LFW data. First, LSSF-normalized and Original LFW synthetic data achieve very comparable performance: 68.97% vs. 69.11%. However, the fusion (averaging) of these two features can significantly improve the face recognition rates. It shows the 56.6 ± 2.4 Dictionary Learning [START_REF] Juefei-Xu | Nir-vis heterogeneous face recognition via cross-spectral joint dictionary learning and reconstruction[END_REF] 78.46 ± 1.67 C-CBFD + LDA [START_REF] Lu | Learning compact binary face descriptor for face recognition[END_REF] 81.8 ± 2.3 CNN + LDML [START_REF] Saxena | Heterogeneous face recognition with CNNs[END_REF] 85.9 ± 0.9 Gabor + RBM [START_REF] Yi | Shared representation learning for heterogeneous face recognition[END_REF] 86.2 ± 1.0 fusion can keep the discriminative facial information but remove the illumination effects. Second, not surprisingly, metric learning can further improve the performance. The metric learning method used here is LDA, which is the most widely used one for face identification. Finally, Table VIII compares the proposed method against the state-of-the-art solutions [START_REF] Lu | Learning compact binary face descriptor for face recognition[END_REF], [START_REF] Juefei-Xu | Nir-vis heterogeneous face recognition via cross-spectral joint dictionary learning and reconstruction[END_REF]. [START_REF] Lu | Learning compact binary face descriptor for face recognition[END_REF] uses a designed descriptor that performs better in this dataset compared with other generic hand-crafted features, and LDA can further improve the accuracy. Our method significantly outperforms [START_REF] Lu | Learning compact binary face descriptor for face recognition[END_REF] when metric learning is not used (79.96% vs. 56.6%), and maintains superior performance when metric learning is used (85.05% vs. 81.8%). [START_REF] Juefei-Xu | Nir-vis heterogeneous face recognition via cross-spectral joint dictionary learning and reconstruction[END_REF] tries to solve the domain shift between two data sources (NIR and VIS) by a cross-modal metric learning: it assumes that a pair of NIR and VIS images shares the same sparse representation under two jointly learned dictionaries. Our method improves over [START_REF] Juefei-Xu | Nir-vis heterogeneous face recognition via cross-spectral joint dictionary learning and reconstruction[END_REF] with a 7% margin without such an extra step of dictionary learning. Concurrently to our work, Saxena & Verbeek [START_REF] Saxena | Heterogeneous face recognition with CNNs[END_REF] obtained a comparable performance of 85.9% using a crossmodal version of LDML metric learning [START_REF] Guillaumin | Is that you? Metric learning approaches for face identification[END_REF], albeit using CNN features learned from the 500,000 face images of the CASIA WEBFACE dataset. Finally, Yi et al. [START_REF] Yi | Shared representation learning for heterogeneous face recognition[END_REF] obtained the best performance of 86.1% using an approach that extracts 40 dim. Gabor features at 176 local face regions, and uses these to train 176 different restricted Boltzmann machines (RBMs) specialized to model the modality shift at each face region. Note that unlike our approach based on feed-forward CNNs, their approach requires Gibbs sampling at test-time to infer the face representations.

VI. CONCLUSIONS AND FUTURE WORK

Recently, convolutional neural networks have attracted a lot of attention in the field of face recognition. However, deep learning methods heavily depend on big training data, which is not always available. To solve this problem in the field of face recognition, we propose a new face synthesis method which swaps the facial components of different face images to generate a new face. With this technique, we achieve stateof-the-art face recognition performance on LFW and CASIA NIR-VIS2.0 face databases.

In the future, we will apply this technique to more applications of face analysis. For example, the proposed data synthesis method can easily be used in training CNN-based face detection, facial attribute recognition, etc. More generally, the method applies to any objects which are well structured. For example, the human body is well structured and human images can be synthesised using this method. The synthetic images can be used to train deep models for pose estimation, pedestrian detection, and person re-identifiation.

Fig. 1 :

 1 Fig. 1: Illustration of the face synthesis process using five parts: left-eye, right-eye, nose, mouth and the rest. Parent images P0 and P1 (top) are mixed by using the eyes of P1 and the other parts of P0 to form the synthetic image (bottom).

Fig. 3 :

 3 Fig. 3: Left: Inter-Synthesis. Middle: cross-modality Intra-Synthesis. Right: Intra-Synthesis.

Fig. 4 :

 4 Fig. 4: Samples of Inter-Synthesis and Intra-Synthesis

Fig. 6 :

 6 Fig. 6: Comparison of synthesis with/without blending. Row 1-2: Input image pairs, Row 3: Our synthesis with 'hard boundaries', Row 4: Poisson blending

Fig. 8 :

 8 Fig. 8: Effect of illumination normalization methods using the CNN-S (top) and CNN-L networks.

TABLE I :

 I Our two CNN architectures

	CNN-L	CNN-S
	conv1
	32×3×3, st.1; 64 × 3 × 3, st.1	16×3×3, st.1; 16 × 3 × 3, st.1
	x2 maxpool, st.2	x2 maxpool, st.2
	conv2
	64×3×3, st.1; 128 × 3 × 3, st.1	32×3×3, st.1
	x2 maxpool, st.2	x2 maxpool, st.2
	conv3
	96×3×3, st.1 ; 192 × 3 × 3, st.1	48×3×3, st.1
	x2 maxpool, st.2	x2 maxpool, st.2
	conv4
	128×3×3, st.1 ; 256 × 3 × 3, st.1 x2 maxpool, st.2	-
	conv5
	160×3×3, st.1 ; 320 × 3 × 3, st.1 x7 avgpool, st.1	-
	fully connected
	Softmax-5000	FC-160 Softmax-5000

TABLE II :

 II Training data synthesized from LFW

			IDs	Images Images/ID
		Intra-Syn	5K	500K	100
	Synthetic	Inter-Syn	5K	1M	200
		Total	10K	1.5M	150
	Raw	5K	10K	2

TABLE III :

 III Table IV compares our method with state-of-the-art methods. All methods listed in Table IV except ours use hand-crafted features, again underlining the difficulty of training deep CNNs with small data. The best deep learning solution [47] recorded in official benchmark achieves 91.75%, and ours is 4% better. In addi-Comparison of synthetic data methods on LFW

	Architecture	Metric learning	Training data	single (%)	32-Avg (%)
			Original	78.97 ± 0.78	-
	CNN-S	-	Intra-Syn+Original Inter-Syn+Original	83.03 ± 0.56 83.18 ± 0.74	83.93 ± 0.49 84.35 ± 0.65
			Intra-Syn+Inter-Syn+Original	85.61 ± 0.71	86.98 ± 0.57
		-	Original	85.03 ± 0.98	-
		JB [5]	Original	87.03 ± 0.69	-
	CNN-L	-JB [5]	Intra-Syn+Inter-Syn+Original Intra-Syn+Inter-Syn+Original	94.88 ± 0.66 95.32 ± 0.38	95.13 ± 0.53 95.77 ± 0.38
		-	Intra-Syn+Inter-Syn+Original (blending)	94.27 ± 0.65	94.46 ± 0.51
		JB [5]	Intra-Syn+Inter-Syn+Original (blending)	94.61 ± 0.35 95.05 ± 0.34
	Intra-Syn		Inter-Syn

TABLE IV :

 IV Comparison with state-of-the-art methods on LFW

	Methods	Accuracy (%)
	Fisher vector faces

TABLE V :

 V Hand-crafted features on LFW

				Methods		Accuracy (%)
				HD-LBP		84.13 ± 1.76
			HD-LBP+JB (original)	89.02 ± 1.11
		HD-LBP+JB (original + synthetic)	91.03 ± 1.06
		97.3
		97.2
		97.1
		97
	accuracy (%)	96.8 96.9
	LFW	96.7
		96.6
		96.5
		96.4
		96.3
		0.5+0	0.5+0.5	0.5+1	0.5+1.5	0.5+2	0.5+2.5	0.5+3
				training images (Million)

TABLE VI :

 VI Synthetic data using CASIA NIR-VIS2.0 database

			IDs	Images	Images/ID
		Intra-Syn	357	90K	250
	Synthetic	Inter-Syn	1K	150K	150
		Total	1.4K	240K	170
	Original	357	8.5K	23

TABLE VII :

 VII Evaluation of the impact of synthetic data

		Training Data CASIA NIR-VIS2.0 LFW	Accuracy(%)
	Baseline	Raw	-	17.41 ± 3.76
		Raw+Syn	-	34.13 ± 2.13
	Synthetic	-	Raw	38.45 ± 2.08
	Data	-	Raw+Syn	66.37 ± 1.45
		Raw+Syn	Raw+Syn	68.97 ± 1.24

TABLE VIII :

 VIII Comparisons with the state of the art on the CASIA NIR-VIS2.0 dataset.

	Method	Accuracy (%)
	CNN-L	Training Data Network Fusion	Original LSSF Original+LSSF	69.11 ± 1.21 68.97 ± 1.24 79.96 ± 1.18
		Metric Learning LDA (Original+LSSF) 85.05 ± 0.83
		C-CBFD [26]
	State-of-the-art

ACKNOWLEDGMENT This work is supported by European Unions Horizon 2020 research and innovation program under grant No 640891, the Science and Technology Plan Project of Hunan Province (2016TP1020), the Natural Science Foundation of China (No. 61502152), and the French research agency contracts ANR-16-CE23-0006 and ANR-11-LABX-0025-01. We gratefully acknowledge NVIDIA for the donation of the GPUs for this research.


Joël Legrand
email: joel.legrand@loria.fr

Yannick Toussaint

Chedy Raïssi

Adrien Coulet

Tree-LSTM and Cross-Corpus Training for Extracting Biomedical Relationships from Text

Keywords: Deep learning, NLP, Tree-LSTM, Cross-corpus training, relationship extraction, pharmacogenomic relationships

A key aspect of machine learning-based relationship extraction algorithms is the availability of training data. Manually annotated corpora are valuable resources for this task, but the time and expertise required for their development explain that still few corpora are available. For tasks related to precision medicine, most of them are rather small (i.e., hundreds of sentences) or focus on specialized relationships (e.g., drug-drug interactions) that rarely fit what one wants to extract. In this paper, we experiment Tree-LSTM, to extract relationships from biomedical texts with high performance. In addition we show that relatively large corpora, even when focusing on a distinct type of relationships, can be use to improve the performance of deep learning-based system for relationship extraction tasks for which initial resources are scarce.

Introduction

With the exponential growth of biomedical publications, it becomes difficult for researchers and clinicians to keep themselves up-to-date. In precision medicine, which takes into account individual variability in genes, environment, and lifestyle for each patient, this problem is even more acute, since it involves many research areas (such as pharmacogenomics, toxocogenomics and environment impact studies), thus many more publications. Relationship Extraction (RE) from the literature has been proposed to extract automatically structured information from unstructured text. In particular, deep learning methods have demonstrated good ability for such task [START_REF] Zeng | Relation classification via convolutional deep neural network[END_REF]. One drawback of these methods is that they generally require a large amount of annotated data in order to obtain reasonable performances. As a first step towards RE for precision medicine, we study the extraction of pharmacogenomic relationship, i.e., relationships between individual gene variations and drug response phenotypes. But to our knowledge, no adequate corpus exists with these specific relationships annotated. Building one may be particularly time consuming and expensive since it involves complex entities and requires trained annotators. However, several large and small corpora have been manually annotated with other type of biomedical relationships and made available. Because these corpora share the same language (i.e., English) and thus a common syntax, one might wonder if these resources developed for slightly different task may be reused for extracting relationships for which resources are scarce, i.e., only few hundred of annotated sentences are available.

Beside deep learning methods mentioned above, many approaches for RE have been proposed. Co-occurrence-based methods assumes that two entities mentioned frequently in the same unit of text (such as a sentence or a paragraph) are related [START_REF] Garten | Pharmspresso: a text mining tool for extraction of pharmacogenomic concepts and relationships from full text[END_REF]. Rule-base methods use manually designed, or learned, rules consisting of word morphosyntactic features or sentence-level syntactic features [START_REF] Coulet | Using text to build semantic networks for pharmacogenomics[END_REF][START_REF] Fundel | Relexrelation extraction using dependency parse trees[END_REF]. These methods have the advantage of requiring few or no annotated data. Also, rules can often be interpreted, since it is possible to associate a meaning to a rule. If the set of rules has been learned, these interpretations can help experts in better understanding a phenomena. In contrast, machine learning methods learn statistical models by training over annotated corpora and then recognize relationships in unannotated text [START_REF] Guodong | Exploring various knowledge in relation extraction[END_REF]. These methods tend to achieve better performances than the formers.

Within machine learning methods, deep learning ones are used to model complex structures such as natural language and successfully applied to various Natural Language Processing (NLP) tasks. In particular, it as been applied to RE with success from annotated corpora . While other methods mainly depend on the quality of extracted features derived from preexisting NLP systems, deep learning models automatically learn lexical features using continuous word vector representations and sentence level features using deep neural network such as Convolutional Neural Network (CNN) [START_REF] Collobert | Natural language processing (almost) from scratch[END_REF] or Recursive Neural Networks (RNN) [START_REF] Socher | Parsing with compositional vector grammars[END_REF]. Despite their advantages, these models strongly depend on the existence of large training corpora, which make them difficult to use for tasks associated with scarce resources.

In this paper we propose, within a specific RE task for which only few training data are available, to investigate how large annotated corpora may be used to improve performances of deep neural networks. For that purpose, we used the treestructured Long Short Term Memory (Tree-LSTM) model [START_REF] Sheng | Improved semantic representations from tree-structured LSTM networks[END_REF], for which variants have proven to be suitable and effective for RE [START_REF] Miwa | End-to-end relation extraction using LSTMs on sequences and tree structures[END_REF]. We experimented using two relatively small corpora (SNPPhenA and EU-ADR) along with two larger corpora (SemEval 2013 DDI and ADE). Section 2 presents various deep learning methods used for RE while Section 3 details the Tree-LSTM used in our experiments. Section 4 describes the set of corpus used in this study and Section 5 presents our experiments and results.

Related work

Deep learning models, based on continuous word representations, have been proposed to overcome the problem of sparsity inherent to NLP [START_REF] Huang | Distributional representations for handling sparsity in supervised sequence-labeling[END_REF]. In Collobert et al. [START_REF] Collobert | Natural language processing (almost) from scratch[END_REF], the authors proposed an unified CNN architecture to tackle various NLP problems traditionally handle with statistical approaches. They obtained state-of-theart performances for several tasks, while avoiding the hand design of task specific features. These results led to progress on NLP topics such as machine translation [START_REF] Cho | On the properties of neural machine translation: Encoder-decoder approaches[END_REF], question-answering [START_REF] Bordes | Question answering with subgraph embeddings[END_REF] and RE.

In particular, Zeng et al. [START_REF] Zeng | Relation classification via convolutional deep neural network[END_REF] showed that CNN models can also be applied to the task of RE. They built such a model to learn a sentence representation, given word and word position embeddings. This representation is then used to feed a softmax classifier [START_REF] Christopher | Pattern recognition and machine learning[END_REF]. To improve the performance of the RE, other authors consider elements of syntax within the embedding provided to the model: Xu et al. [START_REF] Xu | Semantic relation classification via convolutional neural networks with simple negative sampling[END_REF] use the path of grammatical dependencies between two entities, which is provided by a dependency parsing; Yang et al. [START_REF] Yang | A position encoding convolutional neural network based on dependency tree for relation classification[END_REF] include the relative positions of words in a dependency tree. They also take dependency based context (child and parent nodes) into account during the convolution.

While these CNN models incorporate syntactic knowledge in their embeddings, other approaches go further by proposing neural networks which topology is adapt-ing to the syntactic structure of the sentence. In particular, Recursive Neural Networks (RNN) have been proposed to adapt to tree structures resulting from constituency parsing [START_REF] Legrand | Joint RNN-based greedy parsing and word composition[END_REF][START_REF] Socher | Parsing with compositional vector grammars[END_REF]. In that vein, Tai et al. [START_REF] Sheng | Improved semantic representations from tree-structured LSTM networks[END_REF] introduced a Tree-LSTM, a generalization of LSTM for tree-structured network topologies, which allows to process trees with arbitrary branching factors.

The first model to make use of RNN for a RE task was proposed by Liu et al. [START_REF] Liu | A dependency-based neural network for relation classification[END_REF]. The authors introduced a CNN-based model applied on the shortest dependency path, augmented with a RNN-based feature designed to model subtrees attached to the shortest path. Miwa and Bansal [START_REF] Miwa | End-to-end relation extraction using LSTMs on sequences and tree structures[END_REF] introduced a variant of the Tree-LSTM used to compute bidirectional (bottom-up and top-down) tree representations that perform relationship classification. Their model use different weight matrices depending on whether a node belong to the shortest path or not.

For the extraction of biomedical relationships, CNN have been successively applied by adapting the Multichannel Variable-Size CNN [START_REF] Yin | Multichannel variable-size convolution for sentence classification[END_REF] to the extraction of drug-drug interactions (DDI) [START_REF] Liu | Drug-drug interaction extraction via convolutional neural networks[END_REF] and protein-protein interactions [START_REF] Quan | Multichannel convolutional neural network for biological relation extraction[END_REF].

Model

In this paper we used the Tree-LSTM model described in Tai et al. [START_REF] Sheng | Improved semantic representations from tree-structured LSTM networks[END_REF] and more specifically its Child-Sum version. This model is suitable for processing dependency trees since it handles trees with arbitrary branching factors and no order between children of a node. The model computes a score for each possible type or relationship (e.g. negative, positive or speculative) between two identified entities. The number of possible relationship types depends on the task (see Section 4). In this section, we first introduce the features used in this study; then, we detail the Tree-LSTM model; finally, we present the scoring layer.

Input layer

Our Tree-LSTM network is fed with word embeddings (i.e., continuous vectors) of dimension d w . Extra entity embeddings of size d e are used to distinguish the two entities involved in a relation from the other words. The first entity is mapped to the embedding first entity, the second to the embedding second entity and all the other words to the embedding other. Word and entity embeddings are concatenated to form the input of the Tree-LSTM.

Tree-LSTM

The Tree-LSTM model processes the dependency tree associated with an input sentence in a bottom-up manner. This is done by recursively processing the nodes of the tree, using their child representations as input. The transition function for a node j and a set of children C(j) is given by the following set of equations:

ht = k∈C(j) h k i j = σ(W (i) x j + U (i)
hj + b (i) )

f jk = σ(W (f ) x j + U (f ) h k + b (f ) ) o j = σ(W (o) x j + U (o) hj + b (o) ) u j = tanh(W (u) x j + U (u)
hj + b (u) )

c j = i j u j + k∈C(j) f jk c k h j = o j tanh(c j )
, where σ denotes the logistic function, the element-wise multiplication, x j ∈ R dw+de is the input for node j, h k ∈ R d h is the hidden state of the k th child. Each Tree-LSTM unit is a collection of vectors: an input gate i j , a forget gate f jk , an output gate o j , a memory cell c j and and hidden state h j . The matrices W and U and the vectors b are the weight and bias parameters to train.

Scoring layer

The Tree-LSTM layer processes any dependency tree recursively from its leaves to its parent nodes to lastly provide a representation for the root node, i.e., the node that spans all the other nodes. This representation takes the entire sentence into account and is used to feed a single layer neural network classifier, which outputs a score for each possible class. These scores are interpreted as probabilities using a softmax classifier [START_REF] Christopher | Pattern recognition and machine learning[END_REF].

Datasets

We explore how RE tasks that focus on a type of biomedical relationships associated with scarce resources may take advantage from additional resources, in other words how completing a small training corpus with a larger one may help the RE task, even if the latter is annotated with a different type of relationship. For this purpose, we selected two relatively small corpora, SNPPhenA and the EU-ADR corpus, and two larger, the SemEval 2013 DDI corpus and the ADE corpus. All these corpora are publicly available and focus on relationships between two of the three entities of interest in pharmacogenomics (i.e. gene or gene variations, drugs and phenotypes). Table 4.2 summarizes the main characteristics of these four corpora and the following section details them.

Small corpora

-SNPPhenA [START_REF] Bokharaeian | Snpphena: a corpus for extracting ranked associations of single-nucleotide polymorphisms and phenotypes from literature[END_REF] is a corpus of abstracts of biomedical publications, obtained from PubMed1 , annotated with two type of entities: single nucleotide polymorphisms (SNPs) and phenotypes. Relationships between these entities are also annotated and classified in 3 categories: positive, negative and neutral relationships. neutral relationships is used when no relationship is mentioned in the sentence between the two entities. -EU-ADR [START_REF] Van Mulligen | The EU-ADR corpus: annotated drugs, diseases, targets, and their relationships[END_REF] is a corpus of abstracts obtained from PubMed and annotated with drugs, disorders and targets (proteins/genes or gene variants) entities. The corpus is composed of 3 subcorpora, focusing on target-disease, target-drug and drug-disease relationships. Each of them consist of 100 abstracts. Relationships are classified in 3 categories: positive, speculative and negative associations (PA, SA and NA respectively). In [START_REF] Bravo | Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research[END_REF], performances are assessed over the TRUE class, which is composed of the classes PA, SA and NA, in contrast with the FALSE class made with sentences where the two entities co-occur, with no relationship between them.

Experiments

Training

Following Collobert et al. [START_REF] Collobert | Natural language processing (almost) from scratch[END_REF], our LSTM network was trained by minimizing a loglikelihood function over the training data. All parameters, including weight, biases and embeddings were updated via Backpropagation Through Structure (BPTS) [START_REF] Goller | Learning task-dependent distributed representations by backpropagation through structure[END_REF].

Experimental Settings

Dependency trees were obtained using the Stanford Parser [START_REF] Chen | A fast and accurate dependency parser using neural networks[END_REF]. Hyper-parameters were tuned using 10 fold cross-validation and fixed to d w = 100, d e = 10, d h = 200. We applied dropout regularization [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF] after every Tree-LSTM unit and after the embedding layers. The drop probability for each connexion was fixed to 0.25. Word embeddings were pre-trained using the method described in Lebret and Collobert [START_REF] Lebret | Word emdeddings through hellinger pca[END_REF] and trained on PubMed abstracts. We evaluated performances in terms of precision (P), recall (R) and f-measure (F). For multi-label classification, we report the macro-average performance. Because no proper test corpus is provided with EU-ADR, we performed a 10 fold cross-validation using 10% of the corpus for the validation and 10% for the test of our models. For SNPPhenA, we performed a cross-validation using 10% of the corpus for the validation and the provided test corpus for testing. Each result is an average of 5 experiments starting from different random weight initialization.

Cross-corpus study

In this subsection, we present our cross-corpus training strategy and its results. For the cross-corpus experiments, the same network was used for the different corpora (i.e., same embedding layer and Tree-LSTM weights), except for the scorer, which is different for each corpus as the number and types of relationship may change. During the training phase, we randomly pick training sentences from the mixed corpora. Table 2 presents the results of the cross-corpus study. We observe that using additional data consistently improved the performances and interestingly, this phenomenon occurs even for corpora with different types of entities such as the combination of SNPPhenA and SemEval 2013 DDI.

Comparison with the state of the art

Table 3 presents a comparison of performances obtained with our approach versus two state-of-the-art systems applied to the RE task associated respectively with SNPPhenA and EU-ADR, reported in Bokharaeian et al. [START_REF] Bokharaeian | Snpphena: a corpus for extracting ranked associations of single-nucleotide polymorphisms and phenotypes from literature[END_REF] and Bravo et al. [START_REF] Bravo | Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research[END_REF].

Our results for the SNPPhenA corpus are obtained using, for each fold, an ensemble of 5 models starting from different random initialization, following Legrand and Collobert [START_REF] Legrand | Joint RNN-based greedy parsing and word composition[END_REF]. We report the 10 folds average performance. For the EU-ADR experiments, we did not observed any improvement using our ensemble of models. Both state-of-the-art systems use a combination of a shallow linguistic kernel with a kernel that exploits deep syntactic features. Our approach outperforms the performances reported for SNPPhenA and for the drug-disease subcorpus or EU-ADR. For the two other EU-ADR subcorpora, our approach achieves similar performances, while relying on few automatically extracted features whereas the two baselines use several tuned orthographic and morpho-syntactic features. 3. Performance comparison with the state of the art in terms of precision (P), recall (R) and f1-score (F).

Conclusion

In this paper, we empirically demonstrated that corpora developed for specific RE tasks may be use to improve the performance of deep learning-based system for other RE tasks, for which initial resources are scarce. In addition, our rather simple approach led to state-of-the-art results with the SNPPhenA corpus and to near state-of-the-art results with the EU-ADR corpora. This study is an interesting first step for the extraction of pharmacogenomic relationships for which no appropriate corpus is available. Surprisingly, the best results where consistently obtained using the SemEval 2013 DDI corpus as additional data, even for RE tasks that doesn't involve drugs like EU-ADR target-disease. Likewise, one might have thought that the ADE-EXT corpus could have been more asuitable for the EU-ADR drug-disease corpus, since it shares common entities. Several ideas should be explored to better understand this phenomenon, such as the differences of relation and entity types between the different corpora as well as the differences of textual source (e.g. medical case report for ADE-EXT, research articles for the others).

Table 1 .

 1 SemEval 2013 DDI[START_REF] Herrero-Zazo | The DDI corpus: An annotated corpus with pharmacological substances and drug-drug interactions[END_REF] consists of texts from DrugBank and MEDLINE and is annotated with drugs. Drug mentions are categorized in several types: drug, brand, group and drug n (i.e., active substances not approved for human use). Relationships between two drug mentions are annotated and classified in 4 categories: mechanism, effect, advice and int. int is the broader and default category for DDI, when no more detail can be provided.-ADE-EXT (Adverse Drug Effect corpus, extended)[START_REF] Gurulingappa | Extraction of potential adverse drug events from medical case reports[END_REF] consists of MEDLINE case reports, annotated with drug and conditions (e.g., diseases, signs and symptoms) along with relationships between them, when one is mentioned. Main characteristics of the corpora. Two corpora are divided in subcorpora. The sizes of the training and test corpora are reported in term of number of sentences (sent.) and relationships (rel.). EU-ADR and ADR-EXT have no proper test corpus.

	4.2 Large corpora
	-Corpus Subcorpus Train Size Test Size #Entity #Relation
			sent. rel. sent. rel. Types	Types
	SNPPhenA -	362 935 121 365	2	3
		drug-disease 244 176			4	3
	EU-ADR drug-target	247 310	-	-	4	3
		target-disease 355 262			4	3
	SemEval	DrugBank	5,675 3,805 973 889	4	4
	2013 DDI MEDLINE	1,301 232 326 95	4	4
	ADE-EXT -	5,939 6,701 -	-	2	1

Table 2 .

 2 Impact of cross-corpus training in terms of precision (P), recall (R) and f1-score (F). σF is the standard deviation of the f1-score.

	Test Corpus	Train corpus	P	R	F	σF
		SNPPhenA	58.9 73.8 65.5 0.041
	SNPPhenA	+ SemEval 2013 DDI 65.2 71.1 68.0 0.047
		+ ADE-EXT	62.8 72.1 67.2 0.034
		EU-ADR drug-disease 74.8 84.1 79.1 0.123
	EU-ADR drug-disease	+ SemEval 2013 DDI 74.8 90.6 82.0 0.131
		+ ADE-EXT	73.9 88.2 80.4 0.137
		EU-ADR drug-target	72.4 90.6 80.2 0.109
	EU-ADR drug-target	+ SemEval 2013 DDI 71.9 95.5 82.5 0.085
		+ ADE-EXT	70.2 96.7 80.9 0.092
		EU-ADR target-disease 77.0 89.7 82.7 0.064
	EU-ADR target-disease + SemEval 2013 DDI 77.4 91.6 83.9 0.082
		+ ADE-EXT	77.7 89.5 83.3 0.069

  This work EU-ADR target-disease + SemEval 2013 DDI 77.4 91.6 83.9 Table

	Test corpus	Train corpus	P	R	F
		Bokharaeian et al. [2]
		SNPPhenA	56.6 59.8 58.2
	SNPPhenA	This work
		SNPPhenA ensemble + ADE-EXT	64.5 75.2 69.4
		Bravo et al. [4]
	EU-ADR	EU-ADR drug-disease	70.2 93.2 79.3
	drug-disease	This work
		EU-ADR drug-disease + SemEval 2013 DDI	74.8 90.6 82.0
		Bravo et al. [4]
	EU-ADR	EU-ADR drug-target	74.2 97.4 83.0
	drug-target	This work
		EU-ADR drug-target + SemEval 2013 DDI	71.9 95.5 82.5
		Bravo et al. [4]
	EU-ADR	EU-ADR target-disease	75.1 97.7 84.6
	target-disease

https://www.ncbi.nlm.nih.gov/pubmed/