Yaohui Wang

Antitza Dantcheva

Francois Bremond

From attributes to faces: a conditional generative network for face generation

Keywords: Attributes, Soft Biometrics, Generative Adversarial Networks

Recent advances in computer vision have aimed at extracting and classifying auxiliary biometric information such as age, gender, as well as health attributes, referred to as soft biometrics or attributes. We here seek to explore the inverse problem, namely face generation based on attribute labels, which is of interest due to related applications in law enforcement and entertainment. Particularly, we propose a method based on deep conditional generative adversarial network (DC-GAN), which introduces additional data (e.g., labels) towards determining specific representations of generated images. We present experimental results of the method, trained on the dataset CelebA, and validate these based on two GAN-quality-metrics, as well as based on three face detectors and one commercial off the shelf (COTS) attribute classifier. While these are early results, our findings indicate the method's ability to generate realistic faces from attribute labels.

Introduction

Recently attributes or soft biometrics [DER15, [START_REF] Dantcheva | Bag of soft biometrics for person identification[END_REF][START_REF] Chen | Impact of facial cosmetics on automatic gender and age estimation algorithms[END_REF] such as gender, age, ethnicity, height and weight have gained popularity due to their semantic interpretation, i.e., they can provide a description that can be readily understood by humans; for example the description "young, female, tall". While the possibility to extract and classify such attributes has been shown, the inverse problem, namely face generation, given attributelabels is a novel area of high interest, due to related applications in law enforcement and entertainment. One specific application relates to the generation of realistic faces in cases of witness descriptions, where the descriptions are the only available evidence (e.g., in the absence of facial images). Currently in this context, law enforcement utilizes facial composites, which seek to support the process of suspect-identification. Composites are either hand drawn (sketches) or computer-generated 3 , see Fig. 1. Both methods are slow, tedious relatively unrealistic, as well as impeding efficient face recognition (i.e., match such composites with existing mugshot databases maintained by law enforcement agencies). Thus, reliable and automated label-based face generation would be a valuable asset in similar scenarios.

Additionally, label-based face generation accepts the application of visualization of classic fictional characters from novels. Often such characters are verbosely depicted, e.g., "[His] jaw was long and bony, his chin a jutting 'v' under the more flexible v of his mouth. His nostrils curved back to make another, smaller, 'v.' His yellow-grey eyes were horizontal." [START_REF] Tolkien | The Lord of the Rings: One Volume[END_REF]. However there exists the hypothesis that humans are not able to imagine faces, which they have not seen before, and hence an automated visualization based on similar detailed descriptions might be beneficial for an enhanced reading experience. In spite of the aforementioned applications of interest, limited research concerns attributebased face generation. In this context, particularly generative adversarial networks (GANs) and variational auto-encoders (VAEs) are instrumental towards such face generation. Motivated by the above, in this paper we propose to generate faces based on attributelabels. This incorporates two steps: (i) the learning of a text feature representation that captures the important visual details, as well as (ii) given the features, the generation of compelling realistic images. We propose an approach based on deep conditional convolutional generative adversarial network (DCGAN) [START_REF] Perarnau | Invertible conditional GANs for image editing[END_REF], which was introduced to modify images based on attributes (image-to-image translation, see Section 1). We train the proposed GAN with the dataset CelebA and generate faces pertained to two attribute-sets: (a) glasses and gender, as well as (b) glasses, gender, hair color, smile and age. We select these two sets of attributes for the associated high descriptiveness, e.g., such attributes are commonly used by humans to describe their peers. For the experiment (a) we generate 256 images and for (b) 2048 images, which we evaluate by 2 common GAN-quality-scores, as well as by three well established face detectors and an attribute classifier. Results indicate the method's ability to generate realistic faces from attribute labels.

Related work Generative adversarial networks (GANs), as introduced by Goodfellow et al. [START_REF] Goodfellow | Generative adversarial nets[END_REF], incorporate two networks, a Generator, which generates new data instances and a Discriminator, which evaluates them for authenticity. The generator accepts noise as input and generates new samples of data in line with the observed training data. GANs have succeeded in applications such as image generation, image translation, super-resolution imaging, as well as face image synthesis.

Conditional GANs enhance the GAN-concept by providing both, the discriminator and generator with additional class information, in order to generate samples conditioned on different classes. Such text-to-image translation has been beneficial in domain transfer, super-resolution imaging, as well as image editing. Notable approaches include the conditional generative adversarial networks (cGANs) work by Mirza and Osindero [START_REF] Mirza | Conditional generative adversarial nets[END_REF] and the invertible conditional GAN proposed by Perarnau et al. [START_REF] Perarnau | Invertible conditional GANs for image editing[END_REF].

Limited literature concerns attribute-labels based face generation. The work of Gauthier [START_REF] Gauthier | Class Project for Stanford CS231N: Convolutional Neural Networks for Visual Recognition[END_REF] tackles the topic by proposing a conditional GAN in this context and loosely validating the approach qualitatively on Labeled Faces in the Wild (LFW). Deviating from our work, Gauthier added the attribute-information at the last layer of the Discriminator, did not use the set "real images and fake labels" for training, as well as lacked proficient validation. We note here that we tested their architecture and obtained poor results. A further related work presents the VAE-based framework Attribute2Image [START_REF] Yan | Attribute2image: Conditional image generation from visual attributes[END_REF] proposed by Yan et al.. Lu et al. [START_REF] Lu | Conditional cyclegan for attribute guided face image generation[END_REF] also tackle the problem of attributes-based face generation. However, different from them, we apply the state-of-the-art Spectral Normalization [START_REF] Miyato | Spectral Normalization for Generative Adversarial Networks[END_REF] in discriminator and obtain higher quality face images.

As stated above, while limited research concerns attribute-based face generation, extensive research relates to attribute based image-to-image translation. In this context, image-toimage translation refers to the alteration of a particular aspect of a given image to another, e.g., changing the hair color. Notable very recent attribute-translation GANs include Cy-cleGAN [START_REF] Chu | CycleGAN: a Master of Steganography[END_REF], StarGAN [START_REF] Choi | StarGAN: Unified Generative Adversarial Networks for Multi-Domain Imageto-Image Translation[END_REF], as well as MakeupGAN [START_REF] Chang | PairedCycleGAN: Asymmetric Style Transfer for Applying and Removing Makeup[END_REF].

Finally, face sketch synthesis (FSS) has gained interest in the design community (e.g., face artistic styles synthesis [START_REF] Zhang | Compositional model-based sketch generator in facial entertainment[END_REF][START_REF] Zhang | Robust face sketch style synthesis[END_REF]) and is being used for digital entertainment, as well as law enforcement (sketch based face recognition [START_REF] Klum | The FaceSketchID system: Matching facial composites to mugshots[END_REF]). A recent VAE and GAN-based face synthesis approach from sketches is the work of Di and Patel [START_REF] Di | Face Synthesis from Visual Attributes via Sketch using Conditional VAEs and GANs[END_REF].

Proposed method

The proposed conditional GAN aims to fit the conditional probability P(x|z, y), as depicted in Figure 2 and specified in the Tables 1 and2. We let z be the noise vector sampled from N (0, 1) with dimension N = 100, y be the vector representing attribute-labels (with y i ∈ ±1, where i corresponds to the i th attribute). We train a GAN, adding attribute-labels in both, generator and discriminator. While the generator accepts as input the combination of prior noise p(z) and attributes vector y, the discriminator accepts both, real or generated images, as well as the attribute-labels. We have the objective function of our model be: Implementation details Our model is based on the PyTorch implementation of DCGAN. The generator accepts as the input the concatenation of attribute-label-vector y and noise z. On the discriminator-side, we stack the attribute-label-vector with the feature map of the first convolutional layer. We test the insertion of the attribute-label-vector at each discriminator-layer and obtained best results at the first layer-level. We train the model with the Adam optimizer at a learning rate of 0.0002 and a mini batch size of 128. The input and generated image sizes are both 64 × 64. We conduct 2 experiments by applying Batch Normalization(BN) and Spectral Normalization(SN) in discriminator separately. Table 1 2 show the details of model architecture with SN.

Tab. 1: Architecture of Generator Algorithm 1 summarizes the training procedure. When we train the discriminator, we provide two types of negative samples. Firstly the generated images with real labels, secondly real images with fake labels. We observe that such training enforces the discriminator to learn from diverse samples and at the same time enables the generation of realistic samples. (a) no glasses, female, black hair, smile, young (b) glasses, male, no black hair, no smile, young Fig. 3: Example images generated by the proposed approach based on 5 attributes Figure 3 illustrates generated samples of the proposed approach. We observe that the model succeeds specifically in generation of local-attribute-labels based faces (e.g., glasses, smile). Further, based on Figure 4, we found binary labels might be too limiting in describing age. Such labels can guide the model to generate 'younger' or 'older' faces, not exactly 'young' and 'old' faces. In addition, CelebA depicts celebrities, with face alterations inflicted by makeup and plastic surgery, which might affect the reliability of the pair "age-label and image".

We have witnessed a 'mode collapse' after 100 epochs when applying Batch Normalization in discriminator, however when applying Spectral Normalization, 'mode collapse' appears until 300 epochs. It shows that SN can provide more stable training and higher quality generated images.

Evaluation To evaluate how realistic our generated images are, in this section, firstly we report the results based on the pre-trained face and attribute detection models. Then we proceed to present evaluation results of two quality metrics, namely Inception Score (IS) [START_REF] Salimans | Xi: Improved Techniques for Training GANs[END_REF] and Fréchet Inception Distance (FID) [START_REF] Heusel | GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium[END_REF], which have been widely used in image quality evaluation for GANs.

Face Detection We report face detection results based on 3 different face detectors (Face ++4 , DFace5 , dlib6 in Table 3. We observe that DFace has the highest detection rates (for experiment (a) 89%, for (b) 96%), while Face++ has the lowest rates ((a) 58.2%, (b) 53%). 4 gender classification results from Face ++ both on generated dataset and subset of 15735 images from original CelebA. We note the higher true gender classification accuracy in the experiment (b) (generating 5 attributes), which indicates that higher prior information allows the model to generate better targeted images. Figure 4 illustrates the boxplot of estimated age for age estimation by Face ++. Here, we note a shift in estimated age for the old / young labels, but the shift is not profound.

Attribute Estimation We present in Table

GAN quality metrics

We proceed with 2 commonly used GAN -quality measures, namely the Inception Score and the Fréchet Inception Distance, that we firstly proceed to describe.

Inception Score (IS) is a metric for automated quality evaluation of images originated from generative models. The score is computed by an Inception V3 Network pre-trained on ImageNet and Tab. 3: Face detection results. We report the number of detected faces for the three face detectors: dlib, DFace and Face ++. In experiment (a) we generate faces labeled with two attributes corresponding to gender and glasses. Experiment (b) involves 5 attributes corresponding to gender, glasses, age, smile, hair color.

(G) = exp(E x∼p g D KL (p(y|x) p(y))), (2)
where x ∼ p g indicates that x is an image sampled from p g , D KL (p q) is the KL-divergence between the distributions p and q. The score examines generated images for included meaningful objects, as well as a low label entropy.

Fréchet Inception Distance (FID) is an improvement of Inception Score. IS has the limitation of not utilizing statistics of real world samples are not used, and compared to the statistics of synthetic samples. Overcoming that, the Frćhet distance measures the distance between a generated image set and a source dataset, and is calculates as

d 2 ((m,C), (m ω ,C ω )) = m -m ω 2 2 + Tr(C +C ω -2(CC ω ) 1 2 ), (3)
where (m,C), (m ω ,C ω ) represent the mean and covariance of the two respective distributions.

IS and FID for the both experiments are summarized in

Conclusions

In this work we have presented preliminary results on attribute based face generation based on a DCGAN-approach. Results, evaluated by three well established face detectors, an attribute estimator and benchmark quality scores, suggest the method's ability to generate realistic faces from attribute labels. The presented approach can be instrumental in the visualization of witness descriptions. Future work will involve experiments related to matching of generated faces with existing faces.

Fig. 1 :

 1 Fig. 1: Composite images from FACES 4.0 [Sw14].

  Fig.2: Architecture of proposed method consisting of two modules, a discriminator D and a generator G. D learns to distinguish between real and fake images, classifying based on attribute-labels. G accepts as input both, noise and attribute-labels in order to generate realistic face images.

Algorithm 1

 1 Attributes based Conditional DCGAN Input: mini batch x, matching label y, mismatching label ŷ, number of training epochs N 1: for n = 1 to N do 2: z ∼ N (0, 1) Z {Draw sample of random noise} 3: y ← Y {Draw matching attributes label} 4: ŷ ← Ŷ {Draw mismatching attributes label} 5: x ← G(z, y) {Forward through generator} 6: s r ← D(x, y) {Forward through discriminator with real image, right attribute labels} 7: s w ← D(x, ŷ) {Forward through discriminator with real image, wrong attribute labels} 8: s g ← D( x, y) {Forward through discriminator with generated image, right attribute labels} 9: L D ← log(s r ) + (log(1s w ) + log(1s g ))/2 {loss function of discriminator } 10: D ← D -α∂ L D /∂ D {Update discriminator} 11: L G ← log(s f ) {loss function of generator} 12: G ← G -α∂ L G /∂ G {Update generator} 13: end for 3 Experiments We use the benchmark dataset CelebA [Li15] comprising of 202,599 face images annotated with 40 binary attribute labels. We perform two experiments: generation of images given (a) two attributes (glasses, gender) and (b) five attributes (glasses, gender, hair color, smile and age). For (a) we generate 256 images and for (b) 2048 images.

Table 5 .

 5 For IS, higher values indicate a higher quality, while for FID the opposite is the case. Again, we observe that an increased prior information (experiment (b) improves the generated image quality. As a comparison Wasserstein GAN (WGAN) reportedly has IS = 8.42 and FID = 55.2; the Boundary Equilibrium Generative Adversarial Networks BEGAN IS = 5.62 and FID = 71.4.Tab. 5: Inception Score (IS) and Fréchet Inception Distance (FID) quantitatively validating generated images for the two experiments pertained to 2 and 5 attributes.

	Number of Attributes	IS	FID
	2	1.94 63.8
	5	2.20 43.8

INRIA Sophia-Antipolis, STARS, France,{yaohui.wang, antitza.dantcheva, francois.bremond}@inria.fr 3 http://www.iqbiometrix.com

https://www.faceplusplus.com.cn/

https://github.com/kuaikuaikim/DFace

http://dlib.net/

Acknowledgement

This work was supported by the French Government (National Research Agency, ANR) under grant agreement ANR-17-CE39-0002.


Yaohui Wang
email: yaohui.wang@inria.fr

Antitza Dantcheva
email: antitza.dantcheva@inria.fr

Francois Bremond
email: francois.bremond@inria.fr

From attribute-labels to faces: face generation using a conditional generative adversarial network

Keywords: attributes, generative adversarial network, face generation

Facial attributes are instrumental in semantically characterizing faces. Automated classification of such attributes (i.e., age, gender, ethnicity) has been a well studied topic. We here seek to explore the inverse problem, namely given attribute-labels the generation of attributeassociated faces. The interest in this topic is fueled by related applications in law enforcement and entertainment. In this work, we propose two models for attribute-label based facial image and video generation incorporating 2D and 3D deep conditional generative adversarial networks (DCGAN). The attribute-labels serve as a tool to determine the specific representations of generated images and videos. While these are early results, our findings indicate the methods' ability to generate realistic faces from attribute labels.

Introduction

While attribute extraction and classification [START_REF] Dantcheva | What else does your biometric data reveal? a survey on soft biometrics[END_REF][START_REF] Dantcheva | Bag of soft biometrics for person identification[END_REF][START_REF] Chen | Impact of facial cosmetics on automatic gender and age estimation algorithms[END_REF] is a well studied topic, the inverse problem, namely face generation, given attribute-labels is a novel area of high interest, due to related applications in law enforcement and entertainment. One specific application relates to the generation of realistic faces in cases of witness description, where the descriptions are the only available evidence (e.g., in the absence of facial images). Particularly, law enforcement utilizes facial hand-drawn sketches or composites, which seek to support the process of suspect-identification. Such methods for face synthesis are slow, tedious, relatively unrealistic, as well as impeding efficient face recognition (i.e., matching sketches or composites with existing mugshot databases maintained by law enforcement agencies poses a heterogeneous face recognition problem, which is highly challenging). Thus, reliable and automated label-based face generation constitutes beneficial in this context.

In spite of the aforementioned applications of interest, limited research concerns attribute-based face generation. Motivated by the above, in this work we propose to generate faces based on attribute-labels. This incorporates two steps: (i) the learning of a text feature representation that captures the important visual details, as well as (ii) given the features, the generation of compelling realistic images. We propose two approaches based on deep conditional convolutional generative adversarial network (DCGAN) [START_REF] Perarnau | Invertible conditional GANs for image editing[END_REF], which was introduced to modify images based on attributes (image-to-image translation). We train the proposed 2D GAN with the dataset CelebA and generate faces pertained to the attribute-set glasses, gender, hair color, smile and age. We selected these set of attributes for the associated high descriptiveness, e.g., such attributes are commonly used by humans to describe their peers. For the experiment we generate facial images, which we evaluate by 2 common GAN-quality-scores, as well as by well established face detectors and an attribute classifier. More analysis of the 2D model is presented in Wang et al. [START_REF] Wang | From attributes to faces: a conditional generative adversarial network for face generation[END_REF]. In addition we propose a 3D GAN model, trained with the UvA-NEMO3 , generating facial smiling videos pertained to the attributes gender and age. Results indicate the method's ability to generate realistic faces from attribute labels.

Proposed methods

2D model

The proposed conditional GAN aims to fit the conditional probability P (x|z, y), as depicted in Figure 1. We let z be the noise vector sampled from N (0, 1) with dimension N = 100, y be the vector representing attribute-labels (with y i ∈ ±1, where i corresponds to the i th attribute). We train a GAN, adding attribute-labels in both, generator and discriminator. While the generator accepts as input the combination of prior noise p(z) and attributes vector y, the discriminator accepts both, real or generated images, as well as the attributelabels. We have the objective function of our model be: 2), generating videos. In both, generator and discriminator, the convolutional kernels have been expanded onto three dimensions (H, W, T ), where H, W and T represent the height, the width and the temporal step of the receptive fields in each kernel. We feed the attribute vectors into 3D model in a similar manner as the 2D model. Specifically, in the generator we concatenate the attribute vector with the noise vector. In the discriminator, the feature map after the first layer has the dimension of (H, W, C, T ), each (H, W, C, t), t ∈ T containing spatial-temporal features of a certain time period. Our goal is to generate face videos based on attributes, so we proceed to provide each spatial-temporal feature map with the same attribute embedding. Based on this, we insert an attribute embedding onto the spatial-temporal feature map from the first layer of the discriminator, creating a new feature map with the dimension (H, W, C + y, T ), where y is the dimension of the attribute vector. 3 illustrates generated samples of the proposed approach. We observe that the model succeeds specifically in generation of local-attribute-labels based faces (e.g., glasses, smile). The generated glasses appear to be similar, which is a limitation associated to DCGAN and the related loss function we use.

min G max D V (D, G) = E x,y∼p data [log D(x, y)] + E z∼pz,y∼py [log(1 -D(G(z, y), y))]. (1)

Evaluation

To evaluate how realistic our generated images are, in this section, we report the results based on the pre-trained face and attribute classification models. Then we proceed to present evaluation results of two quality metrics, namely Inception Score (IS) [START_REF] Salimans | Improved techniques for training gans[END_REF] and Fréchet Inception Distance (FID) [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a nash equilibrium[END_REF], which have been widely used in image quality evaluation for GANs.

We obtain face detection results of up to 96% by DFace [START_REF] Dface | Deeplearning face[END_REF]) and gender true classification rate of up to 81.8% by the attribute classifier Face++ [START_REF]Face++: Face++ api[END_REF]. For age, note a shift in estimated age for the old / young labels, but the shift is not profound.

GAN quality metrics

We proceed to compute two commonly used GAN -quality measures, namely the Inception Score and the Fréchet Inception Distance, that we firstly proceed to describe. Inception Score (IS) is a metric for automated quality evaluation of images originated from generative models. The score is computed by an Inception V3 Network pre-trained on ImageNet and calculates a statistic of the network's outputs, when applied to generated images. Fréchet Inception Distance (FID) is an improvement of Inception Score. IS has the limitation of not utilizing statistics of real world samples are not used, and compared to the statistics of synthetic samples. Overcoming that, the Frćhet distance measures the distance between a generated image set and a source dataset.

We obtain IS = 2.2 and F ID = 43.8. For IS, higher values indicate a higher quality, while for FID the opposite is the case. As a comparison Wasserstein GAN (WGAN) [START_REF] Arjovsky | Wasserstein gan[END_REF] reportedly achieves an IS = 8.42 and F ID = 55.2; the Boundary Equilibrium Generative Adversarial Network BEGAN [START_REF] Berthelot | Began: boundary equilibrium generative adversarial networks[END_REF] obtains an IS = 5.62 and F ID = 71.4.

Conclusions

In this work we presented 2D and 3D models for attribute based facial images and video generation, both based on DCGAN. Results, evaluated by a face detector, an attribute estimator and benchmark quality scores, suggest the models' ability to generate realistic faces from attribute labels. The presented approaches can be instrumental in the visualization of witness descriptions. Future work will involve experiments related to matching of generated faces with existing faces.

Fig. 1 .

 1 Fig.1. Architecture of proposed 2D method consisting of two modules, a discriminator D and a generator G. While D learns to distinguish between real and fake images, classifying based on attribute-labels, G accepts as input both, noise and attributelabels in order to generate realistic face images.

Fig. 2 .

 2 Fig. 2. Architecture of proposed 3D model for face video generation

  (a) no glasses, female, black hair, smiling, young (b) glasses, female, black hair, not smiling, old (c) no glasses, male, no black hair, smiling, young (d) glasses, male, no black hair, not smiling, old

Fig. 3 .

 3 Fig. 3. Example images generated by the proposed 2D model.

Fig. 4 .

 4 Fig. 4. Chosen output samples from 3DGAN

Table 1 .

 1 Architecture of 2D Generator

	Operation	Kernel Stride Filters Norm Activation
	Concatenation Concatenate z and y on 1st dimension
	ConvTranspose 4 × 4 2 × 2 512	BN	ReLU
	ConvTranspose 4 × 4 2 × 2 256	BN	ReLU
	ConvTranspose 4 × 4 2 × 2 128	BN	ReLU
	ConvTranspose 4 × 4 2 × 2	64	BN	ReLU
	ConvTranspose 4 × 4 2 × 2	3	No	Tahn

Table 2 .

 2 Architecture of 2D Discriminator

	Operation Kernel Stride Filters Norm	Activation
	Conv	4 × 4 2 × 2	64	No	LeakyReLU
	Concatenation Replicate y and concatenate to 1st conv. layer
	Conv	4 × 4 2 × 2 128	SN	LeakyReLU
	Conv	4 × 4 2 × 2 256	SN	LeakyReLU
	Conv	4 × 4 2 × 2 512	SN	LeakyReLU
	Conv	4 × 4 1 × 1	1	No	Sigmoid

3D model We expand the above presented 2D model onto 3D, in order to create a conditional 3DGAN (Figure

Table 3 .

 3 Architecture of 3D Generator

	Operation	Kernel	Stride Filters Norm Activation
	Concatenation	Concatenate z and y on 1st dimension
	ConvTranspose3D 2 × 4 × 4 2 × 4 × 4 512	BN	ReLU
	ConvTranspose3D 4 × 4 × 4 2 × 2 × 2 256	BN	ReLU
	ConvTranspose3D 4 × 4 × 4 2 × 2 × 2 128	BN	ReLU
	ConvTranspose3D 4 × 4 × 4 2 × 2 × 2	64	BN	ReLU
	ConvTranspose3D 4 × 4 × 4 2 × 2 × 2	3	No	Tahn

Table 4 .

 4 Architecture of 3D Discriminator

	Operation	Kernel	Stride Filters Norm Activation
	Conv3D	4 × 4 × 4 2 × 2 × 2	64	No LeakyReLU
	Concatenation Replicate y and concatenate to 1st conv. layer
	Conv3D	4 × 4 × 4 2 × 2 × 2 128	SN LeakyReLU
	Conv3D	4 × 4 × 4 2 × 2 × 2 256	SN LeakyReLU
	Conv3D	4 × 4 × 4 2 × 2 × 2 512	SN LeakyReLU
	Conv3D	2 × 4 × 4 1 × 1 × 1	1	No	Sigmoid

of 202,599 face images annotated with 40 binary attribute labels. We generate images given five attributes glasses, gender, hair color, smile and age), in total 2048 images. Figure

http://www.uva-nemo.org/

Acknowledgement

This work was supported by the French Government (National Research Agency, ANR) under grant agreement ANR-17-CE39-0002.


Yaohui Wang
email: yaohui.wang@inria.fr

Antitza Dantcheva
email: antitza.dantcheva@inria.fr

Jean-Claude Broutart
email: jc.broutart@free.fr

Philippe Robert
email: probert@unice.fr

Francois Bremond
email: francois.bremond@inria.fr

Piotr Bilinski
email: piotr.bilinski@eng.ox.ac.uk

Comparing methods for assessment of facial dynamics in patients with major neurocognitive disorders

Keywords: Facial Dynamics, Facial Expressions, Neurocognitive Disorders, Alzheimer's Disease

Assessing facial dynamics in patients with major neurocognitive disorders and specifically with Alzheimers disease (AD) has shown to be highly challenging. Classically such assessment is performed by clinical staff, evaluating verbal and non-verbal language of AD-patients, since they have lost a substantial amount of their cognitive capacity, and hence communication ability. In addition, patients need to communicate important messages, such as discomfort or pain. Automated methods would support the current healthcare system by allowing for telemedicine, i.e., lesser costly and logistically inconvenient examination. In this work we compare methods for assessing facial dynamics such as talking, singing, neutral and smiling in AD-patients, captured during music mnemotherapy sessions. Specifically, we compare 3D Con-vNets, Very Deep Neural Network based Two-Stream ConvNets, as well as Improved Dense Trajectories. We have adapted these methods from prominent action recognition methods and our promising results suggest that the methods generalize well to the context of facial dynamics. The Two-Stream ConvNets in combination with ResNet-152 obtains the best performance on our dataset, capturing well even minor facial dynamics and has thus sparked high interest in the medical community.

Introduction

Major neurocognitive disorder (NCD), as introduced by the American Psychiatric Association (APA), known previously as dementia, is a decline in mental ability, threatening the independence of a large fraction of the elderly population. Alzheimer's disease (AD) is the most common form of major NCD, associated with loss of short-term-memory, problems with language, disorientation and other intellectual abilities, severely affecting daily life 5 . Worldwide, currently [START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF] Million people have been diagnosed with major neurocognitive disorder, which has been associated with 530 billion Euros in 2010 6 ), tendency increasing 7 . While there is no palliative care, musical therapies have been proposed as efficient therapeutic means, acting as a powerful catalyst for precipitating memories, shown in a number of studies [START_REF] Vink | Music therapy for people with dementia[END_REF][START_REF] Svansdottir | Music therapy in moderate and severe dementia of Alzheimer's type: a case-control study[END_REF][START_REF] Raglio | Music, music therapy and dementia: a review of literature and the recommendations of the italian psychogeriatric association[END_REF]. Specifically mnemotherapy can help elicit autobiographical memories by promoting positive emotional memories [START_REF] Broutart | Démence et perte cognitive: Prise en charge du patient et de sa famille, chap. Mnémothérapie, reviviscence et maladie d'Alzheimer[END_REF]. This and other therapies can improve the quality of life in AD patients [START_REF] Ridder | The use of extemporizing in music therapy to facilitate communication in a person with dementia: An explorative case study[END_REF][START_REF] Ashida | The effect of reminiscence music therapy sessions on changes in depressive symptoms in elderly persons with dementia[END_REF]. However, the assessment of such therapies requires comprehensive manual observation by experienced clinicians [START_REF] Suzuki | Behavioral and endocrinological evaluation of music therapy for elderly patients with dementia[END_REF][START_REF] Folstein | Mini-mental state": a practical method for grading the cognitive state of patients for the clinician[END_REF]. Towards overcoming this limitation, computer vision based methods can offer objective assessment by analyzing affect and expression behaviors, directly related to the effectiveness of therapies.

While expression recognition has attracted significant research attention [START_REF] Sariyanidi | Automatic analysis of facial affect: A survey of registration, representation, and recognition[END_REF][START_REF] Martinez | Automatic analysis of facial actions: A survey[END_REF][START_REF] Walecki | Variable-state latent conditional random field models for facial expression analysis[END_REF], facial behavior analysis from naturalistic videos, associated to illumination changes, partial occlusions, pose variation, as well as low-intensity expressions pose challenges for current existing methods. In addition, while many areas of computer vision have experienced significant advancements with deep neural networks, analysis of facial dynamics has only recently benefited from deep convolutional networks [START_REF] Tran | Deepcoder: Semi-parametric variational autoencoders for facial action unit intensity estimation[END_REF][START_REF] Jung | Joint fine-tuning in deep neural networks for facial expression recognition[END_REF][START_REF] Zafeiriou | Slow features nonnegative matrix factorization for temporal data decomposition[END_REF][START_REF] Saha | Unsupervised deep representations for learning audience facial behaviors[END_REF].

Naturally, the accuracy of facial dynamics classification depends on features, as well as architecture used for assessment. Given the plethora of existing algorithms, exploring different types of features and architectures is necessary to devise a robust solution.

Motivated by the above, in this work we explore and compare computer vision methods, introduced in the context of action recognition in our challenging setting, namely in the context of assessing naturalistic facial dynamics. Specifically we have (a) 3D Convolutional Neural Network (C3D) [START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF], (b) Very Deep Two-Stream Convolutional Network (with VGG-16 and ResNet-152 [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF][START_REF] Wang | Towards good practices for very deep two-stream convnets[END_REF]) and (c) Improved Dense Trajectories (iDT) [START_REF] Wang | Dense trajectories and motion boundary descriptors for action recognition[END_REF], as well as combinations and variations thereof. Given a video sequence, we firstly detect the face and proceed to extract features pertaining to the respective method. The obtained feature set is then classified in one of four facial dynamics categories, namely neutral, smiling, talking and singing. The automatic detection of named facial dynamics indicates the involvement of the patients during mnemotherapy and hence can support the assessment of a therapy session. Specifically, given that AD-patients in later stages of the disease often suffer from apathy, the (frequent) occurrence of smiling, talking and singing indicates that the therapy is effective. Experiments are conducted on a challenging medical unconstrained dataset containing 322 video sequences of 16 AD-patients including continuous pose-changes, oc-clusions, camera-movements and artifacts, as well as illumination changes. In addition, the dataset depicts naturalistic facial dynamics of predominantly elderly subjects, which vary in (generally less profound) intensity and occur jointly (e.g., simultaneous talking and smiling). Moreover, we observe a high level of inter-and intra person variability (e.g., expressive and apathetic AD-patients). We note that, despite that, it is imperative to work with such data, as it is representative for current (vast amount of) video-documentation of medical doctors, requiring automated analysis.

We note that we tested existing methods in expression recognition, such as smile detectors 8 , [START_REF] Dantcheva | Gender estimation based on smile-dynamics[END_REF] on the ADP -dataset, as well as facial-landmark based expression recognition algorithms without success, since already the first incorporated step of face detection failed throughout.

Related Work

Existing approaches for the analysis of facial dynamics are inspired by cognitive, psychological and neuroscientific findings. The most frequent way to describe facial dynamics is based on the Facial Action Coding System (FACS) proposed by Ekman et al. [START_REF] Ekman | Facial action coding system: a technique for the measurement of facial movement[END_REF], representing movements of facial muscles in different terms of action units (AUs). Hence, classical methods analyze sequences of images containing the neutral face and the expression apex [START_REF] Martinez | Automatic analysis of facial actions: A survey[END_REF]. More recent methods involve linear deterministic and probabilistic methodologies including general or special Linear Dynamical Systems (LDS), as well as various extensions of deterministic Slow Feature Analysis (SFA) [START_REF] Zafeiriou | Slow features nonnegative matrix factorization for temporal data decomposition[END_REF]. In addition, HMMs [START_REF] Sandbach | Recognition of 3d facial expression dynamics[END_REF] have been used to capture the temporal segments of facial behaviour.

More recently, learning facial features in supervised and unsupervised manner using deep neural networks has attracted considerable attention. We proceed to provide such notable work, analyzing both, images and video sequences.

Recognizing facial dynamics in images Liu et al. [START_REF] Liu | Facial expression recognition via a boosted deep belief network[END_REF] propose a Boosted Deep Belief Network, integrating three separate training stages for expression recognition in images. Han and Meng [START_REF] Han | Incremental boosting convolutional neural network for facial action unit recognition[END_REF] present the incremental boosting of CNN for AUs recognition. Zhao et al. [START_REF] Zhao | Deep region and multi-label learning for facial action unit detection[END_REF] combine region learning, as well as multiple label learning to detect AUs.

Recognizing facial dynamics in video sequences When analyzing facial behavior in videos, many works usually focus on spatial-temporal feature extraction. Jung et al. [START_REF] Jung | Joint fine-tuning in deep neural networks for facial expression recognition[END_REF] use two neural networks separately to extract temporal appearance features, as well as temporal geometric features for expression recognition. Zafeiriou et al. [START_REF] Zafeiriou | Slow features nonnegative matrix factorization for temporal data decomposition[END_REF] propose a slow-feature-auto-encoder for both supervised and semi-supervised learning of facial behavioural dynamics analysis. Hasani and Mahoor [START_REF] Hasani | Facial expression recognition using enhanced deep 3d convolutional neural networks[END_REF] combine a 3D Inception-ResNet with a Long short-term memory (LSTM) network in order to extract both, spatial and temporal features from videos. Li et al. [START_REF] Li | Action unit detection with region adaptation, multilabeling learning and optimal temporal fusing[END_REF] combine VGG with ROI and LSTM together towards detection of AUs. Most recently, combining Variational Autoencoder (VAEs) and Generative Adversarial Networks (GANs) has allowed for learning a powerful latent representation utilized for facial behavior analysis in audience [START_REF] Saha | Unsupervised deep representations for learning audience facial behaviors[END_REF].

Previous and recent computer vision work related to healthcare have focused on, among others the assessment of: cognitive health in smart home environment [START_REF] Dawadi | Automated assessment of cognitive health using smart home technologies[END_REF], daily activities [START_REF] König | Validation of an automatic video monitoring system for the detection of instrumental activities of daily living in dementia patients[END_REF], AD symptoms [START_REF] Romdhane | Automatic video monitoring system for assessment of Alzheimers disease symptoms[END_REF], depression [START_REF] Zhu | Automated depression diagnosis based on deep networks to encode facial appearance and dynamics[END_REF][START_REF] Dibeklioglu | Dynamic multimodal measurement of depression severity using deep autoencoding[END_REF], assistive technologies [START_REF] Leo | Computer vision for assistive technologies[END_REF], as well as pain [START_REF] Rodriguez | Deep pain: Exploiting long short-term memory networks for facial expression classification[END_REF].

The rest of the paper is organized as follows. Section 3 describes the methods we compare. Section 4 introduces our dataset, assembled for the purpose of medical patient recording. Section 5 presents experiments, validating the effectiveness of the evaluated methods in assessing facial dynamics. Finally, Section 5.3 discusses and Section 6 concludes the paper.

Evaluated Methods

Firstly, we utilize face detection, based on which we crop the faces and proceed to extract facial features using Improved Dense Trajectories, as well as two deep neural network models. The latter have been pre-trained on a large-scale human action dataset UCF101 [START_REF] Soomro | UCF101: A dataset of 101 human actions classes from videos in the wild[END_REF]. We inherit the weights in the neural network models and proceed to extract features of our dataset. Finally, we employ Support Vector Machine (SVM) to classify video sequences into four facial dynamics: smiling, talking, neutral and singing.

Face detection

There exist a large number of face detection algorithms, based on a large number of features and implementations. We compared a number of pre-trained algorithms including VGG [START_REF] Felzenszwalb | Object detection with discriminatively trained part-based models[END_REF], OpenCV [START_REF] Viola | Robust real-time face detection[END_REF], and Doppia [START_REF] Mathias | Face detection without bells and whistles[END_REF] with our ADPdataset (see Section 4). The latter performed the best and was hence included in the pre-processing step.

3D ConvNets

3D ConvNets (C3D) has a simple architecture (see Fig. 1) and has shown a performance of 85.2% accuracy on the UCF101 dataset. We adapt the C3D architecture and extract spatial-temporal features towards categorization of AD patients' facial dynamics.

The original C3D network has 8 convolutional layers, 5 max-pooling layers, 2 fully connected layers and a softmax loss layer. For 5 convolutional layers from 1 to 5, the number of convolutional kernels are 64, 128, 256, 256, 256 respectively. Because all kernels have 3 dimensions, the additional parameter d indicates the kernel temporal depth. Tran et al. [START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF], report for d =3 the best among all experiments, which we also use in the present work. In C3D network, all convolutional kernels are 3 × 3 × 3 with stride 1 in both, spatial and temporal directions, in order to ensure a 3 dimensional output. Such an architecture allows for preserving the temporal information between neighboring frames in a videoclip. With the exception of the first pooling layer, all pooling layers are maxpooling layers, with kernel size 2 × 2 × 2 with stride 1, attaining that the size of the feature map of each layer is reduced by a factor of 8 as opposed to the input(see Figure 2). The kernel size of the first pooling layer is 1 × 2 × 2, which ensures that early merging of the temporal signals is avoided. Since we only need features from the FC6 activation layer in our context, we remove the FC7 and last soft max layers from the original model.

We note that the C3D network was pre-trained on the UCF101 dataset, which contains 13320 videos from 101 action categories including single and multiple person actions. Specifically, each video has been divided into video clips of 16-frames-length, with an 8-frame overlap between two video clips by a sliding window method. All these video clips serve as training-input for the C3D network. The computed C3D-feature of a single video sequence is the average of all these clip FC6 activations followed by an L2 -normalization. Fig. 2: 3D Convolutional kernel and 3D Max-pooling kernel: In each convolutional layer except the first one, the kernel size is 3 × 3 × 3 and in each max-pooling layer the kernel size is 2 × 2 × 2. This 3 dimensional design can preserve both spatial and temporal information. We have blurred the faces of the subject in this figure, in order to preserve the patient's privacy.

Very Deep Two-Stream ConvNets

The second method, which we explore is Two-Stream ConvNets [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF](see Fig. 3a), which has reportedly achieved 88% accuracy on the UCF101 action recognition dataset. It extracts features based on RGB frames, as well as based on optical flow from a video sequence. As reported by Wang et al. We test two variations of Very Deep Two-Stream ConvNets, the first one including VGG-16 in both streams, the second one ResNet-152 in both streams. We note that for both, VGG-16 and ResNet-152, we remove the last fully connected layer and follow a L2 -normalization step after the activations. The pre-trained spatial and temporal ConvNets extract two respective feature vectors, which concatenated serve as input for our classifier that we describe below.

Improved Dense Trajectories

Despite the prevalence of DNN, iDT as introduced by Wang et al. [START_REF] Wang | Action recognition with improved trajectories[END_REF] constitute one of the best hand-crafted feature-based approaches. We employ iDT for their good coverage of foreground motion and high performance in action recognition (competitive to DNN). In addition, it is complementary to DNN and hence a fusion of iDT and DNN has shown to provide improved accuracy. iDT extracts local spatio-temporal video trajectories by applying dense sampling of feature points on multiple spatial scales with subsequent tracking of detected feature points using dense optical flow. We extract dense trajectories and proceed to extract local spatio-temporal video volumes around the detected trajectories. We extract 5 types of features aligned with the trajectories to characterize shape (point shifts), appearance (Histogram of Oriented Gradients (HOG) and motion (Histogram of Optical Flow (HOF) and Motion Boundary Histogram (MBHx, MBHy)). We encode the iDT features with bag-of-features (BOF) in order to represent video sequence using the extracted motion trajectories and their corresponding descriptors.

Classifier

For classification, we train a multi-class SVM classifier for the tested methods. We combine the methods Grid-Search and Cross-validation in order to obtain the best parameters.

Dataset

For this study, we created the Alzheimer's disease patients (ADP) -dataset, comprising of 322 video sequences including 16 female patients, with 5 or more takes of each facial dynamics class. The length of the video sequences ranged from 1.44 seconds to 33.08 seconds. All videos have been recorded on 25fps, with resolution of 576 × 720. Two patients with aphasia endued only video sequences of neutral, smiling and singing. Interestingly, while these two patients were not able to speak, they performed singing-like facial movements, which we labeled as singing. For this study we manually segmented and annotated the data, which was challenging, due to high intra-and inter-class variability of patients, as well as facial dynamics. In terms of annotation, two researchers annotated the data (one working in the area of computer vision, the other one in clinical experiments), overlapping in > 85%. We note that facial dynamics appeared jointly (e.g., singing and smiling), due to the unintrusive nature of the setting. Currently, such overlaps were not considered in the annotation, classes were annotated mutually exclusively.

The patients participated in individual mnemotherapy sessions, located in a small auditorium. Videos of these sessions were acquired with a camescope Sony Handycam DCR-SR 32, placed sideways of patient and clinician, capturing predominantly non-frontal and highly unconstrained videos of the patient.

We identified the most frequent occurring facial dynamics as neutral, smiling, talking and singing / singing-like movements. We note that even "neutral" there are still facial movements (e.g. blinking), hand or head movements.

Experimental Results

Implementation Details

We conduct our experiments on a single GTX Titan X GPU for both face detection and feature extraction, with emphasis on the use of C3D Network and Two-Stream ConvNets.

Face detection is performed by Doppia [START_REF] Mathias | Face detection without bells and whistles[END_REF]. Due to the challenging dataset including among others variations of illumination, patient pose, as well as cameramovements, some faces are not detected in single video frames (constituting to false negatives). In such cases, we remove the concerning frames. In case of prevalence of undetected faces, we exclude the concerning video sequences from the analysis.

To compute optical flow, we follow the work of Wang et al. [START_REF] Wang | Action recognition with trajectory-pooled deepconvolutional descriptors[END_REF] and use the TVL1 algorithm implemented in OpenCV. In our experiments, we set N = 25, for both, RGB and optical flow, following the works [START_REF] Wang | Towards good practices for very deep two-stream convnets[END_REF][START_REF] Wang | Action recognition with trajectory-pooled deepconvolutional descriptors[END_REF]. Each detected face (RGB-frame) is rescaled to 224 × 224 × 3 and optical flow is rescaled to 224 × 224 × 20 in Two-Stream Networks. For C3D we rescale each detected face to 112 × 112 × 3.

For classification we use the scikit-learn library [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF]. We employ a Stratified 10-Folds cross-validation scheme, which preserves the ratio of samples for each category for train and test set (see Table 2). The dataset is divided into 10-folds, 9 folds are used for training and the remaining fold is used for testing. This is repeated 10 times and reported results are the average thereof. We note that video sequences in the test set are not present in the training set. Per split, we compute mean accuracy (MA) (mean accuracy of 10 splits) and we report the MA over all splits.

We test SVM classifier with linear and radial basis function (RBF) kernels. Our experiments show that e.g., for Two-Stream (ResNet-152) the RBF kernel performs best (with C = 25, gamma = 2).

Results

In Table 1 the performance of the facial dynamics-classification is presented as mean accuracy (MA) pertained to C3D, Two-Stream ConvNets (based on VGG-16 and ResNet-152), as well as separately to each spatial and temporal net. We observe that while the spatial and temporal net of Two-Stream ConvNets (ResNet-152) substantially outperform the VGG-16 counterparts, the overall Two-Stream ConvNets (VGG-16) and Two-Stream ConvNets (ResNet-152) perform comparably well. The best performance was obtained by the ResNet-152 based Two-Stream ConvNets, namely M A = 76.40%, marginally outperforming VGG-16 based Two-Stream ConvNets (by 0.4%), and substantially outperforming C3D network (by 9%). In addition, we show performance of C3D and Two-Stream ConvNets fused with iDT. When fusing iDT with the other algorithms, the classification rate consistently increases, up to 79.5% for Two-Stream (ResNet-152). The classification rates for each split of the 10-fold cross-validation are reported in Table 2.

In Fig. 4 we present the overall confusion matrix, associated to the best performing algorithm -Two-Stream ConvNets (ResNet-152). The related results indicate that the highest confusion rates are observed between the dynamics smiling, singing and talking. This may be explainable by the co-occurrence of facial dynamics, as well as with the general low-intensity facial dynamics exhibited by the elderly patients. In some cases, the categories neutral and smiling have been confused. In Table 2 we show accuracy for each facial dynamics-category associated to Fig. 4. We see that the facial dynamic with highest classification rate is neutral, which is intuitive due to the discriminative low-motion. Smiling on the other hand is classified with the biggest error, which might be due to the low-intensity expressions exhibited by the elderly patients (see Figure 5). We here note that annotation was performed utilizing audio and video. In a similar healthcare setting, an algorithm distinguishing between similar facial expressions and activities, based on spatio-temporal Dense Trajectories and improved Fisher Vectors has been proposed [START_REF] Dantcheva | Expression Recognition for Severely Demented Patients in Music Reminiscence-Therapy[END_REF], which we outperform by 16.1% utmost.

Observations and Future Work

In this section we summarize the main findings of this research. Fig. 5: Example images of two subjects from the ADP-dataset. From left to right we depict the classes "neutral", "talk", "smile" and "sing". Low-intensity expressions exhibited by elderly patients impede correct classification in some cases.

-Based on our experiments with iDT and the DNN-architectures, we observe that DNNs contribute highly in obtaining very promising classification rates, despite the small size of our dataset. We note that while the presented results significantly outperform previous methods based on handcrafted features (i.e., [START_REF] Dantcheva | Expression Recognition for Severely Demented Patients in Music Reminiscence-Therapy[END_REF]), when fusing handcrafted features (e.g., iDT) with DNN-based approaches, accuracy increases consistently.

-The methods adapted from action recognition generalize well to classification of facial dynamics. We observe this by the good classification rates, as well as by the facts that (a) temporal ConvNets performs better than spatial ConvNets (cf. [8] [30]), (b) fusion with iDT consistently improves the performance of DNNs (cf. [42] [8]), (c) Two-Stream ConvNets outperforms C3D (cf. [START_REF] Feichtenhofer | Convolutional two-stream network fusion for video action recognition[END_REF]). -Due to the limited size of our dataset, an end-to-end training from scratch of a DNN architecture is not feasible. Large action recognition datasets such as the UCF101 human action dataset offer suitable training alternatives for DNN-architectures. -The accuracy of facial dynamics classification depends on the features, as well as architecture used for assessment. Given the myriad of existing algorithms, exploring different types of features and architectures will be necessary to devise a robust solution. -High inter-and intra-variance of subjects and facial dynamics contribute to the remaining error rates. Further challenges include the low-intensity of facial dynamics exhibited by elderly patients, the unconstrained setting, allowing for facial dynamics to occur jointly, as well as ambiguous human annotation.

However, more work is necessary in this regard. Future work will involve finetuning of the involved methods. In addition, we intend to explore personalized facial dynamics assessment, where we will train algorithms on video sequences related to each patient, individually. Finally, we will design specific neural network models for facial dynamics assessment, placing emphasis on a single endto-end model, incorporating face detection, facial feature extraction, as well as classification.

Conclusions

In this work we have compared three methods for assessment of facial dynamics exhibited by AD-patients in mnemotherapy. The three tested methods include Improved Dense Trajectories, 3D ConvNets and Two-Stream ConvNets, which we have adapted from action recognition. Despite the pre-training of mentioned methods on an action recognition dataset, the methods have generalized very well to facial dynamics. Experiments conducted on an assembled dataset of Alzheimer's disease patients have resulted in a true classification rate of up to 79.5% for the fusion of Two-Stream ConvNets (ResNet-152) and iDT.

Fig. 1 :

 1 Fig. 1: C3D based facial dynamics detection: For each video sequence, faces are detected and the face sequences are passed into a pre-trained C3D network to extract a 4096-dim feature vector for each video. Finally a SVM classifier is trained to predict the final classification result. We have blurred the faces of the subject in this figure, in order to preserve the patient's privacy.

  [START_REF] Wang | Towards good practices for very deep two-stream convnets[END_REF] and Feichtenhofer et al. [8], one successor network, namely the Very Deep Two-Stream ConvNets outperformed the original Two-Stream ConvNets (by 3% on UCF101). Two-Stream ConvNets incorporates a spatial ConvNet, accepting as input single frame with dimension 224 × 224 × 3, as well as a separate stream -a temporal ConvNet, accepting as input stacked optical flow fields, with dimension 224 × 224 × 20. Specifically the optical flow field is composed of horizontal and vertical components D x and D y . A stack of D x and D y of 10 frames together are fed into the following ConvNet. Hence, while the first stream is based on RGB based features, the second stream is based on complementary motion between video frames, resulting to an increased accuracy over each of the streams.

1 N - 1 .

 11 Fig. 3: a) While the spatial ConvNet accepts a single RGB frame as input, the temporal ConvNet's input is the D x and D y of 10 consecutive frames, namely 20 input channels. Both described inputs are fed into the Two-stream ConvNets, respectively. We use in this work two variations of Very Deep Two Stream Con-vNets, incorporating VGG-16 [29] ResNet-152 [13] for both streams respectively. (b) The optical flow of each frame has two components, namely D x and D y . We stack 10 times D y after D x for each frame to form a 20 frames length input volume.

Fig. 4 :

 4 Fig. 4: Confusion matrix for categorized facial dynamics of Two-Stream ResNet-152 + iDT (best performing method) on the ADP-Dataset.

Table 1 :

 1 Classification accuracies of C3D, Very Deep Two-Stream ConvNets, iDT, as well as fusion thereof on the presented ADP-dataset. We report the Mean Accuracy (MA) associated to the compared methods. Abbreviations used: SN...Spatial Net, TN...Temporal Net.

	Method	MA (%)
	C3D	67.4
	SN of Two-Stream ConvNets (VGG-16)	65.2
	TN of Two-Stream ConvNets (VGG-16)	69.9
	Two-Stream ConvNets (VGG-16)	76.1
	SN of Two-Stream ConvNets (ResNet-152)	69.6
	TN of Two-Stream ConvNets (ResNet-152)	75.8
	Two-Stream ConvNets (ResNet-152)	76.4
	iDT	61.2
	C3D + iDT	71.1
	Two-Stream ConvNets (VGG-16) + iDT	78.9
	Two-Stream ConvNets (ResNet-152) + iDT	79.5

Table 2 :

 2 Classification accuracy of Two-Stream ConvNets (ResNet-152) + iDT on the ADP dataset. The numbers in parentheses indicate the number of "neutral", "smiling", "talking" and "singing" samples in each split.

	Split	Train	Test	Accuracy (%)
	1 289 (81, 70, 67, 71) 33 (9, 8, 8, 8)	78.8
	2 289 (81, 70, 67, 71) 33 (9, 8, 8, 8)	75.8
	3 289 (81, 70, 67, 71) 33 (9, 8, 8, 8)	78.8
	4 289 (81, 70, 67, 71) 33 (9, 8, 8, 8)	87.9
	5 289 (81, 70, 67, 71) 33 (9, 8, 8, 8)	78.8
	6 290 (81, 70, 68, 71) 32 (9, 8, 7, 8)	78.1
	7 290 (81, 70, 68, 71) 32 (9, 8, 7, 8)	84.4
	8 290 (81, 70, 68, 71) 32 (9, 8, 7, 8)	71.9
	9 291 (81, 71, 68, 71) 31 (9, 7, 7, 8)	83.9
	10 292 (81, 71, 68, 72) 30 (9, 7, 7, 7)	76.7
		Average		79.5

Table 3 :

 3 Mean Accuracy (%) of Two-Stream ConvNets (ResNet-152) + iDT on the ADP dataset. Assessment by category.

	Method	Neutral Smiling Talking Singing
	iDT	75.6	39.8	68.0	59.5
	C3D + iDT	75.6	53.8	81.3	64.4
	Two-Stream (ResNet-152)+iDT	86.7	65.4	88.0	77.2

https://ibug.doc.ic.ac.uk/resources/smile-detectors/

Acknowledgement

Y. Wang and A. Dantcheva were supported by the French Government (National Research Agency, ANR) under grant agreement ANR-17-CE39-0002.


Manuel Ramos

Marianne Boix

Didier Aussel

Ludovic Montastruc

Patrick Vilamajo

Serge Domenech

Water Exchanges in Eco-industrial Parks through Multiobjective Optimization and Game Theory

Keywords: Eco-industrial Park, Water Network, Multiobjective Optimization, Goal Programming, Game theory

The current environmental context makes urgent the development of robust methodologies able to design innovative industries. Industrial ecology, and most particularly the concept of eco-industrial parks, aims at proposing at several companies to gather in a same geographical site to share several fluxes (water, energy, utilities) in order to decrease environmental impacts of their industrial activities. A recent literature analysis has shown the emergence of new works devoted to the application of optimization methodologies to design greener and more efficient eco-industrial parks. In this work, the method of goal programming is applied for the first time to design optimal exchanges of water in an academic case of park. Goal programming is employed to deal with several conflicting objectives: the cost for each company included in the park. This method is proven to be reliable in this context because it proposes to obtain one solution instead of a set of optimal solutions that takes directly into account the preferences of the decision maker. Although the solution obtained in this study is quite interesting and is a good compromise, the main perspective of this work is to be extended by being coupled with a game theory approach so that an more equilibrate solution can be obtained.

Introduction

Over the past ten years, most industrialized countries have invested heavily in environmental research thanks to a general awareness about natural resources depletion. Especially in the case of fresh water, there is a real need to reduce its consumption by redefining and designing industrial networks with a low environmental impact. In response to these environmental problems, the concept of industrial ecology is born. Frosch and Gallapoulos (1989) initiated the scientific community to look very closely at the gathering of industries with a common goal of sustainable development. During the last twenty years, many terms and concepts have emerged in the broad field of industrial ecology. Eco-industrial parks (EIP) are a particular manifestation of industrial ecology and a definition commonly admitted was given by [START_REF] Lowe | Creating by-product resource exchanges: strategies for eco-industrial parks[END_REF] as: A community of manufacturing and service businesses seeking enhanced environmental and economic performance through collaboration in managing environmental and resource issues including energy, water, and materials. By working together, the community of businesses seeks a collective benefit that is greater than the sum of the individual The approach adopted in this work consists in designing optimal EIP networks (carrying water or energy) before its manufacturing. Optimization methods are then implemented in this context so that optimal networks for EIP can be designed. For a most extended bibliographic review to optimization methods applied to the design of eco industrial parks, the reader can refer to [START_REF] Boix | Optimization methods applied to the design of eco-industrial parks: a literature review[END_REF]. In this study the increasing interest for this research field during the past few years is highlighted. Figure 1 illustrates the number of studies published and cited during the fifteen past years with optimization and eco industrial parks keywords in the ISI Web of Science database. This paper aims at proposing a new methodology to deal with multiobjective optimization of eco-industrial parks. The first part is devoted to the presentation of objective functions, the superstructure and the methodology adopted. Then, a case study illustrates some results applied to the optimal design of a water network in an EIP.

Methodology

Objective functions

The analysis of previous studies has proven the existence of several types of objective functions. The most common is related to the economic cost, one can cite for example: profit of each industry, profit for the local community, transport and logistics or the net present value. However, although the economic cost remains very important for the feasibility of the project, in the context of industrial ecology, many other objectives can also be important. Objective functions in this type of problem are antagonist like for example, the economic cost versus environmental objective. Environmental objectives can be formulated through different ways: minimizing natural resources consumption (water), minimizing greenhouse gases emissions, minimizing water footprint or minimizing health and safety impacts. Another important aspect relative to sustainability development is the social and/or societal criterion. Although most of the time more qualitative than quantitative, social objectives are quite important and include the quality of life of workers, an index of satisfaction for the different participants [START_REF] Aviso | Design of robust water exchange networks for eco-industrial symbiosis[END_REF] or the number of jobs created for example. Finally, topological objectives can also be taken into account as in the work of [START_REF] Boix | Industrial water management by multiobjective optimization: from individual to collective solution through eco-industrial parks[END_REF] so that the network remains feasible.

Modeling eco-industrial parks

Modeling eco-industrial parks is somewhat a complex problem because of its size (thousands of variables, constraints and hundreds of binary variables) and the number of objectives to take into account. In order to model the network of an EIP, the concept of superstructure is used, it represents all the possible alternatives to connect each process of the network, this systemic approach allows to represent process design. Figure 2 shows the superstructure of an EIP including 3 industries and each industry is composed of process and regeneration units. After modeling the superstructure, the next step is to model the network with mass balances so that mathematical programming methods can be applied. A black box approach is adopted so that each parameter of the different processes are known, for more precision about the modeling stage, the reader can refer to [START_REF] Boix | Industrial water management by multiobjective optimization: from individual to collective solution through eco-industrial parks[END_REF].

Multi-objective optimization approaches

Resolution methodologies for multiobjective optimization are generally classified depending on the number of generated solutions and on the importance of the decision maker during the resolution. These approaches can be classified into two categories: Generative approaches: these methods are characterized by the generation of different solutions and the decision maker has to choose one solution among them. These a posteriori methods include scalar approaches like weighting method or the epsilon-constraint strategy coupled with mathematical programming and stochastic methods like genetic algorithm. With these methods, a Pareto front is built to propose a set of non-dominated solutions to the decision maker.

Preference-based methods: in this case, decision maker preferences are included along with the resolution and the optimization. These a priori approaches like for example, goal programming or interactive methods like NIMBUS directly include the decision maker preferences during the resolution so that one solution is finally obtained. In this work, we propose to adopt the goal programming approach to solve the case of water allocation in eco industrial parks. [START_REF] Boix | Optimization methods applied to the design of eco-industrial parks: a literature review[END_REF] highlighted the lack of studies dealing with optimization in order to design optimal configuration of an EIP. However, it is important to develop methodologies able to design an EIP where each industry has a gain compared to the case where it is individual. Furthermore, each company included in a park has its own objective (for example to minimize its costs) and all of these objectives are always antagonists. In this work, a multiobjective optimization approach is adopted in order to take into account each objective function associated to each company of the EIP. This approach has already been applied to the design of industrial water networks but never with the optimal design of the water network in an EIP which is a different problem. The optimization of eco-industrial parks includes several bottlenecks: one of the main characteristic of this problem is that the structure (represented by the existence or not of connections between processes and numerically modeled by binary variables) is dominating the solution. The resulting model is large and of MILP (mixed integer linear programming) type which constitute a great challenge and it is relatively new to solve this problem with the help of preference-based approaches. Indeed, previous studies have widely explored generation based approaches but some numerical problems were encountered especially when problems involve many binary variables. In most of the cases, the solver does not succeed to return a feasible solution and when it succeeds to build the Pareto front, it remains a very long and tedious task. In this work, a goal programming (GP) approach is adopted to obtain one solution that already includes the preferences of the decision maker. This approach is based on a recent study [START_REF] Ramos | Multiobjective optimization using goal programming for industrial water network design[END_REF] where it has been proven that GP can be a very reliable method to design industrial water networks following multiple antagonist objective functions. Based on an academic example, this paper explores 3 different scenarios of an EIP in order to point out the difficulties encountered with multiobjective optimization of EIPs.

Characteristics of eco-industrial parks

Multiobjective optimization through goal programming

Presentation of the case study

In this study, an EIP including 3 companies is designed, this academic example has been widely explored in the literature [START_REF] Olesen | Dealing with plant geography and piping constraints in water network design[END_REF][START_REF] Boix | Industrial water management by multiobjective optimization: from individual to collective solution through eco-industrial parks[END_REF]. For this reason, it remains a good example to test and to validate a new methodology because solutions are well-known. Each company is composed of 5 process units and the parameters are the same as those defined in [START_REF] Boix | Industrial water management by multiobjective optimization: from individual to collective solution through eco-industrial parks[END_REF]. Three different scenarios are explored (Figure 3), where the position of the regeneration unit differs, so that all the configurations are scanned : -Scenario 1: the three companies exchange water and each company owns its regeneration unit, chosen among three available types (depending on the outlet concentration); -Scenario 2: the three companies exchange water and share one regeneration unit; -Scenario 3: a mixed scenario where each company owns its regeneration unit and an additional unit is shared by the three companies. It is important to notice that for each above mentioned scenario, three objective functions are minimized: the total cost of each company involved in the EIP. In order to show that these objectives are antagonists, table 1 represents the pay-off table that include the values of two objectives while the third is minimized for scenario 2. This cost takes into account: fresh water consumption, regenerated water flow rate, cost of external and internal connections between processes and the capital cost of regeneration units.

Results with the goal programming approach

Results obtained with the goal programming approach are summed up in table 3. Monoobjective optimizations were carried out in a first step in order to obtain minimal (utopia) and maximal (nadir) values of each objective function, that is to say the minimal cost of each company when it is included in the EIP. Then, the goal programming methodology allows to have one solution for each scenario. It is important to note that the optimal solution found by GP leads to an intermediate solutions where all the companies are closed to their personal objective. The main interest of this study is to note that for each scenario, one company is favored compared to others. In scenario 1, company A is the closest from its objective whereas in scenarios 2 and 3, it is the company C. Goal programming has been proven to be an efficient approach to design water exchanges in EIP through multiobjective Scenario 1 Scenario 2 Scenario 3

optimization. Furthermore, it proposes a unique solution that satisfies a goal in very low computational times. Although the optimal solutions are intermediate and satisfying in terms of individual costs, it is of great interest to obtain more balanced solutions so that each company is satisfied at the same time.

A game theory perspective

Game theory could be a promising approach, particularly adapted to the case of the design of exchanges in an EIP. Several methods can be adopted like a non cooperative approach to preserve confidentiality of companies (Figure 4). Furthermore, the main barrier to integrate an EIP for industry is the lack of confidentiality and this approach could be very promising to overcome this problem. This approach could be useful to overcome the difficulties linked to information exchanges between companies in an IEP. However, it is important to deal with an authority that attends to minimize environmental impacts of the EIP. The existence of an optimal solution satisfying Nash equilibrium could insure that none of each company is prejudiced compared to others.

Conclusions

In this study, water exchanges in an EIP have been optimized by minimizing three objective functions with a goal programming approach. Goal programming is a reliable method of multiobjective optimization and it had been proven to be efficient to find optimal solutions to complex problems. This study dealt with a well-known example to prove the efficiency of the methodology and to make a preliminary study in order to apply further a game theory approach to this work. The development of this method could be coupled with a game theory approach in order to obtain more equilibrate results where no company is favoured to another following the concept of Nash equilibrium.

Figure 1 .

 1 Figure 1. Number of articles referenced in the last 15 years with the keywords: optimization and eco industrial park (from Boix et al., 2015).

Figure 2 .

 2 Figure 2.General superstructure of an EIP including 3 industries

Figure 3 .

 3 Figure 3. Different configurations of the EIP

Figure 4 .

 4 Figure 4. Example of a non cooperative approach to model an EIP through game theory.

  Since these preliminary studies, a lot of concrete examples have emerged through the world including the most famous example of the EIP of Kalundborg in Denmark. Other successful examples even more numerous are built all over the world. Most of them were built in industrialized countries of North America, Europe, or Australia but more recently it is in developing countries that many parks are born (such as China, Brazil and Korea for example).

	benefits each company would realize if it optimized its individual performance only.
	14 16		Number of publications					160		Number of citations
	12																120
	10
	8																80
	6
	4																40
	2
	0	2000	2001	2002	2003	2004	2005	2006	2007	2008	2009	2010	2011	2012	2013	2014	0	2000	2001	2002	2003	2004	2005	2006	2007	2008	2009	2010	2011	2012	2013	2014

Table 1 .

 1 Pay-off table for scenario 1 (the value in bold is minimized for each line)

	Pay-off table	Cost for ind. A	Cost for ind. B	Cost for ind.C
	Cost A	1.143	4.334	16.985
	Cost B	9.349	1.138	8.223
	Cost C	10.353	9.942	1.154

Table 2 .

 2 Results of the multiobjective optimization of the EIP through goal programming, cost are expressed in M$.

		Scenario 1	Scenario 2	Scenario 3
		Min Max	Solution GP	Min Max	Solution GP	Min Max	Solution GP
	Cost for company A	1.51 6.96	1.85	1.14 13.78	1.98	1.14 16.09	2.85
	Cost for company B	1.14 7.05	1.51	1.14 6.82	1.32	1.14 4.35	1.34
	Cost for company C	2.81 14.13	3.52	1.15 15.36	2.98	1.15 12.42	1.96