O Gaci

S Balev

Proteins: From Structural Classification to Amino Acid Interaction Networks

Keywords: proteins, amino acid, interaction network, structural classification

In this paper we introduce the notion of protein interaction network. This is a graph whose vertices are the protein's amino acids and whose edges are the interactions between them. Using a graph theory approach, we identify a number of properties of these networks. Some of them are common to all proteins, while others depend on the structure arrangement. The last group of properties allows to characterize structural classes, defined by CATH or SCOP, in the terms of interaction network properties.

Introduction

Proteins are biological macromolecules participating in the large majority of processes which govern organisms. The roles played by proteins are varied and complex. Certain proteins, called enzymes, act as catalysts and increase several orders of magnitude, with a remarkable specificity, the speed of multiple chemical reactions essential to the organism survival. Proteins are also used for storage and transport of small molecules or ions, control the passage of molecules through the cell membranes, etc. Hormones, which transmit information and allow the regulation of complex cellular processes, are also proteins. Genome sequencing projects generate an ever increasing number of protein sequences. For example, the Human Genome Project has identified over 30,000 genes which may encode about 100,000 proteins. One of the first tasks when annotating a new genome, is to assign functions to the proteins produced by the genes. To fully understand the biological functions of proteins, the knowledge of their structure is essential.

In their natural environment, proteins adopt a native compact three-dimensional form. This process is called folding and is not fully understood. The process is a result of interactions between the protein's amino acids which form chemical bonds. In this paper we iden-tify some of the properties of the network of interacting amino acids. We believe that understanding these networks can help to better understand the folding process.

There exist different classifications of proteins according to their structure, such as CATH [START_REF] Orengo | CATH -a hierarchic classification of protein domain structures[END_REF] and SCOP [START_REF] Murzin | SCOP: a structural classification of the protein database for the investigation of sequence and structures[END_REF]. Proteins from the same class have similar structures and most often, similar functions. In this paper we show that structure classes can also be defined in the terms of the properties of amino acid networks.

Protein Structure

Unlike other biological macromolecules (e.g., DNA), proteins have complex, irregular structures. They are built up by amino acids that are linked by peptide bonds to form a polypeptide chain. We distinguish four levels of protein structure:

• The amino acid sequence of a protein's polypeptide chain is called its primary or one-dimensional (1D) structure. It can be considered as a world over the 20-letter amino acid alphabet.

• Different elements of the sequence form local regular secondary (2D) structures, such as α-helices or β-strands.

• The tertiary (3D) structure is formed by packing such structural elements into one or several compact globular units called domains.

• The final protein may contain several polypeptide chains arranged in a quaternary structure.

By formation of such tertiary and quaternary structure, amino acids far apart in the sequence are brought close together to form functional regions (active sites). The reader can find more on protein structure in [START_REF] Branden | Introduction to protein structure[END_REF].

Based on the local organization of the secondary structure elements (SSE), proteins are divided in the following four classes [START_REF] Levitt | Structural patterns in globular proteins[END_REF]:

• All α, proteins have only α-helix secondary structure.

• All β, proteins have only β-strand secondary structure.

• α/β, proteins have mixed α-helix and β-strand secondary structure.

• α+β, proteins have separated α-helix and βstrand secondary structure.

From this first division, a more detailed classification can be done. The most frequently used ones are SCOP, Structural Classification Of Proteins [START_REF] Murzin | SCOP: a structural classification of the protein database for the investigation of sequence and structures[END_REF], and CATH, Class Architecture Topology Homology [START_REF] Orengo | CATH -a hierarchic classification of protein domain structures[END_REF]. They are hierarchical classifications of proteins' structural domains. A domain corresponds to a part of a protein which has a hydrophobic core and not much interaction with other parts of the protein.

SCOP

The SCOP classification is built manually from structural information. The process of classification starts by the division into domains of a protein. The protein is then classified on four levels, from the more general to the more specific :

1. Class: There are 4 main classes (see above) and 7 others with very small number of members. A class regroups proteins whose the secondary structure composition is similar.

2. Fold : The secondary structure composition, the spatial arrangement and the connexions are similar.

3. Superfamily: The structures and the functions tend to be similar.

4.

Family: Proteins have at least 30% of their sequence identical or have very similar functions and structures.

In 2007, the SCOP classification has identified 971 fold classes.

CATH

The CATH classification is maintained by both manual and automatic methods. Like SCOP, it is hierarchical and has 4 main levels and three additional levels concerning the similarity of protein sequences. The first 4 levels are the following :

1. Class: Proteins are grouped according to secondary structure composition and interaction between them. There are four classes: mainly α, mainly β, mixed α-β and all the rest.

2. Architecture: The secondary structure organization is the same.

3. Topology: Regroups structures whose foldings in terms of numbers, order and connexions of secondary structure are similar.

4. Homologous surperfamily: Domains which have structure and function very similar.

In 2007, CATH contained 1084 topology families.

Models and methods

The 3D structure of a protein is determined by the coordinates of its atoms. This information is available in Protein Data Bank (PDB) [START_REF] Berman | The protein data bank[END_REF], which regroups all experimentally solved protein structures. Using the coordinates of two atoms, one can compute the distance between them. We define the distance between two amino acids as the distance between their C α atoms. Considering the C α atom as a "center" of the amino acid is an approximation, but it works well enough for our purposes. Let us denote by N the number of amino acids in the protein. A contact map matrix is a N × N 0-1 matrix, whose element (i, j) is one if there is a contact between amino acids i and j and zero otherwise. It provides useful information about the protein. For example, the secondary structure elements can be identified using this matrix. Indeed, α-helices spread along the main diagonal, while β-sheets appear as bands parallel or perpendicular to the main diagonal. There are different ways to define the contact between two amino acids. Our notion is based on spacial proximity, so that the contact map can consider non-covalent interactions.

We say that two amino acids are in contact iff the distance between them is below a given threshold. A commonly used threshold is 7 Å and this is the value we use. Consider a graph with N vertices (each vertex corresponds to an amino acid) and the contact map matrix as incidence matrix. It is called contact map graph. The contact map graph is an abstract description of the protein structure taking into account only the interactions between the amino acids. Now let us consider the subgraph induced by the set of amino acids participating in SSE. We call this graph SSE interaction network (SSE-IN) and this is the object we study in the present paper. The reason of ignoring the amino acids not participating in SSE is simple. Evolution tends to preserve the structural core of proteins composed from SSE. In the other hand, the loops (regions between SSE) are not so important to the structure and hence, are subject to more mutations. That is why homologous proteins tend to have relatively preserved structural cores and variable loop regions. Fig. 1 gives an example of a protein and its SSE-IN. Note that the positions of the vertices in the graph do not correspond to the amino acid positions in the space. The graph is presented in this way only for visualization purposes.

Interaction networks

Many systems, both natural and artificial, can be represented by networks, that is, by sites or vertices bound by links [START_REF] Steven | Exploring complex networks[END_REF]. The study of these networks is interdisciplinary because they appear in scientific fields like physics, biology, computer science or information technology. These studies are lead with the aim to explain how elements interact with each other inside the network and what are the general laws which govern the observed network properties.

From physics and computer science to biology and the social sciences, researchers have found that a broad variety of systems can be represented as networks, and that there is much to be learned by studing these networks [START_REF] Broder | Graph structure in the Web[END_REF]. Indeed, the study of the Web [START_REF] Albert | The Diameter of the World Wide Web[END_REF], of social networks [START_REF] Wasserman | Social network analysis : methods and applications[END_REF] or of metabolic networks [START_REF] Jeong | The large-scale organization of metabolic networks[END_REF] are contribute to put in light common non-trivial properties to these networks which have a priori nothing in common. The ambition is to understand how the large networks are structured, how they evolve and what are the phenomena acting on their constitution and formation [START_REF] Watts | Collective dynamics of 'small-world' networks[END_REF].

In this section we present some measures that we use to describe proteins' SSE-IN. Among these measures, there are simple ones, the most frequently used, but also more subtle, which allow a more precise discrimination between interaction networks.

Diameter and mean distance

The distance in a graph G = (V, E) between two vertices u, v ∈ V , denoted by d(u, v), is the length of the shortest path connecting u and v [START_REF] Diestel | Graph Theory[END_REF]. If there is no path between u and v, we suppose that d(u, v) is undefined.

A graph diameter, D, is the longest shortest path between any two vertices of a graph [START_REF] Diestel | Graph Theory[END_REF]:

D = max{d(u, v) : u, v ∈ V }
The mean distance is defined as the average distance between each couple of vertices:

d G = 2 n(n -1) u,v∈V d(u, v)

Density and mean degree

A degree of a vertex u, k u , is the number of edges incident to u. The mean degree, k G , of a graph G is definied as follows:

k G = 1 n u∈V k u = 2m n
The density, denoted δ, is defined as the ratio between the number of edges in a graph and the maximum number of edges which it could have:

δ(G) = 2m n(n -1) ∼ 2m n 2
The density of a graph is a number between 0 and 1. When the density is close to one, the graph is called dense, when it is close to zero, the graph is called sparse [START_REF] Coleman | Estimation of sparse jacobian matrices and graph coloring problems[END_REF].

Degree distribution

If n k is the number of vertices having degree k, then the degree distribution is given by the next formula:

p k = n k n
The cumulative degree distribution [START_REF] Amaral | Classes of small-world networks[END_REF][START_REF] Erdõs | On random graphs I[END_REF] is defined as follows:

P k = ∞ k =k p k
The law distribution is defined as follows [START_REF] Newman | The strucuture and function of networks[END_REF]:

P k ∼ ∞ k =k k -α ∼ k -(α-1)
This distribution decreases in a polynomial way so that the number of vertices with weak degree is important, while a small number of vertices have high degree (see Fig. 2). The last are called "hubs", that is, sites which have a large connectivity through the network.

The degree distribution can also follow a Poisson law meaning that a small number of vertices have few links, a large number of vertices have a moderate number of incident edges, and a small number of vertices have a large number of incident edges (see Fig. 3). The Poisson distribution law is expressed as following: The degree distribution analysis is an important characteristic of networks because it involves their internal organisation [START_REF] Amaral | Classes of small-world networks[END_REF]. According to the kind of distribution followed, particularly if it's a power law, an interaction network can belong to a general model like scale-free model [START_REF] Barabási | Emergence of scaling in random networks[END_REF][START_REF] Goh | Universal behavior of load distribution in scale-free networks[END_REF] or small-world model [START_REF] Watts | Collective dynamics of 'small-world' networks[END_REF].

p k ∼ e -λ λ k k!

Clustering coefficients

Watts and Strogatz proposed a measure of clustering [START_REF] Watts | Small Worlds[END_REF] and defined it as a measure of local vertices density, thus for each node v, the local clustering around its neighbourhood is defined in the following way:

C v = 1 2 k v (k v -1)
The clustering coefficient is a ratio between the number of edges and the maximum number of possible edges in the vertice neighbourhood. If we extend the previous defintion to the entire graph, the clustering is given by the expression:

C local = 1 n v∈V number of connected neighbour pairs C v
The last defintion is mainly local because for each node, it involves only its neighbourhood.

The global clustering was studied by Newman et al. [START_REF] Newman | The strucuture of scientific collaboration networks[END_REF] and can be mesuared by the following formula:

C global =
3 × number of triangles in the graph number of connected triplets of vertices A triangle is formed by three vertices which are all connected and a triplet is constituted by three nodes and two edges. The global clustering coefficient C global is the mean probability that two vertices that are neighbors of the same other vertex will themselves be neighbors.

Experimental results

The first step before studying the proteins SSE-IN is to select them according to their SSE arrangements. Thus, a protein belongs to a CATH topology level or a SCOP fold level iff all its domains are the same. We have worked with the CATH v3.1.0 and SCOP 1.7.1 files. We have computed the measures from the previous section for three of each hierarchical classification, namely SCOP and CATH (see Table 1). We have chosen these three families by classification, in particular because of their huge protein number. Thus, each family provides a broad sample guarantying more general results and avoiding fluctuations. Moreover, these six families contain proteins of very different sizes, varying from several dozens to several thousands amino acids in SSE.

Name

Diameter and mean distance

Table 2 shows the average diameter for each one of the studied families. We observe very close diameters between TIM Barrel and TIM beta/alpha-barrel and also between Lysozyme and Lysozyme-like families. This is explained by the fact that each pair of families contains almost the same proteins, in other worlds, Lysozyme topology in CATH is the equivalent of Lysozyme-like fold level in SCOP. Figure 4 shows the distribution of the diameter values for two of the studied families. We observe that the distribution follows roughly a Poisson law. These results confirm that the mean diameter is a suitable property to discriminate families between them.

Name

The diameter being an upper bound of distances in interaction networks, we expect that the mean distance d G will be lower than D. Table 3

Density and mean degree

As defined earlier, the density measures the ratio between the number of available edges and the number of all possible edges. Results presented in Table 4 show that the two families TIM Barrel and TIM beta/alphabarrel have the minimum density. It has a consequence on their SSE-IN topology. When the density is low, the network is less connected and consequently, the diameter and the average distance are higher. Comparing these results to Tables 1 and2

Degree distribution

We compute the cumulative degree distribution for all proteins SSE-IN of studied families. A sample of our results is presented on Figure 5. We can remark that the curves follow a power law distribution and can be approximated by the following power-law function: We observe the same results for all studied proteins. To explain this phenomenon, we have to rely on two facts. First, the mean degree of all proteins SSE-IN is nearly constant (see Table 5). Second, the degree distribution, see Figure 6, follows a Poisson distribution whose peak is reached for a degree near k G . These two facts imply that for degree lower than the peak the cumulative degree distribution decreases slowly and after the peak its decrease is fast compared to an exponential one. Consequently, all proteins SSE-IN studied have a similar cumulative degree distribution which can be approximated by a unique power-law function.

Clustering coefficients

The

Conclusion and perspectives

In this paper we introduce the notion of interaction network of amino acids of a protein (SSE-IN) and study some of the properties of these networks. We give different means to describe a protein structural family by characterizing their SSE-IN. Some of the properties, like diameter and density, allow to discriminate two distinct families, while others, like mean degree and power law degree distribution, are general properties of all SSE-IN. Thus, proteins having similar structural properties and biological functions will also have similar SSE-IN properties. In this way our model allows us to draw a parallel between biology and graph theory. The characterization we propose constitutes a first step of a new approach to the protein folding problem. The properties we identified, both general and specific, can give us an insight on the folding process. They can be used to guide a folding simulation in the topological pathway from unfolded to folded state.

Another perspective is to study more deeply the general properties of SSE-IN, in particular degree distribution, and associate them to more general models, such as scale-free or small-world networks, whose behavior and evolution are well known.

Fig. 1 .

 1 Fig. 1. Protein 1COY and its SSE-IN

Fig. 2 .

 2 Fig. 2. Cumulative distribution following power law

Fig. 3 .

 3 Fig. 3. Poisson distribution. Each curve has a peak close to k = λ near the mean degree k G

Fig. 5 .

 5 Fig. 5. Cumulative degree distribution for 1RXC from Rossman fold, top, and 1HV4 from TIM beta/alpha-barrel, bottom.

Fig. 6 .

 6 Fig. 6. Degree distribution for 1RXC from Rossman fold, top, and 1HV4 from TIM beta/alpha-barrel, bottom.

Table 1 .

 1 Families studied, mainly due to their protein number

		Type	Class	Proteins
	Rossmann fold CATH	α β	2576
	TIM Barrel	CATH	α β	1051
	Lysozyme	CATH Mainly α	871
	Globin-like	SCOP	All α	733
	TIM β/α-barrel SCOP	α/β	896
	Lysozyme-like	SCOP	α + β	819

Table 2 .

 2 Average diameter for each family

		D
	Rossmann fold	18.84
	TIM Barrel	19.83
	Lysozyme	12.81
	Globin-like	15.65
	TIM beta/alpha-barrel 20.09
	Lysozyme-like	12.85

Table 3 .

 3 confirms this. Again, we observe very close values between the equivalent SCOP and CATH families for the reasons discussed above. But we can also see that different families have values which allow discrimination between them based on this parameter. It is interesting to note that the ratio D/d G is about 2.5 for all the families. The last property is a characterization of all proteins' SSE-IN. Average of mean distances for each family

	Name	d G
	Rossmann fold	7.26
	TIM Barrel	7.79
	Lysozyme	4.99
	Globin-like	6.64
	TIM beta/alpha-barrel 7.86
	Lysozyme-like	5.03

Table 4 .

 4 one can see the inversely proportional relation between density in one hand, and diameter and average distance on the other. Average density for each family The mean degree, k G is presented in Table5. The observed values are close enough from one family to another. That is why the mean degree is not discriminating property, but rather a property characterizing all proteins' SSE-IN.

	Name	δ(G)
	Rossmann fold	0.033
	TIM Barrel	0.030
	Lysozyme	0.038
	Globin-like	0.034
	TIM beta/alpha-barrel 0.029
	Lysozyme-like	0.042

Table 5 .

 5 Average of mean degrees for each family

Table 6 .

 6 local clustering C local measures the fraction of pairs of a vertex's neighbors and the global clustering C global gives the probability that among three vertices at least two are connected. The results presented in Table6show that the clustering coefficients are close for different families and cannot be correlated to density values. Consequently, the neighbour density remains independent of the previously studied properties. Clustering coefficients for each family

	Name	C local C global
	Rossmann fold	0.63	0.56
	TIM Barrel	0.64	0.57
	Lysozyme	0.65	0.58
	Globin-like	0.63	0.57
	TIM beta/alpha-barrel 0.64	0.57
	Lysozyme-like	0.66	0.58


domain: info scco

text:
Hamza Bouzid
email: hamza-bouzid@um5r.ac.ma

Lahoucine Ballihi
email: lahoucine.ballihi@fsr.um5.ac.ma

Facial Expression Video Generation Based-On Spatio-temporal Convolutional GAN: FEV-GAN

Facial expression generation has always been an intriguing task for scientists and researchers all over the globe. In this context, we present our novel approach for generating videos of the six basic facial expressions. Starting from a single neutral facial image and a label indicating the desired facial expression, we aim to synthesize a video of the given identity performing the specified facial expression. Our approach, referred to as FEV-GAN (Facial Expression Video GAN), is based on Spatio-temporal Convolutional GANs, that are known to model both content and motion in the same network. Previous methods based on such a network have shown a good ability to generate coherent videos with smooth temporal evolution. However, they still suffer from low image quality and low identity preservation capability. In this work, we address this problem by using a generator composed of two image encoders. The first one is pre-trained for facial identity feature extraction and the second for spatial feature extraction. We have qualitatively and quantitatively evaluated our model on two international facial expression benchmark databases: MUG and Oulu-CASIA NIR&VIS. The experimental results analysis demonstrates the effectiveness of our approach in generating videos of the six basic facial expressions while preserving the input identity. The analysis also proves that the use of both identity and spatial features enhances the decoder ability to better preserve the identity and generate high-quality videos. The code and the pre-trained model will soon be made publicly available. .

Introduction

Facial expressions have always been considered one of the essential tools for human interaction. Integrating the ability to recognize and synthesize facial expressions to machines provides a natural and smooth interaction. Which opens the door to many exciting new applications in different fields, including the movie industry, e-commerce, and even in the medical field. Motivated by this, researches have studied facial expression recognition and have already reached a high level of precision, while facial expression generation has been more demanding and less studied in the state of the art. Recently, with the success of Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets, in: Advances in neural information processing systems[END_REF] in data generation, in particular image generation, the task of generating facial expressions has seen tremendous progress.

However, the dynamic facial expressions synthesis is even less studied due to the difficulty of the tasks: 1) learning the dataset distribution (facial structure, background), 2) representing a natural and smooth evolution of facial expressions (Temporal representation), and 3) preserving the same input identity. To address the high complexity of these three tasks, most existing methods tend to treat facial expression generation as a two-steps process. One step for the low dimensional temporal generation (motion) and the other for the spatial generation (content). Such methods [START_REF] Tulyakov | Mocogan: Decomposing motion and content for video generation[END_REF]Wang et al., 2018a;[START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF] are mostly based on 1) the generation of motion as codes in a latent space, thereafter 2) combining it with the input image embedding to generate frames individually, through the use of an Image-to-Image translation network. These methods are efficient in learning facial structure and identity preservation, but they are flawed when it comes to modeling spatio-temporal consistency and appearance retainment. This is caused by the independence between frames generation.

Motivated by the success of Deep spatio-temporal neural arXiv:2210.11182v1 [cs.CV] 20 Oct 2022

network models in recognition and prediction tasks [START_REF] Al Chanti | Deep learning for spatio-temporal modeling of dynamic spontaneous emotions[END_REF][START_REF] Tran | A closer look at spatiotemporal convolutions for action recognition[END_REF][START_REF] Ali | Exploiting dynamic spatio-temporal correlations for citywide traffic flow prediction using attention based neural networks[END_REF][START_REF] Ali | Exploiting dynamic spatio-temporal graph convolutional neural networks for citywide traffic flows prediction[END_REF], researchers have proposed a variety of one-step methods [START_REF] Vondrick | Generating videos with scene dynamics[END_REF][START_REF] Jang | Video prediction with appearance and motion conditions[END_REF]Wang et al., 2020a,b) that use fractionally strided 3D convolutions. Videos generated by these kinds of methods show more spatio-temporal consistency, however, lower video quality, more noise and distortions, and more identity preservation issues compared to two-steps methods. We claim this is due to the high complexity of the three tasks combined in a single network ( learning 1. the spatial presentation, 2. the temporal representation, 3. identity preservation). This requires a large network with high potential complexity and a large amount of data, which significantly increases the difficulty of model optimization.

To solve the issues of the low quality, noise and identity preservation capability faced by one-step methods, We propose encoding the input image into two codes in the latent space, using two feature extractors (E Id identity feature extractor, E s spatial feature extractor). We also suggest exploiting the high performance of state-of-the-art facial recognition systems, by utilizing a pre-trained facial recognition feature extractor as our identity encoder E Id . This grants identity related features that help with identity preservation. In addition, the use of a pre-trained feature extractor allows for applying the optimization process only on the other encoder E S , that is used to extract other spatial features in order to maintain sufficiently good quality while reconstructing facial expression video.

In summary, our contributions include the following aspects: 1. We propose a conditional GAN, with a single generator and a single discriminator, that generates at each time step a dynamic facial expression video, corresponding to the desired class of expressions. The generated videos present a realistic appearance, and preserve the identity of the input image. 2. We investigate the influence of utilizing two encoders E Id and E S , where E Id is a facial identity feature extractor and E S is a spatial feature extractor. 3. We exploit the high potential of state-of-the-art facial recognition systems. We use a pre-trained face recognition model as our generator encoder E Id , which will ensure strongly related identity features. This aims to facilitate the task of the decoder by providing meaningful and structured features. 4. We deeply evaluate our model, quantitatively and qualitatively, on two public facial expressions benchmarks: MUG facial expression database and Oulu-CASIA NIR&VIS facial expression database. We compare it with the recent state-of-the-art approaches: VGAN [START_REF] Vondrick | Generating videos with scene dynamics[END_REF], MoCoGAN [START_REF] Tulyakov | Mocogan: Decomposing motion and content for video generation[END_REF], ImaGINator [START_REF] Wang | Imaginator: Conditional spatio-temporal gan for video generation[END_REF] and [START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF].

Related Work

Static Facial Expression Generation -Facial expressions synthesis was initially achieved through the use of traditional methods, such as geometric interpolation [START_REF] Pighin | Synthesizing realistic facial expressions from photographs[END_REF],Parameterization [START_REF] Raouzaiou | Parameterized facial expression synthesis based on mpeg-4[END_REF], Morphing [START_REF] Beier | Feature-based image metamorphosis[END_REF], etc. These methods show success on avatars, but they fall short when dealing with real faces, and they are unable to generalize a flow of movement for all human faces due to the high complexity of natural human expressions and the variety of identity-specific characteristics. To face these limitations, neural networks based methods have been applied on facial expressions generation, including RBMs [START_REF] Zeiler | Facial expression transfer with input-output temporal restricted boltzmann machines[END_REF], DBNs [START_REF] Sabzevari | A fast and accurate facial expression synthesis system for color face images using face graph and deep belief network[END_REF] and Variational Auto-Encoders [START_REF] Kingma | Auto-encoding variational bayes[END_REF],etc. These methods learn acceptable data representations and better flow between different data distributions compared to prior methods, but they face problems such as the lack of precision in controlling facial expressions, low resolution and blurry generated images.

With the appearance of GANs, multiple of its extensions have been dedicated to facial expressions generation. [START_REF] Makhzani | Adversarial autoencoders[END_REF] and [START_REF] Zhou | Photorealistic facial expression synthesis by the conditional difference adversarial autoencoder[END_REF] exploit the concept of adversity with auto-encoders to present Adversarial Auto Encoders. [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] propose a conditional GAN, namely CycleGAN, that uses Cycle-Consistency Loss to preserve the key attributes (identity) of the data. [START_REF] Choi | Stargan: Unified generative adversarial networks for multi-domain image-to-image translation[END_REF] address the inefficiency of creating a new generator for each type of transformation, by proposing an architecture that can handle different transformation between different datasets. (Wang et al., 2018b) suggest exploiting the U-Net architecture as GAN generator, in order to increase the quality and the resolution of generated images. US-GAN [START_REF] Akram | Us-gan: On the importance of ultimate skip connection for facial expression synthesis[END_REF] uses a skip connection, called the ultimate skip connection, that links the input image with the output of the model, which allows the model to focus on learning expression-related details only. the model outputs the addition of the input image and the generated expression details, improving therefore the identity preservation, but displaying artifacts in areas related to the expression (mouth, nose, eyes). The studies above established the task of generating classes of facial expressions (sad, happy, angry, etc.), but in reality, the intensity of the expression inhibits the understanding of the emotional state of the person. ExprGAN [START_REF] Ding | Exprgan: Facial expression editing with controllable expression intensity[END_REF] used an expression controller module to control the expression intensity continuously from weak to strong. Methods like GANimation [START_REF] Pumarola | Ganimation: Anatomically-aware facial animation from a single image[END_REF], EF-GAN [START_REF] Wu | Cascade ef-gan: Progressive facial expression editing with local focuses[END_REF] used Action Units (AUs) in order to learn conditioning the generation process which offers more diversity in the generated expressions. Other methods such as G2-GAN [START_REF] Song | Geometry guided adversarial facial expression synthesis[END_REF] and GC-GAN [START_REF] Qiao | Geometrycontrastive generative adversarial network for facial expression synthesis[END_REF] exploited Facial Geometry as a condition to control the facial expression synthesis. The objective of the latter models is to take as input a facial image and facial Landmarks in form of binary images or landmarks coordinates, then learn to generate a realistic face image with the same identity and the target expression. [START_REF] Kollias | Deep neural network augmentation: Generating faces for affect analysis[END_REF] and [START_REF] Bai | Data-driven dimensional expression generation via encapsulated variational auto-encoders[END_REF] utilize labels from the 2D Valence-Arousal Space, in which the valence is how positive or negative is an emotion and the arousal is the power of the emotion activation [START_REF] Russell | A circumplex model of affect[END_REF], to guide the process of facial expression generation, enhancing the variety and the control of the generated expressions. All these approaches and others have established the task of facial expression generation, but have not considered the dynamicity of these expressions.

Dynamic Facial Expression Generation -Facial expressions are naturally dynamic actions that contain more information and details than a single pose, e.g. the speed of facial expression transformation, head movements when displaying the expression, etc. This information can be significant in understanding the emotional state of a person. To achieve this, methods like [START_REF] Ha | Marionette: Few-shot face reenactment preserving identity of unseen targets[END_REF][START_REF] Tang | Eggan: Learning latent space for fine-grained expression manipulation[END_REF][START_REF] Li | Facial landmarks and expression label guided photorealistic facial expression synthesis[END_REF][START_REF] Tu | Image-to-video generation via 3d facial dynamics[END_REF][START_REF] Vowels | Vdsm: Unsupervised video disentanglement with state-space modeling and deep mixtures of experts[END_REF] focus on facial expression transfer, in which the facial expression is transferred from a driver to a target face, while aiming to preserve the target identity even in situations where the facial characteristics of the driver differs widely from those of the target. In other methods, the motion is generated separately as codes in the latent space, these codes are then fed to the generator in order to generate frames of the video individually. For example, MoCoGAN [START_REF] Tulyakov | Mocogan: Decomposing motion and content for video generation[END_REF] decomposes the video into content and motion information, where the video motion is learned by Gated RNN (GRU) and the video frames are generated sequentially by a GAN. RV-GAN [START_REF] Gupta | Rv-gan: Recurrent gan for unconditional video generation[END_REF] uses a transpose (upsampling instead of downsampling) convolutional LSTMs as GAN generator to generate frames individually. However, the results of both models present content and motion artifacts, and they both could only be applied to seen-before identities, and a finite number of expressions. In [START_REF] Fan | Controllable imageto-video translation: A case study on facial expression generation[END_REF], the principle of MoCoGAN is extended by adding an encoder that helps preserving the input identity, and a coefficient to control the degree of the expression continuously. The authors of (Wang et al., 2018a) utilize a Multi-Mode Recurrent Landmark Generator to learn generating variant sequences of facial landmarks of the same category (e.g. different ways to smile), translated later to video frames. In [START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF], the authors exploit a conditional version of manifold-valued Wasserstein GAN to model the facial landmarks motion as curves encoded as points on a hypersphere. The W-GAN learns the distribution of facial expression dynamics of different classes, from which new facial expression motions are synthesized and transformed to videos by Texture-GAN. Other works have investigated guiding facial expression generation by speech audio data, such as [START_REF] Chen | Talkinghead generation with rhythmic head motion[END_REF][START_REF] Guo | Ad-nerf: Audio driven neural radiance fields for talking head synthesis[END_REF][START_REF] Wang | One-shot talking face generation from single-speaker audio-visual correlation learning[END_REF][START_REF] Liang | Expressive talking head generation with granular audiovisual control[END_REF], or by a combination of audio and facial landmark information, like [START_REF] Wang | Audio2head: Audio-driven one-shot talking-head generation with natural head motion[END_REF][START_REF] Wu | Imitating arbitrary talking style for realistic audio-driven talking face synthesis[END_REF][START_REF] Sinha | Emotion-controllable generalized talking face generation[END_REF] . All the methods mentioned before are methods that generate a single frame at a time-step, which lowers the dependency between the video frames causing the lack of spatio-temporal consistency. Contrasted to the methods mentioned before, methods like VGAN [START_REF] Vondrick | Generating videos with scene dynamics[END_REF], G3AN (Wang et al., 2020a) and ImaGINator [START_REF] Wang | Imaginator: Conditional spatio-temporal gan for video generation[END_REF]) use a single step for the generation of the whole facial expression video, by employing fractionally strided spatio-temporal convolutions to simultaneously generate both appearance, as well as motion. VGAN decomposes the generated videos into two parts, a static section (the background) and a dynamic section, which imposes the use of a generator composed of two streams, for generating the background and the foreground, that are combined in the output to generate the whole video. G3AN aims to model appearance and motion in disentangled manner. This is accomplished by decomposing appearance and motion in a three-stream Generator, where the main stream models spatio-temporal consistency, while the two auxiliary streams enhance the main stream with multi-scale appearance and motion features, respectively. Both VGAN and G3AN are unconditional models that start from a Gaussian noise input, causing the lack of identity preservation and of control over the generated expression. In order to avoid these problems, ImaGINator uses a blend of auto-encoder architecture and spatio-temporal fusion mechanism, where the low-level spatial features in the encoder are sent directly to the decoder (the same concept as U-Net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]). It also uses two discriminators, one processes the whole video and the other processes frame by frame. Videos generated by these kinds of methods show more spatio-temporal consistency but lower video quality, more noise and less identity preservation compared to two-steps methods.

Motivated by the discussed above, we present a novel onestep approach for facial expression generation based on fractionally strided spatio-temporal convolutions. The rest of the paper is organized as follows. In section 3 we introduce our new FEV-GAN model. Section 4 shows the experimental settings and the quantitative and qualitative analysis of the model. Section 5 concludes the paper and provides perspectives for future research.

Proposed Approach

As stated in the introduction, our main aim is to establish a model that generates dynamic facial expression videos from appearance information and expression category. Thus, we formulate our goal as learning a function G : {I, L} -→ Ŷ, where I is the input image, L is the label vector and Ŷ is is the generated video.

To achieve this objective, we propose a Framework consisting of the following components: a Generator network G built on an encoder-decoder architecture. The encoders E Id and E S take as input a single image I and extract identity features F Id and spatial features F S . The decoder G dec utilizes the extracted features (F Id ,F S ) and a label L to generate a realistic video Ŷ. Finally a discriminator D assists the learning of the Generator for both appearance and expression category. The overview of our approach is shown in Fig. 1.

FEV-GAN Model Description

In the following, the architecture of our network is described, and details on the generator G and the discriminator D are provided.

Generator E Id , E S , G dec : As shown in Fig. 1, our generator consists of three networks, a pre-trained image identity encoder E Id , a randomly initialized encoder E S and a video decoder G dec . The encoder E Id is a well known state-of-the-art facial recognition model VGG-FACE [START_REF] Parkhi | Deep face recognition[END_REF] feature extractor. It takes a (64 × 64 × 3) RGB image I as input and transforms it into 1024 feature maps of (14 × 14), containing the facial identity features F Id . The encoder E S is initialized with a Gaussian noise. It extracts 512 feature maps of (14 × 14) features F S that contain more spatial details beside the identity features. Then, the features F Id , F S and the label vector L are concatenated and utilized by the decoder G dec to generate the new video. The decoder combines spatio-temporal convolutions and fractionally strided convolutions to transform the input tensor into the high dimensional generated video. Tri-dimensional convolution offers spatial and temporal invariance. Fractional convolution is an efficient over-sampling tool, allowing to transform the latent tensor into a (32 × 64 × 64 × 3) video.

Discriminator D: The objectives of the discriminator is to learn the ability (i) to classify realistic scenes from synthesized scenes, (ii) to recognize realistic motion, and (iii) to detect the difference between the different classes of motion. To achieve these three tasks we use a five-layer spatio-temporal convolutional network for the purpose of learning both visual content and motion modeling. The network takes (32 × 64 × 64 × 3) videos Y from the dataset or Ŷ generated by the generator G and the labels vectors as input. It then checks if the video is realistic and both the motion and the label are convenient. The discriminator architecture is designed to almost invert G dec by replacing fractionally strided spatio-temporal convolutions with direct spatio-temporal convolutions (to sub-sample instead of over-sample), and adjust the latter layer to produce a binary real or fake classification.

Loss Function

To train our model G : {I, L} -→ Ŷ, both network D and G are optimized using our objective function Eq 1.

L total (G, D) = L adv (G, D) + λ 1 L rec (G) + λ 2 L id (G), (1)
which consists of three losses: Adversarial loss L adv that helps the generator learn the database distribution. Reconstruction loss L rec that captures the overall structure of the video and improves the quality. Identity loss L id which ensures facial identity details preservation. Due to the large difference we found between the three losses values, λ 1 and λ 2 parameters are used to help stabilize the training and balance the optimization of our model.

We therefore aim to solve

G * = arg min G max D L total . (2)
We note that the parameters of E Id are frozen in the training phase, as it is a pre-trained model that already extracts facial features.

Adversarial Loss: Our conditional adversarial loss is a cross entropy loss, that is applied on both G and D, with the aim that G(I, L) learns to generate videos Ŷ that look similar to real videos Y, and D learns to distinguish between real samples Y and generated samples Ŷ. We train the model so as G aims to minimize the function while D aims to maximize it.

min G max D L adv = min G max D E Y∼P data (Y) [logD( Y ; L)] + E z∼P z (z) [1 -logD( G( I ; L) ; L)]. (3)
Reconstruction Loss: Our reconstruction loss at the video level is defined by

L rec = [ ||Y -Ŷ || 1 ], (4)
with the purpose of capturing the overall structure, video consistency and helping preserve the identity details. This loss is a L 1 norm loss between the generated videos Ŷ and the ground truth videos Y. By combining this loss with L adv , it helps the generator G create more realistic videos and reconstruct smooth expression motion.

Identity Loss: The Identity Loss is used for identity preservation. It is a L 1 norm loss similar to the L rec , but while L rec aims to minimize the L1 distance between pixel values of the generated and the ground truth videos, L id aims to minimize the L1 distance between the identity features of the input image and the frames of generated video. We exploit the same architecture of our VGG-FACE encoder to extract the identity features from both input and output data. This loss is formalized as:

L id = N i=0 [ ||F vgg (I) -F i vgg ( Ŷ) || 1 ], (5)
where F vgg (I) are the identity features of the input image I, and F i vgg ( Ŷ) are the identity features of the i th frame of Ŷ.

We furthermore use L id on 4 frames of the video instead of 32 frames. We count on spatio-temporal consistency of the 3D convolutions to generalize the identity preservation over the rest of the video.

training Algorithm

Algorithm 1 highlights the training process of the proposed FEV-GAN model. The training dataset (I, L, Y) is fed into Algorithm 1. First, the model parameters are initialized. The parameters of E Id are loaded from the pre-trained VGG-FACE model, while the rest of the parameters are initialized from a Gaussian distribution (1;2). The outer for loops are used to learn from the data for a given number of epoch and iterations in each epoch (3;4). The input image I i is fed into the encoders E Id (5) and E s (6), that encode it to identity F Id and spatial F s features, respectively. These features and the given label L i are then used by the decoder G dec to generate a fake video Ŷi (7). Next, the discriminator network D estimates the probability that a video is sampled from the dataset rather than the generator (8;9). The generated videos and the discriminator estimations are used to calculate the losses (10;12;13;14). Finally, the back propagation method and Adam optimizer are used to train the FEV-GAN model (11;15).

Experiment

To evaluate our model we performed an extensive experimental validation. In this following section, the experimental setup of our learning is detailed. Then, the model is evaluated quantitatively and qualitatively and compared to VGAN, MoCoGAN, ImaGINator and [START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF]. Finally, an ablation study is presented to observe the influence of each component of our model. Output: FEV-GAN model; 1 E Id ←initialized with non-learnable pre-trained VGG-FACE parameters; 2 E s , G dec , D ←initialized with learnable parameters sampled from a Gaussian distribution; 3 for the number of epochs do 4 for the number of iterations in an epoch do 5

F Id (I i ) ←-E Id (I i ); 6 F s (I i ) ←-E s (I i ); 7 Ŷi ←-G dec (F Id (I i ), F s (I i ), L i ); 8 E real ←-D(Y i , L i ); 9 E recon ←-D( Ŷi , L i ); 10 L D ←-log(E real ) + (1 -log(1 -E recon )); 11 D ←-D -α(∂L D /∂D); 12 L rec ←-|| Y i -Ŷi || 1 ; 13 L id ←-|| F vgg (Y i ) -F vgg ( Ŷi ) || 1 ; 14 L G ←-log(E real ) + L rec + L id ; 15 G ←-G -α(∂L G /∂G); end end 4.1. Dataset
The evaluation of our method is performed on: the MUG Facial Expression database (Multimedia Understanding Group Facial Expression database) [START_REF] Aifanti | The mug facial expression database[END_REF]: contains videos of 86 people, performing seven facial expressions: "happiness", "sadness", "surprise", "anger", "disgust", "fear" and "neutral". Videos in the database start and end with neutral expressions and display the peak of the expression in the middle. Each video consists of 50 to 160 RGB frames of 896 × 896 resolution. The data of 52 subjects is available to authorized Internet users, 25 subjects are available on request and the remaining 9 subjects are available only in the MUG laboratory. In this work we used the public data of 52 subjects. We only used the first half of the six basic expressions videos, which starts with the neutral expression and ends with the expression peak.

Oulu-CASIA NIR&VIS facial expression database [START_REF] Zhao | Facial expression recognition from near-infrared videos[END_REF]: consists of 480 videos of 80 people, between 23 to 58 years old, performing six facial expressions: "happiness", "sadness", "surprise", "anger", "disgust" and "fear". Each video consists of 9 to 72 RGB frames of 320 × 240 resolution, that begins with a neutral expression and end with the apex of the corresponding expression. The whole database is available to authorized Internet users.

Implementation Details

Before using the data, we first split the data in a subject independent manner into two sets, 80% of the data for the learning and 20% for the testing phases. We then cropped the face area and removed the background using OpenFace [START_REF] Baltrusaitis | Openface 2.0: Facial behavior analysis toolkit[END_REF], normalized all videos to 32 frames using the framework proposed in [START_REF] Karcher | Riemannian center of mass and mollifier smoothing[END_REF], and scaled each frame to 64 × 64 pixels. After the pre-processing phase, we ended up with 32×64×64×3 videos of different expressions and subjects with black background.

The weights of the networks E S , G dec and D are initially sampled from a Gaussian distribution of the mean zero and the standard deviation 0.01. The weights of the network E Id are initialized with the weights of the pre-trained VGG-FACE [START_REF] Parkhi | Deep face recognition[END_REF] and frozen in the training phase. Gradient descent is used for 400 epochs, in order to solve Eq 2. Binary cross entropy loss is used for the adversarial loss, and L 1 norm is used for both reconstruction and identity losses. Additionally, Adam Optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] is used to train the model with a learning rate of 0.0002 and a Momentum of 0.5. The pixels values of the input image and videos are scaled to the interval [-1.1]. Each layer of G dec is followed by the activation function ReLU [START_REF] Agarap | Deep learning using rectified linear units (relu)[END_REF] and batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], except for the output, which use tanh. LeakyReLU [START_REF] Xu | Empirical evaluation of rectified activations in convolutional network[END_REF] and batch normalization are used in the discriminator except for the input layer. The implementation of the network is done with the TensorFlow Framework based on the implementation of [START_REF] Vondrick | Generating videos with scene dynamics[END_REF]. The training on MUG data takes approximately 70 hours on an Nvidia GeForce GTX 1650 GPU (4Gb of memory). The training on Oulu-CASIA NIR&VIS takes approximately 40 hours on an Nvidia Titan V GPU (12GB of memory).

Evaluation Metrics And Baselines

To deeply evaluate our model quantitatively, we use several metrics:

1) PSNR: (Peak Signal-to-Noise Ratio) measures the pixellevel similarity between the generated videos and their ground truth.

2) SSIM: (Structural Similarity Index Measure) represents the structural similarity between real and reconstructed videos.

3) ACD [START_REF] Tulyakov | Mocogan: Decomposing motion and content for video generation[END_REF]: (Average Content Distance) measures content consistency in generated videos, based on the average of all pairwise L 2 distances between facial features of every two consecutive frames in a generated video. However, the ACD only represent identity consistency in the video, lacking information on the identity preservation of the input image.

4) ACD-I [START_REF] Zhao | Learning to forecast and refine residual motion for image-to-video generation[END_REF]: an ACD extension that measures the identity preservation of the input face in the generated video. It calculates the average of L 2 distances between the facial features of the frames of the video and the input image. To extract the facial feature vectors, we use OpenFace [START_REF] Amos | OpenFace: A generalpurpose face recognition library with mobile applications[END_REF], which is a deep model trained for facial recognition that can outperform human performance.

Higher SSIM and PSNR scores indicate better generated videos quality, lower ACD scores indicate similar faces in consecutive generated video frames, and lower ACD-I values indicate higher similarity between faces in the input images and the generated videos.

Regarding the state-of-the-art models used for comparison, we used the public codes of VGAN and ImaGINator provided by the authors with some minor changes. Since we deal with facial expression, we trained two versions of VGAN, the original version and a conditional version. In the conditional VGAN we used an encoder that transforms the input image to a latent code, the latent code is then concatenated with the labels and fed to the generator, that generates videos of the target expressions of the same input face. For MoCoGAN, we utilized the results given in [START_REF] Wang | Imaginator: Conditional spatio-temporal gan for video generation[END_REF]. As for [START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF], we used the results in the original paper.

Experimental Results & Analysis

Quantitative resultsto perform our quantitative analysis, we first generate 106 videos of 18 different subjects from the testing subset performing the six basic facial expressions.

First, we demonstrate that our model FEV-GAN offers better reconstruction capabilities than all baselines using PSNR and SSIM. Table 1 indicates that our model generates better quality videos with less noise, and preserves the general structure of the input through all the video. Then, we analyse the content consistency using ACD metric. Our model achieves similar content consistency to models with spatio-temporal convolutions (VGAN and ImaGINator) that are known to have high content consistency, while it surpasses MoCoGAN that uses Image-to-Image translation. As for identity preservation, our proposed model largely surpasses VGAN, C-VGAN and MoCoGAN, and is competitive with ImaGINator.

We note that VGAN does not preserve the identity. Its score is used for the purpose of representing the metric values where identity is not preserved. The ACD and ACD-I comparison does not include the model used in [START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF] for the reason that the authors have used a different approach to calculate the metrics.

Qualitative results -Fig. 2 presents examples of generated videos of the six facial expressions of the same identity. demonstrate examples of generated videos of the six facial expressions of different given identities. Both figures show results of our proposed model trained on MUG dataset (left) and Oulu-Casia dataset (right). The generated videos are taken randomly from the test results. We recall that the data used for learning and testing the model is subject independent. The synthesized videos display generally natural, smooth and continuous facial variations that can be controlled according to the input label. They also preserve the features of the input images, such as the identity, the beard and the glasses. Our model shows slightly better results on MUG data than on Oulu-Casia data. This is due to the fact that Oulu-Casia contains fewer data instances with a variety of skin colors, lighting conditions and with accessories (glasses) on the face, which increases the difficulty of pattern learning for the model. More examples of videos generated by our model are given in the Fig. 7 in Appendix.

In Fig. 4, a comparison between our results and the stateof-the-art on the MUG database is conducted. The figure illustrates videos generated by our model, VGAN, C-VGAN, MoCoGAN, ImaGINator and [START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF]. On the left side of the image, we show the input images utilized to generate the videos, in the middle, we show the videos generated by our model (a, c, e, g, i), and on the right side we show the videos generated by the baselines (b, d, f, h, j). In each comparison, each line, the videos generated by our model and the baseline contain the same identity and expression. In the first comparison (a,b), VGAN shows a natural smooth expression, but a total loss of the identity. In (c,d), C-VGAN generally preserves the input identity but with detail loss (like skin color), noise and artifacts. In (e,f), MoCoGAN offers sufficient identity preservation, but the identity is already used in the training set. MoCoGAN also suffers from the unnatural expression and distortions in the generated images, we suspect this causes the high ACD-I value even when the identity is preserved better than C-VGAN. In (g,h), ImaGINator preserves the input identity, but changes the skins color and displays an unnatural expression. In (i,j), the baseline shows sufficient identity preservation and structure consistency, but displays artifacts near the mouth and nose area. The figure demonstrates that our model generally surpasses the baselines in terms of identity preservation and quality of the generated videos. It also demonstrate that our model maintains the same contentconsistency and expression naturalness as spatio-temporal models. More qualitative comparison with these methods and others are conducted in Fig. 8 in Appendix.

In addition, we performed a subjective test, in which we aim to obtain opinions of human raters on videos generated by our method in comparison to the state-of-the-art. We asked 17 volunteers to compare videos generated by our method and by the baselines. All the volunteers are PhD students and researchers in the deep learning field, from different university laboratories. The volunteers were asked more than 30 questions, where, each time we offered them two videos, generated by our model and by one of the baselines, without revealing the source of the videos. The question they were asked is to choose the best one of two given videos based on the average of expression naturalness, identity preservation and video quality. As shown in Table 2, our method largely outperforms the cited models. The raters show strong preference for our method over C-VGAN (91.44% VS 08.56%), MoCoGAN (78.09% VS 21.91%), ImaGINator Table 2: Subjective comparison of FEV-GAN and baseline models. The reported results are the mean of the raters preferences.

Models

Rater preference(%) FEV-GAN/C-VGAN 91.44% / 08.56% FEV-GAN/MoCoGAN 78.09% / 21.91% FEV-GAN/ImaGINator 71.07% / 28.93% FEV-GAN / [START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF] 68.05% / 31.95%

(71.07% VS 28.93%), which corresponds to the quantitative results. The raters also commented that 1. in comparison to C-VGAN, we show almost the same expression naturalness and content consistency, while surpassing it in identity preservation and quality of the generated videos. 2. In comparison to MoCo-GAN, we surpass it in all criteria. 3. As for ImaGINator, we have the same level of identity preservation, but we offer better quality and more natural expressions. 4. The raters also mentioned that the closest to our model is the one proposed in [START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF]. they have stated that the baseline slightly outperforms ours in identity preservation, while we generate videos with less noise and artifacts.

The results of the quantitative and the qualitative analysis prove that our model maintain the same content consistency and expressions naturalness as other spatio-temporal convolution models, while offering better quality and identity preservation.

Ablation Study -In this section, we focus on demonstrating the importance of the techniques used to build our model. We showcase the effect of the double encoder method and the effect of the pre-trained facial feature extractor. This is conducted by performing the ablation study on the MUG database. We use the same evaluation metrics we used previously, on multiple versions of our model, where we cancel the target component and observe its effect. We first train two new other versions of our model. In the first one, we discard the spatial encoder E S , while in the second one we remove the identity features encoder E Id . Both networks are trained and evaluated using the same dataset, parameters, losses and for the same number of epochs, as the full network. Fig. 5 shows sequences generated by our full network, by the network w/o E S , and by the network w/o E Id . We deduce that our full network generates videos of the input identity performing the target expression with high quality and better facial details. As revealed in the sequences of Fig. 5.b, the network w/o E S generates videos of lower quality and worse identity preservation, and it does not sufficiently preserve important details like the skin color, facial hair, and minor details like the surrounding areas of the mouth, nose and eyes. As for Fig. 5.c, the videos generated by network w/o E Id show more natural expressions but they also show more noise and distortions and they totally lack the identity preservation, .

The tests shown in Table 3 reveal that the full version of the network gives much better results than the modified versions in terms of quality, content consistency and identity preservation. This can be explained by the existence of the two encoders architecture. E Id capability to guarantee strongly identity related features makes it easier for E S to learn extracting other features that provide more details and information, giving the decoder the capability to learn generating better quality videos.

Discussion and Limitations

From the quantitative and qualitative comparisons, we conclude that our model largely outperforms the benchmarks C-VGAN and MoCoGAN in all criteria. As for the comparison to ImaGINator, we show similar content consistency, however ImaGINator results display some identity detail loss and unnatural expressions with clear distortions in the mouth and eyes area Fig. 4.f Fig. 8.j . For [START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF] , It generates videos with high identity and structure preservation, but it also shows artifacts in the mouth and eyes area . We argue that our proposed model generates generally better quality videos of natural expressions of the input identity, with minimum noise or distortions.

However, there are some inaccuracies that flaw our FEV-GAN model. Fig. 6 illustrates some examples of imprecise video generation. For example, the model learns to synthesis the teeth region when not given in the source image, but fails at generating the eyes when they are not clearly displayed in the input image (1 st example of Fig. 6). Also, the model is trained with neutral expression input images. If given a non-neutral expression, the generated video does not display the transition from neutral to the target expression (2 nd example of Fig. 6). In addition, the model occasionally constructs videos with some minor distortions in the eyes or mouth regions (3 rd and 4 th example in Fig. 6).

Conclusion and perspectives

In this paper, we present a novel Conditional GAN, namely FEV-GAN, for effectively generating the six basic facial expressions videos, given an input single neutral image and a target facial expression category. Specifically, we address the low quality and identity preservation issues encountered by facial expressions generation models that utilize fractionally strided spatio-temporal convolutions.

Based on our state-of-the-art study, these issues are related to the difficulty of the task of generating dynamic facial expressions. Our FEV-GAN model remedies these issues by utilizing two distinct encoders E Id (identity-encoder) and E S (spatial-encoder) that extract respectively the identity features F Id and spatial features F S . These features are given as input to the decoder G dec in order to better preserve the identity and generate high quality facial expression videos.

We have deeply evaluated our method on two benchmark databases, The MUG facial expression database and Oulu-CASIA NIR&VIS facial expression database, quantitatively by using different metrics (PSNR, SSIM, ACD, ACD-I), and qualitatively by using expert human eye rating. The results of these tests confirm our claims and show that our method significantly surpasses the state-of-the-art baselines in dynamic facial expression generation.

To further this research, we plan to test the model with other state-of-the-art facial recognition encoders, such as FaceNet [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF], OpenFace [START_REF] Baltrusaitis | Openface 2.0: Facial behavior analysis toolkit[END_REF], and VGG-FACE2 [START_REF] Cao | Vggface2: A dataset for recognising faces across pose and age[END_REF]. We also project to adapt our model to utilize facial landmarks sequences and/or facial action units instead of one-hot vector as a target category label. This will offer a large variation of possible expressions and more control over the generated expression.Another interesting perspective is to extend this approach to 3D facial expression video and 2D/3D human action generation. We notice MoCoGAN (b) shows large artifacts and generates videos where the intensity of the expression does not increase continuously. The video starts with neutral expression and suddenly turns to the target expression. G3AN (d) shows the transition between the expression, but the transition frames are noisy and contain artifacts. RV-GAN (f) video displays better quality and expression smoothness. However, these methods lack the identity preservation, which is very important when dealing with facial expression generation. C-VGAN (h) shows smooth facial expression and preserves the main structure of the input identity but loses much important details. ImaGINator (j) sufficiently preserves the identity but the expressions are unnatural and the generated frames are distorted. [START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF] (l) and VDSM [START_REF] Vowels | Vdsm: Unsupervised video disentanglement with state-space modeling and deep mixtures of experts[END_REF] (n) show natural smooth facial expressions, and sufficient identity preservation. Nonetheless, VDSM images are blurry and [START_REF] Otberdout | Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets[END_REF] images display artifacts near the mouth area. Our generated samples present smoother and continuous facial expression evolution, sufficient identity details preservation, and they show less noise and artifacts compared to the stat-of-the-art models.

Fig. 1 :

 1 Fig. 1: Overview of the proposed model for generating facial expressions: FEV-GAN. The input image is mapped through two encoders, identity (E Id ) and spatial (E S ) encoders, into the latent codes F Id and F S . Both codes and the label L are fed to the decoder G dec that transforms them to a video of the input identity performing the desired expression. While the discriminator D has the purpose of classifying realistic scenes from synthesized scenes.

Algorithm 1 :

 1 Our Learning Algorithm for the proposed model FEV-GAN Input: Ground-truth Video Y = [Y 0 , Y 1 , ..., Y n ]; Input Image I = [I 0 , I 1 , ..., I n ]; Target expression label L = [L 0 , L 1 , ..., L n ];

  Fig. 3

Fig. 2 :

 2 Fig. 2: The generation of facial expression videos on the MUG database (left) and Oulu-Casia (right). The image sequences show the six basic facial expressions of the same subject on the test dataset. The presented images are sampled every two frames. More examples of different identities are given in supplementary material.

Fig. 3 :

 3 Fig. 3: Examples of videos generated by our model of the six basic facial expression performed by a person given in the input image.

Fig. 4 :

 4 Fig. 4: Qualitative comparison of sequences generated by FEV-GAN model and by the state-of-the-art models on the MUG database. The sequences generated by our model (a, c, e, g, i), by VGAN (b), C-VGAN (d) and by ImaGINator (h) are randomly selected from the test results. The sequence generated by MoCoGAN (f) and by (Otberdout et al., 2019) (j) are taken from the original papers. All the images are sampled with the time step of 4.

Fig. 5 :

 5 Fig. 5: Qualitative comparison of sequences generated by our full FEV-GAN model (a), by FEV-GAN w/o E S (b), and by FEV-GAN w/o E Id (c) on the MUG database.

Fig. 6 :

 6 Fig. 6: Examples of flawed video generation by our model.

Fig. 7 :

 7 Fig. 7: examples of sequences generated by our proposed model. The image sequences in each box show the six basic facial expressions performed by the same subject on the MUG test dataset.

Fig. 8 :

 8 Fig. 8: Qualitative comparison of sequences generated by FEV-GAN model and by the state-of-the-art models on the MUG database. The sequences generated by our model (a, c, e, g, i, k, l), by C-VGAN (h) and by ImaGINator (j) are randomly selected from the test results. The sequence generated by MoCoGAN (b), (Otberdout et al., 2019) (h), G3AN (d), RV-GAN (f), and by VDSM (n) are taken from the original papers. The absence of input identity indicate that the baseline does not preserve the input identity.

Table 1 :

 1 Quantitative comparison results of FEV-GAN and baseline models using PSNR, SSIM, ACD, and ACD-I metrics.

		Trained on MUG		Trained on Oulu-Casia
	Model	PSNR SSIM ACD ACD-I	PSNR SSIM ACD ACD-I
	VGAN (Vondrick et al., 2016)	16.32	0.41	0.14	1.55	15.09	0.61	0.27	1.37
	C-VGAN (Vondrick et al., 2016)	22.30	0.83	0.09	0.73	15.98	0.61	0.25	1.26
	MoCoGAN (Tulyakov et al., 2018)	18.16	0.58	0.15	0.90	-	-	-	-
	ImaGINator (Wang et al., 2020b)	20.29	0.85	0.08	0.29	22.98	0.84	0.07	0.16
	(Otberdout et al., 2019)	25.9	0.90	-	-	24.44	0.89	-	-
	The proposed FEV-GAN	27.10	0.91	0.09	0.23	25.61	0.89	0.12	0.19

Table 3 :

 3 Performance comparison of our model without E Id and E S

	Model	PSNR SSIM ACD ACD-I
	FEV-GAN w/o E S	25.44	0.90	0.11	0.34
	FEV-GAN w/o E Id 19.71	0.69	0.13	1.22
	Full FEV-GAN	27.10	0.91	0.09	0.23


E Espinosa

J Challita

J.-M Desfontaines

M.-E Val-Kennedy

M Marbouty

C Possoz

E Galli

F.-X Barre

MatP-mediated positioning of chromosome terminus regions in V. cholerae

V. cholerae is a curved rod-shaped bacterium with a single polar flagellum. Its genome is distributed on two circular chromosomes, ChrI and ChrII. Each carries a single origin of replication, oriI and oriII, with replication terminating in a diametrically opposite zone, terI and terII. ChrII is derived from a megaplasmid. Its replication begins after duplication of crtS, a sequence located in the middle of the left replication arm of ChrI, and ends at the same time as ChrI replication. Cell division is coordinated with the replication/segregation cycle by SlmA, an inhibitor of FtsZ-ring assembly that is bound all along ChrI and ChrII but terI and terII. -MatP ensures the polar positioning (1) and ordered segregation of terI loci, with dif1 being segregated last (2), as previously observed in E. coli.

-terI and terII both harbour numerous matS sites (3) and are organised in macrodomains by MatP (4). Yet, the deletion of MatP had little effect on the positioning (1) and ordered segregation of ChrII loci (2).

-MatP binds to matS sites even when they are far from the replication terminus and the FtsK activity zone (3,6,7). Its local enrichment depends on the the local density of matS sites (3, 6, 7), which dictates the possibility for MatP to position loci and prolong their cohesion (5 & 7).

-MatP-mediated cohesion does not depend on tetramerisation, nor any interaction with a cell division protein ( 8).

-Study the interplay between MatP, MukBEF, ParAB1/parS1 and ParAB2/parS2 in ChrI/ChrII organisation and segregation.

-Search for the factor that is responsible for the MatP-indepndent cohesion of sister copies of terI and terII loci behind replication forks .

  and terII positioning is key to SlmA-mediated control of cell division. ChrI encodes a homologue of the Escherichia coli MatP protein. E. coli MatP insulates the ter domain of the E. coli chromosome and dictate the positioning of sister ter copies at the centre of the cell by connecting them to the FtsZ-ring.


David Picard

Arnaud Revel

Matthieu Cord

IMAGE RETRIEVAL OVER NETWORKS: ANT ALGORITHM FOR LONG TERM ACTIVE LEARNING

Image retrieval over a network is the focus of this paper, as being a major challenge of content based image retrieval. We present a system that gathers feedbacks given by the users in order to learn the location of the searched images. As a result, the active learning of a content based relevance function is enhanced by the selection of hosts containing relevant examples. We achieve a long term merging of feedbacks over many sessions, without any knowledge about the category being searched at any session. Our system is based on mobile agents crawling the network in an ant-like behavior. Markers are used to learn a routing of the agent leading them to the relevant images. The benefits of the ant-like agents system is a natural parallelization of the processing as well as a distributed approach to the routing learning. We made experiments on the trecvid'05 key-frame dataset showing that the location of the categories were efficiently learned. Furthermore, the long-term learning of categories improves the interaction by reducing the number of labels needed during the interaction to obtain satisfying results.

INTRODUCTION

As multimedia devices (such as mobile phones, digital cameras, etc.) are becoming very usual, huge collections of digital images are available today. Finding images belonging to a specific category in these ever growing collections is a difficult task since searching within by hand has become impossible. Content Based Image Retrieval (CBIR) has been successfully proposed to answer this problem [START_REF] Veltkamp | Content-based image retrieval system: A survey[END_REF]. The main idea is to build a description based on the images content, and to find similarities between descriptions [START_REF] Niblack | The QBIC project: Querying images by content, using color, texture, and shape[END_REF]. The problem of such techniques is the well known semantic gap between the numerical values attached to images and the semantical concepts they belong to. In order to reduce the gap, machine learning techniques have been successfully adapted to train a similarity function in interaction with the user (using her labeling of the results) leading to the so called "relevance feedback" [START_REF] Wood | Iterative refinement by relevance feedback in content-based digital image retrieval[END_REF][START_REF] Huang | Image retrieval with relevance feedback: From heuristic weight adjustment to optimal learning methods[END_REF]. The best improvement has been done with the introduction of active learning, which aims at proposing for labeling the image that will at most enhance the similarity function when added to the training set [START_REF] Tong | Support vector machine active learning with applications to text classification[END_REF].

With the expansion of networks such as the Internet, peerto-peer networks or even personal networks, image retrieval has become a difficult task. As images are split into many collections over the web, the problem of CBIR is not only to find the most relevant images, but also to find the localization of relevant collections. The major part of CBIR computation being dedicated to the processing of the image descriptors, the fact that images are distributed over many sources should be more an advantage than a drawback since it means a possible paralleling. Moreover, we can assume that collections may be coherent in semantics, in other words, that images belonging to a specific concept may be encountered in only few well localized collections. For instance, a site can mostly contain touristic photographs (landscapes, buildings, etc.), while an other can mainly contain photographs of manufactured objects, and a third can only contain old paintings. Although CBIR in a distributed context has been noted as an interesting improvement [START_REF] Smeulders | Content-based image retrieval at the end of the early years[END_REF], it has been, to our knowledge, the focus of a few works. Chen presented a system for image retrieval in p2p network in CBMI'07 [START_REF] Chen | Scalable image retrieval with optimal configuration for p2p network database[END_REF]. In their system, the links between peers of the network are optimized in order to propagate the query to relevant hosts. We have proposed in a previous work [START_REF] Picard | Cbir in distributed databases using a multi-agent system[END_REF] to learn the location of relevant images from the interaction with the user. We carried out a smart cooperation between the interactive CBIR and a localization learning in a global architecture based on mobile agents. However, all the labels gathered during the interaction are forgotten at the end of the session of a classical CBIR system. The major challenge of a widely used CBIR system is to reuse these labels for later sessions in order to benefit from the previous interactive learning [START_REF] Cord | Image retrieval using long-term semantic learning[END_REF]. In a single-user CBIR system, the resulting long-term learning is possible but very slow due to the few labels available. The real interest is in our distributed context since we can gather labels over sessions from many users in parallel as the network is shared between users. In that sense, the knowledge given to the system through the relevance feedback can be gathered from all users. In this paper, we present a generalization for long-term optimization of our previous CBIR over networks system. The localization of the categories are learned over several sessions, enabling a routing of the mobile agents specific to the searched concept. We assume that the categories one might search of are well localized on the networks. Such specialized collections are already available, as for example collections of medical images or aerial images. Based on this assumption, we will learn a category dependent routing of the networks in order to retrieve images from relevant locations.

In the next section, an overview of our system is exposed. The section 3 contains the description of the routing learning algorithm during a session. Section 4 describes the long-term optimization. Finally, we present and discuss the experiments and results we obtained using our system on the trecvid2005 key-frame dataset 1 in section 5.

RETRIEVAL SCHEME

Our system is based on mobile agent technology. A mobile agent is an autonomous computer software with the ability to migrate from one computer to another and to continue its execution there. There are good reasons for using mobile agents in the distributed CBIR context, such as the reduction of the network load (the processing code of the agent being very small in comparison to the feature vector indexes) and the massive paralleling of the computation [START_REF] Lange | Seven good reasons for mobile agents[END_REF]. Functional description of our system showing the user in interaction with the relevance feedback loop (launching of agents, retrieval, display and labeling).

As described in Fig. 1, the user starts his query by giving an example or a set of examples to an interface [START_REF] Veltkamp | Content-based image retrieval system: A survey[END_REF]. A similarity function based on these examples is built [START_REF] Niblack | The QBIC project: Querying images by content, using color, texture, and shape[END_REF]. Mobile agents are then launched with a copy of this similarity 1 see http://www-nlpir.nist.gov/projects/tv2005/ function [START_REF] Wood | Iterative refinement by relevance feedback in content-based digital image retrieval[END_REF]. Every host of the network contains an agent platform in order to be able to receive and execute incoming mobile agents. The agent movements are influenced by markers (a numerical value locally stored on the host) following an ant-like behavior [START_REF] Revel | Web-agents inspired by ethology: a population of "ant"-like agents to help finding user-oriented information[END_REF][START_REF] Revel | From robots to web-agents: Building cognitive software agents for web-information retrieval by taking inspiration from experience in robotics[END_REF], as described in section 3

On each platform, an agent indexing the local images is run, and retrieves the relevant images for the incoming agents. As soon as they receive the answer of the index agent, the mobile agents return to the user's computer [START_REF] Tong | Support vector machine active learning with applications to text classification[END_REF] and the results are displayed on the interface [START_REF] Smeulders | Content-based image retrieval at the end of the early years[END_REF]. The user can label these results (1: relevant, -1: irrelevant), and the similarity function is updated consequently (7a) as well as the relevant paths of the network (7b). As the similarity function we use is based on SVM analysis [START_REF] Cortes | Support-vector networks[END_REF], the update only consists in adding the results and their labels to the training set and to train a new SVM.

Mobile agents are then relaunched with the improved similarity function. To tackle the semantic gap problem, an interactive loop consisting in several launching of mobile agents and labeling of the results is set [START_REF] Picard | Cbir in distributed databases using a multi-agent system[END_REF]. At the end of the interaction, mobile agents are launched for a very last time in order to retrieve the best results from each host [START_REF] Cord | Image retrieval using long-term semantic learning[END_REF]. The number of retrieved images is proportional to the level of the markers leading to this host, assuring that most of the best retrieved images are provided by relevant hosts.

ROUTING LEARNING

We give here a rapid overview of our learning algorithm described in [START_REF] Picard | Cbir in distributed databases using a multi-agent system[END_REF]. During a session, the similarity function is learned as well as a routing of the agents leading them to host containing relevant images. This routing is done by the antlike behavior of the agents. While moving from one host to another, agents are influenced by markers regarding the following rule :

P i = ph i k∈S ph k (1)
Where P i is the probability of an agent to move to host i, ph i being the value of marker of the host i, and S the possible destinations. We use an ant-like algorithm [START_REF] Dorigo | The ant system: Optimization by a colony of cooperating agents[END_REF][START_REF] Bonabeau | Inspiration for optimization from social insect behaviour[END_REF] following the model of the ethologist J. L. Deneubourg [START_REF] Deneubourg | The dynamics of collective sorting: Robot-like ants and ant-like robots[END_REF] to reinforce the markers. The markers act like pheromones for ants, in the sense that ants tend to follow the path containing the highest quantity of pheromone while searching for food. Once they have found food, they go back to the nest laying a trail of pheromone on the path they followed. This trail will influence later ants so as to reflect the direction of the found sources. In our system, the level of markers shall reflect the relevance of the host regarding the searched category. Each time an agent moves towards a host, the selected marker is decreased as follows:

∆ph i = -α • ph i (2)
This rule models the evaporation of the pheromones for real ants. The paths that are not positively reinforced will have their markers level very low due to this rule. Each time the user labels an image, the selected markers on the pathway taken by the agents to retrieve this image are increased:

∆ph i = +γ • u (3)
With u being 1 if the label is positive, 0 otherwise. This rule models the deposit of pheromone by real ants once they have found food. Hosts containing relevant images will be reinforced by this rule, and thus, their markers will have a high level. Using these rules, the estimation of the marker p h i is dependant on the estimation of û the labels given by the user:

p h i = γ • û α (4)
Thus the higher levels of marker will be obtained for the hosts that gave the greatest number of positive labeling, leading to a routing associated with the session's category.

We ran experiments on the Trecvid'05 keyframes dataset to evaluate the quality of the routing learning. In a network consisting of four hosts plus the computer of the user, we hosted the searched category on the fourth host. The remaining images were equally distributed on the four hosts. We then ran a session consisting of 100 labels given by the user. The Fig. 2 shows the mean probability to visit each of the four hosts at the end of the session. Fig. 2. Results of the routing learning system : final probability score P i of an agent to move to each host at the end of a retrieval session depending on the category being searched. In our experiment, fourth host of the four destinations available contained the relevant category. At the end of a session, its probability P 4 of being visited is the highest.

As one can see, the fourth host has always the highest probability of being visited by the agents, showing that the learning of the relevant destination is succesfull. Moreover, some categories were more difficult to localize with the antalgorithm. This phenomenon can be explained by the mean reinforcement û being lower for difficult categories (where more negative examples are found by the active strategy) than for easy categories (where more positive examples are found by the active strategy).

LONG TERM MERGING OF SESSIONS

The main contribution of this paper is the generalization of our previous system to long term leaning. We propose an architecture to merge all the information registered during past retrieval sessions. After several sessions, we dispose of several category related routings as explained in section 3. In order to re-use the routings in sessions concerning an already learned category, we put on each host i of the network N markers {ph i,j } 1≤j≤N , each of them being related to a specific category. Les us denote Ph i the vector concatenating the ph i,j available on the host i. Fig. 3. Long-term system architecture before learning: in this example, the user's computer is linked to three target hosts a, b and c. There are four planes available to catch the distribution of the categories. The probabilities to visit a host are uniform for each plane. For instance, the probability to visit a is 1/3 on all of the four planes.

The use of the markers {Ph i [j]}, j being fixed, for all host i of the network leads to a routing relevant for the as-Fig. 4. Example of the long-term system after learning: Some of the planes have specialized into one category. For instance, the fourth plane has specialized to the category contained on host a and therefore its probability of being visited using this plane is 0.8. sociated category. Let us denote a plane such a routing (see Fig. 3 and4). The association between a plane and the related category is not known, since the system is not aware of which category is being searched at any session (users do not give a label to the session, for instance). The goal of a long term learning is to find, based on the interaction with the user, if the current category has already searched, and if so to reuse the obtained from previous corresponding sessions. In order to benefit from these previously learned routings, we build a function ψ selecting the marker ph i [j] corresponding to the searched category. The goal of ψ is to select a plane related to the searched category. This plane is used by the agents to move. The function ψ is obtained from a vector Ψ containing 1 for the relevant plane, and 0 otherwise :

ψ(Ph i ) = Ψ ⊤ • Ph i (5)
As we do not have any a priori about the category being currently searched and about the categories associated with the available planes, we build the vector Ψ in interaction with the user. We associate with each plane j a probability W j of being related to the currently searched category. We sample the plane to be used by an agent regarding these propabilities using a multinomial law M(1; W 1 , . . . , W N ). Ψ is obtained as the projection on the selected plane: Ψ = {δ j,m } 1≤j≤N , m being the selected plane [START_REF] Smeulders | Content-based image retrieval at the end of the early years[END_REF] Each time the user labels an image, the W j corresponding to the markers that have been used to retrieve this image evolves regarding the following rule:

∆W j = ε(u -W j ) (7)
Where u = 1 if the label is positive, 0 otherwise. Thus, planes that gave a lot of positive labels will have a higher weight, which means a higher chance of being selected.

Ψ is reset each time an agent is launched. A new vector is sampled thanks to M(1, W 1 , . . . , W N ) with the updated weights. All the dimensions are explored until the weights converge to the relevant plane.

As both the function ψ and the markers level are learned at the same time (using the same reinforcement u given by the user), the dynamics of the markers evolution is set slower than the one of ψ. Consequently, a set of marker is chosen (by convergence of the weights) before the markers are evolved.

EXPERIMENTS

We used the trecvid2005 key-frame dataset to test the influence of our category dependent routing on the retrieval. We put the three categories we tested (namely airplane, explosionfire and maps) on three different hosts, and added about 4000 randomly chosen images from the category entertainment to each host (simulating the various content a real host contains). These were the possible destinations of our mobile agents. We ran one hundred of retrieval sessions which consist of launching of agents and displaying of results until 100 labels where obtained. At any session, the searched category was randomly chosen within the three hosted categories.

As shown on Fig. 5, each host had some of the markers specializing in it. For instance, the third host has been routed by the second and last planes, which means that a function ψ choosing one of these two planes will lead the agents to host 3.

To illustrate the evolution of the Ph i [j] during the learning, Fig. 6 shows the distribution of the markers for all sessions. In this diagram, the marker coordinates (x j , y j ) are a projection of their values on each of the 3 host on a 2D space at each of their updates:

(x j , y j ) = ( 2 i=0 Ph i [j] • cos( i • 2π 3 ), 2 i=0 Ph i [j] • sin( i • 2π 3 )) (8)
The hosts where a marker has the highest probability are sectors delimited by lines. Some of the markers specialized very quickly and remained specialized for a single category during all the sessions (like the first plane), while others did Fig. 5. Relative values of each marker on the three hosts. Each of the six planes has been specialized for one host. For example, the first plane has a higher probability to lead to the first host, while the second plane has a higher probability to lead to the third host. move from one category to another (like the third plane). At least one plane keeps the relevant routing for each category (the first plane for the first category, the fifth plane for the second category, and the last plane for the third category) during all the sessions. In that sense, the system is stable. Fig. 6. Distribution of markers on the three hosts for all session. Some markers (for example markers of the first plane) stay in the zone a one host for almost all sessions, consequently routing the agents to this host.

The markers that were used for each category are shown on Fig. 7. As we can see, all categories used a subset of the markers available, meaning that the markers did specialized for a category. We can clearly see the correlation between this specialization and the Fig. 5. The markers used for a session concerning a category where exactly those leading to the host which contained it. Fig. 7. Percentage of markers that were used for each category. Markers have been at most used for the category towards which destination they routed. For example, for the third category, the most used markers where the second and the sixth, which are leading to the host containing the third category as seen on Fig. 5 Fig. 8 shows the recall obtained for 500 images retrieved for each category. In order to see the benefit of our distributed over classical CBIR system, we ran the same experiments with all the images localized on a single system. In this centralized setup, there is no network, and thus no routing learning. Our system performs slightly better, although it is not very pronounced. As discussed in the next figure, the maximum performance is obtained for all categories with both systems, which can explain the small gain obtained after the long-term optimization. The main advantage our distributed system over the classical CBIR setup is the time to perform the retrieval, which was about four time less. Fig. 8. Recall for each category for 500 images retrieved, compared to a classical CBIR system. The distributed system performs a little better on more difficult categories as airplane or explosion-fire.

The improvement of the mean average precision (MAP)

due to long-term learning is shown on Fig. 9, Fig. 10 and Fig. 11. As one can see, the long-term optimization leads to an improvement of the MAP between 5% and 10%. For a difficult category like airplane, the gain is about 5%, whereas for an easy category like explosion-fire, the gain is about 8%. The main consequence is that an equivalent MAP can be obtained with fewer labels after the long-term optimization. For instance, to obtain the same MAP as with 100 labels before long-term optimization, about 40 were needed for the maps category, about 50 for the explosion category and about 70 for the airplane category after the long-term optimization. 11. MAP for the category explosion-fire before and after learning. With the long-term optimization, the gain is about 8% and the speed-up is about 60 labels.

CONCLUSION

In this paper we have presented a CBIR system based on mobile agent technology with an ant-like behavior. Agents crawl the network in search of images using a relevance function learned in interaction with the user. The locations from where the images are retrieved depend on a set of markers. The higher the level of a marker, the higher the probability of the agents to move to this host. Markers are reinforced unsing the labels given by the user. Host containing positively labeled images have their markers increased, whereas host containing more negatively labeled images have their markers level very low. We use several markers on each host, each being associated to a category. Thus, agents using a specific markers set will be routed towards the host containing the associated images. The association between these markers and categories is learned in interaction with the user, resulting into a category dependent routing. Our system carries out a smart cooperation between this routing and the active learning of the similarity function used for the retrieval, leading to an improvement of the recall.

While markers can be naturally shared between users, our system builds an user oriented semantic map of the network that can be used efficiently to improve the retrieval. As shown on the experiments, the association learned between a set of markers and a category improves the retrieval. While the recall gain is little compared to a classical CBIR system, the speed up in learning obtained through the long-term is high. For instance, for the category maps, only 40 labels were needed to obtained about the same MAP as with 100 labels without the long-term optimization.

Fig. 1 .

 1 Fig.1. Functional description of our system showing the user in interaction with the relevance feedback loop (launching of agents, retrieval, display and labeling).

Fig. 9 .

 9 Fig.9. MAP for the category airplane before and after learning. With the long-term optimization, the gain is about 5% and the speed-up is about 30 labels.

Fig. 10 .

 10 Fig.10. MAP for the category maps before and after learning. With the long-term optimization, the gain is 5% and the speed-up is about 50 labels.

  Fig.11. MAP for the category explosion-fire before and after learning. With the long-term optimization, the gain is about 8% and the speed-up is about 60 labels.