Anton Yana

Aymeric Vincent Guillaume

Tatiana Nastya

Lou- Cas Guilhem

Sesh Andrei Pascal

les déformations des objets dans le temps et en extraire des changements intéressants dans le style visuel au fil du temps.

.

Résumé

L'analyse temporelle quantitative à grande échelle de données textuelles a eu un grand impact sur la compréhension des tendances sociales. Une analyse similaire à grande échelle des collections d'images temporelles permettrait de nouvelles applications en médecine, en science ou en histoire de l'art. Cependant, l'analyse temporelle des données visuelles est une tâche notoirement difficile. Le principal défi est que les images n'ont pas un vocabulaire donné d'éléments visuels, par analogie avec les mots du texte, qui pourraient être utilisés pour une telle analyse. De plus, les objets représentés dans les images varient considérablement en apparence en raison du point de vue de l'appareil photographique, de l'éclairage ou des variations intra-classe. L'objectif de cette thèse est de développer des outils pour analyser les collections d'images temporelles afin d'identifier et de mettre en évidence les tendances visuelles à travers le temps.

Cette thèse propose une approche pour l'analyse de données visuelles non appariées annotées avec le temps en générant à quoi auraient ressemblé les images si elles avaient été d'époques différentes. Pour isoler et transférer les variations d'apparence dépendantes du temps, nous introduisons un nouveau module bilinéaire de séparation de facteurs qui peut être entraîné. Nous analysons sa relation avec les représentations factorisées classiques et les auto-encodeurs basés sur la concaténation. Nous montrons que ce nouveau module présente des avantages par rapport à un module standard de concaténation lorsqu'il est utilisé dans une architecture de réseau de neurones convolutionnel encodeur-décodeur à goulot. Nous montrons également qu'il peut être inséré dans une architecture récente de traduction d'images à adversaire, permettant la transformation d'images à différentes périodes de temps cibles en utilisant un seul réseau.

Nous appliquons notre modèle à une collection de plus de 13 000 voitures fabriquées entre 1920 et 2000 et à un ensemble de portraits d'annuaires d'écoles secondaires entre 1930 et 2009. Cela nous permet, pour une nouvelle image d'entrée donnée, de générer une "vidéo historique en continu" révélant les changements dans le temps en variant simplement l'année cible. Nous montrons qu'en analysant ces vidéos générées, nous pouvons identifier This thesis proposes an approach for analyzing unpaired visual data annotated with time stamps by generating how images would have looked like if they were from different times. To isolate and transfer time dependent appearance variations, we introduce a new trainable bilinear factor separation module. We analyze its relation to classical factored representations and concatenation-based auto-encoders. We demonstrate this new module has clear advantages compared to standard concatenation when used in a bottleneck encoder-decoder convolutional neural network architecture. We also show that it can be inserted in a recent adversarial image translation architecture, enabling the image transformation to multiple different target time periods using a single network.

We apply our model to a challenging collection of more than 13,000 cars manufactured between 1920 and 2000 and a dataset of high school yearbook portraits from 1930 to 2009. This allows us, for a given new input image, to generate a "history-lapse video" revealing changes over time by simply varying the target year. We show that by analyzing the generated history-lapse videos we can identify object deformations across time, extracting interesting changes in visual style over decades.

Motivation and objectives

Data-driven analysis through time can have -and already has had -a significant impact on understanding of our world. Analysis of vocabulary used in books through centuries [START_REF] Michel | Quantitative analysis of culture using millions of digitized books[END_REF] shows that large collections of books can be used to perform quantitative analysis and shed light on the evolution of how people write about different topics. Speech analysis has been used to help understand the evolution of language acquisition [START_REF] Roy | The human speechome project. Symbol Grounding and Beyond[END_REF]. Similarly, leveraging large-scale time stamped visual data can enable new applications. Databases of medical imagery have been used to assess disease progression in radiology [START_REF] Marcus | Open access series of imaging studies (oasis): Longitudinal mri data in nondemented and demented older adults[END_REF][START_REF] Tang | Canadian association of radiologists white paper on artificial intelligence in radiology[END_REF]. Art museums have collections of sculptures, paintings, or objects associated with the date of their creation. Recent efforts are making such art collections publicly available [START_REF] Dijkshoorn | The rijksmuseum collection as linked data[END_REF] The Watermark Project; The Visual Arts Data Service]. While art historians are needed to analyze these trends, there is also interest to use machine learning tools to help this analysis [START_REF] Crowley | Of gods and goats: Weakly supervised learning of figurative art[END_REF][START_REF] Gonthier | Weakly supervised object detection in artworks[END_REF][START_REF] Shen | Discovering visual patterns in art collections with spatially-consistent feature learning[END_REF]. Examples of trend analysis include identifying and highliting differences over time. Similarly, in architecture, large scale image collections can be used, with proper annotation, to analyze the characterstic elements of architecture design from place to place [START_REF] Doersch | Mid-level visual element discovery as discriminative mode seeking[END_REF] or through time [START_REF] Lee | Linking past to present: Discovering style in two centuries of architecture[END_REF].

While all these problems touch different domains, they share common characteristics. They concern large collections of images that are time stamped. The objective of this thesis is to develop tools to analyze these different types of time stamped image collections, in order to identify and highlight visual trends over time. Our goal is to take as input (in green) an image of a face or an object such as a car, and generate what it would have looked like in another time-period (in blue). Each row shows temporal translation for a different input portrait or car picture. We further show that analyzing changes between the generated images can reveal structural deformations in object shape and appearance over time.

APPROACH

Approach

In this thesis we propose to perform temporal analysis of visual data by temporal image translation. First, we learn from the input time-stamped image collection a temporal translation model that takes an image as input and translates it to a different time period, while preserving its time-independent characteristics, as illustrated in figure 1.1. Second, we perform temporal analysis by comparing the resulting generated imagery. We describe these two steps next.

Time translation. The time translation is an operation that takes an image and a target time period, and outputs a modified image that has the characteristics of this target time period, while keeping the visual characteristics of the input image that do not depend on time. We illustrate this in figure 1.1 with two examples. We "translate" cars and change their appearance to make them look like they have been built at another period.

In these examples, wheels, headlights, and windshields change their appearance over time. The model has no prior notion of these elements, yet it learns what are the characteristic elements and how they evolve over time from the data. At the same time, the type of car, viewpoint, and general colour don't change, as these visual characteristics are not dependent on the time when the car was made. We also apply the same method to historical yearbook portraits of faces, making them appear as coming from another decade while preserving the face identity. In this thesis we investigate and evaluate different approaches to perform such temporal image translation.

Visual discovery. We aim to use image translation for visual discovery.

Once an image is translated, time related variations for this specific image instance can be identified by simple pixel comparisons. This would be difficult otherwise and would require finding correspondence across different object instances and viewpoints. We can use the images we produce to compare the same car or face at different times (figure 1.2) and highlight the different trends over time. We can also identify typical and atypical images of different time periods (figure 1.3).

Challenges

We proposed to approach temporal analysis of visual data using temporal image translation, that is answering question "what would this picture be, if this car was made 20 years earlier?". This question is, however, a very difficult one to answer and poses the following key scientific challenges. We show how translated images allow us to examine differences between cars manufactured in different time periods. We show the original cars from the 1990s (top row), their translation into the 1940s (middle row) and the absolute differences (bottom row) between the original image and the translated image. The absolute differences are shown as heatmaps where bright yellow color indicates a high difference. In this example the most salient differences are: different appearance of wheels, the presence of a runner board in the 1940s and the bulky distinctive hood in the 1940s.

Portraits that are characteristic of the 2000s that are very different from 1970s portraits. Long, straight or curly blonde hair are common in 2000s yearbooks, but are not featured much in 1970s yearbooks.

Portraits that are characteristic of the 1970s that are very different from 2000s portraits. Short hair is more common in the 1970s than in the 2000s.

Figure (1.3):

Visual mining. Analyzing our translated images, we identify images that are characteristic of a specific time period, but would not pass in another period. This way, we identify the most striking differences between two decades.

Unpaired data. The first difficulty is that while we have pictures of cars made in different years, we do not have any example of a car transforming to appear to be made in a different year. We use the term unpaired to describe our data, meaning that we only observe single instances of objects at a specific time. While we know the time associated to a particular object, we do not have access to a picture of the same object translated to a different time.

Most datasets are naturally unpaired. For example, there is no definitive way to know how a 20 years old woman in 1930 would have dressed and changed her hair, had she been transported to the year 2018 for a yearbook picture. Similarly, while we have pictures of cars manufactured throughout the 20th century, it is not possible to know how a particular car would have looked like had it been made in different time periods. By contrast, data is called paired when it contains the same object with different values for the varying attribute. In some situations, it is possible to obtain paired data. For example, it is possible to take multiple pictures of the same person as they age. Such a dataset could be used in order to try to make people appear older or younger. We illustrate the difference between paired and unpaired data in figure 1.4. Working with unpaired data makes our task harder. If we knew how a Clio-equivalent would look like in 1950, we could train our model to reproduce this known appearance. However, we aim to tackle the problem with unpaired data, as it is more common in practice and often easier to obtain. In addition, evaluation in the unpaired setting is a major challenge. Because we have no direct way to compare each image generated by our model to a ground truth reference, we have to take special care when designing metrics to evaluate our approach. Appearance variations. Another major challenge is the high appearance variations of the objects in the input datasets. For example, cars of the 1990s include very diverse types of cars of cars, ranging from pick-up tracks to sports cars (figure 1.5). This increases the amount of appearance variation the model has to learn in order to translate between different time periods. This variability is intrinsic to the objects we consider.

Other factors of appearance variation include camera viewpoint and illumination, as illustrated in figures 1.6 and 1.7, respectively. Our objective is to isolate the time-dependent variation (e.g. the type of car tyres) from other appearance variation such as viewpoint, illumination or car type that are largely independent of time. The variation in viewpoint means that depicted objects are not aligned, making it difficult to directly compare images at the pixel-level, and to analyze changes in the shape of objects. In addition, analyzing changes in colors and textures is challenging due to variation in illumination. While it is possible to explicitly control for viewpoint and illumination when collecting data, we wish to consider realworld datasets collected in uncontrolled conditions. Generating images. Finally, another challenge is related to the complexity of the output. This is in contrast to image classification that outputs only a score whether an image belongs to a specific class. In other words, our model not only needs to understand that certain parts of the input image 1940s 1970s

Classification. Deciding whether a particular picture comes from 1940s or 1970s is solving the classification problem. The output of this problem is a binary value for each image.

1940s 1970s

Image translation. In temporal translation we wish to translate an input image (green) into a different time period (blue). In contrast to classification the output is another image. are characteristic for a specific time period in order to classify the image correctly, but needs to generate the appearance of those parts the way they look like in another time period, as illustrated in figure 1.8. It needs to understand what elements are typical of the time periods, but also how to change them to make them relate to other time periods (figure 1.8). In detail, in image classification the target output is a label, which is often a scalar.

CHAPTER 1. INTRODUCTION

In image translation the target output is a vector of high-dimension (R h×w ), where (h, w) is the dimension of the target output image.

In image translation, producing images that look realistic is also very challenging. The visual quality of the translated images is important in order to identify the key changes in appearance variation across time. It is hard to produce images that look realistic, sharp and capture the characteristic appearance of the target time period. Producing images as output presents also the additional challenge of performing automatic quantitative evaluation.

Contributions

This section lists the main contributions of this thesis.

The first contribution, described in chapter 4, is a new trainable bilinear factorization module that encourages factor separation by architecture design. We show how this module relates to other existing approaches including principal component analysis. We show its advantages compared to standard concatenation-based factor representation when used in a bottleneck auto-encoder architecture. We then show that our bilinear module can be plugged into a modern unpaired image translation architecture.

Our second contribution is our introduction of models for image translation that features our bilinear module. We investigate an encoder-decoder model with factor separation (section 5.1) and an adversarial traslation model (section 5.2). We demonstrate that learned factors of variation can be used for several visual discovery tasks such as discovering typical or non-typical examples for a particular characteristic attribute (such as a time-stamp).

Our third contribution, described in chapter 6, is the application of image translation models to the task of visual discovery in temporal image collections. We first introduce quantitative metrics that allow us to compare different models for the task of image translation. Then, we apply our image translation models to two temporal datasets: a dataset of portrait pictures acquired from yearbooks in the United States from 1930 to 2009 [START_REF] Ginosar | A century of portraits: A visual historical record of american high school yearbooks[END_REF], and a dataset of cars manufactured between 1920 and 1999 [START_REF] Lee | Style-aware mid-level representation for discovering visual connections in space and time[END_REF]. We compare different models including several baselines using the proposed quantitative metrics. We also show the qualitative results obtained by different methods. We provide different ways to visualize the obtained translation results and show how they can be applied for the task of temporal visual discovery.

Thesis outline

This thesis is organized as follows.

In chapter 2, we overview the previous work this thesis is built upon. In particular, we detail the relevant work in image factorization, image translation and visual discovery. In chapter 3, we overview the general approach common to the different methods we use for image translation, including the necessary notation. In chapter 4, we introduce the bilinear operation as a module for image translation. We explain its relation to other methods, especially the principal component analysis. In chapter 5, we explore image translation models. We investigate different architectures and loss functions. We discuss the strengths and weaknesses of different models.

In chapter 6, we analyze the results of our translation methods, in both a quantitative and qualitative way, and compare them to various baselines. We show results of temporal visual discovery including producing history-lapse videos.

Chapter 2

Related Work

We build upon work on linear and multi-linear models, image translation and temporal analysis. In this chapter we present prior work on these three topics.

(Multi-)linear image models

Linear and multi-linear models are useful for many computer vision tasks, such as classification, detection or recognition. They allow to represent images as combinations of factors that can be manipulated, which is useful for our goals in this thesis. In this section, we first discuss linear models, starting from principal component analysis, as it is the most direct way to factorize images. We then discuss how mixtures of linear models can be used when a single linear model is not enough to represent the data. Finally we discuss multilinear models, and in particular bilinear models, as they can model interactions that are not possible to obtain with linear models. Linear models. In this section, we consider N data points x 1 , . . . , x N ∈ R d . To simplify notation, we consider that these vectors are centered:

1≤i≤N x i = 0. For example, the set {x i } can be a set of images with subtracted average image. We discuss linear and multi-linear representations of these data points and their usage.

Principal Component Analysis (PCA [START_REF] Pearson | On lines and planes of closest fit to systems of points in space[END_REF]) is the change of basis to {e 1 , . . . , e d }, an orthonormal basis that allows to project the data points on smaller subspaces with minimal reconstruction error. More specifically, the PCA minimizes quantities arg min for each 1 ≤ S ≤ d. Thus, PCA allows us to approximate the data points

x i by xi = 1≤k≤S λ ik e k , ( 2.2)
where λ ik = x i , e k . PCA allows us to reduce the dimensionality of the data points x i from d to S in an optimal way, i.e. in a way that minimizes the sum of the reconstruction losses over all data points. The vectors e 1 , . . . , e N are called principal components. They can be obtained by performing Singular Value Decomposition (SVD) of X, as they are the right singular vectors of X. They are also the eigenvectors of the matrix X T X where X ∈ R N ×d is the matrix with rows x 1 , . . . , x N .

PCA has been applied to collections of images in [START_REF] Kirby | Application of the karhunen-loeve procedure for the characterization of human faces[END_REF], and the resulting principal components are called eigenpictures. Examples of eigenpictures are shown figure 2.1. [START_REF] Kirby | Application of the karhunen-loeve procedure for the characterization of human faces[END_REF]] offer a measurement of the error made by projecting these face images into this new space, and show that a linear combination of 50 eigenpictures were enough to have a good reconstruction of any face.

In [START_REF] Matthew | Face recognition using eigenfaces[END_REF], this representation allows the authors to perform classification, detection and recognition tasks. To determine if a query image is an image of a face (classification task), the distance from the image to the face space (i.e. the distance between the image and its projection) is compared to a pre-defined threshold. To recognize a face, the authors compute the Euclidean distance between the projection of the query face image and the projection of the faces from the database. The query image is then assigned to the closest face of the dataset, granted that this minimum distance is below a defined threshold.

Principal component analysis has been successfully used for simple face datasets, but this method is not adequate for more complex datasets, for which a low dimensional orthogonal projection would result in too much image degradation. In this thesis we show relationship of our work and principal component analysis, but we apply PCA on an intermediate representation (composed of features from a convolutional neural network) instead of applying it directly on image pixels.

Mixture of linear models. Instead of using a single linear model, be it PCA or factor analysis, [START_REF] Hinton | Modeling the manifolds of images of handwritten digits[END_REF] has used a mixture of linear models, as illustrated in figure 2.2, to represent images and perform classification. Each different class, e.g. each different digit in a dataset of digit images, uses its own linear model. Each image x i is thus approximated by

xi = 1≤z≤m q i z 1≤k≤S λ ik z e k z , (2.3)
where m is the number of classes, q i z is the coefficient soft-assigning image i to class z, e 1 z , . . . , e S z is the basis obtained by PCA for class z, and the λs are the associated coefficients. The coefficients q i z weigh the different z models for the image of index i, and sum to 1: z q i z = 1 for each i.

This method is used for classification of digits. The mixture model is trained with an expectation-maximization algorithm [START_REF] Arthur P Dempster | Maximum likelihood from incomplete data via the em algorithm[END_REF], which alternates between two phases. The expectation phase consists in assigning each training example to their respective linear models. The maximization phase computes the parameters of each linear model.This model is very close to the Gaussian Mixture Model, although it does not whiten the projections. It has also been used for face detection [START_REF] Yang | Face detection using a mixture of factor analyzers[END_REF].

The mixture of linear models is useful for image classification, but does not fully model the possible interactions between factors of variations in images. We next discuss multilinear factorization, which we use in this thesis, in order to achieve this.

Multilinear factorization.

In order to factor out style and content for font and face analysis, [Tenenbaum and Freeman, 2000] has explored multiplicative interactions in a bilinear model. They introduce a "symmetric" bilinear model (figure 2.3), which consists of describing both a person and a pose by a vector. With this model, the image x ij corresponding to person i and pose j can be approximated by the vector xij with coordinates:

xij = 1≤k≤S 1≤l≤T a ik b jl e kl , , (2.4)
where S and T are the size of the person and pose coefficients, {e kl } 1≤k≤S,1≤l≤T is a set of images, and the image xij is obtained by combination of these image given by the outer product of the person and pose coefficients a i and b j . The set of images and coefficients are learned by performing Singular Value Decomposition repeatedly, fixing alternately the set of coefficient vectors {a i } i and {b j } j .

[Vasilescu and [START_REF] Alex | Multilinear analysis of image ensembles: Tensorfaces[END_REF] extend this idea by considering an arbitrary number of factors of variations. To illustrate this, expression, pose, illumination and person identity are considered. Instead of having eigenfaces, the authors obtain tensorfaces. These tensors can be flattened into vectors, forming images similar to eigenfaces for any considered degree of variation. Images can be reconstructed (bottom right) by making the outer product of these two tensors, and using the resulting matrix as weights for the linear combination of the basis images.

Bilinear models for Convolutional Neural Networks. These models have rarely been used for more complex images than aligned faces, because the complex variations that occur in natural images are hard to represent using linear models in pixel space. In contrast, we formulate bilinear factor translation in a end-to-end trainable encoder-decoder convolutional neural network (CNN) architecture. Bilinear layers have recently been used in CNNs, for example in the context of fine-grained recognition [START_REF] Lin | Bilinear cnn models for fine-grained visual recognition[END_REF] and visual question answering [START_REF] Fukui | Multimodal compact bilinear pooling for visual question answering and visual grounding[END_REF], but to the best of our knowledge, they were not used for visual discovery by image translation.

Deep Auto-Encoders and Image Translation

In this thesis, we translate images, which means that we modify an image so that a specific aspect of it changes, without changing its other characteristics. We use auto-encoders, i.e. neural networks that are trained to learn a coding of an image (with an encoder) and restore images from this coding (with a decoder). In this section, we review auto-encoder models and methods to translate images that are most relevant to our work. We first discuss auto-encoders, and how they can be used to modify images. We then focus on the setup where datasets do not have corresponding images with different attributes. We then discuss ways to improve the visual quality of images, especially perceptual losses and adversarial losses. Finally, we discuss cycleconsistent adversarial networks.

Auto-encoding images. Neural network based models such as Boltzmann machines have been long used for image generation [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF][START_REF] Le Roux | Learning a generative model of images by factoring appearance and shape[END_REF][START_REF] Osindero | Modeling image patches with a directed hierarchy of Markov random fields[END_REF][START_REF] Salakhutdinov | Deep Boltzmann machines[END_REF]. In [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF], a two-layer auto-encoder is used to denoise images. The authors note that different losses can be minimized to learn the parameters of the auto-encoder and generate an image x from an original data image x, in particular the squared Euclidean loss

l(x, x) = x -x 2 2 .
(2.5)

In this thesis we also use, among others, a squared Euclidean loss in order to minimize the difference between a data point and its reconstruction. More recently, [START_REF] Kingma | Auto-encoding variational Bayes[END_REF] has used a variational auto-encoder to learn probabilistic models from large datasets, while [START_REF] Yan | Attribute2Image: Conditional image generation from visual attributes[END_REF] have used a variational auto-encoder to generate images with different attributes.

Image translation with deep auto-encoders. Several works have used auto-encoders in order to modify an input image. In [START_REF] Tatarchenko | Multi-view 3d models from single images with a convolutional network[END_REF], an auto-encoder is trained to modify the viewpoint of an object. They take as input a rendered image of a 3D model of an object, and a target angle for the translated image. The output is the image of the object from a different viewpoint (see figure 2.4 for examples), along with an estimation of the depth. Because the method uses rendered images from a 3D model, it is possible to compare directly the output of the network to an image with the correct attribute. We say that data is paired when it is possible to obtain a ground truth for what a translated image should look like. Because data is paired, the authors train their networks by minimizing the distance between a rotated version of a car obtained by their network and one obtained by a render of the 3D model (ground truth). This distance is a combination of the squared Euclidean loss on pixels and a L1 distance on depth. By contrast, we use unpaired data in this thesis. As we cannot have a reference for what the image translated by our model should be, the method of [START_REF] Tatarchenko | Multi-view 3d models from single images with a convolutional network[END_REF] is not possible to use in our case.

[ [START_REF] Kulkarni | Deep convolutional inverse graphics network[END_REF] propose another approach to build a factorized representation to generate images. They force specific parts of their code variable to explain the variation between paired images that only differ by the respective attribute. Again, this method is only possible to use when paired data is available.

Another approach, proposed by [START_REF] Reed | Learning to disentangle factors of variation with manifold interaction[END_REF], learns a Restricted Boltzmann Machine that models multiplicative interactions between the different factors of variations. This model is also applied on paired data, but the use of multiplicative interactions is also similar to what we use in our convolutional neural networks.

To generate complex photo-realistic without losing low-level details, it is possible to modify the network architecture by adding shortcut connections, as demonstrated for very different sets of paired images in [START_REF] Isola | Image-toimage translation with conditional adversarial networks[END_REF].

Deep auto-encoders for unpaired data. Several works have translated images with unpaired datasets. In [START_REF] Cheung | Discovering hidden factors of variation in deep networks[END_REF], the lack of paired data is solved by using a combination of multiple losses. In addition to a reconstruction loss that is used in paired data setups, they use a covariance loss between two parts of the extracted code, which enforces separation between the different parts of the code. More specifically, their model is an encoder f and a decoder g with (z, ŷ) = f (x) and x = g(z, ŷ) , where x is an image associated with the time labeled y in the dataset, ŷ is the observed attribute computed by the encoder f , z is called a latent variable and x is the reconstruction of image x. The model is trained with a combination of a reconstruction loss between y and ŷ, a cross-entropy loss between y and ŷ, and a cross-covariance loss between z and ŷ:

L COV (ŷ 1,...,N ), (z 1,...,N ) = 1 2N ijn (ŷ n i -ȳi )(z n j -zj ) 2 , (2.6)
where (ŷ 1,...,N ) and (z 1,...,N ) are N associated results of the encoder, and ȳ and z their respective means. This cross-covariance loss is computed on mini-batches of size N . The goal of minimizing the the cross-covariance loss is to disentangle the attribute coded by y from the other factors of variation of the image, which are then encoded by z. After training, the model is able to change ŷ to translate images, without changing the other factors of variation present in the variable z. Examples shown in figure 2.5 show how this model can translate faces to different emotions. In section 5.1, we build on this work by proposing an architecture that uses a separation loss between the attribute and the non-observed factors of variation. We replace the cross-covariance loss with a stronger separation loss in order to increase the variation induced by changing the attribute. , the visual quality of the output imagery can be enhanced using adversarial costs [START_REF] Goodfellow | Generative adversarial nets[END_REF]] that encourage the network to generate images that cannot be distinguished from natural images.

Adversarial networks. As shown by [

While our work is not focused on improving the realism of the translated images, we make use of adversarial networks to improve their visual quality.

Perceptual loss. Auto-encoders are often trained with a reconstruction loss on pixels. In [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF], the authors introduce a perceptual loss to compare feature representations of images. In particular, they use a squared Euclidean loss on a feature reconstruction of both the original and the translated image, as it captures better the perceptual difference between images. Similarly, in our work we use feature construction losses.

Unpaired Translation with Cycle-Consistent Adversarial Networks.

In [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF], the authors translate images from two domains, C 1 and C 2 , with unpaired images. They jointly train two networks, M 1→2 and M 2→1 , translating images from the two domains, such that for an image

I ∈ C 1 , M 2→1 (M 1→2 (I)) is close to I.
They also use a conditional adversarial loss with two discriminators, D 1 and D 2 . D 1 is trained to distinguish between images from the domain 1 C 1 , and images translated from domain 2 to domain 1 M 2→1 (C 2 ). The networks M 1→2 and M 2→1 use residual building blocks from [START_REF] He | Deep residual learning for image recognition[END_REF]. We use a similar network in our work. We extend this work by tackling a dataset with more than two domains.

Visual mining and temporal analysis

Visual mining has been used in computer vision as a tool to accomplish various goals. We call "visual mining" the task of extracting images, or parts of images, from a collection of images that exhibit interesting properties or can be useful for another computer vision task. In this section, we first discuss mining whole images in a dataset of images. Then, we discuss mining discriminative image elements, i.e. parts of images that can be used to differentiate between objects or other attributes. Finally, we discuss how mining discriminative image elements have been used for temporal analysis of visual data.

Mining images.

Extracting relevant images from large image collections has been tackled in a number of related work. In [START_REF] Quack | World-scale mining of objects and events from community photo collections[END_REF], the authors have used external information such as geographical data and Wikipedia articles in order to mine pictures of interesting landmarks from very large collections of images. [Martin-Brualla et al., 2015] is related to both mining and temporal analysis. In this work, the authors reconstruct time-lapses of various landmarks from a large dataset of pictures across the world. To achieve this, they mine multiple pictures of the landmarks, as well as a reference viewpoint for each of them, before warping the images to form a time-lapse of each landmark. Similar work is done in [START_REF] Matzen | Scene chronology[END_REF], in which the authors build a "scene chronology" by clustering 3D points of different scenes obtained from a large set of images.

In [START_REF] Sivic | Video data mining using configurations of viewpoint invariant regions[END_REF], objects are mined across different shots of a movie by clustering feature representations of images. These works concern instance-level matching, i.e. finding the same object or scene across different images captured at different times.

In our work, we also mine images that are characteristic of their time in section 6.2. Beyond this, we focus on finding visual patterns that generalize beyond instance-level matching. This has generally been done by mining discriminative image elements.

Mining discriminative image elements. Patches, windows, parts of images are said "discriminative" if they help distinguishing between attributes, such as the object category (dogs or cats), the location (Barcelona or Casablanca) or the time (which year).

For example, [START_REF] Grauman | Unsupervised learning of categories from sets of partially matching image features[END_REF]] extracts typical features that characterize object categories from unsupervised data. To do this, local features (such as SIFT [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF]) are computed in each image and a similarity score is computed for each pair. Then, based on this score, the features are clustered and represent learned image categories. Still for unsupervised data, [START_REF] Lee | Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations[END_REF]] extracts mid-level visual features from convolutional Deep Belief Networks. Discovering mid-level features from unsupervised data has generally been evaluated by using these features to categorize new images [START_REF] Singh | Unsupervised discovery of mid-level discriminative patches[END_REF][START_REF] Sivic | Discovering objects and their location in images[END_REF][START_REF] Karlinsky | Unsupervised feature optimization (ufo): Simultaneous selection of multiple features with their detection parameters[END_REF]. This discovery of mid-level patches is done without any supervision, so these models are limited to discovering only visual patterns that are both very common and highly visually consistent. To address this issue, another line of work (including ours) uses image-level supervision.

Image datasets can have annotation for each image. This is called weak supervision as it provides annotations only on. This enabled [START_REF] Doersch | What makes paris look like paris[END_REF] to obtain geographically representative elements from different places. This work features a dataset of street level images from cities such as Paris, San Francisco or Prague. For each city, they extract patches from these images that are both common in that particular city and do not appear in other ones. To achieve this, they compute an image representation for patches based on histogram of gradients [Dalal and [START_REF] Triggs | Histogram of Oriented Gradients for Human Detection[END_REF] (HOGs) and a scaled-down version of the patch. Then, using a score based on cross- correlation, they look for the nearest neighbor patches and select patches that are only close to those of the same city. The authors refine this selection several times using discriminative learning to only select patches that would be best to train a linear Support Vector Machine (SVM) detector. Figure 2.6 shows examples of the resulting patches. For example, blue street signs are very frequent in Paris, but will rarely be found in other cities. This approach treats the attribute as a discrete variable (e.g. Paris or London, no in-between). Our work differs as we present methods that treat time as a continuous variable.

Temporal analysis. Another approach for retrieving discriminative parts of objects is examining which parts of images are used by CNNs. In [START_REF] Matzen | Bubblenet: Foveated imaging for visual discovery[END_REF], the authors train a CNN classifier on "foveal images", for which all but a small patch of the image has been blurred, and "bubble images", for which all but a small patch of the image has been grayed out. This network is then used to find the bubble images whose classification prediction matches best the true label of the images. The bubble images are then clustered and, again, used to train SVM classifiers. The authors present an example in which they learn an bubble image classifier to predict the month of the year in which pictures of people in NYC have been taken. They show that it is harder to assign the correct time to two pictures when they have been taken in close months (e.g. July and August) than when they have been taken a long time apart (e.g. July and January). We make similar observations when trying to estimate the time period of pictures in this thesis. [START_REF] Matzen | Bubblenet: Foveated imaging for visual discovery[END_REF]] also analyze temporal trends with their method. Noting that the months of March and April are more distinct than other pairs of consecutive months, they looked at the cluster of patches and saw that the most discriminative one was images of sunglasses. They then correlated the presence of sunglasses with clear weather (see figure 2.7). This work is different from us as we seek to translate full images to exhibit differences instead of extracting patches. Also closely related to our work, [START_REF] Lee | Style-aware mid-level representation for discovering visual connections in space and time[END_REF] aim to discover elements whose appearance changes with time and location. They work on multiple datasets including a dataset of cars that we use. Cars made between 1920 and 1999 are annotated with the time they have been manufactured. We also use this dataset in this thesis. [START_REF] Lee | Style-aware mid-level representation for discovering visual connections in space and time[END_REF] first use HOGs to describe patches, and mine discriminative patches by taking those whose nearest neighours across the dataset are only present in images of similar attribute (such as the year). In a second phase, they establish correspondences between patches across the attributes. They iteratively train detectors of similar visual elements, and gradually enlarge the positive training set of the detector by incorporating matching patches of an increasing range of years. By doing this, they fully leverage the continuous character of their labels.

We differ from these works as we analyze temporal effects by looking at generated images, having both a global and local representation of the changes that occur because of time.

Chapter 3

Approach Overview and Notation

In this chapter, we present an overview of our model for temporal image translation, introduce the key notation and provide a detailed roadmap for the rest of the thesis.

Objectives and approach overview

We seek to discover patterns in changes of appearance related to time in a large collection of images. We address this problem by learning a model that translates images in time to answer question: "What would the input image look like at a different time?". More precisely, we develop a factored model that learns to separate time-specific appearance variation from the rest of the content of the image. This is illustrated in figure 3.1. The outcome of this approach is that, for a given input image, we can produce a "history-lapse video" revealing changes over time by simply varying the target time variable.

More formally, our main hypothesis is that an image is completely determined by a time variable y and a latent content variable z y ∈ R C . For example, the appearance of a particular car from the 1930s is a combination of two sets of factors. The first set of factors, determined by a time vector y, is related to the period the car was built in, due to cars exhibiting time-characteristic features, such as running boards or round mud guards for cars from the 1930s. The second set of factors, encoded in a content vector z y ∈ R C , is independent of the time and includes, for example, viewpoint and body type (e.g. sedan vs. pick-up truck). Our factorization model will be trained in an end-to-end manner from a dataset of images with time stamps. We assume the dataset is unpaired, i.e. each object is observed at only one time. For example, we observe a particular car made at one specific time period and we do not know how that particular car would have looked like if it had been made at a different time period. The objective of the training is to isolate the time dependent appearance variation from such unpaired input training data. This is achieved by optimizing a cost function that includes a reconstruction term, possibly together with optional translation and separation terms. The reconstruction term ensures that the model reconstructs well the input training images at their given time period. The translation term ensures that when the input image is translated to different time periods, the output images exhibit characteristics of the training images from those target periods. The separation terms encourages the independence of factors related to time and factors encoding the time-agnostic content of the image.

Notation

As illustrated in figure 3.1, we assume each input image is associated with a time stamp, represented by a feature vector y ∈ R T . We denote by x ∈ R K a K-dimensional feature encoding of image I obtained with an encoder: x = A(I). Our factorization module (see figure 3.1) computes a mapping of these features into a latent content factor z y = f (x, y). Then the inverse mapping x y = g(z y , y) decodes the content factor z y jointly with the time dependent factor y into a new image feature x y that is reconstructed into the output image I y : I y = A + (x y ). The specific form of f and g is discussed in chapters 4 and 5.

Detailed Roadmap

In chapter 4, we introduce a trainable bilinear factorization module which allows us to represent multiplicative interactions between time and content. used typically in neural network architectures. We also provide details of our efficient implementation of bilinear factorization, which allows us to apply our bilinear module on large-scale datasets.

In chapter 5, we discuss different image translation models. We discuss two different architectures based on an bottleneck auto-encoder and a recent adversarial image translation architecture. We show how our bilinear factorization module can be inserted in both architectures and discuss in details the different loss functions used.

In chapter 6, we discuss the quantitative and qualitative results obtained with our temporal translation approach. We first introduce the two datasets we experimented on, and discuss the metrics that we use to quantitatively evaluate our approach and to compare it other methods. Finally, we demonstrate how to use our temporal image translation model for visual discovery in unpaired image collections annotated with time stamps. We show how to use our model to generate history-lapse videos that translate the input image into different time periods while preserving the identity of the depicted object. We analyze the generated images sequences to extract interesting changes in style over time.

Chapter 4

Bilinear Factorization Module

In this chapter we introduce a trainable bilinear factor separation module that can be plugged into different image translation architectures. In section 4.1 we present and detail this bilinear module. In section 4.2 we describe how this module relates to other factor separation approaches such as Principal Component Analysis, style and content factorization as well as simple feature vector concatenation. Finally, as speed is important for training from large datasets, in section 4.3 we explain our efficient implementation of the bilinear module. Chapter 5 will then show how to employ our bilinear module in different image translation architectures.

Module description

In this section we describe our new bilinear factored representation module, which is visualized in figure 4.1a, and that can be used as the core factorization module in the visual discovery architecture shown in figure 3.1 in chapter 3. Our bilinear factor separation module is composed of two bilinear layers. The first bilinear layer f (x, y), given by eq. (4.2), combines the input feature x and given input year y in a multiplicative fashion into a content vector z y . The second bilinear layer, given by eq. (4.1), combines the content vector z y with the input time y in a multiplicative way into the reconstructed feature x y .

The key idea of our approach is to combine the content vector z ∈ R S and the style vector y ∈ R T in a multiplicative way, using a bilinear layer. In particular, the decoding layer g(z y , y) reconstructs the feature x y from the latent content variable z y , representing the input x interpreted using the 35 given input time vector y, and the given input time variable y as

x k y = g(z y , y) k = S s=1 T t=1 W s,t,k g z s y y t , (4.1)
where W g are the learnable weights of the decoding layer g and a b denotes component b of the vector a (and similarly for tensor components). Note that if y is a one hot encoding of the time period y, matrices W •,t,• g can be interpreted as linear weights specific to the period y.

Before we can proceed with translation using equation (4.1), the latent content vector z y needs to be estimated from the input image feature x. We compute z y using another bilinear layer as

z s y = f (x, y) s = K k=1 T t=1 W k,t,s f x k y t , (4.2)
where W f are the weights of the encoding layer f and z s y is the component s of the latent content variable z y representing the input x interpreted using the given input time vector y. Note also that similarly to W g , if y is a one hot encoding of the time period y, matrices W •,t,• f can be interpreted as weights of a linear layer for time period t. Also note that the latent content variable z y depends on the given input time vector y and can be thought of as a "projection" of the input representation x onto an "image subspace" specific to period y.

Relation to other approaches

In the following we discuss the relation of our bilinear module to (i) image translation based on concatenation, (ii) principal component analysis, and (iii) style and content factorziation.

Relation to concatenation-based generation methods

The standard way to combine variables in a CNN architecture is to concatenate them (or their transformed versions. See for example [Dosovitskiy et al., 2016]. Since the latent variables in concatenation based factorization methods are often estimated independently in a feed forward way, we omit the index y for the z variable in this paragraph. The output of a decoder g(z, y) of a single decoding layer g applied to a concatenation of input features z and y is written as where W g are the weights of the fully connected layer g. This equation is to be compared with equation (4.1), where the interaction between y and z is multiplicative and not additive.

x k y = g(z, y) k = S i=1 W i,k g z i + S+T i=S+1 W S+i,k g y i , ( 4
While the output is in practice often a result of several layers, it builds on this simple additive combination of the latent style and time. The resulting architecture is visualized in figure 4.1b, and we compare its representational power to our bilinear module in chapter 6.

Relation to principal component analysis (PCA)

In the case where y is a one hot vector representing each time period, there is a direct relation between our bilinear module and PCA. Indeed, our module can in this case be rewritten for each time period independently as the succession of two linear layers. If a L2 reconstruction loss is used between the output and the input, these two successive linear layers are equivalent, up to a simple transformation of the feature space, to performing PCA decomposition followed by reconstruction [START_REF] Baldi | Neural networks and principal component analysis: Learning from examples without local minima[END_REF][START_REF] Sanger | Optimal unsupervised learning in a single-layer linear feedforward neural network[END_REF]]. Thus, our architecture is equivalent in this case to performing PCA on each time period independently, a property that we verified experimentally: for each time period y, we reconstruct input feature x in the basis given by the PCA of image features in that period. In contrast to PCA, however, our bilinear module can naturally be inserted in an encoder-decoder architecture. This in turn allows training the entire factored image representation (including the bilinear module, encoder and decoder) in an end-to-end manner using more complex loss functions that go beyond simple L2 reconstruction.

Relation to bilinear style and content factorization

By representing vectors as a bilinear combination of time and latent content latent variables in concatenation based factorization methods, our model is related to the bilinear style and content separation of [START_REF] Joshua | Separating style and content with bilinear models[END_REF]. Our approach has, however, three fundamental differences. First, we assume the style (in our case time) vector is known, given and fixed for each input instance, and thus we only need to estimate the content vector. Second, because we assume the style is known, we can estimate the content vector with a simple linear operation. This is in contrast to [START_REF] Joshua | Separating style and content with bilinear models[END_REF] who have to resort to an algorithm based on an iterative SVD. Because of this key difference the model of [Tenenbaum and Freeman, 2000] also requires multiple observations of the same content with different styles, i.e. paired data, which our model does not and can operate on unpaired data collections. Finally, we include our bilinear separation approach as a module in a CNN architecture, which allows for end-to-end parameter learning.

Efficient implementation of the bilinear module

We implement our module and networks in Torch [START_REF] Collobert | Torch7: A matlab-like environment for machine learning[END_REF].

Torch provides an implementation of the bilinear operation called "nn.Bilinear". However, we found this implementation can be too slow for our purposes, so we implemented another version. In this section, we detail the differences between these two implementations and give performance results.

Let's call n the number of inputs processed simultaneously by the graphics card. This set of inputs is called a minibatch. In equation 4.1, we give the definition of the bilinear layer for a pair of vectors. In the neural network, the operation is implemented for matrices, in order to concatenate several input images into a single minibatch that can be processed more efficiently on graphics processing units. With a minibatch of size n, the bilinear operation can be written as

B(A, B, W ) i,c = 1≤a≤n A 1≤b≤n B B i,b W c,b,a A i,a ∀i = 1, . . . , n , ( 4
B(A, B, W ) i,c = 1≤a≤n A 1≤b≤n B 1≤h≤n A n B I h,b,a B i,b W c,b,a A i,a , (4.6)
where I ∈ R n A n B ×n A ×n B is a sparse indexing matrix given by

I h,b,a = 1 if h = 1 + (b -1)n A + a , 0 otherwise. (4.7)
By switching the order of the sums we can rewrite equation 4.6 as:

B(A, B, W ) i,c = 1≤h≤n A n B 1≤a≤n A 1≤b≤n B B i,b I h,b,a A i,a W h,c , (4.8)
where W ∈ R n A ×n B ,n C is a 2-dimensional matrix with the same coefficients as W :

W 1+(b-1) * n A +a,c = W c,b,a . (4.9)
Finally, we can see that we have re-written the bilinear operation as

B(A, B, W ) = B(A, B, I)W . (4.10)
We have implemented a bilinear layer using this formulation. Note that the bilinear operation B(A, B, I) (which can be seen as an outer product of each line of A and B) takes n A n B loops instead of n C , and can thus lead to significant gain when

n A n B < n C .
In our implementation of the model, the bilinear operation is used twice. First, the code z ∈ R S is obtained by applying the bilinear operation on the input feature x ∈ R K and the year feature y ∈ R T , where S, K and T are the dimensions of the content vector, the input feature vector, and the timed vector, respectively. In this case, using the default implementation of the bilinear module, we have n C = S loops, which is small: typically S = 16 and in all of our tests S ≤ 128. (see section 6.4 for a more detailed discussion about the size of z.). Second, the code is decoded into x y ∈ R K . In this case we have n C = K, which is 64,896 for our auto-encoder model. We thus use our bilinear layer implementation instead, which reduces the number of loops to K × T , which is 128 for a code of size K = 16 and a time feature of size of T = 8. Our new implementation is in this case approximately 300 times faster. With this implementation, each iteration of our bilinear layer is as fast as the baseline concatenation module. For our auto-encoder network, it takes 3 minutes to do a full pass on a dataset of 10,000 images on a GTX 1080 GPU, i.e. a full pass on 1M images would take about 5 hours. Furthermore, compared to a concatenation baseline with a similar number of parameters, our model converges in a smaller number of iterations to a lower value of the objective.

Chapter 5

Image Translation Models

In chapter 4, we have discussed how we separate time and content information using a bilinear module and how this module compares with other factorization techniques and architectures. While it is possible to apply the bilinear factor separation directly on the image pixels [Tenenbaum and Freeman, 2000] with some success on simple images, such as letters or digits, this approach does not work well for complex natural images. Instead, we perform bilinear factor separation on higher-level image features extracted using a convolutional neural network. This is achieved by employing our bilinear factor seperation module as part of a convolutional neural network architecture. This model can be trained end-to-end.In this chapter we plug-in the bilinear factor separation module in two different convolutional network architectures for image translation.

The first one, described in section 5.1, is a bottleneck auto-encoder. We explain how the presence of a bottleneck allows us to perform image translation. We describe how to employ the bilinear module in the autoencoder, compare it with standard concatenation, and discuss different losses used for training.

In section 5.2, we show how to employ the bilinear factorization module in a modern image translation architecture based on CycleGAN [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF]. We show that employing our bilinear factorization module allows us to learn a single encoder/decoder for all time periods instead of having a specific encoder/decoder for each pair of source and target periods. We explain how this model is trained to translate images thanks to a domain-adversarial loss.

Image translation using a bottleneck auto-encoder

In this section we show how to employ our bilinear factorization module in a bottleneck auto-encoder for temporal image translation. We first give an intuition of how to perform image translation using a bottleneck auto-encoder and then discuss the different losses used for training. Finally, we discuss the influence of the bottleneck code size on the model.

Intuition and Architecture

The bottleneck auto-encoder model is composed of three parts (figure 5.1): the feature encoder, the bottleneck factorization module, and the feature decoder. The feature encoder transforms an image into a vector representation that we refer to as "feature vector" that is easier to manipulate than the image itself. The goal of the factorization module is to translate this feature vector from its original time to its target time. We achieve that by compressing the encoded feature into a code vector of small size, before combining it with the year information. Because the code size is small, the model has to be efficient when storing information in the code. The intuition is that forcing the size of the code to be small encourages the code to contain only the time-independent information as the time-dependent information (in the form of the target time period) is added after the bottleneck during decompression. In detail, the time information is added either with the bilinear operation described in chapter 4, or with the standard feature vector concatenation. We will show experimentally in chapter 6 the benefits of our bilinear module. The feature obtained by combining these vectors can be decoded into the translated image.

IMAGE TRANSLATION USING A BOTTLENECK AUTO-ENCODER43

More formally, we denote A the encoder, which takes as input an image I and outputs a feature vector x: x = A(I). Then, the encoding layer f and the decoding layer g form the factorization module: z y = f (x y , y) where z y is the latent content variable, and x y = g(z y , y) where x y is the translated feature. We denote by A + the decoder that takes this translated and outputs a translated image I y : I y = A + (x y ). The factorization, composed of a coding layer f and a decoding layer g, can either be a concatenation module or a bilinear module. The concatenation module is composed of a linear layer for a simple linear layer z = f (x), and a linear layer on the concatenation of z and y for g as described in equation (4.3). For this module the latent code variable z does not explicitly depend on y. The bilinear module described in section 4.1.

Architecture details. Our encoder is based on the AlexNet [START_REF] Krizhevsky | ImageNet classification with deep convolutional neural networks[END_REF] architecture. We use the decoder with a deconvolutional architecture similar to [START_REF] Dosovitskiy | Inverting convolutional networks with convolutional networks[END_REF], but with batch normalization ([Ioffe and Szegedy, 2015]) in order to make the training more stable.

AlexNet has 5 convolutional layers followed by 3 fully connected layers. Features can be extracted from any of these layers, and we have to choose the best features for our task. Using features extracted from the earlier layers make the feature inversion easier and closer to the original image, while deeper features have a higher-level representation of the image. Based on preliminary results as well as the comparison of the reconstructions based on different features by [START_REF] Dosovitskiy | Inverting convolutional networks with convolutional networks[END_REF], we opted to use features from the 4th layer of AlexNet, conv4. After this pre-training, we train the model in two phases. First,we train the factorization module, f and g. Then, we fine-tune the whole model. We discuss the losses that we use for this training and fine-tuning in the next section.

Training loss

Our full loss for training the network can be written as the combination of three losses:

L total (M ) = L image (M ) + λ feat L feat (f, g) + λ sep L sep (f ) , (5.1)
where M is our bottleneck auto-encoder model, and λ feat and λ sep are hyper-paramaters weighing the reconstruction loss on the image L image (M ), the reconstruction loss on the feature vectors L feat (f, g) and the optional separation loss L sep (f ). We detail these losses in the next paragraphs.

Reconstruction loss on the image. The reconstruction loss on the image is a combination of a L1 loss and an adversarial loss:

L image (M ) = E I [ M (I, y I ) -I 1 ] + λ adv L adv (f ) , (5.2)
where λ adv is a hyper-parameter, and y I is the time corresponding to an image I. The goal of the L1 loss is to match an auto-encoded image M (I, y I ) with the original I as much as possible. As it tends to produce blurry images (see figure 6.4), the objective of the adversarial term L adv is to make images look more realistic, i.e. look more like a natural image, for example presenting sharp edges. To improve the stability of the training, we use the least-squares generative adversarial network loss, as described in [START_REF] Mao | Least squares generative adversarial networks[END_REF]. More precisely, we train a fully convolutional discriminator network D that takes as input either a real image I from the dataset, or an image generated by our model M (I, y I ), where y I is the time period corresponding to image I, and evaluates for each location of the image, in a convolutional way, if it comes from the dataset or our model. The generator M is trained to both provide a good reconstruction and confuse the discriminator. The adversarial loss for the generator thus is:

L adv (M ) = E I (D(M (I, y I )) -1) 2 , ( 5.3)
where D(M (I, y I )) is the output of the discriminator for the generated image M (I, y I ). In other words, the generator tries to produce images that are indistinguishable from real images by the discriminator, i.e. the discriminator outputs label 1. Note that this adversarial loss also depends on the discriminator D, which is trained jointly with our model using the following loss function

L disc (D) = E I D (M (I, y I )) 2 + (D(I) -1) 2 , (5.4)
where the expectation is taken over the set of training images I with associated time stamps y I . The first term, D (M (I, y I ) -0) 2 , is the loss on generated images, which the discriminator is trying to map to class 0, and the second term, (D(I) -1) 2 , is the loss on real images I, which the discriminator is trying to map to class 1. The intuition is that the discriminator tries to separate real and generated images while the generator tries to produce images that are indistinguishable from real photographs. Note that losses given by (5.3) and (5.4) are applied in a fully convolutional manner as described in [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF]. This adversarial loss is encouraging the output images to look like real images but does not attempt to discriminate between different time periods. Adversarial loss that discriminates between different time periods will be discussed in section 5.2.2.

Reconstruction loss on the feature vector. In addition to the above reconstruction loss on the pixel-level, we pre-train the bottleneck modules with a reconstruction loss on the feature vector level:

L feat (f, g) = E y∈Y x i y -g(f (x i y ), y) 2 2 , (5.5)
where x i y is the input image and g(f (x i y ), y) its reconstruction with the given ground truth time y. As discussed in chapter 4, this is analoguous to Principal Component Analysis in the case of a bilinear bottleneck module. For both the bilinear and concatenation modules, the feature-level reconstruction loss given by equation (5.5) ensures that the features are preserved when the target year is the year of the original image (no translation).

Separation loss.

Finally, the concatenation model can be complemented with a separation loss L sep . The aim is to learn a model where the code and time vector are two independent variables. For example, when the model is applied to images of faces, the code can capture the identity of the person and attribute the year of the photograph. Here, knowing the year of the photograph does not convey any information about the identity of the person. It is this intuition we would like to capture. We explain in the following paragraph the details of the separation loss. Please note that we only use it for the bottleneck auto-encoder with a concatenation module. As will be seen in chapter 6, we found that this type of separation loss is not necessary to achieve good results with our bilinear factorization module.

More formally, we assume that the code z ∈ R K and time vector y ∈ Y are realizations of random variables Z and Y , with probability distributions p(Z) and p(Y ), respectively. The aim is to enforce that Z and Y are independent, i.e. that P (Z, Y ) = P (Z)P (Y ) .

(5.6)

As by definition the joint probability of Z and Y can be written as

P (Z, Y ) = P (Y )P (Z|Y ) , (5.7)
it can be seen from equation (5.6) that independence can be expressed with the equation P (Z|Y ) = P (Z) .

(5.8)

Thus, Z and Y are independent if and only if the conditional distribution P (Z|Y = y) are the same for all y ∈ Y. In other words, knowing y is not going to bring any additional information about Z. To simplify the computation, we further assume Z has a Gaussian distribution N (µ, Σ).

Since z is the output of a linear layer and an affine transformation can be absorbed by this layer, we can further assume without loss of generality that µ = 0 and Σ = I, i.e. P (Z) = N (0, I). Plugging-in the Gaussian form of P (Z) into constraint (5.8), leads to P (Z|y) = N (0, I), ∀y ∈ Y.

(5.9)

In other words, the code Z is independent of time Y when for each value y of the time the observed conditional distribution of codes z is a unit Gaussian. We translate the constraint (5.9) into the following separation loss

L sep (f ) = y∈Y KL (P (Z|Y = y)) , N (0, I)) , (5.10)
where KL(., .) is the Kullback-Leibler (KL) divergence between two distributions.Note that the separation loss is minimized exactly when the constraint (5.9) is satisfied, i.e. the conditional distribution of Z for each y follows a unit Gaussian. To further simplify the computation, we assume that the dimensions z k of z are independent and Gaussian, which leads to the following form of the separation loss

L sep = y∈Y K k=1 1 2 -log((σ y k ) 2 ) + (µ y k ) 2 + (σ y k ) 2 -1 , (5.11)
where µ y k and (σ y k ) 2 are the mean and variance of the k-th dimension of the codes {z y 1 , . . . , z y i , . . . z y Ny } extracted from the images annotated with time y:

µ y k = 1 N y Ny i z y i,k , (5.12) (σ y k ) 2 = 1 N y Ny i (z y i,k -µ y,k ) 2 .
(5.13) Note that the loss L sep (f ) is summed over all dimensions of z (inner sum) and all possible values of time y (outer sum). Note again that above separation loss is enforcing independence of random variables Z and Y by pulling the mean and variance of codes z y extracted from images annotated with a certain time y towards 0 and 1, respectively.

Discussion

The size of the bottleneck inside the model M , i.e. code z y , is a critical parameter for the bottleneck auto-encoder. It influences the result in two different ways. Decreasing the dimension of z corresponds to decreasing the dimension of the linear space of the possible values of g(z y ) = x y , and leads to the following observations.On one hand, as a result of the information loss corresponding to this dimensionality reduction, the reconstruction error xx y will be larger, and the reconstructed image will lack details. On the other hand, the network has to rely more on the time specific code y to reconstruct the training data. As a result, the reconstructed images will exhibit more strongly the characteristic features of the images in the target period, which is part of our goal. In the extreme case where only the time specific code y is preserved and the dimension of the bottleneck z is zero, the best the network could do in terms of the reconstruction loss on the training data is to produce an output close to an average training image for the given time period.

We find that the concatenation module is more sensitive to the bottleneck code size than the bilinear module. For simple datasets, such as for faces, a balance between preserving image quality and generating typical images for each period can be found with both modules. In other cases, such as with the car dataset we use, we find that only the bilinear module allows us to produce strong appearance variations when changing the time period while preserving the characteristic features of the particular object depicted in the image. We compare results obtained with both modules in experiments presented in section 6.4.

Because the presence of a bottleneck is detrimental to image quality with both modules, we explore another image translation architecture without any bottleneck discussed in the next section.

Adversarial translation

In this section we introduce the domain adversarial translation model. First, we give an intuition of how this model performs image translation. We then discuss the different losses used for training. Finally, we compare this model to the bottleneck auto-encoder presented in section 5.1 and CycleGAN [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF].

Intuition and Architecture

A successful approach to image translation that does not use a bottleneck auto-encoder, CycleGAN, was recently introduced in [ work is focused on the case of two image domains, such as two different time periods in our case. One of the main drawbacks of CycleGAN for our task is that it does not build a shared factored representation, and thus requires to train two translation networks per domain pair. It also does not allow to represent continuous quantities, such as time. We extend the architecture of [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] to multiple domains. To achieve this we (i) share the network weights for translations between all periods, and (ii) add a factorization module as described in chapter 4 as a central block, in parallel to a shortcut connection. We show this model in figure 5.2.

Architecture details.

The network is based on [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF]. It uses residual building blocks ( [START_REF] He | Deep residual learning for image recognition[END_REF]), i.e. blocks of convolutional layers with the same dimension for the input and output, with an additive operation that adds the input of the block to the output. This network allows minimal quality loss in the reconstruction, because there is little dimensionality reduction. Instead, the network can translate images because it is trained with a domain adversarial loss, which forces the result of the translation to appear like images of the target time period. We can thus remove the bottleneck in our bilinear factor separation, which makes it equivalent to a single bilinear layer. Interestingly, while the architecture with a single bilinear layer is simpler than the full bilinear module, it typically has more parameters since it does not have a bottleneck.

Training Loss

To train our translation architecture denoted M , we use a combination of multiple losses:

L total (M ) = L id (M ) + λ cycle L cycle (M ) + λ dom-adv L dom-adv (M ) ,
(5.14)

where L id , L cycle , and L dom-adv are the identity loss, the cycle consistency loss and the domain adversarial loss detailed in the following paragraphs, and the λs are the relative weights of each loss.

Identity loss. We use a L 1 reconstruction loss on the generated image without translation, as we did for the bottleneck model:

L id (M ) = E I [ M (I, y I ) -I 1 ] , (5.15)
where the expectation is taken over all images I and y I is the time period associated to image I. As the target year is the year of the original image, this loss enforces that the image is modified as little as possible. The loss also improves the stability of the training.

Cycle consistency loss. Similar to [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF], we also consider a cycle consistency loss

L cycle (M ) = E I,y
[ M (M (I, y), y I ) -I 1 ] ,

(5.16)

where the expectation is taken over all images I and all possible target time periods y, and y I is the time period associated to image I. This loss ensures that a source image I translated to time y, M (I, y), can be converted back to the source image I when using the source time y I as the input to the generator.

Domain adversarial loss.

Finally, to enforce that the generated images look different for each time period, we rely on a domain adversarial loss

L dom-adv (M ) = E I,y (1 -D y (M (I, y))) 2 , (5.17)
where the expectation is taken over all images I and all possible target time periods y, and D y (M (I, y)) ∈ [0, 1] indicates if M (I, y) resembles images from the dataset with attribute y. The intuition is that the generated image M (I, y) for time period y should be classified correctly by the discriminator D. We train a single discriminator D that takes an image as input and is trained to output a vector that contains one hot encoding of the input image time period. D y denotes the component of the output vector corresponding to time y. The discriminator is trained jointly with the generator M using the following loss

L dom-disc (D) = E I (1 -D y I (I)) 2 + β E I,y D y (M (I, y)) 2 , (5.18)
where β is a hyper-parameter. The first term encourages that the discriminator predicts 1 for the correct time period y I for training images I. The second term encourages that for the translated images M (I, y) the discriminator output for period y, D y should be zero. In other words, the discriminator should be able to detect that the generated image does not belong to the target period. Note again that this loss is evaluated in a fully convolutional manner as described in [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF].

Discussion

This approach has some similarities with the aforementioned bottleneck auto-encoder described in section 5.1. In both cases, a L1 loss is used to compare an original image to the output of the network after translation is performed. With the auto-encoder, an adversarial loss sharpens the results and makes them more realistic. This addition is not necessary in the case of the adversarial translation model, because its domain adversarial loss, which is necessary for translation, also makes images more realistic. With this domain adversarial loss, the adversarial translation model does not rely on a bottleneck and is thus capable of outputting images of better quality.

We note here the differences between CycleGAN and our network. In CycleGAN, only two classes are considered, while we can consider an arbitrary number of classes (in our case time periods) using a single network. We do not need one generator network and one discriminator network for each class, but we use one generator network for all classes and one discriminator for all classes. The only parts of our networks that use more parameters when considering new classes are the bilinear factorization module in the middle of the generator, and the last layer of the discriminator which determines the time period of an image.

Datasets and metrics

To illustrate the generality of our method, we use two datasets of historical images for our evaluation. The historical car dataset [START_REF] Lee | Style-aware mid-level representation for discovering visual connections in space and time[END_REF]] is a set of cars made in a span of 80 years. The historical yearbook portraits [START_REF] Ginosar | A century of portraits: A visual historical record of american high school yearbooks[END_REF] contains pictures of students taken across 80 years. For both datasets, we grouped the images by decades and selected three decades with clear differences to analyse the results of our method (40s, 70s and 2000s for the faces and 40s, 60s and 90s for the cars). We thus obtained 3600 training images and 300 testimages for the face dataset across three decades, and 7299 training images and 1258 test images for the car dataset. Below we give details of the two datasets and introduce appropriate metrics to quantitatively compare the results of the different approaches.

Historical car dataset.

We show results on the challenging car dataset introduced in [START_REF] Lee | Style-aware mid-level representation for discovering visual connections in space and time[END_REF], containing modern images of cars together with their construction date, between 1920 and 1999. This dataset presents several difficulties, with cars having different appearance and different backgrounds. The cars are also imaged in a variety of lighting conditions and from a variety of viewpoints. To localize the cars in the image, we first run a standard Faster R-CNN car detector [Ren et al., 2015] pre-trained on Pascal VOC images. The detected bounding boxes are then resized to fit the input of the networks without preserving the aspect ratio. Despite this pre-processing, this dataset remains a great challenge.

Historical yearbook portraits.

We also applied the proposed model to the historical photograph collection of American high school yearbook portraits from 1930s to 2000s of [START_REF] Ginosar | A century of portraits: A visual historical record of american high school yearbooks[END_REF]. Aligned face images are known to be relatively easy to analyze, and results on this type of data have been demonstrated for example in [START_REF] Ginosar | A century of portraits: A visual historical record of american high school yearbooks[END_REF][START_REF] Reed | Learning to disentangle factors of variation with manifold interaction[END_REF][START_REF] Yan | Attribute2Image: Conditional image generation from visual attributes[END_REF].

Evaluation metrics. Quantitative evaluation for our task is difficult as there is no ground truth for how a specific car or portrait looked like in both 1920s and 1990s. We evaluate the different methods by looking at a reconstruction metric and a classification metric. Our reconstruction metric is simply the L1 distance between the input image and its translation into its ground truth period. While this is not a very precise metric of image similarity, we found that for our task it correlated well with percieved image quality. As pointed above, this reconstruction metric is limited since it can only be used together with the ground-truth time period, and doesn't evaluate wether the generated image shows characteristics of the target time period. This is the goal of our classification metric. Following [START_REF] Isola | Image-toimage translation with conditional adversarial networks[END_REF], we use an off-the-shelf classifier to assess whether our model is able to generate images that "look" like the images of the intended time period. This metric is related to the "inception score" [START_REF] Salimans | Improved techniques for training GANs[END_REF], object detection evaluation in [START_REF] Wang | Generative image modeling using style and structure adversarial networks[END_REF]] and the "semantic interpretability" measure in [START_REF] Zhang | Colorful image colorization[END_REF]. In detail, we trained a classifier on the conv4 features of an AlexNet network trained on ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] to predict time periods on real images and applied it (i) to a validation set of previously unseen real images, and (ii) to images generated by the different methods.

For each model and dataset, we report two numbers: (i) the percentage of correctly classified images in the case when the input image is translated into its ground truth time period ("Same year"), and (ii) the percentage of correctly classified images when the input image is translated to a different time period ("Modified year"). The first score evaluates whether the model preserves the temporal characteristic of the image when the time period is unchanged. The second score evaluates whether the model manages to change the time characteristics of the generated image in a way that fools the classifier. Higher numbers are better for both metrics; they can be interpreted as a success of the translation model to change the time period of the input image. The quantitative results are discussed in sections 6.3 and 6.4. In the following section, we present preliminary results using the bottleneck auto-encoder architecture, which served as a motivation for developing our bilinear factorization module.

Preliminary results with the bottleneck architecture

In this section, we describe the preliminary results obtained with the bottleneck auto-encoder architecture with a concatenation factorization module. We present results only on the easier face dataset, which contains relatively well aligned images. We found that this type of architecture was not able to learn image translation models on more varied image data such as the dataset of historical cars, where objects are not aligned and imaged from a variety of viewpoints and under different illuminations. These negative results served as the main motivation for developing the bilinear factorization module, which is evaluated in sections 6.3 and 6.4.

Architecture details.

In this section, we use the bottleneck auto-encoder architecture described in section 5.1. The year embedding y is a continuous mapping of the time t, defined by: where σ and α are hyper-parameters and K t is a normalization factor set so that y(t) 2 = 1. This mapping provides a soft discretization of continous time into bins representing different time periods. We use concatenation to combine the code z and the year embedding y. Both z and y have a code

y i (t) = K t exp (i -αt) 2 2σ 2 , ( 6

Visualizing differences over time

We first use our network to visualize trends over time from the perspective of a specific image. A first possible approach would be to simply look at the reconstructed images with the same content code z (the same person) for different time periods, as visualized in the top row for each person shown in figure 6.1. Looking at the images, the main observation is the change in the facial expression. As was previously noted in [START_REF] Ginosar | A century of portraits: A visual historical record of american high school yearbooks[END_REF], people in later pictures tend to smile more.

Quantitative analysis of reconstruction errors

While in 6.2 we have shown temporal translation results for specific individuals, here we try to extract patterns by analyzing the reconstruction errors of translated images across decades. In figure 6.2 we show eight plots, one for each time period showing the reconstruction error over time for all 100 testing instances in that time period. Within each graph the curves are shifted so that the error is zero for the ground truth period. The colors of individual curves correspond to the specific years inside the decade -cold colors represent the early years and warm color the later years.

The first observation is that the evolution pattern of errors is consistent for all test samples inside each decade. For example, graph (a) shows that images from 1930s have overall low reconstruction error for the neighbouring 1940s but also 1960s and 1970s. The reconstruction error is higher in 1950s and 1980s. Also, the smallest errors are often attained for the ground truth decade. Second, one can notice clear trends for the years inside the decade shown by consistent pattern of colors for the individual curves, despite the fact that the annotations used during training are exactly the same for all the years inside a decade. This effect is especially evident for the early decades. For example, inspecting graph (a) the images from early 30s encoded by blue and cyan colors have low reconstruction errors for 1950s but are not reconstructed well in 1980s. Interestingly, the reverse is true for the images of late 30s encoded by yellow and red colors that have relatively high reconstruction errors for 1950s bur are reconstructed relatively well in 1980s.

Finally, one can notice three clearly different groups. The first group is formed by 30s, 40s, 60s and 70s for which the errors are generally small but are slightly higher in the 50s and after (including) 80s. The second salient group is formed by the 80s, 90s and 2000s, which have a relatively high error for the earlier periods. Finally, the last group is the 50s, that consistently appeared as an outlier decade in this experiment.

Finding typical and non-typical examples

In this section we report results of visual data mining based on generated imagery. We generate a reconstruction of each test image for all time-periods and mine the test set for example faces that follow a specific temporal pattern in the reconstruction error. Results are shown in figure 6.3 and analyzed Average MSE over time. One of the simplest statistics to look at is the mean square error of the reconstruction, that we can average over all time periods. Figures 6.3a and 6.3b show the images that have a small and large average MSE over time.

PRELIMINARY RESULTS WITH THE BOTTLENECK ARCHITECTURE57

Range of reconstruction errors.

To avoid the bias in the MSE error, we sort here images based on the range of the reconstruction error, i.e. the absolute difference between the reconstruction error in the best and the worst year. Intuitively, if the range is large, then the images are easy to reconstruct in one period but very hard to reconstruct in some other period. On the contrary, if the range of errors is small, the image is just as likely to come from any period. The results are shown in figures 6.3c and 6.3d. We observe that the images with the highest range of errors are the minorities (figure 6.3d) that are badly reconstructed using the early decades because they are absent from the yearbook collections, but are well reconstructed in the recent years. The least distinctive images (figure 6.3c) with low range of errors are faces without explicit features associating them to any specific year.

Comparison of specific decades. The above analysis mines for the most striking differences across the entire test collection. We can, however, use our method to pinpoint images with large differences in reconstruction errors between two specific decades. For example, figures 6.3e and 6.3f highlight the differences between the 1970s and the 2000s. These images highlight two dominant differences: there are much more girls that are smiling and have long hair in the 2000s (figure 6.3e), and much more girls with shorter, dark, but voluminous hair in the 1970s (figure 6.3f). There are some outliers, which is expected as we are mining eight images out of only 100 test images for a specific period. Similarly, we compare in figures 6.3g and 6.3h the 1930s and 1950s and find that faces from the 1930s, which are the most different from the 1950s are the ones with very curly well combed hair. On the other hand, the short hairstyle with more volume around the ears is characteristic of the 1950s compared to the 1930s.

Discussion

In this section we have reported results using the bottleneck autoencoder with concatenation factorization. While these results were promising, we found this architecture has only limited ability to capture appearance variation over time. While it could extract (some) temporal variation from the face dataset, where the faces are fairly well spatially aligned, this model was not able perform temporal translation for the car dataset (results not shown), which has much larger variation of appearance due to intra-class variation, changes in viewpoint, background and illumination conditions. These results led us to develop the bilinear factorization module (chapter 4) which we employ in two different temporal translation architectures (as described in chapter 5. We report results for these models in the following sections.

Analysis of architectures and losses for image translation

In the previous section, we have shown preliminary results obtained with the bottleneck auto-encoder and the concatenation factorization module.

While the results were promising on the aligned face dataset of yearbook portraits, this architecture was not able to capture the larger image variation in the more complex and non-aligned car dataset. These results served as the primary motivation for developing the bilinear factorization module (chapter 4) and incorporating this module in two different temporal translation architectures (chapter 5): (i) the bottleneck auto-encoder (the same as used in the previous section) as well as (ii) the recent adversarial image translation architecture [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF]. In this section we report we report experimental results of comparing these two architectures. We show that the bilinear factorization module makes it possible to learn temporal variation using the bottleneck auto-encoder even from the challenging car dataset as it builds factorization capability directly into the architecture. However, we further demonstrate that even better results can be obtained by inserting our bilinear factorization module in the recent adversarial image translation architecture [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF]. In this case our bilinear module enables image translation to multiple different target time periods using a single network. We begin with the analysis of visual quality of the results, then perform quantitative analysis and finally discuss the typical failure modes.

In the next section 6.4 we analyze the differences between the standard concatenation based factorization module and our new bilinear factorization module when used in the bottleneck auto-encoder. Finally, in section 6.3 we demonstrate how to apply our bilinear adversarial translation model for the task of visual discovery in an unpaired image collection annotated with time stamps.

Qualitative analysis. We focus our qualitative analysis on the more challenging car dataset. Qualitative results on the face dataset are shown in section 6.5 and on our project webpage [web]. First, in figure 6.4, we compare the results of translation of cars from the 1990s to the 1940s using our bilinear factorization module inserted into the bottleneck auto-encoder architecture without (b) and with (c) adversarial loss as well as inserted into the adversarial translation framework (d). The results in column (b) show that using only the L1 reconstruction loss with a bottleneck auto-encoder architecture results in blurry images that show some changes related to the time period but are hard to interpret. As shown in column (c), adding an adversarial loss to the bottleneck auto-encoder produces more realistic images with clearer differences between the time periods. However, there is still significant loss in image quality, due to the dimensionality reduction in the bottleneck. Finally, images produced with the adversarial translation (column (d)) are the most realistic and make changes over time easiest to interpret.

Second, in figure 6.5, we show a comparison between our bilinear adversarial translation model and CycleGAN [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF], which is the state-of-the-art model for image translation. For CycleGAN, we have trained three pairwise models, in order to cover transformations from and to each time period. Note that our bilinear adversarial model is only trained once with the three time periods. While the images are slightly different, the overall image quality and the temporal changes are similar. However, our bilinear adversarial translation architecture has the advantage of being a single model that is faster to train, as most of the weights of the model are shared. Note that the adversarial translation using a concatenation based factorization module produces similar quality of results and is not shown in the figure .   Quantitative analysis. The visual results are supported by the quantitative results reported in table 6.2. First, for both datasets, there is a clear gap between reconstruction errors for methods using adversarial translation and the bottleneck auto-encoder, with more than an order of magnitude difference. This confirms the visual results form figure 6.4. The reconstruction error for the bottleneck auto-encoder architecture is improving with the size of the bottleneck, but does not reach the quality of the result of the adversarial translation model. Second, the classification results, which are much higher than chance for all adversarial translation frameworks and for some bottleneck auto-encoder architectures demonstrate that both the auto-encoder and adversarial translation models can effectively change the appearance of the input image towards the target period. Finally, the joint results of the three independently trained pairwise CycleGAN models are similar to the results of our single bilinear adversarial translation model, which confirms the visual results from figure 6.5. The adversarial translation models, and In each column the top image shows the input (car from the 1940s, green border) and the bottom image shows the output translation into the 1990s (blue border). In all cases the model fails to modify the input image in a substantial way, but often still modifies some small parts, such as the wheels or the headlights.

in particular our bilinear adversarial translation model, show both a high classification score and a low reconstruction error. This makes these models good candidates to perform visual discovery in unpaired temporal image collections, as we will demonstrate in section 6.5.

Failure cases. Figure 6.6 shows several typical failure examples of our best performing approach, the bilinear adversarial translation model. In general, our method fails on uncommon images, where it typically produces a car that is nearly identical to the original input image, with sometimes localized changes on small parts such as the wheels. In other words, the model defaults to the original input image when it doesn't know how to modify the image correctly. These failure cases typically correspond to unusual viewpoints (figure 6.6(a)), open trucks or boots (figure 6.6(b)), or rare or otherwise unique cars in our dataset (figure 6.6(c)).

Comparison of bilinear and concatenation factorization modules

In this section, we analyze the differences between the standard concatenation based factorization module and our new bilinear factorization module. We focus this comparison only on the simple bottleneck auto-encoder architecture. In the more powerful framework of adversarial translation, both types of factorization modules produce similar results. Qualitative analysis. As in the previous section, we focus our qualitative analysis on the more challenging car dataset and provide results on the face dataset on the project webpage [web]. Qualitative translation results for different code sizes are shown in figure 6.7 and demonstrate clear differences between the concatenation-based and bilinear models. Indeed, for the code size 16 and 64, the image quality is similar for both types of factorization modules, but the concatenation based module produces almost no differences between the two target time periods, while very visible changes happen using the bilinear module. By reducing the code size of the concatenation based module further, for example to 4, it is possible to force the model to produce some changes, but only at the cost of very low quality images, where a large part of the image identity has been lost. These results clearly demonstrate the advantage of the bilinear factorization module that we introduced and analyzed in chapter 4.

cars (3 periods)

Quantitative analysis. The visual analysis is supported by the the quantitative results reported in table 6.2a. The reconstruction error is similar using both factorization modules with the same code size, but the classification scores are much higher, on both datasets, using our bilinear module. Obtaining higher classification scores using the concatenation-based model is possible only at the cost of strongly decreasing the code size, and thus reducing the image quality. However, even with a code size of 4, resulting in an important decrease of the reconstruction error, the classification scores of the concatenation module are still lower than the scores of our bilinear module. This demonstrates the clear advantage of our bilinear factorization module when employed in the bottleneck auto-encoder architecture.

Visual discovery via bilinear image translation

In this section we demonstrate how to apply our bilinear adversarial translation model for the task of visual discovery in an unpaired image collection annotated with time stamps. First, we train a bilinear translation model on the input image collection using the time stamp of each image as its (observed) time variable. Note that the model is able to extract temporal changes over time from the unpaired data, i.e. despite never seeing a particular object instance (e.g. a car or a face) evolve over time. rectangular ones in the 1990s; or the windows, initially small, are enlarged over time. Note that the model learns to generate such changes without annotated part correspondence in the training data or seeing a specific object instance evolve over time in training. Please note also that such translation with a continuously changing attribute (here the time variable) would not be possible using the standard CycleGAN approach [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF].

Analyzing trends over time. An analysis of appearing, evolving and disappearing parts can be performed by looking at the differences between the images generated at the different time periods and the original input image. This way we can highlight the most important differences for a specific generated time period. Figure 6.9 shows such difference images for several examples from the car dataset. Please note the consistent changes over time. For example, the strong image differences (bright yellow color) at the base of windows and windshields indicate the evolution of their size over time, clearly showing the trend of having larger windows and windshields in later time periods. This trend is consistently exhibited in the input dataset on many different cars. Some elements are also characteristic for a specific period. For example, the running board on the side of the car is often present in the 1940s, but disappears (and hence is highlighted in bright yellow in the difference images) in the later periods. The wheels and headlights are also elements that have characteristic appearance at a specific time period and are often highlighted in the difference images. Similarly, figure 6.10 reveals the trends in hairstyle and facial expression across time in women's yearbook portraits. More examples of difference images can be found on the project webpage [web].

Please note how the vocabulary of parts (wheels, headlights or windshields for cars and hair, eyes or mouth for faces) as well as their correspondence across different instances is learnt implicitly by our model from the unpaired training data. The outcome is that the model is able to synthesize appearance variation over time for a specific new input object instance despite changes in viewpoint or specific new appearance of the input car. Note also that our model enables pixel-level analysis of temporal trends by just comparing the synthesized time-lapse image sequences.

Chapter 7 Discussion

In this chapter, we summarize the contributions made in this thesis in section 7.1 and present possible extensions of our work in secion 7.2.

Contributions of the thesis

In this thesis, we have developed models for temporal image translation and used them for visual discovery.

In chapter 4, we have introduced a trainable bilinear factorization module that allows us to isolate and transfer time dependent appearance variations. We have articulated the close relation between our bilinear factorization module and principal component analysis. We have compared the bilinear module to other approaches such as the standard concatenation module. Finally, we have shown how to implement the bilinear factorization module in an efficient way to speed up the computation of the bilinear operation.

In chapter 5, we have introduced two different architectures to perform temporal image translation. We have discussed the bottleneck auto-encoder model, which reduces the size of the vector representing the image in order to translate it to different time periods. We have discussed the different losses including a separation loss that can be used to train this model. We have also incorporated our bilinear module in an adversarial translation model to perform temporal image translation.

In chapter 6, we have shown how we can apply our models for temporal analysis of unpaired image collections. We have discussed how a combination of a reconstruction loss and a classification loss were applied to evaluate the different models. We have applied the bottleneck auto-encoder model to translate simple images of faces and to find typical and atypical images in the dataset. We have shown how the bilinear module performs in different architectures, and how it compares to the concatenation module. Finally, we have shown how our models can be used to identify and highlight the differences between time periods and how they can be used to create historylapse videos.

Future work

In this section we discuss different directions for future work. We first discuss two direct extensions of our work: training the model with more time periods or more factors of variation. We then discuss evaluation of translation models and ideas for that could be applied to our models.

Training the translation model with more time periods

We trained the bottleneck auto-encoder with a bilinear module using the whole cars dataset, representing time with a continuous variable. We experienced difficulties doing the same with the domain adversarial translation model, and stabilized the training by using three time periods only. With more than 3 time periods, our current discriminator does not learn to distinguish between the periods, and hence does not provide a good domain adversarial loss for the generator.

Below we discuss a possible change in the training loss in order to improve the training of the discriminator in the domain adversarial translation model and thus enable training the model for a larger number of time periods. The model currently uses for the discriminator the square loss described in equation (5.18), and similarly, the domain-adversarial loss for the generator is the square loss as described in equation (5.17). Instead, one possibility would be to replace the square loss of the discriminator by a cross-entropy loss, as it is commonly used for classification. With a cross-entropy loss, a single discriminator D could be used, which would take as input an image I or a translated image M (I, y) and output a vector of dimension T + 1, where T would be the number of time periods. The discriminator would be trained to output the correct time period for images from the dataset and identify translated images. The cross-entropy loss, replacing equation (5.18), would be given by: where D is the discriminator, M the generator, M (I, y) the output of the translation of image I to time y, y I ∈ R T the one hot vector corresponding to the time period of image I, and β a hyper-parameter weighting the importance of images translated by M and images from the datasets to train the discriminator. Please note that the first term in equation (7.1) is encouraging the discriminator to classify the time period of real images, while the second term encourages the discriminator to identify images translated with the model M . As the cross-entropy loss is commonly used to train classifiers, it may help for training the discriminator which we found difficult to train when using many time periods. Similarly, the domain-adversarial loss would also change to the cross-entropy loss: (7.

2)

The translation model would try to output images that the discriminator classifies as the target time period.

Multiple factors of variation

In this thesis we have focused on the evolution of images through time and introduced a trainable bilinear module to model relations between time, annotated in the datasets we use, and other factors of variations. A possible extension is to introduce other annotated factors of variation. For example, how do building facades vary with time across different locations? The bilinear module can be extended to a multilinear module, with multiplicative interactions between the different factors. This task would be challenging because the size of the parameter tensor would require the computation to be efficient in both memory and speed.

Metrics for quantitative evaluation

In section 6.1, we have discussed metrics to evaluate our models for image translation and proposed to use a combination of a classification loss to check if the image was correctly translated and a reconstruction loss to evaluate if identity can be preserved when the model does not perform translation. These metrics are imperfect as they do not evaluate if the identity is preserved when the model performs translation. Other approaches for evaluation can be considered.

A possible approach is to test our methods on a dataset for which other metrics are available. In [Dosovitskiy et al., 2016], images of cars and chairs are rendered from 3D models. They evaluate their methods of translating from one viewpoint to another by comparing their translations with the ground truth renders that are available for different views of the same object. The advantage of this method is that we could use a square loss to compare directly the translated image with our methods and a generated image of what the object looks like with a different attribute that serves as ground truth. While this metric can only be used on generated datasets, it would allow us a direct evaluation what we want to achieve with our translation model. However, the square loss on pixels is not a direct metric for how humans evaluate proximity between images.

Another possible approach is to use human evaulators. Human perceptual studies have been used to assess the visual quality of generated or translated images. For example, in [START_REF] Isola | Image-toimage translation with conditional adversarial networks[END_REF], Amazon Mechanical Turk (AMT) participants are asked to discern generated images from real images. In [START_REF] Choi | Stargan: Unified generative adversarial networks for multi-domain image-to-image translation[END_REF], AMT participants rank images from different methods for transferring attributes such as the hair color or the gender to images of faces. Similarly, human perceptual studies would help us evaluate and compare the different methods we discuss for temporal image translation. To achieve this, a possible solution would to ask annotators to rank translations of the same image at the same time period by different models, ranking best the translations that convincingly change the time period of the image while preserving its identity. It would also be possible to evaluate the quality of the image by presenting to participants sets of two images of the time period, one translated and one from the dataset, and ask them to identify them.

Figure 1 . 1 :

 11 Figure 1.1: Bilinear image translation. Our goal is to take as input (in green) an image of a face or an object such as a car, and generate what it would have looked like in another time-period (in blue). Each row shows temporal translation for a different input portrait or car picture. We further show that analyzing changes between the generated images can reveal structural deformations in object shape and appearance over time.

  Figure (1.2):Analyzing trends over time. We show how translated images allow us to examine differences between cars manufactured in different time periods. We show the original cars from the 1990s (top row), their translation into the 1940s (middle row) and the absolute differences (bottom row) between the original image and the translated image. The absolute differences are shown as heatmaps where bright yellow color indicates a high difference. In this example the most salient differences are: different appearance of wheels, the presence of a runner board in the 1940s and the bulky distinctive hood in the 1940s.

  data. Portraits of three different people taken in a different time period. It is not possible to know how a portrait of the same person would have looked like had it been taken in a different time period.

Figure ( 1

 1 Figure (1.4): Illustration of the difference between paired (top) and unpaired data (bottom). In this work we focus on unpaired data, which is more challenging.

Figure ( 1

 1 Figure (1.5): Challenge: intra-class variation. The car dataset that we use presents very different types of cars (city cars, sedans, sports, utility cars and others).

Figure ( 1

 1 Figure (1.6): Challenge: viewpoint variation. The object can be imaged from a variety of viewpoints.

Figure ( 1

 1 Figure (1.8): Challenge: High-dimensional output. Contrary to the classification task that has a binary output (top), the translation task (bottom) has a high dimensional output.

e S 1≤i≤N x i - 1≤k≤S x i , e k e k 2 (2.1) 19 Figure

 219 Figure (2.1):The first nine eigenpictures from[START_REF] Kirby | Application of the karhunen-loeve procedure for the characterization of human faces[END_REF], ordered left to right and top to bottom by the order given by the PCA, i.e. by how important they are for accurate reconstruction of pictures. Faces can be reconstructed by linear combinations of these eigenpictures. Adding more eigenpictures will lead to better reconstructions, but with diminishing returns.

Figure ( 2

 2 Figure (2.2): Illustation of how a one-dimensional surface (left) can be captured by a mixture of linear models (right) from [Hinton et al., 1997].

Figure ( 2

 2 Figure (2.3): Illustation from [Tenenbaum and Freeman, 2000] of a symmetric bilinear model. The parameters (top left) of the model are the basis of the low-dimensional space. Each image is characterized by a vector containing pose information (bottom left) and person information (top right).Images can be reconstructed (bottom right) by making the outer product of these two tensors, and using the resulting matrix as weights for the linear combination of the basis images.

Figure ( 2

 2 Figure (2.4): Rotation of cars obtained by [Tatarchenko et al., 2016]. In these two examples, the predictions are on the top row, and the ground truths are on the bottom row; the input image fed to the network is the leftmost image.

Figure ( 2

 2 Figure (2.5): Translation of faces into different emotions in [Cheung et al., 2014]. Input images, in the left most column, are encoded into (z, ŷ). Then, the generator takes z and a different emotion y as input, to decode into images shown on the right.

Figure ( 2

 2 Figure (2.6): Street images of Paris (top) and London (bottom), and results (right) of visual elements that are characteristic of these two cities from [Doersch et al., 2012].

Figure ( 2

 2 Figure (2.7): Correlation between weather and presence of sunglasses in pictures from [Matzen and Snavely, 2015].

  Figure (3.1): Convolutional neural network architecture for factored visual discovery. We assume that an image is completely determined by (i) a latent content variable zy capturing the appearance factors independent of time such as car type and color, and (ii) the time feature y capturing the appearance factors specific to certain time, such as the running boards for cars made in 1930s. The input image I is encoded into a feature vector x, which is transformed together with year variable y using a mapping f (x, y) into a latent content representation zy. A second mapping g(z, y) then combines the latent content with the input time feature y to produce new feature xy. The reconstructed feature xy is finally decoded into image Iy. Parameters of the factorization module together with the encoder and decoder are learnt in an end-to-end manner.

  Concatenation-based network.

Figure ( 4

 4 Figure (4.1): Factorization architectures. Our bilinear factorization module (top) and the standard concatenation-based architecture (bottom) have two principal differences. First, our module captures multiplicative interactions between time y and content zy, while concatenation implies additive interactions. Then, we explicitly include dependency on time y in computing latent content zy. More layers can be included in our bilinear module similar to the concatenation architecture.

  .4) where B(A, B, W ) ∈ R n×n C is the output matrix holding n vectors of output dimension n C , A ∈ R n×n A and B ∈ R n×n B are the input matrices holding n vectors of dimension n A and n B respectively, and W ∈ R n C ×n B ×n A denotes the parameters of the bilinear operation. Note that there is no additive bias in the operation. The default bilinear module in the Torch implementation makes this operation in the following order. First, for each c = 1 . . . n C , the quantityBW c, * , * ∈ R n×n Bis computed with a call to the matrix multiplication routine, where W c, * , * ∈ R n B ×n A is the matrix obtained by taking the c-th sub-matrix of W : (BW c, * , * ) i,a = 1≤b≤n B B i,b W c,b,a ∀i = 1, . . . , n ∀b = 1, . . . , n B . (4.5)Then, the result of the bilinear operation is obtained by making the element wise product between BW c, * , * and A, and summing over the index a, which gives C i,c for each c = 1 . . . n. While the matrix multiplication, element wise product and sum are fast operations done on the GPU, the loop over c = 1 . . . n submatrices in the mini-batch and the associated function call overheads can result in a significant time loss. We can reduce the number of loops by rewriting the bilinear equation, for each c = 1, . . . , n C and i = 1, . . . , n as:

  Figure (5.1):The bottleneck auto-encoder. In this model, modification with time is enforced by the bottleneck and the separation loss Lsep, which separates time and content. The proximity to the original image is enforced by the reconstruction loss on the feature L feat , which compares the input and output of the factorization module, and the reconstruction loss on the image L image , which compares the original image to the output of the model.

Training details.

  The parameters of the encoder are initialized by training it for classification on the ImageNet [Russakovsky et al., 2015] dataset. The decoder is pre-trained to minimize the squared distance between the original image and the auto-encoded image A + (A (I)) -I 2 2 , without any modification of the feature vector.

L

  Figure (5.2):The adversarial translation model. In this model, modification with time is enforced by the domain adversarial loss L dom-adv , which compares the output of the model to other images of the same time period. The proximity to the original image is enforced by the reconstruction loss on the image L image , which compares the original image to the output of the model, and the cycle loss L cycle , which compares the original image to an image translated to another time period then translated again to its original time.

  Face image from the 2000s reconstructed at different time periods. (b) Face image from the 2000s reconstructed at different time periods. (c) Face image from the 1930s reconstructed at different time periods. (d) Face image from the 1930s reconstructed at different time periods.

Figure ( 6

 6 Figure (6.1): Results of the bottleneck auto-encoder with concatenation factorization module. For each face the top row shows the generated appearance over time and the bottom row shows the differences compared to the reconstruction at the true time period. Blue represents darker pixels and red represents lighter pixels in the reconstruction of the specific period compared to the true period. In these examples we can see that most of the difference is located around the lips.

Figure

  Figure (6.2): Reconstruction errors (y-axis) for different time periods (x-axis). See text.

  (a) Test images for which the average reconstruction error across all time periods is the lowest. (b) Test images for which the average reconstruction error across all time periods is the highest. (c) Test images for which the range of the reconstruction errors across time is the lowest. (d) Test images for which the range of the reconstruction error across time is the highest. (e) Test images from the 2000s most different from the 1970s. (f) Test images from the 1970s most different from the 2000s. (g) Test images from the 1930s most different from the 1950s. (h) Test images from the 1950s most different from the 1930s.

Figure ( 6

 6 Figure (6.3):Visual data mining based on generated imagery. We generate a reconstruction of each test image for all time-periods and mine the test set for example faces that follow a specific temporal pattern in the reconstruction error. See discussion in section 6.2.3.

  Figure (6.4): Comparison of different architectures and losses. From left to right: (a) original input image in the 1990s ; (b) translation into 1940s generated with a bilinear bottleneck autoencoder fine-tuned using only a L1 reconstruction loss produces rather blurry output ; (c) Fine-tuning using an additional adversarial loss produces sharper results ; (d) the best results are obtained using the bilinear adversarial translation model.

  Figure (6.5): Comparison between our bilinear adversarial translation model and CycleGAN [Zhu et al., 2017]. (a) The input image (1960s). (b)-(c) translation by our approach into 1940s and 1990s, respectively. (d)-(e) translation by CycleGAN into 1940s and 1990s, respectively. Both our bilinear adversarial translation and CycleGAN produce sharp images and clear changes over time with comparable visual quality of results. However, our bilinear adversarial translation architecture has the advantage of being a single model whereas a separate CycleGAN model needs to be trained for each target time period.

  Figure (6.6): The main failure modes of our bilinear adversarial translation model. Each column shows one failure mode (unusual viewpoint, open hood, rare car type). In each column the top image shows the input (car from the 1940s, green border) and the bottom image shows the output translation into the 1990s (blue border). In all cases the model fails to modifythe input image in a substantial way, but often still modifies some small parts, such as the wheels or the headlights.

  Input images of cars from the 1960s (top row) and their generated translations into 1990s (middle row). The difference images (bottom row) highlight the changes in the headlights of the cars, going from small round ones to large rectangular ones, as well as the increasing size of the car windows and windshield. Input images of cars from the 1940s (top row) and their generated translations into 1990s (middle row). In addition to the changes in the shape of headlights and the size of windows, similar to (a), the difference images (bottom row) highlight the disappearing metallic bumpers as well as the evolution of the shape of the hood, which flattens over time. Input images of cars from the 1990s (top row) and their generated translations into 1940s (middle row). The difference images (bottom row) highlight the appearing running board (or footboard) as well as the changing appearance of wheels.

Figure ( 6

 6 Figure (6.9): Analyzing trends over time. Each plate (a-c) shows consistent changes over time. In each plate, we show the original input images (top row), their translation into another period (middle row) and the absolute differences (bottom row) between the top and middle images. The absolute differences are shown as heatmaps where bright yellow color indicates the maximum absolute difference within the image, while dark blue colors correspond to small differences. We superimpose the heatmaps on a low contrast version of the original input image to further highlight the changes produced by our model. Please see more results on the project webpage [web].

Figure ( 6

 6 Figure (6.10): Analyzing trends over time. Each plate (a-c) shows consistent changes over time. In each plate, we show the original input images (top row), their translation into another period (middle row) and the absolute differences (bottom row) between the top and middle images. The absolute differences are shown as heatmaps where bright yellow color indicates the maximum absolute difference within the image, while dark blue colors correspond to small differences. We superimpose the heatmaps on a low contrast version of the original input image to further highlight the changes produced by our model. Please note consistent changes in hairstyle in plates (a) and (b), and smile in plate (c).

L

  dom-disc (D) = -E I 1≤i≤T [y Ii log (D(I) i )] -β E I,y[log (D(M (I, y)) 0 )] , (7.1)

L

  dom-adv (M ) = E I,y 1≤i≤T [y Ii log (D(I) i )] .

Table 6 .

 6 1: Details of the auto-encoder architecture used in section 6.2.

	Network part			Encoder: z = f θ (A (I))		Concat (z, y)
	layer type	conv	max	conv	max	conv	conv	conv		concat
	kernel size	11	3	5	3	3	3	3		-
	dimension	21	11	11	6	6	6	6		6
	nb features	96	96	256	256	384	384	8		8 + 8
	stride	4	2	1	2	1	1	1		1
	batch norm	-	-	-	-	-	-	-		-
	non-linearity	ReLu	-	ReLu	-	ReLu	ReLu	-		-
	Network part			Decoder: Iy = A + g φ (z, y)
	layer type	conv	conv	up	conv	up	conv	up	conv	up	conv
	kernel size	5	5	2	5	2	5	2	5	2	5
	dimension	6	6	12	12	24	24	48	48	96	96
	nb features	256	256	256	256	256	128	128	64	64	3
	stride	1	1	1	1	1	1	1	1	1	1
	batch norm	BN	BN	-	BN	-	BN	-	BN	-	-
	non-linearity	lReLu	lReLu	-	lReLu	-	lReLu	-	lReLu	-	-

size of 8. Details of this architecture are presented in table 6.1.

Table 6

 6

.2: Quantitative evaluation of our models and baselines. We evaluate both a reconstruction error (normalized L1 loss, lower is better) and a "inception" type of score (percentage of correctly classified images, higher numbers are better).

  Then, we apply the trained model on a new unseen test image. By varying the input time stamp we generate a history-lapse video that translates the input image into different time periods while preserving the identify of the depicted object. Finally, we analyze the generated image sequence to extract interesting changes in style over time. Given a pre-trained bilinear adversarial translation model we generate a history-lapse for a new test image. This is achieved by varying the input time variable. In detail, using soft-assignment of the year y to define the time vector y, we produce history-lapse videos with dense sampling of time. We generate videos at a sampling rate of 1 frame per year. Please see the videos on the project webpage[web]. Example frames are shown in figure6.8. The videos reveal interesting changes of car style learnt from the entire dataset but applied on a specific instance of a car depicted from a given fixed viewpoint and captured in given imaging conditions. Note how some parts appear, disappear or are transformed over time. For example, the running board on the side of the car, characteristic for the 1940s gradually disappears over time; the round lights of the 1940s are transformed into

	History-lapse videos.

Acknowledgement

Chapter 6

Visual Discovery in Unpaired Image Collections

In the previous chapter, we have described how we incorporate the bilinear module in different image translation architectures. The goal of this chapter is to experimentally compare these architectures and demonstrate their use for visual discovery. The objectives are threefold. First, we wish to validate whether the previously described architectures can perform temporal image translation. Second, we wish to compare their performance to sensible baselines on the image translation task. Finally, we wish to demonstrate visual discovery in unpaired image collections via learning and applying the proposed temporal image translation models. This chapter is organized as follows. In section 6.1, we introduce the datasets and metrics that we use to evaluate our models. In section 6.2, we present preliminary results obtained with the bottleneck auto-encoder. These results demonstrate the difficulty of temporal translation on complex visual data and have been for us the motivation for developing our bilinear factorization module (described in chapter 4).

In the remainder of this chapter we present results of our bilinear module incorporated in the two image translation architectures described in chapter 5. In section 6.3, we analyze the effect of different architectures and losses on the results of temporal image translation. In section 6.4, we provide an in depth comparison of our bilinear module with the the standard feature concatenation. In section 6.5, we give examples of the new type of image analysis enabled by our model: generating history-lapse videos and discovering trends by analyzing changes across time. Please note how the shape and appearance of the input cars changes over time. For example, the curved hood gradually fattens, the round lights become rectangular, the windscreen gets bigger, the front grille gets smaller and the running board on the side gradually disappears. Note also that other characteristics of the input car instances, such as viewpoint, color and the overall shape are preserved. History-lapse videos can be found on the project webpage [web].

ABSTRACT

This thesis proposes an approach for analyzing unpaired visual data annotated with time stamps by generating how images would have looked like if they were from different times. To isolate and transfer time dependent appearance variations, we introduce a new trainable bilinear factor separation module. We analyze its relation to classical factored representations and concatenation-based auto-encoders. We demonstrate this new module has clear advantages compared to standard concatenation when used in a bottleneck encoder-decoder convolutional neural network architecture. We also show that it can be inserted in a recent adversarial image translation architecture, enabling the image transformation to multiple different target time periods using a single network.

MOTS CLÉS

Apprentissage, Découverte Visuelle, Vision par Ordinateur RÉSUMÉ L'objectif de cette thèse est de développer des outils pour analyser les collections d'images temporelles afin d'identifier et de mettre en évidence les tendances visuelles à travers le temps. Cette thèse propose une approche pour l'analyse de données visuelles non appariées annotées avec le temps en générant à quoi auraient ressemblé les images si elles avaient été d'époques différentes. Pour isoler et transférer les variations d'apparence dépendantes du temps, nous introduisons un nouveau module bilinéaire de séparation de facteurs qui peut être entraîné. Nous analysons sa relation avec les représentations factorisées classiques et les auto-encodeurs basés sur la concaténation. Nous montrons que ce nouveau module présente des avantages par rapport à un module standard de concaténation lorsqu'il est utilisé dans une architecture de réseau de neurones convolutionnel encodeur-décodeur à goulot. Nous montrons également qu'il peut être inséré dans une architecture récente de traduction d'images à adversaire, permettant la transformation d'images à différentes périodes de temps cibles en utilisant un seul réseau.

KEYWORDS Machine Learning, Computer Vision, Visual Discovery