Anesio De Leles

Ferreira Filho

Andrew Mcpherson

Victor Zappi

Heider Edil Guimar Ães De Medeiros

Marconi Madureira

Christine Guillemot

Yves Wiaux

Yoann Altmann

Soufiane, Maxime Rodolphe Pierre

Nicolas Javiera

Alexis Benjamin

R Émy

Gaston

Louis

Laurane Simon

Anthelme (who I consider a honorary doctorant), and all the others who are too many to name. Thank you for sharing with me many climbing injuries, logic puzzles, break room discussions, and often just a cup of tea or coffee while we got ready to go find more bugs in our codes. Thank you also for your patience with me while I slowly learned your language, and my numerous questions while I did so. Despite my frequent criticisms of some of its n'importe quoi rules, I appreciate it more than you think. I am also thankful to ONERA, who has funded this thesis and all the conferences which I have attended, as well i as the staff that helped me during these years. Thank you also to T él écom Paris, especially the IMAGES group.

Although I did not spend much time there, I have always felt welcome when I did.

I can also say with certainty that I would not have accomplished what I have so far without the support of my family, mainly from my parents Geraldo and Liliana, but also from my brother Gilberto and my grandparents. You may not have taught me computer vision or machine learning or remote sensing, but you have taught me discipline and hard work, and you have been by my metaphorical side, even if physically distant during most of this journey.

Finally, thank you, Dani, for going through these years with me. It was not always easy, especially at the end,

List of Figures

Vision Memo

No. 100 from the Artificial Intelligence Group at MIT contained the goals for the Summer Vision Project, including segmentation between objects and background, and scene analysis containing objects such as balls, bricks, cylinders, and other "objects of known sort". . . . . . . . . . .

Comparison between image segmentation and semantic segmentation from the PASCAL VOC 2012

dataset [EVGW + 12]. In (b), the buses' pixels are separated into two different groups with no semantic information since they come from different objects. In (c), the pixels relative to the regions of both buses are classified as belonging to the semantic class "Bus". . . . . . . . . . . . . . . . . . . . . . . . 2.4 (a) While Rosenblatt's inspiration came partially from retinal nerves, the perceptron model is not restricted to vision. (b) Information flows in a single direction from inputs to outputs and through hidden units in a multilayer perceptron. Images reproduced from [START_REF] Rosenblatt | Perceptron simulation experiments[END_REF] and [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF], respectively. .

2.7

The FCN architecture upsampled feature maps in a single step, which limited the accuracy of predictions around region boundaries. Reproduced from [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF]. . . . . . . . . . . . . . . . . . . . . . . . 2.8 SegNet architecture for semantic segmentation, in which the pooling indices were used for the unpooling operations to recover spatial information at each upsampling step. Reproduced from [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF].

2.9 U-Net architecture schematic. The output of transposed convolutions are concatenated with feature maps produced by the encoder at several levels to combine high spatial accuracy with high-level features. Reproduced from [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.10 Schematics of (a) residual block, (b) dense block, and (c) residual dense block as described in [ZTK + 18].

2.11 Supervision strength paradigms. The challenge of weakly supervised learning is to learn higher precision tasks from lower precision supervision. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.12 Task similarity tree proposed by the Taskonomy project. It was shown that there are several benefits from learning a single latent space representation from which several tasks are performed, such as the need for fewer annotated data. Reproduced from [ZSS + 18]. . . . . . . . . . . . . . . . . . . . . .

2.13

One of the main motivation for domain adaptation is to bridge the domain gap between (a) synthetic data and (b) real data to reduce the cost of data acquisition. One such case is the semantic segmentation of GTA 5 images [START_REF] Richter | Playing for data: Ground truth from computer games[END_REF] and real automotive images from the Cityscapes dataset [COR + 16].

2.14 General Adversarial Networks are trained by forcing two networks to accomplish contradictory tasks.

The generator G attempts to generate realistic images indistinguishable from real samples, while the discriminator D attempts to separate fake from real samples. . . . . . . . . . . . . . . . . . . . . . . .

2.15

Naive change detection methods for producing and thresholding difference images tend to detect either too few or too many changes. Humans are very good at understanding difference based on the context, but the direct pixel colour difference is usually much less discriminative than our intuition suggests. Adaptive thresholding algorithms also assume there are changed and unchanged pixels in the image (often in comparable quantities), which is sometimes a flawed assumption, especially for the automated analysis of large image collections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2. 16 Example of a binary classification evaluation at pixel-level applied to change detection. In (e)-(h), the elements of each group are represented in white. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.1

Example of Sentinel-2 image bands. Most Sentinel-2 spectral bands are of a lower spatial resolution than the visible bands, but they often contain information that can help with the analysis of the images.

3.2 Sometimes, such as between images (a) and (b), it is clear to say whether or not changes have occurred and where they are located. In other cases, such as between images (c) and (d), it may not be clear where the changes are or even if changes have occurred at all. . . . . . . . . . . . . . . . . . + 17] and (e-g) the proposed fully convolutional networks on image Szada/1 from the Air Change dataset.

Comparison between (d) the results obtained by the method presented By Zhan et al. in [ZFY

Changes are marked in white. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + 17] and (e-g) the proposed fully convolutional networks on image Tiszadob/3 from the Air Change dataset.

Comparison between (d) the results obtained by the method presented By Zhan et al. in [ZFY

5.

12 Spatial attention weights that were learned in each of the cross-validation tests. Top row contains all 5 tests using fixed scale ABCD dataset, bottom row are the results using the rescaled version of the dataset. Note that the network was incredibly consistent in identifying the center of the images as most discriminative without any explicit knowledge. These attention matrices are of size 40 × 40. . . . 5.13 Results obtained by using the proposed method. Note that when the attention layer is not used, the network does not learn to localize the features and tends to predict all pixels into the same class.

The attention layer enables the network to localize features much more accurately, and the GAD post-processing further increases the spatial accuracy of such predictions. . . . . . . . . . . . . . . . 96 6.1 DINE aims to extract a code in a latent space where the desired task can be performed regardless of

List of Tables

R ésum é

La t él éd étection est un moyen puissant d'observer les grandes zones avec une grande efficacit é. En utilisant des images prises de tr ès haut, comme celles des avions ou des satellites, il est possible de mesurer avec

Context

Eratosthenes, a Greek polymath, was the first person known to have accurately calculated the circumference of the Earth over two thousand years ago [R + 13]. By measuring the angle of the shadows cast by the Sun at noon on the summer solstice in Alexandria and Syene, the distance between these two cities could be extrapolated to obtain a surprisingly accurate estimate of Earth's size. Much effort has been spent before and since to understand and catalog our planet. Such understanding of our environments, both natural and human-made, is necessary for well-informed planning and decision-making in various situations. Urban planning, warfare, forestry, agriculture, and many other sciences benefit from having information about the current state of the world, its past state, and how it changes.

Remote sensing is a powerful way of observing the large areas with high efficiency. Using images taken from high above, such as from airplanes or satellites, it is possible to accurately measure several characteristics from entire regions from a distance. Imaging techniques can vary significantly, depending on the desired applications.

Passive imaging types include panchromatic, RGB (red, green and blue), multispectral, and hyperspectral. These acquisition methods consist of measuring the intensity of electromagnetic waves emitted by the observed object or area. Active imaging methods, such as SAR (synthetic aperture radar) and LIDAR (light detection and ranging),

emit electromagnetic waves at a target region and observe the reflected response.

Several improvements in imaging technology over the years led to an increase in the quality of the captured images. Figure 1.1 shows images of the Palaiseau Fort acquired during aerial surveys over many decades. Such images were initially obtained by film cameras and in greyscale. Nowadays it is possible to acquire digital photos with color information and at higher spatial resolutions. Other technological improvements, such as better atmospheric correction and registration techniques, further increased our capacity to capture ground-level information from a distance.

The launch of Sputnik 1, the first artificial Earth satellite, marked the beginning of the Space Race in 1957, which has completely changed the field of remote sensing. Before the 1960s, remote sensing of the environment was done predominantly by aerial photography. Nowadays, satellite imaging is widely used to observe the earth. These can monitor atmospheric conditions (e.g. for weather forecasting), as well as ground-level objects. Programs such as Landsat, SPOT, and Sentinel are able to automatically image the Earth at regular intervals. The pair of Sentinel-2 satellites, for example, are able to image all of Earth's main landmasses at intervals of 5-10 days (see Fig. 1.2).

These Earth observation satellites produce extremely large amounts of data. Each of the Sentinel-2 satellites, for example, produces 1.6 TB of multispectral imaging data per orbit, with an orbital period of 100.6 minutes [ESA20a],

resulting in approximately 45.8 TB of data per day. Such large amounts of data are the main motivators for research into automatic Earth observation image analysis. These images contain large amounts of useful information about the Earth and its evolution through time, but the extraction of useful information from remote sensing imagery is a non-trivial problem.

The interpretation of remote sensing images is useful for many applications [START_REF] Audebert | Machine learning for classification of big remote sensing data[END_REF]. Accurate interpretation of such images is extremely important in several impactful areas of study: weather forecasting, climate change, deforestation, monitoring of glaciers, natural disasters, urbanism, agriculture, etc. But the high level or heterogeneity of Earth observation data poses many challenges for its automatic interpretation. Dealing with such large scale and heterogeneous data is a central point of this thesis.

Recently, neural networks have been proven to be extremely powerful in solving several problems, especially in the field of image analysis. Remote sensing is no exception, and the application of neural network based methods to extract meaningful information from Earth observation imagery is currently an extensively researched topic.

Domain

This thesis is located at the intersection of three scientific domains, with different associated practices and traditions:

1. Mathematical sciences: rigorous mathematics is an integral part of image analysis, computer vision, and machine learning. It is the language in which operations, properties, analyses, and results are most often discussed.

2. Engineering: engineering has been defined to me in the past as the study of solving problems given economical constraints. In this thesis, the problem at hand is the extraction of information from bitemporal remote sensing image pairs.

3. Natural sciences: this is the study of objects, structures, behaviours and phenomena through observation and experimentation. The behavior of neural networks is somewhat unpredictable, and their study is often conducted through probing experiments, similarly to what is done in the natural sciences, rather than by mathematical proofs.

These scientific domains often intersect, but studies in each one tend to be approached in different ways. In mathematical sciences, ideas tend to come from first principles (axioms and previous theorems), and derivations follow rigorous rules. In engineering, the main concern is the problem at hand, and the performance of the proposed solution, often with no formal proofs. In natural sciences, experiments are done to better understand a given subject of interest.

Most of the work in this thesis stems from an original problem, the analysis of multitemporal remote sensing images. The methods that are presented in later chapters and in related research are often the result of a loop that iterates between proposing a solution to the problem at hand, and performing experiments to quantify the performance and better understand the workings of the proposed solution. It is important to understand that even a system whose operations are known and deterministically defined may give rise to behaviours that are hard to explain [START_REF] May | Simple mathematical models with very complicated dynamics[END_REF], and may have properties that can't be formally proven using formal axiomatic systems [G öd31]. Formal proofs regarding neural networks often lag behind the ideas that are proposed and tested experimentally. The universal approximation theorem, whose origins can be found in the works of Cybenko [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF] and Hornik [START_REF] Hornik | Approximation capabilities of multilayer feedforward networks[END_REF],

is one such case.

More precisely, this thesis touches three main areas of research: computer vision, machine learning, and remote sensing of the Earth. The motivations and applications start in the domain of remote sensing, where there is currently an abundance in data being acquired from several sources, and fast and accurate interpretation of such data has several impactful applications. Such interpretation is often done using computer vision and image analysis methods, thus many ideas from these fields are frequently applied in remote sensing research. Computer vision, in turn, often brings machine learning into the context of image analysis, especially in the past decade. These domains will be explored further in Chapter 2.

Objectives

The work presented in this thesis orbit around a central point: extraction of change information from bitemporal remote sensing image pairs using convolutional neural networks. This is approached from the following main points of view:

1. Data modalities: are there advantages to using multispectral images over RGB images for change detection?

2. Data semantics: how can we extract semantic information from the image pair to better understand the evolution of the imaged area?

3. Data availability: how can we mitigate the effect of label noise and cope with data scarcity when training convolutional neural networks?

4. Data heterogeneity: how can we create change detection systems that are robust in their generalization to previously unseen events?

The interest in designing automatic change detection systems comes from two main cases: large scale applications and rapid response to events. In both of these cases, the speed of manual image analysis becomes a problem. For example, during the Australian wildfires in 2019, illustrated in Fig. 1.3, remote sensing images were used to obtain up-to-date information about the current situation [START_REF] Boone | I made bushfire maps from satellite data, and found a glaring gap in australia's preparedness[END_REF]. Satellite imagery often provides a fast and accurate way of obtaining precise information over large areas. This is especially valuable for observing areas of difficult access, such as up high mountains, deep into forests, and polar regions.

The application of computer vision and machine learning methods to the analysis of multitemporal remote sensing imagery has a long research history. Chapter 2 contains a summary of the research that is directly related to the work presented in this thesis, as well as a theoretical basis which is necessary for their understanding. This theoretical recapitulation should provide the reader with the minimum knowledge necessary to understand the ideas presented in the subsequent chapters.

The original research that was conducted during this thesis is presented in Chapters 3, 4, 5, and 6. Chapter 3 describes how convolutional neural networks (CNNs) and fully convolutional neural networks (FCNs or FCNNs) can be used to perform change detection as a semantic segmentation problem in a fully supervised manner. Convolutional neural networks and fully convolutional neural networks architectures are trained from scratch using a novel change detection dataset. The impact of architecture variations is studied, as well as the difference in performance between CNNs and FCNs. Chapter 3 also contain comparisons between using the proposed networks with different combinations of bands of multispectral images, and the obtained results show that it is valuable to use bands outside the visible spectrum for change detection when they are available.

Chapter 4 explores the addition of semantic information into the previously presented change detection framework, and how multitask learning affects the networks' performances. A novel large scale high resolution semantic dataset is created to perform these experiments. Different approaches to this problem are analysed, from comparing predicted land cover maps for detecting changes up to a fully integrated FCN for performing simultaneous change detection and land cover mapping of the input images. A staged training scheme is proposed for the integrated multitask network that reduces the number of hyperparameters to be chosen and that improves the performance of the trained network.

Chapter 5 studies the impact of label noise on the learning process for change detection, and borrows ideas from weakly supervised learning to improve the accuracy of the networks' results. Label bias in the previously presented dataset that was automatically generated have a biasing effect on the predicted change maps by the networks.

An iterative training and label cleaning process is employed to increase the network's robustness to label noise, and it is coupled with a novel guided anisotropic filtering algorithm that better fits prediction to boundaries in the input images. Chapter 5 also explores how pixel-level change detection can be performed using only image-level labels as supervision in a weakly supervised setting. A fixed spatial attention layer is learned during the training for a classification network, and the synergy between this attention layer and the network's classifier improve the network's ability to localize changes. These results are also improved by using the guided anisotropic diffusion algorithm.

Chapter 6 describes an adversarial learning approach to bridge the domain gap between images of different types of changes, using various types of natural catastrophes as case studies. Different natural catastrophes can have very different visual appearances in remote sensing images, so neural networks trained to detect changes caused by one type of event don't generalise well to other catastrophes. Adversarial learning is used to project the images into a common latent space where change detection supervision comes from a source domain, and can be applied to images from the target domain. The proposed method is also tested on other domain adaptation tasks in computer vision to show its versatility.

Chapter 7 concludes this manuscript with the main conclusions drawn from the presented research. It also contains perspectives for future studies that could be conducted to continue this line of research.

Publications

Many of the contributions presented in this thesis have been published in peer reviewed venues. Below is a list of the publications and submissions relative to these works at the time of writing. Chapter 2

Journal Articles

Related Work

Isaac Newton famously said that if he saw further, it was by standing on the shoulders of giants [START_REF] Newton | Isaac newton letter to robert hooke 1675[END_REF], and this thesis, as all others, is no different. It is, first and foremost, a work of computer vision and image analysis, with an application in the field of remote sensing and using methods rooted in machine learning. This chapter aims to provide an introduction to the concepts most necessary to understand the contributions that will be presented in the following chapters, as well as a review of recent works that have inspired this work or aim to solve similar problems.

It is often impossible to draw sharp lines between the different scientific domains that are the main pillars of this work. The following sections aim to separate the theoretical foundations into meaningful parts, but the interplay between these research fields makes the information in these sections somewhat interdependent.

The sections below attempt to present the theoretical basis for this work coming from a problem solving perspective. First, a basic summary of the fields of computer vision and image analysis is presented. Then, the foundations of machine learning methods are presented, with special attention to image oriented techniques, along with various alternative data paradigms, as this will be a recurring theme throughout the following chapters. A summary of change detection techniques for remote sensing is then presented to contextualize this work among those that have already studied similar problems. Finally, a section containing definitions of the metrics used for quantitative evaluation of the proposed methods concludes this chapter.

Computer Vision and Image Analysis

It is widely accepted that the field of computer vision had its beginnings at MIT in the 1960s. In his PhD thesis presented in 1963, Lawrence G. Roberts attempted "to make it possible for a computer to construct and display a three-dimensional array of solid objects from a single two-dimensional photograph" [START_REF] Gilman | Machine perception of three-dimensional solids[END_REF]. In 1966, Seymour

Papert conducted "The Summer Vision Project", in which basic vision tasks (e.g. segmentation into 'likely objects', 'likely background areas', and 'chaos') were to be performed by a computer [START_REF] Seymour | The summer vision project[END_REF]. During this project, Marvin

Minsky is reported to have told his undergrad student Gerald J. Sussman to "spend the summer linking a camera to a computer and getting the computer to describe what it saw" [START_REF] Szeliski | Computer vision: algorithms and applications[END_REF]. It is safe to say Sussman did not completely solve this problem that summer.

Figure 2.2: Vision Memo No. 100 from the Artificial Intelligence Group at MIT contained the goals for the Summer Vision Project, including segmentation between objects and background, and scene analysis containing objects such as balls, bricks, cylinders, and other "objects of known sort".

Much of the expertise applied to computer vision came from the field of digital signal processing, specifically digital image processing. One way to separate these two fields of research is to focus on their outputs. Digital image processing tasks usually have their outputs in image form, e.g. image filtering, denoising, contrast enhancement, restoration, inpainting, etc. Computer vision studies the tasks where information is to be extracted from images, usually in a similar fashion to the human visual system. Tasks such as classification, segmentation, stereo depth perception, tracking, and many others are often easy for humans, but defining algorithms that perform these tasks from luminosity levels sampled on a grid is far from trivial.

But it is easy to see the motivation for this pursuit. Our visual system allows for us to sense our environment in an astonishingly precise way. In many tasks and benchmarks, human-level performance is implicitly considered as "the truth". In many others, human-level performance is a baseline performance that is rarely surpassed, except in speed. Even in very specific and well defined contexts, surpassing the human-level performance baseline is something that attracts attention [EKN + 17, FSR + 20].

Richard Szeliski provides a good description of the history of computer vision up to about 2010 in [START_REF] Szeliski | Computer vision: algorithms and applications[END_REF]. He identifies the origin of the difficulty in computer vision stems from it being a study of inverse problems "in which we seek to recover some unknowns given insufficient information to fully specify the solution". Creating a render from a 3D model, for example, can be done by following deterministic, well defined algorithms. Extracting a 3D model from a rendered image, on the other hand, is a much more poorly defined problem. Szeliski then claims physics-based and probabilistic models are then used to disambiguate between solutions.

Early computer vision research focused on mapping the structure of the 3D world using only images. Approaches such as the "blocks world" one proposed by Roberts used topological analysis of 2D lines identified in the images to model 3D objects [START_REF] Gilman | Machine perception of three-dimensional solids[END_REF]. Such approaches motivated research in line extraction techniques, which culminated in algorithms such as Canny's edge detector [START_REF] Canny | A computational approach to edge detection[END_REF] and active contour methods [START_REF] Kass | Snakes: Active contour models[END_REF]. Computer vision quickly branched out to several other application based on real world needs:

• Optical character recognition (OCR) for automated reading of books, letters, envelopes, etc.;

• Face recognition, fingerprint recognition, biometrics;

• Medical image analysis and computer aided diagnostics;

• Machine inspection for improving manufacturing quality control efficiency;

• Video surveillance for widespread security systems;

• Stereo vision for bioinspired 3D vision systems;

• Optical flow estimation, in which the movement of each pixel is estimated between a pair of images;

• Shape from X, which uses physics-based heuristics for shape estimation;

• Image segmentation, which aims to group pixels into meaningful regions coming from the same object;

• Image classification, where the class of an image is inferred among a group of known options;

• Object detection, where the position, number, and usually the class of objects in an image are extracted;

• Semantic segmentation, where each pixel in an image is classified between a set of known possible class options.

This list in no way covers all the problems studied in computer vision, but it illustrates the variety of inverse problems that our visual cortex performs almost effortlessly and that are very hard to solve using computers.

This work focuses mostly on semantic segmentation, while image classification is also a recurrent topic. Modern techniques have their origins in the older problem of image segmentation and pixel clustering. The main difference between image segmentation and semantic segmentation is, conveniently enough, semantics, as depicted in [START_REF] Bay | Surf: Speeded up robust features[END_REF] and ORB [START_REF] Rublee | Orb: An efficient alternative to sift or surf[END_REF]. Such descriptors were often coupled with machine learning methods such as the bag-of-words approach to perform object detection and image classification [START_REF] Sivic | Efficient visual search of videos cast as text retrieval[END_REF].

Neural networks, especially convolutional neural networks (CNNs) have come dominate many of the research topics in computer vision. Their reliance on data rather than handcrafted methods comes with positive and negative sides, as will be discussed in Section 2.2. Modern CNN-based approaches often couple the standard data-based approach with problem-specific heuristics to improve performance at specific problems.

Machine Learning

The field of machine learning studies algorithms that are able to improve its performance at a given task automatically using examples, experience, or data, which is usually referred to as "learning". Machine learning is often conflated with artificial intelligence, although modern machine learning research rarely discusses the philosophical ideas of what intelligence actually is or how to reproduce it. Research in the field of machine learning usually proposes solutions to a given problem, or attempts to better understand previously proposed algorithms.

The value of machine learning algorithms comes firstly from their versatility. In a way, these algorithms solve a meta-problem, and the solution to the specific problem at hand comes from their learning process. Let's take the problem of image classification as a toy example. Imagine an algorithm which was handcrafted to tell apart images of cats and dogs. Such algorithm is unlikely to be useful if we try to use it to separate images of rabbits and foxes, or cars and buses, or benign and malign skin growths. In this problem of image classification, a machine learning algorithm would solve the meta-problem of separating images into two groups, and the exact operations to do so would be extracted from examples. This way, the same algorithm can be used for solving several problems of a certain category.

This means that problems solved with machine learning methods are usually analysed from a farther point of view. These solutions don't require that the machine learning algorithm, and by extension the machine learning engineer, encompass the exact nature of the problem at hand as long as the general structure of the problem is understood. The need for expert knowledge becomes a need for data. Looking at the ears may be the key to differentiating between cats and dogs, but a machine learning engineer need never know that as long as the machine learning algorithm is able to learn that from data.

Most machine learning algorithms are defined by a set of deterministic operations that are performed to the input to attempt to perform the desired task 1 . These operations, which are determined by the machine learning engineer2 , use operation parameters that are learned automatically during the learning process. The process of solving problems with machine learning is then often seen as containing two main parts:

1. Construct an algorithm that, given the appropriate operation parameters, is in theory able to perform the task at hand.

2. Find the operation parameters that produce the desired outcome as well as possible.

There are several sources that provide detailed information on various machine learning systems and their history [START_REF] Mitchell | Machine Learning[END_REF][START_REF] Christopher | Pattern recognition and machine learning[END_REF][START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF][START_REF]Deep learning in neural networks: An overview[END_REF][START_REF] Goodfellow | Deep Learning[END_REF]. For the sake of simplicity, the review in this section focuses on the ideas and methods that support the work presented in the following chapters.

The following subsections will define from the ground up the basic forms or feedforward neural networks, convolutional neural networks, segmentation networks, and finally some alternative learning paradigms. Virtually all definitions that will be made could be appended with "but someone did it differently". For the sake of simplicity, this will only be acknowledged when found to be relevant.

Feedforward Neural Networks

The first model that we will analyse will be the perceptron, or feedforward neural network, first proposed by Frank Rosenblatt in 1957 [START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF][START_REF] Rosenblatt | Perceptron simulation experiments[END_REF]. The understanding of the perceptron begins with the artificial neuron, which was loosely inspired by neuroscience [START_REF] Goodfellow | Deep Learning[END_REF]. The neuron is a function that produces an output y ∈ R from an input vector x ∈ R n and parameters Φ. This is typically done through matrix multiplication, output bias, and a non-linear activation function, here denoted by g(•):

f (x; Φ) = g(w x + b), (2.1)
where Φ contains all the operation parameters, in this case w and b. Most non-linear activation functions don't require any parameters, although that is not always the case. Results discovered by George Cybenko and Kurt

Hornik show that functions with this form have powerful approximation capabilities.

A set of m neurons can be used in parallel to compute different activations from the same input using different parameters. This is straightforwardly done by simply adapting the operations and dimensions in Eq. 2.1:

f (x; Φ) = g(W x + b), (2.2) where x ∈ R n , b ∈ R m , W ∈ R n×m .
Such functions are often referred to as single layer perceptrons. Single layer perceptrons can be chained together to form multilayer perceptrons (MLPs). Given a set of p appropriately dimensioned single layer perceptrons f (i) (x; Φ (i) ), i ∈ {1, 2, . . . , d}, a multilayer perceptron can be defined as:

f (x; Φ) = f (d) (. . . f (1) (x; Φ (1) ) . . . ; Φ (d) ), (2.3)
where n (i) = m (i-1) , i ∈ {2, . . . , d}, n (i) being the number of inputs and m (i) the number of outputs of the i-th perceptron. The number of layers d is often referred to as the depth of the network, and networks with larger The activation functions g(•) are essential to allow the network to perform complex functions. It is easy to see that without the activation functions, the operations performed by all layers chained together (Eq. 2.3) would be equivalent to a linear algebra matrix operation. Common activation functions include:

• Sigmoid function: σ(z) = 1 1+exp(-z) .
• Hyperbolic tangent: tanh(z) = exp(2z)-1 exp(2z)+1 ,

• Rectified linear unit: ReLU(z) = max(0, z).

• Leaky rectified linear unit: LeakyReLU(z) = max(0, z) -α • max(0, -z), α is fixed.

• Parametric rectified linear unit: PReLU(z) = max(0, z) -α • max(0, -z), α is learnable.

The case in which a dataset D = {(x i , y i )}, i ∈ {1, . . . , N } containing N pairs of inputs and associated labels is available for training the machine learning algorithm is called supervised learning. In this case, we aim to find the weights Φ that allow the MLP to approximate the real relationship between x and y:

f (x; Φ) = ŷ ≈ y. (2.4)
This is usually accomplished by searching for Φ that minimises a cost or loss function L(ŷ, y) that measures the similarity between ŷ and y:

Φ = arg min Φ L(f (x; Φ), y).
(2.5)

Global minima are prohibitively costly to find for even moderately sized networks, so we usually settle for finding local minima during the training process. Assuming f and L use only differential operations, this is usually done by calculating the gradients using a backpropagation algorithm, an application of the chain rule for differentiation, and performing some form of gradient descent optimization on Φ. Attributing such procedure to a single author is not a trivial task, the reader is referred to Section 5.5 in [START_REF]Deep learning in neural networks: An overview[END_REF] for a historical summary of backpropagation methods for neural networks.

This form of iterative optimization needs a starting point Φ 0 from which the iterations start. It is necessary to introduce some level of randomness to this initialization of weights, although there are different ways to do so [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF]. The iterative optimization process can then be formalized as

Φ i+1 = Φ i + λ • ∇L(f (x; Φ), y)| Φi = Φ i + λ • ∂L(f (x; Φ), y) ∂Φ Φi , (2.6)
where λ is a tunable hyperparameter that controls the speed and stability of the optimization. There are several improvements to this basic algorithm that incorporate concepts of momentum and adaptable learning rates. A review of such algorithms can be found in [START_REF] Ruder | An overview of gradient descent optimization algorithms[END_REF].

In many cases, the size of the dataset D is too large to evaluate L, the gradients and optimization steps for all data points simultaneously. In these cases, subsets of the (x, y) pairs, called mini-batches, are considered at a time. The resulting problem can then be formalized as a stochastic optimization problem

Φ = arg min Φ E (x,y)∼D [L(f (x; Φ), y)], (2.7)
which can be solved by a stochastic gradient descent (SGD) method similarly to Eq. 2.6 by sampling data from D:

Φ i+1 = Φ i + λ • ∇L(f (x; Φ), y)| Φi = Φ i + λ • ∂L(f (x; Φ), y) ∂Φ Φi , (x, y) ∼ D.
(2.8)

Taking the case of classification as an example, assume D contains pairs of input vectors and associated labels (x, y). The number of considered classes is known to be C and each x is associated to one and only one class.

Classes are numbered from 1 to C. The ground truth output vectors y ∈ R C are one-hot encoded, which is to say

y i = I ci (n) =        1, if n = c i 0, if n = c i , (2.9)
where c i is the index of the class associated with x i .

In this case, it is useful to set the activation function g (d) of the final layer f (d) of the MLP to be the softmax activation function [START_REF] Christopher | Pattern recognition and machine learning[END_REF][START_REF] Goodfellow | Deep Learning[END_REF], which can be defined as

softmax(z) i = exp(z i ) j exp(z j )
.

(2.10)

The dimensionality of the MLP's output must also match the number of classes C. The output ŷi of the MLP for input x i can then be interpreted as predicted probabilities of it belonging to each of the considered classes.

We can then interpret y as a probability distribution, and define a loss function as the cross entropy between y and ŷ:

L(ŷ, y) = - j y j log(ŷ j ) = -log(ŷ c ).
(2.11)

The last part of the equation is a simplification that can only be done due to the one-hot encoding of the labels y for an element belonging to class c. Minimizing the loss function presented in Eq. 2.11 using the procedure described in Eq. 2.8 allows us to find parameters Φ that enables the MLP to predict the relationship between inputs and outputs in D, i.e. f (x, Φ) = ŷ ≈ y.

Nowadays, automatic differentiation computing libraries such as PyTorch [PGC + 17] and TensorFlow [ABC + 16] are able to perform all the operations described above including the calculation of gradients and optimizations, as well as many others. Such libraries are essential for the fast development of complex deep learning systems and are used throughout academia and industry.

Convolutional Neural Networks

The neural architecture described in Section 2.2.1 considers the elements of input x equally and ignores all positional information. While this is appropriate for many applications, image data usually comes in the form

I ∈ R M ×N ×C
, where M and N are the spatial 2D dimensions of the image and C is the number of colour channels (C = 3 for RGB images). Information in x comes not simply from the values attributed to each pixel, but mainly from the relationship between pixels. While images can simply be vectorized as vec(I) ∈ R M •N •C for it to be used as input for an MLP, this completely ignores the structure of the images and makes the MLP extremely sensitive to small translations, rotations, and other transformations. The number of pixels in common images are also large in comparison to what is feasibly tractable with MLPs.

The Neocognitron architecture helped tremendously with these two problems. It used operations that were limited to well defined neighbourhoods and with shared parameters [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition[END_REF]. The Neocognitron proposed by Fukushima et al. was not trained using the procedures described in Section 2.2.1. This was done by LeCun et al. in 1998 in their work in which convolutional neural networks (CNNs) were proposed [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. This was the first time that gradient-based learning was used in conjunction with weight-sharing operations that worked over a sampled 2D grid. This was done through the use of discrete convolutions, which are operations well studied and understood in the field of image processing and digital signal processing, mainly to their application to filtering and interesting properties in Fourier analysis [START_REF] Oppenheim | Discrete-time signal processing[END_REF].

Convolutions for Neural Networks

Two dimensional convolutions are used in CNNs to increase robustness to translations and to massively decrease the amount of learnable parameters compared to the matrix operations used in MLPs. The basic 2D convolutions used in CNNs can be defined as

I out [m, n] = K * I[m, n] = p i=-p q j=-q C c=1 I[m + i, n + j, c] • K[i, j, c],
(2.12)

where K ∈ R (2m+1)×(2n+1)×C is the convolution kernel that contains the learnable parameters. This convolution operation is not exactly equal to the one typically used in digital signal processing, but it follows the same basic idea of applying a multiply-accumulate operation over a sliding window. Padding of the input (zero padding, mirroring, etc.) is often done to maintain the spatial dimensions of the images after the convolution. Note that in this definition the output has a single "colour" channel (although the term "colour" loses its meaning at this point and becomes somewhat interchangeable with "feature"). Similarly to what was done in Eq. 2.2, a set of these operations can be done in parallel, and their results are considered together

I out [m, n, d] = K * I[m, n] = p i=-p q j=-q C c=1 I[m + i, n + j, c] • K[i, j, c, d], (2.13)
where K ∈ R (2m+1)×(2n+1)×C×F can be seen as a collection of F kernels. Combining these convolution operations with bias and activation function operations form what is referred to as convolutional layers.

Pooling operations are also a mainstay of CNNs. These operations aim to increase the robustness of the network to small translations and other spatial transforms, as well as to reduce the size of the calculated features to reduce memory costs and to increase the receptive field of the operations and allow the network to identify larger, more contextual features. The max-pooling operation can be defined as

I out [m, n, c] = max(I[i, j, c]), s.t. s • m ≤ i < s • m + p and s • n ≤ i < s • n + q, (2.14)
where s defines the stride of the operation and p and q define its window size, assuming an indexing that starts at 0.

It is also common to use average-pooling instead of max-pooling, where the maximum operation is replaced by the mean operation, or using convolutions with a stride larger than 1 to reduce the spatial resolution of feature maps.

A reduction in spatial resolution is often done along an increase in feature map space (i.e. number of convolutional filters).

Basic Convolutional Neural Network Architectures

Classification CNNs usually begin with several convolutional layers and pooling layers to calculate hierarchical feature maps, as seen in Fig. 2.5. The number of layers, the number of filters per layer (also known as layer depth), and position and properties of the pooling layers are architecture parameters that can be chosen based on factors such as how much computational power is available and how much training data is available, among other factors.

The output of convolutional and pooling layers does not match the necessary format for classification. This problem is solved by vectorizing the output feature map and feeding it into a classification MLP. Note that a network with this architecture can only process images of a single size. Modern classification CNNs solve this problem through adaptive pooling operations, whose output is of a fixed size regardless of the input size.

Siamese architectures have been proposed in the past to compare images using CNNs [BGL + 94, CHL05a].

In these architectures, the input images are processed separately at first by several convolutional and pooling lay- This is mainly a consequence of the multiplication of gradients through the chain rule for activation functions whose derivatives lie between 0 and 1, such as hyperbolic tangent, sigmoid. The result is that operations far from where the loss function is calculated (the output) are hard to optimize using gradient descent algorithms. The opposite problem of exploding gradients can also be harmful to the training of neural networks.

ers
Several techniques have been proposed to help mitigate these gradient problems when training deep neural networks. These include:

• Layer-wise unsupervised pretraining of networks [START_REF] Schmidhuber | Learning complex, extended sequences using the principle of history compression[END_REF].

• Deep supervision (i.e. using additional losses at intermediary points of the network) [SWY + 15].

• Normalization operations such as batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], group normalization [START_REF] Wu | Group normalization[END_REF], and instance normalization [START_REF] Ulyanov | Instance normalization: The missing ingredient for fast stylization[END_REF].

• Residual convolutional blocks [START_REF] He | Deep residual learning for image recognition[END_REF].

The main motivation for using deeper architectures is that this allows networks to learn more complex operations, as long as they are properly trained. This phenomenon has been repeatedly observed, which explains the modern interest in finding ways to handle larger and deeper architectures [SZ15, HZRS16a, HLvdMW17].

In the early 2010s, a breakthrough that in many ways triggered the modern explosion of research into CNNs was the insight of using graphics processing units (GPUs) for inference and training of neural networks. Their ability to perform massively parallelised calculations allow for these computations to happen much faster than on a regular CPU. While this is usually credited to Krizhevsky et al. [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], it is probably more appropriate to credit Cires ¸an et al. for this idea since they accomplished this one year earlier [CMM + 11]. Regardless, using GPUs and other task-specific hardware is now the standard way of handling the computations needed for large neural networks.

Fully Convolutional Neural Networks

Another problem that is efficiently solved by CNN variants is semantic segmentation [MBP + 20]. Given an image3

I ∈ R M ×N ×C ,
[i -s ≤ m ≤ i + s, j -s ≤ n ≤ j + s], Y[i, j])
for training the network. This requires virtually no change to the previous formulation except for rearranging the dimensions, but it is not a very efficient way of performing semantic segmentation.

Long et al. proposed the first architecture that came to be known as a fully convolutional network (FCN) [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] 4 .

It aimed to predict class probabilities for all the pixels in the input image simultaneously. Instead of feeding the feature maps calculated by convolutional and pooling layers into an MLP, the feature maps were upsampled back to the scale of the input image, thus retaining spatial information in the activations and performing pixel-wise predictions.

It was not only extremely accurate, but it took advantage of redundancies for the calculations of neighbouring pixels which sped up computation speeds compared to competing methods. The decrease in spatial resolution followed by upsampling is why these architectures are also sometimes referred to as encoder-decoder networks. . First, feature maps were upsampled using a transposed convolution (sometimes also written as a convolution with a fractional stride). These feature maps were then concatenated with ones from an earlier point of the network at the same level of spatial downsampling, as depicted in Fig. 2.9. These concatenation operation were named skip connections, since the feature maps "skipped" part of the network at a lower spatial resolution, thus retaining more accurate boundary position information.

Many of the improvements that are currently used in FCNs were first proposed to improve the performance of classification CNNs. Residual connections (or blocks) were proposed by He et al. [START_REF] He | Deep residual learning for image recognition[END_REF] with the motivation of facilitating the propagation of gradients through backpropagation and under the hypothesis that it is easier to learn identity weights using such operations. Densely connected blocks of convolutions were proposed by Gao et al. to allow the network to reuse previously calculated feature maps and reduce the number of parameters of the networks [START_REF] Huang | Densely connected convolutional networks[END_REF]. These ideas are often used in the design of FCN architectures such as was done in [ZTK + 18, ZWZ + 19], as can be seen in Fig. 2.10.

thing". How else could readers possibly know it is a deep learning paper? There several variations to the methods described above, as well as various topics that were not discussed (segmentation post-processing, segmentation variations such as instance segmentation, etc.). An in depth review of semantic segmentation using deep convolutional neural networks by Minaee et al. can be found in [MBP + 20].

Learning Paradigms

The previous sections focused on supervised learning methods. While this covers a large family of machine learning methods, several problems don't fit nicely into this framework. Two of the most common reasons to step out of the supervised learning paradigm are data cost and non-deterministic solutions. While the former is fairly straightforward, the latter is slightly more vague. While the answer to the question "is this a picture of a dog or a cat?" is quite clear (given an appropriate image), other problems can be more complex.

Unsupervised, Semi-Supervised, and Self-Supervised Learning

In some cases, the dataset at hand contains only raw data X with no associated outputs Y. Such data can still be used for training machine learning systems in different ways, such as for clustering [START_REF] Christopher | Pattern recognition and machine learning[END_REF] or representation learning [START_REF] Bengio | Representation learning: A review and new perspectives[END_REF], by extracting domain specific information from the data distribution itself. This is referred to as unsupervised learning, since there is no explicit form of supervision. Unsupervised pre-training of neural network was standard procedure for many years for several reasons, but such procedures are no longer seen as essential in cases where enough data is available [START_REF]Deep learning in neural networks: An overview[END_REF].

At times, only a subset of X is associated to a supervision signal Y. The field of semi-supervised learning studies how to best leverage unlabelled data to improve the performance of purely supervised systems trained using only the supervised part of the dataset [START_REF] Xiaojin | Semi-supervised learning literature survey[END_REF][START_REF] Jesper | A survey on semi-supervised learning[END_REF]. In many cases, leveraging all of the data, usually through an unsupervised learning auxiliary task, leads to better performing machine learning systems, especially in the case of data-hungry deep neural networks. One traditional way of doing so is to pre-train the networks in an unsupervised manner, then fine-tune the network's weights using the supervised data subset [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF][START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF]. Another approach is to consider a composite loss function that is composed of supervised and unsupervised components. The former is calculated only for data points with associated supervision signals, while the latter can be calculated using all the data points in the dataset [START_REF] Jesper | A survey on semi-supervised learning[END_REF].

Weakly Supervised Learning

Data for supervision of machine learning algorithms can come in many different levels of precision. For different levels of spatial classification of images, for example, one possible ladder of complexity is: 1. Image level labels (image classification).

Bounding-box level labels (object detection and classification).

3. Pixel level labels (semantic segmentation).

4. Incomplete pixel level and instance labels (instance segmentation).

Complete pixel level and instance labels (panoptic segmentation).

Higher level supervision usually enables a higher level of image understanding, but acquiring the necessary ground truth for such supervision comes at a similarly higher cost.

The field of weakly supervised learning studies how higher level tasks can be learned from lower level supervision. This allows for cheaper label generation as well as bringing supervision from already existing data from neighbouring domains. It is easy to see how performing semantic segmentation using only image level labels [START_REF] Ahn | Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation[END_REF],

bounding-box annotations [KBO + 16, KBH + 17, DHS15], scribbles [LDJ + 16]. Many weak supervision methods also aim to train networks using user generated tags and labels, which are often free to obtain through crowdsourcing on some online platforms.

Multitask Learning

Machine learning systems can be trained to perform more than one task at once. The term was coined by Richard Caruana in 1993 [START_REF] Caruana | Multitask learning: A knowledge-based source of inductive bias[END_REF], when he noticed evidence that suggested that it is more efficient to learn several tasks at once rather than separately. This topic has been studied in several contexts since [START_REF] Zhang | A survey on multi-task learning[END_REF]. Notably, the Taskonomy project [ZSS + 18] showed it was possible for a network to perform 26 2D, 2.5D, and 3D tasks simultaneously, as well as showing multitask learning resulted in a need for smaller amounts of labelled data points.

Label Noise

Labels used for supervision of classification and segmentation systems are often noisy to different degrees depending on the considered dataset. Automatically generated labels and user generated labels are especially prone to

Adversarial Learning

In 2014, Goodfellow et al. proposed the idea of generative adversarial networks (GANs) [GPAM + 14], which can be seen as a special case of the "adversarial artificial curiosity" according to J ürgen Schmidhuber [START_REF] Schmidhuber | Generative adversarial networks are special cases of artificial curiosity (1990) and also closely related to predictability minimization[END_REF]. The initial objective of GANs were to train generative models that were able to translate noise into realistic images. This could not be done through normal supervision, since there is no "correct" translation from noise to image. Training was done by pitting two networks against each other, as seen in Fig. 2.14, using contradictory or adversarial loss functions. The generator attempts to learn to generate realistic images from input noise in a way that such images trick the discriminator, while the discriminator's task is to correctly identify if an input image is real or was generated by the generator. The training is done my solving a minimax problem by iteratively performing gradient steps on the generator and the discriminator parameters. If properly trained, the generator and the discriminator have a symbiotic relationship where one's good performance helps the other to improve.

There have since been many improvements to this basic idea. Conditional GANs were used to generate images of specific classes [START_REF] Mirza | Conditional generative adversarial nets[END_REF], as well as to perform semantic semantic segmentation [START_REF] Luc | Semantic segmentation using adversarial networks[END_REF].

Change Detection Using Remote Sensing Images

As was introduced in Chapter 1, remote sensing is a vast field of study with various applications [START_REF] Elachi | Introduction to the physics and techniques of remote sensing[END_REF][START_REF] James | Introduction to remote sensing[END_REF][START_REF] Chen | Signal and image processing for remote sensing[END_REF]. While remote sensing can be generally defined as "the acquisition of information about an object without being in physical contact with it" [START_REF] Elachi | Introduction to the physics and techniques of remote sensing[END_REF], the works in this thesis is done considering the more restricted definition of remote sensing that deals with the acquisition, processing and analysis Earth7 observation images taken from an overhead perspective.

Such remote sensing imaging techniques enable us to accurately and frequently survey large areas. These images can be combined with geographical information systems (GIS) and global positional systems (GPS) to create what is usually referred to as geospatial data [START_REF] James | Introduction to remote sensing[END_REF]. Understanding the geographical area that is imaged by remote sensing systems is essential to interpreting a set of images together, as well as for correlating different images that were taken of the same region at different times of by different sensors.

Remote sensing images can have many types. Panchromatic images are grayscale photos that capture a broad range of wavelengths, typically at a higher resolution than other forms of image acquisition. High resolution panchromatic images are often combined with lower resolution multispectral images (including RGB) to produce high resolution multispectral images through the process of pansharpening [START_REF] Chen | Signal and image processing for remote sensing[END_REF]. The spectrum that is observed may be restricted to the human-visible spectrum (i.e. RGB images) or not (multispectral and hyperspectral images). Active methods such as SAR and LiDAR, where the object's response to an emitted electromagnetic signal is observed, are also commonplace in remote sensing. The studies in this thesis consider only RGB and multispectral images, but there is no reason to believe that the proposed ideas can't be used for other types of images.

Remote sensing image analysis has long used techniques and methods developed for generic image analysis and signal processing [START_REF] Chen | Signal and image processing for remote sensing[END_REF]. Recently, with the rise of deep learning techniques, these have also made their way into the field of remote sensing to solve problems such as road detection [START_REF] Mnih | Learning to detect roads in high-resolution aerial images[END_REF], automatic land classification and cartography [Aud18, WZL + 19], and many others [ZTM + 17].

Unlike several other problems in computer vision where it can be easy to define a well behaved use case for algorithms, remote sensing image analysis inherits the heterogeneity from the real world. Images can change drastically depending on seasonal variations, weather patterns, acquisition methods, and several other factors.

Handling these variations is one of the core challenges in remote sensing image analysis.

Figure 2.15: Naive change detection methods for producing and thresholding difference images tend to detect either too few or too many changes. Humans are very good at understanding difference based on the context, but the direct pixel colour difference is usually much less discriminative than our intuition suggests. Adaptive thresholding algorithms also assume there are changed and unchanged pixels in the image (often in comparable quantities), which is sometimes a flawed assumption, especially for the automated analysis of large image collections.

Standard Approaches for Change Detection

The ability to correlate information from different images of a given area taken at different times (and possibly by Many methods avoid the problem of the lack of data by using transfer learning techniques, i.e. using networks that have been pre-trained for a different purpose on a large dataset [START_REF] Arabi | Convolutional neural network features based change detection in satellite images[END_REF][START_REF] Arabi | Zoom out CNNs features for optical remote sensing change detection[END_REF]. While transfer learning is a valid solution, it is also limiting. Firstly, end-to-end training tends to achieve the best results for a given problem when possible [START_REF] Goodfellow | Deep Learning[END_REF]. Transfer learning also assumes that all images are of the same type. As most large scale datasets contain RGB images, this means that extra bands contained in multispectral images must be ignored. It will however be shown in Chapter 3 that using all available multispectral bands for change detection leads to better results.

Unsupervised Change Detection

Supervised Change Detection

Several works have used CNNs to generate the difference image that was described earlier, followed by tradi-

tional

Evaluation Metrics

It is important to define quantitative measures that allow us to precisely evaluate the performance of algorithms. In the binary classification case, consider y n ∼ Y to be the n-th label in the dataset and ŷn ∼ Ŷ to be the n-th label predicted by the considered algorithm, y n , ŷn ∈ {0, 1}. Each of the four possible combinations is given, as described in Tab. 2.1. An illustration applied to change detection can be seen in Fig. 2. 16. The matrix that is represented in Tab. 2.1 is often referred to as the confusion matrix, and it can be extended to a N-class problem by defining a square matrix C N ×N where c i,j represent the total number of elements where y = i and ŷ = j.

y n = 0 y n = 1 ŷn = 0 True negative (TN) False negative (FN) ŷn = 1 False positive (FP)
True positive (TP) Using the number of elements in each group we can calculate useful metrics that let us evaluate the performance of different methods. The first is the total accuracy of the algorithm, which can be defined as

Accuracy = T P + T N T P + T N + F P + F N . (2.15)
Accuracy (also referred to as overall accuracy, global accuracy, etc.) represents the fraction of correctly classified examples considering the whole dataset. The accuracy can range between 0 and 1, with 1 being the best possible value. Accuracy is a class agnostic metric, and can be applied to cases with any number of classes.

Precision is the measurement of how often are positive predictions correct. It can be defined as P recision = T P T P + F P .

(2.16)

High precision scores means we can be highly confident that the "positive" predictions are correct, but it does not provide any information about how many positives have not been detected by the algorithm. Precision scores can range between 0 and 1, with 1 being the best possible value. Precision is not a class agnostic metric, and in its simplest formulation is restricted to binary classification problems.

Recall is the measurement of how many of the positive examples have been detected by the classification algorithm. It can be defined as

Recall = T P T P + F N .
(2.17)

High recall scores means we can be highly confident that most of the positive examples are being detected, but it does not provide any information about whether or not the algorithm is also producing many false positives. Recall scores can range between 0 and 1, with 1 being the best possible value. Recall is not a class agnostic metric, and in its simplest formulation is restricted to binary classification problems.

The Dice score (also referred to as F1 score, or Sørensen-Dice coefficient) is a balance between the precision and recall values, and aims to be the a balanced metric of the performance of classification algorithms. It can be defined as the harmonic mean between precision and recall:

Dice = 2 1 P recision + 1 Recall = 2 • T P 2 • T P + F P + F N . (2.18)
A low score in either precision or recall will bring down the Dice score, which makes it a robust metric for quantifying the performance of classification algorithms. It is also better than the accuracy metric for evaluating algorithms in cases where there is a strong class imbalance, since the accuracy metric will be strongly biased towards the class with more examples. Dice scores can range between 0 and 1, with 1 being the best possible value.

The intersection over union (IoU) serves the same purpose as the Dice score, serving as a single metric that is lowered by the presence of either false positives or false negatives, while being more robust to class imbalances than overall accuracy. It can be defined as

IoU = T P T P + F P + F N . (2.19)
IoU scores can range between 0 and 1, with 1 being the best possible value. IoU is often used to evaluate multi-class classification and semantic segmentation problems by applying it to each class and averaging the results. This is referred to as mean IoU (mIoU).

The kappa coefficient (or Cohen's kappa coefficient) is a measurement of how well the true and predicted classes agree when compared to the probability of agreement given the class distributions. It can be defined as

κ = p o -p e 1 -p e , (2.20)
where p o is the observed agreement probability and p e is the expected agreement probability given the class distributions. In the two class problem, these can be defined as

p o = Accuracy = T P + T N T P + T N + F P + F N , (2.21)
and

p e = (T P + F P )(T P + F N ) + (T N + F P )(T N + F N ) (T P + T N + F P + F N ) 2 . (2.22)
The kappa coefficient can also be directly applied in a multiclass setting. Kappa scores can range between -1 and 1, with 1 being the best possible value.

Chapter 3

Supervised Change Detection Chapter Summary

This chapter presents a complete framework for performing change detection with convolutional neural networks (CNNs) and fully convolutional networks (FCNs) in a supervised manner. Two CNN architectures and three FCN architectures are proposed and compared using a novel dataset that was manually annotated for this purpose.

Results show that both Siamese and early fusion architectures are able to learn the necessary operations to perform change detection. The superiority of FCNs over patch-based CNNs in both performance and speed for such pixellevel tasks is clear in the presented results.

The dataset that was manually annotated for these tests contains Sentinel-2 multispectral images, which contains infrared and ultraviolet bands along with colour channels. We explore the usage of these bands for change detection, and observe the effect of using different numbers of spectral bands against the RGB baseline.

At the time this work was done, and to the best of our knowledge, it was the first time CNNs and FCNs were used for change detection in an end-to-end supervised manner. Previous work had utilised such networks simply for the creation of difference images or change vectors, which were then used along with more traditional methods to obtain the final change maps [EALW16, EALW17, ZFY + 17]. Such methods tend to assume that there are both changed and unchanged pixels in the image, and will perform poorly in cases where no changes are visible. As will be shown in Chapter 4, this is often the case when attempting to automatically analyse large areas, especially at higher resolutions.

The final contribution presented in this chapter is the multispectral change detection dataset that was created by manually annotating selected Sentinel-2 image pairs of urban areas in several different urban areas around the world. The dataset was openly released to serve as a benchmark for change detection methods and has since been used by many for developing and comparing change detection methods.

Introduction

Change Detection (CD) is one of the main problems in the area of Earth observation image analysis. Its study has a long history, and it has evolved alongside the areas of image processing and computer vision [Sin89, HCC + 13].

Change detection systems aim to assign a binary label per pixel or region based on a pair or sequence of coregistered images of a given region taken at different times. A positive label indicates the area corresponding to that pixel has changed between the acquisitions. While the definition of "change" may vary between applications, CD is a well defined pixel-level classification problem. Changes may refer, for example, to vegetation changes, urban expansion, polar ice melting, etc. Change detection is a powerful tool in the production of maps depicting the evolution of land use, urban coverage, deforestation, and other types of multi-temporal analysis.

Programs such as Copernicus and Landsat make available large amounts of Earth observation imagery. These can be used in conjunction with advanced supervised machine learning algorithms that have been on the rise for the past decade, especially in the area of image analysis. It is therefore of interest to find efficient ways to make use of the available data. In the context of change detection there is a lack of large annotated datsets, which limits the complexity of the models that can be used. Nevertheless, there are pixelwise annotated change detection datasets available that can be used to train supervised machine learning systems that detect changes in image pairs, such This is limiting in many ways, as it assumes similarities between these datasets and the relevant change detection data. For example, most of the large scale networks have been trained on RGB images, and cannot be directly applied to SAR or multispectral images, which is the case of the dataset presented later in this chapter. These methods also avoid end-to-end training, which tends to have better results for successfully trained systems. For this reason, we focus on algorithms that are able to learn solely from the available change detection data, and can therefore be applied to any datasets with available supervision. directly applicable to the task that is studied here. The dataset provides a comparison standard for single band, color or multispectral change detection algorithms that are proposed. Since it contains pixel-wise ground truth change labels for each location on each image pair, the dataset also allows for more elaborate supervised learning methods to be applied to the problem of change detection.

CNNs have been applied in various contexts

ONERA Satellite Change Detection Dataset

The ONERA Satellite Change Detection (OSCD) dataset was built using images taken by the Sentinel-2 satellites, which belong to the Copernicus program. The satellites capture images at various resolutions between 10m and 60m in 13 bands between ultraviolet and short wavelength infrared. Twenty-four regions of approximately 500x500 pixels at 10m resolution with various levels of urbanization where urban changes were visible were chosen in various countries and continents. The images of all bands were cropped according to the chosen geographical coordinates, resulting in 26 images for each region, i.e. 13 bands for each of the images in the image pair. These images were downloaded and cropped using the Medusa toolbox2 .

The high variability of the raw data that is available from the Sentinel-2 does not allow a completely scripted generation of image patches. The downloaded images frequently contain large sections of completely black pixels, and the correct images must be selected manually. Furthermore, for the generation of this dataset, it was desired to obtain images with no or very few clouds present in the image. While the sentinelsat API allows some control over the amount of clouds present in the images, this also requires manual verification of each of the downloaded images to ensure the presence of clouds in the downloaded image is not too large. The pixel-wise ground truth labels were manually generated by comparing the true color images for each pair.

Since the creation of this dataset, Papadomanolaki et al. have created an extended version of this dataset by adding appropriately cropped Sentinel-2 images that were acquired between the two dates for each of the regions [PVV + 19]. This addition extends the reach of the dataset beyond bitemporal image analysis, allowing a multitemporal analysis of image sequences to be performed.

Challenges and Limitations

While the dataset is a very valuable tool for methodically comparing different change detection algorithms and for applying supervised learning techniques to this problem, it is important to understand the limitations of this dataset.

First and foremost, the images generated by the Sentinel-2 satellites are not of very high resolution (10-60 m/px depending on the spectral band). This resolution allows the detection of the appearance of large buildings between the images in the image pair. Smaller changes such as the appearance of small buildings, the extension of existing buildings or the addition of lanes to an existing road, for example, may not be obvious in the images as they may occupy a single pixel. For this reason, even the change maps that are generated manually often differ according to its maker. is much agreement between the images, there is also a significant amount of disagreement. This difference comes from the difficulty in finding a clear definition for change that covers all situations, even when the images are being manually labelled. Some differences also come from slight differences on the exact locations of the boundaries of the changes. This means that it may not be reasonable to expect an algorithm to reach perfect quantitative results.

One approach which was explored for generating change maps in an automated manner was comparing Open-StreetMap data from different dates. OpenStreetMap provides open map data, and by comparing the maps for the dates of the images in the pairs it is, in theory, possible to identify what changes occurred in the area. This approach proved unsuccessful for a few reasons. First, most of the changes in the maps between the two dates were actually due to things being added to the map, but which had not been built in the period between the dates when the images were taken. Second, it is not possible to have much precision when it comes to the dates of the older maps, where in many cases only one map was available for each year before 2017.

Finally, the Sentinel-2 satellite was launched in 2015, and therefore the data that is available is not able to go further in the past than June of 2015. This means that the changes contained in the dataset are of a temporal distance of at most two years approximately since the dataset was created in late 2017. In practice, the actual temporal distance between acquisitions is often less than that due to the limited choice in images due to weather effects. This also means that the images contain many times more pixels labelled as no change, and fewer marked as change.

Further information about the OSCD dataset can be found in Appendix A.

Patch Based Architectures

Unlike previous methods which only use CNNs to build difference images which are later thresholded, our methods The second approach is a Siamese (Siam) network. The idea is to process each of the patches in parallel by two branches of four convolutional layers with shared weights, concatenating the outputs and using two fully connected layers to obtain two output values as before. This can be seen as projecting the images into a common latent space where a MLP can be used to detect changes. A schematic can be found in Fig. 3.5(b).

Fully Convolutional Architectures

The proposed fully convolutional architectures are an evolution of the networks presented in Section 3.3. Moving the patch-based architectures to a fully convolutional scheme improves accuracy and speed of inference without affecting significantly the training times. These fully convolutional networks are also able to process inputs of any sizes as long as enough memory is available for the computations, unlike the patch based approaches which require patches with dimensions of exactly 15x15xC.

To extend these ideas we used the concept of skip connections that were used to build the U-Net, which aimed to perform semantic segmentation of images [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. In summary, skip connections are links between layers at the same subsampling scale before and after the encoding part of an encoder-decoder architecture. are also shallower than their U-Net equivalents. As in the patch based EF model, the input of this network is the concatenation the two images in the pair that is to be compared.

The two other proposed architectures are Siamese extensions of the FC-EF model. To do so, the encoding layers of the network are separated into two streams of equal structure with shared weights as in a traditional Siamese network. Each image is given to one of these equal streams. The difference between the two Siamese architectures is only in how the skip connections are done. The first and more intuitive way of doing that is concatenating the two skip connections during the decoding steps, each one coming from one encoding stream. This approach was named Fully Convolutional Siamese -Concatenation (FC-Siam-conc, Fig. 3.6(b)). Since in CD we are trying to detect differences between the two images, this heuristic was used to combine the skip connections in a different way.

Instead of concatenating both connections from the encoding streams, we instead concatenate the absolute value of their difference. This approach was named Fully Convolutional Siamese -Difference (FC-Siam-diff, Fig. 3.6(c)).

Experiments

To evaluate the proposed methods we used two change detection datasets openly available. The first one is the proposed OSCD, and the second is the Air Change Dataset [BS09] (AC). AC contains RGB aerial images, while OSCD contains multispectral satellite images. The networks were also tested using only the RGB layers of the OSCD dataset. The two classes (change and no change) were assigned class weights inversely proportional to the number of pixels in each one. The available data was augmented by using all possible flips and rotations multiple of 90 degrees to the training patches. Dropout with p = 0.2 was used after each activation layer to help avoid overfitting during training. All experiments were done using PyTorch framework and with an Nvidia GTX 1070 GPU.

On the OSCD dataset, we split the data in train and test groups as defined in the dataset guidelines, i.e. 14 image pairs for training and 10 for testing. For the AC dataset, we followed the data split that was proposed in [ZFY + 17]:

the top left 748x448 rectangle of the Szada-1 and Tiszadob-3 images were extracted for testing, and the rest of the data for each location was used for training. This allowed direct comparison to three CD algorithms reported in [ZFY + 17]. Each location (Szada and Tiszadob) was treated completely separately as two different datasets, and the images named "Archieve" were ignored, since it contains only one image pair which is not enough data to train the models presented in this chapter. The DSCN methods was proposed by Zhan et al, in which convolutions are used to project the pixels into a 16dimensional space using a contrastive loss to lead the network to learn similar representations of unchanged pixels and distant representations of changed ones[ZFY + 17]. The table contains the precision, recall and F1 scores from the point of view of the "change" class, as well as the overall accuracy.

The results on the OSCD images show that the fully convolutional methods proposed in this chapter far outperform patch based the ones. While the patch based methods achieve good recall rates due to high numbers of predicted changes, they do poorly on the precision metric, which also reduces the F1 rate. Inference time of the fully convolutional architectures was under 0.1 s per image for all our test cases, while the patch based approach took several minutes to predict a change map for the whole image. On this dataset, the FC-Siam-diff obtained significantly better F1 scores than all other proposed methods, but the other fully convolutional architectures are still clearly superior to the patch based approaches. An illustration of our results on this dataset can be found in Figs. 3.7 and 3.8.Figure 3.9 contains the results for the fully convolutional architectures compared to the annotations of three different manual annotators to illustrate the various possible interpretations.

We then compared the fully convolutional architectures to other state-of-the-art change detection methods using the Air Change dataset. The results obtained on the Air Change dataset also show the superiority of our methods compared to previous ones. For the Szada/1 case, all proposed architectures outperformed the other methods used for comparison in the F1 metric, the FC-Siam-diff being once again the best architecture. For the Tiszadob/3 In these tests, the FC-Siam-diff architecture seems to be the most suited for change detection. This is, we believe, due to three main factors which make this network especially suited for this problem. First, fully convolutional networks were developed with the express purpose of dealing with dense prediction problems. Second, the Siamese architecture imbues into the system an explicit comparison between two images. Finally, the difference skip connections also explicitly guides the network to compare the differences between the images, in other words, to detect the changes between the two images.

The significant speedup of these fully convolutional networks with no loss in performance compared to patchbased methods is a step towards efficient processing of the massive streams of Earth observation data which are available through programs such as Copernicus and Landsat. These programs monitor very large areas with a high revisit rate. Deployment of systems such as these in conjunction with methods such as the ones proposed in this chapter could enable accurate and fast worldwide monitoring.

These results also validate presented dataset's capability of evaluating change detection algorithms. Despite the low number of image pairs and the small disagreements between experts when labelling some parts of the images, supervised learning was done without the need for transfer learning.

Conclusion

We presented the first Sentinel-2 pixel-level urban change detection dataset, which has been openly available, along with methods used for its generation, and the main challenges that were faced. We also presented two patch based and three fully convolutional networks trained end-to-end from scratch. The latter surpassed comparable change detection methods at the time it was developed, both in accuracy and in inference speed without the use of post-processing. Most notably, the fully convolutional encoder-decoder paradigm was modified into a Siamese architecture, using skip connections to improve the spatial accuracy of the results.

A natural extension of the work presented on this chapter would be to evaluate how these networks perform when attempting to detect semantic changes, as will be seen in the next chapter. It would also be interesting to test them with other image modalities (e.g. SAR images), and to attempt to detect changes in image sequences, as was done in [PVV + 19].

High Resolution Semantic Change Detection Dataset

Images

The dataset contains a total of 291 RGB image pairs of 10000x10000 pixels. These are mosaics of aerial images taken by the French National Institute of Geographical and Forest Information (IGN). Each image pair contains an earlier image acquired in 2005 or 2006, and a second image acquired in 2012. They come from a database named BD ORTHO which contains orthorectified aerial images of several regions of France from different years at a resolution of 50 cm per pixel. The 291 selected image pairs are all the images in this database that satisfy the conditions for the labels, which will be described below. The images cover a range of urban and countryside areas around the French cities of Rennes and Caen. Only the images in the regions mapped in the Urban Atlas project and with a maximum temporal distance of one year from either 2006 or 2012 were kept in the dataset.

The dataset contains more than 3000 times more annotated pixel pairs than either OSCD or Air Change datasets.

Also, unlike these datasets, the labels contain information about the types of change that have occurred. This is much more data than was previously available in the context of change detection and it opens the doors for many new ideas to be tested. The amount of labelled pixels and surface area for land cover classification is also about 8 times larger in the proposed HRSCD dataset than in the DeepGlobe Land Cover Classification dataset [DKL + 18],

both of the datasets containing images of the same spatial resolution (50 cm/px).

The BD ORTHO images provided by IGN are available for free for research purposes, but not all images can be redistributed by the users. That is the case for the images taken in 2005 and 2006. Nevertheless, we have made available all the data for which we have the rights of redistribution and the rasters that we have generated for semantic change detection and land cover mapping. The dataset also contains instructions for downloading the remaining images that are necessary for using the dataset directly from IGN's website.

Labels

The labels in the dataset come from the European Environment Agency's Copernicus Land Monitoring Service -Urban Atlas project. It provides "reliable, inter-comparable, high-resolution land use maps" for functional urban areas in Europe with more than 50000 inhabitants. These maps were generated for the years of 2006 and 2012, and a third map is available containing the changes that took place between those dates. The available land cover maps contain several semantic classes, which are in turn organised in different hierarchical levels. By grouping the labels at different hierarchical levels it is possible to generate maps that are more coarsely or finely divided. For example, grouping the labels with the coarsest hierarchical level yields five classes (plus the "no information" class) shown in Table 4.1. This hierarchical level will henceforth be referred to as L1.

Code

These maps are openly available in vector form online. We have used these vector maps and the georeferenced BD ORTHO images to generate rasters of the vector maps that are aligned with the rasters of the images. These rasters allow us to have ground truth information about each pixel in the dataset.

It is important to note that there are slight differences in the semantic classes present in Urban Atlas 2006 and in Urban Atlas 2012. These differences do not affect the L1 hierarchical grouping and therefore had no consequence in the development of this dataset. It may nevertheless affect future work done with the data. We leave it up to the users how to best interpret and deal with these differences. More information is provided in the dataset files.

Dataset Analysis

Despite its unprecedented size and qualities, we acknowledge in this section the dataset's limitations and challenges. Nevertheless, we will show later that despite these limitations, the dataset allows for the boundaries of the state-of-the-art in semantic change detection through machine learning to be pushed.

One issue is the accuracy of the labels contained in the Urban Atlas vector maps with respect to the BD ORTHO images. We do not have access to the images used to build the Urban Atlas vector maps, nor to the exact dates of their acquisitions, nor to the dates of acquisition of the images in BD ORTHO. Hence, there are some discrepancies between the information in the vector maps and in the images. Furthermore, the European Environment Agency only guarantees a minimum label accuracy of 80-85% depending on the considered class. Most of the available data is accurate, but it is important to consider that the labels in the dataset are not flawless. Examples of false negatives and false positives can be see in Fig. 4.2 (d)-(f) and Fig. 4.2 (g)-(i), respectively. It is also worth noting that the labels have been created using previously known vector maps, mostly by labelling correctly each of the known regions. This means a single label was given to each region, and this led to inaccurate borders in some cases. This can be clearly seen in Fig. 4.2 (a)-(c).

One of the main challenges involved in using this dataset for supervised learning is the extreme label imbalance.

As can be seen in Table 4.2, 99.232% of all pixels are labelled as no change, and the largest class is from agricultural areas to artificial surfaces (i.e. class 2 to class 1), which accounts for 0.653% of all pixels. These two classes together account for 99.885% of all pixels, which means all other change types combined account for only 0.115% of all pixels. Furthermore, many of the possible types of change have no examples at all in any of the images of the dataset. It is of paramount importance when using this dataset for supervised learning and algorithm evaluations to take into account this imbalance. This also means that using the overall accuracy as a performance metric with this dataset is not a good choice, as it virtually only reflects how many pixels of the no change class have been classified correctly. Other metrics, such as Cohen's kappa coefficient or the Sørensen-Dice coefficient, must be used instead. This class imbalance is characteristic of real world large scale data, where changes are much less frequent than unchanged surfaces. Therefore, this dataset provides a realistic evaluation tool for change detection methods, unlike carefully selected image pairs with large changed regions. The problem of supervised learning using noisy labels has already been studied and evidence suggests that supervised learning with noisy labels is possible as long as a dataset of a large enough size is used [START_REF] Rolnick | Deep learning is robust to massive label noise[END_REF]. Other works attempt to explicitly deal with the noisy labels present in the dataset and prioritise the correct labels during training [START_REF] Maggiolo | Improving maps from cnns trained with sparse, scribbled ground truths using fully connected crfs[END_REF].

Finally, we acknowledge how challenging it is to use hierarchical levels finer than L1 due to: 1) a massive increase in the number of possible changes, and 2) the difference between similar classes becomes more abstract and context based. For example, the difference between the "Discontinuous Medium Density Urban Fabric" and the the surface at a given pixel (e.g. building or grass), but also by understanding the surroundings of the pixel and calculating the ratio between these two classes at a given neighbourhood that is not clearly defined.

Methodology

Binary Change Detection

We have already showed in the previous chapter the efficacy of using three different architectures of fully convolutional neural networks for change detection [START_REF] Caye Daudt | Fully convolutional siamese networks for change detection[END_REF].

[COA18] also proposed a fully convolutional architecture for change detection that is very similar to one of the three initially proposed architectures. In both of these works, FCN architectures performed better than previous methods for supervised change detection.

Building on this previous work, we have modified the FC-EF architecture to use residual blocks, as proposed by [START_REF] He | Deep residual learning for image recognition[END_REF]. The resulting network is later referred to as FC-EF-Res, and is depicted in Fig. 4.3. These residual blocks were used in an encoder-decoder architecture with skip connections to improve the spatial accuracy of the results [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. Residual blocks are used here to facilitate the training of the network, which is especially important for its deeper variations that will be discussed later. For testing on the OSCD dataset, the size of the network has been kept approximately the same that of FC-EF to avoid overfitting. When using the proposed HRSCD dataset (Section 4.3.2), the larger amount of annotated pixels allows us to use deeper and more complex models. In that case, the number of encoding levels and residual blocks per level has been increased, but the idea behind the network is the same as of FC-EF-Res.

Change Semantics

As was mentioned previously, the efficiency of the proposed architecture for binary change detection and the availability of the HRSCD dataset enable us to tackle the problem of semantic change detection. We consider this problem as two separate but related tasks. The first task is binary change detection, i.e. we attempt to determine whether a change has occurred at each pixel in a co-registered multi-temporal image pair. The second task is to differentiate between types of changes. In our case, this consists of predicting the class of the pixel in each of the two given images. The problem of semantic change detection lies in the intersection between change detection and land cover mapping.

Below we will describe four intuitive strategies to perform semantic change detection using fully convolutional networks. Starting from the plain comparison of land cover maps, we then develop more advanced strategies.

These strategies vary in complexity and performance, as will be discussed in Section 4.3.

Strategy 1: Direct Comparison of Land Cover Maps

The problem of automatic land cover mapping is a well studied problem. In particular, methods involving CNNs have recently been proposed, yielding good performances [START_REF] Audebert | Semantic segmentation of earth observation data using multimodal and multi-scale deep networks[END_REF][START_REF] Audebert | Machine learning for classification of big remote sensing data[END_REF]. When the land cover information is available, as it is the case in the HRSCD dataset, the most intuitive method that can be proposed for semantic change detection would be to train a land cover mapping network and to compare the results for pixels in the image pair (see Fig. 4.4(a)).

The advantage of this method is its simplicity. When using the HRSCD dataset we can assume changes occurred where the predicted class label differs between the two images, and the type of change is given by the predicted labels at each of the two acquisition moments. The weakness of this method is that it heavily depends on the accuracy of the predicted land cover maps. While modern FCNs are able to map areas to a good degree of accuracy, there are still many wrongly predicted labels, especially around the boundaries between regions of different classes.

Furthermore, when comparing the results for two acquisitions the prediction errors would accumulate. This means the accuracy of this change detection algorithm would be lower than the land cover mapping network, and would likely predict changes in the borders between classes simply due to the inaccuracy of the network.

Strategy 2: Direct Semantic Change Detection

A second intuitive approach is to treat each possible type of change as a different and independent label, considering semantic change detection as a simple semantic co-segmentation along the lines of what has been done to binary change detection in the previous chapter, but with an increased number of output classes: one for no change, and one for each type of change.

The weakness of this method is that the number of change classes grows proportionately to the square of the number of considered land cover classes. This, combined with the class imbalance problem that was discussed earlier, proves to be a major challenge when training the network.

Strategy 3: Separate Land Cover Mapping and Change Detection

Since it has been proven before that FCNs are able to perform both binary change detection and land cover mapping, a third possible approach is to train two separate networks that together perform semantic change detection (see

Strategy 4: Integrated Land Cover Mapping and Change Detection

The last of the proposed approaches is an evolution of the previous strategy of using two FCNs for the tasks of binary change detection and land cover mapping. We propose to integrate the two FCNs into a single multitask network (see Fig. 4.4(d) and Fig. 4.5) so that land cover prediction activations can be used for change detection.

The combined network takes as input the two co-registered images and outputs three maps: the binary change map and the two land cover maps.

In the proposed architecture, information from the land cover mapping branches of the network is passed to the change detection branch of the network in the form of difference skip connections, which was previously shown to be the most effective form of skip connections for Siamese FCNs. The weights of the two land cover mapping branches are shared since they perform an identical task, allowing us to significantly reduce the number of learned parameters.

This multitask network gives rise to a new issue during the training phase. Given that the network outputs three different image predictions, it is necessary to balance the loss functions from these results. Since two of the outputs have exactly the same nature (the land cover maps), it follows from the symmetry of these branches that they can be combined into a single loss function by simple addition. The question remains on how to balance the binary change detection loss function and the land cover mapping loss function to maximise performance.

We have proposed and tested two different strategies for training the network. The first and more straightforward approach to this problem is to minimise a loss function that is a weighted combination of the two loss functions. This

L λ (Φ Enc,CD , Φ Dec,CD , Φ Enc,LCM , Φ Dec,LCM ) = L(Φ Enc,CD , Φ Dec,CD ) + λ • L(Φ Enc,LCM , Φ Dec,LCM ) (4.1)
where Φ represents the various network branch parameters, and L is a pixel-wise loss function. Here, the pixel-wise cross entropy function was used. The problem then becomes the search for the value of λ that leads to the best balance between the two loss terms. This can be found through a grid search, but the test of each value of λ is done by training the whole network until convergence, which is a slow and costly procedure. This will later be referred to as Strategy 4.1.

To reduce the aforementioned training burden, we propose a second approach to train the network that avoids the need of setting the hyperparameter λ. We train the network in two stages. First, we consider only the land cover mapping loss

L 1 (Φ Enc,CD , Φ Dec,CD , Φ Enc,LCM , Φ Dec,LCM ) = L(Φ Enc,LCM , Φ Dec,LCM ) (4.2)
and train only the land cover mapping branches of the network, i.e. we do not train Φ Enc,CD or Φ Dec,CD at this stage. Since the change detection branch has no influence on the land cover mapping branches, we can train these branches to achieve the maximum possible land cover mapping performance with the given architecture and data.

Next, we use a second loss function based only on the change detection branch:

L 2 (Φ Enc,CD , Φ Dec,CD , Φ Enc,LCM , Φ Dec,LCM ) = L(Φ Enc,CD , Φ Dec,CD ) (4.3)
while keeping the weights for the land cover mapping Φ Enc,LCM and Φ Enc,LCM fixed. This way, the change detection branch learns to use the predicted land cover information to help to detect changes without affecting land cover mapping performance. This will later be referred to as Strategy 4.2.

Results

Multispectral Change Detection

We first evaluate the performance of the proposed FC-EF-Res network. As explained in Section 4.2.1, this network is an evolution of the fully convolutional architecture FC-EF proposed in Chapter 3, to which residual blocks have been added in place of traditional convolutional layers.

The FC-EF-Res architecture was compared to the previously proposed FCN architectures on the OSCD dataset for binary change detection, which contains lower-resolution Sentinel-2 image pairs with 13 multispectral bands.

As expected, the residual extension of the FC-EF architecture outperformed all previously proposed architectures.

The difference was noted on both the RGB and the multispectral cases. On the RGB case, the improvement was of such magnitude that the change detection performance on RGB images almost matched the performance on multispectral images. The results can be seen in Table 4.4. This corroborates the claims made by [START_REF] He | Deep residual learning for image recognition[END_REF] that using residual blocks improves the training performance of CNNs. For this reason, all subsequent networks that are tested with the HRSCD dataset use residual modules.

Very High Resolution Semantic Change Detection

To test the methods proposed in Section 4. 4.5, and illustrative image results can be seen in Fig. 4.6 and the other examples included at the end of this chapter. The networks were trained using minibatches of four 600x600 patches for a total of 500 epochs.

The ADAM optimizer was used with an initial learning rate of 10 -3 , which was reduced by a factor of 10 every 100 epochs. All tests were done using the PyTorch framework.

As is the case for most deep neural networks, the training times for the proposed methods are significantly larger than the testing times. Once the network has been trained, its fast inference speed allows it to process large amounts of data efficiently. The proposed methods took 3-5 hours of training time using a GeForce GTX 1080 Ti GPU with 11GB of memory. Inference times of the proposed methods were under 0.04 s for 512x512 image pairs using the same hardware.

In Strategy 1, which naively attempts to predict change maps from land cover maps, we can see that the network succeeds in accurately classifying the imaged terrains, but this is not enough to predict accurate change maps. The change detection kappa coefficient for this strategy is very low, which means this method is marginally better than chance for change detection.

The results for Strategy 2 are a fair improvement over those of Strategy 1. The change detection Dice coefficient and the land cover mapping results for this method are not reported due to its nature, since Dice coefficients can only be calculated for binary classification problems, and this strategy bypasses the land cover mapping steps.

Despite achieving a higher kappa coefficient, the network learned to always predict the same type of change where changes occurred. This means that despite using appropriately tuned class weights, the learning process did not succeed in overcoming the extreme class imbalance present in the dataset. In other words, the network learned to detect changes but no semantic information was present in the results.

For Strategy 3, the land cover mapping network that was used was the same as that of Strategy 1, which achieved good performance. A binary change detection network was trained to be used for masking the land cover maps. The performance of this network was better than that of Strategy 1 but worse than that of Strategy 2. The in Fig. 4.5 all at once damages performance in both change detection and land cover mapping.

In Fig. 4.6 we can see the results of the proposed networks on a pair of images from the dataset. Note the amount of false detections by Strategy 1 due to the lack of accuracy of prediction of the land cover maps on region boundaries. The second row shows the predicted classes at each pixel for each image. The semantic information about the changes comes from comparing these two predictions. For example, comparing the images in Fig. 4.6

(k) and (o) we can say that the changes predicted in (g) were from the "Agricultural areas" class to the "Artificial surfaces" class.

In our tests we observed that the trained networks had the tendency to overestimate the size of the detected changes. It is likely that this happens simply due to the nature of the data that was used for training. The labels in the HRSCD dataset, which come from Urban Atlas, mark as a change the whole terrain where a change of class happened. This means that not only the pixels associated with a given change are marked as change, but the neighbouring pixels that are in the same parcel are also marked as change. This leads to the networks learning to overestimate the boundary of the detected changes in an attempt to also correctly classify the pixels surrounding the detected change. This once again illustrates the challenges of the HRSCD dataset. Otsu thresholding to produce the final change maps (CNNF-O), but we have found that such approach leads to overestimating changes. We therefore tuned a fixed threshold (T = 2300) using a few example images and used that value to test the algorithm on all test data (CNNF-F), which significantly increased its performance by reducing false positives. Also included are the results by the method proposed in [START_REF] Celik | Unsupervised change detection in satellite images using principal component analysis and k-means clustering[END_REF], which performs principal component analysis (PCA) and k-means clustering on the pixels to detect changes in an unsupervised manner. Both algorithms perform worse than the proposed method on the HRSCD dataset.

ReCNN

To evaluate the size of the dataset, we have also tested Strategy 4.2 using reduced amounts of data for training the network. The kappa coefficient, in percent, obtained by using the whole training dataset is 25.49. This value is reduced to 23.34 by using half the training data, and is further reduced to 22.18 by using a quarter of the data. This

shows that, as expected, using more data for training the network leads to better results. Nonetheless, it also shows that the dataset is large enough to allow for even more complex and data hungry methods to be trained using the HRSCD dataset in the future.

Finally, it is important to note that the label imperfections in the HRSCD dataset occur not only in the training images, but also in the test images. This means that the performance of the proposed methods may be even higher than the numbers suggest, since some of the disagreements between prediction and ground truth data are actually due to errors in the ground truth data.

Eppalock Lake Images

We compare our method in this section to the one proposed by [START_REF] Mou | Learning spectral-spatial-temporal features via a recurrent convolutional neural network for change detection in multispectral imagery[END_REF], which used recurrent convolutional neural networks for change detection. In that work, pixels were randomly split into train and test sets. We believe that this split leads to overfitting since neighbouring pixels contain redundant information. This is especially true when using CNNs, which take as inputs patches centred on the considered pixels, meaning the network sees the same information for training and testing. It is likely that overfitting takes place, since an accuracy of over 98% is achieved by using only 1000 labelled pixels to train a network with 67500 parameters (for their long short-term memory (LSTM) architecture, which performed the best). The data consists of a single image pair of 631x602 pixels only partially annotated, with a total of 8895 annotated pixels which is much less data than what is required for deep learning methods. The HRSCD dataset presented in Section 4.1 contains over 3 million times more labelled pixels than the Eppalock lake image pair. Despite the flaws of this testing scheme, we have followed it to achieve a fair comparison between the methods.

Using the CNN architecture labelled EF presented in the previous chapter, we have achieved excellent numeric results thanks to this overfitting phenomenon, which discouraged the usage of more complex methods which would lead to even more extreme overfitting. The results achieved by the EF network were better for binary change detection and equivalent for semantic change detection compared to ReCNN-LSTM. The results can be seen in Table 4.6.

Conclusion

This chapter presented the first large scale very high resolution semantic change detection dataset, which has been released to the scientific community. This dataset contains 291 pairs of aerial images, together with aligned rasters for change maps and land cover maps. This dataset allows for the first time for much more complex deep learning methods to be used in this context in a supervised manner than what was possible using the previously available smaller datasets. We have then proposed different methods for using deep FCNs for semantic change detection.

The best among the proposed methods is an integrated network that performs land cover mapping and change detection simultaneously, using information from the land cover mapping branches to help with change detection.

We also proposed a sequential training scheme for this network that avoids the need of tuning a hyperparameter, which circumvents a costly grid search. Chapter 5

Weakly Supervised Change Detection Chapter Summary

In the previous chapter, the HRSCD dataset was presented. Due to the automatic way it was created, i.e. combining two unrelated databases for images and labels, the generated change maps in the dataset were somewhat noisy and biased. Not only can false positives and false negatives be found, but change labels were often larger than the changed object, since the annotations came from parcel-level vector data.

In this chapter, concepts are borrowed from the field of weakly supervised learning, drawing a parallel between these imprecise annotations and bounding box annotations, which are also imprecise when considered at pixel level.

The aim here is to extract the correct information from the annotations, while ignoring the incorrect labels. This is also connected to the problem of supervision using noisy data.

We propose an iterative training method that uses the network predictions, once training has converged, to try to isolate bad labels from good ones or to correct badly labelled samples. Training is then resumed using the cleaner version of the dataset until convergence, and this process is repeated. This training method is coupled with a novel post-processing method named guided anisotropic diffusion, which applies an anisotropic diffusion algorithm to the softmax activations with diffusion coefficients calculated using the input images. This serves as a way to better fit predictions to boundaries present in the input images, which helps reduce the effects of the biased overlarge predictions of the network trained on the original reference data.

Finally, true weakly supervised semantic co-segmentation from image-level labels is performed for the first time, to the best of our knowledge. Using building-centred image patch pairs with binary classification labels, a Siamese network with a novel spatial attention layer is used for this purpose. The attention layer allows the network to learn more localised features, which in turn helps the attention layer to further narrow down the area of interest in a positive feedback loop during the training procedure.

Change Detection with Unreliable Data

The HRSCD that was presented in the previous sections is the first large scale change detection dataset of its kind ever released. It was generated by combining an aerial image database with open change and land cover data. Change maps and land cover maps were generated for almost 30 billion pixels, over 3000 times larger than previous change detection datasets. This dataset, however, contains unreliable labels due to having been generated

automatically. The effects of naively using these data for supervised learning with change detection networks are shown in Fig. 5.1. Inaccuracies in the reference data stem primarily from two causes: imperfections in the vector data at different semantic levels, and temporal misalignment between the annotations and the images. Naive supervision using such data leads to overestimation of the detected changes, as can be seen in Fig. 5.1(e). Nevertheless, there is much useful information in the available annotations that, if used adequately, can lead to better CD systems.

Due to the way the ground truth was generated, the labels in the dataset mark changes at a land parcel level with imprecise boundaries. While useful for global monitoring of changes in land cover, it cannot delineate precise object-level changes. In order to achieve a precise pixel-wise change detection, we propose a weakly supervised learning approach to change detection. We consider the parcel-wise reference data as approximations, similar to bounding-boxes, of an ideal unknown ground truth corresponding to changes at pixel level. For each parcel with detected changes, the reference data in HRSCD contained both good and bad labels. For this reason, the noise in the labels is not randomly distributed, but it is conditioned on the pixels' neighborhoods and highly structured.

Other change detection datasets rely on cross referencing data obtained by on-site surveys with available remote sensing imagery. Such is the case of the ABCD dataset [FSI + 17], which contains image pairs centered on buildings in a region that has been affected by a tsunami. Images before and after the event were taken with different sensors, and were registered and cropped around each known building in the area. Binary change labels for each image pair are available, but segmentation labels are not. This dataset contains over 8000 labelled image pairs, and is available in two versions: fixed scale, where the spatial resolution of the images is kept constant, and resized, where images are resized so that the length of the imaged building takes up roughly a third of the patch size.

Noisy labels for supervised learning is a topic that has already been widely explored [FK + 14, FV14]. In many Results generated by such methods often lack in spatial accuracy and benefit from post-processing methods to increase the precision of the predictions, especially around region boudaries. Post-processing methods that use information from guide images to filter other images, such as semantic segmentation results, have also been proposed [PSA + 04, KCLU07, FRR + 13]. A notable example is the Dense CRF algorithm proposed by Kr ähenb ühl and Koltun, in which an efficient solver is proposed for fully connected conditional random fields with Gaussian edge potentials [START_REF] Kr | Efficient inference in fully connected crfs with gaussian edge potentials[END_REF]. The idea of using a guide image for processing another is also the base of the Guided Image Filtering algorithm proposed by He et al. [START_REF] He | Guided image filtering[END_REF], where a linear model that transforms a guide image into the best approximation of the filtered image is calculated, thus transferring details from the guide image to the filtered image. The use of joint filtering is popular in the field of computational photography, and has been used for several applications [PSA + 04, KCLU07, FRR + 13]. One of the building blocks of the filtering method we propose in this chapter is the anisotropic diffusion, proposed by Perona and Malik [START_REF] Perona | Scale-space and edge detection using anisotropic diffusion[END_REF], an edge preserving filtering algorithm in which the filtering of an image is modelled as a heat equation with a different diffusion coefficient at each edge between neighbouring pixels depending on the local geometry and contrast. However, to the best of our knowledge, this algorithm has not been used for guided filtering previously to this work.

Method

The main contributions of this chapter are: 1) the guided anisotropic diffusion algorithm, which uses information from the input images to filter and improve semantic segmentation results, 2) an iterative training scheme that aims to efficiently learn from inaccurate and unreliable ground truth semantic segmentation data, and 3) a learned spatial attention layer that improves classification and weakly supervised semantic segmentation with class activation maps for datasets with object aligned crops. These contributions are described in detail below. While these ideas are presented in this chapter in the context of change detection, the proposed methods' scope is broader and they could be used for other semantic segmentation problems, together or separately.

Guided Anisotropic Diffusion

In their seminal paper, Perona and Malik proposed an anisotropic diffusion algorithm with the aim of performing scale space image analysis and edge preserving filtering [START_REF] Perona | Scale-space and edge detection using anisotropic diffusion[END_REF]. Their diffusion scheme has the ability to blur the inside of regions with homogeneous colours while preserving or even enhancing edges. This is done by modelling the filtering as a diffusion equation with spatially variable coefficients, and as such is an extension of the linear heat equation, whose solution is mathematically equivalent to Gaussian filtering when diffusion coefficients are constant [START_REF] Koenderink | The structure of images[END_REF]. Diffusion coefficients are set to be higher where the local contrast of the image is lower.

More precisely, we consider the anisotropic diffusion equation

∂I ∂t = div(c(x, y, t)∇I) = c(x, y, t)∆I + ∇c • ∇I (5.1)
where I is the input image, c(x, y, t) is the coefficient diffusion at position (x, y) and time t, div represents the divergence, ∇ represents the gradient, and ∆ represents the Laplacian. In its original formulation, c(x, y, t) is a function of the input image I. To perform edge preserving filtering, one approach is using the coefficient

c(x, y, t) = 1 1 + ||∇I(x,y,t)|| K 2 , (5.2)
which approaches 1 (strong diffusion) where the gradient is small, and approaches 0 (weak diffusion) for large gradient values. Other functions with these properties and bound in [0, 1] may also be used. The parameter K Algorithm 1 Guided Anisotropic Diffusion pseudocode.

1:

Input:I 1 , I 2 , I i n, N , K, λ 2: Output:I f 3: I f ← I i n 4: for (i ← 1; i ≤ N ; i + +) do 5:
for (I j = {I 1 , I 2 }) do 6:

∇I j ← Calculate gradient of I j 7:

c Ij ← Calculate diffusion coefficients using Eq. 5.3 8:

I j ← I j + λ • ∇I j • c Ij 9:
end for 10:

∇I f ← Calculate gradient of I f 11:
c f ← Calculate diffusion coefficients using Eq. 5.4

12:

I f ← I f + λ • ∇I f • c f 13:
end for controls the sensitivity to contrast in the image.

In the guided anisotropic diffusion (GAD) algorithm, the aim is to perform edge preserving filtering on an input image, but instead of preserving the edges in the filtered image we preserve edges coming from a separate guide image (or images). Doing so allows us to transfer properties from the guide image I g into the filtered image I f . An illustrative example is shown in Fig. 5.2, where the image of a cathedral (a) is used as a guide to filter the image of a rough segmentation (b). The edges from the guide image I g are used to calculate c(x, y, t), which in practice creates barriers in the diffusion of the filtered image I f , effectively transferring details from I g to I f . These edges effectively separate the image in two regions, inside and outside the region of interest, and the pixel values in each of these regions experience diffusion, but there is virtually no diffusion happening between them.

Our primary aim is to use this GAD algorithm to improve semantic segmentation results based on the input images. Weakly supervised learning methods are often used when there is an overestimation of the target area: either the whole image is the starting point in classification to segmentation tasks, or the reference region is too large in parcel to region segmentation tasks. GAD provides a way to improve these semantic segmentation results by making them more precisely fit the edges present in the input images. A few design choices were made to extend the anisotropic diffusion from gray level images to RGB image pairs. The extension to RGB image was done by taking the mean of the gradient norm at each location

c I (x, y, t) = 1 1 + C∈{R,G,B} ||∇I C (x,y,t)|| 3•K 2 , (5.3)
so that edges in any of the color channels would prevent diffusion in the filtered image. To extend this further to be capable of taking multiple guide images simultaneously, which is necessary for the problem of change detection, the minimum diffusion coefficient at each position (x, y, t) was used, once again to ensure that any edge present in any guide image would be transferred to the filtered image:

c I1,I2 (x, y, t) = min i∈{1,2} c(I i )(x, y, t).
(5.4)

Guided anisotropic diffusion aims to improve semantic segmentation predictions by filtering the class probabilities yielded by a fully convolutional network. It is less adequate to correct for large classification mistakes, as opposed to non-local methods such as Dense CRF, but it leads to smoother predictions with more accurate edges. It can also be easily extended for any number of guide images by increasing the number of images considered in Eq. 5.4. The pseudocode for the GAD algorithm can be found in Algorithm 1. As mentioned in the original anisotropic diffusion paper, the algorithm is unstable for λ > 0.25 when using 4-neighborhoods for the calculations. For more information the reader can refer to the mathematical derivations presented in [PM90, AK06].

In the following sections, we show two ways to use GAD to improve weakly supervised semantic segmentation and reduce the effect of label noise. First, we address in Section 5.2.2 the inaccurate labelling problem with an iterative data cleansing scheme. Second, in Section 5.2.3 we use GAD to learn to segment changes from classification labels only.

Iterative Training Scheme

The label noise present in parcel-based change detection datasets such as the HRSCD dataset is challenging due to its spatial structure and correlation between neighbors. In the taxonomy presented in [FK + 14, FV14], this type of label noise would be classified as "label noise not at random" (NNAR). NNAR is the most complex among the label noise models in the taxonomy. In the case of HRSCD, most errors can be attributed to one of the following reasons:

the available information is insufficient to perform labelling, errors on the part of the annotators, subjectiveness of the labelling task, and temporal misalignment between the databases used to create the HRSCD dataset.

It is important to note that, as discussed by Fr énay and Kab án in [FK + 14], label noise has an even more powerful damaging impact when a dataset is imbalanced since it alters the perceived, but not the real, class imbalance and therefore the methods used to mitigate class imbalance during training are less effective. In the case of change It has been noticed in this chapter and in the previous one that change detection networks trained directly on the HRSCD dataset had the capacity to detect changes in image pairs but tended to predict blobs around the detected change instances, as is depicted in Fig. 5.7(c), likely in an attempt to minimize the loss for the training images where the surrounding pixels of true changes are also marked as having experienced changes. In many cases, it was observed that the network predictions were correct where the ground truth labels were not. Based on this observation, we propose a method for training the network that alternates between actual minimization of a loss function and using the network predictions to clean the reference data before continuing the training. A schematic that illustrates the main ideas of this method is shown in Fig. 5.3. For the remainder of this chapter, the iteration cycles of training the network and cleaning of training data will be referred to as hyperepochs. Alternating between training a semantic segmentation network and using it to make changes to the training data has already been explored [DHS15, KBH + 17]. Such iterative methods are named "classification filtering" [START_REF] Fr | Classification in the presence of label noise: a survey[END_REF].

Original GT Pred. 0 1 0 0 0 1 0 1 (a) Intersection Original GT Pred. 0 1 0 0 2 1 0 1 (b) FN← Ignore Original GT Pred. 0 1 0 0 2 1 2 1 (c) FN∪FP← Ignore
The main differences between the method proposed in this chapter and previous ones are:

1. No bounding box information is available: we work directly with pixel level annotations, which were generated from vector data; with iterative training. The first is using post-processing techniques that bring information from the input images into the predicted semantic segmentations, improving the results and providing a stronger correlation between inputs and predictions. The GAD algorithm presented in Section 5.2.1 serves this purpose, but other algorithms such as Dense CRF [START_REF] Kr | Efficient inference in fully connected crfs with gaussian edge potentials[END_REF] may also be used. The second way the degradation of results is avoided is by combining network predictions with the original reference data at each iteration, instead of simply using predictions as reference data.

We propose three ways of merging the original labels with network predictions. When merging, each pixel will have a binary label from the original ground truth and a binary label from the network prediction. If these labels agree, there is no reason to believe the label for that pixel is wrong, and it is therefore kept unchanged. In case the labels disagree, the following options to decide the pixel's label are proposed:

1. The intersection of predicted and reference change labels is kept as change: this strategy assumes all changes are marked in both the reference data and in the prediction. It also puts pixels with uncertain labels in the no change class, where they are more easily diluted during training due to the class imbalance. In practice, the ignored pixels are marked as a different class that is given a class weight of 0 during the training.

Tables for the three proposed methods can be found in Fig. 5.4, and an example can be found in Fig. 5.5.

Scene-Invariant Spatial Attention Layer

Many datasets in remote sensing contain georeferenced data. In many cases, patches are cropped from large images using the coordinates of known objects for which a label is known. In the case of the ABCD dataset [FSI + 17],

for example, ground surveys were used to identify which buildings had been damaged by a tsunami, and a dataset was created using crops centred on each of the buildings that were surveyed.

In such cases, objects to which the labels refer are located in the center of the images, while the characteristics of their surroundings are not directly related to the available labels. Pooling techniques such as max pooling and average pooling that are very often used in CNNs are invariant with respect to the image position. These operations fail to make use of the heuristic described above, and do not learn to prioritize some areas of the image over others when making classification predictions.

Attention mechanisms have been successful in applications such as machine translation [VSP + 17] and classattention for classification of aerial images [START_REF] Hua | Recurrently exploring class-wise attention in a hybrid convolutional and bidirectional lstm network for multi-label aerial image classification[END_REF]. We propose a learned spatial-attention layer that can be used to allow the network to learn which positions of the images are more discriminative and should be prioritized over others when making inferences. We also propose to use the GAD algorithm to further focus the attention of the network on the most relevant features. Let's assume that a feature map x of size C × M × N is obtained after any number of convolution, pooling and other operations from an input image (or images in the case of change detection), where C is the number of channels and M and N are spatial dimensions. We define a matrix A of size M × N which will be learned by the network, and will serve as attention weights given to spatial locations. The attention operation f (x) can then be defined as

f (x) c,i,j = α • x c,i,j • σ(a i,j ), (5.5)
where a i,j denotes the element of A in position (i, j), σ denotes the sigmoid function and α is a normalization term defined as

α = M i=1 N j=1
σ(a i,j ).

(5.6)

The sigmoid function is used to ensure the attention weights given to each spatial location is in the range (0, 1).

The matrix A is initialized as a null matrix so that all spatial locations have equal attention values of σ(a i,j ) = 0.5.

Random initialization of A is neither necessary nor recommended. The proposed attention layer is designed to be used after a softmax operation and before a global average pooling (GAP) layer. Doing so will force the network to produce per-pixel classification predictions, which are then put through a weighted average operation whose weights are learnable parameters which depend only on the spatial position of each feature. Global average pooling is preferable to max pooling at the end of a network when we want the network to be able to localize objects, as was discussed in [ZKL + 16]. Note that the number of learnable parameters introduced by this attention layer is only M • N , which is extremely small in the context of deep neural networks. At inference time, the learned attention weights can be further adapted to the input images by using the GAD algorithm proposed in Section 5.2.1. This helps the network to focus its attention at the building at the center of the image pair, further increasing classification performance.

In this work, we incorporate this attention layer into the classification branch of the architecture depicted in Fig. 5.6. The images are processed by two convolutions with stride 1 2 and 5 residual blocks before their features are merged, and 4 residual blocks after. This architecture allows us to perform either classification or segmentation by choosing either of the paths at the end. This architecture is a straightforward Siamese extension of the ideas presented in [ZKL + 16] with residual blocks. Supervision is only available for the classification branch, but the structure of the network allows us to apply equivalent classification operations at each spatial locations by avoiding the attention and global average pooling layers, effectively performing semantic segmentation.

Experiments

The experiments presented in this chapter have been divided into two sections. The first ones explore how the proposed methods allow us to refine approximate labels to obtain pixel-level change detection more accurately than through direct supervision. Then, we present experiments that show the effectiveness of the spatial attention layer in performing weakly supervised change detection, using only image-level labels to perform pixel-level predictions.

The GAD algorithm is used in both cases to improve the obtained results.

Label Refinement Through Iterative Learning

To validate the iterative training scheme proposed in Section 5.2 we adopted the hybrid change detection and land cover mapping fully convolutional network presented in Chapter 4, since it was already proven to work with the HRSCD dataset. We adopted strategy 4.2, in which the land cover mapping branches of the network are trained before the change detection one to avoid setting a balancing hyperparameter. The land cover mapping branches of the network were fixed to have the same parameter weights for all tests presented in this section, and evaluating those results is not done here as the scope of these experiments is restricted to the problem of change detection refinement.

We applied the GAD algorithm to the predictions from a network trained directly on the reference data from HRSCD to evaluate its performance. In Fig. 5.7 there is an example of the obtained results. As noted before, we can see in (c) that the change is detected but unchanged pixels around it are also classified as changes by the network. In (d)-(f) it can be clearly seen how the GAD algorithm improves the results by diffusing the labels across similar pixels while preserving edges from the input images in the semantic segmentation results. As expected, more iterations of the algorithm lead to a stronger erosion of "out-of-bounds" labels. For these results, GAD was applied with K = 5 and λ = 0.24. In Fig. 5.8 we can see a comparison between GAD and the Dense CRF1 algorithm [START_REF] Kr | Efficient inference in fully connected crfs with gaussian edge potentials[END_REF]. While the non-local nature of fully connected CRFs is useful in some cases, we can see the results are less precise and significantly noisier than the ones obtained by using GAD.

To perform quantitative analysis of results, it would be meaningless to use the test data in the HRSCD dataset given that we are attempting to perform a task which is not the one for which ground truth data are available, All tests presented here were done using PyTorch [PGC + 17]. At each hyperepoch, the network was trained for 100 epochs with an ADAM algorithm for stochastic optimization [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF], with learning rate of 10 -3 for the first 75 epochs and 10 -4 for the other 25 epochs. The tests show the performance of networks trained with the proposed method for 5 hyperepochs (iterations of training and cleaning the data), where the first one is done directly on the available data from the HRSCD dataset. For accurate comparison of methods and to minimize the randomness in the comparisons, the obtained network at the end of hyperepoch 1 is used as a starting point for all the methods.

This ensures all networks have the same initialization at the point in the algorithm where they diverge. A baseline network was trained for the same amount of epochs and hyperepochs but with no changes done to the training data.

This serves as a reference point as to the performance of the fully convolutional network with no weakly supervised training methods.

The first comparison, shown in Fig. 5.9(a), compares the three methods proposed in Section 5.2.2 to combine the network predictions with the original ground truth from the HRSCD dataset. We notice that all three strategies surpass the baseline network using the proposed iterative training method, which validates the ideas presented earlier. In Fig. 5.9(b) we see a comparison between a training using the full training scheme proposed in this chapter (without the usage of an ignore class) and the same method but without using the original reference data, i.e.using only network predictions processed by GAD to continue training at each hyperepoch. Our results, which corroborate the ones in [KBH + 17], show that referring back to the original data at each hyperepoch is essential to avoid a degradation in performance.

In Fig. 5.9(c) we show a comparison between using the proposed GAD algorithm versus the Dense CRF [START_REF] Kr | Efficient inference in fully connected crfs with gaussian edge potentials[END_REF] algorithm in the iterated training procedure, as well as using both together. We see that using the Dense CRF algorithm to process predictions leads to good performance in early hyperepochs, but is surpassed by GAD later on. This is likely explained by the non local nature of Dense CRF and its ability to deal with larger errors, but its inferior performance relative to GAD for finer prediction errors.

Figure 5.10 shows the predictions by networks trained by different methods on two example images. We see that the best results are obtained by using the full training scheme with GAD in (d), followed by Dense CRF, which also achieves good results shown in (f). The baseline results in (c), obtained by naively training the network in a supervised manner, and the ones without using the reference data as constraint in the iterative training scheme shown in (e) are significantly less accurate than those using GAD or Dense CRF.

Scene-Invariant Spatial Attention Layer

We tested the proposed method using the ABCD dataset proposed by Fujita et al. [FSI + 17]. This dataset contains pairs of crops of images centered on buildings that have been surveyed to evaluate their destruction after a tsunami.

We have followed the 5-fold cross validation that was defined by the dataset's creators. All networks were trained from scratch using only the ABCD dataset, using an initial learning rate of 0.005 for 10 epochs, then with a linearly decaying learning rate for 90 epochs for a total of 100 epochs. The classification results for these tests are presented in Table 5.1. These results show that our network with the attention module performed very similarly to the ones presented in [FSI + 17]. It is also clear that the proposed attention module improved the classification accuracy of the networks significantly. The obtained results also show that filtering the attention weights using the GAD algorithm further increases the classification performance of the proposed network, improving the quality of the attention weights by using the input images as guides.

Figure 5.12 show the learned spatial attention weights learned in each of the performed tests. We can clearly see how consistent the network was in identifying that the most discriminative region of the images was located in the center. It is also apparent that the network identified that the scale of this discriminative region is larger in the resized version of the dataset compared to the fixed-scale version.

Qualitative analysis of segmentation results show that the usage of the proposed spatial attention operation allowed the network to vastly increase its capacity to localize features in the input images, which led to much more accurate segmentation, as depicted in Fig. 5.13. The results also show how using the GAD algorithm for postprocessing further increased the spatial accuracy of the segmentation results.

These results suggest that there is a positive feedback loop that happens during the training process between the network's ability to localize discriminative features and the spatial attention operation. Once the network develops the ability to roughly localize discriminative features, this allows the training of the spatial attention layer, which leads the network to learn even more local features, and so on.

Two notable examples can be seen in Fig. 5.13. The first one is the example in the fourth row, which shows that the network is not simply finding buildings in the second image and marking those as unchanged. In this example, a building is present in the second image but it is marked as a change nonetheless since it doesn't match the buildings Figure 5.12: Spatial attention weights that were learned in each of the cross-validation tests. Top row contains all 5 tests using fixed scale ABCD dataset, bottom row are the results using the rescaled version of the dataset. Note that the network was incredibly consistent in identifying the center of the images as most discriminative without any explicit knowledge. These attention matrices are of size 40 × 40. in the first image. The second notable example is the one showed in the last row, where a very small change was detected in the center of the image, surrounded only by unchanged buildings. Since the position of this detected change coincided to the spatial attention position, the network was able to mark this image pair as a change, which is correct according to the ground truth label. The same was not accomplished by the network without the attention layer.

Method

Analysis

The experiments presented in the previous section showed how GAD was successfully used in two different weakly supervised change detection settings. The results show an increase in performance in object-level segmentation from parcel-level labels through label cleaning, as well as the seldom explored task of weakly supervised image co-segmentation using classification labels.

The iterative training results made clear that it is of paramount importance to refer back to the ground truth data Figure 5.13: Results obtained by using the proposed method. Note that when the attention layer is not used, the network does not learn to localize the features and tends to predict all pixels into the same class. The attention layer enables the network to localize features much more accurately, and the GAD post-processing further increases the spatial accuracy of such predictions.

every time the training ground truth is being modified. Not doing so leads to a fast degradation in performance, since the network simply attempts to learn to copy itself and stops learning useful operations from the data. The results also showed that separating dubiously labelled pixels leads to a small increase in performance, likely due to the fact that we end up providing a cleaner and more trustworthy dataset at training time.

The guided anisotropic diffusion algorithm was compared against the Dense CRF algorithm for using information from the input images to improve semantic segmentation results. While both algorithms were successful when used in the proposed iterative training scheme, GAD outperformed Dense CRF at later hyperepochs for quantitative metrics. Both algorithms yielded visually pleasing results, each performing better in different test cases.

One possible criticism of the proposed iterative training method is that it would get rid of hard and important examples in the training dataset. It is true that the performance of this weakly supervised training scheme would likely never reach that of one supervised with perfectly clean data, but the results in Section 5.3 show that using the proposed method we can consistently train networks that perform better than those naively trained with noisy data directly.

The proposed spatial attention operation was showed to be useful in improving the classification and weakly supervised segmentation results for datasets which are cropped using object locations as reference points. While this is a particular case, such datasets are often available or can be easily generated for remote sensing applications, where georeferenced data is widely available. The proposed ideas have been only tested in a two-class problem, but there is nothing that indicates that such methods would not work just as well in a multi-class context. Filtering the attention weights with the GAD algorithm further increased the classification performance of the network by increasing the coherence between the attention weights and the region where the building of interest is located in each image.

Conclusion

In this chapter we have proposed the guided anisotropic diffusion algorithm for improving semantic segmentation results by performing a cross-image edge preserving filtering. We have proposed two GAD-based weakly supervised change detection methods to demonstrate how it can help to recover from inaccurate segmentation labels or go beyond the available classification labels.

We first proposed an iterative training method for training networks with noisy data that alternates between training a fully convolutional network and leveraging its predictions to clean the training dataset from mislabelled examples. We showed that the proposed method outperforms naive supervised training using the provided reference data for change detection. The GAD algorithm was used in conjunction with the iterative training method to obtain the best results in our tests. The GAD algorithm was compared against the Dense CRF algorithm, and was found to be superior in performance.

We then proposed a spatial attention operation that can be easily incorporated into existing classification networks that significantly improve the classification and weakly supervised segmentation performances for datasets with object-aligned crops.

The proposed methods are useful when using data-based approaches in data-scarce domains, as is the case of change detection. We have observed improvement in all of our tests when approaching the problem from a weakly supervised perspective, as opposed to naive supervision. This is a step towards more precise systems for accurately measuring and quantifying the surface areas of changes and the evolution of image terrains. The value of this would be to more efficiently extract information from noisy and lower complexity data and to perform analyses The main contribution that is presented in this chapter is a domain adaptation method, named domain invariant encoding, that aims to find a latent representation space where images from either domain are indistinguishable.

The desired task is then performed using the projections of the images in this latent space. Unlike previous methods, feature space alignment is done indirectly through cycle consistent adversarial learning with the addition of a simple L1 similarity loss.

The proposed method is validated in three main experiments:

• Image classification: digit recognition across different datasets, including handwritten and house numbers.

• Semantic segmentation: synthetic to real street scenes interpretation.

• Change detection: damaged building detection using wildfire, hurricane, and tsunami images. . We refer the reader to [START_REF] Csurka | Domain adaptation for visual applications: A comprehensive survey[END_REF] for a more in-depth review of visual domain adaptation methods.

Motivation

Formulation

The unsupervised domain adaptation method we propose in this chapter is based on the CycleGAN framework [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF], where image-to-image translation is learned from unpaired data using cycle-consistent adversarial training. In this section we will briefly summarize the ideas from the CycleGAN framework that are essential to our UDA algorithm. We will then explain in detail how our method learns to perform domain-invariant encoding from unpaired data. and B, respectively. This is done by solving the following problem:

Cycle-Consistent Unpaired Image-to-Image Translation

F * A→B , F * B→A = arg min F A→B ,F B→A max D A ,D B L(F A→B , F B→A , D A , D B ) , (6.2)
where L is composed of adversarial and cycle consistency terms

L(F A→B , F B→A , D A , D B ) = L cyc (F A→B , F B→A ) +L GAN (F A→B , D B ) + L GAN (F B→A , D A ) . (6.3)
The adversarial term L GAN (F A→B , D B ) is defined as

L GAN (F A→B , D B ) = E I A ∼A [log(1 -D B (F A→B (I A )))] +E I B ∼B [log(D B (I B ))] . (6.4)
The symmetric term L GAN (F B→A , D A ) can be obtained by simply flipping the domains in Eq. 6.4. The LSGAN loss function can also be used here instead [MLX + 17]. Finally, the cycle consistency loss term L cyc (F A→B , F B→A ) is defined as

L cyc (F A→B , F B→A ) = E I A ∼A [||F B→A (F A→B (I A )) -I A || 1 ] +E I B ∼B [||F A→B (F B→A (I B )) -I B || 1 ] . (6.5)

Domain-Invariant Encoding

The main idea behind our method is to condition the translation functions to pass through a latent code that is shared between domains. This code can be obtained from the real or translated images, as well as be decoded into either of them. In the CycleGAN framework, F A→B and F B→A are encoder-decoder CNNs. We begin by looking at the encoder (Enc) and decoder (Dec) sections of the networks separately2 :

F A→B (I) Dec A→B (Enc A→B (I)) . (6.6)
This allows us to directly observe the code generated by Enc A→B (I). We aim to condition the network to find a code that is agnostic to the domain of origin of the image. Thus, the code corresponding to the A → B translation should be the same as the code corresponding to the B → A translation:

Enc A→B (I) ≈ Enc B→A (Dec A→B (Enc A→B (I))) . (6.7)
To condition the network to find this domain agnostic encoding, the following code similarity loss functions are defined: L cs,B (Enc B→A , Dec B→A , Enc A→B ) =

L cs,A (Enc A→B , Dec A→B , Enc B→A ) = E I A ∼A [||Enc B→A (Dec A→B (Enc A→B (I A ))) -Enc A→B (I A )|| 1 ] (
E I B ∼B [||Enc A→B (Dec B→A (Enc B→A (I B ))) -Enc B→A (I B )|| 1 ] . (6.9)
Note that this formulation does not increase the number of trainable parameters in the system as a feature discriminator would, nor does it increase the training complexity in any way. It is also important that the code generated by the encoders fully encode the input images, therefore techniques frequently used in encoder-decoder architectures such as skip connections [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF] should not be used.

We now assume that we want to train a network T to perform a task for which supervision labels Y are only available for the data in domain A. The network F takes as input the features obtained previously, i.e.T (Enc A→B (I)).

In the later examples, this task will be image classification, semantic segmentation, and image pair co-segmentation.

The applications of our method are not restricted to these cases, and it could in principle be used for other pattern recognition tasks (e.g.optical flow estimation). The CycleGAN networks are then trained simultaneously with T , which helps Enc A→B and Enc B→A learn task-specific features. For the classification case, the cross-entropy loss function is used:

L task (Enc A→B , T ) = -E (I A ,y)∼(A,Y ) K k=1 1 k=y log(σ(T k (Enc A→B (I A )))) (6.10)
where σ denotes the softmax function, 1 is an indicator function, and K is the number of considered classes. The pixel-wise cross entropy used for semantic segmentation and co-segmentation is nearly identical, except that it is applied at each pixel instead of once for the whole image. Note that this supervised loss is a function of Enc A→B , and therefore has an effect on the CycleGAN networks, helping the encoder learn task specific features.

The total loss function for this problem can then be defined as

L DIN E = L cyc,A + L cyc,B + L GAN,A + L GAN,B + L cs,A + L cs,B + L task , (6.11)
where the function arguments have been omitted for simplicity. In practice, different loss function terms are weighted by hyperparameters that are used to tune the method's performance. We aim to train the networks in an adversarial manner by solving

Φ * = arg min Φ max Ψ L DIN E , (6.12)
where Φ = {Enc A→B , Dec A→B , Enc B→A , Dec B→A , T } and Ψ = {D A , D B }.

Finally, one possible variant of the proposed method is to share the encoder weights between Enc A→B and Enc B→A . In this case, the network attempts to learn domain-invariant operations that extract similar features from each domain, instead of learning domain specific operations that project the images into the same latent space.

Shortcut Decoding

The proposed method opens the door to a validation method that we will refer to as shortcut decoding. Given that the forward and backward translations are supposed to pass through the same domain agnostic code, we can verify feature-space alignment by cross-decoding the encoded image:

I A,shortcut = Dec B→A (Enc A→B (I A )) . (6.13)
The obtained result can then be visually analyzed to qualitatively assess feature-space alignment, or directly compared to I A to obtain quantitative measurements.

Note that an auxiliary loss function comparing I A and I A,shortcut could be used but is not necessary, since feature space alignment implies accurate shortcut decoding, but the opposite is not true. Using a code similarity is therefore more adequate.

Implementation

We adopt for the CycleGAN networks the same architectures as in [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF]. The main difference is that we consider encoder and decoder separately. The encoders Enc A→B and Enc B→A are composed of two convolutional layers with stride 2, followed by 5 residual blocks [START_REF] He | Deep residual learning for image recognition[END_REF]. other methods in a fair way regardless of the backbone architecture.

Results

We use DINE to perform unsupervised domain adaptation for three applications. The first one is classification of digits, both handwritten (MNIST [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] and USPS [START_REF] Hull | A database for handwritten text recognition research[END_REF]) and building identification numbers (SVHN [NWC + 11]).

For segmentation, we explore the adaptation from synthetic images (GTA5 [START_REF] Richter | Playing for data: Ground truth from computer games[END_REF]) to real images (Cityscapes [COR + 16]). For co-segmentation, the adaptation between different geographical regions and natural catastrophes is studied using the xBD dataset [GGP + 19] that was released in late 2019.

Classification

Our classification tests follow the ones presented in [HTP + 18]. UDA is performed between MNIST and USPS datasets of handwritten digits. UDA is also performed from the SVHN dataset [NWC + 11] to MNIST [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. This is a much more challenging case, since SVHN images do not contain handwritten digits, and images are RGB instead of grayscale. Our proposed method is tested both without and with shared encoder weights. The results can be found in Table 6.1.

Our method yields state-of-the-art performances for UDA between the MNIST and USPS datasets. Notably, the performance of DINE when adapting from MNIST to USPS surpasses target only supervision. This is likely a result of two main factors. Firstly, the number of training images is larger in the MNIST dataset. Second, the image translation in DINE can be seen as a self-supervised auxiliary loss that helps the network learn discriminative features. On the difficult task of adapting from street view house numbers to handwritten digits, DINE obtains the third best performance behind [HTP + 18] and [START_REF] Taigman | Unsupervised cross-domain image generation[END_REF]. Here we observed that DINE struggled to stabilize the learning domain-invariant operations.

Co-segmentation

Our final tests for the co-segmentation tasks focus on the problem of change detection on remote sensing image pairs [Sin89, HCC + 13]. We use the xBD dataset, which contains images before and after various natural disasters.

We have chosen three types of natural disaster present in the dataset for our tests: wildfires, hurricanes and tsunamis. From each domain, we have selected the 50 images with the most changed pixels. Change labels were generated using the "destroyed building" annotations present in the dataset. The task here is to attribute dense per-pixel change labels based on an input of two co-registered images.

This task is challenging in several ways. For one, several images are affected by atmospheric conditions. This reflects the use case for such images in the real world, in which timely results are expected and the image acquisition conditions are seldom ideal. Furthermore, class distributions are significantly different between some domains. The fire and hurricane domains contain 7-8% of pixels belonging to the change class, while tsunami images contain about 4% of such pixels. This large class imbalance also means that small differences in total accurately classified pixels may have a strong impact on the Dice score.

The difference between the considered domains is clear when looking at the images. The images of destroyed buildings are extremely different in each of the three cases. For wildfires, most of the buildings is gone but the structure and footprint of the buildings usually remains visible in the same location. For flood images, changes are often inferred from contextual clues, i.e. if water surrounds the building in the second image in the pair, that building has likely suffered heavy damage. In the case of the tsunami images, the affected buildings are almost unrecognisable afterwards. Often, it is impossible to identify any signs of destroyed buildings, as most of it has been washed away or displaced in the image.

Tables 6.3 and 6.4 contain the Dice scores and accuracies obtained in each test, respectively. Domain adaptation is performed in both directions between each pair of domains. Tests of the obtained networks were done using the target domain images as well as the ignored domain images to evaluate how well the networks generalize to unseen cases. Tests on target domain have been highlighted with a yellow background. We observe that the domains of fire and hurricane are not too distant, and that using domain adaptation methods does not improve results over source domain supervision. We do notice that using DINE improves the performances when transferring between either fire↔tsunami and hurricane↔tsunami cases, validating our method when domains are sufficiently distant.

Limitations and Discussion

The proposed method achieved good results in many of the test cases. As is generally the case with adversarial methods, training stability can be an issue. In our tests this was only a problem when bridging the domain shift between the MNIST and SVHN datasets, probably due to the difference in imaged objects (handwritten digits and house numbers), as well as the different number of color channels. DINE obtained excellent results in the MNIST to USPS tests, surpassing even target only supervision.

In the segmentation tests, our method outperformed state-of-the-art UDA methods in tests with a standardized segmentation network. It did not outperform some other methods on the results reported in their original papers. It is possible that with higher computational power and a more advanced segmentation backbone DINE will achieve more comparable results, but that remains to be tested. The co-segmentation tests showed how challenging it is to apply pattern recognition methods to some real world applications. DINE showed a clear improvement in results in the fire↔tsunami and hurricane↔tsunami tests.

Tests like these are still uncommon in the computer vision community, and we hope that the results presented here motivates more applications such as this, as we believe this would help a concrete positive impact in the world.

Our method is lighter than CyCADA [HTP + 18], a well known UDA method that is also based on CycleGAN.

CyCADA requires pre-training a network on source domain data, and using it to preserve semantic information in the images during translation. It is also based on the assumption that a network trained on source domain data should apply directly to target domain data. We make no such assumption, as we only use the domain agnostic code for the desired task. DINE is also lighter at inference time. At test time, CyCADA first translates the image from target to source domain to then apply the desired task. In DINE, image translation is not done at test time since we have trained encoders for each of the considered domains.

Shortcut decoding images also show that DINE is much more effective at aligning the latent spaces of source and target domain representations than adversarial training at feature level. Results that show this phenomenon can be seen in Fig. 6.7.

Unpaired Translation of Change Detection Images

Around the same time this work was being developed, other authors, including Luppino et al. [LKB + 20] and Saha et al. [START_REF] Saha | Unsupervised multiple-change detection in vhr optical images using deep features[END_REF], have explored unpaired image translation using the CycleGAN framework for unsupervised change detection, based on the principle that translated images should more closely match where no changes have occurred,

while not being able to predict changes.

It is easy to see why such translation would be of interest in the case of cross-sensor change detection, where pixel values can't be compared directly. In an ideal scenario, using such image translation techniques would abstract away the different sensor characteristics or even seasonal and other irrelevant variations, allowing the change detection algorithms to compare the images directly.

During the development of the DINE algorithm that was presented above, some tests were conducted using the CycleGAN algorithm to translate images from the ABCD dataset, which contained image pairs taken before and after a tsunami with patch-level change labels. Since the images from before and after were obtained using sensors with different properties, the aim was to translate images taken by the lower quality sensor into higher quality images with similar sensor properties as the other images in the dataset while preserving the content.

An interesting phenomenon was observed when this was attempted, which can be seen in Fig. 6.8. In this case, adversarial training led the CycleGAN translation operations to hallucinate changes to match the statistics in the images taken after the tsunami. This contradicts the original purpose of this operation, since the translation affects not only the sensor statistics (i.e. image style) but also the imaged objects (i.e. image content). This effect was also observed by Luppino et al. in [LKB + 20], where pixel relationships are used in attempt to separate changed pixels from unchanged ones during the adversarial training process to reduce these effects.

In the ABCD dataset, where change labels for the central building in each patch are available, this effect can be mitigated by using only image pairs marked as unchanged for training the image translation algorithm. Figure 6.8

shows the different effects of training the translation algorithm using the full dataset (all) versus that of training using only unchanged patches (clean). Interestingly, the translation algorithm often hallucinates destructed buildings when translating from 1 to 2, but rarely hallucinates real buildings from destructed ones when translating from 2 to 1. The reason for this difference is unclear from these experiments. Furthermore, using only unchanged image patches is not enough to completely stop the network from hallucinating destructed buildings. This is likely due to the fact that these images still contain some destructed buildings surrounding the central building in each patch (which is the one the label refers to).

This highlights the perils of using such adversarial algorithms in the context of image analysis. The fact that a generator network is successful in confounding a discriminator does not necessarily mean that the produced images are "correct" or even realistic. The application of such algorithms in sensitive contexts, such as autonomous driving, must be used very carefully.

Conclusion

The proposed DINE algorithm for unsupervised domain adaptation has been validated for several cases of domain shift, including different data sources, synthetic to real, and different geographical regions. It was also validated for Unlike many other works in this field, the methods presented in this thesis focus on supervised change detection.

Unsupervised methods are valuable, especially when labelled data is not available. But supervised methods are often superior in many ways when successfully applied as they benefit from prior information on the given task. Such methods are able to learn from examples which changes are important and which are not. In a way, everything could be defined as having been changed between two images taken at different times since nothing is exactly the same, but that would hardly be useful. Unsupervised methods decide by themselves what should be considered as a change, simply based on the appearance variance. But what is a change? While the answer to that question in isolation is best left to the philosophers, real applications of change detection methods usually have a concrete applications in mind. Should the algorithm look for new buildings that have been built? Or ones that have been destroyed? Or loss of ice surfaces due to polar melting? Or loss of forest areas due to fires or deforestation?

Each of these applications can be modelled as change detection problems, and each of these events would lead to modified appearances in the imaged areas. This is the value of supervised methods, they can be taught to perform the task that the current application demands. Pixel level was initially studied as a patch classification problem, as well as a two class semantic segmentation problem in Chapter 3. This was, to the best of our knowledge, the first time CNNs were applied in an end-to-end fashion to the problem of change detection in such a framework. Previous CNN-based algorithms mostly focused on generating high dimensional pixel-level descriptors, whose difference could then be analysed using various techniques. Using the OSCD and an earlier dataset for our tests, the superiority of end-to-end methods over other supervised methods became clear. The experiments that were conducted also showed that the information that is present in the bands outside the visible spectrum can also help supervised change detection methods to achieve better performances.

The development of the HRSCD dataset allowed two main improvements to the methods presented in Chapter 3.

First, the unprecedented scale of this change detection dataset allowed for deeper architectures to be trained with a smaller risk of overfitting. Second, the availability of land cover maps allowed the proposed change detection methods to not only detect, but also understand the changes in image pairs. In Chapter 4, several different approaches to harness this semantic information to perform semantic change detection were studied, and it became clear that the integrated multitask approach was superior to the other ones that were considered.

At this point, it became very clear that the considered data were far from ideal, and that simply proposing more advanced methods for change detection without considering such issues would not lead to concrete advancements that would fare well in realistic application cases. Data problems such as label noise, bias, imbalance and other issues had to be considered to produce robust CNN-based change detection algorithms.

To handle the problem of label noise and bias, the work presented in Chapter 5 presented an iterative training algorithm for the multitask network that had been presented in the previous chapter. This training procedure, when coupled with the novel guided anisotropic diffusion algorithm, allowed us to use the network predictions themselves and edge information from the input images to filter out noisy labels from the training dataset, allowing the network to produce much more precise predictions of changes. The guided anisotropic diffusion was also coupled with a class activation map technique and a novel spatial attention layer to perform weakly supervised semantic cosegmentation using only image-level change labels as supervision. This was, to the best of our knowledge, the first time imagelevel to pixel-level weakly supervised change detection was performed.

Chapter 6 presents an exploration of a novel domain adaptation method based on adversarial training for a domain invariant encoding of input images into a common latent space where a given task could be performed.

This method was initially proposed with the aim of performing change detection, but the generality of the proposed method was shown by applying it to classification and semantic segmentation tasks. In the context of change detection, this method allowed the change detection network to learn to detect destroyed buildings using images from one type of natural disaster, and to better generalise to other types of disasters. The range of natural disasters that were considered (wildfires, hurricanes, and tsunamis) illustrated the flexibility of the proposed method.

The exploration of CNN-based techniques for supervised changed detection presented in this thesis encountered several challenges, most of which centred around the properties of the considered data. While many of the recently proposed CNN-based computer vision methods can be somewhat straightforwardly extended to change detection, the peculiar data characteristics often make it quite challenging to actually achieve good results using realistic data.

Nevertheless, end-to-end FCNs were successfully used to perform various change detection tasks.

Future Work

One of the main possible continuations for this work would be to extend the presented methods to deal with multitemporal inputs, i.e. time series of coregistered images. While some work has already been done along those lines, notably in [PVV + 19], the number of time-steps considered remains very low compared to what recurrent neural networks are capable of handling. Such networks are designed to identify long-range relations, which could be very useful when applied to detecting changes using sources with a high revisit rates such as the Sentinel-2 or Landsat satellites. One example would be to input time series of satellite images that have not been selected for good weather, and letting the network identify and interpret images automatically. This would be extremely useful for real time large scale monitoring of landmasses using data streams from Sentinel-2 satellites, for example. Currently, images are still manually selected to avoid images where atmospheric conditions interfere with image interpretation, but such manual selection does not lend itself well to large scale monitoring.

It would also be interesting to extend the methods presented in this thesis to handle multimodal data. Since one can not always have full control over the data that is available to study specific events, it is at times necessary to handle several types of data simultaneously. Data multimodality can be as simple as sensors with different ground sample distance, or as complex as comparing multispectral images to SAR images. Handling such multimodal cases could significantly increase the potential applications of such algorithms. The domain adaptation method presented in Chapter 6 is a good starting point for such approaches. The idea of representing images from different sources in a common latent space could be used to compare Earth observation images. To the best of my knowledge, this

has not yet been done using adversarial training for multitemporal remote sensing image analysis.

Finally, the proposed change detection methods could be integrated into a large scale monitoring system to perform online monitoring of large areas. Such systems could be used for automatically updating maps, monitoring illegal deforestation, and many other applications. The integration of change detection into such a large scale data processing pipeline is a necessary step for a global automatic monitoring system, which would be made possible by harnessing the full potential of Earth observation satellites. This would consist of bridging the gap between purely academic works and real world applications. Automatic monitoring of large areas is a task of interest in many different industries, but it is a challenging task that is hard to fully automate, which subsequently hinders the speed of the developed systems.

Labels

The labels in the OSCD were created using GNU Image Manipulation Program (GIMP)7 . Binary change maps were created by comparing each pair of coregistered true colour images, and marking the observed changes manually as closely as possible to the boundaries in the original images.

It is not always clear to clearly separate changed from unchanged reasons, especially at a resolution of 10 m/px, since reality is not binary. The effects of this ambiguity are illustrated in Fig A .2, where even human analysts disagree about some of the marked changes. At the edge cases, where it is not clear whether a region should or not be marked as changed, a criterion was used to better standardize the creation of such maps. Such regions were marked as changed if it looked like human activities had caused the perceived changes. Other natural changes, such as seasonal variation, was largely ignored.

Size

The detailed size of each region in the dataset at 10 m/px is described in Tab. A.1. The total size of about 9.6 million annotated pixel is larger than previous open datasets, including the Air Change dataset by Benedek et al. [START_REF] Benedek | Change detection in optical aerial images by a multilayer conditional mixed markov model[END_REF].

The complete downloadable version of the dataset amounts to 512.9 MB of data. It occupies 756.7 MB of disk space once uncompressed.

Class Distribution

Table A.4 contains the percentage of changes in each change map in the OSCD dataset. The distribution of changes is not homogeneous across images. Approximately 3.21% of the pixels in the dataset are marked as change, which consists of a 1:30 class imbalance. This should be taken into account when using the dataset.

Contents

The dataset files contain:

1. Information about the suggested train/test split.

2. Information about the acquisition dates of each image.

3. Geojson files for each considered geographical region.

4. All image bands in TIFF format for all images in the dataset at the original resolution cropped according to the geojson files. 5. All image bands in TIFF format for all images in the dataset resampled at 10 m/px for ease of use.

6. All change maps in TIFF and PNG formats.

7. True colour images for all the acquisitions in the dataset, generated from the red, green, and blue bands.

8. A README containing copyright and general information about the dataset.

Access and Distribution

The dataset files can be downloaded from the official IEEE DataPort page 8 or from the mirrors found on the dataset page on my personal website 9 . The dataset is released under a Creative-Commons BY-NC-SA licence 10 . years of 2006 and 2012 were selected. This restricted the selection to only two regions:

1. 14 -Calvados: region around the city of Caen.

2. 35 -Ille-et-Vilaine: region around the city of Rennes.

The selection of images was further narrowed down following two steps:

1. Only the regions for which images were available both around 2006 and 2012 were kept (the imaged regions were not identical in the two years).

2. The regions for which no labels were available were removed from the dataset.

The final size of the dataset is described in Tab. B.1. The HRSCD dataset contains over 3000 times the number of pixels as the OSCD dataset that was presented in Appendix A.

Labels and Class Distribution

The labels in the HRSCD dataset were generated by sampling the Urban Atlas Change 2006-2012 database3 . This database contains vector data for changes that occurred in all major urban areas in Europe between 2006 and 2012.

The annotations come in the form of polygon coordinates, coupled with "from" and "to" information to define what type of change has occurred in that location. These vector data were rasterized using the gdal library4 .

To limit the complexity of data and reduce the number of considered classes, several classes were grouped together according to the hierarchical breakdown presented in [Cop20]. This leaves us with 5 distinct classes (as well as the "no information" class for pixels not covered by any polygons), as can be seen in Tab. B.2.

The land cover maps in the HRSCD dataset were generated in a similar way to the change labels using the Urban Abstract: The analysis of satellite and aerial Earth observation images allows us to obtain precise information over large areas. A multitemporal analysis of such images is necessary to understand the evolution of such areas. In this thesis, convolutional neural networks are used to detect and understand changes using remote sensing images from various sources in supervised and weakly supervised settings. Siamese architectures are used to compare coregistered image pairs and to identify changed pixels. The proposed method is then extended into a multitask network architecture that is used to detect changes and perform land cover mapping simultaneously, which permits a semantic understanding of the detected changes. Then, classification filtering and a novel guided anisotropic diffusion algorithm are used to reduce the effect of biased label noise, which is a concern for automatically generated large-scale datasets.

Weakly supervised learning is also achieved to perform pixel-level change detection using only imagelevel supervision through the usage of class activation maps and a novel spatial attention layer. Finally, a domain adaptation method based on adversarial training is proposed, which succeeds in projecting images from different domains into a common latent space where a given task can be performed. This method is tested not only for domain adaptation for change detection, but also for image classification and semantic segmentation, which proves its versatility.

Institut Polytechnique de Paris 91120 Palaiseau, France

1. 1

 1 Aerial images throughout the years of the Fort de Palaiseau, which became a research center for ONERA in 1947. Images courtesy of IGN's BD Historique. . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 The twin Sentinel-2 satellites regularly image all of Earth's main landmasses. In (b), regions marked in green are imaged every 5 days, and regions marked in yellow are imaged every 10 days (as of October 2019). Images courtesy of sentinel.esa.int. . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Images following the evolution of wildfires near Wooloweyah, New South Wales, Australia. Sentinel-2 images courtesy of ESA's Copernicus program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Excerpt from letter written by Isaac Newton to Robert Hooke in 1675. Image courtesy of https: //discover.hsp.org/. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2. 5

 5 LeNet-5 was the first convolutional neural network to combine bioinspired weight sharing operations with gradient-based learning. Image reproduced from [LBBH98]. . . . . . . . . . . . . . . . . . . . . . 2.6 The CNN proposed by Bromley et al. was the first one to have a Siamese structure. It was used to compare signatures automatically. Image reproduced from [BGL + 94]. . . . . . . . . . . . . . . . . . .

3. 3

 3 Example of change maps between images (a) and (b) manually annotated by three different people, represented by the three colour channels in (c). Even human analysts sometimes disagree at what regions have been changed, especially around the boundaries between changed and unchanged regions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Comparing the OpenStreetMap semantic maps (b) and (d) relative to the images (a) and (c) leads to a change map (f) that does not resemble that which is produced by visual analysis (e). . . . . . . . . . 3.5 Proposed patch based CNN architectures for change detection. Processing both images since the first layer of the network allows for more total comparison operations, but does not allow for the weight sharing properties of Siamese networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Schematics for the three FCN architectures for change detection. These architectures are U-Net inspired extensions of the patch based networks presented previously. In (b) all the activations from both streams are used for skip connections, while in (c) the magnitude of their difference is explicitly calculated. Red arrows represent shared weights between Siamese branches. . . . . . . . . . . . . . 3.7 Results on the OSCD rio image pair using all 13 available spectral channels. White represents true positives, black represents true negatives, green represents false positives, and magenta represents false negatives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Results on the OSCD brasilia image pair using all 13 available spectral channels. White represents true positives, black represents true negatives, green represents false positives, and magenta represents false negatives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Illustrative results on the montpellier test case of the OSCD dataset using all 13 color channels. In (d), each colour channel represents the manual annotations by a different user. In images (d), (e), and (f) white means true positive, black means true negative, green is false positive, and magenta is false negative. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Changes are marked in white. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Examples of image pairs, land cover maps (LCMs) and associated pixel-wise change maps from the HRSCD dataset. In the depicted LCMs, blue represents the "artificial surfaces" class, and orange represents the "agricultural areas" class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Examples of: (a)-(c) overly large change markings, (d)-(f) failure to mark changes, (g)-(i) false positive. 4.3 FC-EF-Res architecture, used for tests with smaller datasets to avoid overfitting. Using residual blocks improves network performance and facilitates training. . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Schematics for all four proposed strategies for semantic change detection. Φ represents the network branch's learnable parameters, "Enc" means encoder, "Dec" means decoder, "LCM" means land cover mapping, and "CD" means change detection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Detailed schematics for the integrated change detection and land cover mapping network (Strategy 4). The encoder-decoder architecture is the same that was used for all 4 strategies. . . . . . . . . . . 4.6 Illustrative images of the obtained results: (a)-(b) multitemporal image pair; (c) ground truth change detection map; (d)-(h) predicted change maps; (i)-(l) ground truth and predicted land cover maps for image 1; (m)-(p) ground truth and predicted land cover maps for image 2. . . . . . . . . . . . . . . . . 4.7 Illustrative images of the obtained results: (a)-(b) multitemporal image pair; (c) ground truth change detection map; (d)-(h) predicted change maps; (i)-(l) ground truth and predicted land cover maps for image 1; (m)-(p) ground truth and predicted land cover maps for image 2. . . . . . . . . . . . . . . . . 4.8 Illustrative images of the obtained results: (a)-(b) multitemporal image pair; (c) ground truth change detection map; (d)-(h) predicted change maps; (i)-(l) ground truth and predicted land cover maps for image 1; (m)-(p) ground truth and predicted land cover maps for image 2. . . . . . . . . . . . . . . . . 4.9 Illustrative images of the obtained results: (a)-(b) multitemporal image pair; (c) ground truth change detection map; (d)-(h) predicted change maps; (i)-(l) ground truth and predicted land cover maps for image 1; (m)-(p) ground truth and predicted land cover maps for image 2. . . . . . . . . . . . . . . . . 4.10 Illustrative images of the obtained results: (a)-(b) multitemporal image pair; (c) ground truth change detection map; (d)-(h) predicted change maps; (i)-(l) ground truth and predicted land cover maps for image 1; (m)-(p) ground truth and predicted land cover maps for image 2. . . . . . . . . . . . . . . . . 4.11 Illustrative images of the obtained results: (a)-(b) multitemporal image pair; (c) ground truth change detection map; (d)-(h) predicted change maps; (i)-(l) ground truth and predicted land cover maps for image 1; (m)-(p) ground truth and predicted land cover maps for image 2. . . . . . . . . . . . . . . . . 4.12 Illustrative images of the obtained results: (a)-(b) multitemporal image pair; (c) ground truth change detection map; (d)-(h) predicted change maps; (i)-(l) ground truth and predicted land cover maps for image 1; (m)-(p) ground truth and predicted land cover maps for image 2. . . . . . . . . . . . . . . . . 4.13 Illustrative images of the obtained results: (a)-(b) multitemporal image pair; (c) ground truth change detection map; (d)-(h) predicted change maps; (i)-(l) ground truth and predicted land cover maps for image 1; (m)-(p) ground truth and predicted land cover maps for image 2. . . . . . . . . . . . . . . . . 5.1 (a)-(b) image pair, (c) change labels from the HRSCD dataset, (d) ground truth created by manually annotating changes, (e) result obtained by naive supervised training, (f) result obtained by our proposed method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Results of guided anisotropic diffusion. Edges in the guide image (a) are preserved in the filtered image (b). (c)-(f) show results using different numbers of iterations. . . . . . . . . . . . . . . . . . . . . 5.3 Iterative training method: alternating between training and data cleaning allows the network to simultaneously learn the desired task and to remove bad examples from the training dataset. . . . . . . . . 5.4 Proposed methods for merging original labels and network predictions. Classes: 0 is no change, 1 is change, 2 is ignore. (a) Intersection between original and detected changes. (b) Ignore false negatives from the perspective of original labels. (c) Ignore all pixels with label disagreements. . . . . 5.5 Example case of the three proposed merge strategies. In (c), black is true negative, white is true positive, magenta is false negative, and green is false positive. In (d)-(f) blue represents the ignore class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Basic schematic of the network used for weakly supervised change detection. Two paths can be taken: the classification path uses the proposed attention layer and global average pooling to produce a classification of the image, while the segmentation path avoids these steps to output pixel-level predictions. Supervision is only available on the classification path. . . . . . . . . . . . . . . . . . . . . 5.7 Guided anisotropic diffusion for filtering a real example of semantic segmentation. The diffusion allows edges from the guide images to be transferred to the target image, improving the results. . . . 5.8 Comparison between (c) original dataset ground truth, (e) prediction filtered by Dense CRF, and (f) prediction filtered with guided anisotropic diffusion for 20000 iterations. . . . . . . . . . . . . . . . . . . 5.9 Ablation studies. (a) Comparison between strategies for merging network predictions and reference data. (b) Comparison between iterative training with and without the usage of original reference data. (c) Comparison between GAD and Dense CRF. Top row contains Dice scores, bottom row contains global accuracy curves. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10 Change maps obtained by using different methods on two image pairs. Detected changes are marked in red color. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.11 Results using the complete inference pipeline. GAD is used to improve predictions during the iterative training process as well as for improving the final segmentations. . . . . . . . . . . . . . . . . . . . . .

  the image's domain of origin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Diagram of the DINE algorithm. The two way image translation allows us to enforce code similarity during the forward and backward translations, forcing the network to find a domain agnostic latent representation space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Shortcut decoding path, used for validating alignment of feature spaces during forward and backward translations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Basic schematic for the architectures of T network for each of the considered tasks. . . . . . . . . . . 6.5 Segmentation results on the Cityscapes dataset using the ResNet-9 backbone for segmentation using supervision from the GTA5 dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Change detection results between images (a) before and (b) after a natural disaster. (d) DINE results show a significant improvement in adapting to new natural disaster with respect to (e) source-only supervision when compared to the (c) ground truth. Changes are marked in red. . . . . . . . . . . . . 6.7 Comparison of shortcut decoded images. (a) Input images. (b) Translated images. (c) Full cycle decoding. (d) Shortcut decoding (DINE). (e) Shortcut decoding (discriminative feature loss). This clearly shows that DINE successfully aligned feature spaces, while a discriminative code loss did not. 6.8 Including changed image pairs when training the CycleGAN networks affects the results, leading to more frequent hallucination of destructed buildings as seen in (b) versus (c). Interestingly, the hallucination of buildings where there are none is much less frequent, as seen in (e) versus (f). . . . . A.1 Example of Sentinel-2 image bands. Differences in spatial resolutions can be perceived as a variance in sharpness in the images above. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 Pair of coregistered images from the OSCD and manual change maps created by three different people. In (c), each colour channel contains the change map produced by a different person. . . . . . B.1 Example of data that can be found in the HRSCD dataset. Coregistered aerial image pairs, change maps, and land cover maps contain semantic information about how the terrain has evolved over a period of six years. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2. 1

 1 Two class confusion matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Quantitative evaluation of the proposed methods on the OSCD and Air Change datasets. . . . . . . . 4.1 Urban Atlas land cover mapping classes at hierarchical level L1, extracted from [Cop20]. . . . . . . . . 4.2 Change class imbalance at hierarchical level L1. Row number represents class in 2006, column number represents class in 2012. Classes were defined in Table 4.1. . . . . . . . . . . . . . . . . . . . 4.3 Summary of proposed change detection strategies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Change detection results of several methods on the OSCD dataset, for the RGB and multispectral (MS) cases. Results are in percent. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Change detection (CD) and land cover mapping (LCM) results of all four of the proposed strategies on the HRSCD dataset. Comparison with the methods proposed by [EALW16] (Otsu [CNNF-O] and fixed [CNNF-F] thresholding) and by [Cel09] ([PCA+KM]) are included. Results are in percent. . . . . 4.6 Change detection results on Eppalock lake test images. Results are in percent. . . . . . . . . . . . . . 5.1 Accuracy and standard deviation for each test on ABCD dataset using 5-fold cross validation. Fixed scale and resized variations of the ABCD dataset were tested. Results from methods proposed by Fujita et al. are included for comparison. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Classification accuracy. Models marked with † share encoder parameters. Models marked with ‡ share encoder parameters and are trained without the code similarity loss. Results marked with * convert SVHN images to grayscale to enable symmetric encoders. . . . . . . . . . . . . . . . . . . . . 107 6.2 GTA5 to Cityscapes segmentation performance. Models marked with † share encoder parameters. Models marked with ‡ share encoder parameters and are trained without the code similarity loss. . . . 108 6.3 Dice score in change detection tests. Models marked with † share encoder parameters. Models marked with ‡ share encoder parameters and are trained without the code similarity loss. Column marked as "none" refers to source domain supervision. Results highlighted in yellow are the ones where target domain and test data are the same. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 6.4 Accuracy in change detection tests. Models marked with † share encoder parameters. Models marked with ‡ share encoder parameters and are trained without the code similarity loss. Column marked as "none" refers to source domain supervision. Results highlighted in yellow are the ones where target domain and test data are the same. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.1 Locations and sizes of the images in the OSCD dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 Dates of acquisition of the images in the OSCD dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . A.3 Sentinel-2 bands and resolutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.4 Percentage of changes in each change map in the OSCD dataset. . . . . . . . . . . . . . . . . . . . . B.1 Size of the HRSCD dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.2 Urban Atlas land cover mapping classes at hierarchical level L1, extracted from [Cop20], and their frequency in the generated land cover maps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.3 Change class imbalance. Row number represents class in 2006, column number represents class in 2012. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  pr écision plusieurs caract éristiques de r égions enti ères à distance. Les techniques d'imagerie peuvent varier consid érablement en fonction des applications souhait ées. Les types d'imagerie passive comprennent les images panchromatiques, RVB (rouge, vert et bleu), multispectrales et hyperspectrales. Ces m éthodes d'acquisition consistent à mesurer l'intensit é des ondes électromagn étiques émises par l'objet ou la zone observ ée. Les m éthodes d'imagerie actives, telles que le SAR (synthetic aperture radar ) et le LIDAR (light detection and ranging), émettent des ondes électromagn étiques dans une r égion cible et observent la r éponse r éfl échie. R écemment, les r éseaux de neurones se sont r év él és extr êmement puissants pour r ésoudre plusieurs probl èmes, notamment dans le domaine de l'analyse d'images. La t él éd étection ne fait pas exception à la r ègle, et l'application de m éthodes bas ées sur les r éseaux de neurones pour extraire des informations significatives des images d'observation de la Terre est actuellement un sujet tr ès recherch é.Les travaux pr ésent és dans cette th èse tournent autour d'un point central: la d ét éction de changements à partir de paires d'images de t él éd étection en utilisant des r éseaux de neurones convolutifs. Cette question est abord ée principalement sous quatre points de vue:1. Modalit és des donn ées: y a-t-il des avantages à utiliser des images multispectrales plut ôt que des images RVB pour la d étection de changements? 2. S émantique des donn ées: comment pouvons-nous extraire des informations s émantiques du paire d'images pour mieux comprendre l' évolution de la zone imag ée? 3. Disponibilit é des donn ées: comment r éduire l'effet du bruit des étiquettes et surmonter la raret é des donn ées lors de l'entraînement des r éseaux neuronaux convolutifs? 4. H ét érog én éit é des donn ées: comment pouvons-nous cr éer des syst èmes de d étection de changements qui soient robustes dans leur g én éralisation à des év énements jusqu'alors inconnus? L'int ér êt pour la conception de syst èmes de d étection automatique de changements provient de deux cas prin-cipaux: les applications à grande échelle et la r éponse rapide aux év énements. Dans ces deux cas, la vitesse d'analyse manuelle des images devient un probl ème. Par exemple, lors des incendies de for êt en Australie en 2019, illustr és dans la Fig.1.3, des images de t él éd étection ont ét é utilis ées pour obtenir des informations actualis ées sur la situation actuelle[START_REF] Boone | I made bushfire maps from satellite data, and found a glaring gap in australia's preparedness[END_REF]. L'imagerie satellitaire offre souvent un moyen rapide et pr écis d'obtenir des informations pr écises sur de vastes zones. C'est particuli èrement utile pour l'observation des zones d'acc ès difficile, comme les hautes montagnes, les for êts profondes et les r égions polaires. L'application de la vision par ordinateur et des m éthodes d'apprentissage automatique à l'analyse des images de t él éd étection multitemporelle a une longue histoire de recherche. Le chapitre 2 contient un r ésum é des travaux qui sont directement li ées aux travaux pr ésent és dans cette th èse, ainsi qu'une base th éorique n écessaire à leur compr éhension. Ce r écapitulatif th éorique devrait fournir au lecteur les connaissances minimales n écessaires pour comprendre les id ées pr ésent ées dans les chapitres suivants.Les travaux originaux qui ont ét é men ées au cours de cette th èse sont pr ésent ées dans les chapitres 3, 4, 5 et 6. Le chapitre 3 d écrit comment les r éseaux de neurones convolutifs (CNN) et les r éseaux neuronaux enti èrement convolutifs (FCN ou FCNN) peuvent être utilis és pour effectuer la d étection de changements comme un probl ème de segmentation s émantique de mani ère enti èrement supervis ée. Les architectures de r éseaux de neurones convolutifs et de r éseaux de neurones enti èrement convolutifs sont entraîn és "from scratch" en utilisant un nouvel ensemble de donn ées de d étection de changements. L'impact des variations d'architecture est étudi é, ainsi que la diff érence de performance entre les CNN et les FCN.Le chapitre 3 contient également des comparaisons entre l'utilisation des r éseaux propos és avec diff érentes combinaisons de bandes d'images multispectrales, et les r ésultats obtenus montrent qu'il est int éressant d'utiliser des bandes en dehors du spectre visible pour la d étection des changements lorsqu'elles sont disponibles. Le chapitre 4 explore l'ajout d'informations s émantiques dans le cadre de d étection de changements pr ésent é pr éc édemment, et comment l'apprentissage multit âche affecte les performances des r éseaux. Un nouvel ensemble de donn ées s émantiques à haute r ésolution et à grande échelle est cr é é pour r éaliser ces exp ériences. Diff érentes approches de ce probl ème sont analys ées, allant de la comparaison des cartes de la couverture terrestre pr édite pour la d étection de changements jusqu' à un FCN enti èrement int égr é pour effectuer simultan ément la d étection de changements et la cartographie de la couverture terrestre des images d'entr ée. Un sch éma d'entraînement par étapes est propos é pour le r éseau multit âche int égr é qui r éduit le nombre d'hyperparam ètres à choisir et qui am éliore les performances du r éseau.

Le chapitre 7 Figure 1 . 1 :

 711 Figure 1.1: Aerial images throughout the years of the Fort de Palaiseau, which became a research center for ONERA in 1947. Images courtesy of IGN's BD Historique.

  Figure 1.2: The twin Sentinel-2 satellites regularly image all of Earth's main landmasses. In (b), regions marked in green are imaged every 5 days, and regions marked in yellow are imaged every 10 days (as of October 2019). Images courtesy of sentinel.esa.int.

Figure 1 . 3 :

 13 Figure 1.3: Images following the evolution of wildfires near Wooloweyah, New South Wales, Australia. Sentinel-2 images courtesy of ESA's Copernicus program.

Figure 2 . 1 :

 21 Figure 2.1: Excerpt from letter written by Isaac Newton to Robert Hooke in 1675. Image courtesy of https: //discover.hsp.org/.

Figure 2 . 3 :

 23 Figure 2.3: Comparison between image segmentation and semantic segmentation from the PASCAL VOC 2012 dataset [EVGW + 12]. In (b), the buses' pixels are separated into two different groups with no semantic information since they come from different objects. In (c), the pixels relative to the regions of both buses are classified as belonging to the semantic class "Bus".

Fig

  Fig. 2.3. In image segmentation, pixels are simply grouped in meaningful sets that belong to the same object or

  Figure 2.4: (a) While Rosenblatt's inspiration came partially from retinal nerves, the perceptron model is not restricted to vision. (b) Information flows in a single direction from inputs to outputs and through hidden units in a multilayer perceptron. Images reproduced from [Ros60] and [Mur12], respectively.

  depths are often referred to as "deep", hence the term "deep learning" [Sch15,[START_REF] Lecun | Deep learning[END_REF][START_REF] Goodfellow | Deep Learning[END_REF]. The activations of perceptrons f (i) , i ∈ {1, . . . , d -1} are called hidden layers, while the activation of perceptron f (d) is referred to as output. These networks are called feedforward networks due to the information flow happening only from the input towards the output. If feedback connections are introduced, the network becomes what is known as a recurrent neural network (RNN)[START_REF] Rumelhart | Learning representations by backpropagating errors[END_REF][START_REF] Goodfellow | Deep Learning[END_REF].

Figure 2

 2 Figure 2.5: LeNet-5 was the first convolutional neural network to combine bioinspired weight sharing operations with gradient-based learning. Image reproduced from [LBBH98].

  which usually share their weights. The images are then compared in feature space to achieve the desired result, such as face verification[START_REF] Chopra | Learning a similarity metric discriminatively, with application to face verification[END_REF], signature recognition [BGL + 94], stereo vision[START_REF] Zagoruyko | Learning to compare image patches via convolutional neural networks[END_REF], or change detection [ZFY+ 17].

Figure 2 . 6 :

 26 Figure 2.6: The CNN proposed by Bromley et al. was the first one to have a Siamese structure. It was used to compare signatures automatically. Image reproduced from [BGL + 94].

  a semantic segmentation algorithm predicts a class for each pixel in the input image, i.e. it performs dense classification among a set of known classes. The ground truth labels Y ∈ R M ×N ×N that are used are the one-hot encoded class of each pixel in I. A naive way to accomplish that using CNNs is to train the network to classify the central pixel of patches by creating a dataset with pairs (I

  Figure 2.7: The FCN architecture upsampled feature maps in a single step, which limited the accuracy of predictions around region boundaries. Reproduced from [LSD15a].

Figure 2 . 8 :

 28 Figure 2.8: SegNet architecture for semantic segmentation, in which the pooling indices were used for the unpooling operations to recover spatial information at each upsampling step. Reproduced from [BKC17].

Figure 2

 2 Figure 2.9: U-Net architecture schematic. The output of transposed convolutions are concatenated with feature maps produced by the encoder at several levels to combine high spatial accuracy with high-level features. Reproduced from [RFB15].

Figure 2 .

 2 Figure 2.12: Task similarity tree proposed by the Taskonomy project. It was shown that there are several benefits from learning a single latent space representation from which several tasks are performed, such as the need for fewer annotated data. Reproduced from [ZSS + 18].

Figure 2

 2 Figure 2.13: One of the main motivation for domain adaptation is to bridge the domain gap between (a) synthetic data and (b) real data to reduce the cost of data acquisition. One such case is the semantic segmentation of GTA 5 images [RVRK16] and real automotive images from the Cityscapes dataset [COR + 16].

  different sensors) allows the study of such images to yield a temporal understanding of the evolution of the imaged area. One such problem that has been researched for many years in this field is change detection, whose study is almost as old as remote sensing itself [Sin89, CJN + 04, HCC + 13, TCT + 15, YCZ20, Alb18]. Change detection is the name given to the task of identifying areas of the Earth's surface that have experienced changes by jointly analysing two or more coregistered images[START_REF] Bruzzone | A novel framework for the design of change-detection systems for very-high-resolution remote sensing images[END_REF]. Changes can be of several different types depending on the desired application, e.g. those caused by natural disasters, urban expansion, polar ice melting, deforestation, and others [XLL + 19, GGP+ 19].The search for ever more accurate change detection comes from the value of surveying large amounts of land and analysing its evolution over a period of time. Detecting changes manually is a slow and laborious process[START_REF] Singh | Review article digital change detection techniques using remotely-sensed data[END_REF] and the problem of automatic change detection using image pairs or sequences has been studied for many decades. Binary change detection attempts to identify which pixels correspond to areas where changes have occurred, whereas semantic change detection attempts to further identify the type of change that has occurred at each location.Many change detection algorithms comprise two main steps [Sin89, HCC + 13]. First, a difference metric is proposed so that a quantitative measurement of the difference between corresponding pixels can be calculated. The image generated from this step is usually called a difference image. Second, a thresholding method or decision function is proposed to separate the pixels into "change" and "no change" based on the difference image. These two steps are usually independent. Post-processing and pre-processing methods are sometimes used to improve results. Many algorithms use out-of-the-box registration algorithms and focus on the other main steps for change detection, or consider that the provided images have already been coregistered [HCC + 13]. Most papers on change detection propose either a novel image differencing method [BB05, EALW16, EALW17, ZFY + 17] or a novel decision function [BP00, Cel09]. A well established family of change detection methods is change vector analysis (CVA),considering the multispectral difference vector in polar or hyperspherical coordinates and attempting to characterise the changes based on the associated vectors at each pixel [LS94, BB07, HCC + 13], which has also been done using CNNs[START_REF] Saha | Unsupervised deep change vector analysis for multiple-change detection in vhr images[END_REF]. The authors of [HCC + 13] and[START_REF] Paul | Evaluation of global image thresholding for change detection[END_REF] noted that the performance of such algorithms is scene dependent. A contrario modelling has also been used to perform change detection by studying the distribution statistics of changes[START_REF] Liu | A contrario comparison of local descriptors for change detection in very high spatial resolution satellite images of urban areas[END_REF].Demir et al. prosed in [DBB13] an active learning method for transferring labels from a source to a target domain. Most methods that propose image differencing techniques followed by thresholding assume that a threshold is chosen based on the difference image. Paris et al. tackled a similar problem by performing domain adaptation to update vector maps using multispectral images[START_REF] Paris | A novel method based on source domain understanding and modeling to transfer labels from land-cover vector maps to classifiers for multispectral images[END_REF].[HCC + 13] categorize change detection algorithms into two main groups: pixel based and object based change detection. The former attempt to identify whether or not a change has occurred at each pixel in the image pair, while the latter methods attempt to first group pixels that belong to the same object and use information such as the object's colour, shape and neighbourhood to help determine if that object has been changed between the acquisitions.As noted in [HCC + 13, BB13], change detection on low resolution images and on very high resolution (VHR) images face different challenges. In low resolution images, pixels frequently contain information about several objects contained within its area. In such cases, a pixel in an image pair may contain both changed and unchanged surfaces simultaneously. VHR images are more susceptible to problems such as parallax, high reflectance variability for objects of the same class, and co-registration problems[START_REF] Bruzzone | A novel framework for the design of change-detection systems for very-high-resolution remote sensing images[END_REF]. It follows that algorithms that perform change detection on very high resolution images must be aware of not only a given pixel's values, but also of information about its neighbourhood. Several change detection algorithms have been developed specifically for VHR images[START_REF] Wiratama | Dual-dense convolution network for change detection of high-resolution panchromatic imagery[END_REF][START_REF] Chen | Deep siamese multi-scale convolutional network for change detection in multi-temporal vhr images[END_REF][START_REF] Fang | Dual learning-based siamese framework for change detection using bi-temporal vhr optical remote sensing images[END_REF] Machine learning algorithms, and notably convolutional neural networks (CNNs) in recent years, also have had great impact on change detection research. CNNs were used on the related task of comparing image pairs for different tasks, usually using a Siamese architecture[START_REF] Chopra | Learning a similarity metric discriminatively, with application to face verification[END_REF][START_REF] Zagoruyko | Learning to compare image patches via convolutional neural networks[END_REF].

  Change detection methods with unsupervised parameter estimation have been used in many different ways [Nie07, HCC + 13, VKKP15, LGT19b]. In the context of change detection, annotated datasets are extremely scarce and often kept private. Thus, unsupervised methods are extremely useful, since, unlike supervised methods, they do not need labelled data for training. Many of these methods automatically analyse the data in difference images and detect patterns that correspond to changes [BBM05, BP00]. Other methods use unsupervised learning approaches such as iterative training [LGQZ18], autoencoders [ZGLJ14], and principal component analysis with k-means clustering [Cel09] to separate changed pixels from unchanged ones. Unsupervised change detection methods often utilize some handcrafted heuristic coupled with unsupervised learning to compare descriptors calculated from different images. Several methods use autoencoders to calculate feature vectors for each pixel, which are then compared to performed change detection [SBB20, SSBB19, SBB19b, ZGZ + 19, BSBB19]. Other accomplish the same task using transfer learning approaches that use CNNs trained for different tasks [EALW16, EALW17, SSBB20, SBB18, YJL + 19]. GANs were also used to obtain pixel-level features for change detection [SBB19b, LKB + 20, LHK + 20]. Alvarez et al. proposed the S 2 -cGAN for self supervised change detection, where an image translation algorithm is learned using unchanged images and the discriminator is used to detect changes by identifying pixels that differ from the expected translated values [ARD20]. Luppino et al. noticed that the changed pixels can have a negative impact in adversarial learning, and worked on mitigating their effects during the training process using a co-attention mechanism [LKB + 20, LHK + 20], an idea which has also been explored by Jiang et al. [JHL + 20]. Adversarial learning was also used by Roy et al. in [RSSD18], which allowed a significant reduction in the amount of data necessary for training the proposed networks.

  Supervised change detection algorithms require labeled training data from which the task of change detection can be learned. Several methods have been proposed for performing change detection using supervised learning algorithms such as support vector machines [HSK + 08, VTK + 09, VTB + 13, LSR13], random forests [SGFT08], and neural networks [GW96, DK99, ZGLJ14]. CNN architectures have also been proposed to perform supervised change detection [ZFY + 17, COA18]. Convolutional neural networks for change detection have been proposed by different authors in the recent years.

  Figure 2.16: Example of a binary classification evaluation at pixel-level applied to change detection. In (e)-(h), the elements of each group are represented in white.

  as the Air Change (AC) dataset[START_REF] Benedek | Change detection in optical aerial images by a multilayer conditional mixed markov model[END_REF].The history of change detection began not long after aerial images became possible [Sin89, HCC + 13]. The proposed techniques have followed the tendencies of computer vision and image analysis: at first, pixels were analyzed directly using manually crafted techniques; later on, descriptors began to be used in conjunction with simple machine learning techniques[START_REF] Le | Urban change detection in sar images by interactive learning[END_REF]; recently, more elaborate machine learning techniques (deep learning) are dominating most problems in the image analysis field, and this evolution is beginning to reach the problem of change detection [SGSC15, EALW16, LGQZ16, GZL + 16, EALW17, ZFY+ 17].Due to the limited amount of available labelled data, most of these methods use various techniques based on transfer learning, taking as a starting point networks that have been trained on larger datasets for different problems.

  Figure 3.1: Example of Sentinel-2 image bands. Most Sentinel-2 spectral bands are of a lower spatial resolution than the visible bands, but they often contain information that can help with the analysis of the images.

  With the rise of open access earth observation from programs such as Copernicus and Landsat, large amounts of unlabelled data are available to be used for different applications. The Sentinel-2 satellites generate time series of multispectral images of Earth's landmasses with resolutions varying between 10m and 60m per pixel. Despite the abundance of raw data, there is a lack of open labelled datasets, which are necessary for developing supervised learning methods. Deep learning techniques and CNNs have been on the rise not only due to the exponential growth of the available computing power, but also to the increasingly large amounts of data that is available. The application of these techniques to the problem of change detection is limited while there is a lack of data that can be used for training, testing and comparing these systems.The lack of common evaluation datasets makes it difficult to quantitatively compare change detection methods.In this section, we present a new change detection dataset that includes manually annotated change maps, aiming to provide a solution to this problem. The objective of this urban change detection dataset is to provide an open and standardized way of comparing the efficacy of different change detection algorithms that are proposed by the scientific community, available to anyone who is interested in tackling the change detection problem 1 . The dataset is focused on urban areas, labelling as "change" only urban growth and urban changes, while ignoring natural changes (e.g. vegetation growth or sea tides).

Figure 3

 3 Figure 3.2: Sometimes, such as between images (a) and (b), it is clear to say whether or not changes have occurred and where they are located. In other cases, such as between images (c) and (d), it may not be clear where the changes are or even if changes have occurred at all.

Figures 3 .

 3 Figures 3.3 and 3.2 show the difficulty in accurately defining and labelling changes in image pairs. In Fig 3.3(c), we can see the change maps done by three different analysts represented in different color channels. While there

Figure 3 . 4 :

 34 Figure 3.4: Comparing the OpenStreetMap semantic maps (b) and (d) relative to the images (a) and (c) leads to a change map (f) that does not resemble that which is produced by visual analysis (e).

  Fig. 3.5(a).

Figure 3 . 5 :

 35 Figure 3.5: Proposed patch based CNN architectures for change detection. Processing both images since the first layer of the network allows for more total comparison operations, but does not allow for the weight sharing properties of Siamese networks.

  Figure 3.6: Schematics for the three FCN architectures for change detection. These architectures are U-Net inspired extensions of the patch based networks presented previously. In (b) all the activations from both streams are used for skip connections, while in (c) the magnitude of their difference is explicitly calculated. Red arrows represent shared weights between Siamese branches.

  Figure 3.7: Results on the OSCD rio image pair using all 13 available spectral channels. White represents true positives, black represents true negatives, green represents false positives, and magenta represents false negatives.

  Figure 3.8: Results on the OSCD brasilia image pair using all 13 available spectral channels. White represents true positives, black represents true negatives, green represents false positives, and magenta represents false negatives.

Figure 4 . 1 :

 41 Figure 4.1: Examples of image pairs, land cover maps (LCMs) and associated pixel-wise change maps from the HRSCD dataset. In the depicted LCMs, blue represents the "artificial surfaces" class, and orange represents the "agricultural areas" class.

Figure 4 . 2 :

 42 Figure 4.2: Examples of: (a)-(c) overly large change markings, (d)-(f) failure to mark changes, (g)-(i) false positive.

Figure 4 .

 4 Figure 4.3: FC-EF-Res architecture, used for tests with smaller datasets to avoid overfitting. Using residual blocks improves network performance and facilitates training.

  Figure 4.4: Schematics for all four proposed strategies for semantic change detection. Φ represents the network branch's learnable parameters, "Enc" means encoder, "Dec" means decoder, "LCM" means land cover mapping, and "CD" means change detection.

Fig. 4

 4 Fig. 4.4(c)). One network performs binary change detection on the image pair, while the other network performs land cover mapping of each of the input images. The two networks can be trained separately since they are independent.In this strategy, the two input images produce three outputs: two land cover maps and a change map. At each pixel, the presence of change is predicted by the change map, and the type of change is defined by the classes predicted by the land cover maps at that location. This way the number of predicted classes is reduced relative to the previous strategy (i.e. the number of classes is no longer proportional to the square of land cover classes) without

Φ

  Figure 4.5: Detailed schematics for the integrated change detection and land cover mapping network (Strategy 4). The encoder-decoder architecture is the same that was used for all 4 strategies.

  2.2 we split the HRSCD images into two groups: 146 image pairs for training and 145 image pairs for testing. By splitting the train and test sets this way we can ensure that no pixel in the test set has been seen during training. Class weights were set inversely proportional to the number of training examples to counterbalance the dataset's class imbalance. The results for each of the proposed strategies can be seen in Table

  Figure 4.6: Illustrative images of the obtained results: (a)-(b) multitemporal image pair; (c) ground truth change detection map; (d)-(h) predicted change maps; (i)-(l) ground truth and predicted land cover maps for image 1; (m)-(p) ground truth and predicted land cover maps for image 2.

  Figure 4.7: Illustrative images of the obtained results: (a)-(b) multitemporal image pair; (c) ground truth change detection map; (d)-(h) predicted change maps; (i)-(l) ground truth and predicted land cover maps for image 1; (m)-(p) ground truth and predicted land cover maps for image 2.

  Figure 4.10: Illustrative images of the obtained results: (a)-(b) multitemporal image pair; (c) ground truth change detection map; (d)-(h) predicted change maps; (i)-(l) ground truth and predicted land cover maps for image 1; (m)-(p) ground truth and predicted land cover maps for image 2.

Figure 5

 5 Figure 5.1: (a)-(b) image pair, (c) change labels from the HRSCD dataset, (d) ground truth created by manually annotating changes, (e) result obtained by naive supervised training, (f) result obtained by our proposed method.

Figure 5 . 2 :

 52 Figure 5.2: Results of guided anisotropic diffusion. Edges in the guide image (a) are preserved in the filtered image (b). (c)-(f) show results using different numbers of iterations.

Figure 5 Algorithm 2

 52 Figure 5.3: Iterative training method: alternating between training and data cleaning allows the network to simultaneously learn the desired task and to remove bad examples from the training dataset.

Figure 5 . 4 :

 54 Figure 5.4: Proposed methods for merging original labels and network predictions. Classes: 0 is no change, 1 is change, 2 is ignore. (a) Intersection between original and detected changes. (b) Ignore false negatives from the perspective of original labels. (c) Ignore all pixels with label disagreements.

2.

  Each annotated region may contain more than one instance: the annotations often group several change instances together; 3. Annotations are not flawless: the HRSCD dataset contains both false positives and false negatives in change annotations. It has also been shown by Khoreva et al. in [KBH + 17] that simply using the outputs of the network as training data leads to degradation of the results, and that it is necessary to use priors and heuristics specific to the problem at hand to prevent a degradation in performance. Here we use two methods to avoid degradation of the results

2.

  Ignore false negatives: using an ignore class for false negatives attempts to keep only good examples in (a) Image 1 (b) Image 2 (c) GT and prediction (d) Intersection (e) FN← Ignore (f) FN∪FP← Ign.

Figure 5 . 5 :

 55 Figure 5.5: Example case of the three proposed merge strategies. In (c), black is true negative, white is true positive, magenta is false negative, and green is false positive. In (d)-(f) blue represents the ignore class.

Figure 5 . 6 :

 56 Figure 5.6: Basic schematic of the network used for weakly supervised change detection. Two paths can be taken: the classification path uses the proposed attention layer and global average pooling to produce a classification of the image, while the segmentation path avoids these steps to output pixel-level predictions. Supervision is only available on the classification path.

Figure 5 . 7 :

 57 Figure 5.7: Guided anisotropic diffusion for filtering a real example of semantic segmentation. The diffusion allows edges from the guide images to be transferred to the target image, improving the results.

Figure 5 . 8 :

 58 Figure 5.8: Comparison between (c) original dataset ground truth, (e) prediction filtered by Dense CRF, and (f) prediction filtered with guided anisotropic diffusion for 20000 iterations.

Figure 5

 5 Figure 5.9: Ablation studies. (a) Comparison between strategies for merging network predictions and reference data. (b) Comparison between iterative training with and without the usage of original reference data. (c) Comparison between GAD and Dense CRF. Top row contains Dice scores, bottom row contains global accuracy curves.

  Figure 5.10: Change maps obtained by using different methods on two image pairs. Detected changes are marked in red color.

  Figure 5.11: Results using the complete inference pipeline. GAD is used to improve predictions during the iterative training process as well as for improving the final segmentations.

  such as urban expansion quantification, deforestation measurement, natural disaster evaluation, etc. It would be interesting to test the efficacy of the proposed ideas outside the context of change detection. The proposed methods could be applied with minor adaptations to other applications to help mitigate the effects of data scarcity and data noise. The proposed attention layer could be used to improve classification results when crops are centred on the object of interest (e.g. a classifier coupled with a region proposal network). GAD and iterative training could be used to improve segmentation results in other cases where overlarge prediction of some classes is a recurring problem. events can lead to changes with different appearances, which can be a problem for neural networks if the test examples differ too much from the ones used for training. Finding ways to bridge this domain gap between different events or even different geographical regions could help networks generalise to unseen cases that are farther from those in the training set. Unlike in Chapter 5, where label quality was the focus, here we deal with issues with the data distribution itself and how to mitigate the effects of domain shifts between training and test data.This chapter studies a novel domain adaptation method based on adversarial learning, with the final aim of performing change detection across datasets containing images from different types of natural catastrophes. The scope of the proposed method is nevertheless wider than just change detection. Thus, this chapter first steps away from the problem of change detection to present the domain adaptation method in a more general formulation, and the experiments contain not only change detection tests but also image classification and semantic segmentation ones.

Zhu

  et al. remark in[START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] that attempting to learn image-to-image translation in an adversarial fashion using unpaired data is very prone to mode collapse, i.e.the networks output very similar images regardless of the given inputs. Their work solves this problem by introducing an auxiliary cycle-consistency loss, which is based on the observation that an image that is put through a forward translation from domain A to domain B and a backward translation back to domain A should remain mostly unchanged 1 :F B→A (F A→B (I)) ≈ I . (6.1) Thus, image translation networks F A→B and F B→A are learned simultaneously. At the same time, discriminators D A and D B are trained in an adversarial manner to differentiate between real and translated image in domains A

Figure 6

 6 Figure 6.2: Diagram of the DINE algorithm. The two way image translation allows us to enforce code similarity during the forward and backward translations, forcing the network to find a domain agnostic latent representation space.

Figure 6

 6 Figure 6.5: Segmentation results on the Cityscapes dataset using the ResNet-9 backbone for segmentation using supervision from the GTA5 dataset.

Figure 6 Figure 6

 66 Figure 6.6: Change detection results between images (a) before and (b) after a natural disaster. (d) DINE results show a significant improvement in adapting to new natural disaster with respect to (e) source-only supervision when compared to the (c) ground truth. Changes are marked in red.

  Figure 6.8: Including changed image pairs when training the CycleGAN networks affects the results, leading to more frequent hallucination of destructed buildings as seen in (b) versus (c). Interestingly, the hallucination of buildings where there are none is much less frequent, as seen in (e) versus (f).

Chapter 7 Conclusion

 7 This thesis began with the aim of bridging the gap between the recent advances in computer vision and machine learning and the application of change detection using pairs of remote sensing images. The breakthroughs that convolutional neural networks had for accomplishing other computer vision tasks had not yet happened in change detection. It quickly became clear that there was one main problem holding back these advances: data. Along with other properties that separate the problem of change detection from other computer vision tasks, data availability and quality for this task is a recurrent issue, especially in the amounts that are usually required to apply deep learning methods. This explains the data-centred point of view from where this work is written. Data modalities, amounts, variability, quality and other aspects were central topics in the analyses presented in the previous chapters.

  Two datasets were created during the development of this thesis. The first, the ONERA Satellite Change Detection dataset (OSCD), contains pairs of multispectral satellite images with manually annotated pixel-level change maps. The second, the High Resolution Semantic Change Detection dataset (HRSCD), combined aerial VHR image pairs with open vector annotations of changes and land cover to obtain the largest openly available change detection dataset at the time of development. Both of these datasets opened the doors to new developments in change detection methods using CNNs. The datasets have been openly released to the scientific community to serve as a benchmark for various change detection algorithms.

Figure A. 1 :

 1 Figure A.1: Example of Sentinel-2 image bands. Differences in spatial resolutions can be perceived as a variance in sharpness in the images above.

Figure A. 2 :

 2 Figure A.2: Pair of coregistered images from the OSCD and manual change maps created by three different people. In (c), each colour channel contains the change map produced by a different person.

Atlas 2006 5

 5 and Urban Atlas 2012 6 land cover mapping vector data. The class distribution for different sections of the dataset can be found in Tab. B.2. The information contained in the generated land cover maps includes the semantic information in the Urban Atlas Change 2006-2012 data. Therefore, change maps were generated as binary maps, and semantic information Titre: R éseaux de Neurones Convolutifs pour l'Analyse de Changements en Imagerie de T él éd étection avec des Annotations Bruit ées et des D écalages de Domaine Mots cl és: T él éd étection, d étection de changements, r éseaux de neurones convolutifs, apprentissage multit âche, apprentissage faiblement supervis é, adaptation de domaine. R ésum é: L'analyse de l'imagerie satellitaire et a érienne d'observation de la Terre nous permet d'obtenir des informations pr écises sur de vastes zones. Une analyse multitemporelle de telles images est n écessaire pour comprendre l' évolution de ces zones. Dans cette th èse, les r éseaux de neurones convolutifs sont utilis és pour d étecter et comprendre les changements en utilisant des images de t él éd étection provenant de diverses sources de mani ère supervis ée et faiblement supervis ée. Des architectures siamoises sont utilis ées pour comparer des paires d'images recal ées et identifier les pixels correspondant à des changements. La m éthode propos ée est ensuite étendue à une architecture de r éseau multit âche qui est utilis ée pour d étecter les changements et effectuer une cartographie automatique simultan ément, ce qui permet une compr éhension s émantique des changements d étect és. Ensuite, un filtrage de classification et un nouvel algorithme de diffusion anisotrope guid ée sont utilis és pour r éduire l'effet du bruit d'annotation, un d éfaut r écurrent pour les ensembles de donn ées à grande échelle g én ér és automatiquement. Un apprentissage faiblement supervis é est également r éalis é pour effectuer une d étection de changement au niveau des pixels en utilisant uniquement une supervision au niveau de l'image gr âce à l'utilisation de cartes d'activation de classe et d'une nouvelle couche d'attention spatiale. Enfin, une m éthode d'adaptation de domaine fond ée sur un entraînement adverse est propos ée. Cette m éthode permet de projeter des images de diff érents domaines dans un espace latent commun o ù une t âche donn ée peut être effectu ée. Cette m éthode est test ée non seulement pour l'adaptation de domaine pour la d étection de changement, mais aussi pour la classification d'images et la segmentation s émantique, ce qui prouve sa polyvalence. Title: Convolutional Neural Networks for Change Analysis in Earth Observation Images with Noisy Labels and Domain Shifts Keywords: Remote sensing, change detection, convolutional neural networks, multitask learning, weakly supervised learning, domain adaptation.

  un nouvel algorithme de filtrage anisotrope guid é qui permet de mieux adapter la pr édiction aux bords des images d'entr ée. Le chapitre 5 explore également comment la d étection des changements au niveau des pixels peut être effectu ée en utilisant uniquement des étiquettes au niveau de l'image comme supervision de fac ¸on faiblement supervis é. Une couche d'attention spatiale fixe est apprise pendant l'entraînement pour un r éseau de classification, et la synergie entre cette couche d'attention et le classifieur du r éseau am éliore la capacit é du r éseau à localiser les changements. Ces r ésultats sont également am élior és par l'utilisation de l'algorithme de diffusion anisotrope guid ée.Le chapitre 6 d écrit une approche d'apprentissage antagoniste pour r éduire le diff érences entre les images de diff érents types de changements dans un espace latent, en utilisant divers types de catastrophes naturelles comme études de cas. Les diff érentes catastrophes naturelles peuvent avoir des apparences visuelles tr ès diff érentes dans

Le chapitre 5 étudie l'impact du bruit des étiquettes sur le processus d'apprentissage pour la d étection de changements, et emprunte des id ées à l'apprentissage faiblement supervis é pour am éliorer la pr écision des r ésultats des r éseaux. Le biais des étiquettes dans l'ensemble de donn ées pr ésent é pr éc édemment et g én ér é automatiquement a un effet de biais sur les cartes de changement pr évues par les r éseaux. Un processus it ératif d'entraînement et de nettoyage des étiquettes est utilis é pour augmenter la robustesse du r éseau au bruit des étiquettes, et il est coupl é à les images de t él éd étection, de sorte que les r éseaux neuronaux entraîn és pour d étecter les changements caus és par un type d' év énement ne se g én éralisent pas bien aux autres catastrophes. L'apprentissage antagoniste est utilis é pour projeter les images dans un espace latent commun o ù la supervision de la d étection de changements provient d'un domaine source, et peut être appliqu é aux images du domaine cible. La m éthode propos ée est également test ée sur d'autres t âches d'adaptation de domaine en vision par ordinateur pour montrer sa polyvalence.

  . 2.3. In image segmentation, pixels are simply grouped in meaningful sets that belong to the same object or region. The definitions of what consists of an object or region can be subjective. Picture an image of a cube that is to be segmented: should the whole cube be a single region, or should each face be separated into a different region? In semantic segmentation, on the other hand, a class label is attributed to each pixel based on a known set of possible classes. Pixels in separate regions can belong to the same class. This problem, given a set of classes, is often more accurately defined than that of image segmentation. Semantic segmentation algorithms often depend on learning from examples from each class. These characteristics will be better discussed later on.

	method only really gained traction after it was used by Dalal and Triggs for human detection in 2005 [DT05]. Such
	works inspired other feature descriptors such as SURF
	Several methods have marked the progress of research into image segmentation algorithms. Mumford and Shah
	tackled this problem through the minimization of an energy functional containing terms for sub-region smoothness,
	boundary length, and other desirable properties. Vincent and Soille proposed the watershed method, inspired by the
	catchment basins of rain water [VS91]. Shi and Malik modelled image segmentation as a graph partitioning problem
	in a method known as normalized cuts [JM00]. The mean shift method was applied for image segmentation by
	Comaniciu and Meer, although the method had already been proposed in different contexts [CM02].
	The Eigenfaces work by Turk and Pentland was one of the first to introduce machine learning to [TP91]. Principal
	component analysis was used on training images to find a set of bases able to represent any face. Distances in
	this new representation space was used to discern between faces of different people and perform facial recognition.
	Another early application of machine learning that had a strong impact in both academia and industry was the object
	detection framework proposed by Viola and Jones [VJ04]. It used cascading classifiers based on Haar features that
	could be efficiently computed, boosted using the AdaBoost algorithm [FSA99]. The efficiency and robustness of this
	method made it the standard for face detection for many years.
	The development of SIFT descriptors in 1999 by Lowe sparked the interest of researchers for local feature
	descriptors due to its excellent performance in applications such as stereo matching, object recognition, tracking,
	stitching, and others [Low99, Low04]. It was technically preceded by HOG descriptors in 1986 [McC86], but this

  What is the best next move in the game of Go [SHM + 16]? What is the most efficient way to cool data centres [Gao14, EG20]? What did Claude Monet see as he placed his easel by the bank of the Seine near Argenteuil on a lovely spring day in 1873 6 [ZPIE17]? It is possible to tackle such problems with machine learning methods if we step outside of supervised learning.

  The training of GANs is notoriously unstable [SGZ+ 16]. If either the generator or the discriminator becomes much more powerful than the other, the training tends to degenerate and produce useless results through mode collapse. Significant work has been put into improving the training of GANs. LSGAN [MLX + 17] and Wasserstein GAN[START_REF] Arjovsky | [END_REF] tackle the problem of vanishing gradients in the discriminator. The pix2pix work proposed a dense discriminator that predicts a label for all overlapping 70x70 patches, producing more robust results[START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF]. The CycleGAN framework proposed a two-way translation coupled with a reconstruction loss that is very effective against mode collapse. Research into the stability and usage of such adversarial networks is still of high interest.

Adversarial training has since been used for many other applications such as face editing [KLA19, YPN + 20], image-to-image translation [IZZE17, ZPIE17], domain adaptation [HTP + 18, VJB + 19], and several others.

Table 2

 2

.1: Two class confusion matrix.

Table 3 .

 3 1 contains the quantitative evaluation of the proposed CD architectures, along the same measures of other state-of-the-art methods. For the AC dataset, the methods user for comparison were DSCN [ZFY + 17], CXM [BS09], and SCCN [LGQZ16], using the values claimed by Zhan et al. in [ZFY + 17]. Benedek et al. proposed the CXM method that uses a conditional mixed Markov model that identify changes using complementary features of global intensity statistics and local correlation [BS09]. Liu et al. proposed the SCCN method, where a symmet-

rical CNN with coupling layers is used to generate pixel-level descriptors in an unsupervised fashion

[START_REF] Liu | A deep convolutional coupling network for change detection based on heterogeneous optical and radar images[END_REF]

.

Table 3

 3

	Data	Network	Prec. Recall Accuracy	F1
	OSCD-3 ch.	Siam. EF FC-EF FC-Siam-conc FC-Siam-diff	21.57 79.40 21.56 82.14 44.72 53.92 42.89 47.77 49.81 47.94	76.76 83.63 94.23 94.07 94.86	33.85 34.15 48.89 45.20 48.86
	OSCD-13 ch.	Siam. EF FC-EF FC-Siam-conc FC-Siam-diff	24.16 85.63 28.35 84.69 64.42 50.97 42.39 65.15 57.84 57.99	85.37 88.15 96.05 93.68 95.68	37.69 42.48 56.91 51.36 57.92
	Szada/1	DSCN [ZFY + 17] CXM [BS09] SCCN [LGQZ16] FC-EF FC-Siam-conc FC-Siam-diff	41.2 36.5 24.4 43.57 62.65 57.4 58.4 34.7 40.93 65.61 41.38 72.38	---93.08 92.46 92.40	47.9 44.9 28.7 51.40 50.41 52.66
	Tiszadob/3	DSCN [ZFY + 17] CXM [BS09] SCCN [LGQZ16] FC-EF FC-Siam-conc FC-Siam-diff	88.3 61.7 92.7 90.28 96.74 85.1 93.4 79.8 72.07 96.87 69.51 88.29	---97.66 93.04 91.37	86.7 74.3 85.8 93.40 82.65 77.78

case, the best F1 score was obtained by our FC-EF architecture, while the other architectures were outperformed by DSCN and SCCN. Once again the inference time of the fully convolutional architectures were below 0.1 s, which .1: Quantitative evaluation of the proposed methods on the OSCD and Air Change datasets. is a speedup of over 500x compared to the processing time of 50 s claimed by Zhan et al. [ZFY

+ 17]

for the SCCN method on a similar setup. The results for these cases can be viewed in Figs. 3.10 and 3.11.

  Table 4.4: Change detection results of several methods on the OSCD dataset, for the RGB and multispectral (MS) cases. Results are in percent.

	Data	Network	Prec. Recall Tot. acc. Dice
	RGB	FC-EF FC-Siam-conc 42.89 47.77 44.72 53.92 FC-Siam-diff 49.81 47.94 FC-EF-Res 52.27 68.24	94.23 94.07 94.86 95.34	48.89 45.20 48.86 59.20
	MS	FC-EF FC-Siam-conc 42.39 65.15 64.42 50.97 FC-Siam-diff 57.84 57.99 FC-EF-Res 54.93 66.48	96.05 93.68 95.68 95.64	56.91 51.36 57.92 60.15

Table 4

 4

		CD			LCM
		Kappa Dice Tot. acc. Kappa Tot. acc.
	Strategy 1 Strategy 2 Strategy 3 Strategy 4.1 Strategy 4.2 CNNF-O CNNF-F PCA+KM	3.99 21.54 12.48 13.79 5.56 -19.13 20.23 25.49 26.33 0.74 2.43 3.28 4.84 0.67 2.31	86.07 98.30 94.72 96.87 98.19 64.54 88.66 83.95	71.92 -71.92 67.25 71.81 ---	87.22 -87.22 85.74 89.01 ---

.5: Change detection (CD) and land cover mapping (LCM) results of all four of the proposed strategies on the HRSCD dataset. Comparison with the methods proposed by [EALW16] (Otsu [CNNF-O] and fixed [CNNF-F] thresholding) and by [Cel09] ([PCA+KM]) are included. Results are in percent.

  Table 4.6: Change detection results on Eppalock lake test images. Results are in percent. The performance of two other change detection methods are also shown in Table 4.5. The first method, proposed in [EALW16], is based on transfer learning and uses features from a pretrained VGG-19 model [SZ15] to create pixel descriptors, whose Euclidean distance is used to build a difference image. The original method uses

			-LSTM	EF
	Binary CD	Tot. acc. Kappa No change Change	98.67 97.28 98.83 98.46	99.35 98.67 99.47 99.19
	Semantic CD	Tot. acc. Kappa No change City exp. Soil change Water change	98.70 97.52 98.49 84.72 100 99.25	98.48 97.10 97.73 100 86.07 99.93

Table 5 .

 5 1: Accuracy and standard deviation for each test on ABCD dataset using 5-fold cross validation. Fixed scale and resized variations of the ABCD dataset were tested. Results from methods proposed by Fujita et al. are included for comparison.

		Fixed scale Resized
	6-ch [FSI + 17] siam [FSI + 17]	94.5 ± 0.5 94.8 ± 0.3	94.7 ± 0.3 94.9 ± 0.4
	89.33 ± 0.79 90.96 ± 0.65 94.36 ± 0.26 94.88 ± 0.18 Attention + GAD 94.58 ± 0.27 94.90 ± 0.22 No attention Attention

  Deep neural networks excel at learning to perform various visual tasks from large amounts of accurately labeled data. Such networks often fail at generalizing to new datasets with different characteristics, even in cases that humans find trivial. Such failures often occur when the target domain is shifted in any way from the source domain, e.g.synthetic to real images, different geographical regions, different weather conditions, or different sensors [PUK+ 17, SDVG18]. Ideally, these systems would be robust to domain shifts or, alternatively, allow for unsupervised domain adaptation (UDA).Most UDA methods fall into one of three main groups. The first group of methods attempt to align the domains in a feature space [LCWJ15, SS16, GL15, THSD17]. Methods in the second group aim to translate target data into the source domain, and use this representation to perform the desired task [HTP + 18, LT16, GKZ+ 16]. Finally, a last group of methods attempt to promote the learning of operations that are robust to domain shifts by means of proxy measurements which do not require labels [LCWJ15, VJB+ 19]. The method presented in this chapter belongs to the first group, although a possible variation that will be presented later belongs to the third group. used for image translation. CyCADA [HTP + 18] performs image translation from target to source domain using a modified version of CycleGAN[START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] that attempts to preserve semantic information using a pre-trained network on source domain data, which assumes a network trained on source data should also apply to target data. ADDA combines discriminative modeling, untied weight sharing, and a GAN loss to approximate source and target domain in feature space[START_REF] Tzeng | Adversarial discriminative domain adaptation[END_REF]. Adapt-SegNet uses discriminators on segmentation maps instead of image features with good results [THS+ 18]. ADVENT uses adversarial training to reduce the entropy of segmentation predictions [VJB+ 19]

	Source
	Domain
		Cycle-Consistent Adversarial	Latent Space	Task
	Target	Training
	Domain

The human visual system possesses an incredible ability to extract abstract representations from images in a very robust way regardless of domain shifts. UDA algorithms in the first family described above attempt to emulate this capability, but they rely on proxies for comparing representations computed from different domains such as maximum mean discrepancy

[START_REF] Long | Learning transferable features with deep adaptation networks[END_REF]

, correlation distance

[START_REF] Sun | Deep coral: Correlation alignment for deep domain adaptation[END_REF]

, or adversarial training

[START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF][START_REF] Tzeng | Adversarial discriminative domain adaptation[END_REF]

. This is necessary since the source and target data are not paired and therefore representations cannot be compared directly.

Many UDA methods rely on adversarial training to approximate source and target domains. Such methods use a discriminator whose task is to separate data coming from source and target domains, be it at image level [HTP

+ 18]

, feature level

[START_REF] Tzeng | Adversarial discriminative domain adaptation[END_REF][START_REF] Hoffman | Fcns in the wild: Pixel-level adversarial and constraint-based adaptation[END_REF]

, or output level [THS + 18, VJB

+ 19]

. As noted in [HTP

+ 18]

, feature-level methods often fail at maintaining semantic consistency when translating domains, and struggle to encode low-level visual appearance variance.

We propose a novel UDA method, named domain-invariant encoding (DINE), that uses cycle-consistent ad-Figure

6

.1: DINE aims to extract a code in a latent space where the desired task can be performed regardless of the image's domain of origin. is

Table 6 .

 6 1: Classification accuracy. Models marked with † share encoder parameters. Models marked with ‡ share encoder parameters and are trained without the code similarity loss. Results marked with * convert SVHN images to grayscale to enable symmetric encoders.

	Input Image	I A	Enc A→B	C A,forward	Dec A→B	G B	T
	Shortcut					Disc B
	Decoded Image	Î A	Dec B→A	C A,backward	Enc B→A
	Figure 6.3: Shortcut decoding path, used for validating alignment of feature spaces during forward and backward translations.

Instance normalization [UVL17] is used after each convolution. The decoders Dec A→B and Dec B→A are composed of 4 residual blocks, followed by two convolutional layers with stride 1 2 . PatchGAN discriminators [IZZE17] are used for D A and D B , where 70 × 70 overlapping patches

Table 6 .

 6 3: Dice score in change detection tests. Models marked with † share encoder parameters. Models marked with ‡ share encoder parameters and are trained without the code similarity loss. Column marked as "none" refers to source domain supervision. Results highlighted in yellow are the ones where target domain and test data are the same.

	Fire

Table 6 .

 6 4: Accuracy in change detection tests. Models marked with † share encoder parameters. Models marked with ‡ share encoder parameters and are trained without the code similarity loss. Column marked as "none" refers to source domain supervision. Results highlighted in yellow are the ones where target domain and test data are the same.

	Fire

Table A .

 A 2: Dates of acquisition of the images in the OSCD dataset.

	Band Central wavelength (nm) Resolution (m/px)
	8b 10 11 12	443 490 (blue) 560 (green) 665 (red) 705 740 783 842 865 945 1375 1610 2190	60 10 10 10 20 20 20 10 20 60 60 20 20
	Table A.3: Sentinel-2 bands and resolutions.
		123

Table A .

 A 4: Percentage of changes in each change map in the OSCD dataset.

	Split	Name	Change percentage
	Train	abudhabi aguasclaras beihai beirut bercy bordeaux cupertino hongkong mumbai nantes paris pisa rennes saclay e	3.76% 1.64% 2.49% 2.69% 0.74% 1.00% 2.37% 3.56% 2.56% 1.14% 0.29% 1.64% 2.58% 0.99%
	Test	brasilia chongqing dubai lasvegas milano montpellier norcia rio saclay w valencia	2.58% 7.21% 9.92% 7.67% 0.80% 6.79% 1.32% 5.69% 1.14% 0.44%
		Total	3.21%

Table B

 B

	Region	Number of images Total number of pixels Total surface area (km 2 )
	D14 -Calvados D35 -Ille-et-Vilaine	101 190	1.01 • 10 10 1.9 • 10 10	2525 4750
	Total	291	2.91 • 10 10	7275

.1: Size of the HRSCD dataset.

There are exceptions to this claim, such as evolutionary algorithms, but these are outside the scope of this work.

There are also exceptions to this, such as neural architecture search algorithms.

Or a set of coregistered images, as will be discussed later.

This architecture is called fully convolutional network, but the group of descending architectures are referred to as fully convolutional networks, which makes the nomenclature a bit confusing at this point.

Around this time it becomes trendy to name your method SomethingNet and title your paper "SomethingNet: A Network That Does Some-

Opening line from Zhu et al.'s work [ZPIE17], a highly recommended read.

Although there is no reason such techniques can be used for other astronomical bodies[START_REF] Henry | Viking landing sites, remote-sensing observations, and physical properties of martian surface materials[END_REF].

https://ieee-dataport.org/open-access/oscd-onera-satellite-change-detection

https://github.com/aboulch/medusa_tb

https://ieee-dataport.org/open-access/hrscd-high-resolution-semantic-change-detection-dataset

https://github.com/lucasb-eyer/pydensecrf

The notation here differs from the one used in[START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] for consistency with the following section.

Due to the dual symmetry of this method, some of the derivations are only shown for one of the two cases for the sake of simplicity.

https://www.gimp.org/

https://ieee-dataport.org/open-access/oscd-onera-satellite-change-detection

https://rcdaudt.github.io/oscd

https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode

https://www.data.gouv.fr/en/datasets/bd-ortho-r-50-cm/

http://www.ign.fr/

https://land.copernicus.eu/local/urban-atlas/change-2006-2009

https://gdal.org/

https://land.copernicus.eu/local/urban-atlas/urban-atlas-2006

https://land.copernicus.eu/local/urban-atlas/urban-atlas-2012

Acknowledgements

Code Max pool Fully con. Softmax Residual Blocks (a) Classification T Code Transp. conv. 1x1 conv. Softmax Residual Blocks (b) Segmentation T Code 1 Code 2 Residual Blocks Transp. conv. 1x1 conv.

Chapter 4

Semantic Change Detection Chapter Summary

The work presented in this chapter expands the fully convolutional network (FCN) architectures presented previously to allow them to understand as well as detect changes in image pairs. This semantic interpretation of image pairs enable a deeper understanding about the evolution of the imaged terrain. Instead of simply separating changed from unchanged regions, such systems could be used to answer questions such as:

• How much has the urban area expanded into agricultural land?

• How much deforestation has occurred?

• How much water surface loss has happened in this region?

• How much have the ice shelves in these images shrunk?

To enable the learning of such semantic understanding of multitemporal image pairs, a novel very high resolution semantic change detection dataset was created by combining two available databases: one which contained aerial images, and another which contained land cover and change vector annotations. This allowed for the creation of a much larger dataset than those which were previously available, and which contains change maps and land cover maps for all regions in the dataset. This dataset was also released to the scientific community.

Different methods are proposed and tested to solve this semantic change detection problem. These solutions range from comparing separately predicted land cover maps for changes, to proposing a single multi-class semantic segmentation network, and finally a multitask network that produces simultaneously a land cover map for each input image, as well as a binary change map. A staged training scheme is also proposed for training this last proposed network, which not only avoids setting a hyperparameter to balance different loss functions, but also leads to better results.

versarial training [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] to obtain a feature-space representation of a scene robust to source-to-target domain shifts. The unsupervised two-way image translation method CycleGAN [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] enables the network to find a domain-invariant feature space into which images can be projected from either the source or the target domains through the use of a simple 1 norm loss. We show that this method is more effective in enforcing feature coherence between source and target features than direct adversarial training, and can provide qualitative and quantitative measurements of how well the feature spaces are aligned.

We validate our domain adaptation method on three image analysis tasks: classification, semantic segmentation, and semantic co-segmentation. Classification tests are done using various digit recognition datasets, validating our method in a case where data sources differ. Our semantic segmentation tests show how effective our method is at bridging the gap between simulated and real data. Finally, we tackle change detection in remote sensing image pairs [Sin89, HCC + 13] as an image co-segmentation problem, showing that our method allows deep neural networks to generalize between different geographical areas and different types of natural disasters. Such tests show how the proposed method can help address harmful accidents that happen worldwide, and may help mitigate the consequences of such events.

Unsupervised Domain Adaptation

The motivation for unsupervised domain adaptation comes from the need to overcome the lack of supervision in some situations. In some cases, labeled data is expensive to acquire, while labeled synthetic data is much cheaper to generate [START_REF] Richter | Playing for data: Ground truth from computer games[END_REF]. The lack of labels is also an issue when timely results are necessary, such as when analyzing natural disaster images [GGP + 19]. UDA methods attempt to bridge the shift between a source domain where labeled data is available and the target domain where a desired task is to be performed. Supervised methods often fail when applied to data significantly different from that which has been seen during training, and UDA methods attempt to bridge that gap [START_REF] Csurka | Domain adaptation for visual applications: A comprehensive survey[END_REF].

Several successful UDA methods have proposed ways to approximate image representations in feature space.

In deep adaptation networks [START_REF] Long | Learning transferable features with deep adaptation networks[END_REF] hidden representations of layers are embedded in a reproducing kernel Hilbert space where the mean embeddings of different domain distributions can be explicitly matched. In deep CORAL [START_REF] Sun | Deep coral: Correlation alignment for deep domain adaptation[END_REF], second-order statistics of the source and target distributions are aligned with a linear transformation. are analyzed in a fully convolutional manner. This discriminator architecture is able to work with images of any size, and has fewer parameters than a full-image discriminator [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF].

Different architectures for T were explored for each task, as the architecture of T is task specific. The chosen architectures are based on the decoder architecture described above. For the classification task, the final two upsampling convolutional layers are replaced by a global max pooling operation, followed by a fully connected layer. To obtain the segmentation network, the decoder architecture's last layer is replaced by a 1 × 1 convolution followed by a softmax layer. For the co-segmentation task, each image is encoded separately, and the codes are concatenated before being decoded by an architecture identical to the segmentation one, except for the number of input and output channels. This effectively creates a Siamese network [START_REF] Chopra | Learning a similarity metric discriminatively, with application to face verification[END_REF][START_REF] Zagoruyko | Learning to compare image patches via convolutional neural networks[END_REF] adversarial training between grayscale and RGB images.

Segmentation

The semantic segmentation tests focus on the adaptation from synthetic GTA5 images to real world Cityscapes images, using the 19 common classes between the two datasets as is done in [HWYD16, THS + 18, HTP + 18, VJB + 19]. Training is done using the training images of each dataset, and testing is done using the validation images from the Cityscapes dataset. While running comparison tests we have realised that the tests presented in most UDA papers vary not only in method, but also in the segmentation architectures that are used. The different backbones have a strong impact in the final results, and these results make it hard to identify if the performance gain comes from the proposed method of from using a better backbone segmentation network. For this reason, we have split our results in two groups. First, results are compared using the reported results for several UDA results and their original segmentation architectures. Second, we used the official codes found online for several UDA methods and tested using a standardized simple segmentation backbone, which should isolate the effect of the considered UDA methods from the chosen architectures.

Table 6.2 contains the obtained semantic segmentation results, as well as a comparison of the used segmentation backbones and number of trainable parameters. The first part of the table contains the results reported in the original paper of each method. The second part contains the results of our tests where several methods were tested using the ResNet-9 segmentation backbone. DINE outperformed all other methods when tested with comparable segmentation architectures. We believe DINE's performance would be more comparable to those reported in other papers if a more advanced segmentation network is used, but our computational setup did not allow for these tests to be made. These results also show that sharing the encoder weights between source and target encoders leads to a performance reduction. This suggests that learning domain-invariant representations is more powerful than the tasks of image classification, semantic segmentation, and co-segmentation. The results were mostly positive, although its performance did not match that of other method in some tests. In one of the classification tests, DINE's performance was superior to target only supervision due to larger training size and the self-supervision of CycleGAN.

Its performance in segmentation was superior to other methods when methods were compared with an identical segmentation network. The co-segmentation tests showed how such methods can help interpret aerial images in the case of natural disasters, even when no disaster-specific labels are available for training neural networks.

Appendix A

ONERA Satellite Change Detection Dataset Images

The ONERA Satellite Change Detection (OSCD) dataset contains image pairs that were captured by the Sentinel-2 satellites 1 . The images were obtained by following two basic steps:

1. Create a geojson file that precisely defines the coordinates of the region of interest using a tool such as geojson.io 2 . Urban areas were chosen in several different continents with the aim of obtaining representative examples of different geographical areas.

2. Images were downloaded using the sentinelsat API 3 with help of the Medusa toolbox 4 . The first and last available images with high visibility (i.e. very small number of clouds) were downloaded and cropped using the coordinates from the geojson files.

Given that the dates of the development of this dataset and the launch of the Sentinel-2 satellites, the image pairs cover a temporal distance of 1-2 years. More information about the chosen regions can be found in Tabs. A.1 and A.2.

Sentinel-2 images contains different different bands inside and outside the visible spectrum [ESA20b]. Spectral bands come at three different resolution, ranging between 10 and 60 meters per pixel as described in Tab. A.3.

The different image resolutions across spectral bands require an additional step for considering the images simultaneously. Each band was cropped according to the chosen geographical coordinates contained in the geojson files, and then were resampled using the zoom 5 function of the scipy python library 6 . Both the original and resampled

Images

The images in the High Resolution Semantic Change Detection (HRSCD) dataset come from the BD ORTHO 1 database, which is provided by the French Institut National de l'Information G éographique et Foresti ère (IGN) 2 . This database contains orthorectified RGB mosaics created using aerial imagery of urban areas in France at a resolution of 50 cm/px split into crops of size 10000x10000 pixels. Some of the images in the BD ORTHO database are under an open licence (licence ouverte), which means they are openly downloadable and redistributable under certain conditions. Others can be accessed (but not redistributed) at no cost for research purposes if a request is made.

For reasons that will become clear in the next section, only regions for which images were available around the

Size

The download size of the HRSCD dataset is 9.2 GB. The dataset files occupy 12.5 GB of disk space when uncompressed. More information about the size of the dataset can be found in Tab. B.1.

Contents

The dataset files contain:

1. 291 RGB aerial image pairs of size 10000x10000 pixels taken in the regions of Calvados and Ille-et-Vilaine in the GeoTIFF format.

2. Binary change maps for all image pairs in the dataset in the GeoTIFF format.

3. Land cover maps for all images in the dataset in the GeoTIFF format.

Access and Distribution

The dataset files, except for the 2006 images, can be downloaded from the official IEEE DataPort page 7 or from the mirrors found on the dataset page on my personal website 8 . The 2006 images can be downloaded using IGN's data portal 9 . The dataset is released under a Creative-Commons BY-NC-SA licence 10 .

7 https://ieee-dataport.org/open-access/hrscd-high-resolution-semantic-change-detection-dataset 8 https://rcdaudt.github.io/hrscd 9 http://professionnels.ign.fr/donnees 10 https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode