domain: spi phys math info

text:
Javiera Castillo-Navarro
email: javiera.castillonavarro@onera.fr

Bertrand Le Saux

Alexandre Boulch

Sébastien Lefèvre

On Auxiliary Losses for Semi-Supervised Semantic Segmentation

Keywords: Semi-Supervised Learning, Semantic Segmentation, Multitask Learning

The development of semi-supervised learning methods is essential to Earth Observation applications. Indeed, labeled remote sensing data are scarce and likely insufficient to train fully supervised models with good generalization capacities. Conversely, raw data are abundant and therefore it is crucial to leverage unlabeled inputs to build better deep learning models. This work addresses the problem of semisupervised semantic segmentation from a multi-task learning perspective. In this context, we explore several auxiliary tasks (reconstruction, unsupervised segmentation or self-supervision), and corresponding unsupervised losses, to perform along with semantic segmentation. Our experiments show the potential of semi-supervised learning approaches in a life-like scenario, outperforming a classical supervised setting.

Introduction

Semantic segmentation, i.e. the problem of pixel-wise image classification, is of special importance in Earth Observation (EO). Indeed, many applications can be addressed as a semantic segmentation task, such as land cover mapping, building recognition or change detection. These tasks help us to deeply analyze and better understand our planet. In the last decade, the development of deep learning techniques has allowed to perform semantic segmentation with impressive success in an automatic way. Unfortunately, most of these methods rely heavily on the availability of large amounts of annotated data to be trained on. For this reason, special attention have been brought recently to the development of semisupervised techniques, that leverage unlabeled data together with labeled data during the learning process. Semi-supervised learning is of significant interest in remote sensing, since labeled data are hard and costly to obtain, as they usually require expertise knowledge to be annotated, while raw data are continuously generated through satellites or drones.

Therefore, this work explores semi-supervised semantic segmentation from a multi-task learning perspective. In this context, a deep neural network is trained to perform two tasks simultaneously: the supervised semantic segmentation, as the main task, and an unsupervised auxiliary task. The latter enables to use prior information from data only -without labels-, in order to improve the results of the main task. Then, the objective function to optimize can be expressed as a weighted sum of two components: the segmentation loss -usually a standard cross-entropy loss-and the auxiliary loss. Yet, it is still unclear which task to perform along with semantic segmentation or which auxiliary loss to optimize.

In this paper, we review several auxiliary tasks and unsupervised losses for semi-supervised semantic segmentation. In particular, we present the Relaxed K-Means loss [START_REF] Castillo-Navarro | Semi-Supervised Semantic Segmentation in Earth Observation: The MiniFrance suite, dataset analysis and multi-task network study[END_REF] and show it is very promising w.r.t. the state-of-the-art. We also highlight the interest of the Chrischurch Aerial Semantic Dataset (CASD) [START_REF] Randrianarivo | Urban Structure Detection with Deformable Part-based Models[END_REF] for evaluating semi-supervised learning. More generally, we demonstrate the relevance of semi-supervised learning over supervised approaches in a life-like scenario where labeled data are scarce, while unlabeled data are abundant.

Related Work

Semi-Supervised Learning for Semantic Segmentation. The general idea is to learn a representation function -mapping a data point to its target-from labeled data and simultaneously leverage unlabeled data to improve this representation by learning the intrinsic properties of data. Existing semi-supervised methods for semantic segmentation in deep learning rely mostly on weak supervision, using scribbles [START_REF] Durand | WILDCAT: Weakly Supervised Learning of Deep ConvNets for Image Classification, Pointwise Localization and Segmentation[END_REF], bounding boxes [START_REF] Khoreva | Simple Does It: Weakly Supervised Instance and Semantic Segmentation[END_REF] or image-level annotations [START_REF] Papandreou | Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation[END_REF] to produce pixel-wise predictions. Fewer are the works that use completely unlabeled data during the learning process. For instance, co-training methods usually train an ensemble of segmentation models and use non-annotated images to exchange information between each other [START_REF] Peng | Deep co-training for semi-supervised image segmentation[END_REF]. Other studies [START_REF] Souly | Semi-supervised Semantic Segmentation using Generative Adversarial Network[END_REF][START_REF] Hung | Adversarial Learning for Semi-Supervised Semantic Segmentation[END_REF] propose adversarial approaches that integrate unlabeled data during training.

Self-Supervised Learning. Self-supervised (or unsupervised) learning aims to learn from the data only, with no labels nor specific task objective [START_REF] Lecun | Self-Supervised Learning[END_REF]. More exactly, it aims to accomplish pretext tasks that are self-induced, such as autoencoding or blank completion, often for pre-training. In computer vision, usual pretext tasks are: predicting rotations [START_REF] Gidaris | Unsupervised Representation Learning by Predicting Image Rotations[END_REF], finding relative position of patches [START_REF] Doersch | Unsupervised Visual Representation Learning by Context Prediction[END_REF], solving the jigsaw puzzle [START_REF] Noroozi | Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles[END_REF], inpainting [START_REF] Pathak | Context Encoders: Feature Learning by Inpainting[END_REF] or colorization [START_REF] Zhang | Colorful Image Colorization[END_REF].

Semi-and Self-Supervised Learning in Earth Observation. In the field of remote sensing, different approaches to benefit from unlabeled data have been developed. Non-annotated examples are used to design sharper feature extractors in [START_REF] Xia | Semi-) Supervised Probabilistic Principal Component Analysis for Hyperspectral Remote Sensing Image Classification[END_REF], or to align manifolds for data coming from different modalities in [START_REF] Tuia | Semi-supervised Manifold Alignment of Multimodal Remote Sensing Images[END_REF]. Lately, deep learning approaches have been developed to leverage weakly annotated data for different applications: land cover classification [START_REF] Nivaggioli | Weakly Supervised Semantic Segmentation of Satellite Images[END_REF], change detection [START_REF] Caye Daudt | Guided Anisotropic Diffusion and Iterative Learning for Weakly Supervised Change Detection[END_REF] or building extraction [START_REF] Bonafilia | Building High Resolution Maps for Humanitarian Aid and Development with Weakly-and Semi-Supervised Learning[END_REF]. Some recent works integrate unlabeled data to the learning process, like [START_REF] Tao | Unsupervised-Restricted Deconvolutional Neural Network for Very High Resolution Remote-Sensing Image Classification[END_REF] that proposes an alternating scheme to perform semisupervised semantic segmentation, or [START_REF] Zhu | Semi-supervised Center-based Discriminative Adversarial Learning for Cross-Domain Scene-Level Land-Cover Classification of Aerial Images[END_REF] that considers an adversarial training strategy and uses unlabeled data for domain adaptation purposes.

Application of self-supervised methods in remote sensing is very recent. For instance, [START_REF] Swope | Representation Learning for Remote Sensing: An Unsupervised Sensor Fusion Approach[END_REF] proposes a new contrastive approach especially designed for multisensor data, while [START_REF] Vincenzi | The Color out of Space: Learning Self-supervised Representations for Earth Observation Imagery[END_REF] introduces a colorization pretext task adapted to remote sensing data, reconstructing visible colors from high-dimensional spectral bands.

Semi-Supervised Semantic Segmentation

In this paper, we approach semi-supervised semantic segmentation as a multitask problem. On the one hand, we learn to perform semantic segmentation under supervision using labeled data, and on the other hand, we learn to solve an auxiliary, unsupervised, task with raw data.

We choose to use BerundaNet-late [START_REF] Castillo-Navarro | Semi-Supervised Semantic Segmentation in Earth Observation: The MiniFrance suite, dataset analysis and multi-task network study[END_REF], a simple and efficient network to address the multi-task semi-supervised semantic segmentation problem. Berun-daNet consists of one decoder and a double decoder, where one decoder learns the supervised semantic segmentation task, while the second decoder is trained to perform an auxiliary task. Let φ s (•) and φ u (•) be the functions learned by the supervised and the unsupervised branch of the network, respectively. Let x be the input image and y the target label, the semi-supervised loss is expressed as a weighted sum of losses for each task:

L(x) = L s (φ s (x), y) + λL u (φ u (x), x) (1)
where L s is a supervised classification loss (usually cross-entropy loss), and thus the only one depending on y, and L u is an unsupervised loss chosen according to the auxiliary task to perform. Section 4 presents some unsupervised tasks that can be performed along with semantic segmentation. λ is an hyperparameter of the model. Since φ s and φ u partially share parameters, the additional unlabeled data will help to capture intrinsic properties through the unsupervised task. BerundaNetlate in particular has an almost-all shared parameters architecture, where only the last layers are split into specific tasks, as shown in Figure 1. Besides, BerundaNetlate is a versatile network and any classic encoder-decoder architecture can be used as backbone.

Auxiliary Tasks and Losses

In the semi-supervised setting described above, one could consider different auxiliary or pretext tasks to perform along with supervised semantic segmentation.

In particular, we study three groups of auxiliary unsupervised tasks, which are associated to different kinds of unsupervised loss functions. unsupervised/supervised shared layers, is used for supervised layers only and is for unsupervised layers.

Notation:

In what follows, we still note x the input image and x i is the i-th pixel of the image. x denotes the reconstructed version of x (x i , respectively).

N is the number of pixels in an image.

Reconstruction Losses

The objective of the reconstruction (or auto-encoding) task is to generate an output as close as possible to the original input. For this reason, reconstruction losses enforce a similarity between the output of the network and the input image. Classical reconstruction losses are based on p-norms. In what follows we consider L 1 and L 2 reconstruction losses as defined in equation ( 2):

L 1 (x) = 1 N N i=1 ||x i -xi || 1 , L 2 (x) = 1 N N i=1 ||x i -xi || 2 2 .
(2)

Segmentation Losses

Image segmentation aims to partition an image into multiple segments, where pixels in a segment share some properties, like color, intensity, or texture. This task can be performed in an unsupervised manner -based on the input image only-and might be a better complement to the supervised semantic segmentation task. We consider in this work two different unsupervised losses to perform unsupervised image segmentation.

Relaxed K-means loss [START_REF] Castillo-Navarro | Semi-Supervised Semantic Segmentation in Earth Observation: The MiniFrance suite, dataset analysis and multi-task network study[END_REF]. The goal is to find an optimal set of K colors for encoding the image. As in the classic k-means algorithm, the relaxed kmeans alternatively optimizes centroids of color clusters c k (k ∈ {1, K}) and membership matrices ŷ(k) of x to cluster k.

Formally, the objective to minimize is given by equation ( 3):

L km (x) = L 1 (x, x c ) + α K k=1 N i=1 ŷ(k) i (1 - ŷ(k) i ), (3)
x c corresponds to a quantized version of the image, that is obtained by

x c = K k=1 c k ⊗ ŷ(k) , with c k = N i=1 x i ŷ(k) i N i=1 ŷ(k) i . (4)
The relaxed k-means setting considers memberships ŷ(k)

i ∈ [0, 1] that can be obtained through a neural network (usually a soft-max output). ⊗ is the outer product.

Mumford-Shah loss. Classical image segmentation methods solve the segmentation problem by minimizing energy functions, such as the Mumford-Shah functional [START_REF] Mumford | Boundary detection by Minimizing Functionals[END_REF]. Recent work [START_REF] Kim | Mumford-Shah Loss Functional for Image Segmentation with Deep Learning[END_REF] shows that this functional can be adapted to be used as an unsupervised loss function in a deep neural network framework.

The unsupervised segmentation loss is then expressed as:

L M S (x) = K k=1 N i=1 ||x i -c k || 2 2 ŷ(k) i + α K k=1 N i=1 |∇ŷ (k) i | (5)
where we kept the same notations as before.

For both unsupervised segmentation losses, there are two hyperparameters to set: α, a regularization weighting parameter and k, the number of unsupervised clusters to identify. In our experiments, we set α = 1 and we compare results for k ∈ {5, 10}. Section 5 reports only results for k = 10 since it exhibits better peformances.

Self-Supervised Losses

Recently, self-supervised methods have shown impressive results on learning data representations. The main idea behind self-supervision is to build a supervised task from completely unlabeled data by producing labels from the data itself. Many self-supervised tasks have been proposed lately, and we explore here two pretext tasks to perform along with semantic segmentation, that can be easily integrated to our semi-supervised framework.

Inpainting Similarly to the context autoencoder [START_REF] Pathak | Context Encoders: Feature Learning by Inpainting[END_REF], we aim to solve the problem of filling in a missing piece in the image. The loss function is then expressed in terms of L 2 distance as

L ca (x) = L 2 (M x, M φ u ((1 -M ) x)) ( 6
)
where M is a binary mask (value of 1 for dropped pixels and 0 for input pixels) and the element-wise product.

There is an intrinsic hyperparameter to the inpainting problem: c, the crop size to mask from the image. In our experiments we try c ∈ {80, 160} and in Section 5 we report results for c = 80 since it led to the best results. In our settings, masks are randomly chosen over the image.

Jigsaw puzzle Solving jigsaw puzzles using neural networks was first proposed by [START_REF] Noroozi | Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles[END_REF] to learn visual representations. In brief, the task consists in cutting out the image into 9 patches, shuffle them and train the network to retrieve the original image.

In practice, we follow here a similar approach to [START_REF] Carlucci | Domain Generalization by Solving Jigsaw Puzzles[END_REF], where a network is trained to solve two tasks simultaneously (in our case, the jigsaw puzzle and the semantic segmentation) and the input is an image with permuted patches. The problem is then formulated as a classification task, using standard cross-entropy loss. We use the maximal Hamming distance algorithm from [START_REF] Noroozi | Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles[END_REF] to define a set of P allowed patch permutations. In our experiments we compare results for P ∈ {30, 100}. Since P = 100 led to the best results, we report them in Section 5.

Experiments

The Christchurch Aerial Semantic Dataset [START_REF] Saux | Christchurch Aerial Semantic Dataset[END_REF] The CASD comprises aerial imagery (≈ 5000 × 4000 px per image) at 10 cm/px resolution over Christchurch, New Zealand. Images were captured after the earthquake that struck the area on February 2011 and made available by Land Information New Zealand1 . Dense semantic annotations were produced by ON-ERA/DTIS on 4 images [START_REF] Audebert | Segment-before-Detect: Vehicle Detection and Classification through Semantic Segmentation of Aerial Images[END_REF][START_REF] Randrianarivo | Urban Structure Detection with Deformable Part-based Models[END_REF], considering 4 classes: buildings, cars, vegetation and background. The dataset also includes 20 aerial images without any kind of annotation, which makes it suitable for semi-supervised learning algorithms. In practice, we use a training partition containing labeled and unlabeled data -2 annotated tiles and 20 non-annotated tiles-, and keep 2 annotated tiles for validation.

Implementation Details

BerundaNet-late is used here with a U-Net [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF] backbone. For all experiments, it is trained using Adam optimizer [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF] with a fixed learning rate of 10 -4 , during 50 pseudo-epochs. Each pseudo-epoch consists in 5000 labeled samples and 5000 unlabeled samples (for the fully supervised experiments only labeled data is used). One sample is a 320 × 320 tile randomly chosen from training data. During testing time, tiles are processed with a sliding window of 320 × 320 pixels and overlap of 75%. PyTorch [START_REF] Paszke | PyTorch: An Imperative Style, High-Performance Deep Learning Library[END_REF] is used for all implementations.

Due to the stochastic nature of the optimization process, all experiments are averaged on 4 runs to obtain statistically significant results.

How to Harness Unlabeled Data?

Table 1 summarizes the results of our experiments with semi-supervision. Interestingly, for every loss studied in Section 4, there exists one (or more) λ value that allows to outperform the supervised setting.

Nevertheless, the best scores are obtained with unsupervised segmentation losses, where the approach with the relaxed K-means loss shows significant improvements, with respect to the supervised setting: mIoU score improves by +3.39%, while overall accuracy increases by +1.97%. Figure 2 shows two visual examples of the different methods. In the first example, the supervised approach is the only one that mistakes the shadow of trees over the river as a building; the supplementary information provided by unlabeled images to the semi-supervised methods allows to prevent this error. In the second image, the L km loss is the only one that correctly segments the central building, likely thanks to the color clustering capacity.

Influence of the λ Hyperparameter

We also study the impact of the weighting parameter λ on the segmentation performance. Figure 3 illustrates the average behavior of each loss with respect to the value of λ.

Three behavioral groups appear. Segmentation losses are robust to the choice of λ and show, in general, better performances. L 1 and L js work better for small λ and require cautious hyperparameter tuning, as they are close to the fullysupervised case. L 2 and L ca losses show the same optimum for λ = 5, which comes likely from the fact that inpainting uses L 2 to estimate discrepancies. In a multi-task setting, different tasks might have very different behaviors and orders of magnitude. Tuning a weighting hyperparameter is not straightforward and further work is needed to find a neat normalization. Some works have even focused on adapting the multi-task loss balancing during training [START_REF] Chen | GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks[END_REF].

Conclusions

In this work, we have explored semi-supervised semantic segmentation in Earth Observation from a multi-task learning perspective. We presented a review of several auxiliary tasks and unsupervised losses to be used in such a setting. We performed experiments over a life-like dataset, the CASD dataset, and showed that it is well-suited for semi-supervised learning. This study also brought out that unsupervised segmentation losses -and in particular the relaxed K-means loss-are suitable as auxiliary losses for semantic segmentation. Indeed, they outperformed other approaches and showed to be robust to the balancing hyperparameter (λ) of the model.

Finally, our experiments have shown the potential of semi-supervised learning approaches over simple supervised learning in a realistic scenario, where labeled examples are limited, while unlabeled images are abundant.

Fig. 1 .

 1 Fig. 1. BerundaNet-late architecture overview.unsupervised/supervised shared layers, is used for supervised layers only and is for unsupervised layers.

Fig. 2 .Fig. 3 .

 23 Fig. 2. Two examples of inference over the CASD dataset. buildings, cars, vegetation and background.

Table 1 .

 1 Results comparison for supervised and semi-supervised methods over the Christchurch Aerial Semantic Dataset.

	Mode	Aux. Task	Aux. Loss	λ	OA (%)	mIoU (%)
	Sup	-	-	-	81.06 ± 0.46	67.43 ± 0.49
		Rec	L1 L2	0.5 5	82.28 ± 0.55 82.36 ± 0.42	68.78 ± 1.27 68.99 ± 0.85
	Semi-sup	Seg	L km LMS	1 1	83.03 ± 0.42 82.94 ± 0.26	70.82 ± 0.35 70.24 ± 0.84
		Self	Lca Ljs	5 0.5	82.57 ± 0.59 82.88 ± 0.95	69.47 ± 0.7 70.17 ± 1.12

https://www.linz.govt.nz/land/maps/linz-topographic-maps/ imagery-orthophotos/christchurch-earthquake-imagery


Nina Cossais

Andrew O Thomas

Frederic Cherqui

Peter Morison

Darren Bos

Denis Martouzet

Elisabeth Sibeud

Anne Honegger

Stephanie Lavau

Tim D Fletcher

.10 Understanding urban water systems Understanding the challenges of managing SUDS to maintain or improve their performance over time Authors

Keywords: Asset management, Integrated Water Management, Public stakeholders, Stormwater, WSUD

A research agreement was signed with the metropolis of Lyon to work on barriers to SUDS implementation, and to search for pathways to

Introduction

Sustainable Urban Drainage Systems (SUDS) are increasingly popular, but their adoption faces many barriers. Available literature suggests that negative perceptions about their operation and maintenance (O&M) limit uptake. Knowing that poor O&M could jeopardise future implementation of SUDS, we set Cossais Nina, Andrew O. Thomas, Cherqui Frédéric, Morison Peter, Bos Darren, Martouzet Denis, Sibeud Elisabeth, Honegger Anne, Lavau Stéphanie, Fletcher Tim D. (2017) Understanding the challenges of managing SUDS to maintain or improve their performance over time, 14th IWA/IAHR International Conference on Urban Drainage, 10-15 September, Prague, Czech Republic. out to answer some fundamental questions about their short, medium and long-term O&M. Are they actually managed? By whom and under what conditions? What barriers limit the adequacy of O&M that SUDS receive? We apply different investigation methods and asset management concepts to overcome barriers and identify strategies that improve the O&M and long-term performance of SUDS.

Materials and Methods

In Melbourne, Australia, we undertook a literature review on the O&M of SUDS and its link with operational performance, and initiated a pilot study investigating the condition of SUDS at eight municipal councils. We then conducted 35 interviews with local government practitioners to better appreciate the perceptions and issues impacting their ability to manage SUDS.

In Lyon, France, we spent four months in the field with SUDS O&M workers to identify implemented techniques, demarcation of tasks, and successes and failures of stormwater management.

Consolidating our findings, we then identified global needs concerning SUDS O&M, and combined these with insights drawn from traditional asset management approaches (e.g. from sewer systems) to identify priority areas for improving the O&M and performance of SUDS.

Results and Discussions

The literature review identified only two studies that directly addressed SUDS O&M practices (Schlüter and Jefferies, 2005;[START_REF] Flynn | Operation and maintenance assessment for structural stormwater BMPs[END_REF]. Both identified a significant deficiency in current practice. The pilot study found similar results: five of the eight councils had no reliable data; the other three had high proportions of assets being in poor condition. Subsequent investigations has identified multiple challenges (Fig. 1) including: inadequate industry-wide capability and capacity; deficiencies in asset management, and a general lack of awareness from the general public of SUDS and their purpose. The fieldwork showed that different types of SUDS had different O&M issues. As SUDS are multifunctional assets, some tasks must be completed by different teams, increasing the complexity of O&M. Tab. 1 shows the demarcation of tasks between services or departments in charge of cleaning public space, sanitation, green spaces and roads in Lyon. Distribution of tasks, cost and responsibilities has been identified as a key issue. Further, SUDS O&M can involve crucial changes in the nature of tasks and their relevant professions.

Tab. 1: distribution of maintenance tasks in Lyon. C=Cleaning task, S=Sanitation task, G=Gardener task, R=Roads task, V=Various tasks. The meaning of each task (e.g C1, S2, etc.) is specified once below the sign. Author: N. Cossais

According to [START_REF] Schulting | Global developments of strategic asset management[END_REF], asset management in the water sector can be described as "managing infrastructure… to minimize the total cost… while delivering the service levels customers desire". In comparison to traditional sewer, this study shows how difficult defining SUDS performance and operational targets can be. SUDS are a special case because they tend: to be diffusely located; have highly variable size, shape, composition and functions; and, thus, require a diverse range of O&M.

Many barriers to O&M appear to be common between the French and Australian cases, including: deficiencies in information and asset management practices, a lack of industry capability and capacity, poor community awareness, and a lack of clarity in the demarcation and delegation of responsibilities between departments and tiers of government. Hence, both countries stand to benefit from working together to addressing these barriers by identifying and trialling different solutions and, in so doing, transitioning SUDS from an innovative concept to a mainstream technology.

We identify the following areas as priorities for further investigation and the exploration of potential solutions:

1. Formal review of classic asset management frameworks in various jurisdictions around the world to identify opportunities for application to SUDS O&M 2. Recognising that SUDS O&M requires specialist skills in both "grey" and "green" components through the establishment of multi-skilled SUDS O&M teams and formal training programs 3. In France, demarcation of responsibilities regarding performance and, in particular, hydraulic aspects 4. Consideration of long-term O&M implications at the planning stage, including "maintenance in design", likely catchment pollutant loads, O&M funding sources, and the commitment of all parties before construction 5. Review of existing government SUDS policy in terms of its impact on long-term O&M.

Conclusions

Despite Melbourne and Lyon having different arrangements for stormwater management, both studies indicate similar challenges. While there are some distinctive differences that can be traced to local factors, most issues are shared between both cities. Overall, this joint study suggests that O&M and the lasting performance of SUDS assets over time may be a universal problem experienced in cities incorporating SUDS across the globe.

Fig. 1 :

 1 Fig. 1: factors affecting SUDS O&M. Author: A. O. Thomas