Yuhang Zhang
email: zhangyuhang2019@email.szu.edu.cn

Shishun Tian
email: stian@szu.edu.cn

Muxin Liao
email: liao-muxin2020@email.szu.edu.cn

Zhengyu Zhang
email: zhengyu.zhang@insa-rennes.fr

Wenbin ) C Zou
email: wzou@szu.edu.cn

Chen Xu
email: xuchen@szu.edu.cn

Guangdong Key

Xu

Fine-grained Self-supervision for Generalizable Semantic Segmentation

Fine-grained Self-supervision for Generalizable Semantic Segmentation

Yuhang Zhang, Shishun Tian, Muxin Liao, Zhengyu Zhang, Wenbin Zou, Chen Xu

Abstract-Unsupervised domain adaptative semantic segmentation is a powerful solution for the distribution shift problem between the source and target domains. However, such methods need specified target domain data that may be unavailable in actual applications due to excess expensive collection. Generalizable semantic segmentation as a new paradigm appears in recent research, which aims to generalize well on distinct unseen domains only using source domain data. The existing methods focus on learning domain-invariant features by using global distribution alignment strategies, which may lead to a decreased discriminability of the model. To cope with this challenge, we propose a fine-grained self-supervision (FGSS) framework for generalizable semantic segmentation that takes into account both discriminability and generalizability from the perspective of the intra-class relationship. The FGSS framework contains singleview and multi-view versions. In the single-view version, we propose a fine-grained self-supervision strategy to distinguish the sub-parts of the semantic class for better class discriminability. In the multi-view version, we propose a class prototype feature enhancement strategy to generate another view (i.e. another representation of the original representation). Then, we propose a multi-view mutual supervision loss to enforce consistency between different views and further enhance the generalizability of the model. Experimental results on five widely-used datasets, i.e., GTAV, SYNTHIA, BDD100K, Cityscapes, and Mapillary, demonstrate that our FGSS framework achieves superior performance compared to state-of-the-art methods.

Index Terms-Semantic segmentation, Domain generalization, Fine-grained, Self-supervision, Intra-class relationship

I. INTRODUCTION

Propelled by the swift progress in deep neural networks [START_REF] Yang | Self-supervised video super-resolution by spatial constraint and temporal fusion[END_REF], Semantic segmentation, a crucial task in computer vision, has made remarkable progress in recent years relying on large amounts of annotations and has been broadly used in lots of applications such as autonomous driving [START_REF] Ji | Encoder-decoder with cascaded crfs for semantic segmentation[END_REF], [START_REF] Hua | Multiple relational learning network for joint referring expression comprehension and segmentation[END_REF]. This task infers a semantic category for each pixel of an image. Despite the rapid development of semantic segmentation in a fully-supervised manner, its drawbacks are also evident. First, labeling pixel-level annotation is laborious and timeconsuming. As mentioned in GTAV [START_REF] Richter | Playing for data: Ground truth from computer games[END_REF], labeling an image from real-world datasets CamVid [START_REF] Brostow | Semantic object classes in video: A high-definition ground truth database[END_REF] and Cityscapes [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF] takes 60 and 90 minutes, respectively. Consequently, some virtual datasets leverage the computer simulator to collect synthetic data containing automatically generated annotations. The model trained on these synthetic datasets is directly evaluated in the real-world environment. However, there is a discrepancy between the distributions of the training set and the validation set, called the distribution shift problem.

From coarse to fine

Unsupervised domain adaptation (UDA) technology partially overcomes these drawbacks [START_REF] Zhou | Context-aware mixup for domain adaptive semantic segmentation[END_REF], [START_REF] Zou | Dual geometric perception for cross-domain road segmentation[END_REF]. Generally, UDA transfers knowledge from a source domain with annotations to a target domain without annotations. It addresses not only the expensive data labeling but also the distribution shift problem in diverse domains, which yields better performance in the target domain. Unfortunately, the target domain data may be unavailable in practical applications. Meanwhile, the UDA model suffers from a dramatic drop in performance when the environment changes. Domain generalization (DG) is another approach to handle these shortcomings, aiming to transfer knowledge from the source data to all unseen scenes [START_REF] Liu | Domain generalization via encoding and resampling in a unified latent space[END_REF], [START_REF] Liao | Domaininvariant information aggregation for domain generalization semantic segmentation[END_REF] without utilizing the target domains at the training stage. In other words, a DG model is trained using only source domain data and then evaluated on the target domain data. Based on the number ACCEPTED MANUSCRIPT / CLEAN COPY of the source domain, DG can be categorized into singlesource DG and multi-source DG, where single-source DG is more challenging than multi-source DG due to its limited domain diversity and less training data. This paper focuses on devising a single-source DG model. Data manipulation and representation learning are two common approaches for DG. The former tries to extend the training set and enhance domain diversity by changing image style to cover new possible domains, while the latter focuses on domaininvariant representation learning. In some circumstances, both types of approaches are co-existing. For example, DRPC [START_REF] Yue | Domain randomization and pyramid consistency: Simulationto-real generalization without accessing target domain data[END_REF] randomized the style of the images using auxiliary datasets and then performed pyramid consistency to learn domaininvariant features. Although some progress have been made in the DG task, the existing approaches [START_REF] Pan | Two at once: Enhancing learning and generalization capacities via ibn-net[END_REF], [START_REF] Choi | Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening[END_REF] encourage global distribution alignment, which may lead to the removal of the discriminative information that helps recognize objects better [START_REF] Jin | Style normalization and restitution for domain generalization and adaptation[END_REF]. Recently, SAN-SAW [START_REF] Peng | Semantic-aware domain generalized segmentation[END_REF] proposed a semanticaware alignment method to align the distribution for each class independently, which alleviated the loss of local discriminative information. More recently, [START_REF] Zhou | Rethinking semantic segmentation: A prototype view[END_REF] pointed out that intra-class has various distributions since there are different meaningful sub-parts in a specific category. Nevertheless, such intra-class relationships are ignored in the existing DG methods. Therefore, it is significant to explore the intra-class relationship for better feature discrimination.

An intuitive idea is to distinguish sub-parts of the coarse semantic class, which belongs to the field of fine-grained recognization. Similarly, fine-grained semantic segmentation aims to assign a fine-grained label for each pixel of an image, where fine-grained labels are sub-parts of the coarse semantic categories. A concrete example is shown in Fig. 1, where a car is grouped into windows, bodies, and chassis. Recently, FGN [START_REF] Chang | Your" flamingo" is my" bird": Fine-grained, or not[END_REF] observed that fine-grained learning is beneficial to enhance the discrimination of coarse-grained classification, which is coincidentally consistent with the intuitive idea. To exploit this assumption, we propose a finegrained self-supervision (FGSS) framework for generalizable semantic segmentation considering both discriminability and generalizability. The FGSS framework contains single-view and multi-view versions. In the single-view version, we propose a fine-grained self-supervision strategy (FSS) to enhance the discriminability of the model. The feature of each coarse semantic class is clustered into distinct object sub-parts by online pseudo-label assignment and then supervised by the fine-grained self-supervision loss. To improve the generalizability, we extend the single-view version to the multi-view version with the proposed Class Prototype Feature Enhancement (CPFE) and Multi-View Mutual Supervision (MVMS). First, CPFE generates an augmented view that can be regarded as a representation with another style but with the same content as the original representation. The augmented features are the intermediate features between the original features and related class prototypes, which are implicitly pulled close to corresponding class prototypes to learn a domain-invariant classifier. Then, MVMS ensures multi-view consistency to learn a generalized representation. In summary, fine-grained segmentation serves as an auxiliary task to improve the generalizability and discriminability for generalized semantic segmentation. This demonstrates for the first time that incorporating fine-grained information is beneficial for generalized semantic segmentation.

Our contributions can be summarized as follows.

• We propose a fine-grained self-supervision framework for generalizable semantic segmentation, which ensures both discriminability and generalizability from the perspective of the intra-class relationships, with both single-view and multi-view versions.

II. RELATED WORK

A. Semantic segmentation

As a dense prediction task, semantic segmentation plays an important role in computer vision and has emerged in many applications, such as autonomous driving and robots [START_REF] Li | Rgb-t semantic segmentation with location, activation, and sharpening[END_REF]. Fully convolutional networks (FCNs) [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] and the encoderdecoder structure [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF] are two types of common basic architectures of semantic segmentation. To incorporate larger context information, feature pyramids were explored in PSP-Net [START_REF] Zhao | Pyramid scene parsing network[END_REF] and DeepLab-series [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF]- [START_REF] Chen | Encoderdecoder with atrous separable convolution for semantic image segmentation[END_REF]. Additionally, selfattention extracted the long-range context by aggregating all pixels of an image, such as DANet [START_REF] Fu | Dual attention network for scene segmentation[END_REF] and CCNet [START_REF] Huang | Ccnet: Criss-cross attention for semantic segmentation[END_REF]. More recently, vision transformer based structures [START_REF] Zheng | Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers[END_REF], [START_REF] Xie | Segformer: Simple and efficient design for semantic segmentation with transformers[END_REF] have been proposed. Such structures split an image into a sequence of tokens and forward it to Transformer layers for feature extraction. These methods achieved impressive performance but got drastic performance drops when the environment changed or with less supervision. Semantic segmentation with less supervision or the distribution shift problem is still challenging in the future.

B. Unsupervised domain adaptation

Unsupervised domain adaptation (UDA) aims to achieve superior performance in the target domain by reducing the domain gap between the source domain and the target domain [START_REF] Zhang | Unsupervised domain adaptation using robust classwise matching[END_REF], [START_REF] Zhao | Source-free open compound domain adaptation in semantic segmentation[END_REF]. In order to solve the time-consuming data labeling, annotations are used only in the source domain and not in the target domain. The semantic knowledge is transferred from the source domain to the target domain. The existing ACCEPTED MANUSCRIPT / CLEAN COPY UDA methods are roughly parted into three categories, namely appearance adaptation, representation adaptation, and selftraining. Changing the image style from the source domain to the target domain is the main idea of the appearance adaptation. Cycada [START_REF] Hoffman | Cycada: Cycle-consistent adversarial domain adaptation[END_REF] proposed a novel cycle-consistent adversarial adaptation. FDA [START_REF] Yang | Fda: Fourier domain adaptation for semantic segmentation[END_REF] replaced the low-frequency information of the source images with that of the target images due to the rich domain-invariant information in low-frequency. DACS [START_REF] Tranheden | Dacs: Domain adaptation via cross-domain mixed sampling[END_REF] performed a cross-domain mixing strategy to solve the class conflation problem. Representation adaptation encourages domain adaptation in deep features between the source and target domains by some domain alignment strategies. ASANet [START_REF] Zhou | Affinity space adaptation for semantic segmentation across domains[END_REF] and DAST [START_REF] Yu | Dast: Unsupervised domain adaptation in semantic segmentation based on discriminator attention and self-training[END_REF] proposed affinity adversarial adaptation and reweighting adversarial adaptation, respectively. ECCA [START_REF] Liao | Exploring more concentrated and consistent activation regions for cross-domain semantic segmentation[END_REF] explored more concentrated and consistent activation regions with kernel-based channel attention and mutual information alignment. RCCR [START_REF] Zhou | Domain adaptive semantic segmentation via regional contrastive consistency regularization[END_REF] proposed a regional contrastive regularization framework to ensure local regional consistency. HDL [START_REF] Zhang | A hybrid domain learning framework for unsupervised semantic segmentation[END_REF] proposed a hybrid domain learning framework with hybrid domain feature generation and triple domain alignment. DTST [START_REF] Wang | Differential treatment for stuff and things: A simple unsupervised domain adaptation method for semantic segmentation[END_REF] proposed stuff and instance matching to improve semantic-level alignment. Prototypical adaptation methods are also the spotlights of representation adaptation, such as CAG [START_REF] Zhang | Category anchor-guided unsupervised domain adaptation for semantic segmentation[END_REF] and BAPA-Net [START_REF] Liu | Bapa-net: Boundary adaptation and prototype alignment for cross-domain semantic segmentation[END_REF], which aim to pull close the samples and their related prototypes. For selftraining, BDL [START_REF] Li | Bidirectional learning for domain adaptation of semantic segmentation[END_REF] adopted the max probability threshold to generate pseudo labels. To denoise the pseudo label, ProDA [START_REF] Zhang | Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation[END_REF] proposed prototypical pseudo label denoising. Meanwhile, the self-training strategy was utilized with many UDA methods to furtherly get performance improvements [START_REF] Zhou | Affinity space adaptation for semantic segmentation across domains[END_REF], [START_REF] Yu | Dast: Unsupervised domain adaptation in semantic segmentation based on discriminator attention and self-training[END_REF], [START_REF] Wang | Differential treatment for stuff and things: A simple unsupervised domain adaptation method for semantic segmentation[END_REF]. In spite of that, in the UDA setting, the target domain is assumed to be known and available at the training stage. Such assumptions are indefensible since the target domain is often variable and unavailable in practice.

C. Domain generalization

Domain generalization (DG) seeks a robust model to generalize well on all possible unseen domains. The target domain data does not participate in the training process of the DG model. Data manipulation and representation learning are two main types of approaches to improve generalization. For data manipulation, the source training set is extended as much as possible to cover all unseen domains at the image level by data augmentation or data generation. For instance, DLOW [START_REF] Gong | Dlow: Domain flow for adaptation and generalization[END_REF] generated a continuous sequence of intermediate domains from one domain to another. DRPC [START_REF] Yue | Domain randomization and pyramid consistency: Simulationto-real generalization without accessing target domain data[END_REF] tried to generate images with different styles using auxiliary datasets and then enforced pyramid consistency across data with distinct styles. FSDR [START_REF] Huang | Fsdr: Frequency space domain randomization for domain generalization[END_REF] randomized image styles in the frequency space by spectrum learning. Besides data manipulation-based methods, the models employing representation learning have been widely devoted due to their effectiveness. For representation learning, the DG model is guided to learn domain-invariant knowledge from different domains or multi-views. IBN-Net [START_REF] Pan | Two at once: Enhancing learning and generalization capacities via ibn-net[END_REF] aggregated Instance Normalization and Batch Normalization to capture and eliminate appearance variance. ISW [START_REF] Choi | Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening[END_REF] proposed an instance selective whitening loss to suppress domain-specific features. SAW-SAN [START_REF] Peng | Semantic-aware domain generalized segmentation[END_REF] encouraged classwise semantic consistency by semantic-aware normalization and whitening, which enhanced local feature discrimination. However, the intra-class relationships are ignored in the existing methods. Therefore, it drives us to explore the intra-class relationship to better learn discriminative features.

D. Fine-grained classification

Fine-grained image classification [START_REF] Wei | Fine-grained image analysis with deep learning: A survey[END_REF] targets classified subordinate categories or sub-parts of the semantic categories. The subordinate categories are presented from a biological perspective, e.g., a dog can be a husky or a teddy. The subparts of the semantic classes refer to the different properties of a semantic category, e.g., a car contains some object parts such as the wheel and car light. Compared to the typical category classification, fine-grained classification is more difficult due to similar visuals between the subordinate categories or sub-parts of a semantic class. The key to fine-grained classification is to extract discriminative features. Recently, FGSN [START_REF] Larsson | Fine-grained segmentation networks: Self-supervised segmentation for improved long-term visual localization[END_REF] proposed a fine-grained segmentation network for visual localization in a self-supervised way. HDNN [START_REF] Zhou | Finegrained segmentation using hierarchical dilated neural networks[END_REF] proposed a hierarchical dilated network with better longterm information flow. RefineMask [START_REF] Zhang | Refinemask: Towards high-quality instance segmentation with fine-grained features[END_REF] fused fine-grained features to supplement lost details for high-quality prediction. Chang et al. [START_REF] Chang | Your" flamingo" is my" bird": Fine-grained, or not[END_REF] observed that the fine-grained information helps to improve performances of the coarse classification, which inspired us to propose the fine-grained self-supervision framework. Different from the fine-grained classification task, the fine-grained label in our task is unavailable and there are domain gaps between the training set and the validation set.

E. Self-supervision learning

Self-supervised methods learn meaningful features by various pretext tasks without human annotation to avoid timeconsuming data annotations [START_REF] Jing | Self-supervised visual feature learning with deep neural networks: A survey[END_REF]. Self-supervised learning was designed initially as the pre-training process for downstream tasks such as image classification and semantic segmentation. In recent research, self-supervised methods are also aggregated in some downstream tasks to improve the performances of downstream tasks. SEAM [START_REF] Wang | Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation[END_REF] proposed an equivariant attention mechanism for weakly supervised semantic segmentation to solve the significant inconsistency of different scale images. SGDepth [START_REF] Klingner | Selfsupervised monocular depth estimation: Solving the dynamic object problem by semantic guidance[END_REF] presented a self-supervised semantically-guided depth estimation to constrain the relationship between the camera pose and geometric projection. SAC [START_REF] Araslanov | Self-supervised augmentation consistency for adapting semantic segmentation[END_REF] proposed a self-supervised augmentation consistency for cross-domain semantic segmentation, which generated pseudo labels by a momentum network allowing stable targets for self-supervised training.

III. FINE-GRAINED SELF-SUPERVISION FRAMEWORK

A. Problem statement

Given a seen source dataset {X s , Y s } ∈ D s and K unseen target domains

{X k t , Y k t } ∈ D k t , k ∈ K,

B. Framework overview

As demonstrated in Fig. 2, the fine-grained self-supervision framework has single-view and multi-view versions. Both versions contain an auxiliary task (i.e., fine-grained segmentation) to improve the discriminability of the model. In the multi-view version, the class prototype feature enhancement generates another view (i.e., representation with different styles but the same content as the original representation). The objective of these two versions can be written as:

L = L s + L f s |L mf s (1)
where L s , L f s , and L mf s are the coarse segmentation loss, single-view fine-grained self-supervision loss, and multi-view mutual supervision loss, respectively. | is the OR operation.

C. Coarse-grained semantic segmentation

Coarse-grained semantic segmentation has fewer semantic categories than fine-grained semantic segmentation, which is our final goal in this paper. Given a batch of data containing images and labels {x s ∈ X s , y s ∈ Y s }, the feature f is first extracted by the feature extractor F (•) and then is classified by the coarse classifier C. The cross-entropy is used as coarse segmentation loss L s , which can be defined as:

L s = - h,w n∈N y (h,w,n) s log(C(f (h,w) )) (2)
where n ∈ N is the coarse semantic class. y

(h,w,n) s
is a one-hot annotation of a pixel at (h, w) position.

D. Fine-grained semantic segmentation

Fine-grained segmentation refers to recognizing the subparts or subordinate categories of the semantic categories. The subordinate categories are not applicated in our case. On the contrary, the sub-parts of semantic categories are used reasonably since the object can be divided into different meaningful sub-parts in an image [START_REF] Zhou | Rethinking semantic segmentation: A prototype view[END_REF]. For example, the person consists of the head, the hand, and the leg visualized in Fig. 8. (c). As pointed out in [START_REF] Chang | Your" flamingo" is my" bird": Fine-grained, or not[END_REF], the fine-grained information helps to improve the discriminability of the coarse segmentation. Consequently, fine-grained semantic segmentation is treated as an auxiliary task to assist the coarse segmentation with a standard cross-entropy L f s , which can be defined as:

L f s = - h,w n f ∈N F y (h,w,n f ) sf log(C f (f (h,w) )) (3)
where n f ∈ N F is the fine-grained class and C f is the finegrained classifier. However, fine-grained annotations is not provided in datasets. To tackle this issue, the fine-grained selfsupervision strategy (FSS) is proposed to supervise the feature by fine-grained pseudo-label. Thus, the Equation (3) can be rewritten as:

L f s = - h,w n f ∈N F q sf (y sf |f ) (h,w,n f ) log(C f (f (h,w) )) (4)
where fine-grained labels y sf are encoded as posterior distributions q sf (y sf |f ) by online pseudo-label assignment described in the following paragraph.

E. Online pseudo label assignment

The annotation assignment is an instance of the optimal transport (OT) problem due to the lack of fine-grained annotation. First, the fine-grained predicton P ∈ R H×W ×N f is extracted by the fine-grained classifier C f to represent the cost of assigning features f (h,w) to n f class, which can be defined as:

P = C f (f ) = ϕ(Conv(δ(BN (Conv(f ))))) (5
Q 1 Q 2 Q 3 Q 4 Clustering
Fig. 3. The structure of online pseudo label assignment, which can be divided into three steps. First, a fine-grained classifier is defined to generate fine-grained prediction P and the fine-grained prediction P n of the coarse class n is obtained. Second, the Sinkhorn-Knopp algorithm is adopted as the clustering solution on P n to get the label assignment matrix Q n of coarse class n. Third, all {Q n , n ∈ N } ∈ Q matrixes are converted to the final fine-grained label q sf for self-supervision.

normalization, respectively. To distinguish different sub-parts of each coarse category, the fine-grained prediction belonging to n category P n is used to perform OT, which is obtained by the element-wise multiplication between the fine-grained prediction and the coarse-grained label. It is denoted as:

P n = P (h,w) 1(y (h,w,n) s == 1) (6)
where 1 refers to the indicator function. Then, this OT problem can be fastly solved by the Sinkhorn-Knopp algorithm [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF], [START_REF] Caron | Unsupervised learning of visual features by contrasting cluster assignments[END_REF] to guarantee online label assignment at the training stage, where the energy function E n sk with entropic constraints can be denoted as:

E n sk = Q n , -logP n + 1 σ KL(Q n ||Rrc T ) s.t. Q n 1 K n = 1 N c 1 Nc , (Q n ) 1 Nc = 1 K n 1 K n (7)
where

Q n ∈ R Nc×K n
is the label assignment matrix. r and c are the marginal projections respectively onto its clusters and feature indices. N c is the number of clusters for each coarse class, i.e., the number of sub-parts for a coarse class. K n is the pixel number of n class. • is the Frobenius dot-product and KL is the Kullback-Leibler divergence. R is a permutation matrix matching clusters to marginals, which represents an arbitrary prior distributions, since the occupied frequencies of the object sub-parts in an image are different [START_REF] Asano | Labelling unlabelled videos from scratch with multi-modal self-supervision[END_REF]. σ is a constant. With the entropic regularization term, the solver in Equation ( 7) can be rewritten as:

Q n = Diag(u)(P n ) σ Diag(v) (8)
where u and v are two renormalization vectors for scaling. After a few iterations, the renormalization vectors are updated to obtain the final assignment matrix Q n . Finally, the finegrained label q sf in Equation ( 4) is converted by these label assignment matrices, which can be denoted as:

q sf = n∈N n=0 Reshape(Q n + N c × n) (9)
where Reshape operation is to match the resolution of the deep feature. Note that the label assignment process does not involve back-propagation. To understand this process clearly, the structure of the online pseudo-label assignment is presented in Fig. 3.

F. Class prototype feature enhancement

The above self-supervised strategy is performed in a single view. Inspired by the DRPC [START_REF] Yue | Domain randomization and pyramid consistency: Simulationto-real generalization without accessing target domain data[END_REF] that generated and aligned multi-view data (i.e., representations with different styles but the same content) for better learning domain-invariant features, the multi-view version of the FGSS framework is extended. Generally, another view is often generated by the photometric transform at the image level as SAC [START_REF] Araslanov | Self-supervised augmentation consistency for adapting semantic segmentation[END_REF] does. However, such methods occupy double memory due to feature extraction twice compared to the single view. To reduce memory cost, we propose the class prototype feature enhancement (CPFE) to generate a new view, which focuses on performing perturbation on the original feature rather than on the image level.

1) Class prototypes generation: Class prototypes should be calculated first before conducting feature enhancement. Class prototypes refer to the class-wise feature centroids and are generated online [START_REF] Zhang | Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation[END_REF]. Given a mini-batch source image x s ∈ X s and a CNN-Based feature extractor F (•), we can obtain the deep feature of an image F (x s ). After that, the prototype of n class p n can be initialized as:

p n = xs∈Xs h w F (x s ) (h,w) 1(y (h,w,n) s == 1) xs∈Xs h w 1(y (h,w,n) s == 1) (10)
To obtain more powerful and generalized class prototypes, the class prototypes are updated using the moving average in two mini-batches, which are defined as,

p n ← λp n + (1 -λ)p n ( 11
)
where p n is the mean feature of class n calculated by Equation ( 10) at the current training state and λ is the momentum term that equals 0.9. 2) Feature enhancement: As pointed out in [START_REF] Chang | All about structure: Adapting structural information across domains for boosting semantic segmentation[END_REF], [START_REF] Song | Achieving domain generalization in underwater object detection by image stylization and domain mixup[END_REF], the latent feature contains style and content information. Since the deep feature of the n class f n and the prototype of n class p n share the same category, their content information is the same. The discrepancy between them is the style information. Thus, the interpolation between f n and p n stands for a new representation (i.e., a new view) with different styles but the same category compared to f n , which can be denoted as:

ACCEPTED MANUSCRIPT / CLEAN COPY

f n aug = λ a F (x s ) (h,w) 1(y (h,w,n) s == 1) + (1 -λ a )p n (12)
where f n aug is the augmented feature of n class and λ a is the random value in the range [0-1]. Finally, the augmented feature is put into the coarse classifier C and supervised by the cross-entropy loss as with Equation (2). The structure of the CPFE strategy is shown in Fig. 4.

The reasons for adopting the interpolation to generate another view are as follow. The deep neural network excels at linearizing the feature and decoupling different potential variables linearly [START_REF] Upchurch | Deep feature interpolation for image content changes[END_REF]- [START_REF] Li | Transferable semantic augmentation for domain adaptation[END_REF]. Meanwhile, the style and semantic information can be linearly separable in deep features [START_REF] Zhang | A hybrid domain learning framework for unsupervised semantic segmentation[END_REF]. The augmented feature can be regarded as the intermediate feature between the feature and its corresponding prototype. The original feature is augmented toward the class prototype rather than toward a non-meaningful direction. The effect of the CPFE strategy is twofold. For one thing, similar to DLOW [START_REF] Gong | Dlow: Domain flow for adaptation and generalization[END_REF], this strategy generates the intermediate feature with various styles that plays a data augmentation role. For another, the augmented feature is another view of the same image, which is expected to provide the same content information as the original feature. The deep features of two views are utilized to perform the multi-view mutual supervision introduced in the following paragraph.

G. Multi-view mutual supervision

Fine-grained self-supervision illustrated in Section III. B can be directly used in multi-view, i.e., the label assignment Extracts feature to get f . Update prototype by Eq 10 and Eq 11. Initilize class prototype by Eq 10. Perform CPFE by Eq 12 to get f aug . 10:

Perform coarse supervision using f aug by Eq 2.

11:

Perform fine-grained prediction using f and f aug by Eq 5.

12: n = 0.

13:

for n ≤ N -1 do 14:

Obtain fine-grained predictions P n and P n aug by Eq 6.

15:

Obtain label assignment maps Q n and Q n aug by Eq 8.

16:

n + +.

17:

Obtain fine-grained labels q sf and q aug sf by Eq 9.

18:

Perform MVMS by Eq 14.

19:

else 20:
Perform coarse supervision by Eq 2.

21:

Perform fine-grained prediction by Eq 5.

22:

n = 0.

23:

for n ≤ N -1 do

24:

Obtain P n by Eq 6.

25:

Obtain label assignment map Q n by Eq 8.

26:

n + +.

27:

Obtain the fine-grained label by Eq 9.

28:

Perform fine-grained supervision by Eq 4.

29:

Update θ f , θ c , θ f c .

30:

iter + +. 31: Return: θ f , θ c , θ f c .

program is performed in two views and then the features of two views are supervised using the fine-grained label generated by themselves respectively. Such naive multi-view self-supervision loss L n mf s can be denoted as:

L n mf s = L f s (q aug sf , f aug ) + L f s (q sf , f ) (13)
where q sf and q aug sf are the pseudo label generated from the original feature f and augmented feature f aug , respectively. However, the label assignment maps of two views may be inconsistent, which may affect performance. Related experiments are provided in Section IV. D. To handle this issue, a multi-view mutual supervision (MVMS) loss is proposed. The original feature and augmented feature are two views to finish this process. The fine-grained label assignment program is conducted twice to get the label assignment maps of two views, respectively. After that, the fine-grained label of one ACCEPTED MANUSCRIPT / CLEAN COPY view is used to supervise another view. The multi-view mutual supervision loss L mf s can be defined as:

L mf s = L f s (q sf , f aug ) + L f s (q aug sf , f ) (14)
The MVMS loss implicitly performs alignment between the original and the augmented views to learn more generalized representation. More details of both versions of the FGSS framework are provided in Algorithm 1.

IV. EXPERIMENT

In this section, we evaluate the effectiveness of our FGSS framework. First of all, the experimental datasets and implemented detail are introduced. Then, the state-of-the-art methods of generalized semantic segmentation are compared to our method. Meanwhile, ablation studies are conducted to verify the influence of proposed components. Finally, segmentation visualization and cluster visualization are provided.

A. Dataset

With the goal of single-domain generalization, the performances of unseen multi-domains need to be evaluated. Five datasets used in the experiments consist of synthetic datasets (GTAV [START_REF] Richter | Playing for data: Ground truth from computer games[END_REF] and SYNTHIA [START_REF] Ros | The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes[END_REF]) and real-world datasets (Cityscapes [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF], BDD100K [START_REF] Yu | Bdd100k: A diverse driving dataset for heterogeneous multitask learning[END_REF] and Mapillary [START_REF] Neuhold | The mapillary vistas dataset for semantic understanding of street scenes[END_REF]).

Synthetic datasets: Several synthetic datasets are created to avoid the amount of human effort in labeling. The GTAV dataset collected 25k images from the computer game Grand Theft Auto V with a high resolution of 1914 × 1052. The SYNTHIA dataset contains 9,400 1280 × 760 images. 19 and 16 common categories with Cityscapes are used in the GTAV and SYNTHIA datasets, respectively.

Real-world datasets: The Cityscapes dataset is a real-world street scene dataset widely used in the semantic segmentation task, which contains 2975 training and 500 validation images with the resolution of 2048 × 1024. The BDD100K driving scene dataset contains 7000 training and 1000 validation images with HD resolution (i.e., 1280 × 720). The Mapillary dataset contains 25k with at least FHD resolution (i.e., 1820 × 1080), which is captured in various environments such as different weathers and seasons. Both BDD100K and Mapillary datasets adopt 19 overlapped classes with the Cityscapes.

B. Implementation Details

The FGSS framework is implemented by the Pytorch library. Three backbones including ResNet-50 [START_REF] He | Deep residual learning for image recognition[END_REF], ShuffleNet-v2 [START_REF] Ma | Shufflenet v2: Practical guidelines for efficient cnn architecture design[END_REF], MobileNet-v2 [START_REF] Sandler | Mobilenetv2: Inverted residuals and linear bottlenecks[END_REF] with DeepLabv3+ architecture [START_REF] Chen | Encoderdecoder with atrous separable convolution for semantic image segmentation[END_REF] are evaluated. The Intersection-over-Union (IoU) of each class and their mean IoU are used as the metric for segmentation accuracy. The model optimization utilizes an SGD optimizer with an initial learning rate of 1e-2, weight decay of 5e-4, and momentum of 0.9. The images are resized to 768 × 768 resolution and the model is trained in a batch size of 4 with 40k iterations. The cluster number N c is set to 3 and the related hyperparameter experiment is conducted in Section IV. E. The fine-grained classifier is implemented as a sequence containing 3 × 3 convolution, batch normalization, ReLU, and 1 × 1 convolution.

ACCEPTED MANUSCRIPT / CLEAN COPY

C. Results

We compare our approach to the other state-of-the-art methods on three backbone networks (i. I and the reported results adopt the model of the multi-view version by default. The performance of our method substantially outperforms the second-best methods [START_REF] Kim | Pin the memory: Learning to generalize semantic segmentation[END_REF] by 1.9% in terms of average mIoU among the above five generalization settings. In the 20 evaluation settings, our FGSS framework achieves 13 best and 3 second-best performances. Compared with ISW [START_REF] Choi | Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening[END_REF], our method achieve 17 better performances in 20 generalization evaluation settings. These results show that our FGSS framework learns a more ACCEPTED MANUSCRIPT / CLEAN COPY Some qualitative examples of different settings (G→C, G→B, G→S, and G→M) are provided in Fig. 5. Some segmentation errors in the Baseline are eliminated in two versions of the FGSS framework. Meanwhile, the multi-view version performs better than the single-view version.

D. Ablation studies

Next, the ablation studies are conducted to investigate the influence of the proposed components including finegrained self-supervision strategy (FSS), class prototypes feature enhancement (CPFE), and multi-view mutual supervision (MVMS). The single-view version of the FGSS framework (SVFGS) represents FSS. The multi-view version of the FGSS framework (MVFGS) consists of CPFE and MVMS. As shown in Table IV, SVFGS has a clear improvement with a 3.1% in terms of average mIoU compared with the Baseline [START_REF] Choi | Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening[END_REF]. Since CPFE generates a new view with various styles and the same semantics as the original view to enlarge available domains, the model improves 1.4% in average mIoU compared with the Baseline. The performance of CPFE + FSS (i.e., naive multiview self-supervision loss in Equation ( 13)) is slightly lower than the SVFGS. It is consistent with the opinion illustrated in Section III. E that inconsistent label assignments may affect performance when the FSS strategy is directly used in two In addition, the model complexity analysis is also provided. The training time and occupied memory are adopted to analyze the model complexity. As shown in Table V, the multi-view version of the FGSS framework cost 892m memory more and 0.06s more in the training time than the Baseline while getting a 4.1% gain in terms of average mIoU. Note that our framework and the Baseline share the same testing time since the fine-grained branch of our FGSS framework is not used in the evaluation stage. Our method gets significant improvement at low consumption.

E. Cluster number validation

Another concern is the cluster number of each coarse class N c . Fig. 6 shows the effect of the cluster number. Note that N c = 1 represents the performance of the Baseline. As shown in Fig. 6, the model achieves the best generalizability when N c equals 3, where the G→B and G→M achieve the best performances compared with other cluster number settings. When N c is greater than 3, the average mIoU has a decreasing tendency and the performance of the model generalizing on different domains is not stable enough. For example, the performance of G→B is lower than the Baseline when N c = 5. Moreover, the average mIoU of our framework in different cluster number settings have a performance improvement compared with the Baseline, which shows the effectiveness of our method.

ACCEPTED MANUSCRIPT / CLEAN COPY

F. Compared with other feature augmentation methods

To verify the advantage of the proposed CPFE, the performance comparison of the feature augmentation methods is shown in Fig. VI. The proposed CPFE achieved the best performance. The augmentation methods designed for the classification task [START_REF] Zhou | Domain generalization with mixstyle[END_REF], [START_REF] Zhang | Exact feature distribution matching for arbitrary style transfer and domain generalization[END_REF] get unsatisfactory performance since semantic segmentation is a pixel-level task simultaneously including several classes in an image instead of including only a class. Directly performing the feature augmentation using the features in the whole spatial location may lead to the class confusion problem. On the contrary, the proposed CPFE generates augmented features at the class level to solve the class confusion problem. The results show the effectiveness of our augmented method.

G. Discrimination validation

As illustrated in Section III, our proposed FGSS framework captures more class discriminative information by distinguishing sub-parts of the semantic categories. An effective manner to verify this viewpoint is to observe the confusion matrix. The confusion matrices generalizing from G to C, B, and M using ShuffleNet-V2 are presented in Fig. 7. The high diagonal value represents the high accuracy of our class predictions. Therefore, the diagonal value is employed as a metric to evaluate the discrimination. Compared with the Baseline, our method achieves 14, 12, and 13 better performances at 19 classes in G→C, B, and M settings. Our proposed FGSS framework performs better in varying categories, showing that our fine-grained self-supervision can enhance the feature discrimination.

H. Cluster visualization

We also present some clustering results with different classes in Fig. 8. The clustering results with different categories are consistent with human cognition from the visual perspective. For example, persons can be divided into head, body, and leg as shown in Fig. 8. (c). Another case is that cars in Fig. 8. (e) consist of windows, bodies, and chassis. The subparts of semantic classes are extracted by label assignment and fine-grained self-supervision is performed to enhance feature discrimination. The visualization of clustering results shows that the FGSS framework can capture the intra-class relationship and ensure discriminability.

I. Extreme case analysis

Given a feature F ∈ R H×W ×C and related annotation y ∈ R H×W ×N , the cluster number N n c of the coarse class n equals the pixel number N n p of coarse class n in the extreme case, where N n N n p = H × W . At this time, the clustering operation lost its meaning and cannot be conducted because the feature of each spatial position can be treated as a finegrained label index. Meanwhile, the fine-grained classifier contains large parameters and greatly reduces the efficiency of the algorithm, which is unreasonable and unfeasible. Concretely, the fine-grained classifier is a mapping function from the original feature F ∈ R H× W ×C to the fine-grained prediction P ∈ R H× W ×(H×W ) , which cannot be implemented due to resource limitations. But we still can try to analyze this situation.

For the single-view version of our proposed method, as we cannot capture the intra-class relationship using clustering, the auxiliary loss cannot work and even hurt performance.

For the multi-view version of our proposed method, if we remove the fine-grained classifier (i.e., the feature of a view directly guides the feature of another view), our proposed method degenerates into the feature distribution alignment of different views. As shown in Table VII, the multi-view version of our proposed method in the extreme case still achieves competitive performance with a clear improvement compared to the Baseline.

V. CONCLUSION

In this paper, we designed a fine-grained self-supervision (FGSS) framework for generalizable semantic segmentation, ACCEPTED MANUSCRIPT / CLEAN COPY which considered both the generalizability and discriminability of the model from the intra-class relationship. The FGSS framework consists of single-view and multi-view versions. In the single-view version, we proposed a fine-grained selfsupervision strategy which treated the fine-grained segmentation as an auxiliary task to improve feature discrimination. In the multi-view version, we furtherly proposed class prototype feature enhancement to generate another view from the feature space, where the augmented feature is close to the related prototype forcing the classifier to capture more domain-invariant information. Moreover, we proposed multiview mutual supervision loss in the fine-grained segmentation task for the label inconsistency problem in two views to implicitly reduce the gap between different views. Extensive experimental results showed that our method achieved superior performance compared to other state-of-the-art methods.

Fig. 1 .

 1 Fig. 1. The illustration of fine-grained segmentation. The pictures from top to bottom are RGB, the car label (coarse-grained label), and the fine-grained label of the car. Fine-grained semantic segmentation segments sub-parts of the coarse semantic category.

Fig. 2 .

 2 Fig. 2. The pipeline of the proposed FGSS framework containing single-view and multi-view versions. CPFE is class prototype feature enhancement and OLA refers to online pseudo label assignment. Fine-grained segmentation is an auxiliary task to improve the discriminability of the model. Cubes are the deep feature.

  ) where ϕ and δ are the Softmax operation and ReLU operation, respectively. Conv and BN are the convolution and batch ACCEPTED MANUSCRIPT / CLEAN COPY

Fig. 4 .

 4 Fig. 4. The structure of the class prototype feature enhancement. The feature combination fills the prototype to a corresponding position according to the ground truth.

Algorithm 1

 1 Fine-grained self-supervision framework Initialize: θ f , θ c , θ f c , max iter, N c , M ultiV iter = 0, Init P roto = 0 Input: Samples { (X s , Y s )} 1: for iter ≤ max iter do 2:

3 :

 3 if Init P roto==1 then

Fig. 5 .

 5 Fig. 5. Qualitative examples. The single-view (SVFGS) and multi-view (MVFGS) versions of the FGSS framework have more right areas in visual.

  e., ResNet-50, MobileNet-V2, ShuffleNet-V2). The GTAV, BDD100K, SYNTHIA, Mapillary, and Cityscapes datasets are denoted as G, B, S, M, and C, respectively. In the ResNet-50, these five datasets are taken turns as the source domain for training and the datasets except the source domain are used for evaluation, which is grouped into G → {C, B, M, S}, S → {G, B, M, C}, C → {G, B, S, M}, B → {G, C, S, M}, and M → {G, B, C, S}. The right arrow → refers to "generalizing to". The performance comparison is shown in Table

Fig. 6 .

 6 Fig.6. The effect of the cluster number for each coarse class (i.e., the number of sub-parts for a coarse class).

Fig. 7 .Fig. 8 .

 78 Fig. 7. Confusion matrices comparison on the task of GTAV generalizing to {C, B, M} using ShuffleNet-V2 backbone.

TABLE I PERFORMANCE

 I COMPARISON ON RESNET-50 BACKBONE. GTAV (G), BDD (B), SYNTHIA (S), CITYSCAPES (C), AND MAPILLARY (M) ARE ADOPTED AS THE SOURCE DATA IN TURN AND EVALUATED AT OTHER DATASETS. RIGHT ARROW → REPRESENTS GENERALIZING TO ANOTHER DATASET. BOLD REPRESENTS THE BEST PERFORMANCE AND UNDERLINE REPRESENTS SECOND-BEST PERFORMANCE. AVG REFERS TO THE AVERAGE MIOU OF ALL

									EVALUATION SETTINGS.
	Methods	Model	Avg	Trained on GTAV (G) →C →B →M →S	Trained on SYNTHIA (S) →C →B →M →G	Trained on Cityscapes (C) →B →M →G →S	Trained on BDD (B) →G →S →C →M	Trained on Mapillary (M) →G →S →C →B
	IBN [12]	ResNet-50 34.2	33.9 32.3	37.8	27.9	32.0 30.6	32.2	26.9	48.6	57.0	45.1	26.1	29.0	25.4 41.1	26.6	30.7	27.0 42.8 31.0
	SW [63]	ResNet-50 32.2	29.9 27.5	29.7	27.6	28.2 27.1	26.3	26.5	48.5	55.8	44.9	26.1	27.7	25.4 40.9	25.8	28.5	27.4 40.7 30.5
	DRPC [11]	ResNet-50 35.8	37.4 32.1	34.1	28.1	35.7 31.5	32.7	28.8	49.9	56.3	45.6	26.6	33.2	29.8 41.3	31.9	33.0	29.6 46.2 32.9
	GTR [64]	ResNet-50 36.1	37.5 33.8	34.5	28.2	36.8 32.0	32.9	28.0	50.8	57.2	45.8	26.5	33.3	30.6 42.6	30.7	32.9	30.3 45.8 32.6
	ISW [13]	ResNet-50 36.4	36.6 35.2	40.3	28.3	35.8 31.6	30.8	27.7	50.7	58.6	45.0	26.2	32.7	30.5 43.5	31.6	33.4	30.2 46.4 32.6
	SAN-SAW [15] ResNet-50 38.5	39.8	37.3	41.9	30.8	38.9	35.2	34.5	29.2	53.0	59.8	47.3	28.3	34.8	31.8 44.9	33.2	34.0	31.6 48.7 34.6
	PinMem [65]	ResNet-50 41.0	41.2 35.2	39.4	28.9	38.2	32.3	33.9	32.1	50.6	57.9	45.1	29.4	42.4	29.1	54.8	51.0	44.1	30.8	55.9 47.6
	Ours	ResNet-50	42.9	44.5 36.5	44.3	31.4	39.2	29.8	33.5	34.5	52.0	58.1	45.1	30.0	47.0	30.8	56.2	55.5	46.7	35.5 57.4 49.8

TABLE II PERFORMANCE

 II COMPARISON ON THE TASK G → {C, B, M} ON SHUFFLENET-V2 AND MOBILENET-V2 BACKBONE NETWORKS. AVG REFERS TO THE AVERAGE MIOU OF ALL EVALUATION SETTINGS.

	Model	Method	→C	→B	→M	Avg
		Baseline	25.6 22.2	28.6	25.4
		IBN-Net [12]	27.1 31.8	34.9	31.3
	ShuffleNet-V2	ISW [13]	31.0 32.1	35.3	32.8
		DIRL [72]	31.9 32.6	36.1	33.5
		Ours	35.3 33.1	37.6	35.3
		Baseline	25.9 25.7	26.5	26.0
		IBN-Net [12]	30.1 27.7	27.1	28.3
	MobileNet-V2	ISW [13]	30.9 30.1	30.7	30.5
		DIRL [72]	34.7 32.8	34.3	33.9
		Ours	35.8 33.2	36.0	35.0
		TABLE III
	PERFORMANCE COMPARISON ON THE TASK C → {G, B, S} ON
	SHUFFLENET-V2 AND MOBILENET-V2 BACKBONE NETWORKS. AVG
	REFERS TO THE AVERAGE MIOU OF ALL EVALUATION SETTINGS.
	Model	Method	→B	→S	→G	Avg
		Baseline	38.1	21.3	36.5	31.9
		IBN-Net [12]	41.9	23.0	40.9	35.3
	ShuffleNet-V2	ISW [13]	41.9	22.8	40.2	35.0
		DIRL [72]	42.6	23.7	41.2	35.8
		Ours	44.1	27.5	40.3	37.3
		Baseline	40.1	21.6	37.3	33.0
		IBN-Net [12]	45.0	23.2	41.1	36.4
	MobileNet-V2	ISW [13]	45.2	22.9	41.2	36.4
		DIRL [72]	47.6	23.3	41.4	37.4
		Ours	46.6	27.6	42.3	38.8

TABLE IV ABLATION

 IV STUDIES ON EACH COMPONENT CONTAINING CPFE, FSS, MVMS. THE GTAV DATASET IS USED AS THE SOURCE DOMAIN WITH THE RESNET-50 BACKBONE. AVG REFERS TO THE AVERAGE MIOU OF ALL EVALUATION SETTINGS.

	Method	CPFE FSS MVMS →C →B →M →S Avg
	Baseline	36.6 35.2 40.3 28.3 35.1
	FSS	41.3 38.9 39.8 32.9 38.2
	CPFE	38.8 36.0 40.0 31.2 36.5
	CPFE + FSS	40.3 36.9 39.9 32.3 37.4
	CPFE + MVMS	44.5 36.5 44.3 31.4 39.2

TABLE V COMPLEXITY

 V ANALYSIS ON EACH COMPONENT CONTAINING CPFE, FSS, MVMS. MEMORY REFERS TO THE OCCUPIED MEMORY IN THE TRAINING STAGE, AND TIME IS THE ITERATION TIME FOR ONE IMAGE. → {C, B, M} and C → {G, B, S} on MobileNet-V2 and ShuffleNet-V2 backbone networks. Related results are shown in Table II and Table III. In the task of G → {C, B, M}, both backbone networks achieve the best performance in 3 evaluation settings and outperform the second-best by at least 1.1% in terms of average mIoU.

	Method	Memory (m) Time (s)
	Baseline	8515	0.30
	FSS	8877	0.32
	CPFE	8985	0.31
	CPFE + FSS	9217	0.39
	CPFE + MVMS	9407	0.36
	generalized semantic segmentation model. Furthermore, we
	conduct G In the task of C →
	{B, S, G}, compared with DIRL [72], our approach achieves
	37.3% and 38.8% average mIoU with a gain 1.5% and 1.4%
	in the ShuffleNet-V2 and MobileNet-V2 backbone networks,
	respectively.

TABLE VII PERFORMANCE

 VII ON THE EXTREME CASE (EC). Method →C →B →M →S Avg Baseline 36.6 35.2 40.3 28.3 35.1 EC 40.6 35.5 41.4 31.1 37.1 SVFGS (Ours) 41.3 38.9 39.8 32.9 38.2 MVFGS (Ours) 44.5 36.5 44.3 31.4 39.2

ACCEPTED MANUSCRIPT / CLEAN COPY

ACKNOWLEDGMENTS

This work was supported in part by the National Natural Science Foundation of China under grants 62171294, 62101344, in part by the key Project of DEGP (Department of Education of Guangdong Province) under grants 2018KCXTD027, in part by the Natural Science Foundation of Guangdong Province, China under grants 2022A1515010159, 2020A1515010959, in part by the Key project of Shenzhen Science and Technology Plan under Grant 20220810180617001, in part by the Interdisciplinary Innovation Team of Shenzhen University and in part by the Tencent "Rhinoceros Birds" -Scientific Research Foundation for Young Teachers of Shenzhen University, China.


Aaron Boone

Joaquim Bellvert

Martin Best

Jennifer Brooke

Guylaine Canut-Rocafort

Joan Cuxart

Oscar Hartogensis

Josep Ramon Miró

Jan Polcher

Jeremy Price

Lleida

Jenni- Fer Brooke

Pere Quintana-Segui

Meeting/Workshop Reports of the 1st LIAISE Conference and Determining Evapotranspiration (dET) CrossCut

published or not. The documents may come

Workshop Objectives

The overall objective of the Land surface Interactions with the Atmosphere over the Iberian Semi-arid Environment (LIAISE) project is to improve our understanding of the impact of anthropization on the water cycle in terms of land-atmospherehydrology interactions, and the limitations of models to represent all aspects of the terrestrial water cycle in a semi-arid environment characterized by strong surface heterogeneity owing to contrasts between the natural landscape and intensive agriculture (https://www.hymex.fr/liaise/index.html). The LIAISE study area is located over the Catalan counties of Urgell and Pla d'Urgell within the Ebro basin in northeastern Spain, which contains a sharp delineation between a vast, nearly continuous intensively-irrigated region and the dryer natural zone to the east. The main water source consists of artificial reservoirs located to the north in the Pyrenees mountains, and this water is brought into the irrigated area using a highly anthropized river and canal network. The ability to make robust estimates of water use efficiency (WUE) is critical for managing the limited water resources in this region. Evapotranspiration (ET) is the process linking the water, energy, and carbon cycles, and modulating the WUE. ET is a challenge to observe and model across a range of spatial scales using a myriad of methods (Cuxart and Boone, 2020): it was studied extensively during the LI-AISE field campaign in 2021 from very different perspectives, ranging from a focus on process understanding and prediction by Earth system modelers, to quantification at a range of spatio-temporal scales for hydrological or agricultural applications, such as reservoir dam regulation or crop irrigation.

The LIAISE field campaign was the first of the GEWEX determining EvapoTranspiration (dET) cross-cut initiative, which focuses on measuring ET, especially in terms of the surface energy balance, ET-atmospheric boundary layer (ABL) interactions, modeling and parameterizations of ET (notably within numerical models), partitioning of ET into its various components (bare soil, transpiration, etc.), and the quantification and study of potentially-significant spatial heterogeneity of ET. The goal of the joint LIAISE-dET meeting was to bring together researchers specializing in estimating ET experimentally using in-situ observations, via remote sensing data and/or models, from the leaf to the regional scale.

Conference Summary

The workshop was held near the end of March at the Parc Científic i Tecnològic Agroalimentari de Lleida, in Lleida, Spain (https://www.gewexevents.org/meetings/liaise-and-det/). There were 41 oral and 15 poster presentations, and over 60 participants attended the 2.5 day event. It brought together experts from diverse disciplines who study ET, consisting mainly of agronomers, hydrologists, meteorologists, soil scientists, remote sensing specialists, and plant physiologists, with a majority presenting results from LIAISE field campaign data analysis and numerical modeling experiments (see [START_REF] Boone | Updates on the international Land Surface Interactions with the Atmosphere over the Iberian Semi-Arid Environment (LIAISE) Field Campaign[END_REF] for a review of the field campaign). It was the first opportunity for many of the researchers present to finally meet face-to-face in the same location, since contact between groups was limited during the campaign due to the COVID-19 sanitary crisis.

Sessions were organized around three topics: (i) land surface ABL conditions during the LIAISE campaign, with a focus on the contrasting conditions over the irrigated and rain-fed areas; (ii) ET derived from observations using both in situ measurements and remote sensing techniques over a range of spatial scales (from plot to region); and (iii) integrated land-atmosphere-hydrology interaction studies. Note that the ET studies were not limited to the LIAISE study area. All of the presentations and posters are available on the LIAISE project page (https://www.hymex.fr/liaise/ LIAISE_conf/Workshop_March_2023/presentations/).

After the end of the conference at mid-day on the 29th, the afternoon was devoted to LIAISE working group (WG) meetings. These groups form the core of the LIAISE project, and tie in with the dET initiative. There are three WGs, and their name and focus are defined as follows: Surface processes (WG1) focuses on water, energy, and carbon balances and related processes in the interaction between the surface and the atmosphere (with ET as a key process that integrates all three balances) from the leaf to the field scale. Meteorology (WG2) deals with ABL processes and mesoscale circulations related to wetdry surface contrasts, topography, and sea breeze interactions. Fire, as a fundamental feature of semi-arid landscapes with strong surface-ABL interactions, is also included. Regional scale hydrology (WG3) explores the linkages between irrigation, streamflow and reservoirs (and their regulation), and regionalscale estimates of surface properties related to soil moisture.

Each WG meeting took place in a parallel session, with the final plenary session devoted to short presentations by each of Quarter 3 2023

Attendees of the 1 st LIAISE Conference and Determining Evapotranspiration (dET) CrossCut Workshop

the group leads and discussions. The main outcomes are as follows: WG1 discussed the first results of the ET-methods (eddycovariance, gradient, lysimeter, scintillometer, remote sensing algorithms) intercomparison effort, how to best organize the various LIAISE data products for general use, the new LIAISE Unified Eddy Covariance flux product, the ecophysiological measurements (leaf area index, vegetation cover fraction, photosynthesis traits), soil moisture, and the development of consistent locally-informed land use and irrigation maps (such maps are generally superior in quality to the default global data sets often used by land surface modelers). WG2 reviewed the progress of the LIAISE mesoscale model intercomparison project (the current pilot study and a future follow-up, which will focus on an Intensive Observation Period from the LIAISE campaign); the atmospheric observational data, which has been processed and is available (e.g., ultra high frequency wind profiler, radio soundings), and the current status of data processing (e.g., turbulence from aircraft and tethered balloon); mesoscale circulations within the LIAISE region (wet-dry surface induced, the sea breeze, nocturnal jets and mountain-induced circulations); plans for large eddy simulations (LES) over the LIAISE domain by several groups; a specific study on two rain events during the special observation period (the focus thus far has been on clear days); and synergies with other projects exploring the impact of land surface heterogeneities on the ABL and subsequent mesoscale circulations [e.g., Model and Observation for Surface-Atmosphere Interactions, MOSAI, https://en.aeris-data.fr/projects/ mosai-2/, or the GEWEX-sponsored Coupling Land and Atmosphere Sub-grid Parameterizations (CLASP) project [START_REF] Chaney | Coupling Land and Atmosphere Sub-grid Parameterizations (CLASP)[END_REF]]. WG3 focused on whether current land surface and hydrological models can simulate a realistic water cycle at km-scale resolutions in mountainous areas. This is especially important in a semi-arid region like the Ebro basin, which has limited water resources and where mountain reservoirs provide drinking water to the local population, generate a substantial amount of electricity, and feed the canals that provide irrigation water for intensive agricultural activities. The main scientific objectives for a regional-scale model intercomparison project (MIP) were discussed. First, adequate (high) spatial resolution input atmospheric forcings are key. Bias correction methods were presented that combine observations gathered in the Système d'analyse fournissant des renseignements atmosphériques à la neige (SAFRAN) reanalysis system [START_REF] Quintana-Seguí | Meteorological Analysis Systems in North-East Spain: Validation of SAF-RAN and SPAN[END_REF] and simulated values from a km-scale model run executed within the European Climate Prediction (EUCP) system project (https://cordis.europa.eu/project/id/776613). The aim is to combine the available observations in low-lying areas with the simulation of near surface conditions over high elevation areas. An analysis of this new data set showed that precipitation is enhanced over high elevation areas as expected, while the evaporative demand, on the other hand, is reduced over these areas. These presentations led to a discussion of the potential coordinated land surface and hydrological MIP that could be proposed to the community. All three groups then met together and summary conclusions were made, mostly related to future WG activities in the upcoming few years (notably for the aforementioned intercomparison projects).

Public Outreach

Drought is predicted to be one of the main effects resulting from climate change in the Mediterranean. The frequency and severity of meteorological and hydrological droughts have increased in the recent past, and it is expected that this trend will continue in the 21st century in the Mediterranean, resulting in a major competition for water resources between different domestic, agricultural, industrial and tourism uses. Taking advantage of the LIAISE conference and the beginning of an uncertain irrigation campaign (owing to historically low reservoirs this year in the LIAISE region), a public outreach day entitled Climate change, drought, meteorology and its impact on water resources in Mediterranean agriculture: the LIAISE project was organized with the aim to present highlights from the LI-AISE conference to the main irrigation actors (water managers, farmers, and local politicians) at the Espai Cultural dels Canals d'Urgell (Urgell Canal Cultural Space) in Mollerussa, Spain, located within the heart of the LIAISE field campaign domain. In addition, this meeting provided a forum for debates with experts on how weather prediction models can be improved, how the drought can affect such irrigated areas of the Mediterranean, and potential strategies for how water resources can be managed best. The meeting was organized by the Institute of Agrifood Research and Technology (IRTA), The Catalonian Meteorological Service, and the Irrigation Authority in collaboration with the LIAISE project steering committee.

Quarter 3 2023

Next Steps

The LIAISE database is being updated continuously with processed field campaign data along with products derived using models and data from local sources, such as regional land cover and soil property maps. The data catalog can be viewed at https://liaise.aeris-data.fr/. Note that there is an embargo in place that restricts the data to those who have requested and been granted access; in less than two years, the data will be available via open access. This data is central to the WG activities, and these groups will continue to function through at least the next two years. Several data product and model intercomparison projects are either already underway (e.g., ET products, the mesoscale MIP) or will begin (regional hydrological MIP) within the next year or so, and they are being run through the WGs. Those interested in following or joining a WG are invited to visit the LIAISE WG webpage for more information. Some of the instrumented sites have remained or will be renewed within the LIAISE region: the Mollerussa potential evaporation site has been taking measurements since the campaign was conducted in 2021 along with the weighing lysimeters; the Els Plans rain-fed super-site 50m tower and several remote sensing instruments remained operational for approximately a year after the campaign; a surface energy balance (SEB) station and lysimeter are currently present and will be operational until at least autumn 2025; and an eddy covariance site will be installed along the shore of Lake Ivars by the end of this year, owing to the interest the campaign data generated for the lake managers. The public outreach was deemed a success, and allowed a lively debate between the scientists involved in LIAISE and local stakeholder and decision makers. As the extreme drought conditions experienced this spring in the region have already resulted in significant crop losses for many of the regional farmers, it was a timely event. LIAISE will continue to remain active in exchanges with the local water management and agricultural sectors. Finally, a future workshop will be planned, likely either in conjunction with or adjacent to the 10th International Conference on Meteorology and Climatology of the Mediterranean (MetMed) meeting, tentatively planned to be held in Toulouse, France, in spring 2025. The first day of the meeting was devoted to reports on the data collected during the TRACER campaign, initial analyses, and research plans, with the second day focused on other ongoing ACPC deep convection research activities and discussions of next steps along the ACPC deep clouds roadmap (Jensen et al., 2022). Ongoing ACPC research activities were presented that focused on disentangling the influences of aerosol, cloud microphysics, and meteorology on convective lifecycle. The relative importance of these processes and their impacts was reported to depend strongly on the character of the convective cloud and its forcing mechanism. The important role of model grid resolution on the representation of convective scale processes was quantified and highlighted as a remaining challenge for convective simulations. Even within these obstacles the significance of the representation of cloud condensation nuclei profiles in the modeling of different characteristics of severe weather was demonstrated. The importance of convective vertical transport for the profile of new particles in the atmosphere produced during both weather events and wildfires was presented. The impacts of these particles on convective clouds were found to depend strongly on the particle size (ul-

The Joint Ninth Aerosol, Clouds, Precipi- tation and Climate (ACPC) initiative and TRacking Aerosol Convection interac- tions ExpeRiment (TRACER) Workshop Houston, TX, USA and Online 16-19 May 2023 Michael P. Jensen 1 , Minghuai Wang 2 , Matthew Chris- tensen 3 , Jiwen Fan 4 , Andrew Gettelman 3 , Meinrat An- dreae 5 , Guy Dagan 6 , Franziska Glassmeier 7 , Johannes Quaas 8 , Daniel Rosenfeld 6 , Philip Stier 9 , Kentaroh Su- zuki 10 , Bethan White 11 , and Robert Wood 12 1

  Brookhaven National Laboratory, 2 Nanjing University, 3 Pacific Northwest National Laboratory, 4 Argonne National Laboratory, 5 Max Planck Institute for Chemistry, 6 Hebrew University of Jerusalem, 7 Delft University of Technology, 8 University of Leipzig, 9 Oxford University, 10 University of Tokyo, 11 Monash University, 12 University of Washington

	The joint ninth Aerosols-Clouds-Precipitation-and-Climate (ACPC; http://www.acpcinitiative.org) initiative and TRack-ing Aerosol Convection interactions ExpeRiment (TRACER; Jensen et al., 2022) workshop was convened from 16-19 May 2023 at Texas Southern University (TSU), Houston, TX, and online to discuss progress towards understanding the interac-tions among aerosols, clouds, precipitation, and climate. The event was organized around the two ACPC working groups, one focusing on aerosol interactions with deep convective clouds (Jensen et al., 2021) and the other focusing on aerosol effects on low clouds, particularly through the study of "natu-ral laboratories" (Christensen et al., 2022). During these ses-sions, participants discussed progress aligned with the ACPC roadmaps and updates on related research activities. Research-ers from around the globe presented their work, using a hier-archy of modeling and observational approaches.