Loic Themyr
email: loic.themyr@lecnam.net

Clement Rambour

Nicolas Thome

Toby Collins

Alexandre Hostettler

Full Contextual Attention for Multi-resolution Transformers in Semantic Segmentation

Transformers have proved to be very effective for visual recognition tasks. In particular, vision transformers construct compressed global representations through self-attention and learnable class tokens. Multi-resolution transformers have shown recent successes in semantic segmentation but can only capture local interactions in highresolution feature maps. This paper extends the notion of global tokens to build GLobal Attention Multi-resolution (GLAM) transformers. GLAM is a generic module that can be integrated into most existing transformer backbones. GLAM includes learnable global tokens, which unlike previous methods can model interactions between all image regions, and extracts powerful representations during training. Extensive experiments show that GLAM-Swin or GLAM-Swin-UNet exhibit substantially better performances than their vanilla counterparts on ADE20K and Cityscapes. Moreover, GLAM can be used to segment large 3D medical images, and GLAM-nnFormer achieves new state-of-the-art performance on the BCV dataset.

Introduction

Transformers have achieved state-of-the-art performances in various Natural Language Processing (NLP) tasks [START_REF] Vaswani | Attention is all you need[END_REF]. Recently, fully transformer-based models have reached excellent performances on vision tasks such as image classification [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] and semantic segmentation [START_REF] Zheng | Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers[END_REF].

The main appeal of transformers is their ability to grasp long-range interactions, which is a crucial point for semantic segmentation. However, this strategy is not easily scalable to high-resolution images involving a large number of patches, due to the quadratic complexity of the transformer's attention module. A simple and efficient strategy to tackle this limitation is to rely on multi-resolution approaches, where the attention in high-resolution feature maps is computed on sub-windows. There have been various recent attempts in this direction [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF][START_REF] Wang | Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[END_REF][START_REF] Zhang | Aggregating nested transformers[END_REF][START_REF] Wang | Pvtv2: Improved baselines with pyramid vision transformer[END_REF][START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF]. However, they limit the interactions of high-resolution features to within each window.

We introduce an approach for semantic segmentation that incorporates global attention in multi-resolution transformers (GLAM). The GLAM module enables full-range interactions to be modeled at all scales of a multi-resolution transformer. As illustrated in Fig. 1, incorporating GLAM into the Swin architecture [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF] enables to jointly capture fine-grained spatial information in high-resolution feature maps and global context, where both elements are crucial for proper segmentation in complex scenes. This concept is illustrated in Fig. 1 where Fig. 1a) shows an input image, and Fig. 1b) shows the self-attention map provided by GLAM in the highest-resolution feature map for the pedestrian region pointed out by the yellow cross in Fig. 1a). We can see that the attention map involves long-range interactions between other visual structures (cars, buildings), in contrast to the Swin baseline, where the window attention at a high-resolution feature map is limited to the small rectangular region in Fig. 1a). Consequently, GLAM has exploited longer-range interactions to successfully segment the image, as shown in 1d).

To achieve this goal, we have made the following novel contributions:

• We introduce the GLAM transformer, able to represent full-range interactions between all local features at all resolution levels. The GLAM transformer is based on learnable global tokens interacting between all visual features. To fully take into account the global context, we also design a non-local upsampling scheme (NLU).

• GLAM is a generic module that can be incorporated into any multi-resolution transformer. It consists of a succession of two transformers applied on the merged sequence of global and visual tokens and in-between global tokens. We highlight that the GLAM transformer can represent full-range interactions between image regions at all scales while retaining memory and computational efficiency. Beyond spatial interactions, global tokens also model the expected scene composition.

• Experiments on various generic (ADE20K), autonomous driving (Cityscape) and medical (Synapse) datasets show the important and systematic gain brought by GLAM when included into existing state-of-the-art multi-resolution transformers including Swin, Swin-Unet, and nn-Former. We also show that GLAM outperforms state-of-the-art methods on Synapse. Finally, ablation studies, model analysis, and visualizations are presented to assess the behavior of GLAM.

Related work

Semantic Segmentation. In the deep learning era, Fully Convolutional neural Networks (FCNs) [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF][START_REF] Shelhamer | Fully convolutional networks for semantic segmentation[END_REF][START_REF] Zhao | Pyramid scene parsing network[END_REF][START_REF] Chen | Encoder-decoder with atrous separable convolution for semantic image segmentation[END_REF][START_REF] Xiao | Unified perceptual parsing for scene understanding[END_REF] have mainly led state-of-the-art performance in semantic segmentation. For example, DeepLab [START_REF] Chen | Encoder-decoder with atrous separable convolution for semantic image segmentation[END_REF] is a model based on an encoder-decoder architecture, and U-shape networks [START_REF] Ronneberger | Unet: Convolutional networks for biomedical image segmentation[END_REF] and 3D variants [START_REF] Milletari | V-net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF][START_REF] Isensee | nnu-net: a self-configuring method for deep learning-based biomedical image segmentation[END_REF] are extremely popular in medical image segmentation. However, those models are limited to a local receptive field which is small for highresolution images. Recently, transformers [START_REF] Vaswani | Attention is all you need[END_REF] have gained a lot of interest from their ability to model long-range interactions, which allows larger spatial context information to be exploited. Vision Transformer backbones. Building on the strong performances of transformers for auto-regressive inference, fully transformer-based models for image generation have been proposed [START_REF] Child | Generating long sequences with sparse transformers[END_REF]. Other early works proposed models combining CNNs and attention for vision tasks such as object detection [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF], disparity estimation [START_REF] Li | Revisiting stereo depth estimation from a sequenceto-sequence perspective with transformers[END_REF] or semantic segmentation [START_REF] Hou | Strip pooling: Rethinking spatial pooling for scene parsing[END_REF]. More recently, fully transformer architectures have outperformed FCN baselines in image classification. ViT [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] is the first pure transformers backbone that achieved state-of-the-art performance for image classification but it requires very large training databases. DeiT [START_REF] Touvron | Training data-efficient image transformers &; distillation through attention[END_REF] managed to reduce this requirement through data-efficient training strategies and distillation. Multi-resolution transformers. Several recent approaches proposed adaptations of the vanilla ViT architecture. In particular, some architectures rely on multi-resolution processing. T2T ViT [START_REF] Yuan | Tokens-to-token vit: Training vision transformers from scratch on imagenet[END_REF] constructs richer semantic feature map through token aggregation while TnT [START_REF] Han | Transformer in transformer[END_REF] and crossViT [START_REF] Chen | Crossvit: Cross-attention multi-scale vision transformer for image classification[END_REF] uses two transformers for fine and coarse resolution. PvT [START_REF] Wang | Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[END_REF] is the first backbone with a fully pyramidal architecture that is based on windowed transformers, allowing to process the images at fine resolution and to build rich feature maps, while reducing the spatial complexity. Other methods kept this hierarchical approach while improving information sharing between the windows. Swin [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF] and its variant [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF][START_REF] Zhou | nnformer: Interleaved transformer for volumetric segmentation[END_REF] proposed to used shifted windows, Twins [START_REF] Chu | Twins: Revisiting the design of spatial attention in vision transformers[END_REF] uses interleaved fine and coarse resolu-Figure 2. The GLAM module for modeling full-range interaction in multi-resolution transformers. GLAM is included at each resolution level of any multi-resolution transformer architecture, e.g. Swin-Unet [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF] or Swin-UperNet [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF]. GLAM includes learnable global tokens, which are leveraged into a succession of two attention steps. We show that this design can indirectly represent long-range interactions between all image regions at all scales, and also external information useful for segmentation while retaining efficiency. We also introduce a non-local upsampling scheme (NLU) to extend the global context modeling in full transformer U-shape architectures such as [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF][START_REF] Zhou | nnformer: Interleaved transformer for volumetric segmentation[END_REF]. tion transformers, and CvT [START_REF] Wu | Cvt: Introducing convolutions to vision transformers[END_REF] replaces linear embedding with convolutions. Efficient Self-Attention. Long sequences have been a challenge for transformers because the original self-attention mechanism has a quadratic complexity in the sequence length. To tackle this, many approaches focus on designing efficient self-attention mechanisms.

Most of them are developed for NLP tasks and can be grouped into four categories. The first category uses a sparse approximation of the attention matrix [START_REF] Qiu | Blockwise self-attention for long document understanding[END_REF][START_REF] Kitaev | Reformer: The efficient transformer[END_REF][START_REF] Parmar | Image transformer[END_REF]. Among these approaches, window-based patch extraction vision transformers recently provided a simple yet efficient approach to compute attention [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF][START_REF] Wang | Pvtv2: Improved baselines with pyramid vision transformer[END_REF][START_REF] Haoqi Fan | Multiscale vision transformers[END_REF]. The second category is composed of methods based on a low-rank approximation of the attention matrix, such as Linformer [START_REF] Wang | Linformer: Self-attention with linear complexity[END_REF]. The third category (memory-based transformers) construct buffers of extra tokens used as static memory [START_REF] Jack W Rae | Compressive transformers for long-range sequence modelling[END_REF][START_REF] Lee | Set transformer: A framework for attention-based permutation-invariant neural networks[END_REF]. The fourth category (kernel-based methods) provides a linear approximation of the softmax kernel [START_REF] Krzysztof Marcin Choromanski | Rethinking attention with performers[END_REF][START_REF] Peng | Random feature attention[END_REF][START_REF] Katharopoulos | Transformers are rnns: Fast autoregressive transformers with linear attention[END_REF]. Some vision transformers have combined multiple efficient attention mechanisms. The recent ViT-inspired backbone PvT [START_REF] Wang | Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[END_REF] is based on windowed self-attention and attention approximation close to Linformer. ViL [START_REF] Zhang | Multi-scale vision longformer: A new vision transformer for high-resolution image encoding[END_REF] balances sparse attention by using a reduced set of global tokens (usually a single one) to extract global representations of the input image.

The GLAM method introduced in this paper is a window-and memory-based transformer that fits well in existing multi-resolution vision backbones. Unlike most other multi-resolution backbones, GLAM fully extends the notion of windowed attention to semantic segmentation by introducing global tokens at the window level, and by design-ing a specific GLAM transformer cascading window (W-MSA) and global (G-MSA) attention. We highlight that GLAM enables global communication across all image regions and also encodes learned information from all the training sets.

The GLAM Method

The main idea in GLAM is to provide a way to represent full range interactions at all feature map resolutions, which is impossible in vanilla models, especially in highresolution feature maps, due to the quadratic complexity of attention transformers.

GLAM is illustrated in Fig. 2, where it has been added to the Swin-Unet architecture [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF]. Note that GLAM can be included in various multi-resolution architectures, e.g. Swin [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF] or PvT [START_REF] Wang | Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[END_REF] and is also applicable for 3D segmentation, e.g. nn-Former [START_REF] Zhou | nnformer: Interleaved transformer for volumetric segmentation[END_REF]. The core idea in GLAM is to design global tokens (in red in Fig. 2), which are leveraged into a succession of two attention steps: first, between visual tokens in each window independently and, second, between global tokens among different windows. We show in Fig. 3.2 that this design enables to represent full range interactions between all image regions at all scales, and also external information useful for segmentation, while retaining efficiency. We also introduce a non-local upsampling scheme (NLU) to extend the full context modeling in Ushape architectures and to provide an efficient interpolation of rich semantic feature maps in an associated decoder.

Multi-resolution transformer architecture

As shown in Fig. 2, GLAM can be included into any multi-resolution transformer architecture [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF][START_REF] Wang | Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[END_REF][START_REF] Zhang | Aggregating nested transformers[END_REF][START_REF] Wang | Pvtv2: Improved baselines with pyramid vision transformer[END_REF][START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF][START_REF] Zhou | nnformer: Interleaved transformer for volumetric segmentation[END_REF]. Transformer. At each resolution level s, given a sequence of visual tokens, a transformer learns representations through Self-Attention (SA). SA is given by the expectation of each token with respect to their probability of sharing the same embedding. The Multi-head Self-Attention (MSA) is obtained from the linear combination of m parallel SA operations. Finally, a complete transformer module is obtained by plugging the output of the MSA into a Multi-Layer Perceptron (MLP). Layer norm operations, as well as residual connections, are added respectively before and after MSA and MLP modules. Windowed attention. MSA cannot be applied to long sequences e.g. patches from high-resolution images becasue the computation of the attention matrix has quadratic memory complexity. To allow high-resolution processing and thus long sequences of small patches, windowed transformers treat the image as a batch of non-overlapping windows [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF][START_REF] Wang | Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[END_REF][START_REF] Wang | Pvtv2: Improved baselines with pyramid vision transformer[END_REF][START_REF] Zhang | Aggregating nested transformers[END_REF]. This approach is combined with a pooling strategy [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF][START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF][START_REF] Wang | Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[END_REF][START_REF] Zhang | Aggregating nested transformers[END_REF] and is well suited to build a multiresolution encoder, able to produce rich semantic maps. Multi-resolution backbones are built by chaining windowed transformer blocks and downsampling. These hierarchical architectures manage to build larger receptive fields in deeper layers, similarly to CNNs. This, however, does not guarantee a global receptive field and the maximal receptive field depends on the model's depth. More importantly, this processing introduces a major modification to the transformer modules. At a finer resolution, only local interactions are considered. With this modification, the processing of isolated patches by self-attention may not be as effective as global self-attention performed on the full image.

Global attention multi-resolution transformers

We show how the GLAM module can provide global attention in all feature maps of multi-resolution transformers. The GLAM transformer is illustrated in Fig. 3, consisting in a sequence of L transformer blocks, processing visual tokens in each region of the multi-resolution maps (shown in blue in Fig. 3) and global tokens (shown in red in Fig. 3).

The basic idea behind GLAM is to associate global tokens to each window that are responsible to encapsulate the local information and transmit it to other image regions by computing MSA between all global tokens. Thus, when information is processed at the window scale, the visual tokens embedding incorporate useful long-range information.

Global Tokens. Global tokens lie at the core of Global Attention (GA). They are specific tokens concatenated to each window and are responsible for communication between windows. We define as N r the number of windows in the feature map, N p as the number of patches per window, and {w l k } 1≤k≤Nr as the sequence of windows after being processed by the l th GLAM-transformer block. We define as {g l k } 1≤k≤Nr the sequence of N g -dimensional global tokens associated to each window. The initialization of the global tokens {g 0 k } 1≤k≤Nr is the same for all windows and is learned by the model. The input of the l th transformer block, defined as z l , is a batch of tokens from each window concatenated with the corresponding global tokens, i.e. z l ∈ R Nr×(Ng+Np)×C with C being the dimension of the tokens. Consequently, the elements in the batch have the form:

∀k ∈ [1. . N r ], z l k = g l k w l k ∈ R (Ng+Np)×C . (1)
GLAM-Transformer. The communication between windows at a given hierarchy level is obtained through the interaction of global tokens. At each block l of the GLAMtransformer, there are two steps: i) visual tokens grasp their local statistics through a local window transformer (W-MSA), and ii) the global tokens are re-embedded by a global transformer (G-MSA), where global tokens from different windows interact with each other. Formally, the l th GLAMtransformer block inputs z l-1 and outputs z l by the succession of a W-MSA and a G-MSA step:

ẑl = W-MSA(z l-1 ) , g l = G-MSA(ĝ l ) , z l = g l T k ŵl T k T (2)
We define as A l r the attention matrix for the window r in the transformer block l. We introduce the following decomposition to express the attention with respect to the global and local tokens:

A l r = A l r,gg A l r,gw A l r,wg A l r,ww . (3)
The square matrices A l r,gg ∈ R Ng×Ng and A l r,ww ∈ R Np×Np give the attention from the global token and the spatial tokens on themselves respectively. The matrices A l r,gw ∈ R Ng×Np and A l r,wg ∈ R Np×Ng are the cross attention matrices between local and global tokens. We define as B l ∈ R (Nr•Ng)×(Nr•Ng) the global attention matrix from all the global token sequence and B l ij ∈ R Ng×Ng as the sub-matrices giving the attention between the global tokens of windows i and j. GLAM-Transformer properties. Putting aside the value matrix, the W-MSA gives the following embedding ĝl r from g l-1 r : ĝr = A l r,gg g l-1 r The G-MSA, i.e. the MSA on the sequence of global tokens gives the following embeddings:

+ A l r,gw w l-1 r . (4)
g l r = Nr n=1 B l rn ĝl n = Nr n=1 B l rn (A l r,gg g l-1 r + A l r,gw w l-1 r ) . (5)
From Eq. ( 5), we have the expression of the global token for a window r processed by the l G-MSA block transformer. Developing this formulation we obtain the following expression for the k th global token in the r th window:

g l k,r = Nr r ′ =1 Ng j=1 b k,r,j,r ′ Ng+Np i=1 a j,r ′ ,i z l-1 i,r ′ = Nr r ′ =1 Ng j=1 b k,r,j,r ′ Ng i=1 a j,r ′ ,i g l-1 i,r ′ + Np i=1 a j,r ′ ,i+Ng w l-1 i,r ′ . ( 6
)
The variables z i,r , g i,r and w i,r corresponds respectively to the visual, global or generic token i in window r. a j,r,i is the attention coefficient given by the token j to the token i inside the window r. b j,r,i,r ′ is the attention coefficient from the global token j in the window r to the global token j in the window r ′ . Re-arranging the indices of equation Eq. ( 6) leads to the following expression for the k th global token in the r th window:

g l k,r = Nr r ′ =1 Np i=1   Ng j=1 b k,r,j,r ′ a j,r ′ ,(i+Ng) w l-1 i,r ′   + Nr r ′ =1   Ng j=1 b k,r,j,r ′ Ng i=1 a j,r ′ ,i g l-1 i,r ′   (7)
This leads to a global attention matrix G k ∈ R (Nr•Np)×(Nr•Np) associated to the k th global token given by [G k ] r ′ ,i = Ng j=1 b k,r,j,r ′ a j,r ′ ,(i+Ng) + Ng j=1 b k,r,j,r ′ Ng i=1 a j,r ′ ,i . Eq. ( 7) gives the embedding of the the global token g l k,r at the l st GLAM-transformer block, with respect to all visual tokens in all feature map windows w l-1 i,r ′ (first row), and all global tokens g l-1 i,r ′ (second row). This rewriting shows that the global embedding g l k,r captures interactions between all image regions independently of the resolution. The different terms in the decomposition are interpreted as an attention map associated with each image region. This is the visualization shown in Fig. 1: the row of the first term corresponds to patch-based attention which depends on all the tokens of the feature map, while the second row represents window-based attention.

Overall, global tokens embedded with GLAMtransformers provide a way for information propagation across all windows (first row in Fig. 7), but also global information (second row) that goes beyond matching visual features in a single image. Especially, this represents global and learned information across the dataset, and can be leveraged as a stabilizing effect in SA, because the information is shared not only from the input but from all the windows in the dataset. This makes them a powerful tool to interpret isolated tokens and to take advantage of redundant structures in the data.

Non-Local Upsampling. We introduce a Non-Local Upsampling (NLU) module for a fully transformer decoder such as [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF][START_REF] Zhou | nnformer: Interleaved transformer for volumetric segmentation[END_REF]. NLU is designed to upsample the semantic features based on all the tokens coming from the skip connection, by drawing inspiration for non-local means [START_REF] Buades | A non-local algorithm for image denoising[END_REF].

The proposed NLU is illustrated in the supplementary material. To perform the upsampling, the skip connections are embedded into a query matrix of size (4N p ) × C while the semantic low-resolution features are embedded into the keys and values of size N p × C. The projection of the values on the resulting attention matrix has the size (4N p ) × C.

Experiments

Experimental Settings

Datasets. We evaluated on three different semantic segmentation datasets: ADE20K [START_REF] Zhou | Semantic understanding of scenes through the ade20k dataset[END_REF], Cityscapes [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF] and Synapse [START_REF] Bennett Landman | Multi-atlas labeling beyond the cranial vault[END_REF]. ADE20K is a scene parsing dataset composed of 20,210 images with 150 object classes. Cityscapes contains driving scenes and is composed of 5,000 images annotated with 19 different classes. Synapse is an abdominal organ segmentation dataset that includes 30 Computerized Tomography (CT) scans which are 3D volumes annotated with 8 abdominal organs. Implementation details. GLAM models were implemented into the mmseg [START_REF]MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark[END_REF] codebase and the models were trained on 8 Tesla V100 GPUs. The layers were pretrained on ImageNet-1K and standard augmentation was used: random crop, rotations, translations, etc. More details are provided in supplementary. We used the Adam optimizer with a weigh decay of 0.01 and a polynomial learning rate scheduler starting from 0.00006 and with a factor of 1.0. The reported segmentation performances are mean Intersection over Union (mIoU) for ADE20k and Cityscapes and Dice Similarity Score (DSC) for Synapse.

GLAM performance

GLAM in multi-resolution transformers. GLAM is well suited to work with window transformers such as PvT [START_REF] Wang | Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[END_REF][START_REF] Wang | Pvtv2: Improved baselines with pyramid vision transformer[END_REF] or Swin [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF] as well as its variants [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF][START_REF] Zhou | nnformer: Interleaved transformer for volumetric segmentation[END_REF]. Due to the top performances of Swin, we incorporated GLAM into this backbone to compute the segmentation of 2D datasets leading to two models: GLAM-Swin-UperNet and GLAM-Swin-Unet. The first one is a hybrid model combining a transformer backbone and a CNN head [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF][START_REF] Xiao | Unified perceptual parsing for scene understanding[END_REF] while the second one is a full transformer model with a decoder symmetric to the encoder [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF]. For 3D images, GLAM was plugged into nnFormer [START_REF] Zhou | nnformer: Interleaved transformer for volumetric segmentation[END_REF] which is designed similarly to Swin-Unet for 3D medical image segmentation. The performances of the Swin and GLAM models are presented in Table 1. GLAM models exhibit important and consistent performance gains compared to their vanilla counter- [START_REF] Shelhamer | Fully convolutional networks for semantic segmentation[END_REF] ResNet-101 41.40 77.34 CCNet [START_REF] Huang | Ccnet: Criss-cross attention for semantic segmentation[END_REF] ResNet-101 43.71 79.45 DANet [START_REF] Fu | Dual attention network for scene segmentation[END_REF] ResNet-101 43.64 80.47 UperNet [START_REF] Xiao | Unified perceptual parsing for scene understanding[END_REF] ResNet-101 43.82 80.10 DNL [START_REF] Yin | Disentangled non-local neural networks[END_REF] ResNet-101 44.25 79.41 PSPNet [START_REF] Zhao | Pyramid scene parsing network[END_REF] ResNet-101 44.39 79.08 DeepLabV3+ [START_REF] Chen | Encoder-decoder with atrous separable convolution for semantic image segmentation[END_REF] ResNet-101 45.47 79.41 Trans2Seg [START_REF] Wang | Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[END_REF] PVT-S 42.60 -FPN [START_REF] Wang | Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[END_REF] PVT-L 42.10 -TNT [START_REF] Han | Transformer in transformer[END_REF] TNT-S 43.60 -SETR-PUP [START_REF] Zheng | Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers[END_REF] DeiT-L 46.34 79.21 Swin-Unet [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF] Swin-B 47.85 -Swin-UperNet [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF] Swin-B 47.99 80.79 Twins-SVT-L [START_REF] Chu | Twins: Revisiting the design of spatial attention in vision transformers[END_REF] Twins-SVT

Methods

Average Dice Score (%) VNet [START_REF] Milletari | V-net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF] 68.81 U-Net [START_REF] Ronneberger | Unet: Convolutional networks for biomedical image segmentation[END_REF] 76.85 Att-UNet [START_REF] Oktay | Attention u-net: Learning where to look for the pancreas[END_REF] 77.77 R50-Deeplabv3+ [START_REF] Chen | Encoder-decoder with atrous separable convolution for semantic image segmentation[END_REF] 75.73 TransUNet [START_REF] Chen | Transunet: Transformers make strong encoders for medical image segmentation[END_REF] 77.48 Swin-Unet [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF] 79.13 TransClaw U-Net [START_REF] Chang | Transclaw u-net: Claw u-net with transformers for medical image segmentation[END_REF] 78.09 nnUNet (3D) [START_REF] Isensee | nnu-net: a self-configuring method for deep learning-based biomedical image segmentation[END_REF] 86.99 nnFormer [START_REF] Zhou | nnformer: Interleaved transformer for volumetric segmentation[END_REF] 87.40 GLAM-nnFormer 88.60

Model Analysis

In this part, we analyze various important aspects of GLAM.

Number of Global Tokens. The number of global tokens directly influences the capacity of GLAM to model global interactions between the windows. Fig. 4 shows the impact of this hyper-parameter on segmentation performances. We can see that using more global tokens improves performances. However, it also increases the number of parameters and memory cost which forces a trade-off. We keep a reasonable value of 10 global tokens, which gives an important performance boost of +1.4pts in both the tiny and base versions of the Swin-Unet model. Impact of NLU. GLAM improves context modeling in multi-resolution transformers thanks to global attention and Non-Local Upsampling (NLU). Table 4 provides an ablation study of these two components. We can see that NLU gives an improvement 0.45pt compared to the original Swin-Unet that uses a patch expension operation for upsampling. GLAM brings another large improvement for a total gain of +1.44pts compared to the baseline.

Long-range interaction. To highlight the impact of G-MSA, Table 5 shows the performances of GLAM backbones using only a W-MSA step but no G-MSA. GLAM backbones show consistent gains compared to their counterparts without G-MSA. This ablation highlight the crucial role of this step to leverage long-range interactions and that the performance gains made by GLAM can not only be explained by the parameter overhead. Parameter and FLOPs overhead. The overhead due to the global tokens is controlled and proportional to the number of GLAM transformer blocks. This overhead brings higher performance gains than increasing the backbone size which validates the model architecture. Tab. 6 illustrates that the GLAM-Swin Base backbones show superior efficiency compared to their vanilla Large counterpart with a superior mIoU increase with respect to additional learnable parameters. The same analysis can be done with FLOPs overhead with a higher mIoU increase per extra-FLOP for GLAM-Swin Base compared to Swin Large. Visualizations. Fig. 6 shows the GLAM attention averaged over the axial direction for the red cross (pancreas). We can see that long-range dependencies are involved, with a much larger spatial extent than the local window (in blue), where attention is given to neighboring organs (stomach and aorta). The full context is crucial to properly segment complex organs with visual local ambiguities such as the pancreas. In Fig. Fig. 5b), we show segmentation results of GLAM-nn-Former for 3D medical image segmentation. We show the results on a given 2D slice. We can notice that GLAM nn-Former is qualitatively much better at segmenting the stomach (in pink) than nn-Former. This can be explained by the global interactions of our model, which enables it to better represent specific interactions between organs.

Conclusion

This paper introduces GLAM, a method for modeling full contextual interactions in multi-resolution transformerbased models. the GLAM transformer leverage learnable global tokens at each resolution level of the model, which allows a complete interaction of the tokens across the image regions, and is further equipped with a non-local upsampling module. Experiments show the large and consistent gain of GLAM when incorporated into several multiresolution transformers (Swin-Unet, nn-Former, Swin) on diverse medical, street, or more general images. Future works includes applying the GLAM idea for modeling full contextual information on very high-resolution images or 3D medical volumes.

Figure 1 .

 1 Figure1. When segmenting the high-resolution image in a) with state-of-the-art multi-resolution transformers, e.g. Swin[START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF], the attention in the highest-resolution feature maps is limited to a small spatial region, i.e. the blue square for the yellow-crossed pedestrian. Our method incorporates GLobal Attention in Multi-resolution transformers (GLAM). The GLAM attention map for the pedestrian in a) is depicted in b): it captures both fine-grained spatial information and long-range interactions, enabling successful segmentation, as shown in d).

Figure

  Figure3. GLAM-Transformer: as in multi-resolution approaches, each input feature map is divided into Nr non overlapping windows (blue). The core idea in GLAM is to design learnable global tokens (in red). The visual tokens from each window are concatenated with the global tokens and processed through a local window transformer (W-MSA). Every W-MSA is followed by a global transformer (G-MSA), where global tokens between different windows interact with each other, which brings a global representation to each window. These two steps give the GLAM-Transformer block; Multiple blocks are chained at every hierarchy level in typical multi-resolution transformer backbones. We show that global tokens learned from GLAM-Transformer indirectly model global interactions between all visual tokens in all widows. The global tokens are also able to represent extra learnable knowledge beyond the patch interactions in a single image.

Figure 4 .

 4 Figure 4. Impact of the number of global tokens on performance (mIoU) using ADE20k. Table 4. Impact of the NLU and the GLAM transformer on a tiny Swin-Unet, 10 global tokens, on ADE20k. Method NLU GLAM mIoU Swin-Unet-T 42.75 Swin-Unet-T ✓ 43.20 Swin-Unet-T ✓ ✓ 44.20

Fig. 5

 5 shows qualitative visualizations of the GLAM method. In Fig. Fig. 5a), we show GLAM attention maps for the highest resolution feature a) Segmentation results and global attention of GLAM on ADE20K. b) Segmentation results on Synapse.

Figure 5 .

 5 Figure 5. Qualitative visualisations of GLAM. We show the ability of GLAM to model full contextual information in high-resolution feature maps on ADE20K (first row), and the ability of GLAM-nn-Former to accurately segment the stomach (in pink).

2 FLOPs 2 Swin

 22 backbone#param. ↑ rel. mIoU / #param ×10 -↑ rel. mIoU / FLOPs ×10 -

Figure 6 .

 6 Figure 6. Averaged GLAM attention map in 3D. The information inside the blue window is ambiguous. To segment the voxel at the red cross, the model leverages long-range dependencies including neighbor organs. The pancreas is in green, the aorta in red, and the stomach in blue.

  [START_REF] Chang | Transclaw u-net: Claw u-net with transformers for medical image segmentation[END_REF]. GLAM-Transformer: as in multi-resolution approaches, each input feature map is divided into Nr non overlapping windows (blue). The core idea in GLAM is to design learnable global tokens (in red). The visual tokens from each window are concatenated with the global tokens and processed through a local window transformer (W-MSA). Every W-MSA is followed by a global transformer (G-MSA), where global tokens between different windows interact with each other, which brings a global representation to each window. These two steps give the GLAM-Transformer block; Multiple blocks are chained at every hierarchy level in typical multi-resolution transformer backbones. We show that global tokens learned from GLAM-Transformer indirectly model global interactions between all visual tokens in all widows. The global tokens are also able to represent extra learnable knowledge beyond the patch interactions in a single image.

Table 1 .

 1 GLAM Improvements on various multi-resolution transformers. Performances are evaluated with respect to mIoU for ADE20k and Cityscapes and average DSC for Synapse.

Table 2 .

 2 Comparison to state of the art methods on ADE20K and Cityscapes. All experiments are made or reported are with single-

	scale inference.
			ADE20K Cityscapes
	Method	Backone	mIoU	mIoU
	FCN

Table 3 .

 3 Comparison to state of the art methods on Synapse.

			48.80	-
	GLAM-Swin-Unet	Swin-B	49.10	-
	GLAM-Swin-UperNet Swin-B	48.44	81.47

Table 5 .

 5 Impact of G-MSA phase on GLAM transformer on different model, 10 global tokens, on ADE20k. GLAM-nogmsa is GLAM without the G-MSA phase.

	Method	mIoU
	GLAM-nogmsa-Swin-Unet B	47.90
	GLAM-Swin-Unet B	49.10
	GLAM-nogmsa-Swin-UperNet B 47.95
	GLAM-Swin-UperNet B	48.44

Table 6 .

 6 Analysis

of the relative mIoU increase with respect to extra learnable parameters and FLOPs compared to the standard Base and Large backbones.

Dataset

Method Size Score Swin-Unet [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF] Tiny 42.75 GLAM-Swin-Unet Tiny 44.19 Swin-UNet [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF] Small 47.49 GLAM-Swin-UNet Small 47.90 Swin-Unet [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF] Base 47.85 GLAM-Swin-Unet Base 49.10 Swin-UperNet [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF] Tiny 43.69 GLAM-Swin-UperNet Tiny 44.16 Swin-UperNet [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF] Small 47.72 GLAM-Swin-UperNet Small 47.75 Swin-UperNet [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF] Base 47.99 ADE20K GLAM-Swin-UperNet Base 48.44

Swin-UperNet [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF] Tiny 78.24 GLAM-Swin-UperNet Tiny 78.64 Swin-UperNet [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF] Base 80.79 GLAM-Swin-UperNet Base 81.47 Swin-Unet [START_REF] Hu Cao | Swin-unet: Unet-like pure transformer for medical image segmentation[END_REF] Tiny 77.

State-of-the-art comparison.

We now compare the GLAM-Swin models with existing approaches on the ADE20K [START_REF] Zhou | Semantic understanding of scenes through the ade20k dataset[END_REF], Cityscapes [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF] and Synapse [START_REF] Bennett Landman | Multi-atlas labeling beyond the cranial vault[END_REF].

ADE20K and Cityscapes. Table 2 summarizes our results. To be fair, we compared models up to ∼ 150M parameters, and we report the top performances from the mmseg [START_REF]MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark[END_REF] benchmark for all methods, with 160K training epochs for all methods. Moreover, we compared only methods trained on 768 × 768 resolution images on Cityscapes. In this setup, GLAM-Swin-Unet yields 49.10% mIoU on ADE20K outperforming its vanilla Swin counterpart with at least 1.10% mIoU. GLAM-Swin-UperNet achieves 81.47 % mIoU on Cityscapes which is 1.58 % better than its Swin-Upernet counterpart.

Synapse. Table 3 reports our results and recent baselines for 3D medical segmentation. GLAM-nnFormer significantly outperforms all other existing methods by at least 1.2% average Dice. To the best of our knowledge, GLAM-nnFormer outperforms state-of-the-art on the Synapse dataset.