Vivien Sainte Fare

Garnot Loic

Landrieu Lastig
email: loic.landrieu@ign.fr

Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks

Unprecedented access to multi-temporal satellite imagery has opened new perspectives for a variety of Earth observation tasks. Among them, pixel-precise panoptic segmentation of agricultural parcels has major economic and environmental implications. While researchers have explored this problem for single images, we argue that the complex temporal patterns of crop phenology are better addressed with temporal sequences of images. In this paper, we present the first end-to-end, single-stage method for panoptic segmentation of Satellite Image Time Series (SITS). This module can be combined with our novel image sequence encoding network which relies on temporal selfattention to extract rich and adaptive multi-scale spatiotemporal features. We also introduce PASTIS, the first openaccess SITS dataset with panoptic annotations. We demonstrate the superiority of our encoder for semantic segmentation against multiple competing architectures, and set up the first state-of-the-art of panoptic segmentation of SITS. Our implementation and PASTIS are publicly available.

Introduction

The precision and availability of Earth observations have continuously improved thanks to sustained advances in space-based remote sensing, such as the launch of the Planet [START_REF] Boshuizen | Results from the planet labs flock constellation[END_REF] and the open-access Sentinel constellations [START_REF] Drusch | Sentinel-2: Esa's optical high-resolution mission for gmes operational services[END_REF]. In particular, satellites with high revisit frequency contribute to a better understanding of phenomena with complex temporal dynamics. Crop mapping-the driving application of this paper-relies on exploiting such temporal patterns [START_REF] Sainte | Time-space tradeoff in deep learning models for crop classification on satellite multispectral image time series[END_REF] and entails major financial and environmental stakes. Indeed, remote monitoring of the surface and nature of agricultural parcels is necessary for a fair allocation of agricultural subsidies (50 and 22 billion euros per year in Europe and in the US, respectively) and for ensuring that best crop rotation practices are respected. More generally, the automated analysis of SITS represents a significant interest for a wide Figure 1: Overview. We propose an end-to-end, singlestage model for panoptic segmentation of agricultural parcels from time series of satellite images. Note the difficulty of resolving the parcels' borders from a single image, highlighting the need for modeling temporal dynamics. range of applications, such as surveying urban development and deforestation.

The task of monitoring both the content and extent of agricultural parcels can be framed as the panoptic segmentation of an image sequence. Panoptic segmentation consists of assigning to each pixel a class and a unique instance label, and has become a standard visual perception task in computer vision [START_REF] Kirillov | Panoptic feature pyramid networks[END_REF][START_REF] Mohan | Efficientps: Efficient panoptic segmentation[END_REF]. However, panoptic segmentation is a fundamentally different task for SITS versus sequences of natural images or videos. Indeed, understanding videos requires tracking objects through time and space [START_REF] Tokmakov | Learning to segment moving objects[END_REF]. In yearly SITS, the targets are static in a geo-referenced frame, which removes the need for spatial tracking. Additionally, SITS share a common temporal frame of reference, which means that the time of acquisition itself contains information useful for modeling the underlying temporal dynamics. In contrast, the frame number in videos is often arbitrary. Finally, while objects on the Earth surface generally do not occlude one another, as is commonly the case for objects in natural images, varying cloud cover can make the analysis of SITS arduous. For the specific problem addressed in this paper, individualizing agricultural parcels requires learning complex and specific temporal, spatial, and spectral patterns not commonly encountered in video processing, such as differences in plant phenological profiles, subpixel border information, and swift human interventions such as harvests.

While deep networks have proven efficient for learning such complex patterns for pixel classification [START_REF] Kamilaris | Deep learning in agriculture: A survey[END_REF][START_REF] Ienco | Land cover classification via multitemporal spatial data by deep recurrent neural networks[END_REF][START_REF] Audebert | Semantic segmentation of earth observation data using multimodal and multi-scale deep networks[END_REF], there is no dedicated approach for detecting individual objects in SITS. Existing work on instance segmentation has been restricted to analysing a single satellite image [START_REF] Rieke | Deep learning for instance segmentation of agricultural fields[END_REF]. In summary, specialized remote sensing methods are limited to semantic segmentation or single-image instance segmentation, while computer vision's panoptic-ready networks require significant adaptation to be applied to SITS.

In this paper, we introduce U-TAE (U-net with Temporal Attention Encoder), a novel spatio-temporal encoder combining multi-scale spatial convolutions [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] and a temporal self-attention mechanism [START_REF] Sainte | Time-space tradeoff in deep learning models for crop classification on satellite multispectral image time series[END_REF] which learns to focus on the most salient acquisitions. While convolutional-recurrent methods are limited to extracting temporal features at the highest [START_REF] Rußwurm | Convolutional LSTMs for cloud-robust segmentation of remote sensing imagery[END_REF] or lowest [START_REF] Rustowicz | Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods[END_REF] spatial resolutions, our proposed method can use the predicted temporal masks to extract specialized and adaptive spatio-temporal features at different resolutions simultaneously. We also propose Parcels-as-Points (PaPs), the first end-to-end deep learning method for panoptic segmentation of SITS. Our approach is built upon the efficient CenterMask network [START_REF] Wang | Centermask: single shot instance segmentation with point representation[END_REF], which we modify to fit our problem. Lastly, we present Panoptic Agricultural Satellite TIme-Series (PASTIS), the first open-access dataset for training and evaluating panoptic segmentation models on SITS, with over 2 billion annotated pixels covering over 4000km 2 . Evaluated on this dataset, our approach outperforms all reimplemented competing methods for semantic segmentation, and defines the first state-of-the-art of SITS panoptic segmentation.

Related Work

To the best of our knowledge, no instance or panoptic segmentation method operating on SITS has been proposed to date. However, there is a large body of work on both the encoding of satellite sequences, and the panoptic segmentation of videos and single satellite images.

Encoding Satellite Image Sequences. While the first automated tools for SITS analysis relied on traditional machine learning [START_REF] Inglada | Assessment of an operational system for crop type map production using high temporal and spatial resolution satellite optical imagery[END_REF][START_REF] Vuolo | How much does multi-temporal Sentinel-2 data improve crop type classification? International journal of applied earth observation and geoinformation[END_REF], deep convolutional networks allow for the extraction of richer spatial descriptors [START_REF] Kussul | Deep learning classification of land cover and crop types using remote sensing data[END_REF][START_REF] Ienco | Land cover classification via multitemporal spatial data by deep recurrent neural networks[END_REF][START_REF] Audebert | Semantic segmentation of earth observation data using multimodal and multi-scale deep networks[END_REF][START_REF] Kamilaris | Deep learning in agriculture: A survey[END_REF]. The temporal dimension was initially dealt via handcrafted temporal descriptors [START_REF] Bailly | Dense bag-of-temporal-siftwords for time series classification[END_REF][START_REF] Tavenard | Efficient temporal kernels between feature sets for time series classification[END_REF][START_REF] Ye | Time series shapelets: a new primitive for data mining[END_REF] or probabilistic models [START_REF] Bailly | Crop-rotation structured classification using multi-source Sentinel images and LPIS for crop type mapping[END_REF], which have been advantageously replaced by recurrent [START_REF] Rußwurm | Convolutional LSTMs for cloud-robust segmentation of remote sensing imagery[END_REF][START_REF] Sainte | Time-space tradeoff in deep learning models for crop classification on satellite multispectral image time series[END_REF][START_REF] Ozgur Turkoglu | Crop mapping from image time series: deep learning with multi-scale label hierarchies[END_REF], convolutional [START_REF] Pelletier | Temporal convolutional neural network for the classification of satellite image time series[END_REF][START_REF] Rustowicz | Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods[END_REF][START_REF] Ji | 3d convolutional neural networks for crop classifi-cation with multi-temporal remote sensing images[END_REF], or differential [START_REF] Metzger | Crop classification under varying cloud cover with neural ordinary differential equations[END_REF] architectures. Recently, attention-based approaches have been adapted to encode sequences of remote sensing images and have led to significant progress for pixel-wise and parcel-wise classification [START_REF] Sainte | Satellite image time series classification with pixel-set encoders and temporal self-attention[END_REF][START_REF] Rußwurm | Self-attention for raw optical satellite time series classification[END_REF][START_REF] Yuan | Self-supervised pre-training of transformers for satellite image time series classification[END_REF]. In parallel, hybrid architectures [START_REF] Stoian | Land cover maps production with high resolution satellite image time series and convolutional neural networks: Adaptations and limits for operational systems[END_REF][START_REF] Rustowicz | Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods[END_REF][START_REF] Papadomanolaki | A deep multi-task learning framework coupling semantic segmentation and fully convolutional LSTM networks for urban change detection[END_REF] relying on U-Net-type architectures [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] for encoding the spatial dimension and recurrent networks for the temporal dimension have shown to be well suited for the semantic segmentation of SITS. In this paper, we propose to combine this hybrid architecture with the promising temporal attention mechanism.

Instance Segmentation of Satellite Images. The first step of panoptic segmentation is to delineate all individual instances, i.e. instance segmentation. Most remote sensing instanciation approaches operate on a single acquisition. For example, several methods have been proposed to detect individual instances of trees [START_REF] Qin | Individual tree segmentation over large areas using airborne lidar point cloud and very high resolution optical imagery[END_REF][START_REF] Zhao | Tree canopy differentiation using instance-aware semantic segmentation[END_REF], buildings [START_REF] Fabien | Unet-id, an instance segmentation model for building extraction from satellite images-case study in the joanópolis city, brazil[END_REF], or fields [START_REF] Rieke | Deep learning for instance segmentation of agricultural fields[END_REF]. Several algorithms start with a delineation step (border detection) [START_REF] Garcia-Pedrero | A machine learning approach for agricultural parcel delineation through agglomerative segmentation[END_REF][START_REF] Mudrik Masoud | Delineation of agricultural field boundaries from Sentinel-2 images using a novel super-resolution contour detector based on fully convolutional networks[END_REF][START_REF] Franc | Deep learning on edge: extracting field boundaries from satellite images with a convolutional neural network[END_REF], and require postprocessing to obtain individual instances. Other methods use segmentation as a preprocessing step and compute cluster-based features [START_REF] Censi | Spatial-temporal GraphCNN for land cover mapping[END_REF][START_REF] Derksen | Spatially precise contextual features based on superpixel neighborhoods for land cover mapping with high resolution satellite image time series[END_REF], but do not produce explicit cluster-to-object mappings. Petitjean et al. [START_REF] Franc ¸ois Petitjean | Spatio-temporal reasoning for the classification of satellite image time series[END_REF] propose a segmentationaided classification method operating on image time series. However, their approach partitions each image separately and does not attempt to retrieve individual objects consistently across the entire sequence. In this paper, we propose the first end-to-end framework for directly performing joint semantic and instance segmentation on SITS.

Panoptic Segmentation of Videos. Among the vast literature on instance segmentation, Mask-RCNN [START_REF] He | Mask R-CNN[END_REF] is the leading method for natural images. Recently, Wang et al.

proposed CenterMask [START_REF] Wang | Centermask: single shot instance segmentation with point representation[END_REF], a lighter and more efficient single-stage method which we use as a starting point in this paper. Several approaches propose extending instance or panoptic segmentation methods from image to video [START_REF] Yang | Video instance segmentation[END_REF][START_REF] Tokmakov | Learning to segment moving objects[END_REF][START_REF] Kim | Video panoptic segmentation[END_REF]. However, as explained in the introduction, SITS differs from natural video in several key ways which require specific algorithmic and architectural adaptations.

Method

We consider an image time sequence X, organized into a four-dimensional tensor of shape T × C × H × W , with T the length of the sequence, C the number of channels, and H × W the spatial extent.

Spatio-Temporal Encoding

Our model, dubbed U-TAE (U-Net with Temporal Attention Encoder), encodes a sequence X in three steps: (a) each image in the sequence is embedded simultaneously and independently by a shared multi-level spatial convolutional encoder, (b) a temporal attention encoder collapses the temporal dimension of the resulting sequence of feature maps into a single map for each level, (c) a spatial convolutional decoder produces a single feature map with the same resolution as the input images, see Figure 2.

(×2) Upsampling T × C 1 × H × W C 1 × H × W D 4 × H 8 × W 8 D 3 × H 4 × W 4 D 2 × H 2 × W 2 D 1 × H × W Attention masks d 4 d 3 d 2 d 1 T e 3
a) Spatial Encoding. We consider a convolutional encoder E with L levels 1, • • • , L. Each level is composed of a sequence of convolutions, Rectified Linear Unit (ReLu) activations, and normalizations. Except for the first level, each block starts with a strided convolution, dividing the resolution of the feature maps by a factor 2.

For each time stamp t simultaneously, the encoder E l at level l takes as input the feature map of the previous level e l-1 t , and outputs a feature map e l t of size C l × H l × W l with H l = H/2 l-1 and W l = W/2 l-1 . The resulting feature maps are then temporally stacked into a feature map sequence e l of size T × C l × H l × W l :

e l = [E l (e l-1 t )] T t=0 for l ∈ [1, L] , (1)
with e 0 = X and [ • ] the concatenation operator along the temporal dimension. When constituting batches, we flatten the temporal and batch dimensions. Since each sequence comprises images acquired at different times, the batches' samples are not identically distributed. To address this issue, we use Group Normalization [START_REF] Wu | Group normalization[END_REF] with 4 groups instead of Batch Normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] in the encoder.

b) Temporal Encoding. In order to obtain a single representation per sequence, we need to collapse the temporal dimension of each feature map sequence e l before using them as skip connections. Convolutional-recurrent U-Net networks [START_REF] Stoian | Land cover maps production with high resolution satellite image time series and convolutional neural networks: Adaptations and limits for operational systems[END_REF][START_REF] Rustowicz | Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods[END_REF][START_REF] Papadomanolaki | A deep multi-task learning framework coupling semantic segmentation and fully convolutional LSTM networks for urban change detection[END_REF] only process the temporal dimension of the lowest resolution feature map with a temporal encoder. The rest of the skip connections are collapsed with a simple temporal average. This prevents the extraction of spatially adaptive and parcel-specific temporal patterns at higher resolutions. Conversely, processing the highest resolution would result in small spatial receptive fields for the temporal encoder, and an increased memory requirement. Instead, we propose an attention-based scheme which only processes the temporal dimension at the lowest feature map resolution, but is able to utilize the predicted temporal attention masks at all resolutions simultaneously. Based on its performance and computational efficiency, we choose the Lightweight-Temporal Attention Encoder (L-TAE) [START_REF] Sainte | Lightweight temporal self-attention for classifying satellite images time series[END_REF] to handle the temporal dimension. The L-TAE is a simplified multi-head self-attention network [START_REF] Vaswani | Attention is all you need[END_REF] in which the attention masks are directly applied to the input sequence of vectors instead of predicted values. Additionally, the L-TAE implements a channel grouping strategy similar to Group Normalization [START_REF] Wu | Group normalization[END_REF].

We apply a shared L-TAE with G heads independently at each pixel of e L , the feature map sequence at the lowest level resolution L. This generates G temporal attention masks for each pixel, which can be arranged into G tensors a L,g with values in [0, 1] and of shape T × H L × W L :

a L,1 , • • • , a L,G = LTAE(e L ) , applied pixelwise. (2)
In order to use these attention masks at all scale levels l of the encoder, we compute spatially-interpolated masks a l,g of shape T × H l × W l for all l in [1, L -1] and g in [1, G] with bilinear interpolation: a l,g = resize a L,g to H l × W l .

(

) 3
The interpolated masks a l,g at level l of the encoder are then used as if they were generated by a temporal attention module operating at this resolution. We apply the L-TAE channel-grouping strategy at all resolution levels: the channels of each feature map sequence e l are split into G contiguous groups e l,1 , • • • , e l,G of identical shape

T ×C l /G × W l × H l .
For each group g, the feature map sequence e l,g is averaged on the spatial dimension using a l,g as weights. The resulting maps are concatenated along the channel dimension, and processed by a shared 1 × 1 convolution layer Conv l 1×1 of width C l . We denote by f l the resulting map of size C l × W l × H l by :

f l = Conv l 1×1   T t=1 a l,g t e l,g t G g=1   , (4)
with [ • ] the concatenation along the channel dimension and the term-wise multiplication with channel broadcasting.

c) Spatial Decoding. We combine the feature maps f l learned at the previous step with a convolutional decoder to obtain spatio-temporal features at all resolutions. The decoder is composed of L -1 blocks D l for 1 ≤ l < L, with convolutions, ReLu activations, and BatchNorms [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF].

Each decoder block uses a strided transposed convolution D up l to up-sample the previous feature map. The decoder at level l produces a feature map d l of size D l × H l × W l . In a U-Net fashion, the encoder's map at level l is concatenated with the output of the decoder block at level l -1:

d l = D l ([D up l (d l+1 ), f l ]) for l ∈ [1, L -1] , (5)
with

d L = f L and [ • ]
is the channelwise concatenation.

Panoptic Segmentation

Our goal is to use the multi-scale feature maps {d l } L l=1 learnt by the spatio-temporal encoder to perform panoptic segmentation of a sequence of satellite images over an area of interest. The first stage of panoptic segmentation is to produce instance proposals, which are then combined into a single panoptic instance map. Since an entire sequence of images (often over 50) must be encoded to compute {d l } L l=1 , we favor a simple approach for our panoptic segmentation module. Furthermore, given the relative simplicity of parcels' borders, we avoid complex region proposal networks such as Mask-RCNN. Instead, we adapt the single-stage CenterMask instance segmentation network [START_REF] Wang | Centermask: single shot instance segmentation with point representation[END_REF], and detail our modifications in the following paragraphs. We name our approach Parcels-as-Points (PaPs) to highlight our inspiration from CenterNet/Mask [START_REF] Zhou | Objects as points[END_REF][START_REF] Wang | Centermask: single shot instance segmentation with point representation[END_REF]. We denote by P the set of ground truth parcels in the image sequence X. Note that the position of these parcels is time-invariant and hence only defined by their spatial extent. Each parcel p is associated with (i) a centerpoint îp , p with integer coordinates, (ii) a bounding box of size ĥp , ŵp , (iii) a binary instance mask ŝp ∈ {0, 1} H×W , (iv) a class kp ∈ [1, K] with K the total number of classes.

Centerpoint Detection. Following CenterMask, we perform parcel detection by predicting centerness heatmaps supervized by the ground truth parcels' bounding boxes. In the original approach [START_REF] Zhou | Objects as points[END_REF], each class has its own heatmap: detection doubles as classification. This is a sensible choice for natural images, since the tasks of detecting an object's nature, location, and shape are intrinsically related. In our setting however, the parcels' shapes and border characteristics are mostly independent of the cultivated crop. For this reason, we use a single centerness heatmap and postpone class identification to a subsequent specialized module. See Figure 3 for an illustration of our parcel detection method. We associate each parcel p with a Gaussian kernel of deviations σ ver p and σ hor p taken respectively as 1/20 of the height and width of the parcels' bounding box. Unlike Law and Deng [START_REF] Law | Cornernet: Detecting objects as paired keypoints[END_REF], we use heteroschedastic kernels to reflect the potential narrowness of parcels. We then define the target centerness heatmap m ∈ [0, 1] H×W as the maximum value of all parcel kernels at each pixel (i, j) in H × W :

mi,j = max p∈P exp - (i -îp ) 2 2(σ ver p ) 2 + (j -p ) 2 2(σ hor p ) 2 (6)
A convolutional layer takes the highest-resolution feature map d 1 as input and predicts a centerness heatmap m ∈ [0, 1] H×W . The predicted heatmap is supervized using the loss defined in Equation 7 with β = 4:

L center = -1 |P | i=1•••H j=1•••W log(m i,j ) if mi,j = 1 (1-mi,j ) β log(1-m i,j ) else. ( 7
)
We define the predicted centerpoints as the local maxima of m, i.e. pixels with larger values than their 8 adjacent neighbors. This set can be efficiently computed with a single max-pooling operation. Replacing the max operator by argmax in Equation 6 defines a mapping H × W → P between pixels and parcels. During training, we associate each true parcel p with the predicted centerpoint c(p) with highest predicted centerness m among the set of centerpoints which coordinates are mapped to p. If this set is empty, then c(p) is undefined: the parcel p is not detected.

We denote by P the subset of detected parcels, i.e. for which c(p) is well defined.

Size and Class Prediction. We associate to a predicted centerpoint c of coordinate (i c , j c ) the multi-scale feature vector dc of size D 1 + • • • + D L by concatenating channelwise the pixel features at location (i c , j c ) in all maps d l :

dc = d l i c /2 l-1 , j c /2 l-1 L l=1 , (8)
with [ • ] the channelwise concatenation. This vector dc is then processed by four different multilayer perceptrons (MLP) to obtain three vectors of sizes 2, K, and S 2 representing respectively: (i) a bounding box size h c , w c , (ii) a vector of class probabilities k c of size K, and (iii) a shape patch s c of fixed size S × S. The latter is described in the next paragraph. The class prediction k c(p) associated to the true parcel p is supervized with the cross-entropy loss, and the size prediction with a normalized L1 loss. For all p in P , we have:

L p class = -log(k c(p) [ kp ]) (9)
L p size = |h c(p) -ĥp | ĥp + |w c(p) -ŵp | ŵp . ( 10
)
Shape Prediction. The idea of this step is to combine for a predicted centerpoint c a rough shape patch s c with a fullresolution global saliency map z to obtain a pixel-precise instance mask, see Figure 4. For a centerpoint c of coordinates (i c , j c ), the predicted shape patch s c of size S × S is resized to the predicted size h c × w c with bilinear interpolation. A convolutional layer maps the outermost feature map d 1 to a saliency map z of size H × W , which is shared by all predicted parcels. This saliency map is then cropped along the predicted bounding box (i c , j c , h c , w c ). The resized shape and the cropped saliency are added [START_REF] He | Mask R-CNN[END_REF] to obtain a first local shape lc , which is then further refined with a residual convolutional network CNN [START_REF] Ienco | Land cover classification via multitemporal spatial data by deep recurrent neural networks[END_REF]. We denote the resulting predicted shape by l c :

lc = resize c (s c ) + crop c (z) (11)
l c = sigmoid( lc + CNN( lc )) , (12)
with resize c and crop c defined by the coordinates (i c , j c ) and predicted bounding box size ( h c , w c ). The shape and saliency predictions are supervized for each parcel p in P by computing the pixelwise binary cross-entropy (BCE) between the predicted shape l c(p) and the corresponding true binary instance mask ŝp cropped along the predicted bounding box (i c(p) , j c(p) , h c(p) , w c(p) ):

L p shape = BCE(l c(p) , crop c(p) (ŝ p )) . (13)
For inference, we associate a binary mask with a predicted centerpoint c by thresholding l c with the value 0.4.

Loss Function : These four losses are combined into a single loss with no weight and optimized end-to-end:

L = L center + 1 |P | p∈P L p class + L p size + L p shape . ( 14
)
Differences with CenterMask. Our approach differs from CenterMask in several key ways: (i) We compute a single saliency map and heatmap instead of K different ones. This represents the absence of parcel occlusion and the similarity of their shapes. (ii) Accounting for the lower resolution of satellite images, centerpoints are computed at full resolution to detect potentially small parcels, thus dispensing us from predicting offsets. (iii) The class prediction is handled centerpoint-wise instead of pixel-wise for efficiency. (iv) Only the selected centerpoints predict shape, class, and size vectors, saving computation and memory. (v) We use simple feature concatenation to compute multiscale descriptors instead of deep layer aggregation [START_REF] Yu | Deep layer aggregation[END_REF] or stacked Hourglass-Networks [START_REF] Newell | Stacked hourglass networks for human pose estimation[END_REF]. (vi) A convolutional network learns to combine the saliency and the mask instead of a simple term-wise product.

Converting to Panoptic Segmentation Panoptic segmentation consists of associating to each pixel a semantic label and, for non-background pixels (our only stuff class), an instance label [START_REF] Kirillov | Panoptic feature pyramid networks[END_REF]. Our predicted binary instance masks can have overlaps, which we resolve by associating to each predicted parcel a quality measure equal to the predicted centerness m at its associated centerpoint. Masks with higher quality overtake the pixels of overlapping masks with lesser predicted quality. If a mask loses more than 50% of its pixels through this process, it is removed altogether from the predicted instances. Predicted parcels with a quality under a given threshold are dropped. This threshold can be tuned on a validation set to maximize the parcel detection F-score. All pixels not associated with a parcel mask are labelled as background.

Implementation Details. Our implementation of U-TAE allows for batch training on sequences of variable length thanks to a simple padding strategy. The complete configuration and training details can be found in the Appendix. A Pytorch implementation is available at https: //github.com/VSainteuf/utae-paps.

Experiments

The PASTIS Dataset

We present PASTIS (Panoptic Agricultural Satellite TIme Series), the first large-scale, publicly available SITS dataset with both semantic and panoptic annotations. This dataset, as well as more information about its composition, are publicly available at https://github.com/ VSainteuf/pastis-benchmark .

Description. PASTIS is comprised of 2 433 sequences of multi-spectral images of shape 10 × 128 × 128. Each sequence contains between 38 and 61 observations taken between September 2018 and November 2019, for a total of over 2 billion pixels. The time between acquisitions is uneven with a median of 5 days. This lack of regularity is due to the automatic filtering of acquisitions with extensive cloud cover by the satellite data provider THEIA. The 10 channels correspond to the non-atmospheric spectral bands of the Sentinel-2 satellite, after atmospheric correction and re-sampling at a spatial resolution of 10 meters per pixel. The dataset spans over 4000 km 2 , with images taken from four different regions of France with diverse climates and crop distributions, covering almost 1% of the French Metropolitan territory. We estimate that close to 28% of images have at least partial cloud cover.

Annotation. Each pixel of PASTIS is associated with a semantic label taken from a nomenclature of 18 crop types plus a background class. As is common in remote sensing applications, the dataset is highly unbalanced, with a ratio of over 50 between the most and least common classes. Each non-background pixel also has a unique instance label corresponding to its parcel index. In total, 124 422 We represent the results of our method in terms of panoptic segmentation (c) and semantic segmentation (d). The parcels' and pixels' color corresponds to the crop type, according to a legend given in the appendix. The predominantly correct class predictions highlight the fact that the difficulty of panoptic segmentation lies in the precise delineation of each individual parcel. We observe cases where the temporal structure of the SITS was successfully leveraged to resolve boundary ambiguities that could not be seen from a single image (cyan circle ). Conversely, some visually fragmented parcels are annotated as a single instance (red circle ).

parcels are individualized, each with their bounding box, pixel-precise mask, and crop type. All annotations are taken from the publicly available French Land Parcel Identification System. The French Payment Agency estimates the accuracy of the crop annotations via in situ control over 98% and the relative error in terms of surfaces under 0.3%. To allow for cross-validation, the dataset is split into 5 folds, chosen with a 1km buffer between images to avoid crossfold contamination.

Semantic Segmentation

Our U-TAE has L = 4 resolution levels and a LTAE with G = 16 heads, see appendix for an exact configuration. For the semantic segmentation task, the feature map d 1 with highest resolution is set to have K channels, with K the number of classes. We can then interpret d 1 as pixel-wise predictions to be supervized with the cross-entropy loss. In this setting, we do not use the PaPs module.

Competing Methods. We reimplemented six of the topperforming SITS encoders proposed in the literature:

• ConvLSTM [START_REF] Rußwurm | Convolutional LSTMs for cloud-robust segmentation of remote sensing imagery[END_REF][START_REF] Shi | Convolutional LSTM network: A machine learning approach for precipitation nowcasting[END_REF] and ConvGRU [START_REF] Ballas | Delving deeper into convolutional networks for learning video representations[END_REF]. These approaches are recurrent neural networks in which all linear layers are replaced by spatial convolutions.

• U-ConvLSTM [START_REF] Rustowicz | Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods[END_REF] and U-BiConvLSTM [START_REF] Andres | Fully convolutional recurrent networks for multidate crop recognition from multitemporal image sequences[END_REF]. To reproduce these UNet-Based architectures, we replaced the L-TAE in our architecture by either a convLSTM [START_REF] Shi | Convolutional LSTM network: A machine learning approach for precipitation nowcasting[END_REF] or a bidirectional convLSTM. Skip connections are temporally averaged. In contrast to the original methods, we replaced the batch normalization in the encoders with group normalization which significantly improved the results across-the-board. • 3D-Unet [START_REF] Rustowicz | Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods[END_REF]. A U-Net in which the convolutions of the encoding branch are three-dimensional to handle simultaneously the spatial and temporal dimensions.

• FPN-ConvLSTM [START_REF] Andres | Fully convolutional recurrent networks for multidate crop recognition from multitemporal image sequences[END_REF]. This model combines a feature pyramid network [START_REF] Lin | Feature pyramid networks for object detection[END_REF] to extract spatial features and a bidirectional ConvLSTM for the temporal dimension.

Analysis. In Table 1, we detail the performance obtained with 5-fold cross validation of our approach and the six reimplemented baselines. We report the Overall Accuracy (OA) as the ratio between correct and total predictions, and (mIoU) the class-averaged classification IoU. We observe that the convolutional-recurrent methods ConvGRU and ConvLSTM perform worse. Recurrent networks embedded in an U-Net or a FPN share similar performance, with a much longer inference time for FPN. Our approach significantly outperforms all other methods in terms of precision. In Figure 5, we present a qualitative illustration of the semantic segmentation results.

Ablation Study. We first study the impact of using spatially interpolated attention masks to collapse the temporal dimension of the spatio-temporal feature maps at different levels of the encoder simultaneously. Simply computing the temporal average of skip connections for levels without temporal encoding as proposed by [START_REF] Stoian | Land cover maps production with high resolution satellite image time series and convolutional neural networks: Adaptations and limits for operational systems[END_REF][START_REF] Rustowicz | Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods[END_REF], we observe a drop of 4.8 mIoU points (Skip Mean). This puts our method performance on par with its competing approaches. Adding a 1 × 1 convolutional layer after the temporal average reduces this drop to 4.2 points (Skip Mean + Conv). Lastly, using interpolated masks but foregoing the channel grouping strategy by averaging the masks group-wise into a single attention mask per level results in a drop of 3.1 points (Mean Attention). This implies that our network is able to use the grouping scheme at different resolutions simultaneously. In conclusion, the main advantage of our proposed attention scheme is that the temporal collapse is controlled at all resolutions, in contrast to recurrent methods. Using batch normalization in the encoder leads to a severe degradation of the performance of 27.1 points (Batch-Norm). We conclude that the temporal diversity of the acquisitions requires special considerations. This was observed for all U-Net models alike. We also train our model on a single acquisition date (with a classic U-Net and no temporal encoding) for two different cloudless dates in August and May (Single Date). We observe a drop of 24.8 and 42.5 points respectively, highlighting the crucial importance of the temporal dimension for crop classification. We also observed that images with at least partial cloud cover received on average 58% less attention than their cloud-free counterparts. This suggests that our model is able to use the attention module to automatically filter out corrupted data.

Panoptic Segmentation

We use the same U-TAE configuration for panoptic segmentation, and select a PaPs module with 190k parameters and a shape patch size of 16 × 16. In Table 2, we report the class-averaged Segmentation Quality (SQ), Recognition Quality (RQ), and Panoptic Quality (PQ) [START_REF] Kirillov | Panoptic feature pyramid networks[END_REF]. We observe that while the network is able to correctly detect and clas- sify most parcels, the task remains difficult. In particular, the combination of ambiguous borders and hard-to-classify parcel content makes for a challenging panoptic segmentation problem. We illustrate these difficulties in Figure 5, along with qualitative results.

Replacing the temporal encoder by a U-BiConvLSTM as described in Section 4.2 (U-BiConvLSTM+PaPs), we observe a noticeable performance drop of 8.4 RQ, which is consistent with the results of Table 1. As expected, our model's performance is not sensitive to changes in the size S of the shape patch. Indeed, the shape patches only determine the rough outline of parcels while the pixel-precise instance masks are derived from the saliency map. Performing shape prediction with a simple element-wise multiplication as in [START_REF] Wang | Centermask: single shot instance segmentation with point representation[END_REF] (Multiplicative Saliency) instead of our residual CNN results in a drop of over -6.8 SQ. Using a single image (August) leads to a low panoptic quality. Indeed, identifying crop types and parcel borders from a single image at the resolution of Sentinel-2 is particularly difficult.

Inference on 490 sequences takes 129s: 26s to generate U-TAE embeddings, 1s for the heatmap and saliency, 90s for instance proposals, and 12s to merge them into a panoptic segmentation. Note that the training time is also doubled compared to simple semantic segmentation.

Conclusion

We introduced U-TAE, a novel spatio-temporal encoder using a combination of spatial convolution and temporal attention. This model can be easily combined with PaPs, the first panoptic segmentation framework operating on SITS. Lastly, we presented PASTIS, the first large-scale panopticready SITS dataset. Evaluated on this dataset, our approach significantly outperformed all other approaches for semantic segmentation, and set up the first state-of-the-art for panoptic segmentation of satellite image sequences.

We hope that the combination of our open-access dataset and promising results will encourage both remote sensing and computer vision communities to consider the challenging problem of panoptic SITS segmentation, whose economic and environmental stakes can not be understated.

between September 2018 and November 2019, forming our four dimensional multi-spectral SITS: T ×C ×H ×W . The publicly available French Land Parcel Identification System (FLPIS) allows us to retrieve the extent and content of all parcels within the tiles, as reported by the farmers. Each patch pixel is annotated with a semantic label corresponding to either the parcels' crop type or the background class. The pixels of each unique parcel in the patch receive a corresponding instance label.

Dataset Extent. The SITS of PASTIS are taken from 4 different Sentinel-2 tiles in different regions of the French metropolitan territory as depicted in Figure 6a. These regions cover a wide variety of climates and culture distributions. Sentinel tiles span 100 × 100km and have a spatial resolution of 10 meter per pixel. Each pixel is characterized by 13 spectral bands. We select all bands except the atmospheric bands B01, B09, and B10. Each of these tiles is subdivided in square patches of size 1.28 × 1.28km (128 × 128 pixels at 10m/pixel), for a total of around 24, 000 patches. We then select 2, 433 patches ( 10% of all available patches, see Figure 6b), favoring patches with rare crop types in order to decrease the otherwise extreme class imbalance of the dataset.

Nomenclature The FLPIS uses a 73 class breakdown for crop types. We select classes with at least 400 parcels and with samples in at least 2 of the 4 Sentinel-2 tiles. This leads us to adopt a 18 classes nomenclature, presented in Figure 7. Parcels belonging to classes not in our 18-classes nomenclature are annotated with the void label, see below.

Patch Boundaries. The FLPIS allows us to retrieve the pixel-precise borders of each parcel. We also compute bounding boxes for each parcel. The parcels' extents are cropped along the extent of their 128 × 128 patch, and the bounding boxes are modified accordingly. Parcels whose surface is more than 50% outside of the patch are annotated with the void label, see Figure 6c.

Void and Background Labels. Pixels which are not within the extent of any declared parcels are annotated with the background "stuff" label, corresponding to all nonagricultural land uses. For the semantic segmentation task, this label becomes the 20-th class to predict. In the panoptic setting, this label is associated with pixels not within the extent of any predicted parcel. We do not compute the panoptic metrics for the background class, since our focus is on retrieving the parcels' extent rather than an extensive land-cover prediction. In other words, the reported panoptic metrics are the "things" metrics, which already penalize parcels predicted on background pixels by counting them as false positives. The void class is reserved for out-of-scope parcels, either because their crop type is not in our nomenclature or because their overlap with the selected square patch is too small. We remove these parcels from all semantic or panoptic metrics and losses. Predicted parcels which overlap with an IoU superior to 0.5 with a void parcel are not counted as false positive or true positive, but are simply ignored by the metric, as recommended in [START_REF] Kirillov | Panoptic feature pyramid networks[END_REF]. Cross-Validation. The 2, 433 selected patches are randomly subdivided into 5 splits, allowing us to perform cross-validation. The official 5-fold cross-validation scheme used for benchmarking is given in Table 3. In order to avoid heterogeneous folds, each fold is constituted of patches taken from all four Sentinel tiles. We also chose folds with comparable class distributions, as measured by their pairwise Kullback-Leiber divergence. We show the resulting class distribution for each fold in Figure 8. Finally, we prevent adjacent patches from being in different folds to avoid data contamination. Geo-referencing metadata of the patches and parcels is included in PASTIS, allowing for the constitution of geographically consistent folds to evaluate spatial generalization. However, this is out of the scope of this paper.

Fold Train Val Test I 1-2-3 4 5 II 2-3-4 5 1 III 3-4-5 1 2 IV 4-5-1 2 3 V 5-1-2 3 4
Table 3: Official 5-fold cross validation scheme. Each line gives the repartition of the splits into train, validation and test set for each fold.

Temporal Sampling. The temporal sampling of the sequences in PASTIS is irregular: depending on their location, patches are observed a different number of times and at different intervals. This is a result of both the orbit schedule of Sentinel-2 and the policy of Sentinel data providers not to process tile observations identified as covered by clouds for more than 90% of the tile's surface. As this corresponds to the real world setting, we decided to leave the SITS as is, and thus to encourage methods that can favourably address this technical challenge. As a result, the proposed SITS are constituted of 33 to 61 acquisitions. In order to assess how our model handles lower sampling frequencies, we limited the number of available acquisitions at inference time 2 , and observed a drop of performance of -0.7, -2.0, -5.5, and -14.6 points of mIoU with 32, 24, 16, and 8 available dates, respectively.

Clouds Cover. Even after the automatic filtering of predominantly cloudy acquisitions, some patches are still partially or completely obstructed by cloud cover. We opt to not apply further pre-processing or cloud detection, and produce the raw data in PASTIS. Our reasoning is that an adequate algorithm should be able to learn to deal with such acquisitions. Indeed, robustness to cloud-cover has been experimentally demonstrated for deep learning methods by Rußwurm and Körner [START_REF] Rußwurm | Convolutional LSTMs for cloud-robust segmentation of remote sensing imagery[END_REF][START_REF] Rußwurm | Self-attention for raw optical satellite time series classification[END_REF].

A.2. Implementation Details

In this section, we detail the exact configuration of our method as well as the competing algorithms evaluated.

Training Details. Across our experiments, we use Adam [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] optimizer with default parameters and a batch size of 4 sequences. The semantic segmentation experiments use a fixed learning rate of 0.001 for 100 epochs. For the panoptic segmentation experiments, we start with a higher learning rate of 0.01 for 50 epochs, and decrease it to 0.001 for the last 50 epochs.

U-TAE.

In Table 4, we report the width of the feature maps outputted by each level of the U-TAE's encoder and decoder. In both networks, we use the the same convolutional block shown in Figure 9 and constituted of one 3 × 3 convolution from the input to the output's width, and one residual 3 × 3 convolution. In the encoding branch, we use Group Normalisation with 4 groups and Batch Normalisation in the decoding branch .

For the temporal encoding, we chose a L-TAE with 16 heads, and a key-query space of dimension d k = 4. We use Group Normalisation with 16 groups at the input and output of the L-TAE, meaning that that the inputs of each head are layer-normalized.

Conv 3x3

Norm ReLu

D in D out D out

Conv 3x3

Norm ReLu Recurrent Models. We use the same U-Net architecture for our models and U-BiConvLSTM and U-ConvLSTM, but simply replace the L-TAE by a ConvLSTM or BiConvL-STM respectively. The hidden state's size of the biCon-vLSTM is chosen as 32 in both directions, and 64 for the convLSTM. For the recurrent-convolutional methods Con-vLSTM and ConvGRU not using a U-Net, we set hidden sizes of 160 and 188 respectively.

3D-Unet. For this network, we use the official PyTorch implementation of Rustowicz et al. [START_REF] Rustowicz | Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods[END_REF]. This network is constituted of five successive 3D-convolution blocks with spatial down-sampling after the 2nd and 4th blocks. Each convolutional block doubles the number of channels of the processed feature maps, and the innermost feature maps have a channel dimension set to 128. Leaky ReLu and 3D Batch Normalisation are used across the convolutional blocks of this architecture. The sequence of feature maps is averaged along the temporal dimension to produce the final embedding of the input image sequence. In their implementation, the authors used a linear layer to collapse the temporal dimension, yet this was not a valid option for PASTIS as the sequences have highly variable lengths and the sequence indices do not correspond to the same acquisition date from one sequence to another.

FPN-ConvLSTM. For this architecture, the input sequence of images is first mapped to feature maps of channel dimension 64 with two consecutive 3 × 3 convolution layers, followed by Group Normalization and ReLu. A 5-level feature pyramid is then constructed for each date of the sequence by applying to the feature maps 4 different 3 × 3 convolution of respective dilation rates 1, 2, 4 and 8, and computing the spatial average of the feature map. These 5 maps are concatenated along the channel dimension, and processed by a ConvLSTM with a hidden state size of 88.

We found it beneficial to use a supplementary convolution before the ConvLSTM to reduce the number of channels of the feature pyramid by a factor 2.

PaPs module. In the PaPs module, the saliency and heatmap predictions are obtained with two separate convolutional blocks operating on the high resolution feature map d 1 with 32 channels. These blocks are composed of two convolutional layers of width 32 and 1 respectively. We use Batch Normalisation and ReLu after the first convolution, and a sigmoid after the second. The 256-dimensional multi-scale feature vector (128 + 64 + 32 + 32) is mapped to the shape, class and size predictions with three different MLPs described in Table 5. The inner layers use Batch Normalisation and ReLu activation.

The residual CNN used for shape refinement is composed of three convolutional layers : 1 → 16 → 16 → 1, with ReLu activation and instance normalisation on the first layer only.

Handling Sequences of Variable Lengths. All models are trained with batches of sequences of variable lengths. To facilitate the handling of batches by the GPU, we append all-zeroes images at the end of shorter sequences to match the length of the longer sequence in the batch. We retain a padding mask to prevent the spatial and temporal encoding of padded values, and to exclude these padded values from temporal averages.

A.3. Additional Results

In Figure 10, we show the class-wise performance of the three best performing semantic segmentation models, displaying an improvement of U-TAE compared to the other methods across all crop types. We also show on Figure 11 the confusion matrix of U-TAE. Unsurprisingly, confusions seem to occur between semantically close classes such as different cereal types, or Sunflower and Fruits, Vegetable, Flower.

In Figure 12, we present qualitative results illustrating the predicted panoptic and semantic segmentations compared to the ground truth. In particular, we show some failure cases in which thin or visually fragmented parcels are not recovered correctly.

In Figure 13, we illustrate the results of the semantic segmentation for our method and three other competing approaches: 3D-Unet, U-BiConvLSTM, and convGRU. We show how our multi-scale temporal attention masks allow our predictions to be both pixel-precise and consistent for large parcels.

Finally, we present in Figure 14 an example of inference using a single image from the sequence. As expected for mono-temporal segmentation, the parcel classification is poor. Furthermore, we show a case of border that is essentially invisible on a single image, but that our full model is able to detect using the entire sequence of satellite images. We represent a single image from the sequence using the RGB channels (a), and whose ground truth parcel's limit and types are known (b). We then represent the parcels predicted by our panoptic segmentation module (c), and the pixelwise prediction of our semantic segmentation module (d). See Figure 7 for the color to crop type correspondence. We highlight with a green circle a large, fragmented parcel declared as one single field. This leads to predictions with low confidence and a low panoptic quality. Conversely, the cyan circle highlights such fragmented parcel which is correctly predicted as a single instance. This suggests that our network is able to use the temporal dynamics to recover ambiguous borders. We highlight a failure case with the red circle , for which many thin parcels are not properly detected, resulting in a low panoptic quality. We observe that the semantic segmentation model struggles as well for such thin parcels. Finally, we highlight with a blue circle an example in which the panoptic prediction is superior to the semantic segmentation, indicating that detecting parcels' boundaries and extent can be informative for their classification. Figure 13: Qualitative Semantic Segmentation Results. We represent a single image from the sequence using the RGB channels (a), and whose ground truth parcel's limit and crop type are known (b). We then represent the pixelwise prediction from our approach (c), and for three other competing algorithms (d-f). The different predictions shown on this figure illustrate the importance of the resolution at which temporal encoding is performed. ConvGRU applies a recurrent-convolutional network at the highest resolution, which results in predictions with high spatial variability. As a consequence, the prediction over large parcels are inconsistent (blue circles ). Conversely, U-BiConvLSTM applies temporal encoding to feature maps with a larger receptive field, resulting in more spatially consistent predictions. Yet, this architecture often fails to retrieve small or thin parcels. In contrast, our U-TAE produces spatially consistent predictions on large parcels, while being able to retrieve such small parcels (green circles ). 3D-Unet also uses temporal encoding at different resolution levels, yet fails to recover these small parcels. First, we observe that many parcels are not detected by the mono-temporal model, indicating an overall low predicted quality. Second, we can see that most detected parcels are misclassified by the mono-temporal model. This is in accordance with the low semantic segmentation score of the mono-temporal model: crop types are hard to distinguish from a single observation. Last, adjacent parcels with no clear borders are predicted as a single parcel, when the multi-temporal model is able to differentiate between the two parcels (cyan circle ). This illustrates how using SITS instead of single images can help resolve ambiguous parcels delineation.

Figure 2 :

 2 Figure 2: Spatio-temporal Encoding.A sequence of images is processed in parallel by a shared convolutional encoder. At the lowest resolution, an attention-based temporal encoder produces a set of temporal attention masks for each pixel, which are then spatially interpolated at all resolutions. These masks are used to collapse the temporal dimension of the feature map sequences into a single map per resolution. A convolutional decoder then computes features at all resolution levels. All convolutions operate purely on the spatial and channel dimensions, and we use strided convolutions for both spatial up and down-sampling. The feature maps are projected in RGB space to help visual interpretation.

Figure 3 :

 3 Figure 3: Centerpoint Detection. The ground truth instance masks (a) is used to construct a target heatmap (b). Our parcel detection module maps the raw sequence of observation (c) to a predicted heatmap (d). The predicted centerpoints (red crosses) are the local maxima of the predicted heatmap (d). The black dots are the true parcels centers.

Figure 4 :

 4 Figure 4: Panoptic Segmentation.The local maxima of the predicted centerness heatmap defines M tentative parcels. For each one, the pixel features at all levels are concatenated and used to predict a bounding box size, a semantic class, and an S × S shape patch. The latter is combined with a global saliency map for predicting pixel-precise masks. The instance predictions are combined into a panoptic segmentation using the centerness as quality.

  (a) Image from the sequence. (b) Panoptic annotation. (c) Panoptic segmentation. (d) Semantic segmentation.

Figure 5 :

 5 Figure 5: Qualitative results.We consider an image sequence (a) with panoptic annotations (b). We represent the results of our method in terms of panoptic segmentation (c) and semantic segmentation (d). The parcels' and pixels' color corresponds to the crop type, according to a legend given in the appendix. The predominantly correct class predictions highlight the fact that the difficulty of panoptic segmentation lies in the precise delineation of each individual parcel. We observe cases where the temporal structure of the SITS was successfully leveraged to resolve boundary ambiguities that could not be seen from a single image (cyan circle ). Conversely, some visually fragmented parcels are annotated as a single instance (red circle ).

Figure 7 :

 7 Figure 7: Color code of our class nomenclature, and the number of parcel per class.

M e a d o w C o r n G r a p e v i n e S o f t w i n t e r w h e a t F r u i t s , v e g e t a b l e s , f l o w e r s O r c h a r d L e g u m i n o u s f o d d e r W i n t e r b a r l e y W i n t e r d u r u m w h e a t W i n t e r r a p e s e e d S o y b e a n s S u n f l o w e r W i n t e r t r i t i c a l e M i x e d c e r e a l S p r i n g b a r l e

 t

Figure 8 :

 8 Figure 8: Class distribution for the five folds (in log-scale).

Figure 9 :

 9 Figure 9: Structure of the convolutional block used in the spatial encoder-decoder network. This block maps a feature map with D in channels to a feature map with D out channels.

WFigure 10 :

 10 Figure 10: Per class IoU of the three best performing semantic segmentation models. Our U-TAE outperforms the other two approaches on every classes, and brings noticeable improvement on hard classes such as Mixed cereal and Sorghum.

Figure 11 :

 11 Figure 11: Confusion matrix of U-TAE for semantic segmentation on PASTIS. The color of each pixel at line i and column j corresponds to the proportion of samples of the class i that were attributed to the class j.

( a )

 a Image from the sequence. (b) Panoptic annotation. (c) Panoptic segmentation. (d) Semantic segmentation.

Figure 12 :

 12 Figure 12: Qualitative Panoptic Segmentation Results.We represent a single image from the sequence using the RGB channels (a), and whose ground truth parcel's limit and types are known (b). We then represent the parcels predicted by our panoptic segmentation module (c), and the pixelwise prediction of our semantic segmentation module (d). See Figure7for the color to crop type correspondence. We highlight with a green circle a large, fragmented parcel declared as one single field. This leads to predictions with low confidence and a low panoptic quality. Conversely, the cyan circle highlights such fragmented parcel which is correctly predicted as a single instance. This suggests that our network is able to use the temporal dynamics to recover ambiguous borders. We highlight a failure case with the red circle , for which many thin parcels are not properly detected, resulting in a low panoptic quality. We observe that the semantic segmentation model struggles as well for such thin parcels. Finally, we highlight with a blue circle an example in which the panoptic prediction is superior to the semantic segmentation, indicating that detecting parcels' boundaries and extent can be informative for their classification.

  (a) Single image. (b) Annotation. (c) U-TAE. (d) 3D-Unet. (e) U-BiConvLSTM. (f) ConvGRU.

  (a) Single observation. (b) Panoptic annotation. (c) Mono-temporal prediction. (d) Multi-temporal prediction.

Figure 14 :

 14 Figure 14: Mono-temporal Panoptic Segmentation. We train our mono-temporal model on a single image (a), with panoptic annotation (b).We then compare the results of the mono-temporal model in (c) with the results our full model when performing inference on the full length sequence (d) from which the single patch (a) is drawn. First, we observe that many parcels are not detected by the mono-temporal model, indicating an overall low predicted quality. Second, we can see that most detected parcels are misclassified by the mono-temporal model. This is in accordance with the low semantic segmentation score of the mono-temporal model: crop types are hard to distinguish from a single observation. Last, adjacent parcels with no clear borders are predicted as a single parcel, when the multi-temporal model is able to differentiate between the two parcels (cyan circle ). This illustrates how using SITS instead of single images can help resolve ambiguous parcels delineation.

Table 1 :

 1 Semantic Segmentation. We report for our method and six competing methods the model size in train-

	Model	# param ×1000	OA mIoU IT (s)
	U-TAE (ours)	1 087 83.2	63.1	25.7
	3D-Unet [37]	1 554 81.3	58.4	29.5
	U-ConvLSTM [37]	1 508 82.1	57.8	28.3
	FPN-ConvLSTM [23]	1 261 81.6	57.1 103.6
	U-BiConvLSTM [23]	1 434 81.8	55.9	32.7
	ConvGRU [4]	1 040 79.8	54.2	49.0
	ConvLSTM [35, 40]	1 010 77.9	49.1	49.1
	Mean Attention	1 087 82.8	60.1	24.8
	Skip Mean + Conv	1 087 82.4	58.9	24.5
	Skip Mean	1 074 82.0	58.3	24.5
	BatchNorm	1 087 71.9	36.0	22.3
	Single Date (August)	1 004 65.6	28.3	1.3
	Single Date (May)	1 004 58.1	20.6	1.3

able parameters, Overall Accuracy (OA), mean Intersection over Union (mIoU), and Inference Time for one fold of ∼ 490 sequences (IT). The second part of the table report results from our ablation study.

Table 2 :

 2 Panoptic Segmentation Experiment. We report class-averaged panoptic metrics: SQ, RQ, PQ.

	SQ	RQ	PQ

Label and Color Class Name Number of parcels

	0	Background	-
	1	Meadow	31292
	2	Soft winter wheat	8206
	3	Corn	13123
	4	Winter barley	2766
	5	Winter rapeseed	1769
	6	Spring barley	908
	7	Sunflower	1355
	8	Grapevine	10640
	9	Beet	871
	10	Winter triticale	1208
	11	Winter durum wheat	1704
	12	Fruits, vegetables, flowers	2619
	13	Potatoes	551
	14	Leguminous fodder	3174
	15	Soybeans	1212
	16	Orchard	2998
	17	Mixed cereal	848
	18	Sorghum	707
	19	Void label	35924

Table 4 :

 4 Width of the feature maps outputted at each level of the encoding and decoding branches of the spatial module.

	Encoder	Decoder
	e 1 64	d 1 32
	e 2 64	d 2 32
	e 3 64	d 3 64
	e 4 128	d 4 128

Table 5 :

 5 Configuration of the four MLPs of PaPs

	MLP	Layers	Final Layer
	Shape 256 → 128 → S 2	-
	Size	256 → 128 → 2	Softplus
	Class 256 → 128 → 64 → K	Softmax

https://scihub.copernicus.eu

This can be interpreted as the test set having an increased cloud cover.

Acknowledgments

The satellite images used in PASTIS were gathered from THEIA: "Value-added data processed by the CNES for the Theia data cluster using Copernicus data. The treatments use algorithms developed by Theia's Scientific Expertise Centres." The annotations of PASTIS were taken from the French LPIS produced by IGN, the French mapping agency. This work was partly supported by ASP, the French Payment Agency.

Supplementary Material

In this appendix, we provide additional information on the PASTIS dataset and our exact model configuration. We also provide complementary qualitative experimental results. Overview. The PASTIS dataset is composed of 2433 square 128 × 128 patches with 10 spectral bands and at 10m resolution, obtained from the open-access Sentinel-2 platform. 1 For each patch, we stack all available acquisitions

A.1. PASTIS Dataset