Bruno Sauvalle Arnaud

Autoencoder-based background reconstruction and foreground segmentation with background noise estimation

Even after decades of research, dynamic scene background reconstruction and foreground object segmentation are still considered as open problems due to various challenges such as illumination changes, camera movements, or background noise caused by air turbulence or moving trees. We propose in this paper to model the background of a frame sequence as a low dimensional manifold using an autoencoder and compare the reconstructed background provided by this autoencoder with the original image to compute the foreground/background segmentation masks. The main novelty of the proposed model is that the autoencoder is also trained to predict the background noise, which allows to compute for each frame a pixel-dependent threshold to perform the foreground segmentation. Although the proposed model does not use any temporal or motion information, it exceeds the state of the art for unsupervised background subtraction on the CDnet 2014 and LASIESTA datasets, with a significant improvement on videos where the camera is moving. It is also able to perform background reconstruction on some non-video image datasets.

Introduction

We consider in this paper the tasks of dynamic background reconstruction and foreground/background segmentation, which can be described in the following way: The input is a sequence X of consecutive frames X 1 , .., X N showing a scene cluttered by various moving objects, such as cars or pedestrians, and the expected output is a sequence X = X1 , .., XN of frames showing the backgrounds of each scene without those objects.

The foreground/background segmentation task similarly takes as input the same kind of frames sequence X 1 , .., X N , but the expected output is a sequence M of foreground masks M 1 , .., M N whose values at the pixel p are equal to zero if this pixel shows the background in the considered frame, and equal to 1 if the background is masked by a foreground moving object at this pixel (Fig. 1). This task is often called background subtraction because the pointwise multiplication of the mask M k and the input image X k gives an image showing only the foreground moving objects present in X k , the input image background being replaced by a black background.

Background substraction is a fundamental tool in image analysis and has been studied for more than 30 years [START_REF] Richard Wren | Pfinder: Real-Time Tracking of the Human Body[END_REF], but is still considered an open problem due to the various challenges appearing in real applications: illumination changes, high level of occlusion of the background, background motions caused by moving trees or water, challenging weather conditions, presence of shadows, etc. The applications of background subtraction are very diverse [START_REF] Garcia-Garcia | Background subtraction in real applications: Challenges, current models and future directions[END_REF]: road, airport, store, maritime or military surveillance, observation of animals and insects, motion capture, humancomputer interface, video matting, fire detection, etc.

The main application of background reconstruction is background subtraction, but other applications such as holefilling in videos [START_REF] Luo | A Hole Filling Approach Based on Background Reconstruction for View Synthesis in 3D Video[END_REF] have also been implemented. Efficient background reconstruction models are also necessary for unsupervised object detection and tracking [START_REF] Jiang | SCALOR: Generative World Models with Scalable Object Representations[END_REF][START_REF] Henderson | Unsupervised object-centric video generation and decomposi-tion in 3D. Advances in Neural Information Processing Systems[END_REF][START_REF] Wu | APEX: Unsupervised, Object-Centric Scene Segmentation and Tracking for Robot Manipulation[END_REF].

The model presented in this paper starts from the classi-cal assumption that the dynamic background of a scene can be modeled as a low dimensional manifold and uses an autoencoder to learn this manifold and perform dynamic background reconstruction. It then compares the input frame with the associated background predicted by the autoencoder to build the foreground segmentation mask. The main contributions of this paper are the following :

• We implement a more robust loss function to train the autoencoder, which gives a high weight to reconstruction errors associated to background pixels and a low weight to reconstruction errors associated to foreground pixels, and shows better performance than the L 1 loss usually considered for this task.

• We train the autoencoder to provide a background reconstruction, but also a background noise estimation, which gives a pixelwise estimate of the uncertainty of the background prediction. This noise estimation map is used to adjust the threshold necessary to compute the background/foreground segmentation mask.

• We reduce the risk of overfitting by developping a method for detecting significant background changes and implementing an early stopping criterion using this method if the video shows a fixed background.

The paper is structured as follows: We first review related work in section 2, then describe the proposed model in section 3. Experimental results are then provided in section 4.

Related work

Background subtraction methods can be split between supervised methods, which require labeled data, and unsupervised methods.

Supervised methods require labeled data as input, which are sets of pairs (X k , M k ), where the image X k is an image extracted from the sequence X 1 , .., X N and the foreground mask M k has to be provided by a human intervention. Supervised algorithms using linear methods such as as maximum margin criterion [START_REF] Li | Efficient and robust feature extraction by maximum margin criterion[END_REF][START_REF] Diana | Background modeling via a supervised subspace learning To cite this version[END_REF] or graph signal reconstruction methods [START_REF] Jhony | Graph-BGS: Background subtraction via recovery of graph signals[END_REF] have been proposed, but the current best performing supervised models use deep learning techniques with convolutional encoder-decoder structures [START_REF] Long | Foreground segmentation using convolutional neural networks for multiscale feature encoding[END_REF][START_REF] Long | Learning multi-scale features for foreground segmentation[END_REF][START_REF] Mandal | Scene Independency Matters: An Empirical Study of Scene Dependent and Scene Independent Evaluation for CNN-Based Change Detection[END_REF], U-net structures [START_REF] Gani Rahmon | Motion U-Net: Multi-cue encoder-decoder network for motion segmentation[END_REF][START_REF] Mondéjar-Guerra | An end-to-end deep learning approach for simultaneous background modeling and subtraction[END_REF] or GANs [START_REF] Sultana | Unsupervised deep context prediction for background estimation and foreground segmentation[END_REF][START_REF] Zheng | A novel background subtraction algorithm based on parallel vision and Bayesian GANs[END_REF].

A spatio-temporal data augmentation strategy has been proposed [START_REF] Ozan Tezcan | BSUV-Net 2.0: Spatio-Temporal Data Augmentations for Video-Agnostic Supervised Background Subtraction[END_REF] to improve generalization. One can also use as additional input to the deep learning model the output of an unsupervised background subtraction model [START_REF] Gani Rahmon | Motion U-Net: Multi-cue encoder-decoder network for motion segmentation[END_REF][START_REF] Pardàs | Refinement network for unsupervised on the scene foreground segmentation[END_REF]. A background subtraction model can be substantially improved by combining its results with the output of a supervised semantic segmentation model [START_REF] Braham | Semantic background subtraction[END_REF][START_REF] Zeng | Background Subtraction with Real-Time Semantic Segmentation[END_REF]. Although supervised models can reach very high accuracy results on a given video after labeling a significant number of frames of this video and training the model with these labeled data, their ability to generalize to new videos remain a major issue, and evaluations on unseen scenes lead to unfavorable results compared to unsupervised algorithms [START_REF] Mandal | Scene Independency Matters: An Empirical Study of Scene Dependent and Scene Independent Evaluation for CNN-Based Change Detection[END_REF]. As a consequence, existing supervised models are not suited for real world applications where it is not possible to provide annotated data for each new input video.

One can classify unsupervised methods as statistical methods or reconstruction methods.

Statistical methods rely on a statistical modeling of the distribution of background pixel color values or other local features to predict whether a particular pixel is foreground or background. These statistical models can be parametric (univariate gaussian [START_REF] Richard Wren | Pfinder: Real-Time Tracking of the Human Body[END_REF], mixture of gaussians [START_REF] Stauffer | Adaptive background mixture models for real-time tracking[END_REF], clusters [START_REF] Link | Real-Time Adaptive Foreground / Background Segmentation[END_REF], Student's t-distributions [START_REF] Mukherjee | Realtime video segmentation using student's t mixture model[END_REF], Dirichlet process mixture models [START_REF] Himani | Background Subtraction with Dirichlet process Gaussian Mixture Model ( DP-GMM ) for Motion Detection[END_REF], Poisson mixture models [START_REF] Faro | Adaptive background modeling integrated with luminosity sensors and occlusion processing for reliable vehicle detection[END_REF], asymmetric generalized gaussian mixture models [START_REF] Elguebaly | Finite asymmetric generalized Gaussian mixture models learning for infrared object detection[END_REF], etc.) or non parametric (pixel value histograms [START_REF] Zhang | Dynamic Background Subtraction Based on Local Dependency Histogram To cite this version[END_REF], kernel density estimation [START_REF] Elgammal | Non-parametric model for background subtraction[END_REF], codebooks [START_REF] Kim | Background modeling and subtraction by codebook construction[END_REF], history of recently observed pixels [START_REF] Barnich | ViBE: A powerful random technique to estimate the background in video sequences[END_REF][START_REF] Hofmann | Background segmentation with feedback: The pixel-based adaptive segmenter[END_REF], etc.). The efficiency of these methods can be increased by using as input not only the pixel color values, but also features attached to superpixels [START_REF] Tzer | SuperBE: computationally light background estimation with superpixels[END_REF] or local descriptors which are robust to illumination changes, such as SIFT [START_REF] Arnold | Efficient projection pursuit density estimation for background subtraction 3 . Projection pursuit density estima-tion[END_REF], LBP or LBSP descriptors [START_REF] Luc St-Charles | SuBSENSE: A universal change detection method with local adaptive sensitivity[END_REF][START_REF] Luc St-Charles | Universal Background Subtraction Using Word Consensus Models[END_REF]. If the camera is static, the segmentation of moving objects on a scene can also be performed by evaluating the motion associated to each pixel, using optical flow or flux tensor models. The blobs produced by these models are generally very fuzzy, but can be used as input to more complex models [START_REF] Bunyak | Flux tensor constrained geodesic active contours with sensor fusion for persistent object tracking[END_REF][START_REF] Wang | Static and moving object detection using flux tensor with split gaussian models[END_REF].

Reconstruction methods use a background reconstruction model to predict the color (or other features) of the background at a particular pixel. The difference between the current image and the predicted background is then computed and followed by a thresholding to decide whether a pixel is background or foreground. Pixelwise reconstruction models try to predict the value of a background pixel at a particular frame from the sequence of values of the pixel of the last frames using a filter, which can be a Wiener filter [START_REF] Toyama | Wallflower: Principles and practice of background maintenance[END_REF], a Kalman filter [START_REF] Ridder | Adaptive Background Estimation and Foreground Detection using Kalman Filtering[END_REF] or a Chebychev filter [START_REF] Chang | Vision modules for a multi-sensory bridge monitoring approach[END_REF]. A global prediction of the background can also be performed using the assumption that the background frames form a low dimensional manifold, which motivates the use of dimensionality reduction techniques such as principal component analysis (PCA) [START_REF] Nuria | A Bayesian Computer Vision System for Modeling Human Interactions[END_REF]. One can add to this approach a prior on the sparcity of the foreground objects by using a L 1 loss term applied to the foreground residuals, which leads to the development of models based on robust principal component analysis (RPCA) [START_REF] Wright | Robust principal component analysis: Exact recovery of corrupted low-rank matrices by convex optimization[END_REF][START_REF] Candès | Robust principal component analysis? Journal of the ACM[END_REF]. More complex norms and additional regularizers have been proposed to improve the performance of this approach [START_REF] Mairal | Network flow algorithms for structured sparsity[END_REF][START_REF] Liu | Background subtraction based on low-rank and structured sparse decomposition[END_REF][START_REF] Xin | Background Subtraction via generalized fused lasso foreground modeling[END_REF][START_REF] Javed | Background-Foreground Modeling Based on Spatiotemporal Sparse Subspace Clustering[END_REF][START_REF] Javed | Moving Object Detection in Complex Scene Using Spatiotemporal Structured-Sparse RPCA[END_REF]. Nonlinear dimensionality reduction using an autoencoder for background reconstruction has been proposed in [START_REF] Farnoosh | Deeppbm: Deep probabilistic background model estimation from video sequences[END_REF][START_REF] Rezaei | G-LBM: Generative Low-Dimensional Background Model Estimation from Video Sequences[END_REF] and is further developed in the proposed model. Several unsupervised models can be also combined to form a more accurate model, such as the IUTIS-5 models, which is an ensemble model combining 5 different unsupervised models [START_REF] Bianco | How far can you get by combining change detection algorithms? Lecture Notes in Computer Science[END_REF].

Background noise estimation Explicit background noise estimation for foreground segmentation has been introduced in [START_REF] Hofmann | Background segmentation with feedback: The pixel-based adaptive segmenter[END_REF]. Estimating the prediction uncertainty of a deep learning model is usually implemented using a negative log-likelihood loss function associated to a probabilistic model which includes a variance or concentration parameter [START_REF] Nix | Estimating the mean and variance of the target probability distribution[END_REF][START_REF] Kendall | What uncertainties do we need in Bayesian deep learning for computer vision[END_REF][START_REF] Bae | Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation[END_REF][START_REF] Moreau | Co-ordiNet: Uncertainty-aware pose regressor for reliable vehicle localization[END_REF][START_REF] Seitzer | On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks[END_REF].

Model description

The proposed model is a reconstruction model and has a general structure similar to the DeepPBM model [START_REF] Farnoosh | Deeppbm: Deep probabilistic background model estimation from video sequences[END_REF]: We assume that the background frames form a low dimensional manifold and train an autoencoder to learn this manifold from the complete video. We however observe that the DeepPBM model described in [START_REF] Farnoosh | Deeppbm: Deep probabilistic background model estimation from video sequences[END_REF] is not really unsupervised since it requires a significant engineering and optimization work for each new video, which is incompatible with any real-world application: The structure of the autoencoder and the number of latent variables have to be defined and fine-tuned on a scene by scene basis, which can be considered as a form of supervision. One also remarks that if the number of latent variables is too high, the autoencoder quiclky learns to reproduce the foreground objects, a phenomenon we call ovefitting, and fails to generate a proper background.

The model proposed in this paper is fully unsupervised: It uses a constant set of hyperparameter, and the structure of the autoencoder, which depends on the size of the image and on the complexity of the background, is defined automatically without human supervision.

Reconstruction loss using background bootstrapping

We implement a reconstruction loss using background bootstrapping, adapted from [START_REF] Sauvalle | Fast and Accurate Background Reconstruction Using Background Bootstrapping[END_REF]. In the case of dynamic background reconstruction, this loss function allows to reduce the risk of overfitting to the foreground objects by giving a higher weight to background pixels than to foreground pixels during the optimization process. This loss is more robust to outliers than the L 1 loss which gives the same weight to small and large errors. The proposed reconstruction loss can be described by the following formulae [START_REF] Sauvalle | Fast and Accurate Background Reconstruction Using Background Bootstrapping[END_REF]: We note x n,c,i,j the pixel color value of the image X n for the channel c at the position (i, j) with 1 ≤ c ≤ 3,1 ≤ i ≤ h and 1 ≤ j ≤ w, and xn,c,i,j the pixel value of the reconstructed background Xn for the same channel and position. The lo-cal L 1 error associated to the pixel (i, j) is

l n,i,j = 3 c=1 |x n,c,i,j -x n,c,i,j |. (1)
The soft foreground masks and spatially smoothed soft foreground masks are defined by the equations

m n,i,j = tanh l n,i,j τ 1 (2)
and

mn,i,j ( Xn , X n ) = 1 (2k + 1) 2 l=k,p=k l=-k,p=-k m n,i+l,j+p , (3)
where τ 1 and r are positive hyperparameters and k = ⌊w/r⌋. The associated pixel-wise weight w bootstrap n,i,j is then defined as

w bootstrap n,i,j = e -β mn,i,j , (4)
where β is another positive hyperparameter. The reconstruction loss of the auto-encoder is then computed by weighting the pixelwise L 1 losses l n,i,j using these bootstrap weights:

L rec ( X , X ) = 1 N hw N,h,w n=1,i=1,j=1 w bootstrap n,i,j
l n,i,j [START_REF] Bianco | How far can you get by combining change detection algorithms? Lecture Notes in Computer Science[END_REF] The main differences between this loss function and the loss function defined in [START_REF] Sauvalle | Fast and Accurate Background Reconstruction Using Background Bootstrapping[END_REF] is that it is a one-to-one loss, whereas the loss defined in [START_REF] Sauvalle | Fast and Accurate Background Reconstruction Using Background Bootstrapping[END_REF] is one-to-many. It also does not use optical flow weights or abnormal image weights. Using optical flow weights would not allow to handle images taken from a moving camera, since it would give a low weight to all pixels associated to the moving background. We do not use abnormal image weights because we want the model to accurately reconstruct the background for each input image, which was not the case in [START_REF] Sauvalle | Fast and Accurate Background Reconstruction Using Background Bootstrapping[END_REF], which is dedicated to fixed background reconstruction.

Optimized thresholding using background noise estimation

We remark that the bootstrap pixel weights w bootstrap n,i,j can be used to get an estimate of the level of background noise of a frame sequence, considering that these weights are close to one when the associated pixel is a background pixel, and close to zero when this is not the case.

We therefore add a fourth output channel to the autoencoder, which is dedicated to give an estimate ln,i,j of the value of the L 1 error l n,i,j for each pixel (i, j) for the frame X n (Fig. 2). The associated loss function is weighted using the bootstrap weights in order to limit its scope to background regions:

L noise = 1 3N hw N,h,w n=1i=1,j=1 w bootstrap n,i,j | ln,i,j -l n,i,j | (6)
When the background is very noisy, the autoencoder is not able to predict accurately the value of a background pixel color. As a consequence, the expectation of l n,i,j is large, which leads to a high value of ln,i,j . One could consider that a more principled method would be to model the background noise as a gaussian distribution and estimate the variance of this distribution by learning the weighted average L 2 error instead of the L 1 error, but we have empirically found that such an approach is not robust to the presence of foreground objects.

The autoencoder is trained using the sum of the reconstruction loss and the loss associated to the background noise estimation. The complete loss function is then

L = L rec + L noise . ( 7
)
The gradients of the weights w bootstrap n,i,j

are not computed during the optimization process [START_REF] Sauvalle | Fast and Accurate Background Reconstruction Using Background Bootstrapping[END_REF]. We also do not use the gradient of l n,i,j in equation 6 because we do not want the quality of the background reconstruction be impacted by the background noise estimation optimization process.

In order to set the pixelwise threshold τ n,i,j associated to the pixel (i, j) of the frame X n and necessary to compute the background/foreground segmentation mask, we also take into account the average illumination În of the reconstructed background Xn , as defined by the formula

În = 1 3hw 3,h,w c=1,i=1,j=1 |x n,c,i,j |. (8)
The threshold τ n,i,j is then set according to the formula

τ n,i,j = α 1 În + α 2 ln,i,j , (9)
where α 1 and α 2 are two positive hyperparameters.The α 1 hyperparameter can then be interpreted as the threshold applicable to a scene showing a noiseless white background.

The motivation of the second term is that if the background noise is high at some pixel, we have to increase the associated threshold for background/foreground segmentation in order to prevent the misclassification of background pixels as foreground caused by background noise. For a given frame sequence X 1 , ..., X n and a reconstructed background sequence X1 , ..., Xn , we then compute the foreground mask M n before post-processing using the thresholding rule M n,i,j = 1 if and only if l n,i,j > τ n,i,j .

A post-processing is then applied in order to remove rain drops, snow flakes, and other spurious detections. It is composed of two morphological operations: a morphological closing using a 5 × 5 square structural element, followed by a morphological opening with a 7 × 7 square structural element.

Detecting significant background changes

The improved reconstruction loss function introduced in 3.1 reduces the risk of overfitting, but is not able to prevent it completely. We observe that the risk of overfitting increases when the number of optimization iterations and the number of parameters of the network increase. This is a significant issue because sequences showing background changes require a high number of training iterations and a model with a large number of parameters. In order to prevent overfitting, the number of training iterations and the complexity of the model are therefore adjusted to the complexity of the backgrounds sequence.

The main challenge here is to estimate without any human supervision whether the video shows substantial background changes or not. Such a task, which is very easy for a human, is far from trivial for a computer. For example, simply taking the variance of the various frames does not allow to estimate the complexity of the background changes because this variance will generally be dominated by foreground objects appearing in the video. More generally, it appears that in order to estimate the importance of the background changes, it is necessary to remove the foreground objects from the estimation process. We observe however that the proposed model can be used to perform this task. We then first train the model for a fixed small number N eval of iterations, which is however sufficient to get a rough evaluation of the background changes. Using this trained model, we compute B eval reconstructed backgrounds Xn using frames X n sampled randomly from the sequence X . Although these backgrounds estimates Xn are not accurate, we are confident that they do not show any foreground objects since a low number of iterations have been performed, so that the risk of overfitting is very low. We then compute the temporal median X of these backgrounds and compare this median background with the reconstructed backgrounds Xn , computing soft masks m n,i,j following the same process as in formula 1 and 2. We then consider the average soft mask value over the B eval reconstructed backgrounds

m = 1 B eval hw Beval,h,w n,i,j m n,i,j . ( 10
)
If m is higher than a threshold τ 0 , we consider that the background is a complex background. The partially trained model is discarded, a new autoencoder is created with more parameters and the number of training iterations is set to N complex with a minimum of E complex epochs for very long sequences. If this ratio is lower than τ 0 , we consider that the background is a simple background, keep the partially trained model, and finish the training, with a total number of training iterations set to N simple . The autoencoder structures for simple and complex backgrounds are described in the supplementary material.

Experimental results

Evaluation method

We consider the CDnet 2014, LASIESTA and BMC 2012 benchmark datasets for background subtraction. We use the public implementations of the algorithms PAWCS [START_REF] Luc St-Charles | Universal Background Subtraction Using Word Consensus Models[END_REF] and SuBSENSE [START_REF] Luc St-Charles | SuBSENSE: A universal change detection method with local adaptive sensitivity[END_REF] provided with the BGS library [START_REF] Sobral | BGSLibrary: An OpenCV C++ Background Subtraction Library[END_REF] to get baseline performance estimates for these methods when they are not available. We rely on published results for the other state of the art methods which do not provide public implementations.

We use the F-measure as main evaluation criteria. To compute the F-measure associated to a sequence of foreground masks predictions M 1 , .., M n , we first compute the sums T P, T N, F P, F N of the true positives, true negatives, false positives and false negatives associated to the sequence of masks M 1 , .., M n , and then compute the Fmeasure associated to this sequence as the harmonic mean of precision and recall.

We provide in Figure 3 some samples of background reconstruction, with the associated predicted foreground mask, and a comparison with foreground masks obtained using PAWCS and SuBSENSE. Other samples are provided in the supplementary material.

CDnet 2014 dataset

The CDnet 2014 dataset [START_REF] Wang | CDnet 2014: An expanded change detection benchmark dataset[END_REF] is composed of 53 videos, for a total of 153 278 frames, selected to cover the various challenges which have to be addressed for background subtraction: dynamic background (scenes with water or trees), camera jitter, intermittent object motion, presence of shadows, images captured by infrared cameras, challenging weather (snow, fog), images captured with a low frame rate, night images, images filmed by a pan-tilt-zoom camera, air turbulence. Ground truth foreground segmentation masks are provided for all frames of the dataset, with specific labels for shadow pixels which are not considered in the F-measure computation. We provide in Table 1 the Fmeasure results per category of the proposed model for each category of the CDnet 2014 dataset, with a comparison with the results obtained by other unsupervised models.

The proposed model gets a higher average F-measure on the CDnet 2014 dataset than all published unsupervised models, including ensemble models such as IUTIS-5, with an average F-measure of 0.784. One can observe a significant improvement in accuracy with the proposed model in the "pan-tilt-zoom" (PTZ) category with an average Fmeasure of 0.800 on this category. To our best knowledge, the proposed model is the first able to correctly handle videos taken from a moving camera.

LASIESTA dataset

The LASIESTA dataset [START_REF] Cuevas | Labeled dataset for integral evaluation of moving object detection algorithms: LASIESTA[END_REF] is composed of 48 videos grouped in 14 categories, for a total of 18 425 video frames. All frames are provided with ground truth pixel labels, with a specific label for pixels associated to stopped moving objects which are excluded from the F-measure computation. These videos are very short (The average number of frames per video is 383), which is challenging for the proposed deep-learning based model. We provide in Table 2 the average F-measure results of the proposed model for all 14 categories. Out of the 48 videos of the dataset, 4 videos are taken with a moving camera (categories IMC and OMC), and 24 videos include simulated camera motion (categories ISM and OSM). These 28 videos which include real or sim- ulated camera motion are very difficult for existing background subtraction models and to our best knowledge, no paper has ever published category-wise evaluation results for these videos. In order to allow a comparison with these published results, we therefore also provide the average Fmeasure over the 10 categories showing only videos taken from a fixed camera. We observe that the proposed model performs better than available unsupervised algorithms on static scenes, and with a significant improvement on scenes where the camera is moving.

BMC 2012 dataset

The BMC dataset [START_REF] Vacavant | A benchmark dataset for outdoor foreground/background extraction[END_REF] contains 9 videos showing real scenes taken from static cameras and including the following challenges: shadows, snow, rain, presence of trees or big objects. Three of these sequences are very long (32 965, 117 149 and 107 815 frames). For fair comparison with other published results for this dataset, we provide the Fmeasure results for our model obtained using the the usual F-measure definition described in 4.1, but also the results obtained using the executable evaluation tool provided with the dataset which does not use the same definition of the Fmeasure [START_REF] Vacavant | A benchmark dataset for outdoor foreground/background extraction[END_REF]. We compute SuBSENSE and PAWCS results on this dataset and provide published evaluation results for other models in Table 3.

We observe that the proposed model gets again a better average F-measure than PAWCS and SuBSENSE on this dataset using the standard definition of the F-measure.

Non-video image datasets : Clevrtex, Object-sRoom, ShapeStacks

The proposed model, which does not use any temporal information, can be adapted to perform background reconstruction and foreground segmentation on some image datasets which are not extracted from video sequences. We have tested this approach on three synthetic image datasets: Clevrtex [START_REF] Karazija | ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation[END_REF], ShapeStacks, [START_REF] Groth | ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking[END_REF] and ObjectsRoom [START_REF] Kabra | Multi-Object Datasets[END_REF]. We use on ShapeStacks and ObjectsRoom the same preprocessing as in [START_REF] Engelcke | GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement[END_REF]. Although each image of these datasets shows a different background, the model is able to recognize that all the backgrounds appearing in a given dataset lie in a low dimensional manifold, which is the case because they have been generated using the same method. These datasets are provided with segmentation annotations for each object appearing in the scenes, which we converted to binary foreground segmentation masks in order to compute the F-measure of the predicted foreground masks.

Considering that on these datasets the risk of overfitting is very low and the background complexity is very high, we substantially increased the number of iterations, which is set to 500 000. We do not use morphological postprocessing on the ShapeStacks and ObjectsRoom datasets, because these images have a very low resolution (64 × 64). We provide in Table 4 the average F-measure obtained on the test sets of these datasets after training on the associated training sets, and in Figure 4 some image samples. To our best knowledge, no other model is able to perform background reconstruction on these datasets.

Robustness to domain shift and fine-tuning

The proposed model is a batch model. In order to see whether it could be adapted for real-time applications, we studied whether a trained model could perform background reconstruction on new unseen images of the scene which do not belong exactly to the same distribution as the images used for training due to various possible domain shifts such as unseen illumination changes. We then have performed the following experiment: We have split each of the 53 videos provided in the CDnet dataset in two videos of equal lengths. The first half of each video is used to train the autoencoder, and the second half is used as a test dataset. The results of this experiment are provided in Table 5 and show stable results on three categories (baseline, bad weather, camera jitter) which do not show noticeable domain shifts, but a significant worsening on the other categories.

We then adopt the pretrain/fine-tune paradigm, consider the models trained on the first half of the videos as pretrained models, and study how many fine-tuning iterations using images randomly sampled from the second half of the videos are necessary to get competitive test results. We observe that the number of required iterations is very low compared to the number of iterations necessary for a full training, and conclude that a trained model is not robust to domain shifts, but can be quickly updated with a small number of fine-tuning iterations.

Implementation details

The proposed model is implemented using Python and the Pytorch framework. The associated code is available on the Github platform. Optimization is performed using the Adam optimizer with a learning rate of 5.10 -4 and batch size equal to 32. The learning rate is divided by 10 when the number of optimization or fine-tuning iterations reaches 80% of the total number of iterations. The most important hyperparameters β, r and τ 1 , which are associated to the loss function, are set to the values recommended in [START_REF] Sauvalle | Fast and Accurate Background Reconstruction Using Background Bootstrapping[END_REF] i.e. β = 6, r = 75, τ 1 = 0.25. The other hyperparameter values, which are related to the segmentation threshold and the detection and management of complex background changes, were found empirically using manual hyperparameter tuning. We then set α 1 = 96/255, α 2 = 7, N eval = 2000, B eval = 480, τ 0 = 0.24, N simple = 2500, N complex = 24000, E complex = 20.

For non-video dataset experiments, which take small images (64 × 64 and 128 × 128) as inputs, the batch size and learning rate are increased to 128 and 2.10 -3 and the number of iterations N complex is set to 500 000. The other hyperparameters remain the same. The autoencoder architecture is described in the supplementary material.

Computation time

We provide in Table 6 some computation time measurements, obtained using an AMD EPYC 7402 2,8 GHz CPU and a Nvidia RTX 3090 GPU. The inference and training times of the proposed model depend on the size of the image and the complexity of the background. The inference speed is between 50 frames per second and 240 frames per second. The time necessary to perform 100 training iterations is between 3,5 and 27 seconds.

Limitations

This model is not suited for night videos, considering the low score obtained on this category on the CDnet dataset. One also notes that although the model is able to handle correctly small objects staying still for a long time, as shown by the good results obtained the intermittent object category of the CDnet dataset, it suffers from overfitting when large foreground objects stay still (or appear to stay still) for a long time in a frame sequence. Out of the 110 tested videos contained in the datasets CDnet, LASIESTA and BMC, we observed this problem on 4 videos: "office", "library" and "canoe" in the CDnet dataset, and "video007" in the BMC dataset (Fig. 5). The proposed model should then not be used when the video is expected to show large objects staying still for a long time. This model is a batch model and adapting it to real-time applications requires further work in order to reduce the latency caused by the fine-tuning iterations described in section 4.6.

Ablation study

In order to assess the impact of the various model features described in this paper, we have implemented sev- Table 5. F-measure results obtained on the CDnet dataset with a model pretrained using the first half of each video as training set, and fine-tuned on the last half using various numbers of fine-tuning iterations. Test results are for the last half of each video. 7. They show that the design of the loss function and the use of the background noise estimation layer have a substantial positive impact on the accuracy of the model. The improvement associated to post-processing is also significant, as already observed for other unsupervised .5% -all backgrounds are considered as simple (τ0 set to 1) 0,7397 -5,6 % -using optical flow weights as in [START_REF] Sauvalle | Fast and Accurate Background Reconstruction Using Background Bootstrapping[END_REF] 0,7701 -1,8% -using abnormal image weights as in [START_REF] Sauvalle | Fast and Accurate Background Reconstruction Using Background Bootstrapping[END_REF] 0,7690 -1,9% background subtraction methods [START_REF] Shahbaz | Evaluation of background subtraction algorithms for video surveillance[END_REF]. The model remains competitive on CDnet if the background complexity of all frames sequence is set to simple, an option which may be considered if training computation time is an issue.

Conclusion

We have proposed in this paper a new fully unsupervised dynamic background reconstruction and foreground segmentation model which does not use any temporal or motion information and is on average more accurate than available unsupervised models for background subtraction. The main strength of the proposed model is that it is able to perform background reconstruction on videos taken from a moving camera. Future works includes adapting the model for real-time applications, and using it to perform unsupervised object detection on real world scenes with complex backgrounds.

Figure 1 .

 1 Figure 1. The proposed model takes as input a frame from the associated video (left column) and provides a reconstruction of the background (middle column) and a foreground mask (right column).

Figure 2 .

 2 Figure 2. Schematic of the proposed model during inference (Error and noise images are normalized in the range [0,1])

Figure 3 .

 3 Figure 3. Examples of background reconstruction and foreground segmentation produced using the proposed model and comparison with PAWCS and SuBSENSE

Figure 4 .

 4 Figure 4. Examples of background reconstruction and foreground segmentations on the datasets Clevrtex (columns 1-4), ObjectsRoom (columns 5-6) and ShapeStacks (columns 7-8)

Figure 5 .

 5 Figure 5. Failure cases due to overfitting on the datasets CDnet 2014 and BMC 2012: sequences "library", "office", "canoe" and "video007" eral modifications of the proposed model and measured the average F-measure (FM) of these models on the CD-net2014 dataset. The results of these experiments are provided in Table7. They show that the design of the loss function and the use of the background noise estimation layer have a substantial positive impact on the accuracy of the model. The improvement associated to post-processing is also significant, as already observed for other unsupervised

  -training with L2 reconstruction loss, α2 set to 0 0,3384 -56,8 % -inference without morphological post-processing 0.7170 -8

Table 1 .

 1 Comparison of top unsupervised BGS algorithms according to the per-category F-measures on CDnet-2014

	Method	Bad weather	Base-line	Camera jitter	Dynamic backgr.	Int. obj. motion	Low framerate	Night	PTZ	Shadow Thermal	Turbu-lence	Overall
	AE-NE (ours)	0.8337 0.8959 0.9230 0.6225	0.8231	0.6771	0.5172 0.8000 0.8947 0.7999	0.8382 0.7841
	IUTIS-5 [5]	0.8248 0.9567 0.8332 0.8902	0.7296	0.7743	0.5290 0.4282 0.9084 0.8303	0.7836 0.7717
	WisenetMD [33] 0.8616 0.9487 0.8228 0.8376	0.7264	0.6404	0.5701 0.3367 0.8984 0.8152	0.8304 0.7535
	SuBSENSE [58] 0.8619 0.9503 0.8152 0.8177	0.6569	0.6445	0.5599 0.3476 0.8986 0.8171	0.7792 0.7408
	PAWCS [59]	0.8152 0.9397 0.8137 0.8938	0.7764	0.6588	0.4152 0.4615 0.8913 0.8324	0.6450 0.7403
	C-EFIC [1]	0.7867 0.9309 0.8248 0.5627	0.6229	0.6806	0.6677 0.6207 0.8778 0.8349	0.6275 0.7307
	MSCL [27]	0.83	0.87	0.83	0.85	0.80	n/a	n/a	n/a	0.82	0.80	0.80	n/a
	B-SSSR [26]	0.92	0.97	0.93	0.95	0.74	n/a	n/a	n/a	0.93	0.86	0.87	n/a

Table 2 .

 2 Average per category of video F-measures on LASIESTA (sources :[START_REF] Cuevas | Labeled dataset for integral evaluation of moving object detection algorithms: LASIESTA[END_REF],[START_REF] Berjón | Real-time nonparametric background subtraction with tracking-based foreground update[END_REF], authors experiments for PAWCS and SuBSENSE)

				static camera		moving camera
						or simulated motion
	Method	ISI	ICA IOC IIL	IMB IBS OCL ORA OSN OSU IMC ISM OMC OSM	Average. 10 categ.	Average. 14 categ.
	AE-NE (ours)	0.91 0.88 0.91 0.81 0.92 0.79 0.94 0.80	0.82 0.91 0.83 0.79 0.86	0.89	0.87	0.86
	PAWCS [59]	0.90 0.88 0.90 0.79 0.81 0.79 0.96 0.93	0.69 0.82 0.48 0.77 0.43	0.75	0.85	0.78
	SuBSENSE [58] 0.90 0.89 0.95 0.65 0.77 0.73 0.92 0.90	0.81 0.79 0.33 0.70 0.31	0.65	0.83	0.73
	Cuevas [4]	0.88 0.84 0.78 0.65 0.93 0.66 0.93 0.87	0.78 0.72 n/a	n/a	n/a	n/a	0.81	n/a
	Haines [22]	0.89 0.89 0.92 0.85 0.84 0.68 0.83 0.89	0.17 0.86 n/a	n/a	n/a	n/a	0.78	n/a
	Maddalena [41]	0.95 0.86 0.95 0.21 0.91 0.40 0.97 0.90	0.81 0.88 n/a	n/a	n/a	n/a	0.78	n/a
	Maddalena [40]	0.87 0.85 0.91 0.61 0.76 0.42 0.88 0.84	0.58 0.80 n/a	n/a	n/a	n/a	0.75	n/a

Table 3 .

 3 Comparison of top unsupervised BGS algorithms according to the video F-measure on BMC 2012

		Method			Video 001	Video 002	Video 003	Video 004	Video 005	Video 006	Video 007	Video 008	Video 009	Average 9 videos
		F-measure (standard definition)
		AE-NE (ours)			0.81	0.72	0.78	0.78	0.60	0.73	0.32	0.84	0.77	0.71
		PAWCS [59]			0.70	0.58	0.85	0.72	0.27	0.79	0.58	0.74	0.80	0.67
		SuBSENSE [58]		0.70	0.62	0.83	0.69	0.21	0.76	0.53	0.68	0.83	0.65
		F-measure (using BMC evaluation tool)
		AE-NE (ours)			0.90	0.86	0.89	0.89	0.80	0.87	0.51	0.92	0.89	0.84
		PAWCS [59]			0.86	0.77	0.93	0.86	0.66	0.89	0.79	0.87	0.90	0.84
		SubSENSE [58]		0.85	0.80	0.92	0.85	0.68	0.87	0.75	0.84	0.91	0.83
		DeepPBM [17]			0.73	0.86	0.94	0.90	0.71	0.81	0.70	0.76	0.69	0.78
		G-LBM [51]			0.73	0.85	0.93	0.91	0.71	0.85	0.70	0.76	0.63	0.79
		MSCL-FL [27]			0.84	0.84	0.88	0.90	0.83	0.80	0.78	0.85	0.94	0.86
		B-SSSR [26]			n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	0.88
	Table 4. F-Measure on the Clevrtex, ShapeStacks and Object-
	sRoom datasets
		number of	number of	average
	dataset	image size	frames	frames	F-measure
		training set	test set	on test set
	Clevrtex	128 × 128	40000	5000	0.78
	ObjectsRoom	64 × 64	980000	20000	0.84
	ShapeStacks	64 × 64	217888	46656	0.83

Table 6 .

 6 Computation time of the proposed model, PAWCS and SubSENSE for some sequences of the CDnet and BMC datasets

		Bad weather	Base-line	Camera jitter	Dynamic backgr.	Int. obj. motion	Low framerate	Night	PTZ	Shadow Thermal	Turbu-lence	Overall no pretraining
	no fine-tuning 0.8114 0.8660 0.8768 0.3845	0.4199	0.5732	0.3998 0.2426 0.7371 0.5872	0.6447 0.5948
	100 iterations 0.8137 0.9063 0.9520 0.5846	0.5956	0.5891	0.4789 0.4723 0.9276 0.7639	0.6639 0.7044 0.4918
	200 iterations 0.8078 0.9105 0.9543 0.6111	0.6536	0.5859	0.4977 0.4969 0.9316 0.7849	0.7523 0.7261 0.5658
	400 iterations 0.8080 0.9125 0.9560 0.6309	0.7298	0.5842	0.5137 0.5465 0.9326 0.7880	0.8560 0.7507 0.6218
	800 iterations 0.8104 0.8965 0.9577 0.6348	0.8212	0.5946	0.5420 0.6403 0.9293 0.7828	0.8763 0.7714 0.6934
	sequence name	highway	Video blizzard	zoomin continuous
			009		zooomout		pan
	image size	240x320 288x352 480x720	240x320	480x704
	number of frames	1700	107817	7000	1130		1700
	background complexity	simple	simple	simple	complex	complex
	computation times (seconds)
	AE-NE (proposed model)
	-training	92	114	394	1443		7175
	-backgrounds
	and masks generation	7	560	139	5		33
	SuBSENSE	92	7161	1586	65		471
	PAWCS	158	11290	2311	164		980
	input	ground	predicted	foreground
	frame	truth	background	mask

Table 7 .

 7 Evaluation of various ablations of the proposed model

Acknowledgment We thank Sascha Hornauer for useful comments on the first draft of this paper.

Supplementary material 6.1. Autoencoder architecture

The autoencoder is deterministic and takes as input a RGB image of size h × w, and produces a RGB image (3 channels) and an error estimation map of the same size (1 channel).

The encoder and decoder structures in the proposed model are computed dynamically using as input the size (height h and width w) of the input frames of the dataset. The number of latent variables produced by the encoder is fixed to [START_REF] Engelcke | GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement[END_REF].

We use a fully convolutional autoencoder architecture, which appears to be more robust to overfitting than architectures including fully connected layers or locally connected layers. We add two fixed positional encoding channels as inputs to all layers of the encoder and the decoder, one chan-nel coding for the horizontal coordinates, the other one for the vertical coordinates .

The encoder is a sequence of blocks composed of a convolution layer with kernel size 5, stride 3 and padding equal to 2, followed by a group normalization layer and a CELU nonlinearity layer. The generator is a symmetric sequence of blocks composed of transpose convolution layers with kernel size 5 and stride 3 and padding equal to 2 followed by group normalization and a CELU nonlinearity, except for the last layer where the transpose convolution layer is followed by a sigmoid to generate the final image. The number of layers of the encoder and the decoder is then equal to 5 or 6 depending on the image size (assuming that the maximum of the image height and image width is in the range 200 -1000). The number of channels per convolutional layer is fixed according to Table 8, depending on the image size and the background complexity.

These channel distributions are motivated by the fact that a larger number of parameters is required in the generator in order to handle complex backgrounds, but that we have experimentally observed that a large number of channels in the last layer of the encoder and the first layer of the decoder increases the risk of overfitting on foreground objects, so that reducing this number for long training schedule is necessary to improve the robustness of the auto-encoder with respect to the risk of overfitting. For example, we have measured that increasing the numbers of channels in the last hidden layer of the encoder and first hidden layer of the decoder to 160 and 256 leads to de 2,3 % degradation of the average F-Measure on the CDnet dataset.

For non-video dataset experiments, which handle small images, we use a smaller stride, set to 2 instead of 3. The autoencoder architectures for 64 × 64 images (ShapeStacks and ObjectRooms datasets) and 128×128 images (Clevrtex dataset) are described in Table 9 and 10:

Additional implementation details

The datasets and preprocessing codes for CLEVRTEX, Shapestacks and ObjectsRoom were downloaded from the following public repositories:

• https://www.robots.ox.ac.uk/ ˜vgg/ data/clevrtex/

• https://ogroth.github.io/ shapestacks/

• https://github.com/deepmind/multi_ object_datasets

Additional image samples

We provide in figures 1 -7 additional samples of background reconstruction and foreground segmentation obtained using the proposed model.


Bruno Sauvalle
email: bruno.sauvalle@mines-paristech.fr

Arnaud De

La Fortelle

Unsupervised multi-object segmentation using attention and soft-argmax

We introduce a new architecture for unsupervised objectcentric representation learning and multi-object detection and segmentation, which uses a translation-equivariant attention mechanism to predict the coordinates of the objects present in the scene and to associate a feature vector to each object. A transformer encoder handles occlusions and redundant detections, and a convolutional autoencoder is in charge of background reconstruction. We show that this architecture significantly outperforms the state of the art on complex synthetic benchmarks.

Introduction

We consider in this paper the tasks of object-centric representation learning and unsupervised object detection and segmentation: Starting from a dataset of images showing various scenes cluttered with objects, our goal is to build a structured object-centric representation of these scenes, i.e. to map each object present in a scene to a vector representing this object and allowing to recover its appearance and segmentation mask. This task is very challenging because the objects appearing in the images may have different shapes, locations, colors or textures, can occlude each other, and we do not assume that the images share the same background. However the rewards of object-centric representations could be significant since they allow to perform complex reasoning on images or videos [START_REF] Ding | Attention over learned object embeddings enables complex visual reasoning[END_REF][START_REF] Tang | Object Dynamics Distillation for Scene Decomposition and Representation[END_REF] and to learn better policies on downstream tasks involving object manipulation or localization [START_REF] Veerapaneni | Entity Abstraction in Visual Model-Based Reinforcement Learning[END_REF][START_REF] Zadaianchuk | Self-supervised Visual Reinforcement Learning with Object-centric Representations[END_REF]. The main issue with objectrepresentation learning today is however that existing models are able to process synthetic toy scenes with simple textures and backgrounds but fail to handle more complex or real-world scenes [START_REF] Karazija | ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation[END_REF].

We propose to improve upon this situation by introducing a translation-equivariant and attention-based approach for unsupervised object detection, so that a translation of the input image leads to a similar translation of the coordinates of the detected objects, thanks to an attention map which is used not only to associate a feature vector to each object present in the scene, but also to predict the coordinates of these objects.

The main contributions of this paper are the following:

• We propose a theoretical justification for the use of attention maps and soft-argmax for object localization.

• We introduce a new translation-equivariant and attention-based object detection and segmentation architecture which does not rely on any spatial prior.

• We show that the proposed model substantially improves upon the state of the art on unsupervised object segmentation on complex synthetic benchmarks.

The paper is organized as follows: In section 2, we provide some theoretical motivation for using attention maps and soft-argmax for object localization. In section 3, we review related work on unsupervised object instance segmentation. In section 4 we describe the proposed model. Experimental results are then provided in section 5.

Motivation for using attention maps and soft-argmax for object localization

It is widely recognized that the success of convolutional neural networks is associated with the fact that convolution layers are equivariant with respect to the action of the group of translations, which makes these layers efficient for detecting features which naturally have this property. It is also easy to show that linear convolution operators are the only linear operators which are equivariant with respect to the natural action of the translation group on feature maps.

We introduce the following notations to describe the action of the translation group: We consider a grayscale image as a scalar-valued function φ(i, j) defined on Z 2 and an element of the group of translations as a vector (u, v) in Z 2 . The natural action T of the group of translations on an image can be described by the formula

T u,v (φ)(i, j) = φ(i -u, j -v). (1)
A model layer L is called equivariant with respect to translations if it satisfies

L(T u,v φ) = T u,v (L(φ)). (2)
Let's now consider a localization model M which takes as input an image φ(i, j) showing one object and produces as output the coordinates of the object present in this image. Such a model does not produce a feature map, so that the previous definition of translation equivariance can not be used for this model. We remark however that the group of translations acts naturally on Z 2 by the action T ′ u,v (i, j) = i + u, j + v, and that the model M should have the equivariance property

M (T u,v φ) = T ′ u,v (M (φ)). (3)
Indeed, if the complete image is translated by a vector (u, v), then the object present in this image is also translated, so that the associated coordinates have to be shifted according to the vector (u, v). It is not difficult to see that in the same way that convolutional operators are the only linear operators equivariant with respect to translations, it is also possible to fully describe which elementary operators follow this specific equivariance property. We first remark however that we have to restrict the space of possible input maps φ: if φ is a constant function, it does not change under the action of the translation group, so that the equivariance property 3 cannot be satisfied with such a function. We then suppose that φ satisfies p φ(p) = 1 and consider that the domain of the operator M is the corresponding affine space A. We also replace the linearity condition by an the following affinity condition:

For all α i ∈ R, φ i ∈ A so that i α i = 1, we have

M ( i α i φ i ) = i α i M (φ i ).
We then have the following proposition:

Proposition 2.1 An affine operator M which satisfies the equivariance property 3 has to be of the form

M (φ) = C + p∈Z 2 φ(p)p (4)
for some constant C in R 2 .

Proof: We write the input map φ as a sum of spatially shifted version of the function δ ∈ A satisfying δ(p) = 1 for p = (0, 0) and δ(p) = 0 for p ̸ = (0, 0):

φ(p) = q∈Z 2 φ(q)δ(p -q) (5)
We then use the the affine property of M and equivariance property 3:

M (φ) = M ( q φ(q)δ(p -q)) (6)
= q φ(q)M (δ(p -q)) = q φ(q)(M (δ) + q) (7) = ( q φ(q))M (δ) + q φ(q)q (8) = M (δ) + q φ(q)q, (9)
which proves the proposition since M (δ) is a constant. The proposition 2.1 can be interpreted as stating that in order to get an equivariant localization operator, the most straightforward method is to build a normalized attention map φ from the input image and compute the coordinates of the detected object using an attention mechanism with φ as attention map and pixel coordinates as target values. One remarks that it is precisely what the soft-argmax operator is doing: It takes an unnormalized scalar map ϕ as input, normalizes it using a softmax operator, and then perform localization using the same formula as in 2.1:

soft-argmax(ϕ) = p∈Z 2 softmax(ϕ)(p)p = p∈Z 2 e ϕ(p) q∈Z 2 e ϕ(q) p (10)
This operation is called soft-argmax because it allows to compute in a differentiable way an estimate of the coordinates of the maximum of the input map ϕ. Using softargmax then appears to be the most natural way to get an equivariant localization operator.

Related work

Unsupervised object detection and segmentation Unsupervised object detection and segmentation models are generally reconstruction models: They try to reconstruct the input image using a specific image rendering process which induces the required object-centric structure. In order to ensure that objects are properly detected, various objectness priors have been defined and implemented:

• pixel similarity priors. Some models consider the task of object segmentation as a clustering problem, which can be addressed using deterministic [START_REF] Jyh | SegSort: Segmentation by discriminative sorting of segments[END_REF][START_REF] Locatello | Objectcentric learning with slot attention[END_REF] or probabilistic [START_REF] Engelcke | GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement[END_REF][START_REF] Greff | Tagger: Deep unsupervised perceptual grouping[END_REF][START_REF] Van Steenkiste | Relational neural expectation maximization: Unsupervised discovery of objects and their interactions[END_REF] methods: If the feature vectors associated to two different pixels of an image are very similar, then it is considered that these pixels should both belong to the same object or to the background.

• independence priors. Some models assume that the images are sampled from a distribution which follows a probabilistic model featuring some independence priors between objects and the background, and use variational [START_REF] Greff | Multi-object representation learning with iterative variational inference[END_REF][START_REF] Engelcke | GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations[END_REF] or adversarial [START_REF] Chen | Unsupervised object segmentation by redrawing[END_REF][START_REF] Bielski | Emergence of object segmentation in perturbed generative models[END_REF] methods to learn these distributions.

• disentanglement of appearance and location. Foreground objects appearing in the scenes of a given dataset can have similar shapes and appearances but very different scales and locations. Object discovery is performed by disentangling the object appearance generation process, which is performed by a convolutional glimpse generator [START_REF] Ali Eslami | Attend, infer, repeat: Fast scene understanding with generative models[END_REF][START_REF] Kosiorek | Sequential attend, infer, repeat: Generative modelling of moving objects[END_REF][START_REF] Crawford | Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks[END_REF][START_REF] Stelzner | Faster Attend-Infer-Repeat with Tractable Probabilistic Models[END_REF][START_REF] Jiang | SCALOR: Generative World Models with Scalable Object Representations[END_REF][START_REF] Jiang | Generative neurosymbolic machines[END_REF] or a learned dictionary [START_REF] Monnier | Unsupervised Layered Image Decomposition into Object Prototypes[END_REF][START_REF] Smirnov | MarioNette: Self-Supervised Sprite Learning[END_REF], from the translation and scaling of the objects appearing in a scene, which is usually done by including a spatial transformer network [?] in the model. The model described in this paper belongs to this category and uses an convolutional glimpse generator.

Object detection and segmentation without spatial prior

State of the art supervised detection and segmentation models usually rely on predefined reference anchors or center points which are spatially organized according to a periodic grid structure. The use of periodic grids has also been proposed for unsupervised object detection [START_REF] Lin | SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition[END_REF][START_REF] Jiang | SCALOR: Generative World Models with Scalable Object Representations[END_REF][START_REF] Jiang | Generative neurosymbolic machines[END_REF][START_REF] Smirnov | MarioNette: Self-Supervised Sprite Learning[END_REF]. Alternative detection methods relying on heatmaps produced by a U-net [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF] or stacked U-nets [START_REF] Newell | Stacked hourglass networks for human pose estimation[END_REF] networks, which predict for each pixel the probability of presence of one object on this pixel have been implemented in the supervised setting [START_REF] Law | CornerNet: Detecting Objects as Paired Keypoints[END_REF][START_REF] Duan | CenterNet: Keypoint triplets for object detection[END_REF]. For some specific applications such as human pose estimation or anatomical landmark localization [START_REF] Tiulpin | KNEEL: Knee anatomical landmark localization using hourglass networks[END_REF], some supervised models predict one heatmap per object. The use of soft-argmax for converting heatmaps to object coordinates has been implemented in the supervised [38, ?, 6], semi-supervised [?] and unsupervised settings [START_REF] Goroshin | Learning to linearize under uncertainty[END_REF][START_REF] Finn | Deep spatial autoencoders for visuomotor learning[END_REF] but has never been proposed for unsupervised object detection or segmentation. More recently, transformerbased [?] models using object [START_REF] Carion | Endto-End Object Detection with Transformers[END_REF][START_REF] Zhu | Deformable DETR: Deformable Transformers for End-to-End Object Detection[END_REF][START_REF] Dong | SOLQ: Segmenting Objects by Learning Queries[END_REF] or mask [START_REF] Cheng | Per-Pixel Classification is Not All You Need for Semantic Segmentation[END_REF][START_REF] Cheng | Masked-attention Mask Transformer for Universal Image Segmentation[END_REF] queries have been proposed which not not rely explicitly on a spatial grid. These models show that transformers are efficient in the supervised setting to avoid multiple detections of the same object.

Description of proposed model 4.1. Model architecture

The overall architecture of the model is described in Fig 1.

The proposed model is composed of a a foreground model and a background model.

The background model is a deterministic convolutional autoencoder: We rely on the classical assumption [START_REF] Wright | Robust principal component analysis: Exact recovery of corrupted low-rank matrices by convex optimization[END_REF] that background images lie on a low-dimensional manifold, and use the autoencoder to learn this manifold.

The foreground model is also deterministic and associates to each object in the scene an appearance vector z what which is used to produce a glimpse of the object, which is then scaled and translated at the right position on the image using a spatial transformer network.

The foreground encoding and reconstruction process can be described as follows: First, a high resolution feature map generator takes a color image of size h × w as input and produces a high resolution feature map Φ of dimension d Φ and several scalar attention logit maps A 1 , ..., A K . We will use in this paper the transformer-based Segformer model [START_REF] Xie | SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers[END_REF], which produces feature maps of size h * ×w * = h/4× w/4. The hyperparameter K is set to the maximum number of objects on a scene in the dataset. The scalar attention logit maps A 1 , ..., A K are transformed into a normalized attention maps A 1 , ..., A K using a softmax operator:

A k (i, j) = e A k (i,j) i ′ ,j ′ e A k (i ′ ,j ′ ) (11)
We normalize the pixel indices (i, j) from the range [1, .., w * ] and [1, .., h * ] to the range [-1, 1] required by spatial transformer networks using the formulas

x(i) = 2 i -1 w * -1 -1 (12)
y(j) = 2 j -1 h * -1 -1, (13)
and predict initial estimates x 0 k , y 0 k of the coordinates of the detected objects as the the center of mass of the attention maps A k :

x 0 k = w * ,h * i=1,j=1 A k (i, j)x(i) (14)
y 0 k = w * ,h * i=1,j=1 A k (i, j)y(j) (15)
We also build K object query feature vectors ϕ 0 1 , ..., ϕ 0 K of dimension d Φ using the same attention maps A 1 , .., A K as weights and the feature map Φ as target values:

ϕ 0 k = w * ,h * i=1,j=1 A k (i, j)Φ(i, j) (16)
A transformer encoder then takes the K triplets (ϕ 0 k , x 0 k , y 0 k ) 1≤k≤K as inputs and produces a refined version (ϕ k , x k , y k ) 1≤k≤K taking into account possible detection redundancies and object occlusions. More precisely, we use a learned linear embedding to increase the dimension of the triplets (ϕ 0 k , x 0 k , y 0 k ) from d Φ + 2 to the input dimension d T of the transformer encoder, and a learned linear projection to reduce the dimension of the outputs of the transformer encoder from d T back to d Φ + 2. The transformer encoder does not take any positional encoding as input, considering that the transformation which has to be performed should not depend on the ordering of the detections. We force the final values of x k and y k to stay in the range [-1, 1] using clamping. Each transformed feature vector ϕ k is then split in three terms:

ϕ k = (s k , α k , z what k ).
• The first term s k is an inverse scaling factor. It is a scalar if objects in the dataset have widths and heights which are similar (isotropic scaling), or a pair of scalars s x k , x y k if this is not the case (anisotropic scaling). We force the values of s k to stay within a fixed range using a sigmoid function. The maximum value of this range ensures that a non-zero gradient will be available. The minimum value is set higher than 1 to make sure that the glimpse generator will not try to generate a full image layer.

• The second term is a scalar which is assumed to predict the activation level α k of the object, which will be used to predict whether it is visible or not. We force this activation value to be positive using an exponential map.

• The remaining coordinates form a vector z what k which codes for the appearance of the object.

We then use a convolutional glimpse generator to build a color image o k of the associated object together with the associated scalar mask m k , using z what k as input. These images and masks are translated to the positions (x k , y k ) and scaled according to the inverse scaling factor s k using a spatial transformer network. We note L k and M k for k ∈ {1, .., K} the corresponding object image layers and masks, and L 0 the background image produced by the background model, so that we have a total of K + 1 image layers.

We now have to decide for each pixel whether this pixel should show the background layer or one of the K object layers. In order to do this in a differentiable way, we multiply the predicted object masks M k with the associated object activation levels α k , and normalize the results to get one normalized weights distribution (w k ) 0≤k≤K per pixel:

w k (i, j) = α k M k (i, j) k ′ ∈0..K α k ′ M k ′ (i, j) , (17)
considering that the mask M 0 associated to the background is set to 1 everywhere and that it has a fixed learned activation factor α 0 . The final reconstructed image X is then equal to the weighted sum of the various image layers using the weights w k :

X(i, j) = K k=0 w k (i, j)L k (i, j) (18)
During inference the segmentation map is built by assigning to each pixel the layer index k ∈ {0, .., K} for which w k (i, j) is the maximum. The background model is not needed to get the segmentation maps during inference.

Model training 4.2.1 loss function

In order to train the proposed model, we use a main reconstruction loss function and an auxiliary loss:

reconstruction loss The local L 1 reconstruction error associated to the pixel (i, j) is

l i,j = 3 c=1 |x c,i,j -x c,i,j |, (19)
where x c,i,j and xc,i,j are the values of the color channel c at the position (i, j) in the input image and reconstructed image.

The reconstruction loss is defined as the mean square of this reconstruction error.

L rec = 1 hw w,h i=1,j=1 l 2 i,j (20)
pixel entropy loss For a given pixel (i, j), we expect the distribution of the weights w 0 (i, j), .., w K (i, j) to be onehot, because we assume that the objects are opaque. We observe that a discrete distribution is one-hot if and only if it has a zero entropy, so that minimizing the entropy of this distribution would be a reasonable way to enforce a stick-breaking process. Considering however that the entropy function has a singular gradient near one-hot distributions, we use the square of the entropy function to build the loss function. We then define the pixel entropy loss as

L pixel = 1 hw w,h i=1,j=1 ( K k=0 w k (i, j) log(w k (i, j) + ϵ)) 2 , (21)
where ϵ = 10 -20 is introduced to avoid any numerical issue with the logarithm function.

This auxiliary loss is weighted using the weight λ pixel before being added to the reconstruction loss. During our experiments, we observed that the pixel entropy loss could prevent a successful initialization of the localization process during the beginning of the training. As a consequence, we smoothly activate this auxiliary loss during initialization using a quadratic warmup of the weight.

The full loss function is then equal to

L = L rec + min(1, step N pixel ) 2 λ pixel L pixel , ( 22
)
where step is the current training iteration index and N pixel is a fixed hyperparameter.

curriculum training

The interaction between the background reconstruction model and the foreground model during training is a very challenging issue, because of the competition between them to reconstruct the image. We handle this problem as in [START_REF] Jiang | Generative neurosymbolic machines[END_REF] by implementing curriculum training. We will then evaluate two methods to train the proposed model:

• baseline training (BT) : The background and foreground models are initialized randomly and trained simultaneously.

• curriculum training (CT):

The training of the model is split in three phases :

1. The background model is pretrained alone, using the methodology and robust loss function described in [START_REF] Sauvalle | Autoencoderbased background reconstruction and foreground segmentation with background noise estimation[END_REF].

2. The weights of the background model are then frozen and the foreground model is trained using the frozen background model.

3. The background and foreground models are then fine-tuned simultaneously.

Experimental results

Evaluation on public benchmarks

We perform a quantitative evaluation of the proposed model on the following datasets: CLEVRTEX [START_REF] Karazija | ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation[END_REF], CLEVR [START_REF] Johnson | CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning[END_REF], ShapeStacks [START_REF] Groth | ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking[END_REF] and ObjectsRoom [START_REF] Kabra | Multi-Object Datasets[END_REF].

We implement on ShapeStacks, ObjectsRoom and CLEVR the same preprocessing as in [START_REF] Engelcke | GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement[END_REF]. We use the same hyperparameter values on these datasets, except for the hyperparameter K related to the number of object queries, which is set to the maximum number of objects in each dataset (i.e. 3 on ObjectsRoom, 6 on ShapeStacks and 10 on CLEVRTEX and CLEVR). We use isotropic scaling on CLEVR and ShapeStacks and anisotropic scaling on the other datasets.

We use the versions B3 of the Segformer model, and rely on the Hugging Face implementation of this model, with pretrained weights on ImageNet-1k for the hierarchical transformer backbone, but random initialization for the MLP decoder which is used as a feature map generator. We use the standard Pytorch implementation of the transformer encoder. The architecture of the backgroud model autoencoder is the same as in [START_REF] Sauvalle | Autoencoderbased background reconstruction and foreground segmentation with background noise estimation[END_REF]. The glimpse generator is a sequence of transpose convolution layers, group normalization [START_REF] Wu | Group Normalization[END_REF] layers and CELU [START_REF] Barron | Continuously Differentiable Exponential Linear Units[END_REF] non-linearities, and is described in the supplementary material.

We use Adam as optimizer. The training process includes a quadratic warmup of the learning rate since the model contains a transformer encoder. We also decrease the learning rate by a factor of 10 when the number of training steps reaches 90% of the total number of training steps. The total number of training steps of the baseline training (BT) scenario is 125 000. In the curriculum training (CT) scenario, the number of training steps for background model pretraining (phase 1) is 500 000 on CLEVRTEX, ShapeStacks and ObjectsRoom, but 2500 on CLEVR, which shows a fixed background, as recommended in [START_REF] Sauvalle | Autoencoderbased background reconstruction and foreground segmentation with background noise estimation[END_REF]. The number of training steps of phase 2 (training with frozen pretrained background model) is 30 000, and the number of training steps of the final fine-tuning phase (phase 3) is 95 000.

Full implementation details and hyperparameter values are provided in the supplementary material, and the model code will be made available on the Github platform.

In order to compare our results with published models, we compute the following evaluation metrics: mean intersection over union (mIoU) and adjusted rand index restricted to foreground objects (ARI-FG). We also provide the mean square error (MSE) between the reconstructed image and the input image, which provides an estimate of the accuracy of the learnt representation. We use the same definitions and methodology as [START_REF] Karazija | ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation[END_REF] for these metrics. We provide the mean segmentation covering (defined in [START_REF] Engelcke | GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations[END_REF]) restricted to foreground objects (MSC-FG) on ObjectsRoom and ShapeStacks where mIoU baseline values are not available.

We call AST-Seg (Attention and Soft-argmax with Transformer using Segformer) the proposed model, and AST-Seg-B3-BT, AST-Seg-B3-CT respectively the models using a Segformer B3 feature map generator trained under the baseline training or curriculum training scenarios.Table 1 and 2 provide the results obtained on these datasets with a comparison with published results.

The proposed model trained under the baseline training scenario gets better average results than existing models on the CLEVR and CLEVRTEX dataset, but shows a very high variance. For example, on the CLEVR dataset, the model may fall during training in a bad minimum where the background model tries to predict the foreground objects. Using curriculum training allows to avoid this issue, get stable results on all datasets, and obtain a very significant mIoU improvement on the most complex datasets CLEVR and CLEVRTEX.

Following the methodology proposed in [START_REF] Karazija | ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation[END_REF], we also evaluated the generalization capability of a model trained on CLEVRTEX when applied to datasets containing out of distribution images showing unseen textures and shapes or camouflaged objects (OOD and CAMO datasets [START_REF] Karazija | ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation[END_REF]). The results of this evaluation are provided in Table 3 and show that the proposed model generalizes well, although it is deterministic and does not use any specific regularization scheme. Some segmentation prediction samples are provided in Fig 8 . Other image samples are available in the supplementary material.The main limitation of the proposed model is the management of shadows, which may be considered by the model as separate objects or integrated to object segmentations.

Ablation study and additional experiments

We provide in Table 4 results obtained using various ablations or modifications on the model architecture or loss function, which show that:

• The model remains competitive if the transformer encoder is removed by setting

(ϕ k , x k , y k ) 1≤k≤K = (ϕ 0 k , x 0 k , y 0 k ) 1≤k≤K .
The results on the ShapeStacks and ObjectsRoom datasets are even improved with this simplified architecture, with a surprisingly strong improvement on the Shapestacks dataset, which shows the efficiency of the attention and soft-argmax mechanism. The transformer encoder is however necessary on the more complex CLEVR and CLEVRTEX datasets.

• Training with a number of slots slightly higher than the maximum number of objects does not lead to significant changes in the results. A more substantial increase of the number of slots however leads to poor results on scenes with complex textures due to the increasing fragmentation the objects. This is very different from the situation observed on query-based supervised detection models like DETR, where the number of queries has to be very high compared to the number of objects.

• It is possible to replace the Segformer high resolution feature map generator with any other generator. The proposed model was originally designed with a custom Unet feature map generator, which gets similar results as the Segformer model on CLEVR, ShapeStacks and ObjectsRoom, but underperforms on the more complex CLEVRTEX dataset. The architecture of this Unet is described in the supplementary material.

• Using a pretrained backbone is necessary to get good performances with a Segformer feature map generator.

• We tested an alternative training scenario where the background model remains frozen during the complete training of the foreground model (125 000 iterations).

The main advantage of this scenario is that it is significantly faster and requires less memory, since the backgrounds of the training images can be pre-computed and memorized. The accuracy of the results is however lower than the curriculum training scenario proposed in this paper, except for the ObjectsRoom dataset.

• Switching between isotropic scaling and anisotropic scaling does not make much difference, except for the ShapeStacks dataset, where the proposed model can consider that each block tower is a single object if anisotropic scaling is enabled.

Computation time

All experiments have been performed using a Nvidia RTX 3090 GPU and a AMD 7402 EPYC CPU. Some train-ing durations are provided in Table 5.

Conclusion

We have described in this paper a new architecture for unsupervised object-centric representation learning and object detection and segmentation, which relies on attention and soft-argmax, and shown that this new architecture substantially improves upon the state of the art on existing benchmarks showing synthetic scenes with complex shapes and textures. We hope this work may help to extend the scope of structured object-centric representation learning from research to practical applications. • The center block is composed of a convolutional layer with stride 1 and kernel size 3 with batch normalization and CELU.

• An upsample block is composed of a residual convolutional layer with stride 1 and kernel size 3 with batch normalization and CELU, followed by a transpose convolutional layer with stride 2 and kernel size 4, with batch normalization and CELU.

Additional image samples

Additional image samples are provided in Figures 123456.

Figure 1 .

 1 Figure1. Overview of proposed model. A High resolution feature map generator (Segformer model) is trained to produce a high resolution feature map Φ and K scalar attention maps (one per object query). These maps are used to predict the coordinates and scales of the detected objects and the associated feature vectors, which are refined by a transformer encoder and then used as inputs to a glimpse generator and a spatial transformer network to produce K object image layers and masks. A convolutional autoencoder is in charge of background reconstruction.

Figure 2 .

 2 Figure 2. Examples of segmentation predictions on CLEVRTEX, CLEVR, ShapeStacks, ObjectsRoom, OOD and CAMO test datasets (Results on OOD and CAMO datasets are obtained using a model trained on CLEVRTEX only)

Table 1 .

 1 Benchmark results on CLEVR and CLEVRTEX. Results are shown (±σ) calculated over 3 runs. Source:[START_REF] Karazija | ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation[END_REF]

	Model	CLEVR ↑mIoU (%) ↑ARI-FG (%) ↓MSE	CLEVRTEX ↑mIoU (%) ↑ARI-FG (%) ↓ MSE
	SPAIR [10]	65.95 ± 4.02 77.13 ± 1.92	55± 10 0.00 ± 0.00 0.00 ± 0.00 1101 ± 2
	SPACE [30]	26.31 ± 12.93 22.75 ± 14.04	63± 3 9.14 ± 3.46 17.53 ± 4.13 298 ± 80
	GNM [23]	59.92 ± 3.72 65.05 ± 4.19	43± 3 42.25 ± 0.18 53.37 ± 0.67 383 ± 2
	MN [36]	56.81 ± 0.40 72.12 ± 0.64	75± 1 10.46 ± 0.10 38.31 ± 0.70 335 ± 1
	DTI [32]	48.74 ± 2.17 89.54 ± 1.44	77± 12 33.79 ± 1.30 79.90 ± 1.37 438 ± 22
	Gen-V2 [15]	9.48 ± 0.55 57.90 ± 20.38 158± 2 7.93 ± 1.53 31.19 ± 12.41 315 ± 106
	eMORL [14]	50.19 ± 22.56 93.25 ± 3.24	33± 8 12.58 ± 2.39 45.00 ± 7.77 318 ± 43
	MONet [4]	30.66 ± 14.87 54.47 ± 11.41	58± 12 19.78 ± 1.02 36.66 ± 0.87 146 ± 7
	SA [31]	36.61 ± 24.83 95.89 ± 2.37	23± 3 22.58 ± 2.07 62.40 ± 2.23 254 ± 8
	IODINE [19]	45.14 ± 17.85 93.81 ± 0.76	44± 9 29.17 ± 0.75 59.52 ± 2.20 340 ± 3
	AST-Seg-B3-BT 71.92 ± 32.94 76.05 ± 36.13	51± 63 57.30 ± 15.72 71.79 ± 22.88 152 ± 39
	AST-Seg-B3-CT 90.27 ± 0.20 98.26 ± 0.07	16± 1 79.58 ± 0.54 94.77 ± 0.51 139 ± 7

Table 2 .

 2 Benchmark results on ObjectsRoom and ShapeStacks. Source:[START_REF] Engelcke | GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement[END_REF].

	Model	ObjectsRoom ↑ARI-FG (%) ↑MSC-FG (%) ↑mIoU (%)	↓MSE	ShapeStacks ↑ARI-FG (%) ↑MSC-FG (%) ↑mIoU (%)	↓MSE
	MONet-g [4, 15]	54 ± 0	33 ± 1	n/a	n/a	70 ± 4	57 ± 12	n/a	n/a
	Gen-v2 [15]	84 ± 1	58 ± 3	n/a	n/a	81 ± 0	68 ± 1	n/a	n/a
	SA [31]	79 ± 2	64 ± 13	n/a	n/a	76 ± 1	70 ± 5	n/a	n/a
	AST-Seg-B3-BT 74.96 ± 10.02	69.86 ± 10.13	74.50± 8.61 11.7 ± 2.1	73.77 ± 7.56	74.12 ± 8.63	70.18±12.68 11.8 ± 7.0
	AST-Seg-B3-CT 87.23 ± 0.88	82.22 ± 0.96	85.02± 0.79	6.7 ± 0.9	79.34 ± 0.73	77.65 ± 1.3	78.84± 0.21	4.5 ± 0.2
		CLEVRTEX		CLEVR			ShapeStacks
	input image
	ground truth
	segmentation
	image
	reconstruction
	predicted
	segmentation
		ObjectsRoom		OOD			CAMO
	input image
	ground truth
	segmentation
	image
	reconstruction
	predicted
	segmentation

Table 3 .

 3 Benchmark generalization results on CAMO, and OOD for a model trained on CLEVRTEX. Results are shown (±σ) calculated over 3 runs. Source:[START_REF] Karazija | ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation[END_REF]

	Model	OOD ↑mIoU (%) ↑ARI-FG (%) ↓MSE	CAMO ↑mIoU (%) ↑ARI-FG (%) ↓MSE
	SPAIR [10]	0.00 ± 0.00 0.00 ± 0.00 1166± 5 0.00 ± 0.00 0.00 ± 0.00 668± 3
	SPACE [30]	6.87 ± 3.32 12.71 ± 3.44 387± 66 8.67 ± 3.50 10.55 ± 2.09 251± 61
	GNM [23]	40.84 ± 0.30 48.43 ± 0.86 626± 5 17.56 ± 0.74 15.73 ± 0.89 353± 1
	MN [36]	12.13 ± 0.19 37.29 ± 1.04 409± 3 8.79 ± 0.15 31.52 ± 0.87 265± 1
	DTI [32]	32.55 ± 1.08 73.67 ± 0.98 590± 4 27.54 ± 1.55 72.90 ± 1.89 377± 17
	Gen-V2 [15]	8.74 ± 1.64 29.04 ± 11.23 539±147 7.49 ± 1.67 29.60 ± 12.84 278± 75
	eMORL [14]	13.17 ± 2.58 43.13 ± 9.28 471± 51 11.56 ± 2.09 42.34 ± 7.19 269± 31
	MONet [4]	19.30 ± 0.37 32.97 ± 1.00 231± 7 10.52 ± 0.38 12.44 ± 0.73 112± 7
	SA [31]	20.98 ± 1.59 58.45 ± 1.87 487± 16 19.83 ± 1.41 57.54 ± 1.01 215± 7
	IODINE [19]	26.28 ± 0.85 53.20 ± 2.55 504± 3 17.52 ± 0.75 36.31 ± 2.57 315± 3
	AST-Seg-B3-CT 67.50 ± 0.75 83.14 ± 0.75 832± 24 73.07 ± 0.65 87.27 ± 3.78 145± 6

Table 4 .

 4 Results of ablation study and additional experiments (results over 1 run, except for starred values, which are averages over 3 runs)

	Dataset	CLEVRTEX	CLEVR	ShapeStacks	ObjectsRoom
		mIoU ARI-FG	mIoU ARI-FG	mIoU ARI-FG	mIoU ARI-FG
	full model AST-Seg-B3-CT (reference)	79.58 * 94.77 *	90.27 * 98.26 *	78.84 * 79.34 *	85.02 * 87.23 *
	model without transformer encoder	75.69	94.41	77.16	93.09	82.99 * 82.29 *	85.51 * 88.49 *
	K = 1 + maximum number of objects	79.11 * 94.78 *	91.03 * 98.17 *	78.87	80.05	82.90	86.45
	K = 2 × maximum number of objects	62.10	89.96	90.56	98.29	54.88	65.16	66.78	78.58
	using a Unet instead of Segformer feature generator 66.82	88.25	90.70	98.17	75.51	77.78	85.59	87.93
	random initialization of Segformer backbone	61.74	80.22	88.94	97.77	62.73	68.40	77.71	79.23
	training without pixel entropy loss	70.18	91.81	85.54	96.09	52.17	60.08	84.21	86.19
	training using frozen pretrained background model 75.30	95.31	81.46	98.29	55.06	66.24	85.82	87.78
	isotropic scaling	78.68	94.78					84.91	87.20
	anisotropic scaling			87.21	98.53	45.47	36.43

Table 5 .

 5 Training computation time with one Nvidia RTX 3090 GPU (curriculum training)

	Dataset	image size	background model pretraining (phase 1)	full model training (phase 2 & 3)
			number of iterations	training time	number of iterations training time
	CLEVRTEX	128 × 128	500000	57 h 47 mn	125000	16 h 00 mn
	CLEVR	128 × 128	2500	20 mn	125000	12 h 03 mn
	ObjectsRoom	64 × 64	500000	14 h 57 mn	125000	6 h 31 mn
	ShapeStacks	64 × 64	500000	14 h 20 mn	125000	6 h 22 mn

Table 6 .

 6 Hyperparameter values

	hyperparameter description

Acknowledgment We thank Sascha Hornauer for useful comments on the first draft of this paper.

Supplementary Material

Hyperparameter values

The hyperparameter values used for the proposed model are listed in Table 6.

Pseudo-code for objects encoder and decoder

The full encoding and rendering process is described in Algorithms 1 and 2.

Additional implementation details

The glimpse convolutional generator is described in Table 7.

Synthetic datasets and preprocessing codes were downloaded from the following public repositories:

• https://www.robots.ox.ac.uk/ ˜vgg/ data/clevrtex/

• https://ogroth.github.io/ shapestacks/

• https://github.com/deepmind/multi_ object_datasets

• https://github.com/applied-ai-lab/ genesis.

The Segformer pretrained weights were downloaded from the following link: https://huggingface.co/nvidia/mit-b3

The architecture of the U-net implemented for the ablation study is described in Table 8. It contains a sequence of downsample blocks which output feature maps of decreasing sizes, a center block which takes as input the feature map produced by the last downsample block, and upsample blocks, which take as input both the output of the previous upsample or center block and the feature map of the same size produced by corresponding downsample block.

• A downsample block is composed of a convolutional layer with stride 2 and kernel size 4, with batch normalization and CELU, followed by a residual convolutional layer with stride 1 and kernel size 3 with batch normalization and CELU.

end // computation of positions and feature vectors before transformer refinement

// transformer refinement of positions and feature vectors


Evelyne Lhoste

Assessing the effects of an experimental policy instrument: associations as intermediary agents?

• what do grant winners do ? What objects do they construct ?

 Hypothesis :

• promote and facilitate interactions between society and sciences through networks, objects and devices (intermediations)

 Methodology :

Conclusion

• Conceptual framework to understand research intermediations in participatory research processes.

• The award-winning associations lay at the border between the worlds of research, third sector, and public authorities. They accomplish a "boundary work" to address the challenge related to the complexity of building an epistemic community, legitimize experiential knowledge, and contribute to acknowledge of their role as institutional entrepreneurs.

Introduction

  This policy instrument = Fonjep-Recherche  Finances participation of associations to research and aims at developing a culture of research and innovation.  Third sector organisations : • Civil society organisations • Associations (80%) • Structured in networks = leader  Public aid rooted in two major strands of State policies :  1. Development of Social Economy Act (2014),  2. Youth and Popular Education Act (1964); a State label "youth and popular education" (JEP) + a Fund (Fonjep) = delegate public parascolar socio-education services to associations. Research question :