Gaétan Bahl

Florent Lafarge

SCANNER NEURAL NETWORK FOR ON-BOARD SEGMENTATION OF SATELLITE IMAGES

Keywords: Image Segmentation, Satellite Imagery, On-board processing, Recurrent Convolutional Network, ConvLSTM, Cloud Segmentation

Traditional Convolutional Neural Networks (CNN) for semantic segmentation of images use 2D convolution operations. While the spatial inductive bias of 2D convolutions allow CNNs to build hierarchical feature representations, they require that the whole feature maps are kept in memory until the end of the inference. This is not ideal for memory and latency-critical applications such as real-time on-board satellite image segmentation. In this paper, we propose a new neural network architecture for semantic segmentation, "Scan-nerNet", based on a Recurrent 1D Convolutional architecture. Our network performs a segmentation of the input image lineby-line, and thus reduces the memory footprint and output latency. These characteristics make it ideal for on-the-fly segmentation of images on-board satellites equipped with push broom sensors such as Landsat 8, or satellites with limited compute capabilities, such as Cubesats. We perform cloud segmentation experiments on embedded hardware and show that our method offers a good compromise between accuracy, memory usage and latency.

INTRODUCTION

Over the past decade, thanks to the increase in computing power and wide availability of data, Convolutional Neural Networks [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] have gradually imposed themselves as the de facto solution for computer vision tasks such as image classification [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], object detection [START_REF] Girshick | Fast r-cnn[END_REF] and image segmentation [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. Image segmentation is the task of assigning a label to every single pixel of an input image. Thus, the output segmentation maps have the same resolution as the input images. This task is not only crucial in remote sensing applications such as cloud segmentation [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF][START_REF] Férésin | In space image processing using ai embedded on system on module: example of ops-sat cloud segmentation[END_REF], land cover estimation [START_REF] Xu | Advanced multi-sensor optical remote sensing for urban land use and land cover classification: Outcome of the 2018 ieee grss data fusion contest[END_REF], or road mapping [START_REF] Bahl | Road extraction from overhead images with graph neural networks[END_REF], but also in other domains, such as medical imagery [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] and self-driving [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF].

As a consequence of the increasing demand for edge processing, techniques such as distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] and quantization Our architecture allows a 280x reduction in memory usage compared to previous works on cloud segmentation at a similar number of parameters. [START_REF] Hubara | Quantized neural networks: Training neural networks with low precision weights and activations[END_REF] have been used as a way to port deep learning models onto low-power devices like UAVs, smartphones or satellites. However, semantic segmentation models remain very compute intensive, and dedicated architectures with a low number of convolution filters had to be developed in conjunction with these techniques in order to be used on low-power satellites [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF][START_REF] Férésin | In space image processing using ai embedded on system on module: example of ops-sat cloud segmentation[END_REF]. Moreover, many earth observation satellites feature push broom sensors. These sensors acquire images of varying height line-by-line (i.e. by "scanning" the ground). In cases where the acquisition runs for a long time, the images created in this way can become too large to be efficiently processed in memory by traditional 2D convolutional neural networks. Finally, since CNNs traditionally process the whole image at once, there is a significant latency between the acquisition and the output of the result, since we have to wait until the whole image is acquired and processed before viewing the result.

In order to alleviate these issues, we propose a semantic segmentation method based on a one-dimensional Convolutional LSTM [START_REF] Hochreiter | Long shortterm memory[END_REF][START_REF] Shi Xingjian | Convolutional lstm network: A machine learning approach for precipitation nowcasting[END_REF] (ConvLSTM), neural network. Our network parses the input image line-by-line, and outputs the result at the same time, thus only keeping the necessary information in memory and achieving a very low latency.

METHOD

In this section, we provide a description of our architecture, shown on Figure 1. Our architecture is based on a two-layer 1D Convolutional LSTM (ConvLSTM), inspired by the work of Shi et al. [START_REF] Shi Xingjian | Convolutional lstm network: A machine learning approach for precipitation nowcasting[END_REF] on 2D ConvLSTM. We implement the Con-vLSTM cells without self-loops as in [START_REF] Hochreiter | Long shortterm memory[END_REF]:

i t = σ(Conv1D([x t ; h t-1 ], W i ) + b i ) f t = σ(Conv1D([x t ; h t-1 ], W f ) + b f ) o t = σ(Conv1D([x t ; h t-1 ], W o ) + b o ) g t = tanh(Conv1D([x t ; h t-1 ], W g ) + b g ) c t = f t • c t-1 + i t • g t h t = tanh(c t ) • o t (1)
where • is the Hadamard product, σ is the sigmoid function, [x t ; h t-1 ] is the concatenation of the input line x t at step t and previous hidden state of the cell h t-1 . We notice that all the convolution operations in Equation 1 can be optimized by combining them into a single 1D convolution with weights

W = [W i ; W f ; W o ; W g ], b = [b i ; b f ; b o ; b g ] and out- put y = [i t ; f t ; o t ; g t ].
We denote as F 1 and F 2 the number of feature maps of the hidden states in layer 1 and 2, respectively. The final output line y t is generated by a 1x1 convolution that maps the F 2 feature maps of the hidden state of layer 2 h t,2 to a single line of the segmentation map, as shown on Figure 1.

In order to obtain results for two different total numbers of parameters and thus easily compare our networks to previous works, we create two networks with varying F 1 , F 2 and filter size. Table 1 shows the details of these two networks.

Network

EXPERIMENTS

In order to validate our design, we perform a cloud segmentation experiment. On-board cloud segmentation is an important topic as it can be the first step in the on-board compression of satellite images. Indeed, a large portion of the Earth is covered by clouds at any given time, thus the removal of cloud pixels translates to large savings in both bandwidth and storage space. We compare our results to the ones from [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF], who also worked on lightweight networks for on-board cloud segmentation.

For this experiment, we use the 38-Cloud dataset [START_REF] Mohajerani | Cloud-net: An end-toend cloud detection algorithm for landsat 8 imagery[END_REF] and implement our architecture in the PyTorch 1. networks shown in [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF] in this framework, in order to ensure a fair comparison. As done in [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF], we only use the RGB bands of the input images. We also implement a non-recurrent 1D convolutional baseline network with a similar number of parameters as our architecture. The 38-Cloud dataset is composed of 384x384 pixel crops from Landsat 8 scenes. We use 15% of the training set as a validation set. All networks are trained until convergence for a maximum of 1000 epochs. We use a batch size of 64 and the Adam optimizer for all networks. Training was done on a machine with an AMD Ryzen 9 3900X CPU, Nvidia GeForce RTX 3090 GPU and 64GB of DDR4 RAM.

In Table 2, we report the number of parameters used by our networks, and the memory required for a single forward pass of each network, as reported by the torchinfo module. We observe that our architecture enables some significant memory savings compared to the already memory efficient architectures presented in [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF]. Indeed, at a similar number of parameters as C-FCN [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF], we obtain a 280x reduction in memory usage. 3. Quantitative results of our architecture on the 38-Cloud dataset [START_REF] Mohajerani | Cloud-net: An end-toend cloud detection algorithm for landsat 8 imagery[END_REF] at a detection threshold of 0.5, compared to networks from [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF]. Our method obtains similar metrics at similar numbers of parameters, while using less memory and having less latency.

Network

Table 3 and Figure 2 show quantitative results using commonly used metrics (see definitions in [START_REF] Mohajerani | Cloud-net: An end-toend cloud detection algorithm for landsat 8 imagery[END_REF]). We observe that our ScannerNet Small network obtains results similar to the ones of C-FCN, even though its memory usage is much lower. Our bigger ScannerNet obtains slightly lower scores than C-UNet++, also while consuming less memory for its forward pass. The baseline 1D convolutional network (without LSTM) obtains much worse results, even though it has a similar number of parameters as ScannerNet Small, which shows that the recurrent part of our architecture is necessary to obtain a decent segmentation.

Figure 3 shows a selection of qualitative results taken from the 38-Cloud test set. We observe that even though it can only use one line of input at any step to generate the output segmentation result, our network manages to generate coherent results that match the output of C-UNet++ [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF]. Moreoever, we can observe some block artifacts in the output of C-UNet++ that do not exist in the output of ScannerNet. Our intuition is that since ScannerNet performs all processing at the same resolution as the input image, it can output a more refined segmentation than C-UNet++.

Next, we use a low-power edge device to measure the latency of our neural networks. In our context, we define the latency as the delay between the start of the processing and the output of the first pixel of the segmentation map, without including data loading time. For our ScannerNet architecture, since the processing is done line-by-line, the latency is thus measured on the computation of a single line of segmentation. For other neural networks, since the processing is done on the whole image at once, the latency is measured on the computation of the whole segmentation.

As our test system, we use an Nvidia Jetson Nano development kit with 2GB of memory, since it is a very popular platform for embedded systems. It features an SoC equipped with a quad-core A57 ARM CPU and a 128-core Nvidia Maxwell GPU, consuming up to 10W. For benchmarking, we lock the clock of the SoC to its maximum using the jetson clocks utility, and cool the board with a Noctua NF-A4x20 fan running at maximum speed. We run each test 500 times in a row and discard the first 10 runs. We use the same software configuration as the one used for training. We test each network on both the CPU and the GPU to take more usage scenarios into account (e.g. systems without GPUs). The results of this latency testing are shown in Table 4. We observe a significant reduction in latency compared to previous works. In particular, our small ScannerNet offers a 3.7x reduction in latency compared to C-FCN [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF], with a similar number of parameters. The improvement is even larger when comparing to C-UNet++. We notice a bigger improvement when running on the CPU than on the GPU. Our intuition is that since GPU architectures are more efficient when asked to process a large amount of data at once, our architecture does not offer as much benefit as on the CPU. Nevertheless, the reduction in latency is still significant.

Network

Latency (CPU) Latency (GPU) C-UNet++ [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF] 638.6 27.9 C-FCN [START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF] 296.

CONCLUSION

In this paper, we have presented ScannerNet, a 1D Recurrent Convolutional neural network architecture with very low memory usage and latency for on-board segmentation of satellite images. We have shown that our networks obtain results comparable to previous works in a cloud segmentation task while occupying 286 times less memory and having a lower latency at a similar number of parameters. We have observed real-world gains on a low-power device, with a reduction of the latency by a factor of 2.4 in the worst case.

Our approach is orthogonal to previous works on compact architectures and quantization of neural network weights. In particular, our networks could be further optimized by using integer or binary weights. We have also shown that 1D convolutions can work just as well as 2D convolutions for image segmentation, a technique that is seldom explored in the literature. We hope to pave the way for more varied neural network architecture designs adapted to different kinds of onboard sensors. Future work includes testing our architecture on FPGA chips and images from push broom sensors.

Fig. 1 .

 1 Fig. 1. Our 1D Convolutional LSTM Neural Network, Scan-nerNet, segments the input images line-by-line to save memory and reduce latency. We use two 1D ConvLSTM [12] layers and a single 1x1 convolution to obtain a binary output. Our architecture allows a 280x reduction in memory usage compared to previous works on cloud segmentation at a similar number of parameters.

Fig. 2 .

 2 Fig.2. IoU (Intersection over Union) of segmentation output at a threshold of 0.5 with respect to memory usage (in Megabytes, log scale), compared to previous works. Our networks achieve IoU results similar to previous works, while having a comparatively very low memory usage.

Fig. 3 .

 3 Fig. 3. Qualitative results of ScannerNet (ours, middle) against C-UNet++ [5] (right) on a selection of images from the 38-Cloud test set. Our architecture performs well under a variety of scenarios, including images with snow backgrounds, even though it only processes one line of the input image at a time.

Table 1 .

 1 Number of filters, filter size and number of parameters for each of our two ConvLSTM networks. F 1 and F 2 are the number of filters in the first and second 1D ConvLSTM layer, respectively.

	ScannerNet	4	8	9	4 521
	ScannerNet Small 4	4	7	1 717

F 1 F 2 Filter size Params.

Table 2 .

 2 10 deep learning framework. We re-implement the C-FCN and C-UNet++ Number of parameters and memory usage (in Megabytes) for a forward pass of our architecture, compared to networks from[START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF]. Our networks allow significant memory savings during inference.

	Network	Param. Memory usage (MB)
	Baseline	1 409	0.08
	C-FCN [5]	1 471	28.62
	C-UNet++[5]	9 129	47.19
	ScannerNet Small (ours) 1 717	0.10
	ScannerNet (ours)	4 521	0.15

Table 4 .

 4 Average latency (milliseconds, lower is better) of our

		1	11.1
	ScannerNet (ours)	91.5	4.6
	ScannerNet Small (ours)	80.3	4.5

networks, measured on the CPU and GPU of a Jetson Nano development kit, compared to networks from

[START_REF] Bahl | Low-power neural networks for semantic segmentation of satellite images[END_REF]

, on 384x384 pixel images of the 38-Cloud dataset. Our networks provide a 3.7x reduction in latency at the same number of parameters (i.e. when comparing ScannerNet Small to C-FCN).

ACKNOWLEDGMENTS

This work is funded by the CIAR project at IRT Saint Exupéry. The authors are grateful to the OPAL infrastructure from Université Côte d'Azur for providing resources and support.