Heng Zhang
email: heng.zhang@irisa.fr

Elisa Fromont
email: elisa.fromont@irisa.fr

Sébastien Lefevre
email: sebastien.lefevre@irisa.fr

Low-cost Multispectral Scene Analysis with Modality Distillation

Despite its robust performance under various illumination conditions, multispectral scene analysis has not been widely deployed due to two strong practical limitations: 1) thermal cameras, especially high-resolution ones are much more expensive than conventional visible cameras; 2) the most commonly adopted multispectral architectures, twostream neural networks, nearly double the inference time of a regular mono-spectral model which makes them impractical in embedded environments. In this work, we aim to tackle these two limitations by proposing a novel knowledge distillation framework named Modality Distillation (MD). The proposed framework distils the knowledge from a high thermal resolution two-stream network with featurelevel fusion to a low thermal resolution one-stream network with image-level fusion. We show on different multispectral scene analysis benchmarks that our method can effectively allow the use of low-resolution thermal sensors with more compact one-stream networks.

Introduction

Vision applications such as autonomous driving or remote surveillance need to maintain high reliability under various conditions, such as insufficient illumination or adverse weather. These situations are challenging for systems using only visible cameras, which is why multispectral systems introduce additional thermal cameras to provide supplementary information. In particular, visible cameras provide visual details of colour and texture, while thermal cameras are sensitive to temperature changes, thus their contributions are complementary and their combination can en-sure reliable recognition performance round-the-clock.

Under the conventional settings of multispectral scene analysis, thermal cameras and visible ones must provide image pairs with identical perception fields and identical spatial resolution. The former requirement can be achieved through camera calibration. However, due to the extreme price gap between high-resolution visible and thermal cameras 1 , the requirement of identical spatial resolution usually leads to either 1) visible image downsampling that may cause information loss or 2) high manufacturing costs for thermal cameras that prevent massive production. From a practical point of view, using a high-resolution visible camera and a low-resolution thermal one would be the best compromise in performance/price.

Another constraint from the current multispectral systems lies in the software part. Nowadays, deep learningbased methods dominate the field of (multispectral) scene analysis. Multispectral information fusion methods can be categorized into: image-level fusion, feature-level fusion and decision-level fusion. Architectures that implement a feature-level fusion, usually within a two-stream network architecture (one dedicated to each source), have been proven to outperform the other strategies, and are currently the most studied in the literature [START_REF] Liu | Multispectral deep neural networks for pedestrian detection[END_REF][START_REF] Konig | Fully convolutional region proposal networks for multispectral person detection[END_REF][START_REF] Li | Multispectral pedestrian detection via simultaneous detection and segmentation[END_REF][START_REF] Zhang | Cross-modality interactive attention network for multispectral pedestrian detection[END_REF][START_REF] Zhang | Guided attentive feature fusion for multispectral pedestrian detection[END_REF][START_REF] Zhou | Improving multispectral pedestrian detection by addressing modality imbalance problems[END_REF][START_REF] Zhang | Multispectral fusion for object detection with cyclic fuse-and-refine blocks[END_REF][START_REF] Ha | Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multispectral scenes[END_REF][START_REF] Sun | Rtfnet: Rgbthermal fusion network for semantic segmentation of urban scenes[END_REF][START_REF] Hazirbas | Fusenet: Incorporating depth into semantic seg-mentation via fusion-based cnn architecture[END_REF]. However, the computational overhead provided by two-stream networks is huge, which is particularly undesirable for software deployment on embedded devices.

In this paper, we propose a novel knowledge distillation framework named Modality Distillation (MD) to tackle the aforementioned hardware and software constraints. This framework follows two steps: Firstly, a multispectral system with high-resolution visible and thermal cameras is used to collect training data and to learn a precise but complex two-stream neural network for scene analysis. This model will be used as a teacher model with fixed weights. Secondly, a more efficient image-level fusion student model is trained with high-resolution visible images and downsampled thermal images to simulate production systems that are equipped with more economical low-resolution thermal cameras. The knowledge from the teacher model is transferred to the student model to mimic the more accurate feature-level fusion architecture and to reconstruct highresolution thermal details. We performed extensive experiments for multispectral pedestrian detection [START_REF] Hwang | Multispectral pedestrian detection: Benchmark dataset and baseline[END_REF] and semantic segmentation [START_REF] Ha | Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multispectral scenes[END_REF]. In both tasks, our model strongly reduces thermal camera requirements (resolution divided by 16) and achieves substantial inference acceleration (runtime divided by, at least, 1.8), with minor precision drop compared to full-resolution two-stream teacher networks which makes the deployment of multiple low-cost student networks on embedded devices much more viable.

The rest of this paper is organized as follows: In Section 2, we provide some representative work on multispectral scene analysis and knowledge distillation. Section 3 details the proposed method. In section 4, we conduct various experiments to study the effects of MD under different thermal resolutions and compare our results to state-of-the-art methods. Section 5 concludes the paper.

Related work

Our review of the related work mainly focuses on multispectral scene analysis and knowledge distillation, the two critical techniques to build the proposed framework.

Multispectral scene analysis

The first dataset for pedestrian detection from visiblethermal image pairs was introduced in [START_REF] Hwang | Multispectral pedestrian detection: Benchmark dataset and baseline[END_REF] and then various deep learning-based methods have been proposed to tackle this problem. In [START_REF] Wagner | Multispectral pedestrian detection using deep fusion convolutional neural networks[END_REF], the authors compared, on this dataset, two different fusion strategies: image-level fusion (called early fusion) and feature-level fusion (there called late fusion because it is at the last possible feature level). Early fusion combines the information from the two modalities by directly concatenating visible and thermal images. Late fusion methods usually apply a two-stream architecture which employs two separate feature extraction networks (for visible and thermal images respectively) and combine the multispectral information by feature concatenation. [START_REF] Wagner | Multispectral pedestrian detection using deep fusion convolutional neural networks[END_REF] concluded that feature-level fusion methods produce superior performance, whereas image-level fusion ones cannot even surpass traditional methods (such as Aggregated Channel Features (ACF) [START_REF] Dollár | Fast feature pyramids for object detection[END_REF]). To the best of our knowledge, worldwide research on image-level fusion for multispectral image analysis has been mostly interrupted since these findings. The research focus has then shifted to feature-level fusion: [START_REF] Liu | Multispectral deep neural networks for pedestrian detection[END_REF][START_REF] Konig | Fully convolutional region proposal networks for multispectral person detection[END_REF] studied the optimal fusion "timing" in the detection network and came to the same conclusion that halfway feature fusion produces better results; [START_REF] Li | Multispectral pedestrian detection via simultaneous detection and segmentation[END_REF] introduced an auxiliary segmentation task on the basis of halfway feature fusion for further performance improvements. [START_REF] Zhang | Cross-modality interactive attention network for multispectral pedestrian detection[END_REF][START_REF] Zhang | Guided attentive feature fusion for multispectral pedestrian detection[END_REF] applied attention mechanisms to adaptively weigh the visible and thermal features in the feature fusion stage; [START_REF] Zhang | Multispectral fusion for object detection with cyclic fuse-and-refine blocks[END_REF][START_REF] Zhou | Improving multispectral pedestrian detection by addressing modality imbalance problems[END_REF] alleviated the inconsistency between visible and thermal features to facilitate the optimization process of a dualmodality network. Apart from these studies on featurelevel fusion, multiple decision-level fusion methods were suggested: [START_REF] Guan | Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection[END_REF][START_REF] Li | Illumination-aware faster r-cnn for robust multispectral pedestrian detection[END_REF] used illumination information to guide the fusion of predictions (decisions) from visible/thermal images or from day/night sub-networks; [START_REF] Zhang | Weakly aligned cross-modal learning for multispectral pedestrian detection[END_REF] discussed a confidence-aware fusion mechanism, where the disagreement between visible and thermal predictions is used to reweigh visible contributions, which could also be regarded as a decision-level fusion approach.

Compared to multispectral pedestrian detection, multispectral semantic segmentation is a younger research topic, and most related methods are based on feature-level fusion. Some relevant datasets and baseline methods were introduced in [START_REF] Ha | Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multispectral scenes[END_REF]. Similarly to other feature fusion methods, the baseline method also adopts two separate feature extractors for visible and thermal images respectively. Moreover, short-cut blocks were designed to concatenate the extracted multispectral feature maps. Based on this, [START_REF] Sun | Rtfnet: Rgbthermal fusion network for semantic segmentation of urban scenes[END_REF] adopted stronger feature extraction networks to further boost segmentation accuracy. However, because of this complexity, the inference speed of their models was much slower. On a different type of data, [START_REF] Hazirbas | Fusenet: Incorporating depth into semantic seg-mentation via fusion-based cnn architecture[END_REF] tackled the RGB-D semantic segmentation task, the architecture of which is also applicable for visible-thermal inputs. Therefore, we also include this model for comparison in our experiments.

Previous studies on multispectral scene analysis aimed to improve detection/segmentation accuracy, neglecting the computational cost and the effectiveness of operational deployment. In this paper, we rather exploit the (forgotten) potentialities of image-level fusion architectures, which have a similar complexity as mono-spectral networks. Moreover, we also deal with the issue of reducing thermal camera resolution, which has never been discussed in the literature despite the great interest in practical multispectral applications.

Knowledge distillation

Knowledge distillation is a method for inheriting the knowledge learnt from one or multiple pre-trained teacher models to a student model. This concept was firstly introduced in [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF], where the training objective of a student model is the prediction produced by the teacher models. The use of this method strongly improved the accuracy of the student model. [START_REF] Romero | Fitnets: Hints for thin deep nets[END_REF] proposed to distil features instead of predictions. Concretely, features from internal layers of a student model should mimic that of a teacher model. [START_REF] Komodakis | Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer[END_REF] suggested that directly parsing features as guidance might be difficult, therefore, they transferred attention maps (generated in an unsupervised manner) that function as a summary of the whole feature maps.

Apart from image classification, knowledge distillation is also widely used in other scene analysis tasks. [START_REF] Chen | Learning efficient object detection models with knowledge distillation[END_REF] proposed the first knowledge distillation framework for object detection, which includes a distillation loss on both the feature network and the detection head. Instead of distilling all feature maps, [START_REF] Li | Mimicking very efficient network for object detection[END_REF][START_REF] Wang | Distilling object detectors with fine-grained feature imitation[END_REF] applied knowledge distillation only on ROI-sampled features or features near object bounding boxes to alleviate the extreme imbalance between foreground and background. In terms of semantic segmentation, [START_REF] Xie | Improving fast segmentation with teacherstudent learning[END_REF] distilled both the fine annotated images and unlabelled auxiliary data to regularize the training of a student segmentation model; [START_REF] Liu | Structured knowledge distillation for semantic segmentation[END_REF] proposed to distil the structured knowledge for a semantic segmentation task.

We take inspiration from previous work and propose two specific knowledge distillation methods for multispectral scene analysis: Attention transfer and Semantic transfer. The former generates teacher attention maps (masks) via Guided Attentive Feature Fusion (GAFF) [START_REF] Zhang | Guided attentive feature fusion for multispectral pedestrian detection[END_REF] from a two-stream teacher network. These masks are then transferred to a one-stream student network (thus performing an image-level fusion). The latter tackles the imbalance problem in feature distillation through a novel "Focal Mean Square Error" loss. Moreover, contrary to previous works where the objective is to transfer knowledge from a "larger" teacher network into a "smaller" student network (e.g., from ResNet-101 to ResNet-18), our objective is instead to transfer from a high thermal resolution two-stream multispectral network into a low thermal resolution one-stream multispectral network, while the "base" network remains unchanged (e.g., we use ResNet-18 for all experiments).

Modality Distillation

This section starts with an overview of the proposed Modality Distillation (MD) framework. We then briefly provide the basic concepts of [START_REF] Zhang | Guided attentive feature fusion for multispectral pedestrian detection[END_REF] that are used in our framework. Finally, the two proposed knowledge distillation modules are described in details.

Overview

As illustrated in Fig. 1(A works, a GAFF [START_REF] Zhang | Guided attentive feature fusion for multispectral pedestrian detection[END_REF] module for multispectral fusion and a task-specific network for pedestrian detection/semantic segmentation. Contrarily to the teacher model, the student model uses a low-resolution thermal input and a onestream feature extraction network that takes as input the image-level fusion of both modalities (through different input channels). We also conduct distillation experiments for a student model without the thermal modality, i.e., in this particular case, we attempt to use a multispectral teacher to improve the performance of a visible-only student.

The proposed MD framework includes two training stages. In the first stage, we train the teacher model and fix its weights, such that the fused features from GAFF module contain the rich semantics of high-resolution thermalvisible image pairs. These features are used to guide the training of the student model; In the second stage, the optimization of the student model is supervised by a taskspecific loss (e.g., pedestrian detection or semantic segmentation loss) as well as the knowledge transfer loss. The objective of the knowledge transfer loss is two-fold: using a more efficient one-stream network to mimic a more precise two-stream network and using the more available lowresolution thermal images to reconstruct high-resolution thermal details. Finally, we obtain a student model that takes low-resolution thermal images as input, and the required parameters and calculations are greatly reduced. Meanwhile, the precision of the low thermal resolution onestream student model is supposed to be close to the high thermal resolution two-stream teacher model.

Guided attentive feature fusion

We carefully follow the implementations of GAFF module from [START_REF] Zhang | Guided attentive feature fusion for multispectral pedestrian detection[END_REF] for multispectral feature fusion in the teacher model. GAFF learns the intra-and inter-modality attention masks to adaptively enhance important areas and to identify reliable modalities, respectively. Intra-modality attention is used to distinguish the foreground and background for each individual modality, and the inter-modality attention is used to select features among visible and thermal modalities according to the dynamic comparison of their prediction quality of the former mask. 2 . The weighted visible and thermal features are obtained via:

f visible weighted = f visible ⊗ (1 + m visible intra ) ⊗ (1 + m visible inter ) f thermal weighted = f thermal ⊗ (1 + m thermal intra ) ⊗ (1 + m thermal inter ) (1)
where f visible and f thermal denote features from visible and thermal feature extraction branches. m intra and m inter are predicted intra-and inter-modality attention masks from GAFF. Their superscript indicates the modality.

The fusion of the weighted visible and thermal features are assigned as the teacher features:

f teacher = f visible weighted + f thermal weighted 2 (2)

Knowledge transfer modules

To preserve the knowledge learnt from the teacher model to the maximum extent, we apply two knowledge transfer strategies: Attention transfer that guides the one-stream model to mimic the two-stream attentive fusion, and Semantic transfer that rebuilds high-resolution visual details from low-resolution thermal images.

Attention transfer. GAFF significantly improves the scene analysis performance in a two-stream teacher model. However, such a multispectral feature fusion module does not exist in a one-stream student model. Thus, as illustrated in the left part of Fig. 1(B), we design the Attention transfer module to simulate this attentive fusion in a one-stream model. The teacher attention mask is the combination of intra-and inter-modality attention masks. To keep the architecture simple, the student attention mask is generated by a 1 × 1 convolution followed by a Sigmoid activation, and is supervised by minimizing the loss [START_REF] Lee R Dice | Measures of the amount of ecologic association between species[END_REF] between 2 Due to space constraint, we refer the reader to [START_REF] Zhang | Guided attentive feature fusion for multispectral pedestrian detection[END_REF] for more details.

the student and teacher attention masks:

m teacher = m visible intra ⊗ m visible inter + m thermal intra ⊗ m thermal inter m student = F(f student ) L attention = 1 - 2|m student ⊗ m teacher | |m student | + |m teacher | (3
) where L attention denotes the Attention transfer loss; m teacher and m student represent the teacher and student attention masks respectively; f student denotes the student features acquired from the joint feature branch; F represents a 1 × 1 convolution followed by a Sigmoid activation; ⊗ and || represent respectively the pixel-wise multiplication and summation operation.

Semantic transfer. To compensate for the resolution reduction of thermal input images, the Semantic transfer module performs an implicit super-resolution of student feature maps. As shown in the right part of Fig. 1(B), we use a basic residual block [START_REF] He | Deep residual learning for image recognition[END_REF] to increase the details in the joint features. Semantic transfer aims to minimize the distance between the student (joint) and teacher (fused) feature maps. However, optimizing this distance has proven to be difficult. At first glance, this is due to the extreme imbalance between the foreground and background areas [START_REF] Li | Mimicking very efficient network for object detection[END_REF][START_REF] Wang | Distilling object detectors with fine-grained feature imitation[END_REF]. Inspired by the Focal loss [START_REF] Lin | Focal loss for dense object detection[END_REF], we argue that the true problem lies in the extreme imbalance between easily-mimic and hardly-mimic areas. Therefore, we propose the Focal Mean Square Error (F-MSE) loss defined as:

d = (f student -f teacher ) 2 L semantic = w h 1 n (δ( n d) × n d) (4)
where L semantic denotes the Semantic transfer loss; d is the squared L2 distance between student and teacher feature maps. δ signifies the Softmax function; w, h, n represent the width, height and depth of feature maps, respectively. The major difference between the proposed F-MSE loss and the standard MSE loss (used in [START_REF] Romero | Fitnets: Hints for thin deep nets[END_REF]) is the spatial reweighting based on feature-mimicking errors. Concretely, the Softmax function generates a 2-D re-weighting mask, where each value reflects the difficulty of feature mimicking on a specific area, and the summation of all values on the mask is equal to 1. In such a manner, the optimization adaptively "focuses" on mis-predicted areas, and the imbalance problem is therefore solved.

Experiments

In this section, we conduct experiments on two multispectral benchmarks [START_REF] Hwang | Multispectral pedestrian detection: Benchmark dataset and baseline[END_REF][START_REF] Ha | Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multispectral scenes[END_REF] and two scene analysis tasks (pedestrian detection and semantic segmentation) to evaluate the effectiveness of the proposed Modality Distillation (MD) framework.

Datasets

KAIST. The KAIST multispectral pedestrian detection dataset [START_REF] Hwang | Multispectral pedestrian detection: Benchmark dataset and baseline[END_REF] (denoted as KAIST Dataset) focuses on the pedestrian detection task based on aligned multispectral image pairs. These image pairs are collected during daytime and nighttime. This dataset contains 21,622 annotated image pairs for training, and 2,252 image pairs for testing. Due to the problematic annotations from the original dataset, we adopt the improved annotations proposed by [START_REF] Zhang | Weakly aligned cross-modal learning for multispectral pedestrian detection[END_REF] and [START_REF] Liu | Multispectral deep neural networks for pedestrian detection[END_REF] for training and evaluation, respectively. Following previous works, we adopt the log-average Miss Rate (computed by averaging the miss rate on false positive per-image points sampled within the range of 10 -2 , 10 0 , lower is better) under a "reasonable" setting [START_REF] Dollar | Pedestrian detection: An evaluation of the state of the art[END_REF] (pedestrians that are occluded or shorter than 55 pixels are eliminated) as the evaluation metric.

MFNet. The Multispectral semantic segmentation dataset [START_REF] Ha | Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multispectral scenes[END_REF] (denoted as MFNet Dataset) targets the semantic segmentation of street scenes for autonomous vehicles. The segmentation labels consist of eight classes: car, person, bike, curve, car stop, guardrail, colour cone and bump. It provides 1,568 aligned multispectral image pairs in the training set, 392 pairs in the validation set and 393 pairs in the test set. Among each subset, half of the image pairs are taken during daytime, and the other half during nighttime. To evaluate the segmentation accuracy, we report the class-wise Mean Accuracy, calculated by averaging the ratio between the number of true positive pixels and the sum of true positive and false negative pixels for each class.

Implementation details

Network architecture. For all experiments, we apply ResNet-18 [START_REF] He | Deep residual learning for image recognition[END_REF] as the feature extraction network, RetinaNet [START_REF] Lin | Focal loss for dense object detection[END_REF] (its label assignment is optimized via Mutual Guidance strategy [START_REF] Zhang | Localize to classify and classify to localize: Mutual guidance in object detection[END_REF]) as the pedestrian detection network and PSPNet [START_REF] Zhao | Pyramid scene parsing network[END_REF] as the semantic segmentation network. For the teacher model (Fig. 1(A) upper model), GAFF [START_REF] Zhang | Guided attentive feature fusion for multispectral pedestrian detection[END_REF] is adopted for the attentive fusion of visible and thermal features. For the student model (Fig. 1(A) lower model), visible and thermal images are concatenated to generate 6-channel multispectral inputs, i.e., 3 channels from each modality. The first convolution layer is modified to suit the 6-channel input 3 . Note that in lieu of generating 4-channel input as done in [START_REF] Wagner | Multispectral pedestrian detection using deep fusion convolutional neural networks[END_REF], we duplicate the single-channel ther- Training details. All models (including teacher and student models) are trained on a single GPU with 16 multispectral image pairs per mini-batch, and with an initial learning rate of 1e-2. To stabilize the training at the beginning, we adopt the warm-up strategy, where the learning rate is linearly increased from 1e-6 to 1e-2 within the first 500 iterations, then decayed with a cosine annealing. Pedestrian detection models are trained for 3,500 iterations, and semantic segmentation models are trained for 14,000 iterations. The whole project is coded in PyTorch 1.20. For fair runtime comparisons, all models' runtimes are measured on an Nvidia GTX 1080Ti GPU. We repeat each training 3 times with different random seed values and report the average performance as well as the standard error (only for the "all" setting because of the space restriction). The best results are shown in bold.

Experimental results

Baseline results. Image-level, feature-level and decisionlevel are the three major fusion methods for multispectral scene analysis. We list in Tab. 1 and 2 their prediction accuracy and inference time on KAIST Dataset [START_REF] Hwang | Multispectral pedestrian detection: Benchmark dataset and baseline[END_REF] and on MFNet Dataset [START_REF] Ha | Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multispectral scenes[END_REF], respectively. The visible-only results are also listed for reference. In order to fairly compare these fusion methods, we use the same feature network (ResNet-18 [START_REF] He | Deep residual learning for image recognition[END_REF]) and detection/segmentation network (RetinaNet [START_REF] Lin | Focal loss for dense object detection[END_REF]/PSPNet [START_REF] Zhao | Pyramid scene parsing network[END_REF]). Specifically, we adopt GAFF [START_REF] Zhang | Guided attentive feature fusion for multispectral pedestrian detection[END_REF] as the feature-level fusion method. For simplicity, we average the prediction from visible and thermal images for decision-level fusion. The tables show that, regardless of the information fusion stage (imagelevel, feature-level or decision-level), multispectral methods greatly improve the detection/segmentation accuracy compared to the visible-only model, especially for nighttime detection/segmentation. Feature-level and decisionlevel fusion methods almost double the execution runtimes (as well as the number of parameters, not reported here because of the space restriction) of a visible-only model. In contrast, the computational overhead for the image-level fusion is negligible, which shows the relevance of this fusion method when fewer computational resources are available.

Distillation results. We list in Tab. 3 and 4 the comparisons between native image-level fusion models and distilled image-level fusion models (i.e., student models) on KAIST Dataset [START_REF] Hwang | Multispectral pedestrian detection: Benchmark dataset and baseline[END_REF] and MFNet Dataset [START_REF] Ha | Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multispectral scenes[END_REF], respectively. It can be observed that MD strategy brings important improvements for all thermal resolutions for both datasets. Specifically, on the multispectral pedestrian detection task (Tab. 3), our full thermal resolution result with MD is already close to that of the feature-level fusion model (i.e., teacher model) shown in Table 1 (7.78% versus 7.77%), while the inference time is almost halved (10.97ms versus 6.55ms). When it comes to the most practical case where thermal resolution is 16 times lower than visible resolution, MD strategy brings 2.14% of Miss Rate improvement, and the performance difference compared to the teacher model is only 0.26% (8.03% versus 7.77%). We show some detection results from native model and distilled model for this practical case in Fig. 2, and it can be observed that our distilled model provides more precise detection results. In- terestingly, the nighttime detection precision is boosted by 27.06% (7.07% versus 34.13%) even if the thermal resolution is reduced to 80 × 64 (i.e., 64 times downsampled), proving the necessity of the thermal modality in nighttime detections. Moreover, our strategy remains helpful when the thermal image is completely removed (e.g., the Miss Rate for visible-only model is reduced from 22.84% to 21.08%). Here the multispectral knowledge from the teacher model allows the visible-only student to perform pseudo-multispectral detection, which is the main reason of improvements.

On the multispectral semantic segmentation task, the improvements are more important (around 5% for all thermal resolutions using MD). It is noteworthy that the performance of the distilled visible-only model is even better than that of the native full-resolution image-level fusion model (60.62% versus 59.42%). Here, our assumption is that the multispectral semantic segmentation task is more critical for the choice of fusion architecture, e.g., according to Tab. 2, native image-level fusion performs 4.03% worse than feature-level fusion. This may be the reason why rare previous work use image-level fusion for multispectral semantic segmentation. However, our MD strategy makes the student model mimic a feature-level fusion teacher model, which compensates its established disadvantage of imagelevel fusion architecture, and thus brings tremendous accuracy improvements. For the practical case (16 times thermal

Method

Miss Rate (%) Runtime Day Night All ACF [START_REF] Hwang | Multispectral pedestrian detection: Benchmark dataset and baseline[END_REF] 42.57 56.17 47.32 2730ms Halfway Fusion [START_REF] Liu | Multispectral deep neural networks for pedestrian detection[END_REF] resolution downsampling), the mean accuracy difference with the teacher model is minor (63.52% versus 63.45%).

We visualize in Fig. 3 the segmentation results from the native model and our distilled model for the practical case, and it could be noted that the improvement from MD strategy on segmentation quality is obvious.

Comparing with state-of-the-art. We compare the results of our distilled models (which adopt the more efficient image-level fusion) with state-of-the-art methods (all adopting the more cumbersome feature-level fusion) on KAIST Dataset (Tab. 5) and MFNet Dataset (Tab. 6). Note that our teacher models use [START_REF] Zhang | Guided attentive feature fusion for multispectral pedestrian detection[END_REF] and this method has already been shown to give better results than its competitors. However, our goal here is to show how our student models (which are supposed to be less good than their teachers) perform compared to their competitors. Specifically, we provide our onestream student models' results using full thermal resolution (same condition as our competitors, denoted as "full") and using 16 times downsampled thermal resolution (denoted as "practical"). We also list our two-stream teacher models' results (denoted as "teacher") for reference. We consider "practical" the most interesting setting for actual multispectral applications.

On the multispectral pedestrian detection task, the achieved Miss Rate from the distilled "practical" model is already better than that of the best feature-level fusion methods in the literature [START_REF] Zhou | Improving multispectral pedestrian detection by addressing modality imbalance problems[END_REF] (8.03% versus 8.13%). It should be noted that our "practical" model takes downsampled thermal images as input and adopts a much simpler architecture (one-stream networks for "full"/"practical" and two-stream networks for others). It is also worth noting that our nighttime detection performance surpasses all previous methods, which proves that the thermal information has been wellpreserved in the student model.

On the multispectral semantic segmentation task, thanks to the substantial accuracy improvements from MD (about 5%), our distilled "practical" model also surpasses the best previous result [START_REF] Sun | Rtfnet: Rgbthermal fusion network for semantic segmentation of urban scenes[END_REF] (63.5% versus 63.1%). For this dataset as well, all our trained models (including the "practical" model with downsampled thermal input) have obvious advantage in nighttime prediction. It should be pointed out that both our distilled models with one-stream ResNet-18 feature network even outperform RTFNet-152 [START_REF] Sun | Rtfnet: Rgbthermal fusion network for semantic segmentation of urban scenes[END_REF] with two-stream ResNet-152 feature network, demonstrating the high efficiency of our distilled models (ours are about 6 times faster than RTFNet-152). More surprisingly, we can see in Tab. 6 that the full student model gives slightly better results than the teacher model. The student model's feature extraction network is the same as the teacher's one, so the student could theoretically achieve similar performance, and the student model has more sources of supervision, i.e., the ground truth and the knowledge learnt from the teacher model, which we believe is the reason for the higher performance shown by the student model.

Ablation experiments. To explore the effects of the proposed Attention transfer and Semantic transfer modules, we conduct ablation experiments on KAIST Dataset (Tab. 7) and MFNet Dataset (Tab. 8), under the most practical case (thermal images are 16 times downsampled). The "S" and According to our experimental results, the latter provides better performance for both tasks. Moreover, we visualize some examples of the 2-D spatial re-weighting mask from F-MSE loss (Eq.4) in Fig. 4, and it can be observed that: 1) the imbalance between easily-mimic and hardly-mimic area is grave, where the former occupies most of a given image;

2) with F-MSE loss, the optimization on the feature mimicking is automatically "focused" on more important areas, e.g., pedestrians, vehicles and colour cones. This specific loss tackles the imbalance problem in the Semantic transfer module. In conclusion, according to our ablation experiments on two datasets, both the proposed Semantic transfer and Attention transfer modules bring notable improvements and their combination leads to the best performance.

Conclusion

In this paper, we identify the hardware and software constraints in today's multispectral scene analysis systems and propose a novel Modality Distillation framework to tackle these constraints. Specifically, this framework distils the knowledge from a high thermal resolution two-stream network with feature-level fusion to a low thermal resolution one-stream network with image-level fusion. The distilled model could perform prediction on widely available lowresolution thermal cameras and shows similar complexity with the mono-spectral models. In this framework, we present two knowledge transfer modules named Attention transfer and Semantic transfer specifically for multispectral learning. Extensive experiments for multispectral pedestrian detection and semantic segmentation demonstrate the efficiency of the proposed framework. In the future, we plan to extend our method for scene analysis with even more modalities such as depth sensor, Doppler radar, LiDAR, etc.

Figure 1 .

 1 Figure1. Overview of the proposed method (A) and details on knowledge transfer modules (B). Blue and orange blocks represent components from teacher and student models, respectively.

Figure 2 .

 2 Figure 2. Visual improvements on KAIST Dataset.

Figure 3 .

 3 Figure 3. Visual improvements on MFNet Dataset.

Figure 4 .

 4 Figure 4. Visualization of the visible-thermal image pairs and the 2-D spatial re-weighting from the proposed F-MSE loss. The first two lines of multispectral images pairs come from KAIST Dataset, and the last two lines come from MFNet Dataset.

Table 2 .

 2 Different fusion methods on MFNet Dataset. For fair comparisons, all listed methods use the same feature network (ResNet-18) and segmentation network (PSPNet).

	Fusion stage	Mean Accuracy (%) Day Night All	Runtime
	Visible-only	50.83 52.26 55.06±0.21	4.57ms
	Image-level	54.97 56.07 59.42±0.22	4.68ms
	Feature-level	57.21 62.18 63.45±0.24	8.94ms
	Decision-level 51.72 53.37 56.21±0.14	9.14ms
	Thermal resolution MD	Miss Rate (%) Day Night All
	Full resolution	✓	10.73 9.45	6.61 9.40±0.39 4.61 7.78±0.28
	4x downsample	✓	11.57 9.39	6.51 9.84±0.72 5.07 7.91±0.11
	16x downsample	✓	12.09 9.85	6.73 10.17±0.42 4.84 8.03±0.19
	64x downsample	14.92 10.66 13.37±0.30 ✓ 10.75 7.07 9.50±0.06

Visible-only 16.95 35.15 22.84±0.77 ✓ 14.74 34.13 21.08±0.21

Table 3 .

 3 Comparison between native models and distilled models on KAIST Dataset under different thermal resolution settings (from full thermal resolution to no thermal scenario). All listed methods use an image-level fusion architecture.

Table 4 .

 4 Comparison between native models and distilled models on MFNet Dataset under different thermal resolution settings (from full thermal resolution to no thermal scenario). All listed methods use an image-level fusion architecture.

	Thermal resolution MD	Mean Accuracy (%) Day Night All
	Full resolution	54.97 56.07 59.42±0.22 ✓ 59.71 62.78 64.93±0.11
	4x downsample	53.89 55.18 57.93±0.27 ✓ 58.32 61.88 64.25±0.11
	16x downsample	53.85 55.43 58.21±0.46 ✓ 58.46 61.37 63.52±0.87
	64x downsample	53.68 53.68 57.06±0.18 ✓ 57.58 59.67 62.62±0.81
	Visible-only	50.83 52.26 55.06±0.21 ✓ 57.74 56.67 60.62±0.42

Table 5 .

 5 Comparison between state-of-the-art multispectral pedestrian detection methods and ours on KAIST Dataset. Our competitors' results are taken from[START_REF] Zhou | Improving multispectral pedestrian detection by addressing modality imbalance problems[END_REF].

		24.88 26.59 25.75	430ms
	FusionRPN+BF [13] 19.57 16.27 18.29	800ms
	IAF R-CNN [15]	14.55 18.26 15.73	210ms
	IATDNN+IASS [6]	14.67 15.72 14.95	250ms
	RFA [28]	16.78 10.21 14.61	80ms
	CIAN [29]	14.77 11.13 14.12	70ms
	MSDS-RCNN [14]	10.53 12.94 11.34	220ms
	AR-CNN [30]	9.94	8.38 9.34	120ms
	MBNet [32]	8.28	7.86 8.13	70ms
	Ours (teacher)	9.37	4.71	7.77	11ms
	Ours (full)	9.45	4.61 7.78	7ms
	Ours (practical)	9.85	4.84 8.03	7ms
	Method	Mean Accuracy (%) Runtime Day Night All
	MFNet [7]	42.6	41.4 45.1	4.35ms
	FuseNet [8]	49.5	48.9 52.4	3.92ms
	RTFNet-50 [21]	57.3	59.4 62.2 11.25ms
	RTFNet-152 [21] 60.0	60.7 63.1 29.35ms
	Ours (teacher)	57.2	62.2 63.5	8.94ms
	Ours (full)	59.7	62.8 64.9	4.92ms
	Ours (practical)	57.6	61.4 63.5	4.92ms

Table 6 .

 6 Comparison between state-of-the-art multispectral semantic segmentation methods and ours on MFNet Dataset. Our competitors' results are taken from[START_REF] Sun | Rtfnet: Rgbthermal fusion network for semantic segmentation of urban scenes[END_REF].

Table 7 .

 7 Ablation experiments on KAIST Dataset. We study the effects of Semantic transfer (with MSE or F-MSE loss) and Attention transfer modules in the proposed MD framework.

	S(M) S(F) A	Miss Rate (%) Day Night All
			12.09	6.73 10.17±0.42
	✓		11.92	6.69 10.09±0.06
	✓		10.24	6.93	9.11±0.15
		✓ 10.61	5.83	8.99±0.23
	✓	✓	9.85	4.84	8.03±0.19

Table 8 .

 8 S(M) S(F) AMean Accuracy (%) Day Night All 53.85 55.43 58.21±0.46 ✓ 56.04 57.87 60.41±0.28 ✓ 57.49 58.47 61.16±0.13 ✓ 57.55 58.46 61.51±0.30 ✓ ✓ 57.58 61.37 63.52±0.87 Ablation experiments on MFNet Dataset. We study the effects of Semantic transfer (with MSE or F-MSE loss) and Attention transfer modules in the proposed MD framework.

	Visible image	Thermal image	Re-weighting mask

A typical thermal camera of resolution 640 × 480 could cost more than 8,000 USD. When the resolution is reduced to 80 × 60, the price becomes much more affordable (around

USD).

Concretely, the pretrained ImageNet[START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] weights for the first convolution layer are duplicated along the input channel dimension, and the values are halved.


Gregory Martin
email: gregory.martin@irisa.fr

Matthieu Donain

Elisa Fromont
email: elisa.fromont@irisa.fr

Tias Guns

Laurence Roze
email: laurence.roze@irisa.fr

Alexandre Termier
email: alexandre.termier@irisa.fr

Prediction-Based Fleet Relocation for Free Floating Car Sharing Services

The success of a free-floating car-sharing service depends on a good allocation of the vehicles across the city, i.e. where and when they are needed by citizens. This requires predicting the demand across the geographical regions and across time, which is challenging due to the sparsity and variability of the data. Furthermore, the purpose of these predictions is to help computing the best possible car positions for the next day, hence the need to model both the prediction task and the optimisation task in a compatible way. As the allocation optimisation involves reasoning about the number of cars to assign to geographical regions, we propose to predict the expected utilisation of a car when added to a region. We discuss the challenges in modeling both the machine learning and the relocation problem, and we propose a integer linear programming method that solves the relocation problem while taking into account the model predictions and relocation distances. We experiment with the datasets from a citywide car sharing company and show how our method can increase the allocation strategies and hence profitability of the service.

I. INTRODUCTION

Free floating car-sharing has emerged as a new public transportation method within urban areas. It contributes to reducing the cities' carbon footprint by allowing shared cars to be freely taken, used and dropped by customers within the city. Contrarily to one-way car-sharing services, in a free-floating service there is no predefined station: the cars can be taken and parked on any public standard parking place, usually within a single city and its surroundings.

Customers of car-sharing services prefer to have available cars in a close vicinity when needed. To this aim, the operators employ staff, called jockeys, to relocate vehicles inside the city. This is usually done empirically, mostly during the night and at a small scale, with a rough knowledge of high-demand areas. Several papers have discussed the lack of precise relocation strategies, either by designing methods an operator needs to apply [START_REF] Weikl | A practice-ready relocation model for free-floating carsharing systems with electric vehicles -Mesoscopic approach and field trial results[END_REF] or by designing incentives so that customers may consider ending their trips in areas selected by the operator [START_REF] Brendel | A Decision Support System for Computation of Carsharing Pricing Areas and its Influence on Vehicle Distribution[END_REF].

In this work, we aim to maximize the use of a service's car fleet in order to make the service sustainable and profitable for the operator. We consider the use of jockeys to relocate cars on a nightly basis, as the concrete daily action taken by car companies to increase shared cars usage. Hence, the goal is to propose a location for all the cars of the fleet at the beginning of the day, so as to maximize expected car use.

There exist multiple strategies to tackle this problem. We choose a two part strategy. First, we predict the expected utilization, in minutes, of cars when placed in defined areas of the city at the beginning of the day. Then, according to these predictions, we design a car relocation strategy to maximize the total expected utilization of the service in one day. The relocation problem is cast as an integer programming problem over the predictions, subject to constraints involved when wishing to re-allocating vehicles in practice.

Section II covers the relevant related work. We propose a general city modeling, a utility prediction strategy and a car assignment methodology in Section III. Both the car utilization prediction and the car relocation are evaluated in Section IV with a proprietary dataset coming from a real service.

II. RELATED WORK

Several works in the bike-sharing literature [START_REF] Raviv | Static repositioning in a bike-sharing system: models and solution approaches[END_REF][START_REF] Ghosh | Dynamic repositioning to reduce lost demand in bike sharing systems[END_REF][START_REF] Ghosh | Improving Customer Satisfaction in Bike Sharing Systems through Dynamic Repositioning[END_REF] are addressing the relocation problem. For example, in [START_REF] Raviv | Static repositioning in a bike-sharing system: models and solution approaches[END_REF] the authors jointly optimize the number of trips per day and some relocation costs (the total distance driven by the relocation trucks that can move multiple bikes per trips). In [START_REF] Ghosh | Dynamic repositioning to reduce lost demand in bike sharing systems[END_REF][START_REF] Ghosh | Improving Customer Satisfaction in Bike Sharing Systems through Dynamic Repositioning[END_REF] the authors improve this solution by proposing continuous relocations during the day while also taking into account the user demand at all times. Numerous relocation strategies have also been proposed for cars: in [START_REF] Zakaria | Car relocation for carsharing service: Comparison of CPLEX and greedy search[END_REF], the authors solve an integer linear programming (ILP) problem to optimize the utilization of cars in a one-way car-sharing service where relocations are done continuously during the day. Their ILP model aims to reduce the number of unfulfilled demand throughout the day, while minimizing the number of relocations made by the staff. The strategies proposed in [START_REF] Weikl | A practice-ready relocation model for free-floating carsharing systems with electric vehicles -Mesoscopic approach and field trial results[END_REF][START_REF] Folkestad | Optimal charging and repositioning of electric vehicles in a free-floating carsharing system[END_REF] were designed for a free-floating car-sharing service, but they only consider the relocation problem per se: given as input an initial distribution of cars in the city and an expected final one, they minimize the relocation costs without taking into account the profit provided by the well-located cars. We take inspiration from all these solutions but cannot apply them directly to our freefloating car sharing problem since, in our problem, there is no fixed stations, the fleet contains few cars (compared to bikes), a jockey can only move one car at a time, relocations occur during the night and we are interested in optimizing the number of driven minutes, not the number of trips. Fig. 1: Overview of the method, a regression model is trained for each cell to predict the driving time of each car, knowing both time and weather-related features. The optimization solver uses both the models learned to predict the driving time of each car and the night car positions to return the best morning car positions.

III. METHOD

Our goal is to propose a morning assignment of all cars of a free floating car sharing service such that it maximizes their expected utilization that day. We first discuss the geographic city modeling, how the prediction problem of the expected utility of a car is modeled and finally define a corresponding car assignment problem.

A. City and Trip Modeling

For efficiency and generality reasons, it is not possible to geographically model a city as the set of all its possible parking spaces. Following a study from [START_REF] Seign | Prescriptions for the successful diffusion of carsharing with electric vehicles[END_REF], we discretized the city surface into an hexagonal grid where each hexagon (also called cell) has a circumradius of 500 meters. This distance is the longest one a customer is willing to walk to find an available car of the service [START_REF] Seign | Prescriptions for the successful diffusion of carsharing with electric vehicles[END_REF]. The set of hexagons is noted K. All the parking spaces within the surface of the cell are labeled with the index k of this hexagon. Note that in our case, we remove unnecessary hexagons, those where no trip has begun or ended in.

B. Utility modeling

We assume that the historical data is a set of user trips, namely a set of tuples as follows: trips = {(id car , ts f rom , ts to , gps f rom , gps to )}. From the timestamp (ts) we compute the rent duration. Location information (gps) is provided as GPS coordinates, which can be mapped to the geographical cells defined earlier.

a) Actual utility: Based on the historical data of a single day, we want to compute the utility of each car for every cell. Since cars can be used for several trips during the day, we compute the actual utility of each car as: U * (id car ) = t∈trips(idcar) utility(t). Where trips(id car ) is the set of trips that have this car ID and the utility utility(t) of a trip t is its net revenue (proportional to the trip duration).

Our goal is to predict the potential utility of a car placed in any cell k. This can be cast as a regression problem, where the training data are the actual utilities defined above. However the number of cars in each cell k influence the potential utility of an additional car, e.g adding a second or fifteenth car will yield a different utility for the added car. In order to take into account this phenomena, we first order the cars of a cell in a deterministic way, namely by pick-up order (1 st car, ..., n th car). This models the temporal relation between cars based on their first pickup. We can now define the added value of an i th car as U i k = utility(id i car ) with id i car the initial car in cell k that was ordered (index i) by first pickup time.

b) Prediction: We call Û i k the predicted utility of a car in cell k at rank i. As cells can behave differently, we will learn a function f k per cell. As features, we use the rank i and a set fs of features including weather features, temporal features and lagged observations of the previous days. Hence, we wish to learn a regression function f k per cell such that Û i k = f k (i, fs). We discuss regression methods to use in Section IV-C. c) Clustering cells: Instead of learning one predictor f k per cell, we can also use similarities in utility behaviour for different cells. By grouping cells, more data becomes available for every function we wish to learn, and ideally more accurate predictions are made. We will hence investigate in Section IV the effect of clustering the cells and learn one function f c per cluster c.

C. Cost-based Car Relocation Problem

To maximize the utilization of cars during the day, we use two sets of Boolean decision variables: F ki denotes that the i th car is relocated away from cell k, while T ki denotes that a car is relocated to cell k to be its i th car. Then by s ki ∈ {0, 1} we denote whether there is an ith vehicle in cell k at the end of the previous day. Using these variables, choosing whether or not there is a car at the i th position in cell k is computed by s ki -F ki + T ki . The decision to add or remove a car from is done knowing the predicted utilization of this car ( Û i k ). Furthermore, since there is a finite number of relocation possible, constrained by the number of jockeys, we limit the number of relocation (constant γ). Moreover, jockeys cannot teleport hence they must be moved by a so-called 'sweeper' car to the next car after dropping off a vehicle. Operationally, these sweeper trips should be minimized, hence we introduce q kl the cost of a jockey being moved by a sweeper car from cell k to cell l; and p the price per minute paid by users renting a car so that profits/costs can be balanced off. Finally, E kl represents the number of sweeper trips necessary between cell k and l, with q kl the cost of one such sweeper trip.

Using the defined variables, the Integer Linear Programming (ILP) formulation of the problem is then:

maximize F,T,E k∈K i∈I p • Ûki • (s ki -F ki + T ki ) - k∈K l∈K q kl • E kl (1)
such that:

s ki -F ki ≥ 0 ∀k ∈ K, i ∈ I (2)
s ki + T ki ≤ 1 ∀k ∈ K, i ∈ I ( 3
)
k∈K i∈I T ki = k∈K i∈I F ki (4)
k∈K i∈I

T ki ≤ γ, k∈K i∈I F ki ≤ γ (5)
(s ki -F ki + T ki ) ≥ (s ki+1 -F ki+1 + T ki+1 ) ∀k ∈ K, i ∈ I \ {m} (6)
l∈K

E kl = j∈I T kj ∀k ∈ K (7) k∈K E kl = i∈I F li ∀l ∈ K (8)
F ki ∈ {0, 1}, T ki ∈ {0, 1} ∀k ∈ K, i ∈ I (9) E kl ∈ {0, . . . , |I|} ∀k, l ∈ K (10)
The objective function in Eq (1) maximizes the total expected revenue from the utility over the computed car placement while taking into account the costs to reposition jockey between cars to relocate. Eq [START_REF] Brendel | A Decision Support System for Computation of Carsharing Pricing Areas and its Influence on Vehicle Distribution[END_REF] states that only vehicles that are present at the end of the day can be moved away overnight, while Eq (3) states that if there is already a vehicle present then another vehicle can not be moved to the location. These two constraints effectively state that depending on s ki either F ki or T ki will be 0. Eq [START_REF] Ghosh | Dynamic repositioning to reduce lost demand in bike sharing systems[END_REF] states that the number of pickups and drop-offs has to be the same, while Eq (5) ensures that both the number of pickups and dropoffs is lower than the relocation budget γ. Eq [START_REF] Zakaria | Car relocation for carsharing service: Comparison of CPLEX and greedy search[END_REF] ensures that there are no 'gap' in the final assignment. Finally Eq [START_REF] Folkestad | Optimal charging and repositioning of electric vehicles in a free-floating carsharing system[END_REF] makes sure that each time a car is relocated to k, then a sweeper car needs to pick up the jockey. Eq [START_REF] Seign | Prescriptions for the successful diffusion of carsharing with electric vehicles[END_REF] assures that there is a sweeper car dropping of a jockey for each car relocated from l.

IV. EXPERIMENTS A. Dataset

Our experiments are done on a dataset from a real freefloating car sharing service. This dataset being sensitive for the service, we call it "CarService".

This dataset accounts for 1 138 246 trips made between the mid 2018 and early 2019. It covers 243 days for an average of 4684 daily trips and has been separated into three subsets: training, validation and test set. The training set regroups the trips from the first 145 days, the validation set are the next 48 days and the test set are the last 50 days.

Each trip in the database is tagged with the car identifier, both the origin and destination GPS positions and both the rent and park timestamps. We further tagged the trips with workday/holiday and day-of-week temporal features.

An additional weather dataset is used as exogenous data to train our prediction models. It has been retrieved from the SYNOP hourly weather broadcasts from the nearest weather station. We extract the temperature ( • C), the relative humidity (%), the pressure (hPa), the wind speed (km/h), the cloud cover (%) and the amount of rain (mm).

B. Cell Clustering Algorithms

We compare a strategy without any grouping (No Clustering) where each function f k is estimated independently for all the 151 cells k of the city grid, and compare it to a strategy based on a K-Medoids clustering. In the latter, each cell is represented by a vector such that the i th coordinate of the vector is the average driven time in minutes of the car of rank i. All the cell vectors do not have the same size. Thus the distance between vectors of different sizes i and i + h (with h > 0) is only computed on the first i common car ranks between both vectors. The Euclidean distance between these two common parts divided by its common length is given to the clustering algorithm. The optimal number (for prediction purpose) of clusters is found using the validation set for each model and feature set. The utility of an cell k becomes U i k = f c (i, fs) where f c is the predicted utility of the cluster c which contains the cell k.

C. Utility Prediction Algorithms

The regression methods tested are 1) the mean of historical values (baseline), 2) a Gradient Boosting Tree Regressor (GBR) and 3) a Support Vector Machine Regressor (SVR). Different sets of features were used to train the models.

Features. The first set of features F S 1 contains no features, such that only the rank i of each car is used to train all the regression models. The second set of features F S 2 contains exogenous information: the day of week, whether this day was a workday or not and a weather information. The weather information is obtained by averaging the hourly broadcasts for each day, thus creating weather vectors for each day. Overall, F S 2 contains 13 attributes. The third set of features F S 3 contains two historical features (in addition to F S 2 ) for a given cell k with a given rank i: the two last (in time) utility values f k (i) known (not predicted) in the data.

Models. Our baseline model (Mean) is the average of the utility values computed for each cell, for a given rank i for all days in the training set. This baseline only uses F S 1 .

We use the regression variant of Scikit-learn implementations of both the Gradient Boosting Tree (GBR) and the Support Vector Machine (SVR) with the default parameters for each model to directly predict the utility of the car in each cell for the four previously mentioned feature sets and with or without clustering the cells beforehand. Each attribute had been normalized by subtracting the average and dividing by the standard deviation.

D. Evaluation Metrics

We use the Mean Absolute Error (MAE) which is the average number of minutes misestimated by the trained regression models on each cell during one day, defined as:

MAE = 1
|K| k∈K i∈I |obs ki -pred ki |, with obs ki the real utility of the i th car in cell k and pred ki the predicted utility of the i th car in cell k.

We also use a ratio for each day between the MAE and the average real driven time per cell, called here the Ratio Mean Absolute Error (RMAE):

RMAE = M AE (1/|K|)• k∈K i∈I obs ki E. Relocation Strategies a) Strategies:
In order to evaluate our approach to relocate the cars of the fleet, we will compare the expected revenue of four different strategies. We denote by Historical the strategy based on the historical fleet position every morning, and its true utilization during the day. The corresponding revenue is the number of driven minutes multiplied by the price paid by users for car renting. Optim MU denotes a relocation strategy which maximizes (our utility prediction and ILP formulation) the car utilities without taking into account any costs. Optim SO denotes the strategy, described in Section III-C, which uses a joint (Single) Objective. This objective maximizes the service utilization revenue minus the costs to reposition the jockeys. Optim DO denotes a two steps (Double) optimization strategy: the first one maximizes the fleet utility (as for Optim MU) while the second one, after fixing the car location found, minimizes the jockeys' repositioning costs.

b) Utility Estimation: The ground truth utility used to evaluate the revenue earned by the service with our approach is a mix between the historical utility and the predicted one: when the solver proposes a relocation solution different from the historical one, the predicted utility is used for the cars for which the historical value is unknown. c) Utility Conversion and Cost Estimation: For all the strategies, we assume that the price p paid by the user per driven minutes is constant. Thus, the fleet expected revenue is the total utility multiplied by p. The sweeper car cost q kl , is computed given the gross hourly salary of two jockeys 2 • c j (a jockey is "repositionned" from k to l by someone driving a sweeper car), the cost c s of running a sweeper car s per km, the average speed (in km/h) of a sweeper car and the distance d kl (in km) between the cell k where a jockey is picked up by the sweeper car and the cell l where he/she is dropped off :

q kl = d kl •(2•cj ) s
+d kl •c s . Furthermore, we make the hypothesis that there are 7 jockeys that can do 10 relocations on average each night, so γ = 70.

F. Case Study

We want to evaluate both the prediction part and the relocation part of our approach. For the former, we evaluate how well can f k (i, fs) be predicted for a cell k, a car rank i and a feature set fs with the regression approaches and baseline (Mean). For the later, we compute the daily expected revenue of the car sharing system when the utility is predicted and used to solve the relocation problem.

1) Regression Model Performance:

The columns of Table I represent the regression models used (historical Mean, Gradient Boosting Tree and the Support Vector Machine). The lines describe the results with or without clustering using the three feature sets. MAE (in min) and RMAE (in %) are related to the true number of minutes driven in each cell and day and the predicted number of minutes on the test set. The historical Mean performance has not been computed with F S 2 and F S 3 since its statistical value could be questionable.

One can notice that with F S 1 , the historical mean gives one of the worst MAE (∼288 mins) showing the potential of the machine learning models and our chosen features.

The addition of the temporal and weather features increases significantly in most cases the performance of the regression models (lines with F S 2 exhibit lower MAE and RMAE than lines with F S 1 ) and especially when using the K-Medoids algorithms to group the cells. Adding information about past values in F S 3 further benefit the SVR regression model (the GBR regression model keeps stable results when using the K-Medoids). In all cases except the most simple (Mean regressor and/or just F S 1 as features), using a clustering over the similar cells allows to reduce the error values, most of the time significantly. The clustering of similar cells allows the regression algorithms to train on more trips with the same features, better generalizing the training set. These results allow to choose for our regression approach the SVR regressor with a K-Medoids clustering of the similar cells while using the feature set F S 3 : this combination offers a 2% reduction in RMAE compared to the baseline, as well as a reduction of 10 minutes (on average per cell and per day) in the MAE. This will be used for the following experiments. 2) Relocation Performance: From now on we call solver car assignment the optimal solution to our optimization problem found by the proprietary solver Gurobi. The utility (number of driven minutes) is estimated using the best regression model found in the previous experiment. Thus, a K-medoids clustering of the city cells is followed by a SVR regressor, using the feature set F S 3 .

For all approaches, the total daily revenue expected by both services are computed. These are plotted per day in Figure 2 and averages per day are reported for in Table II. We expect the revenue to be higher for Optim MU since it does not take into account any relocation cost but this strategy is also less realistic. We also expect Optim SO to give a higher revenue than Optim DO since the one step strategy can take into account the costs while relocating the car which should help reducing these costs.

Table II shows that for "CarService", all the optimization methods could help (on average) to increase the profit of the company. Further details in Figure 2 show that there are only two weeks where this is not true and the gain is almost 10% compared to the current service (historical) for the rest of the test set. Surprisingly, the different between the relocation strategies is small. This is probably because the service is relatively concentrated and the distance between each cell is low enough to have a very light weight on the relocation costs. This also explains why there is no difference between the twostep (Optim DO) and single step (Optim SO) optimization strategies since the cost does not influence the relocations.

V. CONCLUSION

We tackled the problem of optimizing car relocation in a free floating car sharing service in order to increase its daily usage and profitability. We proposed to predict the expected car utilization and showed how such predictions could be used in a ILP solver to optimize the relocations while taking into account the sweeper costs. Although the predicting phase could be further improved by introducing more exogenous data, our approach could already increase the efficiency and profitability of the service to up to 7%.

Our relocation solution could benefit from further operational research formulation to optimize the route of the sweeper car and of all the jockeys (both are pickup and delivery vehicle routing problems [START_REF] Savelsbergh | The General Pickup and Delivery Problem[END_REF]). This would be valuable but non trivial: jointly optimizing a vehicle routing and our assigment problem requires adding non-standard constraints and the problems are of a scale unattainable by generic ILP solvers.

Fig. 2 :

 2 Fig. 2: Daily revenue expected by "CarService" (one x-axis tick is a Sunday), with the Historical strategy (horizontal dashed blue line at 100) as the base index. The red full line, the dashed yellow and green lines are respectively the normalized revenues provided with Optim SO, Optim DO and Optim MU strategies.

TABLE I :

 I Prediction MAE (lower is better) and RMAE of the car utility per day and per (group of) cell(s), using a historical baseline (Mean) and two regression models (GBR and SVR) with different sets of features (F S 1 ,F S 2 and F S 3 ).

TABLE II :

 II Average daily revenue and number of relocations for four relocation strategy in the case of "CarService".

Acknowledgements. This work has been done in collaboration between Stellantis and Inria through the OpenLab AI and was supported by the CIFRE program of ANRT.