domain: math info

text:
Marie Al-Ghossein
email: malghossein@enst.fr

Pierre-Alexandre Murena
email: murena@telecom-paristech.fr

Talel Abdessalem
email: talel.abdessalem@enst.fr

Anthony Barré
email: anthony.barre@live.fr

Antoine Cornuéjols
email: antoine.cornuejols@agroparistech.fr

Ltci

Adaptive collaborative topic modeling for online recommendation

Keywords: Online recommendation, Concept drift, Topic modeling, Collaborative filtering

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

role to help users and empower companies. Approaches to recommendation can mainly be categorized into three classes. First, content-based (CB) approaches rely on information extracted from user profiles and item descriptions. Second, collaborative filtering (CF) approaches make use of user activities and past interactions, e.g., ratings and clicks, to learn preferences and generate recommendations. Lastly, hybrid approaches aim to combine both techniques in order to overcome their weaknesses: While CB methods tend to be overspecialized and lack a sense of novelty, the performance of CF methods drops with an increase of rating sparsity and in the coldstart setting. To get the best of both worlds, hybrid approaches allow CF approaches to exploit auxiliary information like text [START_REF] Wang | Collaborative topic modeling for recommending scientific articles[END_REF][START_REF] Wang | Collaborative deep learning for recommender systems[END_REF] and images [START_REF] He | VBPR: visual Bayesian Personalized Ranking from implicit feedback[END_REF]. Collaborative Topic Regression (CTR) [START_REF] Wang | Collaborative topic modeling for recommending scientific articles[END_REF] is a popular hybrid approach combining probabilistic topic modeling for content analysis [START_REF] David M Blei | Latent dirichlet allocation[END_REF] and latent factor models for CF [START_REF] Pan | One-class collaborative filtering[END_REF].

Most hybrid RS proposed in the literature are meant to work in batch where an initial model is first built from a static dataset and then rebuilt periodically as new chunks of data arrive. Since models are not continuously maintained up to date, they cannot capture the user feedback generated after a model update before the next one. They are also not able to adapt to changes in user preferences and item descriptions which occur due to temporal dynamics.

In real-world applications, the recommendation problem can be formulated as a data stream problem where RS are designed to learn from continuous data streams and adapt to changes in real-time. Recent work [START_REF] Frigó | Online Ranking Prediction in Non-stationary Environments[END_REF] has shown that simple online algorithms can generate better recommendations than more complex ones that are only updated periodically. Online RS are mainly based on incremental learning to continuously update models when receiving new observations. Incremental CF approaches have been proposed in this direction, like incremental neighborhood-based methods [START_REF] Miranda | Item-based and user-based incremental collaborative filtering for web recommendations[END_REF] and incremental matrix factorization [START_REF] He | Fast matrix factorization for online recommendation with implicit feedback[END_REF][START_REF] Vinagre | Fast incremental matrix factorization for recommendation with positive-only feedback[END_REF]. Learning from data streams should also account for concept drifts [START_REF] Gama | A survey on concept drift adaptation[END_REF] which occur when the definition of modeled concepts changes over time. Detecting drifts and adapting to them become thus necessary.

In this paper, we address the problem of online recommendation in a dynamic environment where users interact with items in realtime and where new items are expected to arrive accompanied by a textual description. This setting is common in (but not restricted to) the news and tweet recommendation domains for example, where new articles and tweets are continuously generated and read by users.

We propose an adaptive collaborative topic modeling approach for online recommendation. Our approach combines AWILDA, an adaptive version of online Latent Dirichlet Allocation (LDA) [START_REF] Hoffman | Online learning for latent dirichlet allocation[END_REF] that is able to analyze and model documents arriving in a stream on one side, and incremental matrix factorization [START_REF] Vinagre | Fast incremental matrix factorization for recommendation with positive-only feedback[END_REF] to leverage user interactions for the learning of preferences on the other side. Our approach is adaptive as we actively detect changes of topics that may occur in the document stream and adjust accordingly. We alternate online refinement of the topic model when no drift is found with batch training when it is needed. The decision of retraining and the chunk of data on which we retrain the model are automatically determined by an adaptive sliding window technique [START_REF] Bifet | Learning from time-changing data with adaptive windowing[END_REF].

The purpose of introducing our approach is threefold. First, it is fully incremental and can thus be used in an online setting to generate recommendations. Second, since it is a hybrid approach relying on users' past interactions and on textual information, it addresses in particular the item cold-start problem which, to the best of our knowledge, has not been studied yet in the context of online recommendation. Third, its capacity of automatically detecting and adapting to drifts makes it suitable for real-world scenarios where changes in topics of document streams are frequently happening due to unexpected events and need to be considered for a better quality of recommendation.

The remainder of the paper is organized as follows. In Section 2, we discuss previous work on several topics related to our work. Section 3 proposes a reminder of LDA, ADWIN, and Incremental Matrix Factorization on which our method, detailed in Section 4, is based. Experiments and results are presented and discussed in Section 5. Finally, Section 6 concludes the paper.

RELATED WORK

In this Section, we review related work on hybrid RS, online RS, relevant variants of LDA, and news recommendation.

Hybrid RS. The cold-start problem [START_REF] Andrew I Schein | Methods and metrics for cold-start recommendations[END_REF] occurs in RS when new users or new items are introduced into the system. While recommending new and fresh items is essential, CF methods have difficulties in doing so, as no or few feedback related to these items is observed. Hybrid RS are able to recommend new items by leveraging auxiliary information. They also help alleviate the sparseness of rating or feedback data, thus improving the quality of recommendation.

To this end, previous work has utilized text data such as abstracts [START_REF] Wang | Collaborative topic modeling for recommending scientific articles[END_REF], synopses [START_REF] Wang | Collaborative deep learning for recommender systems[END_REF] or reviews [START_REF] Bao | TopicMF: simultaneously exploiting ratings and reviews for recommendation[END_REF]. Several techniques have been used to model documents like LDA [START_REF] Wang | Collaborative topic modeling for recommending scientific articles[END_REF], stacked denoising autoencoders [START_REF] Wang | Collaborative deep learning for recommender systems[END_REF] or convolutional neural networks [START_REF] Kim | Convolutional matrix factorization for document context-aware recommendation[END_REF]. Images have also been leveraged in this context and visual appearances of items can be added to the preference model [START_REF] He | VBPR: visual Bayesian Personalized Ranking from implicit feedback[END_REF].

While most hybrid RS are designed to work in batch, we propose an online hybrid RS that is able to address the item cold-start problem in a dynamic environment where items are added in real-time.

Online RS. Maintaining recommendation models up-to-date as new data is generated is essential to preserve the quality of recommendation over time [START_REF] Dias | The value of personalised recommender systems to e-business: a case study[END_REF]. Online RS are expected to learn from continuous data streams and adapt to changes in real-time. Elements of a data stream arrive in real-time at high rates and are processed sequentially using limited resources. Online RS are therefore based on incremental models like incremental neighborhood-based models [START_REF] Miranda | Item-based and user-based incremental collaborative filtering for web recommendations[END_REF] and incremental matrix factorization (using stochastic gradient descent [START_REF] Vinagre | Fast incremental matrix factorization for recommendation with positive-only feedback[END_REF] or alternating least squares [START_REF] He | Fast matrix factorization for online recommendation with implicit feedback[END_REF]).

User preferences and item descriptions are expected to change over time in different ways, at different moments, and at different rates. While some existing approaches consider the evolution of entities over time [START_REF] Ding | Time weight collaborative filtering[END_REF][START_REF] Gao | Collaborative dynamic sparse topic regression with user profile evolution for item recommendation[END_REF][START_REF] Koren | Collaborative filtering with temporal dynamics[END_REF], they are not adapted to the online setting. Incremental learning is a way of passively adapting to current changes in the data distribution by continuously learning from new data. Actively accounting for changes in user preferences has been based on the intuition that users' recent observations are more relevant than older ones. Sliding window techniques have been explored in this direction [START_REF] Matuszyk | Forgetting methods for incremental matrix factorization in recommender systems[END_REF][START_REF] Nasraoui | Performance of recommendation systems in dynamic streaming environments[END_REF][START_REF] Zaigham | xstreams: Recommending items to users with time-evolving preferences[END_REF]. We note that these techniques make assumptions concerning the relevance of old observations and the rate at which all preferences drift, which are not always accurate.

From the point of view of the RS, a change of user preferences has an impact on a local scale since it only concerns a single user, whereas a change of item description or popularity affects the recommendation across all users and has a more global impact. Among the few works that considered concept drifts in the context of online recommendation, the focus has been on considering local changes occurring on the user level.

In this work, we manage to detect drifts on the item level in the topic model meant to handle the stream of documents describing items and we adapt the model when it is needed.

Topic modeling and concept drifts. Topic modeling is a machine learning task which consists in associating a document seen as an unordered list of words with a vector of topics, i.e., of word distributions. One of the most influential topic modeling method is the generative model of Latent Dirichlet Allocation (LDA) [START_REF] David M Blei | Latent dirichlet allocation[END_REF]. LDA models a document as a multivariate distribution, the parameter of which is drawn from a Dirichlet distribution. The popularity of LDA is due to its simplicity and modularity, as well as its interpretability.

In se, LDA is not designed for evolving environments but only to infer topics inside a batch of accessible documents. Variants have been proposed to include a temporal aspect to LDA, including situations where the distribution evolves over time. A first variant, called Dynamic Topic Models (DTM) [START_REF] David | Dynamic topic models[END_REF], considers that the wordtopic distribution varies over time. At each time step, this parameter is re-evaluated, conditioned by its value at the previous step. The same idea, but implemented at the level of a paragraph in a book, is proposed by SeqLDA [START_REF] Du | Sequential latent dirichlet allocation: Discover underlying topic structures within a document[END_REF]. A major drawback of these methods and other temporal adaptations of LDA (such as [START_REF] Thomas | Finding scientific topics[END_REF]) is the use of time slices, the size of which is arbitrary and does not depend on the observed data. In particular, changes in topic distribution can happen within a time period significantly smaller than the length of the chosen window. Continuous time models offer solutions to this problem [START_REF] Iwata | Online multiscale dynamic topic models[END_REF]. A pioneer continuous time method [START_REF] Wang | Topics over time: a non-Markov continuous-time model of topical trends[END_REF] modifies LDA by assuming that word distribution over topics depends on RecSys '18, October 2-7, 2018, Vancouver, BC, Canada word co-occurrences, but also on the date of the document. Despite the benefit of this method, it cannot be used in practice for stream analysis since the learning is made offline: it requires the whole dataset to be accessible in one batch to be able to infer the model.

For this reason, existing topic modeling methods are not suitable for data stream mining. Learning from data streams presents two major difficulties: the online nature of learning and concept drift. Concept drift designates the possible change in distribution that can happen in temporal and non-stationary environments [START_REF] Gama | A survey on concept drift adaptation[END_REF]. Two main strategies are adopted to cope with concept drift. On one hand, passive methods adapt the model at each time step, no matter whether a change actually occurred. On the other hand, active methods focus on detecting drifts and adapt the model only when a drift is detected. Among active methods, ADaptive WINdowing (ADWIN) [START_REF] Bifet | Learning from time-changing data with adaptive windowing[END_REF] gained a lot of interest recently for the simplicity of its approach (comparing average values of a time series on subwindows) and for the theoretical guarantees it proposes.

The approach we propose in this paper is based on AWILDA [START_REF] Murena | Adaptive Window Strategy for Topic Modeling in Document Streams[END_REF] which combines LDA model and ADWIN algorithm for drift detection in data streams and which we present in Section 3.

News recommendation. The problem we address in this paper is common in the setting of tweets [START_REF] Diaz-Aviles | Real-time top-n recommendation in social streams[END_REF], articles [START_REF] Wang | Collaborative topic modeling for recommending scientific articles[END_REF], and news recommendation [START_REF] Viorica Epure | Recommending Personalized News in Short User Sessions[END_REF]. Our approach can be used to perform online recommendation of new items in any domain, whenever a textual description is available. We review in particular the related problem of news recommendation since it has been specifically studied in the literature.

News articles are continuously generated and while some of them could be relevant several weeks after the publication date, others have a shorter life cycle. Therefore, news recommendation often considers recency and popularity [START_REF] Ahmed | Fair and balanced: Learning to present news stories[END_REF][START_REF] Doychev | An analysis of recommender algorithms for online news[END_REF]. This is done for example by filtering candidate articles for recommendation based on recency among other criteria [START_REF] Billsus | Adaptive news access[END_REF]. Readers' interests are captured using the categories of the articles (if available) or keywords related to the articles' topics [START_REF] Doychev | An analysis of recommender algorithms for online news[END_REF][START_REF] Liu | Personalized news recommendation based on click behavior[END_REF] and are used for CB recommendation. Hybrid approaches combining CB and CF usually perform best for the problem of news recommendation [START_REF] Abhinandan S Das | Google news personalization: scalable online collaborative filtering[END_REF][START_REF] Lin | Personalized news recommendation via implicit social experts[END_REF][START_REF] Liu | Personalized news recommendation based on click behavior[END_REF]. Combining longterm and short-term preferences has also proven to be beneficial in this context [START_REF] Viorica Epure | Recommending Personalized News in Short User Sessions[END_REF][START_REF] Liu | Personalized news recommendation based on click behavior[END_REF]. In addition, session-aware RS have been used to address this problem, given that users are not always identified when browsing on news platforms. Session-aware RS focus on transitions between items, formulating the problem as a Markov decision process [START_REF] Viorica Epure | Recommending Personalized News in Short User Sessions[END_REF] or using recurrent neural networks [START_REF] Hidasi | Session-based recommendations with recurrent neural networks[END_REF].

Our work addresses the wider and more generic problem of online recommendation that could occur in any domain and with no existence of sessions. We only require a textual description of items which can be easily collected from abstracts or reviews. We leverage CF and online topic models, and we consider the evolution of content through drift detection for modeling items.

PRELIMINARIES

The approach we propose merges three existing techniques of three unrelated domains. The purpose of this section is to present these techniques.

Latent Dirichlet Allocation

Latent Dirichlet Allocation (LDA) [START_REF] David M Blei | Latent dirichlet allocation[END_REF] is a generative model describing text documents and corpora. The key notion involved in the description of a document is the notion of topic. A topic corresponds to a word distribution (for instance in a music-oriented corpus, the word "concert" would have a higher probability to be drawn than in a sport-oriented corpus) and a document is described as a mixture of topics. Topics are learned in an unsupervised fashion and, thus, do not necessarily correspond to human-understandable concepts [START_REF] Chang | Reading tea leaves: How humans interpret topic models[END_REF]. The generative process, presented in Figure 1, is described as follows:

( In terms of document analysis, the parameters of LDA can be understood in the following way:

1) Choose θ ∼ Dirichlet(α) (2
• Vector α represents the global topic trend. For instance, a parameter α = (1, . . . , 1) corresponds to a uniform choice over all topics, on average. The higher the component α i , the more frequent topic i will be in the whole corpus. • Matrix β stores the probability of words inside topics. If a word w is set to belong to topic z, then it will be chosen with probability β z,w . • Vector θ corresponds to the topic distribution inside one document. A training of LDA model is possible based on maximum likelihood principle [START_REF] David M Blei | Latent dirichlet allocation[END_REF]. In practice, it is suggested to use either Gibbs sampling or variational inference for this task. An online version of variational inference is usually preferred [START_REF] Hoffman | Online learning for latent dirichlet allocation[END_REF] based on a stochastic gradient descent. In this version, data are not supposed to be received as a batch, but are processed one by one. However, the context is not the same as stream mining. In online LDA, all data are generated by the same distribution (stationary) and concept drift is not considered.

Adaptive Sliding Window

Adaptive Sliding Window algorithm (ADWIN) [START_REF] Bifet | Learning from time-changing data with adaptive windowing[END_REF] is an active strategy used for classification of data with concept drift adaptation. The algorithm uses a sliding window W to detect a change in a series of one-dimensional observations. In a supervised context, these observations usually correspond to risk measures. The scores are stored by ADWIN in a sliding window W at each observation.

A drift is detected if the window W can be separated into two subwindows W = W 0 W 1 such that the difference of means µ W 0 and µ W 1 in the sub-windows is large enough (above a parameter ϵ).

In addition to the extreme simplicity of the method, ADWIN also benefits from a strong theoretical justification. In particular, it can be shown that, depending on the threshold ϵ only, probabilities of incorrectly detecting a drift (false positive rate) and correctly detecting a drift (false negative rate) are bounded.

Incremental Matrix Factorization

Matrix factorization (MF) is a popular collaborative filtering technique used to model users' interactions by representing users and items in a space of latent factors learned from the data. If R designates the matrix of interactions (where R ui = 1 if the user u interacted with the item i, and 0 otherwise), then MF aims to approximate R as a product of two matrices P and Q by minimizing over P and Q:

(u,i)∈D R ui -P u Q T i 2 + λ u ∥P u ∥ 2 + λ i ∥Q i ∥ 2 ( 1
)
where D is the set of observed interactions, and λ u and λ i are regularization parameters. The score of an item i for a user u, denoted by Rui , is computed using the scalar product between P u and Q T i . Items are ordered by descending proximity of Rui to the value 1, and top-N items are recommended for u.

Classic algorithms for MF are not suitable for a data stream setting. A variant of MF adapted to the incremental nature of data streams [START_REF] Vinagre | Fast incremental matrix factorization for recommendation with positive-only feedback[END_REF] suggests the following procedure. Observations ⟨u, i⟩ are received one after the other and handled by the algorithm. For each received observation, P and Q are updated using the gradient of the objective for this observation only (which corresponds to an estimator of the gradient on the whole dataset). When either a user or an item are observed for the first time, they are added to the matrices with a random initialization, and the values of P and Q are then updated using the observation.

ADAPTIVE COLLABORATIVE TOPIC MODELING

In this Section, we present the proposed algorithm for adaptive collaborative topic modeling. Our algorithm is split into two components: drift detection in topic modeling and collaborative topic modeling.

Adaptive Window based Incremental LDA

We first present the algorithm used for topic drift detection in LDA which merges the drift detection property of ADWIN with parameter estimation in the training of the LDA model. The algorithm, called Adaptive Window based Incremental LDA (AWILDA) [START_REF] Murena | Adaptive Window Strategy for Topic Modeling in Document Streams[END_REF], is based on the idea that a change of distribution will induce a drift in the likelihood of the model. Following this idea, a change in the LDA distribution can be detected by ADWIN by processing the series of likelihoods.

AWILDA is based on two models of LDA. The first model, denoted by LDA m , is used for document modeling. The second model, denoted by LDA d , is used for the detection of drifts only. When a new document is received, the algorithm works as follows: The idea is to split the task of prediction and the task of drift detection by separating the models. The model used for prediction, LDA m , is kept up to date while no drift is detected. On the contrary, the model LDA d is not modified; otherwise, the detected changes might not originate only from a change in the data distribution, but also from the change of the model.

The AWILDA algorithm benefits from all the advantages of AD-WIN. In particular, it offers a strict control of false positive rate and false negative rate. If the underlying data generation process does not change, the distribution of the likelihood is stationary, which makes the theorems relative to ADWIN still valid.

In practice, the likelihood cannot be used directly: Since it measures probabilities, the values of likelihood observed on real (and artificial) data are very low and cannot be well distinguished. For this reason, we use the loglikelihood in practice. The value of the loglikelihood is not bounded which is a problem with respect to the theory of ADWIN. However, we noticed that the values evolve inside an interval of low amplitude and can be lower-bounded.

The loglikelihood of an LDA model is not computable in practice. Consequently, we use the lower-bound proposed by [START_REF] David M Blei | Latent dirichlet allocation[END_REF] and [START_REF] Hoffman | Online learning for latent dirichlet allocation[END_REF] as part of the variational inference training process.

Adaptive Collaborative Topic Modeling

Collaborative Topic Modeling (also called Collaborative Topic Regression, or CTR) [START_REF] Wang | Collaborative topic modeling for recommending scientific articles[END_REF] is a popular framework which was initially introduced to recommend scientific articles. CTR assumes that documents describing items are generated by LDA and it represents users with topic interests. CTR introduces an additional latent variable that offsets the topic proportions when modeling the user interactions. This offset is learned from the feedback data using CF and is added to represent the fact that two items having similar topic proportions can be interesting to different types of users. The topic model used in CTR is LDA and the latent factor model is Probabilistic Matrix Factorization [START_REF] Mnih | Probabilistic matrix factorization[END_REF].

CTR alleviates the problem of feedback sparsity by leveraging auxiliary information. It is also able to recommend existing items and new items in the case of cold start, and proposes an interpretable latent structure for users and items. However, the models used in CTR are learned offline and CTR is not adapted for online recommendation. In the following, we propose CoAWILDA which benefits from the cited advantages of CTR, is adapted to the online setting, and takes into account the non-stationarity nature of data in an evolving environment.

In our setting, observations are supposed to arrive in real-time and are mainly of two types. First, interactions, denoted by ⟨u, i⟩, designate positive actions (clicks, ratings) performed by users and concerning a certain item. Second, additions of items, denoted by ⟨i, doc i ⟩, usually occur when a new item becomes available at a certain time step and we consider cases where a textual description of the new item is provided. if o = ⟨i, doc i ⟩ then ▷ new item added 3:

θ i ← AW ILDA(doc i ) 4: ϵ i ∼ N(0, λ -1 i I K ) 5: Q i ← θ i + ϵ i 6:
end if

7:
if o = ⟨u, i⟩ then ▷ interaction received 8:

if u Rows(P) then ▷ new user observed 9:

P u ∼ N(0, λ -1 u I K ) 10:
end if

11:

e ui ← 1 -P u .Q T i 12: P u ← P u + η(e ui Q i -λ u P u ) 13: ϵ i ← ϵ i + η(e ui P u -λ i ϵ i ) 14: Q i ← θ i + ϵ i 15:
end if 16: end for CoAWILDA is presented in Algorithm 1. When a new item is received, we use AWILDA to model the descriptive document and extract topic proportions θ i . The item latent vector Q i representing an item i results of the addition of the topic proportions θ i and an item latent offset ϵ i . When a new interaction ⟨u, i⟩ is observed, we update the user latent factor P u and the item latent offset ϵ i following the procedure of incremental MF. Recommendation is performed as described in Section 3.3 where Rui = P u .Q

T i = P u .(θ i + ϵ i ) T .

EXPERIMENTS

In this Section, we present the experiments we conducted to prove the effectiveness of our approach. We first show how AWILDA performs when modeling a stream of documents and then discuss how our approach performs when addressing the problem of online recommendation, using real-world datasets.

Datasets. We study the problem of online recommendation where observations are being received in real-time. Data used to evaluate our approach should be chronologically ordered and should include user interactions and the addition of items over time with a corresponding textual description. We note that these two characteristics are not always available in datasets commonly used to evaluate RS.

In our work, we use two real-world datasets: the ml-100k and the plista datasets.

The ml-100k dataset corresponds to the MovieLens 100k dataset1 and gathers 100,000 ratings from 1,000 users on 1,700 movies, spanning over 18 months. Since we are addressing the problem of recommendation with implicit feedback (positive-only data), our goal is to recommend the movies the user is going to rate. We note that the dates reported in ml-100k correspond to the rating date of the movies and not to the actual watching date. Knowing that we are not concerned with the problem of evolution of user preferences in this work, we use ml-100k to evaluate our approach. Movies become available according to their reported release date, and we use DBpedia2 to collect abstracts written in English and describing each one of them.

The plista dataset is described in [START_REF] Kille | The plista dataset[END_REF] and contains a collection of news articles published in German on several news portals. The available dataset captures interactions collected during the month of February 2016. We remove from the dataset interactions corresponding to unknown users, users with less than three interactions, and items with no available textual description. Finally, the dataset gathers 32,706,307 interactions from 1,362,097 users on 8,318 news articles.

Documents from both datasets have been preprocessed by mainly removing stop words, removing words occurring once, and stemming remaining words.

Performance of AWILDA for topic modeling

Evaluation protocol. AWILDA is proposed to model a stream of documents using drift detection. In our experimental setting, we consider that we are receiving documents describing items one after the other, ordered by their availability date (e.g., release or publication date). For each received document, we evaluate the topic model and process the document to update the underlying model. We use the first 20% documents of the stream to train the model which is evaluated on the remaining documents.

Evaluation measure. The goal in document modeling is to maximize the likelihood on unseen documents (D t est ), given a trained topic model. Perplexity measures the ability of a model to generalize to new data and is used to evaluate topic models [START_REF] David M Blei | Latent dirichlet allocation[END_REF]. Perplexity is defined as follows:

perplexity(D t est ) = exp - M d =1 log 2 p(w d ) M d =1 N d (2)
where M designates the total number of documents in D t est . In the actual use of perplexity, the probability p(w d ) is approximated by its upper-bound (given by the variational inference) as explained in Section 4.1. A lower value of perplexity indicates a better generalization capacity. Since we are handling document streams, the perplexity is reported for each received document using the current model.

Methods compared. We compare the performance of AWILDA to the online version of LDA [START_REF] Hoffman | Online learning for latent dirichlet allocation[END_REF], considering its capacity to handle document collections arriving in a stream. We show the results for experiments where we set the number of topics to 10, knowing that similar patterns appear for different values of this parameter.

Results. Figure 2 shows the perplexity measured on the document streams of ml-100k and plista for AWILDA and online LDA. The perplexity is represented as a moving average with a sliding window of 200 observations. The red dotted vertical line marks the detection of a drift by AWILDA. AWILDA detects two drifts for the held-out documents of ml-100k and five drifts for plista. This difference in behavior is expected knowing the volume and nature of both datasets (movies vs. news). Before detecting any drift, online LDA and AWILDA are trained in the same way and on the same data, which explains the close values of perplexity. After detecting the first drift, AWILDA outperforms online LDA for the task of document modeling. As documents continue to arrive, AWILDA is more adapted to the new data. Its drift detection component allows it to adjust to changes after each drift, resulting in a better performance. A further analysis of the datasets with experts from both domains will help to establish link between the detected drifts and real-life events occurring in the same time period, for better understanding and explainability.

Performance of CoAWILDA for online recommendation

Evaluation protocol. RS are traditionally evaluated using holdout methods. These methods are not adapted to the online setting [START_REF] Vinagre | Evaluation of recommender systems in streaming environments[END_REF] mainly because when we randomly sample data for training and testing, we lose the temporal dimension and do not respect the original order of observations. Since the topic model requires an initial phase of training, we adopt the evaluation process introduced in [START_REF] Matuszyk | Selective forgetting for incremental matrix factorization in recommender systems[END_REF]. We sort the dataset chronologically and then split it into the following three subsets:

• Batch Train subset. The first 20% of the dataset is used for the initial training of the models. • Batch Test -Stream Train. The next 30% of the dataset is used for the validation of the initialized models, and for incremental online learning to ensure the transition between the first and the last phase. • Stream Test and Train. The last 50% of the dataset is used for prequential evaluation, which is a test-then-learn procedure performed while iterating over the observations [START_REF] Gama | Issues in evaluation of stream learning algorithms[END_REF]. Each observation ⟨u, i⟩ is used to evaluate the model by generating recommendations for user u and then to update the model using ⟨u, i⟩. Evaluation measures. We use recall@N and DCG@N to measure the quality of recommendation. These metrics are described in [START_REF] Frigó | Online Ranking Prediction in Non-stationary Environments[END_REF] for the online setting. We report the results for the Stream Test and Train subset.

Parameters. We performed a grid search over the parameter space of the methods in order to find the parameters that give the best performance. We report the performance corresponding to the parameters leading to the best results. The parameters are reported along with the methods below.

Methods compared. Since previous work has demonstrated the advantages of using online recommendation compared to batch recommendation [START_REF] Frigó | Online Ranking Prediction in Non-stationary Environments[END_REF][START_REF] Vinagre | Fast incremental matrix factorization for recommendation with positive-only feedback[END_REF], we focus on incremental methods. We also only consider one approach for incremental MF, knowing that our method, CoAWILDA, can integrate any other algorithm for incremental MF or any model-based method. We compare the performances of several incremental methods adapted to the online setting, including variants of the one we propose.

• the collaborative component by representing users in the space of topics and updating their profiles as we get more observations. For ml-100k, we set K = 20, η = 0.04, and λ u = 0.01. For plista, we set K = 10, η = 0.042, and λ u = 0.01. • MF is the incremental MF [START_REF] Vinagre | Fast incremental matrix factorization for recommendation with positive-only feedback[END_REF]. Compared CoAWILDA and CoLDA, MF does not leverage content information about items. For ml-100k, we set K = 50, η = 0.01, λ u = 0.02, and λ i = 0.02. For plista, we set K = 50, η = 0.008, λ u = 0.01, and λ i = 0.01. • Knni is the incremental item-based approach proposed in [START_REF] Miranda | Item-based and user-based incremental collaborative filtering for web recommendations[END_REF].

We set the number of neighbors to 300. • Rand randomly selects items for recommendation.

Results. Figure 3 shows the DCG@Ni of the methods we compare for ml-100k and plista, where Ni is the total number of items included in each dataset. The idea is to evaluate how each approach performs when ranking the items for each user. We report the metric value with respect to the number of observations processed in order to analyze its evolution over the time spanned by the Stream Test and Train set.

CoAWILDA outperforms all the other methods evaluated for both datasets. The comparison between CoAWILDA and CoLDA demonstrates the effectiveness of AWILDA for modeling document streams describing new items and for improving the quality of item modeling and thus recommendation. CoLDA is not able to adjust to drifts occurring in topic modeling which deteriorates the recommendation quality over time.

The performance of CoLDA for plista can be divided into two phases. In the first one, the topic model is still able to carry out good document modeling and is beneficial for the recommendation: CoLDA performs better in terms of item ranking than MF which does not account for content analysis. In the second phase, and with the incapacity of online LDA to adjust to drifts, MF outperforms CoLDA. This means that not only the topic model is not adapted to newly received data, but it is also badly affecting the recommendation quality and there is no interest in using it anymore. We also note the importance of evaluating the evolution of the models over time to show how they are affected by eventual changes occurring in the data. This phenomenon appears for plista where more frequent drifts occur over time, mainly due to the nature of news data. Concerning ml-100k, CoLDA performs better than MF but still worse than CoAWILDA.

AWILDA is a content-based method and only relies on topics extracted from items to model user preferences. It performs poorly compared to the other methods and proves the importance of having a CF component. Knni performs better than AWILDA but is not as robust as MF and the hybrid approaches evaluated.

The number of available items grows significantly over time in plista. This results in the dropping of performance (in terms of ranking) of all methods over time. This is not the case in ml-100k, since only few movies are added in the corresponding time period. More data is received and more learning is done over time, which can explain the improvements in the performance of CoAWILDA and CoLDA. Figure 4 and Figure 5 show the recall@5, recall@10, recall@50, and recall@100 of our approach, CoAWILDA, and its variants on ml-100k and plista respectively.

The experiments for ml-100k confirm the ideas we mentioned before. CoAWILDA outperforms the other variants and performs better than CoLDA which relies on online LDA and does not adapt to changes in the data. CoLDA performs better than MF demonstrating the benefits of using content information. AWILDA relies only on content information which is a weak approach to model user preferences when used alone.

The experiments for plista highlight an interesting behavior. For recall@5 and recall@10, MF performs better than CoLDA for all considered observations. For recall@50 and recall@100, we observe two different behaviors where, first, CoLDA performs better than MF, and then MF outperforms CoLDA. We recall that the reported results are measured on the second half of the dataset (Stream Test and Train subset). Drifts may have occurred during the training phase, which is typically the case for plista. When measuring the recall@N, CoLDA is already weakened by the drifts that have happened and that were not taken into account. This leads to a point Figure 4: Recall@5, recall@10, recall@50, and recall@100 of our approach CoAWILDA and its variants on ml-100k.

Figure 5: Recall@5, recall@10, recall@50, and recall@100 of our approach CoAWILDA and its variants on plista.

where the information learned by the topic model hurts the quality of recommendation, and MF starts performing better than CoLDA. This change of behavior occurs at different moments, depending on the recall we are measuring. For a higher N (i.e., recall@50, recall@100), the performance of CoLDA remains above the performance of MF for a longer time than for a lower N (i.e., recall@5, recall@10). Top list recommendation is thus more affected by the deterioration of the topic model.

All experiments demonstrate the effectiveness of using CoAW-ILDA, the strength of which relies on adapting to changes occurring in the data. Methods that do not detect and adapt to these changes, i.e., CoLDA, perform worse than CoAWILDA.

CONCLUSION

In this paper, we address the problem of online recommendation in dynamic environments and we tackle several subjects at once: hybrid approach for recommendation (merging CF and CB), recommendation with concept drifts, and collaborative topic modeling. The solution we propose, which we call CoAWILDA, is designed for online recommendation where textual descriptions of items are provided as new items are becoming available. It combines the advantages of the drift detection method ADWIN, the flexibility of online LDA, and the online nature of incremental matrix factorization. In the proposed setting, user interactions arrive in a stream and the method adapts to new items becoming available and to user interactions. Leveraging the advantages of both document analysis and users' interactions, CoAWILDA is particularly suitable to alleviate the problem of cold start and offers an elegant and generic solution to deal with drifting item distributions. Since very few training is necessary, the algorithm is truly online and can run in real-time.

An experimental validation on two real-world datasets has shown the actual efficiency of the proposed methodology. CoAWILDA outperforms its variants and other incremental methods we evaluated. In particular, it has been shown that a topic model which is not adjusting to drifts can hurt the quality of recommendation to an extent where its removal results in a much better performance. For that matter, recommendation at the top of the list is firstly affected.

In the context of this paper, we focused on the task of topic mining on textual items, but the idea of combining item modeling with a drift detection method (in particular with ADWIN) can be extended to other domains. Future work will consider new domains and will investigate how adaptive sliding window drift detection can be extended to improve online recommendation.

Figure 1 :

 1 Figure 1: Generative model for Latent Dirichlet Allocation

( 1 )

 1 Compute likelihood L = p(w |LDA d ) for model LDA d . (2) Process L with ADWIN. (3) If ADWIN detects a drift for window decomposition W = W 0 W 1 : • Retrain LDA m based on the documents in W 1 . • Retrain LDA d based on the documents in W 1 . (4) Update LDA m from the new document based on the online LDA algorithm.

Algorithm 1 1 :

 11 Overview of CoAWILDA Data: set of observations O Input: number of latent factors K, learning rate η, regularization parameters λ u and λ i Output: P, Q for o in O do 2:

Figure 2 :

 2 Figure 2: Performance evaluation of online LDA and AW-ILDA for the task of document stream modeling on ml-100k (first subfigure) and plista, (second subfigure) using the measure of perplexity.

  CoAWILDA is the method we propose, combining ADaptive Window based Incremental LDA (AWILDA) for topic modeling and incremental MF for CF. For ml-100k, we set the number of topics K = 20, η = 0.04, λ u = 0.01, and λ i = 0.1. For plista, we set K = 10, η = 0.042, λ u = 0.01, and λ i = 0.1. • CoLDA relies on classical online LDA [23] for topic modeling and incremental MF for CF. It replaces AWILDA from CoAWILDA with classical online LDA. For ml-100k, we set K = 20, η = 0.05, λ u = 0.01, and λ i = 0.1. For plista, we set K = 10, η = 0.045, λ u = 0.01, and λ i = 0.1. • AWILDA denotes the method we propose for adaptive topic modeling. We try to use it for recommendation without

Figure 3 :

 3 Figure 3: DCG@Ni of our approach CoAWILDA and other variants and incremental methods for ml-100k (first subfigure) and plista (second subfigure), where Ni is the number of available items. The evolution of DCG@Ni with the number of evaluated observations is reported.

http://www.movielens.org

http://www.dbpedia.org


Nguyen Anh

Minh Mai

Pierre Duthon

Pascal Housam Salmane

Louahdi Khoudour

Alain Crouzil

Sergio A Velastin

Camera and LiDAR analysis for 3D object detection in foggy weather conditions

Keywords: self-driving car, 3D object detection, foggy weather condition, sensor fusion

Today, the popularity of self-driving cars is growing at an exponential rate and is starting to creep onto the roads of developing countries. For autonomous vehicles to function, one of the essential features that needs to be developed is the ability to perceive their surroundings. To do this, sensors such as cameras, LiDAR, or radar are integrated to collect raw data. The objective of this paper is to evaluate a fusion solution of cameras and LiDARs (4 and 64 beams) for 3D object detection in foggy weather conditions. The data from the two input sensors are fused and an analysis of the contribution of each sensor on its own is then performed. In our analysis, we calculate average precision using the popular KITTI dataset, on which we have applied different intensities of fog (on a dataset we have called Multifog KITTI). The main results observed are as follows. Performances with stereo camera and 4 or 64 beams LiDAR are high (90.15%, 89.26%). Performance of the 4 beams LiDAR alone decreases sharply in foggy weather conditions (13.43%). Performance when using only a camera-based model remains quite high (89.36%). In conclusion, stereo cameras on their own are capable of detecting 3D objects in foggy weather with high accuracy and their performance slightly improves when used in conjunction with LIDAR sensors.

I. INTRODUCTION

Nowadays, self-driving cars are becoming more and more popular in developed countries. These cars have to make accurate predictions such as lane detection, traffic sign detection and obstacle detection. Due to the increasing importance and attractiveness of autonomous vehicles, a number of self-driving car companies have emerged, and almost every traditional automaker has invested in this segment to some extent.

Much has been invested in researching hardware and software for self-driving cars, making it an extremely hot research topic. To better perceive the environment, self-driving cars are equipped with sensors such as cameras, radars and LiDARs. These sensors are used to collect data about the environment. Then, the data is fed into predictive models such as semantic classification, segmentation, and object detection to obtain the information needed for vehicle navigation. In this way, the vehicle can detect lanes, obstacles, and traffic signs to make decisions such as wheel speed or steering wheel angle. These models must perform with very high accuracy while maintaining high safety standards. Therefore, the accuracy of predictive models plays an important role in the operation of self-driving cars.

In previous studies on environment perception, either a camera-based [START_REF] You | Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving[END_REF], [START_REF] Wang | Pseudo-LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving[END_REF], [START_REF] Chen | 3D object proposals for accurate object class detection[END_REF], [START_REF] Chabot | Deep MANTA: A coarse-to-fine many-task network for joint 2D and 3D vehicle analysis from monocular image[END_REF], [START_REF] Xu | Multi-level fusion based 3d object detection from monocular images[END_REF] or LIDAR-based [START_REF] Shi | PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud[END_REF], [START_REF] Yang | Pixor: Real-time 3D object detection from point clouds[END_REF]- [START_REF] He | Structure aware singlestage 3D object detection from point cloud[END_REF] detection algorithms were developed. The data was collected under ideal conditions (daylight, without bad weather). Such studies showed that LiDAR-based models performed better than camera-based ones. This is because LiDAR is an active sensor able to provide a very accurate estimation of the distance from a self-driving car to an obstacle, while a camera does not, especially at long distances. Meanwhile, methods that combine both camera and LiDAR have not produced the desired results that could take advantage of the strengths of both systems [START_REF] Qi | Frustum PointNets for 3D object detection from RGB-D data[END_REF]- [START_REF] Vora | Pointpainting: Sequential fusion for 3D object detection[END_REF]. Since 3D object detection methods now provide stable results on data under normal weather conditions , more attention needs to be paid to study how these models can be used in extreme weather conditions such as rain, snow, or fog [START_REF] Mai | 3D Object Detection with SLS-Fusion Network in Foggy Weather Conditions[END_REF], [START_REF] Bijelic | Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather[END_REF]. Furthermore, we do not know which model (camera-based, LiDAR-based, or fusion-based method) works best under these conditions. There are many studies showing that data collected by LiDAR, and cameras are significantly distorted in extreme weather conditions, e.g. [START_REF] Mai | 3D Object Detection with SLS-Fusion Network in Foggy Weather Conditions[END_REF], [START_REF] Heinzler | Weather Influence and Classification with Automotive Lidar Sensors[END_REF], [START_REF] Kutila | Automotive LiDAR performance verification in fog and rain[END_REF]. However, there are not many studies that show the impact of these distorted data on the performance of the predictive model for 3D object detection. The main objective of this study is to analyze the contribution of the two sensors, i.e. camera and LiDAR, to 3D object detection in foggy weather conditions. The contribution of this paper is threefold:

• We divide and adapt the SLS-Fusion neural network [START_REF] Mai | Sparse LiDAR and Stereo Fusion (SLS-Fusion) for Depth Estimation and 3D Object Detection[END_REF] to take into account stereo camera or LiDAR separately. This leads to two different subneural networks. • We study the performance of each subneural networks in foggy weather conditions in comparison with SLS-Fusion.

• We then analyze the performance of the 3D object detection models (for stereo camera and LiDAR) according to six levels of fog applied to KITTI dataset. After this introduction, in Section II there is a brief summary of the state of the art in 3D obstacle detection by camera and LiDAR, the two sensors being taken together or separately. Section III describes the datasets used for the experiments. Section IV explains the methodology chosen for 3D object detection, the nature of the neural network used, and the methodology for using the network when considering stereo camera and LiDAR both jointly and separately, to analyze their influence. Finally, Section V presents the results on the performance of 3D object detection by camera and LiDAR. A conclusion and short-term perspectives (Section VII) ends the article.

II. RELATED WORK

This section places the proposed work in context. We distinguish between three main relevant topics: camera-based 3D object detection in Section II-A, LiDAR-based 3D object detection in Section II-B and fusion-based 3D object detection in Section II-C.

A. Camera-based 3D object detection

With the success of deep learning-based methods for camera-based classification and object recognition problems, this has been followed up and extended to 3D object detection. However, methods based only on camera sensors [START_REF] You | Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving[END_REF], [START_REF] Wang | Pseudo-LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving[END_REF], [START_REF] Chen | 3D object proposals for accurate object class detection[END_REF], [START_REF] Chabot | Deep MANTA: A coarse-to-fine many-task network for joint 2D and 3D vehicle analysis from monocular image[END_REF], [START_REF] Xu | Multi-level fusion based 3d object detection from monocular images[END_REF] have difficulty determining precise depth, the core element of the 3D vision problem. Deep MANTA [START_REF] Chabot | Deep MANTA: A coarse-to-fine many-task network for joint 2D and 3D vehicle analysis from monocular image[END_REF] is an early work on monocular 3D object detection. It finds the optimal position by matching 2D and 3D keypoints. To recover 3D geometry from 2D data, a 3D template is selected from a template database based on the predicted similarity between 3D templates. However, this requires maintaining a huge database of 3D models and discards valuable information that can be regressed directly from the image. By using several geometric properties, Deep3DBox [START_REF] Mousavian | 3D bounding box estimation using deep learning and geometry[END_REF] utilizes the property that the perspective projection of 3D corners should touch at least one of the two 2D bounding boxes. In the case of deterministic geometry constraints, any errors in 2D object detection are locked into the 3D estimation. In MLF [START_REF] Xu | Multi-level fusion based 3d object detection from monocular images[END_REF], a multi-stage fusion algorithm is used to combine image features with pseudo-LiDAR data. In that work, there is no optimization process, only depth is combined with an RGB image, which leads to suboptimal results. Since depth information is essential for 3D detection, pseudo point cloud using monocular or binocular data has been proposed [START_REF] Wang | Pseudo-LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving[END_REF]. That paper highlights the inefficiency of current methods for 3D object detection based on RGB-D images. A pseudo-3D point cloud in LiDAR coordinate system is generated by first predicting the depth map and then back-projecting. According to [START_REF] Wang | Pseudo-LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving[END_REF], image-based depth maps can be converted into a pseudo-LiDAR representation.

B. LiDAR-based 3D object detection

LiDAR does not provide information about the scene as clearly as an image, but the point clouds obtained from LiDAR are accurate estimations of the location of objects in space. This is an important contribution that helps make LiDARbased methods superior to camera-based methods for 3D object detection. They are usually divided into 3 types: multiview-based methods [START_REF] Yang | Pixor: Real-time 3D object detection from point clouds[END_REF], voxel-based methods [START_REF] Yan | SECOND: Sparsely Embedded Convolutional Detection[END_REF], [START_REF] Lang | Pointpillars: Fast encoders for object detection from point clouds[END_REF], [START_REF] He | Structure aware singlestage 3D object detection from point cloud[END_REF] and point-based methods [START_REF] Shi | PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud[END_REF], [START_REF] Yang | 3DSSD: Point-based 3D single stage object detector[END_REF]. In [START_REF] Yang | Pixor: Real-time 3D object detection from point clouds[END_REF], a new singlestage detector is proposed that performs 2D convolution on the Bird's-Eye-View (BEV). The objects to be detected do not overlap in the BEV representation, which maintains the physical scale. In addition, the BEV has a low computational cost and is one of the fastest detectors. In [START_REF] Yan | SECOND: Sparsely Embedded Convolutional Detection[END_REF], sparse convolution operations are proposed to reduce memory consumption and increase computational speed. In sparse convolution, convolution is performed only over non-empty voxels and not over all voxels, as in traditional 3D convolution. With many optimizations to adjust the representation differences, Faster RCNN [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] and Mask RCNN [START_REF] He | Mask R-CNN[END_REF] have been extended to point cloud representations and are proposed in PointRCNN [START_REF] Shi | PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud[END_REF]. The main advantage is to generate high quality 3D bounding box proposals from a point cloud.

C. Fusion-based 3D object detection

Since both camera and LiDAR data have their own advantages and disadvantages, various methods [START_REF] Qi | Frustum PointNets for 3D object detection from RGB-D data[END_REF]- [START_REF] Vora | Pointpainting: Sequential fusion for 3D object detection[END_REF] have been explored to incorporate both data into the same model, but overall no significant superiority has been shown. The Frustum PointNets method [START_REF] Qi | Frustum PointNets for 3D object detection from RGB-D data[END_REF] uses a standard 2D CNN object detector to extract 2D regions, and then converts the coordinates of the 2D regions into 3D space to create Frustum proposals. A PointNet-like block is used to segment each point within the frustum and obtain points of interest. Sophisticated 2D recognition methods are used to obtain prior knowledge, reducing the 3D search space and paving the way for its successors. This method depends heavily on the accuracy of 2D detectors, which is the main drawback of such a cascade approach. In PointPainting [START_REF] Vora | Pointpainting: Sequential fusion for 3D object detection[END_REF], semantic segmentation information from images is used to consolidate point clouds. To be precise, PointPainting first applies semantic information to each pixel to classify it. Then, a segmentation result, which is actually a compact outline of the image features, is "painted" onto the LiDAR points by projecting the LiDAR points directly onto the segmentation mask. Finally, a 3D detector can be used with LiDAR for applications such as localization and classification. PointPainting uses segmentation values instead of RGB attributes to enhance existing LiDAR-based networks. Pseudo-LiDAR++ [START_REF] You | Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving[END_REF] uses Pseudo-LiDAR [START_REF] Wang | Pseudo-LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving[END_REF] as the basis for developing a 3D detection architecture. To rectify the depth estimation, Pseudo-LiDAR++ uses sparse 3D measurements (synthetic 4 beams LiDAR). The architecture is independent of any kind of LiDAR. However, this camera and LiDAR fusion needs to be studied further for harsh weather conditions like fog.

III. DATASETS USED

Detecting objects in foggy weather conditions is an important task for a self-driving car. For this reason, datasets collected from sensors such as LiDAR and cameras must be effectively fused into the model to improve the perception system of autonomous vehicles. To achieve better 3D object detection via depth maps, we use a new approach called Sparse LiDAR and Stereo Fusion (SLS-Fusion) [START_REF] Mai | Sparse LiDAR and Stereo Fusion (SLS-Fusion) for Depth Estimation and 3D Object Detection[END_REF] to fuse the data from stereo camera and LiDAR sensors.

To enable the operation of the SLS-Fusion system, the synthetic Scene Flow dataset [START_REF] Mayer | A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[END_REF] is first used to train the neural network model for depth estimation. This dataset collection contains more than 35,000 stereo images with a size of 960×540 and a large ground truth variety for optical flow and disparity. After depth prediction, the KITTI dataset [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF] is used to train the 3D object detection part of the SLS-Fusion method. KITTI was chosen because it is one of the most common datasets for autonomous driving. It contains hours of real traffic scenarios recorded with both stereo and LiDAR sensors under normal weather conditions. A total of 7,481 training samples and 7,518 test samples are included. As detailed in [START_REF] Chen | Multi-view 3d object detection network for autonomous driving[END_REF], the training dataset is divided into two parts: training (3,712 samples) and validation (3,769 samples). On the other hand, there are few datasets recorded in extreme weather conditions. To address this problem, we decided to create a dataset, derived from KITTI, augmented with foggy weather conditions. To do this, we use a physics-based fog simulator [START_REF] Bijelic | Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather[END_REF] that converts normal weather data from KITTI to foggy weather data (this modified dataset is then called Multifog KITTI). In this dataset, the level of fog is characterised according to a visibility distance expressed in meters. Based on the results of the simulated foggy system, we divide foggy weather into six conditions or levels according to visibility (level 1: 20-29 m, level 2: 30-39 m, level 3: 40-49 m, level 4: 50-59 m, level 5: 60-69 m, and level 6: 70-79 m).

The detection results of the SLS-Fusion method were obtained by training with the Multifog KITTI data [START_REF] Mai | 3D Object Detection with SLS-Fusion Network in Foggy Weather Conditions[END_REF]. The performance of the 3D object detector can be analyzed by comparing the 3D predictor results with the annotated KITTI data results. However, after creating the new dataset by using the fog simulator, we found that many labeled objects (ground truths) that were visible without fog became invisible in fog, resulting in incorrect ground truths. For this reason, the ground truths were filtered. The bounding boxes, which have a greater distance to the sensor than the meteorological visibility, are removed and not taken into account in the evaluation. When we manually review the LiDAR and camera data, we see that this method does remove objects that are completely invisible.

IV. METHODOLOGY

This section presents the 3D object detection algorithm that we used in all experiments. The original model, which considered stereo camera and LiDAR simultaneously, was modified. Namely, a new representation was used for either the stereo camera or the LiDAR. The model combines stereo images (I l , I r ) with the corresponding stereo images (S l , S r ) generated by re-projecting LiDAR. Both images and point clouds are extracted using an encoder-decoder network. The backbone is inspired by that of DeepLiDAR [START_REF] Qiu | DeepLiDAR: Deep surface normal guided depth prediction for outdoor scene from sparse LiDAR data and single color image[END_REF]. Instead of only using left and right images as in [START_REF] Chang | Pyramid Stereo Matching Network[END_REF] and [START_REF] You | Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving[END_REF], the proposed network uses a weight-sharing pipeline for both LiDAR and images (I l and S l ) and (I r and S r ). Following the decoding, the left and right features are fed into the Depth Cost Volume (DeCV) found in [START_REF] You | Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving[END_REF] to calculate the depth information loss. Like in [START_REF] You | Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving[END_REF], we use the smooth

A. Initial model representation: SLS-Fusion

L1 loss function (u,v)∈I l |d(u, v) -D(u, v)| , (1)
where a valid depth ground truth is denoted by d(u, v). The depth map is denoted by D, where D(u, v) represents the depth corresponding to pixel (u, v) on the left image I l . Based on a pinhole model, pseudo point clouds are generated. Given the depth D(u, v) and camera intrinsic matrix, the 3D position (X c , Y c , Z c ) in the camera coordinate system for each pixel (u, v) is given by

(depth) Z c = D(u, v), (2a) (width) X c = (u -c U ) × Z c f U , ( 2b
) (height) Y c = (v -c V ) × Z c f V , (2c)
where c U and c V are the coordinates of the principal point and f U and f V are the focal length in pixel width and height, respectively. Following [START_REF] You | Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving[END_REF], LiDAR is used to improve the quality of the pseudo point cloud. Then, each point (X c , Y c , Z c , 1) is transformed into (X l , Y l , Z l , 1) in the LiDAR coordinate system (the real world coordinate system). Given

the camera extrinsic matrix C = R t 0 1
, where R and t are respectively the rotation matrix and the translation vector. The pseudo point cloud can be determined as follows

    X l Y l Z l 1     = C -1     X c Y c Z c 1     . (3)
Once the pseudo point cloud is obtained, it can be treated as a normal point cloud, although its accuracy depends on the quality of the predicted depth. Similar to Pseudo-LiDAR++ [START_REF] You | Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving[END_REF], the input (point clouds) is used to correct errors in the pseudo point cloud. This is a refinement step to obtain a more accurate point cloud. Then the depth map is converted to a pseudo point cloud. The idea is to leverage the power of the top leading LiDAR-based methods such as PointRCNN [START_REF] Shi | PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud[END_REF] to eventually detect objects.

B. SLS-Fusion utilization for LiDAR and camera separately

Like [START_REF] Wang | Pseudo-LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving[END_REF] and [START_REF] You | Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving[END_REF], SLS-Fusion [START_REF] Mai | Sparse LiDAR and Stereo Fusion (SLS-Fusion) for Depth Estimation and 3D Object Detection[END_REF] involves 3 main steps: depth estimation, data conversion, and LiDAR-based 3D object detection. In our experiments, we try to analyze the contribution of camera and LiDAR for the 3D object detection task. Thus, the two sensors are considered separately. shows the network representation for the stereo camera only.

We start from the global representation in Figure 1 and consider the depth estimation task. In this global representation, we indeed have 2 inputs to set up the depth map estimation: one for the LiDAR and one for the stereo camera. As it can be seen in Figures 2 and3, in the networks for the LiDAR and the stereo camera, the part that allows the prediction of the depth maps changes. The last step, obstacle detection, remains the same whether the depth map obtained from LiDAR or the depth map obtained from the stereo camera is used. This is the method used in SLS-Fusion.

As can be seen in Figure 2, the LiDAR has been eliminated. Thus, the output depth map is based on the stereo camera as input. In contrast, Figure 3 shows the model that uses LiDAR only. In this case, the input data is the LiDAR and the output depth map is based on the LiDAR only.

V. EXPERIMENTS A. Evaluation Metrics

Common metrics [START_REF] Everingham | The Pascal visual object classes (VOC) challenge[END_REF], [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF] are used to evaluate object detection algorithms. The Average Precision (AP) for both 3D and Bird's Eye View (BEV) is reported as AP 3D and AP BEV , respectively, with thresholds of 0.5 and 0.7 for Intersection over Union (IoU). The objects are classified into three difficulty levels: Easy, Moderate and Hard, depending on the size of the 2D bounding box, the occlusion, and the degree of truncation of the object appearing in the RGB image according to [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF]. The experiments are performed using the KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF] and Multifog KITTI [START_REF] Mai | 3D Object Detection with SLS-Fusion Network in Foggy Weather Conditions[END_REF] datasets. The original 3D object detection model SLS-Fusion [START_REF] Mai | Sparse LiDAR and Stereo Fusion (SLS-Fusion) for Depth Estimation and 3D Object Detection[END_REF] and its modified version presented in Section IV are used for all experiments.

B. Experimental Protocols

It has been shown that the combination of LiDAR and camera does not outperform the results of models using only LiDAR in normal weather conditions (without rain, fog). On the other hand, LiDAR and camera data are strongly affected by noise in foggy weather. In this case, we do not know what is the best choice: to continue to fuse the two sensors or to use them separately. That is the question we want to answer here. We perform the following experiments: we test the SLS-Fusion model with only camera or LiDAR inputs to see how each sensor contributes to the performance of the model when the data is affected by fog. To analyse the contribution of each sensor, we first compare the results on the KITTI dataset with those on the Multifog KITTI dataset. Knowing that the LiDAR gives poor results in foggy weather, we try to see if a LiDAR with more laser beams (64 instead of 4) would give better results in those adverse conditions.

When analysing fog data, the visibility interval is included in the data. For each level of visibility (level 1 to level 6), object detection performance is considered. This can then determine up to what visibility range the models can provide acceptable performance.

C. Implementation Details

The procedure used is quite similar to the common pipeline of Pseudo-LiDAR [START_REF] Wang | Pseudo-LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving[END_REF].

The depth estimation network is implemented in Pytorch. To achieve faster convergence, complete depth maps from the Scene Flow dataset are first used for training. Fine-tuning is then performed on the 3,712 training samples in the KITTI dataset for 100 epochs, with the batch size set to 4 and the learning rate set to 0.001. Work reported by You et al. [START_REF] You | Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving[END_REF] is followed to generate the simulated 4 beams laser LiDAR (which is as similar as possible to the 4 beams sensor called ScaLa) from the 64 beams LiDAR and then project it onto the left and right image planes to feed into the depth estimation network.

To make the pseudo point cloud closer to the real LiDAR signals, following [START_REF] Wang | Pseudo-LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving[END_REF], reflectance is set to 1 and points higher than 3 m from the ground are removed. Then the LiDARbased 3D object detector can be applied to these remaining points.

For 3D object detection, PointRCNN [START_REF] Shi | PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud[END_REF], a LiDAR-based method with a high performance, is used as a basis by many other methods. The method is developed to consider the sparse point cloud. In order to use the pseudo point cloud, we have to make a pre-processing. This latter consists in subsampling the pseudo point cloud in order to obtain a sparser point cloud compatible with a 64 beams LiDAR. Then, the released implementations of PointRCNN are directly used, and their guidelines followed to train it on the training set of KITTI object detection dataset only for the class "Car", since car is one of the main objects and occupies the largest percentage in KITTI dataset, which causes imbalance between "Car" and other classes.

VI. RESULTS AND DISCUSSION

As shown in Tables I, II and III, the results of different tests using the SLS-Fusion method are shown for IoU (Intersection over Union) of 0.5 and 0.7, respectively. In each cell of these tables, a pair of numbers A/B corresponds to the results obtained with the AP BEV /AP 3D metrics on the KITTI or Multifog KITTI datasets. In describing the results, we have considered the AP 3D metric.

In Table I, the experiments are divided into 2 parts. The upper part consists of experiments (1.1, 1.2, 1.3 and 1.4) on the KITTI dataset. In contrast, the experiments in the lower part (2.1, 2.2, 2.3, 2.4 and 2.5) were performed on the Multifog KITTI dataset.

The upper part 1.1 reports the experiment mentioned in [START_REF] Mai | Sparse LiDAR and Stereo Fusion (SLS-Fusion) for Depth Estimation and 3D Object Detection[END_REF]. It is a test of the SLS-Fusion model, with a stereo camera and a 4 beams LiDAR fused (L4+S) as inputs experimented on the KITTI dataset. In the next two experiments, we present the results in the same way, but 1.2 only takes a 4 beams LiDAR as input (L4) and 1.3 only takes stereo camera as input (S). In this part which concerns using the KITTI dataset without fog, one notices, as expected, that the fusion of the stereo camera and LiDAR provides good results 93.02% for Easy and IoU of 0.5. In this configuration, a stereo camera provides better results than a 4 beams LiDAR (89.39% versus 78.16%). Separating the two sensors results in lower performance. This prompts us to consider the two sensors jointly. On the other hand, if we consider the 64 beams LiDAR (line 1.4), the results are very promising (97.3% for Easy objects and an IOU of 0.5). Nonetheless, the 64 beams LiDAR price is very high compared to that of cameras and 4 beams LiDARs.

In the second part of Table I, we calculated the same indicators as before, but on data with fog (Multifog KITTI). Performance when fusing a stereo camera and a 4 beams LiDAR (L4+S) remains high (90.15%) for Easy objects, which demonstrates the viability of using both sensors at the same time. The performance of the 4 beams LiDAR alone (L4) decreases sharply (13.43%), which seems to show that the 4 beams LiDAR is not useful in fog, and it is the stereo camera (S) that has the highest contribution, as its performance remains very high (89.36%). The combination of stereo camera and 64 beams LiDAR provides good results (89.26% for Easy objects) and for most of the types of objects, but the stereo camera still plays the most important role.

Tables II, III and IV, show results on AP and according to the visibility distances (from level 1 to level 6). In this configuration, we consider 3D detection performance by combining a stereo camera and a 64 beams LiDAR analyzed by visibility distance. For the Easy category (IoU of 0.5), the combination of the two sensors (S+L64) leads to an AP greater than 82.52%. On the other hand, for the Moderate category (with an IoU of 0.7), visibility distance plays a role, and we have an AP gain of about 7.4% (difference between 46.71% at level 1 and 54.10% at level 6).

If we now look at the sensors separately and depending on the visibility distance, we see the following. For the 64 beams LiDAR (Table III), objects classified as Easy (with an IoU of 0.5) are detected at 71.11% for visibility level of 20 m (level 1), up to 84.95% for level 6. On the other hand, if we consider Moderate objects (with an IoU of 0.7) we see that at low visibility level 1, the AP is 17.24%. Conversely, the AP at level 6 is 35.19%. We see that the results are better when the visibility is better. However, the two results are bad given the safety requirements of autonomous vehicles. All this shows that in the presence of fog, the use of a LiDAR alone does not provide satisfactory results.

Table IV provides the results for the stereo camera taken into account alone, in foggy weather and as a function of the visibility distance. For Easy objects (IoU of 0.5) we notice that the AP does not vary much depending on visibility (maximum 96.17% of AP ). If we consider Moderate objects (with an IoU of 0.7) we can note that the AP varies according to the visibility distance, ranging from 46.70% to 54.09%. If we compare the results to those of a 64 beams LiDAR, we can clearly see the superiority of the stereo camera. Therefore, in foggy weather, LiDAR alone is not suitable.

to which fog has been added. The model used is a simple model, which considers only the macroscopic attenuation phenomenon. This model has been calibrated on tests performed in a controlled environment on standard sensors [START_REF] Bijelic | Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather[END_REF]. However, this model has two limitations. (i) First, it should be verified that it is valid for the sensors used in the KITTI dataset, because, depending on the sensor (brand, type, internal settings), the impact of fog can be more or less strong. In the case of the LiDAR, the threshold effects when passing from the raw signals to the point cloud can have a strong impact on the model we use. For the camera, the exposure setting is not considered here, and may once again have an impact not modeled here. (ii) The model used here does not consider the microscopic phenomena of light diffusion. Thus, the halo effects for the camera and the backscattering effects for a LiDAR sensor are not simulated here. These different elements are known limitations of the model and clearly explained. They can have an impact on the results, so the results presented here should not be taken as categorical, but as initial results allowing to compare sensors, and to find data fusion solutions adapted to the autonomous vehicle.

For future work, two options can be considered to circumvent the limitations of the model used here. The first would be to perform acquisitions on real site, under real adverse weather conditions. The second, more promising option, would be to use more complex 3D models that implement microscopic scattering phenomena and simulate the complete path of light from objects to the sensor and the sensor itself.

Fig. 1 :

 1 Fig.1: Initial part of the depth estimation of the SLS-Fusion algorithm[START_REF] Mai | Sparse LiDAR and Stereo Fusion (SLS-Fusion) for Depth Estimation and 3D Object Detection[END_REF] with stereo camera and LiDAR as inputs.

Fig. 2 :

 2 Fig. 2: Modified part of the depth estimation of the SLS-Fusion algorithm to adapt it for using only the camera as input.

Fig. 3 :

 3 Fig. 3: Modified part of the depth estimation of the SLS-Fusion algorithm to adapt it for using only the LiDAR as input.

VII. CONCLUSION

In this study, we have tested the capabilities of 3D obstacle detection using two types of sensors: two versions of LiDAR (4 beams and 64 beams) and a stereo camera. The developed algorithms were tested on the KITTI dataset and then with additional fog (Multifog KITTI). Based on obstacle detection performance, we have analyzed several aspects: the contribution of the two types of sensors both in normal weather and in fog, when they are combined and when they are used separately. The main result is that using LiDAR in foggy weather leads to a slightly worse obstacle detection performance (even worse when the LiDAR is a 4 beams laser sensor). On the other hand, results based on stereo camera are promising in foggy weather, regardless of level of visibility.

The results in Tables I, II, III and IV contain a lot of information that needs to be interpreted in detail. In the context of this article, and given the limited number of pages, we presented summarised results, with main tendencies, and we were not able to go in more depth in the interpretation. We plan to do this shortly in a separate publication.

As a reminder, the results of this study are obtained on real LiDAR and camera data, initially acquired in clear weather,