Giulia Marchello

Alexandre Destere

Marco Corneli

Charles Bouveyron

Deep dynamic co-clustering of count data streams: application to pharmacovigilance

Keywords: Co-clustering, change point detection, zero-inflated distributions, online inference, VEM algorithm, data streams

Co-clustering is a widely used technique that allows the analysis of complex and high-dimensional data in various domains. However, existing models mostly concentrate on continuous and dense data in fixed time situations, where cluster assignments remain unchanged over time. For example, in the field of pharmacovigilance, it is crucial to cluster in real time drugs and adverse effects simultaneously, facilitating the automation of safety signal detection processes. However, traditional co-clustering methods require all the data to be loaded into memory, which can be a challenge for large datasets or even impossible in certain scenarios. The proposed online co-clustering model is designed to overcome this challenge by processing the data incrementally, one step at a time. This work introduces a novel inference process for the latent block model that addresses the challenge of online coclustering of sparse data matrices. To properly model this type of data, we assume that the observations follow a time and block dependent mixture of zero-inflated distributions, thus combining stochastic processes with the time-varying sparsity modeling. To detect abrupt changes in the dynamics we make use of a Bayesian online change point detection method on both cluster memberships and data sparsity estimations. The inference relies on an original variational procedure whose maximization step trains a LSTM neural network in order to solve the dynamical systems. Numerical experiments on simulated datasets demonstrate the effectiveness of the proposed methodology in the context of count data streams. Then, we fit the model to a large-scale dataset supplied by the Regional Center of Pharmacovigilance of Nice (France), providing meaningful online segmentation of drugs and adverse drug reactions.

Introduction

The use of unsupervised machine learning techniques, such as clustering, to summarize data is increasingly needed. A topical application area is pharmacovigilance, whose main activity concerns in gathering drug associated adverse events to detect safety signals about drugs. Currently, the detection of safety signals heavily relies on manual expert analysis, leading to potential incompleteness due to the workload involved and the need for substantial data before critical events can be detected. Such methods would enable pharmacovigilance experts to focus on searching for ADRs in all the digital documents and reports generated by healthcare establishments. Consequently, there is a pressing need to develop automated methods for safety signal detection in pharmacovigilance. Clustering techniques can help in this task to effectively summarize data to detect safety signals along the time. In addition to pharmacovigilance, clustering methods are increasingly important in various domains such as social media, e-commerce, biomedical data, finance, and genetics. Traditional clustering approaches may struggle with high-dimensional sparse datasets, making co-clustering an appealing alternative. Co-clustering is useful in this context since it clusters both observations and features simultaneously, providing useful data summaries. Although many notable methods have been introduced in this field in recent decades, the development of dynamic coclustering methods remains largely unexplored. Also, in many applications and particularly in the case of pharmacovigilance, the data generating process can change over time, which can result in changes in the clustering patterns. For example, drugs may have unexpected adversarial effects in response to changes in the molecular principle, or new drugs may be introduced to the market. Online change point detection algorithms can detect when the data generating process has changed and trigger further investigations. In this paper, we address this challenge by exploring the development of an online model-based co-clustering tool for real-time safety signal detection. By treating adverse drug reaction (ADR) notifications as count data observed over time, our approach allows for the identification of temporal breaks in the safety signals. This facilitates the creation of alerts and provides room for further investigation by medical authorities. The primary objective of this research is to showcase the potential of our proposed method as a routine tool in pharmacovigilance.

Related Work

This section summarizes the related work in dynamic co-clustering and change point detection.

Co-clustering and Latent Block Models Co-clustering is a valuable tool for analyzing datasets as it allows for the simultaneous clustering of both observations and features. There are various types of co-clustering methods, that can be distinguished into metric based approaches, such as non-negative matrix tri-factorization (NMTF, [START_REF] Labiod | Co-clustering under nonnegative matrix tri-factorization[END_REF][START_REF] Ding | Orthogonal nonnegative matrix t-factorizations for clustering[END_REF], spectral co-clustering [START_REF] Dhillon | Co-clustering documents and words using bipartite spectral graph partitioning[END_REF], information theory [START_REF] Dhillon | Information-theoretic co-clustering[END_REF], and model-based co-clustering approaches (e.g. [START_REF] Bouveyron | Model-based clustering and classification for data science: with applications in R[END_REF].Among those approaches, model-based co-clustering is widely appreciated for its robust statistical foundations and adaptability to various data types and levels of sparsity. The cornerstone of model-based co-clustering is the popular latent block model (LBM) [START_REF] Govaert | Clustering with block mixture models[END_REF] that was initially introduced for the co-clustering of binary data matrices. LBM is based on the assumption that rows and columns are grouped into hidden clusters and that observations within a block (intersection of a row cluster and a column cluster) are independent and identically distributed. Whereas the original formulation of the model dealt with binary data only, the model has been extended in the last two decades to count data [START_REF] Govaert | Latent block model for contingency table[END_REF], continuous data [START_REF] Lomet | Sélection de modèle pour la classification croisée de données continues[END_REF], categorical data [START_REF] Keribin | Estimation and selection for the latent block model on categorical data[END_REF], ordinal data [START_REF] Jacques | Model-based co-clustering for ordinal data[END_REF][START_REF] Corneli | Co-clustering of ordinal data via latent continuous random variables and not missing at random entries[END_REF], functional data [START_REF] Bouveyron | The functional latent block model for the co-clustering of electricity consumption curves[END_REF], textual data [START_REF] Bergé | The latent topic block model for the co-clustering of textual interaction data[END_REF] and mixed-type data [START_REF] Selosse | Model-based co-clustering for mixed type data[END_REF]. Recently, [START_REF] Boutalbi | Tensor latent block model for co-clustering[END_REF] also proposed the tensor latent block model (TLBM) for coclustering, whose aim is to simultaneously cluster rows and columns of a 3D matrix, where covariates represent the third dimension. TLBM, in the same paper, is also implemented for different types of datasets: continuous data (Gaussian TLBM), binary data (Bernoulli TLBM) and contingency tables (Poisson TLBM).

Dynamic models for clustering and co-clustering While there has been an extensive body of literature, over the past decade, on static model-based clustering and co-clustering techniques the use of dynamic models in this context is a more recent development. It is worth noting that a more plentiful work has been made in the context of network clustering, in particular, for the Stochastic Block Model (SBM, [START_REF] Nowicki | Estimation and prediction for stochastic blockstructures[END_REF] than for LBM, although SBM is a special case of LBM, which does not need that data matrices to be square and/or symmetrical. [START_REF] Yang | Detecting communities and their evolutions in dynamic social networks-a bayesian approach[END_REF] proposed a dynamic version of SBM by allowing the cluster of each node to switch at time t+1 depending on its current state at time t, in a Markovian framework, where the switching probabilities are collected into a transition matrix. In a more general framework, [START_REF] Matias | Statistical clustering of temporal networks through a dynamic stochastic block model[END_REF] showed that, in dynamic SBMs, it is not possible to let vary over time both the connectivity parameters and cluster memberships without incurring into identifiability issues. Recently, [START_REF] Marchello | Co-clustering of evolving count matrices with the dynamic latent block model: application to pharmacovigilance[END_REF] proposed an extension of LBM allowing one to perform the simultaneous clustering of rows, columns and slices of a three dimensional counting tensor. Although being a first attempt to extend LBM to the dynamic case, this model has the limitation of not allowing cluster switches of rows/columns. Later the same authors proposed Zip-dLBM [START_REF] Marchello | A deep dynamic latent block model for the co-clustering of zero-inflated data matrices[END_REF], a further dynamic extension of LBM. In Zip-dLBM, the authors allow observations and features to change clusters over time, this paper has been proposed in a general way for any Zero-Inflated distribution, to model highly sparse scenarios, typical of interaction datasets. In a different framework, [START_REF] Casa | Co-clustering of time-dependent data via the shape invariant model[END_REF] extended the latent block model to deal with longitudinal data, relying on the shape invariant model [START_REF] Lindstrom | Self-modelling with random shift and scale parameters and a free-knot spline shape function[END_REF] and [START_REF] Boutalbi | Implicit consensus clustering from multiple graphs[END_REF] developed a model-based co-clustering method for sparse three-way data, where the third dimension can be seen as a discrete temporal one. Here, the sparsity is handled following the same assumption as in [START_REF] Ailem | Sparse poisson latent block model for document clustering[END_REF] that all blocks outside the main diagonal share the same parameter. However, while being a first step toward the dynamic expansion of co-clustering methods, these methods can be used mostly for macroscopic analysis, as they do not allow for the temporal dependence of variables.

Change point detection Change points refer to sudden shifts in the pattern of a time series dataset. Identifying these points can be valuable in analyzing and forecasting time series data. Also, these algorithms are specifically designed to detect the precise moment when a process that is evolving over time has undergone a significant change. This specific moment indicates a shift in the underlying process generating the observed data points. Several change point detection algorithms have been proposed in the literature (e.g. [START_REF] Basseville | Detection of abrupt changes: theory and application[END_REF]; [START_REF] Cheng | Multivariate max-cusum chart[END_REF]; Van den Burg and Williams (2020); [START_REF] Montgomery | Introduction to statistical quality control[END_REF]). These algorithms can be categorized into two types: online and offline. Online algorithms are designed to operate in real-time, with upcoming time series data. In contrast, offline algorithms are intended to run after the entire dataset has been collected. Since the goal here is to detect on the fly pharmacovigilance events, we focus hereafter on online methods only. Regarding online change point detection algorithms, recent studies have shown that among the most effective methods there are the likelihood and probabilistic approaches [START_REF] Kondratev | A comparison of online methods for change point detection in ion-mobility spectrometry data[END_REF][START_REF] Kavitha | Clustering time series data stream-a literature survey[END_REF]. On this subject, a seminal paper was proposed by [START_REF] Adams | Bayesian online changepoint detection[END_REF] introducing the Bayesian Online Change Point Detection (BOCD). The idea of BOCD is to identify change points using the so-called run lengths or segment. Whenever a new data point becomes available, the algorithm determines the likelihood of the corresponding run length increasing by one. If the probability of change is higher than that of growth, the run length resets to zero, and a change point is identified. Alternatively, another approach has been proposed by [START_REF] Kawahara | Change-point detection in time-series data by direct densityratio estimation[END_REF], introducing subspace identification for change point detection. Here, the change points are detected by estimating a state-space model behind time series.

Our contribution

This paper proposes an online extension of Zip-dLBM [START_REF] Marchello | A deep dynamic latent block model for the co-clustering of zero-inflated data matrices[END_REF], a coclustering technique designed to handle time-evolving data matrices that may contain many empty entries. We aim to introduce three novelties in this regard. The first is the ability of the estimation algorithm to work online, with streams of data. The second is the addition of an online change point detection method. By capturing the data dynamic behavior, the method can identify abnormal events that affect the generative process. To detect these changes we make use of the Bayesian Online Change Point Detection (BOCD, [START_REF] Adams | Bayesian online changepoint detection[END_REF] that runs on the estimated model parameters in real time. The algorithm iteratively updates the posterior probabilities of the change points, based on the data observed so far. The third novelty relies on a different modeling choice for the time evolving parameter. In fact fully connected neural networks are substituted with LSTMs, as their structure is deemed more appropriate for the purpose. Therefore, this model introduces a new approach by incorporating an online inference method for Zip-dLBM and online changing point detection.

Organization of the paper

This paper is organized as follows. Section 2 recalls the generative model Zip-dLBM. Section 3 introduces the proposed online inference for stream data. Section 4 presents various experiments on simulated data to test and evaluate the model performances. In Section 5, an application on a real ADRs dataset is presented to illustrate the potential of Stream Zip-dLBM in pharmacovigilance. Section 6 provides some concluding remarks.

Stream Zip-dLBM

The following section reminds the Zero-Inflated Poisson Dynamic Latent Block Model (Zip-dLBM, [START_REF] Marchello | A deep dynamic latent block model for the co-clustering of zero-inflated data matrices[END_REF] for batch processing. Notice that, although this model has been proposed in a general way for any Zero-Inflated distribution, we focus here on the Zero-Inflated Poisson (ZIP) distribution, since count data are considered in pharmacovigilance. In Zip-dLBM the observed data are assumed to be collected into time evolving matrices, over the interval [0, T ]. Being in a discrete time setting, we assume a time partition of equally spaced points:

0 = t 0 < t 1 < t u ≤ t U = T.
With a slight abuse of notation we denote by t the time point t u . At time t, the incidence matrix X(t) ∈ N N ×M has X ij (t) as generic element that counts the number of interactions between the observation i and feature j that took place between t-1 and t. The rows of X(t) are indexed by i = {1, ..., N } and the columns by j = {1, ..., M }. The goal of Zip-dLBM is to cluster both the rows and columns of a series of data matrices, {X(t)} t , with t ∈ [0, T ].

A Zero-Inflated Dynamic Latent Block Model

First, let us now consider that we have a time interval of fixed lenght, [0, T ], and that the model does not run online. The goal of Zip-dLBM is to cluster both the rows and columns of a series of data matrices, {X(t)} t , that change over time. The numbers of clusters for rows and columns are represented by Q and L, respectively, and remain constant over time. However, the cluster membership of each row and column is allowed to change in [0, T ].

Clusters modeling

To track the changes in cluster memberships over time, we use two evolving random matrices, Z(t) and W (t), for rows and columns, respectively. With Z(t) ∈ {0, 1} N ×Q , its the i-th row is denoted by Z i (t) and W (t) ∈ {0, 1} M ×L , its j-th row is denoted by W j (t). The two distributions are respectively parametrized by α(t) and β(t). More formally:

Z i (t) ∼ M(1, α(t) := (α 1 (t), . . . , α Q (t))), independently ∀i, ∀t ∈ {0, . . . , T }, (1)
where M(•, •) denotes the multinomial probability mass function and α q (t) = P{z iq (t) = 1}, with Q q=1 α q (t) = 1. Thus Z(t) := {z iq (t)} i∈1,...,N ;q∈1,...,Q represents the clustering of N rows into Q groups at a given time point t.

In a similar fashion, for the column clusters, we assume:

W j (t) ∼ M(1, β(t) := (β 1 (t), . . . , β L (t))), independently ∀j, ∀t ∈ {0, . . . , T }, (2)
where

β (t) = P{w j (t) = 1} and L =1 β (t) = 1.
The two random vectors Z and W are further assumed to be independent.

Sparsity modeling

In order to model a potentially extreme sparsity, the observed data are assumed to follow a mixture of block-conditional Zero-Inflated Poisson (ZIP) distributions, where the entries X ij (t) are conditionally independent in (i, j, t):

X ij (t)|Z i (t), W j (t) ∼ ZIP (Λ Z i (t),W j (t) , π(t)) , (3)
where Λ is the Q × L block-dependent intensity function of the Poisson component, and π(t) is a vector of length T that indicates the level of sparsity at any given time period. In order to ease the inference, we finally provide an equivalent formulation of the above equations in terms of a hidden random matrix, A ∈ {0, 1} N ×M , where, independently for all i and j:

A ij (t) ∼ B(π(t)),
with B(•) denoting the Bernoulli distribution of parameter π(t) and such that:

A ij (t) = 1 ⇒ X ij (t)|Z i (t), W j (t) = 0 A ij (t) = 0 ⇒ X ij (t)|Z i (t), W j (t) ∼ P(X ij (t), Λ Z i (t),W j (t) ), (4)
where P(•, Λ) denotes the Poisson distribution with intensity parameter Λ.

Modeling the temporal evolution of the parameters

We assume that the evolution of the mixing proportions α, β, and the sparsity parameter π are governed by a system of ordinary differential equations (ODEs). By using ODEs, we can model the temporal evolution of both the composition of clusters and sparsity. Since we work in discrete time we discretize the dynamic systems by making use of their Euler schemes:

     a(t + 1) = a(t) + f Z (a(t)), b(t + 1) = b(t) + f W (b(t)), c(t + 1) = c(t) + f A (c(t)), (5)
where f Z , f W and f A are assumed to be three continuously differentiable functions and:

     α q (t) = softmax(a q (t)) = e aq(t) / Q q=1 e aq(t) , β (t) = softmax(b (t)) = e b (t) / L =1 e b (t) , π(t) = e c(t) /1 + e c(t) . (6)

The joint distribution

The set of the model parameters is denoted by θ = (Λ, α, β, π) and the latent variables used so far are Z, W , and A, where we denote α {α(t)} t , β {β(t)} t , π {π(t)} t . Thus, the likelihood of the complete data reads:

p(X, Z, W, A|θ) = p(X|Z, W, A, Λ, π)p(A | π)p(Z|α)p(W |β). (7)
The terms on the right hand side of the above equation can be further developed, as in [START_REF] Marchello | A deep dynamic latent block model for the co-clustering of zero-inflated data matrices[END_REF].

Online inference for stream data

In this section, we present the online extension of the Zip-dLBM method, called Stream Zip-dLBM. The objective is to perform co-clustering of rows and columns in real-time as new data become available. To prevent memory overload, we have revisited the original inference algorithm of Zip-dLBM, enabling the data to be processed without the need to store it in memory. To allow the algorithm to update the parameter estimates continuously as a new data is incorporated, we use a moving window, G d (t), of size d. In more detail, at time t, we keep in memory only the data in the interval [t-d, t], namely X(t-d), X(t-d+1), . . . , X(t), that will be used for the estimation of the model parameters. The data outside the interval can be discarded to prevent memory overloads and maintain the algorithm's functionality. Once a time point t quit the time window (after passing through it) the parameter estimates at that point become fixed, and the "past" data can be discarded by the inference procedure.

Variational Assumption

The traditional approach to estimate the model parameters involves maximizing the loglikelihood p(X|θ). However, in our case a direct maximization is not feasible. Neither we can adopt the EM-algorithm, because the joint conditional distribution of the latent variables p(Z, W, |X, θ) cannot be computed, due to the interdependent double missing structure of (Z, W ) and their continuous evolution. Additionally, we cannot update α, β and π using closed formulas because of their connection with systems of ordinary differential equations. Therefore, we use a combination of Variational-EM inference and Stochastic Gradient Descent (SGD) to estimate the parameters. Let us thus introduce a variational distribution q(.), over (A, Z, W ), in order to decompose the observed data log-likelihood as follows:

log p(X|θ) = L(q, θ) + KL(q(.)||p(.|X, θ)),
where L denotes a lower bound of p(X|θ) and is defined as:

L(q, θ) = Z W A q(Z, W, A) log p(X, A, Z, W |θ) q(Z, W, A) = E q(A,Z,W ) log p(X, A, Z, W |θ) q(A, Z, W = E q(A,Z,W ) [log(p(X, A, Z, W |θ)] -E q(A,Z,W ) [log(q(A, Z, W ))], (8)
and KL indicates the Kullaback-Liebler divergence between the true and the approximate posterior q(•):

KL(q(.)||p(.|X, θ)) = - A Z W q(A, Z, W ) log p(A, Z, W |X, θ) q(A, Z, W ) .
Now, the objective is to find a distribution q(.) that maximizes the lower bound L(q, θ). In order to allow the optimization of L(q, θ), we further assume that q(A, Z, W ) factorizes as follows for all t:

q(A(t), Z(t), W (t)) = q(A(t))q(Z(t))q(W (t)) = N i=1 M j=1 q(A ij (t)) N i=1 q(Z i (t)) M j=1
q(W j (t)).

(9)

Variational E-Step

The optimal variational updates of q(•), under the assumption in Eq. ( 9), can be obtained as in Bishop (2006, Ch.10). Denoting by δ ij (t) := q(A ij (t) = 1) the variational probability of success for A ij (t), the optimal update is:

δ ij (t) = exp(R ij (t)) 1 + exp(R ij (t)) , (10)
with:

R ij (t) = log(π(t)1 {X ij (t)=0} ) + Q q=1 L =1 -E q(W,Z) [Z iq (t)]E q(W,Z) [W j (t)]X ij (t) log Λ q + + E q(W,Z) [Z iq (t)]E q(W,Z) [W j (t)]Λ q + log X ij (t)! -log(1 -π(t)).
Note that, formally, when X ij (t) = 0, R ij (t) = -∞ and δ ij (t) = 0, which makes sense: nonnull observations in X come from a Poisson distribution with probability one (see Eq. ( 4)).

Regarding q(Z), let us denote by τ iq (t) := q(Z iq (t) = 1) the variational probability of success of Z iq (t). The optimal update can be written as:

τ iq (t) = r iq (t) Q q 0 =1 r iq 0 (t) , (11)
with r iq (t) is denoted by:

r iq (t) ∝ exp M j=1 L =1 (1 -E q(A,W ) [A ij (t)]) E q(A,W ) [W j (t)]X ij (t) log(Λ q )+ -E q(A,W ) [W j (t)]Λ q + log(α q (t)) .
Similarly for the latent variable W , denoting by η j (t) := q(W j (t) = 1) the variational probability for W j (t), the optimal update of q(W ) is:

η j (t) = s j (t) L o=1 s j o (t) , (12)
where :

s j (t) ∝ exp N i=1 Q q=1 (1 -E q(A,Z) [A ij (t)]) E q(A,Z) [Z iq (t)]X ij (t) log(Λ q )+ -E q(A,Z) [Z iq (t)]Λ q + log(β (t)) .
The proofs of Equations 10, ( 11) and ( 12) are provided in [START_REF] Marchello | A deep dynamic latent block model for the co-clustering of zero-inflated data matrices[END_REF]. It is worth noting that these update equations can be executed step by step, allowing for incremental updates of the variational parameters. Also, note that the update in these equations can be computed independently for any pair (i, j), at any time point t.

Online variational M-Step

While the updates in the E-step for τ (t), η(t), and δ(t) depend solely on the current time instant t, the same cannot be said for the updates in the M-step. The M-step involves updating the model parameters, such as θ = (Λ, α, β, π), based on the current estimates obtained in the E-step. In order to obtain the updates of the parameter set θ, the objective of the M-Step is the maximization of the lower bound L(q, θ) with respect to θ = (Λ, α, β, π), while holding the variational distribution q(•) fixed. Denoting t as the current time instant, we develop Eq.( 8) such that the variational lower bound L(q, θ) can be written as:

L(q, θ) = t u=1 N i=1 M j=1 δ ij (u) log(π(u)1 {X ij (u)=0} ) + (1 -δ ij (u)) log(1 -π(u))+ + Q q=1 L =1 τ iq (u)η j (u)X ij (u) log Λ q -τ iq (u)η j (u)Λ q -(1 -δ ij (u)) log(X ij (u)!) + t u=1 N i=1 Q q=1 τ iq (u) log(α q (u)) + t u=1 M j=1 L =1 η j (u) log(β (u)) - t u=1 N i=1 Q q=1 τ iq (u) log(τ iq (u))+ - t u=1 M j=1 L =1 η j (u) log(η j (u)) - t u=1 N i=1 M j=1 δ ij (u) log(δ ij (u)) + (1 -δ ij (u)) log(1 -δ ij (u)) .
(13)

Update of Λ

Here our goal is to derive the online update of the Zero-inflated Poisson intensity parameter, Λ. The variational distribution q(A, Z, W ) is kept fixed, while the lower bound is maximized with respect to Λ at every time instant t, to obtain its update, Λ. To find the optimal update we compute the derivative of the lower bound L(q, θ) in Eq. ( 13) with respect to Λ and set it equal to zero, as follows:

∂ log L(q, θ) ∂Λ q = N i=1 M j=1 t u=1 (1 -δ ij (u)) τ iq (u)η j (u)X ij (u) Λ q -τ iq (u)η j (u) = 0 ⇔ N i=1 M j=1 t u=1 (1 -δ ij (u)) τ iq (u)η j (u)X ij (u) -τ iq (u)η j (u)Λ q = 0 ⇔ N i=1 M j=1 t u=1 (1 -δ ij (u))τ iq (u)η j (u)Λ q = N i=1 M j=1 t u=1 τ iq (u)η j (u) X ij (u) -X ij (u)δ ij (u) ⇒ Λq = N i=1 M j=1 t u=1 τ iq (u)η j (u) X ij (u) -δ ij (u)X ij (u) N i=1 M j=1 t u=1 τ iq (u)η j (u) 1 -δ ij (u) . ( 14
)
Although the update of Λq sums over all the past observations (X ij (1), . . . , X ij (t)), we can develop Eq. ( 14) as follows:

Λq = N i=1 M j=1 (t-1) u=1 τ iq (u)η j (u) X ij (u) -δ ij (u)X ij (u) + N i=1 M j=1 τ iq (t)η j (t) X ij (t) -δ ij (t)X ij (t) N i=1 M j=1 (t-1) u=1 τ iq (u)η j (u) 1 -δ ij (u) + N i=1 M j=1 τ iq (t)η j (t) 1 -δ ij (t) = N old q + N (t) q D old q + D (t) q = N old q D old q + D (t) q + N (t) q D old q + D (t) q . ( 15
)
By splitting Λ in two different parts, we can distinguish between a part known at time t -1, namely N old q and D old q , and the current updates at time t, N

q and D (t)

q . Then, we divide and multiply the first term for D old q , such that, denoting Λold q = N old q D old q , we obtain the final online update:

Λq = Λold q • D old q D old q + D (t) q + N (t) q D old q + D (t) q , (16)
Hence, when a new observation comes Λ can be updated thanks to Eq.( 16), along with the update of D old q .

3.3.2 Update of α, β and π through deep neural networks

As mentioned in Section 2.1.3, the mixture proportions α and β, as well as the sparsity parameter π are driven by three systems of differential equations, respectively. Hence, the update process for these parameters in the online inference algorithm poses a challenge because they lack closed-form updating formulas. As a result, the decomposition strategy used for updating Λ cannot be applied. To address this issue, we introduce an approximation technique that leverages a moving window G d (t) of size d, allowing us to update the parameters based on the most recent d observations. In addition to its role in parameter updates, the moving window G d (t) serves another purpose as the input for a deep neural network. As we assumed that the functions f A , f W and f Z are continuous, we propose to parametrize them with three LSTM networks [START_REF] Hochreiter | Long short-term memory[END_REF]. LSTM is a type of recurrent neural network that operates on sequences of a specific length and produces a sequence of the same length, shifted one time step ahead. For instance, let's consider the current time t and the time window G d (t) with a length of d. The input for the LSTM networks consists of a series of values ranging from t -1 -d to t -1, representing the historical observations within the window. The LSTM networks then predict a sequence of values from t -d to t, which correspond to the updated parameter values for α, β, and π. Optimizing the lower bound in Eq.( 13) with respect to α, β, and π reduces to maximize it with respect to the parameters of the neural networks. For instance, the loss function for α can be expressed as follows:

L = u∈G d (t) N i=1 Q q=1 τ iq (u) log α q (u), (17)
The loss functions of β and π can be similarly derived using their respective distributionspecific equations. In the experiments, this update is implemented in PyTorch via automatic differentiation [START_REF] Paszke | Automatic differentiation in pytorch[END_REF] and relies on stochastic optimisation (ADAM, [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. Once the neural nets are trained via back-propagation (SGD) they provide us with the current ML estimates of α(t), β(t) and π(t). The whole inference procedure is summarized in Algorithm 1.

Initialization and model selection

When it comes to clustering methods based on the EM algorithm, the initialization process and selecting the appropriate number of clusters (for rows and columns in this case) are two important aspects that require careful consideration. These issues become slightly more complex when deep neural networks are employed to model cluster dynamics and sparsity proportions. Despite the inherent complexity introduced by these networks, they offer unexpected flexibility that can help reduce the computational burden of the entire algorithm. This becomes evident through the results of numerical experiments, as shown in Section 4.2.

In this section, modifications to the initialization method presented in [START_REF] Marchello | A deep dynamic latent block model for the co-clustering of zero-inflated data matrices[END_REF] are proposed to make the algorithm adaptable to run online. First, we select the first slice of the data X t 0 and apply on it a static LBM algorithm for a list of pairs of cluster numbers, i.e. (q, ) for q = 2, . . . , Q max and = 2, . . . , L max . Subsequently, we employ the ICL criterion (Integrated Completed Likelihood, [START_REF] Biernacki | Assessing a mixture model for clustering with the integrated completed likelihood[END_REF]) to determine the optimal number of row and column clusters for this particular subset of data. The ICL criterion approximates the integrated log-likelihood of the complete dataset and can be derived as follows:

ICL(Q, L) = log p(X, Ẑ, Ŵ ; θ) - Q -1 2 log N - L -1 2 log M - QL 2 log(N M ) - 1 2 log(N M ). (18)
The pair ( Q, L) that leads to the highest value for the ICL is considered as the most meaningful cluster numbers for the considered slice of data X t 0 . However, as we expect that the choice of Q row and L column cluster components could not be the best for all the future time instants, the VEM-SGD algorithm (see Algorithm 1) will be then run with more components than the ones found by the ICL. Indeed, we run the VEM-SGD algorithm with Q max ≥ Q and L max ≥ L cluster components. Then, every time there is a new data entry, part of the model parameters are initialized with θ(t) obtained via a static run of LBM and the remaining parameters, corresponding to the additional row and column clusters are set to zero.

Therefore, our objective is to leverage the advantage of using deep neural networks, which enables our VEM-SGD algorithm to initialize with empty clusters. These empty clusters can potentially be activated in the future, if required. As a result, we can avoid the typically computationally intensive task of running the entire algorithm with all possible combinations of row and column cluster numbers for the complete dataset. This strategy enables our approach to handle large-scale datasets within a reasonable computation time while achieving satisfactory results, as demonstrated in the forthcoming section. Also, at the initial stage of the algorithm (i.e. the first d time points), the parameters α(t), β(t) and π(t) are modeled via a two-layer fully connected neural networks, following the approach of [START_REF] Marchello | A deep dynamic latent block model for the co-clustering of zero-inflated data matrices[END_REF], until t = d. Then, as explained in Section 3.3.2, at t = d + 1, the estimates from the previous time step, obtained via fully connected networks, serve as input to LSTM, which is used for online parameter estimation from this point on.

Algorithm 1 VEM-SGD Algorithm for Stream Zip-dLBM Require: X, Q, L, Q max , L max , max.iter, G d (t).
Initialization of τ (t = 0) and η(t = 0): sampling from M(α(t = 0)) and M(β(t = 0)), respectively; Initialization of δ(t = 0): matrix of 1, then setting δ(t = 0) = 0 when X > 0; while New observations X(t) come: do Initialization of α(t), β(t), π(t), Λ with LBM; % with Q, and L Add Q max -Q columns of zeros to α(t); Add L max -L columns of zeros to β(t); Add Q max -Q rows and L max -L columns of zeros to Λ; for it = 1 to max.iter do VE-Step: for p = 1 to Fixed.Point do alternatively update δ(t), τ (t), η(t); % fix point eqs end for M-Step: Update θ = (Λ, π(t), α(t), β(t)).

Λq = Λold q • D old q D old q + D (t) q + N (t) q D old q + D (t) q .
Update α(t), β(t), π(t) %LSTM on the moving window t ∈ G d (t) end for Discard all the observation before G d (t) end while

Bayesian online change point detection

As previously stated, one of the aims of Stream Zip-dLBM is to perform multivariate online change point detection. To accomplish this task, we combine the Bayesian Online Change Point Detection (BOCD) method, proposed in a seminal paper by [START_REF] Adams | Bayesian online changepoint detection[END_REF], with our strategy. BOCD detects change points based on the estimation of the posterior distribution over the current "run length", or time segment since the last change point, given the data observed so far, using a simple message-passing algorithm. Essentially, the run length is used to determine if a new data point belongs to the current partition based on previous observations. If the new data point belongs to the current partition, the run length will increase by 1 at the next time step, otherwise it will reset to 0. This process is continuously repeated at each time step. It is worth noticing that the BOCD algorithm is typically implemented in an online fashion, analyzing the data as it streams in. However, in our case, we directly apply the algorithm to detect change points on the estimates of α(t), β(t), and π(t) that are generated by the LSTM. To prevent detecting change points on parameters that will be recalculated in future time steps, we run the BOCD algorithm only on time points "behind" G d (t). Stated differently, at time t, BOCD operates on parameter values at time instances t -d.

Numerical experiments

The main purpose of this section is to highlight the most important features of Stream Zip-dLBM over simulated datasets and to demonstrate the validity of the inference algorithm presented in the previous sections. The first experiment consists in applying Stream Zip-dLBM to a specific dataset with evolving block pattern and sparsity to show that it recovers the data structure in real-time. While the second experiment demonstrates the model selection procedure on a simulated dataset.

Introductory example

A simulated dataset with dimension 350 × 300 × 150 has been generated according to our model to perform this experiment. The simulated dynamics of α, β and π can be seen on the left-hand side of Figure 2. Concretely, α, β and π are three time series independently fluctuating around constant trends. Fluctuations are obtained at each time by means of independent Gaussian distributions with constant variance. The mean levels change when a change point occurs. The levels of the simulated change points and the values of the simulated parameter Λ are indicated in Table 1. Based on the mixture proportions α, β, the values of the latent variables were then simulated through their distributions. Next, we used the sparsity proportions, π, and the intensity parameter, Λ, to simulate the three-dimensional tensor X as Zero-Inflated Poisson variables. We then applied the Stream Zip-dLBM inference algorithm to the simulated dataset, using the actual values of Q = 3 and L = 2 to demonstrate the model's ability to recover the parameters. Figure 2 on the left side and the online estimates on the right side. The red dashed lines depict the simulated and estimated change points, respectively. From these results we see that Stream Zip-dLBM perfectly recovers the evolution of the mixing proportion and the sparsity parameter over time, including the change points.

Model selection experiment

In this experiment, we assess the ability of Stream Zip-dLBM to determine the optimal number of clusters for rows and columns. Initially, we utilize the Integrated Completed Likelihood (ICL) criterion to compute the optimal number of clusters on the first slice. This ensures that the algorithm is initialized with the best possible parameters. Once initialized, the algorithm maintains consistent results without making any alterations, thus retaining the optimal number of clusters. To evaluate the effectiveness of the algorithm and verify if it activates new clusters, we generate a dataset based on the configuration described in Section 4.1. The dataset consists of 3 row clusters (Q=3) and 2 column clusters (L=2). We then apply Stream Zip-dLBM to this dataset, with maximum values of Q max = 7 and L max = 7. Figure 3 provides an illustrative demonstration of the algorithm's behavior, specifically regarding the activation of clusters. It is clear from the figure that the unnecessary clusters remain empty and that the estimates of the α, β, and π parameters are also accurate. Finally, it is worth noticing that Stream Zip-dLBM successfully identifies the changing points in α, β and π over time, despite not using the optimal number of input clusters. Finally, to evaluate the performance of the model in identifying the correct rows and columns partitions, we use the adjusted Rand index (ARI) [START_REF] Rand | Objective criteria for the evaluation of clustering methods[END_REF]. The adjusted Rand index, from a mathematical point of view, is closely related to the accuracy measure, however it is a commonly used method for evaluating clustering problems since it can be applied for measuring the similarity between two partitions even with different number of clusters and it is invariant to label switching. We also use a measure called CARI, recently introduced by [START_REF] Robert | Comparing high-dimensional partitions with the Co-clustering Adjusted Rand Index[END_REF]. This new criterion is based on the Adjusted Rand Index [START_REF] Rand | Objective criteria for the evaluation of clustering methods[END_REF] and it was developed especially for being applied to co-clustering methods. The closer these indexes are to 1, the more two label vectors are similar to each other. We compared the original matrices Z and W , with the estimates τ and η given by the output of Stream Zip-dLBM. We evaluate the performance indexes at each time step, obtaining the following results: ARI rows ARI columns CARI 0.99 ± 0.03 1 ± 0 0.99 ± 0.02 Thus, we can conclude that our algorithm satisfyingly identifies the composition of the original clusters in time.

Analysis of the adverse drug reaction dataset

This section focuses on the online application of Stream Zip-dLBM to a large-scale pharmacovigilance dataset, with the aim of illustrating the potential of the tool for such studies.

Protocol and data

This section considers a large dataset consisting of an adverse drug reaction (ADR) dataset, collected by the Regional Center of Pharmacovigilance (RCPV), located in the University Hospital of Nice (France). The RCPV survey an area of three departments totalling 2.3 millions inhabitants. A time horizon of 7 years is considered, from January 1 st , 2015 to March 3 rd , 2022. Since the data are extremely sparse, we aggregate them summing up along the time dimension, such that one time instant corresponds to one month. The overall dataset is made of 39,267 declarations, for which the market name of the drug, the notified ADR and the reception date are considered. Moreover, we only considered drugs and ADRs that were notified more than 10 times over the 7 years. The resulting dataset contains 419 drugs, 614 ADRs and 87 time intervals with 23,264 non-zero entries. To prevent medicines with the same molecule but marketed under different brand names from being considered more than once, we decided to use the international nonproprietary name (INN) of the drug (to simplify the comprehension, the INNs would be referred as drugs for the rest of the study). Looking at Figure 4, it can be clearly noticed that there are at least two peaks to detect, one in 2017 and the other one in 2021. In 2017, an unexpected rise of reports for ADRs happened concerning a specific drug called Lévothyrox ® , containing the active compound levothyroxine. This has been marketed in France for about 40 years as a treatment for hypothyroidism and, in 2017, to correct drug stability problems, a new formula (with an innocuous change of excipients) was introduced on the market. The Lévothyrox ® case had an extremely high media coverage in France: Lévothyrox ® spontaneous reports represent almost the 90% of all the spontaneous notifications that the RCPV received in 2017 [START_REF] Viard | Spontaneous adverse event notifications by patients subsequent to the marketing of a new formulation of levothyrox® amidst a drug media crisis: atypical profile as compared with other drugs[END_REF]. In addition, since the end of the year 2020, vaccinations against Covid-19 have been introduced. At that time, three vaccines are licensed in Europe, Comirnaty ® was the first Covid-19 vaccine available in France in December 2020, followed by Moderna ® in January 2021 and Vaxzevria ® in February 2021.

From Figure 4, one can understand the difficulty to work with such data which contain signals of very different amplitude. Indeed, behind those very visible effects, many other ADR signals, somehow less trivial, need to be detected for obvious public health reasons. In particular, those data also contain ADR reports regarding another public health problem which occurred in 2017, involving Mirena ® , which is here far less visible than Lévothyrox ® , but also led to many avoidable health policy concerns. This is why, we expect Stream Zip-dLBM to be a useful tool to reveal such hidden signals. It is important to highlight that the data being used exhibit extreme sparsity, ranging from a minimum of 99.25% to a maximum of 99.98% per month. To avoid encountering numerical issues, the LSTM network was employed only for inferring the parameters α and β. Point estimates of π were used in the inference process. -12-31 2015-08-31 2016-04-30 2016-12-31 2017-08-31 2018-04-30 2018-12-31 2019-08-31 2020-04-30 2020-12-31 2021-08-31

Summary of the results

To initialize the algorithm, as explained in Section 3.4, we computed the ICL criterion on one data slice, corresponding to the first month, where the optimal numbers of clusters identified by the model selection criterion are Q = 3 and L = 3. Then, we run Stream Zip-dLBM and every time a new entry is added in the tensor X. We initiate the model parameters through the process described in Section 3.4. Also, we ran Stream Zip-dLBM with Q max = 7 and L max = 7 to allow the model to fill or empty clusters as needed. The process takes a running time of about one hour on a MacBook Pro, 2020, with a processor of 2,3 GHz Quad-Core Intel Core i7 and 16 GB of RAM. In Figure 5a, the frequency of declarations received by the RCPV from 2015 to 2022 is depicted, organized by month. Here, the identified change points are represented by dashed lines. Specifically, the green dashed lines indicate the change points detected in the evolution of drug clusters, while the blue dashed lines indicate the change points detected in the evolution of ADR clusters. Figure 5b displays the estimated Poisson intensity parameter, Λ. This figure only focuses on the 3 groups of drug clusters, denoted by the letter D, and the 3 groups of ADR clusters, denoted by the letter A, that have been activated in the inference. This representation provides valuable insights for model interpretation as it gives an overview of the relationships between drug clusters and ADR clusters and how they evolve over time. Each color refers to a drug (rows) or ADR (columns) cluster and the higher is the value in each block, the strongest is the relationship (i.e the expected number of declarations received in the time unit) between the related pair of clusters. Looking at this figure, it can be seen that some clusters are strongly related whereas others are not at all. Figures 5c and5d display the estimated mixture proportions of drug clusters (α) and ADR clusters ( β) respectively. The dashed lines in the figures represent the change points identified by the BOCD algorithm. Figure 6 illustrates the estimated sparsity parameter, with the y-axis scale ranging from 0.99 to 1. It is worth noting that no change points have been detected in π because the values did not exhibit sufficient variability. By examining the information provided in Figures 5b,5c, and 5d, we can observe an interesting pattern. The clusters with the highest intensity are also the least populated. For instance, cluster D3 (drug cluster) demonstrates a remarkably high intensity of interactions with clusters A1 and A2 (ADR clusters). Despite its high intensity, cluster D3 appears to be relatively small in Figure 5c. This phenomenon is attributed to the presence of drugs associated with significant health crises that happened during the reporting period. Notably, Mirena ® in the first half of 2017, Lévothyrox ® in the latter part of 2017, and Covid-19 vaccines throughout 2021 were the primary drivers of these crises. Interestingly, each crisis period is marked by a detected change point in both the drug cluster proportions (Figure 5c) and ADR cluster proportions (Figure 5d). Similarly, by analyzing the composition of clusters A1 and A2, it is possible to identify which ADRs were the most reported in each of the aforementioned crises. For instance, from the composition of cluster A2 we notice that the most reported side effects during the Mirena ® health crisis are mostly hormonal ones, such as anxiety, heat shock, and aggressive behavior.

Then, looking at Figure 5d, during the Lévothyrox ® health crisis we notice a peak in the A3 cluster of adverse drug reactions, probably because the great media coverage that the scandal had in those years made people declare the most disparate side effects. Also, we see that in 2021 there is another peak, corresponding to the period of the Covid-19 vaccination. Here, the adverse drug reactions found in cluster A1 are mostly linked to problems related to the injection of the vaccine (e.g. arm pain, arm inflammation, skin reaction) or flu syndrome as a result. Cluster D2 presents an interesting contrast as it remains empty until August 2017. Subsequently, it contains a few but widely used drugs that are frequently reported, such as paracetamol, amoxicillin, and some popular antidepressants. The evolution of cluster D2's proportion, as shown in Figure 5c, aligns with the change points identified by the algorithm. This cluster begins to populate after the Lévothyrox ® health crisis, with a peak in the early stages of the Covid-19 crisis, before the introduction of the vaccines. From Figure 5b, it can be observed that the intensity of interactions with clusters of undesirable effects does not differ significantly for cluster D2. Cluster D1, on the other hand, exhibits remarkably low interaction intensities and is densely populated by all other drugs. Initially, from the beginning of the analysis until 2017, it contains all drugs. However, after 2017, it includes only drugs with a low frequency of reports. From Figure 5c, we see that the evolution of Cluster D1 over time aligns with the change points identified by the algorithm. Upon examining Figures 5b and5d, a distinct behavior is observed in the clusters of adverse effects. Initially, until June 2017, cluster A1 contains all adverse effects. However, it gradually empties over time. Specifically, during the Mirena ® crisis, only the adverse effects not associated with that particular health scandal remain in cluster A1. Subsequently, a significant change in the cluster membership occurs following the change point identified in October 2017. From this point onward, the number of ADRs in cluster A1 decreases significantly, and they are specifically related to Lévothyrox ® (e.g. hair loss, cramps, insomnia, etc.). After the Lévothyrox ® crisis, cluster A1 becomes empty until the subsequent change point detected in January 2021. From this moment until the peak of Covid-19 vaccine reports in February 2022, cluster A1 includes the main adverse effects reported for Covid-19 vaccines (e.g. pain at the vaccination site, skin rash, pericarditis, etc.). Also, from Figure 5d we clearly notice as the Lévothyrox ® crisis marked a cornerstone in the history of pharmacovigilance, probably because from this moment on people realized its importance and began to declare side effects of drugs and vaccines much more frequently. Lastly, Figure 6 provides the estimated evolution of the sparsity parameter over time. The y-axis scale is set from 0.99 to 1 in order to visualize the changes over time. Notably, no change points were detected in the estimated parameter π due to insufficient variability in the values. Initially, in 2015, the sparsity is recorded at 99.83%. As we approach the peak in 2017, the number of declarations increases, leading to a decrease in sparsity, reaching a local minimum of 99.38%. Subsequently, as we approach the peak related to Covid-19 vaccines, the sparsity level reaches its global minimum at 99.25% in March 2021. As a summary, Stream Zip-dLBM method successfully identified meaningful clusters within the extensive initial data matrix of ADR reports. This provides a proof of concept of the possible use of this algorithm to detect public safety events from streams of ADR data.

Conclusions

This paper is born out of the need to analyze and summarize observations and features of a dynamic matrix in an online setting for an application to pharmacovigilance. We have proposed an online dynamic co-clustering technique that enables simultaneous clustering of rows and columns along the time dimensions. As observations and features can change their cluster memberships over time, detecting structural changes in cluster interactions throughout the time period becomes crucial. We have introduced a generative zero-inflated dynamic latent block model as online extension of Zip-dLBM. The time modeling approach relies on three systems of ordinary differential equations. Inference is conducted using a Variational EM algorithm, combined with stochastic optimization of LSTM network parameters for the dynamic systems. Also, we added an online change points detection method to the process such that Stream Zip-dLBM is able to detect abrupt changes and create alerts in real time. The performance of our approach is evaluated through applications to some simulated data scenarios. Then, Stream Zip-dLBM was fit to a large-scale dataset supplied by the Regional Center of Pharmacovigilance of Nice (France). In this context, the model provided meaningful online segmentation of drugs and adverse drug reaction. Its potential use by medical authorities for identifying meaningful pharmacovigilance patterns or emerging public health concerns looks very promising. In fact, we are actively developing a web platform based on the Stream Zip-dLBM model for the Regional Center of Pharmacovigilance in Nice, France.

Once implemented, it will regularly run on a center machine, automatically fitting the model to incoming data. The goal is to promptly identify structural changes and send email notifications with concise reports. The software effectiveness will be assessed over 6 months, with plans to expand its use nationally. The online inference algorithm, combined with change point detection, allows Stream Zip-dLBM to operate in real-time, continuously analyzing the flow of ADR declarations and triggering alerts as soon as a change point is detected. This provides an opportunity for further investigation and intervention by medical authorities.

Figure 1 :

 1 Figure 1: Simulated time instants for change points of α, β and π and simulated values of Λ.

Figure 2 :

 2 Figure 2: Evolution of the true (left) and estimated (right) proportions of the parameters α, β and π, respectively.

Figure 3 :

 3 Figure 3: Evolution of the true (left) and estimated (right) proportions of the parameters α, β and π, respectively.

Figure 4 :

 4 Figure 4: Number of declarations received by the pharmacovigilance center from 2015 to 2022, sorted by month.

  Evolution of ADR clusters proportions.

Figure 5 :

 5 Figure 5: Histogram of declarations over time, with the change points detected for the drugs and ADRs (top left) and estimated Poisson intensities, each color represents a different drug (ADR) cluster (top right); evolution of the estimates of α (bottom left); evolution of the estimates of β (bottom right).

  Figure 6: Evolution of the estimates π.


Hedi Harzallah

Frédéric Jurie

Cordelia Schmid

Combining efficient object localization and image classification

In this paper we present a combined approach for object localization and classification. Our contribution is twofold. (a) A contextual combination of localization and classification which shows that classification can improve detection and vice versa. (b) An efficient two stage sliding window object localization method that combines the efficiency of a linear classifier with the robustness of a sophisticated non-linear one. Experimental results evaluate the parameters of our two stage sliding window approach and show that our combined object localization and classification methods outperform the state-of-the-art on the PASCAL VOC 2007 and 2008 datasets.

Introduction

Over the past years, there has been increasing interest in object category recognition. Two major tasks are image classification and object localization. Image classification is defined as the task of assigning an image one or multiple labels corresponding to the presence of a category in the image. Many recent papers exist on this topic [START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF][START_REF] Perronnin | Universal and adapted vocabularies for generic visual categorization[END_REF][START_REF] Van Gemert | Kernel codebooks for scene categorization[END_REF][START_REF] Zhang | Local features and kernels for classification of texture and object categories: a comprehensive study[END_REF] and several evaluation campaigns, such as TRECVID, Pascal VOC, and ImageCLEF, demonstrate the good performance of these algorithms. Object localization detects instances of a given category in the image, in many cases up to a bounding box. Recent techniques combine efficient image descriptors with machine learning techniques [START_REF] Chum | An exemplar model for learning object classes[END_REF][START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF][START_REF] Ferrari | Groups of adjacent contour segments for object detection[END_REF][START_REF] Heitz | Learning spatial context: Using stuff to find things[END_REF][START_REF] Laptev | Improvements of object detection using boosted histograms[END_REF][START_REF] Viola | Robust real-time face detection[END_REF]. The task remains challenging due to intra-class variations, viewpoint changes and deformations of the objects.

Even if the two tasks are different, they are obviously related. If one has a good object detector, it becomes easy to predict image labels when some objects are detected with high scores (Figure 1-a) . Inversely, knowing the class of an image can help to detect hardly visible objects (Figure 1-b). This interdependence has been studied in context-based approaches, where context can model properties of the entire image [START_REF] Hoiem | Putting objects in perspective[END_REF][START_REF] Torralba | Contextual priming for object detection[END_REF], the relationship between objects [START_REF] Carbonetto | A statistical model for general contextual object recognition[END_REF][START_REF] Kumar | A hierarchical field framework for unified context-based classification[END_REF] or the background surrounding the candi- date detections [START_REF] Crandall | Composite models of objects and scenes for category recognition[END_REF]. Convincing results have been obtained by approaches where the image is segmented into labeled regions and the labels are used for the localization of surrounding objects [START_REF] Heitz | Learning spatial context: Using stuff to find things[END_REF][START_REF] Wolf | A critical view of context[END_REF]. A recent paper [START_REF] Divvala | An empirical study of context in object detection[END_REF] presents an empirical evaluation of the role of context in object detection, evaluating several sources of different context and ways to utilize it.

Because of the interdependence of the two tasks, the combination of object detection and image classification became a topic of interest in the recent literature. Li and Fei-Fei [START_REF] Li | What, where and who? classifying events by scene and object recognition[END_REF] propose a graphical model of events in images where event is a latent factor conditioning the generation of objects and scene categories. Shotton et al. [START_REF] Shotton | Semantic texton forests for image categorization and segmentation[END_REF] propose a related idea in the context of image segmentation: the likely categories are emphasized by multiplying the local segmentation and global image classification distributions. This is a principle we find again in [START_REF] Heitz | Learning spatial context: Using stuff to find things[END_REF], in another context, for improving object detections using labels of surrounding regions. We build on the idea that classification and detection can be considered as independent knowing a latent property of the observed scene, leading to a simple but powerful combination scheme. The main contribution of this paper is to show that such a combination can improve the results of state-of-the-art algorithms, both for classification and detection.

This, obviously, requires state-of-the-art approaches for classification and localization. For image classification we rely on [START_REF] Marszałek | Learning object representations for visual object class recognition[END_REF], one of the state-of-the-art approaches of the PASCAL VOC 2007 and 2008 challenges. For object localization we build on and improve existing sliding window approaches [START_REF] Bosch | Representing shape with a spatial pyramid kernel[END_REF][START_REF] Chum | An exemplar model for learning object classes[END_REF][START_REF] Ferrari | Groups of adjacent contour segments for object detection[END_REF][START_REF] Rowley | Human face detection in visual scenes[END_REF]. Firstly, we implement an efficient two stage approach which uses a linear support vector machine (SVM) classifier for pre-selection and a non-linear SVM for scoring. This allows an excellent trade-off between speed and accuracy. We believe that our two stage cascade is simpler than recent cascades [START_REF] Bourdev | Robust object detection via soft cascade[END_REF][START_REF] Brubaker | On the design of cascades of boosted ensembles for face detection[END_REF][START_REF] Dundar | Joint optimization of cascaded classifiers for computer aided detection[END_REF] while giving remarkably good results. Secondly, we evaluate extensively different ways of describing the image and propose a simple and efficient image representation. Our window descriptor builds on recent work [START_REF] Bosch | Representing shape with a spatial pyramid kernel[END_REF][START_REF] Chum | An exemplar model for learning object classes[END_REF][START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF][START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF] and combines the key ideas of these approaches into a simple but efficient descriptor. A comparison with the state-of-the-art shows that our detector gives better results for most of the object categories. This detector represents the second contribution of the paper.

The paper is organized as follow. Section 2 presents the datasets used for our experiments. A description of our efficient object detector and its experimental evaluation is given in section 3. Section 4 then describes the model we propose for combining classification and localization and evaluates its performance. Finally, in section 5 we present a comparison with the state-of-the-art methods and conclude.

Dataset and evaluation criteria

The PASCAL visual object class datasets are today probably the most widely used reference datasets for category recognition. Objects are present in realistic conditions under scale, viewpoint and illumination changes as well as with significant amounts of background clutter, see figure 4 for an illustration. The PASCAL VOC 2007 & 2008 datasets [START_REF] Everingham | The PASCAL Visual Object Classes Challenge Results[END_REF] used in this paper contain twenty object classes: person, animals (bird, cat, cow, dog, horse, sheep), vehicles (aeroplane, bicycle, boat, but, car, motorbike, train), and indoor objects (bottle, chair, dining table, potted plant, sofa and TV/monitor).

The main PASCAL challenge tasks are image classification and object localization. Training is in both cases supervised. For object localization the annotations include the bounding box, the object pose (left, right, front, back, other) and a flag indicating whether the object is truncated (only part of the object is visible). Training, validation and test sets are available for PASCAL VOC 2007, but not for 2008 for which the test annotations have not yet been made available. 1 We, therefore, run our evaluations in sections 3 and 4 on the 2007 dataset. We, then, compare in section 5 to the state-of-the-art on the VOC 2007 and 2008 datasets.

Our performance metrics, both for classification and de-tection, follow the PASCAL VOC ones. The average precision (AP) is computed from the precision/recall curve and is an approximation of the area under this curve. The mean AP (mAP) measures the mean of the APs over all categories.

Efficient object localization

Our object localization approach builds on the now standard sliding window approach [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF][START_REF] Ferrari | Groups of adjacent contour segments for object detection[END_REF][START_REF] Laptev | Improvements of object detection using boosted histograms[END_REF][START_REF] Viola | Robust real-time face detection[END_REF]. Such an approach evaluates a score function for all positions and scales in an image and detects local maxima of this function. Its performance depends on: (a) an efficient search strategy; (b) a robust image representation; (c) an appropriate score function for comparing candidate regions with object models; (d) a multi-view representation and (e) a reliable nonmaxima suppression.

Our approach has two contributions: a robust window representation and an efficient search strategy. For all the remaining components we use the standard techniques. Our two contributions are described and evaluated below.

Image representation

Our approach uses two complementary descriptors, shape and appearance descriptors described in the following. The combination of the two descriptors is presented and evaluated in the next section.

Shape descriptor (HOG). The shape descriptor is a histogram of oriented gradients (HOGs) [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF]. We apply the fast HOG implementation of [START_REF] Zhu | Fast human detection using a cascade of histograms of oriented gradients[END_REF]: after quantizing the gradient orientation at each pixel, we compute and store an integral image for each discrete orientation. These integral images are used to efficiently compute the HOG for any rectangular image region, i.e., in 4 × number of orientations basic operations.

The division of the description window into subwindows (or tiles) is referred to as geometry. The geometry is determined by three main parameters: [START_REF] Bosch | Representing shape with a spatial pyramid kernel[END_REF] The number of tiles that determine the resolution of the tiling. (2) The organization of the tiles which can be adapted or regular. Adapted tiles are as square as possible and are obtained as follows: given the average aspect ratio of the object category and the number of desired tiles T , we seek to have tiles that are as square as possible with round( T W H ) tiles along the width, and round( T H W ) along the height. Regular tiles are taken on a grid with the same number of tiles along the height and the width. [START_REF] Brubaker | On the design of cascades of boosted ensembles for face detection[END_REF] The overlap between tiles: when overlapping, a tile shares 50% of its surface with each of its four neighbors .

We build different configurations with the number of tiles ranging from 40 to 350, with overlapping or not overlapping tiles, and with adapted or regular grids. We evaluate these configurations with the filtering classifier (linear kernel). To compare the results obtained with the different configurations, we perform the Friedman statistical test. This test, based on the ranks on the different classes, separates the configurations into significantly different sets (here with probability of 95%).

We obtained 12 groups of equivalent configurations and observed that the difference between the best and the worst one results in a significant difference in mAP (7%). Three main observations are: (1) Tilings should contain at least 150 tiles, but having more has no impact. (2) Tiles should be overlapping; we observed a difference up to 3% just by adding the overlap between tiles. (3) Tiles should be as square as possible, but the difference is less significant.

We also evaluated the influence of the number of discrete HOG orientations. We used several configurations where the number of bins varied from 8 to 32 with signed or unsigned orientations. Using signed orientations appeared to be better (+1.3%), 16 bins outperformed 8 bins by 3% and 32 bins lead to the same performance as 16.

Appearance descriptor (BOF). The appearance descriptor builds on the spatial pyramid over quantized SIFT descriptors [START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF]. We first extract multi-scale patches from the image and describe them using the SIFT descriptor [START_REF] Lowe | Distinctive image features from scaleinvariant keypoints[END_REF]. The descriptors then quantized into visual words using k-means and a histogram of visual words summarizes the content of the window. Instead of building one global histogram for the window, we compute one histogram per tile, using the spatial pyramid tiling. This technique consists in partitioning the image into increasingly finer subregions and computing histograms of local features inside each subregion.

We vary the level of the pyramid and calculate the mAP on the PASCAL VOC 2007 dataset. Using only one level leads to poor results, the mAP is less than 3%. Using two levels rises the mAP to 7.6% and the best configuration uses 3 levels, resulting in a mAP of 15%. Adding more levels does not improve the results further. These experiments were obtained with a linear classifier and a visual vocabulary of 100 words.

We observe that the BOF descriptor performs slightly better than the HOG (14.6% mAP against 15%, see table 1). On the other hand, the BOF descriptor with 3 levels is more expensive computationally as well as memory-wise.

Normalization of the descriptors. Normalization of the descriptors makes them robust to contrast changes (HOG) or to scale changes (BOF and HOG). Previous work [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF][START_REF] Zhu | Fast human detection using a cascade of histograms of oriented gradients[END_REF] showed that local L1 and L2 norms give comparable results. In the following, we use the L2 norm, which can be used in different ways. We can either normalize each cell or perform a global normalization of the descriptor. It is also possible to not normalize the descriptor at all. Experimental results show that (a) not using any normalization reduces the results by 14% (HOG) and 4% (BOF) and (b) the per tile normalization is about 2% better than global normalization for both HOG and BOF.

Two stage object localization 3.2.1 Search strategy

As the number of windows per image is huge, a technique for reducing this number should be applied. While some authors suggest methods avoiding to scan exhaustively the image [START_REF] Chum | An exemplar model for learning object classes[END_REF][START_REF] Lampert | Beyond sliding windows: Object localization by efficient subwindow search[END_REF], the most popular technique is the cascade, introduced by Viola and Jones [START_REF] Viola | Robust real-time face detection[END_REF]. It decomposes a strong classifier into several classifiers arranged in a cascade, each of which decides if the window contains the object or not. This type of approach has received a lot of attention during the past five years [START_REF] Bourdev | Robust object detection via soft cascade[END_REF][START_REF] Brubaker | On the design of cascades of boosted ensembles for face detection[END_REF][START_REF] Dundar | Joint optimization of cascaded classifiers for computer aided detection[END_REF][START_REF] Zhu | Fast human detection using a cascade of histograms of oriented gradients[END_REF]. Cascades have, however, several limitations. Training a cascade is slow, taking on the order of weeks; determining the target false positive rate and detection rate at each stage in the cascade is often empirical; and finally, a cascade always reduces the overall performance.

We propose to use a very simple but efficient two stage cascade. We first apply a linear SVM for each window of the image. This is fast due to the simplicity of the classifier and the use of fast descriptors implemented with integral images. We then apply the final score function (a non-linear SVM with a χ 2 kernel) only on candidate regions, i.e., those that obtained good scores during the first stage. Our experimental results confirm that this approach has an excellent trade-off between speed and accuracy.

Linear classifier

The filtering classifier is a linear SVM classifier that is used to rapidly scan the image and reject windows unlikely to contain objects. Its performance depends on the data used to train it. We artificially increase the positive training set by following a procedure proposed by Laptev [START_REF] Laptev | Improvements of object detection using boosted histograms[END_REF]. The negatives examples are obtained by an iterative procedure. The initial training set consists of randomly chosen background windows and objects from other classes. The resulting classifier is used to scan images and select the top false positives or hard examples. These hard examples are added to the negative set and a new classifier is learned. This procedure is repeated several times to obtain the final classifier.

The performance of the filtering classifier is reported in Table 1. The mAP is 14.6% for HOG features and increases by 3% when combining HOG and BOF features. The combination is obtained by concatenating the two feature vectors.

However, the main interest of a filtering classifier is its ability to filter out the majority of the windows, the scoring classifier being applied only on the remaining small fraction of these windows. It is, therefore, more important to measure the capability of the linear classifier to select windows From these experiments, we can draw two conclusions. First, we observe that the different possible descriptors (HOG, BOF or the combination of the two) give similar results in terms of the highest possible recall, see Figure 2. Therefore, we use in the filtering stage the least expensive descriptor, i.e., HOG. Second, the mAP does not progress significantly for value above N = 100.

Improvement due to non-linear classifier

The scoring classifier is based on a non-linear SVM with a χ 2 kernel: K(x, v) = exp(-γ ch .d χ 2 (x ch , v ch )) where γ ch is a channel dependant parameter, x ch and v ch are the components of vectors x and v corresponding to the channel ch, here shape (HOG) or appearance (BOF), and d χ 2 is the χ 2 distance defined by

d χ 2 (x, v) = N i=1 (x i -v i ) 2 |x i + v i | . (1)
The γ value is defined by the heuristic of [START_REF] Zhang | Local features and kernels for classification of texture and object categories: a comprehensive study[END_REF] as follows:

γ = N b 2 N b i=1 N b j=1 d χ 2 (x i , x j ) , (2)
where N b the number of training examples and x i the i th training example.

To train the scoring classifier, we use the same positive training examples as for the filtering classifier. For the negative examples, we do not find additional hard examples due to the computational cost, but use the negative examples of the filtering classifier (background windows and hard examples) to which we add a large number (here 70K) of background windows.

Table 1 shows the performance increase due to the nonlinear scoring stage. The table reports the results obtained using a χ 2 kernel and three different descriptors combinations when filtering with a linear SVM + HOG and keeping 200 windows per image. The best mAP (26.3%) is obtained for the combination of HOG and BOF. Using only one of these representations reduces the performance by more than 3%. The improvement due to the non-linear classifier is very significant (8.7%).

Note that there is also an increase of performance when combining HOG and BOF for a linear SVM classifier. The combination of the two features allows to improve detection score, but does not help to improve the filtering efficiency of the linear SVM, see figures 2 and 3.

Discussion

Our final detector, i.e., χ 2 , HOG + BOF, gives excellent results on the PASCAL VOC 2007 database (see Table 1, bottom line). The proposed detector is efficient due to the two stage sliding window algorithm and its results compare favorable to the state-of-the-art. Experimental results show the benefit of using complementary image descriptors. Figure 4 shows a few localization examples. The first row shows two correctly detected objects, while the second row shows two strong false positives. Interestingly, the detector confuses a bus with a car and includes context information in the case of chairs. The last row shows missed objects and illustrates the complexity of the task: the car is very small and the chair hardly visible.

Contextual combination of localization and classification

In the following we describe our approach for combining localization and classification and present an experimental validation.

Our approach

The idea that classification and detection can benefit from and contribute to each other's successes relies on the assumption that they use different information. This assumption can be verified by observing that, often, a single image is classified differently (i.e. with a significantly different probability) by the detector and the classifier. Figure 5 shows the probability density of true positives windows (all classes of the Pascal VOC 2007 dataset being merged) as a function of the probabilities given by the classifier (applied to the whole image) and the detector (applied to the window). Interestingly, we can see that our hypothesis is valid: for many true positives only one of the modalities has a high probability. Indeed, if an object is occluded or truncated, it will be hardly detectable by the detector while the classifier could still have enough information (context, object parts) to decide on the presence of the object. Inversely, if the object is small and appears in a non standard context, the detector will still be able to find it while it would be non-detectable for the classifier (see the figure) that for most false positives both probabilities are low.

In the following, we construct a model for combining lo- calization and classification results that takes into account this notion of detectability. Let us denote P (D i ) as the probability that the presence of an object can be detected by the classifier applied to the entire image. In the same way, P (D w ) is the probability that the presence of an object can be detected by the sliding window detector. We obtain an approximation of the conditional probability P (O|D i , S i ), where S i is the score of the classifier, by histogramming the scores of positive and negative training examples. Objects of all training images are assumed to be detectable. This gives us the probability of having the object in the image knowing it is detectable. In the same way, we compute an approximation of P (O|D w , S w ), the probability of having an object knowing the score S w of the sliding window detector and assuming it is detectable.

Formally, the probability of having an object in the image, given the classification score, is therefore P (O|Si) = P (Di)P (O|Si, Di) + P (Di)P (O|Si, Di) [START_REF] Brubaker | On the design of cascades of boosted ensembles for face detection[END_REF] and similarly, the probability of having an object in a window given the detection score is P (O|Sw) = P (Dw)P (O|Sw, Dw) + P (Dw)P (O|Sw, Dw)

where P (O|S i , D i ) (resp. P (O|S i , D w )) is the probability that the object is present when it is not detectable by our image classifier (resp. object detector). They are supposed here to be constant values.

The final probability is task dependent. For the localization task, we consider the score S w of the window obtained by the detector as well as the score S i of the image obtained by the classifier, and compute the probability of having an object in the window as: P (O|S w , S i ) ∝ P (O|S i ) × P (O|S w ).

(

) 5
For the classification task, we consider the score of the window having the best score, denoted S bw , as well as the score S i obtained by the image classifier. The probability of having an object in the image is then computed as:

P (O|S bw , S i ) ∝ P (O|S i ) × P (O|S bw ). (6)
For our experiments, conditional probabilities P (O|S i , D i ), P (O|S i , D w ) as well as the priors P (D i ) and P (D w ) are constant values obtained by cross-validation, i.e., we take the values that maximize the AP on a validation set.

Experiments

Classification experiments. We compare the classification performance obtained by our detection framework, by the INRIA Flat V2 approach and by the combination of both. To obtain classification scores based on detection we keep for each object class only the best scored window in the image. The INRIA Flat V2 approach [START_REF] Marszałek | Learning object representations for visual object class recognition[END_REF], for which we have obtained the results (classification scores) from the authors, is based on the work of [START_REF] Zhang | Local features and kernels for classification of texture and object categories: a comprehensive study[END_REF], i.e., it integrates a set of different image features with a χ 2 kernel. V2 signifies the version used in the PASCAL VOC 2008 challenge which adds additional channels to the 2007 version and improves the performance. Table 2 shows the results for the PASCAL VOC 2007 dataset. The first row is obtained by detection. In terms of mean average precision (mAP), it gives poor results. However, for some classes it gives very good results and even outperforms the INRIA Flat V2 (second row) approach, namely for the classes car and bottle.

When combined ("Our combination"), we observe a significant gain of 3.4% in term of mAP over the IN-RIA Flat V2 classifier. Furthermore, for some classes (e.g bottle, plant) we observe a gain of 10% in AP which represents a significant improvement. The row "Product" shows the results obtained with a simpler combination rule where we multiply the two probabilities. As expected, the results are not as good as with our combination method. We also evaluated other combination methods such as the MIN or MAX rules [START_REF] Kittler | On combining classifiers[END_REF], and an SVM classifier for a prediction based on two scores. All of them performed slightly worse than our approach.

Detection experiments. We compare the results obtained by our detector to combinations with the classification score given by the INRIA Flat V2 classifier. Table 3 presents the results. The first row gives the average precision obtained by our detector, the third one shows results for a combination performed by simply computing the product of the two probabilities, and the last one present the results of our method. Our method improves the results of the base detector by 2.6%. Best improvement are observed for the animal classes cow (7.6%) and sheep (6.1%) as well as for indoor classes bottle (4.5%) and chair (3.4%) which suggests that the context information offers most of the improvement.

The results submitted by the authors of [START_REF] Felzenszwalb | A discriminatively trained, multiscale, deformable part model[END_REF] to the PAS-CAL VOC 2008 used contextual information to reweight detection scores: the final score is computed from the initial score, the position of the window and the best detection scores in the image for all the other object categories. The results given by this approach, reported Table 3-row 2, show that the information due to the image classifier is more useful than the contextual information due to object detections of other categories.

Comp. with state-of-the-art & Discussion

As stated in the introduction, we believe that to clearly demonstrate the importance of contextual information or of any combination of methods we need to experimentally demonstrated that it produces results better than existing state-of-the-art methods. Starting from any baseline algorithm and improving its performance is indeed easy. Surpassing state-of-the art results is more difficult.

Classification. Table 4 compares our method to the five top methods of the PASCAL VOC 2008 challenge [START_REF] Everingham | The PASCAL Visual Object Classes Challenge Results[END_REF]. The mAP we obtain is 57.7%, which improves by 2.8% over the best competing approach. Furthermore, our approach obtains best results for 13 out of 20 categories. On the PASCAL VOC 2007 dataset we obtain 63.5%, see table 2. Compared to the results obtain in the 2007 challenge, we improve by 4.1% in terms of mAP over the best method and obtain best performance on 15 out of 20 classes.

Localization. We compare our localization performance to the best results of the PASCAL VOC 2008 challenge [START_REF] Everingham | The PASCAL Visual Object Classes Challenge Results[END_REF] in Table 5. Compared to these results, we achieve best performance for 11 out of 20 classes and obtain results comparable with the method [START_REF] Felzenszwalb | A discriminatively trained, multiscale, deformable part model[END_REF].

On the PASCAL 2007 VOC dataset, we obtain 28.9% mAP, see table 2. When compared to the challenge results in 2007, we obtain the best performance for 18 out of 20 classes and a gain of 11.8% in mAP with respect to the best method. This confirms the big improvement of localization methods over the past year. Discussion. This paper has shown that there exists a potential for combining the presence and location of objects, resulting in a significant improvement of classification and localization performance. We also introduced an efficient two stage sliding window detector. Future work will explore the combination of different types of information. We will, for example, investigate image classification methods that explicitly take into account the results of an object detector, i.e., for example by griding the image based on the detected region.

Figure 1 .

 1 Figure 1. Complementarity of image classification and object localization: (a) Cars are detected with strong scores, while the image obtained a poor classification score. (b) Cars are detected with low scores due to partial visibility, but the image got a very high classification score.

Figure 3 .

 3 Figure 3. mAP of the scoring classifier (χ 2 kernel + SVM with HOG & BOF features) versus number of windows per image selected by the filtering classifier (linear SVM and HOG features). Results are presented for PASCAL VOC 2007.

  Figure 1 for an illustration). We also observed (not shown on Car Chair (a) Good localizations (b) Strong false positive (c) Missed objects

Figure 4 .

 4 Figure 4. Localization examples for the classes car and chair with our detector (χ 2 , HOG + BOF) on PASCAL VOC 2007. (a) Examples of objects correctly detected. (b) Examples of false positives with high score. (c) Examples of objects that were missed.

Figure 5 .

 5 Figure 5. Probability density of true positive windows, as a function of the probabilities given by the image classifier and the object detector. Warm (resp. cold) colors mean high (resp. low) densities. Results are presented for the PASCAL VOC 2007 dataset.

Table 1 .

 1 plane bike bird boat bottle bus car cat chair cow table dog horse moto person plant sheep sofa train tv mAP Linear, HOG 10.0 27.8 04.7 00.6 11.4 31.7 33.9 02.6 10.1 14.9 09.7 01.8 28.1 22.6 12.2 09.9 10.0 04.3 19.3 26.1 14.6 Linear, BOF 16.9 21.2 04.9 04.8 07.3 25.2 28.4 06.9 09.8 10.3 06.7 06.9 30.5 26.6 13.1 09.4 12.5 12.1 17.0 28.5 15.0 Linear, HOG+BOF 22.4 30.5 03.3 01.8 11.2 26.4 36.7 06.0 11.1 14.3 10.9 07.6 33.8 27.2 14.7 09.8 15.1 14.7 22.4 32.2 17.6 χ 2 , HOG 18.4 39.5 09.8 02.0 18.2 42.2 47.5 02.5 13.6 22.1 10.5 10.7 43.5 34.6 14.5 11.7 12.7 14.2 31.8 37.2 21.9 Localization performance on PASCAL VOC 2007 for the different image representations with/without using the non-linear scoring classifier.

		χ 2 , BOF		29.8 33.3 11.1 04.2 09.5 39.7 42.3 14.4 12.7 20.4 13.3 15.5 40.5 37.6 16.8 11.4 19.8 18.8 34.4 35.6 23.1
	χ 2 , HOG+BOF		33.8 43.0 09.7 09.6 18.7 41.9 50.4 15.0 14.6 23.9 15.1 15.4 48.2 41.7 20.2 16.1 21.2 20.3 29.1 38.2 26.3
		1
		0.8
	Maximum recall	0.4 0.6
					HOG
		0.2			BOF
					HOG-BOF
		0
		0	50	100	150	200
			Number of windows per image

Figure 2. Maximum recall versus number of windows per image for filtering with a linear SVM with HOG and/or BOF features. Results are presented for PASCAL VOC 2007.

than to measure its mAP, even if the two are related. We measured this capability with two criteria:

[START_REF] Bosch | Representing shape with a spatial pyramid kernel[END_REF]

The highest recall the scoring classifier can reach using the top N windows selected by the linear classifier, for different values of N . Results are given Figure

2

. (2) The mAP obtained by the scoring classifier if only the top N windows are selected. These results are reported in Figure

3

for filtering with a linear SVM + HOG and scoring with χ 2 SVM with HOG & BOF features.

Table 2 .

 2 plane bike bird boat bottle bus car cat chair cow table dog horse moto person plant sheep sofa train tv mAP Localization 58.2 59.5 19.3 26.9 37.5 60.8 80.2 36.7 39.3 40.7 28.3 35.2 67.9 63.8 74.0 38.4 40.1 35.7 56.7 54.3 47.7 INRIA Flat V2 76.7 64.4 55.8 69.3 34.1 62.6 76.0 58.4 55.9 44.3 60.3 48.3 77.4 63.5 85.7 42.9 48.3 49.0 76.4 54.5 60.1 Product 75.0 69.0 51.7 67.7 48.8 68.7 83.3 54.6 57.6 53.7 56.6 46.4 78.7 69.4 84.8 51.8 54.4 55.5 73.0 62.8 63.2 Our combination 77.2 69.3 56.2 66.6 45.5 68.1 83.4 53.6 58.3 51.1 62.2 45.2 78.4 69.7 86.1 52.4 54.4 54.3 75.8 62.1 63.5 Image classification performance on the PASCAL VOC 2007 dataset obtained by localization and classification methods (1st and 2nd rows) and by combination (last 2 rows). plane bike bird boat bottle bus car cat chair cow table dog horse moto person plant sheep sofa train tv mAP Context of [11] 35.4 44.7 10.6 05.5 20.3 42.1 50.7 18.7 16.2 26.2 14.2 16.3 49.0 42.9 20.5 17.0 23.4 21.0 30.7 39.3 27.2 Product 33.9 40.7 10.8 11.9 22.7 36.8 47.1 18.7 17.4 31.9 17.0 16.5 48.0 38.1 20.9 18.4 26.6 23.6 28.6 37.2 27.3 Our combination 35.1 45.6 10.9 12.0 23.2 42.1 50.9 19.0 18.0 31.5 17.2 17.6 49.6 43.1 21.0 18.9 27.3 24.7 29.9 39.7 28.9

	Our detector	33.8 43.0 09.7 09.6 18.7 41.9 50.4 15.0 14.6 23.9 15.1 15.4 48.2 41.7 20.2 16.1 21.2 20.3 29.1 38.2 26.3

Table 3 .

 3 Object localization performance on the PASCAL VOC 2007 dataset obtained by our two stage detector alone (1st row) and combined with context (last 3 rows).

An evaluation of the results on the VOC

test set can be obtained from M. Everingham.

Acknowledgements This research has been funded by MBDA.