Salma Moujtahid
email: salma.moujtahid@liris.cnrs.fr

Stefan Duffner
email: stefan.duffner@liris.cnrs.fr

Atilla Baskurt
email: atilla.baskurt@liris.cnrs.fr

Classifying Global Scene Context for On-line Multiple Tracker Selection

In this paper, we present a novel framework for combining several independent online trackers using visual scene context. The aim of our method is to decide automatically at each point in time which specific tracking algorithm works best under the given scene or acquisition conditions. To this end, we define a set of generic global context features computed on each frame of a set of training videos. At the same time, we record the performance of each individual tracker on these videos in terms of object bounding box overlap with the ground truth. Then a classifier is trained to estimate which tracker gives the best result given the global scene context in a particular frame. We experimentally show that such a classifier can predict the best tracker with a precision of over 80% in unknown videos with unknown environments. The proposed tracking method further filters the classifier responses temporarily using a Hidden Markov Model in order to avoid rapid oscillations between different trackers. Finally, we evaluated the overall tracking system and showed that this scene context-based tracker selection considerably improves the overall robustness and compares favourably with the state-of-the-art.

Introduction

We consider the problem of on-line visual object tracking in unconstrained environments, which raises many challenges. First, the arbitrary nature of the object makes it difficult to model its appearance only from the first video frame and implies the use of on-line learning as opposed to off-line trained classifiers for specific types of objects (e.g. faces or pedestrians). Additionally, the potential changes in the scene appearance, camera motion or other acquisition conditions during a video make background subtraction algorithms unsuitable for the tracking task. In order to successfully track an object in these unconstrained conditions, a tracking algorithm needs to adapt its model based on discriminant, descriptive features while minimising model drift. In our work, we leverage the fact that different tracking algorithms are specialised on different types of scenes and acquisition conditions. We show that, by quantifying these scene context parameters and training a discriminative classifier that decides at each moment which specific algorithm will perform best under the given conditions, we are able to successfully combine several independent trackers in a way that considerably improves the overall robustness in highly varying environments. c 2015. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms.

Related Work

In the literature, many ways of combining, fusing or selecting visual features have been presented. An example of low-level fusion of features is the Bayesian framework introduced by Yilmaz et al. [START_REF] Yilmaz | Object contour tracking using level sets[END_REF] fusing probabilistic density functions based on texture and colour features for object contour tracking. Collins et al. [START_REF] Collins | On-line selection of discriminative tracking features[END_REF] used likelihood maps to rank features in order to select the most discriminant one. Other existing works (e.g. [START_REF] Perez | Data fusion for visual tracking with particles[END_REF][START_REF] Stalder | Beyond semi-supervised tracking: Tracking should be as simple as detection, but not simpler than recognition[END_REF][START_REF] Triesch | Democratic integration: Self-organized integration of adaptive cues[END_REF][START_REF] Yin | Likelihood map fusion for visual object tracking[END_REF]) fuse different modalities, like motion or shape, in order to improve the overall foreground-background discrimination. Fusion is also possible at a higher level, where several trackers are run in parallel in order to select or combine their respective results. A probabilistic combination has been proposed by Leichter et al. [START_REF] Leichter | A general framework for combining visual trackers -"black boxes" approach[END_REF] where parallel trackers use different features and output a probabilistic density function of the tracked state. Another probabilistic formulation has been presented by Kwon et al. [START_REF] Kwon | Visual tracking decomposition[END_REF][START_REF] Kwon | Tracking by sampling trackers[END_REF], where different motion and appearance models are combined by sampling from different trackers. More recently, the method Bailer et al. [START_REF] Bailer | A superior tracking approach: Building a strong tracker through fusion[END_REF] fuses the bounding boxes of multiple state-of-the-art trackers by an off-line training step and trajectory optimisation. In terms of model or feature fusion, our previous work Moujtahid et al. [START_REF] Moujtahid | Coherent selection of independent trackers for real-time object tracking[END_REF] concentrated on using confidence values of several individual trackers to select the most suitable one at a given instant. Each tracker is independent and relies on a different visual feature, like colour or texture. The similar approach from Stenger et al. [START_REF] Stenger | Learning to track with multiple observers[END_REF] also used confidences but has been applied to the particular case of face tracking and involves off-line trained classifiers, whereas in [START_REF] Moujtahid | Coherent selection of independent trackers for real-time object tracking[END_REF] we used an additional spatial-temporal coherence criteria to enforce the continuity of tracking.

In contrast to these existing works, in our proposed framework, the selection of the most discriminant and most suitable tracker is based on the visual scene context in the video. Context has been used previously for object tracking in different forms and has shown to improve the overall tracking performance. Some works propose to detect image regions or interest points that move similarly to the tracked object [START_REF] Grabner | Tracking the invisible : Learning where the object might be[END_REF][START_REF] Wen | Robust online learned spatio-temporal context model for visual tracking[END_REF][START_REF] Yang | Context-aware visual tracking[END_REF] in order to assist the tracking, so-called supporters, contributors or helper objects. Other methods seek image regions that have similar appearance, so-called distractors, in order to avoid confusion [START_REF] Tb Dinh | Context tracker: Exploring supporters and distracters in unconstrained environments[END_REF][START_REF] Hong | Dual-force metric learning for robust distracter-resistant tracker[END_REF]. However, these approaches are computationally expensive, due to the more complex data association and modelling of spatial and temporal relationships between the different tracked objects or interest points.

In our approach, we are not trying to detect and track supporting or distracting image regions but we are classifying the general scene context and conditions in order to select the most appropriate visual cue or tracker for a given situation. To this end, we compute global image descriptors based on colour, intensity and motion at each video frame. In the past, other global image descriptors (sometimes called gist features) have been proposed (e.g. [START_REF] Oliva | Modeling the shape of the scene: A holistic representation of the spatial envelope[END_REF][START_REF] Siagian | Rapid biologically-inspired scene classification using features shared with visual attention[END_REF][START_REF] Torralba | Context-based vision system for place and object recognition[END_REF]) mostly for fixed images to classify scenes into different semantic categories, such as open, closed environments, indoor, outdoor etc. To our knowledge, no other work exists that extracts and classifies global scene context features for visual object tracking.

Motivation

Different tracking algorithms have different strengths and weaknesses. Some cope well with different lighting conditions, some are particularly robust to object deformations or occlusions etc. but will fail in other conditions. Due to the compromise between invariance and discriminative power, despite recent progress in on-line tracking algorithms, it is hard to design models that perform well in very different environments and contexts. Although the fusion of different visual cues generally improves the performance in that regard, it remains difficult to decide on the importance or on the weight each modality should get when the overall scene context changes, especially within a given video. Moreover, the on-line updating of the underlying models is not straightforward. A given modality or tracker might be impaired by the update under conditions that it has not been designed for.

To this end, we propose a framework that combines several independent and complementary trackers, each specialised on different image and scene conditions. The decision on which tracker to select is proposed by an off-line trained classifier which, in turn, is based on general scene context features that are independent from the individual trackers. To summarise, our contributions are the following:

• a set of general scene context features that describe the global conditions (such as lighting, camera motion) that are relevant to track an object in a video,

• a framework that combines independent trackers by using a classifier that estimates at each frame the most suitable tracker for the given context, and by filtering the classifier responses using a temporal Hidden Markov Model (HMM),

• and a thorough evaluation of the different components of the proposed approach and comparison with state-of-the-art methods.

Overall approach

The general procedure of the proposed tracking framework is illustrated in Fig. 1. On a given video, N independent trackers T n , (n ∈ 1..N) run in parallel and, at every frame t, produce each an estimate of the object's state. This is usually a bounding box B n t with an associated confidence value (or score) c t,n . The objective is to select at each frame the best tracker, i.e. the one that outputs the bounding box that fits best the object to track.

At the same time, M scene context features f t are extracted and concatenated with additional measures like the trackers' confidences c t and the identifier of the last selected tracker s t-1 to form a large feature vector i t . An N-class classifier, that has been trained off-line on annotated data, is then applied on these features to estimate the best tracker for the given scene context. The classifier responds with y t , a probability for each class which is subsequently filtered by a HMM to ensure some temporal continuity of the tracker selection and reject outliers. Finally, a Kalman Filter is applied as a post-processing step to temporally smooth the resulting object bounding box B s t from the selected trackers T s . The result of the Kalman filter represents the final output of our tracking algorithm, and is further used to update the models of the individual trackers T n . Apart from this last update step, all the trackers are completely independent and do not cooperate or interact with each other. It is also important to mention that this approach is very generic, and in theory any on-line tracking algorithm can be integrated in this framework.

Visual scene context

In most existing computer vision algorithms, for example image classification or object recognition, some common visual features are used to concisely describe the content of an image or a region, such as SIFT, HOG, Haar-like features, LBP, or more roughly using Gist descriptors. However, to our knowledge, there are no established descriptors that capture global scene information regarding the overall environment or acquisition conditions, like lighting, camera motion, background uniformity etc. To quantify these phenomena is of particular interest for video analysis and object tracking algorithms. In the following, we will propose a set of such descriptors that we call "scene context features".

Scene context feature extraction

The proposed scene context features are designed to help predicting the best tracker in a given environment. Most of them correspond to first and second order statistics of a given imagerelated variable (e.g. intensity, hue, saturation, motion vectors), and are straightforward to compute. Let's define Ω as a region of the input image, and f Ω t,k as our feature computed on Ω. To simplify the notation, we omit the frame index t in this section. We propose to use the following set of features, grouped into three categories and defined in Equations 1-4:

Intensity features

-Average brightness ( f Ω 1 ): the mean grey-scale pixel value over region Ω (see Eq. 1). -Average contrast ( f Ω 2 ): the mean squared value of the difference of each grey-scale pixel and the average brightness over Ω (see Eq. 4).

Chromatic features

-Average saturation ( f Ω
3 ): the mean pixel value of the saturation channel in HSV colour space over the region Ω (Eq. 1).

-Saturation variance ( f Ω 4 ): the variance of saturation over Ω (Eq. 2). -Dominant hue ( f Ω 5 ): the dominant colour of the region Ω extracted from a histogram of quantised hue pixel values in HSV colour space (Eq. 3).

-Hue variance ( f Ω 6 ): the variance of the pixel values in the hue channel (Eq. 2).

Motion features

-Average motion ( f Ω 7 ): the mean of the norm of optical flow vectors densely computed over the region Ω (Eq. 1).

motion variance ( f Ω 8 ): the variance of the norm of dense optical flow vectors over Ω (Eq. 2).

Let p i denote the pixel value of a given image channel (e.g. H,S,V) and Ω the number of pixels in the region Ω. The above mentioned features are then defined as follows: AVERAGE:

f Ω k = ∑ i∈Ω p i Ω , for k = 1, 3, 7 (1)
VARIANCE:

f

Ω k = ∑ i∈Ω (p i ) 2 Ω - ∑ i∈Ω p i Ω 2 , for k = 4, 6, 8 (2)
DOMINANT CUE:

f Ω k = argmax i∈Ω (p i ), for k = 5 (3)
CONTRAST:

f Ω k = 1 Ω . ∑ i∈Ω p i - ∑ i∈Ω p i Ω 2 , for k = 2. ( 4
)
Each of these features k is computed on three different image regions Ω. We define a global value as the feature computed on the whole image: f G k . The local value is the feature computed on the Region Of Interest (ROI), i.e. the region defined by the bounding box of the tracked object: f L k . And a differential value as the difference between the feature computed on the foreground region (i.e. the ROI) and the background region (i.e. the image not including the ROI): f D k . Not every combination of feature and region is used as some them don't show semantic meaning. The concatenation of these features gives us:

f G = f G 1 , . . . , f G 3 , f G 6 , . . . , f G 8 f L = f L 1 , . . . , f L 8 f D = f D 1 , . . . , f D 7 .
Finally, we obtain M = 21 scene context features

f t = f G t , f L t , f D t for frame t.

Scene Context Classifier

To learn the different patterns that show high correlation between the information extracted from the scene context and the performance of a tracker in a particular set of conditions, we employ a multi-class classifier. We chose a fully connected Multi-Layer Perceptron (MLP) with one hidden layer of N h neurons and N output neurons. Any other algorithm could be used. In fact, a multi-class SVM showed equivalent performance, in our experiments. However, it was relatively sensitive to the choice of hyper-parameters (e.g. the type of kernel). The input to the classifier at frame t consists of the scene features f t and two additional components: First, the confidence values of the N trackers c t = (c t,1 ..c t,N ); they provide the classifier with a measure of reliability of each tracker's result. And second, the identifier s t-1 of the tracker that has been selected in the previous frame. We will experimentally show that this recursion highly contributes to learning the correlation between the scene context features and the selected tracker in a given frame.

Furthermore, in order to give the scene context classifier information on the evolution of the context over time, we additionally provide it with the features from the two previous frames t -1 and t -2 forming the vectors:

F t = {f t , f t-1 , f t-2 } C t = {c t , c t-1 , c t-2 } S t-1 = {s t-1 , s t-2 , s t-3 }
Incorporating the temporal aspect has proven to be very effective, as shown in our experimental results. The final feature vector given as input to the classifier is the following:

i t = {F t , C t , S t-1 }.
The classifier is trained off-line with a set of training samples i j , o * j N t j=1 containing the input features i j and labels o * j computed from a separate set of videos with annotated object bounding boxes. To construct the scene context features i j = F j , C j , S j-1 , we run the N trackers on each video, and at each frame, extract the scene context features F j as well as the trackers' confidences C j . For the vector S j-1 , i.e. the previously selected tracker, we used the identifiers of the best tracker in the respective preceding frames according to the ground truth. The desired classifier output class o * t is the most accurate tracker, i.e. the one with the highest overlap with the ground truth. We optimise the neural network parameters with standard stochastic gradient descent by minimising the mean squared error between the networks response vector y t = {y t,1 ..y t,n } and the desired output vector y * t ∈ {-1, +1} N with +1 for the component corresponding to o * t and -1 otherwise. The network's final class prediction is simply o t = argmax n∈N y t,n . We perform early stopping using a separate validation set. At each validation step, we update the component s (the previously selected tracker) in the input vectors according to the maximum of the actual output of the classifier.

Tracking procedure

The proposed algorithm uses N independent on-line trackers that are initialised with the bounding box of the object in the first video frame. Then, as illustrated in Fig. 1, the trackers and the context feature extraction operate in parallel providing at each frame N confidence values C t and M scene context features F t respectively.

At each video frame, the scene context classifier estimates the best tracker y t to select for the given scene context. To avoid frequent and unnecessary switching between different trackers, we filter the classifier responses in time using a Hidden Markov Model (HMM).

The HMM is used to estimate the discrete hidden variable x t ∈ {1..N} corresponding to the best tracker selection, and it receives the observations y t being the output of the scene context classifier. Another observation variable d t = (d t,1 ..d t,N ) is added and defined as the normalised distances of each tracker's resulting bounding box B n t to the previous estimated object position. Using the HMM, we want to estimate the posterior probability distribution of x t , which we can compute recursively:

p(x t |y t , d t ) = p(y t |x t ) p(x t |x t-1 , d t )p(x t-1 |y t-1 , d t-1 )dx t-1 .
(

) 5
To simplify model parameter estimation, we assume that observations d t and the hidden variable x t-1 are independent. This gives us:

p(x t |y t , d t ) = p(y t |x t )p(x t |d t ) p(x t |x t-1 )p(x t-1 |y t-1 , d t-1 )dx t-1 (6)
The likelihood function p(y t |x t ) of the observed classifier responses and the probability p(x t |d t ) of selecting a tracker given the distances to the previous bounding box are modelled by histograms computed on a separate training set. The transition probability p(x t |x t-1 ) is set empirically to achieve a reasonable continuity of the HMM responses. Then, the best tracker selection according to the HMM is computed as the maximum a posteriori probability:

s t = argmax s p(x t = s |y t , d t ) . (7)
and the bounding box B t from the selecting tracker T s t is passed to a Kalman Filter to provide a smoother final trajectory.

Experiments

For our experiments, we used N = 3 On-line AdaBoost (OAB) trackers [START_REF] Grabner | Real-time tracking via on-line boosting[END_REF] with different visual cues for each: Haar-like features (HAAR), Histograms of Oriented Gradients (HOG) and Histograms of Colour (HOC). The evaluation of the performance of our framework consists of two parts. First, we measured the classification rate of the proposed scene context classifier when trained on the different groups of features described in Section 3.1. Secondly, we evaluated the overall tracking algorithm on a public benchmark analysing the contribution of the different components, i.e. scene context features, HMM, and Kalman filter.

Training dataset. The context-based classifier is trained on the Princeton Tracking Benchmark Dataset [START_REF] Song | Tracking revisited using RGBD camera: Unified benchmark and baselines[END_REF]. It contains 100 RGB-D videos with a diverse set of object types, backgrounds, and changes in illumination, appearance and motion.

Evaluation dataset. The evaluation of context-based classifier, as well as for the overall tracking framework was conducted on the publicly available Visual Object Tracking (VOT2013) benchmark [START_REF] Kristan | The Visual Object Tracking VOT2013 challenge results[END_REF]. The VOT2013 dataset contains 16 image sequences collected from well-known tracking evaluations, they cover most of the challenging situations in object tracking: scale variance, complex backgrounds, occlusions, object deformation etc.

Classifier evaluation

In order to understand the relevance of the different scene context features, we conducted a series of experiments related to the classifier. We first separated the scene features into the three group presented in Section 3.1: local, global and differential. Then we added several time steps F t , the confidences C t and the previous tracker identifiers S t-1 . For each feature set, the classifier is trained on the Princeton dataset and tested on the VOT2013 dataset. The results for the different combinations are shown in Table 1. The recognition rate represents the proportion of frames where the classifier has successfully predicted the best tracker.

The combination of local, global and differential scene features gives only a low recognition rate of 30.03%, however when introducing features over several time steps F t , we achieve a higher rate of 35.80%. Adding the confidence values C t and the previous tracker identifier S t-1 , the rate is considerably increased to 81.80%. Given the low recognition rates without S t-1 , one might think that the classifier's decision is mainly relying on this particular feature. However, when training solely on the features S t-1 , the classifier does not converge to a viable solution as S t-1 alone does not allow for a good generalisation. It tends to just respond the identifier of the previously selected tracker. In fact, it is the association of both the context scene features and tracker features that enables the classifier to extract and learn the correlations between the scene information and the trackers' performance.

Tracking evaluation

We further evaluated the performance of the tracking framework and its different components following the protocol of the VOT2013 benchmark [START_REF] Kristan | The Visual Object Tracking VOT2013 challenge results[END_REF] bounding box in the videos' first frame and re-initialised whenever the target is lost. The benchmark provides two evaluation measures: is the average overlap with the ground truth, and "failures" represents the robustness of the algorithm counting the number of times the tracking is lost. Here, we present the (sequence-based) average values for the whole dataset. See [START_REF] Kristan | The Visual Object Tracking VOT2013 challenge results[END_REF] for more details.

In Table 2, the results for our proposed tracking method and its different components are presented along with a baseline method called "Best Confidence" (BC) that selects the best tracker only by the maximum confidence value. A classification based on the global scene context features considerably reduces the number of failures compared to using only the confidence values in BC. We can also see that both the HMM and the Kalman Filter greatly improve the robustness. However, when decreasing the number of failures the accuracy decreases slightly as well. These two additional components of the algorithm are important for the continuity of the tracking and, at the same time, ensure that a different, more suitable tracker can be selected whenever drastic scene changes occur.

We further compare the proposed framework (i.e. including HMM and Kalman Filter) with other state-of-the-art tracking algorithms, as well as the three individual (OAB-based) trackers. Figure 2 shows the ranking and Accuracy-Robustness plots. The proposed algorithm increases the robustness of our individual trackers HAAR, HOG and HOC. In fact, our method ranks among the top trackers of the challenge in terms of robustness, outperforming for example EDFT [START_REF] Felsberg | Enhanced distribution field tracking using channel representations[END_REF], STC [START_REF] Moujtahid | Coherent selection of independent trackers for real-time object tracking[END_REF], Struck [START_REF] Hare | Struck: structured output tracking with kernels[END_REF] and FoT [START_REF] Matas | Robustifying the flock of trackers[END_REF] methods. On the other hand, the accuracy of our method is directly linked and dependent on the accuracy of the individual trackers. Note that we fixed the scale of the trackers in our experiments as the robustness of OAB generally decreased when adapting to scale. Using more accurate trackers would increase the general accuracy of our proposed method as well as the robustness. Nevertheless, we demonstrated that our framework is able to combine the strengths of each OAB tracker and enhance the overall robustness (Fig. 3). [START_REF] Moujtahid | Coherent selection of independent trackers for real-time object tracking[END_REF], Struck [START_REF] Hare | Struck: structured output tracking with kernels[END_REF], MIL [START_REF] Babenko | Robust object tracking with online multiple instance learning[END_REF], TLD [START_REF] Kalal | Tracking-learning-detection[END_REF], Meanshift [START_REF] Comaniciu | Kernel-based object tracking[END_REF], PLT [START_REF] Kristan | The Visual Object Tracking VOT2013 challenge results[END_REF], EDFT [START_REF] Felsberg | Enhanced distribution field tracking using channel representations[END_REF], FoT [START_REF] Matas | Robustifying the flock of trackers[END_REF], LGT++ [START_REF] Cehovin | Robust visual tracking using an adaptive coupled-layer visual model[END_REF].

Conclusion

In this work, we proposed a novel tracker selection framework based on scene context. We used a classifier to learn the patterns that relate the scene context information with the "suitability" of specific independent trackers under the conditions at a given video frame. We also introduced a HMM to eliminate outliers and enforce the continuity in our tracking. In our experiments with the VOT2013 benchmark, the proposed method ranks among the top state-of-the-art trackers, and we showed the effectiveness of generic scene context features in challenging tracking environments. Future work will concentrate on studying the effect of using diversified state of the art trackers, increasing the number of trackers, as well as using other types of scene context features.

Figure 1 :

 1 Figure 1: Overall framework of the proposed scene context-based tracking algorithm.

Figure 2 :

 2 Figure 2: VOT2013 Benchmark ranking (left) and Accuracy-Robustness (right) plots: Comparison of our proposed framework (Ours) with Online AdaBoost trackers HAAR, HOG, HOC ; baseline BC ; and state of the art trackers: STC[START_REF] Moujtahid | Coherent selection of independent trackers for real-time object tracking[END_REF], Struck[START_REF] Hare | Struck: structured output tracking with kernels[END_REF], MIL[START_REF] Babenko | Robust object tracking with online multiple instance learning[END_REF], TLD[START_REF] Kalal | Tracking-learning-detection[END_REF], Meanshift[START_REF] Comaniciu | Kernel-based object tracking[END_REF], PLT[START_REF] Kristan | The Visual Object Tracking VOT2013 challenge results[END_REF], EDFT[START_REF] Felsberg | Enhanced distribution field tracking using channel representations[END_REF], FoT[START_REF] Matas | Robustifying the flock of trackers[END_REF], LGT++[START_REF] Cehovin | Robust visual tracking using an adaptive coupled-layer visual model[END_REF].

Figure 3 :

 3 Figure 3: Illustration of our proposed framework's tracking results on the "David"(1 st row) and "Bicycle"(2 nd row) videos. Different scene context variations in lighting, texture or background are present throughout the videos. Our framework selects the most suitable tracker in each scenario (pink: HAAR, blue: HOG, green: HOC).

Table 1 :

 1 . It's a well-known benchmark among the tracking community, which help us compare our proposed framework to the state of the art tracking methods. The tracking algorithm is initialised with the ground truth object Classifier input i t Recognition rate Recognition results for different classifier inputs on the VOT2013 database.

	f L t f L t , f G t f L t , f G t , f D t	26.28% 27.25% 30.03%
	{F t }	35.80%
	{F t , C t }	40.06%
	{F t , C t , S t-1 }	81.80%
	Method		Accuracy Failures
	Best Confidence (BC)		0.559	3.513
	Context Classifier		0.540	1.617
	Context Classifier with HMM		0.574	0.887
	Context Classifier with HMM and Kalman Filter	0.553	0.583

Table 2 :

 2 VOT2013 Benchmark results for the proposed method and baseline (BC).