Mickael Guironnet

Denis Pellerin

Michele Rombaut

VIDEO CLASSIFICATION BASED ON LOW-LEVEL FEATURE FUSION MODEL

This article presents a new system for automatically extracting high-level video concepts. The novelty of the approach lies in the feature fusion method. The system architecture is divided into three steps. The first step consists in creating sensors from a low-level (color or texture) descriptor, and a Support Vector Machine (SVM) learning to recognize a given concept (for example, "beach" or "road"). The sensor fusion step is the combination of several sensors for each concept. Finally, as the concepts depend on context, the concept fusion step models interaction between concepts in order to modify their prediction. The fusion method is based on the Transferable Belief Model (TBM). It offers an appropriate framework for modeling source uncertainty and interaction between concepts. Results obtained on TREC video protocol demonstrate the improvement provided by such a combination, compared to mono-source information.

INTRODUCTION

To respond to the increase in audiovisual information, various methods for indexing, classification and retrieval have emerged. The need to analyze the content has appeared to facilitate video understanding, which will contribute to a better automatic classification.

Recent advances in content analysis have allowed video annotation systems to be developed. However, the difficulty consists in bridging the gap between low-level features and semantic concepts. Semantic video classification techniques are divided into two categories: (i) Rules-based approaches exploit a priori knowledge on a particular domain (like for example sport video) for extracting concepts [START_REF] Shih | Semantic network modeling for understanding baseball video[END_REF][START_REF] Zhou | Rule-based video classification system for basketball video indexing[END_REF], (ii) Statistical approaches try to achieve annotations with an independent analysis of videos. The statistical methods generally require a training to categorize video scenes from low-level features automatically. Different classifiers can be proposed, for example, Bayesian networks [START_REF] Shih | Semantic network modeling for understanding baseball video[END_REF] or Support vector machine (SVM) classifier [START_REF] Naphade | On supervision and statistical learning for semantic multimedia analysis[END_REF]. In this last case, different combinations of features (only color descriptor, concatenation of color and texture descriptors,. . . ) are considered to create the SVM model and the combination that gives the best results is chosen as the optimal combination. In [START_REF] Al-Ani | A new technique for combining multiple classifiers using the Dempster-Shafer theory of evidence[END_REF], several features are extracted and each feature is used to train an Artificial Neural Networks (ANN) classifier. From these classifiers, a combination method is achieved based on Dempster-Shafer theory. However this combination requires a training again to minimize the mean square error between the combined output and the target output of a training set.

In this paper, a system framework for automatically semantic video annotation is described. The system architecture is depicted in figure 1. We distinguish three steps: sen-sor, sensor fusion and concept fusion. Firstly, the sensor step consists in creating a set of sensors from a low-level descriptor (based on color or texture) and a Support Vector Machine (SVM) learning to allow a given concept (for example, boat, beach or basketball. . . ) to be predicted. Secondly, the sensor fusion step corresponds to the combination of sensors for each concept. Finally, the concept fusion step models interaction between concepts. The fusion method is achieved using the Transferable Belief Model (TBM). This is appropriate to model source uncertainty and combine information. We have applied our system using TREC video protocol [START_REF] Smeaton | TRECVID 2004 An introduction[END_REF]. The interest of working with the TREC video base is the great quantity of data (reference video segmentation, ground truth for high-level concept extraction. . . )

The rest of this paper is organized as follows: In section 2 the image descriptors are presented. Section 3 describes the method of high-level concept extraction. In section 4, the results of the method are shown. Finally, we present our conclusions.

LOW-LEVEL DESCRIPTORS

It is well-known that color and texture are visual cues used for image classification or similarity searches. Color and texture are important features in image perception.

Color descriptor

Among the color descriptors, we retain color histogram which captures global color distribution in an image. Selected color space is YCbCr space, which is used in compression MPEG. However, we do not use a uniform quantification of the color space which gives the same weight to the pixels near the centre of a bin as those that are located at the edges. The use of fuzzy sets allows each pixel to associate a membership degree to each bin. Each component of YCbCr The histogram is updated as follows:

h(b i ) = h i h(b i+1 ) = 1 -h i (1)
For i, j, k = 0 to 1, h 3d (b i , b j , b k ) = h(b i )h(b j )h(b k )+h 3d (b i , b j , b k ) (2)
Finally, after a normalization by the image size, a fuzzy 3D histogram with 8x8x8 components is obtained.

Texture descriptor

Texture has been widely studied in recognition tasks. While many computational approaches have been proposed, we chose to design a descriptor inspired by human perception and adapted to describe video content. This descriptor is divided into two steps: retinal filter followed by a Gabor decomposition.

At the first level of image processing, the retinal filter [6] performs an adaptive compression of brightness intensity followed by high-pass filtering. It provides a relative insensitivity to local illumination variations and then carries out a spectral whitening compensating for the 1/f image amplitude spectrum of natural images.

In the primary visual cortex, cells are sensitive to stimuli having a certain orientation and a certain frequency with a specific position in the visual field. Here, we chose to model this using two-dimensional Gabor function. Figure 3 shows the bank of Gabor filters used in our experimentation. We carried out this filtering by directly multiplying the retina output image weighted by a Hanning window with the Gabor filter in the Fourier domain. Finally, we obtained the energy E( f k , θ i ) according to 7 spatial frequencies and 7 orientations. A normalization [START_REF] Guyader | Representation espacefrequence pour la categorisation d'images[END_REF] is then applied to be invariable to image quality. Indeed, the blur is an isotropic function of the G( f ) frequency and the normalization carried out by frequency band removes this term.

E( f k , θ i ) = E( f k , θ i )G( f k ) ∑ j E( f k , θ j )G( f k ) = E( f k , θ i ) ∑ j E( f k , θ j ) (3)
Each keyframe is characterized by a matrix 7x7 where each component corresponds to energy for an orientation and a frequency.

HIGH-LEVEL FEATURE EXTRACTION

The fusion method aims to combine the response of the different sensors for a given concept and then model interaction between concepts.

Sensors

The sensors are created from low-level descriptors (color or texture). After a dimensionality reduction of descriptors, a classifier is used to recognize each concept. A principal component analysis (PCA) was performed to reduce feature dimensionality. The training is only applied on the keyframes of TREC 2004 development data. For each descriptor, the selected dimension is chosen to retain at least 99% of the variance. In our experiments, we went from 512 to 64 components for color descriptor and from 49 to 32 for texture descriptor.

Then, the Support Vector Machine (SVM) classifier was applied to learn each concept of TREC video. SVM is successfully used in a variety of pattern recognition tasks. We will quickly describe the principle of the SVM, a more detailed description can be found in [START_REF] Schlkopf | SVMs -a practical consequence of learning theory[END_REF]. Let {x 1 • • • x n } be a set of training data which are feature vectors of labeled images. We are also given their labels {y 1 • • • y n } where y i ∈ {-1, 1}. The problem consists in approximating an unknown function g such as:

g(x) = L ∑ i=1 y i • w i • K(x i , x) + b (4)
where K( , ) is the kernel function, x i are called support vectors determined from training data, L is the number of support vectors, y i is the label associated with each x i , and w i , b are constants determined from training. In this study, the commonly used radial basis function (RBF) kernel is considered. SVM classifier consists in finding the hyperplane that separates the training data with a maximal margin. The classification of a new vector x is given by the sign of decision function g. Nevertheless we prefer to consider a confidence measure for classification. The perpendicular distance from the hyperplane to vector x is used as a confidence measure. We apply the SVM light developed by Thorsten Joachims [START_REF] Joachims | Advances in Kernel Methods -Support Vector Learning[END_REF] with the default parameters.

A ground truth of each concept is carried out on TREC 2004 development data and allowed SVM model to be created. The classifiers are then applied on the keyframes of TREC 2004 test data. The important thing in this article is not the classifier used but the way in which we combine the classifiers. Finally, for each concept, two sensors are created from color descriptor and texture descriptor.

Sensor fusion step

The fusion method is based on Transferable Belief Model [START_REF] Smets | The transferable belief model[END_REF] coming from the Dempster-Shafer theory. This tool is adapted to deal with imprecise information.

Transferable Belief Model

The set of hypotheses is defined:

Ω = {H 1 , • • • , H n }.
The different sources of information will give a belief to subsets A i ∈ Ω. For each source, a Basic Belief Assignment (BBA) is defined as:

m : 2 Ω → [0, 1] A i → m(A i ) (5)
where 2 Ω is the set of all subsets of Ω and m(A i ) is called basic belief masses and represents a confidence measure that is assigned to the subset A i . The attribution of BBA for each information source is constrained by the following rules:

m( / 0) = 0 ∑ A i ∈2 Ω m(A i ) = 1 (6)
where / 0 is the empty set. Let m 1 and m 2 be the BBA respectively attributed by source 1 and source 2, their conjunctive combination is defined as:

m 12 (A i ) = ∑ A j ∩A k =A i m 1 (A j ) • m 2 (A k ) (7)

BBA definition

Without statistical knowledge, the fuzzy sets can be used to model the BBA from the output x of the SVM learned on a given concept. Figure 4 shows how SVM output is used to attribute a confidence measure. The distribution of SVM output (fig 4.a) corresponds to the perpendicular distance from the hyperplane of SVM model to the image vector. The closer the distance is to zero, the more likely the classifier is to make an error and reciprocally. We then defined the BBA as shown in figure 4.b. If the output of the "Beach" concept is considered, the set of hypotheses is

Ω = {H 1 = Beach, H 1 = Beach}. m x (H 1 )
is the confidence that is assigned to the "Beach" concept , m x (H 1 ∪ H 1 ) represents the doubt about the "Beach" concept and m x (H 1 ) is the confidence that is not the "Beach" concept.

Sensor fusion

The purpose of sensor fusion is to model the fusion of several sensors on the same concept. Each sensor carries out an observation and assigns its confidence over the set Ω. In our experimentation, two sensors are defined from color descriptor and texture descriptor. Table 1 illustrates the combination of two sensors which relates the same concepts H i . A mass is often assigned to the empty set. This is interpreted such as a conflict between the sensors. The conflict means that one of the sensors makes a mistake but that one does not know which. The mass in conflict H i ∩ H i = / 0 is then transfered on H i ∪ H i . This avoids making decision. Thus the combination with other sensors will be able to allow a correct classification. (a)

0 -2 2 0 1 x m x H 1 U H 1 H 1 H 1 (b)
m 2 H i H i ∪ H i H i m 1 H i H i H i H i ∪ H i H i ∪ H i H i H i ∪ H i H i H i H i ∪ H i H i H i

Decision

We use the TREC protocol to evaluate classification and this requires a ranked shot list per concept to be submitted. The rules of decision are then applied to singleton hypotheses H i . The mass m(H i ) is selected for the final decision and represents the degree of confidence placed exactly on the concept H i . A shot can contain one or more keyframes and the passage of keyframes to shot is carried out in the following way:

v i = max I k ∈S i (m(H){I k }) (8)
where I k keyframes belong to a S i shot and v i is the confidence degree of the concept H placed on the S i shot.

Concept fusion step

The concept fusion step models interaction between concepts. The principle consists in combining a concept with another concept having a good reliability to improve the classification. The BBA on a set Ω 1 = {H 1 , H 1 } can also be combined with the BBA of another set Ω 2 = {H 2 , H 2 } if a relation exists between H 1 and H 2 . For instance, if they are exclusive, Table 2 shows how the combination is carried out. Several strategies can be adopted to deal with the empty set. As previously, the mass is transfered on the union H 1 ∪ H 1 .

EXPERIMENTS AND DATA ANALYSIS

In this section, our purpose is to judge the effectiveness of the fusion method previously described.

Results of fusion

The evaluation of the ranked shot list is performed by average precision (AP) and total number of relevant documents (NRD) returned for a given concept. Average precision is the mean of the precision value obtained after each relevant shot is retrieved. Table 3 illustrates the method of fusion. The results show that the number of documents found by combining the color and texture sensors (Sensor fusion) is higher or equal to the number of documents found independently by the sensors. Obviously, the results depend on the kind of descriptor and the training carried out with the classifier. If the sensors taken separately are not very effective, their combination will not be able to find all the documents. The important point is that the combination of sensors improves the number of documents found. A studied concept can be put in competition with another concept of better quality in order to improve the precision. It allows false alarms of ranked shot list to be removed. A new concept is then defined containing mono-color and few textured images because often black images are inserted in videos. The concepts interact with the "Monocolor" concept and the results are shown in Table 3 (Concept fusion). The results show that the number of documents found and the precision increases.

This process is iterative and the concept fusion output can be combined with other concepts. For example, if a "Natural landscapes" concept is created, we will be able to combine it with the "Basket scored" concept but not with the others because they are not exclusive of landscapes. This combination still improves the results with 37 relevant shots found against 34 and a mean average precision equals 0.0117 against 0.0065. The difficulty consists in finding new concepts which will be able to remove the false alarms as well as possible.

CONCLUSION

We have presented a method of high-level concept extraction. This approach is divided into three steps. First, sensors are created for each concept from color or texture descriptors and SVM learning. Then, the sensor fusion is performed for each concept to improve the classification. Finally, the concept fusion models interaction between concepts. The fusion method is based on the Transferable Belief Model. This tool is adapted to model imprecise information of sensors. Results obtained on TREC video protocol demonstrate the improvement provided by such a combination, compared to mono-source information. The fusion method can be applied with other sensors (audio information) and can also be used to model other interactions between concepts.

Figure 2 :

 2 Figure 2: A membership degree of a pixel to each bin. is quantified into 8 bins as shown in figure 2. For each pixel p, the bins b i , b j and b k are computed, respectively for each component of YCbCr as:

Figure 1 :

 1 Figure 1: System architecture for sensor fusion and concept fusion.

Figure 3 :

 3 Figure 3: Bank of 49 Gabor filters.

Figure 4 :

 4 Figure 4: (a) Example of distribution of the SVM output on the training set of "Beach" concept. (b) BBA definition from SVM output.

Table 1 :

 1 Interaction of two sensors m 1 and m 2 for the same concept.

Table 2 :

 2 Combination of the m 1 concept with a m 2 concept having good reliability.m 2 H 2 = H 1 H 2 ∪ H 2 = H 1 ∪ H 1 H 2 = H 1 ∪ H 1

	m 1
	4.1 Data
	We considered a high-level feature extraction task over a
	keyframe dataset of TREC Video data. The elementary unit
	in the context of TREC is a shot and a shot can contain one or
	more keyframes. This data is provided by the National Insti-
	tute of Standards and Technology (NIST). The TREC 2004
	development data consists of 254 videos which is represented
	by 138823 keyframes and the TREC 2004 test data has 128
	videos with 48818 keyframes. The video collection contains
	CNN or ABC news and advertisements.

Table 3 :

 3 Results of sensor and concept fusion. The first line of each concept corresponds to total number of relevant documents and the second one is average precision.

	Concept	Tot. Number	Texture	Color	Sensor	Concept
		Relevant Doc.	sensor	sensor	fusion	fusion
	Boat	441	62	120	120	120
	Ship		0.0054	0.0185 0.0182	0.0191
	Beach	374	84	139	139	145
			0.0143	0.0358	0.036	0.0381
	Basket	103	11	22	34	34
	scored		0.0006	0.0049 0.0065	0.0071
	Airplane	62	18	14	20	28
	takeoff		0.0045	0.002	0.0049	0.0327
	People	1695	153	167	171	191
	walking		0.009	0.0084 0.0081	0.0102
	Physical	292	28	41	50	54
	violence		0.0016	0.0024 0.0036	0.0048
	Road	938	205	162	243	279
			0.0418	0.0128 0.0322	0.0429


Mickael Guironnet
email: guironnet@lis.inpg.fr

Nathalie Guyader
email: nguyader@yahoo.fr

Denis Pellerin
email: pellerin@lis.inpg.fr

Patricia Ladret
email: ladret@lis.inpg.fr

STATIC AND DYNAMIC FEATURE-BASED VISUAL ATTENTION MODEL: COMPARISON TO HUMAN JUDGMENT

In this paper, a novel bottom-up visual attention model is proposed. By using static and dynamic features, we determine salient areas in video scenes. The model is characterized by the fusion of spatial information and moving object detection. The static model, inspired by the human system, is achieved by a retinal filtering followed by a cortical decomposition. The dynamic model is carried out by an estimation and a compensation of camera motion. Although several approaches to visual attention were developed in various applications, few compared their model to human perception. A psychophysical experiment is then presented to compare our model with human perception and to validate it. The results provide a quantitative analysis and show the efficiency of this approach.

INTRODUCTION

Visual attention has become an important feature in image analysis processing. It aims at reducing the quantity of data and considering only relevant information. Indeed, attention represents the capacity of humans to focus on a visual object by watching a scene. Attention shifts our gaze according to two kinds of processing: task dependent processing (top-down) and primitive selective attention processing (bottom-up). In this last case, the saliency is determined unconsciously by the visual system. It is obtained from basic information of input image (intensity, orientation...) and driven by the attributes of stimuli in a scene.

Despite recent advances in video content analysis and event detection, understanding of videos is still far from being achieved. To try to bridge the semantic gap between low-level visual features and high-level concepts, attention is used to facilitate understanding of videos and thus their indexing. Attention model allows regions of interest to be selected in scenes and reduces the number of locations to be analyzed. Saliency maps have been studied in computer vision. The first models relied on spatial information. From different low-level features (orientation, intensity and color), models were defined combining them as for example in [START_REF] Itti | Target detection using saliencybased attention[END_REF]. Other approaches created more elaborated models. Chauvin et al. [START_REF] Chauvin | Natural scene perception: visual attractors and image processing[END_REF] proposed a model inspired by the retina and the primary visual cortex cell functionalities. Then, attention was introduced into video analysis. The methods exploited the temporal component and were based on a motion estimation [START_REF] Ma | A user attention model for video summarization[END_REF]. New systems appeared by combining maps of static and dynamic visual attention [START_REF] Ho | A user-attention based focus detection framework and its application[END_REF][START_REF] Courty | A new application for saliency maps: Synthetic vision of autonomous actors[END_REF]. In [START_REF] Chen | Image adaptation based on attention model for small form factor devices[END_REF], the authors proposed an attention model based on visual static features but also on face and text detection. In general, these computational models are not inspired by functionalities of hu-man visual system cells and the saliency maps obtained are not compared to human perception. These maps are directly used through various applications like video summarization, encoding, watermarking or surveillance.

In this paper, we propose a new framework to model bottom-up visual attention. It relies on the fusion of a static model inspired by the human system and a model of moving object detection in a scene. The static model is based on retinal filtering followed by a bank of Gabor filters. The moving object detection is carried out by compensation of camera motion. Once the visual spatio-temporal attention model was built, a psychophysical experiment allowed us to validate the proposed model. The main contributions of our work are a new user attention model and the building of an experiment to judge the effectiveness of the method.

ATTENTION MODEL

In this section, we describe our attention model. This model extracts the salient areas from videos. It is divided into two parts: a static and a dynamic one. The model architecture is depicted in figure 1.

Static part of the model

This part is inspired by biology and functionalities of human visual system cells (from the retina to the primary visual cortex). This part of the model concerns each frame of the videos.

Retinal filtering

At the first level of information processing, the retinal photoreceptors carry out an adaptive compression process followed by high-pass filtering [START_REF] Beaudot | Sensory coding in the vertebrate retina: towards an adaptive control of visual sensitivity[END_REF]. This results in contrast equalization of the image, providing a relative insensitivity to local illumination variations. This pre-processing is interesting for extracting saliency because it is invariable to some modifications, such as luminance variations between images.

Then, the parvocellular pathway provides a spatial high-pass filter (known to whiten an image's frequency spectrum) that compensates for the 1/f image amplitude spectrum.

Primary visual cortex

Primary visual cortex cells are sensitive to visual signal orientations and spatial frequencies. Here, we chose to model simple cell receptive fields. These cells are sensitive to stimuli having a certain orientation and a certain frequency with a specific position in the visual field, which is modelled by a two-dimensional Gabor filter. A Gabor function is defined by a gaussian with spatial extents σ x and σ y modulated by a complex exponential with frequency f in a direction θ . We chose 4 frequency bands f 0 = [0.0375, 0.075, 0.15, 0.3] and 8 orientations

θ i = iπ 8 . g(x, y, f 0 , θ i , σ x , σ y ) = 1 2πσ x σ y e 2π j f 0 x e -( x 2σ 2 x + y 2σ 2 y ) x = x • cos θ i + y • cos θ i y = y • cos θ i -x • cos θ i (1)
We carried out this filtering by directly multiplying the retina output image with the Gabor filter in the Fourier domain. Before achieving the Fourier transform, we multiplied the image by a Hanning window to remove edge effects. We chose here to decompose each image of the video using thirty-two Gabor filters (four different spatial frequencies and eight different orientations). So, we obtained thirty-two maps (thirty-two images), depending on the frequency and the orientation of the original image, for each frame of the video.

Interactions

A neuron is, by definition, a contact cell. Thus, the response of a cell is always dependant on a neuronal environment. Then, the neuron's activity is modeled by the visual field neighborhood and so is dependent on lateral connections. With regard to the orientations, the computed interactions preferentially connect neurons devoted to the same orientation. These interactions, which symbolize both excitatory and inhibitory connections, are modeled by a linear combination of the simple cells (fig 2):

E int ( f i , θ j ) = ∑ k,l w • E( f k , θ l ), w = 0 -0.5 0 0.5 1 0.5 0 -0.5 0 (2)
Figure 2 illustrates the interactions. In this example, the Gabor filter with weight 1 in direction π/4 interacts with its neighbours. The Gabor filters with the same direction symbolize excitatory and those with different directions represent inhibitory. The output of this stage is composed of 32 maps for each image of a video. These maps put into relief the image energy in function of the spatial frequency and the orientation of the signal in the original image and take into account the interaction between the orientation maps.

Static saliency map

We extract a static saliency map for each image as the sum of the 32 energy maps described above:

E f = ∑ i, j E int ( f i , θ j ) (3)
The regions having the highest energy are considered to be salient. Figure 3 shows examples of static saliency maps. The content of the images is rather varied: a boat and surfers close to a beach, a rugby match, a person in a hall. We can observe on the bottom row that the energy is located on objects which seem to be salient.

Dynamic part of the model

The dynamic part of the model detects the moving objects in a scene. In fact, we assume that the location where something moves is salient. Next, it is necessary to estimate camera motion. We use the 2D motion estimation algorithm developed in [START_REF] Odobez | Robust multiresolution estimation of parametric motion models in complex image sequences[END_REF]. This algorithm provides the dominant motion between two successive frames. A 2D parametric motion model between two successive frames is then defined and a robust multiresolution estimation of parametric motion models is carried out. We chose affine motion model to represent the camera motion.

v x = a 1 + a 2 • x + a 3 • y v y = a 4 + a 5 • x + a 6 • y (4)
Once we had the coefficients [a 1 , ..., a 6 ], we computed the motion compensated frame. Compensation of camera motion was formed by bilinear interpolation I c (x, y,t + 1) = I(x + v x , y + v y ,t + 1). The previous frame was then subtracted from the motion compensated frame to generate Displaced Frame Difference (DFD):

DFD(x, y,t) = I c (x, y,t + 1) -I(x, y,t) (

Finally, the absolute value of DFD informs about regions that do not follow camera model and corresponds to displacement of objects. Figure 4 shows examples of object detection. We can see that the surfers are correctly detected as well as the rugby players, and the person in hall.

Spatio-temporal attention model

Before dealing with map fusion, it is necessary to carry out a temporal filtering of each map. Indeed, the maps are computed locally, either on one frame (for the static model) or on two successive frames (for dynamic model), and the salient regions must be temporally coherent inside a window of duration L. The temporal continuity of the video prevents the appearance of salient areas on one or two images only (two frames correspond to 2/25 = 0.08s). This is why a median filtering of width L is carried out. In our experiment, the size of window L equals five frames. Since the maps do not have the same magnitude, a standardization stage is necessary before carrying out the map fusion. This stage is carried out by the following:

S n = S/T h if S < T h 1 if S ≥ T h ( 6
)
where S is a static or dynamic saliency map and T h is a predefined threshold (25 in the two cases).

Once the maps have been normalized, a fusion stage is achieved to combine all maps into a final saliency map. The fusion is performed using the max operator which can be interpreted as an "or" logical operator. Thus, the final map contains static and dynamic information. Finally, the map obtained is a gray image with a higher value for salient zones.

Thanks to image processing techniques, we detect the regions of attention. The following steps are achieved: thresholding, morphological operation (close and open), region selection. In the last step, we determine the regions according to 4-connected neighbourhood. The regions with area lower than a threshold are removed. Finally, the remaining regions are selected and defined as masks. If the number of masks is greater than five, we keep only the five biggest masks. Figure 5 illustrates the fusion of maps and the selection of attention masks: boat and surfers, crowd and players, and the person.

EXPERIMENTAL RESULTS

In order to test and to validate our model, we carried out a psychophysical experiment. The goal of the experiment was to know if the areas defined as salient by the model are indeed salient. We tried to compare the model with human perception.

Method

Subjects

Sixteen naive subjects underwent the experiment. All subjects had normal or corrected to normal vision.

Stimuli

The subjects saw nine different videos displayed in the middle of the screen with a frequency of 25 images per second. Videos were composed of 288x352 pixel-images in 256 graylevels. For one randomly selected image of one video, we associate to it the spatio-temporal attention map using the presented model. In order to test if the salient areas provided by the model are in agreement with the human visual perception we take exactly the same image and the same masks but we apply the masks to random positions as shown in figure 6. The principle is as follows: the mask of model having highest area is first randomly moved in the image; the second mask having highest area is then moved but without possible overlapping with the other mask and so on. . . Finally the two images (fig 6) have the same masks but at different places in the image.

Procedure

The experiment was processed with a computer with a Pentium III processor. The stimuli were presented on a 21" screen (Mitsubishi Diamond Pro2020u) with a resolution of 1024 by 768 pixels and a frame rate of 100 Hz. Subjects were placed at a distance of approximately 50 cm from the screen. Figure 7 describes the events for one trial: a fixation point appeared (here a small black cross) in the middle of the screen for two seconds, followed by a 1.2s video still in the middle of the screen. Then, two images were presented symmetrically in the middle of the screen. These images belonged to the previous video and were masked in different ways: one following the model and the other with random position masks. The subject had to choose which one seemed to him to be the closest to the video. The selected image should have represented the best video content. His response and the reaction time were measured with a response box and E-Prime software. His answer had to be given as quickly as possible.

Each video appeared four times, with two different target images in the two possible positions on the screen. So in two cases the same images are used and in one case the image provided by the model is on the right side of the screen and in the other case is on the left. This allows us to have more answers for one condition and to see if a subject gave his answer randomly. The experiment is divided into three phases. Each phase contains three videos and so twelve random trials. During one experiment, each subject answered 36 trials.

Results

For the analysis we only kept fourteen subjects (two subjects had random responses). We measured the percentage of correct responses per subject (the correct response is in the case where the subject chose the model mask). Over all subjects the mean correct response percentage is 88% with a mean standard deviation of 5%. As we expected, for all subjects the model masks correspond more to the video than the random masks. We can refine these results. For some masks there is an overlapping between the model masks and the random ones. So we should find fewer correct answers when the model masks are overlapped with random masks rather than when the masks are separated. So we added a condition: when masks had more than 50% of overlapping and when masks had less than 50% of overlapping. We made an analysis of variance (ANOVA) for the percentage of correct response as a function of these two conditions. The overlapping influences the correct response (F(1, 13) < 0.001). So the percentage of correct responses is lower when the model masks and the random masks are overlapped, which consolidates the model.

CONCLUSION

We have presented a spatio-temporal attention model. It relies on the fusion of a static model inspired by the human system with a model of moving object detection. A psychophysical experiment was proposed to judge the effectiveness of the model. The proposed model provides good results with a precision of 88%. These results are promising. In addition, the model can be used in many applications such as video indexing, summarization, watermarking and surveillance. One of the future works would be to use this model to provide a video summary and to use an experimental paradigm to test the efficiency of the method.

Figure 1 :

 1 Figure 1: Principle of the spatio-temporal attention model

Figure 2 :

 2 Figure 2: Example of interactions. The Gabor filter with weight 1 in direction π/4 interacts with its neighbours.

Figure 3 :

 3 Figure 3: Examples of static attention maps. Top row: video frame. Bottom row: the associated static attention map.

Figure 4 :

 4 Figure 4: Examples of moving object detection. Images represent the absolute value of Displaced Frame Difference.

Figure 5 :

 5 Figure 5: Examples of spatio-temporal attention masks.

Figure 6 :

 6 Figure 6: Example of the target images. The left image is the output of the presented model. The right one is the same image with the same masks but placed in random positions.

Figure 7 :

 7 Figure7: Experimental design: one trial sequence is illustrated. First, a video appeared for 1.2s, and then two images appeared in the middle of the screen. The task was to choose the image that is closest to the video.

Acknowledgment

The authors would like to thank Christian Marendaz (Laboratoire de Psychologie et NeuroCognition, Grenoble, France) for having received us and let us use the material for the experiments. The authors also thank the Vista Research team at Irisa/Inria Rennes for the use of the Motion2D software.