Bryan Prosser

Shaogang Gong

Tao Xiang
email: txiang@dcs.qmul.ac.uk

Multi-camera Matching under Illumination Change Over Time

Illumination differences between disjoint cameras can have a dramatic effect on the appearance of objects, thus increasing the difficulty of multi-camera object association. Although methods to model these inter-camera illumination conditions exist, they often rely on static illumination conditions and are unable to copewith unpredictable illumination changes over time. In this paper we propose a novel method for multi-camera object association based on adapting a learned intercamera illumination mapping function to new illumination conditions over time without the need for a manual training stage using new foreground objects. Comparative experiments are carried out using challenging data taken from a disjoint camera network. The results demonstrate that the proposed method outperforms a number of existing methods given changing illumination conditions.

Introduction

A typical video surveillance system employs a number of networked cameras, many of which have disjoint views. One of the key problems of behaviour monitoring using a networked cameras is to track people across camera views, known as the person re-identification problem. Specifically, to re-establish a match of the same person over different camera views located at different physical sites, one aims to track individuals either retrospectively or on the fly when they move through different sites. Due to considerable changes in object orientation, pose and lighting conditions between camera views, this task is non-trivial. Since real world camera networks rarely have overlapping views, the key challenge for reidentification is to establish object correspondence given these changes. Among these conditions that vary across cameras, dealing with the lighting condition change is particularly challenging. This is because light conditions at different camera views can change over time in a unknown manner. While methods exist to address the problem of illumination change between camera views [START_REF] Javed | Appearance modeling for tracking in multiple non-overlapping cameras[END_REF][START_REF] Ilie | Ensuring color consistency across multiple cameras[END_REF][START_REF] Cheng | Matching of objects moving across disjoint cameras[END_REF], none of them consider the lighting condition changes over time. These changes at each camera view result in additional changes across cameras not modelled by existing techniques. (see examples in Figure 1). Our aim is therefore to enhance multi-camera re-identification by modelling temporal illumination changes.

Simple appearance based methods currently exist to handle the lighting condition changes between cameras. Cheng et al [START_REF] Cheng | Matching of objects moving across disjoint cameras[END_REF] cluster colours into a subset of major colours, and to alleviate the effect of illumination changes, they employ a histogram equalisation technique. A more sophisticated method is to model the illumination differences between each pair of camera views explicitly. Javed et al [START_REF] Javed | Appearance modeling for tracking in multiple non-overlapping cameras[END_REF] proposed a subspace based colour brightness transfer function (BTF). They use probabilistic PCA to calculate the subspace of BTFs for a set of known correspondences and compare the BTF of a test object pair against this subspace to determine a correspondence. Prosser et al [START_REF] Prosser | Multi-camera matching using bi-directional cumulative brightness transfer functions[END_REF] also use a BTF-based approach but accumulate training data before computing the BTF. This Cumulative BTF (CBTF) enables sparse colour information to be preserved through the BTF calculation process. Gilbert and Bowden [START_REF] Gilbert | Tracking objects across cameras by incrementally learning inter-camera colour calibration and patterns of activity[END_REF] model inter-camera colour transformations using an incrementally updated transformation matrix. A similar model was proposed in [START_REF] Ilie | Ensuring color consistency across multiple cameras[END_REF] without incremental learning. Instead, it employs a hardware calibration phase to learn colour differences, which is impractical because many real world systems do not offer the access to the hardware parameters.

The main shortcoming of the previous approaches is that they are trained for a single constant lighting condition at each individual view. If the illumination at any one of the cameras changes over a period of time, the BTF based approaches would require manual selection of a substantial set of corresponding object observations in each camera to re-learn the brightness mapping under the new conditions. It is therefore tedious, time-consuming, and above all is not even possible when illumination condition changes are non-gradual. The histogram equalisation technique used by Cheng et al [START_REF] Cheng | Matching of objects moving across disjoint cameras[END_REF] assumes some arbitrary a priori knowledge of a suitable colour mapping function which again would need to be updated manually.Gilbert and Bowden's incremental colour mapping method [START_REF] Gilbert | Tracking objects across cameras by incrementally learning inter-camera colour calibration and patterns of activity[END_REF] has the potential to cope with temporal illumination changes provided such changes are gradual. However, their method requires thousands of object appearances to learn an accurate brightness mapping between camera views, which are unlikely to be available especially when the lighting conditions changes are non-gradual.

In this paper we propose a novel method for multi-camera people association based on adapting cumulative brightness transfer function (CBTF) to new illumination conditions without the need for a manual training stage using new foreground objects. This method therefore can run in real time even given con-stantly changing lighting conditions. More specifically, by modelling the temporal changes in background illumination, the updated CBTF is estimated using a combination of the original CBTF and the background illumination changes at each camera. The proposed method is evaluated using challenging datasets obtained from a real world CCTV camera network. The results demonstrate that the adaptive CBTF estimation is accurate and the proposed method significantly outperforms existing approaches.

Inferring Illumination Changes Over Time

Given a pair of camera views i and j and the CBTF cf ij obtained using a set of object correspondences, we aim to adaptively update the CBTF to any change of illumination condition without collecting new object correspondences. The new camera views under a different illumination condition is denoted as i ′ and j ′ , and the updated CBTF cf i ′ j ′ . This is achieved through calculating a colour mapping function for each of the two camera views over time, denoted as f i ′ i and f j ′ j respectively. This enables us to convert object images under a different illumination condition back to the illumination conditions under which the original CBTF was learned. The approach is illustrated in Figure 2. Fig. 2. Camera views i,j and i ′ ,j ′ under different illumination conditions. By modelling the illumination change for each camera view (f i ′ i and f j ′ j ) we are able to utilise the original inter-camera cfij to infer the new inter-camera cf i ′ j ′ without re-training.

Brightness Transfer Function

First let us formally define the multi-camera person re-identification problem. A camera network has m cameras C 1 , ..., C m all of which are assumed to have non-overlapping views. We break down each view into entry/exit regions as illumination varies between both inter-and intra-camera regions. Specifically, for each of the camera views we define its set of n entry/exit regions as E 1 C1 , ..., E n C1 . We then simplify this by describing the global set of g entry/exit regions as E 1 , ..., E g . Next we define a set of k object observations in each entry/exit region E i as {O i,1 , ..., O i,k }. Using an existing single camera view tracking system we can obtain these observations by taking a colour histogram of a target object as the centre of its bounding box passes through an entry/exit region. In order to solve the multi-camera re-identification problem we form a set of correspondence hypotheses Q where each Q j,b i,a indicates a potential match between observations O i,a and O j,b . We consider the solution space S as a set of all possible correspondence hypotheses between E i and E j . Assuming that an object in E i is seen no more than once in E j we aim to find the subset of S, s where each Q j,b i,a ∈ s, if and only if observations O i,a and O j,b are the same person. The solution of the multi-camera re-identification problem is then defined as s ∈ S which maximises an observation similarity measure:

s = arg max s Q j,b i,a ∈s Similarity(O i,a , O j,b ) (1)
where Similarity() is the similarity between O j,b and O i,a in the testing data.

A vital part of the re-identification process lies in the BTF. Javed et al [START_REF] Javed | Appearance modeling for tracking in multiple non-overlapping cameras[END_REF] suggested that a BTF f ij ( ) between cameras C i and C j can be constructed by sampling values from a set of fixed increasing brightness levels B i (1), ..., B i (d), and

(B j (1), ..., B j (d)) = (f ij (B i (1)), ..., (f ij (B i (d))
). In the case of a common 8-bit per channel image, d is set to 256. To establish such a mapping function between views, a pair of known correspondence must be available. These correspondences are represented as normalised histograms of RGB brightness values. Computing a mapping function can be achieved as follows. It is assumed that the percentage of pixels in an observation O i with the brightness value less than B i is equal to the percentage of image points seen in O j of brightness less than or equal to B j . H i and H j are then defined as cumulative histograms. More specifically, for H i each bin of brightness value B 1 , ..., B m , ..., B 256 in one of the three colour channels is obtained from the colour histogram h i as follows:

H i (B m ) = m k=1 I i (B k ) (2)
where

I i (B k ) is the count of brightness value B k in O i .
Each bin is then normalised using the total number of pixels in O i . H i (B i ) represents the proportion of H i less than or equal to B i , then

H i (B i ) = H j (B j ) = H j (f ij (B i ))
and the BTF mapping function can be defined as:

f ij (B i ) = H -1 j (H i (B i )) (3)
with H -1 representing the inverted cumulative histogram. An example BTF and the corresponding observation images can be seen in Figure 3.

In order to produce a more accurate transfer function, multiple BTFs can be estimated. Prosser et al [START_REF] Prosser | Multi-camera matching using bi-directional cumulative brightness transfer functions[END_REF] show that mean BTF based methods rely on having a consistent set of coloured individuals to accurately model the BTF and that taking the mean of a set of BTFs actually removes vital colour information that may only be contained in a small subset of the training data. Rather than computing a BTF for each training pair they accumulate the brightness values of the whole training set before the BTF computation. The cumulative histogram

After obtaining this cumulative histogram using all the training image pairs, the CBTF is computed as follows

cf ij (B i ) = Ĥj -1 ( Ĥi (B i )) (5)
This CBTF is used as the cf ij from Figure 2 in the following section.

Modelling Temporal Illumination Changes

The first stage in modelling the illumination change is to derive a single median background image from each of the two datasets collected under different illumination conditions. To do this we collect a set of background images, I 1 , ..., I n , by using a background/foreground subtraction technique to find images containing minimal foreground regions. In our case we chose a set size of 20 images. From this set of images we compute the median RGB values at each pixel location and produce a median background image. The two median background images for the two different illumination conditions are denoted as M i (x, y) and M i ′ (x, y).

In each of the median background images we extract regions of interest R that corresponds to entry/exit regions of a camera. In our work these regions are manually defined, however we are aware that work exists to extract these automatically, such as [START_REF] Makris | Automatic learning of an activity-based semantic scene model[END_REF][START_REF] Li | Scene segmentation for behaviour correlation[END_REF]. As the background of a scene may change over time due to reasons other than illumination change, e.g. the movement of a static object, we perform frame differencing to remove these areas so that they do not pollute the final colour mapping. Let Mi (x, y) denote a region of interest R after frame differencing, we have:

∀x, y ∈ R, Mi (x, y) = M i (x, y) if abs(M i (x, y) -M i ′ (x, y)) < σ 0 otherwise ( 6
)
where σ was typically between 30 and 50. An example of regions of interests extracted from median background images of an entry/exit region of a camera is shown in Figure 4. From Mi (x, y) and similarly calculated Mi ′ (x, y) we can then estimate the illumination change for each camera. To model the illumination changes we work on the same principle as the brightness transfer function outlined in Section 2.1. That is we assume that the percentage of pixels in background image M i ′ (x, y) with the brightness value less than B i ′ is equal to the percentage of image points seen in M i (x, y) of brightness less than or equal to B i . And thus we use Equation (3) to compute f i ′ i and f j ′ j from Figure 2 as follows:

f i ′ i (B i ′ ) = H -1 i (H i ′ (B i ′ )) (7)
A similar approach was proposed by Grossberg et al [START_REF] Grossberg | Determining the camera response from images: What is knowable[END_REF], however their method does not consider significant background changes between images. As this mapping may not contain one-to-one brightness mappings we use linear interpolation to estimate unmapped regions. A sample illumination mapping can be seen in Figure 5. The mapping between j ′ and j is then calculated in the same way. Once f i ′ i , f j ′ j and the inter-camera CBTF cf ij have been calculated using Equations ( 7) and (3) respectively objects can be mapped into the previous illumination conditions and objects in view i can be mapped to view j for comparison. Specifically, in order to compare two observations O i ′ ,a and O j ′ ,b , their colours are converted to the corresponding colours in E i and E j , i.e. Òi ′ ,a (B i ′ ) and Òj ′ ,a (B j ′ ), using f i ′ i and f j ′ j respectively:

∀B ′ i , Òi ′ ,a (B i ′ ) = f i ′ i (O i ′ ,a (B i ′ ))∀B ′ j , Òj ′ ,b (B j ′ ) = f j ′ j (O j ′ ,b (B j ′ )) (8)
Next Òi ′ ,a (B i ′ ) must be converted to the illumination conditions of E j , becoming Ôi ′ ,a (B i ), using the learned inter-camera CBTF: Òi ′ ,a (B i ′ ) has now undergone transformation by a combination of f i ′ i and cf ij as depicted in Figure 2.

∀B i , Ôi ′ ,a (B i ) = cf ij ( Òi ′ ,a (B i )) (9
Note that we have assumed so far that the CBTF contains only one-to-one colour relationships. However, in reality the mapping function obtained from the training set often contains cases of many-to-one colour correspondences due to incomplete ranges of colour values found in the training data. To address this problem, a nearest neighbour smoothing function is employed to smooth out the noisy peaks in the resulting histograms. Figure 6 shows an example of the process of converting a potential observation pair for comparison.

Once both O i,a and O j,b are converted to the same illumination conditions we can compare them directly using the Bhattacharya distance measure D() and thus the similarity measure from Equation ( 1) can be defined as follows:

Similarity(O i,a , O j,b ) = 1 -D( Ôi,a , Òj,b ) (10)
Note that in order to compare two colour objects, we must apply this process to each of the three RGB channels. Thus the overall similarity measure becomes the mean of the similarity values obtained in all three channels.

Automatic Model Updating

The CBTF updating process described above is triggered automatically by the detection of an illumination change in each camera site. A background modelling approach such as [START_REF] Russell | Minimum cuts of a time-varying background[END_REF] can be deployed to construct an empty background from a stack of frames containing foreground objects. From this automatically generated empty background region we can calculate the brightness histograms for the entry/exit region over a temporal sliding window. The brightness histograms are then compared against the brightness histograms of the background region from the previous period. Illumination change is detected when the difference between the brightness histograms is larger than a threshold.

Experiments

Two sets of experiments were carried out using challenging datasets collected from a distributed camera network. First, we compare the the proposed adaptive Fig. 6. Example of the conversion from the new illuminations (bottom row) to the old (top row). From here the image from camera i is converted to the illumination conditions of j for comparison using the similarity measure.

CBTF (A-CBTF) method against standard Bhattacharya distance and CBTF colour transfer [START_REF] Prosser | Multi-camera matching using bi-directional cumulative brightness transfer functions[END_REF] without temporal illumination change modelling. Secondly, we compare our method against current inter-camera colour compensation methods [START_REF] Javed | Appearance modeling for tracking in multiple non-overlapping cameras[END_REF][START_REF] Madden | Tracking people across disjoint camera views by an illumination-tolerant appearance representation[END_REF]. In each of these experiments, the BTFs and CBTFs for each colour channel were estimated from a set of training pairs with known correspondences from Day 1 and tested using the observations in Day 2 which has different illumination conditions in each camera view (an example illumination change is shown in Figure 7). In each set of results we show both rank 1 and rank 5 results indicating the presence of the correct match as the highest and top 5 highest similarity scores respectively. Comparing Bhattacharya distance, CBTF, and Adaptive CBTF: Here we demonstrate that temporal illumination change modelling improves on the CBTF, which in turn is an improvement over Bhattacharya distance alone. Each observation was decomposed into its RGB and component histograms at each entry/exit region and compared against all other observations. For the Bhattacharya distance experiment, no colour mapping is performed. For the CBTF we use only the inter-camera CBTF learned from Day 1 (cf ij ) as an estimation of the colour changes between views in Day 2 (cf i ′ j ′ ). Figure 9 shows that the proposed method achieved an approximately 15% improvement in overall matching rate against Bhattacharya distance and CBTF. This validates our assumption that changes in illumination can be approximated using a linear combination of foreground and background changes. Example of object association results obtained using the three methods are shown in Figure 11. ure 9(c) show the three methods give a similar result for camera sites 2 and 3. This is because incidentally the illumination conditions at Sites 2 and 3 are similar, which causes the Bhattacharya distance result to be higher while the minor inaccuracies in the CBTF-based methods cause a slightly lower result.

Comparison with alternative approaches: In this experiment, our adaptive CBTF method is compared against other reported approaches. We have implemented the BTF subspace approach [START_REF] Javed | Appearance modeling for tracking in multiple non-overlapping cameras[END_REF] and the Major Colour Spectrum Histogram (MCHR) approach [START_REF] Madden | Tracking people across disjoint camera views by an illumination-tolerant appearance representation[END_REF], in which object colour histograms are equalised before being decomposed into major colours. Note, as there is no assumed knowledge of the relationship between cameras, our equalisation graph for the MCHR was based on a standard linear equalisation, whilst the graph in [START_REF] Madden | Tracking people across disjoint camera views by an illumination-tolerant appearance representation[END_REF] was nonlinear based on some rather arbitrary a priori knowledge. In addition, as the number of frames in which an object is captured passing through our entry/exit zones is low, the incremental MCHRs cannot be used. More critically though, as our model is designed for online processing, we have excluded their batch-based post matching integration part which cannot be performed online. The results in Figure 10 show that the equalisation based MCHR does not cope well with this challenging data set. Although slightly better, the BTF sub- space approach suffers due to its inability to adapt to the difference between the illumination conditions changes over time. Overall our approach outperforms the two alternative methods by approximately 20% in rank 1 and 15% in rank 5.

Conclusions and Future Work

We have demonstrated that by modelling background illumination changes we can infer new brightness mapping functions between cameras from the original CBTF. In particular, by using background illumination we are able to estimate the changes on the foreground objects without the need for manual association of foreground objects each time these illumination conditions change, which would be required by other approaches. The datasets used provide a challenging test for object association due to the sparse colour information of the objects observed. Although our method produces relatively low matching rates its ability to adapt to new illumination conditions allows it to significantly outperform existing methods. In order to improve inter-object discrimination we plan to model the colour distribution of individuals. This would help us distinguish between objects with similar colour histograms but different colour layouts as can be seen in Figures 11(g) and 11(j). Currently our matching algorithm uses a brute force approach to finding object correspondences. We also plan to incorporate temporal information [START_REF] Makris | Bridging the gaps between cameras[END_REF] to reduce the number of initial correspondence hypotheses.

Fig. 1 .

 1 Fig. 1. Illumination condition can change over time especially when outdoor lighting plays a part. In this case Day 1 was a dull cloudy day and Day 2 was sunny.

Fig. 3 .

 3 Fig. 3. (a) and (c) show a corresponding person in 2 camera views, (b) shows the BTF converting between the two illumination conditions. Ĥi of N training samples in camera view i can be computed from the brightness values B 1 , ..., B m , ..., B 256 as:

Fig. 4 .

 4 Fig.4. Corresponding regions of interest from the same entry/exit region of a camera site on Day 1 and 2 respectively with pixels with large value changes removed. Those removed pixels correspond to an LCD display, a chair, and some magazines, all of which have been changed/moved over the two days.

Fig. 5 .

 5 Fig. 5. Background illumination BTF from the blue channel of Site 3. Note the values on the x-axis (Day 2) corresponding to lower (darker) values on the y-axis (Day 1).

Fig. 7 .

 7 Fig. 7. Example RGB histograms of a single camera view on both days. Day 2 shows significant change in global illumination. Datasets: We obtained two sets of data from inside an office building observed by three cameras. Example views are shown in Figure 8. The illumination conditions and colour quality vary between each views. Camera 1 displays a corridor scene where objects are periodically lit by spotlights causing darker regions in

Fig. 8 .

 8 Fig. 8. Sample frames from two days showing the differing lighting conditions between days in addition to the inter-camera illumination changes. The yellow boxes show the entry/exit zones.

Fig. 9 .

 9 Fig. 9. Comparison of Bhattacharya distance, CBTF and A-CBTF with temporal illumination change modelling.

Fig. 10 .

 10 Fig. 10. Comparitive results from the MHCR-based method, the BTF subspace method and our adaptive CBTF method.

Fig. 11 .

 11 Fig. 11. (a) and (b): the same individual appeared at entry/exit regions 1 and 2 respectively.; (c): BTF(subspace) match; (d): MCHR match: (e): CBTF match (correct one). (f) and (g): A much more challenging case from due to the presence of similar coloured objects in the testing set. (h)-(j): all three methods found the wrong match.


Jordi Salvador

Josep R Casas

Shape from Probability Maps with Image-Adapted Voxelization

This paper presents a Bayesian framework for Visual Hull reconstruction from multiple camera views with a 3D sampling scheme based on an irregular 3D grid, which becomes regular once projected onto the available views. The probabilistic framework consists in establishing a foreground probability for each pixel in each view rather than segmenting in order to obtain binary silhouettes of the foreground elements. Next, a Bayesian consistency test labels the occupancy of each image-adapted 3D sample. The proposed method, using image-adapted 3D sampling in the Bayesian framework, is compared to a shape-fromsilhouette implementation with image-adapted voxelization, where the input data are binary silhouettes instead of probability maps; we also compare its performance to a state-of-the-art method based on regular 3D sampling with binary silhouettes and SPOT projection test.

Introduction

When trying to extract the Visual Hull [START_REF] Laurentini | The visual hull concept for silhouette-based image understanding[END_REF] from images captured by a set of calibrated cameras, numerous approaches consist in, firstly, obtaining a silhouette image of the foreground object for which we want to obtain the Visual Hull and, secondly, applying a shape-from-silhouette algorithm on the segmented images (silhouettes), typically labelling elements of a regular 3D sampling grid (voxels) as being occupied or empty.

2D Foreground Segmentation

An extended method to extract foreground silhouettes from an image is to model the color propoerties of each background pixel as an independent Gaussian random variable and classify pixels in new images as foreground or background depending on their similarity to the generated pixel models [START_REF] Stauffer | Adaptive background mixture models for real-time tracking[END_REF]. For this, training images containing only background should be available.

The obtaining of silhouette images with this method is error-prone in certain scenarios, leading to silhouette images with mislabeled pixels where both false detections, in highlighted or shadowed areas, and misses, when foreground colors are highly similar to background ones, occur.

As a possible solution, we propose avoiding binary thresholding at pixel level when detecting foreground silhouettes. Instead, a foreground probability map can be obtained and used in a probabilistic Visual Hull computation.

Shape from Silhouette

Typically, shape-from-silhouette implementations rely, as many other multi-view analysis techniques, on a regular voxelization of 3D space, which is non-adapted to image data. In settings such as a smart room with a small number of evenly distributed cameras, voxels project to unbalanced sets of image data in each camera, from which analysis algorithms, icluding shape-from-silhouette, have to work out feature matches and consistency checks.

A common strategy to compute the voxelized Visual Hull from a set of silhouettes via shape-from-silhouette is to define a projection test where the center of each voxel is projected onto the images in order to be tested, assuming small or distant voxels. This method has the advantage of its low computational cost, but an important drawback is the undersampling of the actual voxel projections, which leads to wrong decisions, mainly in contours. An alternative method that shows better performance is the SPOT projection test [START_REF] Cheung | Visual Hull Construction, Alignment and Refinement for Human Kinematic Modeling, Motion Tracking and Rendering[END_REF], which takes samples of the actual voxel's projection at random positions and considers aspects such as the distance from a voxel to a camera or the accuracy of the binary silhouettes.

Image-Adapted Voxelization. An alternative strategy to regular voxelization is image-based scanning of the 3D scene. Matusik introduced the concept of image-based Visual Hulls to render an observed scene in real time from a virtual camera point of view without constructing an explicit auxiliary volumetric representation [START_REF] Matusik | Image-based visual hulls[END_REF]. He claims that the advantage of performing geometric computations based on epipolar geometry in image space is the elimination of resampling and quantization artifacts in other volumetric approaches.

Image-adapted voxelization [START_REF] Casas | Image-based multi-view scene analysis using 'conexels[END_REF] follows the concept of image-based processing, but focusing on volumetric analysis applications. In particular, an image-based progressive scanning algorithm for multi-view analysis is used, and the corresponding geometry in 3D space is derived. This provides a volumetric representation for image data functionally equivalent to regular voxelization as volumetric data support for analysis algorithms, with the benefit that the 3D scanning procedure is better adapted to the image data than regular voxelization. The principles underlying this technique and its associated algorithm with detailed explanations can be found in [START_REF] Salvador | Image-adapted voxelization in multicamera settings[END_REF] and [START_REF] Casas | Image-based multi-view scene analysis using 'conexels[END_REF].

Bayesian framework. The method proposed in this paper, Shape from Probability Maps, relies on the image-adapted 3D scanning procedure described above, extrapolating the shape-from-silhouette algorithm for Visual Hull reconstruction to a probabilistic framework with Bayesian formulation.

Document Organization

The document is organized as follows: in Section 2, we introduce the formulation and principles of shape-from-probability-maps. Then, in Section 3, the method is compared to two different implementations of standard shape-fom-silhouette, one based in image-adapted voxelization and the other in regular voxelization with SPOT projection test. To conclude, the main features of shape-from-probabilitymaps are summarized.

Shape from Probability Maps

In order to obtain a foreground segmentation of an image when background images are available, a simple yet functional approach consists in:

1. modelling each background pixel as a Gaussian random variable which describes its color, with mean and variance extracted from the background images 2. Classifying pixels from new images, testing whether they fit in the background model or not Furthermore, the previous approach allows updating the background models with color information from pixels classified as belonging to the scene's background. This extension allows correctly segmenting images in scenarios where, for example, light conditions suffer slow variations.

An undesired characteristic of the method is that background pixels that are shadowed by the presence of a new element in the scene are also classified as belonging to the scene projection's foreground. Furthermore, foreground pixels which were highlighted because of surface reflections when the background model was generated, can also be classified as foreground pixels if the light ray that originated the highlight is blocked by some foreground element. For simplicity, we call these two phenomena natural and blocking shadows, respectively.

Foreground Detection with Natural and Blocking Shadow Modelling

A possible solution to this undesired behavior consists in modelling natural and blocking shadowed zones using a model similar to that used for background pixels. Thus, the proposed model for both of them is a scaling of the measured color when none of these two phenomena occurs. As a result, the probabilistic models for a background pixel's natural and blocking shadowed states consist in two gaussians with means and variances proportional to those of the corresponding basic background model. This approach for modelling shadows has already been used in [START_REF] Wang | A probabilistic approach for foreground and shadow segmentation in monocular image sequences[END_REF]. Here, we generalize it for the blocking shadow case. Let x p be a {R, G, B} pixel in an image and µ p and σ 2 p the mean and variance of its associated background model. We model natural background shadowing as a factor scale α ν multiplying the original background color. Hence, the resulting model is a gaussian variable with mean α ν ×µ p and variance α 2 ν ×σ 2 p . Equivalently, the blocking shadow background model becomes a gaussian varaible with α β ×µ p for the mean and α 2 β × σ 2 p for the variance. Intuitively, it is clear that α β must be smaller than α ν .

The resulting expression for the pixel's foreground probability depends on three priors (foreground, and natural and blocking shadowed background probabilities). A uniform random variable modelling foreground pixels, a normal or Gaussian model for each background pixel and two scaled versions of each background model for both natural and blocking shadowed background modelling are used to complete the pixel's foreground probability model:

p(f |x p ) = p(x p |f )p(f ) p(x p |b)p(b) + p(x p |ν)p(ν) + p(x p |β)p(β) + p(x p |f )p(f ) , (1)
with p(x p |f ) a uniform pdf and p(x p |b), p(x p |ν) and p(x p |β) gaussian probability density functions for background, natural shadowed background and blocking shadowed background respectively. All the pdfs are in {R, G, B} space. This improvement of the 2D foreground detection procedure provides better background detection and, consequently, more reliable probability maps, although it does not constitute our main contribution.

Visual Hull Computation with A-priori Knowledge

The proposed analysis method relies on the progressive 3D scanning procedure previously introduced, an idea similar to that of octree-based analysis for voxels [START_REF] Meagher | Geometric modeling using octree encoding[END_REF], but adapted to the available image settings and their associated calibration data.

The aim of the progressive scanning is to simplify the analysis in those areas where coarse resolutions are sufficient, either because of the emptyness of the scene in the feature space we are considering or because of the completeness of the observations of the features. On the opposite case, finer resolutions are used to scan space in those regions where the features measured from the available images do not show uniformity.

Consistency Test

The consistency test is responsible of marking a 3D region (like a voxel, in the case of regular voxelization) as belonging to the scene's foreground or background. In this case, the available data are groups of pixels in each camera view, with their associated pixel foreground probabilities.

As an extension to the 2D foreground segmentation procedure, we also propose a Bayesian formulation in order to describe the occupancy of a 3D region. Let X c be the group of pixels containing the projection of an analyzed 3D region onto a camera c, then:

p(f 3D |X 1 , ..., X N C ) = p(X 1 , ..., X N C |f 3D )p(f 3D ) p(X 1 , ..., X N C ) , (2)
with

p(X 1 , ..., X N C ) = p(X 1 , ..., X N C |f 3D )p(f 3D ) + p(X 1 , ..., X N C |b 3D )(p(b 3D )), (3)
where p(f 3D ) and p(b 3D ) stand for the 3D foreground probability and the 3D background probability, respectively, and N C stands for the number of cameras. As a result, those 3D regions where Eq. 2 lies above 0.5 are marked as belonging to the Visual Hull of the scene.

3D Probability Model. The 3D foreground conditional probability is modelled as a function of the actual measured data: the cameras' foreground probabilities. We assume independence between the assigned probabilites in each view despite the foreground objects are the same in all views. This is reasonable since the foreground detection step in a view is not influenced the results of the foreground detection in other views. As a result, we model the foreground conditional probability of a 3D region as the product of each of the cameras' projection models:

p(X 1 , ..., X N C |f 3D ) = N C c=1 p(X c |f 3D ). (4)
Equivalently, for the 3D background conditional probability:

p(X 1 , ..., X N C |b 3D ) = N C c=1 p(X c |b 3D ). (5)
Camera Projection Model. We define the probabilistic model of the projection of a scanned 3D region on a camera image as the average foreground probability of the pixels lying inside of the projection. The 3D probabilities, conditioned to 3D background and foreground, result in

p(X c |f 3D ) = 1 N p Np p=1 max (p(f |x p ), ǫ) , ǫ ≪ 1 ( 6
)
for 3D foreground, and

p(X c |b 3D ) = 1 N p Np p=1 max ((1 -p(f |x p )) , ǫ) , ǫ ≪ 1 (7)
for 3D background, with N p as the number of pixels lying inside of the projection of the 3D region onto the probability map, in general different for each camera c. The term ǫ is introduced in order to balance the effect of erroneus 2D segmentations as background when computing 3D foreground conditional probabilities.

Experiments and Results

In this section we test shape-from-probability-maps and compare it to standard shape-from-silhouette methods in two different configurations. In the first one, we use regular voxelization with SPOT as the projection test, whereas in the second one image-adapted 3D scanning is used.

Performance Metrics

In order to estimate the actual performance from indirect measurements in 2D projections, we propose to evaluate the system employing verification measures commonly used in information retrieval and also curves derived from these measures, obtained from the projection of the reconstructed Visual Hulls onto the original points of view of the captured images.

-Recall measures the ability of the method to detect a high number of foreground regions:

Recall = #correct foreground #correct foreground + #wrong background (8)
-Precision measures the ability of the system to detect accurately the actual foreground regions:

Precision = #correct foreground #correct foreground + #wrong foreground (9)
-F-measure, also known as the harmonic mean of the two previous quantities, measures the combination of Precision and Recall and gives a more reliable measure of the performance of the system: F-measure = 2 × Recall × Precision Recall + Precision (10)

Data Preparation

The data set was chosen to allow quantitatively measuring the performance by using the proposed metric.

Testing Data. The testing data is composed by ten sets of five images captured in a smart room. Four of the cameras are installed at the corners of the room and the fifth is configured as a zenithal camera, with a focal length shorter than that in the corner cameras. For all the cameras the calibration parameters were obtained as in [START_REF] Bouguet | Camera calibration toolbox for matlab[END_REF], and the distortion parameters were used to obtain undistorted versions of the captured images. In Figure 1, both one of the captured images and its undistorted version are shown.

Groundtruth. The groundtruth is composed by ten sets of five binary silhouettes obtained by manual segmentation of the undistorted images. In Figure 5 (d), the groundtruth silhouette for one of the undistorted images is shown. Camera's Background Models Generation. In addition, 200 frames from each camera were captured with the room completely empty. These training frames can be used to model each camera's background. The parameters that we used for the model generation were a learning rate ρ = 0.01 and an initial variance σ 2 = 20. This initial variance equals the minimum variance accepted for modelling each pixel. This way we ensure that the resulting Gaussian model will have a slightly smoother slope than the one we would obtain by not setting a minimum variance when background pixels show little fluctuations between frames.

The prior probabilities have been arbitrarily assigned in order to be able to obtain reasonably accurate silhouettes, while keeping a realistic scenario including typical artifacts (misses and false detections) that appear when segmenting images with a method based on each pixels' background models. The chosen default values for these priors are p(f ) = 0.2 for the foreground probability, p(ν) = 0.15 for the natural shadow probability and p(β) = 0.15 for the blocking shadow probability. The resulting background prior is 1-p(f )-p(ν)-p(β) = 0.5. The chosen scaling factors for modelling natural and blocking shadowed background are α ν = 0.8 and α β = 0.65, respectively.

Shape from Silhouette

In order to compute the Visual Hull with this method, either with regular or image-adapted voxelization, it is necessary to provide binary images (silhouettes) marking each pixel of the captured projections of the 3D scene as belonging to foreground or background. We decide for thresholding the foreground probability maps with a confidence threshold equal to 0.5. After this step, we obtain 2D silhouettes showing both misses and false detections that can be used for computing the foreground scene's Visual Hull. A 2D silhouette obtained by this method is shown in Figure 2. For a fair comparison (actually slightly favorable to regular voxelization), an intra-voxel distance of 1 cm is chosen as equivalent to the maximal resolution (3 × 3 pixels) used in the image-adapted voxelization algorithm.

Regular Voxelization with SPOT Projection Test. The best performance for this method with the given configuration is obtained with a number of tests per projection equal to 20, with a false alarm rate equal to 0.1, a miss rate also equal to 0.1 and a foreground probability equal to 0.01. From these values, SPOT's optimal threshold was obtained as specified by its author. The results of this best configuration, averaged over 10 scenes with 5 images each, are shown in Table 1. The reconstruction of the Visual Hull for one of the scenes from a virtual camera position is shown in Figure 3. Note that severe errors in 2D foreground segmentation cause splits in the reconstructed volumes.

Image-Adapted Voxelization. In this case, the projection test consists in checking if any of the pixels lying in the projection of the analyzed 3D region belongs to the silhouette's foreground, once the finest resolution of 3 × 3 pixels has been set. As a consistency test, when in each of the available images the projection test is positive, the 3D region is marked as belonging to the Visual Hull. The results of this configuration, again averaged over the same 10 scenes with 5 images each, are shown in Table 2. As it can be seen, this method slightly improves the performance of the previous one, thanks to the more balanced usage of the available data.

Shape from Foreground Probability Maps

We have performed several runs, with different 3D foreground probability priors, in order to characterize in a higher level of detail the new method with its Precision-Recall curve. In order to better depict the obtained results, they are grouped in Figure 4.

We also include the averaged results for the several runs of the probabilistic method parameterized with different priors. They are listed in Table 3. As it can is obviously too small to describe correctly the scene's actual 3D foreground probability.

In Figure 5, the projections of one of the reconstructed scenes onto one of the available cameras are shown for shape-from-silhouette with regular voxelization and SPOT projection test, shape-from-silhouette with image-adapted voxelization and shape-from-probability-maps with 3D foreground prior equal to 0.00005 (our best run). Please note that we have not applied contour smoothing in our reconstructions. As it can be seen, shape-from-probability-maps is able to reconstruct the right arm of the men at the left side of the images more accurately then the other methods.

Limits. As it can be seen, the maximum Recall for shape-from-probabilitymaps, obtained when 3D foreground priors are high, is clearly above the maximum achievable Precision, obtained when priors are low. This is consistent with the fact that the finest level of detail of the image-adapted voxelization scanning procedure is set to 3 × 3 pixels. This condition introduces some additional foreground pixels when projecting a 3D region which is in contact with the Visual Hull's surface onto the original cameras' points of view. Nevertheless, as the results with image-adapted and regular voxelization show, the performance of this 3D sampling method is superior to that of regular voxelization.

Conclusions

An alternative method to compute the Visual Hull of a 3D scene captured by several cameras, shape-from-probability-maps with image-adapted voxelization, has been presented. Its main features are:

1. the use of a 3D scanning algorithm better adapted to the available information than the commonly used regular voxelization, and 2. the introduction of a Bayesian formulation that avoids early image segmentation in favor of considering pixels as a foreground probability values Compared to the standard shape-from-silhouette algorithm with voxelization adapted to images, the input data for the proposed method offer more information by not thresholding foreground detection results, delaying this decision to allow for contributions from the other views before the foreground-background classification. We have to point out that the probability maps obtained from 2D foreground detection are rather binary, allowing for just a marginal gain when comparing our new method with shape-from-silhouette. Thus, the next steps in order to exploit the proposed Bayesian framework should be in the direction of enhancing the foreground detection. Comparison with regular voxelization shows that the performance of image-adapted 3D analysis also goes beyond that of regular voxelization, due to the more balanced usage of available image data, which does not introduce an additional arbitrary sampling in 3D space. To sum up, as seen in the results provided, shape-from-probability-maps delivers a Recall-Precision curve above the performance of any of the other state-of-the-art methods included in the comparison. This work has been partially supported by the Spanish Ministerio de Educación y Ciencia, under project TEC2007-66858/TCM, and by the Spanish Administration agency CDTI, under project CENIT-VISION 2007-1007. The authors would also like to thank Marcel Alcoverro for his help with parameters optimization in the tests with SPOT projection test for regular voxelization.

Fig. 1 .

 1 Fig. 1. A captured image as seen from one of the cameras (a) and its corresponding undistorted version (b)

Fig. 2 .

 2 Fig. 2. Binary segmented silhouette for one of the captured images

Fig. 3 .

 3 Fig. 3. Visual Hull reconstructed by shape-from-silhouette with regular voxelization and SPOT projection test

Fig. 5 .

 5 Fig. 5. Projections of one of the reconstructed scenes onto one of the available cameras for shape-from-silhouette with regular voxelization and SPOT projection test (a), shape-from-silhouette with image-adapted voxelization (b), and shape-fromprobability-maps with p(f3D) = 0.00005 (c). The corresponding groundtruth silhouette (d) is also shown

Table 1 .

 1 SfS with SPOT projection test for regular voxelization's average performance

	Precision Recall F-measure
	87.34% 81.01% 83.61%

Table 2 .

 2 SfS for image-adapted voxelization's average performance Precision Recall F-measure 84.87% 84.37% 84.25%

Table 3 .

 3 Shape from Foreground Probability Maps for image-adapted voxelization's average performance Precision-Recall curve for different values of the prior 3D foreground probability p(f3D). Results for SfS with image-adapted voxelization (deterministic) and SfS with regular voxelization and SPOT projection test (spot) are also used for comparison

	Prior p(f3D) Precision Recall F-measure
	0.0005	75.32% 97.23% 84.71%
	0.0004	75.51% 97.17% 84.81%
	0.0003	75.83% 97.06% 84.97%
	0.0002	81.45% 88.33% 84.32%
	0.0001	85.20% 85.15% 84.79%
	0.00005	87.51% 83.51% 85.08%
	0.00003	89.08% 82.06% 85.04%
	0.00001	91.16% 78.68% 84.05%
	be seen, the performance of the new method is consistently above the results
	obtained with any of the two shape from silhouette methods included in the
	comparison. An exception appears with the last prior included, 0.00001, which


Eric Sommerlade

Ian Reid

Cooperative Surveillance of Multiple Targets using Mutual Information

This work presents a method to control multiple, but diverse pan-tilt-zoom cameras which are sharing overlapping views of the same spatial location for the purpose of observation of this scene. We cast this control input selection problem in an information-theoretic framework, where we maximise the expected mutual information gain in the scene model with respect to the observation parameters. The scene model yielding this information comprises several dynamic targets, augmented by one which has not yet been detected. The information content of the former is supplied directly by the uncertainties computed using a Sequential Kalman Filter tracker for the observed targets, while the undetected is modelled using a Poisson process for every element of a common ground plane. Together these yield an informationtheoretic utility for each parameter setting for each camera, triggering collaborative explorative behaviour of the system. Overall this yields a framework in which heterogeneous active camera types can be integrated cleanly and consistently, obviating the need for a wide-angle supervisor camera or other artificial restrictions on the camera parameter settings.

Introduction

Successful observation of a scene of multiple moving targets is pervasive in any surveillance scenario, and its success a prerequisite for further interpretation of the gathered data, e.g. classification. In an active vision setting, this success highly depends on the sensible selection of control actions for the monitoring process. In many application areas -such as sport events, surveillance, and patient monitoring -camera control can be seen as a simple example for arbitration of different interests. One interest is to obtain the maximum resolution of a target to facilitate classification. Examples are identification of people, closeups to disambiguate specific gestures, or properties such as view direction. The second interest is to minimise the risk of losing a target once it has been detected. Here zoom is an important factor. When a target remains static, the zoom can be safely increased. Once a target starts moving, small mistakes in following the object can result in a loss of sight. For example, following an object with a fixed zoom telescope gets harder the more erratic this object moves. A third interest is an ongoing observation of the environment, in order to minimise the risk of not recording events of importance. Previous authors have put forward solutions to control a specific setup of cameras such as a fixed supervisor camera directing multiple active sensors, and special rules have been found to schedule cameras to observe a number of people. In contrast, our work focusses on a generic setup of heterogeneous active cameras observing a number of targets, making use of different camera types in a homogeneous manner.

For a set of dynamic targets, the increase of accuracy by a zoom process has to be arbitrated with the chances of losing lock and missing targets of interest. The latter -exploration of the scene -is explicitly addressed by a dynamical model of the scene, modelling the appearances of actors in the environment. A yet undetected target is integrated into a decision process by the information to be potentially gained upon detection. We have presented a method for a single camera in our previous work [START_REF] Sommerlade | Information theoretic active scene exploration[END_REF]. Here, we extend this work towards multiple cameras, and argue that mutual information is the right metric to handle more than one target. We furthermore correctly include the performance of the detector in the acquisition process.

The remainder of this paper is structured as follows. Related work is discussed in the next section, followed by a quick review of the decision process for camera parameter selection. The Sequential Kalman Filter and its resulting mutual information gain are discussed in section 3.1, and section 3.2 gives examples how the approach responds sensibly in situations requiring camera hand-off. In section 3.3, we explain how the observation likelihood of a yet undetected target adds an exploratory behaviour to the parameter selection process. We then present an experiment on a stereotypical situation.

Related Work

This work touches several areas. One is finding the optimal zoom setting of a camera. Both Tordoff and Murray [START_REF] Tordoff | A method of reactive zoom control from uncertainty in tracking[END_REF] as well as Denzler et al. [START_REF] Denzler | Information theoretic focal length selection for real-time active 3-d object tracking[END_REF] use probabilistic reasoning for camera zoom control, effectively minimising the chance of losing the target while maximising zoom level at the same time, but address only a single camera. In particular, Denzler et al. [START_REF] Denzler | Information theoretic focal length selection for real-time active 3-d object tracking[END_REF] were the first to minimise entropy for the control of a pan, tilt and zoom camera. Deutsch et al. [START_REF] Deutsch | Multi-step multi-camera view planning for real-time visual object tracking[END_REF] extend Denzler's work towards multiple cameras, but also consider a single target only. Mutual information and information theoretic measures are also used for view planning in classification tasks [START_REF] Paletta | Active object recognition by view integration and reinforcement learning[END_REF][START_REF] Denzler | Information theoretic sensor data selection for active object recognition and state estimation[END_REF] in the presence of a single, static target.

When there are more targets to be observed than sensors available, a decision has to be made which target to observe with which sensor. This camera assignment problem is phrased as a dynamic optimisation problem by Bagdanov et al. [START_REF] Bagdanov | Acquisition of high-resolution images through on-line saccade sequence planning[END_REF]; specifically Isler et al. [START_REF] Isler | Target tracking with distributed sensors: The focus of attention problem[END_REF] address the computational issue of assignment of a single target to a single camera. Zhang and Qi [START_REF] Zhang | Active and dynamic information fusion for multisensor systems with dynamic bayesian networks[END_REF] use mutual information from a Dynamic Bayesian Network to decide upon use of different sensors for a specific target type.

This touches upon the area of camera scheduling, which has been picked up in some recent works by the vision community [START_REF] Bagdanov | Acquisition of high-resolution images through on-line saccade sequence planning[END_REF][START_REF] Qureshi | Surveillance in virtual reality: System design and multi-camera control[END_REF][START_REF] Hampapur | Face cataloger: Multi-scale imaging for relating identity to location[END_REF]. These all use at least one specific supervisor camera and specific, hand-crafted rules to control the individual sensors, for example choosing the zoom setting via geometric reasoning. Another area this work touches is long term surveillance of a scene, where efficient placement of cameras is vital. Here, the installation cost is minimised with respect to a maximisation of the quality of the recorded data [START_REF] Yao | Sensor planning for automated and persistent object tracking with multiple cameras[END_REF][START_REF] Erdem | Automated camera layout to satisfy task-specific and floorplan-specific coverage requirements[END_REF]. Since this is a long term goal, short term behaviour by active parameter changes are usually not addressed, and the optimum is found for a static environment with temporal average of target behaviour. An exception is Bodor et al. [START_REF] Bodor | Optimal camera placement for automated surveillance tasks[END_REF], who use their method to place a robot for optimal surveillance; still, no zoom parameters are adjusted.

Control methods benefit from a feedback of learned scene parameters, in the surveillance domain this is usually pedestrian activity [START_REF] Bodor | Optimal camera placement for automated surveillance tasks[END_REF][START_REF] Migdal | Moving object segmentation using super-resolution background models[END_REF][START_REF] Davis | An adaptive focus-of-attention model for video surveillance and monitoring[END_REF] or saliency [START_REF] Gould | Peripheral-foveal vision for real-time object recognition and tracking in video[END_REF]. In the latter work a sophisticated perceptual model is learned and used to drive the focus of attention. Our work provides a facility to integrate such acquired knowledge by using an appearance rate of objects in the supervised area. This information can then be used to drive other observation priorities than target tracking. Elfes [START_REF] Elfes | Occupancy Grids; A Probabilistic Framework for Robot Perception and Navigation[END_REF] pioneered such dynamic certainty grids to model information obtained about scene areas. This approach has also been used by Bourgault et al. [START_REF] Bourgault | Information based adaptive robotic exploration[END_REF], where different objectives in robotic exploration are arbitrated in likewise fashion. Grocholsky [START_REF] Grocholsky | Information-theoretic control of multiple sensor platforms[END_REF] uses mutual information in an optimal control setting with moving sensors and static targets.

Multi-target tracking in general is a very active field of research [START_REF] Black | Multi view image surveillance and tracking[END_REF][START_REF] Khan | A multiview approach to tracking people in crowded scenes using a planar homography constraint[END_REF][START_REF] Yang | Game-theoretic multiple target tracking[END_REF][START_REF] Mittal | M2tracker: A multi-view approach to segmenting and tracking people in a cluttered scene[END_REF]. A good overview of the mathematical prerequisites for multi camera tracking can be found in Calderara et al. [START_REF] Calderara | Hecol: Homography and epipolar-based consistent labeling for outdoor park surveillance[END_REF]. Recently, Fleuret et al. [START_REF] Fleuret | Multi-camera people tracking with a probabilistic occupancy map[END_REF] have presented a sophisticated tracking method using probabilistic occupancy grids. However, these works are usually constrained to static cameras, whereas we focus on the control of active ones.

Camera Parameter Selection

One goal of our system is to track objects and obtain images at a high resolution to aid in processing steps, e.g. identification or classification. For this, we desire minimal uncertainty in the location of the objects. This evokes the problems of target assignment to each camera, and parameter selection for each camera. The first summarises which camera is to be used to track a number of targets. The second problem addresses the choice of zoom and other parameter for each camera, which are bounded by the uncertainty of the object's motion, as well as its spatial extent.

We phrase these problems in a coherent information-theoretic manner. The procedure is as follows: Before making an observation at time k, we choose the best parameter a k for the observation. The parameter a k summarises the different settings for the observation process, i.e. assignment of targets to cameras, and the respective pan, tilt and zoom settings. Among all choices, this parameter will maximally reduce the expected uncertainty in a given probability distribution of the true state x k . Applying the chosen parameter yields an observation o k which is finally used to update the distribution p(x k ).

A measure for uncertainty of the state is Shannon entropy, and since the decision for an action has to be made before observing the target, the appropriate value is the expected conditional entropy. This quantifies the average uncertainty in state x when an observation o is made, and is independent of any actual observation:

Ĥa (x|o) = - ∞ -∞ p a (x, o) log(p a (x|o)) dx do (1)
When looking at a set of multiple targets X = x 1 . . . x n , the currently known information is of importance. In a surveillance context, following a target at highest zoom may lead to other targets passing through the wider scene unnoticed. This evokes the use of expected mutual information gain, which measures the information potentially gained by making an observation, and is simply the difference between the expected uncertainty and the current uncertainty:

I(x; o) = H(x) -Ĥ(x|o) = -E log p(x|o) log p(x) (2)
If the random variables x and o are independent, the information about x potentially gained from an observation is zero. The resulting distributions p a (x, o) and p a (x|o) depend on the chosen action a. The optimal action -the best parameter settings for all sensors -is finally obtained from maximising the expected mutual information gain

a * = arg max a I a (x; o) (3)
over all possible actions. This can be interpreted as choosing the action which makes the distribution p(x|o) more peaked relative to p(x).

The parameter a hides the intricacies of the target assignment problem. Here, a decision has to be made about which sensor is to observe a subset of targets. For example, one camera can focus onto a single target, another one is directed towards a group of targets. When assigning targets to sensors in such a way, resulting measurements are expected. Maximising the expected gains from these measurements by a full search quickly becomes prohibitive, since the number of options scales with the factorial of the number of targets and cameras.

When instead maximising over all different parameter settings, the advantage is that no such decision has to be made a priori. The benefit of looking at a set of targets, and ignoring others is given by the objective function. For example, directing a camera away from a target does not gain any information about this particular one, and -as can be seen in equation 2 -an observation will never be detrimental to the mutual information, no matter how unlikely the chance of making it, we can therefore assign all targets to all cameras. When using mutual information to weight the importance of a single target in the observation process, the information already gathered about this target is incorporated. It is less sensible to follow a target at highest zoom level when this also risks the loss of another target.

Tracking with multiple cameras

We represent the motion of a target in the scene in ground plane coordinates, facilitating integration of measurements from different cameras. Furthermore, we assume that these cameras are calibrated and have a negligible positional error. Whereas this is a strong requirement compared to other methods [START_REF] Calderara | Hecol: Homography and epipolar-based consistent labeling for outdoor park surveillance[END_REF] the camera calibration needs to be obtained for every parameter setting of the camera -the benefit is the ease of data fusion in a common ground plane.

For tracking we use a sequential Kalman filter, which is a simple extension of the standard Kalman filter. Its derivation and resulting equations can be found in textbooks, e.g. [START_REF] Bar-Shalom | Tracking and data association[END_REF], so that we can confine ourselves to the notation required. The sequential Kalman filter makes a single prediction step, taking target state x+ k-1 and covariance matrix P+ k-1 to a predicted position xk and Pk , taking into account the uncertainty of the motion model. This prediction is updated once for every observation made by each camera, which is valid if the measurement noise of the cameras is uncorrelated. This is a common assumption [START_REF] Deutsch | Multi-step multi-camera view planning for real-time visual object tracking[END_REF][START_REF] Bar-Shalom | Tracking and data association[END_REF]. For each update, the observation matrix H is linearised anew at the estimate produced by the incorporation of the previous observation.

The resulting covariance of a single target observed by a set C = {c 1 , . . . , c n } of cameras, which can be a subset of all cameras C * , is thus the product of all Kalman filter gains:

P+ k = c∈C (I -K c H c ) P- k . (4)
Since the differential entropy of a Gaussian distributed state vector x ∈ R n with covariance matrix P is

H(x) = n 2 + 1 2 log((2π) n |P|), (5)
the conditional entropy of the sequential Kalman filter with covariance given in equation 4 and observations {o}, reduces to

Ĥa (x|{o}) = n 2 (1 + log(2π) + c∈C log |I -K c H c | + log | P- k |). (6)
The resulting covariance depends on the order of the updates, and the probability of actually making an observation with each sensor. The former is resolved by assuming a sufficiently stable linearisation point, and the latter is obtained from the overall chance of making an observation within the field of view Ω c a camera can supervise for a given setting a:

w(a) = Ωc p a (o) do (7)
Full evaluation of equation 6 for every possible combination of expected target observability is of exponential complexity in the number of cameras. We therefore approximate (see [START_REF] Deutsch | Multi-step multi-camera view planning for real-time visual object tracking[END_REF]) a single covariance matrix P+ k,c resulting from a single observation process of a single camera by:

P+ k,c = w c (a)(I -K c H c ) P- k,c + (1 -w c (a)) P- k,c . (8)
The mutual information for all targets is then

I a (x; o) = H(x) -Ĥa (x|o) = -n/2 c∈C * log |I -w c (a)K c H c |. (9)

Examples

To highlight the influence of this objective function on the parameter selection, we created some simple artificial scenes. Here, the target is simply moving on the ground plane and has no spatial extent, i.e. the observation yields a point in the image plane. The target moves on a predefined spline curve, which model second order behaviour, whereas the motion model of the Kalman filter is linear.

In the first scene setup, a single moving target is observed by two fixedzoom cameras, both looking in the same direction. The first one is fixed, and the second one can move along a part of the x-axis to follow the target, but otherwise has the same observation parameters as the first. A top view of this scene is shown in figure 1(a). Given that the cameras are sufficiently far away from the target, the choice of the camera position will not have an influence on the linearised observation model and in turn the target's covariance matrix; instead, it will have an impact on the visibility term in equation 7 only. Thus the maximal mutual information for a single target will simply maximise the probability of making an observation, which is given in equation 7. The contour plot 1(b) shows this visibility term for varying positions at different time steps. It forms a plateau, where the chance of making an observation is 1, which falls off to 0 according to the predicted positional uncertainty and observation noise. The selected camera position is anywhere on this plateau, which follows the expected target movement.

In the second artificial scene, the second camera is now also allowed to change its zoom factor between 1 and 12. This setup and the resulting camera positions are shown in figure 2(a). The zoom setting influences the Jacobian of the observation model, and therefore the expected covariance of the target, as in equation 6. The objective function for these parameters is shown as a contour plot for three frames in figure 2(b). The initial zoom value is limited because of the high uncertainty after initialisation of the Kalman filter. Later, the tracking is good enough to guarantee successful tracking at highest zoom level. Finally, the target left the field of view of the first camera and the second one has to zoom out to maintain successful tracking. This is also portrayed in figure 2(c), where the maximum mutual information is shown for the given observation parameters. For a single target, the resulting camera settings are the ones which allow for maximum visibility at highest zoom. As another example, showing collaborative tracking of cameras with this formulation, we set up an artificial scene motivated by a real world data set with multiple cameras and manually annotated ground truth 1 .

We model the state of the targets as a 3-dimensional bounding box moving on a ground plane with linear dynamics. The observation model yields the two-dimensional bounding box of the projected vertices of the target. The only parameters varied here are the zoom settings for both cameras. Figure 3 shows the tracking of a pedestrian in said scenario. The rectangles mark the bounding box of the actor, whose path is artificially occluded in the centre. At the start of the sequence, the mutual information gain for the second camera is close to zero, because the view to the target is occluded. As long as the other camera observes the target, the position estimate is accurate enough to be sure that the target is still blocked from view. Once the target is lost by the first camera, the mutual information gain raises since the uncertainty of the target's position raises -hence an observation might be made in the area surrounding the blocked view. This behaviour is shown in the close-up of the development of the mutual information in 3 (c). This behaviour is sensible in that there is no other objective for the first first camera. As soon as another target of interest is available, the camera will focus on this. We will give an example for this in the next section.

Searching for unobserved targets

When zooming onto a single target or a subset of targets, a wider field of view is sacrificed, which increases the risk of missing other, possibly more interesting targets or events in the scene. We thus add the chance of the appearance of another, yet undetected target to our scene model. New targets appear regularly, but unpredictably. The absolute times of two appearances are independent random variables -the number of appearances before an occurrence is independent of the number of the following ones. Similar to [START_REF] Sommerlade | Information theoretic active scene exploration[END_REF], we assume that at a given position y, the waiting time T until the next appearance of an object at location has an exponential distribution with the appearance rate λ(y). This yields the prior p(e, y) on actual target existence at this position.

In many recent tracking systems, a tracking method is initiated after prior detection. These detection methods can be relatively simple and computationally cheap, such as background subtraction, or harness the robustness of detectors for specific object classes. Once an object is detected, a tracking method is instantiated on this object, e.g. a Kalman filter. We assume that a detector can potentially find an object if it is in the field of view Ω c of a single camera. The detection performance, i.e. the chance of making a detection d, can be described by a likelihood p(d|e) for a prior target existence e.

We now discretise the ground plane into N locations y. The expected information gain at each location is then

I at (e|y; d) = H(e|y) -Ĥ(e|d, y) = H(e) + e,d
p at (e, d|y) log p at (e|d, y) [START_REF] Qureshi | Surveillance in virtual reality: System design and multi-camera control[END_REF] The joint probability p at (e, d|y) is trivially obtained via Bayes from the detection performance and the chance that an object has appeared at this location. Assuming that detections in all cameras and probabilities of appearance at all locations y are independent, the expected information gain ÎN,at for given parameter settings is the sum of all information gains for all N locations and all cameras.

Experiments

To compare active vision algorithms in a fair manner, all methods should be presented with the same input. When this video data portraits human actors, such a live evaluation becomes difficult and straining due to continuous repetition of the same tasks. We therefore use pre-recorded video for our evaluation. If the resolution is high enough to support a "virtual zoom" approach, where the image is down-sampled and cropped to a desired field of view, object detectors or blob trackers can be run on the down-sampled image. Unfortunately, few of such data sets exist -either the annotation is very limited, or the data is at a very low resolution. One of these is the PETS 2001 data mentioned in section 3.1, where only one sequence is annotated. To compare the tracking performance with respect to ground truth, we use these data sets in the following manner. For every annotated target in the scenes, we add Gaussian noise of 1 pixel to the labelled bounding boxes. Detections are randomly generated upon the first entrance of an object into the field of view of a camera. Each detection is assigned a Kalman filter, which is used to obtain the uncertainty of the tracking as described in section 3.1. The Kalman filter uses an observation noise of 1 pixel, and process noise of 0.05 units (both for 1σ). A track is lost if for more than 10 frames no observation has been made, the target leaves the maximum field of view, or the expected measurement does not overlap with the actual measurement. This approach finally removes other sources of error in the evaluation, e.g. from data association.

We measure the performance of the method with two metrics. The first is the relative size increase of the observed targets, compared to the ground truth values without zoom control. We justify this choice by reported increase of identification capabilities of state of the art systems [START_REF] Phillips | FRVT 2006 and ICE 2006 large-scale results[END_REF]. When evaluating the observed area of multiple cameras, we only use the maximum area of the target in each of the The first experiment compares the performance of the proposed tracking method for different target assignments and data fusion methods. Either all or a single targets are assigned to each camera, and these targets are then tracked by a common Kalman filter, or one for each camera. We furthermore compared this control rule with one that favours zooming out when the mutual information gain to be expected is very small (as described in section 3.1).

Here, the reward for zooming out was set to a constant value, to emphasise the behaviour resulting from the Kalman filter control rule. The only parameter which was optimised is the zoom setting of the cameras, which was allowed to vary between 1 and 9. Figure 5 shows the results of an evaluation on the PETS data set for a varying number of actors, approximating an increase in scene complexity. Whereas the average increase in observation area of all targets is similar, the approach of assigning a single target to a camera without sharing the information among the cameras results in a delay in acquiring new targets.

Conclusion

In this paper we have introduced how mutual information can be used to control multiple cameras to track multiple targets, and how entropy based measures can be used to combine different objectives, such as exploration vs. tracking. Whereas here only two objectives have been investigated and the numerical evaluation is far from complete, we would like to point out the unifying approach and extensibility of this method.

Whereas the resulting evaluations of the work presented have been carried out on synthetic data only, the setup has been motivated by publicly available data sets depicting "real world" scenes. Nonetheless, we are currently working on the integration of this method into an active vision system.

Our future work finally focusses on an integration of the data association into the mutual information term, efficient ways to reduce the action space of this optimisation problem, and a step away from the omniscient, or "super-Bayesian", approach of a state model in every camera, facilitating control of widely distributed camera networks.

Figure 1 .Figure 2 .

 12 Figure 1. (a): Top view of the simple scene setup. The first camera (red) remains fixed. The second camera (blue) follows the target, maximising expected visibility. Lighter tones correspond to earlier frames. (b):The resulting visibility term for a number of camera positions at every step, given for a likelihood of 0.1 and 0.9. The predicted target position is drawn in red/bold.

Figure 3 .

 3 Figure 3. Trajectories of first actor of the PETS 2001 data set with superimposed ground plane in left and right view. Only every 10th frame is shown. Plot 3(e) shows the likelihood of making an observation for both cameras (left:red, green/light: right). The resulting zoom setting for the two cameras is shown in plot 3(f). See text for details.

Figure 4 .

 4 Figure 4. Discretised coordinates for actor appearance modelling in ground plane view of the PETS data set, showing the priors after 5 zoom steps into the centre of the image by each of the cameras.

  avg. relative observation size all targets, common filter all targets, single filter 1 target, common filter 1 target, single filter (b) relative increase in maximum observation area

Figure 5 .

 5 Figure 5. Performance on data set for varying scene complexity recorded images. The second metric is the average delay of the initial acquisition of a target.The first experiment compares the performance of the proposed tracking method for different target assignments and data fusion methods. Either all or a single targets are assigned to each camera, and these targets are then tracked by a common Kalman filter, or one for each camera. We furthermore compared this control rule with one that favours zooming out when the mutual information gain to be expected is very small (as described in section 3.1).Here, the reward for zooming out was set to a constant value, to emphasise the behaviour resulting from the Kalman filter control rule. The only parameter which was optimised is the zoom setting of the cameras, which was allowed to vary between 1 and 9. Figure5shows the results of an evaluation on the PETS data set for a varying number of actors, approximating an increase in scene complexity. Whereas the average increase in observation area of all targets is similar, the approach of assigning a single target to a camera without sharing the information among the cameras results in a delay in acquiring new targets.

PETS

data set: http://pets2001.visualsurveillance.org