Damien Kelly
email: kelly16@tcd.ie

François Pitié
email: fpitie@tcd.ie

Anil Kokaram
email: anil.kokaram@tcd.ie

A Comparative Error Analysis of Audio-Visual Source Localization

This paper examines the accuracy of audio-video based localization using multiple cameras and multi-microphones. Covariance mapping theory is used to determine the accuracy of audio and video based localization. Both modalities are compared in terms of their ability to provide accurate location estimates of a moving audio-visual source. Relatively, video is found to be significantly more accurate than audio. The problem of audio-video fusion is also examined. The fusion of audio and video location estimates is applied in the audio domain, the video domain and the positional domain. The accuracy of these three fusion strategies for 3D localization are examined from a theoretical basis. The best localization performance is found when fusion is applied in the positional domain. Fusing audio and video data in the video domain is found to exhibit the worst localization performance. This analysis is confirmed by measuring the accuracy of each fusion strategy in localizing a moving audio-visual source.

Introduction

There is a current trend in the research community towards the use of multiple modalities in tracking. Particularly, the use of both audio and video in tracking people is gaining significant interest. Literature proposes a variety of Kalman filter based [START_REF] Strobel | Joint Audio-Video Signal Processing for Object Localization and Tracking[END_REF][START_REF] Katsarakis | Polymenakos: 3D Audiovisual Person Tracking using Kalman Filtering and Information Theory[END_REF] and particle filter based [START_REF] Zotkin | Joint Audio Visual Tracking Using Particle Filters[END_REF][START_REF] Checka | Multiple Person and Speaker Tracking with a Particle Filter[END_REF] techniques for joint audio-visual tracking. In general these approaches show improved tracking performance beyond that possible through the use of either modality alone.

This result is mainly due to the complementary nature of the audio and video modalities. The performance of audio-based tracking is dependent on the quality of the received signals at the microphones which is affected by the distance from the speaker to the microphones [START_REF] Gustafsson | Source Localization in Reverberant Environments: Modeling and Statistical Analysis[END_REF], background noise and most significantly distortion due to reverberation [START_REF] Chen | Benesty: A Comparative Study on Time Delay Estimation in Reverberant and Noisy Environments[END_REF]. Such issues do not affect video-based tracking. Similarly, factors affecting video-based tracking such as varying illumination, visual clutter and visual occlusions have no implications on audio-based tracking. In a joint audio-video based system where audio and video fail independently therefore it is reasonable to assume an overall improvement in tracking reliability.

There is conflicting evidence in literature that improved tracking accuracy is achieved through a joint audio-video based approach. Specifically, Strobel et al. report improved tracking reliability, but note no clear improvement in accuracy beyond the best available single-modality position estimate [START_REF] Strobel | Joint Audio-Video Signal Processing for Object Localization and Tracking[END_REF]. The general approach in evaluating the performance of joint audio-video trackers is to determine the tracking performance against some ground truth (eg. [START_REF] Strobel | Joint Audio-Video Signal Processing for Object Localization and Tracking[END_REF][START_REF] Katsarakis | Polymenakos: 3D Audiovisual Person Tracking using Kalman Filtering and Information Theory[END_REF][START_REF] Checka | Multiple Person and Speaker Tracking with a Particle Filter[END_REF]). Tracking performance however is not only dependent on the accuracy of audio and video-based position estimates but also on the employed motion model. Thus a well chosen motion model can result in undue credit being attributed to a multi-modal approach in improving tracking accuracy.

Furthermore, this measure of performance can only give an indication to the expected accuracy of the resulting fused track. It can not be used to determine how well a system is performing in relation to its best possible performance. Currently, little effort has been made in literature to evaluate the expected performance of joint audio-video based tracking systems. This means that such systems are being proposed without any theoretical basis on which to measure performance.

It is informative to examine the performance of a joint audio-video tracking system by examining the localization accuracy in each domain individually. In this way the contribution of both audio and video in improving localization accuracy through a fused estimated can be determined. Literature proposes techniques for predicting the 3D error associated with localization using multiple cameras [START_REF] Olague | Optimal Camera Placement for Accurate Reconstruction[END_REF] and multiple microphones [START_REF] Brandstein | Microphone Array Localization Error Estimation with Application to Sensor Placement[END_REF]. The incorporation of such theory in the analysis of joint audio-video based tracking to date has not been adequately considered. This work aims at addressing this issue.

In this paper covariance mapping theory is used to examine the error of localizing an audio-visual source using multiple microphones and multiple cameras. This mapping theory is used to determine the 3D localization error associated with audio-based localization using time-delay estimation and video-based localization through triangulation from multiple cameras. Given this, a direct comparison is made between the localization accuracy of both modalities in terms of their ability to provide accurate location estimates of a moving audio-visual source.

Covariance mapping is also used to determine a representation of uncertainty on the time delay estimates in the video domain and similarly to determine a representation for uncertainty on pixel measurements in the audio domain. In essence, audio and video localization uncertainty is examined in the positional domain, audio domain and video domain. Maximum likelihood data fusion is applied in these three domains and a resulting 3D fused localization estimate is obtained in each case. The effectiveness of these fusion strategies is examined from a theoretical basis and their ability to provide accurate location estimates of a moving source is evaluated.

In order to make this analysis tractable the general assumption of Gaussian observation noise is assumed on time delay estimates in the audio domain and also in the video domain on pixel measurements obtained from each camera.

Relatively, video-based localization is found to be significantly more accurate than audio-based localization. The contribution of audio data in a fused estimate is also found to vary significantly over the track duration. The analysis of audio and video fusion shows that for the given problem the best localization accuracy is achieved when fusion is applied in the positional domain.

The remaining sections of this paper are organized as follows. Sec. 2 examines the process of uncertainty propagation in a multi-camera multi-microphone configuration. Sec. 3 outlines the experimental setup used in the comparative error analysis and also examines the validity of the uncertainty propagation for this tracking environment. A comparative error analysis of audio-video based localization is made in Sec. 4 and finally, conclusions are presented in Sec. 5.

Uncertainty Mapping

In this analysis we examine the 3D position S ∈ R 3 of an audio-visual source observed indirectly by pixel measurements x = f (S) in the video domain and time delay estimates τ = g(S) in the audio domain. Using these measurements we wish to map their respective covariances Σ x and Σ τ into positional space in order to estimate the associated covariances Σ V

x of a video-based location estimate and Σ A τ of an audio-based location estimate. The covariance of a fused maximum likelihood estimate in the positional domain is then obtained by,

Σ P os S = ((Σ A S ) -1 + (Σ V S ) -1 ) -1 . (1)
Fusion in this application could also considered in two additional domains, the audio domain corresponding to time delays and the video domain corresponding to pixel measurements. This requires the transformation of audio and video measurements to equivalent levels of representation. For instance, for fusion in the video domain an equivalent representation of audio-based measurements and uncertainty must be determined in the image plane. Similarly, for fusion in the audio domain video-based localization measurements and uncertainty must be transformed into an equivalent representation in the audio domain. In the video domain, in addition to the covariance of pixel measurements Σ x the covariance Σ A

x of audio-based 3D localization transformed into the image plane is determined. Likewise, in addition to the covariance of time delay estimates Σ τ the covariance Σ V τ of video-based 3D localization transformed into the audio domain can be determined. Within both the audio domain and video domain then fusion is applied using [START_REF] Strobel | Joint Audio-Video Signal Processing for Object Localization and Tracking[END_REF]. The resulting covariance of the fused estimate is then mapped from each domain into positional space. This enables the covariance Σ V id S of 3D localization through fusion in the video domain and the covariance Σ Aud S of 3D localization through fusion in the audio domain to be evaluated. The relevant mappings considered are illustrated in Fig. 1.

Linear Approximation Mapping

The mapping of covariances between the positional domain, audio domain and video domain can be achieved through a first order Taylor series expansion of the audio and video measurements functions and their inverses. Here the process of mapping the covariance Σ S of the source position S to obtain a corresponding measure of pixel uncertainty Σ x in the video domain is presented. If the measurement function f (S) has a continuous first order derivative then a first order Taylor series expansion of f (S) enables the mean and covariance of x to be approximated. The mean of x is approximated by

E[x] = f (E[S]
) and its covariance Σ x by,

Σ x = ∂f (E[S]) ∂S Σ S ∂f (E[S]) ∂S T . (2)
The inverse mapping of ( 2) is difficult to obtain in cases where the inverse measurement function S = f -1 (x) can not be explicitly defined. For an implicitly defined function F (x, Ŝ) = x-f ( Ŝ) where Ŝ minimizes the criterion function C(x, S) = |F (x, S)| 2 , the first order derivative of the inverse measurement function can be approximated using the implicit functions theorem [START_REF] Faugeras | Three-Dimensional Computer Vision: A Geomteric Viewpoint[END_REF]. This is found to be,

∂f -1 (E[x]) ∂x ≈ - ∂F ∂S † ∂F ∂x . ( 3
)
where † is used to denote the pseudo inverse. The concept in using this mapping of uncertainty between the positional, audio and video domains in illustrated in Fig. 1.

Audio-based Measurement Function

Time-delay based acoustic source localization infers the position of an active source from time delay estimates between spatially separated microphones. These time delays arise due to the spatial separation of the microphones whereby the source signal is received at each microphone at different points in time. The audio measurement function in this case therefore describes the vector of time delays τ associated with the 3D point S. This function is completely described by the positions of the microphones, the sampling frequency and the speed of sound.

Let

m ij = [X ij , Y ij , Z ij ]
T j = 1, 2 denote the positions of the microphones of the ith microphone pair configuration and τ = [τ 1 , .., τ i , ..., τ N ] T be the vector of time-delay estimates for N microphone pairs. For the ith microphone pair, the expected time delay τ i given the source position S may be determined by,

τ i = f s c [((X i1 -X) 2 + (Y i1 -Y ) 2 + (Z i1 -Z) 2 ) 1 2 -((X i2 -X) 2 + (Y i2 -Y ) 2 + (Z i2 -Z) 2 ) 1 2 ] = g i (S). ( 4
)
where f s = 48kHz is the sampling rate and c = 343m/s is the speed of sound.

The time delays referred to therefore are in units of audio samples. Using Fig. 1: The mapping of uncertainty between the audio, video and positional domains through a first order Taylor series expansion of the measurement functions and their inverses. The video measurement function is denoted f (S) and the audio measurement function is denoted g(S). The corresponding inverse measurement functions are denoted f -1 (x) and g -1 (τ ) respectively.

estimates. This linearization of the audio-based measurement function follows in principle the theory of extended Kalman filtering [START_REF] Gehrig | Tracking Multiple Speakers with Probabilistic Data Association Filters[END_REF]. Given (4) a set of implicit functions can be defined, one for each microphone pair as,

G i (τ, Ŝ) = τ -g i ( Ŝ), ( 5
)
where S is the 3D location estimate which minimizes the criterion function

C g (τ, Ŝ) = N i G i (τ, Ŝ) 2 .
Using ( 5) and ( 3) enables uncertainty on time delay estimates to be propagated into the 3D positional domain. This approach to predicting the error region associated with a time-delay based location estimate is not readily available in literature therefore a complete derivation of this result is presented in Appendix A.

Video-based Measurement Function

The video measurement function relates the 3D point S to a vector of pixel measurements x = [x 1 , .., x i , .., x N ] T , where x i = [x i , y i ] T is the 2D pixel measurement corresponding to the ith camera view. This function is dependent on the camera matrices of the multi-camera views and their associated distortion parameters. Assuming that the distortion characteristics are known and distortion therefore can be corrected, the projection of a 3D point to the ith image plane is described by [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF],

x i = f i (S) =   p i
where p i uv is the (u, v) entry in the camera matrix corresponding to the ith camera view. Using ( 6) and (2) 3D positional uncertainty can be propagated into the video domain.

Generally, S is determined as the point Ŝ which satisfies the implicit function,

F i (x i , S) = X(p i 31 x i -p i 11 ) + Y (p i 32 x i -p i 12 ) + Z(p i 33 x i -p i 13 ) + p i 14 X(p i 31 y i -p i 21 ) + Y (p i 32 y i -p i 22 ) + Z(p i 33 y i -p i 23 ) + p i 24 ( 7
)
such that some criterion function 7) and ( 3) enables uncertainty on pixel measurements to be propagated into the 3D positional domain.

C f (x, Ŝ) = N i F i (x, Ŝ) 2 is minimized. Using (

Evaluation of Audio-Visual Source Localization

Three 720 × 576 resolution cameras and six microphones were used to record an audio-visual source moving along a 3D path. The cameras were calibrated using 37 point correspondences from known 3D points within a 5.33m×6.98m×2.45m room. The 3D positions of the points were measured using a measuring tape, a square and a level using a single wall as a datum plane. A linear estimate of the camera matrices was used to initialize a bundle adjustment minimization procedure which minimized the reprojection error of the 37 points in the three views [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF]. The overall reconstruction error was found to be 5mm and over an additional set of 37 test points it was found to be 13mm.

The six microphones were arranged into two 3-element microphone arrays. The array geometry used was that of a vertical equilateral triangle with the relative spacing between the microphones set to 340mm.

The audio-visual source followed a known 3D trajectory in space. The source was localized in each camera view through intensity thresholding and blob extraction. The center of mass of the blob was then taken as the object's location in the frame and its 3D location was determined through triangulation. Recursive least squares using the Levenberg-Marquardt algorithm was used to localize the source in the acoustic domain. An audio sampling frequency of 48kHz was used and the time-delay estimates were obtained using GCC-PHAT [START_REF] Knapp | The Generalized Correlation Method for Estimation of Time Delay[END_REF] from 40ms audio data frames i.e. a single audio-based location estimate for each video frame. The video frame rate was 25 f ps and the total track duration was 2594 f rames.

Given both the audio-based and video-based tracking results, further calibration procedures were applied so as to optimize the relative calibration between the audio and video tracking spaces. This was necessary so as to ensure no bias existed between the tracking spaces.

Sufficient reconstruction accuracy was achieved from multi-view visual reconstruction such that any error in localization relative to audio based localization was deemed negligible. The visually reconstructed track therefore was taken as the true track's 3D position. Gaussian noise was added synthetically to pixel measurements to simulate noisy visual localization. In the audio-domain the variance of time-delay estimates was measured empirically using a running variance estimate.

Validity of First Order Error Propagation

Of particular concern in the application of the error propagation techniques presented in Sec. 2.1 is the validity of using a first order Taylor series expansion. From Fig. 2 however it can be seen that under the assumption of Gaussian noise first order uncertainty propagation is sufficient for the propagation of uncertainty into each domain.

Shown in Fig. 2a and Fig. 2b are the results of 100 Monte Carlo simulations for the localization of 12 positions in the room from noisy time-delay estimates and noisy pixel measurements respectively. Also shown in these figures are the predicted 95 percentile error regions. The variance of time-delay estimates in this case was set to 1 audio sample. The variance of pixel measurements was set to 5 pixels in both x and y. Shown in Fig. 2c are the audio based localization estimates mapped into the image plane. Also shown are the predicted error ellipses in the image plane. In Fig. 2d the result of mapping the video based location estimates to time delays in the audio domain is given. Also seen in this figure are the predicted error bars representing the 95 percentile error regions associated with the mapped time delays. In all of the described cases the predicted error regions are seen to match the localization results. through fusion in the video domain it can be seen that little improvement is achieved. This suggests that in the image plane the contribution of audio data in improving localization accuracy is small.

Using the trace of the 3D localization covariance matrix as a measure of localization accuracy, audio-based localization was compared to simulated videobased localization whereby noise was added to the true pixel measurements. The percentage of frames for which audio-based localization was found to be more accurate than video-based localization for varying pixel measurement noise is shown in Fig. 4a. From this it can be seen that even with a pixel measurement noise of 20 pixel 2 the percentage of frames for which audio localization is more accurate over the track duration is less than 40%. Although this is the case, for 5% of the track duration audio-based localization was found to be more accurate than video-based localization for a pixel measurement variance of 1 pixel 2 . This reveals significant variation in audio-based localization accuracy over a typical track duration.

The percentage of frames where fusion resulted in improved tracking accuracy can be seen in Fig. 4b. This figure reveals that the overall best improvement in accuracy over the track duration was obtained through fusion in the positional domain. The worst performance occurred for fusion in the video domain. These results are in accordance with the analysis presented in Fig. 3.

Conclusions

The use of error propagation was presented in this paper as a useful means of evaluating the performance of a joint audio-video based tracking system. First order error propagation was shown to adequately map audio and video measurement uncertainty across the positional, video and audio tracking domains. In the comparison of audio-based and video-based localization accuracy video-based localization was found to consistently outperform that of audio-based localization in terms of accuracy and consistency. Maximum likelihood fusion of location estimates in the audio domain, video domain and positional domain was examined. The best 3D localization accuracy for the described multi-camera and multi-microphone setup was found to be achieved where fusion is applied in the positional domain. Little contribution from audio in improving localization accuracy through a joint audio-video estimate was found where fusion is applied in the video domain.

The analysis presented in this paper can also be seen to provide a convenient basis for determining optimal sensor placement configurations [13]. APPENDIX A.

Given G(τ, S) and C g as defined in Sec. 2.2 the implicit functions theorem implies [START_REF] Faugeras | Three-Dimensional Computer Vision: A Geomteric Viewpoint[END_REF],

∂g -1 (τ ) ∂τ = -H -1 ∂Φ ∂τ ( 8
)
where Φ and H are derived in the following as, where † is used to denote the pseudo inverse.

Φ = ∂C1 ∂S T = 2 N i Gi(S, τ ) ∂Gi ∂S T , (9)
H = ∂Φ ∂S = ∂ ∂S ∂C1 ∂S T (10)

  [START_REF] Checka | Multiple Person and Speaker Tracking with a Particle Filter[END_REF] and (2), 3D positional uncertainty can be propagated into the domain of time-delay Mapping audio and video uncertainty into 3D space. Here Σ V S represents the covariance of a video-based location estimate and Σ A S denotes the covariance of an audio-based location estimate. Mapping video uncertainty into the audio domain. The covariance of a video based 3D location estimate in the audio domain is denoted Σ V τ . Mapping audio uncertainty into the video domain. The covariance of an audio based 3D location estimate in the video domain is denoted Σ V τ .

  Video-based localization error mapped to time delay domain of microphone pair m1 -m2.

  (e) 3D Error ellipsoids for audio (green) and video (red) based localization.

Fig. 2 :

 2 Fig. 2: Validity of first order error propagation.

Fig. 4 :

 4 Fig. 4: Comparison of audio-based, video-based and fusion-based localization performance.

This research is funded by the Irish Research Council for Science Engineering and Technology (Grant No. RS/2005/95) and Science Foundation Ireland.

Comparative Error Analysis and Discussion

Using the covariance mapping techniques presented in Sec. 2.1 and given the covariance of time delay estimates, the error associated with audio-based localization can be determined. Similarly, given the covariance of pixel based measurements the error associated with video-based localization can be determined. Shown in Fig. 3a are the resulting 95 percentile error ellipsoids for points along the track as described in Sec. 3. The variance of pixel measurements is assumed to be 5 pixels in both the x and y image axes and the variance of time delay estimates is measured empirically using a running variance estimate over the track duration.

From this it can be seen that in each of the cases of both audio and videobased localization the error associated with a location estimate is non-uniform in space. Also, the orientation of the error regions is dependent on the configuration of the sensors. Direct comparison of the associated 3D localization covariance matrices therefore can not be made. The trace of the covariance matrix defining the error ellipsoids was chosen as a performance measure for the accuracy of localization.

Shown in Fig. 3b is a table quoting this performance measure for the numbered points along the track as shown in Fig. 3a. Included in this table are the expected values of this performance measure of 3D localization accuracy for the three fusion strategies as outlined in Sec. 2. From these results it can be seen that as expected the theoretical performance of 3D localization using audio-video fusion is greater than the use of audio or video alone. It can also be seen that the best overall performance for 3D localization is expected where fusion is applied in the positional domain. Of the fusion strategies considered the worst 3D localization accuracy is expected where fusion is applied in the video domain. In comparing the results between localization using video data only to localization


Taketoshi Mori
email: tmori@ics.t.u-tokyo.ac.jp

Takashi Matsumoto
email: matumoto@ics.t.u-tokyo.ac.jp

Masamichi Shimosaka
email: simosaka@ics.t.u-tokyo.ac.jp

Hiroshi Noguchi
email: noguchi@ics.t.u-tokyo.ac.jp

Tomomasa Sato
email: tomomasasato@jcom.home.ne.jp

Multiple Persons Tracking with Data Fusion of Multiple Cameras and Floor Sensors Using Particle Filters

Successful multi-target tracking requires locating the targets and labeling their identities. For the multi-target tracking systems, the latter becomes more challenging when the targets frequently interact with each other. In this paper, we propose a method for multiple persons tracking using multiple cameras and floor sensors. Our method estimates 3D positions of human body and head, and labels their identities. The method is composed of multiple particle filters that interact only in the exclusion occlusion model. Each particle filter tracks each person correctly by integrating information from floor sensors and the targetspecific information from multiple cameras. Integration of these two types of sensors enables complement of each weak point and the correct tracking of the target. Moreover, we develop a new particle filter framework that tracks the human head by using the estimated human body position simultaneously. Our experimental results demonstrate the effectiveness and robustness of the method against several complicated movements of multiple persons. The results also demonstrate that this method can maintain correct tracking when the targets are in close proximity.

Introduction

Recently, expectations towards robotic systems which enables daily life assistance are rising. Among many, researches on home environment with distributed sensors are active, and believed to offer practical applications. Aware Home [START_REF] Kidd | The Aware Home: A Living Laboratory for Ubiquitous Computing Research[END_REF] and Sensing Room [START_REF] Mori | Sensing Room: Distributed Sensor Environment for Measurement of Human Daily Behavior[END_REF] are examples of such systems. When considering the appropriate support by such systems, position and ID of the targets are the most important information. Without tracking position and ID, inappropriate support and accidents might occur. In addition to the human body position, tracking a human part (i.e. a head, a hand) also enables a wide variety of support.

There are a number of researches on human tracking. Various sensors are tried for human tracking in those researches. Cameras [START_REF] Kobayashi | 3d head tracking using the particle filter with cascaded classifiers[END_REF][START_REF] Kim | Multi-camera tracking and segmentation of occluded people on ground plane using search-guided particle filtering[END_REF][START_REF] Yang | Fast multiple object tracking via a hierarchical particle filter[END_REF][START_REF] Perez | Color-based probabilistic tracking[END_REF], RFID [START_REF] Mori | Multiple people tracking by integrating distributed floor pressure sensors and rfid system[END_REF], laser range finder [START_REF] Fod | A laser-based people tracker[END_REF], and floor pressure sensors [START_REF] Addlesee | The orl active floor[END_REF][START_REF] Fukui | Expression method of human locomotion records for path planning and control of human-symbiotic robot system based on spacial existence probability model of humans[END_REF][START_REF] Murakita | Human tracking using floor sensors based on the markov chain monte carlo method[END_REF] are used.

Among a variety of sensors, cameras are especially used in many works. Most of these methods use particle filter framework [START_REF] Isard | Condensation-conditional density propagation for visual tracking[END_REF][START_REF] Vermaak | Maintaining multi-modality through mixture tracking[END_REF][START_REF] Du | Tracking by cluster analysis of feature points using a mixture particle filter[END_REF]. Kobayashi et al. [START_REF] Kobayashi | 3d head tracking using the particle filter with cascaded classifiers[END_REF] used cascaded classifier on AdaBoost in the framework. Kim et al. [START_REF] Kim | Multi-camera tracking and segmentation of occluded people on ground plane using search-guided particle filtering[END_REF] incorporated color histograms of the multiple camera images and segmentation on the ground plane into the framework. These methods succeeded in tracking a single person robustly or multiple persons in each environment. However, they are not sufficient to track against complicated movements of multiple persons.

On the other hand, Fukui et al. [START_REF] Fukui | Expression method of human locomotion records for path planning and control of human-symbiotic robot system based on spacial existence probability model of humans[END_REF] proposed a method for human locomotion with probability potential maps by the sensor floor and Murakita et al. [START_REF] Murakita | Human tracking using floor sensors based on the markov chain monte carlo method[END_REF] proposed a method for human tracking using floor sensors based on the Markov Chain Monte Carlo Method. These methods show the effectiveness of floor pressure sensors against a single person tracking. However, it is very difficult to label their identities by using only floor pressure sensors.

Sensor fusion techniques for human tracking are also proposed. Dore et al. [START_REF] Dore | A particle filter based fusion framework for video-radio tracking in smart spaces[END_REF] proposed a video-radio fusion technique using particle filter framework. In [START_REF] Mori | Multiple people tracking by integrating distributed floor pressure sensors and rfid system[END_REF], floor pressure sensors and RFID System are used for multiple persons tracking. However, these methods are not sufficient to estimate identities and positions of multiple persons correctly when the targets are in close proximity.

We then propose a method for multiple persons tracking using multiple cameras and floor sensors, which estimates 3D positions of human body and head, and labels their identities. The method is composed of multiple particle filters. In the particle filter framework, we integrate multiple camera images and floor sensors. Camera images are influenced by illumination, but a great deal of information is provided by cameras. Floor sensors are resistant to environmental changes and cover wide area, but it is difficult to discriminate multiple persons. Integration of these two types of sensors enables complement of each weak point and the correct tracking of the targets. Moreover, to track both human body and head simultaneously, we develop a new particle filter framework that tracks the human head by using the estimated body position.

This paper is organized as follows. We briefly give an overview of the particle filter in section 2. In section 3, we propose a framework for multiple persons tracking. In the framework, section 4 presents a method for human body tracking using multiple cameras and floor sensors. Then section 5 presents extension to head tracking. We mention experimental results of the proposed method in section 6. Finally, conclusion is discussed in section 7.

Particle Filter

The particle filter is a Bayesian sequential importance sampling technique, which recursively approximates the posterior distribution using a finite set of weighted samples. The posterior distribution at time t is approximated by a set of discrete samples { s

(n) t } (n = 1...N ) with importance weights { π (n) t } (n = 1...N ).
Particle filter simulates this distribution by the following three-step recursion.

Selection

: Select samples { s (n) t-1 } (n = 1...N ) in proportion to weight { π (n) t-1 } (n = 1...N ) corresponding to sample { s (n) t-1 } (n = 1...N ).

Prediction

(n) t ; π (n) t )} (n = 1...N ).

Multiple persons tracking

Traditional particle filters perform poorly at consistently maintaining the multimodality in the target distribution that often results from multiple targets. Vermaak et al. [START_REF] Vermaak | Maintaining multi-modality through mixture tracking[END_REF] introduced a mixture particle filter (MPF), where each component is modeled with an individual particle filter that forms part of the mixture. The MPF enables tracking multiple targets simultaneously. However, this method is inappropriate for multi-target 3D tracking that requires locating the targets and labeling their identities.

In this paper, our method is composed of multiple single-target particle filters as the MPF. Each of the filters tracks the same single-target and labels same identity through all tracking by holding the target-specific information.

Moreover, for discriminating the distribution of multiple filters, we applied the exclusion occlusion model as the interaction between multiple filters.

Target-specific information and detection

Each of the filters holds the target-specific information provided by cameras. In our method, we obtain color histogram of the person as the target-specific information and use it in the samples evaluation. The reference color histogram is gathered from the pixels detected by background subtraction of each camera image. When detecting a person, the filter obtains color histogram of the person and labels identity. The filter then starts tracking.

Person is detected by floor sensors. The sum of weight on floor sensors is in proportion to the number of persons, and floor sensors enable correct detection.

Occlusion exclusion model

When tracking multiple persons, samples of multiple particle filters are often mixed so that multiple particle filters track the same person by mistake. The remedy for this problem is exclusive segmentation of the targets' region. We then proposed the occlusion exclusion model.

When samples of the filters tracking a different person respectively overlap on the ground plane, the confidence of the samples is not high. Then we give samples the square area which side is 100mm, and the weights of the areaoverlapped samples are 0. The length of the side is determined by the size of a person. Such method enables exclusive segmentation. Fig. 1 shows the occlusion exclusion model.

Tracking by integration of cameras and floor sensors

Our method uses particle filter framework to track a single human body. We estimate human body position, which is represented as a 3D vector (x, y, z) T in a 3D state. In the framework, we integrate both information from floor sensors and the target-specific information from multiple cameras. We show a transition model and a observation model which are important when implementing the particle filter.

Transition model

We assume uniform straight motion of a target position between two successive image frames. Transition model p(x t |x t-1 = s t-1 ) is denoted as below.

s t = s t-1 + τv t-1 + ω, ( 1
)
v t = p t -p t-1 τ , ( 2
)
where τ is the time interval between frames, v t-1 is the previous velocity of the target, ω is a system noise added to s t-1 , and p t is the estimated target's position at time t. We control the diffusion factor ω adaptively by the velocity of the target. Such control of the system noise ω contributes to improving the robustness against sudden abrupt motion and the accuracy of the estimation.

Observation model for body tracking

In the observation model, cameras and floor sensors are integrated.

In the model, the evaluations of samples are hierarchically executed. The hierarchical evaluation enables to decrease calculation cost. The concrete evaluation procedures are given below.

1. Eliminate a sample when a location of the sample is out of the room.

2. Evaluate a weight π f,t by floor sensors as like section 4.3. When the evaluation score is 0, the sample is eliminated. 3. Project a sample into the i-th camera coordinates. When the sample is out of the field of view of the camera, the sample is eliminated. 4. Evaluate a weight π bg,i,t by background subtractions of the i-th camera image. The evaluation score is binary. When the evaluation score is 0, the sample is eliminated. 5. Evaluate a weight π clr,i,t by color histogram of the i-th camera image as like section 4.4. 6. Repeat the procedures from step 3 to 5 for each camera. Compute a weight π b,t for body tracking at time t by multiplying these likelihoods. A weight π b,t is π f,t i π bg,i,t π clr,i,t .

We incorporate the estimation using floor sensors into the particle filter framework and tracking with data fusion of multiple cameras and floor sensors is realized by multiplying the weights. Floor sensors are resistant to environmental changes and are not influenced by occlusion. Even when the multiple persons are in close proximity and occlusion ambiguities occur in camera images, the evaluation using floor sensors enables estimating true position.

In the evaluation by background subtractions, we improve the background subtraction process to avoid eliminating correct samples.We perform background subtraction in HSV color space. After the background subtraction, we use erosions and dilations of the image to decrease noise. Moreover, we remove small regions through the contour process.

The evaluation using color histogram in camera images is indispensable for tracking the correct target. Color histogram gives the target-specific information and enables labeling their identities correctly. The evaluation also contributes to the estimation of human position in high resolution.

Evaluation using floor sensors and evaluation using color histogram are described in the following section.

Evaluation of samples using floor sensors

The weight π f,t is evaluated by weight on floor sensors. The weight π f,t is computed by

π f,t = f t (x, y) m∈L f m t , (3)
where f t (x, y) is the weight at a location (x, y) and is included in the cluster given a label L in floor sensors. m is the elements of the cluster L. m∈L f m t is the weight of the cluster L and π f,t is the proportion of the weight at a location (x, y) to the weight of the cluster L.

Such evaluation contributes to a dense set of samples around the true position, which consequently enables the amelioration of the estimation accuracy and the tracking rate of multiple persons. Fig. 2 shows floor sensors images and camera images. Here, we convert weight of floor sensors into gray scale image.

Evaluation of samples using color histogram

Our method uses the data similarity between the reference color histogram p obtained in the initialization and a candidate color histogram q around a sample for samples evaluation. A candidate color histogram q is gathered within a 30 × 90 pixel-sized region set around a sample. Following Perez's method [START_REF] Perez | Color-based probabilistic tracking[END_REF], our method adopt a sample evaluation based on HSV color histograms. HSV is insensitive to illumination effects. Color information is however only reliable when both the saturation and the value are not too small. Hence, we populate an HS histogram with N H N S bins using only the pixels with saturation and value larger than two thresholds set to 0.1 and 0.07 respectively in our experiments. The remaining "color-free" pixels can however retain crucial information when tracked regions are mainly black and white. We thus populate N V additional value-only bins with them. The resulting histogram is composed of N = N H N S + N V bins, where N H , N S , N V are set to 10.

We apply the Bhattacharyya similarity coefficient. We thus compute the weights π clr,i,t by color histogram in the i-th camera image as below.

D(p, q)

= 1 - N k=0 √ p k q k (4)
π clr,i,t ∝ e -λD 2 (p,q) , (

where both p and q are normalized and λ is a constant which is experimentally determined.

Extension to head tracking

Our method estimates not only human body position but also human head position. In the framework of single person tracking, to track both body and head simultaneously, we develop a new particle filter framework.

Head tracking using estimated body position

In addition to a sample set represents body state, another sample set represents head state s h,t = (x, y, z, θ) T is added, where θ denotes the pan of the head. In our method, the filtering step for head tracking is also repeated individually. In sampling and evaluation step, we link head tracking to body tracking by using the estimated body position. By using the estimated body position, tracking performance when the head moves up and down is significantly improved. Moreover, the link with body tracking enables labeling their identities of the head. First, we show the single-target tracking procedures below. The details of the sampling and evaluation using body position are described in the following section.

(n) h,r,t } (n = 1...αM ) in proportion to weight { π (n) h,t } (n = 1...M ). (f) Propagate samples { s (n) h,

Sampling using body-head model

In the sampling step of samples for head tracking, a new sample set is composed of the mixture of samples s h,r,t generated by normal resampling step and samples s h,b,t generated around the position based on the estimated body position.

Samples s h,b,t are generated around the estimated body position and propagated with body-head model h t-1 . Body-head model h t denotes the distance between head and body of the target at time t. Samples s h,b,t are given by

s h,b,t = s h,b,t + h t-1 + ω, ( 6
) h t = p h,t -p b,t , (7)
where ω is a 4D Gaussian noise which is adaptively controlled by the velocity of the target as like section 4.1. Samples s h,r,t are selected in proportion to weight π h,t and propagated with state transition probability p(x t+1 |x t ). Transition model p(x t |x t-1 ) is the same as that of body tracking in Eq.1.

Observation model for head tracking

In the observation model, the evaluation of samples s h,t by using the estimated body position is incorporated. The evaluation procedures are given below.

1. Eliminate a sample when a location of the sample is out of the room. 2. Evaluate a likelihood by distance from body position. When the evaluation score is 0, the sample is eliminated. 3. Project a sample into camera coordinates. When the sample is out of the field of view of the camera, the sample is eliminated. 4. Evaluate a likelihood by background subtractions of the camera image. The evaluation score is binary. When the evaluation score is 0, the sample is eliminated. 5. Evaluate a likelihood by color histogram. 6. Repeat the procedures from step 3 to 5 for each camera. Compute a weight π h,t for head tracking at time t by multiplying these likelihoods.

In the second step of the evaluation, the distance from body position is used. Such evaluation enables head tracking linked to body tracking. The evaluation of color histogram is based on that of body tracking. Our method uses the data similarity between the reference color histogram obtained in the initialization and a candidate color histogram around a sample as like section 4.4. The reference color histogram is changed into histogram around human head. A candidate color histogram is gathered within a 20 × 20 pixel-sized region set around a sample. In the evaluation of the sample s h,t , the reference color histogram corresponding to the pan θ of the sample is selected from multiple reference color histograms obtained by multiple camera images.

Experimental Results

The tracking system

Floor pressure sensors are distributed on the floor at 200mm intervals. There are 252 pressure sensors. This floor detects pressure data with 200mm spatial resolution. Force Sensing Resisters (FSRs) are utilized as the pressure sensor. Pressure can be converted into the voltage value, because a FSR has a character that its electric resistance decreases by the increase in the power added. Pressure sensitivity range is 10kg/cm 2 . It produces an 8-bit output and can detect human existing areas by finding high-pressure areas. Sampling rate of the pressure sensors is 10Hz.

We used four cameras on the four corners of the ceiling of the room. The camera is IEEE1394camera DragonFly2. Each image was captured at a resolution of 320×240. 4 server PCs (Dell Precision 690, Intel Xeon Processor 5060 3.20 GHz × 2, 4GB Memory) were prepared, and a camera was connected to each server PC. Cameras were completely synchronized during capture, and intrinsic / extrinsic camera parameters were calculated beforehand.

A client PC was also prepared in the same specification. The client PC integrated data of two sensors and executed the tracking process. In this configuration, we processed 150 samples represent head state and 250 samples represent body state per one particle filter within 50ms when three persons tracking. The tracking runs online at a frame rate of 20 Hz. Fig. 3 shows their configuration.

Tracking multiple persons

To demonstrate the effectiveness and robustness of the method against the complicated movements of multiple persons, we performed several experiments. In the experiments, we assumed a variety of movements of multiple persons: crossing, turning after coming close, going around, and going back after coming close.

Fig. 4 shows the tracking results against two movements: crossing, going back after coming close. Tracking against going back after coming close is very difficult because the movement is contrary to the transition model. Even against the complicated movements, our method was able to estimate positions of multiple persons and label their identities correctly. This is attributed to data fusion of multiple cameras and floor sensors. When the multiple persons are in close proximity and occlusion ambiguities occur in camera images, floor sensors give true positions of the persons and cameras gives the target-specific information.

Next, the tracking experiment in a home environment was performed. Fig. 5 shows an example of the images obtained in the experiment. We can see from Fig. 5 that the human head position is well estimated even when the head moves up and down.

Estimation errors of human position

To investigate the estimation accuracy of the human body by the proposed method, we conducted comparative experiments. We confirm the effectiveness of the evaluation using floor sensors and the evaluation using color histogram. However, it is difficult to measure human body positions accurately. Therefore, we assumed that a test subject walks along the straight test course precisely. Fig. 6 shows the trajectory of the estimated body position by three comparative methods. We can see from Fig. 6 that the estimation accuracy is sufficient to estimate the body position. The effectiveness of the evaluation using color histogram can be also confirmed.

Next, order to investigate quantitatively the estimation accuracy of the human head by the proposed method, the human head in the image was manually specified. The 3D coordinates obtained by inverse projection from 4 camera images was assumed to be the true position, and was compared to the result of estimation. The means of the error on the X, Y and Z axis are 4.9, 5.4 and 4.5 cm respectively. These errors are below the radius of human head and the tracking can be considered sufficiently accurate.

Conclusion

In this work, we have proposed a method for tracking multiple persons by integrating multiple cameras and floor sensors. Our method estimates positions of human body and head, and labels their identities. Our method is composed of multiple particle filters, and each filter tracks each person correctly by integrating information from floor sensors and target-specific information from cameras.

Successful tracking was demonstrated in a tracking experiment for multiple persons. Even when the targets are in close proximity and move complicatedly, our method was able to estimate positions of the targets and label their identities correctly. This is due to the integration of multiple cameras and floor sensors. Because floor sensors are resistant to environmental changes and is not influenced by occlusion, the evaluation using floor sensors enabled estimating true position. Camera provides a great deal of information such as color histogram and background subtraction, and the evaluation using color histogram enabled labeling their identities correctly. It was shown that robust tracking of multiple persons was realized by the integration of these two types of sensors.

Fig. 1 .

 1 Fig. 1. Occlusion exclusion model

Fig. 2 .

 2 Fig. 2. Weight on floor sensors

  } (n = 1...N ) for body tracking in proportion to weight { π (n) b,t } (n = 1...N ). 5. Propagate samples { s (n) b,t } (n = 1...N ) with state transition probability p(x t+1 |x t ) and generate new samples { s (n) b,t+1 } (n = 1...N ).

Fig. 3 .

 3 Fig. 3. Configuration of cameras and floor sensors

Fig. 4 .

 4 Fig. 4. Tracking of multiple persons

Fig. 5 .

 5 Fig. 5. Tracking of multiple human heads in indoor environment

Fig. 6 .

 6 Fig. 6. Walking trajectory of a person

  : Propagate samples { s

	(n) t-1 } (n = 1...N ) with state transition
	probability p(x t |x t-1 = s t-1 ) and generate new samples { s	(n) t } (n = 1...N )
	at time t. 3. Update: Update weights π t (n) by evaluating a likelihood through observations. Normalize π = p(y t |x t = s (n) t ) corresponding to sample s (n) so that the (n) t t sum of { π (n) t } (n = 1...N ) is equal to 1. As a result, estimated state at time
	t is equal to the expectation of the set of samples { (s

  } (n = 1...M ) as like section 5.3. (d) Estimate the head position p h,t at time t.

	1. Apply the exclusion occlusion model to the sample set for body tracking
	{ s	(n) b,t } (n = 1...N ) generated at time t -1.
	2. Update weights { π b,t } (n = 1...N ) as like section 4.2. (n)
	3. Estimate the body position p b,t at time t.
	(a) Generate the new sample set { s around the body position p b,t and propagate the samples with body-head (n) h,b,t } (n = 1...αM ) for head tracking
		model h t-1 .
	(b) Mix samples { s	(n) h,b,t } (n = 1...αM ) with samples { s	(n) h,r,t } (n = 1...(1 -
		α)M ) generated at time t -1 by resampling, and generate the sample
		set { s	(n) h,t } (n = 1...M ).
	(n) h,t (c) Update weights { π

(e) Select samples { s

  r,t } (n = 1...αM ) with state transition probability p(x t+1 |x t ) and generate new samples { s

	(n) h,r,t+1 } (n = 1...αM ).
	4. Select samples { s