Antoine Deleforge
email: antoine.deleforge@inria.fr

Radu Horaud
email: raduhoraud@inria.fr

The Cocktail Party Robot: Sound Source Separation and Localisation with an Active Binaural Head *

Keywords: G.3 [Mathematics of Computing]: Probability and Statistics-Multivariate statistics, Probabilistic algorithms; I.2.7 [Natural Language Processing]: Speech Recognition and Synthesis Algorithms, Theory, Experimentation Blind source separation, computational auditory scene analysis, EM algorithm, learning

Human-robot communication is often faced with the difficult problem of interpreting ambiguous auditory data. For example, the acoustic signals perceived by a humanoid with its on-board microphones contain a mix of sounds such as speech, music, electronic devices, all in the presence of attenuation and reverberations. In this paper we propose a novel method, based on a generative probabilistic model and on active binaural hearing, allowing a robot to robustly perform sound-source separation and localization. We show how interaural spectral cues can be used within a constrained mixture model specifically designed to capture the richness of the data gathered with two microphones mounted onto a human-like artificial head. We describe in detail a novel EM algorithm, we analyse its initialization, speed of convergence and complexity, and we assess its performance with both simulated and real data.

INTRODUCTION

There is an increasing interest in robots able to communicate with people in the most natural way, e.g., Fig. 1. * This work was supported by the European project HU-MAVIPS, under EU grant FP7-ICT-2009-247525.

Figure 1: A cocktail party robot should be able to communicate with people in the most natural way. One fundamental task that such a robot should be able to accomplish is to localize the speakers in a room and to separate their emitted speech signals, all in the presence of music, background noise, and reverberations.

Such robots must be endowed with the ability to reliably process and understand sensory inputs, e.g, visual or auditory data, gathered in unconstrained physical situations. Within the field of computational auditory scene analysis (CASA) tremendous progress was made in speech recognition. Nevertheless, current approaches work well with a single sound source often recorded with a close-range microphone. A more natural setup, e.g., humanoids with their own on-board microphones, implies to deal with much more complex and challenging acoustic inputs involving auditory data of various kinds (speech, prosody, music, electronic devices, etc.) and originating from sparsely located multiple sound sources, all in the presence of noise, attenuation and reverberations.

A classical example illustrating the difficulty of modeling such situations is the well known cocktail party problem (CPP) [START_REF] Haykin | The cocktail party problem[END_REF]. While human listeners solve this problem routinely and effortlessly, we note that it has not been properly addressed from the perspective of robot audition. Two key aspects of CPP are localization and separation of several sound sources. We believe that principled solutions to these problems are some of the prerequisites for addressing higher-level tasks in HRI such as speech and music recognition, verbal communication, dialog handling, etc.

In this paper we propose a new method for solving for sound-source separation and localization based on a gen-erative probabilistic model and on active binaural hearing. More precisely, we promote a novel robot audition paradigm based on interaural spectral features, namely the interaural level difference (ILD) and the interaural phase difference (IPD) and on a constrained mixture model specifically designed to capture the richness of the binaural data recorded with a robot head endowed with a human-like head related transfer function (HRTF). We formally derive an EM procedure that alternates between separation (E-step) and localization (M-step). We show how our system can be fully automatically and efficiently trained using an audiomotor map. We analyse our algorithm in detail and we assess its performance with both simulated and real data.

There is behavioral and physiological evidence that humans use binaural cues in order to infer the direction of a sound. Two such cues seem to play an essential role, namely the ILD (already mentioned) and the interaural time difference (ITD). A number of computational models have been recently developed for robust sound localization and sound tracking based on ITD and/or ILD [START_REF] Roman | Binaural tracking of multiple moving sources[END_REF][START_REF] Willert | A probabilistic model for binaural sound localization[END_REF]. However, it is well known that the spatial information provided by interaural-difference cues within a restricted band of frequency is spatially ambiguous, particularly along a roughly vertical and front/back dimension [START_REF] Middlebrooks | Sound localization by human listeners[END_REF]. To avoid these ambiguities, more accurate sound localization models incorporate the HRTF, e.g. [START_REF] Keyrouz | A new method for binaural 3D localization based on HRTFs[END_REF]. These approaches are based on the fact that the particular shape of the head, pinna and torso act as a filter depending on the emitting 3D location of the sound source (distance, azimuth, and elevation). However, HRTF databases are subject-specific and room-dependent (noise, reverberations, room geometry, etc.), and only a handful of HRTF databases are available in practice [START_REF] Algazi | The CIPIC HRTF Database[END_REF][START_REF] Shinn-Cunningham | Localizing nearby sound sources in a classroom: Binaural room impulse responses[END_REF]. This makes them hardly applicable to a real robotic application. To overcome these issues, the HRTF of a specific robot can be automatically learnt using audio-motor maps [START_REF] Deleforge | Learning the direction of a sound source using head motions and spectral features[END_REF][START_REF] Hörnstein | Sound localization for humanoid robots -building audiomotor maps based on the HRTF[END_REF]. Such maps are built by recording a static full-spectrum sound source from different motor states of the robot.

The problem of sound source separation has been thoroughly studied in the last decades and several interesting approaches were proposed. For example, [START_REF] Bensaid | Single microphone blind audio source separation using EM-Kalman filter and short+long term AR modeling[END_REF][START_REF] Roweis | One microphone source separation[END_REF] and many others achieve separation with a single microphone, based on known acoustic properties of speech signals, and are therefore limited to a specific type of input. Other techniques such as independent component analysis (ICA) [START_REF] Comon | Handbook of Blind Source Separation, Independent Component Analysis and Applications[END_REF] or multimicrophone techniques require as many microphones as the number of sources. Several other methods use binaural localization cues for source separation [START_REF] Keyrouz | Robotic localization and separation of concurrent sound sources using selfsplitting competitive learning[END_REF][START_REF] Mandel | Modelbased expectation-maximization source separation and localization[END_REF][START_REF] Viste | On the use of spatial cues to improve binaural source separation[END_REF]. In [START_REF] Keyrouz | Robotic localization and separation of concurrent sound sources using selfsplitting competitive learning[END_REF] acoustic inputs at different frequency channels are clustered over time by means of some assumptions on the emitted signals, and an HRTF data look-up table is used to find their corresponding positions in space. Once exact locations are known, up to two sources can be separated using the HRTF at each frequency channel.

Our method is based both on clustering (localization) and on spectral masking (separation). Spectral masking, also called binary masking, allows the separation of an arbitrary number of sources from a mixed signal, with the only assumption that a single source is active at every frequencytime point (f, t). This is referred to as the W-disjoint orthogonality assumption [START_REF] Yılmaz | Blind separation of speech mixtures via time-frequency masking[END_REF] and it has been shown to hold, in general, for simultaneous speech signals; It is particularly well suited for binaural recordings in realistic environments. Recently, [START_REF] Mandel | Modelbased expectation-maximization source separation and localization[END_REF] proposed a probabilistic model for multiple sound source separation based on interaural spatial cues. For each sound source, a binary mask and a discrete distribution over interaural time delays is provided. This can be used to approximate the azimuth angle of the source with a front-back ambiguity, if the distance between the microphones is known.

It appears that there are very few methods that formally combine 2D localization and separation. The first originality of our approach is to formally cast the localization and separation tasks into a generative probabilistic model that is solved very efficiently with a novel EM algorithm. Full details of this algorithm and its initialization as well as favorable comparisons with recent binaural-based separation methods are presented in a companion paper [START_REF] Deleforge | A latently constrained mixture model for audio source separation and localization[END_REF]. The second originality of our approach is to use an active robotic head in order to automatically learn audio-motor maps for a very large set of sound-source directions located in the far field. Such a training phase incorporates an implicit model of the HRTF and leads to a high precision both in terms of source separation and of localization. A thorough evaluation of the method with simulated and real data is provided, and puts forwards this approach as a promising future tool for auditory human-robot interactions.

The remainder of this paper is organized as follows: section 2 describes a binaural sound representation, section 3 presents in detail the formal model as well as its associated EM algorithm, section 4 describes the data acquisition and recording technique, section 5 presents our validation method, experiments and results. Concluding remarks and directions for future work are discussed in section 6.

BINAURAL SOUND REPRESENTATION

Both sound source localization and separation require a proper representation of the perceived data. For localization, one needs a content-independent representation that contains as much spatial information as possible. For separation, one needs a representation preserving all the richness of the original signals. In this paper we put forward two binaural representations, namely ILD and IPD spectrograms.

As already mentioned in section 1, our sound source separation method is based on binary masking. This technique consists in "filtering" the original signal by weighting all frequency-time points corresponding to the target source with 1 and all the other points with 0. Consequently, the perceived signal needs to be described within a time-varying spectral representation, a spectrogram. Spectrograms associated with each one of the two microphones are computed using a short-term fast Fourier transform (FFT) algorithm [START_REF] Allen | Short-term spectral analysis, synthesis, and modification by discrete fourier transform[END_REF]. We use a time window of 64ms with 8ms overlap between two consecutive windows, thus yielding T = 126 time-windows for a signal lasting 1s. Since sounds are recorded at a sample rate of 24,000Hz, each time-window contains 1,536 samples, multiplied by a Hann window padded with zeros, for a total length of 2,048 samples. Each window is then transformed via FFT to obtain complex coefficients of F = 1, 024 positive frequency channels between 0 and 12,000Hz. We denote with s

(k) f,t | 2 ) (1)
while the spectrogram of the perceived acoustic level is defined by a (LR) f,t = 10 log(|s The W-disjoint orthogonality assumption implies that a single sound source emits at a given frequency-time point (f, t). Let k be that source, the HRTF model provides a relationship between emitted and perceived spectrogram points at (f, t):

(L) f,t | 2 + |s (R) f,t | 2 ) (2)
s (L) f,t = h (L) (x k , f )s (k) f,t and s (R) f,t = h (R) (x k , f )s (k) f,t (3)
where x k ∈ R 3 denotes the 3D position of sound source k in a robot-centered coordinate frame (the origin of this frame is the midpoint between the two microphones and the x, y, and z axes pointing towards the left microphone, the headtop and in front of the head) and h (L) and h (R) denote the left and right HRTF. The latter functions depend on both the emitter's position and the frequency and they act as linear filters on the emitted signal. The HRTFs are mainly determined by the shape of the head, pinna and torso of the listener, e.g, the robot-mounted dummy-head in our case. The interaural transfer function (ITF) Î is defined by the ratio between the left-and right-HRTF:

I(x k , f ) = h (R) (x k , f ) h (L) (x k , f ) ∈ C (4)
We can now define the interaural spectrogram as the ratio between the left-and right-microphone spectrograms:

Îf,t = s (R) f,t s (L) f,t (5)
From ( 3), ( 4) and ( 5) one may notice that if the source k emitting at time-frequency point (f, t) is located at x k then Îf,t ≈ I(x k , f ). The equality is only approximate due to the sensor noise and FFT errors. Therefore, at a given frequency-time point, the interaural spectrogram value I f,t does not depend on the emitted spectrogram value s

(k) f,t
but only on the source position x k . We finally define the ILD spectrogram α and the IPD spectrogram φ as the logamplitude and phase of the complex interaural spectrogram Îf,t , e.g., Fig. 2-(c) and (d):

α f,t = 20 log | Îf,t | ∈ R and φ f,t = arg( Îf,t ) ∈ ]-π, π] (6)

SEPARATION AND LOCALIZATION

This section starts by describing how a training set of interaural parameters can be built out of a sound dataset annotated with sources positions. After formally stating the problem, we then introduce our novel probabilistic model for interaural parameters in a mixture of point sound sources. We finally detail a new version of the EM algorithm for this model, allowing to achieve simultaneous sound sources separation and localization based on learned positions.

Building a Training Set

We first explain how a training set of interaural parameters can be built out of a sound dataset annotated with sources positions. Let X = {xn} N n=1 be a set of 3D coordinates in the robot centered frame that we already described above. If a single sound source n emits white noise from xn ∈ X , let {α n f,t } F,T f =1,t=1 and {φ n f,t } F,T f =1,t=1 be the perceived ILD and IPD spectrograms. The mean ILD vector µ(xn) = (µ n 1 . . . µ n f . . . µ n F ) ⊤ ∈ R F associated with xn is defined by taking the temporal mean of α at each frequency channel:

µ f (xn) = 1 T T X t=1 α n f,t (7)
Similarly, the mean IPD vector ξ(xn) = (ξ n 1 . . . ξ n f . . . ξ n F ) is the temporal mean of the IPD spectrogram φ at each frequency channel:

ξ f (xn) = arg 1 T T X t=1 exp(jφ n f,t ) ∈ ]-π, π] (8)
The components of the mean IPD vector are estimated in the complex domain in order to avoid problems due to phase circularity, as suggested in [START_REF] Nix | Sound source localization in real sound fields based on empirical statistics of interaural parameters[END_REF]. White noise is used because it contains equal power within a fixed bandwidth at any center frequency: the source n is therefore the only source emitting at each frequency-time point (f, t), and µ f and ξ f are thus approximating the log-amplitude and phase of Î(., f ). The set X of 3D sound-source positions as well as the mappings µ and ξ will be referred to as the training data to be used in conjunction with the sound-source separation and localization algorithm described below.

Problem Formulation

We suppose that there are K sound sources randomly located in the robot's environment and that these K sources emit simultaneously sounds with unknown spectrograms from the unknown locations {x k } K k=1 ⊂ X . From the acoustic inputs perceived by the robot's microphone pair, one can build the ILD and IPD spectrograms {α f,t } F,T f =1,t=1 and {φ f,t } F,T f =1,t=1 as already described. The goal of the soundsource separation and localization algorithm is to associate each perceived frequency-time point (f, t) with one of the sound sources and to estimate their 3D locations.

The observed α f,t and φ f,t are useful only if there is a significant signal at (f, t). To identify such significant observations we estimate the perceived acoustic level a LR f,t with (2) and we retain only those frequency-time points for which the acoustic level is above a threshold, a LR f,t > ǫ f . One empirical way to choose the thresholds (one at each frequency) is to measure the highest perceived acoustic level at each f in the absence of emitting sound sources. These thresholds are typically very low compared to perceived acoustic levels due to natural sounds, and allow to filter out frequencytime points corresponding to the "room silence". We call level mask the associated spectral mask. We denote by M f the number of significant observation at f and by α f,m and φ f,m the m-th significant ILD and IPD observations at f . The observed data will be denoted by

A = {α f,m } F,M f f =1,m=1 and Φ = {φ f,m } F,M f f =1,m=1 .
We also introduce the missing data, i.e., a set of unobserved variables z f,m ∈ {0, 1} K , such that z f,m,k = 1 if observations α f,m and φ f,m were generated by source k, and z f,m,k = 0 otherwise. The W-disjoint orthogonality constraint can therefore be written as:

K X k=1 z f,m,k = 1 ∀(f, m) (9)
Variables z f,m are also called data-to-source assignments at (f, m), and

M k = {z f,m,k } F,M f f =1,m=1
corresponds to the binary spectral mask associated with sound-source k. Finally, we denote with

Z = {z f,m } F,M f f =1,m=1
the set of all missing data. The problem of simultaneous localization and separation amounts to estimate the sound-source locations {y k } K k=1 and the masking variables Z, given the number of sound sources K and the observed data A and Φ.

A Constrained Mixture Model

We assume that both the ILD and IPD observed data are drawn from normal distributions. The conditional likelihood of the observation α f,m given its assignment to sound source k, i.e., z f,m,k = 1 and located at x k is therefore drawn from a 1D Gaussian distribution centered at µ f (x k ) and with variance σ 2 f,k :

P (α f,m |z f,m,k , x k , σ f,k ) = N (α f,m |µ f (x k ), σ 2 f,k ) = 1 (2π) 1/2 σ f,k exp - (α f,m -µ f (x k )) 2 2σ 2 f,k ! (10)
For simplicity, the expression z f,m,k = 1 is replaced by z f,m,k in probabilities. Since the IPD data lie on the circle ]-π, π], they should be modeled by a circular normal distribution. As proposed in [START_REF] Mandel | Modelbased expectation-maximization source separation and localization[END_REF], we approximate the wrapped normal distribution with a 1D Gaussian. The conditional likelihood of the observation φ f,m given its assignment to sound source k located at x k is given by:

P (φ f,m |z f,m,k , x k , ρ f,k ) = ∠N (φ f,m |ξ f (x k ), ρ 2 f,k ) = 1 (2π) 1/2 ρ f,k exp - ∆(φ f,m , ξ f (x k )) 2 2ρ 2 f,k ! (11)
where ρ 2 f,k is the IPD variance associated with source k at frequency f and the ∆ function is defined by

∆(x, y) = arg(e j(x-y) ) ∈ ]-π, π] (12)
The distribution ∠N (ξ, ρ 2 ) approximates the normal distribution on the circle ]-π, π] when ρ is small relative to 2π. Preliminary experiments on IPD spectrograms of white noise showed that this assumption holds in the general case.

Note that in our model, the source positions act as latent constraints on Gaussian means and thus tight the observations at different frequency channels. Such an approach can also be used to tight auditory and visual observations, as recently done in [START_REF] Khalidov | Conjugate mixture models for clustering multimodal data[END_REF].

As emphasized in [START_REF] Mandel | Modelbased expectation-maximization source separation and localization[END_REF], the well known correlation between ILD and IPD does not contradict the assumption that Gaussian noises corrupting the observations are independent. Under this assumption, the conditional likelihood of the observed data (α f,m , φ f,m ) given its assignment to source k located at x k and with variances σ 2 f,k and ρ 2 f,k is

P (α f,m , φ f,m |z f,m , x k , σ f,k , ρ f,k ) = N (α f,m |µ f (x k ), σ 2 f,k ) × ∠N (φ f,m |ξ f (x k ), ρ 2 f,k ) (13)
The prior probability of assigning an observation to a sound source writes:

P (z f,m,k ) = π f,k (14)
Finally, we denote by Θ the set of all the parameters of our binaural mixture model:

Θ =  {x k }; {π f,k }; {σ 2 f,k }; {ρ 2 f,k } ff F,K f =1,k=1 (15)

The Separation-Localization Algorithm

The problem of both source separation and source localization can now be expressed as an optimal parameter estimation problem, namely the maximization of the observeddata log-likelihood over the parameters Θ:

e Θ = arg max Θ L(A, Φ; Θ) (16)
In order to keep the model as general as possible, there is no assumption on the emitted sounds as well as the way their spectra are spread across the frequency-time points. Therefore, we assume that all the observations are independent:

L(A, Φ; Θ) = log P (A, Φ|Θ) = F X f =1 M f X m=1 log P (α f,m , φ f,m |Θ) (17)
It is well established that the direct optimization of ( 17) is difficult because of the presence of many local maxima. Therefore we recast the problem within the framework of maximum likelihood with missing data, i.e., Z in [START_REF] Khalidov | Conjugate mixture models for clustering multimodal data[END_REF], which is traditionally solved via expectation-maximization (EM).

The EM algorithm alternates between estimating the expected complete-data log-likelihood (E-step) using the current model parameters and maximizing this likelihood over the model parameters given the current posterior probabilities (M-step). The posterior probability r f,m,k of z f,m,k is defined by:

r f,m,k = P (z f,m,k |α f,m , φ f,m ; Θ) (18)
As it will be detailed below, the E-step of the proposed algorithm computes the posterior probabilities of assigning each spectrogram point to a sound source k (separation-step), while the M-step maximizes the expected complete-data loglikelihood with respect to the model parameters Θ, namely the priors, the variances and, most notably, the source locations {x k } K k=1 (localization-step). One of the most interesting properties of EM algorithms is that the log-likelihood L in ( 16) is increased at each EM iteration and hence it converges to a local maximum. An outline of the proposed EM method is provided in Algorithm 1. The finally estimated posterior probabilities allow to compute the binary spectral masks M k associated with each source k while the final parameters x k provide estimates for the source locations. v (p) denotes the value of variable v at iteration p, and v (P) denotes its final value. The E-step, M-step, initialization and convergence check are detailed below.

Algorithm 1 Separation-Localization EM 1: Input: A, Φ, {µ(xn)} N n=1 , {ξ(xn)} N n=1 , K 2: Output: {x (P) k } K k=1 , {M k } K
(p) f,m,k } f,m,k ) 9: end while 10: M k := (k == arg max k ′ r (P) f,m,k ′ ) F,M f f =1,m=1
Based on Bayes' formula and on marginalization rules, the E-step computes the current posterior probabilities conditioned by the previously estimated parameters, i.e., [START_REF] Khalidov | Conjugate mixture models for clustering multimodal data[END_REF]:

r (p) f,m,k := π f,k P (α f,m , φ f,m |z f,m ; Θ (p-1) ) P K i=1 π f,i P (α f,m , φ f,m |z f,m ; Θ (p-1) ) (19)
The expected complete-data log-likelihood can now be written as:

Q(Θ|Θ (p-1) ) = E (Z|A,Φ;Θ) [log P (A, Φ, Z|Θ)] = F X f =1 M f X m=1 K X k=1 r (p) f,m,k log π f,k P (α f,m , φ f,m |z f,m ; Θ) (20)
The M-step maximizes [START_REF] Roweis | One microphone source separation[END_REF] with respect to Θ:

Θ (p) := e Θ = arg max Θ Q(Θ|Θ (p-1) ) (21)
By combining ( 13) with ( 20) the problem becomes equivalent to minimizing:

F X f =1 M f X m=1 r (p) f,m,k log σ 2 f,k π f,k ! + log ρ 2 f,k π f,k ! + (x f,m -µ f (x k )) 2 σ 2 f,k + ∆(φ f,m , ξ f (x k )) 2 ρ 2 f,k ! (22)
which can be easily differentiated with respect to {π f,k } f , {σ f,k } f and {ρ f,k } f to obtain closed-form expression, conditioned by x k , for the optimal values of these parameters. These expressions are then subtituted in [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF], which is evaluated for all x k ∈ X to find the optimal position xk , and deduce all the other optimal parameters. Interested readers can find a detailed solution in [START_REF] Deleforge | A latently constrained mixture model for audio source separation and localization[END_REF].

As already mentioned, EM converges to a local maximum of the observed data log-likelihood function L. However, the non-injectivity of the interaural functions µ f and ξ f leads to a very large number of these maxima, especially when the set of learned positions X , i.e., section 3.1, is large. This makes the algorithm to be very sensitive to initialization. A common way to avoid being trapped in local maxima may be to initialize the parameters at random, but such a strategy cannot be directly applied here: First, because the cardinality of the parameter set Θ is very large and second, because there is no straightforward way to initialize the variances σ 2 f,k and ρ 2 f,k . Another possibility may be to randomly initialize the source assignment variables Z and then proceed with the M-step, but extensive experiments with simulated data revealed that the algorithm very rarely converged to a global maximum (in less than 0.1% of the cases). We therefore decided to adopt a method that combines these two initialization strategies by randomly perturbating both the source locations and the source assignments.

This lead us to develop a stochastic initialization procedure similar to the stochastic EM (SEM) algorithm [START_REF] Celeux | A classification EM algorithm for clustering and two stochastic versions[END_REF]. The idea of exploiting stochasticity to escape from local maxima is a commonly used principle in global optimization [START_REF] Zhigljavsky | Stochastic Global Optimization[END_REF]. The SEM algorithm includes a stochastic step (S) in between the E and the M steps, during which random samples R f,m,k ∈ {0, 1} are drawn from the posteriors r f,m,k . These samples are then used instead of the posterior probabilities during the M-step. To initialize our algorithm, we first set all the posterior probabilities to 1/K and then proceed through the following step sequence: S M* E S M, where the M*-step is a variation of the M-step in which the sources' positions are drawn randomly from X instead of computing {x k } k . Experiments with simulated data showed that this technique converged to a global optimal solution in over 10% of the cases. More advanced techniques may also be used to improve the convergence rate, and are detailed in [START_REF] Deleforge | A latently constrained mixture model for audio source separation and localization[END_REF].

In practice, twenty stochastic initializations are used in order to increase the chances of correct convergence, and only the one providing the best log-likelihood after two iterations is eventually iterated until the convergence criteria is satisfied. The algorithm stops either when the log-likelihood gain is less than 1%, or after pmax = 20 iterations. Indeed, from two hundred simulated experiments (see section 5.2), we concluded that, on an average, the algorithm converges in eleven iterations.

Algorithm Complexity

Both the E-and S-steps are linear in the total number of significant observations, P f M f and in the number of sound sources, K. The M-step is linear in the number of frequency channels F and in the number N of source locations available in the training set X . In the case of simulated data using one-second long signals composed of two sources, the values of these parameters are: K = 2, F = 1024, P f , M f = 70, 000, and N = 10, 800. With a Matlab implementation executed on a 2.53GHz Intel-Xeon processor, we obtained the following average running times: 49ms (E), 1030ms (S), 2520ms (M) and 24ms (M*). The most time-consuming part of the algorithm is its initialization (20 stochastic initializations iterated 2 times) which takes 195.9 seconds, while each subsequent EM iteration takes 2.569 seconds, amongst which 98% corresponds to selecting all optimal xk in X . Careful algorithm and software optimization will allow, however, to obtain realistic execution times needed for human-robot interactions scenarios, e.g., one to two seconds.

DATA ACQUISITION

In order to build the training set introduced in section 3.1, we developed a technique to learn a large number of sound source locations in an entirely unsupervised and automated way using the motor system of a binaural robot head. This technique is initially inspired from the sensorimotor theories of early development in psychology, suggesting that experiencing the sensory consequences of voluntary motor actions was necessary for an organism to learn the perception of space [START_REF] O'regan | A sensorimotor account of vision and visual consciousness[END_REF]. In particular [START_REF] Aytekin | A sensorimotor approach to sound localization[END_REF] argued that naive organisms such as humans and echo-locating bats could learn sound localization based solely on acoustic inputs and their relation to motor states. This idea was experimentally validated using a robot system in [START_REF] Deleforge | Learning the direction of a sound source using head motions and spectral features[END_REF]. Sound acquisition is performed with a Sennheiser MKE 2002 acoustic dummy-head linked to a computer via a Behringer ADA8000 Ultragain Pro-8 digital external sound card. The head is mounted onto a robotic system with two rotational degrees of freedom: a pan motion and a tilt motion (see Fig. 3). This device was specifically designed to achieve precise and reproducible movements. The emitter -a loudspeaker -is placed at approximately 2.7 meters ahead of the robot, as showed on Fig. 3. The loud-speaker's input and the microphones' outputs were handled by two synchronized sound cards in order to simultaneously record and play. All the experiments were carried out in real-world conditions, i.e., a room with natural reverberations and background noise due to computer fans. All the recordings are publicly available 1 . We believe that such a large audio-motor data set has no equivalent today, and is therefore a contribution in its own right.

Rather than placing the emitter at known 3D locations around the robot, it was kept in a fixed reference position while the robot recorded emitted sounds from different motor states. Consequently, a sound source direction is directly associated to a pan-tilt motor state (ψ, θ) rather than a 3D point in space. A robot trained in such a way would therefore be able to perform the head movement pointing to-ward an emitting sound source in an entirely unsupervised way, without needing the inverse kinematics, the distance between microphones, or any other parameters.

Recordings were made from 10, 800 uniformly spread motor states: 180 pan rotations ψ in the range [-180 • , 180 • ] (left-right) and 60 tilt rotations θ in the range [-60 • , 60 • ] (top-down). The associated set of 3D source positions X ⊂ R 3 could be deduced using the direct kinematic model of the robot given in [START_REF] Deleforge | Learning the direction of a sound source using head motions and spectral features[END_REF]. However, the speaker was located in the far field of the head during experiments (> 1.8 meters), and [START_REF] Otani | Numerical study on source-distance dependency of head-related transfer functions[END_REF] showed that HRTFs mainly depend on the sound source direction while the distance has fewer impact in that case. That is why sound source locations will be expressed with angles in the rest of the paper.

At each motor state, five binaural recordings of one second each were made while the speaker emitted different sounds. Sound 1 corresponds to white noise, and was used to build the training set (section 3.1). Sounds 2, 3 and 4 form the test set (see section 5.3), and correspond respectively to a woman pronouncing "Bonjour!", a man pronouncing "Un petit café?", and a flute melody. Sound 0 corresponds to "room silence" and was used to determine the perceived acoustic level threshold ǫ f during tests (see section 3.2).

EXPERIMENTS AND RESULTS

Performance Evaluation

Algorithm 1 was tested and validated using the database described in the previous section. In all presented results, a source is considered as correctly localized if the algorithm could locate it within 2 • of absolute angular error in both azimuth and elevation. For real mixtures (section 5.2), the separation was evaluated using standard metrics, namely Source-to-Distortion Ratio (SDR) and Source-to-Interferences Ratio (SIR), proposed in [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF]. These metrics are based on the decomposition of the estimated signal into the target signal, the error term coming from other interfering signals, and the error term coming from unexplained artifacts. Another term can be added to evaluate background noise reduction, but this is outside the scope of this work. Once the decomposition is achieved, the SDR is defined by the ratio of energy between the target and the error terms, while the SIR is the ratio of energy between the target and the interference term only. Both these metrics are expressed in decibels. As they are computed from 1D signals, the masked left and right microphone spectrograms were converted back to temporal signals using the inverse FFT, and concatenated to evaluate the quality of binaural-based separation.

SDR and SIR scores obtained with our algorithm are given together with upper and lower bounds. The upper bound corresponds to the SDR of the ground truth mask or oracle mask [START_REF] Yılmaz | Blind separation of speech mixtures via time-frequency masking[END_REF], which is set to 1 at every spectrogram point in which the target signal is at least as loud as the combined other signals and 0 everywhere else. The lower bound corresponds to the SDR and SIR in the original mixture. The level mask described in section 3.2 was applied to all signals so that only separation of significant observations could be evaluated.

For tests made on simulated spectrograms, since there is no 1D signal to compare with, the separation was evaluated by a pointwise = operation between estimated and oracle binary masks: we define the mask error by the ratio of points in the estimated mask that differs from the oracle mask.

Simulated Data

To validate our model we first tested the algorithm on simulated data. We simulated ILD and IPD spectrograms with F = 1024 frequency channels from 0 to 12, 000Hz, and 10 to 126 significant observations per channel (M f randomly drawn for each f ). First, K source positions x1 . . . xK are randomly drawn from X . Then, each spectrogram point (f, m) is randomly assigned to one of the K sources, and we generate ILD and IPD observations α f,m and φ f,m using the distributions N (µ f (x k ), σ 2 f (x k )) and ∠N (ξ f (x k ), ρ 2 f (x k )), where σ 2 f (x k ) and ρ 2 f (x k ) correspond to ILD and IPD variances of white noise emitted from x k , and are estimated from the dataset. A total of 200 mixtures composed of 2 to 5 sources were generated, and K was set to the correct number of sources for each test. Table 1 shows the percentage of correctly localized sources, the mean mask error obtained over all sources, and the mean mask error obtained with randomly generated binary masks. These results show that our model is very well suited for both separation and localization tasks. A very high localization accuracy is achieved for mixtures composed of 2 and 3 sources. When the number of sources is higher, the number of observations per source decreases while the number of local maxima of the log-likelihood function increases, leading to poorer results.

Real Mixtures from Learned Positions

Three sounds were used to test the algorithm with real data: male speech, female speech, and flute melody (see section 4). Mixtures were obtained by summing raw binaural signals from the dataset, corresponding to randomly drawn positions in X . ILD and IPD spectrograms were computed from these mixtures, and only significant observations corresponding to points with a higher acoustic level than the corresponding background mixtures were kept, as described in section 2.

First, 200 mixtures for each possible pair of test sounds were generated, resulting in 1, 200 localization-separation tasks. Mean results obtained with our method for all these tasks as well as mean results for correctly localized sources only are shown in Table 2. SDR and SIR scores are compared to those of the original mixture and oracle mask.

A very high localization rate is obtained, with 84.6% of the 1, 200 test sound sources localized with less that 2 • error in both azimuth and elevation. In addition, significant improvements of SDR and SIR ratios over the original mixtures are achieved. This is particularly true for SIR ratios that almost reached oracle ratios in some tests. This shows that although our algorithm generates some artifacts in the target sound signal during the spectral masking process, it is able to considerably reduce the volume of the interferer and thus significantly improve the perceived quality. Finally, we note that SDR and SIR ratios are always better when the algorithm could correctly localize the sound source, which demonstrates the importance of localization cues for sound sources separation.

A second experiment was made with 200 mixtures of the three test sounds emitting altogether from random positions in X , resulting in 600 localization-separation tasks. Results are displayed in a similar fashion in Table 3. Although performances significantly decrease for this very challenging task (a 1s mixture of 3 equally loud sounds is hard to decipher even for humans), we note that the algorithm could still accurately localize almost half of the 600 individual sources, while significantly improving their SDR and SIR with respect to the original mixture.

Real Mixtures from Unlearned Positions

The last experiment is also the most challenging one. Indeed, we tested the ability of our algorithm to separate sound sources emitting from positions outside the training set. More precisely, we made several test recordings in which two loud speakers were simultaneously emitting different sounds while the robot stayed in its reference position (0 • , 0 • ). One of the loud speaker was placed in a frontal position corresponding to the training dataset, while the second one was manually placed in 21 side positions around the robot, with a 10 • to 90 • azimuth distance and 0 • to 30 • elevation distance. These experiments were repeated for the six possible pairs of test sounds, resulting in 252 separationlocalization tasks. The frontal loud speaker was correctly localized at (0 • , 0 • ) in 95.2% of the mixtures tested. Localization performances for the side loud speaker were not evaluated, as no ground truth was available. Table 4 shows the mean SDR and SIR scores of each source obtained with our approach, the original mixture, and the oracle mask.

Despite a decrease of performances with respect to sound mixtures from learned positions, the significant improvement of SIR obtained in realistic cocktail-party like scenarios puts forward our approach as a promising tool for auditory human-robot interactions.

CONCLUSION AND FUTURE WORK

Traditionally, computational auditory scene analysis was addressed with either a close-range or an array of microphones and using simulated or anechoic audio data. We propose to bridge the gap between constrained and unconstrained audio analysis and to apply CASA to HRI. We propose a system integrating the audio-motor abilities of a robot within a unified framework, and performing sound-source separation and localization in a realistic cocktail-party like scenario. By providing a genuine audio-motor database and presenting encouraging results obtained from these data, we presented a benchmark for the unexplored field of sensorimotor learning for robot audition.

One of the most interesting and promising directions will be to extend our model to a continuous space of sound source positions. This could be done using the manifold structure of interaural parameters studied in detail in [START_REF] Deleforge | Learning the direction of a sound source using head motions and spectral features[END_REF]. By approximating this manifold by local tangent spaces, the size of the training set could be considerably reduced, thus speeding up the M-step, while improving the localization of sound sources from unknown places. Dynamic models incorporating moving sound sources and head movements could also be included based on this idea. Finally, a "garbage" source class could be added to our model in order to better deal with background and non-point sources. We believe that these ideas combined with careful algorithm and software optimization could lead to a novel robot hearing paradigm within the emerging field of human-robot interaction.

  ∈ C the spectrogram value at frequency-time point (f, t) of a signal emitted by a sound source k and with s

  the spectrogram values perceived by the leftand right-microphone respectively. The spectrogram of the emitted acoustic level is defined by: a (k) f,t = 10 log(|s

Fig. 2 -

 2 Fig. 2-(a) and (b) show an example of spectrograms of emitted and perceived acoustic levels.

Figure 2 :

 2 Figure 2: Spectrograms corresponding to a male utterance emitted from the left-hand side of the binaural head.

k=1 3 :

 3 Θ (0) := initialize 4: p := 0 5: while !converged do 6: p := p + 1 7: {r (p) f,m,k } := E-step(Θ (p-1) ) 8: Θ (p) := M-step({r

Figure 3 :

 3 Figure 3: A binaural head is placed onto an agile device that can perform precise and reproducible pan and tilt motions (left). The emitter (a loud-speaker) is placed in front of the robot head at approximately 2.7 meters (right).

Table 1 :

 1 Mean localization and separation results for 200 simulated mixtures of 2 to 5 sources.

	K	Correctly Localized(%)	Estimated Mask Random Mask Error(%) Error(%)
	2	99.2	20.7	50.0
	3	91.2	25.4	55.6
	4	54.1	28.9	62.5
	5	14.4	29.3	68.0

Table 2 :

 2 Mean angle error, SDR (Source-to-Distortion Ratio) and SIR (Source-to-Interferences Ratio) for real mixtures of 2 sound sources.

		Ang Err( • ) SDR(dB) SIR(dB)
	Original	-	0.05	0.05
	Us (All sources)	11.5	4.11	14.3
	Us (Loc: 84.6%)	0.18	5.17	15.2
	Oracle	-	17.9	39.6

Table 3 :

 3 Mean angle error, SDR and SIR for real mixtures of 3 sound sources.

		Ang Err( • ) SDR(dB) SIR(dB)
	Original	-	-3.61	-3.61
	Us (All sources)	35.5	-4.03	3.21
	Us (Loc: 45.2%)	0.18	0.09	7.29
	Oracle	-	14.5	32.7

Table 4 :

 4 Mean SDR and SIR of frontal and side speakers

		Frontal	Frontal	Side	Side
		SDR(dB) SIR(dB) SDR(dB) SIR(dB)
	Original	-0.28	-0.28	0.32	0.32
	Us	-1.09	3.01	-3.30	5.69
	Oracle	18.29	40.64	18.63	40.13

http://perception.inrialpes.fr/~Deleforge/CAMIL_ Dataset/


Jordi Sanchez-Riera
email: jordi.sanchez-riera@inria.fr

Jan Čech
email: jan.cech@inria.fr

Radu Horaud
email: radu.horaud@inria.fr

Action Recognition Robust to Background Clutter by Using Stereo Vision

An action recognition algorithm which works with binocular videos is presented. The proposed method uses standard bag-of-words approach, where each action clip is represented as a histogram of visual words. However, instead of using classical monocular HoG/HoF features, we construct features from the scene-flow computed by a matching algorithm on the sequence of stereo images. The resulting algorithm has a comparable or slightly better recognition accuracy than standard monocular solution in controlled setup with a single actor present in the scene. However, we show its significantly improved performance in case of strong background clutter due to other people freely moving behind the actor.

Introduction

An extensive research has been done in action recognition throughout recent years, which is well documented in survey papers [START_REF] Weinland | A survey of vision-based methods for action representation, segmentation and recognition[END_REF][START_REF] Poppe | A survey on vision-based human action recognition[END_REF]. Most of the methods work with monocular videos only. Very successful methods use image retrieval techniques, where each video sequence is represented as a histogram of visual words [START_REF] Laptev | On space-time interest points[END_REF], and large margin classifier is then used for recognition.

In particular, spatiotemporal interest points [START_REF] Laptev | On space-time interest points[END_REF] are detected in the image sequence. These points are described by a descriptor HoG (Histogram of Gradients)/HoF (Histogram of Optical Flow) [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] which captures the surrounding of an interest point. The descriptors are quantized by K-means clustering and each videoclip is represented as a histogram with K bins. Support Vector Machine is then used for classification.

Further research to improve the recognition accuracy went in the direction of densifying the interest points and enhancing the local descriptors. The interest points employed in [START_REF] Laptev | On space-time interest points[END_REF] are spatiotemporal extensions of a Harris corner detector, i.e. locations in a video stream having large local variance in both spatial and temporal dimensions, representing abrupt events in the stream. This is in order to achieve high repeatability of the detection. However, such points are quite rare and important relevant information can be missed. Therefore there were alternatives to these interest points, e.g. based on Gabor filters [START_REF] Dollár | Behavior recognition via sparse spatio-temporal features[END_REF][START_REF] Bregonzio | Recognising action as clouds of space-time interest points[END_REF], or even simply using a regular dense sampling [START_REF] Wang | A spatio-temporal descriptor based on 3D-gradients[END_REF] to reach higher coverage, or a hybrid scheme by [START_REF] Tuytelaars | Dense interest points[END_REF], which start by dense sampling and optimize the position and scale within a bounded area in order to increase the coverage and preserve the repeatability of the interest points. An extension of the original HoG/HoF descriptor was proposed e.g. by spatiotemporal gradients [START_REF] Klaser | A spatio-temporal descriptor based on 3D-gradients[END_REF], or motion boundary histograms [START_REF] Wang | Action recognition by dense trajectories[END_REF].

However these methods can be quite sensitive to background clutter present in populated scenes, since interest points are detected not only in the actor but on the background as well. This causes the global histogram representation to be corrupted and the accuracy is significantly decreased.

Stereo vision or multiple view vision have not been much used in action recognition. Using stereo, the existing methods typically try to make the algorithm insensitive to a camera viewpoint [START_REF] Roh | View-independent human action recognition with volume motion template on single stereo camera[END_REF]. Similarly [START_REF] Weinland | Action recognition from arbitrary views using 3D exemplars[END_REF] uses a special room and a multi-camera setup to construct viewpoint invariant action representation, and [START_REF] Yan | Learning 4D action feautre models for arbitrary view action recognition[END_REF] incorporate temporal information to the multi-view setup. Work [START_REF] Uddin | Human activity recognition using body joint-angle features and hidden Markov model[END_REF] uses the depth map obtained by stereo matching to fit an articulated body model and use joint trajectories for action recognition.

An alternative to stereo vision is using RGB-D sensor, which provides a depth image besides the color/intensity image. It is based on time-of-flight or structured light technology. This research is vivid nowadays due to the recent irruption of Kinect device. For instance [START_REF] Holte | View-invariant gesture recognition using 3d optical flow and harmonic motion context[END_REF] constructs 3D motion primitives from a cloud of 3D points. Work [START_REF] Li | Action recognition based on a bag of 3D points[END_REF] extends 2D silhoutte by projection of the point cloud into three orthogonal planes. In [START_REF] Zhang | 4-dimensional local spatio-temporal features for human activity recognition[END_REF] the authors uses local interest point descriptors which are computed from spatiotemporal image and depth gradients for each pixel of a spatiotemporal neighbourhood of interest points. Since the neighbourhood is large, they use PCA for dimensionality reduction prior to quantization. In [18], spatiotemporal interest points are divided into different layers based on depth and a multichannel histogram is created. Another direction is to estimate the body skeleton from the depth data. Commercially successful real-time game controller uses skeleton model from body part labelling of depth data of Kinect [19]. Joint trajectories are used for action or gesture recognition in e.g. [20,21]. However, for some applications such active sensors are not suitable. For example, in outdoor setup or in a scenario with multiple autonomous robots whose active sensors would interfere to each other.

Therefore we propose a simple stereo vision based method, which can focus the algorithm to an active actor while disregarding the background activity based on completely passive system, see Fig. 1. Our contribution is extending the original successful action recognition framework [START_REF] Laptev | On space-time interest points[END_REF] with descriptors based on stereo-vision and the scene-flow. We observed a significant improvement of the proposed method in the robustness to the perturbations due to the uncontrolled motion of other people behind the actor.

The rest of the paper is structured as follows: The proposed method is described in detail in Sec. 2. Experimental validation which includes a comparison with the state-of-the-art algorithm is presented in Sec. 3. Finally, Sec. 4 concludes the paper.

Method Description

Before we give details on the proposed descriptor, we briefly revise the bag-ofwords (BoW) paradigm for action recognition. Following [START_REF] Laptev | On space-time interest points[END_REF] it requires to:

1. Collect a set of local descriptors associated to the interest points for image frames of all training action video clips. 2. Apply clustering algorithm to these descriptors, for instance, K-means.

3. Quantize the descriptor to get the 'visual words'. For each descriptor, assign label according to its nearest cluster centroid. 4. Represent a video clip as a K-bins histogram of the quantized descriptor ('bag of words'). 5. Train a classifier with these histograms, for instance, SVM.

In Steps 1-3, the the visual word vocabulary (or the codebook) is constructed. The dimensionality of the local descriptor is typically high and the space is consequently sparse, that is why it is represented by K clusters of observed data. In Steps 4-5, a compact (K-length vector) representation of training videoclips with annotated labels is used to train a classifier. The 'bag of words' representation encodes a relative frequence of occurences of the quantized desriptors and it turns out to be discriminative among action classes. Later for recognition, an unknown videoclip is first represented as the K-length histogram and then it is fed to the classifier which assigns the class label.

We follow exactly this framework, except for the Step 1. Unlike the monocular HoG/HoF descriptor [START_REF] Laptev | On space-time interest points[END_REF], we introduce a new descriptor based on the Scene-Flow [22].

Local Descriptor based on the Scene-flow

The Scene-flow is a 3D extension of the optical flow. We represent a scene-flow as depth and optical flow, which together with a camera calibration is equivalent to a vector field of 3D position and associated 3D velocities of reconstructed surface points. This intrinsic representation is potentially less sensitive to the changes of texture and illumination in the action dataset than the representation which relies solely on the intensity images. Moreover, with the notion of depth, it is straightforward to focus the actor performing the action to be recognized while discarding any activity from the background clutter. We assume the action performing actor is the person which is the closest to the camera. We believe this is a reasonable assumption, which is typically the case of human-robot interaction or movies.

The proposed descriptor is constructed as follows, see Fig. 2: 1. Get the synchonized sequences of the left I l and right images I r . For each frame compute the disparity map D and optical flow maps F h , F v by the algorithm [22]. 2. Find the actor's face with a face detector [23]: (x 0 , y 0 ) = FD(I l ). In case of multiple faces detected, the one with the highest disparity d 0 = D(x 0 , y 0 ) is selected 1 . In case no face is detected, if the actor turns or the detector miss the face, we simply assume a previous face position. [START_REF] Laptev | On space-time interest points[END_REF]. Segment the scene using disparity and optical flow: (1) Only pixels with magnitude of optical flow greater than zero are considered, (2) Only pixels with disparity greater or equal to the disparity of the actor's face are considered. So the set of valid pixels

S = {(x, y) : F h (x, y) 2 + F v (x, y) 2 > 0 and D(x, y) > d 0 -µ},
where µ = 5 is a small margin to ensure the entire actor's body is included. 4. At each reconstructed pixel passing the above test (x, y) ∈ S , the local descriptor is 5-dimensional only: Notice the face-normalized position of the pixels, brings a kind of global information into the local descriptor. Following the BoW procedure described above, after building the codebook and subsequent quantization of pixel descriptors, the resulting histograms of their occurrences in the action video sequence intuitively encodes the activity of actor's body parts in the sense of 3D motion. See Fig. 3 for an illustration.

L(x, y) = x -x 0 , y -y 0 , D(x, y) -d 0 , F h (x, y), F v (x, y) .

Experiments

To evaluate the performance of the proposed binocular method and compare it with a state-of-the-art monocular method [START_REF] Laptev | On space-time interest points[END_REF], we use the Ravel2 dataset [24]. The Ravel dataset consists of 7 actions (talk phone, drink, scratch head, turn around, check watch, clap, cross arms) performed by 12 actors in 6 trials each. First 3 trials are with stable static background without other people in the scene (we denote as 'Controlled'), while next 3 trials are performed with motion background clutter due to arbitrary activity of the people behind the actor (we denote as 'Clutter'). See Fig. 4 and Fig. 5 for respective examples. The dataset is challenging due to the strong intra-class variance, strong dynamic background in the 'Clutter', and unstable lighting conditions.

We will show results of two baseline algorithms. The first one is the algorithm described in [START_REF] Laptev | On space-time interest points[END_REF] works with monocular (left camera) stream only and uses the sparse spatiotemporal interests points and HoG/HoF descriptors, we denote as 'STIPs'. The other baseline is the same algorithm, however we ran it in both left and right camera sequences, matched the detected points along the epipolar lines, and removed the interest points which have smaller disparity than the Fig. 4: Ravel dataset examples -controlled setup. Note that different actors perform the same action quite differently as for example in "cross arms". Actions: "cross arms", "check watch", "scratch head", "cross arms", "talk phone", "cross arms", "scratch head", "clap".

disparity of the actor's face. The motivation behind is to remove the irrelevant interest points detected on the background clutter. The rest of the algorithm [START_REF] Laptev | On space-time interest points[END_REF] remains the same. We call this algorithm 'STIPs-stereo'. The proposed method described in Sec. 2, is denoted as '5DF'.

The codebook was built in a sequence of a single actor, namely 'character-09'. This actor was not later used either for learning a classifier or for testing. We believe a single actor performing the same set of actions as all other actors sweeps the space of local descriptors is enough and also K-means algorithm is run only once and not in the leave-one-out loop (see later), which would be too time consuming. The size of the codebook K was optimized for all the methods in the logarithmic range from K = 10 to K = 10000 and the optimum was found for K = 1000, the same for all the methods.

Learning a classifier and testing was performed in a standard leave-one-actorout scenario. One actor was removed from the set, the linear SVM classifier was trained in the sequences of remaining actors and then tested on the sequence of the left actor and this was repeated for all actors. The recognition rate reported is the average error over all actors.

Results are shown in Tab. 1. We can see the proposed method (5DF) performs comparably in the setup when there is a single actor in the scene only. This proves the proposed descriptor computed in the meaningful semi-dense locations is informative. Furthemore, we can see the recognition accuracy of the proposed method does not drop much in cases of the background clutter of other people freely moving behind the actor. This demonstrates that the algorithm can properly focus the active actor while disregarding the background activity using the depth information from stereo. The monocular baseline method [START_REF] Laptev | On space-time interest points[END_REF] (STIPs) is naturally very sensitive to this type of the background clutter. The algorithm cannot distinguish the informative interest points of the clutter from Table 1: Recogntion accuracy of the tested methods. The proposed (5DF) method has comparable results with state-of-the-art method (STIPs) in the controlled setup with only one actor in the scene, while it much less sensitive to the strong dynamic background clutter. The other baseline (STIPs-stereo) is less sensitive to the background by using the stereo information, however due to insufficient coverage of interest points the recognition accuracy is lower.

corresponding descriptors on other people in the scene, which contaminates the histograms and the recognition accuracy drops significantly. The second baseline (STIPs-stereo), which attempts to remove the interest points detected on the background by stereo matching, is less sensitive to the background clutter, however its recognition accuracy is slightly lower for 'controlled' setup. The reason is that the sparse spatiotemporal interest points become even sparser, since the stereo matching may discard also points on the foreground due to matching ambiguity. Notice that in STIPs method, we have about 10 interest points per frame, but in our method we have about 10000 locations per frame where descriptors are computed.

For more insight, we show confusion matrices of both methods for both 'controlled' and 'clutter' setups, see Fig. 678. For instance, we can see that scratch head is confusing with talk phone. This is not so surprising since these actions starts with the hand at the level of the pocket and is directed to the head, where the difference is whereas it remains static (talk phone) or moving (scratch head). Again, there is significantly much less confusion in case of the background clutter in the proposed binocular method compared to the state-of-the-art method which only uses a monocular video. This corroborates that stereo vision brings an important extra information.

Conclusion

We presented an action recognition method which uses the scene-flow computed from binocular video sequences. Experimentally we proved that the extra information from stereo significantly improves the recognition accuracy in the presence of strong background clutter. The proposed method requires the actor's face is detected in majority of the frames. We expect that a tracker with a motion model would help to localize the face if it is turned away. Future work includes an elaboration on the design of the local descriptor. Combination of the local descriptor with the proposed one could further improve the recognition accuracy.

Fig. 1 :

 1 Fig. 1: Example of data for one sequence. The input data consists of sequences of (a) Left and (b) Right images. The maps of (c) disparity, (d) horizontal, (e) vertical component of the optical flow computed by algorithm [22]. The maps are color-coded: gray color means unassigned value, for disparity warmer colors corresponds to points closer to the camera, for optical flow warmer colors corresponds to motion to the left and up respectively.

Fig. 2 :

 2 Fig. 2: Construction of the proposed descriptor. The actor's face is detected from the left input image (a). The raw disparity map (b) is segmented, such that all pixels having the lower disparity than the actor's face are discarded (c). The descriptor is then computed for all remaining pixels undergoing non-zero motion, such that it consists of the pixel's position relative to the face, it's disparity (d), and horizontal (e) and vertical (f) components of optical flow.

Fig. 3 :

 3 Fig. 3: Histograms of visual words and corresponding assignment to pixels for frames of two actions: clap (top) and turn-around (bottom). The color encodes the indices of visual words 1, . . . , K. The coloring is such that similar visual words have similar color. We can see typical visual words occuring during the actions.

Fig. 5 :

 5 Fig.5: Ravel dataset examples -cluttered setup. Actions: "turn around", "clap", "talk phone", "talk phone", "turn around", "drink", "check watch", "drink". Note different illumination conditions.

Fig. 6 :

 6 Fig. 6: Confusion Matrix for the proposed method (5DF) for a) Controlled and b) Cluttered setup.

Fig. 8 :

 8 Fig. 8: Confusion Matrix for the state-of-the-art method [3] (STIPs) for a) Controlled and b) Cluttered setup.

The disparity of the face is estimated as an average disparity inside the bounding box obtained from the face detection. The center of the bounding box is the pixel (x0, y0).

http://ravel.humavips.eu

Acknowledgements. This research was supported by EC project FP7-ICT-247525-HUMAVIPS.