Stéphane Lathuilière
email: stephane.lathuiliere@inria.fr

Georgios Evangelidis

Radu Horaud

Recognition of Group Activities in Videos Based on Single-and Two-Person Descriptors *

Group activity recognition from videos is a very challenging problem that has barely been addressed. We propose an activity recognition method using group context. In order to encode both single-person description and two-person interactions, we learn mappings from highdimensional feature spaces to low-dimensional dictionaries. In particular the proposed two-person descriptor takes into account geometric characteristics of the relative pose and motion between the two persons. Both single-person and two-person representations are then used to define unary and pairwise potentials of an energy function, whose optimization leads to the structured labeling of persons involved in the same activity. An interesting feature of the proposed method is that, unlike the vast majority of existing methods, it is able to recognize multiple distinct group activities occurring simultaneously in a video. The proposed method is evaluated with datasets widely used for group activity recognition, and is compared with several baseline methods.

Introduction

Human activity recognition has been an active topic in computer vision over the past years. Although considerable research has been devoted to the recognition of singleperson [START_REF] Turaga | Machine recognition of human activities: A survey[END_REF][START_REF] Poppe | A survey on vision-based human action recognition[END_REF][START_REF] Wang | Action recognition with improved trajectories[END_REF][START_REF] Lan | Beyond Gaussian pyramid: Multi-skip feature stacking for action recognition[END_REF] and two-person [START_REF] Ryoo | Recognition of composite human activities through context-free grammar based representation[END_REF][START_REF] Ryoo | Spatio-temporal relationship match: Video structure comparison for recognition of complex human activities[END_REF] activities, less attention has been paid to group activities. The latter is however important for a number of applications, e.g. video surveillance, video monitoring, human-robot interaction and video indexing, etc.

In this paper we aim to recognize activities that are performed jointly by groups of people, such as dancing, talking and so forth. More precisely, we propose a model that Figure 1: High-dimensional feature vectors are mapped onto a low-dimensional space that encodes both singleperson and two-person descriptors. Based on these descriptors, a group-activity label is assigned to each person by solving an energy optimization problem. can handle videos that contain one or even several distinct group activities that occur either simultaneously or in sequence. Such a problem cannot be solved by simply considering multiple instances of single-person activities as long as the collective context carries out important information [START_REF] Choi | What are they doing?: Collective activity classification using spatio-temporal relationship among people[END_REF]. As recently shown, the inter-person distance, the motion, and the relative pose of group members seem to be important cues for good recognition performance [START_REF] Sun | Activity group localization by modeling the relations among participants[END_REF][START_REF] Lan | Discriminative latent models for recognizing contextual group activities[END_REF][START_REF] Choi | Discovering groups of people in images[END_REF]. However, the way these cues should be combined together is crucial for robust recognition.

In this paper we cast the group-activity recognition problem into a structured-output labelling problem that is solved within an optimization framework. A two-stage mapping is used to obtain low-dimensional features and, in turn, to define the unary/pairwise potentials of the energy terms, e.g. Fig. 1. First, spatiotermporal motion features, e.g. [START_REF] Wang | Action recognition with improved trajectories[END_REF] are used in conjunction with bounding boxes which are provided in advance (person detection is not addressed in this paper. Second, we carefully investigate the two-person geometry, motion and relative pose, in order to describe highlevel cues such as persons A and B face each other, or stand side-by-side, or move in the same direction'. These descriptions are further encoded into pose-activity words in an unsupervised manner. Third, such a multi-stage mapping pro-vides low-dimensional yet meaningful interaction descriptors that are eventually used (along with single-person descriptors) to define unary and pairwise potentials of an energy function. The parameters of the latter are learned using structured SVM. It is worth noticing that the estimation of person poses needed to compute two-person descriptors is only performed for model training and not for testing (recognition).

The remainder of the paper is organized as follows. Sec. 2 discusses the related work. In Sec. 3, the energy function is presented. Sec. 4 details the pairwise interaction description, while model learning is explained in Sec. 5. Sec. 6 discusses the features that encode the raw data and Sec. 7 presents the experimental results. Finally, Sec. 8 concludes this work.

Related Work

Early human action recognition methods could only handle simple activities performed by a single person in in controlled environments, e.g. [START_REF] Schuldt | Recognizing human actions: a local SVM approach[END_REF]. More complex scenarios have been subsequently addressed, such as presence of occlusions, changing illumination conditions, moving cameras, dynamic background, e.g. [START_REF] Marszalek | Actions in context[END_REF]. We note that challenging single-person activities can be dealt with using sophisticated feature representations, e.g. [START_REF] Wang | Action recognition with improved trajectories[END_REF]. At the same time, significant effort has been put toward using contextual information to improve activity and object recognition [START_REF] Rabinovich | Objects in context[END_REF][START_REF] Murphy | Using the forest to see the trees: a graphical model relating features, objects and scenes[END_REF]. Such strategies benefit from using the global image content, thus not suffering from low-quality appearance, small objects, or occlusions. The object-action context is addressed in [START_REF] Kjellström | Simultaneous visual recognition of manipulation actions and manipulated objects[END_REF][START_REF] Marszalek | Actions in context[END_REF][START_REF] Gupta | Observing humanobject interactions: Using spatial and functional compatibility for recognition[END_REF][START_REF] Yao | Recognizing human-object interactions in still images by modeling the mutual context of objects and human poses[END_REF][START_REF] Jain | What do 15,000 object categories tell us about classifying and localizing actions?[END_REF] while spatial coherence constraints may be enforced as well [START_REF] Gupta | Observing humanobject interactions: Using spatial and functional compatibility for recognition[END_REF].

When it comes to group activity, the recognition performance benefits from a cross-person context [START_REF] Amer | A chains model for localizing participants of group activities in videos[END_REF][START_REF] Odashima | Collective activity localization with contextual spatial pyramid[END_REF][START_REF] Ryoo | Stochastic representation and recognition of high-level group activities[END_REF][START_REF] Lan | Beyond actions: Discriminative models for contextual group activities[END_REF][START_REF] Sun | Activity group localization by modeling the relations among participants[END_REF]. As a first attempt, [START_REF] Choi | What are they doing?: Collective activity classification using spatio-temporal relationship among people[END_REF] showed that the collective behavior, via modeling relative poses of persons, improves the classification of group activities. While this framework can deal with multiple group activities that are performed simultaneously, the classification is still done individually, i.e. for each detected person. Subsequent work focused on the simpler scenario of a single group activity per frame. Any person interaction description can then be simply integrated in the model, e.g. [START_REF] Lan | Beyond actions: Discriminative models for contextual group activities[END_REF][START_REF] Sun | Activity group localization by modeling the relations among participants[END_REF][START_REF] Hajimirsadeghi | Visual recognition by counting instances: A multi-instance cardinality potential kernel[END_REF] and not only in feature encoding. This led to the introduction of pairwise potentials within energy-based formulations, so that persons involved in the same activity are jointly considered. Group structure analysis and selection of meaningful pairs of persons [START_REF] Lan | Discriminative latent models for recognizing contextual group activities[END_REF][START_REF] Sun | Activity group localization by modeling the relations among participants[END_REF] seem to further help at the expense of higher computational complexity. However, these methods relies on tracking each person, that can be inaccurate in crowded scenarios. Note that [START_REF] Choi | Discovering groups of people in images[END_REF] models groups of humans in still images. More recently, deep structured models have been used for single group activity [START_REF] Deng | Deep structured models for group activity recognition[END_REF][START_REF] Ibrahim | A hierarchical deep temporal model for group activity recognition[END_REF]. While these methods reach satisfying results, their generalization to the multiple activity case is not straightforward. Actually, when several activities can occur simultaneously, the size of the label space grows exponentially with the number of persons. Thus, complex or restrictive formulations cannot be used in practice. Temporal information has been also used in [START_REF] Cupillard | Group behavior recognition with multiple cameras[END_REF][START_REF] Choi | A unified framework for multitarget tracking and collective activity recognition[END_REF][START_REF] Khamis | Combining perframe and per-track cues for multi-person action recognition[END_REF] to deal with issues that arise from the social context, e.g. occlusions, gathering etc. In a similar manner, [START_REF] Li | Learning multi-modal densities on discriminative temporal interaction manifold for group activity recognition[END_REF] considers temporal interactions between persons to analyse team sports. In some sense, however, these two strategies contradict each other since the finer the group modelization, the more difficult its integration into a temporal framework. Conversely, adding the time dimension into graph formulations leads to computationally demanding solutions, in particular when several groups of different activities appear at the same time. As a consequence, one may need to properly combine these two approaches.

Instead, we propose to investigate the two-person (or person-person) geometry, the motion and the relative poses in order to describe precisely the interactions. The proposed model exploits temporal information only when building descriptors, thus avoiding inference on temporal models. Moreover, the proposed descriptors can be used efficiently and without computationally demanding optimization algorithms in difficult scenarios in which several group activities occurs simultaneously. In addition, and unlike [START_REF] Choi | What are they doing?: Collective activity classification using spatio-temporal relationship among people[END_REF][START_REF] Choi | Understanding collective activities of people from videos[END_REF][START_REF] Choi | Discovering groups of people in images[END_REF], body poses are used for training purposes only, so that the body pose is not required at test time to estimate our pairwise descriptor.

Modeling Group Activity

Provided that I < I max persons are detected in an image, we extract high-dimensional single-person feature vectors, x i ∈ R D , from each bounding box and highdimensional pairwise feature vectors, y ij ∈ R F , from each pair of boxes, so that the sets X = {x i | 1 i I} and Y = {y ij | 1 i, j I, i = j} are available. The goal is to find a set of activity labels A = {a i | 1 i I} (one label per detected person) where a i ∈ L = {l m | 1 m M }, that is an M -activity label set. To this end, we define an energy function E(X, Y , A) and we seek the optimizer A * that best fits with the features:

A * = argmax A∈A I E(X, Y , A), (1)
where A I is the activity label set. The energy function is defined as:

E(X, Y , A) = I i=1 Ψ 1 (x i , a i ) individual potential + I×I i =j Ψ 2 (y ij , a i , a j ) pairwise potential + Ψ 3 (A) regularizer . (2)
The individual potential Ψ 1 (x i , a i ) models the compatibility of label a i with x i ; the pairwise potential Ψ 2 (y ij , a i , a j ) models the compatibility of a pair of labels (a i , a j ) with y ij ; the term Ψ 3 (A) enforces grouping, i.e., by assigning the same activity label to multiple persons. A log-linear model leads to the following energy function:

E(X, Y , A) = I i=1 w 1 ai φ 1 (x i ) + I×I i =j w 2 ai,aj φ 2 (y ij )1(a i = a j ) + w 3 φ 3 (A). (3)
The indicator function in the pairwise term makes the sum valid for the pairs of persons that perform the same activity. Grouping is then implicitly considered without adding group variables. Note that the parameters w 1 ai ,w 2 ai,aj and w 3 are learned on a training set (Sec. 5). As for the feature functions, they are defined as follows.

Single-person mapping φ 1 : Let B = {b i | 1 i I} be the discrete body poses of the detected persons where b i ∈ Π = {π q | 1 q Q} ⊂ [-π, π], i.e., Π denotes a set of possible discrete body poses. The poses are modeled by an angle between the body orientation and a reference orientation. As in [START_REF] Lan | Beyond actions: Discriminative models for contextual group activities[END_REF], φ 1 maps the high-dimensional feature space onto a low-dimensional space of posterior probabilities:

φ 1 (x i ) = P (b i = π q , a i = l m |x i ) (πq,lm)∈Π×L
The mapping φ 1 is estimated via a linear SVM on training data in the second stage of the pipeline. Two-person mapping φ 2 : This mapping projects a pairwise high-dimensional feature vector onto a lowdimensional space that describes the interaction between two detected persons, e.g., facing each other. The explicit design and estimation of φ 2 is one of the main contributions and it is explained in detail in Sec. 4.

Regularization term φ 3 : The regularizer is simply de-

fined as φ 3 (A) = 1(ν(A) = i) 1 i Imax , where ν(A) is the number of different activities in A. For instance, if A = {l 1 , l 3 , l 3 , l 5 }, then ν(A) = 3 and φ 3 (A) = (0, 0, 1, 0 • • • , 0).

Two-Person Mapping

As with the single-person case, we want to map the highdimensional pairwise feature space into a low-dimensional space of posterior probabilities, that is,

φ 2 (y ij ) = P (ξ ij = k|y ij ) k∈K (4)
where K is a pose-activity dictionary and ξ ij ∈ K defines the pose-activity word associated to feature y ij . However, the construction of such a dictionary is not straightforward as long as we need to learn relevant pose-activity words. Therefore, we learn this dictionary in an unsupervised manner using a spectral clustering method. Suppose two persons are observed in a frame. Using the method of [START_REF] Hoiem | Putting objects in perspective[END_REF], we extract their 3D positions P i and P j as illustrated in Fig. 2. We define an interaction descriptor p ij that encapsulates their relative distance and pose, as well as the associated activities: p ij = (d ij , θ i , θ j , a i , a j ), where d ij is the relative distance d ij = P i -P j 2 and θ i , θ j are the angles spanned between the line P i P j and the respective pose vectors. Fig. 2 (right) illustrates the geometry of the model. To learn the pose-activity words, we use a spectral clustering method [START_REF] Yu | Multiclass spectral clustering[END_REF] with a similarity function

δ(p ij , p kl ) = e -f (p ij ,p kl ) 2
, where f is defined by:

f p ij , p kl = min{2 -(1(a i = a k ) + 1(a j = a l )) + λ 1 (µ(θ i , θ k ) + µ(θ j , θ l )), 2 -(1(a i = a l ) + 1(a j = a k )) + λ 1 (µ(θ i , θ l ) + µ(θ j , θ k ))} + λ 2 |d ij -d kl | (5) with µ(θ, θ ) = min k∈{1,0,-1} |θ -(θ + 2kπ)|. (6)
The parameters in (6), λ 1 and λ 2 are weights and the min operator is used to make the function f invariant to 2π shift-ing and symmetric, that is, f p ij , p kl is equal to zero if p ij = p kl or if p ij = p lk . The value of f p ij , p kl increases with the distance and the pose/action difference. As a consequence, the spectral clustering method tends to group together pairs of persons with the same activity, same distances and same relative poses. Considering a training set of interaction descriptors pij , the spectral clustering algorithm provides a cluster assignment label ξij ∈ K for each descriptor. This label is used as a pairwise poseactivity word of the respective dictionary. As with singleperson mapping, φ 2 is learned via a linear SVM. At test time, the assignment labels ξ ij are not computed. Instead, φ 2 maps directly the high-dimensional representation onto the low-dimensional space where energy optimization is performed.

We illustrate our clustering approach on an example in order to clarify its contribution. For visualization simplicity we consider a dataset in which there is only one activity per image. As a consequence, a i = a j holds for all interaction descriptors p ij . After extracting each interaction descriptor, we apply the above described clustering method. Fig. 3 shows which cluster each interaction descriptor is assigned to. The interaction descriptors are displayed on one of the two plots according to the a i value. In the case of talking activity, most of points are gathered in a cyan cluster where (θ i , θ j ) ≈ (0, 0). This confirms the intuitive idea that the persons are likely to face each other when they are talking. On the contrary, the area (θ i , θ j ) ≈ (0, 0) is almost empty in the case of the queueing activity since the persons are supposed to be in single-file when they are in a queue. Moreover, the only parameter that distinguishes the two clusters of the queueing activity is the distance d ij .

Model Learning

We use structured support vector machine (SSVM) [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF] to estimate the vectors w 1 ai ,w 2 ai,aj and w 3 . To this end, Equation ( 3) is rewritten as

E(X,Y , A) = M m w 1 lm I i=1 φ 1 (x i )1(l m = a i ) + M m w 2 lm I×I i,j i =j φ 2 (y ij )1(a i = a j )1(l m = a i ) + w 3 φ 3 (A) = w φ(X, Y , A), (7)
Considering a set of N training images, From each of the Ĩ(n) bounding boxes, we extract the individual feature vectors X(n) and the pairwise feature vectors Ỹ (n) to build Figure 3: Clustering results obtained on dataset A' (see Sec. [START_REF] Choi | What are they doing?: Collective activity classification using spatio-temporal relationship among people[END_REF] for more details about the datasets) for two activities: talking, queueing. Each color represents one cluster and each point is an interaction descriptor. θ i and θ j are measured in radians and d ij in meters.

our training set { X(n) , Ỹ (n) , Ã(n) } 1 n N .
We estimate w * such that:

w * = argmin w 1 2 w 2 + C N n=1 max A∈A I(n) w φ( X(n) , Ỹ (n) , A) -w φ( X(n) , Ỹ (n) , Ã(n) ) + ∆( Ã(n) , A) (8)
Commonly, C is a hyper-parameter to adjust the SVM fitting, while ∆( Ã(n) , A) is the loss function that should penalize different labelings. Here, the following loss function

is used: ∆( Ã(n) , A) = Ĩ(n) i=1 1(ã (n) i = a i ).
This function counts the number of incorrect labels. It is deliberately not normalized to account the fact that the model can learn more from images with a large number of persons.

Feature Extraction

In this section we describe how we extract the individual (single-person) features x i and the pairwise features y ij , that is the first-level features of our pipeline (see Fig. 1). The individual features x i are built on two state-of-the-art feature extraction methods. The first one is based on point trajectories [START_REF] Wang | Action recognition with improved trajectories[END_REF] and catches the local body motion whereas the second one is based on histogram of oriented gradients (HOG) [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] and catches the posture of a person. While local spatial gradients are described by the trajectories, HOG descriptor carries complementary information that is missing from trajectory features and is mainly useful for static activities. For each bonding box, we extract a trajectorybased feature vector x traj i and a HOG feature vector x hog i . Two single-person mappings φ traj 1 and φ HOG 1 are learned. We then concatenate the two outputs (the low-dimensional vectors) to obtain φ 1 (x i ). Next, we explain how x traj i is built.

In order to use temporal information, we use the information from neighboring frames to build our box-wise descriptor. Thus, we propose to adapt the improved trajectory features of [START_REF] Wang | Action recognition with improved trajectories[END_REF]. Improved trajectories consist of tracking over time densely sampled image points at different scales, while the local volume around the trajectory is encoded via several descriptors: HOG, histogram of optical flow, motion boundary histograms and trajectory shape, thus leading in a vector of dimension D. However, to build our single-person feature vector x traj i , we proceed as follows. To describe a bounding-box of a person at frame t, we consider the interval [t -∆t, t + ∆t]. First, sampled points at frame t -∆t are tracked until frame t, and only trajectories that cross the bounding box of a person at frame t are kept. The remaining points are tracked until the frame t + ∆t. Fig. 4 illustrates trajectories that are kept at frame t while their end points are further tracked. Recall that this allows us to use temporal information without building a complete temporal model. In [START_REF] Wang | Action recognition with improved trajectories[END_REF], trajectories with low variance are rejected based on the assumption that they most likely belong to the background. Here, we keep all the trajectories since our region of interest (bounding box) does not contain lots of background information. Moreover, lack of motion seems to be informative for some activities like waiting or queuing. The dimension of the feature vectors is halved (D/2) via PCA. We then encode the individual features of each person into Fisher vectors of size κD using κ Gaussian components [START_REF] Sánchez | Image classification with the fisher vector: Theory and practice[END_REF].

We also use trajectory-based features to obtain the pairwise (two persons) feature vectors y ij . Considering two persons (and their bounding boxes) detected in frame t-∆t, we densely sample points and track them until frame t as with the individual features. Here, only trajectories that cross one of the bounding boxes at frame t are kept. As before, the remaining points are tracked until frame t + ∆t. We encode the descriptors into Fisher vectors using κ Gaussian components to obtain the pairwise vector y ij .

Experiments

We first describe the datasets used to evaluate our model and give the implementation details. Choi et al. first published a collective activity dataset in [START_REF] Choi | What are they doing?: Collective activity classification using spatio-temporal relationship among people[END_REF]. This dataset is composed of 44 videos with 5 activities: crossing, waiting, queueing, walking, talking, and 8 poses: right, frontright, front, front-left, left, back-left, back, back-right. We refer here to this dataset as dataset A. This dataset is used to evaluate our model for the problem of recognizing multiplegroup (distinct) activities. In order to compare with methods that can only deal with the single-group activity problem, two other datasets are used. Choi and Savarese published a new collective activity dataset [START_REF] Choi | A unified framework for multitarget tracking and collective activity recognition[END_REF], which we refer to as dataset B. This dataset consists in 33 videos with 6 activities: gathering, talking, dismissal, walking together, chasing, queueing. It is important to note that even if different activities can occur sequentially in a video of dataset B, all persons perform the same activity, which means that there is a single activity at each frame. Instead, dataset A may contain one or several distinct activities that are performed in parallel by different groups. At the same time, Choi and Savarese [START_REF] Choi | A unified framework for multitarget tracking and collective activity recognition[END_REF] published new annotations for dataset A in which they only consider one dominant activity per frame.In other words, if two groups of people perform two different activities, all the persons are considered as belonging to the activity of the largest group. We refer to this dataset as dataset-A'. As with dataset -emphA, eight pose labels are considered. Note that in both datasets A' and B, annotations are provided for one out of three frames, whereas one out of ten frames are annotated in dataset A.

As in [START_REF] Choi | A unified framework for multitarget tracking and collective activity recognition[END_REF], we split each dataset in training and test sets. One of the main difficulties with these datasets is that the classes are unbalanced. To figure out this issue, we use synthetic data augmentation as proposed in [START_REF] Chawla | SMOTE: Synthetic minority over-sampling technique[END_REF] and we produce new features that belong to the convex hull of the original features for minority classes, before training with SVM. While data augmentation does not directly apply to structured data, we augment the training examples of SSVM by perturbing the data of minor classes with Gaussian noise.

For the trajectory features, we use the values D = 426 and ∆t = 7 in order to track points over 15 frames, as recommended in [START_REF] Wang | Action recognition with improved trajectories[END_REF]. Then, Fisher vectors are built based on mixture of κ = 128 Gaussian components, while its dimensionality is reduced to 1500 using PCA, trained on a balanced set of features. As for pairwise features, we use weights λ 1 = 0.33 and λ 2 = 0.25. The weights have been chosen such that each term of the function f in ( 5) is equally important. The camera parameters used for the projectionfree map estimation have been chosen to give results which look realistic but we cannot evaluate the predicted positions quantitatively because ground-truth values are not provided. However the satisfying clustering results, e.g. Fig. 3, confirm the quality of the 3D position estimations. We chose K = 30 for dataset A and K = 12 clusters for datasets A' and B.

Evaluation of the Proposed Model

In this section, we show results from a series of experiments in order to make clear the contribution of each component of the proposed methodological pipeline. We compare six variants of the proposed framework:

• Single person: Here, only the individual terms are used, namely the first double sum of (2). In the case of dataset A this is equivalent to a single SVM estimation per person. For dataset B, since we use the prior that all the persons are perform the same activity, our method yields better performance than single SVM estimations.

• Static Pairwise: Here we use simpler pairwise features on the following grounds. Instead of using the trajectories, we use HOG features and a linear SVM to estimate person poses, then we deduce an interaction descriptor p ij . We then estimate (P (ξ ij = k|p ij )) k∈K with a second SVM. The posterior probabilities of each pose-activity word are then used as a two-person mapping φ 2 . This way, only one frame is used to extract the pairwise features.

• Without regularization: We use the model as described above but the regularization term in (2) is omitted.

Note that only the performance with dataset A is affected by this modification. As long as only one group activity occurs at a time in datasets A' and B, the regularizer has no impact.

• Full model: We make use of the full model, as proposed above.

• 4 meter threshold: Since [START_REF] Lan | Beyond actions: Discriminative models for contextual group activities[END_REF] observe some benefits from pruning, we test the full model along with the constraint that only person pairs whose distance is less than 4 meters are summed up in [START_REF] Chawla | SMOTE: Synthetic minority over-sampling technique[END_REF]. We consider this threshold as a reasonable assumption for the used activities, i.e., people that are at least4 meters away from each other are not taken into account. The performance of each one of these variants is evaluated on the three datasets. We summarize the results in Table 1. We can see that pairwise features drastically help the recognition in datasets A and A'. Fig. 5 illustrates this gain with a few examples. The gain is smaller for dataset B. This can be explained by the fact that dataset B constitutes an easier case compared to datasets A and A': stable camera, simpler action scenarios, fewer persons, and less overlap. It validates the principle that taking pairwise interactions into consideration helps in complex and natural environments. Unlike [START_REF] Sun | Activity group localization by modeling the relations among participants[END_REF][START_REF] Lan | Beyond actions: Discriminative models for contextual group activities[END_REF], the pruning strategy is not very helpful in our case, i.e., we do not consistently notice improvement when using a distance threshold. This may mean that other methods rely more on the used features compared to the proposed one. Our pairwise features succeed in capturing the information of the distance between persons, and Table 3: Classification accuracies based on per-class averages for single group-activity recognition. Fig. 6 plots the confusion matrices obtained with three methods and with our method. Note that it is not possi-ble to show the confusion matrix obtained with [START_REF] Amer | HIRF: Hierarchical random field for collective activity recognition in videos[END_REF] because only average results are provided in this paper. On dataset A, we compare our method with [START_REF] Sun | Activity group localization by modeling the relations among participants[END_REF] which uses extra annotations. The very good recognition scores obtained with the proposed method for the queueing activity shows the added value brought in by the use of pairwise features. As expected, both methods have difficulties in discriminating between crossing and walking. It seems difficult to reduce the confusion between these two categories in the absence of context information. On dataset A', the proposed method discriminates between crossing and walking slightly better then [START_REF] Choi | Understanding collective activities of people from videos[END_REF] which, again, uses extra annotations, namely interaction labels: facing-each-other, standing-side-by-side, and standing-still. In contrast, our method relies on two-person descriptors with no extra annotations.

On dataset B, the proposed method recognizes well activities for which motion plays an important role, such as chasing and walking. It performs less well whenever there are large motion variations during the activity, e.g. gathering. Nevertheless, our method yields a sensible result for this activity which is split into two parts: when the participants are spread out, the activity is labeled as walking, and once they get close to each other, it is labeled as talking. We believe that these are valid recognition results, although the activity is annotated as gathering. Nevertheless, activities with large dynamic variations might be dealt with by using a more sophisticated temporal model.

Conclusions

In this paper we proposed to model the group activity recognition problem as a structured labeling problem which is solved via an energy optimization framework. The proposed model can handle videos with either single or multiple group activities that occur simultaneously. Once high-dimensional feature vectors are calculated for each bounding-box and each for each pair of bounding boxes, we learn mapping functions in order to get meaningful representations in low-dimensional spaces. The geometry of the relationship between pairs of persons is included in the mapping function by learning a dictionary of relevant interactions. The geometry of the activity is thus taken into account without relying on pose estimations. We propose to use trajectory features to encode individual person description as well as the interactions between pairs of persons. The proposed framework outperforms several state-of-theart methods for the recognition of multiple group-activities and compares favorably with other methods for the recognition of a single group activity. Future work includes the further use of group features as well as adding a temporal model to the energy formulation.

Figure 2 :

 2 Figure 2: We estimate the projection-free 3D positions P i and P j considering that the persons are standing on the ground (left image). We then extract the angles θ i and θ j as well as the distance d ij . The right image shows a top-view of the geometry between two persons and their associated variables.

Figure 4 :

 4 Figure 4: Trajectory-based description of (pink) boundingbox at frame t: Points of frame t -∆t are densely sampled and tracked up to frame t. At time t, (red) trajectories out of the bounding box are rejected. The remaining end points of trajectories (green) are further tracked until t + ∆t. The local space-time area around the green trajectories is then encoded into the individual features x i .

Figure 5 :

 5 Figure 5: Example frames from sequences of dataset A with single person model (first row) and the proposed model (second row) . The color of the box represents the predicted activity (red for Crossing,yellow for Waiting, green for Queueing, blue for Walking, purple for Talking). On top of each bounding box, the letters show the ground truth activity (better viewed on screen)

Figure 6 :

 6 Figure 6: Confusion matrices for dataset A (left column), dataset A' (middle column) and dataset B (right column).

•

  Temporal smoothing: A median filter is applied on the full model output to smooth predictions. Only the past predictions are used within a 12-frame window. As we choose ∆t = 7 to extract the trajectories, a 26-frame window is used in total for each prediction. We test however the contribution of temporal smoothing only for datasets A' and B, since different activities occur simultaneously in dataset A and this would require to track all the persons.

		dataset A dataset A' dataset B
	Single person	56.3%	59.6%	74.6%
	Static pairwise	66.6%	73.8%	75.4%
	Without regularization	73.3%	-	-
	Full model	74.5%	79.8%	78.8%
	4 meter threshold	74.8%	78.9%	78.4%
	Temporal smoothing	-	81.5%	80.2%

Table 1 :

 1 Average per-class activity recognition accuracy on three datasets for the single and multiple group activity problems

* Funding from the European Union FP7 ERC Advanced Grant VHIA (#340113) is greatly acknowledged.

SSVM has learned that pairs with high distances have to be ignored. A temporal yet non-sophisticated smoother helps on dataset A' and B and even increases the performance of the proposed model.

Comparison with State of the Art Methods

Designing a benchmark for group activity recognition is difficult for the following reasons. Some authors use additional information/annotation for training or for testing, or they use different evaluation measures. For example, group labels are required to evaluate the method proposed in [START_REF] Sun | Activity group localization by modeling the relations among participants[END_REF]. Table 2 and Table 3 show the performance of several methods for the single and multiple group activity problems. We use the per-class average to compute the performance considering the unbalanced number of available classes for each example. In the case of multiple activity recognition (Table 2) our method is the second-best performing method. Note however that additional group annotations are used by [START_REF] Sun | Activity group localization by modeling the relations among participants[END_REF]. In the case of the recognition of single group activities (Table 3), our method also yields the second-best results. It should be noted that the best performing method, i.e. [START_REF] Amer | HIRF: Hierarchical random field for collective activity recognition in videos[END_REF], uses its own person detections which biases the comparison with the other methods.


Xiaofei Li

Laurent Girin

Radu Horaud

AUDIO SOURCE SEPARATION BASED ON CONVOLUTIVE TRANSFER FUNCTION AND FREQUENCY-DOMAIN LASSO OPTIMIZATION

Keywords: Source separation, convolutive transfer function, 1 -norm regularization

This paper addresses the problem of under-determined convolutive audio source separation in a semi-oracle configuration where the mixing filters are assumed to be known. We propose a separation procedure based on the convolutive transfer function (CTF), which is a more appropriate model for strongly reverberant signals than the widely-used multiplicative transfer function approximation. In the short-time Fourier transform domain, source signals are estimated by minimizing the mixture fitting cost using Lasso optimization, with a 1 -norm regularization to exploit the spectral sparsity of source signals. Experiments show that the proposed method achieves satisfactory performance on highly reverberant speech mixtures, with a much lower computational cost compared to time-domain dual techniques.

INTRODUCTION

In this paper we address the problem of multichannel audio source separation (MASS) from, possibly underdetermined, convolutive mixtures (the number of sensors is lower than the number of sources). This problem is often divided into two subproblems, which are both difficult: identification of mixing filters and estimations of source signals. This paper focuses on audio source estimation assuming that the mixing filters are either known or an estimation is available.

Most of convolutive source separation techniques are designed in the short time Fourier transform (STFT) domain. In this domain, the convolutive process is assumed to be well approximated at each TF bin by a product between the source STFT coefficient and the Fourier transform of the mixing filter. This assumption is called the multiplicative transfer function (MTF) approximation [START_REF] Avargel | On multiplicative transfer function approximation in the short-time Fourier transform domain[END_REF], or the narrowband approximation. Moreover, the sparsity of the audio signals in the timefrequency (TF) domain is a desirable effect. Based on these properties, the separation methods rely on binary masking of the mixture STFT bins, e.g. [START_REF] Yilmaz | Blind separation of speech mixtures via time-frequency masking[END_REF][START_REF] Mandel | Model-based expectation-maximization source separation and localization[END_REF], on the 1 -norm minimiza-tion, e.g. [START_REF] Winter | Map-based underdetermined blind source separation of convolutive mixtures by hierarchical clustering and 1norm minimization[END_REF], on probabilistic models for source signals, e.g. [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF][START_REF] Duong | Underdetermined reverberant audio source separation using a full-rank spatial covariance model[END_REF], or on a combination of these methods.

The MTF approximation is theoretically valid only if the length of the mixing filter impulse response is smaller than the length of the STFT window. In practice, this is very rarely the case, even for moderately reverberant environments, since the STFT window is limited to assume local stationarity of audio signals. Hence the MTF can be a poor approximation, fundamentally endangering the separation performance; this becomes critical for strongly reverberant environments. Yet, the MTF is poorly questioned in the MASS literature, and only a few studies attempted to tackle its limitations. In [START_REF] Duong | Underdetermined reverberant audio source separation using a full-rank spatial covariance model[END_REF], the use of a full-rank spatial covariance matrix for the source images, instead of the rank-1 matrix corresponding to the MTF model [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF], is claimed to overcome to some extent the limitations of MTF. A more direct approach to the problem is proposed in [START_REF] Kowalski | Beyond the narrowband approximation: Wideband convex methods for under-determined reverberant audio source separation[END_REF] where the source signals are estimated in the time domain by minimizing a wide-band 2 -norm mixture-fitting cost, in which the exact source-filter convolution is used, using a Lasso optimization technique. [START_REF] Arberet | Sparse reverberant audio source separation via reweighted analysis[END_REF] improved this wideband Lasso (W-Lasso) technique by a re-weighted scheme. The W-Lasso technique achieves quite good source separation performance in reverberant environments, at the price of a tremendous computation time.

To represent convolution more accurately in the STFT domain, especially for the long filter case, cross-band filters (CBFs) were introduced in [START_REF] Avargel | System identification in the short-time Fourier transform domain with crossband filtering[END_REF] in the context of linear system identification, as an alternative to MTF. Using the CBFs, an output STFT coefficient is represented as a summation over frequency bins of multiple convolutions between the input STFT coefficients and the TF-domain filter impulse response, along the frame coordinate. A convolutive transfer function (CTF) approximation is further introduced in [START_REF] Talmon | Relative transfer function identification using convolutive transfer function approximation[END_REF] to simplify the analysis. Here, at each frequency, the output STFT coefficient is modeled as a (unique) convolution of the input STFT coefficients and the CTF, along the frame axis. The CBFs were recently considered for solving MASS [START_REF] Badeau | Multichannel highresolution NMF for modeling convolutive mixtures of non-stationary signals in the time-frequency domain[END_REF], in combination with a high-resolution non-negative matrix factorization model of the source signal. A variational EM algorithm was proposed to estimate the filters and infer the source signals. Unfortunately, this method was observed to perform well only in a semi-blind setup where both filters and source parameters are initialized from the individual source images.

In this paper we propose to use the CTF for MASS quite differently than what was proposed in the past. Following the spirit of W-Lasso, the source signals are estimated, at each frequency, by minimizing a CTF-based 2 -norm mixture fitting cost in the STFT domain. In addition, we add to the fitting cost a 1 -norm regularizer such as to exploit the sparsity of TF-domain audio signals along the frames. By circumventing the MTF approximation, the proposed method achieves satisfactory source separation performance in reverberant environments, likewise W-Lasso, but since the STFT frame level is considered instead of the time-domain sample level, separation is obtained at a much lower computational burden, as shown by our experiments. Another potential advantage of the proposed method is that, compared to the timedomain mixing filters in W-Lasso, it may be easier to identify the CTF mixing filters from the mixture signals based on the TF-domain sparsity.

The rest of this paper is organized as follows. Section 2 presents the CTF model. The proposed source separation method is given in Section 3. Experiments are presented in Section 4. Section 5 concludes the paper.

CONVOLUTIVE TRANSFER FUNCTION

In a reverberant and noise-free environment, a source image y(n) is, in time domain, given by

y(n) = a(n) s(n) (1)
where s(n) and a(n) are the source signal and the impulse response of the propagating filter, respectively, and denotes linear convolution. With the usual MTF approximation, (1) is approximated in the STFT domain as

y p,k = a k s p,k (2)
where y p,k and s p,k are the STFT of the corresponding signals, and a k is the Fourier transform of the filter a(n), p ∈ [1, P ] is the frame index, N is the frame length, and k ∈ [0, N -1] is the frequency bin index. As discussed above, this approximation is only valid when the filter a(n) is shorter than the STFT window, which is often questionable. In this paper we therefore use the CTF model, i.e. y(n) is approximated in the STFT domain by:

y p,k = p a p ,k s p-p ,k = a p,k s p,k . (3)
The filter CTF, i.e. the TF-domain impulse response a p ,k , is related to the time-domain impulse response a(n) by:

a p ,k = (a(n) ζ k (n))| n=p L , (4)
which represents the convolution with respect to the time index n evaluated at multiples of the frame step L, with

ζ k (n) = e j 2π N kn +∞ m=-∞ ω(m) ω(n + m), (5)
where ω(n) and ω(n) denote the STFT analysis and synthesis windows, respectively. The CTF can be interpreted as: the time-domain convolution is transformed into a TF-domain convolution with a certain approximation error.

SEMI-BLIND SOURCE SEPARATION BASED ON CTF

Mixture model and source separation formulation

We consider a multi-channel underdetermined convolutive mixture with J sources and I sensors (I < J). Based on the CTF model ( 3), in the STFT domain the sensor signals

x i p,k , i ∈ [1, I]
are given by:

x i p,k = J j=1 y j p,k + e i p,k = J j=1 a i,j p,k s j p,k + e i p,k , (6)
where a i,j p,k is the CTF from source j to sensor i, and e i p,k denotes the noise signal. For frequency k, let x ∈ C I×P , s ∈ C J×P and e ∈ C I×P denote the matrices of sensor signals, source signals and noise signals, respectively, and let A ∈ C I×J×P denote the three-way CTF array. Since the proposed algorithm is frequency-wise, the frequency index k is omitted from now on. Then ( 6) can be written as:

x = A s + e. (7)
For underdetermined mixtures, the source signals cannot be estimated by inverting the filters. Instead, we estimate the source signals by minimizing a 2 -norm mixture fitting cost. Moreover, the TF-domain sparsity of the speech sources is enforced using an 1 -norm regularization term. Overall, the source separation is carried out by solving the following convex optimization problem:

min s∈C J×P A s -x 2 2 +λ s 1 , (8)
where the free parameter λ is chosen as a trade-off between the fitting term and the regularization term. Note that both the 2 and 1 norms on matrices in (8) are defined here as vector norms. As mentioned in the introduction, this problem was considered in [START_REF] Kowalski | Beyond the narrowband approximation: Wideband convex methods for under-determined reverberant audio source separation[END_REF] in the time-domain, as a "wide-band" (convolutive) version of the Lasso problem [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] or of the basis pursuit denoising problem [START_REF] Chen | Atomic decomposition by basis pursuit[END_REF]. [START_REF] Kowalski | Beyond the narrowband approximation: Wideband convex methods for under-determined reverberant audio source separation[END_REF] involved a large data size, leading to a low optimization convergence rate and a huge computational cost. Applying the "convolutive" Lasso to MASS in the STFT domain, thanks to the CTF model proposed here, enables to considerably reduce the computational burden and to directly exploit the STFT sparsity of audio signals.

Optimization using FISTA

To solve the optimization problem (8), we adopt the fast iterative shrinkage-thresholding algorithm (FISTA) [START_REF] Beck | A fast iterative shrinkagethresholding algorithm for linear inverse problems[END_REF], as al-ready done in [START_REF] Kowalski | Beyond the narrowband approximation: Wideband convex methods for under-determined reverberant audio source separation[END_REF] in the time-domain, with the notable difference that here the optimization process is carried on the complex domain. Let F(s) = A sx 2 2 denote the fitting cost function, which is L-Lipschitz differentiable. Its derivative is

∆F(s) = Ã (A s -x), (9)
where the adjoint matrix à is obtained by conjugate transposing the source and channel indices, and then temporally reversing the filters.

Similar to [START_REF] Kowalski | Beyond the narrowband approximation: Wideband convex methods for under-determined reverberant audio source separation[END_REF], the Lipschitz constant L is computed using the power iteration algorithm, which is summarized in Algorithm 1. In this work, v is initialized as a matrix composed of I replications of the first channel sensor signal. The convergence criterion consists in testing if the difference in amplitude of the inner product between the values of v, at the current and previous iterations, is larger than a threshold δ.

The 1 -norm regularization term in ( 8) is a lower semicontinuous function, non-differentiable at 0. The proximity operator of the function γ • 1 at point z, aka the shrinkage operator, is defined as:

Prox γ • 1 (z) = argmin y 1 2 z -y 2 2 +γ y 1 . (10)
The closed-form entrywise solution is given by:

y i = z i |z i | max(0, |z i | -γ). (11)
Based on the derivative operator (9), on the Lipschitz constant L, and on the shrinkage operator, FISTA is summarized in Algorithm 2. s 0 is also initialized as the matrix composed of I replication of the first channel sensor signal. The convergence criterion is chosen as a threshold over the relative decrease of the objective cost in (8) at each iteration.

Algorithm 1 Power iteration.

Input: A, Ã Initialization: v ∈ C I×P repeat w = Ã (A v) v = w/ w 2 until convergence L = w 2

EXPERIMENTAL EVALUATION

To test the efficiency of the proposed source separation method, experiments were conducted with simulated binaural signals, under various acoustic conditions. Binaural room impulse responses (BRIR) were generated with the ROOMSIM simulator [START_REF] Campbell | The ROOMSIM user guide (v3.3)[END_REF] and with the head related impulse

Algorithm 2 FISTA Input: x, A, Ã, L Initialization: k = 1, s 0 ∈ C I×P , z 0 = s 0 , t 0 = 1 repeat 1. ∆F(s k-1 ) = Ã (A s k-1 -x) 2. s k = Prox (λ/L) • 1 (z k-1 -∆F(s k-1 )/L) 3. t k = (1 + (1 + 4t 2 k-1 ))/2 4. z k = s k + ((t k-1 -1)/t k )(s k -s k-1 ) 5. k = k + 1 until convergence
response (HRIR) of a KEMAR dummy head [START_REF] Gardner | HRTF measurements of a KEMAR dummy-head microphone[END_REF]. The size of the room was 8 m × 5 m × 3 m. The KEMAR dummy head was located at (4 m, 1 m, 1.5 m). Speech signals from the TIMIT dataset [START_REF] Garofolo | Getting started with the DARPA TIMIT CD-ROM: An acoustic phonetic continuous speech database[END_REF] and then sampled at 16 kHZ were used as sources convolved with the simulated BRIRs to generate sensor signals. The speech sources were located at 1 m away from the dummy head with azimuth directions varying from -90 • to 90 • , spaced by 5 • , and at an elevation of 0 • . Three reverberation times were tested, namely T 60 = 0.22 s, 0.5 s and 0.79 s. Moreover, the anechoic case was also tested. A set of underdetermined mixtures with 3, 4 and 5 sources were processed. For each experiment, 50 mixtures were generated. The STFT window was a Hamming window of 512 samples (32 ms), with 50% overlap. The free parameter λ was set at a fixed value of 10 -3 through the whole set of experiments, as it was shown to be suitable for all the tested conditions. The power iteration convergence threshold δ was set to 0.999. The FISTA convergence threshold was set to 10 -6 . The CTF coefficients were computed from the known filter impulse responses using [START_REF] Winter | Map-based underdetermined blind source separation of convolutive mixtures by hierarchical clustering and 1norm minimization[END_REF].

For comparison, we tested four state-of-the-art source separation methods, all using in the same semi-blind configuration as the proposed method (i.e. with known mixing filters): the degenerate unmixing estimation technique (DUET) [START_REF] Yilmaz | Blind separation of speech mixtures via time-frequency masking[END_REF], which is based on time-frequency masking, assuming only a single source is active in each TF bin, the 1 -norm minimization method ( 1 -MIN) [START_REF] Winter | Map-based underdetermined blind source separation of convolutive mixtures by hierarchical clustering and 1norm minimization[END_REF], which assumes that at most I sources are active in each TF bin, the full-rank spatial covariance matrix (FR-SCM) method of [START_REF] Duong | Underdetermined reverberant audio source separation using a full-rank spatial covariance model[END_REF], and the wide-band Lasso (W-Lasso) method of [START_REF] Kowalski | Beyond the narrowband approximation: Wideband convex methods for under-determined reverberant audio source separation[END_REF] with 1 -norm regularization term on source STFT coefficients, and trade-off factor set to 10 -5 . DUET and 1 -MIN methods are based on the MTF approximation, i.e. the instantaneous mixing matrix in each frequency bin is employed. Since the BRIRs are longer than the STFT window, they have to be truncated to generate the mixing matrix with the Fourier transform. However we obtained better results using the Fourier transform of the HRIRs. For FR-SCM, the SCMs were individually estimated using each separate image source signal, following the line of the semi-oracle experiments in [START_REF] Duong | Underdetermined reverberant audio source separation using a full-rank spatial covariance model[END_REF]. Then an EM was applied with the SCMs being kept fixed to the semi-oracle T 60 (s) Methods Anechoic 0.22 0.50 0.79 DUET [START_REF] Yilmaz | Blind separation of speech mixtures via time-frequency masking[END_REF] 0.01 0.02 0.02 0.02

1 MIN [4] 25.1 23.5 24.9 24.6 FR-SCM [START_REF] Duong | Underdetermined reverberant audio source separation using a full-rank spatial covariance model[END_REF] 345 361 373 396 W-Lasso [START_REF] Kowalski | Beyond the narrowband approximation: Wideband convex methods for under-determined reverberant audio source separation[END_REF] 2694 2810 2975 3232 Proposed 18.1 22.2 28.1 32.9

values. Note that we did not compare our method with [START_REF] Badeau | Multichannel highresolution NMF for modeling convolutive mixtures of non-stationary signals in the time-frequency domain[END_REF], since this method involves a specific rank-1 source model that is poorly appropriate for speech signals.

The signal-to-distortion ratio (SDR) [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF] in decibels (dB), averaged over 50 mixtures for each condition, is used as the separation performance metric (only 5 mixtures are tested for W-Lasso due to its high computation time). Fig. 1 plots the SDR obtained for 3-source mixtures and for the 3 reverberation times. It can be seen in these plots that all five methods achieve high SDR in the anechoic case. DUET performs the worst, due to its limiting assumption that only a single source is active at each TF bin. 1 -MIN and FR-SCM perform better by assuming that more sources can co-exist in a TF bin. W-Lasso achieves the highest SDR, thanks to the exact (time-domain) convolution model. The proposed method achieves a lower SDR than W-Lasso due to the CTF approximation error. As the reverberation time increases, the SDRs of DUET, of 1 -MIN and to a least extent of FR-SCM, dramatically decrease. For DUET and for 1 -MIN, the MTF approximation is no longer suitable when the filter impulse response is (much) longer than the STFT window. FR-SCM mitigates the problem by using a full-rank spatial covariance matrix, which models the reverberations, although to a limited extend. In contrast to these three methods, both W-Lasso and the proposed method achieve remarkably stable performances: the SDR actually increases with T 60 , which is a bit surprising at first sight. This can be explained as follows: (i) the mixture models in W-Lasso and in the proposed method fit the actual mixture better than the MTF, and (ii) given a good fit of the mixture model, the longer the filter, the more information availalble to discriminate and separate different sources. Again, due to the CTF approximation error, the proposed method performs worse than W-Lasso by 2 to 5 dB, depending on T 60 . Table 1 shows the average computation time needed to process one mixture (averaged over the 50 test mixtures; the average mixture duration is about 4 s) for each method and each tested T 60 . We can see in this table that the W-Lasso method is much more time-consuming than the others. DUET is the fastest. The computation time of FR-SCM is about 12% of the computation time of W-Lasso. The computation time of the proposed method and of 1 -MIN is comparable, and is less than 1% of the computation time of W-Lasso.

Finally, Table 2 displays the SDR for various number of sources, for T 60 = 0.5 s. As expected, the SDR of all five methods degrades when the source number increases. W-Lasso has the fastest degradation with increasing number of sources, whereas the proposed method seems more robust, so that for 5 sources, W-Lasso achieves only about 1.1 dB SDR improvement over the proposed method.

CONCLUSION

In this paper, we have proposed a semi-blind source separation method based on the CTF model and STFT-domain Lasso optimization. Overall, the proposed method and the timedomain W-Lasso perform prominently better than DUET, 1 -MIN and FR-SCM. The proposed method drastically improves the computation efficiency over W-Lasso with an acceptable decrease in separation performance. It thus seems to be an excellent trade-off between the fast but inaccurate MTF model, and the exact but highly greedy time-domain W-Lasso, especially for filters that are much longer than the STFT window. Recently, it was proposed a CTF-based method to extract the direct path of a single source [START_REF] Li | Estimation of the direct-path relative transfer function for supervised sound-source localization[END_REF]. Based on the STFT-domain sparsity, the extension to multiple sources is under development, which altogether will enable the development of a CTF-based blind source separation method.

Fig. 1 :

 1 Fig. 1: Source separation performance (SDR) for 3-source mixtures as a function of reverberation time.

Table 1 :

 1 Computation times, in seconds, for a 3-source mixture and for various reverberation times. All the algorithms were implemented in Matlab.

Table 2 :

 2 Source separation performance (SDR in dB) for various number of sources (T60 = 0.5 s).

		Number of sources
	Methods	3	4	5
	DUET [2]	-2.35 -4.54 -5.64
	1 MIN [4]	-1.79 -3.56 -4.79
	FR-SCM [6] -0.86 -2.50 -4.75
	W-Lasso [7] 13.87 7.71	5.58
	Proposed	9.43	5.94	4.46

This research has received funding from the ERC Advanced Grand VHIA (#340113) and from EU-FP7 STREP project EARS (#609465).


Dionyssos Kounades-Bastian

Laurent Girin

Xavier Alameda-Pineda

Sharon Gannot

Radu Horaud

AN EM ALGORITHM FOR JOINT SOURCE SEPARATION AND DIARISATION OF MULTICHANNEL CONVOLUTIVE SPEECH MIXTURES

Keywords: Audio source separation, speaker diarisation, local Gaussian model

We present a probabilistic model for joint source separation and diarisation of multichannel convolutive speech mixtures. We build upon the framework of local Gaussian model (LGM) with non-negative matrix factorization (NMF). The diarisation is introduced as a temporal labeling of each source in the mix as active or inactive at the short-term frame level. We devise an EM algorithm in which the source separation process is aided by the diarisation state, since the latter indicates the sources actually present in the mixture. The diarisation state is tracked with a Hidden Markov Model (HMM) with emission probabilities calculated from the estimated source signals. The proposed EM has separation performance comparable with a state-of-the-art LGM NMF method, while outperforming a state-of-the-art speaker diarisation pipeline.

INTRODUCTION

Multichannel audio source separation (MASS) aims at recovering unobserved source signals from observed mixtures [START_REF] Comon | Handbook of Blind Source Separation -Independent Component Analysis and Applications[END_REF]. MASS is mainly concerned with mixtures of speech, music, and ambient noise. Speaker diarisation is the segmentation and labelling of an audio signal emitted during multi-party conversations [START_REF] Tranter | An overview of automatic speaker diarization systems[END_REF][START_REF] Miro | Speaker diarization: A review of recent research[END_REF]. In short, speaker diarisation is answering to the question "who is talking, and when?" while MASS tries to recover the emitted signals. Both processes are crucial front-ends for higher-level processes such as speech recognition, human-computer or human-robot interaction.

There has been extensive research addressing independently either MASS [START_REF] Vincent | Probabilistic modeling paradigms for audio source separation[END_REF][START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF][START_REF] Duong | Underdetermined reverberant audio source separation using a full-rank spatial covariance model[END_REF][START_REF] Ozerov | A general flexible framework for the handling of prior information in audio source separation[END_REF], or speaker diarisation [START_REF] Tranter | An overview of automatic speaker diarization systems[END_REF][START_REF] Miro | Speaker diarization: A review of recent research[END_REF] tasks. Currently, most of MASS methods implicitly assume all sources as continuously emitting. Besides, state-of-theart methods on diarisation, e.g. [START_REF] Vijayasenan | Multistream speaker diarization of meetings recordings beyond mfcc and tdoa features[END_REF] consists of a pipeline that starts by extracting features from the audio mixture, e.g. Mel frequency cepstral coefficients, and proceeds with speech/non-speech segmentation of the audio stream, and clustering of the speech segments into associated speakers.

Obviously, the two tasks are highly inter-related. Indeed, knowing the separated sources of an audio mixture helps assessing when each source is active/inactive. On the other hand, knowing the diarisation of the sources within the mixture determines how many sources need to be separated and when. Therefore, a joint formulation of MASS and diarisation can be beneficial for both problems. Except for a series of Higuchi et. al. [START_REF] Higuchi | A unified approach for underdetermined blind signal separation and source activity detection by multichannel factorial hidden markov models[END_REF][START_REF] Higuchi | Underdetermined blind separation and tracking of moving sources based on DOA-HMM[END_REF][START_REF] Higuchi | Unified approach for audio source separation with multichannel HMM and DOA mixture model[END_REF], a framework addressing jointly the two problems seems overlooked in the literature; in [START_REF] Higuchi | A unified approach for underdetermined blind signal separation and source activity detection by multichannel factorial hidden markov models[END_REF][START_REF] Higuchi | Underdetermined blind separation and tracking of moving sources based on DOA-HMM[END_REF] the active/inactive state of a source is independently modeled with a factorial HMM in a MASS framework. This independent modeling of the activity of a source with respect to the activity of the other sources may be unrealistic for multiparty conversations. In [START_REF] Higuchi | Underdetermined blind separation and tracking of moving sources based on DOA-HMM[END_REF] the source activity detection is combined with a direction-of-arrival-dependent HMM for the propagation model. A variational expectation maximization (EM) is presented that infers the sources, their activity and the model parameters, although under the assumption of a single active source per time-frequency bin.

In this paper we propose a probabilistic model for simultaneous diarisation and separation for multichannel audio mixtures that enjoys the following merits: We consider all possible combinations of simultaneous active sources and process them jointly, as the overall state of diarisation. An EM algorithm is designed for model parameter estimation. We compare the performance of the proposed EM with [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF] in terms of separation, and with [START_REF] Vijayasenan | Multistream speaker diarization of meetings recordings beyond mfcc and tdoa features[END_REF] in terms of diarisation.

The proposed model is presented in Section 2. The EM algorithm that estimates the parameters and that infers the source signals and the diarisation is presented in Section 3. Experimental evaluation reported in Section 4 shows competitive performance on both source separation and diarisation.

MODELS

Audio Mixtures with Diarisation

As in many source separation methods, the observed signal is modeled as a multichannel time-invariant convolutive noisy mixture of the source signals. We work with the short time Fourier transform (STFT) representation of the input audio, where x f = [x 1,f . . . x I,f ] ∈ C I is the I-channel vector of Fourier coefficients at frequency bin f ∈ [1, F ] and time frame ∈ [1, L]. Relying on the narrow-band assumption, i.e. the channel impulse response is shorter than the STFT analysis window, x f writes [START_REF] Parra | Convolutive blind separation of non-stationary sources[END_REF]: x f = A f s f +b f , where s f = [s 1,f . . . s J,f ] ∈ C J is the latent vector of source coefficients, A f ∈ C I×J is the mixing matrix, and b f ∈ C I is residual noise. In the present study we want to express x f with a formulation that explicitly encodes the activity of the sources, in a binary manner; we have N = 2 J possible combinations, or "states", for source activity. Let us represent a state n ∈ [1, N ] with a [J × J] diagonal matrix D n with j th entry D jj,n set to:

D jj,n = 1, if the j th source is active at state n 0, otherwise. (1)
For example, with J = 2, the N = 4 possible matrices are:

D 1 = 0 0 , D 2 = 0 1 , D 3 = 1 0 , D 4 = 1 1 . (2)
Incorporating D n we rewrite x f as:

x f = J j=1 D jj,n a j,f s j,f + b f = A f D n s f + b f , (3)
with a j,f ∈ C I the j th column of A f . By choosing the matrix D n for a frame , we select which of the J sources compose the mixture at that frame, as D n zeroes out inactive sources.

Selecting the Diarisation

The activity of each sound source varies over time, hence the state is to be estimated for each frame . For this we define a latent categorical variable Z = n, n ∈ [1, N ] indicating the state at -th frame. The Z follows a first-order HMM with:

p(Z 1 = n) = λ n , p(Z = n|Z -1 = r) = T nr , (4)
with λ n , T nr ∈ R + , n, r ∈ [1, N ] being the prior and transition parameters to be estimated. Let us assume that b f follows a zero-mean proper complex-Gaussian distribution 1 .

We can now express the mixture of (3) probabilistically:

p(x f |s f , Z = n) = N c (x f ; A f D n s f , v f I I ) , (5)
with A f and v f ∈ R + parameters to be estimated and I I is the identity matrix of dimension I.

The Source Model

Let {K j } J j=1 denote a non-trivial partition of {1 . . . K}, with K ≥ J the number of latent components that is known in advance. Following [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF][START_REF] Kounades-Bastian | A variational EM algorithm for the separation of time-varying convolutive audio mixtures[END_REF][START_REF] Arberet | Nonnegative matrix factorization and spatial covariance model for under-determined reverberant audio source separation[END_REF], s j,f is modeled as the sum of the latent components c k,f , k ∈ K j :

s j,f = k∈Kj c k,f ⇔ s f = Gc f , (6)
where G ∈ N J×K is a matrix with entries G jk = 1 if k ∈ K j and G jk = 0 otherwise, and

c f = [c 1,f , . . . , c K,f ] ∈ C K is the vector of component coefficients. Each component c k,f
follows a zero-mean proper complex Gaussian distribution with variance w f k h k , and w f k , h k ∈ R + parameters to be estimated. The components are assumed to be mutually independent and individually independent across frequencies and over time; the probability density function (pdf) of the component vector therefore is:

p(c f ) = N c c f ; 0 K , diag K (w f k h k ) , (7)
with 0 K the zero-vector of dimension K and diag K (d k ) the K × K diagonal matrix with respective entries {d k } K k=1 . Eq. ( 7) corresponds to a non-negative matrix factorization (NMF) model placed on the F × L matrix of variances of the source coefficients; a now common practice in audio signal processing, e.g. [START_REF] Benaroya | Non negative sparse representation for Wiener based source separation with a single sensor[END_REF][START_REF] Févotte | Nonnegative matrix factorization with the Itakura-Saito divergence. With application to music analysis[END_REF][START_REF] Mohammadiha | Supervised and unsupervised speech enhancement using nonnegative matrix factorization[END_REF].

EM FOR SEPARATION AND DIARISATION

We derived an EM algorithm to infer the hidden variables

H = {c f , s f , Z } F,L f, =1 and estimate the parameters θ = {A f , v f , T nr , λ n , w f k , h k } F,L,K,N,N f, ,k,n,r=1
. The E-step and Mstep are given below. The complete EM is given in Algorithm1. The iteration index is omitted for clarity.

E-step

The E-step consists of computing the joint posterior probability:

p(c f , Z = n|{x f } F,L f, =1
). This is done by first computing the posterior over the component coefficients:

p(c f |Z = n, {x f } F,L f, =1
) and then computing the posterior over the states:

η n = p(Z = n|{x f } F,L f, =1
). After a few manipulations, it turns out that for every state n ∈ [1, N ] we obtain a different complex Gaussian pdf for the components:

p c f |Z = n, {x f } F,L f, =1 ∝ p(c f )p(x f |c f , Z = n) = N c c f ; ĉf n , Σ c f n , (8)
with ĉf n ∈ C K and Σ c f n ∈ C K×K given by:

Σ c f n = diag K 1 w f k h k + G D n A H f A f v f D n G -1 , ĉf n =Σ c f n G D n A H f x f v f . (9)
From ( 6), we easily deduce that the source posterior distribution is a complex Gaussian with mean vector ŝf n ∈ C J , and covariance matrix Σ s f n ∈ C J×J calculated with:

ŝf n = Gĉ f n , Σ s f n = GΣ c f n G . (10)
From ( 4), the posterior probability η n of the n th state is calculated by decoding a first-order HMM (using the forwardbackward algorithm [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]), with emission probabilities

ι n = p({x f } F,L f, =1 |Z = n)
given by:

ι n ∝ exp   F f =1 log Σ s f n + x f H v f A f D n ŝf n   , (11)
with |.| the matrix determinant. Then we run the forward and backward recursions to compute the probabilities φ n , β n :

φ n ∝ ι n N r=1 T nr φ ( -1)r , (12)
β n ∝ N r=1 T rn ι ( +1)r β ( +1)r . (13)
Multiplying φ n with β n and normalizing, we obtain:

η n ∝ φ n β n . (14)
The recursions require initialization of φ 1n and β Ln . We observed faster convergence by, at each EM iteration, setting φ 1n = ι 1n λ n , running the forward recursion, and then setting β Ln = φ Ln to initialize the backward recursion.

M-step

In the M step, the parameters maximizing the expected complete-data log-likelihood are computed. M-w f k , h k step: The parameters w f k and h k are coupled in the objective function and an alternation strategy is required, i.e. fixing one parameter to estimate the other. Similar to [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF][START_REF] Kounades-Bastian | A variational EM algorithm for the separation of time-varying convolutive audio mixtures[END_REF] the updates for w f k and h k are:

w f k = 1 L L =1 u k,f h k , h k = 1 F F f =1 u k,f w f k , (15)
with u k,f ∈ R + being here the posterior PSD of c k,f averaged with the posterior probability of the states:

u k,f = N n=1 η n Σ c kk,f n + |ĉ k,f n | 2 , (16)
with Σ c kk,f n ∈ R + being the k th diagonal entry of Σ c f n and ĉk,f n ∈ C being the k th entry of ĉf n . [START_REF] Tranter | An overview of automatic speaker diarization systems[END_REF]. repeat E step C: Compute Σ c f n and ĉf n with ( 9). E step S: Compute Σ s f n , ŝf n with ( 10). E step Z: Compute ι n with [START_REF] Higuchi | Unified approach for audio source separation with multichannel HMM and DOA mixture model[END_REF], set φ 1n = λ n ι 1n , for : 2 to L. Compute φ n with [START_REF] Parra | Convolutive blind separation of non-stationary sources[END_REF]. end.

Algorithm 1 Separation & diarisation of J sound sources input {x f } F,L f, =1 , binary matrix G, initial parameters θ. construct: The 2 J matrices D n , n ∈ [1, 2 J ] with
Set β Ln = φ Ln .

for : L -1 to 1. Compute β n with [START_REF] Neeser | Proper complex random processes with applications to information theory[END_REF]. end. Compute η n with ( 14). M-w f k , h k step: Update w f k , and then h k , with [START_REF] Arberet | Nonnegative matrix factorization and spatial covariance model for under-determined reverberant audio source separation[END_REF]. M-T nr , λ n step: Update ξ ,nr , then T nr , λ n , with [START_REF] Févotte | Nonnegative matrix factorization with the Itakura-Saito divergence. With application to music analysis[END_REF]. [START_REF] Sturmel | Linear mixing models for active listening of music productions in realistic studio conditions[END_REF]. until convergence return The source images, the diarisation D n ∀ . M-T nr , λ n step: Classically, for the HMM parameters, we calculate ξ ,nr = p(Z = n, Z -1 = r|{x f } F,L f, =1 ) and then update T nr and λ n with:

M-A f step: Compute o f , R f with (18), A f with (19). M-v f step: Update v f with
ξ ,nr ∝ β ( +1)n ι ( +1)n T nr φ r , T nr ∝ L-1 =1 ξ ,nr , λ n = η 1n . (17)
M-A f step: As the mixing matrix is common for all diarisation states, we need to first define the "final" source estimate o f ∈ C J , i.e. average source estimate over the states, and the corresponding second-order statistic R f ∈ C J×J :

o f = N n=1 η n D n ŝf n , R f = N n=1 η n D n Σ s f n + ŝf n ŝH f n D n . (18)
Then the optimal value for A f is:

A f = L =1 x f o H f L =1 R f -1 , (19)
which is a standard form of least square estimator [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF]. M-v f step: The optimal noise variance is:

v f = 1 LI L =1 x H f x f -2Re x H f A f o f + tr A f R f A H f . (20)
where tr{.} denotes the trace and Re{.} denotes the real part.

Estimation of Separated Sources and Diarisation

The separation performance is assessed with the time domain source images, i.e. the multichannel source signals at all microphones [START_REF] Duong | Underdetermined reverberant audio source separation using a full-rank spatial covariance model[END_REF][START_REF] Sturmel | Linear mixing models for active listening of music productions in realistic studio conditions[END_REF], estimated by applying the inverse STFT with overlap-add on {o j,f a j,f } F,L f, =1 . The diarisation output n is given at each frame by selecting the higher value of η n , n ∈ [1, N ]. Then from D n we have the active sources at th frame. Frames where η 1 is dominant are non-speech frames.

EVALUATION

To assess the performance of the proposed method, we simulated the challenging task of separating and diarizing J = 3 sources from a convolutive stereo mixture (I = 2). Each source signal was a 27s speech signal made by concatenating utterances chosen from the TIMIT database [START_REF] Garofolo | Timit acoustic-phonetic continuous speech corpus[END_REF] (one different speaker for each source). As mixing filters, we used binaural room impulse responses from [START_REF] Hummersone | A comparison of computational precedence models for source separation in reverberant environments[END_REF] having RT 60 ≈ 0.68s. We generated two types of mixtures: Mix-DC where all sources are emitting continuously. Mix-8 where each source has balanced portions of speech and silence so that all N = 8 states appear.

MASS performance is assessed with the signal-to-distortion (SDR), signal-to-interference (SIR), and signal-toartefact (SAR) measures (in dB) [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF]. Diarisation is assessed with accuracy (Acc) defined as the percentage of frames for which a source was correctly identified (as either active if actually active, or inactive if actually inactive). As baseline, we used [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF] for source separation and [START_REF] Vijayasenan | Multistream speaker diarization of meetings recordings beyond mfcc and tdoa features[END_REF] for speaker diarisation. Both baselines were provided with the correct number of sources. Because [START_REF] Vijayasenan | Multistream speaker diarization of meetings recordings beyond mfcc and tdoa features[END_REF] is designed for non-overlapping audio streams, we considered each of the 2 J source combinations as a virtual speaker, and we translated the result of the clustering over virtual speakers into clustering of individual sources (we tested all possible associations and reported the one giving the highest accuracy). Afterwards, we use a median filter on the estimated label of each source to remove spikes.

The initialization of the parameters is crucial for EM. We initialized the parameters w f k , h k of both the proposed EM and [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF], by applying the KL-NMF algorithm [START_REF] Févotte | Nonnegative matrix factorization with the Itakura-Saito divergence. With application to music analysis[END_REF] on corrupted versions of the true source spectra, using |K j | = 20 components per source. The other parameters were randomly initialized. For the STFT analysis we used a sine analysis window with 512 taps and 50% frame overlap, leading to L = 1697 frames.

In Table 1 we report detailed MASS and diarisation scores. Each value is an average measure over 10 mixture realizations with different speakers. Fig. 1 illustrates one diarisation result for Mix-8. In terms of MASS, we see that the proposed method performs equally well with [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF] on both Mix-8 and Mix-DC. E.g. on Mix-8 the avg. SDR of the proposed method is 0.2dB higher (8.3dB versus 8.1dB). This is encouraging, considering that the proposed method has estimate the additional parameters needed for diarisation. In terms of diarisation, the proposed method higher accuracy than [START_REF] Vijayasenan | Multistream speaker diarization of meetings recordings beyond mfcc and tdoa features[END_REF] on Mix-8 (91.7% versus 74.9%) and on Mix-DC (97.3% versus 77.3%). This justifies the joint modeling of the source activity detection and the source signal recovery. Qualitatively, we see from Fig. 1 that the activity pattern is tracked with only a few misdetections.

CONCLUSIONS

We introduced an LGM based probabilistic framework for joint MASS and diarisation of the audio sources in a multichannel mix. Experiments on underdetermined speech mixtures showed competitive performance of the proposed method compared to the state-of-the-art, in particular in diarisation. Future research will investigate the ability of the proposed model to automatically determine the number of sources J (via D n ). We will benchmark the performance of alternative source models, e.g. [START_REF] Mysore | Nonnegative hidden markov modeling of audio with application to source separation[END_REF], when bundled with a diarisation scheme. We will explore realistic initialization schemes, so to create a fully blind joint MASS and diarisation method. We espy also the simultaneous use of source localization cues to improve separation and diarisation.

3 GT = E = 1 GT = 1 ,Fig. 1 .

 3111 Fig. 1. Timing diagram of per source estimated diarisation for Mix-8 when using the proposed method (top) and the baseline (bottom). The graphic is color-coded as a function of the ground-truth (GT) and the estimate (E) per each segment.

Table 1 .

 1 Average source separation & diarisation scores.

				Mix-8			Mix-DC
			SDR SIR SAR Acc.(%)	SDR SIR SAR Acc.(%)
		s 1	7.7	11.6 12.1	93.5	7.8	12.4 12.2	99.5
	Ours	s 2 s 3	7.9 9.2	14.9 16.6 13.4 14.1	94.3 87.5	7.3 8.9	14.0 15.1 13.3 14.0	93.2 99.3
		avg.	8.3	13.3 14.3	91.7	8.0	13.3 13.7	97.3
		s 1	7.6	12.6 12.4	89.0	7.7	12.6 12.7	87.8
	Base	s 2 s 3	7.6 9.0	13.5 15.9 13.1 14.8	68.4 67.4	7.3 8.8	13.1 16.0 13.0 14.8	82.2 61.8
		avg.	8.1	13.1 14.4	74.9	7.9	12.9 14.5	77.3

The proper complex Gaussian distribution is defined asNc(x; µ, Σ) = |πΣ| -1 exp -[x -µ] H Σ -1 [x -µ], with x, µ ∈ C I and Σ ∈ C I×I being the argument, mean vector, and covariance matrix respectively[START_REF] Neeser | Proper complex random processes with applications to information theory[END_REF].

This research has received funding from the EU-FP7 STREP project EARS (#609465) and ERC Advanced Grant VHIA (#340113).