Samuele Cornell
email: s.cornell@pm.univpm.it

Maurizio Omologo
email: omologo@fbk.eu

Stefano Squartini
email: s.squartini@univpm.it

Emmanuel Vincent
email: emmanuel.vincent@inria.fr

Overlapped speech detection and speaker counting using distant microphone arrays

Keywords: voice activity detection, overlapped speech detection, speaker counting, distant microphones, spatial features

We study the problem of detecting and counting simultaneous, overlapping speakers in a multichannel, distant-microphone scenario. Focusing on a supervised learning approach, we treat Voice Activity Detection (VAD), Overlapped Speech Detection (OSD), joint VAD and OSD (VAD+OSD) and speaker counting in a unified way, as instances of a general Overlapped Speech Detection and Counting (OSDC) multi-class supervised learning problem. We consider a Temporal Convolutional Network (TCN) and a Transformer based architecture for this task, and compare them with previously proposed state-of-the art methods based on Recurrent Neural Networks (RNN) or hybrid Convolutional-Recurrent Neural Networks (CRNN). In addition, we propose ways of exploiting multichannel input by means of early or late fusion of single-channel features with spatial features extracted from one or more microphone pairs. We conduct an extensive experimental evaluation on the AMI and CHiME-6 datasets and on a purposely made multichannel synthetic dataset. We show that the Transformer-based architecture performs best among all architectures and that neural network based spatial localization features outperform signal-based spatial features and significantly improve performance compared to single-channel features only. Finally, we find that training with a speaker counting objective improves OSD compared to training with a VAD+OSD objective.

Introduction

Motivation

In spontaneous human conversations different speakers tend to overlap with each other and, in meeting scenarios with more than two participants, the amount of overlapped speech can account for a significant portion of the total speech time, usually between 10% and 20% [START_REF] Mccowan | The AMI meeting corpus[END_REF][START_REF] Watanabe | CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings[END_REF]. This phenomenon is one of the main obstacles towards fully reliable multi-party speech diarization [START_REF] Ryant | Enhancement and analysis of conversational speech: JSALT 2017[END_REF][START_REF] García-Perera | Speaker detection in the wild: Lessons learned from JSALT[END_REF] and recognition (Watanabe et al., 2017;Vincent et al., 2018;[START_REF] Haeb-Umbach | Speech processing for digital home assistants: Combining signal processing with deep-learning techniques[END_REF]. In fact, most current techniques for speech recognition and diarization are not designed to deal directly with overlapped speech. As a result, their performance can degrade significantly in such conditions.

For this reason, Overlapped Speech Detection (OSD) is crucial to prevent back-end task performance degradation. This can be accomplished by including a reliable OSD algorithm together with Voice Activity Detection (VAD) in the very front-end part of the pipeline, possibly followed by speech separation [START_REF] García-Perera | Speaker detection in the wild: Lessons learned from JSALT[END_REF][START_REF] Watanabe | CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings[END_REF]. Speaker counting [START_REF] Stöter | CountNet: Estimating the number of concurrent speakers using supervised learning[END_REF] is a closely related task, which can be seen as an extension of VAD+OSD. Instead of merely identifying when there is speech and overlapped speech, speaker counting aims to directly estimate the actual number of concurrent speakers. This task, which is directly related to the problem of counting the number of coherent sources is of crucial importance for modern spoken language understanding applications, such as voice-based virtual assistants. In fact, in many real far-field speech recognition applications, encountering three or more simultaneous sources is quite common, for example in an home environment, where sound could also come from electronic devices. Speaker counting could be also employed to further help back-end tasks such as speech separation [START_REF] Stöter | Separation and count estimation for audio sources overlapping in time and frequency separation and estimation of the number of audio signal sources with time and frequency[END_REF] and also diarization, by extending previously proposed methods which employed only OSD information, such as [START_REF] Bullock | Overlap-aware diarization: Resegmentation using neural end-to-end overlapped speech detection[END_REF]. The accuracy of OSD, VAD and speaker counting algorithms is critical, as errors can propagate to the subsequent processing blocks, severely impacting, for example, speech recognition performance when speech segments are missed [START_REF] Tong | Evaluating VAD for automatic speech recognition[END_REF].

Related works

The research towards reliable OSD spans more than one decade, with the first systems relying on handcrafted features and classical machine-learning approaches. Most of these early studies focused on Gaussian Mixture Model (GMM) or Hidden Markov Model (HMM) based classifiers [START_REF] Boakye | Improved overlapped speech handling for speaker diarization[END_REF][START_REF] Vipperla | Speech overlap detection and attribution using convolutive non-negative sparse coding[END_REF][START_REF] Charlet | Impact of overlapping speech detection on speaker diarization for broadcast news and debates[END_REF][START_REF] Yella | Overlapping speech detection using long-term 1070 conversational features for speaker diarization in meeting room conversations[END_REF][START_REF] Lee | Overlapping speech detection with cluster-based HMM framework[END_REF] with the exception of [START_REF] Geiger | Detecting overlapping speech with long short-term memory recurrent neural networks[END_REF] who showed a Long-Short Term Memory (LSTM) neural network to outperform a GMM-HMM system. [START_REF] Boakye | Improved overlapped speech handling for speaker diarization[END_REF], [START_REF] Vipperla | Speech overlap detection and attribution using convolutive non-negative sparse coding[END_REF], and [START_REF] Yella | Overlapping speech detection using long-term 1070 conversational features for speaker diarization in meeting room conversations[END_REF] reported a substantial reduction of the Diarization Error Rate (DER) on the AMI meeting corpus [START_REF] Carletta | The AMI meeting corpus: A preannouncement[END_REF] by removing overlapped speech segments from the segment clustering phase and performing overlap attribution afterwards.

When multiple microphone channels are available, speaker counting can be performed by clustering interchannel features [START_REF] Drude | Source counting in speech mixtures using a variational em approach for complex watson mixture models[END_REF][START_REF] Pasha | Blind speaker counting in highly reverberant environments by clustering coherence features[END_REF] or explicitly localizing the speakers in space [START_REF] Brutti | Multiple source localization based on acoustic map de-emphasis[END_REF][START_REF] Pavlidi | Source counting in realtime sound source localization using a circular microphone array[END_REF], both in the single-array and multiple-array scenarios. Singlechannel speaker counting is more challenging, with early works focusing on handcrafted features such as the modulation index [START_REF] Arai | Estimating number of speakers by the modulation characteristics of speech[END_REF], the mean and variance of the 7th Mel filter [START_REF] Ouamour | Pens: a confidence parameter estimating the number of speakers[END_REF] or the cosine similarity between Mel Frequency Cepstrum Coefficient (MFCC) feature vectors along with pitch [START_REF] Xu | Crowd++: unsupervised speaker count with smartphones[END_REF]. More recently, [START_REF] Andrei | Counting competing speakers in a timeframe -human versus computer[END_REF] estimated the number of speakers by computing the distance between the mixture and a reference single-speaker utterance in the magnitude spectral domain.

CountNet [START_REF] Stöter | CountNet: Estimating the number of concurrent speakers using supervised learning[END_REF]) marked a significant departure from these previous works by showing that a neural network can be trained to perform speaker counting without relying on handcrafted features, and it can even outperform humans. [START_REF] Andrei | Overlapped speech detection and competing speaker counting -humans versus deep learning[END_REF] also showed that a neural network based speaker counting algorithm can defeat human ability especially when more than three speakers are active. [START_REF] Kanda | Joint speaker counting, speech recognition, and speaker identification for overlapped speech of any number of speakers[END_REF] took a different direction: they trained a neural network to perform joint speaker counting, speech recognition and speaker identification in a fully end-to-end fashion. In all these works, synthetic mixtures are employed for both training and testing and, crucially, the datasets are designed with balanced proportions of single-speaker speech, two-speaker overlapped speech, three-speaker overlapped speech, and so on. This does not match the characteristics of real-world datasets where singlespeaker speech is more frequent than two-speaker overlapped speech, which is itself much more frequent than three-speaker overlapped speech. Also related to speaker counting, End-to-End Neural Diarization (EEND) firstly proposed in Fujita et al. (2019a) and subsequently improved by Fujita et al. (2019b) by using a Transformer-based architecture. In such works a neural network is trained to perform diarization by using a permutation invariant multi-label training scheme. As these systems are trained to perform diarization they can be easily adapted to perform speaker counting. In a similar manner also the recently proposed Target-Speaker VAD (TS-VAD) framework of [START_REF] Medennikov | Target-speaker voice activity detection: A novel approach for multi-speaker diarization in a dinner party scenario[END_REF] can be employed for the purpose of counting speakers. This technique has been shown to be particularly effective even in challenging scenarios such as CHi-ME-6 [START_REF] Watanabe | CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings[END_REF]. TS-VAD employs a neural network to estimate each speaker speech activity. However, unlike EEND and CountNet, it requires, as an additional input to the network, i-vectors [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF] for each speaker. This additional requirement makes TS-VAD more cumbersome to implement, especially for streaming applications, than both EEND and CountNet which employ end-to-end neural approaches.

Regarding OSD, [START_REF] Andrei | Detecting overlapped speech on short timeframes using deep learning[END_REF] and [START_REF] Sajjan | Leveraging LSTM models for overlap detection in multi-party meetings[END_REF] reduction on the AMI headset mix. In another vein, [START_REF] Málek | Voice-activity and overlapped speech detection using x-vectors[END_REF] addressed VAD+OSD by employing simple classifiers on top of pre-trained xvector speaker embeddings [START_REF] Snyder | Xvectors: Robust DNN embeddings for speaker recognition[END_REF] and evaluated them on synthetic data corrupted by noise and artificial reverberation.

Our contribution

In this article, we unify VAD, OSD, joint VAD+OSD, and speaker counting as instances of a general Overlapped Speech Detection and Counting (OSDC) supervised classification task. We study two TCN [START_REF] Bai | An empirical evaluation of generic convolutional and recurrent networks for sequence modeling[END_REF] and Transformer [START_REF] Vaswani | Attention is all you need[END_REF] based architectures for this task, compare them with the LSTM-based architecture of [START_REF] Sajjan | Leveraging LSTM models for overlap detection in multi-party meetings[END_REF] and CountNet, and present an in-depth study of their computational efficiency. In addition, we explore the use of spatial features to aid VAD+OSD and speaker counting. As mentioned above, a number of works have shown that spatial features can be used for counting [START_REF] Drude | Source counting in speech mixtures using a variational em approach for complex watson mixture models[END_REF][START_REF] Pasha | Blind speaker counting in highly reverberant environments by clustering coherence features[END_REF][START_REF] Brutti | Multiple source localization based on acoustic map de-emphasis[END_REF][START_REF] Pavlidi | Source counting in realtime sound source localization using a circular microphone array[END_REF] and VAD (Vecchiotti et al., 2019b). However, to our knowledge, no study has yet been performed where spatial features are used in conjunction with deep neural networks to tackle OSD and speaker counting directly. We perform an extensive experimental evaluation using a purposely made multichannel synthetic dataset and two real-world, multi-microphone, distant-speech datasets: AMI [START_REF] Mccowan | The AMI meeting corpus[END_REF] and CHiME-6 [START_REF] Watanabe | CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings[END_REF]. This article significantly extends and improves upon our preliminary study [START_REF] Cornell | Detecting and counting overlapping speakers in distant speech scenarios[END_REF], which did not include the Transformer-based architecture, was restricted to single-channel input and a single type of single-channel features, did not analyze the results as a function of speaker distance or angle, and did not report computational efficiency.

In detail, we first evaluate the different architectures on AMI and CHiME-6 for both VAD+OSD and speaker counting, considering single-channel features only for the sake of comparison with [START_REF] Sajjan | Leveraging LSTM models for overlap detection in multi-party meetings[END_REF] and [START_REF] Cornell | Detecting and counting overlapping speakers in distant speech scenarios[END_REF]. We show that the proposed Transformer-based network, despite having the lowest computational footprint, achieves the best performance on all tasks. We then study how its real-world performance can be further improved by adding spatial features. We examine different such features, including classical interchannel features and neural network based localization features. Also suitable early fusion and late fusion schemes for combining single-channel spectral features and spatial features are compared. The synthetic dataset is used to further study and validate our findings in a controlled environment where oracle speaker locations are known. For the sake of reproducibility, the code used to perform the experiments and to generate the synthetic dataset is made publicly available. 1The remainder of this paper is structured as follows. In Section 2, we explain the multi-class classification framework adopted through this work for supervised VAD+OSD and speaker counting. Section 3 presents the proposed and existing neural architectures and Section 4 introduces the spatial features we explore for this purpose. Then, in Section 5, we describe the datasets used for the experiments and, in Section 6, we report and discuss our extensive experimental evaluation, including the comparison of computational requirements and the results achieved by single-channel and multichannel systems. Finally, in Section 7, we summarize the results obtained, draw conclusions and outline possible future work directions.

Overlapped Speech Detection and Counting Framework

In this work, we treat supervised VAD, OSD, VAD+OSD, and speaker counting in a unified way, as special instances of a general OSDC task. This task can be formulated as a multi-class supervised sequence labeling problem, with a different number of classes for VAD, OSD, joint VAD+OSD, and speaker counting.

We consider a parametric model F(X; θ) which takes as input a sequence of frame-level feature vectors X = {x 1 , x 2 , . . . , x m } and outputs a sequence of class posterior probabilities. We assume that the model may perform internal subsampling, i.e., one output frame is provided every K input frames. This is because frame-level estimation is unnecessary for most speech segmentation applications and, by employing subsampling operations, the computational burden can be reduced.

In the supervised setting, we are given the ground-truth class label sequence y = {y 1 , y 2 , . . . , y l } of length l ≤ m, and we wish to estimate the optimal model parameters θ according to a certain criterion. As in this paper we focus on neural approaches, the optimal model parameters are estimated on a suitable training set composed of N pairs of input feature sequences and corresponding class label sequences T = {(X 1 , y 1 ), . . . (X N , y N )} by using Stochastic Gradient Descent (SGD) to minimize the cross-entropy loss between the estimated framelevel posterior probabilities and the true class distribution.

In this framework, VAD and OSD can be treated either separately as binary classification tasks (speech vs. non-speech, overlap vs. non-overlap), or jointly as a three-class (non-speech, single speaker, overlapped speech) problem. Similarly, speaker counting can be formulated as an C-class classification task where C is equal to the maximum possible number of overlapping speakers plus one. While this approach is not the only one for supervised speaker counting, it has been found to be the most effective [START_REF] Stöter | CountNet: Estimating the number of concurrent speakers using supervised learning[END_REF], provided the maximum possible number of concurrent speakers is known.

Neural Architectures for OSDC

We study four neural network architectures for tackling the OSDC task.

Long-Short Term Memory (LSTM)

The first one is the best neural network for joint VAD+OSD among the ones examined by [START_REF] Sajjan | Leveraging LSTM models for overlap detection in multi-party meetings[END_REF] which, to our knowledge and with the exception of our preliminary work [START_REF] Cornell | Detecting and counting overlapping speakers in distant speech scenarios[END_REF], achieves the best reported performance on AMI single-channel distant-speech data.

It consists of a unidirectional LSTM layer with a hidden size of 512 neurons, followed by 3 dense layers with 1024, 512 and 256 neurons, respectively. A final 256 × N pointwise convolutional layer along with softmax is used to output the probability of each frame belonging to one of the N classes (e.g., N = 3 for VAD+OSD). This network features a total of 2 million parameters and generates one output vector for every input frame given a context of 11 frames (current frame plus 5 past and 5 future frames).

As the original architecture lacked any normalization technique, in our experiments we added batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] before each dense layer activation as well as layer normalization [START_REF] Ba | Layer normalization[END_REF] on the input features. This, coupled with data-augmentation, allows us to improve performance over the original network as it will be shown in Section 6.5.

Hybrid Convolutional-Recurrent Neural Network

We also consider the best CountNet architecture among the 5 different networks compared by [START_REF] Stöter | CountNet: Estimating the number of concurrent speakers using supervised learning[END_REF]. This network is a hybrid Convolutional-Recurrent Neural Network (CRNN) , composed of a 2-D Convolutional Neural Network (CNN) block followed by an RNN block. The main idea behind this architecture is that the CNN extracts a local representation of the input features while the RNN deals with long-term temporal modeling, thus combining the advantages of both CNNs and RNNs.

Input features of shape F × T are fed to the CNN which is composed of two blocks, each composed of two 2-D convolutional layers with kernel size 3 × 3 followed by ReLU activation and a 3 × 3 max-pooling subsampling operation. A total of 4 convolutional layers is thus employed with 64, 32, 128 and again 64 channels, respectively. Dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF] is applied on the output of the CNN and the representation is fed to an LSTM layer with a hidden size of 40. As an LSTM operates on 2-D sequences while the output of the CNN is a 3D tensor with channel, frequency, and time dimensions, a 2-D sequence is obtained by stacking the frequency dimension onto the channel dimension. [START_REF] Stöter | CountNet: Estimating the number of concurrent speakers using supervised learning[END_REF] performed an additional max-pooling operation on the whole time dimension in order to output a single prediction for the entire input because they aimed to count the maximum number of speakers in the whole sequence. Here, as explained in Section 2, we are interested in estimating the number of speakers in each time frame instead so we omit this final pooling layer. In this way, the network generates one output vector for every 6 feature vectors in input. As this architecture also originally lacked any normalization strategy, we added batch normalization after each convolutional layer and layer normalization at the input.

Temporal Convolutional Network

In addition to the above two state-of-the-art architectures, we consider a TCN architecture for the OSDC task. This type of architecture has been shown to achieve state-of-the-art performance in many sequence-related tasks [START_REF] Bai | An empirical evaluation of generic convolutional and recurrent networks for sequence modeling[END_REF] and for source separation [START_REF] Luo | Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF].

TCNs rely on multiple stacked dilated convolutional layers whose dilation factor increases progressively as depth increases. This makes it possible to greatly expand the receptive field, such that upper layers can have access to longterm contextual information without any pooling operation. This in turn allows TCNs to outperform recurrent models in some tasks [START_REF] Bai | An empirical evaluation of generic convolutional and recurrent networks for sequence modeling[END_REF]. In fact, because they are based only on convolutional operations, TCNs have several benefits with respect to RNNs. First, being feedforward, they are not affected by the vanishing gradient problem which plagues RNNs, as skip-connections and residual connections can be used to backpropagate the gradient unscathed down to the very first layers. Second, in RNNs the information about the past must be contained in the hidden state. This makes it difficult to learn very longterm dependencies as all relevant information about the past must be squeezed into this finite-sized representation. On the contrary, TCNs process the whole sequence and, because no downsampling is performed, the information at all steps is preserved in all layers. Finally, as no recurrent operations are employed, TCNs are significantly faster than recurrent models in both the training and inference phases. However, the fact that the representation is not pooled leads TCNs to have large memory requirements in general, especially if a very wide receptive field is desired.

The architecture we employ here [START_REF] Cornell | Detecting and counting overlapping speakers in distant speech scenarios[END_REF] is depicted in Fig. 1. It is inspired from MobileNet [START_REF] Howard | Mobilenets: Efficient convolutional neural networks for mobile vision applications[END_REF] and Conv-TasNet [START_REF] Luo | Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF]. Input frame-level feature vectors of size F (e.g., log-Mel filterbanks) are fed to a layer normalization [START_REF] Ba | Layer normalization[END_REF] layer followed by an F × 64 1D pointwise convolutional layer (denoted as conv 1x1 ) and by R = 3 blocks of X = 5 residual blocks (res blocks) with 1D dilated convolutions, where the dilation factor increases in each block as 2 0 , 2 1 , . . . , 2 X-1 . Each residual block consists of a 64 × 128 pointwise convolutional layer followed by batch normalization and activation, a dilated depthwise separable 128×128 convolutional layer (d-conv ) followed by batch normalization and activation, and another 128 × 64 pointwise convolution which squeezes the representation back so that it can be summed with the input. We use PReLU [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification[END_REF] as the activation function in all residual blocks and a kernel size of 3 in depthwise dilated convolutions.

Transformer

Finally, we propose a Transformer-based architecture for OSDC. Transformers, which were originally proposed by [START_REF] Vaswani | Attention is all you need[END_REF] for natural language processing applications, are pure attention-based models which have been shown recently to achieve state-of-the-art performance in many speech processing tasks including diarization (Fujita et al., 2019b). They have several advantages over recurrent models, including faster inference speed and better modeling of longterm dependencies. Being feedforward models, the whole sequence is attended at once, eliminating any recurrence and any need for an internal hidden state to keep track of past elements. For this reason, they exhibit the same advantages as TCNs over RNNs, even if their inherent functioning is significantly different. Similarly to TCNs and while being much faster than RNNs, Transformers also have higher memory requirements, due to the fact that the attention mechanism grows as O(n 2 ) in memory with n the length of the input sequence.

Our Transformer-based architecture is depicted in Figure 2 and, as it can be seen, has some input and output blocks in common with the previously described TCN network. To counter the quadratical memory growth induced by the attention mechanism, we adopt a concatenate-subsample (cat-pool ) operation over the input feature vectors. For each frame, we concatenate the feature vectors from C past frames and C future frames with the current one. Afterwards, we subsample this representation on the frame axis by a factor of S. In this way, the information contained in the temporal dimension is effectively transferred to the feature dimension with a resampling factor of C/S the original rate. This concatenated and pooled representation is then fed to a layer normalization layer followed by a pointwise convolutional layer (conv 1x1 ) which shrinks the representation to a predefined size H to reduce the memory requirements of subsequent blocks, allowing us to process longer sequences or, alternatively, to reduce the computational footprint of the model as it will be shown in Section 6.4. Sinusoidal positional encoding is added right after this bottleneck convolutional layer and the result is fed to a succession of R Transformer Encoder blocks, each composed of two residual sub-blocks.

The structure of each Transformer Encoder block is identical to the one proposed by [START_REF] Vaswani | Attention is all you need[END_REF] with the exception that, in our architecture, layer normalization is performed at the beginning of each residual block rather than in the end. Indeed, [START_REF] Nguyen | Transformers without tears: Improving the normalization of self-attention[END_REF] recently found that this results in better performance as well as faster convergence. The first residual block consists of a normalization layer followed by a Multi-Head Attention (MHA) layer and dropout. The second one consists of a normalization layer followed by a position-wise feedforward neural network (FFN) composed of one dense layer,2 a ReLU activation followed by dropout, and another dense layer which projects the hidden representation back. As in the TCN model, a final H × N pointwise convolutional layer followed by softmax is used at the output.

Spatial Features and Feature Fusion Schemes for OSDC

Intuitively, spatial features can help VAD, OSD and speaker counting. For example, OSD and speaker counting can benefit from knowing whether the sound comes from one or more Directions of Arrival (DoAs). VAD can also benefit from spatial features to distinguish speech, which is usually directional, from noise, which can be spatially diffuse.

In fact, as mentioned in Section 1.2, many works have tackled speaker counting by framing it as a localization problem. These works resort to DoA estimation methods based on generalized cross-correlation with phase transform (GCC-PHAT) [START_REF] Knapp | The generalized correlation method for estimation of time delay[END_REF] as in [START_REF] Brutti | Multiple source localization based on acoustic map de-emphasis[END_REF][START_REF] Drude | Source counting in speech mixtures using a variational em approach for complex watson mixture models[END_REF], magnitude-squared coherence (MSC) [START_REF] Pasha | Blind speaker counting in highly reverberant environments by clustering coherence features[END_REF] or simple crosspower spectrum [START_REF] Pavlidi | Source counting in realtime sound source localization using a circular microphone array[END_REF][START_REF] Walter | Source counting in speech mixtures by nonparametric Bayesian estimation of an infinite Gaussian mixture model[END_REF]. The speaker number is estimated via a direct approach such as in [START_REF] Brutti | Multiple source localization based on acoustic map de-emphasis[END_REF] by counting peaks in GCC-PHAT based acoustic maps or by clustering methods, where speaker clusters are identified by iterative grouping of complex-valued time-frequency coefficients [START_REF] Drude | Source counting in speech mixtures using a variational em approach for complex watson mixture models[END_REF], magnitude squared coherence feature vectors [START_REF] Pasha | Blind speaker counting in highly reverberant environments by clustering coherence features[END_REF], or DoAs estimated over single-source time-frequency zones [START_REF] Pavlidi | Source counting in realtime sound source localization using a circular microphone array[END_REF] or individual time-frequency bins [START_REF] Walter | Source counting in speech mixtures by nonparametric Bayesian estimation of an infinite Gaussian mixture model[END_REF].

Recently, a series of works have proven that neural network based localization is more robust than signal-based methods in reverberant and noisy environments. In these works, a neural network is trained to estimate the DoA on a synthetic dataset for which the true position of the sources is known. Input features include GCC-PHAT [START_REF] Xiao | A learningbased approach to direction of arrival estimation in noisy and reverberant environments[END_REF], cosine-sine interchannel phase difference (CSIPD) features [START_REF] Sivasankaran | Keyword-based speaker localization: Localizing a target speaker in a multi-speaker environment[END_REF], the phase spectra of all channels (phasemap) [START_REF] Chakrabarty | Broadband DOA estimation using convolutional neural networks trained with noise signals[END_REF], the magnitude and phase spectra [START_REF] Adavanne | Direction of arrival estimation for multiple sound sources using convolutional recurrent neural network[END_REF], or the raw waveform (Vecchiotti et al., 2019a).

Signal-based Spatial Features

In this paper, for what concerns signal-based spatial features, we explore the interchannel phase difference (IPD) and CSIPD, as they have been shown in the aforementioned works to work well in reverberant and noisy environments. In particular, our choice of IPD instead of phasemap is justified by the fact that, both in AMI and CHiME-6, microphones are close to each other and thus some microphone pairs can be discarded as they do not add much spatial diversity at 16 kHz. On AMI, we consider only those pairs of microphones with maximal distance from each other, i.e., the 4 pairs formed by opposite microphones in each circular array instead of all 28 possible pairs. On CHiME-6, due to the asymmetrical placement of microphones in Kinect devices, we consider the 3 pairs formed by channels 1 and 4, channels 2 and 4, and channels 3 and 4. The IPD or CSIPD features of all pairs are then concatenated together over the frequency dimension. Thus, in these contexts, using interchannel features allows us to reduce the feature size with respect to the phasemap and hence save computational resources with practically no loss in spatial information.

IPD and CSIPD features are tightly related and derive from the phase spectrum. Denoting by x i (n, f ) and x j (n, f ) the STFT of the i-th and j-th microphone signals, where n and f are respectively the frame and frequency index, the IPD φ i,j (n, f ) between channel i and j is given by

φ i,j (n, f ) = x i (n, f ) -x j (n, f ), (1)
where (.) is the function returning the phase from the input complex value. The IPD feature vector in time frame n is then defined as

IPD(n) = [φ i,j (n, 0), φ i,j (n, 1), . . . , φ i,j (n, F/2)] T , (2)
with F the FFT size. The CSIPD feature vector in time frame n can be obtained directly from the IPD feature vector and is another way of encoding the information contained in it by using its cosine and sine values:

CSIPD(n) = [cos φ i,j (n, 0), sin φ i,j (n, 0), . . . , sin φ i,j (n, F/2)] T . (3)
An important property of CSIPD is that the GCC-PHAT angular spectrum for a given microphone pair (or the SRP-PHAT spectrum when there are 3 or more microphones and all pairs are considered) can be expressed as a linear transformation of the CSIPD feature vector [START_REF] Sivasankaran | Localization guided speech separation[END_REF]. When these features are to be input to a neural network model, there is therefore no benefit in using the GCC-PHAT or SRP-PHAT angular spectra as inputs instead, since this linear transformation can be learned by the neural network itself. This was confirmed by our experiments, so we do not report results obtained with GCC-PHAT or SRP-PHAT features in the following.

Neural Network-based Localization Features

As an alternative, we also consider the strategy of training a neural network to estimate the DoAs of multiple overlapped speakers on a suitable synthetic dataset for which the true DoAs are known. The embeddings extracted by some intermediate layer of this network can then be used as "higher-level", possibly more robust spatial features to be employed in the OSDC system. In this work, we adopt the multi-speaker localization method of [START_REF] Chakrabarty | Broadband DOA estimation using convolutional neural networks trained with noise signals[END_REF], where the space of DoAs is discretized and the neural network is trained to estimate the posterior probability that a speaker is active for each discrete DoA by minimizing the sum of binary cross-entropies across all discrete DoAs. Binary cross-entropy is used as the cost function since multiple concurrent speakers with different DoAs can be active at the same time.

In detail, even for localization, we use the networks outlined in Section 3 by modifying the output layer which is replaced with mean pooling over the sequence dimension and a new linear layer with output size D followed by sigmoid activation, where D is the number of discrete DoAs considered. The network representation before the mean pooling operation is then employed as a spatial feature vector for OSDC systems.

One advantage of neural network-based features over signal-based features is that joint fine-tuning of the two networks can be performed, thus optimizing the localization feature extraction network for OSDC applications. However, it must be noted that the computational footprint significantly increases by using neural network based features. Also, the fact that true source DoAs are needed for training necessitates the use of a synthetic training dataset, which can be mismatched with real-world data.

Fusion schemes

Spatial features are not sufficient for reliable OSDC when used alone. For example, directional noise sources may sometimes be confused with speech, or concurrent speakers can have the same DoA. They must hence be combined with single-channel spectral features, such as log-Mel spectra. We consider two different fusion schemes for this combination, which we call early and late fusion.

These fusion schemes are illustrated in Figure 3 for the Transformer-based network. In early fusion, the two features are stacked together in the very first layer of the neural network. Layer normalization on spatial features is performed separately prior to concatenation. In late fusion, after layer normalization, the spatial features are injected before each Transformer Encoder Block (TE Block ), using Feature-wise Linear Modulation FiLM [START_REF] Perez | Film: Visual reasoning with a general conditioning layer[END_REF]. In this way, each block of the architecture can focus on a different aspect of the input spatial features since they are available even in deeper layers. As the spatial and singlechannel features are concatenated together in early fusion, they must have same temporal length. Thus, for proposed Transformer network we employ the same cat-pool operation also on spatial features prior to concatenation. The same argument applies also for late fusion where instead spatial features are used to modulate activations at multiple layers.

Datasets

We conduct experiments on two real-world multi-microphone datasets: AMI and CHiME-6. Moreover, we also use a synthetic dataset to further study, in a controlled situation, the use of spatial features to improve the performance of OSDC systems.

Synthetic Dataset

We simulate multi-speaker mixtures captured by a single microphone array.

Clean speech utterances are taken from Librispeech [START_REF] Panayotov | Librispeech: an ASR corpus based on public domain audio books[END_REF] train-clean-100 for training, dev-clean for validation, and test-clean for test. The Montreal Forced Aligner (MFA) [START_REF] Mcauliffe | Montreal Forced Aligner: Trainable text-speech alignment using Kaldi[END_REF] is used to split these original Librispeech utterances in order to obtain shorter "sub-utterances" for each speaker. This splitting is performed whenever pauses of more than 150 ms are encountered. MFA is also used, in parallel, to obtain ground truth word-level speaker activity. For each mixture, we sample from 1 to 4 different speakers, and, for each speaker one clean speech sub-utterance is sampled. The starting time of each speaker sub-utterance is sampled independently from an exponential distribution. In this way, by varying the decay rate parameter, the amount of overlap between the speakers and the amount of silence can be controlled. A different acoustic scenario is sampled for each mixture. We simulate a rectangular room whose size is varied between 10 and 60 m 2 . The position of each speaker is chosen randomly inside the room but with some constraints. Namely, the speakers cannot be less than 0.5 m from each other and from the walls. We consider a 4-microphone linear array placed randomly with respect to the walls, whose height with respect to the floor can vary between 1.7 and 2 m and whose distance to the closest wall is larger than a minimal distance which is varied between 10 and 30 cm. We use the gpuRIR [START_REF] Diaz-Guerra | gpuRIR: A Python library for room impulse response simulation with GPU acceleration[END_REF] toolkit for room simulation with a T60 reverberation time uniformly sampled between 0.2 and 0.6 s. Anechoic noise from [START_REF] Furnon | Distributed speech separation in spatially unconstrained microphone arrays[END_REF] is also employed to make the dataset more realistic. The positions of noise sources inside the room are selected with the same criteria as the speakers' ones. The whole synthetic dataset consists of a total of 10 k mixtures (∼ 23 hours) for training, 2 k for validation and 2 k for test (∼ 4.6 hours).

AMI

The AMI Corpus [START_REF] Mccowan | The AMI meeting corpus[END_REF] is over 100 h of meeting recordings. Each meeting has been recorded by a variety of devices including cameras, microphone arrays, and per-speaker headset and lapel microphones and has from 3 to 5 participants. Ground truth speaker activity was obtained by human annotators from close-talk speaker-worn microphones while distant speech was recorded by two 8-microphone circular arrays, each with a 10 cm diameter: one placed at the end and another at the centre of the meeting table used by the participants.

CHiME-6

The CHiME-6 corpus comprises dinner party recordings. The recordings are divided into 20 sessions for a total of more than 60 h of data. In each session, 4 speakers are recorded in a real home environment consisting of different rooms. Due to the particular setting, it features conversational speech and low Signalto-Noise Ratio (SNR). Recordings from binaural microphones worn by each speaker are provided along with distant speech captured by 6 array devices with 4 microphones each for a total of 24 microphones. Two different annotations are provided for the start and end time of every utterance: looser ones geared towards Automatic Speech Recognition (ASR) and tighter ones obtained via forced-alignment. The latter ones are more suitable for evaluating VAD and diarization systems and we use them in the following.

Experimental Results

In the following, we evaluate the neural architectures in Section 3 and the spatial features and feature fusion schemes in Section 4 on the datasets described in Section 5. Firstly, in Section 6.1, we define and motivate the chosen performance metric. In Section 6.2, we outline the training and testing procedure adopted in our experiments and, in Section 6.3, we highlight the impact of different choices of hyperparameters and single-channel input features for the Transformer-based architecture. Then, in Section 6.4, we provide an analysis of the computational footprint of the four considered neural architectures when applied to single-channel data and, in Section 6.5, we report their OSDC performance on AMI and CHiME-6. Finally, in Section 6.6, we assess the impact of spatial features on the best single-channel system: we explore different spatial features, fusion schemes and number of microphone pairs, and evaluate the results on AMI, CHiME-6 and the proposed synthetic dataset.

Evaluation Metric

On real-world data, VAD, OSD and speaker counting tasks are affected by class imbalance. This imbalance, which arises from intrinsic characteristics of human conversations, can be more or less severe depending on the context. This can be seen in Table 1, which reports the class statistics on AMI and CHiME-6 for the counting task.3 Due to its informal, "cocktail-party" scenario, the CHi-ME-6 dataset exhibits a slightly higher proportion of overlapped speech than the AMI dataset, which consists of meetings. Nevertheless, in both datasets, the proportion of 4-speaker and 3-speaker overlap is very small. The imbalance is less severe for VAD and OSD tasks but, even for these, the choice of the evaluation metric can be crucial. We argue that metrics such as accuracy and precision-recall, as used respectively by [START_REF] Sajjan | Leveraging LSTM models for overlap detection in multi-party meetings[END_REF] and by [START_REF] Kunešová | Detection of overlapping speech for the purposes of speaker diarization[END_REF] and [START_REF] Bullock | Overlap-aware diarization: Resegmentation using neural end-to-end overlapped speech detection[END_REF], do not provide a fair evaluation of OSDC algorithms on real-world data due to this fundamental imbalance. For example, concerning OSD on the AMI evaluation set, an accuracy of 83.7% can be reached by labeling all the material as no-overlap. As it has been observed by Cornell et al. (2020, Table 5), this leads to small accuracy differences even for classifiers with drastically different performance. In this scenario, precision and recall are a better choice than accuracy. However, similarly to accuracy, their value depends on the choice of the detection threshold which can be application-specific (e.g., a different threshold for diarization and speech recognition is often desirable). This does not allow for a fair comparison between different OSDC algorithms.

For these reasons, we propose the use of Average Precision (AP) metric which summarizes the precision-recall curve and is widely used, for example, in object segmentation [START_REF] Lin | Microsoft COCO: Common objects in context[END_REF], information retrieval [START_REF] Kishida | Property of average precision and its generalization: An examination of evaluation indicator for information retrieval experiments[END_REF] and other tasks exhibiting strong class imbalance. It can be obtained from precision P and recall R at the k-th threshold as:

AP = M k=1 (R k -R k-1 )P k , ( 4
)
where M is the total number of unique thresholds considered. The number of elements in this set is upper bounded by the number of unique elements in the classifier output probabilities vector. In this work we use each time the maximum number of possible thresholds to compute AP. As it can be seen, AP has the advantage that it does not depend on a particular threshold, making it both more robust to imbalanced data and more suitable for comparison purposes. In all experiments, AP scores are computed on 10 ms time frames.4

AP in Equation ( 4) is suitable only for binary classification tasks such as VAD and OSD. However, it can be extended easily to multi-class classification problems such as Speaker Counting with a leave-one-out classification strategy: e.g. AP for the 1-spk class can be obtained by considering an equivalent binary task where the true positives are the frames correctly classified as 1-speaker and the true negatives are the frames classified as silence (0-spk), 2-speaker (2-spk), 3-speaker (3-spk), or 4-speaker (4-spk). In a similar way one can compute VAD and OSD AP from a neural network trained to perform Speaker Counting or VAD+OSD. For example, for a Speaker counting algorithm, VAD predictions can be obtained by summing the probabilities for the classes with at least one speaker: 1-spk, 2-spk, 3-spk, and 4-spk, thus obtaining the total probability of speech; OSD by summing classes with at least 2 speakers: 2-spk, 3-spk, and 4-spk.

Unless stated otherwise, in each Table , we highlight in bold font the best result and the ones which are statistically equivalent to it (if any) with p = 0.001.

Because we found the distribution of the AP metric to be highly non-gaussian, we use the Wilcoxon-Signed Rank non-parametric test [START_REF] Demšar | Statistical comparisons of classifiers over multiple data sets[END_REF].

Training and Testing Procedure

In the following experiments, we use the exact same training and testing procedure as in our preliminary work [START_REF] Cornell | Detecting and counting overlapping speakers in distant speech scenarios[END_REF]. This allows the results to be directly comparable. In detail, we train all models using RAdam [START_REF] Liu | On the variance of the adaptive learning rate and beyond[END_REF] on 5 s chunks obtained from training signals with 50% overlap. The last chunk is discarded if shorter. Hyperparameters such as batch size, learning rate and dropout rate are tuned for each network, dataset and training objective (speaker counting or VAD+OSD) on the development set. Inference is performed by using a sliding window approach is used where the logits of overlapping blocks are averaged to obtain the final estimate. Popular speech processing toolkits such as Pyannote [START_REF] Bredin | Pyannote.audio: neural building blocks for speaker diarization[END_REF] use this approach. In this work we use a sliding window of 3 s with 50% overlap.

In our preliminary work [START_REF] Cornell | Detecting and counting overlapping speakers in distant speech scenarios[END_REF], we found that using training targets obtained via Forced-Alignment (FA) brings considerable improvement even when manual annotation is used as the ground truth in the testing phase. We also studied the efficacy of FA as an automatic labeling procedure for speech segmentation applications using synthetic data and we found that, when close-talk worn microphones are employed, it can be considered reliable even in overlapped speech regions and challenging SNR conditions. Thus, we employ FA labels to train OSDC models on both AMI and CHiME-6. In detail, we use the Kaldi [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF] recipes for AMI and CHiME-6 and get the segmentation from the tri3 GMM-HMM speech recognition model. The results on the test set are evaluated using the official annotation, which is manual in the case of AMI and FA-based in the case of CHiME-6. In fact, the FA-based annotation of the CHiME-6 development and evaluation sets was obtained with similar FA procedure as used here. Moreover, to further improve performance on real-world data and counteract class imbalance, we resort, in our experiments, to the data-augmentation strategy described by [START_REF] Cornell | Detecting and counting overlapping speakers in distant speech scenarios[END_REF], where it was shown to bring significant improvements. This data-augmentation technique, which is itself an extension of the one proposed by [START_REF] Bullock | Overlap-aware diarization: Resegmentation using neural end-to-end overlapped speech detection[END_REF], consists of on-the-fly creation, at training time, of new concurrent speaker examples by overlapping 2, 3, and 4 random single-speaker chunks from the original dataset in order to re-balance the classes. To further increase the training material, a random gain factor sampled from N (µ = -16.7, σ = 4) in dB scale is applied to each chunk independently. In this way, we augmented the original AMI data by a factor of 70% and CHiME-6 data by 40 %. This augmentation factor is tuned for each dataset using the development set. In parallel, to improve generalization, we also use SpecAugment [START_REF] Park | SpecAugment: A simple data augmentation method for automatic speech recognition[END_REF] on both single-channel and spatial features separately.

Choice of Transformer Hyperparameters and Single-Channel Features

In Table 2, we show the hyperparameter space explored for the proposed Transformer-based architecture. We varied number of future and past frames (C) and subsampling factor (S) used in cat-pool operation as well as size of hidden representation (H), number of attention heads, size of feed-forward neural network hidden layer (FFN size) and number of transformer encoder blocks (R). The hyperparameters were tuned on the development set of AMI, for fair comparison with [START_REF] Sajjan | Leveraging LSTM models for overlap detection in multi-party meetings[END_REF] who also optimized his LSTM model on AMI.

The models were trained to perform VAD+OSD according to the framework introduced in Section 2. The best combination was selected using two criteria: overall VAD+OSD performance and inference-time computational footprint, to give an overview of how much demanding the model is when used in practical applications. In fact, if the OSDC model has a modest computational burden, using it at the very first stage of a speech processing pipeline has the advantage of lowering the computational requirements of the whole pipeline, as subsequent processing can be applied only when needed. Moreover, models with modest computational requirements allow for deployment on mobile and edge-computing devices. In Table 3, we show the VAD and OSD performance on the AMI development set, as well as the total number of floating point operations (FLOP) and total memory consumption (Mem) with the best combination of hyperparameters (Best) and when changing the value of one hyperparameter at a time. FLOP and Mem are computed with a 300-frame (3 s) dummy 80-dimensional feature sequence (matching the 80 log-Mel features used in the following experiments), generated from a uniform distribution. These computational footprint figures are estimated using the built-in profiler in the Pytorch toolkit and the Performance Application Programming Interface [START_REF] Terpstra | Collecting performance data with PAPI-C[END_REF]. Several observations can be made. First, the choice of hyperparameters does not affect the VAD performance, which is arguably a simpler task than OSD and is more easily tackled by the network. Second, doubling the number of Transformer Encoder blocks only marginally improves performance at the cost of a significant increase of the computational footprint. Third, increasing time resolution by halving the sub-sampling rate also significantly increases the computational requirements without bringing significant benefits, meaning that a resolution in the order of 100 ms is enough in the application scenario considered here. In Table 4, we report the results achieved by the proposed Transformer-based architecture on the AMI development set for different choices of single-channel input features. In the past, [START_REF] Sajjan | Leveraging LSTM models for overlap detection in multi-party meetings[END_REF] 2019) explored magnitude STFT spectra, log spectra and 40 Mel-scale filterbanks. In both studies, the features were extracted with a 25 ms window and 10 ms hop-size. Hereafter, we consider magnitude spectra computed over 32 ms and 64 ms windows (512 and 1024 samples respectively), 40 and 80 log-Mel, 40 and 80 gammatone filterbanks, and 20 and 40 MFCCs instead. All these features were computed with a 10 ms hop-size. Regarding MFCCs, we used 20 and 40 Mel bands, respectively. A window of 25 ms was used for log-Mel, gammatone and MFCCs. We can see that OSD and to a lesser extent VAD performance correlate with frequency resolution. In fact, especially for OSD, the use of compact features such as MFCCs, 40 log-Mel or 40 gammatone filterbanks leads to a loss in performance. These results partially agree with the findings of [START_REF] Sajjan | Leveraging LSTM models for overlap detection in multi-party meetings[END_REF], who found 64 gammatone filterbanks to be superior to 40 log-Mel features for OSD. Because no statistical difference was found between 80 gammatones and 80 log-Mel and higher-resolution features (e.g., 64 ms magnitude spectra) did not result in higher performance, we ultimately decided to use 80 log-Mel features in the following.

Computational Footprint Comparison Across Architectures

In Figure 4 we report the total number of floating point operations (FLOP), the total memory usage and the inference time in clock cycles for the four considered network architectures as a function of the input signal duration from 1 s to 100 s. Inference time is computed over batches of 64 examples in order to get reliable estimates. As we are interested in comparing only the architectures, we use the same single-channel features for all architectures, namely 80 log-Mel features with 25 ms window and 10 ms hop-size. An Intel i9-10920X CPU is employed to perform the comparison.

As expected, regarding inference speed, the RNN-based architectures (LSTM and CRNN) are slower than the TCN and the Transformer, which do not employ recurrence. A similar trend is observable in the FLOP plot, with the difference that the CRNN has a much higher FLOP count than the other architectures due to the use of 2-D convolutions, despite the fact that it is slightly faster than the LSTM architecture as it employs pooling operations and the CNN part is parallelizable. The use of 2-D convolutions also increases the CRNN memory footprint with respect to the other architectures. The small number of parameters employed in the TCN leads to similar memory footprint as the LSTM architecture.

Overall, the proposed Transformer architecture is the most efficient according to the three criteria despite having the second largest number of parameters 625 after the LSTM. Due to the cat-pool operation, the total memory usage is kept contained and grows almost linearly until a duration of 100 s. In practice, due to the fact that OSDC typically requires a context of a few seconds only, inference is never performed directly over such long signals. In fact, a sliding window approach, as explained in Section 6.2 is employed. More generally, all architectures, including previously proposed LSTM and CRNN ones, attain overall computational resources figures which are suitable for edge devices deployment. For example, the FLOP count for one second of audio is comparable to the one reported by [START_REF] Choi | Temporal convolution for real-time keyword spotting on mobile devices[END_REF] for keyword spotting in smartphone devices.

An important take from these results is also that the number of parameters, which is widely used as a gauge for model computational burden, does not correlate well with the latter and can be deceptive when comparing very different architectures.

Single-Channel Experimental Results

We now evaluate the performance achieved by the four architectures on the AMI and CHiME-6 distant speech datasets. For the sake of comparison with [START_REF] Sajjan | Leveraging LSTM models for overlap detection in multi-party meetings[END_REF] and [START_REF] Stöter | CountNet: Estimating the number of concurrent speakers using supervised learning[END_REF], we use single-channel features only, namely 80 log-Mel features with 25 ms window and 10 ms hop-size. Each architecture is trained and evaluated according to two different tasks: VAD+OSD and speaker counting. Indeed, we are interested in assessing the feasibility of VAD+OSD and speaker counting on real-world data. Speaker counting, as already said, has the advantage of providing more information to downstream tasks, but it is plagued by extreme class imbalance. VAD+OSD, by contrast, does not provide any clue about concurrent speakers, but exhibits a less extreme class imbalance.

Concerning AMI, to allow direct comparison with previous works [START_REF] Sajjan | Leveraging LSTM models for overlap detection in multi-party meetings[END_REF][START_REF] Cornell | Detecting and counting overlapping speakers in distant speech scenarios[END_REF], data from all microphone channels is used during training while testing is performed on the first microphone of array 1. Regarding CHiME-6, training is also performed using all microphone channels from all array devices but, when evaluating, we consider for each array the first channel and then average the outputs of single-channel systems across all arrays because of the multi-room environment of CHiME-6. 5In Table 5, we report the VAD and OSD results obtained when training the models with a VAD+OSD objective. It can be seen that the AP figures on both datasets are considerably higher for VAD than for OSD. This is expected since OSD is inherently a more challenging task than VAD. As also expected, the performance is better on AMI than CHiME-6, as CHiME-6 is arguably a much more challenging dataset, having lower SNR due to the more unconstrained setting. The proposed Transformer architecture performs on-par or better than the other architectures, with the TCN architecture closely following. LSTM and CRNN perform significantly worse, despite the addition of normalization layers which were not present in the respective original works of [START_REF] Sajjan | Leveraging LSTM models for overlap detection in multi-party meetings[END_REF] and Stöter et al. (2019). 6 Table 5: VAD and OSD AP (%) achieved by the four considered neural network architectures on the AMI and CHiME-6 evaluation sets using single-channel features and VAD+OSD as a training objective. Similarly, Tables 6 and7 report the speaker counting results achieved on the evaluation sets of AMI and CHiME-6, respectively, when training the models with a counting objective. The fact that the AP for the 0-spk class is remarkably lower on AMI is a rather unexpected result, as it features a much higher SNR than CHiME-6 overall. This could be explained by class imbalance since, as reported in Table 1, the proportion of 0-spk in AMI is significantly lower than in CHiME-6. The proposed Transformer architecture achieves the best figures overall on both datasets. In general, compared to the 0-spk and 1-spk classes, the AP degrades considerably for the 2-spk class and even more so for the 3spk and 4-spk classes. This suggests that the data-augmentation strategy, is only able to partially compensate for the extreme imbalance of 3-spk and 4-spk classes. Therefore, it can be said that speaker counting is still far from being reliable on real-world data. In Table 8 we compare the performance of Transformer models trained to perform either VAD, OSD, VAD+OSD or counting for the VAD and OSD tasks. For each dataset, we report the evaluation set performance and, in parentheses, the development set performance. Regarding VAD, the choice of the training objective has little impact on performance on all datasets. Regarding OSD, interestingly, the model trained to perform speaker counting, which is inherently a more difficult task, leads to better OSD performance than the model trained directly with a VAD+OSD or OSD objective on the AMI development and evaluation sets and on the CHiME-6 evaluation set. This is especially evident In the following, we select the best model found in Section 6.5, namely the proposed Transformer model trained with a speaker counting objective, and we show how its performance can be improved by employing spatial features along with single-channel features. To do so, we evaluate the IPD, CSIPD and neural network-based spatial features and the early and late fusion schemes described in Section 4 using AMI, CHiME-6 and the proposed synthetic dataset.

VAD+OSD

In order to allow direct comparison with single-channel results, we adopt the same training strategy as above. Data augmentation is extended to the multichannel scenario by overlapping multichannel audio chunks and being careful, when mixing, in maintaining the array topology (i.e., the first channel is always mixed with the first channel). Training is performed by considering each array separately and using the same FA-based targets as above. Testing is performed, on AMI and CHiME-6, by averaging the predictions made independently for each array across all arrays (i.e., 2 devices for AMI and 6 for CHiME-6).

The IPD and CSIPD features are computed with an STFT window length of 50 ms and the same 10 ms hop-size as single-channel log-Mel features. The corresponding feature vectors, for each microphone pair, are thus of size 801 and 1602, respectively.

Neural network based localization features are extracted using the same Transformer-based architecture as for OSDC, but with R = 2 and the modifications outlined in Section 4.2. The network takes CSIPD features relative to most distant microphone pairs with the same STFT window length and hop-size as above, and it outputs D = 181 discrete DoAs. It is trained on matched synthetic datasets. More specifically, concerning AMI, we use our synthetic dataset by simulating a circular array instead of the linear one and compute CSIPDs over the 4 pairs obtained by taking opposing microphones in the circular array.

Regarding CHiME-6, we perform training on the Kinect-WSJ2Mix dataset [START_REF] Sivasankaran | Analyzing the impact of speaker localization errors on speech separation for automatic speech recognition[END_REF] which involves simulated Kinect devices and real CHiME-6 noise and we use CSIPD features between the 3 microphone pairs with largest distance, as explained in Section 4.1. Because Kinect-WSJ2Mix involves at most 2 overlapping speakers while in CHiME-6 up to 4 concurrent speakers can be present, we extend the original data by creating on-the-fly mixtures of up to 4 overlapped speakers and use this newly generated data to train the localization network.

Regarding the experiments performed on the synthetic dataset, we also use CSIPD features between the 3 microphone pairs with largest distance as inputs. Contrary to the AMI and CHiME-6 real-world datasets, in which the localization network is trained on a separate dataset, here we use the same synthetic data for both the OSDC and the localization network.

In addition, to avoid possible domain mismatch between the simulated training dataset for the localization network and the test dataset for the OSDC network, we fine-tune the localization network with the OSDC model by joint optimization with respect to the speaker counting task on the OSDC training dataset. This fine-tuning step is critical to achieve good performance when applying the OSDC network to real-world datasets: for example, on CHiME-6 without fine-tuning the resulting AP is in the order of 50% only. We summarize the datasets used for training the neural localization network, fine-tuning and testing with the back-end OSDC system in Table 9. In Tables 10 and11, we report the performance achieved for the VAD and OSD tasks, respectively, with different spatial features, fusion schemes, and numbers of microphone pairs. Microphone pairs are selected as described in Section 4.1, by considering, as the upper bound (all ), only pairs which add significant spatial diversity, i.e., from 1 to 4 pairs formed by opposing microphones in AMI and from 1 to 3 pairs in CHiME-6 and the synthetic dataset.

We also include in the comparison of a single-channel ensemble system with no spatial features, where ensembling is done by averaging the OSDC network outputs over all microphones in the array and a single-channel system trained on beamformed audio using BeamformIt [START_REF] Anguera | Acoustic beamforming for speaker diarization of meetings[END_REF]). For what concerns VAD performance in Table 10, it can be seen that neural network-based localization features result in on-par or higher performance than the other spatial features, and they outperform single-channel systems by a significant margin on CHiME-6 and the synthetic dataset. Regarding AMI, the AP saturates for most models due to the fact that, as noted previously in Section 6.5, silence is under-represented in the material. An interesting trend which appears on CHiME-6 and synthetic data is that the performance of signalbased spatial features improves when increasing the number of microphone pairs and by using late fusion. Especially on models with late fusion, using more microphones considerably boosts the performance for IPD and CSIPD features. Instead, a smaller improvement is noticeable when early fusion is employed, due to the fact that the size of CSIPD and IPD features grows linearly with the number of pairs but the bottleneck convolutional layer applied in early fusion maps them to a fixed-size representation (384 neurons, as reported in Table 2). Thus some information is inevitably lost in early fusion. On top of that, in late fusion spatial features are available at multiple stages of the architecture. Similar trends can be also observed for OSD performance in Table 11 regarding the number of microphone pairs and early fusion versus late fusion. Notably, neural network-based spatial features are outperformed by signal-based ones on AMI and CHiME-6 when late-fusion is used but reach on-par or top performance when early fusion is employed instead. This suggests that fine-tuning the localization network compensates for the synthetic/real domain mismatch only up to a certain point regarding OSD. It can also be observed that the performance gain achieved by late fusion with respect to early fusion appears modest for neural spatial features, while it is substantial for signal-based ones. This is explained by the fact that neural network-based features are less affected by the aforementioned "bottleneck issue" in early fusion, as they have a more compact size than signal-based ones and, moreover, are jointly fine-tuned with the OSDC system. Again, models with spatial features are able to outperform the single-channel systems and ensembles of single-channel systems. This is notable, as the ensemble is performed using all channels in the array and it comes at the cost of increasing the computational footprint linearly in the number of channels. By contrast, spatial features allow us to boost performance with a smaller increase in computational requirements. The use of beamformed audio degrades OSD performance but not VAD performance with respect to the single-channel only baseline system. This could be explained by the fact that BeamformIt tends to enhance the source with the highest energy and attenuate the rest.

In Tables 12 and13 we report the counting performance achieved for different spatial features on AMI and CHiME-6, respectively, using two microphone pairs and late fusion. On both datasets, a similar trend can be noticed. On the one hand, neural network based localization features achieve the best figures regarding the 0-spk and 1-spk classes which are the most represented ones. This is in accordance with the VAD results in Table 10 where neural spatial features have in general higher scores. On the other hand, CSIPD and IPD obtain similar or higher AP values for 2 and 3 concurrent speakers. This is in accordance with the OSD results in Table 11. Nonetheless, while systems based on spatial features are able to substantially increase the speaker counting performance over single-channel systems, the observations made in Section 6.5 are still valid, and reliable speaker counting remains out of reach on real-world data. Finally in Figure 5 we use the synthetic dataset to further explain the benefit of spatial features. Using mixtures of two speakers, we report the OSD AP values obtained by the system using single-channel features only versus the ones obtained with late fusion and CSIPD features computed using the 3 microphone pairs with largest distance. The OSD AP performance is plotted against the mean distance of the two speakers from the array and the angle between them as seen from the array. It can be seen that, for the single-channel model, performance degrades to some extent as the speaker distance increases (i.e., colors become darker from bottom to top), but it is largely independent of the angle between the speakers. By contrast, for the model employing spatial features, performance still degrades as the speaker distance increases but at the same time it clearly improves as the angle between the speakers increases (i.e., colors become lighter from left to right). In fact, the AP is significantly boosted for angles greater than 30 degrees, indicating that spatial features offer complementary information which allows the model to more effectively discriminate frames with overlapped speech.

Conclusions

In this paper we studied the problem of performing VAD+OSD and speaker counting on real-world data featuring distant microphone arrays. We focused on neural network based approaches and compared different architectures for the two tasks, on AMI, CHiME-6 and a purposedly developed synthetic dataset. Among the neural networks compared we introduced two novel architectures: one based on TCNs and another based on the Transformer. In parallel we explored the use of spatial features, both signal-based and neural-based, to aid in the VAD+OSD and speaker counting tasks when multiple microphones are available. We conducted an extensive experimental evaluation by comparing the models' computational footprint and VAD, OSD and counting performance on single-channel and multichannel distant speech data. On CHiME-6, our proposed TCN and Transformer-based architectures achieve an absolute improvement in AP of 15% and 16% over previous techniques, respectively. Overall, we found the proposed Transformer-based architecture to be the most promising as it was shown to be able to reach on-par or better results than the other architectures with a significantly lower computational footprint. In general, in comparing VAD+OSD and speaker counting tasks we found that, due to class imbalance, speaker counting performs poorly on real-world data, but, on the other hand, it is desirable to use a speaker counting objective to train a system to perform VAD+OSD as it is shown to improve OSD. Finally, concerning spatial features, we found that significant further improvements can be obtained by using a late-fusion strategy and by increasing the number of microphone pairs considered. Neural-based spatial features show a clear advantage over signalbased ones for VAD across all datasets, but no spatial feature shows a clear advantage over another for OSD or counting. Future work includes fusing estimates over multiple arrays in a way that favors arrays closer to the speakers and exploits the relative positions and orientations of the arrays whenever they are known, and exploring suitable techniques to counteract the class imbalance problem.

Figure 1 :

 1 Figure 1: Proposed TCN architecture for the OSDC task.

Figure 2 :

 2 Figure 2: Proposed Transformer architecture for the OSDC task.

Figure 3 :

 3 Figure 3: Fusion strategies for single-channel features and spatial features for the proposed Transformer architecture: a) early fusion, b) late fusion. TE stands for Transformer Encoder.

  and Stöter et al. (2019) explored different single-channel features for the LSTM and CountNet architectures: Sajjan et al. (2018) used gammatone filterbanks, log-Mel and other features such as kurtosis and spectral flatness, while Stöter et al. (

Figure 4 :

 4 Figure 4: Inference-time computational footprint for the four considered neural network architectures as a function of the input signal duration. Top: number of floating point operations (FLOP). Middle: Total memory usage in GB. Bottom: number of CPU clock cycles. The numbers in parentheses in the legend indicate the number of model parameters. The two axes are in log-scale.

Figure 5 :

 5 Figure 5: OSD AP (%) achieved on the synthetic evaluation set by the Transformer-based architecture trained with a speaker counting objective as a function of the mean distance of the speakers from the array and the angle between the speakers. Left: single-channel features only. Right: CSIPD spatial features and late fusion.

Table 1 :

 1 Frame-level class frequency (%) for the speaker counting task on the AMI and CHi-ME-6 development and evaluation sets.

	Class frequency 0-spk 1-spk 2-spk 3-spk 4-spk
	AMI	dev 15.87 67.17 13.95 2.59 eval 15.12 68.39 12.63 3.09	0.42 0.76
	CHiME-6	dev 24.05 54.25 17.74 3.49 eval 33.47 51.52 12.03 2.46	0.47 0.51

Table 2 :

 2 Hyperparameter space explored for the Transformer-based architecture. The best combination of hyperparameters is highlighted in bold.

	Hyperparameter	C	S	H	heads	FFN size	R
	Values	(7, 5) (10, 5) (256, 384) (4, 8, 16) (1024, 2048) (2, 4, 8)

Table 3 :

 3 VAD and OSD AP (%) and computational footprint of the Transformer-based architecture on the AMI development set for different architecture hyperparameter values.

	Model Parameters	FLOP [10 6 ]	Mem [10 6 ]	AP
				VAD	OSD
	Best	85.6	3.3	98.5	57.4
	S = 5	166.8	6.9	98.5	57.5
	R = 8	161.0	6.2	98.5	57.8
	heads = 8	85.4	3.6	98.5	56.9
	FFN size = 2048	153.1	5.1	98.5	57.6

Table 4 :

 4 VAD and OSD AP (%) achieved by the Transformer-based architecture on the AMI development set with different choices of single-channel features.

	AP	MagSpec	Log-Mel	Gammatone	MFCC
		512	1024	40	80	40	80	20	40
	VAD	98.5	98.5	98.4	98.5	98.4	98.5	98.3	98.4
	OSD	61.1	61.0	58.2	61.0	58.0	59.8	56.8	58.4

Table 6 :

 6 Speaker counting AP (%) achieved by the four considered neural network architectures on the AMI evaluation set using single-channel features and counting as a training objective.

	Counting Model 0-spk 1-spk 2-spk 3-spk 4-spk
	LSTM	47.0	82.4	24.7	6.4	0.02
	CRNN	49.8	84.2	34.8	9.2	0.03
	TCN	50.7	86.1	40.4	11.3	0.03
	Transformer	50.9	87.2	41.8	11.2	0.03

Table 7 :

 7 Speaker counting AP (%) achieved by the four considered neural network architectures on the CHiME-6 evaluation set using single-channel features and counting as a training objective. where a larger gap between the two models is observed. So, while speaker counting performs poorly on real-world data, it can be convenient to use models trained to perform speaker counting to perform VAD and OSD instead. This may be explained by the fact that speaker count labels provide the model with more information during training than mere OSD labels.

	Counting Model	0-spk	1-spk	2-spk	3-spk	4-spk
	LSTM	79.1	69.7	20.5	6.1	0.002
	CRNN	86.2	73.8	25.4	8.5	0.003
	TCN	88.3	77.3	30.0	12.3	0.003
	Transformer	88.2	77.3	30.6	12.5	0.003
	on AMI,

Table 8 :

 8 VAD and OSD AP (%) achieved by the Transformer-based architecture on the AMI and CHiME-6 development and evaluation sets when using single-channel features and either VAD, OSD, VAD+OSD or counting as the training objective. The values obtained on the development sets are in parentheses.

	Method	VAD		OSD
		AMI	CHiME-6	AMI	CHiME-6
	Transformer-VAD	98.5 (98.6) 94.3 (93.2)	n.a.	n.a.
	Transformer-OSD	n.a.	n.a.	57.8 (61.0) 50.2 (55.4)
	Transformer-VAD+OSD 98.5 (98.6) 94.3 (93.1) 57.8 (61.0) 49.9 (55.1)
	Transformer-Counting 98.5 (98.5) 94.3 (93.2) 59.1 (64.3) 50.8 (55.8)
	6.6. Multichannel Experimental Results

Table 9 :

 9 Datasets used for the neural localization network experiments: training (train), finetuning (adapt) with OSDC back-end and testing (test) dataset splits. The total number of hours for each dataset is reported in parenthesis.

		Datasets
	Localization Network	OSDC Network
	train	adapt	test
	Synthetic (23h)	AMI (81h)	AMI (9h)
	Reverberated WSJ-2mix (47h)	CHiME-6 (40.3h)	CHiME-6 (5.2h)
	Synthetic (23h)	Synthetic (23h)	Synthetic (4.6h)

Table 10 :

 10 VAD AP (%) achieved on the AMI, CHiME-6 and synthetic evaluation sets by the Transformer-based architecture trained with a speaker counting objective for different spatial features, fusion schemes, and numbers of microphone pairs (1, 2 or all), as compared to singlechannel features only (None, 1 ch.), an ensemble of single-channel systems (None, all ch.) and a single-channel system + BeamformIt (None, enh).

	Dataset	Fusion	IPD			CSIPD		Neural	None
		1	2	all	1	2	all	all	1 ch. all ch. enh
	AMI	early 98.6 98.7 98.7 98.6 98.7 98.7 late 98.6 98.7 98.7 98.6 98.7 98.7	98.7 98.7	98.5	98.6	98.5
	CHiME-6	early 94.7 94.8 94.8 94.7 94.9 95.1 late 94.8 95.4 95.4 94.9 95.4 95.4	95.4 95.5	94.3	94.5	94.3
	Synth	early 96.3 96.8 97.2 96.1 96.4 96.8 late 96.5 97.2 97.4 96.3 97.1 97.4	97.5 97.5	96.4	96.6	96.4

Table 11 :

 11 OSD AP (%) achieved on the AMI, CHiME-6 and synthetic evaluation sets by the Transformer-based architecture trained with a speaker counting objective for different spatial features, fusion schemes, and numbers of microphone pairs (1, 2 or all), as compared to singlechannel features only (None, 1 ch.), ensemble of single-channel systems (None, all ch.) and a single-channel system + BeamformIt (None, enh).

	Dataset	Fusion	IPD			CSIPD		Neural	None
		1	2	all	1	2	all	all	1 ch. all ch. enh
	AMI	early 58.1 58.6 59.4 57.8 58.4 58.9 late 58.4 59.5 60.3 58.1 59.6 60.4	59.3 59.7	57.8	58.6	57.6
	CHiME-6	early 51.4 51.5 51.6 51.3 51.4 51.5 late 51.6 52.4 52.4 51.7 52.3 52.2	51.8 51.9	50.8	51.2	50.2
	Synth	early 81.8 82.3 82.7 81.6 82.0 82.4 late 82.8 83.4 84.2 82.9 83.6 84.4	83.8 84.3	82.4	83.1	82.1

Table 12 :

 12 Speaker counting AP (%) achieved on the AMI evaluation set by the Transformerbased architecture trained with a speaker counting objective for different spatial features, as compared to single-channel features only (None, 1 ch.) or an ensemble of single-channel systems (None, all ch.).

	Spatial Features	0-spk	1-spk	2-spk	3-spk	4-spk
	IPD	52.8	88.3	45.0	12.8	0.03
	CSIPD	52.9	88.4	45.1	12.7	0.03
	Neural	53.1	88.8	44.9	11.8	0.03
	None, 1 ch.	50.9	87.2	41.8	11.2	0.03
	None, all ch.	51.3	87.9	42.4	11.5	0.03
	None, enh	50.8	87.4	41.6	10.8	0.03

Table 13 :

 13 Speaker counting AP (%) achieved on the CHiME-6 evaluation set by the Transformer-based architecture trained with a speaker counting objective for different spatial features, as compared to single-channel features only (None, 1 ch.) or an ensemble of single-channel systems (None, all ch.).

	Spatial Features	0-spk	1-spk	2-spk	3-spk	4-spk
	IPD	89.9	78.8	32.6	12.4	0.003
	CSIPD	90.1	78.7	32.5	12.4	0.002
	Neural	90.2	79.0	32.2	11.9	0.003
	None, 1 ch.	88.2	77.3	30.6	12.5	0.003
	None, all ch.	90.1	78.4	31.4	11.9	0.003
	None, enh	88.1	77.4	30.3	11.8	0.003

github.com/popcornell/OSDC

Note that dense layers are equivalent to conv 1x1.

We disregard the 5-speaker overlap class on AMI since it does not occur in practice.

The sequence output by the Transformer model is stretched by a factor of S, in order for the number of input and output frames to be equal, similarly to the other models.

The single-channel evaluation protocol for CHiME-6 differs from the multichannel protocol adopted by[START_REF] Cornell | Detecting and counting overlapping speakers in distant speech scenarios[END_REF], who averaged the outputs of single-channel systems across all 24 microphones instead.

These normalization layers do improve performance, as can be seen by comparison with the results reported in our preliminary work[START_REF] Cornell | Detecting and counting overlapping speakers in distant speech scenarios[END_REF] which did not include them.