Dimitrios Giannoulis

Dan Stowell

Emmanouil Benetos

Mathias Rossignol

Mathieu Lagrange

Mark D Plumbley

A DATABASE AND CHALLENGE FOR ACOUSTIC SCENE CLASSIFICATION AND EVENT DETECTION

Keywords: Computational auditory scene analysis, acoustic scene classification, acoustic event detection

An increasing number of researchers work in computational auditory scene analysis (CASA). However, a set of tasks, each with a well-defined evaluation framework and commonly used datasets do not yet exist. Thus, it is difficult for results and algorithms to be compared fairly, which hinders research on the field. In this paper we will introduce a newly-launched public evaluation challenge dealing with two closely related tasks of the field: acoustic scene classification and event detection. We give an overview of the tasks involved; describe the processes of creating the dataset; and define the evaluation metrics. Finally, illustrations on results for both tasks using baseline methods applied on this dataset are presented, accompanied by open-source code.

INTRODUCTION

Computational auditory scene analysis (CASA) includes a wide set of algorithms and "machine listening" systems that deal with the analysis of acoustic scenes. Most of them model to some extent the human auditory system and its mechanisms and aim to detect, identify, separate and segregate sounds in the same way that humans do [START_REF] Wang | Computational Auditory Scene Analysis: Principles, Algorithms, and Applications[END_REF].

Certain practical applications that fall under the umbrella of CASA, such as noise-robust automatic speech recognition and automatic music transcription, have seen a high amount of research over the last decades, and state-of-the-art approaches for both are able to achieve satisfactory performance, comparable to that of humans (see the MIREX evaluation 1 and the CHiME challenge2 ). However, the field of CASA involves a much wider set of tasks and "machine listening" systems, many of which are far from being fully explored at a research level yet.

Over the last few years the tasks of identifying auditory scenes, and that of attempting to detect and classify individual sound events within a scene, have seen a particular rise in research, mainly due to them being interdependent with other tasks of high interest such as blind source separation. Despite an increasing number of attempts by the community for code dissemination and public evaluation of proposed methods [START_REF] Stiefelhagen | The CLEAR 2006 evaluation[END_REF][START_REF] Over | Trecvid 2012-an overview of the goals, tasks, data, evaluation mechanisms and metrics[END_REF][START_REF] Rodriguez-Fuentes | The Albayzin 2010 language recognition evaluation[END_REF], it is evident that there is not yet a coordinated, established, international challenge in this particular area with a thorough set of evaluation metrics that fully define the two tasks. By organising the present "IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events" [START_REF] Giannoulis | Detection and classification of acoustic scenes and events, an IEEE AASP challenge[END_REF] we aim to do exactly that. In the rest of the paper, we present the datasets created, the evaluation metrics used, and provide evaluation results using two baseline methods.

BACKGROUND

Acoustic scene classification aims to characterize the environment of an audio stream by providing a semantic label [START_REF] Aucouturier | The bagof-frames approach to audio pattern recognition: A sufficient model for urban soundscapes but not for polyphonic music[END_REF]. It can be conceived of as a standard classification task in machine learning: given a relatively short clip of audio, the task is to select the most appropriate of a set of scene labels. There are two main methodologies found in the literature. One is to use a set of low-level features under a bag-of-frames approach. This approach treats the scene as a single object and aims at representing it as the long-term statistical distribution of some set of local spectral features. Prevailing among different features for the approach is the Mel-frequency Cepstral Coefficients (MFCCs) that have been found to perform quite well [START_REF] Aucouturier | The bagof-frames approach to audio pattern recognition: A sufficient model for urban soundscapes but not for polyphonic music[END_REF]. The other is to use an intermediate representation prior to classification that models the scene using a set of higher level features that are usually captured by a vocabulary or dictionary of "acoustic atoms". These atoms usually represent acoustic events or streams within the scene which are not necessarily known a priori and therefore are learned in an unsupervised manner from the data. Sparsity or other constraints can be adopted to lead to more discriminative representations that subsequently ease the classification process. An example is the use of non-negative matrix factorization (NMF) to extract bases that are subsequently converted into MFCCs for compactness and used to classify a dataset of train station scenes [START_REF] Cauchi | Non-negative matrix factorisation applied to auditory scenes classification[END_REF]. Building upon this approach, the authors in [START_REF] Benetos | Characterization of acoustic scenes using a temporally-constrained shift-invariant model[END_REF] used shift-invariant probabilistic latent component analysis (SIPLCA) with temporal constrains via hidden Markov models (HMMs) that led to improvement in performance. In [START_REF] Chu | Environmental sound recognition with time-frequency audio features[END_REF] a system is proposed that uses the matching pursuit algorithm to obtain an effective time-frequency feature selection that are afterwards used as supplement to MFCCs to perform environmental sound classification.

The goal of acoustic event detection is to label temporal regions, such that each represents a single event of a specific class. Early work in event detection treated the sound signal as monophonic, with only one event detectable at a time [START_REF] Mesaros | Acoustic event detection in real life recordings[END_REF]. Events in a typical sound scene may co-occur, and so polyphonic event detection, with overlapping event regions, is desirable. However, salient events may occur relatively sparsely and there is value even in monophonic detection. There has been some work on extending systems to polyphonic detection [START_REF] Heittola | Sound event detection in multisource environments using source separation[END_REF]. Event detection is perhaps a more demanding task than scene classification, but at the same time heavily intertwined. For example, information from scene classification can provide supplementary contextual information for event detection [START_REF] Heittola | Contextdependent sound event detection[END_REF]. Many proposed approaches can be found in the literature among which spectrogram factorization techniques tend to be a regular choice. In [START_REF] Mesaros | Latent semantic analysis in sound event detection[END_REF] a probabilistic latent semantic analysis (PLSA) system, a closely related approach to NMF, was proposed to detect overlapping sound events. In [START_REF] Cotton | Spectral vs. spectro-temporal features for acoustic event detection[END_REF] a convolutive NMF algorithm applied on a Mel-frequency spectrum was tested on detecting nonoverlapping sound events. Finally, a number of proposed systems focus on the detection and classification of specific sound events from environmental audio scenes such as speech [START_REF] Barker | Decoding speech in the presence of other sources[END_REF], birdsong [START_REF] Briggs | Acoustic classification of multiple simultaneous bird species: A multi-instance multilabel approach[END_REF], musical instrument and other harmonic sounds [START_REF] Giannoulis | Recognition of harmonic sounds in polyphonic audio using a missing feature approach[END_REF] or pornographic sounds [START_REF] Kim | Automatic extraction of pornographic contents using radon transform based audio features[END_REF].

CHALLENGE

This section presents the proposed IEEE-sponsored challenge in acoustic scene classification and event detection [START_REF] Giannoulis | Detection and classification of acoustic scenes and events, an IEEE AASP challenge[END_REF]. Firstly, the datasets for the two aforementioned tasks are described, followed by definitions on the employed evaluation metrics.

Scene classification datasets

In order to evaluate Scene Classification systems we created a dataset across a pre-selected list of scene types, representing an equal balance of indoor/outdoor scenes in the London area: bus, busystreet, office, openairmarket, park, quietstreet, restaurant, supermarket, tube, tubestation. To enable participants to further explore whether machine recognition could benefit from the stereo field information available to human listeners [1, Chapter 5], we recorded in binaural stereo format using a Soundman OKM II microphone.

For each scene type, three different recordists (DG, DS, EB) visited a wide variety of locations in Greater London over a period of months (Summer and Autumn 2012), and in each scene recorded a few minutes of audio. We ensured that no systematic variations in the recordings covaried with scene type: all recordings were made in moderate weather conditions, and varying times of day and week, and each recordist recorded each scene type.

We then reviewed the recordings to select 30-second segments that were free of issues such as mobile phone interference or microphone handling noise, and collated these segments into two separate datasets: one for public release, and one private set for evaluating submissions. The segments are 30-second WAV files (16 bit, stereo, 44.1 kHz), with scene labels given in the filenames. Each dataset contains 10 examples each from 10 scene types. The public dataset is published on the C4DM Research Data Repository (accessible through [START_REF] Giannoulis | Detection and classification of acoustic scenes and events, an IEEE AASP challenge[END_REF]).

Event detection (office) datasets

For the Event Detection task, we addressed the problem of detecting acoustic events in an office enviromnent. In order to control the degree of polyphony in the dataset, so that algorithms' performance can be evaluated using different polyphony levels, we followed two related approaches: we recorded live, scripted, monophonic sequences in real office environments; and we also recorded isolated events as well as background ambience, and artificially composed these into scenes with controllable polyphony.

For the scripted recordings, we created scripts by random ordering of event types, and then recruited a variety of paid participants to perform the scripts in various office rooms within QMUL. For each script, multiple takes were used, and we selected the best take as the one having the least amount of unscripted background interference. Event types used were: alert (short alert (beep) sound), clearthroat (clearing throat), cough, doorslam (door slam), drawer, keyboard (keyboard clicks), keys (keys put on table), knock (door knock), laughter, mouse (mouse click), pageturn, (page turning), pendrop (pen, pencil, or marker touching table surfaces), phone, printer, speech, switch. To capture the spatial layout of the acoustic environment, recordings were made in first order B-format with a Soundfield model SPS422B microphone placed in an open space in the room, with events spatially distributed around the room. Recordings were mixed down to stereo (using the common "Blumlein pair" configuration). The challenge is conducted using the stereo files, with scope for future challenges to be extended to full B-format and take into account spatial information for event detection.

Since there is inherent ambiguity in the annotation process, we recruited two human annotators to annotate the onset and offset times of events in the recordings. Annotators were trained in Sonic Visualiser 3 to use a combination of listening and inspecting waveforms/spectrograms to refine the locations. We then inspected the two annotations per recording for any large discrepancies, which allowed us to detect any instances of error. The remaining small deviations between the annotations reflect the ambiguity in event boundaries.

For the second approach, we designed a scene synthesizer able to easily create a large set of acoustic scenes from many recorded instances of individual events. The synthetic scenes are generated by randomly selecting for each occurrence of each event we wish to include in the scene one representative excerpt from the natural scenes, then mixing all those samples over a background noise. The distribution of events in the scene is also random, following high-level directives that specify the desired density of events. The average SNR of events over background noise is also specified and, unlike in the natural scenes, is the same for all event types (this is a deliberate decision). The synthesized scenes are mixed down to mono in order to avoid having spatialization inconsistencies between successive occurrences of a same event; spatialization including room reverberation is left for future work. The resulting development and testing datasets consist of scripted/synthetic sequences with varying durations, with accompanying ground-truth annotations. The development dataset is published on the C4DM Research Data Repository (accessible through [START_REF] Giannoulis | Detection and classification of acoustic scenes and events, an IEEE AASP challenge[END_REF]).

Challenge evaluation metrics

For the scene classification task, participating algorithms will be evaluated with 5-fold stratified cross validation. The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm will be computed.

For the event detection tasks, in order to provide a thorough assessment of the various systems three types of evaluations will take place, namely a frame-based, event-based, and class-wise event-based evaluation. Frame-based evaluation is performed using a 10ms step and metrics are averaged over the duration of the recording. The main metric used for the frame-based evaluation is the acoustic event error rate (AEER) used in the CLEAR evaluations [START_REF] Temko | CLEAR evaluation of acoustic event detection and classification systems[END_REF]:

AEER = D + I + S N ( 1
)
where N is the number of events to detect for that specific frame, D is the number of deletions (missing events), I is the number of insertions (extra events), and S is the number of event substitutions, defined as S = min{D, I}. Additional metrics include the Precision, Recall, and F-measure (P-R-F).

3 http://www.sonicvisualiser.org/ By denoting as r, e, and c the number of ground truth, estimated and correct events for a given 10ms frame, the aforementioned metrics are defined as:

P = c e , R = c r , F = 2PR P + R . ( 2
)
For the event-based metrics, two types of evaluation will take place, an onset-only and an onset-offset-based evaluation. For the onset-only evaluation, each event is considered to be correctly detected if the onset is within a 100ms tolerance. For the onset-offset evaluation, each event is correctly detected if its onset is within a 100ms tolerance and its offset is within 50% range of the ground truth event's offset w.r.t. the duration of the event. Duplicate events are counted as false alarms. The AEER and P-R-F metrics for both the onsetonly and the onset-offset cases are utilised.

Finally, in order to ensure that that repetitive events do not dominate the accuracy of an algorithm, class-wise eventbased evaluations are also performed. Compared with the event-based evaluation, the AEER and P-R-F metrics will be computed for each class separately within a recording and then averaged across classes. For example, the class-wise Fmeasure is defined as:

F = 1 K k F k (3)
where F k is the F-measure for events of class k.

BASELINE SYSTEMS

Scene classification

The widespread standard approach to audio classification is the "bag-of-frames" model discussed above. Its modelling assumptions imply among other things that the sequence ordering of frames is ignored [START_REF] Foote | Content-based retrieval of music and audio[END_REF][START_REF] Aucouturier | The bagof-frames approach to audio pattern recognition: A sufficient model for urban soundscapes but not for polyphonic music[END_REF]. Foote [START_REF] Foote | Content-based retrieval of music and audio[END_REF] is an early example, comparing MFCC distributions via vector quantisation. Since then, the standard approach to compare distributions is by constructing a Gaussian Mixture Model for each instance or for each class [START_REF] Aucouturier | The bagof-frames approach to audio pattern recognition: A sufficient model for urban soundscapes but not for polyphonic music[END_REF][START_REF] Wegener | On the robustness of audio features for musical instrument classification[END_REF].

The MFCC+GMM approach to audio classification is relatively simple, and has been criticised for the assumptions it incurs [START_REF] Aucouturier | Improving timbre similarity: how high's the sky?[END_REF]. However, it is quite widely applicable in a variety of audio classification tasks. Aucouturier and Pachet [START_REF] Aucouturier | The bagof-frames approach to audio pattern recognition: A sufficient model for urban soundscapes but not for polyphonic music[END_REF] specifically claim that the MFCC+GMM approach is sufficient for recognising urban soundscapes but not for polyphonic music (due to the importance of temporal structure in music). It has been widely used for scene classification among other recognition tasks, and has served as a basis for further modifications [START_REF] Chu | Environmental sound recognition with time-frequency audio features[END_REF]. The model is therefore an ideal baseline for the Scene Classification task.

Code for the bag-of-frames model has previously been made available for Matlab. 4 However, for maximum repro-ducibility we wished to provide simple and readable code in a widely-used programming language. The Python language is very widely used, freely available on all common platforms, and is notable for its emphasis on producing code that is readable by others. Hence we created a Python script embodying the MFCC+GMM classification workflow, publicly available under an open-source licence, 5 and designed for simplicity and ease of adaptation.

Event detection

As mentioned in Sec. 2, the NMF framework is a useful one for event detection as it can deal with polyphonic content and the low-rank approximation it provides can efficiently model the underlying spectral characteristics of sources hidden in an acoustic scene. Therefore, we chose to provide an NMFbased baseline system 6 that performs event detection in a supervised manner, using a pre-trained dictionary.

Our algorithm is based on NMF using the β-divergence as a cost function [START_REF] Kompass | A generalized divergence measure for nonnegative matrix factorization[END_REF]. As a time-frequency representation, we used the constant-Q transform (CQT) with a log-frequency resolution of 60 bins per octave [START_REF] Schörkhuber | Constant-Q transform toolbox for music processing[END_REF]. The training data were normalized to unity variance and NMF with Kullback-Leibler (KL) divergence (β = 1) was used to learn a set of N bases for each class. The numbers of bases we tested was 5, 8, 10, 12, 15, 20 and 20i, the latter corresponding to learning individually one basis per training sample, for all 20 samples. Putting together the sets for all classes we built a "fixed" dictionary of bases used subsequently to factorize the normalized development set audio streams. Afterwards, we summed together the activations per class obtained from the factorization. We tested the use of median filtering for smoothing purposes but this did not improve the classification. Finally a threshold T was chosen to be applied in order to give us the final class activations. The optimal N and T values were chosen empirically by maximizing the F -measure for the two annotations on the development set.

RESULTS

The two baseline systems were tested using the public datasets of the challenge described in Sec. 3. In scene classification, where chance performance is 10%, our baseline system attained 52 ± 13% (95% confidence interval). Table 1 breaks down these results as a confusion matrix. It shows, for example, that supermarket was the true class most often mislabelled, most commonly as openairmarket or tubestation.

Results for the event detection system are shown in Table 2. These include the computed metrics as presented in Sec. 3.3, as well as the optimal system parameters determined Table 2. Detection accuracy (%) of the NMF system for the Event Detection task for the monophonic Office Live Dataset.

separately for both annotations (1-2), calculated as mentioned in Sec.4.2. We found that learning basis vectors from individual sounds resulted in better performance. It is also worth highlighting that the event-based metrics lead to lower reported performance than the frame-based metric.

Finally, not all classes were detected equally well. The F k was 0% for certain classes, which were: keyboard, keys, mouse, printer, and switch. All these sounds are characterised by a short-lived and highly transient nature and very low SNR levels that might be potential reasons for failing to be detected by the system. A further set of sounds with an overall poor F k were: alert, laughter, and pageturn. We believe that the big variation that characterises these sounds could be the reason behind the low performance of the baseline system.

CONCLUSIONS

In this paper, we presented a newly launched public evaluation challenge for the classification of acoustic scenes and the detection of acoustic events. We presented the datasets, evaluation metrics, and finally offered evaluation results using an MFCC+GMM system for scene classification and an NMFbased system for event detection. The challenge datasets and the code for both systems are available online and third parties are welcome to use it as the basis for challenge submissions as well as for future research in the CASA field.

Possible extensions for the scene classification system may include a wider set of features, addition of temporal features (such as ∆MFCC) or the use of HMMs to model the various acoustic scenes. For event detection, possible extensions could be to remove or de-emphasize lower frequencies that mainly capture ambient background noise, to try different β values, or to add constrains in the NMF algorithm such as sparsity on the activation matrices. Of course, these baseline systems are just examples, and we welcome approaches to the tasks that differ radically from the baseline systems we have implemented.

At the time of writing, the challenge is still running; results and descriptions of submitted systems will be made available online [START_REF] Giannoulis | Detection and classification of acoustic scenes and events, an IEEE AASP challenge[END_REF]. In the future, we aim to release detailed challenge results and create a code repository for all opensource submissions, which can serve as a point of reference for the advancement of CASA research.

Table 1 .

 1 Confusion matrix for scene classification with baseline MFCC+GMM classifier. Rows are ground-truth labels.

		Evaluation Method
	Metrics	Event	Class-Wise Frame
	Parameters	Based Event Based Based
	R	16.8	21.7	16.0
	P	15.9	11.6	29.1
	F -measure	15.4	13.5	20.6
	AEER *	2.51	2.94	1.62
	Offset R	4.4	7.1	-
	Offset P	4.5	3.6	-
	Offset F -meas.	4.2	4.0	-
	Offset AEER *	2.88	3.37	-
	Optimal N	20i	20i	20-20i
	Optimal T	750-650	400-700	400-550

* Not measured in (%)

http://music-ir.org/mirexwiki/

http://spandh.dcs.shef.ac.uk/projects/chime/ challenge.html

http://www.jj-aucouturier.info/projects/mir/ boflib.zip

http://code.soundsoftware.ac.uk/projects/smacpy

http://code.soundsoftware.ac.uk/projects/ d-case-event

This work has been partly supported by ESPRC Leadership Fellowship EP/G007144/1, by EPSRC Grant EP/H043101/1 for QMUL, and by ANR-11-JS03-005-01 for IRCAM. D.G. is funded by a Queen Mary University of London CDTA Research Studentship. E.B. is supported by a City University London Research Fellowship.

Label

bus busystreet office openairmarket


Dimitrios Giannoulis

Emmanouil Benetos

Dan Stowell

Mathias Rossignol

Mathieu Lagrange

Mark D Plumbley

Detection and classification of acoustic scenes and events: An IEEE AASP challenge

Keywords: Computational auditory scene analysis, acoustic scene classification, acoustic event detection

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Over the last few years, there has been an increased interest in the speech and audio processing community in code dissemination and reproducibility of results as a means to improve the quality and relevance of published results. This can be attributed to accumulating evidence of the benefits of performing research with reproducibility in mind and making well-documented code and data publicly available [START_REF] Vandewalle | Reproducible research in signal processing[END_REF][START_REF] Kovacevic | How to encourage and publish reproducible research[END_REF]. Public evaluation of proposed methods, especially if accompanied with open-source submissions is a key component in the move towards this reproducibility. It can serve as a reference point for the performance of proposed methods and can also be used for studying performance improvements throughout the years. Numerous initiatives have reached maturity, for example the SiSEC evaluation for signal separation [START_REF] Araki | The 2011 signal separation evaluation campaign (SiSEC2011)[END_REF], the MIREX competition for music information retrieval [START_REF]Music Information Retrieval Evaluation eXchange (MIREX)[END_REF] and the CHiME speech separation and recognition challenge [START_REF] Barker | The PASCAL CHiME speech separation and recognition challenge[END_REF]. The research problems related with these evaluations are well-defined and have their own performance metrics established. However, for researchers working on modeling and classifiction of acoustic scenes, containing non-speech and non-music, and detecting sound events within a scene, there is not yet a coordinated established international challenge in this area, with the exception of the now discontinued CLEAR evaluations [START_REF] Stiefelhagen | The CLEAR 2006 evaluation[END_REF] funded by the CHIL project and the Multimedia Event Detection of the TRECVID video retrieval evaluations, where the focus is on audiovisual, multi-modal event detection in video recordings [START_REF] Over | TRECVID 2012an overview of the goals, tasks, data, evaluation mechanisms and metrics[END_REF].

In this paper, we describe a newly-launched public evaluation challenge on acoustic scene classification and event detection, both for monophonic and polyphonic audio [START_REF] Giannoulis | Detection and classification of acoustic scenes and events[END_REF]. In Section 2, we present the datasets that were created for the challenge, as well as the employed evaluation metrics. Participating systems are then outlined in Section 3, and evaluation results are presented and discussed in Section 4.

CHALLENGE DESCRIPTION

Acoustic scene classification and detection of sound events within a scene are well defined engineering tasks that both fall under the "umbrella" of computational auditory scene analysis (CASA) [START_REF] Wang | Computational auditory scene analysis: Principles, algorithms, and applications[END_REF]. The first task aims to characterize the acoustic environment of an audio stream by providing a semantic label to it [START_REF] Aucouturier | The bag-offrames approach to audio pattern recognition: A sufficient model for urban soundscapes but not for polyphonic music[END_REF]. The second one aims to label temporal regions within the audio, within which a specific event class is active, by estimating the start and end time of each event and if necessary (i.e. for audio extraction purposes) separate it from other overlapping events.

The present challenge consists of a set of three subtasks. The first one addresses the problem of identifying and classifying acoustic scenes or soundscapes. The other two subtasks address the problem of identifying individual sound events that are prominent in an acoustic scene: one focuses on monophonic event detection without overlapping sounds and the other focuses on polyphonic scenarios. The polyphonic case could be considered more interesting, as in realistic everyday scenarios most of the sounds that reach our ears tend to stem from a multitude of sources, but at the same time it consists of a much more challenging problem. More details about the proposed tasks along with baseline results can be found in [START_REF] Giannoulis | Detection and classification of acoustic scenes and events[END_REF][START_REF] Giannoulis | A database and challenge for acoustic scene classification and event detection[END_REF].

Datasets

Each of the tasks is accompanied by its own dataset. The datasets for Scene Classification (SC) consists of two equally proportioned parts each made up of ten 30 seconds recordings for each scene (class), for a total of 100 recordings per dataset. One part has been made publicly available [START_REF] Giannoulis | Detection and classification of acoustic scenes and events[END_REF] and serves as the development set for participants to investigate the performance of their system, whereas the other is kept private and used for a train/test (K-fold) evaluation. The two datasets span a pre-selected list of scene types, representing an equal balance of indoor/outdoor scenes in the London area: bus, busystreet, office, openairmarket, park, quietstreet, restaurant, supermarket, tube, tubestation. These recordings were made with a set of Soundman OKM II binaural microphones. These microphones imitate a pair of in-ear headphones that the user can wear for added portability and subtlety. Furthermore, the data carries also binaural information about the sound that could be utilized as cues for the sound event and scene detection or simply be ignored by adding the two channels together in order to obtain a mono recording.

Participants

The datasets for event detection were built from audio collected in office environments because of the interest of such audio to certain applications such as audio-conferencing systems etc. Two event detection tasks are proposed, a monophonic task denoted as Office Live (OL) and a polyphonic task denoted as Office Synthetic (OS). Polyphonic data for the OS task was created using a scene synthesizer, concatenating recordings of isolated events. Each dataset consists of three subsets (a training, a development and a testing dataset). The training set contains instantiations of individual events for every class and is shared between the OL and OS tasks to allow for single training of event detection systems. The development and testing datasets consist of roughly 1 minute long scripted recordings of everyday audio events in a number of office environments (different size and absorbing quality rooms, different number of people in the room and varying noise level). Event types used were: alert (short alert (beep) sound), clearthroat (clearing throat), cough, doorslam (door slam), drawer, keyboard (keyboard clicks), keys (keys put on table), knock (door knock), laughter, mouse (mouse click), pageturn, (page turning), pendrop (pen, pencil, or marker touching table surfaces), phone, printer, speech, switch. To capture the spatial layout of the acoustic environment, recordings were made in first order B-format (4-channel), with a high-quality Soundfield SPS422B microphone system, placed in an open space in the room, with events spatially distributed around the room. Recordings were mixed down to stereo (using the common "Blumlein pair" configuration). The challenge is conducted using the stereo files, with scope to extend the challenge to full B-format in future if there is demand.

More details about the creation of the datasets, the annotation process and the audio recording process can be found in [START_REF] Giannoulis | Detection and classification of acoustic scenes and events[END_REF].

Evaluation Metrics

For the scene classification task, systems are evaluated with 5-fold stratified cross validation. The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm is computed.

For the event detection tasks, in order to provide a thorough assessment of the various systems, three types of evaluations take place, namely a frame-based, event-based, and class-wise eventbased evaluation. The main metrics used for each evaluation type are the F-measure (F ) and the acoustic event error rate (AEER) as described in [START_REF] Giannoulis | Detection and classification of acoustic scenes and events[END_REF]. For the event-based and class-wise event-based metrics, two types of evaluation will take place, an onset-only and an onset-offset-based evaluation. Results to onset-based metrics (denoted without any subscript) and onset-offset-based metrics (denoted as F offset and AEER offset ). For a complete and analytic description of the evaluation metrics employed the reader is referred to [START_REF] Giannoulis | Detection and classification of acoustic scenes and events[END_REF][START_REF] Giannoulis | A database and challenge for acoustic scene classification and event detection[END_REF]. Dashed lines indicate (left to right): chance performance (black); baseline system performance (light blue); mean accuracy of human listener (orange). "Majority vote" is a meta-classifier using the majority decision over all submissions.

SUBMITTED SYSTEMS

Overall, 11 systems were submitted to the SC task, 7 systems were submitted to the OL task, and 3 systems to the OS task. Variants for each system were allowed that increased the total number and variety considerably. The systems submitted for the scene classification and event detection tasks are listed in Tables 1 and2, respectively, along with a short description of each system and the programming language in which it was written. Apart from the submitted systems, performance on the test sets is also reported for baseline systems for the two tasks. These systems were made publicly available as open source software [START_REF] Giannoulis | A database and challenge for acoustic scene classification and event detection[END_REF].

CHALLENGE RESULTS

Results were computed by running all the submitted systems on the held-back testing datasets and computing the metrics as in Sec. 2.2. Figure 1 shows the overall performance of submitted systems for the scene classification task. Most systems were able to outperform the baseline system, and some matched or even slightly excelled the mean accuracy we found in human listeners (71%; results in preparation). The strongest performers are notably diverse in their choice of features and their use of temporal information, though often using SVMs for classification. Two submissions achieved good results on the development data but not on our held-out test data.

Table 3 shows a confusion matrix for the scene labels as round percentages of the sum of all confusion matrices for all submissions. Confusions are mostly concentrated over classes that share some acoustical properties such as park/quietstreet and tube/tubestation. Values are expressed as percentages rounded to the nearest integer.

For the event detection OL and OS tasks, results are summarized in Tables 4 and5, respectively. The baseline was outperformed by most systems for these tasks too. The best performance for the OL task using all types of metrics is achieved by the SCS submission, which used a Gabor filterbank feature extraction step with by 2-layer hidden Markov models (HMMs) for classifying events, followed by the NVM submission, which used a meta-classifier combining hierarchical HMMs and random forests. For the OS task, the best performance in terms of F-measure is achieved by the DHV system, which used an iterative scheme with HMMs. It should also be noted that submitted systems performed better with lower polyphony, with the exception of the DHV system, which had better performance with higher polyphony levels. As expected, the onset-offset evaluation produced worse results compared to onsetonly evaluation for both tasks, although the performance difference is rather small. This may be explained by the percussive nature of most events.

The challenge website [START_REF] Giannoulis | Detection and classification of acoustic scenes and events[END_REF] gives detailed system descriptions and extensive results, analytic breakdown of performance per system, as well as further error analysis.

CONCLUSIONS

In this paper we presented a challenge on the detection and classification of acoustic scenes and events. We ran a scene classification (SC) challenge, and two event detection and classification challenges: office live (OL) and office synthetic (OS). Our goal was to provide a focus of attention for the scientific community in developing systems for CASA that will encourage sharing of ideas and improve the state of the art, potentially leading to the development of systems that achieve a performance close to that of humans.

The results enable us to draw some interesting conclusions about the different problems. For scene classification, we found that although simple systems can do relatively well, the improvement that more complex systems achieve can bring performance to the levels achieved by human listeners. For event detection, which is a more challenging task, performance is much worse although we have not performed a direct comparison with human listeners at present. For the monophonic case, systems are able to achieve satisfactory performance with scope for improvement. For the polyphonic case, the task of recognising individual potentially overlapping sounds becomes significantly challenging and the performance of systems that are even prepared to deal with polyphonic content falls dramatically. More details for all the submitted systems can be found on the challenge website in [START_REF] Giannoulis | Detection and classification of acoustic scenes and events[END_REF]. At this point, we have just completed running the challenge. For future work, we will consider producing a detailed performance evaluation, creating a code repository, releasing test sets, doing a Bformat challenge, running the challenge again or doing a challenge on world synthetic sounds (WS) as proposed in [START_REF] Giannoulis | Detection and classification of acoustic scenes and events[END_REF].

Figure 1 :

 1 Figure 1: Classification accuracy(%) for the SC task. Plot shows mean over 5-fold cross-validation with 95% confidence intervals.Dashed lines indicate (left to right): chance performance (black); baseline system performance (light blue); mean accuracy of human listener (orange). "Majority vote" is a meta-classifier using the majority decision over all submissions.

Table 1 :

 1 Summary of submitted scene classification systems.

	Code Method

Table 2 :

 2 Summary of submitted event detection systems.

Table 3 :

 3 Aggregate confusion matrix for scene classification across all submissions. Rows are ground truth, columns the inferred labels.

	Label	bus	busystreet	office	openairmarket	park	quietstreet	restaurant	supermarket	tube	tubestation
	bus	81	3	0	4	1	0	0	4	6	2
	busystreet	1	69 14	2	1	2	1	3	3	5
	office	1	0	55 13	9	12	4	3	1	3
	openairmarket	1	2	0	59 13	0	9	12	3	2
	park	1	1	8	3	51 29	3	2	1	1
	quietstreet	0	5	4	3	29 43	9	5	0	1
	restaurant	1	1	0	16	5	0	53 21	2	3
	supermarket	6	5	6	6	4	7	10 42	7	7
	tube	7	7	1	1	2	2	5	3	44 28
	tubestation	5	16	1	4	1	2	3	8	19 41

Table 4 :

 4 Evaluation metrics for the participating systems for the (monophonic) Office Live Event Detection task. F offset (%) AEER AEER offset F (%) F offset (%) AEER AEER offset F (%) AEER

						Evaluation Method
			Event-Based			Class-Wise Event-Based		Frame-Based
	System	F (%) F offset (%) AEER AEER offset	F (%) F offset (%) AEER AEER offset	F (%) AEER
	CPS	2.23	1.65	2.285	2.301	0.65	0.49	1.872	1.891	3.82	2.116
	DHV	26.67	22.43	2.519	2.676	30.72	25.29	2.182	2.370	26.0	3.128
	GVV	15.52	13.46	1.779	1.831	13.21	12.03	1.556	1.606	31.94	1.084
	NVM 1	32.57	24.95	1.864	2.095	29.37	21.80	1.639	1.899	40.85	1.115
	NVM 2	34.16	26.28	1.852	2.095	33.05	24.88	1.602	1.877	42.76	1.102
	NVM 3	34.51	27.01	1.827	2.052	33.52	24.65	1.575	1.846	45.50	1.212
	NVM 4	30.47	24.68	1.906	2.083	28.17	21.62	1.650	1.849	42.86	1.360
	NR2	19.21	15.26	3.076	3.244	21.54	17.64	2.857	3.010	34.66	1.885
	SCS 1	39.47	36.74	1.669	1.749	36.33	34.20	1.579	1.677	53.02	1.167
	SCS 2	45.17	41.06	1.601	1.727	41.51	38.32	1.511	1.646	61.52	1.016
	VVK	30.77	25.40	2.054	2.224	24.55	20.36	1.762	1.949	43.42	1.001
	Baseline	7.38	1.58	5.900	6.318	9.00	1.86	5.960	6.462	10.72	2.590
			Event-Based			Class-Wise Event-Based		Frame-Based
	System F (%) DHV 8.45	6.18	4.741	4.860	9.73	7.58	4.028	4.147	13.08	8.426
	GVV	7.69	7.33	1.913	1.920	6.69	6.51	1.584	1.591	10.30	1.553
	VVK	5.80	5.28	1.885	1.895	5.10	4.77	1.436	1.445	5.77	2.106
	Baseline	4.98	0.24	6.507	6.895	6.69	0.18	5.389	5.782	6.88	3.047

Table 5 :

 5 Evaluation metrics for the participating systems for the (polyphonic) Office Synthetic Event Detection task.

October 20-23, 2013, New Paltz, NY

ACKNOWLEDGMENTS

The authors would like to thank all of the contributors, for taking part in the challenge and participating in email discussions as we were developing the challenge, and Daniele Barchiesi, for his contribution in running the evaluation of the submitted systems and providing useful feedback.