Claudia Gutierrez Rodriguez

Sylvie Servigne
email: sylvie.servigne@insa-lyon.fr

Claudia C Gutiérrez Rodriguez

Sensor Data Quality for Geospatial Monitoring Applications

Keywords: Data Quality, Sensor Data, Metadata, Geospatial Applications, Environmental Monitoring Systems, Decision Making Support

published or not. The documents may come

Introduction

Embedded with mobility, communication and processing capabilities, sensors and sensor networks are more and more employed in numerous applications domains such as military, environmental, healthcare, etc. More particularly, the use of sensors in geospatial applications like environmental monitoring (i.e. floodings, volcanoes...), allows a better interpretation of the real world and supports crisis and risk management. However, and despite a strong interest from geospatial domain, the integration of sensors and sensor networks raises new technical and data management challenges. In fact, sensor data turns imprecise and uncertain considering the hardware restrictions of sensor devices and the collection of data in hostile environments at variable rates and positions. These backwards thus necessitate for quality mechanisms capable to guarantee the achievement of monitoring applications and assist expert in decision making.

Data quality management has been strongly integrated on geographic domain. Actually, thanks to research and standardisation programs in this domain [START_REF] Devillers | Fundamentals of Spatial Data Quality[END_REF]5], Geographic Information Systems (GIS) are able to assess and improve the quality of geographic data. Moreover, other international organizations as Open Geospatial Consortium (OGC) or the National Oceanic and Atmospheric Administration (NOAA), have also recently raised the importance of sensor data quality issue and strongly advice to perform further studies.

In this paper, we address sensor data quality issue with an approach inspired both by current methodologies and approaches managing data quality. This approach focuses on the specificities of sensor data in a monitoring context.

The rest of this document is organized as follows: Section 2 introduces an overview of our research context. Section 3 details the specificities of sensor data and monitoring systems. Section 4 introduces our approach for sensor data quality analysis. In Section 5 we describe a proposal of visualization interface for sensor data quality discovery. We conclude in Section 6.

Overview

For years, data quality characterizes a key problem for all kind of organizations [START_REF] Wang | Beyond accuracy: what data quality means to data consumers[END_REF]. Actually, emerging applications and manipulating sensor data, like environmental monitoring, also reveal this important issue. Considering monitoring as a primary key on environmental crisis management system, an early and reliable detection of critical events is crucial for the achievement of such kind of systems. Environmental monitoring thus requires an efficient acquisition of information coming from sensors (spread over large areas), the interpretation of complex observation patterns at different temporal and spatial scales, as well as reliable and understandable results. These facts led us to wonder how to evaluate and provide users with quality information. We believe that such complementary information can support users (i.e. environmental scientists, urban planners...) to make decisions in a more trustworthy way.

According to the literature, data quality issue has been the subject of numerous studies, especially for Information Systems [START_REF] Naumann | Information Quality[END_REF]. As a result, several models and methodologies oriented to manage quality in different application domains (e.g. medical, geographic…) have been proposed [START_REF] Batini | Methodologies for data quality assessment and improvement[END_REF]. Also, evaluation and improvement techniques have been subject of standardizations and being the core of data management approaches as for Geographic Information Systems [START_REF] Devillers | Fundamentals of Spatial Data Quality[END_REF][START_REF] Goodchild | Data quality in geographic information : from error to uncertainty[END_REF]. We note that these studies provide excellent quality approaches, enhancing the basic perception of data quality and mainly focused on so-called traditional applications, where static data is managed in databases management systems or datawarehouses. Withal, we identify that such approaches do not sufficiently take into account the properties of sensor data, namely their dynamicity, temporality and heterogeneity. Besides, several sensor management approaches propose to analyze sensor as a service and qualify them as such [START_REF] Ahluwalia | Composite quality of service and decision making perspectives in wireless networks[END_REF]; others propose filtering, cleaning or clustering techniques to limit faulty data propagation [START_REF] Ni | Sensor network data fault types[END_REF]. Although these approaches contribute to the management of sensor data, the necessity to tackle the quality of sensor data over monitoring systems remains a relevant research topic.

Geospatial Monitoring Applications: Environmental Phenomena Surveillance

In our case of study, we are particularly interested on the study of data quality for geospatial monitoring applications. With our approach, we aim to identify and tackle the most transcendental aspects of this problematic.

Geospatial monitoring applications refers to the environmental monitoring systems composed by a set of wireless, geolocalised and heterogeneous sensors (i.e. geosensors) [START_REF]Proc. GeoSensor Networks, 3 rd Intl. Conf. GSN[END_REF]. These sensors are typically organized in networks, measuring one or several parameters like temperature, movement and especially deployed in contexts where an environmental activity carry a risk of harmful effects both for human safety and/or for the natural environment. Figure 1 depicts a typical framework for this kind of systems. Here, sensor data are collected and transferred (directly or through a gateway) to the data processing and management center. Data can be visualized and further analyzed by a Geographic Information System (GIS), for example. Data can be also integrated with other information as satellite images and maps or published and analyzed through the Web by specialized services. This kind of framework is also employed to monitor farmlands, animals or smart buildings, for example [START_REF]Proc. GeoSensor Networks, 3 rd Intl. Conf. GSN[END_REF].

Data Quality Issues and Requirements

The surveillance of environmental phenomena is a critical process and using monitoring systems based on sensors and sensor networks raise also several concerns. First, such monitoring systems impose sensors to operate in difficult conditions. In addition, sensors perform measurements according to their restricted storage, processing and communication capabilities. In both cases, sensors can provide unreliable information. This effect is very critical because, an erroneous data can be propagated and provide a misinterpretation of the real world.

However, we note that a given data may not only be erroneous because of a faulty sensor; deployment conditions of the sensor and their use have also an impact. Generally, erroneous data coming from sensors can be identified in two categories: intentional or unintentional errors [START_REF] Ni | Sensor network data fault types[END_REF][START_REF] Shi | Designing secure sensor networks[END_REF]. Intentional data errors can be caused by physical or logical attacks over sensors (i.e. malicious attacks) and unintentional data errors are generally caused by hardware malfunction (or stochastic errors), misplacement (conditional errors) or exhausted resources (systematic errors).

Regarding unintentional errors, sensors used to monitor environmental phenomena are exposed to unpredictable situations. For example, if we consider a motion sensor localized on the surrounding of a volcano, it may temporarily experiment some vibrations that are not necessarily related to the volcano activity. These vibrations can come from a helicopter landing and performing a field survey. Further, multiple problems can also be derived from or caused by animals surrounding the detection area.

Considering these facts, monitoring systems based on sensors and sensor networks must be reliable enough to guarantee system achievement and assist experts on decision making. We have to provide and communicate, as much as possible, the quality of the information managed through the system.

In order to better understand the elements required to define and manage the quality of sensor data, we introduce next the study of sensor data specificities in a monitoring context.

Sensor Data Modeling and Specificities

In a monitoring context, sensor data has several specificities comparing to data exploited in traditional applications. To the best of our knowledge, existing approaches base their data analysis only on sensed data and do not take into account all the life cycle of data through the system [START_REF] Ni | Sensor network data fault types[END_REF][START_REF]Sensor Web Enablement[END_REF]. To better explain this aspect, we propose the modeling of sensor data specificities, from its acquisition to its discovery.

Sensor Data Specificities

We analyze sensor data from a product-oriented point of view: from acquisition to discovery. For this, we organize the typical framework of environmental monitoring systems (Figure 1) in three main layers: acquisition, processing and discovery. This organization allows us to analyze sensor data all through the system.

The acquisition layer refers to the sensor data collection system and thus to raw (or sensed) and pre-processed data. The processing layer involves data resulting from data processing and management center where energy, storage and analyze capabilities are more significant. Finally, at discovery layer, we talk about delivered data (or post-processed data) exploited over a GIS or combined with a Web service or application.

According to these layers, we distinguish several specificities. For example, data coming from sensors are geolocalised and time stamped values. Sensor data have then spatiotemporal properties and mainly stored over temporal and spatial relations. Also, for environmental monitoring systems, "real-time" processing does not means "fast", it means that processing time is limited, predictable and manageable (soft real-time). In fact, to process all data, monitoring systems must adopt priority policies as preferring most recent data and store them according to temporality properties. Finally, sensor data can be further processed by users, together with complementary information as producing statistics, sorting or geo-locating data in a map.

Sensor Data Modeling : Observation Data

The spatial and temporal properties of data coming from geosensors introduce a new scheme of data collection. Since observation is the principal goal of monitoring systems, we base our modeling on this concept (also used by OGC -SWE [START_REF]Sensor Web Enablement[END_REF]). We call observation data those data used to describe a phenomenon. Such data has spatial, temporal, semantic and dynamic properties as well as complementary information contained in metadata.

With this modeling (Figure 2), we attempt to provide a picture of data coming from sensors and the complementary information (metadata) which characterizes data in a monitoring context. Being compatible with current applications [START_REF]Sensor Web Enablement[END_REF][START_REF]Semantic Sensor Network Incubator Group[END_REF]13] in geospatial domain, our model consider that a sensor network is composed by a set of sensors, located on the same observation area and allowed to collect and transfer data at fixed and variable positions (fixed, agile and mobile). Such sensors are related to observation stations (meteorological, agricultural stations…) responsible for observing different phenomena (i.e. tsunamis, volcanoes …), and where one or more elements are used to determine the evolution of such phenomena (i.e. temperature, gas, etc.).

We note in this sensor data modeling, that not all observation data has the same behavior over time and belong to different aspects of the system. We thus identify dynamic elements which refer to objects changing over time and according to the observed phenomenon (e.g. measures, agile or mobile sensor location…); and static elements which remain the same throughout an observation (e.g. observation station, phenomenon, measured elements).

In the remaining of this paper, we will note that sensor data quality management requires also an adapted management of dynamic and static attributes of data.

Quality for Sensor Data in Geospatial Monitoring Applications

Traditionally, data quality was perceived as a simple correctness [START_REF] Wang | Beyond accuracy: what data quality means to data consumers[END_REF]. Nevertheless, quality is more than that, it is a multifaceted concept, in which different elements (as dimensions, attributes, criteria…) are used to described it [START_REF] Naumann | Information Quality[END_REF]. Accordingly, to estimate the quality of sensor data, we must consider also the quality of their sources and process. In other words, we believe that the quality of data from an intrinsic (or internal) point of view is not sufficient to describe the quality of a system like an environmental monitoring system. This fact reveals the necessity to determine and formalize the set of attributes and properties required to evaluate the quality of sensor data.

In order to reach our goal, we propose in this section an approach to define and manage the quality of sensor data. This approach takes mainly into account: the characteristics of sensor data, a set of criteria related the factors impacting the quality of sensor data, as well as the use of metadata to manage quality information.

Data quality Model for Sensor Data

In order to categorize the set of quality elements necessaries to define and evaluate the quality of sensor data, we first expose the set of factors impacting the quality of sensor data through the system. Based on these factors, the specificities of sensor data and inspired by quality modelling approaches and standardizations, we associate a set of quality criteria capable to estimate their impact. A data quality model will formalize all these elements.

Impacting the Quality of Sensor Data

Sensor data is processed at different levels of the monitoring system (see Section 3.1). At each level, the quality of sensor data can be impacted by several factors and thus, erroneous or poor quality of data can reach the final user.

First, we analyze data at acquisition layer according to three main aspects: measurement context, sensor and transmission. The analysis of these aspects allow us to identify several factors such as sensor calibration and performance, battery level, storage and processing capacity, measurement rate, accuracy and precision, transmission rate and type. Secondly, at processing layer the quality of data strongly depends on the processing mechanisms used to transform data as: raw sensor data gathering and validation, data processing and storage. More explicitly, like processing mechanisms (e.g. aggregation), the presence or absence of data validation mechanisms (e.g. filtering), the quality of service of the main server (storage level, availability, server load...) as well as the processing time and temporality of sensor data (i.e. update, historical, recent). Finally, at the discovery layer quality factors are related to how data is extracted, represented and queried. In this instance, we identify factors like automatic extraction mechanisms, representation models or the human factor.

Sensor Data Quality Criteria

Considering the nature of impact factors, we estimate that sensor data quality is mainly defined by "the accuracy and reliability of sensor data sources and concerns also the temporality and adequacy of data used for an intended goal".

In fact, from acquisition to delivery, sensor data can experiment quality degradation. Inspired on existing data quality management approaches [START_REF] Batini | Methodologies for data quality assessment and improvement[END_REF], we describe the quality criteria adapted in our approach.

In accordance with the nature of impact factors, we have analyzed the criteria proposed to evaluate the quality of geographic data and the quality of services [START_REF] Devillers | Fundamentals of Spatial Data Quality[END_REF]5,[START_REF]Quality of service -Guide to methods and mechanisms[END_REF]. We illustrate in Figure 3 the results of our study. Regarding the three processing layers (acquisition, processing and discovery) and their impact factors, we classify quality criteria into three categories: context, internal and usage. The first category regroups the set of criteria selected to estimate the quality of raw sensor data at the acquisition layer: accuracy, reliability, spatial precision, completeness and communication reliability. Selected criteria allow us to estimate the quality on data sources, their context of acquisition and their transmission to the data management and processing center. The internal category integrates quality criteria such as consistency, currency and volatility. Their main goal is to avoid inconsistent information and to maintain the temporality of sensor data at a processing level. Finally, usage category includes criteria such as timeliness, availability and adequacy. We formalize such categories and criteria as follows.

Sensor data quality modeling

According to the literature, each data quality model has its own description level according to their goals and application domain [START_REF] Batini | Methodologies for data quality assessment and improvement[END_REF]. Our concept of data quality is hence inspired on several data quality management approaches and standardizations defined for the GIS [START_REF] Devillers | Fundamentals of Spatial Data Quality[END_REF]5,[START_REF]Quality of service -Guide to methods and mechanisms[END_REF].

Even if a generic model seems, at present, difficult to conceive, we propose a vision of data quality providing important genericity and enabling us to include this model at different application contexts. This model is mainly characterized by quality categories, criteria, indicators and measures. Each category can be associated to a particular property of data and each criterion can be associated to one or more indicators accordingly. A given indicator may correspond to a measure or a set of measures related to several quality criteria. In Figure 4, we depicted the correlation of our quality principles and the sensor data modeling. Here, quality information is related to sensor data at different granularity levels. A data quality criterion can be defined to evaluate measure series during an instant or period of time (i.e. phenomenon observation). Thus, one or several criteria can be related to a particular measure, to a set of measures or to a stream of measures. Evaluation results can thus be depicted by quality indicators.

Accordingly, a data quality category may refers to a quality component within a monitoring system (i.e. context, internal and usage). A data quality criterion must be considered as an extension of data (qualitative or quantitative) and referring to factors impacting quality as reliability, accuracy, etc. A data quality indicator is a value resulting from a measure or a set of quality measures and a data quality measure corresponds to the evaluation method applied to data in order to qualify it. Due to the lack of space, evaluation algorithms are not described in this paper.

Management of sensor data quality

In a dynamic context as environmental monitoring, sensor data are enhanced with contextual or complementary information as sensor battery level, position, etc. In our approach, we consider also quality information as complementary information describing the reliability of sensor data. Any further information describing data is called metadata, which means "data about data" [START_REF]Understanding Metadata[END_REF].

Manage data together with quality information, and thus with metadata in a real-time context reveals several challenges. We propose as follows, a metadata based approach oriented to manage quality information in real-time.

Modeling quality information : sensor data and metadata

Managing quality information in a monitoring context implies a strong correlation between sensor data and the processing of data quality. Here, sensors mean the source of measures and quality information is considered as complementary information to enhance sensor data understanding.

It is worth to mention that in monitoring applications, the management of quality information requires updated metadata, capable to represent and explain the evolution and behaviour of observed data. In this context, metadata are data related to sensors behaviour and monitoring context as well as closely linked to data quality information. We hence consider metadata as "data about data that meets user requirements for a specific location and instant of time".

According to these facts, we propose a metadata modeling oriented to structure complementary information about sensor data in a monitoring system. This model respects the dynamicity (time and space), granularity (abstraction level) and generality (generic or applicative) of data.

Basically, metadata in our context must take into consideration information about: Observation, Sensor and Quality (Figure 5). Observation metadata refers to information describing the specificities of an observation. For this kind of metadata, complementary information is identified as general or referring to observation's responsible. Sensor metadata refers to static and dynamic information about a sensor, allowing us to identify and evaluate the capacities of a sensor at an instant of time. Data quality metadata refers to information indicating the properties of data that we are evaluating such as criteria, measure, indicator etc. Such metadata includes the description of quality assessment principles.

Managing the quality of sensor data

The management of sensor data quality implies information about the dynamic changes of quality values as well as the methods used to access to information. In a general way, our solution to manage data quality consists in the estimation of quality criteria at each processing layer. During this process, quality information is cached at a sensor, gateway and server level. Here, our quality model is applied and supported by metadata in order to provide users with more complete information about sensor data.

More specifically, if we consider that in an observation, a sensor or sensor node provides information related to measured values and metadata. Measures are the sensed values (temperature, pressure…) and metadata refers to information like sensor lineage, behavior or the quality. During sensor data processing, a complex quality function is employed to determine the quality of data sensor data at each processing layer and according to the selected quality criteria. In this way, each object resulting from a data, a dataset or a data stream is linked to quality criteria and then allowed to be estimated. Results from the estimation of sensor data quality criteria will be transmitted to the user by the means of quality indicators or data quality reports.

In our approach, we chose to characterize indicators in a graphical way, using icons exploited in an user interface.

Besides, data quality reports can be setting and generated according to experts requirements.

Discovery of Sensor Data Quality Information

In order to assist experts in the assessment of sensor data quality, we propose a prototype of visualization interface. Our prototype allows the visualization and discovery of data coming from sensors together with the communication of contextual information such as monitoring information and the quality of sensor data. This prototype allows us to validate our approach and implement several mechanisms as geospatial data discovery, sensor data quality management, as well as the use of visual quality indicators and reports. This prototype proposes users a way to interact with sensor data and quality information in a monitoring system.

Scenario: volcano activity

We take as an example the surveillance of a Mexican volcano: Popocatepetl. The Popocatepetl is localized at 60 kilometers from Mexico City and placed under the surveillance of the National Center for Disaster and Prevention of Mexico (Cenapred). The Cenapred uses a set of sensors distributed on 25 stations and process approximately 64 telemetry signals with 16 computers. Such monitoring system supervises in a visual, seismic, geodetic and geochemical way the behavior of the volcano. As this case of study shows, users of environmental monitoring systems are interested on discovering information about the evolution of the observed phenomena.

MoSDaQ prototype

The MoSDaQ (Monitoring Sensor Data Quality) prototype refers to a web-oriented user interface intended to monitor natural phenomena. This prototype attempts to be exploited by experts (locally or remotely) at a client side (Figure 6). This proposal is composed by five main sections: mapping, observation, sensor information, data querying and quality indicators). Mapping section refers to the localization of sensor objects in the cartographic space, what we call observation zone. In this zone, we place several sensors in a given position (by coordinates) and characterized by an icon according to its type and status. By clicking on each sensor icon, we have access to sensor information (static and dynamic). Observation section introduces all the information related to observed phenomena and elements. This section notify four important aspects: observed phenomenon, observation features, measured elements and coordinates of observation zone. Also, query section allows querying current or historical observation data according to its spatial, temporal, quality or semantic properties.

Sensor description section provides all technical information about each sensor located in the observation zone, as type, supplier, operational features and constraints. Besides, quality indicators section enables the access to quality properties of sensor data, to setting them and to visualize them. A data quality report can be also produced and visualized (Figure 7).

Conclusion

This paper introduces our proposal for the definition and evaluation of sensor data quality in a geospatial monitoring context. Our study of quality over sensor data implied the analysis of sensor data specificities. Accordingly, we propose a model attempting to formalize sensor quality properties (categories, criteria and indicators) and the corresponding measures. Our approach allows representing such properties and employs a product approach to evaluate them; qualifying data from acquisition to discovery. This contribution is mainly supported by metadata as a complementary information source. Regarding sensor data specificities, we analyzed and modeled metadata from a dynamic point of view, considering their granularity, generality and their spatiotemporal properties.

In order to complement our approach, a web-oriented user interface for sensor data quality discovery in real-time has been implemented. This interface is based on the specificities of a volcano monitoring system.

Figure 1 :

 1 Figure 1: Natural phenomena monitoring system framework.

Figure 2 :

 2 Figure 2: Sensor data model for monitoring systems

Figure 3 :

 3 Figure 3: Sensor Data Quality Criteria

Figure 4 :

 4 Figure 4: Sensor Data Quality Model

Figure 5 :

 5 Figure 5: General Metadata Model

Figure 6 :

 6 Figure 6: MoSDaQ web-based user interface.

Figure 7 :

 7 Figure 7: Sensor Data Quality Report -MoSDaQ


Etienne Thoret
email: etienne.thoret@mail.mcgill.ca

Baptiste Caramiaux

Philippe Depalle

Stephen Mcadams

Learning metrics on spectrotemporal modulations reveals the perception of musical instrument timbre

Humans excel at using sounds to make judgements about their immediate environment. In particular, timbre is an auditory attribute that conveys crucial information about the identity of a sound source, especially for music. While timbre has been primarily considered to occupy a multidimensional space, unravelling the acoustic correlates of timbre remains a challenge. Here we re-analyze 17 datasets from published studies between 1977 and 2016 and observe that original results are only partially replicable. We use a data-driven computational account to reveal the acoustic correlates of timbre. Human dissimilarity ratings are simulated with metrics learned on acoustic spectrotemporal modulation models inspired by cortical processing. We observe that timbre has both generic and experiment-specific acoustic correlates. These findings provide a broad overview of former studies on musical timbre and identify its relevant acoustic substrates according to biologically inspired models.

Introduction

The human auditory system processes acoustic information through several levels of increasing complexity [START_REF] Huang | Connecting deep neural networks to physical, perceptual, and electrophysiological auditory signals[END_REF][START_REF] Kell | A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy[END_REF] . At the first levels, transformations of the physical vibrations into neural activity conveyed by the peripheral auditory system are reasonably well known [START_REF] Moore | An introduction to the psychology of hearing[END_REF] . Yet, nonlinear transformations carried out at higher cortical levels remain poorly understood from a computational point of view. This is all the more striking when considering the perception of complex and rich sounds such as those from musical instruments, which involves a complex combination of both bottom-up and top-down processes. As such, achieving a unified comprehension of how human auditory perception responds to such complex sounds has the potential to critically advance our global understanding of the processing of acoustic information by the auditory system. We propose a meta-analysis of 17 published datasets and apply metric distance learning to acoustic models of spectrotemporal modulations similar to those represented in primary auditory cortex. Our main contributions are thus: 1) to address theoretically the historical question of defining the acoustical basis of musical instrument timbre, and 2) to develop a data-driven method to reveal the acoustic correlates of sound perception and interpret these correlates with spectrotemporal modulation (STM) models. More generally, this study implements a data-driven procedure to address the relationship between the physical properties of complex sounds and their perceptual ratings.

Musical instrument sounds have perceptually salient properties like loudness (perceived energy), pitch (perceived frequency), and timbre (related to their "sound quality"). Although the first two properties are well understood, the timbre of musical instrument sounds remains complex and ill defined [START_REF] Siedenburg | Four distinctions for the auditory "wastebasket" of timbre[END_REF] . Wherein lies the difference between the sounds produced by two musical instruments playing the same pitch at the same loudness? Participants are usually asked to rate the dissimilarity between pairs of sounds (Figure 1A) from which relevant perceptual dimensions of a timbre space are revealed by multidimensional scaling (MDS) (Figure 1B) [START_REF] Plomp | Timbre as a multidimensional attribute of complex tones[END_REF][START_REF] Wessel | Timbre space as a musical control structure[END_REF][START_REF] Grey | Perceptual effects of spectral modifications on musical timbres[END_REF][START_REF] Grey | Multidimensional perceptual scaling of musical timbres[END_REF][START_REF] Krumhansl | Why is musical timbre so hard to understand[END_REF][START_REF] Iverson | Isolating the dynamic attributes of musical timbre[END_REF][START_REF] Mcadams | Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes[END_REF][START_REF] Lakatos | A common perceptual space for harmonic and percussive timbres[END_REF][START_REF] Barthet | From clarinet control to timbre perception[END_REF][START_REF] Patil | Music in our ears: the biological bases of musical timbre perception[END_REF][START_REF] Elliott | Acoustic structure of the five perceptual dimensions of timbre in orchestral instrument tones[END_REF][START_REF] Siedenburg | Acoustic and categorical dissimilarity of musical timbre: Evidence from asymmetries between acoustic and chimeric sounds[END_REF][START_REF] Ogg | Acoustic Correlates of Auditory Object and Event Perception: Speakers, Musical Timbres and Environmental Sounds[END_REF] . MDS produces a low-dimensional parametric space (typically with two or three dimensions) in which sounds are assigned coordinates, and the distances between sounds reflect their perceptual dissimilarities. Relevant dimensions of these timbre spaces are then correlated with audio descriptors computed from the sound signal, enabling a psychoacoustic interpretation of musical timbre perception. However, this approach suffers from severe limitations. First, it has not been tested simultaneously across a wide variety of musical instrument timbre datasets, which renders its generalizability uncertain. Second, among a plethora of audio descriptors, only two descriptors have been found to correlate well with the two first dimensions of timbre spaces: the attack time, corresponding to the initial onset portion of the temporal envelope, and the centroid of the spectral content [START_REF] Mcadams | Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes[END_REF][START_REF] Mcadams | The perceptual representation of timbre[END_REF] . Despite more than 40 years of research on timbre, the acoustic correlates for higher dimensions remain unclear [START_REF] Elliott | Acoustic structure of the five perceptual dimensions of timbre in orchestral instrument tones[END_REF] . Finally, and most problematically, by reducing timbre spaces to two main dimensions, the subtle differences between musical instrument timbres are lost. A rigorous argument comes from a series of experiments done on cochlear implanted patients. A study [START_REF] Macherey | Perception of musical timbre by cochlear implant listeners: a multidimensional scaling study[END_REF] indeed observed that dissimilarity rating experiments run with hearing-impaired participants led to the same main two dimensions in timbre spaces, although these listeners are clearly unable to perceive the subtleties of musical timbre.

Here, we present the meta-analysis on 17 datasets, comprising a wide range of stimuli and dissimilarity measures, stemming from eight historical studies on timbre between 1977 and 2016 [START_REF] Grey | Perceptual effects of spectral modifications on musical timbres[END_REF][START_REF] Grey | Multidimensional perceptual scaling of musical timbres[END_REF][START_REF] Iverson | Isolating the dynamic attributes of musical timbre[END_REF][START_REF] Mcadams | Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes[END_REF][START_REF] Lakatos | A common perceptual space for harmonic and percussive timbres[END_REF][START_REF] Barthet | From clarinet control to timbre perception[END_REF][START_REF] Patil | Music in our ears: the biological bases of musical timbre perception[END_REF][START_REF] Siedenburg | Acoustic and categorical dissimilarity of musical timbre: Evidence from asymmetries between acoustic and chimeric sounds[END_REF] . The studies employed very different stimuli ranging from recorded notes to synthesized and hybrid sounds (see Extended Data Figure 1 for details on the datasets). We trained an interpretable distance metric that could reveal latent acoustic dimensions involved in sound perception. In other words, rather than determining the salient dimensions of dissimilarity measures and correlating these with scalar acoustic features [START_REF] Peeters | The timbre toolbox: Extracting audio descriptors from musical signals[END_REF] , we computationally train a distance kernel function k (similar to a weighted Euclidean distance) between acoustic models to maximally correlate with human dissimilarity measures (Figure 1C). The trained weights of this function highlight acoustic information fitting human perceptual distance.

We apply metric learning to acoustic models as achieved by primary auditory cortices, highlighting the spectrotemporal modulations of a sound event [START_REF] Chi | Multiresolution spectrotemporal analysis of complex sounds[END_REF] . Spectrotemporal modulation (STM) models have been shown to be relevant to the study of musical sounds [START_REF] Patil | Music in our ears: the biological bases of musical timbre perception[END_REF][START_REF] Albouy | Distinct sensitivity to spectrotemporal modulation supports brain asymmetry for speech and melody[END_REF] . It is however unknown which parts of these acoustic models are relevant from a perceptual point of view.

Results

Multidimensional scaling analysis.

To assess the replicability of previous studies, we first reanalyzed the 17 datasets with the classical MDS-based approach. Although parameters are usually adjusted differently in each study, here we used the same method for all the datasets (see Methods for details). We evaluated the Spearman correlations of the positions along the dimensions of the MDS solutions with the two most basic acoustic descriptors (hereafter expressed as ! 2 ): the logarithm of the attack time (LAT) and the spectral centroid (SC) [START_REF] Peeters | The timbre toolbox: Extracting audio descriptors from musical signals[END_REF] . Globally, the LAT correlates moderately with the first dimension of the timbre spaces [! 2 : Mdn = .41, IQR = .29, Extended Data 1 for the full statistics]. In addition, when available, the original acoustical analyses can be replicated more or less well depending on the study (Figure 2, Extended Data Figure 3). In order to optimize the correlation between LAT and SC with one of the two first dimensions, we chose a method that automatically finds the best set of MDS parameters, i.e. the number of projected dimensions and the optimal rotation of the MDS dimensions (see Methods).

Nevertheless, replicating the published correlation values remains very difficult. This is a major limitation of the MDS-based approach. Indeed, it critically depends on implicit decisions made by the authors, often not accurately and/or comprehensively reported in the published papers: the experimenter always makes choices to optimize the final outcomes, e.g., by choosing one particular type of MDS, adapting its parameters, or hand-tuning the audio descriptors used for the correlations. Taken together, these observations show the limitation of the dimensional approach and the need for an alternative to reveal the acoustical substrates of musical timbre.

Spectro-Temporal Modulations of musical sounds. Spectro-Temporal Modulation (STM) models are produced by mathematical tools that mimic the output of the processing of sounds by primary auditory cortical neurons. These models refer to Spectro-Temporal Receptive Fields (STRFs) [START_REF] Theunissen | Spectral-temporal receptive fields of nonlinear auditory neurons obtained using natural sounds[END_REF][START_REF] Shamma | On the role of space and time in auditory processing[END_REF] , which are tuned to fire for specific STM patterns in sounds. Practically, the auditory spectrogram, representing the cochlear processing of a sound, is projected into a four-dimensional representation space characterized by time, frequency (in Hz), temporal modulations (rate in Hz), and spectral modulations (scale in cycles/octave) (Figure 1). This STM representation (STMF: Spectro Temporal Modulation transfer Function) reflects the multiresolution analysis of the spectrotemporal information [START_REF] Chi | Multiresolution spectrotemporal analysis of complex sounds[END_REF][START_REF] Chi | Spectro-temporal modulation transfer functions and speech intelligibility[END_REF] , which has already provided insights into the identification of musical instrument sounds [START_REF] Patil | Music in our ears: the biological bases of musical timbre perception[END_REF][START_REF] Suied | Auditory sketches: sparse representations of sounds based on perceptual models[END_REF][START_REF] Isnard | Auditory sketches: very sparse representations of sounds are still recognizable[END_REF][START_REF] Thoret | Perceptually salient spectrotemporal modulations for recognition of sustained musical instruments[END_REF][START_REF] Thoret | Perceptually salient regions of the modulation power spectrum for musical instrument identification[END_REF] and correlates with human brain activity [START_REF] Halpern | Behavioral and neural correlates of perceived and imagined musical timbre[END_REF][START_REF] Allen | Representations of pitch and timbre variation in human auditory cortex[END_REF][START_REF] Ogg | Separable neural representations of sound sources: Speaker identity and musical timbre[END_REF] . In this study, 128 frequency channels, 22 rates, and 11 scales were chosen to compute the representations (see Methods). As these representations are high-dimensional, they are averaged over time leading to tensors of dimension 128 × 22 × 11 (called the Full STMF below; further technical details on computations are provided in Methods) (Figure 1C, left box). In order to understand the role of the three acoustic dimensions embedded in the global 3D tensor, representations projected onto each pair of the three dimensions will also be considered, i.e., scale/rate (averaged over frequency), frequency/rate (averaged over scale) and frequency/scale (averaged over rate) (Figure 1C, middle box).

Optimized metrics simulating human dissimilarity ratings. The human ratings are perceptual distances between sounds. To understand the extent to which these ratings are derived from the acoustic difference between sounds, we fit a distance kernel (radial basis function) between STM representations that best approximate the perceptual distances (see Methods and Supplementary Figure 1). This method seeks the parts of the STMFs that are the most relevant to reproduce the ratings of the stimuli. On median, across the 17 datasets, 78% of the variance in dissimilarity ratings is explained by the optimized metrics [r 2 : Mdn=.78, IQR=.58] (see Table 1 and Supplementary Table 2). We control for overfitting of each metric with the leave-one-sound-out cross-validation method (see Methods, Extended Data Figure 4 and Supplementary Table 3).

Notably, the 2D projections scale-rate, freq-rate and freq-scale do not explain a large proportion of variance alone [scale-rate: r 2 : Mdn=.20, IQR=.25; freq-rate: r 2 : Mdn=.32, IQR=.49; freq-scale: r 2 : Mdn=.55, IQR=.57] (see Supplementary Table 4, Supplementary Table 5, Supplementary Table 6). Lastly, as frequency and rate have more dimensions than scale, we tested for another potential overfitting in these two dimensions by running the previous analysis after having downsampled the rates and frequencies to the same number of dimensions as the scales (10 frequencies × 10 rates × 10 scales, see Methods for details). The explained variances obtained with this test are lower on average than with the higher-dimensional representation (see comparison in Table 1, Supplementary Table 7, Supplementary Table 8, Supplementary Table 9, Supplementary Table 10) but respect the same trends. As a first control, we ran the optimization process with the timeaveraged auditory spectrum, which revealed lower proportion of variance explained [r 2 : Mdn = .19, IQR = .48] (see Supplementary Table 11). As a second control, we computed the Euclidean distance between Full STMFs of pairs of stimuli to show the interest of learning the distance. This does not allow us to accurately simulate the human dissimilarities [r 2 : Mdn = .11, IQR = .20] (see Table 1 andSupplementary Table 12, Supplementary Table 13). These results support the relevance of the distance metric learning approach to suggest a link between human behavioral data with acoustic representations of the stimuli.

Acoustic interpretations of the metrics. We first assess whether the kernels fitted to each individual dataset show similarities between datasets. We computed pairwise Spearman (!) correlations between the fitted weights w (see Methods). Spearman correlations were computed as the different fitted metrics may have different scales that may bias Pearson's correlations values.

A summary of the full statistics is available in Supplementary Table 14. The fitted kernels for the Full STMF do not generalize well across the different datasets [! 2 (30,974): Mdn=.25, IQR=.17].

Nevertheless, if we inspect each 2D projection of the metrics, as observed in Figure 4, for example, the fitted weights appear to have qualitatively similar traits. In particular, the weights fitted on the scale/rate projection are very similar across all of the datasets [! Clustering of optimized metrics. In order to understand in depth how the stimuli of the experiments drive the properties of the metrics, clustering analyses were run based on the previous correlational analyses (Figure 3). For the Full STMF and each of its three projections, a cluster analysis was conducted based on the pairwise Spearman correlations. This analysis aims to understand whether: 1) the stimulus sets of similar stimuli group together; 2) edited, recorded and synthesized stimuli group together; 3) fundamental frequency affects grouping; and 4) which representation captures general and more stimulus-set-specific features. Based on the clusters determined with a hierarchical clustering of the Spearman correlation coefficients with a complete linkage method, the first two factors do not affect grouping in the case of the datasets under consideration. It's worth noting that 10 out of 17 datasets have the same fundamental frequency (311Hz) which partly limits the potential conclusions regarding the effect of fundamental frequency. Further research should therefore be conducted to investigate more systematically the relationship between fundamental frequency and timbre. However, scale-rate is the representation that captures most of the generality across the datasets as shown by the tight clustering of all but five datasets. This reinforces the idea that timbre has generic dimensions embedded in the scalerate projection and that the subtler aspects of timbre that are specific to the stimulus set are embedded in the frequency-rate and frequency-scale projections.

Timbre perceptual metrics are experiment-specific. In order to understand the links between the stimuli and the metrics for each dataset, Pearson correlations between the fitted metrics and the sample-wise standard deviations of the stimuli were computed. Practically, for each dataset and each metric, we first compute the standard deviation for each bin of the representation across the stimuli, i.e., the standard deviation "pixel per pixel". Then, the Pearson correlation between the vectorized arrays of standard deviations and the vectorized version of the metrics was computed. This revealed that the full STMF is strongly correlated with the stimulus standard deviations [r 2 (30,974): Mdn=.75, IQR=.12] (Figure 4, Supplementary Figures 23456789101112131415161718, Supplementary Table 15). If we inspect these correlations for each of the three 2D representations, all three strongly correlate with the stimulus sample-wise standard deviations [scale-rate-r 18). This result strongly suggests that perceptual metrics are experiment-dependent and can be derived from the stimulus variability. In order to understand more precisely the link between the optimized metrics and the stimuli of each experiment, we performed a multiple linear regression between the fitted metrics and the stimuli for the four representations (Full STMF, scale-rate, freq-rate, freq-scale) for each dataset. We observed that each metric can be decomposed (Supplementary Table 19). This analysis reinforces the fact that the metrics fitted for each dataset is principally driven by the information embedded in the range of freq-rate and freq-scale projections in each dataset. Conversely, scale-rate, which is more generic across the different datasets, does not share common properties with freq-rate and freq-scale decompositions.

Discussion

In this paper, we addressed a historical question about musical instrument timbre perception: is the timbre of sounds best represented as a rigid acoustical space, or is it composed of both generic acoustical correlates embedded in the spectral and temporal envelopes and stimulus-set-specific ones embedded in the frequency/scale and frequency/rate patterns of STM representations.

Through a computational meta-analysis of 17 experiments, this study showed that the classical MDS-based approach of correlating positions of stimuli along a given perceptual dimension with univariate acoustic descriptors is only partially replicable for most of the studies, and that it is only adapted to investigate the generic acoustic correlates of timbre. With the distance metric learning method, we determine which information in the STM representations of sounds best simulates human dissimilarity ratings of musical instrument timbres. This study revealed that STM representations are crucial tools, and in particular we uncovered the role of each of their three projections for the perception of musical instrument timbre. This makes a significant step forward in understanding the acoustic correlates of musical instrument timbres by showing how listeners use the variability of the frequency/scale and frequency/rate representations of the stimuli to make their perceptual ratings. More globally, this study supports a generic data-driven procedure to address the relationship between the physical properties of sounds and their perceptual ratings.

Most of the studies in the last 40 years have considered timbre as a fixed perceptual space with dimensions correlating with acoustic descriptors [START_REF] Grey | Perceptual effects of spectral modifications on musical timbres[END_REF][START_REF] Grey | Multidimensional perceptual scaling of musical timbres[END_REF][START_REF] Iverson | Isolating the dynamic attributes of musical timbre[END_REF][START_REF] Mcadams | Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes[END_REF][START_REF] Lakatos | A common perceptual space for harmonic and percussive timbres[END_REF][START_REF] Patil | Music in our ears: the biological bases of musical timbre perception[END_REF] , see 4 for a review. Although it has been efficient for the two first dimensions, this led to a never-ending debate concerning the number of perceptually relevant dimensions of the underlying musical instrument timbre spaces [START_REF] Elliott | Acoustic structure of the five perceptual dimensions of timbre in orchestral instrument tones[END_REF][START_REF] Terasawa | The thirteen colors of timbre[END_REF] . In addition, as shown in the MDS-based meta-analysis, the replicability of these studies is not obvious and is sensitive to the MDS model parameters and acoustic descriptors used by the authors of each study. Nevertheless, the present results confirm the relevance of the temporal and spectral envelopes: the fitted metrics indeed generalize in the scale/rate representation. But here, rather than focusing on a reduced dimensional space, we observed that human dissimilarity ratings are explained by the variability of the STM representations of the stimuli in each experiment, whereas the MDS-based approach excludes significant information by reducing a complex spectrotemporal morphology to a small number of dimensions and scalar descriptors. We argue here that in addition to these main acoustic dimensions, listeners adjust the range of their dissimilarity ratings based on the variability of more subtle sub-representations of the STM representation: the frequency/scale and frequency/rate projections, which can be directly observed by way of the optimization of Gaussian kernels. Practically, this adjustment of the dissimilarity range by the listeners may depend on the specific stimulus set, either during the training preceding each experiment or even over the course of the experiment itself.

The previous acoustic considerations support the use of STM representations and this notably relates to neurophysiological observations. In particular, the neuronal populations of primary auditory cortex-the spectrotemporal receptive fields-are indeed plastic and driven by attention [START_REF] Fritz | Rapid task-related plasticity of spectrotemporal receptive fields in primary auditory cortex[END_REF][START_REF] Kraus | Experience-induced malleability in neural encoding of pitch, timbre, and timing: Implications for language and music[END_REF][START_REF] David | Task reward structure shapes rapid receptive field plasticity in auditory cortex[END_REF][START_REF] Mesgarani | Selective cortical representation of attended speaker in multi-talker speech perception[END_REF][START_REF] Kaya | Modelling auditory attention[END_REF] . Brain imaging studies also support the hypothesis that musical timbre is specifically encoded in the human auditory cortex [START_REF] Halpern | Behavioral and neural correlates of perceived and imagined musical timbre[END_REF][START_REF] Allen | Representations of pitch and timbre variation in human auditory cortex[END_REF][START_REF] Allen | Encoding of natural timbre dimensions in human auditory cortex[END_REF] . Human dissimilarity ratings could be adjusted based on this plasticity and through attentional processes that focus on different acoustic factors, which may depend on selective attention processes. These findings support the hypothesis that relevant STM cues can thus be opportunistically probed by attentional processes according to the listening situation and to the task, here judging the dissimilarity between two sounds. The STM representation thus provides a relevant unbiased acoustic input to more complex modelling of auditory processes as those performed by higher cortical levels. More complex tasks would indeed need more complex models given that the acoustic correlates feeding the Gaussian kernel metrics are not complex enough to reflect the globality of the cortical processes' nonlinear behaviours 2 .

Understanding which acoustic information is relevant to perform an auditory task is a crucial issue in auditory cognitive neuroscience that goes well beyond musical instrument perception to speech [START_REF] Albouy | Distinct sensitivity to spectrotemporal modulation supports brain asymmetry for speech and melody[END_REF][START_REF] Flinker | Spectrotemporal modulation provides a unifying framework for auditory cortical asymmetries[END_REF][START_REF] Ponsot | Cracking the social code of speech prosody using reverse correlation[END_REF] , environmental sounds [START_REF] Nelken | An ear for statistics[END_REF] , and even animal bioacoustics [START_REF] Bregman | Songbirds use spectral shape, not pitch, for sound pattern recognition[END_REF] . The method implemented here allows one to determine: (1) whether an acoustic representation is relevant to simulate a given dissimilarity rating perceptual task, and (2) which information is used to reproduce the task. In the present case, STM representations were used. While the gold-standard approach has mainly used combinations of large numbers of audio descriptors to explain auditory tasks [START_REF] Lartillot | A matlab toolbox for music information retrieval[END_REF] , the tools clearly lack interpretability regarding which biologically plausible representations are useful to model auditory cognition [START_REF] Aucouturier | Seven problems that keep MIR from attracting the interest of cognition and neuroscience[END_REF] . Here we implement a framework in which we interpret the weights of a Gaussian kernel on STM representations. The strength of this approach can be generalized with other metrics [START_REF] Bellet | A survey on metric learning for feature vectors and structured data[END_REF] , to other acoustic models, e.g., statistical representations [START_REF] Mcdermott | Sound texture perception via statistics of the auditory periphery: evidence from sound synthesis[END_REF][START_REF] Anden | Joint time-frequency scattering[END_REF] , and to other parametric models, e.g., speech prosody [START_REF] Ponsot | Cracking the social code of speech prosody using reverse correlation[END_REF] . The method implemented here thus aims to provide a way to investigate the links between low-level acoustic representations and high-level auditory judgements, which was not easily achieved with the statistical methods used up to this point. It nevertheless captures a significant part of the variance in human dissimilarity ratings and links it to a potential acoustical substrate. This approach has the potential to be applied in many other domains of cognitive neuroscience, in particular to make non-intuitive hypotheses on a possible link between high-dimensional models and global dissimilarity data.

Methods

Ethics.

Our study reports an analysis of quantitative experimental data collected from human participants who rated dissimilarity between pairs of sounds. No participants were actually run in this study as the data were provided by the original authors. For details on ethical regulations and editorial board references, see the original papers.

Acoustic representations of sounds. To enhance the speed of processing in the modelling effort, the sound files of the different studies, initially sampled at 44.1 kHz, were first down-sampled to 16 kHz. The auditory representations were computed with an adapted version of the NSL Tools [START_REF] Chi | Multiresolution spectrotemporal analysis of complex sounds[END_REF] .

Cochlear processing is modelled as a bank of 128 constant-Q asymmetric bandpass filters equally spaced on a logarithmic frequency scale spanning 5.3 octaves. Inner hair cell potentials and lateral inhibitory networks are modelled by a high-pass/low-pass filter and spectral sharpening, respectively. The loss in phase locking operating at the midbrain level is performed by using a short-term integration of 4 ms. These computations result in the auditory spectrogram, a twodimensional time-frequency array. The auditory spectrum is computed by averaging the auditory spectrogram along the temporal dimension. The spectrotemporal modulation model (STM) is computed by applying a spectrotemporal modulation filterbank to the spectrogram. A detailed description is provided by Patil and colleagues [START_REF] Patil | Music in our ears: the biological bases of musical timbre perception[END_REF] . Practically, a two-dimensional Fourier transform is first applied to the spectrogram, which results in a two-dimensional array, also called the Modulation Power Spectrum (MPS) [START_REF] Thoret | Perceptually salient spectrotemporal modulations for recognition of sustained musical instruments[END_REF][START_REF] Flinker | Spectrotemporal modulation provides a unifying framework for auditory cortical asymmetries[END_REF] whose dimensions are spectral modulation (scale) and temporal modulation (rate). Because of the spectral symmetry of the spectrogram, only positive scale values are kept, whereas both positive and negative rates are kept. Finally, the Spectro-Temporal Modulation transfer Function (STMF) representation is a multiresolution analysis of the spectrogram which is derived by filtering the MPS according to different rates and scales and then coming back to the time-frequency domain. The final STMF representation can be seen as a series of spectrograms filtered according to different rates and scales. Here, we chose the following scale (s) and rate (r) center values as 2D Gaussian filters to generate the STM representation: s = [0.25, 0.35, 0.50, 0.71, 1.00, 1.41, 2.00, 2.83, 4.00, 5.66, 8.00 cyc/oct], r = ± [4.0, 5.7, 8.0, 11.3, 16.0, 22.6, 32.0, 45.3, 64.0, 90.5, 128.0 Hz]. The resulting representation thus corresponds to a 4D matrix with time, frequency, scale, and rate dimensions. This representation is then averaged across time to make the Full STMF representation (frequency, scale, rate). The Full STMF is then averaged across each of its three dimensions to make the 2D scale/rate, frequency/rate, and frequency/scale representations. Decimated representations were also used with the following parameters: s = [0.25, 0.50, 0.71, 1.00, 1.41, 2.00, 2.83, 4.00, 5.66, 8.00 cyc/oct], r = ± [4.0, 11.3, 22.6, 45.3, 128.0 Hz] and frequency channels = [1, 15, 29, 43, 57, 72, 86, 100, 114, 128].

Stimulus datasets. We investigated 17 published timbre spaces that used different kinds of sounds: natural musical instrument sounds that have been analyzed and then resynthesized with simplifications or systematic modifications (Grey, 1977

MDS-based analysis.

For each dataset, eight standard non-metric MDS analyses were performed with Matlab (The MathWorks). The choice of dimensionality onto which the MDS is projected is generally based on the curve representing stress as a function of number of dimensions. Here we chose to use eight values, from 2 to 10 dimensions, which is clearly higher than the average number of dimensions generally sufficient to fit an MDS solution to a perceptual dataset. In addition, this allows us to be less user-specific. For each dataset and each MDS dimension, we then evaluated the Spearman correlation of the first two dimensions with the two most classical descriptors: the logarithm of the attack time (LAT) and the spectral centroid (SC). We finally assessed the correlation between the descriptors and the first two perceptual dimensions by choosing the solution from among the eight with the maximum correlation (! 2 ). As the non-metric MDS is based on ranks we here report the Spearman correlation which provides a non-parametric assessment of correlation between two variables. The LAT was computed as in Patil et al. (2012) by taking the logarithm of the time to increase the temporal envelope from -40 dB to -12 dB relative to the maximum waveform amplitude. The SC was computed as the centroid of the Fast-Fourier Transform averaged over all time frames. Optimization of Gaussian kernels. For each dataset, we learn a distance metric mimicking human ratings from a STM representation. Distance metric learning is a well-known problem in machine learning [START_REF] Vinyals | Matching networks for one shot learning[END_REF][START_REF] Goldberger | Neighbourhood components analysis[END_REF] , which aims to learn the coefficients of a distance in order to fit with a given distance.

These coefficients can be interpreted as weights on the representations of the stimuli. Here the learned distance, the kernel, is a radial basis function (see below) between STMFs of two sounds x and y:

k(x, y) = exp ,-∑ |" ! #$ ! | " & ! " ' ()* / (Eq. 1)
where x and y are the vectorized version of the STMFs defined in the previous section. Practically, each stimulus is initially a three-dimensional array with 22 rates, 11 scales, and 128 frequencies, which is reshaped into a one-dimensional array of size N = 22 x 11 x 128. w is the array of coefficients (weights). From a generic point of view, the distance metric learning here can be interpreted as a way to optimize weights on a Euclidean distance passed through an exponential function. The main interest here is that the relative importance of each feature can be directly observed as masks on the STMFs, which makes the method fully interpretable. More precisely, the learned coefficients w array has the same dimensionality as the STMF and can then be directly observed by reshaping it to the original 22 x 11 x 128 three-dimensional matrix.

The loss function used to fit the kernel (learning weights w) is the correlation between the simulated distances between sounds and the mean dissimilarity ratings. In other words, the objective is to learn the weights w by maximizing the Pearson correlation between the kernel distances k and the human ratings r, which can be formulated as:

J = ∑ ∑ ,-(" ! ," # )#12,3(" ! ," # ) # 42 $ #%& $ !%& 5 ' 5 ( (Eq. 2)
where 1 and 2 and (σ -and σ 3 , respectively) are the means (standard deviations) of the representations and human dissimilarity ratings across sounds, respectively. Optimizing the weights w thus allows one to observe how the kernel emphasizes certain dimensions of the STM representation to mimic the human ratings (Figure 4 & Supplementary Figure 23456789101112131415161718).

The optimization of the Gaussian kernel weights, i.e., wi in Eq. 1, is performed by maximizing the correlation between the human dissimilarities and the kernel distance. In practice we considered the log-kernel, which simplified the computation of the first-and second-order derivatives and stabilized the optimization. The optimization is performed using the limitedmemory Broyden-Fletcher-Goldfarb-Shannon (L-BFGS) algorithm. This is a particularly fast gradient-descent algorithm that is adapted for parameter estimation in machine learning with very large numbers of variables 51 . For each dataset, the generalizability of the fitted kernel is first cross-validated with the leaveone-sound-out method. The metric is then refitted on the whole dataset and the correlation with the cross-validated set is evaluated. a The values in parentheses correspond to the explained variance with the downsampled representations.

Figure 2 ]

 2 Figure 2] and moderately with the second dimension [! 2 : Mdn = .20, IQR = .32, Extended Data Figure 2]. Mdn is the median and IQR is the Interquartile Range of the Spearman correlations for the 17 datasets. The degrees of freedom ranges between 9 and 18. The detailed full statistics (significance, statistical power and 95% Confidence Intervals) for each dataset are reported withinthe Supplementary Table1. It must be noted that datasets including sounds whose temporal

  as a linear combination of the different stimuli representations [Full STMF-R 2 : Mdn=.86, IQR=.05; scale-rate-R 2 : Mdn=.98, IQR=.04; freq-rate-R 2 : Mdn=.96, IQR=.05;freq-scale-R 2 : Mdn=.92, IQR=.05]. Secondly, in order to understand whether the four representations led to the same linear combination of stimuli, we computed the pairwise Spearman correlations between the regression weights obtained with the four different projections for each dataset. High correlations reflect that the global metric (Full STMF) is decomposed in the same way as the other considered metric. We observed that the weights of the linear combinations explaining the Full STMF metric correlate moderately with the weights of the regressions obtained with the freq-rate and freq-scale representations [Full STMF/freq-rate-! 2 : Mdn=.59, IQR=.33; Full STMF/freq-scale-! 2 : Mdn=.55, IQR=.38] (Supplementary

  ; Grey & Gordon, 1978; Barthet et al., 2010; Siedenburg et al., 2016, Exp. 2A Sets 2 and 3), imitations and hybrids of musical instruments synthesized with a frequency-modulation algorithm available on a commercial synthesizer (McAdams et al., 1995), and recorded and edited natural sounds (Iverson & Krumhansl, 1993; Lakatos, 2000; Patil et al., 2012; Siedenburg et al., 2016, Exp. 2A Set 1).

Figure 1 .

 1 Figure 1. Two different approaches to investigate the auditory perception of musical instrument timbre. Each approach aims to fit a model (green box) to human participants' dissimilarity ratings between pairs of sounds (blue box) on the basis of the acoustic information (pink box). (A) Experiments: listeners make dissimilarity ratings on all pairs of sounds to produce a dissimilarity matrix. (B) Historical approach: dissimilarity ratings are analyzed with multidimensional scaling and the dimensions are correlated with scalar audio descriptors computed from basic acoustic representations. This has several limitations; in particular, the audio descriptors and the MDS parameters are hand-tuned by experimenters leading to difficulties in replicating the findings in the literature. (C) Distance metric learning on the STM representation: this data-driven approach simulates human dissimilarity ratings from the STM representations of sounds, which can then be interpreted. Computation of the STM representation. The waveform is first transformed into a time-frequency representation-an auditory spectrogram-and then into a STM representation embedding frequency, rate (temporal modulations), and scale (spectral modulations). A distance k(x,y) is optimized by learning the weighting coefficients wi that best mimic the human dissimilarity ratings, providing an interpretable metric revealing the relevant information embedded in the STM representations.

Figure 2 .

 2 Figure 2. Replicability of the MDS-based approach. Spearman correlation (! 2 ) with the Logarithm of the Attack Time (LAT) (A) and the Spectral Centroid (SC) (B) in the meta-analysis vs. explained variance in the original study. The computed correlations in the meta-analysis are generally lower than the explained variance provided in the original studies. Eleven studies are considered for the LAT and nine for the spectral centroid corresponding to those available in the original papers (Extended Data Figure 3) -B2010: Barthet et al. (2010); P2012A3: Patil et al. (2012) A3 dataset; P2012GD4: Patil et al. (2012) GD4 dataset; P2012DX4: Patil et al. (2012) DX4 dataset; L2000P: Lakatos (2000) Percussive dataset; L2000H: Lakatos (2000) Harmonic dataset; L2000C: Lakatos (2000) Combined dataset; I1993W: Iverson & Krumhansl (1993) Whole dataset; I1993O: Iverson & Krumhansl (1993) Onset dataset; I1993R: Iverson & Krumhansl (1993) Remainder dataset; McA1995: McAdams et al. (1995).

Figure 3 .

 3 Figure 3. Generalizability of the metrics learned for the different dataset. Fitted weights are correlated between all pairs of datasets. In the dissimilarity matrices, yellow indicates a perfect correlation (identity on the diagonal) and dark blue a zero correlation. For each representation: (A) Full STMF, (B) scale-rate, (C) freq-rate, and (D) freq-scale, dendrograms with complete linkage from the Spearman correlations were computed. Dendrograms are presented on the top of the correlation matrices and represent the similarity structure of the optimized metrics among the datasets. The fundamental frequency of the stimuli in each dataset are mentioned at the right of each label.

Figure 4 .

 4 Figure 4. Correspondence between fitted metrics and standard deviations of the stimuli. Projections of optimized metrics (upper panels) and standard deviations between stimuli (lower panels) (from the A3 dataset in Patil et al., 2012) in the three projections of the STM representation, scale/rate (S/R -cyc/oct vs. Hz), frequency/rate (F/R -Hz vs. Hz) and frequency/scale (F/S -Hz vs. cyc/oct). The similarity of these representations is particularly high [r 2 (30,974) = .83 in the case of the Patil et al., 2012, A3 dataset and r 2 (30,974) = .74 on average across all the datasets] (Supplementary Figures 2-18 for the other datasets and Supplementary

Table

  . It must be noted that datasets including sounds whose temporal envelopes have been manipulated (Iverson & Krumhansl, 1993, Onset dataset; Iverson & Krumhansl, 1993, Remainder dataset)

  [START_REF] Grey | Perceptual effects of spectral modifications on musical timbres[END_REF][START_REF] Iverson | Isolating the dynamic attributes of musical timbre[END_REF] failed to provide high correlations with MDS dimensions, whereas those including unmanipulated stimuli had higher correlations. The correlations between SC and the first dimension are generally poor [! 2 : Mdn = .06, IQR=.07, Extended Data Figure2and Supplementary Table1for the full statistics], whereas it correlates well with the second dimension [! 2 : Mdn = .61, IQR=.45, Extended Data Figure2and see Supplementary Table

  2 (240): Mdn=.68, IQR=.33]. In addition, for each metric, most of the energy is centered on low temporal modulations (< 15 Hz) and spectral modulations around 1 cycle/oct. Conversely, the weights fitted on the frequency/rate and frequency/scale projections are specific to each dataset [! 2 (2,814): Mdn=.50, IQR=.23, and ! 2 (1406): Mdn=.28, IQR=.29, respectively]. As a control, we also equalized the dimensionalities of each representation to evaluate a potential influence of this factor on the correlation. Overall, it confirms our conclusions by showing very similar trends between the different projections [Downsampled Full STMF (10×10×10)-! 2 (98): Mdn=.30, IQR=.18; scale/rate-! 2 (98): Mdn=.58, IQR=.32; freq/rate-! 2 (98): Mdn=.55, IQR=.26; freq/scale-! 2 (98): Mdn=.45, IQR=.37]. We further refined this analysis by considering only the 10 datasets with stimuli that share the same fundamental frequency (Eb4 = 311 Hz). This confirms the previous observations: scale-rate metrics generalize between the datasets better than frequency-rate and frequency-scales [Full STMF: ! 2 (30,974): Mdn=.30, IQR=.18; scale/rate-! 2 (240): Mdn=.74, IQR=.25; freq/rate-! 2 (2,814): Mdn=.55, IQR=.22; frequency/scale-! 2 (1406): Mdn=.38, SD=.31].

  2 (240): Mdn=.90, IQR=.11; frequency-rate-r 2 (2,814): Mdn=.83, IQR=.04; frequency-scale-r 2 (1406): Mdn=.79, IQR=.10] (Figure 4, Supplementary Table 16, Supplementary Table 17, Supplementary Table

Table 19 )

 19 . Conversely, these weights have a lower correlation with those obtained with the scale-rate projections [Full STMF/scale-rate-! 2 : Mdn=.17, IQR=.31] (Supplementary Table 19). Lastly, the weights obtained with freq-rate and with freq-scale projections correlates moderately with each other [freq-rate/freq-scale-! 2 : Mdn=.31, IQR=.25] and do not correlate with the weights obtained with the scale-rate projection [scale-rate/freq-rate-! 2 : Mdn=.08, IQR=.18; scale-rate/freq-scale-! 2 : Mdn=.05, IQR=.24]

Table 2 )

 2 .

	Tables

Table 1 . Explained variance (r 2 ) of the human ratings by the optimized Gaussian kernels.

 1

	Study	Dataset	Degrees	Auditory	Euclidean	Full	Scale/	Freq/	Freq/
		name	of	Spectrum	distance on	STMF a	Rate a	Rate a	Scale a
			freedom		Full STMF a
	Grey, 1977 -	118	.47	.17 (.17)	.78	.10	.46	.57
						(.61)	(.10)	(.45)	(.48)
	Grey &	-	118	.11	.00 (.00)	.30	.09	.12	.18
	Gordon,					(.15)	(.08)	(.10)	(.09)
	1978
	Iverson &	Whole	118	.16	.26 (.26)	.83	.35	.40	.61
	Krumhansl,					(.48)	(.37)	(.24)	(.25)
	1993
		Onset	118	.08	.01 (.01)	.21	.06	.07	.13
						(.16)	(.08)	(.05)	(.08)
		Remainder	118	.03	.02 (.04)	.27	.04	.06	.10
						(.14)	(.03)	(.04)	(.04)
	McAdams	-	151	.30	.08 (.13)	.73	.20	.25	.42
	et al., 1995					(.52)	(.18)	(.22)	(.26)
	Lakatos,	Harmonic	134	.19	.06 (.02)	.83	.35	.40	.61
	2000					(.48)	(.37)	(.24)	(.25)
		Percussive	151	.18	.02 (.01)	.29	.08	.19	.17
						(.23)	(.08)	(.16)	(.15)
		Combined	188	.14	.08 (.09)	.33	.00	.17	.24
						(.20)	(.00)	(.15)	(.16)

Acknowledgments

. This work was supported by the Canadian Natural Sciences and Engineering Research Council awarded to SMc (grants RGPIN-2015-05280 and RGPAS 478121-15) and to PD (RGPIN-2018-05662), as well as a Canada Research Chair (grants 950-223484 and 950-231872) awarded to SMc. ET was funded through an ILCB/BLRI grant ANR-16-CONV-0002 (ILCB), ANR-11-LABX-0036 (BLRI) and the Excellence Initiative of Aix-Marseille University (A*MIDEX), BC was founded through EU Marie Skłodowska-Curie fellowship (Project MIM, H2020-MSCA-IF-2014, GA no. 659232). BC acknowledges STMS IRCAM-CNRS-Sorbonne Université in Paris where he was a Marie Sklodowska Curie research fellow at the beginning of the project. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. We thank the authors of the different studies reanalyzed for providing the stimuli and data from their experiments, Guillaume Mestdagh, Emmanuel Ponsot, and Benjamin Morillon for helpful discussions on earlier versions of the 20 manuscript, Mounya Elhilali and Daniel Pressnitzer for help in the initial implementation of the optimization framework.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon request and on: https://github.com/EtienneTho/musical-timbre-studies

Code Availability Statement

Custom codes that supports the findings of this study are available from the corresponding author upon request and on: https://github.com/EtienneTho/musical-timbre-studies

In this paper, we use the metric optimized on the whole set of sounds within each dataset.

However, this method may be prone to overfitting, which we assessed as follows. We tested the metric learning approach using a leave-one-sound-out cross-validation protocol. More precisely, for a dataset with N sounds, N folds were considered. For each fold, the metric was fitted considering N-1 sounds from the dataset, and tested on the remaining sound. Early stopping was used when the testing correlation reach a local minimum and then went up during 200 iterations.

Early stopping thus maximizes the cross-validation of the metric within each dataset. We found that the median of the N-1 training correlations is strongly correlated to the testing correlations across datasets (r 2 (15)=.93: CI95%=[.82;.98]; p < .001, power=1). Then, the fitted weights on each fold for a given dataset were correlated to the fitted weights on the whole set of sounds within the same dataset (r 2 : Mdn=.92, IQR=.109, Extended Data Figure 4, Supplementary Figure 19). We found that the metrics strongly correlate within the N folds (r 2 : Mdn=.85, IQR=.210, Extended Data Figure 4, Supplementary Table 3) showing that the metric learned on all the sounds within a dataset doesn't overfit. We then used this metric for the subsequent analyses.

Statistics

All the correlations reported in the paper are supported by Full Statistics: degrees of freedom (df), 95% Confidence Interval (95%CI) or 95% Confidence Interval range (95%CI range), and statistical power (power). They are either reported in the main article or in the supplementary materials. All the statistical tests were two-tailed. The results can also be replicated with the scripts openly available on: https://github.com/EtienneTho/musical-timbre-studies

The corresponding author can be contacted for further details in order to replicate the results or to use the scripts.

Author

Competing Interests

The authors declare no competing interests.