Justin Mathew
email: justin.mathew@lri.fr

Stéphane Huot
email: stephane.huot@lri.fr

Alan Blum
email: al.blum@ens-louis-lumiere.fr

A Morphological Analysis of Audio Objects and their Control Methods for 3D Audio

Keywords: Object-Based 3D Mixing, Design Space, Control Methods

Recent technological improvements in audio reproduction systems increased the possibilities to spatialize sources in a listening environment. The spatialization of reproduced audio is highly dependent on the recording technique, the rendering method, and the loudspeaker configuration. While object-based audio production reduces this dependency on loudspeaker configurations, related authoring tools are still difficult to interact with. In this paper, we investigate the issues of spatialization techniques for object-based audio production and introduce the Spatial Audio Design Spaces framework (SpADS), which describes the spatial manipulation of object-based audio. Based on interviews with professional sound engineers, our morphological analysis clarifies the relationships between recording and rendering techniques that define audio objects for 3D speaker configurations, allowing the analysis and the design of advanced object-based controllers as well.

INTRODUCTION

Multi-dimensional audio recording and rendering techniques make it possible to capture sound scenes and to position sound sources in a realistic manner in 3D speakers configurations. However, content creators need to account for the diversity of speaker configuration methods in order to make the listener experience consistent across different audio spatialization formats. Technically, in current multi-channel audio production systems, achieving this spatial consistency requires unique audio mixes and output files for each speaker configuration. As an alternative, many higher-level objectbased file formats have been introduced to reduce the complexity of creating several consistent audio mixes across different speaker systems [START_REF] Claypool | Auro 11.1 versus object-based sound in 3d[END_REF][START_REF] Laboratories | Authoring for Dolby Atmos Cinema Sound Manual[END_REF][START_REF] Geier | Object-based audio reproduction and the audio scene description format[END_REF]. Full support for object-based mixing is however still limited in current 3D audio authoring systems, thus reducing the possibilities of manipulating audio objects in 3D spaces and making complex 3D auditory designs difficult to achieve.

The goal of our study is to identify the limiting factors of current object-based 3D audio authoring techniques from a user's point of view. In fact, while many types of audio objects are well specified at the technical level [START_REF] Laboratories | Authoring for Dolby Atmos Cinema Sound Manual[END_REF][START_REF] Iosono | Spatial audio workstation 2 operation manual[END_REF], they are vaguely defined at the user level. As a result, low-level channel-based production systems are still the norm. We believe that by better understanding J. Mathew, S. Huot, A. Blum. A Morphological Analysis of Audio Objects and their Control Methods for 3D Audio. In NIME'14 : Proceedings of the International Conference on New Interfaces for Musical Expression, May 2014. Authors Version object-based production outside of the constraining channel-based approach, we can clarify "what 3D audio objects are" at a higherlevel, for users, and alleviate current limitations with new methods for designing appropriate controllers.

We first conducted informal interviews of sound engineers in order to get insights into the way they interact with audio objects, as well as the issues they encounter. We found that they interact with audio objects along three main stages of 3D audio production: Recording/Rendering, Mixing, and Monitoring.

1. The recording/rendering aspect focuses on how they capture audio content through different microphone techniques and how they are rendered to a speaker configuration, mostly relying on the channel-based approach; 2. The mixing aspect describes how they manipulate and control the recorded content in a 3D space; 3. Finally, monitoring is the method used to listen and analyze the mix. From the relationships between these steps and the corresponding interaction methods, we define the Spatial Audio Design Spaces (SpADS) conceptual framework that includes a morphological definition of audio objects as well as two design spaces for analyzing both the audio objects and their control methods. Beyond its descriptive power, we expect SpADS to help the design of new 3D object-based mixing methods that better suit the needs of professional sound engineers.

In the next section, we review related work before we discuss the findings from our interviews of sound engineers. We describe our morphological analysis of current 3D audio techniques and define audio objects. We then introduce the SpADS framework made of two design spaces: Audio Object Design Space and Audio Object Controller Design Space. Finally, we discuss the use of SpADS for analyzing an existing system before to conclude.

RELATED WORK

Object-based audio mixing came into focus with the growth of multi-dimensional diffusion systems. Very few commercial systems integrate support for this approach, which is still mostly built on top of traditional channel-based solutions (e.g. Auro 11.1 1 and Dolby Atmos 2 ). As for research, audio objects were mainly investigated from a descriptive point of view (i.e. file format) [START_REF] Breebaart | Spatial Audio Object Coding (SAOC) -the upcoming MPEG Standard on Parametric Object Based Audio Coding[END_REF][START_REF] Laboratories | Authoring for Dolby Atmos Cinema Sound Manual[END_REF][START_REF] Geier | Object-based audio reproduction and the audio scene description format[END_REF][START_REF] Peters | Towards a spatial sound description interchange format (SPATDIF)[END_REF] rather than on how to interact with them. There are however some notable exceptions.

New Mixing Interfaces

Geier et al. proposed the SoundScape Renderer authoring method [START_REF] Geier | The SoundScape Renderer: A Unified Spatial Audio Reproduction Framework for Arbitrary Rendering Methods[END_REF] as a modular rendering system for localized sound sources, with its associated file format: the Audio Scene Description Format [START_REF] Geier | Object-based audio reproduction and the audio scene description format[END_REF]. While they compared the capabilities of their format to others, they did not report on any study of the interface and interaction techniques. Jang et al. [START_REF] Jang | Object-based 3d audio scene representation[END_REF] introduced a method to interact with multiple types of audio objects through geometrical features to represent sound sources (i.e. point, line, plane, or volume). They also proposed several examples of audio-visual applications, but did not conduct any evaluation of these techniques.

Alternative Input Methods

Alternative input methods such as multi-touch or haptic feedback have also been investigated. Carrascal and Jordà introduced a multitouch mixing interface [START_REF] Carrascal | Multitouch interface for audio mixing[END_REF] that they compared with a standard mixing console to achieve a predefined mixing task. Their results showed that non-expert participants completed the tasks faster with their design, suggesting that it eases the accessibility of objectbased mixing to casual users. However, there is no evidence that it would be beneficial to expert users in a professional working environment. Melchior et al. compared the use of a haptic feedback mixing technique against a standard mouse in a qualitative study with experts [START_REF] Melchior | On the use of a haptic feedback device for sound source control in spatial audio systems[END_REF]. The tasks consisted of controlling the 3D position, translation and trajectory of a sound source. While the haptic device was preferred by the participants, the mouse was assessed as being better for predictability and manageability. Gelineck et al. also investigated the use of smart tangible objects as controllers on a 2D surface [START_REF] Gelineck | Towards an interface for music mixing based on smart tangibles and multitouch[END_REF]. The results of a qualitative study showed that tangibles were preferred to typical controllers, but the participants felt the system did not scale in terms of functionality.

Overall, several new object-based mixing interfaces were investigated but focused on very specific and low-level mixing tasks, without a thorough analysis of 3D audio production as a whole. A notable exception is the study conducted by Peters et al., which consisted in an on-line questionnaire about the use of tools for sound spatialization [START_REF] Peters | Current technologies and compositional practices for spatialization: A qualitative and quantitative analysis[END_REF]. They identified issues in existing mixing tools which, as we will see in more detail later, are in-line with our analyses and also ground our study.

HCI Methodologies

Our work also relates to the adaptation of Human-Computer Interaction methodologies for the design and evaluation of new musical instruments [START_REF] Kiefer | HCI Methodology For Evaluating Musical Controllers : A Case Study[END_REF][START_REF] Stowell | Evaluation of live human-computer music-making: Quantitative and qualitative approaches[END_REF]. Our approach was inspired by Wanderley and Orio's use of HCI tools for evaluating input devices for musical expression [START_REF] Wanderley | Evaluation of input devices for musical expression: Borrowing tools from hci[END_REF]. They analyzed common contexts found in scenarios of interactive computer music to investigate the use of input devices from a higher level of user interaction. While we are not focusing on the formal evaluation of 3D audio production systems, such methodologies can also help to better characterize these systems and to inform their design. In particular, we adopted Card et. al.'s morphological analysis approach [START_REF] Card | A morphological analysis of the design space of input devices[END_REF] in order to better define audio objects and characterize the related control methods within our SpADS framework. Beyond its descriptive power, our objective is also to provide designers with a tool with generative power -i.e. "the ability to help designers create new designs" [START_REF] Beaudouin-Lafon | Designing interaction, not interfaces[END_REF] -for the exploration of new 3D object-based mixing methods.

INTERVIEWS WITH PROFESSIONAL SOUND ENGINEERS

Previous studies highlighted general issues with tools for spatialization, and especially the use of low-level channel-based methods to manipulate the higher-level concept of audio object. As reported in Peters et al.'s study [START_REF] Peters | Current technologies and compositional practices for spatialization: A qualitative and quantitative analysis[END_REF], the bus architecture in Digital Audio Workstations is limiting and requires "[to develop] input devices that are tailored to the specific needs of controlling spatialization" [START_REF] Peters | Current technologies and compositional practices for spatialization: A qualitative and quantitative analysis[END_REF]. Similar concerns were raised at the FISM 2013 conference where invited speakers from a round table discussion about 3D audio in cinema and broadcasting highlighted the need to capture 3D content in a correct manner, but also that there is no convenient data structure or tool for such content [START_REF] Nicol | Round table discussion: 3D sound and its application[END_REF].

These general observations provided us with directions into the areas of current production methods to focus on in our study. To gather more details about the actual use of audio objects, we conducted informal interviews with two experienced sound engineers, E1 and E2, at their place of work. E1 works at a concert venue and has more than 10 years experience in 3D audio and acoustics. We also observed one of his students working on a project. E2 works in a radio station and has been working in spatial audio for over 5 years, but started working specifically on 3D audio within the past year. We asked them what type of projects they were currently working on and what tools they use. Our goal was also to observe them in situ and to assess what parts of channel-based systems are difficult to use when working with 3D audio techniques.

The key points extracted from these discussions highlighted details that were missing in previous work to ground our analysis of object-based tools in channel-based systems.

E1 introduced us to his work in a concert venue that broadcasts live shows with multi-channel and binaural playback. In their current projects, they have been experimenting with several types of High Order Ambisonic (HOA) microphones and comparing their use while mixing for multiple speaker configurations. They use custom-built plug-ins within a traditional DAW to properly decode the recordings for comparison. E1 presented this work with a student's project where the recordings were decoded and mixed into different buses to be played on four different loudspeaker configurations. Individual mixes are done in software with a standard mouse and keyboard setup, and a MIDI controller to switch between speaker configurations for comparison.

E2 discussed the same type of mixing setups for multiple outputs when he works on broadcasting shows in multi-channel and binaural playback. He uses a traditional DAW with specific plugins and a standard channel mixer. This engineer only focused on the mixing process, but said that he works closely with recording engineers who provide him with detailed notes on how the audio was recorded. One interesting observation he made is that there is "no relationship of audio objects with output file-formats", highlighting the inconsistent mapping between the recording methods and output file-formats that are specific to speaker configurations.

From these answers and previous work, summarized in Figure 1, we identified common issues in 3D audio production: a need to understand how the audio was recorded and will be rendered, the lack of mixing controllers that are tailored to specific microphone techniques, and a complicated bussing system to monitor. These issues and needs are directly related to well-identified stages of audio production: Recording/Rendering, Mixing, and Monitoring. (green for Mixing, red for Recording/Rendering, blue for Monitoring)

Rendering/Recording

Understanding how the audio was recorded, and thus understanding "what is the audio object", greatly influences how it will be mixed. However, characterizations of audio objects are too vague because of their numerous but limited definitions, as well as the variety of recordings and rendering methods. This was especially obvious in E1's project where different types of HOA recordings are used in the channel-based system. Each recording needs to be considered individually and independently in the bussing architecture to be rendered correctly. Having clear details on the recorded objects was also pointed out by E2 as a crucial part of the process.

Mixing

Mixing is done through plug-ins in DAWs that are typically controlled by a mouse (and sometimes standard faders and knobs). In this channel-based approach, each mix is specific to a loudspeaker configuration and the user has to restart the mixing process each time he has to consider another type of output. We observed this constraint in the need to decode the HOA recordings several times for each speaker configuration in E1's student's project.

Monitoring

Mixing for different speaker configurations is a common concern for both E1 and E2, and this obviously implies that monitoring the mix for each speaker configuration is important. For each speaker configuration, outputs of the channel-based system must be accurately mapped to the speakers and need to be re-mapped when changing among systems. Consistent knowledge of each speaker and its relative position in each configuration is needed for the engineers to correctly create bussing paths and to switch among them when monitoring.

These interviews revealed some previously unidentified issues in 3D audio manipulation techniques within channel-based systems that cover the three stages of audio production. Analyzing their relationships from an object-based point of view shows how the Recording/Rendering stage dictates the spatial capabilities of an audio object, that are in turn controlled through the Mixing stage. The Monitoring stage provides information about the speaker locations that influences the spatial capabilities of audio objects when the speaker configuration changes. Figure 2 summarizes how audio objects are related to each stage, as well as their relationships. Additionally, the variety of techniques in the Recording/Rendering stage leads to question what audio objects are, making it difficult to assess which control methods are appropriate. In the next section, we address this issue of clearly defining audio objects by conducting a morphological analysis of current production tools for 3D audio in the Recording/Rendering and Mixing stages from an object-based point of view.

A MORPHOLOGICAL ANALYSIS OF AUDIO OBJECTS

As we already pointed out, spatial audio recordings highly depend on the recording techniques that were used and will thus be perceived differently by the listener. Sound engineers must be aware of that fact when they spatially mix these recordings in order to have a consistent perception of their spatial properties over different rendering systems. Audio objects could help to bridge this gap between recording and rendering methods by providing a higherlevel and consistent abstraction, which the engineer would manipulate independently of the technologies that are used upstream and downstream (see Figure 2). It would require however a better definition of these audio objects and of their possible control methods due to the variety of current techniques. To this end, we conducted a morphological analysis of the different tools in the Recording/Rendering stage and how they are controlled in the Mixing stage. By analyzing the functionalities of these tools, our goal is to define the relationships between audio data and rendering methods, thus defining audio objects, their corresponding manipulable properties and their control methods.

Audio-Objects

Recording/ Rendering Mixing

Decoding & Rendering Monitoring

Defining Audio Objects

To connect recording techniques with rendering methods, Geier et al. distinguished between Data-Based and Model-Based audio objects [START_REF] Geier | Object-based audio reproduction and the audio scene description format[END_REF]. Data-Based rendered objects are audio captured with a microphone technique that encodes spatial information into the audio data. In model-based rendering, the recording technique is modeled as virtual sound sources within the scene. This higherlevel representation is unlike many current systems whose audio objects' properties are closer to the technical aspects of the system.

Data-Based

One method to capture an acoustic space is to place microphones in the same configuration as the speaker playback configuration, resulting in a one-to-one mapping between microphones and speakers that we call Direct Rendering. This method is commonly used with microphone arrays to recreate the sound field that is traveling based on Huygens' Principle [START_REF] Baalman | WONDER -a software interface for the application of Wave Field Synthesis in electronic music and interactive sound installations[END_REF]. However, it is inflexible to other speaker configurations and requires many microphones to achieve high spatial resolution. A typical example is the Holophone H3-D microphone, which captures a 2D space that is directly rendered to a 5.1 Surround Sound speaker format [START_REF] Systems | Holophone H3-D Surround Sound Microphone User Guide[END_REF]. Conversely, the Spatially Encoded method uses the directivity and orientation of multiple microphone capsules to encode a space within the audio data. Decoding audio data with different methods allows the captured space to be rendered on any configuration. The Double M/S microphone for instance is a spatially encoded technique that can be decoded to Surround Sound or B-format for rendering on any speaker configuration. This technique enables the control of the characteristics of the microphone's polar patterns, giving users the ability to rotate the captured sound scene [START_REF] Wittek | Double m/s -a surround recording technique put to test[END_REF]. Another common spatially encoded technique is 1 st order Ambisonic recording, which captures a 3D sound scene that is decoded to B-format for rendering on any speaker configuration [START_REF] Geier | Object-based audio reproduction and the audio scene description format[END_REF]. The Ambisonic Studio plug-in allows manipulation of the spatial parameters of Ambisonic type recordings and renderings, such as orientation and directivity [START_REF] Coureville | Ambisonic Studio B2X Plug-in Suite for Mac OSX[END_REF]. From a user's point of view, these types of recordings need to be treated collectively as one unit, to ensure spatial accuracy of the captured scene. Any type of manipulation must be done with knowledge of how it will affect the scene during playback.

Model-Based

Model-Based rendering methods include VBAP, WFS, and Ambisonic Panning [START_REF] Baalman | WONDER -a software interface for the application of Wave Field Synthesis in electronic music and interactive sound installations[END_REF][START_REF] Geier | Object-based audio reproduction and the audio scene description format[END_REF][START_REF] Geier | The SoundScape Renderer: A Unified Spatial Audio Reproduction Framework for Arbitrary Rendering Methods[END_REF]. Typical recordings with these techniques are mono, as in in the Dolby Atmos Authoring system [START_REF] Laboratories | Authoring for Dolby Atmos Cinema Sound Manual[END_REF]. Iosono's Spatial Audio WorkStation 2 plug-in takes a different approach with two basic types of audio objects: Channel Objects and Event Objects. Channel Objects are "sources of audio that may contain one or more sounds mixed together [...] coming from one location that may be moved around", and Event Objects " are singular sounds [...] with specific start and end times in certain location" [START_REF] Iosono | Spatial audio workstation 2 operation manual[END_REF]. From an objective point of view, the audio object in these tools is a mono audio signal and spatial manipulations are only 3D panning capabilities. This is limiting when considering the palette of possible spatial properties. An exception is the IRCAM Spatialisateur [START_REF] Jot | Spatialisateur Ircam/Espaces Nouveaux User Manual[END_REF], which considers audio objects as localized sound sources as well, but defines their spatial parameters from their interaction within a room, thus allowing more spatial manipulations such as orientation. However, Model-Based rendered audio does not have to be restricted to just mono recordings. Linear or geometrical microphone arrays could be rendered this way, each microphone being a point in a line or shape, creating the same contexts that Jang et al. referred to [START_REF] Jang | Object-based 3d audio scene representation[END_REF]. While these audio objects are more flexible in terms of the type of rendering to use, the spatial capabilities of most tools are still restricted to 3D positioning only.

Spatial Parameters

Overall, model and data-based software only support subsets of all the possible spatial parameters, which we define as Translation, Orientation, and Area/Volume. Translation (1D, 2D, and 3D) refers to how many dimensions an audio object can move in. The Orientation parameters (Pitch, Yaw, and Roll) refer to the direction of orientation of sound sources or sound scenes, depending on the type of the object. Area/Volume parameters describe the shapes an audio object can have, with Width, Height, and Depth. These parameters are however perceived and thus manipulated differently by sound engineers, depending on whether they use a databased or model-based approach in the Monitoring stage. Common data-based audio objects immerse listeners in the center of a sound scene, whereas typical model-based audio objects are localized as spatial sound sources [START_REF] Jot | A comparative study of 3-d audio encoding and rendering techniques[END_REF]. This difference in perception creates different contexts for manipulating the spatial parameters, which are thus dependent on the type of audio object.

Spatial Audio Design Spaces (SpADS)

From this analysis of the nature of audio objects and their spatial parameters in the Recording/Rendering and Mixing stages, we define the dimensions of two design spaces for SpADS: Design Space of Audio Objects (SpADS-A) and Design Space of Control Methods (SpADS-C).

SpADS-A (Audio Objects)

SpADS-A defines and classifies audio objects as audio data along two dimensions: their Rendering method and their Spatial Parameters. Various audio objects are then made of different combinations of audio data and rendering methods. As presented in Figure 3, SpADS-A helps gain a better understanding of how audio objects of similar or different natures compare to each other. SpADS-A allows quick identification of how recording techniques can be rendered, and the corresponding 3D mixing capabilities with spatial parameters. For example, a mono audio signal is only used in Model-Based renderers, which can be translated up to three dimensions and oriented in three directions. The Stereo A/B recording is considered to be two point sources that are linked, and differs slightly from mono recording. Stereo recording is a two point source that can be translated together, oriented from an anchor point, and increased/decreased in width between the two points. Increasing the number of point sources creates microphone arrays, adding more capabilities in the spatial manipulations. It seems that only model-based audio objects provide this ability, but some encoded data-based renderers can provide the same spatial capabilities. For instance, the M/S recording technique is a two-channel microphone technique that is decoded into stereo, allowing it to have the same parameters as the Stereo A/B method. However, the need to decode the M/S technique categorizes it as a data-based object rather than a model-based object [START_REF] Wittek | Double m/s -a surround recording technique put to test[END_REF]. Double M/S is an improvement over the M/S technique that captures audio with 2D spatial qualities [START_REF] Wittek | Double m/s -a surround recording technique put to test[END_REF], and can only be translated in one dimension, oriented, and area manipulated. This increase of dimensionality provides different spatial capabilities from the perspective of the listener, which is situated at the center of the recreated area. Ambisonic recordings capture a 3D space and locate the listener similarly but can only be oriented and not translated.

Overall, SpADS-A shows how audio objects can be rendered and manipulated, highlighting similarities and differences among them. Now, we can use these spatial parameters to explore how different input devices can be used to manipulate these objects during the Mixing stage.

SpADS-C (Controllers)

Our study of controllers for audio objects is inspired by the use of traditional mixing consoles in the three stages of production. In this context, the audio data used in the Recording/Rendering stage is a mono signal, and the typical Monitoring speaker configuration is stereo. The resulting audio object is rendered with left/right equal power panning that positions it on the line between the speakers. A simple one-dimensional potentiometer is normally used as positioning input device, and its physical position gives visual feedback of the audio object's location. Additionally, multiple potentiometers can be controlled at once with two hands. Our SpADS-C design space extends this analysis to 3D speaker configurations and classifies input devices according to their capabilities for controlling the spatial parameters of audio objects (see Figure 4). The Input and Visual Feedback dimensions respectively describe how the user controls the device and the produced visual feedback. Input can be Single or Multiple, whether the user can control one or several controllers at once, and have defined degrees of freedom (i.e. how many parameters each input can control). The Visual Feedback3 can be provided by the Physical position of the controller or a Virtual representation within the software GUI, which can have several dimensions (e.g. 2D vs 3D graphics). As presented in Figure 4, SpADS-C helps to compare how some input devices can be used to control spatial parameters. For example, a Haptic Device with 6 degrees of freedom can provide integral control of multiple parameters, but can only control one audio object at a time. However, a multi-touch tablet can provide multiple inputs, but has only two degrees of freedom for each input. Also, virtual feedback can be coupled with different kinds of controllers, but only some of them provide physical feedback: In a typical mixing console for instance, faders and knobs provide physical visualizations augmented with level meters that can be considered as virtual feedback. For input devices without physical feedback, such as a tablet, virtual visualization is often mandatory. SpADS-C can categorize and compare different types of controllers but does not help to assess which are better suited for the manipulation of specific audio objects and parameters. Such assessment will require further investigation and evaluation. However, this high-level design space of controllers already provides valuable insights into the matching between the spatial capabilities of audio objects and the functionalities of current systems.

DISCUSSION

The SpADS framework is a tool to explore the spatial capabilities of audio objects and to compare and match controllers to their spatial parameters. As an example, we used SpADS to analyze the Neve DFC console for the authoring of Dolby Atmos content [START_REF] Critchley | Working with the Dolby Atmos format on the AMS-NEVE DFC console[END_REF]. We considered only the use of mono and surround sound recording techniques and subsequent rendered audio objects that we placed in SpADS-A and SpADS-C (see Figures 5 &6).

We analyzed three examples of recording and rendering techniques that can create Surround Sound objects: Double M/S, Holophone, and mono audio recordings. We assumed that the mono audio recordings are rendered with VBAP, but another rendering technique may be used. For the Holophone recording, the audio is mapped out directly to the Surround Sound object through channel configuration, but the Dolby Surround Sound [START_REF] Laboratories | Authoring for Dolby Atmos Cinema Sound Manual[END_REF] or Schoeps' Double M/S plugin tools [START_REF] Wittek | Double m/s -a surround recording technique put to test[END_REF] could be used as well. Both plug-ins can be controlled with a mouse, but the Dolby Surround Sound plugin can position mono audio objects with the joystick in the Neve DFC Console (see Figure 6). Feedback is always virtual within the plug-in, but the joystick also provides an additional virtual 2D grid. It also gives physical visual feedback, but only during interaction, and it has to be re-aligned when changing between controlled audio objects [START_REF] Critchley | Working with the Dolby Atmos format on the AMS-NEVE DFC console[END_REF]. Finally, the Dolby Atmos plug-in must be used to manipulate 3D localized mono audio objects. They can be controlled with the mouse or the two joysticks on the console, providing the user with both physical and virtual feedback [START_REF] Critchley | Working with the Dolby Atmos format on the AMS-NEVE DFC console[END_REF].

The SpADS-A analysis highlights that some spatial properties cannot be controlled with this production system (boxed out in red in Figure 5). Once transposed to SpADS-C (boxed out in red in Figure 6), it gives the opportunity to explore and increase the functionality of the system in terms of spatial mixing capabilities. For instance, a mouse with simple GUI controls could increase the missing spatial capabilities, or a tablet could be used to control the Area/Volume parameters for the Surround Sound object. These examples illustrate how SpADS can help to describe existing systems for spatial mixing from an audio object based point of view, and potentially improve them. Beyond its descriptive and generative powers, we believe that SpADS can be extended to support the evaluation and comparison of control methods and input devices for the manipulation of audio objects [START_REF] Wanderley | Evaluation of input devices for musical expression: Borrowing tools from hci[END_REF][START_REF] Card | A morphological analysis of the design space of input devices[END_REF]. At a higher level, SpADS addresses the issue pointed out by E2 in our preliminary study: there was "no relationship of audio objects with output file-formats". SpADS' object-based approach clearly defines the necessary relationship between the recording and rendering techniques that define audio objects in a user-centered approach. Using this relationship, designers can explore control methods for the spatial parameters of an audio object or a combination of audio objects, and design new authoring systems that can render and mix a variety of audio objects together for 3D audio.

CONCLUSION

The development of multi-channel mixing has provided sound engineers with the ability to spatially manipulate audio in 3D, but from our interviews of content creators, we highlighted several issues of current object-based production systems. The vague definition of audio objects, the lack of well-suited controllers, and the need to listen to multiple mixes led us to define the three stages of 3D audio production to focus on: Recording/Rendering, Mixing, and Monitoring. Our morphological analysis of audio objects allowed us to better define them as well as their spatial capabilities at a higher-level. The resulting Spatial Audio Design Spaces (SpADS) conceptual framework introduces two design spaces for analyzing the spatial parameters of audio objects (SpADS-A) and the spatial parameters that an input device can control (SpADS-C).

This initial proposal of SpADS focuses on the Recording / Rendering and Mixing stages in audio production, and does not yet include Monitoring. Extending SpADS to account for the Monitoring stage would enable the analysis of complete audio production systems. We also plan to conduct experiments to better analyze the possible matchings between the spatial properties of audio objects and several types of controllers/input devices. This will complement the descriptive power of SpADS with comparative and generative capabilities [START_REF] Beaudouin-Lafon | Designing interaction, not interfaces[END_REF], and ultimately make it possible to explore and evaluate new audio production software focused on the flexibility of fully object-based Recording/Rendering, Mixing, and Monitoring.

•

  Previous study (Peter's et. al) • Multi-speaker tools for current software are too inflexible • Bussing architecture is limiting • "Input devices are tailored to specific needs of controlling spatialization" • FISM Conference 2013 • Need to capture audio in a correct manner • No tool or representative data structure of the captured audio • E1 • Experimenting with different microphone techniques (High Order Ambisonics) • Mixes the audio into different speaker formats using current DAW software and compares the mixes • E2 • Not able to interact with audio objects within the whole of a DAW system • "No relationship of audio objects with the output file-formats" • There's a need to know how the audio was recorded during mixing process • E1's Student • Comparison of Ambisonic Recording techniques for the same audio content for different speaker configuration • Need different control over all speaker levels for different speaker configuration

Figure 1 :

 1 Figure 1: The major issues of 3D object-based production in channel based systems, from related work and our interviews.

Figure 2 :

 2 Figure 2: Block diagram showing the relationships between three stages of object-based production (Recording/Rendering, Mixing, Monitoring) .

Figure 3 :

 3 Figure 3: An example of an analysis comparing different types of audio objects using SpADS-A.

Figure 4 :

 4 Figure 4: An example of SpADS-C with some input devices for controlling spatial parameters with different visual feedback.

Figure 5 :

 5 Figure 5: SpADS-A analysis of possible audio objects that can be used in the Dolby Atmos with the Neve DFC console.

Figure 6 :

 6 Figure 6: SpADS-C analysis for Dolby Atmos with the Neve DFC console which highlights missing controls for audio objects parameters.

http://www.barco.com/en/auro11-1

http://www.dolby.com/us/en/professional/technology/cinema/dolby-atmos-creators.html

While promising for 3D audio object manipulation, we do not consider advanced feedback technology such as force-feedback since visual feedback is still the norm in professional audio production.

ACKNOWLEDGEMENTS

We thank the sound engineers who participated in our study for their time and fruitful discussions. Thanks to Halla Olafsdottir, Joseph Malloch and Michel Beaudouin-Lafon for helpful comments on this paper. This research is partially supported by the French Agency for Technical Research (ANRT) and DMS-Cinema.


Hedi Tabia

Hamid Laga

David Picard

Philippe-Henri Gosselin

Covariance Descriptors for 3D Shape Matching and Retrieval

Several descriptors have been proposed in the past for 3D shape analysis, yet none of them achieves best performance on all shape classes. In this paper we propose a novel method for 3D shape analysis using the covariance matrices of the descriptors rather than the descriptors themselves. Covariance matrices enable efficient fusion of different types of features and modalities. They capture, using the same representation, not only the geometric and the spatial properties of a shape region but also the correlation of these properties within the region. Covariance matrices, however, lie on the manifold of Symmetric Positive Definite (SPD) tensors, a special type of Riemannian manifolds, which makes comparison and clustering of such matrices challenging. In this paper we study covariance matrices in their native space and make use of geodesic distances on the manifold as a dissimilarity measure. We demonstrate the performance of this metric on 3D face matching and recognition tasks. We then generalize the Bag of Features paradigm, originally designed in Euclidean spaces, to the Riemannian manifold of SPD matrices. We propose a new clustering procedure that takes into account the geometry of the Riemannian manifold. We evaluate the performance of the proposed Bag of Covariance Matrices framework on 3D shape matching and retrieval applications and demonstrate its superiority compared to descriptor-based techniques.

Introduction

Quantifying shape similarity between 3D objects, hereinafter refereed to as shape analysis, is central to many computer vision and pattern recognition tasks. The common factor in existing techniques is the use of shape signatures that capture the main properties of 3D objects. While early methods focused on global descriptors that are invariant under rigid transformations [19], the use of local features has gained a significant momentum in the past few years [5, [START_REF] Toldo | Visual vocabulary signature for 3d object retrieval and partial matching[END_REF][START_REF] Tabia | Local visual patch for 3D shape retrieval[END_REF].

Due to their good performance in many image analy-sis settings, many authors attempted to adapt popular local image descriptors to 3D shapes. Examples include the 3D SIFT [START_REF] Zaharescu | Surface feature detection and description with applications to mesh matching[END_REF] and the 3D Shape Context [START_REF] Frome | Recognizing objects in range data using regional point descriptors[END_REF][START_REF] Kokkinos | Intrinsic shape context descriptors for deformable shapes[END_REF]. One of the main strengths of local features is their flexibility in terms of type of analysis that can be performed with. For instance they can be used as local descriptors for shape matching and registration but can be also aggregated over the entire shape, using the Bag of geometric Words (BoW), to form global descriptors for recognition, classification and retrieval [5]. These techniques, however, face two main concerns. First, local features do not capture the spatial properties or structural relations between shape elements. Second, often 3D shape collections exhibit large inter-class and intra-class variability that cannot be captured with a single feature type. This triggers the need for combining different modalities or feature types. However, different shape features have different dimension and scale, which makes their aggregation difficult without normalizing or using blending weights.

To overcome some of these shortcomings, Behmo et al. [2] proposed to compute a graph based representation that captures the spatial relations between features. Bronstein et al. [5] extended this approach and proposed spatially-sensitive bags of features by considering pairs of geometric words. Laga et al. [START_REF] Laga | Geometry and context for semantic correspondences and functionality recognition in manmade 3d shapes[END_REF] combined both geometric and structural features to capture the semantics of 3D shapes. While these representations take into account the spatial relations between features of the same type, they are limited when it comes to handling multimodal features.

Recently, the image analysis community showed a growing interest in characterizing image patches with the covariance matrix of local descriptors rather than the descriptors themselves. They have been used for object detection and tracking [29,[START_REF] Tuzel | Region covariance: a fast descriptor for detection and classification[END_REF][START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF][START_REF] Li | Visual tracking via incremental log-euclidean riemannian subspace learning[END_REF], texture classification [START_REF] Tuzel | Region covariance: a fast descriptor for detection and classification[END_REF], action recognition and face recognition [13]. The use of covariance matrices has several advantages. First, they provide a natural way for fusing multi-modal features without normalizing. Second, covariance matrices extracted from different regions have the same size. This enables comparing any regions without being restricted to a constant window size or specific feature dimension.

Covariance matrices, however, lie on the manifold of Symmetric Positive Definite (SPD) tensors (Sym + d ), a special type of Riemannian manifolds. This makes the development of classification and clustering methods on the space of SPD matrices quite challenging. Many authors attempted to generalize computational methods, originally designed for Euclidean spaces, to Riemannina manifolds. Jayasumana et al. [START_REF] Jayasumana | Kernel methods on the riemannian manifold of symmetric positive definite matrices[END_REF], for example, generalized the powerful kernel methods to manifold-valued data, such as covariance matrices, and has demonstrated their usage in pedestrian detection, visual object categorization, texture recognition and image segmentation. Similarly, Faraki et al. [START_REF] Faraki | Fisher tensors for classifying human epithelial cells[END_REF] demonstrated the usage of bag of covariance matrices in the classification of human epithelial cells in 2D images.

In this paper, we generalize the ideas presented in [29,[START_REF] Tuzel | Region covariance: a fast descriptor for detection and classification[END_REF] to 3D shape analysis problems, particularly to 3D shape matching and recognition and to 3D shape retrieval. Our idea is to represent a 3D model with a set of n landmarks sampled (uniformly or randomly) on its surface. Each landmark has a region of influence, which we characterize with the covariance of features of different types. Each type of features captures some properties of the local geometry. Covariance matrices are not elements of the Euclidean space, they are elements of the Lie group, which has a Riemannian structure. Therefore, matching with covariance matrices requires the computation of geodesic distances on the manifold using a proper metric. We show how such dissimilarity measure can be computed in an efficient way and demonstrate its performance in 3D face matching and recognition. We also propose an extension of the BoW approach to nonlinear Riemannian manifolds of covariance matrices. Our dictionary construction procedure uses geodesic distances and thus it captures effectively the structure of the manifold.

We demonstrate the performance of the proposed framework in several 3D shape analysis applications. We show that covariance descriptors perform better than individual features on 3D correspondence and registration tasks and on 3D face matching and recognition. We also evaluate the retrieval and classification performance of the Bag of Covariance matrices on 3D shape retrieval and recognition using standard benchmarks.

Our contributions are three-fold. First, we propose covariance matrices as new descriptors for 3D shape analysis. While similar descriptors have been proposed for object tracking and texture analysis in 2D, it is the first time that covariance based analysis is explored for 3D shape analysis. Second, the framework enables the fusion of multiple and heterogeneous features without the need for normalization. Finally, using a Riemannian metric on the manifold of covariance matrices, we introduce the concept of Bag of Covariance matrices for 3D shape classification and retrieval.

The rest of the paper is organized as follows. We intro-duce covariance matrices for 3D shape analysis in Section 2 and review the basic concepts of Riemannian geometry in Section 3. Section 4 presents how covariance matrices can be used in comparing 3D models. Section 5 extends the Bag of Features paradigm to the non-linear space of covariance matrices of 3D shapes. We evaluate our approach on various datasets in Section 6 and conclude in Section 7.

Covariance matrices on 3D shapes

We represent a 3D shape as a set of overlapping patches {P i , i = 1 . . . m}, each patch P i is extracted around a representative point p i = (x i , y i , z i ) t . For each point p j = (x j , y j , z j ) t , j = 1 . . . , n i , in the patch P i , we compute a feature vector f j , of dimension d, that encodes the local geometric and spatial properties of the patch. The following properties can be used:

• F1: The location of the point p j with respect to the patch center

p c = 1 ni ni k=1 p k .
It is given by p j -p c . • F2: The distance of the point p j to p i .

• F3: The volume of the parallelepiped formed by the coordinates of the point p j .

Note that our framework is generic and thus other features can also be added to the representation. Examples include the norm of the gradient, the two principal curvatures k 1 and k 2 , the mean and Gaussian curvatures, the shape index [START_REF] Koenderink | Solid shape[END_REF], the curvedness at p i , which is defined as (k 2 1 + k 2 2 )/2, the shape diameter function [START_REF] Gal | Pose-oblivious shape signature[END_REF][START_REF] Shapira | Contextual part analogies in 3d objects[END_REF], and the scale-invariant heat kernel signature [START_REF] Bronstein | Scale-invariant heat kernel signatures for non-rigid shape recognition[END_REF].

Let {f j } j=1..n be the d-dimensional feature vectors computed over all points inside P i . We represent a given 3D patch P i with a d × d covariance matrix X i :

X i = 1 n n j=1 (f j -µ) (f j -µ) T ( 1
)
where µ is the mean of the feature vectors computed in the patch P i . The covariance matrix X i is a symmetric matrix. Its diagonal elements represent the variance of each feature and its off-diagonal elements represent their respective correlations. It has a fixed dimension (d × d) independently of the size of the patch P i .

Riemannian geometry of SPD matrices

For the sake of completeness, we present here the mathematical properties of the space of covariance matrices and the metrics that have been proposed for comparing them.

The space of covariance matrices

Let M = Sym + d be the space of all d × d symmetric positive definite matrices and thus non-singular covariance matrices. Sym + d is a non-linear Riemannian manifold, i.e. a differentiable manifold in which each tangent space T X at X has an inner product •, • X∈M that smoothly varies from point to point. The inner product induces a norm for the tangent vectors y ∈ T X such that y 2 = y, y X . The shortest curve connecting two points X and Y on the manifold is called a geodesic. The length d(X, Y ) of the geodesic between X and Y is a proper metric that measures the dissimilarity between the covariance matrices X and Y . Let y ∈ T X and X ∈ M. There exists a unique geodesic starting at X and shooting in the direction of the tangent vector y. The exponential map exp X : T X → M maps elements y on the tangent space T X to points Y on the manifold M.

The length of the geodesic connecting X to Y is given by d(X, exp X (y)) = y X .

Machine learning algorithms require the definition of metrics for comparing data points. When data lie in the Euclidean space, Euclidean distance is often used as the natural choice of metric. Covariance matrices, however, lie on a non-linear manifold and thus, efficient algorithms are required for computing geodesics and their lengths. Geodesic lengths can then be used as a metric for subsequent machine learning algorithms. In this paper, we focus on the computation of geodesic lengths. Most of the existing classifiers (e.g. the nearest neighbor classifier) only requires a notion of distance between points on the manifold M. We use the distance proposed in [START_REF] Pennec | A riemannian framework for tensor computing[END_REF] to measure the dissimilarity of two covariance matrices.

Geodesic distance between covariance matrices

The Riemannian metric of the tangent space T X at a point X is given as y, z X = trace X -1 2 yX -1 zX -1 2 . The exponential map associated to the Riemannian metric exp X (y) = X

1 2 exp X -1 2 yX -1 2 X 1
2 is a global diffeomorphism (a one-to-one, onto, and continuously differentiable mapping in both directions). Thus, its inverse is uniquely defined at every point on the manifold:

log X (Y ) = X 1 2 log X -1 2 Y X -1 2 X 1 2
. The symbols exp and log are the ordinary matrix exponential and logarithm operators, while exp X and log X are manifold-specific operators, which depend on the point X ∈ Sym + d . The tangent space of Sym + d is the space of d×d symmetric matrices and both the manifold and the tangent spaces are of dimension

m = d(d + 1)/2.
For symmetric matrices, the ordinary matrix exponential and logarithm operators can be computed in the following way. Let X = U DU T be the eigenvalue decomposition of the symmetric matrix X. The exponential series is defined as:

exp (X) = ∞ k=0 X k k! = U exp (D) U T ,
where exp (D) is the diagonal matrix of the eigenvalue exponentials. Similarly, the logarithm is given by log

(X) = ∞ k=1 -1 k-1 k (X -I) k = U log (D) U T . The exponential
operator is always defined, whereas the logarithms only exist for symmetric matrices with strictly positive eigenvalues.

The geodesic distance between two points on Sym + d is then given by:

d 2 g (X, Y ) = log X (Y ) , log X (Y ) X = trace log 2 X -1 2 Y X -1 2 (2)
An equivalent form of the affine-invariant distance metric has been given in [START_REF] Förstner | A metric for covariance matrices[END_REF] and [25] in terms of joint eigenvalues of X and Y . We will use this metric to derive algorithms for 3D shape analysis using covariance matrices as local descriptors.

3D shape matching using SPD matrices

In this section, we propose a metric for matching 3D shapes using covariance matrices as descriptors and the Riemannian metric as a measure of dissimilarity. Let us consider a point p i represented by the covariance descriptor X i on the first 3D model and a point q j represented by the covariance descriptor Y j on the second 3D model. Let c (p i , q j ) = d g (X i , Y j ) denote the cost of matching these two points. It is defined as the geodesic distance on the Riemannian manifold between the two descriptors X i and Y j .

Given the set of costs c ij between all pairs of points p i on the first 3D shape and q j on the second 3D shape, we define the total cost of matching the two 3D shapes as:

C(ϕ) = i c p i , q ϕ(i) (3)
Minimizing C(ϕ), subject to the constraint that the matching is one-to-one, gives the best permutation ϕ(i). This is an assignment problem, which can be solved using the Hungarian method. In our implementation, we use the shortest augmenting path algorithm [START_REF] Jonker | A shortest augmenting path algorithm for dense and sparse linear assignment problems[END_REF]. The input to the assignment problem is a square cost matrix with entries c ij . The result is a permutation ϕ(i) such that Equation 3 is minimized. The minimum of this cost function is then used as the dissimilarity between the two 3D models.

Bag of Words on Riemannian manifold

We extend the Bag of Words (BoW) approach [32, 24] to covariance matrices for 3D shape analysis. In the classical BoW approach, a 3D model is represented as a collection of local descriptor prototypes, called visual words, given a well chosen vocabulary, i.e. the codebook. BoW representations have been widely adopted by the 3D shape analysis community [START_REF] Toldo | Visual vocabulary signature for 3d object retrieval and partial matching[END_REF]5]. One of their advantages in 3D shape retrieval is their ability to aggregate a large number of local descriptors into a finite, low dimensional, histogram. The dissimilarity between a pair of 3D models becomes then a distance between their respective histograms, which is computationally more efficient than minimizing a cost function.

The first step of the BoW approach consists of approximating a set of training samples with a finite set of visual words, the centers, which form the codebook. The codebook is often obtained using clustering algorithms, such as k-means. A good codebook should represent the original data with minimum distortion. The distortion is usually defined as the average mean square distance of all the training points to the centers:

M SD = 1 |S| X∈S d 2
g (X, X) where the set S denotes the input data set, |S| its cardinality, and X is the center of the cluster to which X is assigned. The classification of a query model involves the assignment of the descriptors computed on the model into one of the codebook elements. This process involves the computation of distances between feature points. When these points lie in Euclidean spaces, the computation of Euclidean distances in straightforward. In what follows, we describe how to extend this concept to covariance matrices, which are elements of the non-linear Riemannian manifold Sym + d and thus require efficient algorithms for computing geodesic distances.

Center computing

Let S = {X i } i=1..N be a set of points on M = Sym + d . The intrinsic mean of points in Riemannian manifold is the point on Sym + d that minimizes the sum of squared geodesic distances:

X = arg min X∈M N i=1 d 2 g (X i , X) , (4)
where d g is the geodesic distance between two points on the manifold. The mean is the solution to the nonlinear matrix equation N i=1 log X (X i ) = 0, which can be solved using a gradient descent procedure. At each iteration, the estimate of the mean is updated as follows:

Xt+1 = exp Xt 1 N N i=1 log Xt (X i ) . (5)
Equation 5 starts by mapping the points onto the tangent space to the manifold at the current estimate of the mean (using inverse exponential map), computes an estimate of the mean in the tangent space, and maps back the estimated mean onto the manifold using exponential map. This process is iterated until convergence. The iterative use of the logarithmic and exponential maps makes this method computationally expensive. To overcome this, many solutions can be used:

• One can use the k-medoid algorithm instead of kmeans. When computing the codebook, k-means tries to find the true mean of the data. Thus, the minimization in Equation 4 is over the entire manifold. Kmedoids on the other hand constrains the center to be one of the data points and thus the search space for the best solution is significantly reduced. Using K-medoid requires the computation of the pairwise distances between all training data, which can be time consuming.

• Alternatively, one can map all the training points to the tangent space of the manifold at one point (e.g. mean point), obtaining an Euclidean representation of the manifold-valued data. As stated in [START_REF] Jayasumana | Kernel methods on the riemannian manifold of symmetric positive definite matrices[END_REF], this mapping does not globally preserve distances, resulting in a poor representation of the original data distribution.

• The third solution is to use the geodesic distance d g with the Frobenius distance when computing the centers of k-means. The idea is that the Euclidean average of covariance matrices lies in the Riemannian manifold, as stated in [4]. Indeed, any non-negative linear combination of symmetric positive definite (SPD) matrices is an SPD matrix. This implies that the linear average X of X 1 , ..., X N given by: X = 1 N (X 1 + ... + X N ) is an SPD matrix, hence belongs to the Riemannian manifold. The Frobenius distance from which the linear average came is given by:

d 2 F ro (X 1 , X 2 ) = 1≤i,j≤n (X 1 -X 2 ) ij 2 . ( 6
)
We tested those three codebook constructions and studied the repartition of the final set of centers obtained from each solution in the space of Hermitian forms, SPD matrices.

Center repartitions

After running the three earlier mentioned solutions, we evaluate the distance between each center and the identity element I n of this space. This distance is computed according to Equation 2 when the geodesic metric is used, and Equation 6 when the Frobenius metric is chosen.

Figure 1 shows the repartition of the centers after running k-means on a sample of 10 4 Hermitian forms, using three variants based on geodesic distance (presented in the previous section), and a fourth variant based on the Frobenius distance (cf. Eq. 6). In each sub figure, the origin represents the identity matrix I n and a green symbol represents a center Xi placed on a circle of radius equal to its distance from the identity. The angles, α( Xi ) = 2π

card( Xi) k card( Xk )
(where Xk 1≤k≤K are the centers), that surround the different centers are proportional to the size of their corresponding cluster.

From Figure 1-(a) and (c), one can notice that the size of different classes is almost equal. This demonstrates that the data points are equally partitioned when using the geodesic distance. On the other hand, Figure 1-(b) and (d) show that large classes contain most of the data samples. This shows that the use of Frobenius distance when assigning the points to their closest centers is not suitable for covariance matrices. It also shows that approximation using the tangent space of the mean data point voids one of the benefits of using the geodesic distance. Ensure: labeled data points and

K centers iter = 1; M SE iter = 0; M SE iter-1 = 1; while (iter ≤ maxIter) && (M SE iter = M SE iter-1 ) do M SE iter = 0 for i = 1 to N do d = M AXV AL for k = 1 to K do d k = dg( Xiter k , X i ) using eq. 2 if d > d k then d = d k k min = k end if end for Assign X i to the center Xiter kmin M SE iter + = d kmin end for for k = 1 to K do Xiter+1 k = 1 N k i (X ik ) end for iter + + end while
Another observation that can be made from Figures 1-(a), (b) and(c) is that the centers are located at different distance levels from the identity matrix. This demonstrates that the geodesic distance is more discriminant for covariance matrices than the Frobenius one as shown in Figure 1-(d). We can conclude that the geodesic distance gives better repartition of the data samples than the Frobenius distance. We can also see that the K-medoid algorithm (Figure 1-(a)) behaves better than the other solutions. However, as mentioned earlier, using the K-medoid algorithm, all pairwise distances between training data have to be computed, which is time consuming when dealing with large data samples. The third solution (see Figure 1-(c)) behaves similarly to Kmedoid solution. In this paper, we used the third solution, which is detailed in Algorithm 1 for codebook construction.

Signature computing

We describe 3D shapes using vectors of visual word frequencies. For a given 3D model, each point P i represented by its covariance descriptor is assigned to its closest center using the geodesic distance of Equation 2. The frequency of a visual word in a given shape is simply the number of times a given visual word appears in that model. For best performance, we perform a normalization step of the signature. In our experiment, we use the 0.3 power norm. At the retrieval stage, 3D shapes are ranked using as metric the cosine of the angle between the query vector and all shape vectors in the database.

Experimental results

We carried out an extensive set of experiments to evaluate and verify the effectiveness of the proposed method in matching and retrieving 3D models.

3D face matching and recognition

We analyzed the performance of the proposed covariance descriptors on 3D face recognition tasks. We used the GavabDB1 dataset, which is among the most noise-prone datasets currently available in public domains. It contains 549 three-dimensional facial surfaces of 61 different individuals (45 males and 16 females). Each individual has nine facial surfaces with two frontal views at neutral expression, two views rotated with +35 degrees around the x-axis, one looking up and another looking down, both in neutral expression, two profile views (left and right), both in neutral expression, and three frontal images with facial expressions (laugh, smile and a random expression chosen by the user). Thus, the database provides large variations with respect to pose and to facial expressions. All the individuals are Caucasian and their age varies between 18 and 40 years old.

We have first preprocessed the surfaces to remove spikes, fill in the holes and align them. We have then uniformly sampled m = 100 feature points on the 3D face scan. To ensure a uniform sampling, we first select a random set of vertices on the face and then apply few iterations of Lloyd relaxation algorithm. We then generate a set of patches {P i , i = 1 . . . m}, each patch P i has a radius r = 10% of the radius of the shape's bounding sphere, and it is identified with its corresponding seed point p i . For each patch we compute a 5 × 5 covariance matrix that encodes the features F1, F2 and F3 defined in Section 2. We then use the matching method presented in Section 4 to compare 3D faces.

In this experiment, the first frontal facial scan of each subject was used as gallery while the others were treated as probes. To objectively evaluate our method for face recognition, we present results in the form of recognition rate, see Table 1. We can see that our method achieves a recognition rate of 100% on the faces of neutral expressions. This performance is similar to the approach of Drira et al. [START_REF] Drira | Pose and Expression-Invariant 3D Face Recognition using Elastic Radial Curves[END_REF]. Our approach performs slightly lower than the approach of Drira et al. on faces with expressions (93.3% vs. 94.54%). It is interesting to note that in [START_REF] Drira | Pose and Expression-Invariant 3D Face Recognition using Elastic Radial Curves[END_REF], authors manually landmarked points on all the face scans in the dataset to ensure an efficient face registration. Table 1 also shows that the proposed method outperforms all the other methods and achieves the best overall result when the neutral and expressive faces are merged together.

3D shape retrieval

We have analyzed the performance of the proposed covariance-based descriptors on 3D shape retrieval tasks using three different databases:

• The McGill2 database [31], which contains 255 objects divided into ten classes. Each class contains one 3D shape under a variety of poses.

• The SHREC07 dataset3 for global 3D shape retrieval. It contains 400 models evenly distributed into 20 shape classes. The experiment was designed so that each model was used in turn as a query against the remaining elements of the database, for a total of 400 queries.

• The SHREC07 dataset4 for partial 3D shape retrieval. It contains a database of 400 models and a query set of 30 composite models. The dataset exhibits diverse variations including pose change, shape variability, and topological variations (note that four of the 20 classes contain non-zero genus surfaces) [START_REF] Marini | Shape retrieval contest 2007: Partial matching track[END_REF]. We randomly sample m = 600 points on the 3D mesh. We assign for each point p i a patch P i , i = 1 . . . m, on which we compute a covariance descriptor. We extract the local patch P i by considering the connected set of facets belonging to a sphere centered at p i and of radius r = 15% of the radius of the shape's bounding sphere. We then compute a 5 × 5 covariance matrix from the features F1, F2 and F3 defined in Section 2. We then use our Riemannian BoW method (Section 5) to compare 3D shapes. We use a codebook of size 250 visual words.

We have evaluated the proposed method using various performance measures, namely: Precision-Recall graphs, Nearest Neighbor (NN), the First Tier (1-Tier), The Second Tier (2-Tier), and the Discounted Cumulative Gain (DGC).

Watertight 3D shape retrieval

Table 2 summarizes the retrieval performance of the proposed method on the McGill dataset. We compare our results to various state-of-the-art methods: the Hybrid BoW [START_REF] Lavoué | Combination of bag-of-words descriptors for robust partial shape retrieval[END_REF], the PCA-based VLAT approach [START_REF] Tabia | Compact vectors of locally aggregated tensors for 3d shape retrieval[END_REF], the graphbased approach of Agathos et al. [START_REF] Agathos | Retrieval of 3d articulated objects using a graph-based representation[END_REF], and the hybrid 2D/3D approach of Papadakis et al. [START_REF] Papadakis | Perantonis. 3d object retrieval using an efficient and compact hybrid shape descriptor[END_REF]. Although the proposed method does not consider structural information of shapes, it achieves the best performance on NN and DCG. The graph-based algorithm performs slightly better than ours on 1-Tier and 2-Tier. The robustness of the Riemannian BoW method to nonrigid deformations of shapes is probably due to the local descriptors we used in building the covariance matrices.

Global shape retrieval

We used the SHREC07 dataset to evaluate the performance of our method in global shape retrieval tasks. Figure 2 summarizes and compares the precision-recall performance of our approach against three state of the art methods: the Hybrid BoW of Lavoué [START_REF] Lavoué | Combination of bag-of-words descriptors for robust partial shape retrieval[END_REF], the curve based method of Tabia et al. [START_REF] Tabia | A new 3dmatching method of nonrigid and partially similar models using curve analysis[END_REF] and the BoW method of Toldo et al. [START_REF] Toldo | Visual vocabulary signature for 3d object retrieval and partial matching[END_REF]. We can clearly see that covariance-based BoW achieves significantly better precision than previous methods for low recall values. The precision is kept over 70% when half of the relevant objects have been returned (recall equals to 0.5). Since low recall values correspond to the first objects retrieved, this result shows that our method is significantly Methods NN 1-Tier 2-Tier DCG Covariance Method 0.930 0.623 0.737 0.864 Hybrid BoW [START_REF] Lavoué | Combination of bag-of-words descriptors for robust partial shape retrieval[END_REF] 0.918 0.590 0.734 0.841 Tabia et al. [START_REF] Tabia | A new 3dmatching method of nonrigid and partially similar models using curve analysis[END_REF] 0.853 0.527 0.639 0.719 Table 3. Results on SHREC07 for global 3D shape retrieval. better than the other methods in retrieving relevant models at the top of the ranked list of results. Table 3 also shows that the proposed approach outperforms the state-of-the-art on NN, 1-Tier, 2-Tier and DCG measures.

Partial shape retrieval

Partial shape similarity is a complex problem, which has not been fully explored in the literature. We have tested the performance of the proposed method on this problem using the SHREC07 partial retrieval dataset. Figure 3 shows some query models and the top-6 results returned by our method. We can observe that all the retrieved models are highly relevant to the queries. We conducted a quantitative evaluation using the Normalized Discounted Cumulated Gain vector (NDCG) [START_REF] Marini | Shape retrieval contest 2007: Partial matching track[END_REF]. G [i] × log(i) otherwise where G [i] is a gain value depending on the relevance of the i th retrieved model (2 for highly relevant, 1 for marginally relevant and 0 otherwise). The NDCG is then obtained by dividing the DCG by the ideal cumulated gain vector.

Figure 4 shows the NDCG curves of the proposed method and six other state-of-the-art methods: the Hybrid BoW [START_REF] Lavoué | Combination of bag-of-words descriptors for robust partial shape retrieval[END_REF], the curve-base method [START_REF] Tabia | A new 3dmatching method of nonrigid and partially similar models using curve analysis[END_REF], the BoW method of Toldo et al. [START_REF] Toldo | Visual vocabulary signature for 3d object retrieval and partial matching[END_REF], the graph-based technique of Tierny et al. [START_REF] Tierny | Partial 3D Shape Retrieval by Reeb Pattern Unfolding[END_REF], the extended Reeb graphs (ERG) [START_REF] Biasotti | Sub-part correspondence by structural descriptors of 3d shapes[END_REF] and the curveskeleton based many-to-many matching (CORNEA) [START_REF] Cornea | 3d object retrieval using many-to-many matching of curve skeletons[END_REF]. We can clearly see that our method outperforms them. The high performance of the proposed method is due to the descriptive power of the covariance descriptor that efficiently discriminates relevant regions of each model.

Discussion. Analyzing shapes with covariance matrices has several advantages compared to individual descriptors: (1) covariance matrices enable the fusion of multiple heterogeneous features of arbitrary dimension without normalization or blending weights, and (2) spatial relationships can be encoded in the covariance matrices. In 3D shape analysis tasks, it is often desirable to design descriptors that are invariant to shape-preserving transformations. Covariancebased descriptors inherit their invariance properties from the features that are used to build them. Finally, building covariance-based descriptors requires local features that are correlated to each other otherwise covariance matrices become diagonal and will not provide additional benefits compared to using the individual features instead of their covariances.

Conclusion

We proposed in this paper a new approach for comparing 3D shapes using covariance matrices of features instead of the features themselves. As covariances matrices lie on a Riemannian manifold, we use the geodesic distance on the manifold to compute the distance between two descriptors. We proposed two ways of computing the similarity between 3D models. First, we presented a matching method based on covariance descriptors and the associated Riemannian metric. We then presented an extension to the BoW model using covariance descriptors and the Riemannian metric. Experimental results have demonstrated the performance and the potential of the two methods.

Figure 1 .Algorithm 1

 11 Figure 1. (a) k-medoid using geodesic distance, (b) k-means on the tangent space of the mean data point (c) k-means using Geodesic distance and linear update of centers and (d) k-means using Frobenius distance, K = 20 classes.

Figure 2 .

 2 Figure 2. Precision vs recall curves of our method compared to recent state of the art method [34], for the SHREC 2007 global shape retrieval dataset.

Figure 3 .

 3 Figure 3. Some examples of query objects (the most-left object in each row) from the SHREC07 Partial retrieval dataset and the top-6 retrieved models.

Figure 4 .

 4 Figure 4. NDCG curves of our method compared to recent state of the art methods, for the SHREC 2007 Partial retrieval dataset.

Table 1 .

 1 Results on Gavab dataset.

	Methods	Neutral Expression	Overall
	Proposed method	100%	93.30%	94.91%
	Drira et al. [8]	100%	94.54%	94.67%
	Li et al. [40]	96.67%	93.33%	94.68%
	Mahoor et al. [22] 95.00%	72.00%	78.00%
	Moreno et al. [26] 90.16%	77.90%	NA

http://www.gavab.es/recursos en.html

http://www.cim.mcgill.ca/ shape/benchMark/

http://watertight.ge.imati.cnr.it/

http://partial.ge.imati.cnr.it/

Acknowledgements.

The authors would like to thank Guillaume Lavoué and Umberto Castellani for sharing some of their evaluation data. Hamid Laga is partiallyfunded by the Australian Research Council (ARC), Grains Research and Development Corporation (GRDC) and the South Australian Government, Department of Further Education, Employment, Science and Technology.