2017sacls328

Justin D Mathew

M Emmanuel Pietriga

M Sylvain

Marchand Professeur

M Thierry

Duval Professeur

M Florent Berthaut

M Stéphane Huot

M Brian

F G Katz

Huot Blum Mathew

Huot Katz Mathew

A Design Framework for User Interfaces of 3D Audio Production Tools

Titre : Un cadre de conception pour les interfaces utilisateur des outils de production audio 3D Keywords : Interaction Homme-Machine, Audio 3D, Informatique, Traitement de Signal Résumé : Il y a un intérêt important et croissant aà procurer des expériences déecoute immersives pour une variété dapplications, et les améliorations constantes des technologies de reproduction audio 3D permettent de produire des scenes auditives immersives à la fois créatives et réalistes. Mais bien que ces technologies de rendu audio 3D soient maintenant relativement disponibles pour les consommateurs, la production et la création des contenus adéquats restent difficiles en raison de la variété des techniques de rendu, des considérations perceptives et des limites des interfaces utilisateur disponibles. Cette thèse traite de ces problèmes en développant un cadre de conception basé sur deux points de vue : l'analyse morphologique des méthodes et des pratiques audio 3D, et la conception d'interaction. À partir du recueil de données ethnographiques sur les outils, les méthodes et les pratiques pour la production de contenu audio 3D, de considérations sur la perception spatiale liée a laudio 3D, et d'une analyse morphologique sur les objets dintéret connexes (objets audio 3D, paramètres interactifs et techniques de rendu), nous avons identifié les taches que doivent supporter les interfaces utilisateur audio 3D et propose un cadre de conception qui caractérise la création et la manipulation des objets audio. Ensuite, nous avons concu plusieurs techniques dinteraction pour la création audio 3D et avons étudié leurs performances et leur facilité dutilisation selon différentes caractéristiques des méthodes dentrée et de 'mapping' (multiplexage, intégralité, 'directitude'). Nous avons observé des différences de performances lors de la création et de lédition de trajectoires audio suggérant que laugmentation de la sensibilité de la technique de 'mapping' améliore les performances, et quun équilibre entre la séparabilité et lintégralité des méthodes dentrée peut résulter en un compromis satisfaisant entre la performance de lutilisateur et la simplicité matérielle de la solution. Plus généralement, à partir de ces perspectives, nous avons identifié les critères de conception requis pour les interfaces utilisateur audio 3D en vue de compléter notre cadre de conception. Ce dernier, associé à nos résultats expérimentaux, sont un moyen d'aider les concepteurs à mieux prendre en compte les dimensions importantes dans le processus de conception, analyser les fonctionnalités et améliorer les interfaces utilisateur pour les outils de production audio 3D. Title : A Design Framework for User Interfaces of 3D Audio Production Tools Keywords : Human-Computer Interaction, 3D Audio, Computer Science, Signal Processing Abstract : There has been a significant interest in providing immersive listening experiences for a variety of applications, and recent improvements in audio production have provided the capability for 3D audio practitioners to produce realistic and imaginative immersive auditory scenes. Even though technologies to reproduce 3D audio content are becoming readily available for consumers, producing and authoring this type of content is difficult due to the variety of rendering techniques, perceptual considerations, and limitations of available user interfaces. This thesis examines these issues through the development of a framework of design spaces that classifies how 3D audio objects can be created and manipulated from two different viewpoints : Morphological Analysis of 3D Audio Methods and Practices and Interaction Design. By gathering ethnographic data on tools, methods, and practices of 3D audio practitioners, overviewing spatial perception related to 3D audio, and conducting a morphological analysis on related objects of interest (3D audio objects, interactive parameters, and rendering techniques), we identified the tasks required to produce 3D audio content and how 3D audio objects can be created and manipulated. This work provided the dimensions of two design spaces that identify the interactive spatial parameters of audio objects by their recording and rendering methods, describing how user interfaces provide visual feedback and control the interactive parameters. Lastly, we designed several interaction techniques for 3D audio authoring and studied their performance and usability according to different characteristics of input and mapping methods (multiplexing, integrality, directness). We observed performance differences when creating and editing audio trajectories, suggesting that increasing the directness of the mapping technique improves performance and that a balance between separability and integrality of input methods can result into a satisfactory trade-off between user performance and cost of equipment. This study provided results that inform designers on what they might expect in terms of usability when designing input and mapping methods for 3D audio trajectory authoring tasks. From these viewpoints, we proposed design criteria required for user interfaces for 3D audio user production that developed and improved the framework of design spaces. We believe this framework and the results of our studies could help designers better account for important dimensions in the design process, analyze functionalities in current tools, and improve the usability of user interfaces for 3D audio production tools.

A D E S I G N

F R A M E W O R K F O R U S E R I N T E R FA C E S O F 3 D A U D I O P R O D U C T I O N T O O L S
I would like to thank the many friends I have been fortunate to have made during this time and all of my friends from back home. In particular, I would like to personally thank Rudi, Bobby, Kirk, Andrew, Antoine, Marie, Zack, and Alix for being such great friends. In addition, I would like to also thank Laura for always supporting me and helping me take my mind off work when I needed it. Lastly, I would like to thank my parents, brothers, and my cousin, Sini. I could not imagine doing this without you all. v C O N T E N T S The 3D audio renderers referenced as being used by the participants in their projects. The classification of various types of audio objects by their recording methods, rendering methods, and possible interactive spatial parameters in a 3D scene using SpADS-A. The classification of various types of input devices by their input method and visual feedback that can be directly mapped to possible interactive spatial parameters of audio objects using SpADS-C. This flowchart describes one method of designing a rendering space, where the rendering techniques are chosen first, then a matching speaker configuration is designed. These choices define the listener's properties and the room properties are then finally considered. 71 Figure 5.11

SpADS-A v2 with new descriptions shows how different audio objects can be classified based on their recording method and whether it is rendered as a Field-Based or Position-Based with more descriptive spatial parameters of Translation, Directivity, and Orientation. 75 Figure 5. [START_REF] Bøcadker | The human-artifact model: An activity theoretical approach to artifact ecologies[END_REF] SpADS-C v2 with new descriptions (highlighted in yellow) that describe how input devices can be mapped to control different spatial properties and the types of visual feedback through defining control methods as Input Method, Mapping Method, and their Visual Feedback. 76 Figure 5. [START_REF] Marinus | Spatial Sound-Field Reproduction by Wave-Field Synthesis[END_REF] SpADS-A v2: This analysis classifies possible audio objects for the Dolby Atmos and Neve DFC system by their recording and rendering method, and possible spatial parameters that could be interacted with in a 3D virtual scene.

Figure 5.14 SpADS-C v2: This analysis classifies the mouse and the dual joysticks with new descriptions of mapping method and associated visual feedback. 79 Figure 5. [START_REF] Bovair | The acquisition and performance of text-editing skill: A cognitive complexity analysis[END_REF] Side by side comparison of the V1 and V2 version of SpADS highlights compares its analytical power. SpADS v2, on the right, describes the parameters of the Dolby Atmos and Neve DFC-Gemini productions in more detail through a more accurate analysis of the Double M/S object's directivity description (a) and through the visual feedback of the audio object, 3D scene, and rendering space (b). This is indicated through parameters boxed in green. 80 Figure 5. [START_REF] Bredies | The multi-touch soundscape renderer[END_REF] Example showing how SpADS-C can be used to describe new input devices and visualizations that could be used for 3D audio user interfaces. This provides a space on how spatial parameters can be controlled and how visual feedback can assist in the 3D audio tasks. The shapes with black numbers describe the degrees of freedom of the input device, which parameter it can be mapped to, and whether that mapping is direct or augmented. The shapes that describe the spatial parameters within the visual feedback use white numbers to describe the dimension of the visualization. 82 Figure 5.17

Examples of new 3D visual representations with mono object (sphere), Double MS object (circular plane), and speaker configuration (boxes) that could be used for the visual feedback in the control methods of audio objects and rendering space properties. 82 Figure 6.1

Human-computer communication loop for 3D audio production tools where practitioners use input and mapping methods to control audio objects and receive visual and audio feedback from the computer monitor and the audio display system (renderer and speaker configuration) in the rendering space.

Figure 6.2 Human-computer communication loop for pointing interaction technique on a computer monitor that consists of a 2D relative mouse input device that is mapped with a computer-to-display gain transfer function to control the 2D position of the computer cursor on the computer monitor. 90 Figure 6. 3 Human-Computer communication loop for operating a 3D audio application with a mouse input device and a computer monitor for visual feedback and headphones for audio feedback. 90 Figure 6. 4 Examples of a spatially multiplexed MIDI keyboard and a temporally multiplexed mouse-based input device to control the same virtual piano keys of Steinberg's The Grand 3 VST plug-in. 92 Figure 6. 5 Examples of input methods that have separated input methods (slider (1D), knobs (1D), buttons (1D)) to increasing integral input methods (mouse (2 DoF), SpaceMouse (6 DoF)). 93 Figure 6. [START_REF] Michel Beaudouin-Lafon | GlideCursor: Pointing with an Inertial Cursor[END_REF] Example of interaction technique with a direct mapping method to change the width of a rectangle by selecting a boundary point of rectangle and dragging and dropping to a desired width. 96 Figure 6. [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF] Example of interaction technique with an intermediary mapping method to change the width of a rectangle by manipulating the the value of a GUI slider. 97 Figure 6. [START_REF] Augustinus | A holographic approach to acoustic control[END_REF] Examples of augmented mapping methods, such as physics-based engines, divergent, and convergent, that can be used to map how input methods control audio object spatial parameters. 98 Figure 6.9

SpADS-C with the input and mapping method dimensions highlighted. 99 Figure 6. [START_REF] Bier | Skitters and Jacks: Interactive 3D Positioning Tools[END_REF] Human-Computer communication loop for 3D audio production with classifications of input and mapping methods. 100 Figure 6.11

The new input and mapping descriptions of SpADS-C without the visual feedback description. 100 Figure 6. [START_REF] Bøcadker | The human-artifact model: An activity theoretical approach to artifact ecologies[END_REF] SpADS-C Input and Mapping Method analysis of the intermediate methods of the Dolby Atmos and Neve DFC production system with new descriptions. 101 Figure 6. [START_REF] Marinus | Spatial Sound-Field Reproduction by Wave-Field Synthesis[END_REF] SpADS-C Input and Mapping Method analysis of the direct methods of the Dolby Atmos and Neve DFC production system with new descriptions. 102 GUI Slider interaction technique using an intermediate mapping strategy using a trackpad to control a mouse cursor that controls GUI sliders to manipulate spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b). 108 Figure 7. 3 GUI Timeline interaction technique using a trackpad, keyboard shortcuts, and GUI timeline to control spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b). 110 Figure 7. 4 Zonal Trackpad interaction technique using predefined zones on the trackpad to control spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b). 111 Figure 7. 5 Physical Slider interaction technique where each slider is mapped directly to 3D position of an audio object (a) and a SpADS-C analysis of the technique. 112 Figure 7. [START_REF] Michel Beaudouin-Lafon | GlideCursor: Pointing with an Inertial Cursor[END_REF] Phantom controller interaction technique that is directly mapped to the 3D position of an audio object (a) and a SpADS-C analysis of the technique. 113 SpADS-C provides a space to describe control methods of audio objects by their characteristics of the visual feedback, input method, and mapping method. 138 Figure 8. 3 Model that identifies stages in audio production and how audio objects are created and manipulated. 140 Figure 8. 4 Classification of three major tasks in 3D audio production. 141 Figure 8. 5 Flowchart on how to use virtual speakers to decode any audio object onto any auditory display. 142 Table 4.2 Visual cues that have been tested to affect the perception of distance and depth in visual images. Monocular (or pictorial) cues provide depth information from the 2D image created by the vision. Binocular cues provide perception of depth based on the differences in information captured by each eye, and Motion cues provide sense of depth and shape with based on movement and orientation of objects in the image. The advancement of media technologies has provided us the capability to experience a vast amount of content, and recent trends in the development of immersive media technologies have been increasingly popular with consumers. Immersive technologies simulate our spatial perception of our surrounding, and it can provide experiences that blur the notion between simulation and reality through the stimulation of our senses. This changes our sense of spatial presence from being in the physical world to a spatial presence of a different environment [START_REF] Mestre | Immersion et présence[END_REF]. While a major focus of immersive technologies has been to improve the spatial experience through visual displays, such as large 2D visual displays, 3D glasses, and virtual reality head sets, there has been significant interest in providing a sense of spatial immersion aurally. 3D audio technology gives listeners the impression of sonic environments with sound sources originating from various directions using 3D auditory displays. Currently, technologies to reproduce 3D audio content are becoming readily available for consumers. In cinema and home theaters, audio systems that use 3D speaker configurations coupled with 3D audio rendering techniques are becoming popular methods to increase the sense of immersion. In video games, spatial audio technologies are used to improve the experience of gameplay scenarios, and more recently, virtual reality content has used spatial audio to increase the immersion of the virtual environment. Even though 3D audio technology can be applied across a wide range of applications that use visual displays, there has been significant interest for some time from composers and audio engineers to provide a sense of acoustic space through performances of musical compositions and audio recording/broadcasting technology.

spatial music and 3d audio technology

The concept of space in music productions can have two different meanings. One paradigm of spatial music focuses on the use of geometric space for composing musical scores, and the other focuses on the spatial acoustics of instruments and environments to insert a sense of spatial presence in the audience's listening experience [START_REF] Solomon | Spatialization in music: The analysis and interpretation of spatial gestures[END_REF][START_REF] Trochimczyk | From circles to nets: On the signification of spatial sound imagery in new music[END_REF]. Many 1 composers have used the latter paradigm as an important part of the creative process in their compositions because, as Trochimczyk [START_REF] Trochimczyk | From circles to nets: On the signification of spatial sound imagery in new music[END_REF] eloquently noted, the "idea of immersing the listeners in stationary, circular sound fields may have quasi-mystical overtones, as this design serves to evoke other-worldly stillness and to induce a state of contemplation". It was reported that the musical composer, Charles Ives, spatially separated musicians in his compositions for the purpose of facilitating "the differentiation of two or more simultaneous -and relatively independent -layers of sound" by freeing "music from being strictly a foreground phenomenon and to give it a background layer of depth" [START_REF] Solomon | Spatialization in music: The analysis and interpretation of spatial gestures[END_REF]. Popular examples of this were seen in Iannis Xenakis's compositions of Terretektorh (1965Terretektorh ( -1966) ) and Nomos Gamma (1967)(1968)) that dispersed musicians among the audience in circular performance spaces to give different spatial listening experiences to each audience member. Other compositions, such as Karlheinz Stockhausen's Carré (1959)(1960), also used a similar technique by placing performers on the sides of performance spaces to immerse the audience members [START_REF] Trochimczyk | From circles to nets: On the signification of spatial sound imagery in new music[END_REF]. Similarly, Pauline Oliveros used a strategy of physically surrounding audience members with "meditating performers" in a circle in her composition Zina's Circle from Sonic Meditations (1972) [START_REF] Trochimczyk | From circles to nets: On the signification of spatial sound imagery in new music[END_REF]. As audio technology advanced, composers started to use loudspeakers to spatialize their compositions. One of the most famous uses of this method was Edgar Varèse's composition Poème Électronique, which was made specifically for the pavilion of the 1958 World Fair in Brussels, Belgium. This was an installation where Iannis Xenakis and Le Corbusier worked on the architectural acoustics of the pavilion and used 350 loudspeakers situated along the walls to provide an immersive listening experience of moving sounds [START_REF] Lombardo | A virtual-reality reconstruction of poeme electronique based on philological research[END_REF]. More recently, Smalley [START_REF] Smalley | Space-form and the acousmatic image[END_REF] described a model of composing with space for acousmatic music, which are compositions that are made to be played out of loudspeakers in a performance space to force audience members to focus on the listening experience of the composition rather than the visual experience of seeing live musicians. A majority of musical compositions normally focus on musical elements, but these examples highlight the attraction to use acoustic space as an extra creative dimension to physically separate musical elements from each other. While there was interest from many musical composers in using spatial acoustics as part of their creative process, there was also early interest in the development of spatial audio technologies for recording and broadcasting purposes.

Throughout the development of audio technologies to capture (microphones), store (wax cylinders, vinyl, digital formats), and broadcast (loudspeakers) sound, there has been significant effort in improving the quality and spatial listening experience for audience members. Early experimentation and demonstration of 3D audio technologies used micro-phone pairs that simulated audience members to capture an orchestral performance at the La Lumière Électrique exhibition at the Paris Grand Opera in 1881. This system, developed by Clement Ader, routed the microphones to matching telephone receivers where listeners could put a receiver at each ear to listen to the performance, which was one of the first designs of binaural audio technology. One listener commented that the use of two telephone receivers causes the "sound [to take] a special character of relief and localization" [START_REF] Torick | Highlights in the history of multichannel sound[END_REF]. Other early spatial audio technologies focused on the use of multi-channel recordings and transmission through loudspeakers. In 1933, researchers at Bell Laboratories demonstrated the transmission of a three-channel recording of a performance in Philadelphia to a three-channel loudspeaker system in Washington D.C. [START_REF] Torick | Highlights in the history of multichannel sound[END_REF]. Broadcasting with two-channel stereo, patented by Alan Dower Blumlein in 1933, eventually became popular, but it took more than 20 years for it to be the standard storage and broadcasting format for vinyl records and radio. Even though it has been more than 60 years since the adoption of this technique, music and cinema industries continually use the stereo listening format. There was an interest in quadrophonic display systems by many engineers and musicians to augment the spatial experience from stereo to four-channel speaker systems arranged in a square with listeners situated in the center [START_REF] Torick | Highlights in the history of multichannel sound[END_REF]. However, the cost of the technology and lack of interest in the radio broadcasting industry kept the stereo format as the industry standard [START_REF] Torick | Highlights in the history of multichannel sound[END_REF].

Further advancements of spatial audio production occurred within the cinema industry, where 5 speaker surround sound display technology was developed by engineers at Disney Hyperion Studios and introduced in 1940 with the movie Fantasia to increase the immersive experience of movie. This technique arranged three speakers in front of the audience (left-center-right) with two speakers situated in the left and right back of the audience. Engineers would mix audio to two channels for this speaker format. One channel was normally designated for speech and routed to the center speaker, and the other was designated for ambient sound and routed to the rest of the speakers [START_REF] Torick | Highlights in the history of multichannel sound[END_REF]. This production technique was further developed with Dolby 5.1 surround sound format that used five audio channels for each of the speaker and provided a Low-Frequency Effect (LFE) channel for bass effects [START_REF] Spath | Dolby E-A Practical Way to Distribute Multichannel Audio[END_REF]. This improved the placement accuracy of onscreen sounds by providing more freedom and flexibility for engineers to control the location of sounds between the three front speakers, and it provided the capability to improve the immersion of listeners with a designated speaker for low-frequency content.

While surround sound has been a standard in cinema audio formats and has been used for musical purposes, 3D auditory displays were developed to provide a sense of height. For the Osaka World Fair in 1970, Stockhausen used over 50 loudspeakers along the surface of a spherical auditorium for his composition Spiral (1960) [START_REF] Solomon | Spatialization in music: The analysis and interpretation of spatial gestures[END_REF]. While other composers also experimented with the sensation of height, these speaker configurations and production methods were designed experimentally and eventually more methodological techniques for rendering 3D audio content were developed. Gerzon [START_REF] Michael | Ambisonics in Multichannel Broadcasting and Video[END_REF] developed ambisonic methods that focus on the use of spherical harmonics to capture sound in three dimensions with ambisonic microphones and reproduce it across three dimensional speaker configurations. Pulkki [START_REF] Pulkki | Virtual Sound Source Positioning Using Vector Base Amplitude Panning[END_REF] developed Vector-Based Panning Algorithm (VBAP), which provides an algorithm to position sound sources between triplets of speakers, and the arrangement of multiple triplets of speakers can form an enclosed geometric configuration to provide a 3D immersive listening experience. Berkhout [START_REF] Augustinus | A holographic approach to acoustic control[END_REF] proposed a holographic method of spatializing audio using a wave-field synthesis method and dense linear speaker arrays to model how sound waves travel in air. Further work on binaural reproduction created renderers that use perceptual filters to model how sound waves travel from a source to listeners [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF].

On the commercial side, companies have created 3D audio digital formats and 3D speaker configurations to produce and broadcast 3D audio content, such as Dolby Atmos and Auro3D [START_REF] Claypool | Auro 11.1 versus Object-Based Sound in 3D[END_REF][START_REF]Dolby Atmos Cinema Technical Guidelines[END_REF]. In addition, there has been significant interest in broadcasting radio productions binaurally due to the popularity of using headphones [START_REF] Nicol | A Roadmap for Assessing the Quality of Experience of 3D Audio Binaural Rendering[END_REF]. For these technologies, content can be created from a wide variety of tools that range between libraries for programming languages, digital audio production software, and user interfaces with advanced input devices. However, the development of these systems has focused on displaying a spatial listening experience, and many user interfaces for 3D audio production tools are still underdeveloped with limited functionality. This forces practitioners to configure various tools together to spatially mix 3D audio content.

user interfaces for 3d audio production tools

Audio production tools have been developed with a focus on controlling the properties of audio signal algorithms. These algorithms were designed based on the recording methods and how the production will be broadcast and stored. Initial audio productions consisted of a single microphone that converted sound waves generated by performers and musicians to analog electrical signals that can be measured by voltage, current, and power, which were then directly routed to the broadcasting system or a storage medium [START_REF] Owsinski | The Recording Engineer's Handbook[END_REF]. As engineers started to use more microphones to capture a higher sound quality, electrical signal processing techniques were used to mix the signals from each microphone. Mixing audio signals consisted of using a multi-channel mixer that used a signal processing summing algorithm to add audio signals together, which consisted of routing and adjusting the power of each signal through gain amplifiers to create a balance for clarity and intelligibility [START_REF] Owsinski | The Mixing Engineer's Handbook[END_REF]. In the mental model of this design, audio signal routing and summation are considered like pipes that users must connect together for mixing. This model eventually developed to include electrical signal processing techniques that would divert the routing to manipulate the audio signal and then route it back to original path. These audio effects are considered as "plug-ins" that are inserted into the main path of the routing, allowing users to manipulate audio signal properties, such as equalization filtering (low-pass, band-pass, band-stop, high-pass, multi-parametric, etc.) [START_REF] Owsinski | The Mixing Engineer's Handbook[END_REF].

As two-channel stereo broadcasting was adopted, panning algorithms were developed that allowed engineers to interact with the routing using a ratio that determines how much of an audio signal is routed to the left and right speaker channels. Rather than relying on the recording method to provide a spatial sensation, panning methods allowed engineers to modify the position of the sound source in the space between the speakers [11,[START_REF] Michael | Panpot laws for multispeaker stereo[END_REF][START_REF] Owsinski | The Mixing Engineer's Handbook[END_REF]. Common interaction designs for stereo panning used rotary knobs for each audio channel within mixing consoles, which provided a physical visual feedback of the direction the audio channels were panned towards. This feedback presents information on where to expect an audio channel within the sound field generated by stereo speakers and provides information on how much an audio signal is sent to the left and right speaker channels. The use of a rotary knob also matches well with surround sound formats as the rotary knob can provide information in a similar manner. This design still provides a mental model of routing, but routing can be done with a knob rather than connecting signals together.

In general, the design of mixing consoles matches with the mental model of the audio signal flow when mixing for stereo and surround sound formats, as they provide physical visual feedback that can be understood easily and quickly. With the digital revolution, Digital Audio Workstations (DAWs) were modeled after multi-channel mixers and use graphical user interface (GUI) elements based off of these common interaction methods. They also provide increased functionality of authoring how signal-processing parameters can change over time. This can be done through automation drawing where users define how a single parameter changes along a timeline or sequencer. However, as practitioners started to use 3D speaker configurations, the design of panning, authoring, and low-level interaction techniques were still the standard methods for mixing. This increased the complexity of producing 3D audio content as no standard 3D panning and authoring method has been built into the design of mixing consoles and DAWs in the same manner as rotary knobs and GUI elements. Instead, practitioners were required to properly route the audio channels to create a 3D listening experience, which can become complex and tedious when mixing a large number of audio channels. To help solve this, plug-ins and external software were developed for DAWs that rendered the audio signals and provided 3D computer graphics for a visual feedback of the location of the audio channel within the 3D audio scene [START_REF] Claypool | Auro 11.1 versus Object-Based Sound in 3D[END_REF][START_REF]Authoring for Dolby Atmos Cinema Sound Manual[END_REF]. These types of software are used alongside standard designs of mixing consoles and DAWs, but their design still complicates the mental model of producing content for 3D audio speaker configurations due to their limited flexibility with routing and bussing of audio channels. This forces practitioners to mix 3D audio content through routing and configuring with separated interactions rather than through an integral method that is similar to panning. However, the design of the 3D audio rendering techniques used within the plug-ins and external software provides an opportunity for improving the interactions towards an integral 3D panning and authoring technique.

For 3D audio practitioners, content can currently be created from a wide variety of tools that range from programming libraries to 3D user interfaces. Within these tools, spatial parameters can be assigned to an audio signal, which will then be interpreted by the renderers to compute how the signal should be routed to specific speakers in order to create the intended spatial experience. This creates a different paradigm of interaction, where user interfaces for 3D interaction can be used rather than control of low-level signal processing and routing within multichannel mixers or DAWs. However, due to 3D audio technology being relatively new in the consumer market and the current popularity of standard multi-channel mixers and DAWs, user interfaces for 3D audio production have had little advancement in this direction. Instead, practitioners have been forced to configure various tools to properly produce 3D audio content in manners they would like. In this thesis, we investigate this issue by studying current practitioner's tools, methods, and practices, spatial perception during interaction, and the usability of user interfaces within these tools. Through this work, we developed a design framework, Spatial Audio Design Spaces (SpADS), which organizes and classifies design criteria of user interfaces that are important when producing 3D audio content.

a design framework for user interfaces of 3d audio production tools

Design frameworks, commonly used in the field of Human-Computer Interaction (HCI), provide methodologies on how to design a particular product, functionality, or interaction. These can be found in different forms and levels of detail, such as Model-View-Controller (MVC) methodology for creating software applications [START_REF] Krasner | A description of the model-view-controller user interface paradigm in the smalltalk-80 system[END_REF] or the Genex four phase framework for user interfaces that support innovation [START_REF] Shneiderman | Creating Creativity: User Interfaces for Supporting Innovation[END_REF]. Another common methodology is to conduct a morphological analysis that characterizes the intrinsic properties of the objects of interests and their dependencies through design spaces. This latter strategy was initially applied to the design of input devices to make better sense of how their various characteristics affect usability [START_REF] Stuart | A Morphological Analysis of the Design Space of Input Devices[END_REF]. We adopted this method due to various 3D audio tools and production techniques currently available and used this strategy to develop a methodology on how to create user interfaces for 3D audio production tools. However, many factors must be considered due to the variety of 3D audio techniques, the design of production systems, the multi-modal perceptual nature of the interaction, and how the interfaces allow practitioners to interact with the objects of interest. Throughout this work, we observed that there are dependencies between recording and rendering techniques and how the user interfaces are designed. We used these assertions as a basis to develop the SpADS framework as a set of two design spaces that classify 3D audio objects as the object of interest during interaction (SpADS-A) and how user interfaces are designed to control them (SpADS-C). In this thesis, we present the development of this framework in two parts: 3D Audio: Tools, Methods, and Practices and Interaction Design for 3D Audio Authoring.

Part I -3D Audio: Tools, Methods, and Practices

In this part of the thesis, we discuss the state of 3D audio production tools, methods, practices, and spatial perception to better understand how practitioners produce 3D audio content. In Chapter 2, we present a review on the state of the art in 3D audio rendering techniques, 3D audio user interfaces in consumer productions, studies and projects in 3D audio user interfaces, and their relationship to research in 3D User Interfaces (3DUI) from other domains. In Chapter 3, we present results of conducted interviews and an online survey of current 3D audio practitioners that identified limitations in current tools and their methodolo-gies in producing 3D audio content. From practitioner comments reported in Chapter 3, we identified issues in relating the visual feedback of many current user interfaces with the listening experience. To better understand how to improve the design of visual feedback, we provide an overview of the related work on spatial perception in auditory and visual modalities and higher-level spatial comprehension in Chapter 4.

In Chapter 5, we present a morphological analyses from the work presented in Chapters 2 to 4 that informed the development of the SpADS framework. The first version of the framework presents a set of two design spaces that classify audio objects by their recording and rendering methods (SpADS-A) and how user interfaces are designed to control them (SpADS-C). In addition to this, responses from the online survey informed a high-level classification of tasks that are required for 3D audio production, which describes how practitioners must define the properties of the space for rendering the production, create and manipulate audio objects, and monitor the properties of the production with feedback from the auditory and visual displays. A second version of the SpADS framework was developed with revised and added dimensions that better describe audio objects and the properties of the user interface based off of the responses of the online survey and literature on spatial perception. The classification of tasks provides a model of how to produce 3D audio content, while the SpADS framework provides design spaces that define the relationships of audio objects and their control methods within the design of user interfaces.

Part II -Interaction Design for 3D Audio Authoring

Practitioners comments discussed in Chapter 3 also mentioned limitations in interaction techniques within 3D audio tools. In particular, comments and responses focused on interaction techniques for controlling audio objects and authoring 3D trajectories identifying limitations in their usability. We investigated this issue by analyzing how different characteristics of input and mapping methods of user interfaces affect tasks related to authoring and editing 3D trajectories. In Chapter 6, we first examine design methodologies, input methods, and mapping strategies that are important in the design of 3D audio user interfaces. Through this examination, we again improved the SpADS framework to include important characteristics of input and mapping methods as dimensions within the SpADS-C design space.

We then explored some of the dimensions of the design space through a study that investigated different mapping strategies for desktop inter-faces based on their mapping strategies and varying characteristics of multiplexing and integrality of input methods in Chapter 7. These results suggest that increasing directness of the mapping and integrality of the input methods improves completion time of tasks. Observations and participant comments also suggest that the integrality of input methods affect camera movements, the references in the visual feedback are important for completing the tasks, and the structure of the task may be operationalized into separated tasks rather than fully integral. In addition to these results, this study demonstrates how varying the input and mapping methods can affect usability in completing specific 3D audio production tasks.

Overall, the work presented in this thesis explores different design criteria that should be considered for improving the current state of 3D audio production tools. From a practitioner's perspective, it is important to have the flexibility of current production systems in order to include advanced functionalities with 3D audio recording and rendering techniques, authoring, and the capability to produce across various speaker configurations. Within 3D audio production user interfaces, improvements to the visual feedback are required for them to be more informative and relatable to listening experiences. In addition, we classified and investigated how input and mapping methods of 3D audio user interfaces could affect accuracy and completion time in 3D audio trajectory authoring tasks. This work provided results that help organize and classify design considerations to form the design spaces of the SpADS framework. This framework provides descriptive and analytical powers that allow designers to describe the capabilities of 3D audio production tools, analyze their functionalities, and help evaluate how changes in the dimensions affect usability. In doing so, this framework can help designers identify limitations, suggest improvements, and identify design choices that improve usability of 3D audio production tools.

Part I

3 D A U D I O : T O O L S , M E T H O D S A N D P R A C T I C E S
Advanced user interfaces for 3D audio production and authoring tools are relatively new within consumer products, but there has been significant interest in their development within the research and academic field. However, there are difficulties for current practitioners to adopt many of the new tools being developed. In Chapter 2, we survey the state of the art in user interfaces of 3D audio production and authoring tools. This is accomplished by identifying interaction techniques from consumer tools and previous studies, their parallels to 3DUI for other domains, and different evaluation techniques. Afterwards, we discuss the results of two studies that resulted in identifying 3D audio practitioner tools, methodologies, missing functionalities, and current limitations (Chapter 3). In Chapter 4, we overview human visual and aural perception in comprehending 3D space and how they relate to spatial perception when producing 3D audio content to better understand issues within the design of audio and visual feedback. This is followed by a presentation of higher level analyses in Chapter 5 that identified tasks for 3D audio production and formed the SpADS design framework to help better analyze and inform the design of user interfaces for 3D audio production. The results of these studies and analyses help organize issues and limitations in the development of 3D audio production tools, and they suggest direction on how to better develop these tools. 3D audio has been applied to a diverse range of applications. Practitioners employ spatial audio techniques across multiple domains, such as cinema, video games, virtual reality simulators, computer-aided composition, audio engineering, and sounds installations. 3D audio tools allow practitioners to create this type of content and vary between plugins for DAWs to low-level libraries for programming languages, which renders audio data to a 3D auditory display. Practitioners can choose a number of different ways to capture audio content and render them across different speaker configurations and listening formats. However, many 3D audio tools tailor the design of their user interfaces for the specific rendering method they use. This creates two paradigms practitioners have to consider when creating and editing 3D audio content:

1. How is the 3D audio content created and how will it be rendered? 2. How does the user interface control properties of 3D audio content?

To better understand these paradigms, we adopt definitions used in audio file-format design by Geier, Ahrens, and Spors [START_REF] Geier | Object-based Audio Reproduction and the Audio Scene Description Format[END_REF] to classify and overview common 3D audio recording, rendering, and interaction techniques. Even though these definitions were initially defined for fileformat storage, they help to define what are 3D audio objects, how can they be created and rendered, and how they can be manipulated through a user interface. After this, we overview how 3D audio user interfaces have been developed and evaluated, and then finally examine the similarities and differences in the development of 3D user interfaces for other domains. This overview explains how current 3D audio systems allow users to create and edit content, the development of the user interface for these systems, and identifies design and evaluation techniques that designers of 3D audio user interface do not commonly use.

3d audio objects

Audio objects are high-level objects of interest that were initially proposed for the design of audio file-formats that are renderer flexible [START_REF] Geier | Object-based Audio Reproduction and the Audio Scene Description Format[END_REF].

Just as in the development multi-channel audio technology, Geier, Ahrens, and Spors [START_REF] Geier | Object-based Audio Reproduction and the Audio Scene Description Format[END_REF] developed their format for audio storage and broadcasting purposes, but this concept can be adapted to the purpose of interaction where audio objects are objects of interest for user interaction. Adopting this concept for interaction design helps ease the spatialization process by providing users with direct manipulation of spatial parameters similar to panning algorithms for stereo and surround sound instead of low-level control of audio signal routing. However, the majority of standard audio rendering techniques require a specific auditory display design (speaker configuration or headphones) and their interactive spatial parameters are dependent on the recording and rendering methods. This requires practitioners to consider the spatial characteristics of the recording and rendering method together when creating 3D audio content, which can be classified into two methods: Field-Based and Position-Based. Using this classification, we explain how current rendering techniques are designed to create 3D audio objects.

Field-Based

This method of creating audio objects captures the sound of a specific acoustic space through the placement and orientation of microphones. Using the information of the microphone placement and orientation, practitioners can reproduce the recorded acoustic space with a respective rendering methods using the spatial characteristics encoded in the audio data. Each audio recording is assigned an encoding label and rendered through the respective decoding techniques that decide how the audio signal is processed and routed to the auditory display. One common method of creating Field-Based audio objects is through Direct Channel-Based encoding and decoding, which assigns the audio from a specific microphone, like a Decca Tree Surround Sound microphone 1 , to be directly routed to a specific output channel of a common speaker configuration. This technique normally uses audio recordings from microphone configurations that are positioned to represent speaker configurations in order to capture and reproduce a realistic impression of an acoustic scene [START_REF] Sawaguchi | Multichannel sound mixing practice for broadcasting[END_REF]. Another method is through ambisonic encoding and decoding, which uses the arrangement and directivity of the microphone capsules, like the TetraMic 2 , to encode more spatial information in the audio recordings. The recordings from each microphone capsule are encoded into an ambisonic format (A-Format, B-Format, etc.) depending on the order of the spherical harmonics used. These formats provide the necessary information to decode the recordings through ma- trixing algorithms onto a speaker configuration [START_REF] Michael | Ambisonics in Multichannel Broadcasting and Video[END_REF][START_REF] Malham | 3-D Sound Spatialization using Ambisonic Techniques[END_REF] (see Figure 2.1 for examples of Field-Based techniques).

Position-Based Methods

This method contrast the Field-Based methods by ignoring the spatial data encoded in the audio recording itself and treats recordings as individual sound sources that originate from a point in space. Through these techniques, renderers take a single audio signal and a user defined position to compute the routing to the output channels. Through this, users can model 3D audio scenes by positioning multiple audio signals to be like point sources in a 3D space. We classify these common techniques by rendering methods due to their similarity of rendering mono audio signals. Amplitude Panning (Vector-Based and Distance-Based) rendering techniques is based on the technique of positioning the sound to be perceived between speakers. Vector-Based Amplitude Panning (VBAP) uses the equal power panning algorithm (same as standard stereo panning) to compute the ratio of gains between triplets of speakers to pan with height (see Figure 2.2(a)). This method can provide the impression that a sound source is originating anywhere between the triplets of speakers based on the summing localization theory [11], which we discuss in more detail in Chapter 4. Using a user defined position and positional information of the speakers, the audio signal can be routed to the correct set of speaker triplets [START_REF] Pulkki | Virtual Sound Source Positioning Using Vector Base Amplitude Panning[END_REF]. Distance-Based Amplitude Panning (DBAP) extends this by adding distance attenuation and spatial blur in the algorithm to create the perception of distance [START_REF] Lossius | DBAP -Distance-Based Amplitude Panning[END_REF]. High-Order Ambisonic (HOA) Panning is based on ambisonic recording and decoding methods to simulate how an audio source would be captured in an ambisonic recordings through encoding and decoding functions (see Figure 2.2(b)). In this method, an audio signal is first encoded into an ambisonic format that simulates how an ambisonic microphone captures the sound and then automatically routes it onto the speaker configuration through matrixing functions. The order of the spherical harmonics used for the encoding and decoding decides the resolution of panning [START_REF] Neukom | Ambisonic Panning[END_REF].

Wave Field Synthesis (WFS) simulates a sonic wave-field using Huygen's principle, where the speaker configuration is considered as a wavefield boundary (see Figure 2.2(c)). The algorithm renders an audio signal across a speaker array to simulate how a source sound produces a wave front that originates behind the speaker configuration boundary. By calculating how the wave front reaches the speaker configuration boundary, the rendering algorithm simulates how the wave front would continue past the boundary and simulates this through the speakers.

For accurate spatial rendering, a dense loudspeaker array is required for this technique [START_REF] Augustinus | A holographic approach to acoustic control[END_REF][START_REF] Augustinus | Acoustic control by wave field synthesis[END_REF]. Lastly, Binaural rendering algorithms use the perceptual theory of how sound travels from the source and arrives at the listener's ear to simulate a more accurate spatial listening experience through headphones. This technique uses binaural perceptual cues of time difference of arrival (Interaural-Time Difference) and difference in intensity and level (Interaural-Intensity Difference or Interaural-Level Difference) between the two ears to replicate a spatial listening experience. This is implemented using a dataset of Head-Related Transfer Functions (HRTFs) that consists of ITD information and IID filters that describe these cues across different positions surrounding the listener. A user-defined position identifies which HRTF filters are required to render the audio onto the left and right channels of the headphones (see Figure 2.2(d)). Since the HRTFs are based on perceptual factors specific to the listener, proper HRTF matching and other features are important to ensure the rendering matches the listener's perception [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF][START_REF] Kendall | A 3-D Sound Primer: Directional Hearing and Stereo Reproduction[END_REF][START_REF] Laurent | Perceptual attributes for the comparison of head-related transfer functions[END_REF].

Both Field-Based and Position-Based methods allow the creation of a 3D audio scene through different paradigms, which is determined by the design of rendering techniques and its interactive spatial parameters. Position-Based methods mostly focus on user-defined positional parameters of audio sources, and Field-Based methods mostly focus on the decoding of a recorded scene. We classified common rendering techniques under a specific methodology due to how they are currently been applied in various tools and plug-ins, but the majority of these recording and rendering techniques can be used with both methodologies. For example, a linear microphone array can be rendered using a Position-Based method where each microphone or the full array has a specific 3D position, or it could rendered using a Field-Based method using the spatially encoded data and simulating a wave-field (WFS). This classification mostly depends on the intention of the practitioner and the design of recording and rendering technique. Even though the design of rendering techniques provide the ability to control and edit spatial parameters, many current 3D audio user interfaces vary in the levels of interactions from low-level signal processing control to higher-level spatial control.

3d audio user interfaces

Geier, Ahrens, and Spors [START_REF] Geier | Object-based Audio Reproduction and the Audio Scene Description Format[END_REF] differentiated how original audio fileformat methods would require the audio channels to be defined to a specific output channel when designing an audio file-format for storage of spatial audio production, which they described as Channel-Based fileformats. This differed with file-formats that used audio objects for rendering across multiple listening formats, which they described as Object-Based file-formats. We adopt these terminologies to describe how the design of user interfaces allows practitioners to create and edit 3D audio content. Initially, common audio production systems provided user interfaces with low-level audio signal processing and channel routing to a specific listening format. This Channel-Based approach to content creation requires users to record audio into specific input channels and route the audio signals to a specific output channel of the listening format. This method has been consistently used in the design of multichannel mixing consoles, like the Neve DFC PS-1 3 (see Figure 2.3(a)). Consoles like this are commonly paired with DAW software, such as ProTools HD 4 (see Figure 2.3(b)), that enable users to digitally record, edit, and manipulate audio. The Channel-Based production methodology pairs well with the design of mixing consoles due to the design of the signal processing and interface being similar and the predefined listening formats. In this typical set-up, users route individual channels to specific output channels through either built-in stereo/surround sound panning techniques or through bussing/routing of channels. With a large number of speakers, this method can become tedious and complicated. On the other hand, Object-Based interaction techniques ease this process by allowing users to define a spatial position that is used to automatically route to the proper channels for any auditory display design. Many 3D audio consumer software incorporate object-based tools with advanced control methods as plug-ins to traditional mixing consoles and DAWs, but issues with bussing/routing, limited output channels, and the need to configure external software and devices can add to the complexity of producing 3D audio content for various types of auditory displays. However, there has been research and development of tools and systems to produce 3D audio content in an Object-Based approach that includes advanced interaction techniques for manipulating audio objects within a 3D audio scene.

Consumer Products

Many of the advanced consumer products available have incorporated Object-Based interaction techniques for creating content to specific listening formats and domains, such as Dolby Atmos . Within these tools, GUIs are provided to control spatial parameters, but many only provide simple GUI controls, such as sliders and knobs found in the Dolby Atmos plug-in (see Figure 2.4(a)).

In addition to the use of simple GUI controls, other tools like the Spatial Audio Designer 9 and IRCAM Spat 10 , provide interactions along two 2D planes that allow the user to control two spatial parameters simultaneously. Some more advanced interaction techniques have been used as well, such as Longcat's H3D and AudioStage 11 (see Figure 2.5), that use a more direct interaction paradigm by allowing users to interact with 3D axes within a 3D visualization. 3D axes are provided for each selected audio source, and users can reposition and reorient them by using a method similar to the Skitter and Jacks techniques [START_REF] Bier | Skitters and Jacks: Interactive 3D Positioning Tools[END_REF]. This allows users to select one of the three planes on the cursor that then maps the 2D position of the mouse to the two dimensions within the selected plane (see Figure 2.5). In addition to GUIs, many of these tools allow parameters to be controlled with external input devices, such as standard mixing consoles, joysticks, or 3 DoF input devices. The SSL Duality 12 console augments standard mixing consoles (knobs, sliders, and buttons, routing, etc.) with a trackball, and the Neve DFC 13 mixing console provides two joysticks to directly manipulate the 3D position of an audio source. Recent software, such as Dolby Atmos and Sonic Emotion Wave I, provide the capability to control parameters with multi-touch tablets through an Open Sound Control (OSC) network connection.

The majority of these tools focus on the interaction with the spatial parameters of position and orientation, but some are more advanced with the ability to interact with the acoustic parameters of the sound source. Tools, like the IRCAM Spat and Sonic Emotion Wave 1, allow users to manipulate the directivity of the sound source so that it can be perceived as more narrow or wider. Similar techniques can be seen in other programming language libraries, but they are not widely implemented in consumer products. 3D visualization is used for 3D interaction and a sequencer is used for temporal properties of the production.

Design and Evaluation of 3D Audio Interfaces

In addition to available consumer-based products, various types of systems for 3D audio have been proposed through research and academic institutions. The SLABScape is a real-time virtual reality environment that uses binaural rendering and a 3D scene that can be manipulated in real-time [START_REF] Miller | Recent developments in slab: A software based system for interactive spatial sound synthesis[END_REF]. The SoundScape Renderer GUI (see Figure 2.6(a)), allows users to render audio objects across multiple renderers and provides a 2D GUI to position sound sources [START_REF] Geier | The SoundScape Renderer: A Unified Spatial Audio Reproduction Framework for Arbitrary Rendering Methods[END_REF]. The OMPrisma library [START_REF] Schumacher | Spatial Sound Synthesis in Computer-Aided Composition[END_REF], a library of objects for the OpenMusic visual programming language, allows composers to spatialize compositions using GUI editors (see Figure 2.6(b)). GUI objects for 3D audio tools have also been proposed for MaxMsp and PureData visual programming languages. For example, the HOA Library [START_REF] Sèdes | The HOA library, review and prospects[END_REF] and ICST Ambisonic Externals [START_REF] Peters | A stratified approach for sound spatialization[END_REF] provide a library of objects with ambisonic-based rendering and GUIs to control and monitor spatial parameters. Lastly, Holo-Edit provides a graphical and algorithmic editor to author and draw control parameters of audio trajectories that resemble DAW and automation techniques (seen Figures 2.6(c) & 2.6(d)) [4]. These projects provided a set of tools where users can create, edit, and render 3D audio content. Other works focused more on the design of new interaction techniques for controlling sound source position. More specifically, there has been an interest in studying the use of Mid-air Gestures, High DoF Input Devices, and Multitouch Interfaces for spatial mixing.

Mid-air Gestures

Churnside, Pike, and Leonard [START_REF] Churnside | Musical Movements-Gesture Based Audio Interfaces[END_REF] studied the use of skeletal-tracking and gestures to control the position of a sound source. In their pro- posed system, they mapped the orientation of the elbow and wrist to azimuth and elevation position of the sound source. This technique was evaluated with three professional audio engineers by allowing them to explore the control of a sound source within a mix of audio and recording their subjective feedback. This resulted in mixed opinions from the three subjects about the use of mid-air interaction for its practicality and its effects on helping creativity [START_REF] Churnside | Musical Movements-Gesture Based Audio Interfaces[END_REF]. In a later study by Gelineck and Korsgaard [START_REF] Gelineck | An Exploratory Evaluation of User Interfaces for 3D Audio Mixing[END_REF], a traditional mixing console was compared to mid-air tracking for positioning, selecting, and de-selecting audio objects, and they compared differences in using multiple 3D views versus a single 3D view. Participant comments indicated that it was easier to use the mixing console for the mixing task of positioning a sound, but that mid-air gestures gave them a better understanding of the 3D space. In addition, the selection technique was deemed too difficult by participants, but they enjoyed the bi-manual interaction of selecting with one hand and freeing the other hand for other purposes. The multiple view windows was preferred by participants, but they still noted issues regarding occlusion with the 3D graphics [START_REF] Gelineck | An Exploratory Evaluation of User Interfaces for 3D Audio Mixing[END_REF].

High DoF Input Devices

Selfridge and Reiss [108] compared the use of a Wii controller to the use of a mouse for controlling a sound source in an audio mix. Two different pieces of music were mixed with each input device by a set of participants and were evaluated for speed and accuracy. They found that participants were less accurate and took longer to complete the tasks with the Wii controller technique. In addition to this evaluation, a separate listening test on separate participants was conducted to test whether they could differentiate the interaction techniques from the mixes created by the earlier participants. These results showed that there was a significant preference for conventionally mixed pieces over those made with the Wii controller based interface [108]. In a study by Melchior et al. [START_REF] Melchior | On the Use of a Haptic Feedback Device for Sound Source Control in Spatial Audio Systems[END_REF], a comparison was conducted between a desktop mouse and a six DoF haptic device for the creation of audio trajectories in a 3D space. Participants were asked to create trajectories by separately and simultaneously defining position and velocity. Participants were asked to rate different heuristics about each interaction technique. Results revealed better predictability and manageability with a mouse even though participants preferred the haptic device overall [START_REF] Melchior | On the Use of a Haptic Feedback Device for Sound Source Control in Spatial Audio Systems[END_REF]. The creation and editing of audio trajectories were also explored by Garcia, Bresson, and Carpentier [START_REF] Garcia | Towards Interactive Authoring Tools for Composing Spatialization[END_REF] for the OpenMusic software. Using interviews and participatory design, they developed a visual representation of 3D audio scenes and trajectories and Open Sound Control (OSC) functionalities to control positional parameters with a GameTrack input device [START_REF] Garcia | Towards Interactive Authoring Tools for Composing Spatialization[END_REF].

Multi-Touch Interfaces

In a study by Favory, Garcia, and Bresson [START_REF] Favory | Trajectoires: a mobile application for controlling and composing sound spatialization[END_REF], a multi-touch interface was explored to draw and edit 3D audio trajectory parameters within the OpenMusic software. This was developed through participatory design and subjective feedback from participants. Multi-touch interfaces have also been used to design and evaluate new interaction techniques for spatial audio mixing. For instance, Carrascal and Jordà [START_REF] Carrascal | Multitouch interface for audio mixing[END_REF] designed a multi-touch interface with surround sound rendering capabilities. In their study, participants were given four audio signals to complete a standard mixing task that they completed faster with the multi-touch interface, preferring it to the standard mixing interface. A multi-touch surface was proposed by Bredies et al. [START_REF] Bredies | The multi-touch soundscape renderer[END_REF] for controlling audio objects in the SoundScape Renderer application, which controlled audio objects through 2D positioning and the use of menus. They proposed two different methods of menu controls tailored to single and multiple users, but user evaluations were not conducted for a comparison analysis. Gelineck et al. [START_REF] Gelineck | Towards an Interface for Music Mixing based on Smart Tangibles and Multitouch[END_REF] also investigated the use of a multi-touch interface for 2D interaction comparing it to a traditional mouse-based GUI interaction, a tangible object to control position and orientation, and a smart tangible object with extra sensors that measure how users grasp the object. Participants in the study preferred the use of the tangible-based user interface over the multi-touch and mouse-based GUI design.

Consumer products for 3D audio typically have Object-Based interaction techniques where they interact with the spatial interactive parameters of the audio object regardless of the speaker configuration, but many of them are designed to be incorporated into Channel-Based production systems, like standard mixing consoles and DAWs. This configuration can make the production process difficult and complicated due to users needing to switch between the two paradigms. Alternatively, researchers have proposed Object-Based libraries and GUIs for customizable systems. Within this interaction paradigm, new control techniques were evaluated with tasks that resemble practical audio mixing scenarios. However, the creation and edition of 3D audio trajectories has not been equally investigated, while it is an important functionality for 3D audio due to the temporal nature of audio. Even though these studies provided interesting evaluation methods by comparing to traditional techniques, the majority of them provided inconclusive subjective results. However, various research into interaction techniques with spatial parameters and their evaluation methods in other domains could be applied for 3D audio user interfaces due to their similar spatial manipulation tasks.

3d user interfaces

3D user interfaces in general have been studied for some time now. More recently, there has been strong interest, both from industry and academia, to develop user interfaces for 2D visual displays and 3D virtual reality displays. These works are interesting in the context of 3D audio due to the similar tasks of controlling spatial parameters. Jankowski and Hachet [START_REF] Jankowski | A survey of interaction techniques for interactive 3D environments[END_REF] classified the state of the art of 3D interaction techniques from a general point of view for interactive 3D environments by proposing that all 3D user interfaces need to provide interaction techniques for Navigation, Selection & Manipulation, and System Control. These basic tasks can be seen within 3D audio tasks when it comes to interaction with the 3D audio scene and listener location (Navigation), selecting and editing the spatial parameters of audio objects (Selection & Manipulation), and switching between various modes like playback, editing, or recording (System Control). In the context of this study, we focus on commercial products and research of 3D user interfaces that are related to 3D audio tasks: Manipulation of Spatial Parameters, Drawing & Editing Trajectories, and Evaluation Methods.

Manipulation of Spatial Parameters

3D audio user interfaces enable the manipulation of spatial parameters of audio objects, but the interaction with 3D spatial parameters has been studied in more depth in the context of 3D user interfaces for virtual objects. For 2D GUI and mouse based interaction, direct manipulation of 3D graphical objects using 3D cursors were proposed by Nielson and Olsen [START_REF] Nielson | Direct manipulation techniques for 3D objects using 2D locator devices[END_REF], and more advanced techniques, such as ArcBall [START_REF] Shoemake | ARCBALL: a user interface for specifying threedimensional orientation using a mouse[END_REF], have also been proposed. With techniques like these, users can select widgets to manipulate the geometric and spatial parameters. These are popular methods that have been implemented in open-source tools, such as OpenGL, and commercial tools, such as the Jitter visualization library for Max 14 and AutoDesk 3DS Max 15 . The use of touch-based interfaces has also been explored for 3D manipulations. Martinet, Casiez, and Grisoni [START_REF] Martinet | The design and evaluation of 3D positioning techniques for multi-touch displays[END_REF] proposed to separate the control of the z-parameter along with the use of multi-touch viewport for 3D positioning, and they proposed the DS3 interaction techniques to explore separating the control of position and orientation in a separate study [START_REF] Martinet | Integrality and separability of multitouch interaction techniques in 3D manipulation tasks[END_REF]. Cohé, Dècle, and Hachet [START_REF] Cohé | tBox: a 3d transformation widget designed for touch-screens[END_REF] presented the tBox that provides new 3D transformation widgets to manipulate 3D spatial parameters. In addition, the use of high-DoF input devices, such as multi-sensor gloves (MITS Glove, Polhemus Microsensor, CyberGlove, etc.), advanced joystick devices (Space-Ball, Space Mouse, etc.), and armatures devices (MicroScripe, Phantom haptic devices, etc.), have been examined for the performance of manipulating 3D virtual objects [START_REF] Zhai | User performance in relation to 3D input device design[END_REF]. Motion tracking through camera devices, such as the 6D Hands, have also been explored for 3D manipulation [START_REF] Wang | 6D hands: markerless hand-tracking for computer aided design[END_REF]. These interaction techniques have strictly focused on the manipulation of 3D virtual objects within a 3D virtual scene, and have provided different performances in completing basic manipulation tasks. These tasks can be found in 3D audio production tasks as well, such as manipulating a sound source's 3D position, orientation, and acoustic parameters (i.e., directivity, timbre, etc.). However, 3D audio production requires the ability to alter these parameters in time, which creates additional complications that can require interaction with an extra dimension. Even though DAWs and other production systems allow users to define how a single parameter changes in time through drawing and editing automation parameters [START_REF] Metatla | Audio-haptic interfaces for digital audio workstations[END_REF], defining and editing the 3D position of sound sources in time (3D audio trajectories) with more advanced technqies is a feature that has an interest for 3D audio. It is closely related to the task of path and trajectory drawing found in 3D user interfaces for virtual reality. In their survey of interaction techniques for 3D curves, Jacob et al. [START_REF] Jacob | Un espace de caractérisation pour l'édition de courbes à trois dimensions[END_REF] classified creation tasks as sequential, defining constraints, techniques that infer drawing gestures, and choosing elements from a list. They also categorized editing tasks as selection, manipulation, and deletion of control points (keyframes) of trajectories [START_REF] Jacob | Un espace de caractérisation pour l'édition de courbes à trois dimensions[END_REF]. Sequential techniques define trajectories with drawing techniques [START_REF] Zeleznik | SKETCH: an interface for sketching 3D scenes[END_REF], and defining constraints are techniques that define points along the line [START_REF] Bruno | Mockup builder: direct 3D modeling on and above the surface in a continuous interaction space[END_REF]. Inference techniques interpret user-drawn strokes or constraints to automatically develop the curve [START_REF] Sachs | 3-draw: A tool for designing 3d shapes[END_REF]. Choosing elements and templates provides predefined trajectories, which can be seen in software such as Iannix [START_REF] Jacquemin | Iannix 0.8[END_REF]. Editing is a comprehensive task that consists of selection, manipulation, and deletion of control points. Techniques specifically for drawing focused on the selection of control points or constraint parameters [START_REF] Bruno | Mockup builder: direct 3D modeling on and above the surface in a continuous interaction space[END_REF], which are then manipulated or deleted. Many of the proposed techniques have focused on desktop interfaces, but tangible interaction techniques, such as ShapeTape [3], have also been proposed and investigated as methods of editing 3D curves.

Evaluation Methods

In evaluating these interaction techniques, quantitative and qualitative methods have been common practice to measure usability. Informal user feedback has been used as popular method to quickly evaluate interaction techniques through opinions and comments of participants [START_REF] Jankowski | A survey of interaction techniques for interactive 3D environments[END_REF]. Another common method for all interfaces is to investigate how well user interfaces comply with usability heuristics [START_REF] Nielsen | Usability Engineering[END_REF], which has also been extended specifically for 3D user interfaces [START_REF] Alistair | Heuristic evaluation of virtual reality applications[END_REF][START_REF] Alistair | Evaluating the usability of virtual reality user interfaces[END_REF]. The goal of these evaluation methods is to gather qualitative understanding on the factors being tested. It can provide quick results to highlight which factors are important for the task or interaction technique. On the other hand, performance measures investigate the design of the interaction technique in performing specific tasks. Previous work investigated the task of moving a 3D virtual object from one point to another (docking task) by measuring accuracy, precision, and completion time. In addition to these, degree of freedom coordination measures have been used by computing the ratio of the path that the user took over the shortest path possible [START_REF] Zhai | Quantifying coordination in multiple DOF movement and its application to evaluating 6 DOF input devices[END_REF] or through analyzing how multiple-dimensions were controlled simultaneously by the user [START_REF] Robert | Integrality and Separability of Input Devices[END_REF][START_REF] Martinet | Integrality and separability of multitouch interaction techniques in 3D manipulation tasks[END_REF][START_REF] Maurice | Measuring the allocation of control in a 6 degree-of-freedom docking experiment[END_REF][START_REF] Veit | Influence of degrees of freedom's manipulation on performances during orientation tasks in virtual reality environments[END_REF][START_REF] Veit | An experimental analysis of the impact of touch screen interaction techniques for 3-d positioning tasks[END_REF]. These measures help evaluate the interaction by how accurate, precise, and fast participants can complete the task, and they can help determine how well participants can control all parameters simultaneously to better understand user behavior. Though the ratio of path taken over the shortest path provides a useful general measure for comparison, analyzing how many dimensions are controlled simultaneously provides more insight into how users perform the task with the interaction technique. Common practice has been to combine these measures with subjective informal feedback or heuristics ratings to provide a more comprehensive analysis of interaction techniques.

Informal user feedback and heuristic measure have been popular evaluation techniques for 3D audio user interfaces, while performance measures have not been. Many of the studies on 3D audio user interfaces have used an audio mixing scenario, allowing participants to explore within that scenario rather than explicitly defining a common and practical task in that scenario for a performance measure. This was likely chosen due to performance measures being conducted on focused tasks, while creative user interfaces require the need to explore and experiment [START_REF] Terry | Recognizing Creative Needs in User Interface Design[END_REF]. However, performance measures can be useful for evaluating the usability, and we suggest applying quantitative evaluation methods similar to the studies of general 3DUI for 3D audio user interfaces. This requires a priori knowledge of tasks and design factors specific to 3D audio production, which we present in the following chapters.

state of the art overview

When creating an audio object, there are two paradigms that a practitioner can choose from. One paradigm is the method of capturing and reproducing an acoustic space (Field-Based), and the other allows users to models a 3D audio scene by placing individual sound sources in different locations (Position-Based). With either method, both the recording and rendering method need to be considered to understand the spatial quality that will be perceived. There are also two different paradigms when interacting with the audio object's parameters. Channel-Based requires users to control the low-level signal processing and routing of the audio object for it to be rendered spatially to a specific format, while Object-Based allows users to focus on the spatial parameters of the audio object which processes and routes it automatically to any format. The majority of 3D audio tools provide Object-Based interaction, but many of them are designed to be used in a Channel-Based production system for a limited number of formats that provides a complicated mental model of interaction. In addition, advanced techniques for the creation and edi-tion of 3D audio trajectories have not been largely explored compared to audio mixing tasks. 3D audio user interface development has also lacked in comprehensive evaluation methods in many of their studies, which may be the cause of inconclusive results from participants. However, similar tasks and functionalities have been evaluated in more detail for 3D user interfaces in other domains, but this requires knowledge of common production tasks for it to be adopted for evaluation of 3D audio user interfaces. Overall, this survey of the state of the art identified aspects of 3D audio user interfaces that have not been explored in greater detail:

1. How to classify the spatial properties of an audio object?

2. How to comprehensively analyze the usability of 3D audio user interfaces?

3. What are required tasks supporting 3D audio practices and methods?

In the following chapters, we'll address these questions through a presentation and analyses of results from interviews and a larger online survey with professional 3D audio practitioners. In addition, we overview the auditory and visual spatial perception and how they are used within 3D audio production settings. With the ethnographic results from practitioners and the spatial perception overview, we present work that classified production methods through a higher-level morphological analysis that defines the spatial properties of an audio objects, how to analyze its control capabilities, and the tasks required to produce 3D audio content.

S T U D Y O F C U R R E N T P R A C T I C E S
Despite a wide variety of tools being available to produce 3D audio content, it is still unclear how practitioners spatially characterize and interact with audio objects. For this reason, we conducted interviews and an online survey with 3D audio practitioners to better understand common 3D audio practices. The results indicated that practitioners consider the recording and rendering method when creating audio objects as well as limitations and opinions on the workflow design in 3D audio production tools. The information gained from these studies highglight needs for improvement within rendering, mixing functionality, visual feedback, and workflow of current production systems.

previous surveys and studies

Gathering ethnographic information on practitioners has been a useful method for improving and designing interactions techniques [START_REF] Mackay | The Interactive Thread: Exploring Methods for Multi-disciplinary Design[END_REF]. This type of study can be administered through interviews, surveys, questionnaires, or observational sessions, and many of these methods have been previously used to gather information on 3D audio practices. In two separate studies, interviews with contemporary composers who use OpenMusic and spatialization identified a need for temporal visual feedback of the spatial scene in 3D audio composition [START_REF] Favory | Trajectoires: a mobile application for controlling and composing sound spatialization[END_REF][START_REF] Garcia | Towards Interactive Authoring Tools for Composing Spatialization[END_REF]. Using a critical incident technique as described by Mackay [START_REF] Mackay | The Interactive Thread: Exploring Methods for Multi-disciplinary Design[END_REF], these interviews resulted in the design of new visualizations to inform how sound sources change in time [START_REF] Garcia | Towards Interactive Authoring Tools for Composing Spatialization[END_REF] and an initial design of a mobile multi-touch interaction techniques for drawing and editing of spatial audio trajectories [START_REF] Favory | Trajectoires: a mobile application for controlling and composing sound spatialization[END_REF].

In a more general approach, Peters, Marentakis, and McAdams [START_REF] Peters | Current Technologies and Compositional Practices for Spatialization: A Qualitative and Quantitative Analysis[END_REF] used an online survey for musical composers who use 3D audio technology to collect information on their tools, methods, and limitations. Their results show that the technical properties of their tools were an important consideration when composing, such as the need for real-time audio rendering, an improved bussing functionality, and 3D visual representation. A similar methodology with an online survey was used to gather information on how to archive spatial audio productions for future performances. Boutard and Guastavino [14] tested whether similarities exist among participants regarding how they organize their productions, and proposed a framework that classifies required information for archiving. Their results found that technical skills, specifications, mapping strategies, characteristics of the performance venue, and perceptual effects are essential to perform a spatial audio composition [14].

A majority of these studies identified specific limitations in the practices of 3D audio, with some of them proposing interesting solutions to those issues. Even though they focused their surveys and interviews for a specific application of 3D audio production, their techniques provided useful insights on how to gather exploitable results from the responses. By borrowing these methods, we conducted an initial study through interviews that focused on how professional mixing and recording engineers create and use audio objects, which was then extended to general practitioners of 3D audio production through an online survey.

interviews

As mentioned in Chapter 2, audio objects were initially proposed for file formats to describe sound sources' spatial parameters for flexible rendering across different listening formats. From an interactive point of view, audio objects have mostly been described as a single point sound source that can be positioned using Position-Based methods. Other methods, such as Field-Based techniques, have been developed to encode spaciousness into the audio to be decoded onto various speaker configuration [START_REF] Malham | 3-D Sound Spatialization using Ambisonic Techniques[END_REF]. However, Peters, Marentakis, and McAdams [START_REF] Peters | Current Technologies and Compositional Practices for Spatialization: A Qualitative and Quantitative Analysis[END_REF] reported issues with the bussing architecture of DAWs and how many 3D audio tools use "input devices that are tailored to the specific needs of controlling spatialization". In addition, concerns that there is not a convenient data structure or tool for 3D audio content production were mentioned from a round table discussion at the FISM 2013 conference in Paris, France. Participants in this discussion also emphasized the importance of capturing 3D audio content in a correct manner for cinema and broadcasting industries [START_REF] Nicol | Round Table Discussion: 3D Sound and its application[END_REF]. Even though Geier, Ahrens, and Spors [START_REF] Geier | Object-based Audio Reproduction and the Audio Scene Description Format[END_REF] provides an insightful classification of audio objects that can be adopted for interaction purposes, there is still limited information on the factors 3D audio practitioners consider in creating and manipulating 3D audio content. To gain a better insight into this, we conducted informal interviews with professional audio engineers on the topic of how they construct and interact with audio objects to search for similarities among their methods.

Methodology

Sessions with two experienced professional audio engineers, E1 and E2, along with one student of E1 were conducted in their place of work. Each session began with asking the participant to show a current project that used 3D audio techniques, which resulted in an informal discussion and demonstration of how they use their tools. Observation and key points from these discussions helped better clarify audio objects and how these engineers use them.

Findings from Interviews

E1 presented his work on experimenting with different types of High Order Ambisonics (HOA) microphones and comparing their use for broadcasting concerts to multi-channel and binaural listening formats. Using a traditional DAW, E1 uses custom-built plug-ins to properly render and compare each recording technique. An example of this method was further observed through a project of E1's student. Within this project, the recording techniques were mixed and rendered on multiple audio channels in a traditional DAW. The routing of the channels was designed to play the mixed production on three different loudspeaker formats and headphones. The project was created through a standard mouse and keyboard setup, but the student mapped a MIDI mixer controller to interact with the volume levels and quickly switch between the auditory display designs.

E2 demonstrated a similar channel routing design in the project example when producing content across different listening formats. E2 also uses a traditional DAW with specific plug-ins and a standard mixing console for audio mixing, but he mentioned working closely with the recording engineer for detailed notes on how the audio was recorded. He also provided an interesting critique on how current system provide "no relationship of audio objects with output file-formats". This suggests an inconsistent mental model between the recording methods and the fileformats associated with a speaker configuration, which is likely due to the difficulty of switching between Object-Based and Channel-Based interactions techniques during production.

Combining comments by the engineers with previously mentioned needs and limitations, similarities are highlighted in Figure 3.1. Comments about Recording/Rendering (red), Mixing (green), and Monitoring (blue) are classified to show a need to understand how the audio was recorded, how it can be rendered, that mixing controllers are too inflexible or limiting, and complicated routing/bussing architectures are needed for popular DAWs software to produce across different listening formats.

Recording/Rendering

Understanding how audio is recorded provides information on how it can best be rendered, which then defines the manipulable spatial parameters for mixing. This requires each recording to be considered independently in the bussing architecture to be rendered properly. This was obvious in the project of E1's student, but E2 also pointed this out when discussing the notes he received from his recording engineer.

Mixing

Adjusting the spatial parameters of each audio recording is typically done through the mouse and keyboard, but occasionally it is done with knobs, faders (sliders), and other controllers. A major part of this process is to correctly route the audio channels for proper rendering onto a specified loudspeaker format. However, the engineer must adjust the mixes and sometimes restart the routing process for every configuration as seen in the student's project where he needed to route and decode the same HOA recordings for different listening formats.

Audio-Objects

Recording/ Rendering

Mixing

Monitoring

Producing the same content onto different speaker configurations was a major part of the works of both E1 and E2, which requires constant monitoring by the user. Continuous monitoring of the routing design and outputs is required, which was seen in the student's project through level meters for each stage of routing from the audio object to the listening format.

In addition, these classifications help specify important concepts in audio production stages. Figure 3.2 explains this visually by showing how the interview participants define and interact with audio objects within an audio production workflow. Participants consider both the Recording/Rendering that define interactive spatial parameters for Mixing. The audio is then rendered and participants use Monitoring techniques to evaluate the listening experiences. The importance of the Recording/Rendering in every project highlights their connection to each other in defining the spatial parameters of an audio recording. This matches the definition of audio objects by Geier, Ahrens, and Spors [START_REF] Geier | Object-based Audio Reproduction and the Audio Scene Description Format[END_REF], which classify audio data and their spatial parameters with their recording and rendering techniques. Even though these results provided interesting insight into methods for creating and manipulating audio objects, they were limited to just a few audio engineers. In addition, discussions were informal and specifically about 3D audio projects for broadcasting. There are many other professions that use 3D audio for their projects, which led us to design and conduct and online survey for general 3D audio practitioners.

online survey

Building upon the results of the interviews, the online survey widened and increased the participant pool to find general trends among many types of 3D audio practitioners. The results of the survey identified more limitations within the flexibility and functionality of recording and rendering techniques, interaction design, and workflow design.

Methodology

Professionals were recruited through emails to academic institutions and online communities related to spatial audio. Responses were accepted for 3 months, during which they could edit responses at anytime within that period. The questionnaire was divided into five sections:

1. Profession and Experience -to gather demographic information about the participants, such as profession, age, location, gender, roles they take during projects, and years of experience in using spatial tools.

2.

Hardware and Software Tools -to gather a list of current tools participants use for spatialization.

3.

Composing/Planning and Editing -to gather information and opinions on how participants sketch, compose, and edit spatial parameters.

4. Notation and Performance -to gather information on how participants visually express, communicate, and perform spatial ideas in their projects.

5.

Listening/Monitoring -to gather insight into their methods and tools' capabilities for listening and monitoring spatialization choices.

Questions were recorded as multiple-choice with only a single response, multiple-choice with multiple responses, Likert Scale (1-5), and free responses. Most multiple-choice questions gave the opportunity to provide an answer different from the choices provided. The survey concluded with an opportunity for participants to provide additional comments and feedback, but participants were not required to answer every question (refer to Appendix A for the full questionnaire). The multiple-choice questions were analyzed through percentage of selected responses, and free responses were analyzed to find similarities. Analyzing across all of the responses, we present findings across the Demographics of Participants, Audio Rendering, Visual Feedback, Functionality, and Workflow Integration.

Findings of Survey

Demographics of Participants

The survey had 48 participants (47 male and 1 female) from Europe (62.5%), North American (27.1%), Australia (4.2%), and Jordan (2.1%). Ages of participants varied between 18 and 55+ years old with 29.2% having 0-4 years, 22.9% having 5-10 years, and 47.9% having 10+ years of experience. Many participants chose that they take on multiple roles in their projects, and use spatialization for a variety of projects, listening formats, and virtual spaces (see Figure 3.3). The majority of participants identified themselves as Music/Audio Researchers (62.5%), Composers (58.3%), Recording/Mixing Engineer (50.0%), and Sound/Installation Designer (41.7%) with many identifying themselves with multiple roles. Types of projects identified were Record Music Performances (66.7%), Live Music Performances (56.3%), Sound Installations (56.3%), and Computer-Aided Compositions (35.4%). Many indicated using spatialization techniques on Stereo (58.3%), Surround Sound (54.2%), Loudspeaker Arrays (47.9%), 3D Multichannel Reproduction (43.8%), and Bin- aural (35.4%) listening formats, and they indicated using Simple Cubic Rooms (45.8%), Speakers (39.6%), Theaters or Auditoriums (31.3%), and Spheres (29.2%) as virtual spaces for those projects. In reference to their hardware and software tools, all participants mentioned using computers, but they also use other types of hardware in their production setups, such as physical controllers (47.9%), multi-channel mixers (39.6%), multi-touch tablet (33.3%), and effects processors (29.2%). In regards to software, participants indicated using DAWs (45.8%), textual (16.7%) programming languages, and visual (27.1%) programming languages. However, visual programming languages, MaxMSP and PureData, were heavily mentioned when asked to list types of software used for 3D spatialization, which is most likely due to the survey being advertised at institutions and online communities where these softwares are commonly used. In addition to those languages, many participants also listed various types of plug-ins for DAWs, ambisonic-based software, and programming libraries for spatialization (see Figure 3.4). The variety of projects, listening formats, virtual spaces, and roles identified suggest that there is a need for flexible tools. In addition, the variety of low-level software referenced, the high level of experience, and the technical roles of many participants suggest that many of them have a strong technical background in spatialization technologies. well... but [cannot] get sounds to come close to the listener" and that Distance-Based Amplitude Panning (DBAP) leaves a "bigger gap between speakers than Ambisonics" (P 18 ). Another participant mentioned that "binaural rendering needs head tracking to accurately experience the 3D audio project" (P 6 ). Specific limitations within the design of rendering systems were mentioned as well, including how some were designed for listeners to be in the center of the configuration (sweet spot) P 38 & P 45 , and that there is a need for "easy adaptability of 'completed' work from one playback space to another" (P 35 ). This highlights the lack of homogeneity in 3D audio rendering systems, which therefore requires the design of specific tools to be flexible from one system to the other. Addressing these issues would improve the accuracy in the intended spatialization and the flexibility to compose for various systems and rooms. In addition, an improvement requires more flexible user interfaces that can be adapted to interact with productions across a variety of systems, formats, virtual spaces, and projects.

audio recordings:

Fifty percent of the participants identified with the role of Recording/Mixing Engineer in their projects (see Figure 3.3(a)), and some listed the use of proper microphone placement to capture spaciousness in their audio recordings (see Figure 3.4). In addition, participants commented on the use of advanced recording techniques, associated limitations, and their effects on the listening experience. For instance, a few participants described using ambisonic recordings to capture space: "In my model, sounds are usually treated as spaces rather than as point sources. This is because I mostly work with stereo or [ambisonic] recordings of environmental sound[s]. Occasionally a sound is defined as a point or as a directional source" (P 6 ). In addition, P 35 noted the use of "ambisonic recording of a particular environment..." to create a listening experience that "is hopefully very true to the recorded environment". However, one participant pointed out the inability "to adapt [ambisonic] recordings to non-standard reproduction systems, and being able to use them as a source with richer and more detailed spatial information, while not being tied to the sweet-spot regime" (P 8 ). In addition, one participant mentioned that there are "difficulties to qualify [recording methods] and the listening experiences" (P 38 ). Even though recording techniques provide capabilities to capture and encode spaciousness in the audio data, there is a need to improve their flexibility and adapt them to custom speaker configurations.

Visual Feedback

Visual feedback provides the representations that allow one to monitor and evaluate objects of interest. For spatial audio parameters, visual representations of position, orientation, directivity, and time provide important information on the state of the values being entered into the renderer. Participants noted that most of their authoring tools provide a 2D visual representation (68.8%), loudness levels (54.2%), and 3D visual representation (47.9%) to represent source position or movement (see Figure 3.6(a)), but many of them do not use visual representation for orientation (37.5%) or directivity (41.7%). Some participants (22.9%) noted not having any type of visual feedback for trajectories, but 62.5% of the participants indicated that their tools display a moving point in space to represent trajectories (see Figure 3.6(b)).

Even though many noted having some type of visual feedback available, their issues with regard to their design were mentioned in free responses: "deceptive spatialization tools (for example, panners which suggest sounds can be positioned in the center of an array)"(P 14 ); "many tools to monitor the spatialization in [MaxMSP,] but all these tools have advantages and drawback[s] and I'm forced to use several tools to monitor (and sometimes reedit) the spatialization"(P 10 ). In addition, some participants felt that their visualizations lacked a "direct view of the surround speakers from audience positions"(P 35 ), representations of the "position of audience members in the room relative to speaker placement"(P 35 ), and "quality of the recordings and the decoders"(P 38 ). A lack of interactive features was also described in order to better comprehend the visualizations in the feedback, such as the ability "to zoom in a specific area or to choose to visualize only one or a bunch of selected sources"(P 38 ) and "efficient visualization interfaces and a good gestion of time (easy zooming in a specific part of the timeline)"(P 26 ) for notating spatialization. Addressing these limitations and suggestions can help with the effectiveness of visual feedback design, but an overall need in visual feedback can best be described as "a proper visualization for multichannel setups that displays what is happening in a room in a way that is easy and fast enough to read while providing sufficient detail"(P 8 ).

Authoring Functionality

In addition to the concerns with visual feedback, comments highlighted intentions and issues in producing 3D audio content, which we categorized into methods and limitations within Trajectories and Input Devices.

trajectories:

As we discussed before in Chapter 2, there has been interests in developing the interaction techniques to define 3D audio trajectories due to the temporal nature of audio production. For this, we asked the participants several questions about how they create and edit trajectories. Various patterns and methods for inputting and editing parameters of trajectories were indicated (see Figure 3.7), and participant comments highlight inconsistencies in composing and editing methods. For predefined patterns, participants indicated using circular path in azimuth (64.6%), line (56.3%), circular path in elevation (33.3%), and spiral (25%) trajectory patterns (see Figure 3.7(a)). When not using patterns, 31.3% selected position and velocity at the same time, 27.1% selected automation lines not in time, 25.0% input each positional parameter in time, and 18.8% input path and velocity separately (see Figure 3.7(b)). Figure 3.7(c) shows how participants chose methods similar to the ones they use to initially input parameters for editing trajectories, such as redraw the automation lines not in time (25.0%), re-record automation lines for each parameter (25.0%), redraw path or control velocity separately (25.0%), and redraw the path and velocity at the same time (22.9%).

Participants indicated interacting with the parameters both in realtime and not in real-time when producing (18.8%) and editing (16.7%) trajectories, which was highlighted by one comment on how the participant "often [makes] a recording or two, and then just adjust automation" (P 43 ). Some difficulties for producing and editing trajectories when only using a computer for production were also mentioned: the "process to define source trajectories is very time consuming and not very intuitive" (P 11 ); "It can be rather time consuming [to generate] dynamic sound scenes since the changes in source properties must all be [inputted] in a somewhat unintuitive manner by hand" (P 36 ). Others mentioned some missing functionalities, such as: "controlling trajectories of groups of sounds with a kind of group behavior." (P 44 ); "tools with more options of controllable random position in controllable area, probabilistic control, typical trajectories such as line, circle, ellipse, square, rectangle, go back and forth..." (P 26 ). Overall, these results suggest that many interaction methods to define and edit trajectory parameters are under developed, especially for standard mouse-desktop interaction due to the limited number of inputs.

input devices:

Limitations in defining and editing trajectories may be partially due to the input devices or control methods. Participants indicated using software controllers (52.1%), physical controllers (39.6%), textually (29.2%), drawing (33.3%), and pointing (25.0%) devices to control trajectory parameters (see Figure 3.7(d)). In the context of performance, the majority of participants selected to perform spatialization through computer software (54.2%), pre-defined in the software (25.0%), or with midi-controllers (22.9%), but there were participants who mentioned general issues within the input devices they use. One participant, P 40 , felt that the typical computer mouse was limiting and that a tactile interface would offer more degrees of freedom. Others mentioned that "distance control" (P 31 ) and "3D gestural controls" (P 3 ) were limiting factors of their input devices. Even though more standard mixing interfaces are now including advanced input devices (see Chapter 2), many participants still find their functionality limiting in general. This was summarized best by one participant who noted "comprehensive interfaces are lacking in control of the many aspects of sound fields and the sources within them" (P 39 ).

Workflow Integration

Even though DAWs are widely used for spatial audio production, many participants also use programming languages and their dedicated libraries (see Figure 3.4). In addition, some comments highlighted difficulties in audio spatialization with DAWs: "DAWs are inadequate tools. Having to arrange rendered audio to suit a specific number of output channels is tedious in the extreme" (P 23 ); "tools [are] still often experimental... [and] difficult to integrate with classical tools like [ProTools] or other sequencers" (P 26 ). The complexity can become greater with larger amounts of audio due to the "huge amount of data to control [all the] objects to spatialize" (P 47 ). Even though there are difficulties within DAWs, their usefulness as an editing tool was explained by P 7 who "use[s] the HOALibrary and ICST Ambisonics externals for MaxMSP to do the actual spatializing", but uses "Reaper for any editing (and some occasional spatializing with their built-in tools)". Another participant mentioned that their tool choices were dependent "on the production and purpose. For real time manipulation, [MaxMSP] is mostly preferred. For precise [non-real] time productions, [Nuendo] is a powerful tool" (P 18 ). There is still an attraction to use DAW software for authoring and editing 3D audio, but many of their designs have made it inflexible with configuring 3D audio renderers and interaction. Instead, many practitioners appropriate DAWs for basic audio production tasks and use specific 3D audio libraries and programming languages for spatialization rendering, which might interrupt and limit their creative workflow.

The results of this survey provided an interesting view in the understanding of practitioners' tools and methods. They also further enhanced the findings from the interviews by identifying specific limitations, reaffirming the importance in considering the rendering techniques in production, the heterogeneity of rendering techniques, and the tediousness of using traditional DAWs. In addition, it further detailed the limited interactivity of visual feedback, audio recordings, creation and edition of trajectories, and input devices. This provided more detailed responses that can be used to better improve the general design of 3D audio user interfaces and production systems.

informing design from user feedback

Overall, these findings helped to identify specific needs for improving audio recording and rendering, visual feedback, functionality, and workflow of 3D audio tools, which are summarized below as:

1. 3D Audio Rendering & Recording Flexibility -The heterogeneity of current 3D audio rendering and recording techniques cause specific limitations for moving productions across different listening formats.

2.

Comprehensive Visual Feedback -The visual representation within visual feedback of many authoring tools are not easily comprehensible, and there are limitations in helping users understand how the tool and its parameters affect the listening experience.

3. Limited Authoring Functionality -Current interactive 3D audio tools have limited functionality in the spatial manipulation of audio recordings, adaptation of various recording formats to renderers, authoring 3D trajectories, and the control capabilities of available input devices.

4.

Inflexible Workflow Integration -The lack of flexibility to integrate advanced rendering, authoring, and editing methods tailored for spatialization causes 3D audio tasks to be complex and tedious. This causes practitioners to custom configure various types of tools and methods to realize their creative ideas.

These findings can be individually used to inform new designs that address many of the current limitations, but further analysis can help better organize these needs to address the limitation of heterogeneity and workflow. In the next chapter, we investigate this issue in more detail through a review of spatial perceptual studies to better understand issues within audio and visual feedback.

When producing 3D audio content, practitioners use their spatial perception to analyze the spatial characteristics of the rendered 3D audio production and visual representations within 3D audio user interfaces. As indicated by the participants of the online survey in Chapter 3.3, practitioners mentioned issues with the ability of recording and rendering techniques to produce perceptions of auditory spatial cues and higherlevel spatial comprehension. They also mentioned difficulties in relating the visual spatial representation found in their user interfaces to what they spatially perceive when listening to the rendered outcome. These comments overall identified one of the major findings from our studies on practitioners, which was that comprehensive visual feedback in 3D audio user interfaces needs to accurately reflect the spatial perception when listening to the rendered production (see Chapter 3.4). This is important for advancing 3D audio productions which use object-based production methods and allow spatial mixing with 3D interactions, as it would require understanding the spatial visual and auditory perceptual factors involved when producing 3D audio content. Understanding the auditory spatial perception provides the basis of how practitioners understand the spaciousness of a 3D audio production, and the visual spatial perception is important to understand due to many user interfaces employing 3D graphics within their visual feedback. For this reason, we overview the auditory and visual cues that form spatial perception and how they are used to understand spatial characteristic of scenes and objects at a higher level. With this basis, we can better understand how practitioners evaluate and monitor their productions and tools and identify drawbacks in spatial comprehension within the design of current 3D audio production tools.

spatial perception

The understanding of 3D space is required for various simple and complex tasks in our daily lives. In a simple task, such as grasping a pencil off a desk, an individual needs to understand spatial concepts (size, shape, orientation, distance, etc.) to complete the task of moving one's arm, hand, and fingers to positions and orientations that allow the individual to complete the task. In a more complicated spatial task, such as playing the guitar, an individual needs to comprehend complex spatial relationships of the strings, strum area, and chord formations to play a note, chord, or a full song. Simple tasks become ingrained in our daily experiences and become easy to conduct without major difficulties, but complex tasks require more practice and training for individuals to conduct them with relative ease. Although movement and memory (proprioception kinesthesia) can play a significant role in spatial interactive tasks [START_REF] Goldstein | Sensation and Perception[END_REF], the primary information used to comprehend space comes from aural and visual cues. In the context of 3D audio production, aural spatial cues are the primary focus, but visual spatial cues have an important role during interacting and monitoring the production properties with 3D audio user interfaces. For this reason, we review how the auditory modality perceives the surrounding 3D space in detail and then overview how we perceive 3D objects through the visual modality. We follow this by reviewing how cues from both modalities provide higher-level spatial comprehension of a 3D environment and how practitioners use them when producing 3D audio content with current 3D audio production tools.

Aural Spatial Cues

Many people rely on visual perception to provide spatial comprehension in our daily lives, but auditory cues from our hearing provide information for understanding the surrounding auditory space. In the context of the 3D audio production, this is the main method of evaluating and monitoring the spatial characteristics of the audio production. In addition, this perception is important when designing visual feedback of user interfaces to accurately represent the auditory spatial perception. Goldstein [START_REF] Goldstein | Sensation and Perception[END_REF] described that one aspect of spatial listening that has been heavily studied is auditory localization, which describes the task of identifying the azimuth, elevation, and distance of a sound source with regards to a listener's head being the origin (see Figure 4.1). The ability to localize a sound source is dependent on auditory cues that describe how the sound waves travel to the ears and the content of the sound. These cues consist of differences between how sound waves arrive at each ear (Binaural cues), spectral filtering due to the shape of the ears (Monaural Pinnae cues), head and sound source movement (Dynamic cues), and properties that affect distance perception (Distance cues). Even though other factors, such as familiarity, can play a role in localization, previous studies have found that these principle cues have the most significant effect on the performance of localizing a single sound source [11]. binaural cues: These cues describe the differences of how sound waves from a source arrive at each ear, which studies have observed to have a strong effect on the ability to localize a sound in the azimuth dimension [11]. Begault [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF] explained how the anatomy of the listener creates these cues and affects how the sound waves arrive at the ear canals. When a sound source is directly in front of the listener's head, the sound waves will arrive at each ear at the same time with similar interference. When a sound source is located to either side of the head, like in the diagram in Figure 4.2, the sound waves arrive at the closest (ipsilateral) first before the arriving at the furthest (contralateral) ear. In addition, the head and body interfere the sound wave arriving at the further ear more than closer ear. The difference between the times of arrival of the sound wave is called Interaural-Time-Differences (ITD), and the difference in intensity is called Interaural-Level-Differences (ILD) or Interaural-Intensity-

Differences (IID).

There can be confusions in localization due to certain positions providing similar ITD and ILD values. This can occur when sound sources are located in the front or back, a phenomenon called front/back reversals, and when sound sources are located along the surface of the cone of confusion [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF]11]. The cone of confusion references a hyperbolic surface radiating outwards from each ear where positions along the surface produce similar ITD cues. The similarity of cues causes a spatial blur and creates difficulties in accurately localizing sound sources within this area [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF][START_REF] Mills | Foundations of Modern Auditory Theory[END_REF]. A majority of studies have tested localization blur to understand the localization precision in non-reverberant environments or anechoic room [11], but these cues can still hold true in many reverberant environments due to the precedence effect. This effect explains how we differentiate the arrival of a sound source's first sound wave front and from the following sound wave fronts caused by reflections within a reverberant room [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF][START_REF] Wallach | The precedence effect in sound localization (tutorial reprint)[END_REF]. In addition to this, it is important to note the phenomenon of Summing Localization, which describes how listeners perceive a single sound source from two spatially separated sound sources that emit identical sounds. If the ITD and ILD cues are below certain values, the listener will perceive a phantom source that is located between the two real sources. This is the basis of audio panning algorithms, which the majority of production systems use to position sound sources between speakers [11]. In binaural rendering on headphones, the ITD and ILD cues are implemented as a transfer function, called Head-Related Transfer Function (HRTF), that uses a set of HRTF filters in order to simulate the localization of sound sources from user defined positions [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF]. In general, previous work has identified that these cues provide the most precision in auditory localization, but they are limited due to the confusion possibilities.

monaural pinnae cues:

Studies have observed that reflections and filtering from the listener's pinnae have an effect on localization along the elevation dimension. In particular, the shape of the pinnae has a significant effect on the perception of elevation. Gardner and Gardner [START_REF] Gardner | Problem of localization in the median plane: effect of pinnae cavity occlusion[END_REF] studied the effect of the pinnae by comparing localization before and after smoothing the shape of the pinnae by occluding its cavities, which resulted in poorer localization in the elevation plane and no significant effect in the azimuth plane. In a similar study, Hofman, Van Riswick, and Van Opstal [START_REF] Paul | Relearning sound localization with new ears[END_REF] showed that changing the shape of the pinnae with ear molds had a negative effect on the perception along the elevation dimension. However, their results indicated that people can adapt to new spectral cues after a period of time and can return to understanding their original cues quickly when the inserts were removed [START_REF] Paul | Relearning sound localization with new ears[END_REF]. Even though the shape of the pinnae has been found to have an effect when localizing, listeners are less accurate along the elevation dimension than the azimuth dimension where ITD and ILD cues dominates [11].

dynamic cues:

Head and sound source movements provide changes in ITD and ILD that increase the capability to localize sounds. Begault [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF] explained that in natural environments we tend to move our head towards the direction of a sound source to acquire it visually. In doing this, the ITD and ILD values minimize and the spectral cues from the pinnae change, which help better localize and reduce the number of reversals coverage. If there is localization confusion, the movement of the head and how the ITD and ILD cues change provide more information on the location of the source. Movement of the sound sources also affects the cues in a similar manner and helps localize the sound, but transient characteristics of the sound also have an affect on the localization ability. A moving sound whose amplitude envelope rapidly changes (pulsed or rhythmic) is easier to localize than a sound whose amplitude envelope slowly changes (continuous sound). The reason for this is because rapid changes in the amplitude creates more significant binaural cues that can be processed due to the precedent effect. If the amplitude changes slowly, the ITD cues become blurred due to the difficulty in processing the precedent effect [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF]. The Doppler effect, which describes how the sound source's frequency content changes as it is passing by a listener, provides information that the source is approaching and receding. However, it has been observed to have little effect on localization and mostly informs listeners of the moment when the sources passes the listener (transition between approaching and receding) [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF][START_REF] Zahorik | Auditory distance perception in humans: A summary of past and present research[END_REF].

distance cues: A primary cue that affects the perception of auditory distance is the intensity or loudness of the sound source. Sound sources that are further in distance have lower intensity due to the attenuation of sound wave energy in distance. Another cue is the Direct-to-Reverberant energy ratio, which is the ratio between the energy of the direct sound waves to the indirect sound waves (see Figure 4.3) [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF][START_REF] Zahorik | Auditory distance perception in humans: A summary of past and present research[END_REF]. According to Zahorik, Brungart, and Bronkhorst [START_REF] Zahorik | Auditory distance perception in humans: A summary of past and present research[END_REF], the spectrum of the sound source, binaural cues, and motion cues provide information on distance cues as well. The distance attenuation of the sound also affects the spectrum of the sound when it arrives at the ear, where higher frequencies attenuate more quickly than middle to lower frequencies due to absorption properties of air. As well, it has been proposed that comparisons of ITD and ILD cues can help listeners perceive distance for closer locations, but it is still unclear how much it affects distance perception [START_REF] Zahorik | Auditory distance perception in humans: A summary of past and present research[END_REF].

Individuals rely on all these cues to localize sound sources in real-life (see Table 4.1), but binaural cues provide more precision on determining the direction of a sound source's position along the horizontal plane. However, localizing on the horizontal plane is more precise in the front and back of the listener compared to the sides, and there can be instances of front-back confusions and localization blur along the cone of confusion [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF][START_REF] Mills | Foundations of Modern Auditory Theory[END_REF]. Localizing in elevation is less precise and dependent on the shape of the pinnae, while motion cues depend on the change of both binaural and monaural cues [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF]. When testing auditory distance perception, results indicate that individuals tend to overestimate distances when sources are located within 1 meter and underestimate when distances are further than 1 meter [START_REF] Zahorik | Auditory distance perception in humans: A summary of past and present research[END_REF]. Understanding these factors that affect auditory spatial perception is important to consider when designing the auditory displays and rendering spaces so that they can produce accurate spatial cues in the listening experience of the audience. It is also important to consider this when designing 3D audio user interfaces for practitioners in order to better understand how they and audience members will perceive the spatial characteristics of the production. In production methods where practitioners can interact with 3D virtual spaces within their user interfaces, it is also important to understand spatial perception from the visual modality.

Visual Spatial Cues

The dominant sensation used for spatial perception is vision for normal sighted or corrected vision individuals. Light is absorbed by the eyes and processed to form 2D images, providing several cues within that image that inform perception of distance, depth, and size that come from monocular (single eye), binocular (both eyes), and motion cues [START_REF] Goldstein | Sensation and Perception[END_REF].

Monocular cues provide information of depth and distance because we perceive further objects to be smaller. This comes from cues of brightness (illumination), shadowing, shading, contour, and texture gradient that help provide a sense of distance and depth that help identify the size and shape of objects. From these priori cues, comparisons of objects provide further information of distance and depth. Focus distance cues show objects that are at the fixation point clearer in the image, while objects further away from the fixation point are blurrier. Perspective cues compare sizes of objects that change when an individual moves to different viewpoints. Depth and distance perception also comes from the familiarity of the size of objects and distances, which can have an influence on whether the object is closer or further away. Also, relative height of a familiar object in the 2D image provides information of object distances, and partially occluded objects provide information that they are further than the object that is causing the occlusion. When using both eyes, binocular cues of convergence and disparity of objects in the image also help provide a sense of distance. In these cues, an object is perceived closer when there are differences of the object within each eye's image (disparity) and objects that are closer require inward eye movement in order to fixate on the object (convergence) [START_REF] Goldstein | Sensation and Perception[END_REF]. During the moment an individual is moving, objects that are moving faster within the formed image are perceived to be closer compared to objects that are moving slower within the image (motion parallax). This is commonly experienced when a person traveling in a car or train looks out the window and sees closer objects, such as road signs, moving faster in the formed image compared to further objects, such as mountain ranges, moving slower [START_REF] Rogers | Motion parallax as an independent cue for depth perception[END_REF]. In the opposite scenario when objects are moving and the individual is stationary, depth and distance perception can come from kinetic depth effect cues caused by a rotating or moving object. This cue uses the changing of brightness, shading, contour, and texture to inform shape and size of the object [START_REF] Wallach | The kinetic depth effect[END_REF].

All of these combined cues help visually identify objects and locations in space (see Table 4.2), however monocular and motion cues are typically used for providing 3D spatial perception in computer graphics. Wanger, Ferwerda, and Greenberg [START_REF] Wanger | Perceiving spatial relationships in computer-generated images[END_REF] evaluated the effect of some of these cues on accuracy when determining the position, rotation, and size of virtual 3D geometric objects. For determining position, shadowing and perspective cues had an effect in improving accuracy. For determining orientation, perspective had a negative effect on accuracy, but motion and shadowing had a small positive effect. Lastly, cues from shadowing, motion, and texture gradient improved accuracy in determining size, but perspective reduced accuracy. Overall, they concluded that all cues were most likely needed to increase the accuracy of perceiving spatial relationships in 3D computer graphics even though some cues have more effect in determining either position, orientation, or size [START_REF] Wanger | Perceiving spatial relationships in computer-generated images[END_REF]. It is important to consider these factors when designing 3D visual representations. However, as the primary goal of 3D audio production is to provide an aural sense of spatial properties, it is important to know how audio objects, rendering spaces, and interaction with the user interfaces affect the listening experience.

Monocular

Higher-Level Spatial Relationships

In realistic scenarios, our surrounding environment is complex with many objects, and interacting with this environment requires understanding the spatial relationships of objects. We accomplish this through using spatial cues to organize objects in regards to the individual (egocentric point of view) or in regards to objects and spaces themselves (exocentric point of view). In doing so, higher-level definitions of the scene can be used that provide a language that is more practical at conveying spatial characteristics of the scene. This is important for the design of 3D audio production systems in order to understand how practitioners spatially comprehend 3D visual representations in user interfaces and spatial listening experience during playback of the 3D audio production.

In auditory perception, we use a process called auditory scene analysis to analyze the surrounding sounds and spaces from an egocentric point of view. Bregman [START_REF] Bregman | Auditory scene analysis: The perceptual organization of sound[END_REF] described this process to be when listeners perceptually organize the sound sources and space by grouping them based on their location, spatial characteristics, timbre (spectral characteristic), and reverberation. We differentiate sound sources based on their location and spatial characteristics that originate from our auditory localization capability, and we can differentiate sounds sources that have different timbre. In addition, a continuous sound is a strong indication that a single source produces it. Familiarity of sounds or spaces helps to distinguish them from other sound sources or spaces, while the reverberation of a room provides cues that describe the enclosed space. The timbre of late reflections (late reverberation) and the decay rate of the reverberation energy can help inform the size, absorptive/reflective properties of the room, and the location of a sound source within the room. However, studies have observed that highly reverberant spaces can negatively affect auditory localization ability [START_REF] Bregman | Auditory scene analysis: The perceptual organization of sound[END_REF][START_REF] Goldstein | Sensation and Perception[END_REF]. In addition, Rumsey [START_REF] Rumsey | Spatial quality evaluation for reproduced sound: Terminology, meaning, and a scene-based paradigm[END_REF] proposed scene-based spatial attributes to evaluate an auditory scene for evaluation purposes of auditory display through apparent width, distance, depth, and immersion attributes. Rumsey also provided definitions that describe the stability and focus of sources and scenes. Studies have also explored the use of other attributes to evaluate the spatial quality of auditory displays and rendering spaces [START_REF] Guastavino | Perceptual evaluation of multi-dimensional spatial audio reproduction[END_REF][START_REF] Kendall | The artistic play of spatial organization: Spatial attributes, scene analysis and auditory spatial schemata[END_REF][START_REF] Laurent | Perceptual attributes for the comparison of head-related transfer functions[END_REF]. However, these attributes evaluate spatiality differently than auditory scene analysis, which is more similar to how we use visual cues to analyze a spatial scene.

Individuals with normal or corrected vision typically focus on the visual cues to analyze the surrounding space. Studies have observed that this may be due to the modality-appropriateness theory [START_REF] Welch | Contributions of audition and vision to temporal rate perception[END_REF], which ar- gues that we weigh the precision of perceptual information from each modality to determine which modality to use to complete a task. This could explain how normal sighted individuals rely more on visual cues than aural cues for spatial perception due to their higher precision [START_REF] Wada | Audiovisual integration in temporal perception[END_REF][START_REF] Welch | Contributions of audition and vision to temporal rate perception[END_REF]. In using visual spatial cues, we can analyze a spatial scene through organizing objects based on points in space, objects, direction, and orientation. Klatzky [START_REF] Klatzky | Allocentric and Egocentric Spatial Representations: Definitions, Distinctions, and Interconnections[END_REF] classified these spatial concepts through two manners -egocentric and exocentric (allocentric). As mentioned before, egocentric comprehension considers the spatial cues with regards to the individual (see Figure 4.4(a)), and exocentric comprehensions compares spatial cues of objects within the space regardless of the individual (see Figure 4.4(b)). Within these viewpoints, Klatzky described that individuals analyze the spatial relationships by defining points (locations in space) and objects (multiple points that form a coherent entity). Using these definitions, we can form spatial values of distance between points or objects, object orientation, heading of an object (direction it faces), and bearing between two points (angle). In the egocentric view, the heading and bearing of objects describe the angle of the object's direction in relation to a reference formed by the individual's direction. In the exocentric point of view, these angles and spatial relationships are described with regards to an exocentric reference origin and direction. If an individual is observed from an exocentric point of view, the observed individual's position, heading, and bearing are treated like other objects in the space [START_REF] Klatzky | Allocentric and Egocentric Spatial Representations: Definitions, Distinctions, and Interconnections[END_REF]. This classification provides a useful language on how to visually analyze a complex space with regards to the individual or to the external scene, which can be used to describe the design of visual feedback that represents the 3D audio scene. Higher-level spatial definitions help describe the spatial perception from the aural and visual cues in a language that is more practical to real-life scenarios. The use of attributes like the ones proposed by Rumsey [START_REF] Rumsey | Spatial quality evaluation for reproduced sound: Terminology, meaning, and a scene-based paradigm[END_REF] can be seen as a way of interacting and evaluating properties, such as the interactive perceptual parameters in IRCAM's Spat [START_REF] Jot | Spatialisateur Ircam/Espaces Nouveaux User Manual[END_REF]. However, it is still uncommon in many commercial products. Instead, the majority of 3D audio production tools have focused on the development of the rendering techniques and auditory display to ensure a high quality of spatial perception in the production, and the majority of their user interfaces provide only basic spatial representations as visual feedback.

spatial perception in 3d audio production & authoring

The difference between visual and aural spatial perception can make designing user interfaces difficult. Spatial listening experiences are an egocentric spatial comprehension, but 3D spatial visual representations in current user interfaces with 2D visual displays provide an exocentric point of view of the audio objects within the rendering space. Since visual spatial cues are normally more precise than spatial aural cues, user attention will tend to focus on the visual representation in the user interfaces for spatial information. This can have an impact on the spatial comprehension during interaction and monitoring if visual representations do not accurately represent the auditory spatial perception since the primary goal is to produce spatial perception within the auditory modality. When interacting with audio object parameters during realtime playback of the audio, like in the scenario of recording automation parameters, users can ignore the visual representation in the user interface and focus on the interaction with the input method and listening experience. When examining how many standard production systems, such as DAWs, provide authoring functionalities, users focus their interaction with the spatial parameters within a GUI and then listen to the rendered audio playback afterwards [START_REF]Authoring for Dolby Atmos Cinema Sound Manual[END_REF]. This can create a complicated spatial task that is exocentric during interaction with the user interface, and then egocentric when listening to the rendered outcome. This requires two spatial comprehension sub-tasks that practitioners must evaluate with different perspectives. In addition, the spatial comprehension of the 3D visual representation in the user interface may not accurately represent the spatial comprehension when listening to the 3D audio production, as mentioned by participants of our online survey in Chapter 3.3. This can add another level of complexity to the design of 3D audio visualizations due to the variability of rendering space properties and limited understanding on how to visualize spatial listening experiences. We did conduct a study that investigated a common scenario for 3D audio production of how practitioners relate the 3D computer graphics in their visual feedback with their spatial aural perception of listening experiences, but the design of the experimental user interface introduced a bias that encouraged participants to map the target modality to the response mechanism rather than attempting to relate the two modalities. This study provided preliminary results about relationships between 3D aural and visual feedback. In particular, it provided information on the difference between the perceived spatial resolution from the aural and visual feedback, and how this can be dependent on the auditory display technology. However, we do not report these results here due to the experimental bias from the user interfaces, but details concerning this study can be found in Appendix B. In the next chapter, we present an analyses on the 3D audio production that classified a workflow of major tasks and present the SpADS design framework to analyze the design of interaction techniques of audio objects.

From the interviews in Chapter 3.2, three important stages of production were identified that highlight how audio objects are created, mixed, and monitored. The results of the online survey in the same chapter added to this finding by identifying needs for experimentation and exploration of renderers, listening formats, and reproduction spaces. Using the state of the art and the results of the interviews, we conducted a HCI conceptual morphological analysis to better classify how different audio objects (objects of interest) can be created and controlled. This led to the development of a framework of design spaces, Spatial Audio Design Spaces (SpADS), that helps analyze how possible and existing production systems can create audio objects and how their user interfaces control audio objects. Afterwards, we included the findings of the online survey from Chapter 3.3 into the analysis, which led to a high level classification of tasks that are required for all types 3D audio production. We then improved the SpADS framework to incorporate the findings of the online survey, spatial perception overview, and task classification. The results from these morphological analyses help organize the required functionalities of 3D audio practitioners, and provide a set of design spaces with descriptive, analytical, and suggestive capabilities.

design spaces for creating and manipulating audio objects

The following work builds upon HCI methodologies for the design and evaluation of new musical instruments [START_REF] Kiefer | HCI Methodology For Evaluating Musical Controllers : A Case Study[END_REF][START_REF] Stowell | Evaluation of Live Human-computer Music-making: Quantitative and Qualitative Approaches[END_REF]. This approach was inspired by Wanderley and Orio's use of HCI tools for evaluating input devices for musical expression [START_REF] Mortensen | Evaluation of Input Devices for Musical Expression: Borrowing Tools from HCI[END_REF], which analyzed common contexts found in scenarios of interactive computer music to investigate the use of input devices from a higher level of user interaction. Such methodologies can help better characterize creative systems and inform their design. For the purpose of this work, we adopted the morphological analysis approach by Card, Mackinlay, and Robertson [START_REF] Stuart | A Morphological Analysis of the Design Space of Input Devices[END_REF] to create a design space that classifies audio objects and their related control meth-59 ods. With this approach, an initial framework of two design spaces was developed to describe how audio objects are created and used [START_REF] Mathew | A Morphological Analysis of Audio-Objects and their Control Methods for 3D Audio[END_REF].

SpADS Framework

Card, Mackinlay, and Robertson [START_REF] Stuart | A Morphological Analysis of the Design Space of Input Devices[END_REF] focused on the generation of the design space for input devices through reflection on the dialogue based communication between the human and computers. We take the same approach in forming a language to describe and analyze audio objects. In the context of interaction, audio objects are high-level objects that provide users the ability to manipulate their spatial parameters instead of controlling low-level routing in audio signal processing. The spatial parameters of audio objects are closely dependent on the capabilities of the recording and rendering method, which has to be accounted for when designing interaction techniques for manipulating audio objects. The initial analysis resulted in the first version of the SpADS framework [START_REF] Mathew | A Morphological Analysis of Audio-Objects and their Control Methods for 3D Audio[END_REF] that consists of two design spaces, SpADS-A (see Figure 5.1) and SpADS-C (see Figure 5.2), that help classify the spatial capabilities of audio objects and how they can be controlled. In this initial version, the two spaces are connected by the same Spatial Parameter dimension (see Figure 5.3), which describes possible interactive parameters of audio objects by using the analogy that audio objects are spatially similar to virtual geometric objects. In this context, audio objects can be controlled through Translation, manipulation of its Area/Volume, and through Orientation, which we can use to classify how user interfaces are designed to control those parameters.

Descriptive Power

SpADS-A defines and classifies audio objects by their recording method along two dimensions: their Rendering method and their interactive Spatial Parameters. By placing the possible combinations of recording methods with rendering methods, we can describe how audio objects can be spatially manipulated. We define Rendering methods as Data-Based and Model-Based as described by Geier, Ahrens, and Spors [START_REF] Geier | The SoundScape Renderer: A Unified Spatial Audio Reproduction Framework for Arbitrary Rendering Methods[END_REF]. Data-Based audio objects use techniques to encode the audio data to be directly routed to a specific speaker or spatially encoded to be rendered across a speaker configuration. Model-Based audio objects are defined by their rendering techniques used to position it within a 3D audio scene. As presented in Figure 5.4, we can use this space to describe and compare various types of recording methods by possible rendering methods and possible interactive spatial parameters in a 3D virtual scene.

For example, a mono audio recording can be only used in Model-Based renderers and can be treated as a point source in space, which can be translated and oriented up to three dimensions. The stereo recording can be considered as two mono audio recordings that are linked together, which can be translated together, oriented from an anchor point, and increased/decreased in distance between the two points. Increasing the number of linked mono audio recordings can be considered as a microphone array recording that adds more possibility in spatial manipulations. It can be seen that only Model-Based audio objects provide this ability, but some encoded Data-Based renderers can provide the same spatial capabilities. For instance, the M/S recording technique is a twochannel microphone technique that is decoded into stereo, allowing it to have the same parameters as the stereo recording. However, the need to decode the M/S technique categorizes it as a Data-Based object rather than a Model-Based object [START_REF] Wittek | Double M/S -a Surround recording technique put to test[END_REF]. Double M/S advances on the M/S technique by capturing the 2D surrounding space [START_REF] Wittek | Double M/S -a Surround recording technique put to test[END_REF], and can only be treated as a 2D circle in the 3D scene. This means that its depth can be increased or decreased to manipulate its area from the perspective of a listener situated in the center of the 2D geometric shape. This type of audio object can be translated in one dimension across a 3D scene and oriented in three dimensions. Audio objects like this have increased geometric dimensionality in the 3D scene and normally provides listeners with a reproduction of a sonic environment. Ambisonic recordings that capture a 3D sound scene can be considered as 3D geometric objects in the 3D scene and can only be oriented in this analogy. Using this space helps by providing a language to describe 3D audio objects, and allows us to classify them as geometric objects with specific interactive spatial properties. This helps to initially address the issue of how to define the spatial properties of audio objects stated in Chapter 2, and provides the necessary definitions to investigate how their interactive spatial parameters can be controlled.

This study of controllers for audio objects is inspired by the use of traditional mixing consoles in the three stages of production found in the results of our interviews. In standard mixing consoles, the audio data used in the Recording/Rendering stage is a mono signal rendered to a stereo speaker configuration. The resulting audio object is rendered with left/right equal power panning that positions it on the line between the speakers. A simple one-dimensional potentiometer is normally used as a positioning input device, and its angular rotation and level meters of the speakers gives visual feedback of the audio object's location along that line. Additionally, multiple potentiometers can be controlled at once with two hands to pan multiple audio channels simultaneously. Our SpADS-C design space extends this analysis to 3D speaker configurations and classifies input devices according to their capabilities for controlling the spatial parameters of audio objects in a 3D virtual scene (see Figure 5.5). We classify the control methods by their Input method and Visual Feedback, where the input can be either Single or Multiple to describe whether the user can control one or several controllers at once and its degrees of freedom or DoF (i.e. how many parameters each input can control). Visual Feedback can be provided by the Physical position of the controller or a Virtual representation within the software GUI, which can have several dimensions (e.g. 2D vs. 3D graphics). In the context of professional audio production, we do not consider advanced feedback technology, such as virtual reality displays or force-feedback, since visual feedback from 2D computer monitors is still the norm.

Figure 5.5 compares how a 6 DoF input device, multi-touch tablet, and physical sliders can be used to directly control various spatial parameters of audio objects. 6 DoF controllers are normally designed with a single input method that can control up to 6 parameters simultaneously, while multi-touch tablets are designed to have multiple touch inputs with 2 parameters to control simultaneously. Multiple physical sliders can be present on a single input device (e.g. mixing console) where each one controls a single parameter, but users can control more than one at a time using bi-manual interaction. Both the 6 DoF device and physical slider can provide physical visual feedback on the state of the spatial parameters from the position of the device, but 3D virtual scenes can also be designed as visual feedback that react to these input devices. Due to the nature of multi-touch tablets not having a tangible object to grab and move, only virtual visual feedback can be designed for this type of device. Analyzing control methods with this space provides a useful and quick method to compare the interaction techniques of different user interfaces for 3D audio production. In addition, describing both audio objects and control methods using both design spaces together can help quickly analyze the functionalities and interaction of existing production systems.

Analytical Power

This method to describe the spatial capabilities of audio objects and compare how controllers can be mapped directly to their spatial parameters can help describe existing production systems and analyze their functionalities. As an example, we analyze the Neve DFC-Gemini console for the authoring of Dolby Atmos 3D audio content for cinema and broadcasting productions [START_REF] Critchley | Working with the Dolby Atmos Format on the AMS-NEVE DFC Console[END_REF]. In this scenario, we considered only the use of mono and Double M/S recording techniques with the Dolby Atmos software tools and Schoeps Double M/S plug-in for DAWs, and we consider the control methods through a desktop mouse and the Neve DFC-Gemini joystick controllers (see Figure 5.6).

We assume that the mono audio recordings are rendered with VBAP through the Dolby Atmos plug-in, and the Double M/S audio recording is rendered through the decoding method in the Schoep's Double M/S plug-in tool (see Figure 5.7) [START_REF] Wittek | Double M/S -a Surround recording technique put to test[END_REF].

In Figure 5.7(a), we classify the recording techniques by their rendering methods and possible spatial parameters that can be manipulated in a 3D audio scene. In the descriptive analysis in Section 5.1.1.1, the mono audio recording is treated as a point source that can be translated and rotated in three dimensions, and the Double M/S object is considered a 2D plane that can only be translated in one dimension across the speaker configuration, manipulated by its area that increases depth, and rotated up to three dimensions across an enclosed speaker configuration. When analyzing the associated control methods in Figure 5.7(b), we examine how the input devices can manipulate the spatial parameters and what types of visual feedback are provided. Both plug-ins can be controlled with a mouse, but the Dolby Atmos plug-in can also position mono audio objects with the joystick in the Neve DFC-Gemini Console (see Figure 5.6(d)). Virtual visual feedback is provided within the plug-in, but the mixing console also provides an additional virtual 2D grid for the mono audio recording. The mixing console also gives physical visual feedback, but only during interaction, and it has to be re-aligned when switching between audio objects [START_REF] Critchley | Working with the Dolby Atmos Format on the AMS-NEVE DFC Console[END_REF]. Practitioners can use a mouse and the Dolby Atmos plug-in's GUI sliders or two joysticks on the mixing console to manipulate 3D mono audio objects. They provide both virtual and physical feedback through the 3D graphics in the plug-in software and positioned of the unhinged joysticks [START_REF] Critchley | Working with the Dolby Atmos Format on the AMS-NEVE DFC Console[END_REF]. Within this analysis, we can observe that the design of the rendering methods in this production system has missing functionality to control some of the possible spatial parameters of the audio objects. The mono audio object has no capability for orientation control and the Double M/S object is lacking control methods for the pitch, roll, and depth. This highlights that there are possible areas of improvements in the rendering techniques in order to increase the number of possible interactive parameters and the design of their control methods.

Suggestive Power

The SpADS-A analysis highlights that some spatial properties cannot be controlled with this production system (highlighted in red in , it gives the opportunity to explore and increase the functionality of the system in terms of spatial mixing capabilities. Improving the rendering capabilities of both objects would provide extra spatial dimensions for exploration and creativity. It is possible to interact with these parameters through a simple design that consists of a mouse with simple GUI controls, or a tablet that could be used to control the Area/Volume parameters for the Double M/S object. 3D graphical representation within the visual feedback would provide the capability to monitor the states of the parameters. Using the design spaces in this manner provides an ability to quickly analyze potential improvements on existing systems and may generate designs of new production systems. Beyond these capabilities, we believe that SpADS can be extended to support the evaluation and comparison of control methods and input devices [START_REF] Stuart | A Morphological Analysis of the Design Space of Input Devices[END_REF][START_REF] Mortensen | Evaluation of Input Devices for Musical Expression: Borrowing Tools from HCI[END_REF]. Analyzing possible control methods of specific audio objects across the dimensions in SpADS-C can help classify the differences between the control methods, which can be evaluated through usability studies to better understand the effect of the classifications on 3D audio mix-ing tasks. Through these evaluations, dimensions of the design space can be validated, identified as unimportant, or used to discover new dimensions to include within the design space. When dimensions are not validated or new dimensions are discovered, it provides an opportunity to create an iteration of the design spaces to revise or improve its capabilities.

Using this object-based approach in SpADS, the relationship between the recording and rendering techniques can be used to define audio objects in a user-centered approach. Through this relationship, designers can explore control methods of a single or combination of audio object to design new authoring systems that can render and mix a variety of 3D audio objects together. Though this was helpful in analyzing existing tools, overall responses from the online survey in Chapter 3.3 helped clarify tasks from a top-down approach (higher-level to lower-level tasks) required for practitioners to produce 3D audio content.

task classification

When designing user interfaces for creative purposes, Terry and Mynatt [START_REF] Terry | Recognizing Creative Needs in User Interface Design[END_REF] identified that users require the ability to experiment, explore variations, and evaluate states. These same requirements were observed in the results of our survey in Chapter 3 where practitioners spatialize audio for various creative projects that require flexible tools (see Figure 3.3). To organize the necessary requirements in a comprehensive approach, we further developed our classification that described stages of production (Recording/Rendering, Mixing, and Monitoring) for audio engineers to three general tasks in 3D audio production for all types of practitioners: Defining the Rendering Space, Audio Object Creation and Manipulation, and Real-time Monitoring of Audio/Visual Feedback.

Defining the Rendering Space

The rendering space consists of the properties that affect the rendering of the 3D audio production and the listening experience. This is typically considered when designing the listening environment and hardware systems, which consists of the auditory display system (renderers and speakers), listener properties, and room acoustic properties. There are dependencies between the properties that are important to consider that causes this task to be complex. define the renderer(s): This defines subsequent type(s) of audio objects that practitioners can use. The most popular rendering methods based on the tools listed by participants and previous work are VBAP/DBAP, Ambisonic Panning, Ambisonic Decoding, WFS, and Binaural, and each method has their pros and cons in creating a spatial listening experience that are dependent on speaker, listener, and room properties. define speaker(s) properties: This task requires practitioners to define spatial parameters, such as position, orientation, frequency response, and directivity. Typically enclosed geometrical speaker configurations (e.g. spherical, hemi-spherical, cubic, etc.) are required for VBAP/DBAP, Ambisonic Panning, and Ambisonic Decoding to properly produce the spatial listening experience [START_REF] Lossius | DBAP -Distance-Based Amplitude Panning[END_REF][START_REF] Malham | 3-D Sound Spatialization using Ambisonic Techniques[END_REF][START_REF] Neukom | Ambisonic Panning[END_REF][START_REF] Pulkki | Virtual Sound Source Positioning Using Vector Base Amplitude Panning[END_REF], and WFS renderers require a dense loudspeaker array [START_REF] Marinus | Spatial Sound-Field Reproduction by Wave-Field Synthesis[END_REF]. Orientation, directivity, and frequency response define the direction, directionality, and the frequency range of the sounds that the speakers produce [START_REF] Toole | Sound Reproduction: Loudspeakers and Rooms[END_REF]. In Binaural rendering for headphone production, frequency response for equalization of the headphones is a dependent factor [START_REF] Møller | Fundamentals of binaural technology[END_REF], while head tracking and reverberation have also been found to increase spaciousness [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF].

define listener properties: This task normally defines the listener position, but it can also consist of other properties related to the spatial auditory perception. For loudspeaker production, listener(s) position can affect the spatial impression of the listening experience, such as a listener situated outside of the sweet spot for enclosed geometrical configurations [START_REF] Jot | A Comparative Study of 3-D Audio Encoding and Rendering Techniques[END_REF]. Other listener properties are more important for Bin- aural rendering, such as properly matching the HRTF to the listener to accurately simulate the immersion of the 3D audio scene [START_REF] Kim | Head Movements Made by Listeners in Experimental and Real-Life Listening Activities[END_REF][START_REF] Laurent | Perceptual attributes for the comparison of head-related transfer functions[END_REF].

define room properties: The acoustic properties of the space are normally characterized by the natural modes of the room and reverberation, which determines the resonant frequency of the room and the impression of space and distance. These characteristics are determined by the geometric shape of the room and the acoustic absorptive properties of room materials [START_REF] Toole | Sound Reproduction: Loudspeakers and Rooms[END_REF]. Normally these properties are altered and adjusted physically, but having knowledge of these properties provides the information to compensate or alter for unwanted or desired effects. However, tools that model this are not common in standard production tools and Binaural productions are normally not dependent on a physical room. Rather, room properties are either embedded in the HRTFs or added to the production to increase the sensation of an acoustic room [START_REF] Møller | Fundamentals of binaural technology[END_REF].

These properties and their relationships to one another are important to understand when designing or working with an existing rendering space as they can have a significant effect on the listening experience.

Consider the choice of the VBAP/DBAP (green) example for a rendering space in Figure 5.10. The flow of the green line indicates that an enclosed geometrical speaker configuration is needed for VBAP/DBAP renderers. Within this configuration, it is designed for speakers to be oriented towards the center where the listener(s) are situated, and the directivity and frequency response of the speakers will affect the impression of the audio object's timbre. In addition, speaker placement in the configuration is important to consider since the algorithm is designed to pan between triplets of speakers. This makes the panning dependent on the spacing between the triplets of speakers [START_REF] Lossius | DBAP -Distance-Based Amplitude Panning[END_REF][START_REF] Pulkki | Virtual Sound Source Positioning Using Vector Base Amplitude Panning[END_REF]. In addition, considering the renderers and speaker configuration, the modes and the reverberation of the room have an effect on the sound's timbre and the spatial quality [START_REF] Toole | Sound Reproduction: Loudspeakers and Rooms[END_REF]. These factors are important when considering practitioners need to understand how the rendering space affects the listening experience, which was further emphasized by comments from the online survey in Chapter 3.3 that indicated practitioners require the need to interact with these properties especially when producing content between different rendering spaces.

Audio Object Creation and Manipulation

The creation and manipulation of audio objects in a 3D audio scene are the primary creative tasks (experimenting and exploring variations) in 3D audio productions. In these tasks, users initially create audio data and configure it to a required or chosen renderer. Users then explore variations of the audio objects through manipulating the audio data with audio effects (gain, equalization, delay, etc.) or through manipulation of the spatial parameters as mentioned in the SpADS framework. creation of audio objects: Audio objects can be created through either audio recording techniques, existing audio data, synthesis, virtual instruments, or other sonification methods. Their manipulable parameters are determined by the configured renderer's capabilities. manipulation of audio objects: This task consists of manipulating the audio data through interactive signal processing techniques or the audio object's spatial parameters through spatial mixing. In the context of manipulating spatial parameters, practitioners can manipulate their parameters for stationary audio objects or record their changes in time through authoring and editing techniques along a sequencer (e.g. automation lines), advanced trajectory authoring and editing techniques, higher-level mappings, or a combination of them for dynamic audio objects.

As mentioned by the participants of the online survey in Chapter 3.3, DAWs are a popular method for creating and manipulating audio data through the channel routing and plug-in design, but it was mentioned that it was difficult to include and interact with 3D audio tools within this production setup. Novel interaction techniques have been proposed in recent research, but many of those have had inconclusive results when studying their usability (see Chapter 2).

Real-time Monitoring with Audio/Visual Feedback

Evaluation of past, current, and future choices is an important part of the creative process [START_REF] Terry | Recognizing Creative Needs in User Interface Design[END_REF]. As with all user interfaces, feedback should provide information to monitor the current states of interactive parameters, normally through visual displays [START_REF] Norman | The Design of Everyday Things: Revised and Expanded Edition[END_REF]. However, 3D audio production requires monitoring the state of the rendering space, audio object parameters, and their effects on the listening experience. For this, the user needs audio feedback of the rendered audio objects in the rendering space and simultaneous visual feedback of their parametric values. Even though many tools and systems mentioned in the survey provide rendered audio playback and spatial representations within the visual feedback, practitioners still mentioned that there are difficulties in associating the rendering techniques and the visual representations. Furthermore, many standard DAWs only provide synchronized audio/visual feedback during recording of audio data or automation parameters. Otherwise, users typically interact with the parameters and then listen to the rendered audio playback afterwards.

Results from the online survey in Chapter 3.3 identified a need to develop visual feedback methods to better represent the listening experience. In addition, providing synchronized audio/visual feedback can help users monitor and better understand how the current state of the rendering space and audio objects affect the listening experience in real-time. However, interaction techniques that provide both audio and visual feedback to help users to compare past and future choices more efficiently still need to be investigated. In subsequent chapters of this thesis, we will investigate the manipulation of audio object spatial parameters by investigating human visual and aural perception during interaction and control methods for manipulating spatial parameters of audio objects. Overall, this classification improves on the classification of audio production stages for 3D audio engineers by highlighting how types of 3D audio practitioners go between defining a rendering space, creating and manipulating audio objects, and evaluating the creative intentions through the use of both audio and visual feedback. Developing a more comprehensive and flexible user interface requires improvements across these basic tasks, which fulfills the needs for creative user interfaces while also addressing specific needs for 3D audio production.

spads v2

In the initial version of the framework, we classified rendering techniques as either Data-Based or Model-Based based on the classifications of Geier, Ahrens, and Spors [START_REF] Geier | Object-based Audio Reproduction and the Audio Scene Description Format[END_REF], but this does not take account of situations when recording techniques can be rendered with multiple methodologies. For this reason, we simplified the classification in this version as Field-Based or Position-Based as discussed in Chapter 2, which is a better description of audio objects from the practitioner's point of view. In addition to these observations, certain findings from the survey also indicated limitations in some of its descriptive power. In comparing the results of the survey to the description of the spatial parameters, participants identified using 2D Cartesian (45%), 3D Cartesian (51%), Polar (51%), and Spherical (38%) parameters to define position, and they selected Yaw, Pitch, Roll (50%) and Vector Pointing (37%) to define orientation. Even though other parametric coordinates of 3D position and orientation are used, a significant number of participants used the parameters of Translation and Orientation with 3D cartesian and yaw-pitch-roll angles that we initially defined in the first version. However, this differed with the Area/Volume dimension, which was initially used to provide an analogy to describe audio object perception of size with 3D graphical parameters. In the survey, a significant number of participants selected that they use Spread (64%), Width (61%), and Frequency Dependent Parameters (22%) to interact with directivity parameters, which are more practical and useful dimensions that better describe how recording techniques capture sound waves. From the overview of spatial perception and task classification, we suggest the design of the visual feedback to include representations of the rendering space properties and auditory spatial perception. From the results of the survey and task classification, we observed that there are additional factors besides representations of the 3D audio scene that practitioners interact with, and they mentioned using Geometrical Algorithms (19%), Probabilistic Trajectories (23%), and Physical Dynamics Simulation (19%) for higher-level mappings to control trajectories of audio objects. These responses coupled with comments of new ideas for tools, such as "Random spatialization generator based on sound parameters (frequencies, amplitude, phase, etc.)" (P 15 ) and "tools where the notes contains their fundamental information and spatial information at the same place" (P 46 ) identified a missing dimension that describes the mapping of the input method in the first version of SpADS-C.

spads-a v2:

In this revision, seen in Figure 5.11, we simplified the classification of Rendering Method to be Field-Based and Position-Based where recording techniques can be either type of audio object depending on how the system renders it. For example, audio data from a Stereo A/B (square symbol) can be rendered as a Field-Based where it is rendered to reproduce the acoustic field it represents, of it can be rendered as a Position-Based where the left and right channel can be considered point sources that are linked together. For the spatial parameters, the Translation and Orientation dimensions stayed unchanged by defining how many dimensions in a 3D space the audio object can be translated and oriented. The major revision in this iteration changes the definitions used to describe how an audio object produces sound waves. Area/Volume parameters may be useful to define geometrical objects in 3D graphics, but these parameters do not create a realistic analogy on the perception of audio objects and their sound. Instead, we substitute this for an acoustic definition of Directivity, which more accurately describes how an audio object's sound waves are acoustically dispersed with Width and Height angles that are based off of the beam dimensions of acoustic directivity of radiators [START_REF] Toole | Sound Reproduction: Loudspeakers and Rooms[END_REF]. Even though this is a minor revision, this change improves the description of spatial parameters for audio objects by providing more realistic interactive properties.

spads-c v2:

The second iteration of SpADS-C (see Figure 5.12) uses the new Directivity dimension in the spatial parameters discussed in the previous section.

The dimension to describe Input Method remains unchanged. The parameters of Single and Multiple describe how many inputs and their degrees of freedom. To improve the description of the tool's control method, we include a new dimension of Mapping Method and add more definitions in the Visual Feedback dimension. The Mapping Method dimension describes how the input method is mapped to the spatial parameters of the audio object through either Direct mapping or Augmented mapping.

Augmented mapping methods map higher level parametric values to generate changes in the spatial parametric values of audio objects. This can be seen in the use of OpenMusic and its various programming objects that allow users to map outputs of programming objects (e.g. random generator function, sound synthesis function, or musical notes) to spatial parameters for rendering audio objects [START_REF] Schumacher | Spatial Sound Synthesis in Computer-Aided Composition[END_REF]. In addition to this added dimension, we add new definitions within the Visual Feedback to include our findings from the task classification and our overview of spatial perception. The first version of SpADS focused on whether the feedback was provided through the Physical input device or through Virtual representations. In this revision, we classify this type of visual feedback to describe visual representation of the 3D Scene and extend the overall dimension to include feedback of the Audio Object itself and the Rendering Space. The visual feedback of the Audio Object gives information on its Content and Spatial characteristics. The content dimension describes visualizations that represent the audio properties and the spatial dimension describes visualizations of the spatial parameters of the audio object. Visual feedback of the Rendering Space describes how information in relation to the rendering methods, speakers, listeners, rooms, and expected spatial perception is provided. Including these new definitions provides a more complete description of a 3D audio tool's capabilities in controlling the spatial parameters of audio objects, and how the visual feedback helps practitioners understand the tools' effect on the listening experience.

Analytical Power

To demonstrate the improved descriptive power of this SpADS revision, we analyzed the same example of the Dolby Atmos and Neve DFC-Gemini Console production system (see Figure 5.6) and compare its results to the first version of SpADS. The audio objects are classified with the revised spatial parameters and their control methods are analyzed with the new classifications. With the second version of SpADS-A, the two types of audio objects (mono and Double M/S) that the production system can render are classified with the new descriptions (see Figure 5.13). A mono audio object is an audio recording from a mono microphone whose 3D position can be manipulated through the Dolby Atmos plug-in and assumed to be rendered through VBAP in this instance. A Double M/S audio object is a set of audio recordings from a Double M/S microphone setup that uses encoding and decoding techniques to capture the audio originating from the front, back, and side of the microphones. In this example, it is decoded through the Schoep's Double M/S plug-in that controls the object's orientation (yaw) and directivity (width angle) [START_REF] Wittek | Double M/S -a Surround recording technique put to test[END_REF]. With this analysis, we also identify the interactive parameters provided by the renderer of the Dolby Atmos system (in red) and the Schoep's Double M/S renderer (in green). In addition, the Double M/S audio object can be better described with the inclusion of the width angle to manipulate the spatial directivity of the of audio data captured, which provides the necessary classification to analyze the system's control methods.

The analysis of the control methods and visual feedback, seen in Figure 5.14, provides more description through the dimensions of the visual feedback and mapping method. The system provides 1D GUI sliders (see Figure 5.6(a)) that can be controlled by a mouse or two physical joysticks (see Figure 5.6(d)) to control a mono audio object's 3D position [START_REF]Authoring for Dolby Atmos Cinema Sound Manual[END_REF]. The Dolby Atmos plug-in's visual feedback provides a 3D virtual representation of the mono audio object position in a 3D virtual scene, and the joysticks from the Neve DFC-Gemini console provide physical feedback since the joysticks do not revert back to a stationary position. The user can also use the visualization of the 3D scene in the Dolby Atmos plug-in while interacting with the joysticks, and the Dolby Monitoring Application [START_REF]Authoring for Dolby Atmos Cinema Sound Manual[END_REF] provides visual feedback of the 3D scene with information of the room shape and volume of the speakers through 1D level meters. This separate application provides visual information on how the sound is rendered to the output speaker channels and how objects are positioned within the space of the room. The only interaction is rotating the scene for a different camera view (see Figure 5.6(b)). However, the system does not provide any spatial visual feedback of the mono audio object besides its content through the DAW. The Double M/S audio object and plug-in [START_REF] Wittek | Double M/S -a Surround recording technique put to test[END_REF] provide visual feedback of the decoded polar patterns mapped to a surround sound configuration, which can be manipulated with a mouse cursor. The focus parameter slider controls the directivity azimuth angle, and the orientation parameters are controlled through the angle slider or on the visualization itself. The visual feedback of the spatial characteristics are achieved through volume level meters of the encoded audio signals and the decoded audio signals, which provides information on how the recording method is decoded to a surround sound output (see Figure 5.6(c)). However, it does not provide visual representations of the 3D rendering space or how it is situated within the 3D audio scene.

Overall, this analysis describes the interaction of the control method in more detail than SpADS V1. In Figure 5.15, a side by side comparison shows an increased description of this 3D audio production system when using the second version of SpADS. Additional descriptions that do not exist in the first version are boxed in green. It can be seen that the second version of SpADS describes this example's control method in more detail. Even though this iteration provides more description, miss- ing functionalities to manipulate audio objects can still be seen just as in the first version, and highlighting this provides the capability to suggest and quickly analyze new ways to improve the system.

Suggestive Power

Using the suggestive capabilities of this framework, we can investigate how new input devices and visual feedback can be designed to improve the example production system. In Figure 5.16, we examine how a 3D SpaceMouse and multi-touch surfaces with 3D visualizations can be used to interact with existing (highlighted in blue) and missing (highlighted in red) interactive parameters of a mono and Double M/S audio object. The 3D SpaceMouse has a single 6 DoF input that can be mapped directly to existing functionalities that translate a mono audio object up to the three dimensions simultaneously and rotate the yaw of the Double M/S audio object. Multi-touch surfaces provide multiple touch inputs (typically up to 10) with 2 DoF for each input, which can translate audio objects up to three dimensions when using more than one input. In addition, it can be directly mapped to control the orientation and directivity parameters of mono and Double M/S audio objects using pinch gestures. However, we can see that these input and mapping methods can also be designed to control the missing functionality of interacting with height, pitch, and roll parameters, which suggests one aspect of improvement in both the rendering and user interface of the system.

3D visualizations within the visual feedback, such as the examples in Figure 5.17 that are inspired by the sound source models by Jang et al. [START_REF] Jang | Object-based 3D Audio Scene Representation[END_REF], provide a representation of each audio object in a 3D scene and the speaker positions of a rendering space. In these visualizations, a mono audio object is represented as a white 3D sphere point source, a Double M/S audio object is represented as a white 2D circular plane surrounding the listener, and the yellow colored boxes represent the speakers. In a visual feedback system like this, users could monitor the state of spatial parameters of the mono audio object through its position and orientation in the 3D scene. They could monitor the state of the Double M/S object by its rotation, size, and placement above or below the listener (1D translation). In addition, users could interact with the speaker configuration to change the properties of the rendering space. Newer models of the Neve DFC consoles (Neve DFC3D and Neve DFC PS-1) include 3D visual feedback with rendering space information within the mixing console, but speaker properties are constrained to the Dolby Atmos predefined speaker configurations and are not interactive1 . Using 3D visual representations, such as these in Figure 5.17, could help users monitor all the audio objects within a 3D scene and rendering space simultaneously, but there can be issues of occlusion and depth perception that limit the amount of spatial information that can be perceived at once. As in the suggestive analysis of SpADS V1, exploring new design ideas like these within this framework can provide a quick analysis on functionalities of new designs. As well, this improvement of the dimensions in the second version of the SpADS framework can be more useful for evaluation purposes by classifying the control methods in more detail, and in turn, can help more accurately describe the effects of these input devices and visualizations that could be used for 3D audio user interfaces. This provides a space on how spatial parameters can be controlled and how visual feedback can assist in the 3D audio tasks. The shapes with black numbers describe the degrees of freedom of the input device, which parameter it can be mapped to, and whether that mapping is direct or augmented. The shapes that describe the spatial parameters within the visual feedback use white numbers to describe the dimension of the visualization. classifications in usability studies.

conclusion

The overall results of the morphological analysis helped classify the multifaceted considerations needed to organize the design process of 3D audio production tools. In addition to the development of the design framework, we identified important tasks and factors that practitioners consider when producing 3D audio content. This classification of tasks along with the SpADS framework can help improve the design process of 3D audio user interfaces to increase usability. Our analysis of the Dolby Atmos and Neve DFC-Gemini systems demonstrates its ability to describe its functionality, highlight missing functionalities, and suggest how other input, mapping, and visual feedback methods could be used to improve the current design of the system. In addition, this analysis highlighted how production systems can be heterogeneous. This analysis also suggested how to create more homogeneous production tools by increasing rendering system capabilities to provide more manipulable spatial parameters, configuring advanced input devices, and providing more comprehensive visual feedback. In the following chapters of this thesis, we turn our focus towards the interaction with audio objects within 3D audio production tools and explore the dimensions of the Input and Mapping Method of SpADS-C.

Part II

I N T E R A C T I O N D E S I G N F O R 3 D A U D I O A U T H O R I N G
In the following chapters, we study how interaction techniques are designed for 3D audio production and invesitgate the usability of different interaction techniques for 3D audio authoring tasks. In Chapter 6, we present an overview of design methodologies for interaction techniques for desktop interfaces and apply these methodologies to explain the design of 3D audio production tools. In Chapter 7, we present a study that classifies standard and new interaction techniques for 3D audio authoring by their input methods and mapping strategies and evaluates them in the context of basic atomic tasks within 3D audio authoring of positional parameters. In doing so, we highlight criteria that should be considered in the design of 3D audio user interfaces, and present results that suggest direct mapping strategies and increasing the integrality of input methods improve performance of authoring tasks. In addition, we identify an efficient combination of direct mapping and integrality using a multi-touch trackpad that provides a balance between the performance of GUIbased interaction techniques and the cost advanced input devices. From the work in these chapters, we identify specific and important design criteria for the design of 3D audio user interfaces and demonstrate how some of them might affect usability in 3D audio authoring tasks.

As we identified in Chapter 5.2, one of the major tasks in 3D audio production is Audio Object Creation and Manipulation. When manipulating audio objects, practitioners can alter the object's content with audio effects and alter the object's spatial properties with spatial mixing techniques. Spatial mixing consists of adjusting the position, orientation, and altering directivity of audio objects within the audio scene and authoring the changes of these spatial parameters in time. The design of interaction methods to spatially mix and author audio objects can have a major influence in the usability of user interfaces. To classify this, we initially characterized interaction methods within SpADS-C through general definitions of input and mapping methods. These dimensions provide a quick method to classify interaction techniques of 3D audio user interfaces, but it is also limited in the description of the input and mapping methods that designers must consider. Improving this aspect of the framework is important due to many inconclusive results of evaluating new designs when compared to traditional techniques, which we observed in previous studies on input devices for 3D audio mixing and authoring in Chapter 2. To better understand which parameters influence the usability of interaction techniques for 3D audio production and authoring, we review frameworks that define Interaction Design, Input Method Design, and Input Device Mapping Strategies that have been studied in HCI. From these frameworks, we adopt terminologies to classify design criteria of input and mapping methods for 3D audio user interfaces, which helped improve the descriptive power of the SpADS-C dimensions.

interaction design for 3d audio objects

There has been significant interest in the field of HCI in studying the input and mapping methods of interaction techniques with desktop computers. Dix et al. [START_REF] Dix | Humancomputer Interaction[END_REF] described the interaction as input-output channels of information between users and computers, which have been primarily studied by defining the hierarchy of tasks and subtasks as in the GOMS (Goals, Operator, Methods, and Selection) [START_REF] Stuart | The Psychology of Human-Computer Interaction[END_REF], CCT (Cognitive Com-plexity Theory) [START_REF] Bovair | The acquisition and performance of text-editing skill: A cognitive complexity analysis[END_REF], and linguistic based models [START_REF] Reisner | Formal grammar and human factors design of an interactive graphics system[END_REF]. These models help design complex but usable interaction techniques with computers by designing task-based mental models, methods to input information to the computer, and methods to receive information from the computer displays. Hinckley et al. [START_REF] Hinckley | Input/Output Devices and Interaction Techniques[END_REF] described this fusion of human-input and computer-output as a communication loop between the human and the computer. In this loop, the user communicates to the computer through manipulation of the physical artifact of the input device, and the computer communicates to the user through sound, text, or graphical representations from audio and visual displays. There are input methods that do not use physical artifacts to grasp, such as motion tracking cameras or microphones, and there are methods for the computer to communicate to the user, such as through haptic force-feedback. In audio production, multi-channel mixers have been the standard method for practitioners to interact with audio signals and parameters, but many standard 3D audio production tools have been developed only for DAWs and programming languages. The use of 3D audio software tools creates a communication loop between a practitioner and computer using an input and mapping method, a visual display for monitoring the visual feedback, and an audio display (renderers and speaker configuration) within a rendering space for monitoring the 3D audio production (see Figure 6.1). There has been interest in studying how to include other channels of communication, such as the use of motion tracking for 3D audio mixing [START_REF] Churnside | Musical Movements-Gesture Based Audio Interfaces[END_REF][START_REF] Gelineck | An Exploratory Evaluation of User Interfaces for 3D Audio Mixing[END_REF] and haptic feedback for authoring the velocity of audio trajectories [START_REF] Melchior | On the Use of a Haptic Feedback Device for Sound Source Control in Spatial Audio Systems[END_REF]. Even though the communication loop for 3D audio production tools can be adapted to include non-standard input and output methods, we concentrate this work on standard communication loops that use common physical input and mapping methods and 2D visual and 3D audio displays.

A common example of this communication loop is user interaction with a computer through a pointing device (computer mouse, trackpad, touchpad, etc.). In this example, a user can input 2D positional information through displacement of the input device, which changes the position of the operating system's cursor on the computer monitor (see Figure 6.2). With the standard model of the WIMP (Windows, Icons, Menus, Pointer) graphical user interfaces, users can manipulate an onscreen cursor to interact with GUI elements displayed on the monitor through actions, such as clicking or dragging [START_REF] Hinckley | Input/Output Devices and Interaction Techniques[END_REF]. In this model of interaction, Hinckley et al. [START_REF] Hinckley | Input/Output Devices and Interaction Techniques[END_REF] focused on input device properties as the physical property sensed, number of dimensions (DoF), transfer function, and the state of the input device for the common interaction task of pointing and interacting with GUI elements. The physical property sensed of the device refers to how users manipulate the physical artifact of the input device and how the transducers of the input device sense those manipulations.

The input device senses the physical artifact's position, motion, or force through a number of integrated transducers that describe its DoF. The device is considered isotonic when it is designed to sense the change in position of a physical artifact within input device's workspace area (e.g. mouse), and it is considered isometric when a physical artifact senses changes in position, motion, or force from a stationary reference position (e.g. joystick). This characterization describes the continuum of elasticity of the physical artifact and how the user controls it [START_REF] Zhai | User performance in relation to 3D input device design[END_REF]. Input devices sense the change and convert them to digital values that are then mapped to the range of possible 2D positions on the computer monitor through a transfer function. Transfer functions either scales the range of the input device to the range of the cursor on the monitor (absolute) or uses more advanced functions, such as control-display gain (relative) or acceleration functions (augemented) [START_REF] Casiez | No more bricolage!: methods and tools to characterize, replicate and compare pointing transfer functions[END_REF].

In this human-computer model of interaction, Hinckley et al. [START_REF] Hinckley | Input/Output Devices and Interaction Techniques[END_REF] focused on lower level engineering properties of pointing devices to describe factors important to the interaction technique, but higher-level interaction models have also been proposed. With Instrumental Interaction, Beaudouin-Lafon [5] proposed an interaction model to extend WIMP-based interfaces that focused on the interaction of domain objects through interaction instruments. In this model, the user interacts with the instrument that comprises of an input device (physical part of the instrument) and a GUI element (logical part of the instrument), and converts the actions into commands that affect the domain object (primary focus of the user). An instrument can be classified by its degree of directness (or indirection) that characterizes the spatial and temporal offset of the GUI elements and the domain object. It can also be described by its degree of integration that describes a ratio between the DoF of the input device and the DoF provided by the GUI element and its degree of compatibility that describes the similarity of the gestures on the instrument with the response of the object [5]. Other models, such as Activity Theory [START_REF] Kuutti | Context and Consciousness[END_REF] and the Human-Artifact model [START_REF] Bøcadker | The human-artifact model: An activity theoretical approach to artifact ecologies[END_REF], describe even higher levels of interaction by taking a broader scope and combining cognitive, behavioral, and societal factors to describe interaction models. Even though these higher-level models are interesting to consider in the cultural and usability aspects of designing user interfaces for 3D audio production tools, we focused our classification of input and mapping methods with lower-level design criteria by borrowing definitions from Hinckley et al. [START_REF] Hinckley | Input/Output Devices and Interaction Techniques[END_REF] and Instrumental Interaction [5].

input method design

Interaction technique models have considered the characteristics of input devices differently. Hinckley et al. [START_REF] Hinckley | Input/Output Devices and Interaction Techniques[END_REF] used an overview of the physical characteristics of the input for the specified task of pointing on a computer display, and Beaudouin-Lafon [5] combined the input device and a logical operation to consider it as an instrument. In the context of our work, we adopt terminologies from both models that define the input method as a physical artifact or sensor that reads a user's physical action. This can vary between the number of physical artifacts to manipulate, their degrees of freedom, and the resolution of values they sense and output. Using these characteristics, we classify input methods as: Spatially-Temporally Multiplexed, Integral-Separable, and Resolution of Acquisition.

Spatial-Temporal Multiplexing

Fitzmaurice and Buxton [START_REF] Fitzmaurice | An empirical evaluation of graspable user interfaces: towards specialized, spacemultiplexed input[END_REF] described an input device that is spatially multiplexed when each transducer has a dedicated control function defined by the space the transducer operates on. On the other hand, an input device that is temporally-multiplexed uses one device that is dedicated to different control functions at different times [START_REF] Fitzmaurice | An empirical evaluation of graspable user interfaces: towards specialized, spacemultiplexed input[END_REF]. For example, a standard keyboard MIDI-Controller mapped to a digital piano Virtual Studio Technology (VST) software for DAWs (e.g. Steinberg's The Grand 3 plug-in1 ) is spatially-multiplexed between the basic piano keys. A mouse-based input device is temporally-multiplexed since the same input method must be used to play a single piano key at a time on the GUI of the VST (see Figure 6.4). When comparing spatially-multiplexed and temporally-multiplexed interaction designs, Fitzmaurice and Buxton [START_REF] Fitzmaurice | An empirical evaluation of graspable user interfaces: towards specialized, spacemultiplexed input[END_REF] observed that spatially-multiplexed input methods can have better performance than temporally-multiplexed input methods in certain tasks due to minimal cost of completion time between switching physical input methods. However, there is an increased cost of material and work space with spatially-multiplexed input methods that is minimal with time-multiplexed input methods [START_REF] Fitzmaurice | An empirical evaluation of graspable user interfaces: towards specialized, spacemultiplexed input[END_REF].

Integral-Separable

Within the design of input methods, integrality and separability is used to describe the DoF of the input device and the structure of the task [53].

Structure of Input Devices

The integrality or separability of input methods describes the DoF of the transducers or sensors within the input device. In the keyboard MIDI-controller example, the piano keys, sliders, and buttons are highly separated input methods that control specific parameters (e.g. on/off musical note, volume parameter, on/off effect, etc.). In more advanced controllers, input methods are more integrated to provide a higher DoF to control multiple parameters simultaneously. In Figure 6.5, examples of input devices with increasing integrality are shown with highly separable sliders, knobs, and buttons (Palette Controllers2 ) that all have 1 DoF, a computer mouse that has 2 DoF, and a 3D mouse with 6 DoF (SpaceMouse3 ). For computer-based interaction, devices with high DoF were based on the original mouse-based input devices (mouse with scroll-wheel, mouse with tilt sensors, Fingerball, 3D mouse, etc.) or armature devices (6 DoF devices) [START_REF] Zhai | User performance in relation to 3D input device design[END_REF]. Subsequently, multi-touch (multitouch tablets, smartphones, etc.) and motion tracking (camera based, gloves, etc.) devices have been explored to input 3D spatial parameters [START_REF] Jankowski | A survey of interaction techniques for interactive 3D environments[END_REF][START_REF] Martinet | The design and evaluation of 3D positioning techniques for multi-touch displays[END_REF].

Structure of Tasks

A logical assumption in the design of 3D interaction is to use input devices with high DoF and map them directly to 3D position and orientation parameters. However, it has been shown that the perceptual structure of the task and its relationship to the control structure of the input device are important factors in predicting performance of devices with high DoF [53]. The structures of the tasks describe the DoF or integrality/separability of the mental model of operations used to complete the task. Jacob et al. [53] observed that if the task requires manipulating 3D position, a high DoF input device would be better suited for the task. If the task requires manipulating 1D position, a low DoF input device would be better suited for the task, and mismatching the structures negatively affects usability. Take the example of the Etch-A-Sketch toy discussed by Buxton [START_REF] Buxton | There's more to interaction than meets the eye: Some issues in manual input[END_REF], which uses two rotary knobs that each control a single dimension when drawing on the screen. If the task is to draw horizontal and vertical lines on the 2D screen (separated structure of task), the structure of the separated rotary knobs matches well. If the task is to draw a slanted or curved line on the screen (integrated structure of task), users have to separate the operations or require additional practice to control both knobs simultaneously due to the separated structure of the rotary knobs. Instead, a 2D pointing devices would be better suited to match the structure of the task. However, there may be limits to the level of integration within the structure of a task. In analyzing the effect of integral input methods in integral tasks, Veit, Capobianco, and Bechmann [START_REF] Veit | Influence of degrees of freedom's manipulation on performances during orientation tasks in virtual reality environments[END_REF] suggested that it was not possible to control all three dimensions in a 3D orientation task and that separated interaction techniques can lead to significant improvements in task completion time.

Within spatial mixing tasks, higher DoF input methods seem a logical choice since a majority of operations on audio objects are 3D positioning, but this may be too complicated for users and there may be instances when the practitioner only needs to manipulate the audio object along one or two dimensions.

Resolution of Acquisition

The resolution of acquisition characterizes how transducers or sensors generate digital values to be sent to the computer. This is dependent on the physical and electrical properties of the transducer, conversion from analog to digital, and the conversion of the digital values to any specific transmission protocol. There are various ways to acquire and measure physical actions, such as through cameras (capture changes in light waves), microphones (capture changes in air pressure), displacement/rotation of physical artifacts (slider/rotary potentiometers), or through touching capacitive/resistive surfaces. In the context of this study, the input devices we investigated use the displacement/rotation of physical artifacts or touch on capacitive/resistive surfaces. Firmware of the device reads the changes in analog electrical signals to determine the position or displacement of the physical artifact or touch. The device converts analog signals to digital values that are then transmitted through a communication protocol (e.g. MIDI, HID, etc.) to the computer. Each protocol has a rate that determines how frequent the digital values are transmitted. The majority of the input devices use protocols that are continuously sending values, but the MIDI protocol only sends values when a change is observed in the physical artifact. In addition, the MIDI protocol confines all MIDI control values to be in the range of 0-127. Even though this may be constraining, it is still a very popular protocol for consumer products in audio production. This is an important characteristic of user interfaces to consider as it determines the range of possible values mapped to the object of interest's parameter values, which could affect accuracy in tasks.

Other studies have considered the compatibility of gestures required to interact with the input method as an important factor in the usability of the interaction technique [START_REF] Aceituno | Direct and expressive interaction on the desktop: increasing granularity, extent, and dimensionality[END_REF]5], but we focus on the multiplexing, integrality, and resolution of input methods in the context of this work. In 3D audio production tools, temporally-multiplexed mouse-based input devices and spatially-multiplexed controllers have been popular choices for 3D audio mixing. Due to the increased number of parameters to control, a natural assumption is to increase the intregrality of the input method and map them directly to the spatial parameters to control. Based on the results of Jacob et al. [53], this would work well for input devices that have an equal number of inputs to interactive parameters, but audio objects can have up to 8 basic interactive parameters to control (3D position, 3D orientation, and two directivity angles) depending on the rendering capabilities. A design solution for controlling a large number of parameters could be to increase the number of inputs (spatiallymultiplexed or integrated) or to temporally-multiplex the existing input methods. Another possible solution is to use different mapping strategies in order to map the existing input methods to the spatial parameters. To understand mapping solutions in more detail, we review different strategies for mapping inputs and their applications in controlling 3D audio objects.

input device mapping strategies

Mapping strategies are an important design factor of interaction techniques that determines how values of input methods control interactive parameters. For the interaction task of pointing on a computer monitor, Hinckley et al. [START_REF] Hinckley | Input/Output Devices and Interaction Techniques[END_REF] described that control-display gain transfer functions are used to convert the output values of the input devices to ranges of the cursor on the computer monitor. This is a common method of configuring input devices to control interactive parameters and works well with input devices that have a DoF that equals the number of parameters to control. Strategies for WIMP based applications and in the Instrumental Interaction model use a cursor and GUI elements that can control the interactive parameters of the main object of interest, such as the use of a scroll bar to navigate a text document [5]. In addition, other mapping strategies has been studied in the context of HCI when using higher-level language of interactive parameters and the interdependency between them [START_REF] Dragicevic | Icon: input device selection and interaction configuration[END_REF][START_REF] Hunt | Mapping strategies for musical performance[END_REF][START_REF] Huot | The MaggLite post-WIMP toolkit: draw it, connect it and run it[END_REF][START_REF] Malloch | Libmapper: (a Library for Connecting Things)[END_REF]. With further analysis on common mapping strategies and how they have been used for 3D audio production, we classified them into two mapping methods: Direct and Indirect.

Direct Mapping

Mapping the values output from input devices to directly control interactive parameters of the object of interest form a direct mapping paradigm. This occurs when each transducer of the input device is mapped to control a single interactive parameter through either absolute or relative methods. Direct-absolute mapping methods convert the range of the input device values to the range of the interactive parameters. Direct-relative mapping methods convert the relative change in the transducer to define the amount of change in the interactive parameter through a transfer function [START_REF] Hinckley | Input/Output Devices and Interaction Techniques[END_REF]. Many of the examples of input methods described before use either direct-absolute or direct-relative mapping

Indirect Mapping

Indirect mapping techniques converts data from the input device to other contextual data that in turns controls the parameters of the object of interest. This is commonly implemented in two manners: Intermediary and Augmented.

Intermediary Mappings

Intermediate mapping strategy occurs when the interaction techniques use a primary interaction method to control an intermediary interaction method that in turns manipulates the interactive parameters. This occurs when the visual representation of the object of interest and intermediary control method is non-collocated. Using the same example of a computer-aided drawing application to manipulate the width of a rectangle, one method of mapping the position of the cursor to control the width of the rectangle is through a GUI slider. In this strategy, the primary interaction technique is the control of the mouse cursor, and the GUI slider is considered the intermediary control method that re-4. http://www.adobe.com/products/illustrator.html quires changing the state of the input device to change the slider value (down-click or drag & drop). This mapping design has been standard in WIMP-based applications where the designs of the intermediary controls are interacted through pointing at GUI elements. In Instrumental Interaction, the intermediary interaction method is considered the logical part of the instrument. The design of the intermediary interaction method has a significant factor in the usability of this mapping strategy, which has the same properties of multiplexing, integrality, and resolution as the initial input method. In the context of this example, using a GUI slider to manipulate the width of a rectangle lowers the degree of directness (see Figure 6.7).

Augmented Mapping

There has been interest in using augmented mapping strategies for 3D animation, music, and audio production software, which are done by mapping input methods to higher-level parametric values of functions that control the interactive parameter of the object of interest. In this mapping strategy, the higher-level function that controls the parameters of objects of interest are designed into the mapping method. For a common example in 3D animation, input methods or intermediary controls can be mapped to higher-level parameters of a physics-based engine, such as velocity, acceleration, or friction, that control how the position of a geometric object on the computer screen changes position in time. In the context of pointing with a cursor on a visual display, transfer functions that use acceleration or inertia to map input methods to cursor positions are classified as augmented mapping strategies, which can be seen in the GlideCursor application by Beaudouin-Lafon et al. [START_REF] Michel Beaudouin-Lafon | GlideCursor: Pointing with an Inertial Cursor[END_REF]. In the context of music and audio production, augmented mappings have been used to connect values of input methods through complex routing and functions to control basic sonic parameters of audio data. Within this type of mapping strategy, levels of convergence and divergence have been used to characterize the number of inputs compared to the number of parameters of interest. Hunt and Kirk [START_REF] Hunt | Mapping strategies for musical performance[END_REF] described convergent mapping to be when multiple inputs are used to control one sonic parameter, while divergent mapping describes when singular inputs are mapped to control multiple sonic parameters. In controlling the spatial parameters of 3D audio objects, augmented mapping strategies can vary across levels of convergence and divergence through the design of transfer functions. For example, convergence mappings tend to use multiple input methods that are either spatially-multiplexed or integrated to control acceleration, friction, and gravity parameters of a physics engine to control the 3D position of an audio object. Divergent methods use a small number of input methods to control multiple parameters, such as initializing a random number generator to provide random 3D position and orientation values (see Figure 6.8). The design of these augmented mapping strategies adds interesting functionalities and ways to interact with spatial parameters of audio objects. As we saw from participant comments from the online survey in Chapter 3.3, augmented mapping strategies were functionalities that current practitioners use and desire.

Many of the existing input methods for 3D audio user interfaces use direct mapping from advanced input devices to spatial parameters or through intermediate mappings that use a combination of a mousebased input device and GUI elements. While a divergent-augmented mapping strategy can be seen in the interaction with the "source presence" parameter in the IRCAM Spat GUI operator, which controls the Figure 6.9 -SpADS-C with the input and mapping method dimensions highlighted.

energy of the source and early reflections of the reverberation [START_REF] Jot | Spatialisateur Ircam/Espaces Nouveaux User Manual[END_REF]. There can be various reasons to choose a mapping strategy, but decisions depend on the input methods and the required task. Certain mapping strategies are designed for a specific input method, such as using directabsolute or direct-relative mapping for absolute or relative input devices respectively. Understanding the relationship between the two and how this affects the usability for 3D audio user interfaces can help describe, analyze, compare, and design new interaction techniques. For this reason, we revised the classification of input and mapping methods in the SpADS-C design space.

spads-c: input and mapping method

When initially defining the classifications of SpADS-C, simple descriptions of input and mapping methods were used that were based on interaction designs of standard mixing consoles, designs proposed in research, and comments by current practitioners (see Figure 6.9). Describing the input methods by the number of inputs and their DoF and whether those inputs are mapped directly or through augmented mapping strategies provides a quick manner to analyze and differentiate different input devices and interaction techniques. However, a more detailed description is required to investigate and understand the usability of different input and mapping methods. By using the classification we presented, we can describe the design criteria of input and mapping methods that are important to consider in the human-computer communication loop when interacting with audio objects (see Figure 6.10).

We revised the input and mapping design dimensions in SpADS-C to include these classification, which further improves the descriptive capability of the framework (see Figure 6.11). Input methods are described by their resolution of acquisition, spatial-temporal multiplexing, and integrality. Mapping methods are described by a direct, intermediary, or augmented mapping strategy. In this space, an input method is placed along the interactive spatial parameters that it can control, and it can describe its resolution of acquisition by the number of discrete values the input method can output where larger values have more resolution and smaller values have less resolution. The multiplexing of the input methods are classified by describing the number of physical artifacts the user can control. An input method that is spatially multiplexed would have an equal or more number of inputs than the number of interactive spatial parameters it can control. An input method that is temporally-multiplexed has a low number of inputs, but can control more spatial parameters than the integrality of the input. The integrality of the input method describes the DoF of each physical artifact of the input device. The mapping method describes whether the corresponding input method is mapped through an intermediary interaction method, directly, or with an augmented mapping strategy. The resolution, multiplexing, and integrality describe the intermediary interaction method, and then it describes if that intermediary method uses direct mapping or augmented mapping. The direct mapping can be absolute or relative, and the augmented mapping can be convergent or divergent. Using these classifications, we can further classify the Dolby Atmos and Neve DFC production example that was used in Chapter 5.1. The previous analysis of the input and mapping methods of this production system highlighted the mouse-based input device as a single input method mapped directly to control spatial parameters. In addition, the previous analysis classified the double joysticks in the Neve DFC console as multiple inputs mapped directly to the audio object's positional parameters. In the new analysis, the interaction techniques with the GUI of the software and the joysticks are described in more detail by their input and mapping characteristics in Figure 6.12 & 6.13 respectively.

In Figure 6.12, we assume the mouse-based input device is a trackpad with high resolution and a single touch input (multiplexing) with 2 DoF (integrality) in this analysis. This is then mapped to the operating system's cursor, which can then control the GUI elements within each DAW plug-in. For the Schoep's Double MS plug-in, the trackpad is a single touch input device with a resolution of 387 dot-per-inch (dpi) 5 with 2 DoF. It uses intermediary mapping techniques (GUI sliders with a resolution dependent on the window size) to directly control the yaw and width angle. Even though the trackpad has two DoF to control two spa-5. https://support.apple.com/kb/TA21609?locale=en _ US tial parameters in this plug-in, the trackpad is temporally-multiplexed due to the intermediary mapping consisting of GUI sliders for each parameter. However, the GUI sliders within this plug-in are spatiallymultiplexed since there are more than one slider, with each being responsible for an individual parameter. In the case of the Dolby Atmos plug-in, the intermediary mapping consists of three GUI sliders with a resolution dependent on the window size, spatially multiplexed with 1 DoF each, and a direct-absolute mapping to control the audio object's 3D position. Similar to the Schoep's Double MS plug-in, the trackpad input device is temporally-multiplexed because it has less DoF than the parameters to control, and it is used with intermediary mapping strategies for the Dolby Atmos plug-in. The intermediary mapping strategy for this plug-in is spatially-multiplexed since there are three GUI sliders where each one is responsible for a single dimension. In Figure 6.13, we analyzed the two joysticks as spatially-multiplexed with a resolution of 210 possible points, two DoF each, and uses a direct-absolute mapping strategy where one joystick controls along the X-Y dimension and the other controls along the X-Z dimension.

With this analysis, we can observe that the joysticks provide a more direct interaction technique when compared to the intermediate mapping methods with the trackpad, but the joysticks also create a mismatch between the integral structure of the devices and the task of 3D translation. Even though the use of the trackpad and cursor matches well with the integrality of the input and the pointing tasks on GUI elements, the intermediary control methods separate the main task of positioning a 3D audio object. Describing interaction techniques in this manner can help further understand how these properties of interaction techniques affect the usability of different interfaces in conducting the 3D audio production tasks. This could help designers to better understand which combination of interaction strategies is more useful with 3D audio production tasks. In the next chapter, we present the results of a user study that evaluated interaction techniques by varying input and mapping methods from our classification for the purpose of authoring and editing 3D audio trajectories.

E VA L U AT I O N O F I N P U T A N D M A P P I N G M E T H O D S F O R A U T H O R I N G A U D I O O B J E C T S
In standard 3D audio production systems, interaction technique designs have focused on desktop computer interfaces using GUI elements or through drawing and editing automation lines along a timeline or sequencer. Designs have also used advanced input devices, but only to position a stationary audio object or to record the change of spatial parameters during the real-time recording of automation parameters. Recording spatial parameters for automation creates a scenario where the practitioner must control the change in real-time. Authoring tasks differ by requiring practitioners to plan how the audio object's spatial parameters will change in time. Within DAW software, practitioners can define the change of a single parameter along a timeline, but this can be limiting and tedious for authoring 3D spatial parameters due to the integral structure of the task. This design requires practitioners to define the change of individual spatial parameter (e.g. x-position vs. time, y-position vs. time, z-position vs. time, etc.) separately rather than defining the change of multiple spatial parameters simultaneously (e.g. {x, y, z}-position vs. time).

In this chapter, we present a study that investigated the usability of different interaction techniques for conducting tasks related to 3D audio authoring. In total, five different interaction techniques were studied that compared varying mapping strategies and input methods. Two standard interaction techniques that used intermediate mapping strategies were compared to designs that used direct mapping strategies. In addition, we compared the direct mapping strategies to each other based on the integrality and spatial-multiplexing of their input methods. We compared these interaction techniques for the task of manipulating 3D position when conducting atomic tasks of authoring and editing 3D audio trajectories (adding, inserting, and selecting control points).

Results suggested that there were differences between the tasks across all interaction techniques, and that direct mapping strategies reduce completion time. Results also indicated that increasing the integrality of input methods for direct mapping strategies reduces completion time across these tasks. In addition to these results, participant comments and observations suggested that they relied on the visual references within the 3D virtual scene to complete the tasks and that tasks were deconstructed into separated operations. These results suggest how direct mapping strategy, integral input methods, and visual references improve performance and suggest directions to improve 3D audio user interfaces for authoring spatial parameters.

related work

Standard audio production software allows authoring of spatial parameters, but only through mouse-based drawing and editing of automation lines. Other works have investigated this through drawing and editing of audio trajectories. Jacob et al. [START_REF] Jacob | Un espace de caractérisation pour l'édition de courbes à trois dimensions[END_REF] proposed a design space specifically for editing 3D curves for the purpose of 3D audio trajectories where primary dimensions consisted of modeling 3D curves, visualization of curve information, 3D navigation, using curves for animation, and camera perspective of the 3D curve. In addition, user interfaces with interaction techniques for creating and editing trajectories have been proposed, such as using the mouse and ray-casting to compose and edit trajectories in the OpenMusic software [START_REF] Schumacher | Spatial Sound Synthesis in Computer-Aided Composition[END_REF]. In the same programming environment, Garcia, Bresson, and Carpentier [START_REF] Garcia | Towards Interactive Authoring Tools for Composing Spatialization[END_REF] used interviews and participatory design to develop a visual representation of 3D audio scenes and trajectories for the OpenMusic software, and also developed an OSC functionality to control positional parameters with external input devices [START_REF] Garcia | Towards Interactive Authoring Tools for Composing Spatialization[END_REF]. Favory, Garcia, and Bresson [START_REF] Favory | Trajectoires: a mobile application for controlling and composing sound spatialization[END_REF] extended this work and investigated the use of a multi-touch tablet to author and edit trajectories. In both of these user studies, observation of user interaction and participant feedback were collected to propose directions for improvement. Melchior et al. [START_REF] Melchior | On the Use of a Haptic Feedback Device for Sound Source Control in Spatial Audio Systems[END_REF] compared a desktop mouse and a 6 DoF input device with haptic feedback for the task of creating trajectories by defining position and velocity either separately or simultaneously, evaluating them through heuristic ratings by participants. In this study, participants rated the mouse-based interaction technique as having better predictability and being easier to manage, but the majority still preferred the 6 DoF input device overall [START_REF] Melchior | On the Use of a Haptic Feedback Device for Sound Source Control in Spatial Audio Systems[END_REF]. In addition to these studies, authoring techniques of paths have been studied through mouse-based interfaces through sequential techniques that define trajectory paths through gestures [START_REF] Zeleznik | SKETCH: an interface for sketching 3D scenes[END_REF], defining constraints that define points along the line [START_REF] Bruno | Mockup builder: direct 3D modeling on and above the surface in a continuous interaction space[END_REF], and inference techniques that interpret user-drawn strokes or constraints to automatically develop a curve [START_REF] Sachs | 3-draw: A tool for designing 3d shapes[END_REF]. Choosing trajectory elements and templates provides predefined trajectories, which can be seen in software such as Iannix [START_REF] Jacquemin | Iannix 0.8[END_REF] and Sonic Emotion Wave 1 Designer1 . Other 3D audio consumer tools, such as Longcat's Audio Stage2 , provides a timeline sequencer and a 3D scene to author and control the spatial parameters of an audio object (see Figure 7.1).

Even though studies have investigated interaction techniques for authoring and editing 3D audio and visual trajectories, quantitative evaluations are missing as they focused mostly on the designs of the interaction technique and qualitative evaluations.

As the majority of consumer tools use mouse-based input methods to interact with GUI elements, there are interests in finding ways to improve this interaction technique. While the majority of studies focused on one design or comparison of a new technique with traditional mousebased technique, we used a holistic approach by investigating multiple interaction techniques (standard and new) with specific variations of their input method and mapping strategies.

input and mapping methods for computer interfaces

Three of the interaction techniques used a standard laptop trackpad as the input device. Two of these techniques used intermediary mapping strategies that are standard interaction techniques for 3D audio production tools, and the third used a direct mapping strategy that uses the multi-touch capability to directly control spatial parameters. The other two interaction techniques used external input devices with direct mapping and were chosen based on their integrality characteristic. In general, we chose these interaction techniques to compare direct mapping strategies against traditional intermediate strategies and how the input methods of direct mapping strategies affect usability for a more comprehensive comparison. We named these interaction techniques as: GUI Sliders, GUI Timeline, Zonal Trackpad, Physical Sliders, and Phantom.

GUI Sliders

This interaction technique uses an input and intermediate mapping strategy that is a standard design of controlling interactive parameters using non-collocated GUI elements. In this design, the primary input method is the trackpad as a pointing device, and the intermediate interaction methods are the GUI sliders that are spatially-multiplexed across a computer monitor (see Figure 7.2(a)). When analyzing this interaction technique with SpADS-C, the trackpad has a high resolution of 387 dpi with one touch input method (spatial-multiplexing) and two DoF (integrality) that control the cursor on the computer monitor. This is used to control the three GUI sliders that have one DoF each, spatiallymultiplexed on the screen, and have a resolution of its pixel length. Through this, we can see that this interaction technique is temporallymultiplexed since it has a lower number of input methods than the 3D position it is designed to control. However, we can see that the intermediary interaction method is spatially-multiplexed since it has the same number of inputs as the parameters it controls (see Figure 7.2(b)). In this interaction technique, a touch on the trackpad is mapped to the mouse cursor on a computer monitor, which can control GUI sliders that in turn control spatial parameters of an audio object. Users must move the mouse cursor to point to a GUI slider, acquire its control through mouse down action (pressing the trackpad button), drag the control to the desired value, and release the control through the mouse up action (releasing the trackpad button).

GUI Timeline

We focus on the standard method for authoring automation lines on a timeline in this interaction technique with the capability to control different spatial parameters by changing its mode of operation through keyboard shortcuts. In this design, the trackpad input method and cursor is temporally-multiplexed and has the same characteristics as before, but the intermediary interaction method is also temporally-multiplexed as it controls all three positional parameters in the same location through changes in its mode of operation. The resolution of the GUI Timeline is based on its pixel height dimension, and it can control one positional parameter at a time (see Figure 7.3(b)). In this interaction technique, the trackpad is used to control the cursor, which is then used to manipulate the 2D control points that are placed along the timeline's x-axis by changing their position along timeline's y-axis. This action manipulates the control point in the 3D virtual scene in real-time. Users can change the operation mode of the GUI Timeline through keyboard shortcuts of 'x', 'y', and 'z' (see Figure 7.3(a)). As with the GUI Slider, users must move the mouse cursor to point to a control point on the timeline, acquire its control through mouse down action, drag the control point to the desired value, and release it through the mouse up action.

Zonal Trackpad

This new interaction technique was inspired by Aceituno and Potier [2] that exploits the multi-touch capability of trackpad in order to improve 3D positioning tasks without the need for external input devices. This design spatially-multiplexed the surface of trackpad by defining zones that are directly mapped to control the x, y, and z parameter of an audio object. Figure 7.4(a) describes how touching the upper right zone designates that touch to control the x and y parameter, and the touch's movement on the trackpad's x-axis and y-axis is relatively mapped to change the x and y parameter. The same touch method is used for the left zone, which is relatively mapped to change the z parameter. Both zones can be interacted simultaneously, allowing users to control all three parameters simultaneously (see Figure 7.4(b)). This technique is a semi-integrated input method as it has spatially-multiplexed inputs with varying DoFs. The bottom zone has been designed for controlling the time of control points along the timeline, but we observed that controlling spatial parameters and temporal parameters are separated controls after an initial pilot study. Therefore, we deactivated this zone and focused on the input methods to control 3D positional parameters.

Physical Sliders

This interaction techniques used a fully separated controller consisting of three physical sliders that are spatially-multiplexed and directly mapped to a single positional parameter. This devices uses the MIDI protocol to send MIDI values (0-127) when the controller changes rather than continuously sending values. The values of the slider are scaled to have a direct-absolute mapping to the range of the spatial parameters (see Figure 7.5). A device like this is commonly used for all types of audio production due to its flexible mapping and multiple inputs, providing a direct manner of controlling sliders compared to the use of the cursor and GUI sliders.

Phantom Desktop

Input devices that use a single input method with a DoF that matches the number of parameters to be controlled are fully integrated. In this study, we used the 3D positional input of the Phantom Desktop device that has a high resolution of acquisition and has a large workspace for positional control. It uses scaling with direct-absolute mapping to the range of the 3D positional parameters (see Figure 7.6). For this study, we only employed it as a 3D positional controller and added a small continuous resistance to improve the control mechanism of the device.

When comparing the characteristics of these interaction techniques using SpADS-C (see Figure 7.7), we see that direct mapping strategies occupy less space than the intermediate mapping strategies, and the differences of interaction techniques based on their integrality and spatialmultiplexing. Larger numbers in the multiplexing dimension describes the number of spatially-multiplexed inputs and the number in the integrality dimension describes the DoF of each input. This study tested participant's ability and preferences with these interaction techniques for completing atomic tasks of adding, inserting, and selecting 3D control points that are related to authoring and editing 3D positional parameters (3D trajectory path).

evaluation of interaction techniques

The experiment simulated how practitioners could compose and edit 3D audio trajectories through a common production scenario. Participants were given a 3D audio mix that consisted of a Position-Based audio object of a female voice counting from zero to nine that was mixed with Direct Channel-Based binaural recordings of a drum ensemble and environmental noise. In this 3D audio mix, participants controlled the position of the female voice by virtually positioning the 3D cursor (gray 3D control point) within the 3D audio scene in the visual display. This simulates how users could virtually position a 3D cursor to a location where they can use action commands (add, insert, select) to compose and edit control points of a 3D audio trajectory. Each interaction technique was tested within this scenario using a visual feedback that provided visual representations of a 3D virtual scene, designed with Apple's SceneKit framework 3 , which included a grid box to define the range of positional parameters, 3D model of a head to indicate the origin of the scene and listener location, and the audio content along a GUI Timeline . The grid lines indicated distance intervals across each dimension, where each grid value was equal to a virtual distance of 0.2 meters. In total, the range of values for each dimension was from -1 to +1 virtual meters from the center of the scene. Within the GUI Timeline, visual representations of the audio waveform of the female voice and 2D representations of control points were presented. A light gray set of control points with colored borderlines (red for x, green for y, blue for z) represented the position of the 3D cursor within the timeline and reacted in real-time when changing the position of the 3D cursor. In addition, participants were allowed to rotate the camera viewport around the 3D graphics scene for a different perspective. This could be done using the up/down and left/right arrows on the keyboard, but rotation was constrained between a bird's eye view of the transverse plane (x & y axes), frontal plane from behind the head (x & z axes), and the median plane from the left and right of the head (y & z axes). In addition, participants were provided with real-time audio feedback to increase the practicality of the scenario, which was a one second loop of where the gray set of control points were located on the timeline. The 3D audio mix was presented through 3D binaural rendering with headphones. Even though 3D audio feedback was provided to the participants, we instructed them to use the 3D graphical scene in the visual display as the main source of information for each task due to the modality-appropriateness theory [START_REF] Welch | Contributions of audition and vision to temporal rate perception[END_REF] indicating that participants would likely tend to use the visual feedback as their main source of information. In addition, this instruction allowed us to limit the effect of auditory spatial perception biases from the binaural rendering.

Methodology

Participants with experience in 3D audio or 3D graphics were recruited from research centers and universities. They were given an overview of the full experiment, the risks and benefits, and an informed consent form to sign to agree to participate in the study. Since the experiment was designed to focus on the visual feedback of the 3D scene to conduct the task, participants that had only 3D graphics were recruited as well. Within each session, participants followed a protocol that was divided into five blocks for each interaction technique. Within each interaction technique block, participants conducted a training phase to practice and learn the interaction technique for each task, which was followed by the experimental trials (see Figure 7.9).

At the beginning of each training phase, we provided a detailed explanation on how to use the interaction technique to complete each task. In addition, participants were informed that the goal of each trial is for short completion time and high accuracy. Afterwards, participants went through a training session that consisted of 10 repetitions of each task in order to learn and become comfortable with the interaction technique. Once the participants confirmed that they understood the inter-action technique and finished the training session, they conducted the experimental trials. They were required to conduct a set of 10 repetitions for each task for a total of 30 trials for each interaction technique block. They had an opportunity to take breaks between each interaction technique block and task block, and participants responded to a brief survey after each interaction technique block that allowed them to rate how difficult it was to learn and to complete each task with the interaction techniques. Once the participant finished an interaction technique block, they proceeded to the next interaction technique where they went through the same procedure -explanation, training, trials, and survey. After finishing all five interaction techniques, the participants answered an overall survey to compare all five interaction techniques to each other. In total, each participant conducted 150 trials for all five interaction techniques along with six surveys.

Inspired by Metatla et al. [START_REF] Metatla | Audio-haptic interfaces for digital audio workstations[END_REF]'s classifications of atomic or sub-tasks of editing automation lines in DAWs, we classified atomic tasks of composing and editing a 3D audio trajectory within a 3D virtual scene. In this study, we evaluated participant performance in conducting these scenarios: creating a new 3D control point (Add-Task), inserting a new 3D control point within an existing trajectory (Insert-Task), and selecting a 3D control point (Select-Task).

add-task:

Participant were given a task that simulated the addition of a 3D control point to an existing trajectory. Two consecutive control points (control point A and B) of a trajectory path followed by the 3D cursor that was connected to control point B were presented at the start of each trial. Participants were asked to position the 3D cursor to form a straight line with the distance between the 3D cursor and control point B (distance BC) to be equal to the distance of control point A and control B (distance AB) while maintaining the order of control point A (blue) -control point B (red) -3D cursor (grey). Participants could identify the control points by the color of the spheres and each trial provided a task label above to remind participants of the color order. Figure 7.10 shows an example of how an Add-Task should look upon completion using the GUI Sliders interaction technique, with a label This task simulated editing a trajectory path where the participants had to create a new control point between existing control points. Participant were given two control points (control point A and B) with the 3D cursor in between and connected to both points. The goal of the task, as seen in Figure 7.11, was to position the 3D cursor halfway between the two control points to form straight light where the distance between con-trol point A and the 3D cursor (distance AC) was equal to the distance between the 3D cursor and control point B (distance CB) keeping the order of control point A (blue) -3D cursor (grey) -control point B (blue). The grey cursor always stayed connected to both blue control points to help complete the tasks.

select-task:

This task simulated the selection of control point in an existing trajectory path. Participants were required to position the 3D cursor at the same location as the middle control point (control point B). Figure 7.12 shows an example of this task right before the 3D cursor was positioned at control point B.

The provided conditions and the target position randomly changed per trial for all three tasks using a random generator function that was confined to ensure the given control points and target position was within the range of the gridded box. Participants could validate the tasks by pressing the Enter key on the keyboard for all interaction techniques, but they were also allowed to use a mouse-click for the Zonal Trackpad, or a button on the Physical Sliders and Phantom Desktop. This was provided to participants for easier mechanism to validate their responses. If participants situated the 3D cursor within the predefined degree of error (within a virtual distance of 0.3 meters), a green light flashed on the screen indicating the task was completed and participants could move to the next trial. The degree of error was chosen from an analysis of a pilot Figure 7.12 -Example of a completed Select-Task study to create a balance that ensured participants attempted the complete task with speed, accuracy, and precision. If participants validated outside the degree of error, a red light would flash and participants were required to better position the 3D cursor and try again.

During the time of each trial, quantitative and qualitative data were recorded to measure participant performance, analyze participant behavior, and measure participant preferences. Quantitative data consisted of values sent by the input device (trackpad touches, physical slider, phantom, and keyboard values), the target position for the trial, the entered position by the participant, the position of 3D gray cursor throughout the trial, and the sequential order of operations by participants. The surveys after each interaction technique block used Likert scale (1-5) questions to record participants' impressions of the difficulty of the interaction technique in completing each task. We also collected information of the participants' experience with 3D audio and 3D graphics (composing, developing, or basic experience) and their overall preferences between the five interaction techniques through the final survey. In addition, observations of participant strategy and comments were recorded during the experiment.

Results

We base our analyses and discussions on estimation of the effect of each factor on measures by computing effect sizes using percent differences and confidence intervals using bootstrap method [START_REF] Cumming | The new statistics: Why and how[END_REF][START_REF] Cumming | Inference by eye: confidence intervals and how to read pictures of data[END_REF][START_REF] Kirby | BootES: An R package for bootstrap confidence intervals on effect sizes[END_REF], which allows us to report our findings based on factors that might be of interest (effect sizes) and their range of plausible values (confidence intervals) across participants [START_REF] Dragicevic | Fair statistical communication in HCI[END_REF]. Percent differences were calculated by computing the symmetric effect size by taking the difference of means normalized by their average. We chose this method rather than using a null hypothesis significance (NHS) testing due to concerns of the limits of NHS testing within HCI experimental designs [START_REF] Dragicevic | Fair statistical communication in HCI[END_REF] (see Appendix C for more detail).

Data was collected from 12 participants (9 men, 3 women) with a majority of them having 5+ years of 3D graphic experience (92%), but only 17% of them had 5+ years of 3D audio experience. Even though a small minority of participants had 3D audio experience, their results did not significantly differ from the rest of the participants due to our instructions of using the visual feedback to monitor and validate the tasks. Completion time was measured from the start of the trial to when the participants validated their entered position within the acceptable degree of error, and distance error was calculated by computing the dis- tance between the entered position and the target position. The study was partially counter-balanced due to number of participants and factors, but the average distance error and completion time across task blocks showed consistency until the last three blocks (last interaction technique block), where completion time decreased and distance error increased (see Figure 7.13). This suggests that participants experienced fatigue in the last interaction technique block, but average distance error in these last three blocks were not significantly larger than the previous blocks suggesting that this could be due to a learning effect for some subjects and fatigue for others.

Comparison of Tasks

Performance measures were analyzed per task by taking a log-transformation on the measures. This compensates for the positive skew of the distribution and allows asymmetrical confidence intervals [START_REF] Dragicevic | Fair statistical communication in HCI[END_REF]. Figure 7.14 shows the log-transforms of the distributions, averages, and confidences intervals by task represented by the histograms, numerical values, and the the light gray bars respectively.

To compare the tasks against each other, we computed effect sizes using percent differences on the distance error and completion time. In this analysis, participants performed more accurately on the Select-Task (0.046 distance error) compared to the Add-Task (0.091 distance error) with a 67% difference and compared to the Insert-Task (0.063 distance error) with a 33% difference. They performed the Insert-Task more accurately compared to the Add-Task with a 36% difference. Completion time was measured from the start of the trial to when participants validated their entered position within the acceptable degree of error. As with the error analysis, we observed a similar trend of average comple-tion time of each task. Participants completed the Select-Task (10.17 seconds) faster than the Add-Task (15.41 seconds) and the Insert-Task (12.51 seconds) with a 41% and 18% difference respectively. They conducted the Insert-Task faster than the Add-Task with a 24% difference. These differences may be due to the differences in the structure of the tasks and the visual references presented within each of them. The Add-Task had a single line from the 3D cursor to control point B (see Figure 7.10) while the Insert-Task had two lines from the 3D cursor to control point A and B (see Figure 7.11) to complete the task. The Select-Task had a target control point where participants were required to dock the 3D cursor (see Figure 7.12). Even though this analysis was done with aggregating all of the interaction techniques, we observed similar differences in performance across tasks within each interaction techniques.

Comparison of Interaction Techniques

Analysis of performance measures across interaction techniques revealed an effect on completion time, but it did not show an effect on distance error. Figure 7.15 and 7.16 shows the distribution of distance error and completion time using the log-transformation by interaction technique along with their averages and confidence intervals. There was little variations of distance errors between interaction techniques. When comparing interaction techniques against each other, participants performed the tasks on the GUI Timeline (0.072) and Phantom (0.072) with slightly higher average distance error compared to the Zonal Trackpad (0.065), GUI Sliders (0.062), and Physical Sliders (0.061). There are overlapping confidence intervals and the distributions are very similar, which gave an effective 0% difference and suggest little effect on the accuracy of completing the tasks (see Figure 7.15). However, the average completion times between the mapping strategies were significantly different as predicted. In Figure 7.16, the distribution and averages of completion time across interaction techniques shows that participants conducted the tasks in order of fastest to slowest with the Phantom (6.56 s), Zonal Trackpad (8.14 s), Physical Sliders (11.55 s), GUI Sliders (15.32 s), and GUI Timeline (21.31 s). When comparing them to each other, participants performed with the Phantom faster than the GUI Sliders and GUI Timeline with an 80% and 106% difference respectively. They performed with the Zonal Trackpad faster than the GUI Sliders and GUI Timeline with a 61% and 90% difference respectively, and they completed the task faster with the Physical Sliders than the GUI Sliders and GUI Timeline with a 28% and 59% difference respectively. These comparisons indicate that the Phantom, Physical Sliders, and Zonal Trackpad designs perform better than the standard design of GUI software that uses Intermediate Mapping techniques. When comparing the interaction techniques with direct mapping to each other, participants performed with the Phantom faster than the Zonal Trackpad and Physical Sliders with a 22% and 55% difference respectively, and they performed with the Zonal Trackpad faster than the Physical Sliders with a 35% difference. This comparison suggests that increasing the integrality of the input method improves performance in 3D positioning tasks, which is consistent to the findings from a previous study by Jacob et al. [53].

Observations and Subjective Feedback of Participants

Observations and participant comments were noted throughout the experiment, and further feedback on the difficulty of the tasks and interaction techniques were rated and compared. In particular, specific comments and observations indicated a difference between the integrality of input methods (Physical Sliders, Zonal Trackpad, and Phantom) and that many participants completed the tasks using a strategy of separated operations rather than a single 3D operation. In addition, many participants commented the use of visual references to complete the tasks and had comments on the use of the camera rotation. Lastly, participants provided subjective feedback through Likert scale rating of the difficulty of tasks within each interaction technique and the overall preferences between all five interaction techniques. integrality of input methods: We observed differences between time of engaging with the 3D cursor (movement time) and time disengaged with the 3D cursor (stationary time). The movement time measured the time duration the participant was actively engaging with the device and moving the 3D cursor on the screen, while stationary time measured the time duration where participants were not moving the 3D cursor and either interacting with the keyboard or evaluating the 3D scene. We analyzed this by computing the percentage of movement time and stationary time within each trial. When the percentage of stationary time is larger than the percentage of movement time, it suggests that the device is more difficult for the task and participants require more time to think more about how they can use the device to accomplish the task. When analyzing this with interaction technique with direct mapping strategies, we observed that integrality may have an effect on the ratio between stationary time and movement time (see Figure 7.17). When using the Physical Sliders, it was observed that participants spent 71.20% of the time stationary and 21.80% of the time moving on average per trial. Participants spent However, we observed at times that participants would engage with the input devices and interact with the keyboard simultaneously. This means that there may have been times where we measured a movement time due to the 3D cursor being manipulated, but participants were engaging with the keyboard to rotate the camera view. We measured the average number of camera movements across participants for the Phantom, Zonal Trackpad, and Physical Sliders (see Figure 7.18) to see if the integrality of the device reduces camera movements. A camera movement was measured as the action of pressing the arrow keys to rotate the camera. Average number of camera movements for the Physical Sliders were 10.69 movements for the Add-Task, 7.68 movements for the Insert-Task, and 3.99 movements for the Select-Task. Average number of camera movements for the Zonal Trackpad were 8.29 movements for the Add-Task, 7.06 movements for the Insert-Task, and 4.02 movements for the Select-Task. Lastly, the participants moved the camera on average 1.38 movements for the Add-Task, 1.29 movements for the Insert-Task, and 0.28 movements for the Select-Task with the Phantom. We observed that in general, average number of camera movements were smaller with the Phantom compared to the Zonal Trackpad with a 143% (Add-Task), 138% (Insert-Task), and 174% (Select-Task) difference and Physical sliders with a 154% (Add-Task), 142% (Insert-Task), and 174% (Select-Task) difference, but also the structure of the Select-Task had an effect on camera movements. Participants used the camera less with the Select-Task compared to the Add-Task and Insert-Task with a 133% and 129% difference with the Phantom, 69% and 55% difference with the Zonal Trackpad, and 91% and 33% difference with the Physical Sliders. Though the confidence intervals are large and overlapping, this was due to some participants not rotating the camera viewport at all in certain trials. When aggregating all three tasks, average camera movements reduced significantly with the Phantom (0.91 average camera movements) compared to the Physical Slider (7.17 average camera movements) and Zonal Track- pad (6.20 average camera movements) with a percent difference of 155% and 149% respectively. This analysis suggests that the structure of the Select-Task and the input method of the Phantom helped reduce camera movements allowing participants to complete tasks faster.

Even though differences in distance error were negligible across interaction techniques, the average completion time, ratio of stationary time over movement time, and average camera movements suggest that using direct mapping strategies and increasing the integrality can improve usability for 3D audio authoring tasks. In addition to these quantitative data, recorded observations and qualitative data were analyzed to understand user strategies and general preferences of the input and mapping strategies.

number of degrees of freedom combined (ndc): Since the structure of the task required participants to manipulate three dimensions, the analysis of completion time across input methods falls in line with Jacob et al. [53] theory on integrality and separability, which states that matching the integrality of the task structure with the input method provides better performance. However, participants were allowed to use the Physical Slider and Zonal-Trackpad as a bi-manual input method, which allowed them to interact with more than one dimension at a time. An interesting observation was that many participants positioned the 3D cursor along a 2D plane first, and then adjusted the third dimension to complete the task. In addition, there were participants who explicitly stated that they used this strategy to complete the tasks. One participant stated that they "aligned then pushed towards target" for the Select-Task, while another participant mentioned "2D + 1D controls is what I have been doing since the beginning" when commenting on the design of the Zonal-Trackpad. Another participant mentioned that they only use "2 axes at once" for the Physical Slider, Zonal Trackpad, and Phantom. This indicates that participants deconstructed the structure of the task as separated operations rather than an integrated operation.

To explore this in more detail, we conducted a Number of Degrees of Freedom Combined (NDC) analysis, proposed by Veit, Capobianco, and Bechmann [START_REF] Veit | An experimental analysis of the impact of touch screen interaction techniques for 3-d positioning tasks[END_REF], on each input method that calculated the average percentage of time that participants interacted with either of seven possible combinations of parameters (X, Y, Z, XY, XZ, YZ, XYZ) within each trial. Figure 7.19 shows a NDC analysis of each input method with varying thresholds. A zero threshold indicates a dimension is being interacted with whenever there was a change within each parameter, and increased threshold values indicates a dimension is being interacted with when a change within a parameter is greater than the threshold values. We observed with zero threshold that participants interacted with all three parameters (XYZ) the majority of time (94.94%) with the Phantom input device, interacted with the XY parameters 41.83% of the time with the Zonal-Trackpad, and interacted mostly with the X (32.28%), Y (36.49%), and Z (26.11%) parameters separately with the Physical Sliders. The large amount of time spent manipulating all three parameters with the Phantom was due to its very integral design with a high precision, making it difficult to stay along 1 or 2 axes at a time.

When increasing the threshold values, the NDC-XYZ value decreased from 94.94% (0.0 m) to 68.64% (0.0025 m), 57.67% (0.005 m), and 45.9% (0.01 m) for the Phantom input device. However, the NDC-XY value showed almost no difference with higher thresholds from 41.83% (0.0 m) to 41.45% (0.0025 m), 41.11% (0.005 m), and 40.42% (0.01 m) for the Zonal Trackpad, and the NDC-X, NDC-Y, NDC-Z values stayed consistent with the Physical Sliders with increased threshold values. This suggests that participants may have used the Phantom as a separable input method similar to that of the Zonal Trackpad rather than an integral task. This analysis along with the observations and participant comments suggest that participants might still have reduced the task into separate opera-tions even if the structure of the task and input method have matching integrality, which provides insight on how participants methods and strategies structured to complete the tasks. In addition to these analyses, other observations and comments indicated that participants used certain visual references in the design of the visual feedback to help complete the tasks.

use of visual references and perspective in tasks:

The design of each task themselves provided different visual references as mentioned before. One participant noted this by commenting that "Select-Task was easier to complete, maybe because there was a clear target", which was not the case for the Add-Task and Insert-Task. In those tasks, the use of the lines between 3D control points and the 3D cursor were required to complete the task rather than providing a visual reference of a clear position on where to move the 3D cursor. In addition, participants commented on the use of shadows of the 3D cursor and control points to help localize their positions in the 3D scene. One participant mentioned that "the shadows helped a lot, but a z-axis shadow could help better perform the task". Another participant explained that they used the shadows for the tasks, but that the "problems with the shadows is that it is hard to differentiate which shadow is associated to which control point". Another interesting comment mentioned that they did not use the camera rotation functionality with the Phantom and relied only on the "shadows for this technique because it was integrated". Even though shadows may have helped complete the majority of the tasks, instances of the cursor and control points being occluded required participants to change the camera for a different perspective of the 3D scene. We observed that some participants used one hand to control the camera with the arrow keys on the laptop keyboard and one hand for controls of the position during the interaction technique to quickly change perspective for a better view. One participant commented that it would be useful to have "camera control within the same [input] device", suggesting that camera control input methods need to be quickly accessed to complete these tasks faster and more accurately. Overall, these observational and subjective results suggest that the use of visual references and camera rotation are important features in completing these tasks. In addition, participants also provided subjective feedback on the input and mapping methods.

difficulty and preferences:

Participants provided feedback on the difficulty in learning the interaction techniques through comments and the survey. One participant stated that "there is a learning curve" involved with the Phantom device while another also mentioned the "Zonal-Trackpad and Phantom input devices [have] a learning curve, but [they] were the easiest afterwards". From the responses of the survey, Figure 7.20 shows participants ratings of the difficulty in learning each interaction technique. The majority of participants found that the input methods with direct mapping were easier to learn compared to the two intermediate mapping strategies. When explicitly asked to choose which of the five interaction techniques was easiest to learn, participants selected the Physical Sliders and Zonal-Trackpad (33.3% each) followed by the GUI Sliders and Phantom (16.7% each).

Participants were asked which interaction technique they preferred in completing each task. Figure 7.21 shows that participants selected the Zonal-Trackpad (50%), Physical Sliders (25%), and Phantom (25%) as their preferred interaction technique for completing the Add-Task, the Physical Sliders (50%), Zonal-Trackpad (25%), and Phantom (16.7%) for completing the Insert-Task, and Phantom (58.3%), Zonal-Trackpad (25%), and Physical Sliders (16.7%) for completing the Select-Task. Overall, participants preferred the direct mapping strategy and varying input methods and did not select the GUI Slider or GUI Timeline as interaction techniques they prefer for these tasks. Participants responses on the difficulty to use or learn the interaction techniques for each task showed clear preferences for the Phantom, Physical Slider, and Zonal Trackpad. Figure 7.22 shows the results of participant responses when asked about the difficulty of interaction techniques. In these questions, participants were asked to rate using a Likert Scale [START_REF] Aceituno | Direct and expressive interaction on the desktop: increasing granularity, extent, and dimensionality[END_REF](2)(3)(4)(5), where 1 indicated "Easy" and 5 indicated "Difficult", in using the interaction technique in completing the Add-Task (see Figure 7.22(a)), Insert-Task (see Figure 7.22(b)), and Select-Task (see Figure 7.22(c)). The responses were analyzed through a boxplot analysis that shows participants selected the GUI-Timeline as the most difficult interaction technique for each task and selected the input methods with direct mapping to be easier across all tasks. We can also see that the majority of participants rated the difficulty of tasks in the order of Add-Task (most difficult), Insert-Task, and Select-Task (easiest). This is in line with the trends observed with participant performances and observations across tasks.

The overall results of this study provided an interesting analysis on the differences of tasks, the usability of different interaction techniques, and how their characteristics of the input and mapping methods affect the performance of these tasks. Participant feedback and further analy- sis indicated a significant preference towards direct mapping strategies and that many participants might have deconstructed the task into separate operations even though increasing the integration of input methods improved performance. We observed that participants relied on the visual references of shadows and changes in perspective of the 3D virtual graphic scene to better understand the position of the 3D cursor and control points. Lastly, we observed that participants performed well with the Zonal-Trackpad when compared to the intermediate mapping strategies (GUI Slider and GUI Timeline) and the Physical Sliders.

discussion and conclusions

Combining the quantitative and qualitative results, we extracted four major findings that should be considered in the design of 3D audio authoring interaction techniques:

1. Difficulty of Atomic Tasks -The differences in the structure of the task can vary accuracy and completion time.

2. Direct Mapping Strategy -Use of direct mapping strategies can reduce completion time.

3. Integrality of Inputs and Task -Increasing the integrality of input methods can reduce completion time, but observations suggest participants may have separated the structure of the task.

4.

Visual References -Visual references of shadows and the ability to quickly change perspective are important functionalities.

These are important factors to consider when designing the functionalities of authoring 3D audio trajectories, input and mapping methods, and visual feedback. When comparing between tasks, the major differences are the visual references provided to complete the task. Even though Select-Task is a possible functionality of 3D audio authoring, it is more likely that participants would select control points or key-frames along the GUI Timeline since it reduces the complexity of the task to a 2D pointing task. However, Add-Task and Insert-Task are still realistic functionalities, and user performance on these tasks suggests that improvements are needed to reduce completion time towards the level of the Select-Task. Providing targets or snapping constraints [START_REF] Shtof | Geosemantic Snapping for Sketch-Based Modeling[END_REF] are some possible ideas for providing visual references to improve Add-Task and Insert-Task. Also, we observed that direct mapping strategies and increasing the integrality of input methods improved completion time across all tasks. However, the Zonal-Trackpad can provide a satisfactory solution between the cost of external input devices and performance through intermediate mapping strategies. Even though integrating input methods improved performance, participants' comments and NDC analyses with higher thresholds suggest participants might have separated the structure of the task. This could be due to a need for further practice and training with the integrated input methods by some participants, or that a single 3D operation is too difficult for participants to conduct. In either case, integrated input methods still resulted in shorter completion times, but improving their functionality to provide separated interaction can provide flexibility for users to match the input method with their mental model on to complete the task. These considerations can improve the usability of user interfaces for 3D audio productions, but it is important to note that we designed this experiment to study the tasks with the visual feedback as the main source of spatial information and did not report results on the temporal interaction of control points along the timeline. Within a realistic 3D audio production scenario, practitioners would also evaluate the spatial information aurally and adjust positional parameters accordingly. This would require additional understanding of the resolution of aural spatial listening experiences from a 3D audio display and rendering space, but evaluating these interaction techniques with only 3D audio feedback would help further understand their usability in 3D audio authoring tasks. In addition, the temporal nature of 3D audio trajectories requires users to also define the timing of the position, which would further evaluate the usability of these interaction techniques for authoring and editing 3D audio trajectories.

C O N C L U S I O N

Designing interactive 3D audio production tools requires the understanding of production methods of practitioners, the ability to perceive and understand the spatial characteristics of the production properties, and designing usable interaction techniques for audio mixing. In this thesis, we analyzed the current state of 3D audio production tools by gathering information on the state of the art and practitioner methods and opinions, and we explored and studied how different interaction techniques affect a particular 3D audio authoring tasks. From this, we presented three high-level 3D audio production tasks and the development of the design framework, SpADS, that better characterizes the production methods, interactive parameters of audio objects (SpADS-A), and their control methods (SpADS-C). These classifications provide designers of 3D audio production tools with design spaces that help describe audio objects and their control methods (see Figure 8.1 and 8.2), which they can use to analyze the design of current production systems, suggest improvements and new designs, and help better understand how different design choices may affect usability. With SpADS-A, we can classify a recording technique with a rendering method and its interactive spatial parameters (translation, orientation, and directivity). This can accurately describe the spatial capabilities of the production system, which define the spatial dimensions of audio objects that practitioners can experiment, evaluate, and explore. In SpADS-C, we can describe how user interfaces provide visual feedback and control methods to interact with audio objects within the production system. When using this space to analyze existing production systems, we can describe existing and missing interactive functionalities that help suggest and analyze ways to further improve production systems and suggest directions to design new 3D audio production tools.

Through the studies presented in this thesis, we formulated and improved the design spaces of the SpADS framework. Interviews initially provided information that led to the first version of SpADS. Practitioner responses from an online survey and an overview of spatial perception provided insight that improved the descriptions of interactive spatial parameters of audio objects and control methods. Our work investigating control methods of user interfaces helped improve descriptions on input and mapping methods and demonstrated how these design dimensions can affect the usability of user interfaces for trajectory authoring and editing. In addition to the SpADS framework, the results of the work presented suggested new directions on the development of 3D audio production tools through: 3D Audio Production Functionalities, Spatial Perception, and Interaction Design.

3d audio production functionalities

Responses from practitioners through both interviews and the online survey provided information on how 3D audio content can be produced through an Object-Based approach that helped better describe how audio objects are created and their interactive techniques. From the interviews, we classified three important processes of 3D audio production for engineers. The first being that they invest time to plan how the production will be recorded and which speaker configuration it will be rendered upon, which is the basis of a 3D audio object as described by Geier, Ahrens, and Spors [START_REF] Geier | Object-based Audio Reproduction and the Audio Scene Description Format[END_REF]. Within the listening formats they choose to render the production, engineers use their standard production tools to spatially mix the 3D audio objects, and they use the visual feedback from the user interfaces and the auditory display to monitor spatial choices. This provided an initial concept of how audio objects are created and mixed by audio engineers (see Figure 8.3). This model of 3D audio production was further developed with responses from the online survey that identified how general practitioners produce 3D audio content overall. From these responses, we observed that there are practitioners who experiment and produce content for a variety of standard and non-standard auditory displays and rendering spaces. We found that practitioners also put an emphasis on how audio content is recorded and rendered, which reaffirms the definition of 3D audio objects. We also observed that practitioners find certain recording methods limited to their rendering methods and would like to adapt the recorded data to form other audio objects. In addition, we observed that practitioners spatially mix audio objects with augmented methods through algorithmic and musical mappings. These findings helped formalize three general high-level tasks that practitioners must conduct when producing content with flexibility, functionality, and creativity. Practitioners must Define the Rendering Space(s) that will be used for experiencing the production, Create and Manipulate Audio Objects, and Monitor with Audio and Visual Feedback (see Figure 8.4).

Audio-Objects

Recording/ Rendering

Mixing

In developing tools for this type of model of production, practitioners should be able to define the properties of any rendering space with flexibility and irrespective of the audio objects. In addition, practitioners should be able to create and manipulate audio objects without being dependent on the properties of the rendering space. With tools that provide these functionalities, practitioners can focus on their creative intentions when creating and manipulating audio objects through spatial mixing rather than low-level limitations of their production systems. However, the current state of many standard audio production systems and 3D audio tools force practitioners to consider audio objects by their recording techniques and how they can be routed and rendered onto an auditory display. This can be limiting when certain recording techniques are designed for specific types of auditory displays. One solution is to use virtual speakers, a feature in IRCAM Spat [START_REF] Jot | Spatialisateur Ircam/Espaces Nouveaux User Manual[END_REF], which renders audio objects on a virtual speaker configuration where each virtual speaker is rendered onto the actual auditory display as audio objects (see Fig- Another method is to increase the capabilities of recording and rendering methods for them to be adjusted and adapted to various rendering spaces, which would allow more flexibility for practitioners in creating and manipulating audio objects. This requires further research and development into the recording and rendering technology and their effects on the spatial listening experience. In addition to improving flexibility of audio objects, improvements in monitoring techniques from a spatial perceptual point of view is required to develop the user interfaces of 3D audio production tools.

spatial perception

A major focus in the development of 3D audio production has been to increase the spatial perception and immersion through designs of auditory display and renderers. Even though this is an important factor in providing an accurate auditory spatial listening experience, we observed that practitioners are sometimes restricted to certain types of auditory displays and renderers due to the heterogeneity of current 3D audio tools. When there are restrictions on the auditory display and renderers, information on the spatial perceptual capabilities of the auditory display and rendering could help practitioners anticipate the listening experience. Current technologies have used 3D computer graphics to monitor the state of the 3D audio scenes, but responses from our online survey in Chapter 3 indicated that many of them do not represent the listening experience of the rendered production very well. From our review of aural and visual spatial perception in Chapter 4, we can see there are major differences in how we perceive the spatial properties from 3D computer graphics from an exocentric point of view and from 3D auditory displays with an egocentric point of view. Due to the nature of aural perception being the dominant modality in 3D audio production, we suggest designing visual feedback with representations of the rendering space properties and spatial perception to improve the relationship between visual and audio feedback. However, further investigation is required in determining how the design of auditory displays (speakers and renderers) affects the resolution of spatial aural cues. Even with the development of 3D visual representations to include this information, there are still issues with quickly perceiving spatial relationships in 3D computer graphics. We observed in our studies of interaction techniques that visual references (shadows, lines, targets) within the 3D virtual scene improved performance of authoring tasks, and changes in perspective through camera rotation controls were often used for certain interaction techniques. The use of visual references and quick access to camera controls can improve interaction with the visual representations within the visual feedback, but the characteristics within the input and mapping methods may have a greater effect on the usability of user interfaces.

interaction technique design

The design of the Phantom input device provides an interesting solution for 3D audio production mixing consoles since it is isometric (position controlled) with a 3D workspace. Within this design, practitioners can use the physical visual feedback of the position of the controller in the workspace to represent the audio object within the 3D audio scene in a similar manner as rotary knobs with stereo and surround sound formats. Similar to rotary knobs, this device can stay fixated in a certain location with the use of its haptic feedback functionality even though we did not investigate this. Designing methods so that the controller could be spatially fixated could provide a physical visual feedback similar to the use of rotary knobs for panning. In terms of its usability, we observed in Chapter 7 that its integrality and direct mapping method reduces completion time in 3D trajectory authoring tasks. The difference in integrality of physical sliders, multi-touch zonal trackpad, and Phantom input devices suggested that higher DoF input methods improved the completion time in conducting add, insert, and select tasks. However, we observed that participants might have used the Phantom Desktop with separated operations rather than an integrated operation, suggesting that there was a mismatch between the structure of the input device and the task for certain participants.

According to Jacob et al. [53], performance of user interfaces is optimized when the integrality of the task and input device match. This would suggest that if participants were operationalizing the authoring tasks into separated spatial tasks, then designing the integrality of the input methods to match how they operationalize the task would be optimal for them. This may vary between practitioners though. To compensate for this, we suggest designing high DoF input methods with flexibility to separate the dimensions of control, which would be more practical for the design of input devices as it could be adapted to practitioner's needs. The 3D SpaceMouse does provide functionality to interact with a dominant axis, which defines an axis of interaction if that axis is first manipulated. If this functionality is enabled, this makes the input device fully separable as users can limit interaction to a single dominant axis at a time. However, combinations of dimensions are still lacking in the design of the 3D SpaceMouse. Improving the flexibility of integrality within input devices could be a useful design for various purposes, as it would allow participants to switch modes of interaction that best suit their mental model of the task's structure.

In addition to this, we observed that utilizing the multi-touch capability of the trackpad could provide a suitable solution for 3D audio trajectory authoring based on participant performance and cost of the device compared to other evaluated input devices. The Phantom input device with three DoF was observed to have the best performance based on completion time, but participant completion time with the Zonal Trackpad was close to the Phantom and better than the Physical Sliders. As the inclusion of multi-touch surfaces within laptop user interfaces has become standard, this technique is a good balance that minimizes the cost compared to high DoF devices and some MIDI controllers. We also observed this benefit when comparing the performance of the intermediate mapping methods using the GUI Sliders and GUI Timeline. As previously described by Aceituno [START_REF] Aceituno | Direct and expressive interaction on the desktop: increasing granularity, extent, and dimensionality[END_REF] and Shneiderman [START_REF] Shneiderman | Direct manipulation for comprehensible, predictable and controllable user interfaces[END_REF], there are improvements in usability when increasing the directness of the interaction. Exploiting the multi-touch capability of the trackpad and directly mapping it to the interactive spatial parameters of the audio object helps remove the spatial and temporal offsets of pointing and manipulating GUI elements (sliders and key-frames on a timeline) with the cursor. Even though there is a logical assumption in designing interaction techniques with high-end input devices, there are still benefits in further exploring and exploiting the capabilities of desktop interfaces for cost and performance purposes.

Though this is one method of increasing the directness, there are other techniques of increasing the directness with the use of the cursor that were not tested, such as using the operating system's cursor within the 3D virtual scene to select and manipulate objects through the use of raycasting or axes handles like Skitter and Jacks [START_REF] Bier | Skitters and Jacks: Interactive 3D Positioning Tools[END_REF]. In addition, we can increase the directness of GUI sliders and timeline interaction through a multi-touch tablet by using touch inputs and removing the use of the cursor. We predict that this would improve performance to a similar level of the physical sliders since similar gestures are employed, but it would be dependent on the resolution of the tablet surface, pixel size of the GUI elements, and the effect of not having a physical artifact to manipulate. Using techniques like ray-casting or axes handles within a tablet interface would further increase the directness of the interac- tion, but issues with exocentric interactions and egocentric evaluations still exists. Increasing directness of the total interaction with 3D audio production tools would require developing the use of egocentric visual displays (virtual/mixed reality headsets) for this purpose. There can be a cost in using this type of display in terms of materials and potential motion sickness [START_REF] Rizzo | Virtual Reality and Cognitive Assessment[END_REF], but this is an interesting solution for increasing the directness of 3D audio user interfaces.

implication for other domains

Even though the focus of these studies has been on 3D audio production, some of these results can inform the development of other production technologies for immersive content creation. The model of general 3D audio production tasks of defining the rendering space, creating and manipulating audio objects, and monitoring the production in real-time could also be applied to producing content for immersive visual and haptic experiences (see Figure 8.6).

For both visual and haptic experiences, content producers must define the technology to render and display the immersive content. Visual display technologies, such as 3D glasses or virtual/mixed reality headsets, display the visual experiences in a certain format that the content producers must consider when creating and editing the content. Software tools, such as Unity3D 1 or AutoDesk 3DS Max 2 , already provide capabilities to render projects across 2D visual displays or virtual/mixed reality headsets, which allows them to focus on producing content specifically for both displays. For technologies that provide haptic experiences, such as devices with vibrating physical elements or force-feedback, content producers must keep a similar consideration in mind of how the haptic-feedback will be rendered and displayed. In addition to this, the vast amount of available display technologies for consumers requires immersive content producers to consider perceptual differences between the display technologies. Content producers might require information on the limitations of the visual and haptic display technology as well. With this information, user interfaces can be designed to provide information on how an object of interest (3D graphics, 3D video recording, force-feedback) and any manipulations of the object will be rendered. In addition, providing real-time monitoring tools that include perceptual information on the renderer and display technology can further aid the production of content without the actual physical display. From a control method point of view, the results of the authoring tasks can suggest improved performance based on the integrality of input methods and directness of mapping methods for all types of 3D user interfaces. The results of the authoring tasks we tested could be applied to 3D trajectories in general and for 3D positional manipulation, and the proposed flexible integrality within input methods could be found useful for 3D user interfaces for other domains. However, the structure of the task and control method is dependent on the properties of the object of interest and how practitioners would like to manipulate those properties. Defining exactly what the object of interest is, how it will be rendered, and how practitioners use them are important when developing production tools for other modalities in the same manner as 3D audio production.

future work

In the context of further developing 3D audio production tools, followup studies on the spatial perception of auditory displays, monitoring of visual feedback, and interaction techniques for controlling audio objects could further explore the design dimensions of the SpADS frame-work and better inform the development of 3D audio user interfaces. In particular, studying the auditory localization perception changes across varying properties of auditory display and rendering spaces would help better characterize how audio objects would be spatially perceived. In turn, this could provide the necessary information to better develop visual representations within visual feedback that accurately reflect the aural spatial perception. In addition, studying which characteristics of the visual feedback improves the performance in 3D audio production tasks could help improve its usability. From the perspective of input and mapping methods, investigating the scenario of authoring 3D audio trajectories with only the use of the 3D auditory display without visual feedback could help better understand the effect of input and mapping methods when only using the auditory modality. To conduct a study like this with quantitative evaluation, priori knowledge of the auditory spatial resolution from the auditory display is required. This would help further investigate the usability of input and mapping methods when practitioners only use auditory spatial perception to monitor the production and provide a more accurate understanding of the expected spatial resolution of auditory displays. Within these directions, we will be able to better describe how the design dimensions of the SpADS framework affect performances in different 3D audio production tasks.

In this thesis, we demonstrated the use of SpADS in exploring user interfaces for 3D audio trajectory authoring and editing purposes, but the dimensions of SpADS-C can be used for evaluating user interfaces for other mixing functionalities. Other possible user studies could include evaluating characteristics of user interfaces for mixing different types of audio objects, recording a 3D audio trajectory, and the usability of visual representations within the visual feedback. In addition to exploring how to improve 3D audio production tools, it is also important to reflect on how production tools for immersive technologies in general will move forward. Currently, many production tools focus on a single perceptual modality, but it is more realistic to use multiple perceptual modalities to improve the sense of presence in an immersive virtual environment, such as the Google Daydream virtual headsets with controllers that can be coupled with binaural headphones 3 . For this reason, it will be important to envision the development of technologies to produce multi-modal or multi-media content (audio, visual, haptic, etc.) in a comprehensive and homogeneous manner. This would require extensive knowledge of the display technology of each perceptual modality, the perceptual limitations of the modalities and the display technology, and the perceptual characteristics of the objects of interest for content producers. By classifying immersive technologies with low-level engi-neering and perceptual factors, higher-level creative functionalities can be further developed to provide content producers the flexibility to experiment, explore, and evaluate their creative intentions [START_REF] Terry | Recognizing Creative Needs in User Interface Design[END_REF].

Part III

A P P E N D I X A A P P E N D I X A : O N L I N E S U RV E Y
The following describes and provides a copy of the online survey that was reported in Chapter 3. The survey was conducted through Google Surveys and structured into five sections:

1. Profession and Experience -to gather demographic information about the participants.

2.

Hardware and Software Tools -to gather a list of current tools participants use for spatialization.

3.

Composing/Planning and Editing -to gather information and opinions on how participants sketch, compose and edit spatial parameters.

4. Notation and Performance -to gather information on how participants visually express, communicate, and perform spatial ideas in their projects.

5.

Listening/Monitoring -to gather insight into their methods and tool's capability for listening and monitoring spatialization choices through questions.

Responses were accepted for 3 months where participants could edit responses at anytime within that period. They were not required to answer every question, except on the first page when agreeing to participate in the survey. The questions were recorded as multiple-choice with only a single response ("Mark only one oval"), multiple-choice with multiple responses ("Check all that apply"), Likert Scale (1-5), and free responses. Most multiple-choice question gave the opportunity to provide a different answer than the choices provided ("Other:"). The following pages provide a copy of the survey. DESCRIPTION: This questionnaire is a part of "Removed for Reviewing Process" PhD research on understanding the use of audio spatialization in order to gain a broad understanding of how audio spatialization is currently done in various projects and the intention of using these techniques. The results of this study will hopefully enable us to design new interactive and visualization interfaces for audio spatialization for various types of users.

RISKS AND BENEFITS: The study consists in answering an online questionnaire. There are no known risks involved in this procedure and there are no direct benefits to participation. TIME INVOLVEMENT: Your participation will take approximately 20 to 30 minutes.

PARTICIPANT'S RIGHTS: If you have read this form and have decided to participate in this study, please understand your participation is voluntary and you have the right to withdraw your consent or discontinue participation at any time without penalty. You have the right to edit your responses until 31Dec2014 as well as to refuse to answer particular questions. All collected raw data will be kept confidential and your individual privacy will be maintained in all published and written data resulting from the study.

You can choose to pause and complete the rest of questionnaire at a later time. If so, please go to the final page and press submit when you choose to pause. You can then edit the blank responses later to complete the rest of the questions.

If you have any questions regarding the purpose or problems with completing the questionnaire, please feel free to contact me at "Removed for Reviewing Process".

Reverberation

These questions are related to reverberation, which is the decay of sound over time.

What architectural-acoustic space do you take into consideration?

Check all that apply. rather than matching it to the curvatures in the other modality. As well, analyses suggested that the Aural-to-Visual task was dependent on how well participants could aurally discriminate the differences between the curves of the audio trajectory path that we suggest is dependent on the auditory display capability to produce aural spatial cues for each trajectory path.

b.1 related work

Previous studies have investigated the auditory and visual perception separately, but there has been interest in studying the effects of using both modalities. When developing the modality-appropriateness hypothesis, Welch, DutionHurt, and Warren [START_REF] Welch | Contributions of audition and vision to temporal rate perception[END_REF] investigated the synchronization of aural and visual modalities in spatial and temporal perception that resulted in participants changing their attention to the modality that was more sensitive for the task. If the goal of the task was to identify a spatial property or a temporal duration, participants were more accurate when using hearing for temporal perception and visual for spatial perception. However, Wada, Kitagawa, and Noguchi [START_REF] Wada | Audiovisual integration in temporal perception[END_REF] reported results that suggested that hearing can affect spatial perception when visual information is less precise. Hairston et al. [START_REF] Hairston | Visual localization ability influences cross-modal bias[END_REF] studied cross-modal bias with tasks using both auditory and visual modalities. In this study, participants were asked to locate an auditory, visual, and audio-visual stimulus and they were asked to locate an auditory stimulus when the visual stimulus had a different location (spatial disparity). In addition, participants conducted the latter task again and responded whether the audio-visual stimulus originated from the same location. Hairston et al. observed that there was a large variability in accuracy when only localizing auditory targets, a small variability when only localizing visual targets. They also observed that there was a significant bias towards the visual stimulus on tasks when the audio-visual stimulus had spatial disparity. Odegaard, Wozny, and Shams [START_REF] Odegaard | Biases in visual, auditory, and audiovisual perception of space[END_REF] found similar results when studying spatial perception biases in localization tasks of objects in visual, auditory, and audio-visual modalities. In this study, they collected a large data set of observations, and their analysis identified a bias towards the center when localizing visual stimuli and the periphery when localizing auditory stimuli. In addition, they suggested the visual modality can reduce the bias in the auditory modality for localizing an audio-visual stimuli and the integration of both modalities improved precision and accuracy in localization tasks.

Mycroft, Reiss, and Stockman [START_REF] Mycroft | The influence of graphical user interface design on critical listening skills[END_REF] studied how complex GUIs in DAW software affect critical listening skills within audio production.

Participants were asked to identify which sound source was being panned (auditory task) while matching frequency curves of a multi-parametric equalizer to a target multi-parametric equalizer (visual task). Their results suggested that the use of the scroll bar negatively affects reaction times of the auditory task, which they proposed is due to the additional cognitive load when navigating visually while critically listening. In a study more related to 3D audio production, Nguyen et al. [START_REF] Khoa-Van Nguyen | Spatial audition in a static virtual environment: the role of auditoryvisual interaction[END_REF] tested the accuracy of localizing a visual, auditory, and audio-visual target within a virtual reality headset binaural audio reproduction. They observed similar performances within in each modality to results from studies that investigated this task in real-life, which suggested that the use of virtual reality displays could be a used for psychophysics studies in general. Other works presented designs of visual representations of 3D audio scenes, but in-depth studies on the usability of designs and their relationships to listening experience are missing [START_REF] Bresson | Spatial Structures Programming for Music[END_REF][START_REF] Garcia | Towards Interactive Authoring Tools for Composing Spatialization[END_REF][START_REF] Schumacher | Spatial Sound Synthesis in Computer-Aided Composition[END_REF]. Overall, these works focused on the bias or effect when using singular or bi-modalities in their tasks. Instead, we focus our study on tasks relating between the auditory and visual modalities, which is more realistic to how 3D audio practitioners currently produce content.

b.2 methodology

This experiment was designed to simulate a common scenario when practitioners use a spatial visual representation and listen to the 3D audio playback separately. We designed a rendering space where partici- The auditory display was gain calibrated to be 60 dB at the sweet spot, but no speaker equalization was applied due to the having very similar responses. Participants were seated next to a small desk where a 13-inch laptop computer was placed for them to complete the two cross-modal tasks:

1. Visual-to-Aural -Participants were given a target curvature of a trajectory path in the visual modality and were asked to select the curvature of a trajectory path from a set of audio trajectories that best matched the target curvature.

2. Aural-to-Visual -Participants were given a target curvature of an audio trajectory path and were asked to select the curvature of a trajectory from a set of visualizations of trajectory paths that best matched the target curvature.

Trajectory paths consisted of five different start and stop positions, and each path had a set of 15 trajectories that differed in curvature. These start and stop positions were chosen based on the likelihood of use in 3D audio production that consisted of start and stop points with regards to the listeners: Front Left -Front Right (FL-FR), Front Right -Front Left (FR-FL), Front Left -Back Left (FL-BL), Front Right -Back Right (FR-BR), or Front Overhead -Back Overhead (FO-BO). The FL-FR and FR-FL paths were chosen because of the popularity of panning between left and right in stereo productions. The FB-BL and FR-BR paths were chosen because of a popular scenario of an object moving past the side of the listener (e.g. moving cars), and the FO-BO path was chosen because of the scenario of an object flying over the listener (e.g. planes). The curvature conditions of each path were created by computing a cubic spline curve using end points of each path and the midpoint as the peak of the curve. Each curvature condition had a different midpoint that changed along one of the dimensions of the path. Midpoint values differed by a virtual distance of 0.25 m, which were informally chosen during a pilot study. In the aural modality, each playback of an audio trajectory curvature condition lasted 2 seconds long. Though this varied the velocity when the curves were greater, it removed the possibility of making choices based on the length of time rather than spatial perception and fell within the discrimination ability of moving sounds [START_REF] François-Xavier Féron | Upper limits of auditory rotational motion perception[END_REF]. The stimulus sound of each audio trajectory was a pulsed pinked noise repeated every 100 ms, which was chosen to help participants localize the trajectory path due to the rapid transient pulses. This was rendered across the 32-speaker configuration using MaxMSP and IRCAM's Spat rendering engine. In the visual modality, a single 3D trajectory path was displayed through three 2D views and one 3D view using MaxMSP's Jitter framework and OpenGL. Both visual feedback and response method were provided within the GUI of the experiment on the 13-inch laptop screen (see Participants were initially asked to conduct an informal listening test that asked them to listen to the audio trajectory paths and asked them to identify their start and stop point: Front Left -Front Right, Front Right -Front Left, Front Left -Back Left, Front Right -Back Right, or Front Overhead -Back Overhead. This was to ensure participants had basic ability to localize the sound source in the audio trajectories. If participants answered correctly, they proceeded to the experiment.

Using a mixed participant design, half of the participants conducted the experiment with Vector-Based Amplitude Panning (VBAP) and the other half with High-Order Ambisonic Panning (HOA) 3D audio rendering algorithms. The order of the tasks was counterbalanced among participants and paths, and the whole experiment was broken into two blocks (one for each task) that consisted of three sessions for each block. At the start of each block, participants were required to complete six training trials on the task and path. Afterwards, participants conducted each session on the task with either the combination of FL-FR and FR-FL paths, combination of FL-BL and FR-BR paths, or the FO-BO path. For simplicity, we classify the combination of FL-FR and FR-FL paths as FL-FR and the combination of FL-BL and FR-BR paths as F-B (see Figure B.3 for a workflow of experimental protocol). Within each session, participants conducted the task on nine test curvature conditions. These conditions included a zero curvature path, four paths with increasing midpoint intervals (0.25 m) in one direction, and four paths with increasing midpoint intervals in the opposite direction. We classify each condition as a curvature interval that varied between -4 to 4, where curve 0 has no curvature and curve -4 and 4 are the extremes. In each session, participants repeated the task 3 times for the nine test curvature conditions, resulting in 27 trials per session. Between each block and session, participants were encouraged to take a break to counter fatigue.

In each trial, participants were asked to select a curvature out of a set of 13 curvatures through a GUI Slider that best matched the curvature in the target modality. The tick marks were designed to indicate where on the slider a curvature value could be selected, but the choices did not stay consistent for each trial. The nine tested conditions were always available for selection within each trial and the order of the curvatures stayed consistent. However, the set of 13 curvatures presented in each trial randomly shifted across the set of 15 curvatures (see This was implemented in order to prevent participants from relying on their memory from previous trials to complete each task and force participant to rely on their spatial perception of the visual feedback and audio trajectories. Participants were informed at the beginning of the experiment that the choices on the slider were not consistent from trial to trial and to not to rely on their memory of the mapping of curvature conditions to slider positions from previous trials to complete the task. Even though the set of curvature conditions shifted across the total 15 curvatures, the order of choices on the slider stayed consistent. For the FL-FR and F-B trajectories, moving the slider up changes the curvature in the upward direction and moving the slider down changes the curvature in the downward direction (curvature along the z-axis). For the FO-BO trajectory, moving the slider up changes the curvature in the right direction and moving the slider down changes the curvature in the left direction (curvature along the x-axis). The entire experiment lasted approximately 1-1.5 hours for each participant. Responses were analyzed to investigate the success of each tested curvature condition across each task, rendering method, and path. We base our analyses on the distribution of participant responses.

b.3 results

Twenty-four participants were recruited to conduct the experiment (12 for each rendering technique) with seven having 5+ years of experience, ten having 1-4 years experience, and seven having no experience with producing 3D audio content. Participant performance was analyzed by comparing their responses with the correct response. Even though experienced participants were expected to perform more accurately in both tasks, an initial analysis did not indicate any significant differences between experience level. Instead, we observed differences in distribution between tasks and trajectory paths.

b.3.1 Comparison of Tasks

There was a strong difference between participant responses in the Visual-to-Aural and Aural-to-Visual tasks. In Figure B.5, the responses for both renderers and trajectory paths were aggregated and analyzed with boxplots for each task. The tested curvature conditions are represented on the x-axis and the curvature conditions chosen by participants are represented on the y-axis. The dashed lines indicate the correct choices are located in the analyses, and the circle within each box represents the median curve chosen across participants. The length of the box represents the 75% and 25% quantiles of participant responses. Successful trials with zero error would show the median value intersect with the dashed line. The results of this analysis indicate that participant responses in the Visual-to-Aural task are more consistent and closer to the target curvature compared to the Aural-to-Visual task where responses are less consistent and further from the target curvature. However, participants performed similarly on both tasks for the condition of no curvature (curve 0). By comparing these tasks based only on human performance and not rendering quality, this analysis suggests that the task of matching a curvature of an audio trajectory path to a target visual image (Visual-to-Aural) is easier than matching a curvature of visual images to a target curvature of an audio trajectory path (Aural-to-Visual). As we saw before when comparing tasks, participants performed more consistently and accurately on the Visual-to-Aural task compared to the Aural-to-Visual tasks for these paths. In the Aural-to-Visual task, participants tended to underestimate the curvature amount, and while tending to overestimate in the Visual-to-Aural task. However, we observed that the performance across the FO-BO paths was more consistent and accurate than the other paths. Figure B.9 shows the boxplot analysis of participant responses, which are generally closer to the target curvature and more consistent when compared to the FL-FR and F-B paths.

It suggests that paths that curve left or right of the listener's head are more easily perceived than paths that curve up and down. This is in line with aural localization perception [11], where sources that differ in azimuth are easier to discriminate than sources that differ in elevation. In addition, it suggests that certain areas in the design of the auditory display are more difficult to aurally discriminate than others. To investigate why there are differences in performance between paths, we analyzed how both rendering techniques produce Binaural cues on each trajectory path and curvature.

We believe the results of this study are due to the design of the user interface, which biased the participant responses by allowing them to ignore the matching modality and just focus on the target modality and the GUI slider. This allowed them to use their memory of previous trials to respond even though we tried to compensate for this using the shifting slider values. If we focus on how participants conducted the task (target curvature in one modality to GUI Slider) instead of how we intended (target curvature in one modality to the other modality), discriminable auditory spatial cues are needed to conduct the Aural-to-Visual task. The performance in this task in the FO-BO paths suggest that this task is easier when these cues are discernible and dependent on the positions of the trajectory path. As mentioned before, we did not test the ability to discriminate the curves of trajectory paths in this study, but testing the Just-Noticeable-Difference (JND) of the curves for each trajectory path in the aural modality could further explain the required precision of curvature needed to properly complete this task. In addition, improvements in the user interface and experimental design could help enforce participants to relate the modalities to better understand how 3D graphics are related to 3D audio listening experience. These improvements along with measurements and analysis of auditory displays, rendering techniques, room acoustics, and psychoacoustics of listeners could provide the necessary information to design visual representations that better inform practitioners how their spatial mixing decisions will affect their auditory spatial perception. The following further explains the analysis methodology of distributions, effect sizes, and confidence intervals used to report the data in Chapter 7. For further details on the use of this analysis method, please refer to the references of Cumming and Finch [START_REF] Cumming | Inference by eye: confidence intervals and how to read pictures of data[END_REF], Cumming [START_REF] Cumming | The new statistics: Why and how[END_REF], Kirby and Gerlanc [START_REF] Kirby | BootES: An R package for bootstrap confidence intervals on effect sizes[END_REF], and Dragicevic [START_REF] Dragicevic | Fair statistical communication in HCI[END_REF].

c.0.1 Distributions

Probability distributions are continuous functions used to model observations and report statistical analyses, where the normal distribution function is commonly used. Normal distributions describe the probability of an observation around an average mean, µ, and the probability of an observation reduces when it is further from the mean. The amount of reduction is characterized by its standard deviation, . The Central Limit Theorem describes that the average mean of an infinite number of samples drawn from a distributed random variable makes is normally distributed. This is under the assumption that the normal distribution is independent of the source distribution, which would make the computed measures normally distributed in experimental analyses. However, they are more likely to produce an approximate normal distribution due to many uncontrollable factors in many user-based studies.

In common HCI-based studies, non-negative measures (completion time and distance error) are used to measure performance of a task, but they cannot be normally distributed due to their non-negative values, which skews the distribution towards a positive direction. Instead, we use a log-transform for analyzing the distribution of these measures, which takes the natural log of the measures and then the inverse for reporting the averages and confidence intervals. This allows the mean averages to be less affected by large values and makes the confidences intervals assymetric. c.0.2 Effect Sizes and Confidence Intervals Effect sizes are average means of anything of interest (measures) that may be affected by factors of an experiment, such as accuracy, precision, completion time, etc. If the average mean vary greatly across these factors, then it suggest that these factors have an effect on the measure of interest [START_REF] Cumming | The new statistics: Why and how[END_REF]. For our purposes, we computed the symmetric effect size through percent differences, which is the difference of means normalized by their average (see Equation C.1). This computation allows to better account for cases when one mean is much larger than the other.

Effect Size = (µ 1 -µ 2 ) (µ 1 + µ 2 )/2 ⇤ 100 (C.1)
Confidence intervals provide an added detail to the effect sizes to show how far the measures are from the average mean. Commonly, 95% confidence intervals are used to analyze the range of measures, where smaller confidence intervals that are closer to the average mean suggest that the measures are more predictable and larger confidence intervals suggest that the measure are less predictable [START_REF] Cumming | The new statistics: Why and how[END_REF].

The effect sizes and confidence intervals were computed using bootstrap methods due to the small number of samples within our study and the Central Limit Theorem requiring an infinite number of samples. Bootstrap methods approximate the unknown distribution of samples by repeatedly drawing random samples from the original data set to increase the number of samples to 2000 samples. The larger data set estimates the distribution of measures if more subjects participated in the study. From this approximate distribution, effect sizes and confidence intervals can be computed and reported that provides a better prediction of the expected results with additional subjects [START_REF] Kirby | BootES: An R package for bootstrap confidence intervals on effect sizes[END_REF].

D

A N N E X E D : S Y N T H È S E E N F R A N Ç A I S L'avancement des technologies des médias nous a permis d'avoir une grande quantité de contenu et les tendances récentes dans le développement de technologies de médias immersives ont été de plus en plus populaires auprès des consommateurs. Les technologies immersives simulent notre perception spatiale de notre environnement et elles peuvent fournir des expériences qui brouillent la notion de simulation et de réalité grâce à la stimulation de nos sens. Cela modifie notre sens de la présence spatiale d'être dans le monde physique à une présence spatiale d'un environnement différent [START_REF] Mestre | Immersion et présence[END_REF]. Un objectif majeur des technologies immersives est d'améliorer l'expérience spatiale grâce à des affichages visuels, comme de grands écrans visuels 2D, des lunettes 3D et des casques de réalité virtuelle. Il y a eu des intérêts significatifs à fournir une sensation d'immersion spatiale auriculaire aussi. La technologie audio 3D donne aux auditeurs l'impression d'environnements sonores avec des sources sonores provenant de différentes directions à l'aide affichages auditifs 3D. En ce moment, les technologies permettant de reproduire le contenu audio 3D sont facilement disponibles pour les consommateurs. Dans les salles de cinéma et à domicile, les systèmes audio qui utilisent des configurations de haut-parleurs 3D couplés à des techniques de rendu audio 3D deviennent des méthodes populaires pour augmenter le sentiment d'immersion. Dans les jeux vidéo, les technologies de l'audio spatial sont utilisées pour améliorer l'expérience des scénarios de jeu, et plus récemment, le contenu de la réalité virtuelle a utilisé l'audio spatial pour augmenter l'immersion de l'environnement virtuel. Même si la technologie audio 3D peut être appliquée à des applications utilisant des affichages visuels, les compositeurs et les ingénieurs audio ont suscité beaucoup d'intérêt pour créer un sens de l'espace acoustique grâce à des performances de compositions musicales et d'une technologie d'enregistrement/diffusion audio.

d.1 interfaces utilisateur pour les outils de production audio 3d

Des outils de production audio ont été développés en mettant l'accent sur le contrôle des propriétés des algorithmes de signaux audio. Ces algorithmes ont été conçus en fonction des méthodes d'enregistrement et de la façon dont la production sera diffusée et stockée. Les productions audio initiales consistaient en un seul microphone qui convertissait les ondes sonores générées par les artistes et les musiciens en signaux électriques analogiques pouvant être mesurés par la tension, le courant et la puissance, qui étaient ensuite directement dirigés vers le système de diffusion ou un support de stockage [START_REF] Owsinski | The Recording Engineer's Handbook[END_REF]. Comme les ingénieurs ont commencé à utiliser plus de microphones pour capter une qualité sonore supérieure, les techniques de traitement du signal électrique ont été utilisées pour mélanger les signaux de chaque microphone. Le mixage des signaux audio consistait à utiliser un mélangeur multicanal qui utilisait un algorithme de sommation de traitement du signal pour additionner des signaux audio qui consistaient à diriger et ajuster la puissance de chaque signal à travers des amplificateurs de gain pour créer un équilibre de clarté et d'intelligibilité [START_REF] Owsinski | The Mixing Engineer's Handbook[END_REF]. Dans le modèle mental de cette conception, le routage et la sommation du signal audio sont considérés comme des tuyaux que les utilisateurs doivent connecter ensemble pour le mélange. Ce modèle a finalement été développé pour inclure des techniques de traitement du signal électrique qui détourneraient le routage pour manipuler le signal audio, puis le rediriger vers le routage d'origine. Ces effets audio sont considérés comme des «plug-ins» insérés dans le routage principal, permettant aux utilisateurs de manipuler les propriétés du signal audio, comme le filtrage d'égalisation (passe-bas, passe-bande, arrêt de bande, passe-haut , multi-paramétrique, etc.) [START_REF] Owsinski | The Mixing Engineer's Handbook[END_REF].

Pour les professionnels de l'audio 3D, le contenu peut actuellement être créé à partir d'une grande variété d'outils allant des bibliothèques de programmation aux interfaces utilisateur 3D. Avec ces outils, les paramètres spatiaux peuvent être assignés à un signal audio, qui sera ensuite interprété par les systèmes de rendu afin de calculer comment le signal doit être dirigé vers des haut-parleurs spécifiques afin de créer l'expérience spatiale. Cela crée un paradigme d'interaction différent, où les interfaces utilisateur pour l'interaction 3D peuvent être utilisées plutôt que le contrôle du traitement et du routage du signal de bas niveau au sein des mélangeurs multicanaux ou stations de Digital Audio Workstations (DAWs). Cependant, la technologie audio 3D étant relativement nouvelle sur le marché grand public et la popularité actuelle des mixeurs multicanaux standard et DAWs, interfaces utilisateur pour la production audio 3D ont eu peu d'avancement dans cette direction.

Au lieu de cela, les praticiens ont été obligés de configurer divers outils pour produire correctement du contenu audio 3D de la manière qu'ils aimeraient. Dans cette thèse, nous étudions ce problème avec des outils, les méthodes et les pratiques du praticien actuel, la perception spatiale pendant l'interaction et la convivialité des interfaces utilisateur au sein de ces outils. Grâce à ce travail, nous avons développé un cadre de conception, Spatial Audio Design Spaces (SpADS), qui organise et classe les critères de conception des interfaces utilisateur qui sont importants lors de la production de contenu audio 3D. d.2 un cadre de conception pour les interfaces utilisateur des outils de production audio 3d

Les cadres de conception sont couramment utilisés dans le domaine de l'interaction homme-machine (IHM) qui fournissent des méthodologies sur la façon de concevoir un produit particulier, une fonctionnalité ou une interaction. Ceux-ci peuvent être trouvés sous différentes formes, comme la méthodologie Model-View-Controller (MVC) de Krasner and Pope [START_REF] Krasner | A description of the model-view-controller user interface paradigm in the smalltalk-80 system[END_REF] pour la création d'applications logicielles ou le cadre Genex de Shneiderman [START_REF] Shneiderman | Creating Creativity: User Interfaces for Supporting Innovation[END_REF] en quatre phases pour les interfaces utilisateurs qui soutiennent l'innovation. Une autre méthodologie commune consiste à effectuer une analyse morphologique qui caractérise les propriétés intrinsèques des objets d'intérêt et de leurs dépendances à travers les espaces de conception. Cette dernière stratégie a été initialement appliquée à la conception de dispositifs d'entrée pour mieux comprendre comment leurs diverses caractéristiques affectent la convivialité [START_REF] Stuart | A Morphological Analysis of the Design Space of Input Devices[END_REF]. Nous avons adopté cette méthode en raison de divers outils audio 3D et des techniques de production actuellement disponibles et avons utilisé cette stratégie pour développer une méthodologie sur la façon de créer des interfaces utilisateur pour les outils de production audio 3D. Cependant, de nombreux facteurs doivent être pris en considération en raison de la diversité des techniques audio 3D, de la conception des systèmes de production, de la nature perceptuelle multimodale de l'interaction, et de la façon dont les interfaces permettent aux praticiens d'interagir avec les objets d'intérêt. Dans ce travail, nous avons observé qu'il existe des dépendances entre les techniques d'enregistrement et de rendu et la façon dont les interfaces utilisateur sont conçues. Nous avons utilisé ces assertions comme base pour développer le framework Spatial Audio Design Spaces (SpADS) comme un ensemble de deux espaces de conception qui classent les objets audio 3D comme des objets d'intérêt pendant l'interaction (SpADS-A) et comment les interfaces utilisateur sont conçues pour les contrôler (SpADS-C). Dans cette thèse, nous présentons le développement de ce cadre en deux parties: Audio 3D: outils, méthodes et pratiques et Conception d'interaction pour la création audio 3D. tive qui peut aider les concepteurs à améliorer la conception actuelle et à concevoir de nouveaux outils de production.

d.2.2 Partie II -Conception d'interaction pour la création audio 3D

Commentaires des praticiens discutés au chapitre 3 mentionné les limitations des techniques d'interaction dans les outils audio 3D. En particulier, les commentaires et les réponses se sont concentrés sur les techniques d'interaction pour contrôler les objets audio et créer des trajectoires 3D identifiant les limitations dans leur facilité d'utilisation. Nous avons étudié ce problème en analysant comment les différentes caractéristiques des méthodes d'entrée et de mappage des interfaces utilisateur affectent les tâches liées à la création et à l'édition de trajectoires 3D. Dans le chapitre 6, nous examinons d'abord les méthodologies de conception, les méthodes d'entrée et les stratégies de cartographie qui sont importantes dans la conception des interfaces utilisateur audio 3D. Grâce à cet examen, nous avons encore amélioré le cadre SpADS pour inclure des caractéristiques importantes d'entrée et de cartographie (voir figure D.3). Les descriptions d'entrée sont caractérisées comme Resolution of Acquisition, Multiplexing, et Integrality qui décrit la résolution de la méthode d'entrée, si les entrées sont multiplexées spatialement ou temporellement et le degré de liberté des entrées respectivement. Les descriptions de mappage sont caractérisées comme Direct (valeurs de l'entrée mappées directement à l'objet audio), Intermediate (les valeurs de l'entrée sont converties en d'autres données contextuelles puis mappées à l'objet audio), et Augmented (les valeurs de l'entrée passent par des fonctions de transfert avancées).

Nous avons ensuite exploré certaines des dimensions de l'espace de conception à travers une étude qui a étudié différentes stratégies de cartographie pour les interfaces de bureau en fonction de leurs stratégies de cartographie et des caractéristiques variables du multiplexage et de l'intégralité des méthodes d'entrée dans le chapitre 7. Deux techniques d'interaction avec des méthodes de mapping intermédiaires utilisant des curseurs de base (GUI Sliders) et timeline (GUI Timeline) ont été comparées à trois techniques d'interaction avec des méthodes de mapping direct. Ces techniques d'interaction de cartographie directe consistaient en des curseurs physiques spatialement multiplexés et totalement séparés (Physical Sliders), des zones spatialement multiplexées et semi-intégrées sur un trackpad (Zonal Trackpad) et une entrée totalement intégrée 3DoF périphérique (Phantom). Ces techniques ont été évaluées à travers trois tâches liées à la création d'une trajectoire 3D, que nous avons classée en ajoutant un point de contrôle à une trajectoire existante (Add-Task), en insérant un point de contrôle dans une trajectoire existante (Insert-Task) et en sélectionnant un point de contrôle dans une trajectoire existante (Select-Task). Nous avons mesuré la capacité des participants à effectuer ces tâches rapidement en fonction du temps d'exécution et de la précision en fonction de l'erreur de distance entre les tâches (voir figure D.4) et les techniques d'interaction (voir figure D.5).

Ces résultats suggèrent que l'augmentation de la directivité de la mapping et l'intégralité des méthodes d'entrée améliore le temps d'achèvement des tâches. Les observations et les commentaires des participants ont également suggéré que l'intégralité des méthodes d'entrée affecte les mouvements de la caméra, les références dans le retour visuel sont importantes pour accomplir les tâches et la structure de la tâche peut être opérationnalisée en tâches séparées plutôt qu'intégrale. En plus de ces résultats, cette étude démontre comment la variation des méthodes d'entrée et de mappage peut affecter la facilité d'utilisation lors de l'exécution de tâches de production audio 3D spécifiques.

d.3 conclusion

En conclusion, les travaux présenté dans cette thèse explore différents critères de conception qui devraient être considérés pour améliorer l'état actuel des outils de production audio 3D. Du point de vue d'un praticien, il est important d'avoir la flexibilité des systèmes de production actuels afin d'inclure des fonctionnalités avancées avec des techniques d'enregistrement et de rendu audio 3D, la création et la capacité de produire dans diverses configurations de haut-parleurs. Dans les interfaces utilisateur de production audio 3D, il est nécessaire d'améliorer le retour visuel pour qu'ils soient plus informatifs et puissent être comparés à des expériences d'écoute. De plus, nous avons classé et étudié comment les méthodes d'entrée et de mappage des interfaces utilisateur audio 3D peuvent affecter la exactitude et le temps d'achèvement des tâches de création de trajectoire audio 3D. Dans l'ensemble, ce travail a fourni des résultats qui aident à organiser et classifier les considérations de conception pour former les espaces de conception du cadre SpADS. Ce cadre fournit des pouvoirs descriptifs et analytiques qui permettent aux concepteurs de décrire les capacités des outils de production audio 3D, d'analyser leurs fonctionnalités et d'évaluer comment les changements de dimensions affectent la convivialité. Ce cadre peut aider les concepteurs à identifier les limites, suggérer des améliorations et identifier les choix de conception qui améliorent la convivialité des outils de production audio 3D.

Figure 2 . 1 15 Figure 2 . 2 16 Figure 2 . 3 19 Figure 2 . 4 20 Figure 2 . 5 21 Figure 2 . 6 22 Figure 3 . 1 32 Figure 3 . 2 33 Figure 3 . 3 35 xFigure 3 . 4 36 Figure 3 . 5

 211522162319242025212622313232333335343635 Figure 2.1 Examples of Direct Channel-Based recording and decoding technique with a Decca Tree Surround Sound microphone for a 5 channel Surround Sound speaker configuration and an ambisonic recording and decoding technique with TetraMic for a 3D speaker configuration. 15 Figure 2.2 Images describing Position-Based techniques using VBAP borrowed from Pulkki [99] (a), the spherical nature of HOA panning borrowed from Neukom [87] (b), Wave-Field Synthesis borrowed from Berkhout, Vries, and Vogel [9] (c), and Binaural rendering technique (d). 16 Figure 2.3 Screenshots of the Neve DFC PS-1 mixing console (a) and the ProTools HD Digital Audio Workstation (b). 19 Figure 2.4 Screenshots of Dolby Atmos (a & b) and Sonic Emotion Wave 1 (c & d) software tools that are examples of consumer-based 3D audio tools for Digital Audio Workstations. 20 Figure 2.5 Screenshot of a production in Longcat's AudioStage software. A 3D visualization is used for 3D interaction and a sequencer is used for temporal properties of the production. 21 Figure 2.6 Screenshot of SoundScape Renderer (a), OMPrisma for Open Music (b), and Holo-Edit 3D audio software, which are examples of software applications that have been developed from research institutions. 22 Figure 3.1 Classification of comments of interviews with previous mentioned limitations that defined needs and limitations in Recording/Rendering (red), Mixing (green), and Monitoring (blue). 32 Figure 3.2 Model that identifies stages in audio production and how audio objects are created and manipulated. 33 Figure 3.3 The variety of Roles (a), Projects (b), Listening Formats (c) and Virtual Spaces (d) in 3D audio highlights the need for flexible spatialization tools. 35

37 Figure 3 . 6 38 Figure 3 . 7 41 Figure 4 . 1 47 Figure 4 . 2 48 Figure 4 . 3 50 Figure 4 . 4 3D 55 Figure 5 . 1 60 Figure 5 . 2 61 Figure 5

 37363837414147424843504455516052615 Figure 3.4 Word cloud visualization of the spatialization software used by participants. Text size relates to the number of participants who mentioned using the corresponding software. 36 Figure 3.5 The 3D audio renderers referenced as being used by the participants in their projects. 37 Figure 3.6 Types of visualizations participants use to indicate location in a trajectory (a) and verify placement and movement (b). 38 Figure 3.7 The way participants edit trajectories with Patterns (a), Input (b), Edit (c) and Control methods, highlights the variety of methods to define parameters in time with existing tools. 41 Figure 4.1 Diagram of dimensions used to define the auditory localization position by azimuth, elevation, and distance of a sound source with regards to the listener's head. 47 Figure 4.2 Diagram of how binaural cues of ITD and ILD/IID are formed by the anatomy of the listener, and how it affects how the sound waves arrive at each ear. 48 Figure 4.3 Diagram of how listeners receives information from sound waves that travel the direct path of the sound to the listener and the indirect paths (early and later reflections) in a reverberant room. The comparison between the two creates the direct to reverberant energy cue. 50 Figure 4.4 3D graphical renderings of virtual objects from an egocentric point of view (a) and of an individual surrounded by virtual objects from an exocentric point of view (b) that are based on the definitions of Klatzky [65]. 55 Figure 5.1 Version 1 of the SpADS-A that classifies and describes audio objects by their recording and rendering techniques. 60 Figure 5.2 Version 1 of the SpADS-C that classifies and describes audio object control methods by their input devices and visual feedback. 61 Figure 5.3 3D SpADS Representation 61 Figure 5.4The classification of various types of audio objects by their recording methods, rendering methods, and possible interactive spatial parameters in a 3D scene using SpADS-A. 62

Figure 5 . 5

 55 Figure 5.5The classification of various types of input devices by their input method and visual feedback that can be directly mapped to possible interactive spatial parameters of audio objects using SpADS-C. 64 Figure 5.6 Images of the Neve DFC-Gemini console joysticks, and images of the Dolby Atmos and Schoep's Double M/S software tools. 65 Figure 5.7 Analysis of the Neve DFC-Gemini and Dolby Atmos system through SpADS with possible audio objects it can render in SpADS-A (a), and the associated control methods in SpADS-C and (b). 66 Figure 5.8 Example of how the SpADS framework can be used to suggest improvements on the rendering techniques to provide additional manipulable parameters in SpADS-A (a), and how those parameters can be controlled with other input devices in SpADS-C (b). 68 Figure 5.9 Classification of three major tasks in 3D audio production. 70 Figure 5.10 This flowchart describes one method of designing a rendering space, where the rendering techniques are chosen first, then a matching speaker configuration is designed. These choices define the listener's properties and the room properties are then finally considered. 71 Figure 5.11 SpADS-A v2 with new descriptions shows how different audio objects can be classified based on their recording method and whether it is rendered as a Field-Based or Position-Based with more descriptive spatial parameters of Translation, Directivity, and Orientation. 75 Figure 5.12 SpADS-C v2 with new descriptions (highlighted in yellow) that describe how input devices can be mapped to control different spatial properties and the types of visual feedback through defining control methods as Input Method, Mapping Method, and their Visual Feedback. 76 Figure 5.13SpADS-A v2: This analysis classifies possible audio objects for the Dolby Atmos and Neve DFC system by their recording and rendering method, and possible spatial parameters that could be interacted with in a 3D virtual scene. 78

Figure 5 . 6 65 Figure 5 . 7 66 Figure 5 . 8 68 Figure 5 . 9 70 Figure 5 . 10

 5665576658685970510 Figure 5.5The classification of various types of input devices by their input method and visual feedback that can be directly mapped to possible interactive spatial parameters of audio objects using SpADS-C. 64 Figure 5.6 Images of the Neve DFC-Gemini console joysticks, and images of the Dolby Atmos and Schoep's Double M/S software tools. 65 Figure 5.7 Analysis of the Neve DFC-Gemini and Dolby Atmos system through SpADS with possible audio objects it can render in SpADS-A (a), and the associated control methods in SpADS-C and (b). 66 Figure 5.8 Example of how the SpADS framework can be used to suggest improvements on the rendering techniques to provide additional manipulable parameters in SpADS-A (a), and how those parameters can be controlled with other input devices in SpADS-C (b). 68 Figure 5.9 Classification of three major tasks in 3D audio production. 70 Figure 5.10 This flowchart describes one method of designing a rendering space, where the rendering techniques are chosen first, then a matching speaker configuration is designed. These choices define the listener's properties and the room properties are then finally considered. 71 Figure 5.11 SpADS-A v2 with new descriptions shows how different audio objects can be classified based on their recording method and whether it is rendered as a Field-Based or Position-Based with more descriptive spatial parameters of Translation, Directivity, and Orientation. 75 Figure 5.12 SpADS-C v2 with new descriptions (highlighted in yellow) that describe how input devices can be mapped to control different spatial properties and the types of visual feedback through defining control methods as Input Method, Mapping Method, and their Visual Feedback. 76 Figure 5.13SpADS-A v2: This analysis classifies possible audio objects for the Dolby Atmos and Neve DFC system by their recording and rendering method, and possible spatial parameters that could be interacted with in a 3D virtual scene. 78

Figure 7 . 1

 71 Figure 7.1 Screenshots of Longcat's AudioStage (a) and Sonic Emotion Wave 1 Performer (b), that have trajectory authoring capabilities through automation drawing and trajectory templates respectively. 107 Figure 7.2GUI Slider interaction technique using an intermediate mapping strategy using a trackpad to control a mouse cursor that controls GUI sliders to manipulate spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b). 108 Figure 7.3 GUI Timeline interaction technique using a trackpad, keyboard shortcuts, and GUI timeline to control spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b). 110 Figure 7.4 Zonal Trackpad interaction technique using predefined zones on the trackpad to control spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b). 111 Figure 7.5 Physical Slider interaction technique where each slider is mapped directly to 3D position of an audio object (a) and a SpADS-C analysis of the technique. 112 Figure 7.6 Phantom controller interaction technique that is directly mapped to the 3D position of an audio object (a) and a SpADS-C analysis of the technique. 113 Figure 7.7 SpADS-C analysis and comparison of the five interaction techniques for 3D translation. 114 Figure 7.8 GUI used to test each interaction technique. 115 Figure 7.9 Flowchart of experimental protocol during each session. 116 Figure 7.10 Example of a completed Add-Task 118 Figure 7.11 Example of a completed Insert-Task 118 Figure 7.12 Example of a completed Select-Task 119 Figure 7.13 Completion time (a) and distance error across task blocks (interaction technique block = 3 task blocks). 121 Figure 7.14 Comparison of accuracy and completion time by tasks. 122 Figure 7.15 Comparison of average distance error by interaction techniques. 123 Figure 7.16 Comparison of average completion time by interaction techniques. 124

Figure 7 . 7 115 Figure 7 . 9 116 Figure 7 . 10 122 Figure 7 . 15

 7711579116710122715 Figure 7.1 Screenshots of Longcat's AudioStage (a) and Sonic Emotion Wave 1 Performer (b), that have trajectory authoring capabilities through automation drawing and trajectory templates respectively. 107 Figure 7.2GUI Slider interaction technique using an intermediate mapping strategy using a trackpad to control a mouse cursor that controls GUI sliders to manipulate spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b). 108 Figure 7.3 GUI Timeline interaction technique using a trackpad, keyboard shortcuts, and GUI timeline to control spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b). 110 Figure 7.4 Zonal Trackpad interaction technique using predefined zones on the trackpad to control spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b). 111 Figure 7.5 Physical Slider interaction technique where each slider is mapped directly to 3D position of an audio object (a) and a SpADS-C analysis of the technique. 112 Figure 7.6 Phantom controller interaction technique that is directly mapped to the 3D position of an audio object (a) and a SpADS-C analysis of the technique. 113 Figure 7.7 SpADS-C analysis and comparison of the five interaction techniques for 3D translation. 114 Figure 7.8 GUI used to test each interaction technique. 115 Figure 7.9 Flowchart of experimental protocol during each session. 116 Figure 7.10 Example of a completed Add-Task 118 Figure 7.11 Example of a completed Insert-Task 118 Figure 7.12 Example of a completed Select-Task 119 Figure 7.13 Completion time (a) and distance error across task blocks (interaction technique block = 3 task blocks). 121 Figure 7.14 Comparison of accuracy and completion time by tasks. 122 Figure 7.15 Comparison of average distance error by interaction techniques. 123 Figure 7.16 Comparison of average completion time by interaction techniques. 124

Figure 7 . 17 126 Figure 7 . 18

 717126718 Figure 7.17 Comparison of average percentage of movement time and stationary time per trial across Physical Sliders, Zonal-Trackpad, and Phantom Desktop. 126 Figure 7.18 Average camera movements per trial across Physical Sliders, Zonal-Trackpad, and Phantom Desktop for each task. 127 Figure 7.19 NDC analysis of average time spent interacting with a combination of dimensions with threshold of 0 m (red), 0.0025 m (green), 0.005 m (blue), and 0.01 m (purple). 129 Figure 7.20 Rating of how difficult it was to learn each interaction technique. 131 Figure 7.21 Preferences of interaction techniques within the Add-Task (a), Insert-Task (b), and Select-Task(c). 132 Figure 7.22 Difficulty rating of interaction techniques within the Add-Task (a), Insert-Task (b), and Select-Task(c). 133 Figure 8.1 SpADS-A provides a space to describe the interactive spatial parameters of audio objects by their recording and rendering techniques. 137 Figure 8.2 SpADS-C provides a space to describe control methods of audio objects by their characteristics of the visual feedback, input method, and mapping method. 138 Figure 8.3 Model that identifies stages in audio production and how audio objects are created and manipulated. 140 Figure 8.4 Classification of three major tasks in 3D audio production. 141 Figure 8.5 Flowchart on how to use virtual speakers to decode any audio object onto any auditory display. 142 Figure 8.6 Model of production tasks for multi-modal immersive content. 145 Figure B.1 3D model of the cylindrical speaker configuration used for rendering the audio trajectories (a), and photo of a participant within the real speaker configuration and experimental room (b). Not all speakers shown in the photo of the participant were used for the rendering space in this study. 175 Figure B.2 Screenshot of GUI used in the experiment 178 Figure B.3 Flowchart of experimental protocol. 180 Figure B.4 Slider mapping method 180 Figure B.5 Boxplot analysis of Aural-to-Visual task (a) and Visual-to-Aural task (b) across each curvature amount. 182

Figure 8 . 6 Figure B. 6

 866 Figure B.6 Boxplot analysis of HOA across Aural-to-Visual task (a) and Visual-to-Aural task (c), and VBAP across Aural-to-Visual task (b) and Visual-to-Aural task (d). 183 Figure B.7 Analysis of FL-FR & FR-FL paths across renderers and tasks. 184 Figure B.8 Analysis of FL-BL & FR-BR paths across renderers and tasks. 184 Figure B.9 Analysis of FO-BO path across renderers and tasks. 185 Figure D.1 SpADS-A avec la façon dont différents objets audio peuvent être classés en fonction de leur méthode d'enregistrement et si elle est rendue sur Field-Based ou Position-Based avec des paramètres spatiaux descriptifs de translation, directivité et d'orientation. 192 Figure D.2 SpADS-C v2 qui décrivent comment les périphériques d'entrée peuvent être mappés pour contrôler différentes propriétés spatiales et les types de rétroaction visuelle en définissant des méthodes de contrôle comme méthode d'entrée, méthode de mappage et leurs commentaires visuels. 193 Figure D.3 Les nouvelles descriptions d'entrée et de mappage de SpADS-C sans la description de retour visuel. 194 Figure D.4 Comparaison de la précision et du temps de réalisation par tâches. 196 Figure D.5 Comparaison de la précision et du temps d'achèvement par des techniques d'interaction. 197

53 A 1 I

 531 C R O N Y M S A N D T E R M S HCI -Human-Computer Interaction DAW -Digital Audio Workstation Listening Format -Audio file format and associated speaker configuration GUI -Graphical User Interface 3DUI -3D User Interfaces VBAP -Vector-Based Amplitude Panning DBAP -Distance-Based Amplitude Panning HOA -High-Order Ambisonics WFS -Wave-Field Synthesis HRTF -Head-Related Transfer Function ITD -Interaural-Time-Difference IID -Interaural-Intensity-Difference ILD -Interaural-Level-Difference VST -Virtual Studio Technology xix N T R O D U C T I O N

Figure 2 . 1 -

 21 Figure 2.1 -Examples of Direct Channel-Based recording and decoding technique with a Decca Tree Surround Sound microphone for a 5 channel Surround Sound speaker configuration and an ambisonic recording and decoding technique with TetraMic for a 3D speaker configuration.

Figure 2 . 2 -

 22 Figure 2.2 -Images describing Position-Based techniques using VBAP borrowed from Pulkki [99] (a), the spherical nature of HOA panning borrowed from Neukom [87] (b), Wave-Field Synthesis borrowed from Berkhout, Vries, and Vogel [9] (c), and Binaural rendering technique (d).

Figure 2 . 3 -

 23 Figure 2.3 -Screenshots of the Neve DFC PS-1 mixing console (a) and the Pro-Tools HD Digital Audio Workstation (b).

2. 4

 4 (b)) and Auro 3D 6 for cinema, Spatial Workstation 7 for virtual reality, and Sonic Emotion Wave I 8 tools for venue performances and installations (see Figures 2.4(c) & 2.4(d))

  (a) Dolby Atmos Panner Plug-in (b) Dolby Atmos Monitoring Application (c) Sonic Emotion Wave 1 Designer (d) Sonic Emotion Wave 1 Performer

Figure 2 . 4 -

 24 Figure 2.4 -Screenshots of Dolby Atmos (a & b) and Sonic Emotion Wave 1 (c & d) software tools that are examples of consumer-based 3D audio tools for Digital Audio Workstations.

Figure 2 . 5 -

 25 Figure 2.5 -Screenshot of a production in Longcat's AudioStage software. A

Figure 2 . 6 -

 26 Figure 2.6 -Screenshot of SoundScape Renderer (a), OMPrisma for Open Music (b), and Holo-Edit 3D audio software, which are examples of software applications that have been developed from research institutions.

•••Figure 3 . 1 -

 31 Figure 3.1 -Classification of comments of interviews with previous mentioned limitations that defined needs and limitations in Recording/Rendering (red), Mixing (green), and Monitoring (blue).

Figure 3 . 2 -

 32 Figure 3.2 -Model that identifies stages in audio production and how audio objects are created and manipulated.

Figure 3 . 3 -

 33 Figure 3.3 -The variety of Roles (a), Projects (b), Listening Formats (c) and Virtual Spaces (d) in 3D audio highlights the need for flexible spatialization tools.

Figure 3 . 4 -

 34 Figure 3.4 -Word cloud visualization of the spatialization software used by participants. Text size relates to the number of participants who mentioned using the corresponding software.

Figure 3 . 5 -

 35 Figure 3.5 -The 3D audio renderers referenced as being used by the participants in their projects.

Figure 3 . 6 -

 36 Figure 3.6 -Types of visualizations participants use to indicate location in a trajectory (a) and verify placement and movement (b).

Figure 3 . 7 -

 37 Figure 3.7 -The way participants edit trajectories with Patterns (a), Input (b), Edit (c) and Control methods, highlights the variety of methods to define parameters in time with existing tools.

Figure 4 . 1 -

 41 Figure 4.1 -Diagram of dimensions used to define the auditory localization position by azimuth, elevation, and distance of a sound source with regards to the listener's head.

Figure 4 . 2 -

 42 Figure 4.2 -Diagram of how binaural cues of ITD and ILD/IID are formed by the anatomy of the listener, and how it affects how the sound waves arrive at each ear.

Figure 4 . 3 -

 43 Figure 4.3 -Diagram of how listeners receives information from sound waves that travel the direct path of the sound to the listener and the indirect paths (early and later reflections) in a reverberant room. The comparison between the two creates the direct to reverberant energy cue.

  Binaural ITD, ILD/IID, Precedence Effect Monaural Pinnae Spectral Dynamic Changes in ITD and ILD/IID Distance Loudness, Direct-to-Reverb Ratio

Figure 4 . 4 -

 44 Figure 4.4 -3D graphical renderings of virtual objects from an egocentric point of view (a) and of an individual surrounded by virtual objects from an exocentric point of view (b) that are based on the definitions of Klatzky [65].

Figure 5 . 1 -

 51 Figure 5.1 -Version 1 of the SpADS-A that classifies and describes audio objects by their recording and rendering techniques.

Figure 5 . 2 -

 52 Figure 5.2 -Version 1 of the SpADS-C that classifies and describes audio object control methods by their input devices and visual feedback.

Figure 5 . 3 -

 53 Figure 5.3 -3D SpADS Representation

Figure 5 . 4 -

 54 Figure 5.4 -The classification of various types of audio objects by their recording methods, rendering methods, and possible interactive spatial parameters in a 3D scene using SpADS-A.

Figure 5 . 5 -

 55 Figure 5.5 -The classification of various types of input devices by their input method and visual feedback that can be directly mapped to possible interactive spatial parameters of audio objects using SpADS-C.

Figure 5 . 6 -

 56 Figure 5.6 -Images of the Neve DFC-Gemini console joysticks, and images of the Dolby Atmos and Schoep's Double M/S software tools.

Figure 5 . 7 -

 57 Figure 5.7 -Analysis of the Neve DFC-Gemini and Dolby Atmos system through SpADS with possible audio objects it can render in SpADS-A (a), and the associated control methods in SpADS-C and (b).

  Figure 5.8(a)). Once transposed to SpADS-C (highlighted in red in Figure 5.8(b))

Figure 5 . 8 -

 58 Figure 5.8 -Example of how the SpADS framework can be used to suggest improvements on the rendering techniques to provide additional manipulable parameters in SpADS-A (a), and how those parameters can be controlled with other input devices in SpADS-C (b).

Figure 5 . 9 -

 59 Figure 5.9 -Classification of three major tasks in 3D audio production.

Figure 5 . 10 -

 510 Figure 5.10 -This flowchart describes one method of designing a rendering space, where the rendering techniques are chosen first, then a matching speaker configuration is designed. These choices define the listener's properties and the room properties are then finally considered.

Figure 5 .

 5 Figure 5.11 -SpADS-A v2 with new descriptions shows how different audio objects can be classified based on their recording method and whether it is rendered as a Field-Based or Position-Based with more descriptive spatial parameters of Translation, Directivity, and Orientation.

Figure 5 . 12 -

 512 Figure 5.12 -SpADS-C v2 with new descriptions (highlighted in yellow) that describe how input devices can be mapped to control different spatial properties and the types of visual feedback through defining control methods as Input Method, Mapping Method, and their Visual Feedback.

Figure 5 .

 5 Figure 5.13 -SpADS-A v2: This analysis classifies possible audio objects forthe Dolby Atmos and Neve DFC system by their recording and rendering method, and possible spatial parameters that could be interacted with in a 3D virtual scene.

Figure 5 .

 5 Figure 5.14 -SpADS-C v2: This analysis classifies the mouse and the dual joysticks with new descriptions of mapping method and associated visual feedback.

Figure 5 . 15 -

 515 Figure 5.15 -Side by side comparison of the V1 and V2 version of SpADS highlights compares its analytical power. SpADS v2, on the right, describes the parameters of the Dolby Atmos and Neve DFC-Gemini productions in more detail through a more accurate analysis of the Double M/S object's directivity description (a) and through the visual feedback of the audio object, 3D scene, and rendering space (b). This is indicated through parameters boxed in green.

Figure 5 . 16 -

 516 Figure 5.16 -Example showing how SpADS-C can be used to describe newinput devices and visualizations that could be used for 3D audio user interfaces. This provides a space on how spatial parameters can be controlled and how visual feedback can assist in the 3D audio tasks. The shapes with black numbers describe the degrees of freedom of the input device, which parameter it can be mapped to, and whether that mapping is direct or augmented. The shapes that describe the spatial parameters within the visual feedback use white numbers to describe the dimension of the visualization.

Figure 5 . 17 -

 517 Figure 5.17 -Examples of new 3D visual representations with mono object (sphere), Double MS object (circular plane), and speaker configuration (boxes) that could be used for the visual feedback in the control methods of audio objects and rendering space properties.

Figure 6 . 1 -

 61 Figure 6.1 -Human-computer communication loop for 3D audio production tools where practitioners use input and mapping methods to control audio objects and receive visual and audio feedback from the computer monitor and the audio display system (renderer and speaker configuration) in the rendering space.

Figure 6 . 2 -

 62 Figure 6.2 -Human-computer communication loop for pointing interaction technique on a computer monitor that consists of a 2D relative mouse input device that is mapped with a computer-to-display gain transfer function to control the 2D position of the computer cursor on the computer monitor.

Figure 6 . 3 -

 63 Figure 6.3 -Human-Computer communication loop for operating a 3D audio application with a mouse input device and a computer monitor for visual feedback and headphones for audio feedback.

Figure 6 . 4 -

 64 Figure 6.4 -Examples of a spatially multiplexed MIDI keyboard and a temporally multiplexed mouse-based input device to control the same virtual piano keys of Steinberg's The Grand 3 VST plug-in.

Figure 6 . 5 -

 65 Figure 6.5 -Examples of input methods that have separated input methods (slider (1D), knobs (1D), buttons (1D)) to increasing integral input methods (mouse (2 DoF), SpaceMouse (6 DoF)).

Figure 6 . 6 -

 66 Figure 6.6 -Example of interaction technique with a direct mapping method to change the width of a rectangle by selecting a boundary point of rectangle and dragging and dropping to a desired width.

Figure 6 . 7 -

 67 Figure 6.7 -Example of interaction technique with an intermediary mapping method to change the width of a rectangle by manipulating the the value of a GUI slider.

Figure 6 . 8 -

 68 Figure 6.8 -Examples of augmented mapping methods, such as physics-based engines, divergent, and convergent, that can be used to map how input methods control audio object spatial parameters.

Figure 6 .

 6 Figure 6.10 -Human-Computer communication loop for 3D audio production with classifications of input and mapping methods.

Figure 6 . 11 -

 611 Figure 6.11 -The new input and mapping descriptions of SpADS-C without the visual feedback description.

Figure 6 . 12 -

 612 Figure 6.12 -SpADS-C Input and Mapping Method analysis of the intermediate methods of the Dolby Atmos and Neve DFC production system with new descriptions.

Figure 6 .

 6 Figure 6.13 -SpADS-C Input and Mapping Method analysis of the direct methods of the Dolby Atmos and Neve DFC production system with new descriptions.

Figure 7 . 1 -

 71 Figure 7.1 -Screenshots of Longcat's AudioStage (a) and Sonic Emotion Wave 1 Performer (b), that have trajectory authoring capabilities through automation drawing and trajectory templates respectively.

  Figure 7.2 -GUI Slider interaction technique using an intermediate mapping strategy using a trackpad to control a mouse cursor that controls GUI sliders to manipulate spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b).

  Figure 7.3 -GUI Timeline interaction technique using a trackpad, keyboard shortcuts, and GUI timeline to control spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b).

  Figure 7.4 -Zonal Trackpad interaction technique using pre-defined zones on the trackpad to control spatial parameters of an audio-object (a), and the SpADS-C analysis of this interaction technique (b).

  (a) Physical Slider Configuration (b) SpADS-C: Physical Slider

Figure 7 . 5 -

 75 Figure 7.5 -Physical Slider interaction technique where each slider is mapped directly to 3D position of an audio object (a) and a SpADS-C analysis of the technique.

Figure 7 . 6 -

 76 Figure 7.6 -Phantom controller interaction technique that is directly mapped to the 3D position of an audio object (a) and a SpADS-C analysis of the technique.

Figure 7 . 7 -

 77 Figure 7.7 -SpADS-C analysis and comparison of the five interaction techniques for 3D translation.

Figure 7 . 8 -

 78 Figure 7.8 -GUI used to test each interaction technique.

Figure 7 . 9 -

 79 Figure 7.9 -Flowchart of experimental protocol during each session.

Figure 7 . 10 -Figure 7 . 11 -

 710711 Figure 7.10 -Example of a completed Add-Task

Figure 7 . 13 -

 713 Figure 7.13 -Completion time (a) and distance error across task blocks (interaction technique block = 3 task blocks).

  (a) distance error by tasks (b) completion time by tasks

Figure 7 . 14 -

 714 Figure 7.14 -Comparison of accuracy and completion time by tasks.

Figure 7 . 15 -

 715 Figure 7.15 -Comparison of average distance error by interaction techniques.

Figure 7 . 16 -

 716 Figure 7.16 -Comparison of average completion time by interaction techniques.

Figure 7 . 17 -

 717 Figure 7.17 -Comparison of average percentage of movement time and stationary time per trial across Physical Sliders, Zonal-Trackpad, and Phantom Desktop.

52 .

 52 30% of the time stationary and 47.70% of the time moving when using the Zonal Trackpad, and 7.15% of the time stationary and 92.85% of the time moving on average per trial when using the Phantom. As we can see from this analysis, participants spent a much larger amount of time moving the 3D cursor with the Phantom compared to the Zonal Trackpad and Physical Slider, and participants spend more time moving the 3D cursor with the Zonal Trackpad than Physical Slider.

Figure 7 . 18 -

 718 Figure 7.18 -Average camera movements per trial across Physical Sliders, Zonal-Trackpad, and Phantom Desktop for each task.

Figure 7 .

 7 Figure 7.19 -NDC analysis of average time spent interacting with a combination of dimensions with threshold of 0 m (red), 0.0025 m (green), 0.005 m (blue), and 0.01 m (purple).

Figure 7 . 20 -

 720 Figure 7.20 -Rating of how difficult it was to learn each interaction technique.

Figure 7 . 21 -

 721 Figure 7.21 -Preferences of interaction techniques within the Add-Task (a), Insert-Task (b), and Select-Task(c).

Figure 7 . 22 -

 722 Figure 7.22 -Difficulty rating of interaction techniques within the Add-Task (a), Insert-Task (b), and Select-Task(c).

Figure 8 . 1 -

 81 Figure 8.1 -SpADS-A provides a space to describe the interactive spatial parameters of audio objects by their recording and rendering techniques.

Figure 8 . 2 -

 82 Figure 8.2 -SpADS-C provides a space to describe control methods of audio objects by their characteristics of the visual feedback, input method, and mapping method.

Figure 8 . 3 -

 83 Figure 8.3 -Model that identifies stages in audio production and how audio objects are created and manipulated.

Figure 8 . 4 -

 84 Figure 8.4 -Classification of three major tasks in 3D audio production.

  ure 8.5).

Figure 8 . 5 -

 85 Figure 8.5 -Flowchart on how to use virtual speakers to decode any audio object onto any auditory display.

Figure 8 . 6 -

 86 Figure 8.6 -Model of production tasks for multi-modal immersive content.

appendix a: online survey 1 / 20 Spatial

 120 Audio Survey and Questionnaire Informed Consent Form for Participants Please read the following information carefully. You can also request a copy for future reference. Study Title: Spatial Audio Content Creation and Manipulation Involved Researchers: "Removed for Reviewing Process"

Free-field

  assumption (ideal anechoic world) Acoustic of the physical reproduction space Acoustics of your studio Simple virtual geometry (rectangular room) Virtual represention of the reproduction space Complex virtual architecture Other:

  Figure B.1 -3D model of the cylindrical speaker configuration used for rendering the audio trajectories (a), and photo of a participant within the real speaker configuration and experimental room (b). Not all speakers shown in the photo of the participant were used for the rendering space in this study.

1 .

 1 FL-FR -Front Left to Front Right that curve up and down.

  (a) 2D view of FL-FR trajectories (b) 3D view of FL-FR trajectories

  Figure B.2 for a screenshot of the GUI).

Figure B. 2 -

 2 Figure B.2 -Screenshot of GUI used in the experiment

  Figure B.4 for diagram of mapping).

Figure B. 3 -

 3 Figure B.3 -Flowchart of experimental protocol.

Figure B. 4 -

 4 Figure B.4 -Slider mapping method

b. 3 . 2

 32 Comparison of Rendering Techniques Within these tasks, we observed little difference between the HOA and VBAP rendering techniques. Figure B.6 shows the boxplot analysis of all the paths aggregated under the HOA and VBAP rendering condition for both tasks. Responses for VBAP (Figures B.6(b) and B.6(d)) were slightly more consistent than HOA (Figures B.6

  (a) and B.6(c)) for certain curvature conditions, but average responses (mid-line within box) were similar to between rendering techniques. However, differences between tasks can still be seen within this comparison.

3 . 3

 33 Figure B.5 -Boxplot analysis of Aural-to-Visual task (a) and Visual-to-Aural task (b) across each curvature amount.

Figure B. 6 -

 6 Figure B.6 -Boxplot analysis of HOA across Aural-to-Visual task (a) and Visualto-Aural task (c), and VBAP across Aural-to-Visual task (b) and Visual-to-Aural task (d).

  Figure B.7 -Analysis of FL-FR & FR-FL paths across renderers and tasks.

Figure B. 8 -

 8 Figure B.8 -Analysis of FL-BL & FR-BR paths across renderers and tasks.

  Figure B.9 -Analysis of FO-BO path across renderers and tasks.

C

  A P P E N D I X C : A N A LY S I S M E T H O D S

d. 2 . 1

 21 Partie I -Audio 3D: outils, méthodes et pratiques Dans cette partie de la thèse, nous discutons de l'état des outils de production audio 3D, des méthodes, des pratiques, et de la perception spatiale pour mieux comprendre comment les professionnels produisent du contenu audio 3D. Dans le chapitre 2, nous présentons un état des lieux des techniques de rendu audio 3D, des interfaces utilisateur audio 3D dans les productions grand public, des études et des projets d'interfaces utilisateur audio 3D et leur relation à la recherche dans les interfaces 3D domaines. Dans le chapitre 3, nous présentons les résultats d'entrevues menées et un sondage en ligne auprès des praticiens actuels de l'audio 3D qui ont identifié les limites des outils actuels et leurs méthodologies pour produire du contenu audio 3D. D'après les commentaires des praticiens rapportés au chapitre 3, nous avons identifié les problèmes liés au retour visuel de nombreuses interfaces utilisateur actuelles avec l'expérience d'écoute. Pour mieux comprendre comment améliorer la conception du retour visuel, nous donnons un aperçu des travaux connexes sur la perception spatiale dans les modalités auditives et visuelles et la compréhension spatiale de haut niveau dans le chapitre 4.

Figure D. 1 -

 1 Figure D.1 -SpADS-A avec la façon dont différents objets audio peuvent être classés en fonction de leur méthode d'enregistrement et si elle est rendue sur Field-Based ou Position-Based avec des paramètres spatiaux descriptifs de translation, directivité et d'orientation.

Figure D. 2 -

 2 Figure D.2 -SpADS-C v2 qui décrivent comment les périphériques d'entrée peuvent être mappés pour contrôler différentes propriétés spatiales et les types de rétroaction visuelle en définissant des méthodes de contrôle comme méthode d'entrée, méthode de mappage et leurs commentaires visuels.

Figure D. 3 -

 3 Figure D.3 -Les nouvelles descriptions d'entrée et de mappage de SpADS-C sans la description de retour visuel.

  Figure D.4 -Comparaison de la précision et du temps de réalisation par tâches.

Table 4 . 1

 41 Auditory cues that have been tested to have an effect on the ability to localize sound sources. Binaural cues provide ITD and ILD/IID that help localize on the azimuth plane.

	Monaural Pinnae

Table 4 .

 4 1 -Auditory cues that have been tested to have an effect on the ability to localize sound sources. Binaural cues provide ITD and ILD/IID that help localize on the azimuth plane. Monaural Pinnae cues provide spectral cues that affect localization on the elevation plane. Motion cues from head or source movement provide changes in the ITD and ILD/IID to help localize a sound source, and loudness and Directto-Reverberant ratio provides cues on a sound source's distance.

Table 4 .

 4 2 -Visual cues that have been tested to affect the perception of distance and depth in visual images. Monocular (or pictorial) cues provide depth information from the 2D image created by the vision. Binocular cues provide perception of depth based on the differences in information captured by each eye, and Motion cues provide sense of depth and shape with based on movement and orientation of objects in the image.

		Binocular	Motion
	Perspective, Familiarity	Convergence	Motion Parallax
	Height & Relative Size	Disparity	Dynamic Occlusion
	Occlusion, Brightness		Kinetic Depth Effect
	Shadowing, Shading
	Contour, Texture Gradient
	Focus Distance

you agree with the abovestated conditions and are willing to participate in the study, please check the 3 boxes below. By checking the following boxes, you confirm that you meet the following conditions: *

  Check all that apply.

	33. What control parameters of directivity do you use?
	Check all that apply.
	Spread
	Width
	Frequency dependent parameters
	I don't consider/change source directivity
	Other:
	34. Cite
	Thank you for your time!
	* Required
	1.
	2. If You are at least 18 years old.
	You have read the above consent form, understood it and agree to it.
	You want to participate in the abovementioned study.
	10/20

some of the reasons why you would vary the source directivity over time. 35. Cite some of the reasons why you would define the source directivity and leave it fixed.

http://www.ams-neve.com/products/post-production/dfc-gemini

http://ams-neve.com/dfc3d/

https://www.steinberg.net/en/products/vst/thegrand3/new _ features.html

https://palettegear.com/

http://www.3dconnexion.com/products/spacemouse/spacemousewireless.html

http://www2.sonicemotion.com/professional/

http://www.longcat.fr/

(a) Phantom Controller Configuration (b) SpADS-C: Phantom

https://developer.apple.com/reference/scenekit

https://vr.google.com/daydream/

2/20

Profession and Experience

These are basic questions to gather information on your experience and the type of projects you do with spatialization.

Gender

Mark only one oval.

Male Female

Prefer not to answer

Hardware and Software Tools

These are basic questions on the equipment you use for your projects. [START_REF] Bier | Skitters and Jacks: Interactive 3D Positioning Tools[END_REF]. What listening format or scenarios are your spatialized projects typically for?

Check all that apply.

Stereo

Binaural

Surround Sound (5.1, 7.1, etc.)

3D Multichannel Reproduction Loudspeaker Arrays

Other:

Composing/Planning & Editing Techniques

Please answer these questions related to how you create spatial ideas.

Positioning of localized sounds

These questions are related to the placement of sounds in 2D or 3D at a specific location relative to a listener.

Orientation

These questions are related to the direction at which a directional source (i.e. not omnidirectional sound source) is pointed.

How important is consideration/control of the orientation of a sound source in your projects?

Mark only one oval.

Not Important Very Important

For what types of sources do you control orientation?

Check all that apply.

Live instruments

Loudspeakers

Virtual sources Other:

What parameters do you use to express orientation?

Check all that apply.

Yaw, Pitch, Roll

Vector Pointing Target Orientation (orient towards a given object)

Quaternions

Other:

29. Cite some of the reasons why you would vary the orientation in time.

30.

Cite some of the reasons why you would define the orientation and leave it fixed.

Directivity

The following questions are related to the modification of the directivity pattern of sound sources (i.e. not omnidirectional sound source).

General

Listening/Monitoring

Please answer questions related to methods of listening and evaluating your spatial ideas. The following presents a study that investigated two scenarios where practitioners relate one modality to the other for the objective to identify what causes difficulties in relating visual feedback to audio feedback when producing 3D audio content. The first scenario examines when practitioners use the visual modality first to understand the spatial representations of the 3D computer graphics within user interfaces and then to relate this to their spatial perception in the aural modality when listening to the rendered 3D audio production. We classify this task as Visual-to-Aural, which can occur when monitoring or evaluating a production and during authoring of spatial parameters that require practitioners to focus on the interaction with visual representations and then listen to the rendered outcome afterwards.

Positioning and Movement of Localized Sounds

The second scenario represents when practitioners use the aural modality first to spatially characterize the production and then relate their characterization to the spatial relationships within the visual feedback. We classify this task as Aural-to-Visual, which can also occur during monitoring and evaluating purposes and when practitioners attempt to recreate the spatial perceptions of a previous listening experience through the user interface.

In both scenarios, this can result in a mismatch between the user's expectations and the actual result due to spatial visual cues being more precise than aural cues, especially when the visual feedback misrepresents the spatial perception in the listening experience. To understand in more detail what causes these discrepancies, we conducted an experiment to simulate these two cross-modal tasks. This was done using a 3D graphical representation of the path displayed on a 2D computer monitor and the playback of a moving sound source from a 3D auditory display. Results of this study indicated that the Visual-to-Aural task was typically easier for participants than the Aural-to-Visual, which was most likely due to the design of the user interface allowing participants to map the target curvature to the GUI slider values used for responding