David Thery
email: david.thery@limsi.fr

Brian Fg Katz
email: brian.katz@sorbonne-universite.fr

Anechoic audio and 3D-video content database of small ensemble performances for virtual concerts

Keywords: Anechoic, Recordings, Point-Cloud, Auralization, Virtual Reality

This paper presents the details related to the creation of a public database of anechoic audio and 3D-video recordings of several small music ensemble performances. Musical extracts range from baroque to jazz music. This work aims at extending the already available public databases of anechoic stimuli, providing the community with flexible audio-visual content for virtual acoustic simulations. For each piece of music, musicians were first close-mic recorded together to provide an audio performance reference. This recording was followed by individual instrument retake recordings, while listening to the reference recording, to achieve the best audio separation between instruments. In parallel, 3D-video content was recorded for each musician, employing a multiple Kinect 2 RGB-Depth sensors system, allowing for the generation and easy manipulation of 3D point-clouds. Details of the choice of musical pieces, recording procedure, and technical details on the system architecture including post-processing treatments to render the stimuli in immersive audio-visual environments are provided.

Introduction

Auralization is based on the convolution of anechoic recordings with room impulse responses (RIR). This technique allows one to simulate how the recorded material would be heard in the measured room, for a given source/receiver pair. These RIRs can either be measured in the actual space or simulated, by employing various numerical methods [START_REF] Vorländer | Computer simulations in room acoustics: concepts and uncertainties[END_REF]. Over the last decade, the perceptual quality, and hence the ecological validity, of such acoustical simulations has improved, as recent research has shown that simulated auralizations can be subjectively comparable to measured ones on a given set of attributes [START_REF] Postma | Perceptive and objective evaluation of calibrated room acoustic simulation auralizations[END_REF]. Nevertheless, the primary source (the anechoic material) remains one of the key factors to achieving realistic simulations. Several public sources exist for anechoic recordings which could be used for auralization purposes. Examples span almost 50 years. The BBC Orchestra conducted early anechoic recordings in 1969, however these recordings do not appear to be available anymore [START_REF] Burd | Nachhallfreie Musik fur Akustische Modelluntersuchungen[END_REF]. The Japan Audio Society released a CD in 1985 including various extracts of solo instrument music [START_REF]Various -Impact[END_REF]. The Archimedes project from Bang and Olufsen followed in 1992 [START_REF] Hansen | Making recordings for simulation tests in the Archimedes project[END_REF], providing highquality solo instruments and speech recordings. Soon after, Denon published a well-known number of recordings of orchestral music [START_REF] Anazawa | An Anechoic Digital Recording of Orchestral Music[END_REF], including classical, romantic and post-romantic pieces. However, these full orchestral recordings have a poor signal-to-noise ratio. In addition, the entire orchestra was recorded simultaneously using close-microphones on an acoustically damped stage and then down-mixed to a stereo recording, with the results not being well-suited to detailed auralizations due to the inability to separate instrument tracks. Multi-channel recordings of symphonic music were recorded by [START_REF] Vigeant | Multi-channel orchestral anechoic recordings for auralizations[END_REF] in 2005, but these recordings have not been made readily available, mainly due to copyright permission from the orchestra. A choir was recorded in an anechoic chamber in 2005, providing six choral arrangements with 80 singers [START_REF] Freiheit | Making an anechoic choral recording[END_REF]. One of the most thorough and high quality resource to date are the recordings described in [START_REF] Pätynen | Anechoic recording system for symphony orchestra[END_REF] in 2008: instruments were individually recorded providing high-quality and perfectly separated instrument recordings of classical music from various periods, including Mozart, Beethoven, Mahler, and Bruckner. Lastly and most recently, in 2016, [START_REF] D'orazio | Recordings of Italian opera orchestra and soloists in a silent room[END_REF] conducted recordings of opera orchestra and soloists, from composers including Donizetti, Verdi, and Puccini. These last recordings are not perfectly anechoic, being carried out in a dry room. Overall, few high-quality anechoic recordings are available, especially for multiple instruments. It should be noted that a majority of recently built concert halls are multi-purpose halls, welcoming both classical and jazz ensembles. Yet, few if any anechoic jazz ensemble recordings have been made publicly available. Aside from basic audio auralizations, there is growing interest and effort to couple audio renderings with associated visual models, potentially rendered in virtual reality (VR) [START_REF] Poirier-Quinot | Augmented auralization: Complimenting auralizations with immersive virtual reality technologies[END_REF][START_REF] Katz | Experience with a virtual reality auralization of Notre-Dame Cathedral[END_REF][START_REF] Pelzer | Auralization of virtual rooms in real rooms using multi-channel loudspeaker reproduction[END_REF]. It is generally acknowledged that auditory and visual modalities interact with each other, and both visual and auditory perceptions are significantly improved when the audio-visual stimuli are coherent (when the sound matches the visual). Research into the spatial coherence of audio-visual renderings in auralizations has shown that the visual distance between source and listener affect the perceived auditory distance and loudness, though this effect has shown inter-individual variations [START_REF] Postma | The influence of visual distance on the room-acoustic experience of auralizations[END_REF]. In addition to the inclusion of visual rendering, recent studies have shown that the inclusion of dynamic voice directivity in simulated auralizations enhances the perception of envelopment and apparent source width, as well as the plausibility of the simulation [START_REF] Postma | Subjective Evaluation of Dynamic Voice Directivity for Auralizations[END_REF]. In the context of these recent findings, this work presents the acquisition of synchronized audio and video recordings of small ensembles playing a variety of musical pieces, including different orchestrations, with two main goals:

• Extending the range of publicly available databases of anechoic audio recordings to cover a wider range of musical styles, hence enabling a more adapted music selection for a given space under study.

• Providing the community with audio and 3D-video recording datasets that can be easily integrated into VR environments (potentially including dynamic instrument directivity, see description in Sec. 3.4).

This paper begins with a description of the musical selections, followed by the procedure employed for the recordings. Technical details are provided in Sec. 3. Finally, potential applications and future work are discussed in Sec. 4.

MUSIC SELECTION AND MUSICIANS

2.1 Criteria of selection Several requirements were defined for the selection of the musical pieces: they needed to provide valuable acoustical interest (presence of tutti, solo passages, wide occupation of the frequency spectrum), to represent different periods and musical styles (particularly those missing from currently available anechoic recordings), and to stage different orchestrations including various instruments. Particular attention was given to movable instruments (like saxophone, clarinet), with the final aim being the inclusion of coherent directivity in the audio that follow the musicians movement, also represented in the visual rendering. A summary of the different musical excerpts is provided in Tab. 1.

Classical Baroque

One of the missing musical period in the work of [START_REF] Pätynen | Anechoic recording system for symphony orchestra[END_REF] was the Baroque era; which they decided not to record mainly because of the presence of the harpsichord, which was deemed difficult to record properly. Given the numerous composers, different geographical influences, number of orchestrations, and overall richness of this period, it was decided to acquire Baroque music material. The 3 pieces were interpreted by the same quartet, comprising 2 violins, 1 viola, and 1 cello. The first selected piece was the second movement of the 3 rd Orchestra Suite (BWV 1068 no.3) called Aria in D-minor. This 1'10" long piece is one of the most famous piece from J.S Bach, and providing various harmonies with a slow tempo (Largo/Larghetto). The second selected piece was part of the Art of the fugue, the last piece of work from J.S Bach. An extract of 1'07" from the Canon Alla Ottava in D-minor was selected. The first phrase of this Canon is written for the violin alone, followed by the cello; the different voices alternate often, at a moderate tempo (Adagietto), enabling one to focus on the different instruments and their frequency ranges successively. The third and final piece from the Baroque era was the 3 rd movement of the Summer of the 4 seasons from Vivaldi, in G-minor. This piece of 2'35", played presto, comprises a variation of dynamics, including violin solo passages, crescendos, and tutti. These pieces, being interpreted with 2 violins, 1 viola, and 1 cello, are also interesting from the viewpoint of moving instruments.

Jazz

Common jazz instruments include saxophone, trumpet, trombone, clarinet, piano, guitar, double-bass, voice, and drums. Different periods can be characterized by different orchestrations.

It was decided to record Manouche jazz, a style from the 1930's characterized by a rythmic basis played by 2 guitars and a double-bass, accompanying a melodic violin, with the absence of percussion, woodwinds, and brass. The chosen title was the well-known Minor Swing in A-minor, interpreted by a trio comprising 1 doublebass, 1 guitar, and 1 violin, covering a large part of the audible frequency range. The recorded extract lasted 1'15". Two pieces were selected to extend the range of instruments available, as well as other styles of jazz music: the first 2 min of Don't mean a thing, written in 1931 by Duke Ellington, played by a trio comprising a tenor saxophone, guitar, and double-bass. This piece was selected to represent the Swing jazz period. Finally, the first 1'30" of a Sydney Bechet song, recorded in 1952 (New Orleans period), played with a saxophone alto, guitar, and double bass was chosen for the particular timbre of the saxophone alto and the low density of notes in this piece, allowing for more perception of room effects.

Musicians

To cover the different styles chosen described above, 8 musicians from the Sorbonne University's Choir and Orchestra (COSU) were recruited and paid, including: 1 double-bass player, 1 cello player, 1 guitarist, 3 violonists, 1 altist, and 1 saxophonist. Their mean age was ≈ 20 years. All musicians had previous recording experience, though not in anechoic conditions.

RECORDINGS

Procedure

The objective of this work was to record both the audio and video of each musician, to allow maximum flexibility for future simulations. To achieve the high quality and perfect separation of instruments in the audio recordings, it was decided to individually record each instrument/musician. The recording procedure was defined as follows:

• A preliminary reference take in which all musicians were aligned in a linear row and played together was recorded, in the anechoic room. Except the saxophone, violins, and viola which were recorded with a microphone mounted on the instrument, each instrument was recorded using figure-of-8 pattern close-microphones, with the nulls pointing perpendicular to the associated instrument, to reduce the sound captured from neighboring instruments. In order to provide a natural acoustic for the musicians, rather than the very dry anechoic chamber's acoustics, the live captured audio of each instrument was processed in real time by adding a reverberation of 1 sec to simulate a small studio (usual in studio recordings) and rendered over headphones identically to all musicians. The listening level was adjusted to maximize the musicians' playing comfort, optimizing the balance between their own instrument sound, the added reverberation, and the surrounding instruments. This live reverb processing was achieved using SPAT 1 , with one instance per instrument. The processing was adjusted with the live feed virtually placed at 1 m in front of the listener, with a room size of 600 m 3 and an aperture of 90°.

• To provide a performance reference for the musicians when they were subsequently recorded in isolation, comparable to the reference piano track and conductor video in [START_REF] Pätynen | Anechoic recording system for symphony orchestra[END_REF], a downmix of the reference performance was created for each musician for which their individual contribution was excluded. In this way, it was intended that musicians would be able to perform in a more "natural" way, mimicing their prior performance when playing in ensemble.

• A second recording session was then performed in which each musician was individually recorded while listening to their individual monitor mix. The same live reverberation processing was employed for the individual recording sessions. Subsequent to this recording, each musician was allowed to listen to their individual take and to the whole. They were able to retake the session if desired.

• For each of these individual takes, the video capture from the Kinect cameras system 2 was manually started and stopped.

Audio anechoic recordings 3.2.1 Anechoic chamber

All recordings were been carried out in the anechoic chamber of the Sorbonne Université, interior working dimensions of 7.95×5.5×4.07 m, excluding the surrounding wedges of 0.85 m depth. The chamber is therefore assumed anechoic for frequencies above 80 Hz. Musicians were always located at least 1.5 m away from the tip of the anechoic wedges. The chamber has an equivalent background noise level of LAeq = 16.6 dBA. A surveillance camera installed in the chamber enabled visual monitoring of the recordings sessions from the adjacent control room.

Microphones

Several options were considered for micing the instruments. Use of purely electric instruments to avoid capturing other instruments sounds was discarded for two reasons: first, the altered timbre of such instruments as compared to acoustic ones, and second, the fact that even professional musicians are not used to playing this Figure 1. Musicians wearing headphones and playing together for the reference recording (1st session). Saxophone and violin were recorded using DPA4060 mounted on the instrument, while double-bass and guitar were recorded using Schoeps MK-8 figure-of-8 microphones.

type of instrument. Use of piezo-electric microphones was discarded due to the cost/lack of availability in the lab of high-quality microphones of this type and the additional processing required to transform the signal into natural instrument sound. After discussions with a few recording professionals 3 , it was decided to record the instruments using closely placed microphones. The exact micing configuration was adapted for each instrument, following guidelines from recording engineers [START_REF] Owinski | The recording engineer's handbook[END_REF].

For the reference recording, as the goal was to record the ensemble with minimal cross-talk between pickups, the musicians were aligned in a single row. In addition to close-mounted microphones, figure-of-8 microphones pointing towards the instruments were used, to provide rejection of sounds coming from the adjacent musicians, as described in the procedure, and depicted on Fig. 1.

For the second recording phase, solo instrument recordings, protocols depended on the instrument category: static versus moving instruments with respect to needing the microphones being fixed to the instrument or not. For static instruments, including double-bass, cello, and guitar, one fixed omni-directional microphone (DPA 4006) was placed approximately 1 m away, pointing towards the instrument. Two supplementary microphones were placed off-center at 2 m distance from the instrument, to provide additional resources for any subsequent equalization (while also providing minimal data for comparing with theoretical radiation patterns of each instrument [START_REF] Otondo | A New Method for the Radiation Representation of Musical Instruments in Auralizations[END_REF]). For moving instruments, such as violin, viola, and saxophone, closely placed miniature microphones (DPA 4060) were mounted on the instruments. These microphones were chosen for their high-quality: linear frequency response, good directional pattern 4,5 , and low noise-floor (respectively LAeq = 15 dBA and 23 dBA).

Hardware/Software details

Audio and video were recorded using two distinct and independent systems. A dedicated computer (Macbook 2.5GHz, 16 GB RAM) managed all the audio, using a custom Max6 patch. The signals from the microphones recording the instruments were acquired using a Fireface 802 7 audio interface, at 44.1 kHz/24bits. The audio computer also provided the monitor output mix to the headphones worn by the musicians, as well as a general monitor for the recording engineer.

RGB-Depth video recordings

In parallel to the individual audio recordings, each musician was also visually recorded in order to acquire 3D images for use as realistic performance avatars playing on stage in VR simulations. Visual recordings were carried using three Kinect v2 RGB-D sensors, allowing for the acquisition of 3D point-cloud representations of each the musician. Such visual data can be integrated, for example, into a Unity8 VR scene without the need to create an animated avatar. The remainder of this section describes the architecture of the system used for the 3D-video recordings as well as the RGB-D image post-processing applied to obtain the final 3D-point-cloud. Recording from multiple Kinects was achieved thanks to the LiveScan3D library9 developed in C++/C# and the OpenCV library10 . This system enabled for the production of a single fused 3D point-cloud of each musician from the data acquired from the separate cameras, viewable from different viewpoints (see Fig. 2). The LiveScan3D library is based on a client/server architecture, the clients communicating with the server via TCP/IP. In principle, it allows for the use of any number of Kinect camera clients, with each Kinect connected to a separate computer (a limitation originating from the Kinect v2 SDK that does not allow multiple Kinect's on a single PC). For this database, 3 sensors were used, resulting in a few small shadow zones behind the musician in the 3D reconstruction of the point-cloud [START_REF] Kowalski | Livescan3D: A Fast and Inexpensive 3D Data Acquisition System for Multiple Kinect v2 Sensors[END_REF].

The server manages all the clients simultaneously for recording sequences. The composite point-cloud can directly be streamed in Unity 11 or to a viewer available in the LiveScan3D library. For the creation of the visual part of the database, the sequences were saved as .ply format files 12 , a format which supports colored point-cloud data (either binary or ASCII format). These files can then, for example, be imported in Unity and played frame by frame to provide the visuals of each musician playing. A similar approach, though with only a single Kinect, has previously been used for the insertion of visual actors into an audio-visual rendering of a theatre auralization [START_REF] Postma | The influence of visual distance on the room-acoustic experience of auralizations[END_REF][START_REF] Thery | Impact of the Visual Rendering System on Subjective Auralization Assessment in VR[END_REF][START_REF] Katz | Virtual reconstructions of the Théâtre de l'Athénée for archeoacoustic study[END_REF].

Post-processing

The added value of having video recordings is that dynamic source positions and orientations can be extracted and employed to provide dynamic acoustic directivity in auralizations, following the method described in [START_REF] Postma | Subjective Evaluation of Dynamic Voice Directivity for Auralizations[END_REF], based on a 12 beams angularly equi-repartited decomposition of impulse responses. Briefly, after convolving the mono recorded instrument track with the HOA-RIR of each source beam and filtering into separate frequency bands, dynamic source directivity can be adjusted in real-time by altering the gains of the different source beams to create the desired pattern and orientation. In contrast to the method proposed by [START_REF] Vigeant | Multi-channel orchestral anechoic recordings for auralizations[END_REF], which used multi-channel anechoic recordings, this method offers a better representation of source directivity in the rendering architecture. RIR simulations need only to be calculated once, for a set of directional sources on a sphere. Current research efforts involve the development of a real-time implementation of this method in order to study musician/room interactions [START_REF] Katz | EVAA: A platform for Experimental Virtual Archeological-Acoustics to study the influence of performance space[END_REF].

Outlook

This paper presented the individual recordings (both 3D video and anechoic audio) of small musician ensembles (trios and quatuors) playing 6 extracts of musical pieces comprising Baroque and Jazz music, including the following composers: Bach, Vivaldi, Django Reinhardt, Duke Ellington, and Sydney Bechet. Close-mounted microphones were used to record each instrument individually in a methodical manner. Accompanying simultaneous RGB-depth-video recordings enabled for the creation of 3D point-clouds which can be easily incorporated into future virtual acoustic simulations combined with auralizations.

As introduced in Sec. 3.4, dynamic instrument directivity can be included in the simulations. This first perspective will allow for the conduction of subjective experiments to assess the impact of the incorporation of such dynamic instrument directivity on auralization perceptions, with a comparison to results previously obtained with voice stimuli [START_REF] Postma | Subjective Evaluation of Dynamic Voice Directivity for Auralizations[END_REF]. Other potential uses of this multimodal dataset include investigating the impact of changing the spatial arrangement of musicians and instruments to simulate different orchestrations, or providing coherent and incoherent audio/visual cues to investigate multimodal effects, in particular for the auditory perception of dynamic source directivity.

Resources

Synchronized audio, 3D-video point-cloud, and source tracking information are made freely available for research purposes. Files can be downloaded via the institutional site www.lam.jussieu.fr/Projets/AVAD-VR. html.

Figure 2 .

 2 Figure 2. Example of cello player 3D point-cloud reconstituted image, viewed from two different angles.

Table 1 .

 1 List of selected musical pieces, including title of the piece, composer, period/style, instruments used in the recorded interpretation, and the duration of the extract.

	Musical piece 1	Composer	Period/Style	Instruments	Length
	BWV 1068 no.3	J.S. Bach	Baroque (1739)	2 Violins, 1 Viola	1'10"
	Aria			1 Cello
	BWV 1080/15	J.S. Bach	Baroque (1721)	2 Violins, 1 Viola	1'07"
	Canon alla ottava			1 Cello
	RV 315	A. Vivaldi	Baroque (1723)	2 Violins, 1 Viola	2'35"
	Opus 8			1 Cello
	Minor Swing 2	D. Reinhardt Manouche Jazz (1937)	2 Guitars	1'15"
				Violin, Double-Bass
	Don't mean a thing	D. Ellington	Swing Jazz (1931)	Saxophone, Guitar	2'00"
	(If Ain't Got That Swing)			Double-Bass
	Si tu vois ma mère	S. Bechet	New Orleans (1952)	Clarinet, Guitar (Tenor) 1'30"
				Double-Bass

http://www.forumnet.ircam.fr/product/spat-en/

https://www.ens-louis-lumiere.fr/en/

https://www.dpamicrophones.com/ddicate/4006-omnidirectional-microphone

https://www.dpamicrophones.com/dscreet/4060-series-miniature-omnidirectional-microphone

https://cycling74.com/products/max-features

http://www.rme-audio.de/en/products/fireface_802.php

https://unity3d.com

https://github.com/MarekKowalski/LiveScan3D

https://opencv.org/

https://github.com/MarekKowalski/LiveScan3D-Hololens

http://paulbourke.net/dataformats/ply

ACKNOWLEDGEMENTS

The authors acknowledge the musicians for their participation in the creation of this database.


Brian Fg Katz

David Poirier-Quinot

N J Barteld

Postma

Virtual reconstructions of the Théâtre de l'Athénée for archeoacoustic study

Keywords: Archeoacoustics, Archaeological Acoustics Auralization, Virtual Reality

The French ECHO project studies the use of voice in the recent history of theater. It is a multi-disciplinary project which combines the efforts of historians, theater scientists, and acousticians. In the scope of this project an audio-visual simulation was created which combines auralizations with visualizations of former Théâtre de l'Athénée configurations issue from a series of renovations, enabling researchers to realistically perceive theater performances in foregone rooms. Simulations include the room, 2 actors on stage, and an audience. To achieve these simulation, architectural plans were studied from archives providing various details of the different theater configurations, from which the corresponding visual and room acoustic geometrical acoustics (GA) models were created. The resulting simulations allow for 360°audio-visual presentations at various positions in the theater using commercial standard hardware.

INTRODUCTION

The ECHO project studies the use of voice in the recent history of theater. This multi-disciplinary project combines the efforts of theater scientists, historians, and acousticians. In order to enable theater scientists and historians to study the use of voice in historic theater conditions an audio-visual framework which enables virtual reality visits to historic configurations of the Théâtre de l'Athénée-Louis-Jouvet 1 was created. For this purpose, first the architectural history of the theater was studied (see Sec. 2). Based on these studies three Geometrical Acoustic (GA) models were created corresponding to different epochs of this theater (see Sec. 3). These various models were used to compare the acoustic conditions across renovations. As studies have shown that visuals influence the room acoustical experience, the auralizations of the GA models were combined with visuals employing an audio-visual framework (see Sec. 4).

ARCHITECTURAL HISTORY OF THE THÉÂTRE DE L'ATHÉNÉE

On 7-Jan-1883 the Éden-Théâtre was inaugurated in Paris. The 4000-seat theater suffered from continual financial difficulties and was therefore closed in 1894 and demolished in May 1895. In 1893 a foyer of the Éden-Théâtre was converted into a much smaller theater called the Comédie-Parisienne which was inaugurated on the 31-Dec-1894. The architect was Stanislas Loison, with further modifications carried out under the architect Paul Fouquiau's guidance in 1894. This newly created four-floor theater seated approximately 600 attendants. It was closed again on 9-May-1895, after two bankruptcies. In 1896, the Athénée Comique took possession of this room which was renamed the "Comédie Parisienne" on 28-Dec-1898, and then on 25-Oct-1899 as the "Athénée" [START_REF] Lecomte | Histoire des Théâtres de Paris 1402-1904[END_REF]. When Louis Jouvet became director of the theater in 1934, the room changed its name for the last time: the Théâtre de l'Athénée-Louis-Jouvet.

Original configuration

In the original configuration (see Fig. 1(a)) there was an orchestra pit meant for ≈ 25 musicians. Walls were covered with light velour and seating was lightly upholstered. During the first years of the Théâtre de l'Athénée, there was no amplification. With the appointment of Louis Jouvet in 1937 the theater underwent its first physical alterations [START_REF] Knapp | Louis Jouvet Man of the Theatre[END_REF]. Under his guidance, the stage was rebuilt, the old electrical fixtures rewired, and the electrical power increased to support stage lighting. Furthermore, Louis Jouvet decided to install more luxurious chairs. Finally the Théâtre de l'Athénée was brought completely up to date with a loudspeaker intercommunication system. A second larger renovation was carried out at a yet undetermined time. Pictures and plans show that it was carried out between 1962 and 1985 (see Fig. 1(b)). The stage edge was straightened and the orchestra pit was removed. On the first floor the loges were replaced by folding seats and the orchestra pit by regular seating. The loges at the second and third floor were moved back, accommodating regular seating. The fourth floor back wall seating was replaced by a glass-framed technical room for sound and light direction. The Théâtre de l'Athénée underwent its third renovation between 1995-1998 (see Fig. 1(c)). An orchestra pit was re-installed in front of the stage: larger than its predecessor, it could now be covered by extra rows of seats when no orchestra was needed. Furthermore, the loges on the first floor were reinstalled and lightly upholstered chairs were brought back. The other floors of the theater remained as they were.

CREATION AND CALIBRATION OF THE GA MODELS

To create GA models for the various architectural and acoustic configurations of the theater, a base model was created and calibrated from room-acoustic measurements performed at the theater in its present state. For details of this procedure, see [START_REF] Postma | Creation and calibration method of virtual acoustic models for historic auralizations[END_REF]. GA models of the former configurations were derived from this model, adapting its geometry and materials to match the observations reported in Sec. 2

Creation

The geometry of the Théâtre de l'Athénée's current configuration was determined from architectural plans (see Fig. For the original configuration, an orchestra pit was modeled in front of the stage, the loges on the second and third floor were modeled according to Fig. 1(a). For the pre-1995-1998 configurations the loges along the side and back wall of the first floor were replaced by seating and the stage was straightened. The modular design of the GA model ensured coherence of the unmodified elements between configuration (see Figs. 2(a), 2(b), and 2(c)). Surface materials were determined from visual inspection. The Théâtre de l'Athénée has a wooden stage floor and concrete side and back stage walls. The audience area has a wooden floor, light velour walls, plastered balcony fronts, and plastered ceiling with a large chandelier. Upholstered chairs are positioned throughout the room. Few changes were observed in material usage throughout the history of the Théâtre de l'Athénée. The pre-1995-1998 renovation contained heavier upholstered chairs, modeled as medium upholstered [START_REF] Beranek | How they sound: Concert and Opera halls[END_REF]. The frequency-dependent scattering coefficient (scatt coe f ) was estimated as a function of a given characteristic depth (char depth ) representative of the surface's depth variations or roughness. The estimation algorithm (see Eq. 1, [START_REF] Dalenbäck | CATT-A v9: User's Manual CATT-Acoustic v9[END_REF]) is available in CATT-Acoustic via the estimate function, though specific values can also be directly assigned as a function of frequency.

scatt coe f ( f ) ≤0.99 ≥0.10 = 0.5 char depth λ ( 1
)
where λ is the wavelength. This method of defining scatt coe f was selected as it provided a more intuitive and physically relevant control parameter and reduces the possibility of creating unrealistic frequency variations in scattering properties for general materials. Employing this formula results in an increasing scattering with higher frequency, as is typically found in scattering measurements.

Measurement

A room-acoustic measurement was performed in order to serve as a reference for the calibration. A summary of the calibration reference values are provided in Tab. 1. Full details of the calibration measurements are provided in [START_REF] Postma | Perceptive and objective evaluation of calibrated room acoustic simulation auralizations[END_REF].

Calibration

Details of the applied calibration procedure can be found in [START_REF] Postma | Creation and calibration method of virtual acoustic models for historic auralizations[END_REF]. The procedure consists in adapting the acoustic surface parameters to minimize the mean error on simulated acoustic parameter estimation (EDT, T20, C50, and C80) compared to those extracted from RIR measurements across source-receiver pairs. Subsequently, a perceptual validation was performed using binaural auralization, ensuring that participants could not distinguish auralizations using the measured RIRs from those using the simulated ones (ratings based on eight perceptual parameters, see [START_REF] Postma | Perceptive and objective evaluation of calibrated room acoustic simulation auralizations[END_REF]).

Occupied theater

To simulate the acoustics of the theater during actual performances, curtains were added to the back and side walls of the stage in the GA model. In addition to changing the audience absorption between conditions to take into account the change in seating design, values were also adjusted to occupied absorption seating characteristics. Resulting simulated T20 values of the current configuration model closely resembled measured reverberation times in occupied conditions [START_REF] Polack | Statistical analysis of a set of parisian concert halls and theatres[END_REF].

Comparative acoustic study

With these calibrated models it is possible to study the influence of the renovations on the acoustical conditions. Fig. 3 presents a mapping of T30 and C80 values in the 1 kHz-octave band for the 3 configurations. The quantified results show that the reverberation time has not changed over time while clarity has improved for positions at the far end of the audience area.

AUDIO VISUAL RENDERINGS

Multiple studies have shown the influence of visuals on the room acoustic experience in terms of distance [START_REF] Zahorik | Estimating sound source distance with and without vision[END_REF][START_REF] Calcagno | The role of vision in auditory distance perception[END_REF][START_REF] Postma | Influence of visual rendering on the acoustic judgments of a theater auralization[END_REF][START_REF] Thery | Impact of the visual rendering system on subjective auralization assessment in vr[END_REF], loudness [START_REF] Barron | Subjective study of british symphony concert halls[END_REF][START_REF] Postma | Influence of visual rendering on the acoustic judgments of a theater auralization[END_REF], ASW [START_REF] Larsson | Ecological acoustics and the multi-modal perception of rooms: real and unreal experiences of auditory-visual virtual environments[END_REF], and room size perception [START_REF] Larsson | Ecological acoustics and the multi-modal perception of rooms: real and unreal experiences of auditory-visual virtual environments[END_REF]. Therefore, attention is subsequently focused on an associated visual model. A framework was created to compliment the simulated auralizations with visual renderings of the theater, its actors, and an audience. A 3ds Max2 visual model was provided by the Théâtre de l'Athénée. This model was imported into Blender and adjusted to match the various configurations of the theater discussed in Sec. 2, as illustrated in Fig. 4. While not accurate to every detail, the model does reflect the general visual nature of the theater. The resulting models served as scenery for a Blender Game Engine scene implementation, relying on the OSC protocol for real-time update of both listener position and orientation in a Max/MSP3 client handling the 2 nd order Ambisonics auralizations. This general augmented auralization framework has been presented in detail in [START_REF] Poirier-Quinot | Augmented auralization: Complimenting auralizations with immersive virtual reality technologies[END_REF]. It has been employed for various perceptual tests [START_REF] Postma | Influence of visual rendering on the acoustic judgments of a theater auralization[END_REF][START_REF] Postma | Subjective evaluation of dynamic voice directivity for auralizations[END_REF][START_REF] Thery | Impact of the visual rendering system on subjective auralization assessment in vr[END_REF], and allowed for multiple audio and visual presentations. The visuals can be rendered on either a 3 screen video-wall setup, an Oculus Rift DK2 Head Mounted Display (HMD), or a high definition virtual cave [START_REF] Thery | Impact of the visual rendering system on subjective auralization assessment in vr[END_REF] (see Fig. 5). The audio can be presented using either headphones (binaural) or a speaker array (Ambisonic).

In order to create a realistic sound source in terms of visuals and acoustics, a 5 min extract of the play "Ubu Roi", by Alfred Jarry, was performed by two actors and recorded in the Théâtre de la Reine Blanche, using two headset microphones and a Kinect 2 sensor. Voice directivity was incorporated according to [START_REF] Postma | Dynamic voice directivity in room acoustic auralizations[END_REF][START_REF] Postma | Subjective evaluation of dynamic voice directivity for auralizations[END_REF]. As the direct-to-reverberant ratio is high for close mic recordings, these were employed as approaching anechoic recordings. These close-miked voice recordings were convolved with simulated 2 nd order Ambisonic RIRs from the GA models. Voice directivity variations were controlled via gain modulation on the components of a 12-beam directivity sphere. Source rotation, applied to the gain modulations via a spherical harmonic interpolation, employed the currently speaking actors' head orientation based on tracking information extracted from the Kinect 2 video. The sum of the 12-beam×2 nd order Ambisonic convolutions, properly weighted and rotated, resulted in a single Ambisonic stream. The video stream of the Kinect 2 sensor was processed by a script based on the libfreenect2 library4 , recording both RGB and Depth videos. Both videos were combined during real-time rendering in BlenderVR [START_REF] Katz | BlenderVR: Open-source framework for interactive and immersive VR[END_REF] to produce a 512×424 pixel point-cloud of the actors. A set of fixed position 360 • videos was generated off-line from the same material to support a study on the influence of the listening position on perceived acoustics in immersive renderings. To provide a listening experience more comparable to attending a performance, rather than a rehearsal, animated simplified spectator avatars were added to the visual scene, coupled with audience noise. The audience noise was a collection of randomly played studio recordings made of typical audience activity: moving, rustling, coughing, scratching, mumbling, etc. These dry sounds were convolved with calculated RIRs for source positions distributed throughout the audience areas, further improving the verisimilitude of the overall rendering. The resulting Ambisonic streams were combined with the actors streams and the visual rendering to create 360°-HOA examples. The spectator avatars where animated using the Blender CrowdMaster5 add-on, assigning different animation sequences to each armatured mesh from a set matching the selected audience noises (scratching, moving, coughing, etc.). The final scene is illustrated in Fig. 6, the 360 • videos are available online at www.lam.jussieu.fr/Projets/CloudTheatre.html.

Conclusion

In order to enable audio-visual perceptual studies in historic theaters, a virtual reality framework was created which enables visits to former configurations of the Théâtre de l'Athénée. Both realistic room-acoustic and visual models were created and calibrated to match the considered configurations of the theater. These were combined using a virtual reality platform relying mainly on BlenderVR for the visual rendering and Max/MSP for the auralization. Subsequent studies in the field of experimental virtual archaeological acoustics will extend this framework to interactive real-time audio processing and an up to date game engine [START_REF] Katz | EVAA: A platform for experimental virtual archeological-acoustics to study the influence of performance space[END_REF].

(a) 1

 1 st , 2 nd , 3 th , and 4 th floor of the Théâtre de l'Athénée in the original configuration. (b) 1 st , 2 nd , 3 th , and 4 th floor of the Théâtre de l'Athénée in the pre-1995 configuration. (c) 1 st , 2 nd , 3 th , and 4 th floor of the Théâtre de l'Athénée in the current configuration.

Figure 1 .

 1 Figure 1. Floor plans of the Théâtre de l'Athénée at different periods.

  1(c)) and sections (minimum modeled dimension = 0.34 m). Figs. 2(c) and 2(d) compare the interior of the room to the geometrical model. The stage opening of this 4-floor theater is ≈ 7.7 m, stage depth ≈ 7.8 m, and height over the stage ≈ 13.8 m. The audience area is 'horse shoe' shaped with a maximum width of ≈ 11.0m, length ≈ 14.3 m, and height ≈ 13.6 m. Box seats are present on the side and back walls on the first, second, and third floor.

Figure 2 .

 2 Figure 2. (a), (b), and (c): Geometrical model of the Théâtre de l'Athénée. Volume ≈ 2500 m 3 , ≈ 1300 planes. (d) Photo depicting the audience area as seen from the stage (22-Jul-2014).

Figure 3 .

 3 Figure 3. Reverberation time T30 and Clarity C80 simulation results for 1 kHz-octave band. 500 000 rays used, ray truncation time 3000 ms.

Figure 4 .Figure 5 . 3 -

 453 Figure 4. Visual models of past Théâtre de l'Athénée configurations

Figure 6 .

 6 Figure 6. Snapshot of actors performing "Ubu Roi" in the virtual Théâtre de l'Athénée for different listening positions, including audience avatars.

Table 1 .

 1

		125	250	500	1000	2000	4000
	T 20	2.06	1.74	1.53	1.31	1.17	1.03
	EDT 1.91	1.81	1.56	1.30	1.20	0.98
	C50	1.71	1.04	1.08	1.63	1.93	2.57
	C80	2.32	1.73	2.40	3.45	3.28	4.99

Average T20, EDT, C50, and C80 of the measurement in the Théâtre de l'Athénée for various frequency bands (in Hz).

* brian.katz@sorbonne-universite.fr[START_REF] Barron | Subjective study of british symphony concert halls[END_REF] For brevity called Théâtre de l'Athénée here.

https://www.autodesk.com/products/3ds-max/overview

https://cycling74.com

https://github.com/OpenKinect/libfreenect2.

http://crowdmaster.org/

ACKNOWLEDGEMENTS

Our thanks go out to Francis Cousson who created the original visual model and Dominique Lemaire, technical director of the Théâtre de l'Athénée, who provided the model. This work was funded in part by the ECHO project: "The emergence of modern orality and aurality. Phonic movements in the scenic image" (ANR-13-CULT-0004, echo-projet.limsi.fr).


Yukio Iwaya
email: iwaya@ieee.org

Brian Fg Katz
email: brian.katz@sorbonne-universite.fr

Analysis of head rotation trajectories during a sound localization task

Keywords: Dynamic sound localization, Head rotation, Reaction time

Dynamic changes of the Head-Related Transfer Function renderings as a function of head movement have been shown to be an important cue in sound localization. To investigate the cognitive process of dynamic sound localization, quantification of the characteristics of head movements is needed. In this study, trajectories of head rotation in a sound localization task were measured and analyzed. Listeners were asked to orient themselves towards the direction of active sound source via localization, being one of five loudspeakers located at 30 • intervals in the horizontal plane. A 1 s pink noise burst stimulus was emitted from different speakers in random order. The range of expected head rotations (EHR) for a given stimulus were, therefore, from 30 • to 120 • . Head orientation was measured with a motion capture system (yaw, pitch, and roll). Analysis examined angular velocity, overshoot, and reaction time (RT). Results show that angular velocity increased as EHR increased. No relationship between overshoot and EHR was observed. RT was almost constant (≈260 ms) regardless of EHR. This may suggest that dynamic sound localization studies could be difficult for a short stimulus with duration less than 260 ms.

INTRODUCTION

Sound localization of human beings can be accomplished via the Head-Related Transfer Functions (HRTFs) empirically [START_REF] Blauert | Spatial Hearing[END_REF] in our daily lives. When we can move our head, the accuracy of sound localization improves [START_REF] Wallach | The role of head movements and vestibular and visual cues in sound localization[END_REF][START_REF] Thurlow | Head movements during sound localization[END_REF][START_REF] Thurlow | Effect of induced head movements on localization of direction of sounds[END_REF]. This tendency can be observed in virtual conditions with dynamic binaural renderings [START_REF] Iwaya | Effects of head movement on front-back error in sound localization[END_REF]. In consequence, it can be concluded that dynamic cues in sound localization are important in addition to HRTFs. Several previous studies have investigated the perceptional process of dynamic sound localization using the notion of 'active listening' [START_REF] Honda | Detection of Sound Image Movement during Horizontal Head Rotation[END_REF][START_REF] Honda | Effects of Head Movement and Proprioceptive Feedback in Training of Sound Localization[END_REF]. However, the details of these dynamic cues has not been sufficiently detailed. In this study, therefore, the cognitive process of dynamic sound localization was quantitatively investigated with regards to head rotation specifically. In previous studies [START_REF] Iwaya | Effects of head movement on front-back error in sound localization[END_REF], front-back confusions in sound localization were analyzed. We observed that front-back confusion rates decreased when the duration of test stimulus was long. This result indicates that head rotation in dynamic sound localization tasks needs sufficient time to be beneficial. In other words, we must be aware of the existence of a sound and roughly localize the corresponding direction of the sound before we rotate our head. In this paper, the time trajectories of head rotation in a sound localization task were measured and analyzed.

METHOD

Six university students (in their 20's with normal hearing) participated in the study. Five loudspeakers (C50A, Beringer) were arranged on the horizontal plane at a radius of 2 m at -60 • intervals to 60 • , where 0 • indicated the direction in-front of a listener. A listener sat on a chair at the center of the circle. The height of listener's ear entrance and center of loudspeakers were 1.2 m. A 1 s pink noise at 70 dBA was emitted randomly through one of the five loudspeakers, via a USB audio interface (UR824, Steinberg) controlled by a PC1 (Microsoft Windows 10, Core i7-4790, 16GB Memory). Five markers were placed on a headband attached to the listener enabling monitoring of and head movement, captured via a motion capture system (V120 Trio, OptiTrack) on another PC2 (Microsoft Windows 10, Core i7-4790, 16GB Memory). The electric signal of the stimulus from a loudspeaker was fed back to a USB audio interface (UR824, Steinberg) attached to PC2, while the stimulus and head movement data were simultaneously recorded (Motive, OptiTrack). The update rate of the motion capture system was 1/120 s. The listener was asked to localize sound direction by facing (nose pointing protocol) to the direction of the source. In each trial, three stimuli were emitted continuously at 1 s intervals. For each listener, 35 trials were performed, with a total of 105 localizations.

RESULTS AND ANALYSIS

3.1 Abstract of head movement An exmaple of a typical head movement data log is shown in Fig. 1. Head rotation (yaw, pitch, and roll) and the recorded sound envelope are plotted in Fig. 1. We observed small overshoot in roll and yaw angle. Furthermore, start time is delayed from sound emitted time. These tendencies were observed for all participants. Therefore, we analyzed, overshoot, reaction time (RT), and angular velocity of roll.

Reaction Time

We defined reaction time as followings: First, we calculated the amount of angle change of roll angle between five frames, as shown in Fig. 2 (one frame is 1/120 s). Small changes were eliminated as data noise using threshold processing. Subsequently, a tangent line was drawn at the time of 2 • , when we decided the participant started movement. Finally, the moment of the 0 • crossing of the tangent was calculated as the starting time for head movement. The difference between the calculated start time and the sound envelope rise time was defined as the reaction time. The five loudspeakers were arranged at 30 • intervals. The range of expected head rotations for a given stimulus was, therefore, from -120 • to 120 • . Table 1 shows the average RT for expected amount of head rotation. Positive and negative angular signs indicate right and left rotations, respectively. From Table 1, it can be observed that RT is almost constant, ≈0.26 s. Table 2 shows the results of RT averaged across trials for each participant. Subject B was the fastest (0.18 s) with Subject C the slowest (0.30 s). As a result, it can be assumed that there may be individual difference among individuals. We calculated the time to the maximum value of each movement in Fig. 2 and the slope at the time was used to determine the maximum angular velocity. 3, the maximum angular velocities were larger than that of expected amount of head rotation in 1 s. It was observed that as the total amount of head movement increased, the maximum angular velocity tended to increase.

Overshoot

Participants did not always face the emitted speaker correctly. To quantify this, the difference between the angle of the peak in one localization and the angle at the start time of the next localization was calculated as the overshoot.

Average overshoot for each expected amount of head rotation is shown in Table 4. No systematic change in overshoot was observed as a function of the target position.

SUMMARY

In this paper, the time trajectories of head rotation in a sound localization task were measured and analyzed. Reaction time, overshoot, and maximum angular velocity of the head rotation were investigated. From the results, reaction time was more than 0.2 s, irrespective of the expected amount of rotation. Results show that angular velocity increased as the angular distance to the target increased. No relationship was observed between overshoot and rotation. Reaction time was almost constant (about 260 ms) regardless of expected rotation. This may suggest that dynamic sound localization studies could be difficult for a short stimulus. However, further investigations are needed because the duration 260 ms in this paper might be a task dependent result.

Figure 1 .

 1 Figure 1. Example of head movement (yaw, pitch, roll, and sound envelope) data log. Target angles were 30 • , 60 • , and -60 • .

Figure 2 .

 2 Figure 2. Example of amount of change of roll angle in 5 frames.

Table 1 .

 1 Average RT to amount of head rotation.

	Expected amount of head rotation ( • ) RT (s)
	30	0.31
	-30	0.27
	60	0.27
	-60	0.27
	90	0.24
	-90	0.24
	120	0.22
	-120	0.23
	Average	0.26
	3.3 Maximum Angular velocity.

Table 2 .

 2 Average RT of each participant.

	Participant RT (s)
	A	0.29
	B	0.18
	C	0.30
	D	0.29
	E	0.24
	F	0.22
	Average	0.26

Table 3 .

 3 Maximum angular velocity to expected head rotationExpected amount of head rotation ( • ) max. angular velocity of rotation ( • ).

	30	139.8
	-30	143.6
	60	200.1
	-60	189.6
	90	251.8
	-90	239.5
	120	300.6
	-120	288.1

Table 4 .

 4 Maximum angular velocity to expected head rotation.

	Expected amount of head rotation ( • ) average overshoot ( • ).
	30	2.4
	-30	0.4
	60	2.4
	-60	1.8
	90	2.6
	-90	2.4
	120	2.6
	-120	1.7

ACKNOWLEDGEMENTS

Authors would like to thank Mr. Shoji and Mr. Otake for their contribution in conduction experiments. This study was partially supported by JSPS Kakenhi (19K12286).