Lei Yu

Jean Pierre Barbot
email: barbot@ensea.fr

Gang Zheng
email: gang.zheng@inria.fr

Hong Sun
email: hongsun@whu.edu.cn.

J- P Barbot

Compressive Sensing with Chaotic Sequence

Keywords: Compressive Sensing, Chaos, Logistic Map

Compressive sensing is a new methodology to capture signals at sub-Nyquist rate. To guarantee exact recovery from compressed measurements, one should choose specific matrix, which satisfies the Restricted Isometry Property (RIP), to implement the sensing procedure. In this letter, we propose to construct the sensing matrix with chaotic sequence following a trivial method and prove that with overwhelming probability, the RIP of this kind of matrix is guaranteed. Meanwhile, its experimental comparisons with Gaussian random matrix, Bernoulli random matrix and sparse matrix are carried out and show that the performances among these sensing matrix are almost equal.

I. INTRODUCTION

Over the recent years, a new sampling theory, called Compressive Sensing [START_REF] Candès | Decoding by linear programming[END_REF], [START_REF] Candès | Near-optimal signal recovery from random projections: Universal encoding strategies? Information Theory[END_REF], [START_REF] Candès | An introduction to compressive sampling[END_REF] (CS for short), has attracted lots of researchers. The central goal of CS is to capture attributes of a signal using very few measurements: for any Ndimensional signal v (w.l.g. v is s-sparse vector), the measurement y ∈ R M is captured through Φv, where s < M < N and Φ ∈ R M ×N is a well chosen matrix satisfying the Restricted Isometry Property (RIP) [START_REF] Candès | The restricted isometry property and its implications for compressed sensing[END_REF].

Definition 1.1: Matrix Φ ∈ R m×n satisfies the Restricted Isometry Property of order s if there exists a constant δ ∈]0, 1[ such that

(1 -δ)∥v∥ 2 2 ≤ ∥Φv∥ 2 2 ≤ (1 + δ)∥v∥ 2 2 (1) for all s-sparse vectors v.

In CS framework, finding a proper sensing matrix Φ satisfying RIP is one of the central problems. Candès and Tao have proposed that matrix with elements drawn by Gaussian distribution or Bernoulli distribution satisfies RIP with overwhelming probability, providing that sparsity s ≤ O(M/ log N ) [START_REF] Candès | Near-optimal signal recovery from random projections: Universal encoding strategies? Information Theory[END_REF]. And the randomly selected Fourier basis also retains RIP with overwhelming probability with sparsity s ≤ O(M/(log N ) 6 ) [START_REF] Candès | Near-optimal signal recovery from random projections: Universal encoding strategies? Information Theory[END_REF]. On the other hand, many researchers have employed some other techniques to construct deterministic sensing matrix: one group satisfying Statistical Isometry Property (StRIP) [START_REF] Calderbank | Construction of a large class of deterministic sensing matrices that satisfy a statistical isometry property[END_REF], such as Chirp Sensing Codes, second order Reed-Muller code, BCH code by R. Calderbank et. al [START_REF] Applebaum | Chirp sensing codes: Deterministic compressed sensing measurements for fast recovery[END_REF], [START_REF] Howard | A fast reconstruction algorithm for deterministic compressive sensing using second order reed-muller codes[END_REF], [START_REF] Calderbank | Construction of a large class of deterministic sensing matrices that satisfy a statistical isometry property[END_REF]; one group satisfying RIP-1 [START_REF] Berinde | Combining geometry and combinatorics: A unified approach to sparse signal recovery[END_REF], such as sparse random matrix by P. Indyk et. al [START_REF] Berinde | Sparse recovery using sparse random matrices[END_REF] and LDPC by D. Baron et. al [START_REF] Baron | Bayesian compressive sensing via belief propagation[END_REF]; one group satisfying deterministic RIP, such as finite fields by R. A. Devore [START_REF] Devore | Deterministic constructions of compressed sensing matrices[END_REF]; etc.

In this paper, we employ chaotic sequence to construct the sensing matrix, called chaotic matrix. Comparing to other techniques, chaotic system generates the "pseudo-random" matrix in deterministic approach and hence verifies the RIP similar to Gaussian or Bernoulli matrix. Moreover, it is easy to be implemented in physical electric circuit and only one initial state is necessary to be memorized. Based on the statistical property of chaotic sequence, it is shown that chaotic matrix satisfies RIP with overwhelming probability, providing that s ≤ O(M/ log (N/s)).

The main contribution of this paper is to make a connection between chaotic sequence and CS. It is shown by the experiments that the performance of chaotic matrix is somewhat equal to the famous Gaussian random matrix and sparse random matrix. The paper is organized as below. In section II, one chaotic system is recalled and its statistical property is presented. Section III shows the construction of chaotic matrix and proves its RIP. In section IV, experiments are carried out to simulate the performance of chaotic matrix. At the end, the conclusion is given.

II. CHAOTIC SEQUENCE AND ITS STATISTICAL PROPERTY

Let us consider the following quadratic recurrence equation

z n+1 = rz n (1 -z n ) ( 2
)
where r is a positive constant sometimes known as the "biotic potential" giving the so-called Logistic map. For the special case r = 4, the solution for system (2) can be written as below [START_REF]logistic map[END_REF]:

z n = 1 2 [1 -cos(2πθ2 n )] (3)
where θ ∈ [0, π] satisfying z 0 = 1 2 [1 -cos(2πθ)] with z 0 the initial condition of (2). It is well known that chaotic system (2) can produce very complex sequences. Even more, it is often used as the random number generator in practice since (2) takes a very simple dynamics [START_REF]logistic map[END_REF]. In this section, we will analyze its statistical properties, the distribution, the correlations and the sampling distance which guarantees the statistical independence. Denote

x n = cos(2πθ2 n ) (4)
obviously, z n takes the similar statistical property with x n since the linear transformation, for instance the fact that x n and x m are statistically independent, would result in z n and z m are statistically independent.

A. Distribution

Function (4) possesses the following features: zero mean, values bounded within the interval -1 ≤ x n ≤ 1, invariant density given by ρ(x) = 1 π (1 -x 2 ) -1/2 and x 0 = cos(2πθ).

B. Correlations

It can be checked that the m-th moment of

x n satisfies E(x m n ) = 0 if m is odd and E(x m n ) = 2 -m ( m m 2 ) ( 5
)
if m is even.

C. Statistical independence

In [START_REF] Wang | Statistical independence in nonlinear maps coupled to noninvertible transformations[END_REF], it has been proved that sequence generated by ( 4) is not independent. However, we can measure its independence through the high order correlations, which is determined by the sampling distance. We have the following lemma.

Lemma 2.1: Denote X = {x n , x n+1 , ..., x n+k , ...} the sequence generated by (4) with initial state x 0 = cos(2πθ), and integer d the sampling distance, then for any positive integer

m 0 , m 1 < 2 d , it has E(x m0 n x m1 n+d ) = E(x m0 n )E(x m1 n+d ). (6)
Proof: If there exists at least one odd number in m 0 , m 1 , the right side of ( 6) is equal to 0. For the left side, we have

E(x m0 n x m1 n+d ) = ∫ 1 -1 ρ(x 0 )x m0 n x m1 n+d dx 0 = ∫ 1 0 cos m0 (2πθ2 n ) cos m1 (2πθ2 n+d )dθ = 1 2 (m0+m1) ∑ σ δ ( 2 n m0 ∑ i=1 σ ni + 2 n+d m1 ∑ i=1 σ (n+d) i )
where the last equation uses the fact that cos θ = (e iθ +e -iθ )/2 and

∫ 1 0 e i2πθk dθ = δ(k), with δ(k) = 0 if k ̸ = 0 otherwise equals to 1.
∑ σ is the summation over all possible configurations, where σ ni = ±1 and σ (n+d) i = ±1. All possible cases for m 0 and m 1 are analysis as below:

1)

m 1 is odd: | ∑ m0 i=1 σ ni | ≤ m 0 and | ∑ m1 i=1 σ (n+d) i | ≥ 1, hence 2 n ∑ m0 i=1 σ ni + 2 n+d ∑ m1 i=1 σ (n+d) i ̸ = 0; 2) m 1 is even: it is possible that ∑ m1 i=1 σ (n+d) i = 0,
while since m 0 is odd (the assumption at the beginning of this part of proof),

∑ m0 i=1 σ ni ̸ = 0, hence 2 n ∑ m0 i=1 σ ni + 2 n+d ∑ m1 i=1 σ (n+d) i ̸ = 0.
Then we can conclude that the left side of ( 6) is also equal to 0.

If both m 0 and m 1 are even numbers, after a trivial combinatorial analysis, we get

E(x m0 n x m1 n+d ) = 2 -(m0+m1) ( m 0 m 0 /2 ) ( m 1 m 1 /2 )
Compare it with equation ( 5), we have ( 6). Remark 2.2: Lemma 2.1 implies that x n and x n+d are statistically independent when d → ∞, and this result corresponds to that given in [START_REF] Vlad | Computational measurements of the transient time and of the sampling distance that enables statistical independence in the logistic map[END_REF]. Approximately, if the sampling distance is chosen large enough, for instance d = 15, E(x m0 n x m1 n+d ) = E(x m0 n )E(x m1 n+d ) for all m 0 , m 1 < 32768, hence x n and x n+d can be considered approximately independent, as illustrated in Fig. 1.

III. CHAOTIC SENSING MATRIX

Let Z(d, k, z 0 ) = {z n , z n+d , ..., z n+kd } be the chaotic sequence sampled from the output sequence produced by Logistic map (2) with sampling distance d and initial condition z 0 , and let x k ∈ X(d, k, x 0 ) denote the regularization of Z(d, k, z 0 ) as below

x k = 1 -2z n+kd (7)
where X(d, k, x 0 ) just corresponds to equation ( 4) and hence fulfils the statistical properties discussed in the previous section.

To construct the sensing matrix Φ ∈ R M ×N , generate sampled Logistic sequence X(d, k, x 0 ) with length k = M × N , then create a matrix Φ column by column with this sequence, written as

Φ = √ 2 M      x 0 . . . x M (N -1) x 1 . . . x M (N -1)+1 . . . . . . . . . x M -1 . . . x M N -1      (8)
where the scaler

√ 2
M is for normalization. By chosen sampling distance d = 15, then elements of sequence X(d, k, x 0 ) are approximately independent and satisfy identical distribution ρ(x), i.e. a.i.i.d, and hence elements of matrix Φ are a.i.i.d. Theorem 3.1: Chaotic matrix Φ ∈ R M ×N constructed following (8) satisfies RIP for constant δ > 0 with overwhelming probability, providing that s ≤ O(M/ log(N/s)).

Remark 3.2: Inherently, this matrix Φ is sub-gaussian with a.i.i.d elements. In [START_REF] Shahar Mendelson | Uniform uncertainty principle for bernoulli and subgaussian ensembles[END_REF], A. Pajor et. al have proved that all sub-gaussian matrix verify the RIP from geometrical point of view. In what follows, a brief proof following R. Baraniuk's idea [START_REF] Baraniuk | A simple proof of the restricted isometry property for random matrices[END_REF] connecting Johnson-Lindenstrauss property [START_REF] Johnson | Extensions of Lipschitz maps into a Hilbert space[END_REF], [START_REF] Achlioptas | Database-friendly random projections[END_REF] and RIP, is presented. Moreover, we can see what Lemma 2.1 implies for RIP.

Before giving the proof, let us recall a lemma stated in [START_REF] Achlioptas | Database-friendly random projections[END_REF].

Lemma 3.3: For h ∈]0, 1/2[, E[exp (hQ 2 )] 1 √ 1 -2h , E[Q 4 ] 3
where Q = ⟨x, u⟩ with x being any row vector of Φ and u being any unit vector. Remark 3.4: In Lemma 3.3, represents approximately less, which goes to be strictly ≤ when sampling distance d → ∞.

Proof for Theorem 3.1: The proof contains two parts: first prove the J-L property for any sub-matrix of Φ, then conclude the RIP using permutation theory. 1) J-L property: Denote Φ T the arbitrary column sub matrix of Φ, with index set |T | = s. For any unit vector u ∈ R s , from Chernoff's inequality, given some positive value h, it has

Pr [ ∥Φ T u∥ 2 ≥ 1 + δ ] ≤ exp (-hM (1 + δ)) E [ exp ( hM ∥Φ T u∥ 2 )] ≈ exp (-hM (1 + δ)) ( E [ exp ( hQ 2 )]) M exp (-hM (1 + δ)) ( 1 √ 1 -2h ) M exp ( - M 2 ( δ 2 /2 -δ 3 /3 ) ) = exp(-c 1 (δ)M )
where the last inequality is obtained by Taylor expansion and setting h = 1 2 δ 1+δ , which is the extremum point, and

c 1 (δ) = δ 2 /4 -δ 3 /6.
Similarly, we can calculate the lower bound of its probability as follows

Pr [ ∥Φ T u∥ 2 ≤ 1 -δ ] ≤ exp (hM (1 + δ)) E [ exp ( -hM ∥Φ T u∥ 2 )] ≈ exp (hM (1 + δ)) ( E [ exp ( -hQ 2 )]) M exp(hM (1 -δ)) ( 1 -h + 3 2 h 2 ) M = exp(-c 2 (δ)M )
where the last inequality is obtained by Taylor expansion and

setting h = h opt = -2-δ+ √ 4+8δ-5δ 2 3(1-δ)
, which is the extremum point, and

c 2 (δ) = h opt (1 -δ)(1 -h opt + 3h 2 opt /2). Choose c(δ) = min{c 1 (δ), c 2 (δ)}, then one finally gets Pr [ ∥Φ T u∥ 2 -1 ≥ δ ] ≤ 2 exp(-c(δ)M ) (9)
2) RIP: For any s-sparse vector v, denote T the set of locations where elements are nonzero, then |T | = κ ≤ s ≪ N . The column sub matrix Φ T defined in previous part can be set up and satisfies [START_REF] Candès | Decoding by linear programming[END_REF]. Let us denote E κ one complementary event of condition in (1), i.e.

E κ = { ∥Φ T u∥ 2 -1 ≥ δ }
, and denote E the union of all possible complementary events, i.e. E = ∪ s κ=1 E κ . Then one obtains

Pr [E] = ∪ i Pr [E Ti ] 2 exp (-c(δ)M ) s ∑ κ=1 ( N κ ) ≤ 2s ( n s ) exp (-c(δ)M ) ≤ 2s(eN/s) s exp (-c(δ)M ) = exp (log 2 -c (δ) M + s (log (N/s) + 1) + log s)
where, for a fixed constant c 3 > 0, whenever s ≤ c 3 M/ log(N/s), the bound will only have the exponent with the exponential

≤ -c 4 M provided that c 4 ≤ c(δ) -c 3 [1 + (1+(log s)/s)/ log N/s].
Hence we can choose c 3 sufficiently small to ensure that c 4 > 0.

Consequently, the probability for satisfying RIP is at least

1 -Pr [E] 1 -2e -c4M .

IV. EXPERIMENTS

As presented in section III, we choose sampling distance d = 15, then generate the chaotic matrix following [START_REF] Candès | The restricted isometry property and its implications for compressed sensing[END_REF]. The synthetic sparse signals v adopted throughout this section are with only ±1 nonzero entries. The locations and signs of the peaks are chosen randomly. The measurement vector y is computed by y = Φv. Then the reconstruction v * from y is solving by Linear Programming, which is accomplished using the SparseLab [START_REF] Donoho | Sparselab[END_REF]. The decision for failure reconstruction is ∥v * -v∥ > 0.1.

One interest is the maximum sparsity s which allows exact reconstruction of the signal. The results are given in Fig. 2 and show that the maximum sparsity s in the case of chaotic matrix is similar to that in the case of Sparse matrix [START_REF] Berinde | Sparse recovery using sparse random matrices[END_REF], Gaussian random matrix and Bernoulli random matrix. The delicate experiment for the maximum sparsity s with respect to measurements M for chaotic matrix is given in Fig. 3.

Also the probability of successful recovery (recovery rate) for fixed signal size N = 100 and fixed measurement number M = 50 is compared among these matrices, shown in Fig. 4. The result shows that chaotic matrix performs similar to the other 3 matrices.

In addition, to evaluate the influence of the initial condition of the chaotic system (2), we set the initial state respectively to z 0 ∈ {0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9}, and redo the experiment to test the recovery rate, shown in Fig. 5. The result shows that the initial state takes no influence to the recovery rate.

V. CONCLUSION

In this paper, we firstly recall the statistical property of one special chaotic system -Logistic map and prove that the generated sequence is approximately independent with sampling distance large enough (for instance d = 15). Then we prove that matrix constructed with this sampled chaotic sequence also satisfies RIP with overwhelming probability.

From the experiments, it shows that chaotic matrix has the similar performance to Sparse matrix, Gaussian random matrix and Bernoulli random matrix.

Fig. 1 .

 1 Fig. 1. Probability density ρ(xn) (a) and ρ(x n+d ) (b); (c) and (d) joint probability density P (xn, x n+d ) for sampling distance d = 5, 15.

Fig. 2 .

 2 Fig. 2. Maximum sparsity for fixed signal size N = 800 and variable number of measurements M ∈ [100, 500] (left) and for variable signal size N ∈ [300, 1000] and fixed number of measurements M = 200 (right).

Fig. 3 .

 3 Fig. 3. Probability of correct recovery for fixed signal size N = 1000.

This work was supported by "Bourses Doctorales en Alternance", PEPS-A2SDC of INSIS-CNRS, NSFC (No. 60872131) and FEDER through CPER 2007-2013 with INRIA Lille-Nord Europe.

L.


Florent Berthaut
email: berthaut@labri.fr

Myriam Desainte-Catherine
email: myriam@labri.fr

Martin Hachet
email: hachet@labri.fr

DRILE: an immersive environment for hierarchical live-looping

Keywords: Drile, immersive instrument, hierarchical live-looping, 3D interaction

We present Drile, a multiprocess immersive instrument built upon the hierarchical live-looping technique and aimed at musical performance. This technique consists in creating musical trees whose nodes are composed of sound effects applied to a musical content. In the leaves, this content is a one-shot sound, whereas in higherlevel nodes this content is composed of live-recorded sequences of parameters of the children nodes. Drile allows musicians to interact efficiently with these trees in an immersive environment. Nodes are represented as worms, which are 3D audiovisual objects. Worms can be manipulated using 3D interaction techniques, and several operations can be applied to the live-looping trees. The environment is composed of several virtual rooms, i.e. group of trees, corresponding to specific sounds and effects. Learning Drile is progressive since the musical control complexity varies according to the levels in live-looping trees. Thus beginners may have limited control over only root worms while still obtaining musically interesting results. Advanced users may modify the trees and manipulate each of the worms.

INTRODUCTION

In the past decade, the live-looping technique has become more and more used in musical performances, either for solo singers who create their accompaniment or for instrumental or electronic improvisations. However, it has some limitations due to the use of hardware controllers, such as the impossibility of creating and manipulating complex musical structures or the impossibility of modifying all the parameters of recorded loops. We believe that 3D immersive environments can support the evolution of this technique into a more advanced system for musical performances. Indeed these environments provide interesting possibilities for 3D interaction with multiprocess instruments, i.e instruments that are composed of several sound synthesis processes. They also enable new way of building, visualizing and navigating complex musical structures as 3D shapes. Immersion, e.g using large stereoscopic displays, may be as valuable for the audience as it is for musicians. For example, comprehension of what musicians are doing and involvment in the musical performance may be improved by the use of stereoscopic vision and large screens. Finally, these immersive environments are often used in other fields for collaborative tasks, and thus they may be appropriate for collaborative musical performance.

In section 3, we describe an evolution of the live-looping technique, called hierarchical live-looping technique. Then, in section 4, we present Drile, our immersive musical instrument which relies on this technique. As it can be seen on figure 1, the musician wears 6DOF tracked stereoscopic glasses, and he interacts with the instrument, displayed on a large screen, using the Piivert input device. We define the components of our instrument in section 4.2, then we describe how we represent and interact with the 3D livelooping trees in section 4.3. Finally, we present the live-looping scenes in section 4.4 and the collaboration/learning possibilities brought by our approach in section 4.5.

Figure 1: Drile used by one expert musician.The live-looping scene contains a three-level tree on bottom left, a two-level tree on top right, 1 leaf node/worm and 2 tunnels(scattering/reverb and color hue/pitch).

RELATED WORK 2.1 3D Virtual Instruments

Many existing immersive instruments focus on navigation in musical environments, like the virtual groove in the Phase project [START_REF] Rodet | Study of haptic and visual interaction for sound and music control in the phase project[END_REF] or the audiovisual grains in Plumage [START_REF] Jacquemin | Plumage: Design d'une interface 3d pour le parcours d'échantillons sonores granularisés[END_REF]. These applications allow musicians to play precomposed musical structures, but they do not give access to the structure of the synthesis processes.

Other immersive instruments are single process instruments, i.e. instruments that allow to interact with only one synthesis process, such as the Virtual Xylophone, the Virtual Membrane, or the Virtual Air Guitar developed by Mäki-Patola et al. [START_REF] Mäki-Patola | Experiments with virtual reality instruments[END_REF] and the sculpting instruments developed by Mulder [START_REF] Mulder | Design of virtual three-dimensional instruments for sound control[END_REF]. The application devel-oped by Mike Wozniewski et al. [START_REF] Wozniewski | A spatial interface for audio and music production[END_REF] relies on users movements to either control the spatialization of pre-recorded sound sources, or apply effects on the sound of an accoustic instrument. All these applications aim at being ordinary instruments, with more control over synthesis processes than navigation tools, that one may use for musical performance. However, they do not take advantage of the possibilities brought by 3D environments in terms of multiple processes handling and structures creation.

An interesting application designed by Polfreman [START_REF] Polfreman | Frameworks 3d: composition in the third dimension[END_REF] takes advantage of 3D environment to build complex musical scores including hierarchical organisation of elements. Nevertheless it is not designed for musical performance but rather for composition.

Live-Looping

Live-Looping is a musical technique that consists in recording musical loops from an audio or control(e.g MIDI messages) input and stacking these loops to quickly build musical structures or sound textures. It has several advantages over other electronic playing techniques such as triggering sequenced loops, or applying effects to different tracks of a prepared song. First of all, there is usually no temporal quantization so it enables more natural musical patterns as opposed to midi sequences. It may fit any musical genre since any musical input and any time-signature can be used. Finally, one can rely on prepared loops or improvise every part of a musical structure. It is mostly used for live performances by different instrumentalists such as guitar players, beat-boxers, or electronic musicians 1 . It is also an interesting tool for music writing, allowing quick sketching of songs.

Live-Looping originates from tape-delay systems and was explored by contemporary composers such as Terry Riley or Steve Reich. This looping system was then implemented in hardware effects racks or guitar pedals as the Gibson Echoplex, Digitech Jamman, Boss RC20 and so on. Usual operations include record, play and stop/mute. One can also overdub, i.e add material to an existing loop, multiply the length of a loop, and reverse a loop. Livelooping has evolved with digital loopers, such as Sooperlooper2 or Freewheeling 3 , which include new possibilities such as synchronization, timestretching, graphical interfaces and so on.

Most live-looping systems deal with audio input, so that the loops are audio buffers. However, some of them, such as LiveLoop 4 , deal with control inputs, such as MIDI or OpenSoundControl messages or other events. They record and playback series of events, which may be either sound triggers or synthesis/effects parameters controls. This control live-looping has many advantages over audio live-looping. First of all, loops can be easily modified since each event is separable. This enables timestretching and addition/removal of events. Notes and effects controls may be recorded independently, so that they can be modified separately. Finally the same recorded loops can easily be rerouted on different synthesis processes.

Some novel instruments are based on live-looping. For example, the BeatBugs [START_REF] Weinberg | The beatbug network: a rhythmic system for interdependent group collaboration[END_REF] are small input devices that record rhythms played by users and repeat them. Recorded rhythms can then be modified and several devices can be synchronized, enabling collaborative musical interaction. Another very interesting application is Fijuu [START_REF] Olive | [END_REF], in which users manipulate 3D audiovisual shapes using a gamepad. Each shape is associated with a specific sound synthesis process which is triggered when the shape is distorted. Audiovisual effects can also be applied on produced sounds and audio loops can be recorded for each shape. These loops can then be accelerated, muted and amplified. However, creation and modification of musical structures remain limited, as well as the inter-action possibilities.

HIERARCHICAL LIVE-LOOPING

Basic live-looping does not enable the creation and manipulation of complex musical structures, but only lists of recorded sequences. It also reduces the possibilities of advanced modification of recorded loops, especially when the content is audio data. We propose to expand the advantages of control live-looping with what we call hierarchical live-looping. The tree structure used in hierarchical live-looping is based on the study of Marczak [5]. This section focuses on the concept of hierarchical live-looping while section 4 describes its implementation as a 3D immersive instrument.

Principle

We define live-looping trees containing leaves l :< c l , x > and nodes n :< ch, c, x >. x is a list of audio effects. c l is a one-shot, i.e no sequences or loops, musical content which can be a soundfile or another synthesis process. ch is a list of children nodes. c is a musical content composed of live-recorded sequences of children events. These events are effects parameters and triggers of the musical content. The audio effects available for all nodes are pitch, volume, distortion and reverb. In addition to these audio effects, the tempo of sequences can be modified, and other high-level musical effects could be added. Each node and leaf also has a content play mode, which can be trigger, i.e the content is played until its end, normal, i.e the content is played until its end or the reception of a stop event, and loop, i.e the content is looped until it receives a stop event. The loop mode is not used for leaves, Control and audio data follow different paths in hierarchical live-looping. Control data go top-down the tree because nodes contain their children sequences. Audio data go bottom-up the tree, which means that the sequences of sounds contained in the leaves will be successively modified by the effects of their parent nodes. The audio and control data flows and the structure of a live-looping trees can be seen on figure 2 Hierarchical live-looping has several advantages over traditional live-looping approaches. First of all, it allows musicians to build musical sequences of any complexity by creating and manipulating trees. A simple example can be seen in figure 3.

It also provides direct access to any level of the trees. One may trigger only the snare or kick to produce a fill, or the drums node to completely modify the rhythm. Effects can be applied to any node independently, whether they are leaves or high-level nodes, in which case the effects will affect all their children. For example one may filter all the drums by modifying the drums node, or pitchshift only the hi-hat.

Live-looping trees also adapt to user with different levels of experience. Beginners can play with tree roots, thus triggering musically interesting parts and applying simple effects to it. Advanced users can go down the tree and manipulate each node. They may change any sequence by re-recording it, or modify the effects associated to the different nodes.

Furthermore, advanced users can easily collaborate to build musical structures. For example, they may add nodes to existing trees, or duplicate nodes to remix loops prepared by others.

Finally, several live-looping trees can coexist, allowing musicians to experiment with non-synchronized loops. This may be useful to create complex sonic textures.

Node operations

Two operations are available for the nodes. Nodes content can be triggered, and nodes audio effects can be modulated. Triggering a leaf node will play its sound whereas triggering a higher-level node will play its sequences of events and parameters. Audio effects parameters can also be modified. In leaves, this will directly affect the musical content, i.e the audio file, whereas in higherlevel nodes it will affect the resulting audio output of all children. Musical effects, such as tempo modification, affect higher-level nodes sequences.

Tree operations

We propose a set of operations for the live-looping trees, which are shown in figure 4: build, merge, duplicate, extract. The build operation has two steps. The first step starts the recording of the events of a node. The second step stops the recording, creates a parent node and define the recorded sequences of events as its content. The playing mode of the parent content is automatically set to loop so that its children sequences keep playing, as it would occur with a traditional live-looping application. The play-ing mode can then be changed to trigger or normal. When a build operation is performed on a node that already has a parent, this operation simply records sequences over existing ones.

The merge operation consists in linking two trees by merging two parent nodes into a single parent node. The oldest sequence of these nodes, i.e the one which was recorded first, is taken as a pulse. Another sequence could be chosen as the pulse, but the usual playing technique in live-looping is to first record the pulse, then the other loops. All the other sequences are then synchronized to multiples of this pulse according to their length, as depicted in figure 5. Thus the parent node synchronizes all its children together, which allows musicians to build regular rhythms. The build and merge operations can be combined if multiple sequences are recorded at the same time. In that case, all simultaneously built parent nodes are merged into a single node. One must note that this operation is not possible if both nodes have parents, since it may lead to too much changes in the live-looping tree.

The duplicate operation consists in duplicating a part of a livelooping tree. This is a useful operation because hierarchical livelooping results in nodes being locked to their parents content, as one node can not have two parents, for musical structure understanding reasons. Thus nodes can not be used in two different sequences at the same time. Using the duplicate operation, one can copy a node in order to use it to build another sequence, with different effects parameters for example.

With the extract operation, one may remove any node and its children from a live-looping tree, making it the root of a new tree, or a leaf if it has no children.

Implementation

Hierarchical live-looping adds new theoretical possibilities to basic live-looping. However, its relevance heavily depends on its implementation. Indeed operations on trees and manipulations of nodes need to be efficient. 3D immersive environments provide new possibilities in terms of interaction, visualization and immersion. They are thus fitted to the control of complex musical structures, especially in a performance context.

DRILE

Drile implements the hierarchical live-looping technique in a 3D immersive environment. In this implementation, leaves contain multisampled sounds which can be played normally or using granular synthesis. Live-looping trees are represented by 3D virtual structures. Musicians interact with these structures in front of a large screen. They are immersed in the 3D environment thanks to stereoscopic glasses and head-tracking. A video of Drile can be watched on http://vimeo.com/9206485.

Technical setup

Drile is composed of three applications which run on Gnu/Linux: drile-audio, drile-ui and drile-ui-display. Drile-audio handles the sound processes and it outputs audio via the Jack sound server. It also uses LV2 audio effects plugins, and VAMP audio analysis plugins. Drile-ui handles the scene graph of the 3D environment. They may run on two separate computers and communicate via OpenSoundControl messages. The stereoscopic display is done by two instances of drile-ui-display synchronized with drileui and connected to separated projectors, one for each eye. In addition, one user (or several with split screens) can be equipped with tracked glasses, to improve immersion and interaction with the environment. Users wear passive stereoscopic glasses and the sound is played using an external usb soundcard and two active speakers.

Worms, Tunnels and Piivert

In Drile, nodes of live-looping trees are represented by what we call worms. These worms, which can be seen on figure 1, have several graphical parameters such as color hue, size, transparency, scattering and so on. These parameters are associated to the nodes audio effects parameters, so that modifying the worms appearance modifies the nodes audio effects. Current mappings are size/volume, color hue/pitch, transparency/distortion and scattering/reverb. They were chosen as a result of a user study, following user preferences and trying to preserve the independance of graphical perceptual dimensions. One must note that the mappings are relative, not absolute. For example, two worms with the same color hue do not necessarily have the same pitch, but their initial pitch is modified by the same amount. Graphical parameters thus give the current value of the effect, like a cursor on a graphical slider, which is essential for efficient interaction.

Worms shapes are associated to the analysed spectrum of nodes audio outputs, to allow users to identify which worm is associated to a node. Finally, each worm rotation on the y-axis is associated to the position of the read pointer in its associated node content. This allows musicians to follow the playback of nodes sequences.

Worms graphical parameters, and thus their associated sound parameters, can be modified by grabbing and sliding the worms through what we call tunnels. Two of them can be seen on figure 1. Each tunnel is associated to one or several graphical parameters, and one or several scales for each parameter. The tunnels are made of a succession of thin cylinders which reflect the values of the parameters scales. The scales can be continuous or discrete. For example, if one associates size with pitch, one may define musical notes with an array of sizes. One may modify only one worm at a time by passing it through a tunnel, but one may also grab and move a tunnel to modify several worms at the same time, for example to mute several loops.

To interact with the environment, we use an input device called Piivert [START_REF] Berthaut | Piivert: Percussion-based interaction for immersive virtual environments[END_REF]. Piivert is a bimanual input device that combines 6DOF tracking, using infrared targets detection, with high-sensitivity pressure sensing. It allows musicians to benefit from the possibilities of graphical interaction while preserving accurate musical controls. Force-resistive sensors are positionned under the thumb, index, middle and ring fingers, allowing us to detect different gestures. These include low-level gestures such as hit and pressure, and high-level percussion gestures such as flam, three-strikes roll and four-strikes roll. This device allows us to select and grab worms and tunnels using a ray-casting technique, i.e a virtual ray that sticks out of the device and goes through the 3D environment. To grab a worm or tunnel, users perform a pressure gesture with their thumb. Low-level gestures transmit vibrations to the worms with the virtual ray, and thus trigger the associated node. Hit gestures play the entire content of the node while pressure gestures play the content using random small grains of the content, which results in granular synthesis for the leaves. Tunnels scale presets can also be changed by selecting the tunnels and performing a hit gesture. Finally, high-level gestures are used to trigger hierarchical live-looping operations. A user holding this device can be seen in figure 1.

Hierarchy

Common 3D selection and manipulation techniques such as the virtual ray technique are more efficient for objects at short distances, as evaluated by Poupyrev et al. [START_REF] Poupyrev | Egocentric object manipulation in virtual environments: Empirical evaluation of interaction techniques[END_REF]. Furthermore, as the size parameter is used to control the volume of worms, allowing continuous modification of worms depth may disrupt the perception of worms size. Thus we propose to keep most worms and tunnels on what we call the interaction plane, at a fixed distance. This preserves spatial and temporal accuracy for musical actions such as selecting and manipulating worms and tunnels. Depth and navigation in the environment may then be used for actions that require less temporal accuracy.

Live-looping trees

As depicted in figure 6, live-looping trees are represented by connected worms organized by depth. Each level of the tree is displayed at a discrete depth. The level that is the closest to the users is on the interaction plane. Worms children are placed in kind of appendages. Musicians may only select and grab the worms on the interaction plane and their direct children. By physically moving in front of the screen, musicians can easily point directly at children with the virtual ray, thanks to head-tracking. When grabbed, the children jump to the depth of the interaction plane so that they can be modified using the tunnels. To access lower-level and higher-level worms, musicians respectively pull and push one of the worms, as depicted in figure 6. When going down the tree, high-level worms move towards users and disappear. Their appendage remain visible, to indicate that displayed worms are children. When users grab and move the root worm of a tree, all the tree follows its translations. When users grab a children worm, move it and release it, it jumps back to its parent appendage.

The immersive environment allows us to display the live-looping hierarchy without overloading the interface and disrupting the interaction. Indeed, manipulation of nodes is done within the interaction plane whereas access to the trees levels relies on the depth of the environment and on the use of head-tracking. These simultaneous manipulations of trees and nodes would be more complicated with a 2D interface, since they could not be done in separated dimensions.

Tree Operations

The build operation needs to integrate closely in a sequence of musical gestures since it records a musical loop. Thus we have defined a high-level gesture with Piivert, actually a ring-middle finger flam. First the user needs to select a worm with the virtual ray. When the record step is triggered, the worm changes shape to highlight its rotation around the y-axis. When the create step is triggered, the created parent node appears and the child worm moves behind it, in its appendage, as depicted on figure 7.

The merge operation is a less temporally critical operation. This it is done by colliding worms. When two worms collide for more than 1 second, the system will try to merge them. All children of one of the worms are transferred to the other, and this empty node is deleted from the scene. Children worms are automatically arranged behind their parent worm in its appendage, as depicted in figure 7. The duplicate operation is also done graphically by grabbing a worm with two rays, i.e two hands, and stretching it, as it can be seen on figure 7. A copy of the stretched worm and all these children, with all contents, is then added to the scene. The stretched worm thus become the root of a new live-looping tree, and is brought to the interaction plane.

Since the extract operation is the opposite of the build operation, it is also triggered by a high-level Piivert gesture. This gesture is a flam with the same fingers but in the opposite direction, i.e middle finger before ring finger. The extracted worm is brought to the interaction plane, its children remaining behind it, as depicted in figure 7. This results in a new live-looping tree.

Live-looping scenes

As explained in section 2.2, the live-looping technique often includes the idea of scenes. A scene is composed of several loops which are synchronized, or which simply fit together musically. With hardware systems, like guitar pedals, scenes are selected using buttons or rotary encoders. Thus each scene is only associated with a number. With 2D looping software, like Freewheeling, scenes are selected with a menu, and are associated with a name. Their musical features, such as tempo or mood, may thus be better described.

In Drile, we define scenes as 3D rooms containing several worms, which can be leaves or high-level nodes of pre-built live-looping trees, and several tunnels.

When the user walks backward from the screen and reaches a specific distance, the camera moves backward, revealing the grid of scenes, as depicted in figure 8. A scene can then be selected simply by looking at it. The head movement needs to be a little exagarated though, as the selection is computed from the position and orientation of the user's head. Finally, the user move to the selected scene simply by walking towards the screen.

Such an approach has several advantages. Visual properties of each scene/room, such as colors, 3D objects, walls and so on, may be used to reflect the musical "mood" of the live-looping trees. Users can refer to these properties to choose the musical content they want to play with. The appearance of the scenes is defined in a configuration file with a XML syntax.

Different tunnels can be chosen for each scene, as audio effects needs may be different for different musical content. A worm can be brought from a previous scene, to make smooth transitions between musical parts. This worm can be a leaf, with a single sound, so that musicians may play it a last time while starting the new musical part. It may also be the root of complex tree, so that the new musical part will consist in modifying the previous tree by progressively adding new sounds.

Performances may follow very different paths, as any transition between scenes is possible and may be done at anytime.

Finally, each scene has a floor. Musicians can bypass worms or complete trees by moving them below this semi-transparent floor. Their audio data is then sent another stereo Jack output, which can be connected to earphones for example. This may be used as a monitor system to prepare trees and then play them, or it can be simply used to quickly mute trees.

Learning and Collaboration

Thanks to hierarchical live-looping, learning Drile is progressive. Indeed, following Birnbaum et al. classification of new instruments in dimension spaces [START_REF] Birnbaum | Towards a dimension space for musical devices[END_REF], Drile happens to be between two different configurations, as depicted in figure 9. New users may begin by interacting only with the root worms of pre-built live-looping trees. This allows them to manipulate musically interesting sequences without much expertise. To push this idea further, we have developed an interaction technique in addition to the ones available with Piivert, which we call the bucket. Users manipulate a virtual bucket using a 6DOF tracked device with a single button. They may then grab and move only worms on the interaction plane. They are not allowed to trigger the sequences, but only pass these worms through tunnels, or bypass them. Interaction is quite simple, i.e grabbing and moving worms, and the musical possibilities are reduced to high-level processes manipulation. Thus required expertise, musical control and degrees of freedom are low.

When users get accustomed to this subset of Drile interaction techniques, they can switch to Piivert. By going down the tree, musicians progressively access rawer musical content, i.e simpler sequences and finally single sounds. Symmetrically, the degree of interaction needed to produce an interesting result, i.e the required expertise, rises. At first, they may interact only with the root worms, this time being able to trigger the musical content. When gaining expertise, they may access lower levels of pre-built live-looping trees, to trigger their nodes and make more complex manipulations. Finally, after more learning, they may modify trees structures by using the available operations, and even build new trees. Obviously required expertised increases as hierarchical livelooping operations and low-level worms manipulations are complex. The number of degrees of freedom also increases because Piivert enables new musical gestures and 3D interaction techniques. Symmetrically, the musical control evolves from high-level control of a musical process to a complete control of the sound and musical parameters of each worm. Collaborative performances may rely on these different expertise levels. An advanced user can easily collaborate with one or several beginner users, as seen in figure 10. In this configuration, the expert builds live-looping trees, and pass them to the beginners so that they can manipulate worms at the foreground, i.e interesting musical sequences. As the bucket does not allow to manipulate the tunnels, the expert may also choose which tunnels should be used by the beginners.

CONCLUSION

In this paper, we presented Drile, an immersive multiprocess instrument especially suited for musical performances. It relies on the concept of hierarchical live-looping, which enhances basic live-looping by allowing musicians to create and manipulate complex musical structures. Drile enables efficient use of this technique thanks to the possibilities brought by immersive environments. In particular, musicians make use of head-tracking, navigation in 3D environments and combination of hierarchy visualization with 3D interaction. Moreover, Drile is well adapted to learning and collaboration between users with different levels, because live-looping trees include different steps of musical interaction complexity.

Collaboration between several beginners users and a single advanced user is simple to setup with one screen, as beginners do not need head-tracking to use the bucket technique. Collaboration between advanced users, i.e using virtual rays and head-tracking, on the same screen could be implemented with a more complex hardware setup.3D avatars may also be used for distant collaboration, as multiple environments could be synchronized using Open-SoundControl messages together with OpenSG internal synchronization features. Following Birnbaum et al. classification, distribution in space would then be maximized. Another interesting issue is the immersion of the audience. They may obviously be equipped with stereoscopic glasses, but one can imagine various stage setups. Musicians and the audience may both face the same screen, though musicians may occlude parts of the screen. 3D avatars may also be used to display musicians while they interact in front of another screen. Future work will investigate these different issues.

Figure 2 :

 2 Figure 2: Basic live-looping data flow (left) and hierarchical live-looping control (in black) and audio (in red) data flows.

Figure 3 :

 3 Figure 3: Hip-Hop live-looping tree example.

Figure 4 :

 4 Figure 4: Hierarchical live-looping operations.

Figure 5 :

 5 Figure 5: Example of sequences synchronization when merging several nodes. Here the oldest sequence is S1.

Figure 6 :

 6 Figure 6: Three-levels live-looping tree (left). Pulling a worm goes down the tree (right).

Figure 7 :

 7 Figure 7: Live-looping trees operations (from right to left, top to bottom): Build, Merge, Duplicate, Extract

Figure 8 :

 8 Figure8: View of all scenes (here 4). Here one tree is being moved from one scene to another.

Figure 9 :

 9 Figure9: Drile Dimension Space, in purple for beginners with buckets and only access to root worms, in blue for advanced users with piivert and access to all worms of the live-looping trees.

Figure 10 :

 10 Figure 10: Collaboration between one advanced user with Piivert (left) and two beginner users with buckets

  .

	Basic Live-Looping	Hierarchical Live-Looping
	Input (Audio/Control)	Audio Output Control Input
						Node2
						Effects
						Sequences2
						Node1
	Buffer1	Buffer2	Buffer3	Buffer4	Buffer5	Sequences1 Effects
					Leaf1		Leaf2
					Effects	Effects
					Sound1	Sound2

http://www.livelooping.org/

http://www.essej.net/sooperlooper/

http://freewheeling.sourceforge.net/

http://code.google.com/p/liveloop/


Florent Berthaut

Myriam Desainte-Catherine

Martin Hachet

COMBINING AUDIOVISUAL MAPPINGS FOR 3D MUSICAL INTERACTION

3D environments provide new possibilities for musical interaction. They allow musicians to manipulate and visualize large sets of sound processes associated to 3D objects by connecting graphical parameters to sound parameters. Several of these audiovisual mappings can be combined on a single 3D object. However, this brings up the issues of the choice of these mappings and of their combinations. We conducted a user study on sixteen musicians to evaluate audiovisual mappings and their combinations in the context of 3D musical interaction. This user study is composed of three experiments. The first experiment investigates subjects preferences for mappings between four perceptual sound parameters (amplitude, pitch, spectral centroid and noisiness) and ten graphical parameters, some of them specific to 3D environments. The second experiment focuses on efficiency of single mappings in an audiovisual identification task. The results show almost no significant differences, but some tendencies, which may indicate that the choice of mappings scales is more important than the choice of the mappings themselves. The third experiment investigates the efficiency of mappings combinations. The results indicate no significant differences, which suggest that it may be possible to combine up to four audiovisual mappings on a single graphical object without any performance loss for musicians, if they do not disrupt each other.

INTRODUCTION

The past decade has seen an important development of graphical musical interfaces. These interfaces don't share the physical limitations of hardware controllers. Consequently, they give access to a high number of control parameters of an unlimited number of sound processes. Each musical parameter can be modified by interacting with the graphical components of the interface. Furthermore, in opposition to traditional instruments, graphical interfaces allow users to display precise information about the musical events, such as their activity, their current parameters and so on. Immersive virtual environments, used in the field of virtual reality, add even more possibilities for musical applications. In particular, they enable the visualization of large sets of audiovisual objects, with additional parameters such as distance, 3D orientation, 3D shape, transparency. 3D interaction techniques, such as virtual sculpting, ray-casting, or navigation, may also be favourably used for musical purposes. These interfaces bring up the issue of the choice of audiovisual mappings.

Which graphical parameters of 3D objects should be used to display sound parameters for visualization and manipulation ? Appropriate mappings and visual organisation may improve interaction efficiency as it is done in the information visualization field. Moreover, using combinations of audiovisual mappings, as is it done for generic data by Healey [START_REF] Healey | Building a perceptual visualisation architecture[END_REF], users may manipulate and visualize many musical parameters on a single 3D object, in opposition to simple sliders controlling only one parameter each. Thus it is important to evaluate these different mappings, in terms of subjects preferences and performances, in the context of musical interaction, i.e with varying sound parameters.

In this paper, we present a user study that investigates user rates, in section 3.5, and efficiency, in section 3.6, of audiovisual mappings for 3D graphical musical interaction. It is also aimed at evaluating the quantity of musical information, i.e sound parameters, that a graphical component may display without causing disruption. This last evaluation is done in the third experiment, described in section 3.7.

RELATED WORK

A lot of work has been done on linking images to sound or music [START_REF] Lemi | Reviewing the transformation of sound to image in new computer music software[END_REF], from Isaac Newton who noticed the correspon-dence between prismatic rays and musical scales, to the first color-music instruments such as "Le clavecin oculaire" from Louis-Bertrand Castel, to the extensive work of John Whitney [START_REF] Whitney | Digital Harmony: on the Complementarity of Music and Visual Art[END_REF].

Existing 3D instruments rely on traditional visualization of sounds, such as 3D spectrograms or 3D speakers for spatialization [START_REF] Chaudhary | Visualization, Editing and Spatialization of Sound Representations using the OSE Framework[END_REF]. None of them combine several audiovisual mappings on the 3D objects, thus they reduce the visualization and interaction possibilities. For example, 3D spectrograms must be viewed from a certain angle to be understandable, so that users cannot see or manipulate many sounds processes representations at the same time. Other applications use simple sliders transposed to 3D, reducing the control possibilities brought by 3D objects to a single parameter.

Many studies were done on audiovisual mappings. Walker [START_REF] Walker | The effects of culture, environment, age, and musical training on choices of visual metaphors for sound[END_REF] studied extensively both musicians and non-musicians preferences concerning audiovisual mappings, with a musical notation perspective, using sequences of static 2D graphical signs matched with sound sequences. He found significant preferences, but only within the musicians groups. Another study was made by Lipscomb [START_REF] Lipscomb | Perceived match between visual parameters and auditory correlates: an experimental multimedia investigation[END_REF], in order to build a visualization tool. Variations between six distinct instruments timbres were matched to shape variations. In opposition to previous studies, no significant differences were found between musicians and non-musicians. According to the author, this could be related to the low number of subjects. Visual stimuli were three states animations of 2D shapes. Finally, very interesting work was done by Giannakis [START_REF]Imaging soundscapes: Identifying cognitive associations between audatory and visual dimensions[END_REF][START_REF] Giannakis | Towards a theoretical framework for sound synthesis based on auditory-visual associations[END_REF][START_REF] Giannakis | A comparative evaluation of auditoryvisual mappings for sound visualisation[END_REF], that led to Sound Mosaic, a tool for synthesis and composition. He concentrated on perceptual dimensions of color and texture as visual parameters, thus following research of the information visualization field. Subjects had to choose sequences of images from palettes to represent sequences of sounds consisting of linear or random variations of the auditory parameters. The results of these studies are presented in figure 1.

However these user studies do not answer our specific questioning, which is the audiovisual mappings of dynamic parameters on 3D objects and their combinations.

Author

Auditory Feature Visual Feature Walker [START_REF] Walker | The effects of culture, environment, age, and musical training on choices of visual metaphors for sound[END_REF] loudness size pitch vertical position timbre pattern duration horizontal length Lipscomb [START_REF] Lipscomb | Perceived match between visual parameters and auditory correlates: an experimental multimedia investigation[END_REF] This study tried to answer three questions. Do mappings preferences observed in other studies still appear with 3D representations ? Are some audiovisual mappings more efficient for an identification task than others, especially in a dynamic context ? Does combining several mappings on a single graphical component improve performances of an identification task ?

In the first experiment, we studied the preferences of the subjects on the mappings between four audio parameters and ten graphical parameters. These first results were used to reduce the number of mappings, keeping the preferred ones. In the second experiment, we studied the efficiency of these mappings. To do so, we measured the subjects' performances in an audiovisual object identification task. The results were then used to choose one graphical parameter for each audio parameter. In the third experiment, we studied the effect of combining the chosen mappings on the subjects' performances.

Stimuli

Auditory stimuli

Several types of sound parameters could be controlled and visualized. Perceptual parameters are for example loudness, pitch, and the different dimensions of timbre. Low-level physical parameters include the waveform and the spectrum. There are also parameters of the different sound synthesis techniques, such as additive, substractive, granular and so on. Finally there are sound processing parameters, including the parameters of effects, for example pitchshifting, distortion and filter.

Perceptual parameters provide numerous advantages. They are more general because they can be extracted from any synthesized or recorded sound source, more easily than sound processing parameters. Moreover, they are understandable, as opposed to some synthesis parameters, so that one clearly perceives the effect of their variations. Some of them can be modified by a single standard audio effect, such as pitchshifting, volume, distortion or low-pass filter. Thus they can be controlled as well as visualized.

Four auditory perceptual parameters were used in the experiments: pitch, amplitude/loudness, brightness / spectral centroid and noisiness / irregularity. Seven audio sequences of 7 seconds were generated for each parameter using the Renoise1 tracker, starting with a simple sinusoid and then using Renoise's internal effects to produce variations. The audio loops were played and analysed in realtime to get the perceived audio parameters in PureData2 .

The amplitude parameter featured random discrete and continuous variations ranging from 0dB to 30dB and analysed with the env∼ object. The discrete variations of the pitch parameter were values randomly chosen within the notes {60, 63, 67, 68, 72}, the corresponding frequencies being {261.6Hz, 311.1Hz, 391.9Hz, 415.3Hz, 523.2Hz}, and analysed with the fiddle∼ external [START_REF] Puckette | Realtime audio analysis tools for pd and msp[END_REF]. The pitch variations were small (within one octave) in order to prevent variations of perceived loudness. The brightness / spectral centroid changed with random discrete and continuous variations ranging from 948Hz to 998Hz and analysed with the sc∼ external from the flib library. Finally, the noisiness / irregularity had random discrete and continuous variations ranging from 3.6 to 170 and analysed with the irreg∼ external from the flib library, calculated according to Jensen [START_REF] Jensen | Timbre models of musical sounds[END_REF] and based on the original formula by Krimphoff [START_REF] Krimphoff | Caractérisation du timbre des sons complexes. ii: Analyses acoustiques et quantification psychophysique[END_REF].

Seven sequences of 7 seconds were also made by combining variations of the four parameters.

Visual stimuli

In order to provide fast and precise graphical control and visualization, perceptually salient parameters are needed. The field of information visualization has proven the efficiency of preattentive visual features, as described by Healey [START_REF] Healey | High-speed visual estimation using preattentive processing[END_REF]. These visual properties are detected without the need for focused attention, independently from the number of displayed elements, so that tasks involving them, such as detecting a filled circle in a group of empty circles, take 200ms or less. Target detection or estimation of the number or percentage of visual elements can thus be improved by using them. Examples of preattentive features are 2D/3D orientation, length, size, curvature, number, color hue, intensity, direction of motion, stereoscopic depth, lighting direction. Some of these features can be organized into a hierarchy, for example color luminance affects color hue perception which in turn affects visual texture perception. As explained in the introduction, these features are also often combined, for example in [START_REF] Healey | Building a perceptual visualisation architecture[END_REF], to enable efficient visualization of complex data. Furthermore, studies by Luck and Vogel [START_REF] Luck | The capacity of visual working memory for features and conjunctions[END_REF] demonstrated that it is possible to retain information about four visual objects defined by a conjunction of four features in working memory.

For our experiment, ten graphical parameters were chosen, some of them being preattentive features. The graphical parameters were applied to cubes rendered in a 3D environment using the OpenSG 3 scenegraph library. Distance on the z-axis ranged from the positions -1 to -3, camera being on 0. Ground and shadows were also used in order to improve its perception. Translations on x and y axis were not tested, in order to save them fo the spatial organisation of the 3D objects. The orientation parameter was a rotation around the z-axis ranging from 0rad to 1rad, so that 3 http://opensg.vrsource.org/trac the rotation was understandable. The size parameter was a scale operation ranging from 0.4 to 0.9, so that the object did not disappear. The texture rugosity parameter was rendered using relief bump mapping with scale parameter ranging from 0 (flat) to 10 (very rough). The scale of the object's texture varied for the texture scale parameter from original size to eight times this size. The speed parameter was rendered by a circular motion ranging from 0 to Pi unit per second. One dimension only was selected for the color parameter, color lightness, mostly because we wanted to keep the experiments short enough. There is a hierarchy between features, as said before luminance perturbs hue perception, and that would have cause problems in the combinations experiments. The color was ranging from {0,0,0} to {255,255,255}. The shininess parameter ranged from 0 to 1. It corresponded to the amount of specular reflection of the 3D objects surfaces. The shape distortion parameter corresponded to the scale of a random modification of the object's vertices' positions, ranging from 0 to 0.5 . Simple generative shapes [START_REF] Ebert | Procedural shape generation for multi-dimensional data visualization[END_REF] or more complex shapes could have been experimented, but it was instead decided to choose a parameter which could affect any complex shape. Indeed, because complex shapes allow for multiple dimensions whithout disturbing other features, they could be used to represent multidimensional perceptual phenomena, such as the sound spectrum divided into the twenty-four critical bands of hearing [START_REF] Zwicker | Subdivision of the audible frequency range into critical bands[END_REF], and thus they were not selected for this experiment. The transparency parameter varied from 0 to 0.8, so that the object did not completely disappear. To make sure that transparency was correctly perceived, a randomly generated background was added.

The 3D animations were rendered in realtime, visual parameters being connected to the audio parameters by Open-Soundcontrol4 messages which were sent from PureData to the 3D software. The scales of the graphical parameters were linear and chosen in order to fit the ranges of the audio parameters.

Subjects

There were sixteen subjects (13 males and 3 females) aged between 22 and 54. All of them were trained musicians (having at least taken music lessons), and they all had already used music software. None of them were regular users of 3D software.

Experimental Setup

Subjects sat in front of a 12,1" laptop screen and they were equipped with Beyerdynamic DT-770 headphones. They went through the experiments and entered their answers using a modified computer keyboard on which all the keys were removed, except the spacebar and the four keys used to select the objects in the second and third experiments.

First experiment: Single mappings -Preferences

Goals

The goal of this first experiment was to test subjects preferences for all the possible mappings. Our hypothesis was that several significant preferences would appear, similar to what has been found in existing user study.

Experimental procedure

This experiment was composed of four tests, one for each sound parameter, i.e amplitude, pitch, centroid and noisiness. Ten cubes were displayed in each test, each cube using a different graphical parameter to display the sound parameter variations, as seen on figure 2. The subjects were asked to rate verbally the relevance of each mapping between 1 and 5. The order of the tests was randomly chosen.

Results and analysis

The results are shown in figures 3, 4 and 5. Kruskal-Wallis tests indicated a significant effect for Amplitude (χ 2 (9) = 57.81, p < 0.0001), Pitch (χ 2 (9) = 32.178, p = 0.0002), Spectral Centroid (χ 2 (9) = 29.932, p = 0.0005), but no significant effect for Noisiness. Wilcoxon analysis for any two mappings of the significant tests revealed the significant differences shown below each figure.

Subjects comments

Some subjects would have preferred an inverted scale for some mappings. For example, they felt that higher pitch should correspond to smaller object. This is consistent with the the fact that high frequencies are perceived louder than low frequencies. Most subjects assimilated distance to a 1D fader. They also reported that changes in the sound identity, ie. timbre parameters, should be associated to changes in the object's identity, i.e shape. Subjects felt that the speed parameter should have been applied to something else than a circular movement.

Discussion

The results indicated significantly preferred graphical parameters for amplitude, pitch and spectral centroid. In particular, subjects associated amplitude with size as in Walker's experiments, but also with distance and transparency. The tendency for the pitch parameter was to associate it with distance, which subjects in their comments described as a 1D fader. Thus it seemed to correspond to the vertical position used in Walker's study. Color lightness was significantly preferred for spectral centroid and the tendency for noisiness was to associate it with texture rugosity. That could be explained by the correlation seen in the other studies between the inner aspect and the timbre parameters. We chose to remove the graphical parameters with the worst ranks for these four audio parameters and keep the five best ones, to evaluate the mappings with our performance experiment.

3.6. Second experiment: Single mappings -Performances

Goals

The goal of this second experiment was to test, for each sound parameter, the efficiency of the five graphical parameters that had been identified as the preferred ones in the first experiment. This was done by measuring the subjects performances in an identification task. For the amplitude parameter, size, distance, color, transparency and shininess were tested. For the pitch parameter, distance, color lightness, shininess, size and orientation were tested. For the spectral centroid parameter, the tested parameters were color, . The significant differences between the mappings from a Wilcoxon analysis are also given.

shininess, size, transparency and distortion. And for the noisiness parameter, texture rugosity, color, distortion, size and transparency were tested.

Our hypothesis was that some graphical parameters would be more efficient than others, because musicians were used to some representations.

Experimental procedure

The second experiment was composed of four tests, one for each sound parameter. Each test involved 5 conditions, one for each selected graphical parameter, and each condition involved seven trials. During each trial, an audio sequence corresponding to the sound parameter was played and four cubes were displayed, as seen in figure 6. Subjects were asked to identify as quickly as possible the cube whose graphical variations were connected to the sound parameter. They selected it with the corresponding key of the keyboard. Response times and error rates were recorded. At the end of each condition, subjects were also asked to give an evaluation of its easyness between 1 and 5. To avoid a learning effect, the order of the sequences and of the conditions for each audio parameter were randomly chosen. A practice trial was also added at the beginning of each condition.

Results

The response times are shown in figure 7. An analysis of variance (ANOVA) indicated a significant effect of the mapping choice for Amplitude (F = 6.131 > F( 4 An ANOVA indicated no significant differences for the error rates of the four tests.

A test of correlation was done for each user between their preferences in the first experiment and their response times of the performances experiment. It revealed that there was no correlation between these results, with an average value of 0.16.

Subjects comments

Subjects preferred mappings that had an analogy with the real world, for example when sound got brighter as the object got closer, when sound got louder when the object got bigger or when sound got higher pitched when the object got smaller. They indicated that the scale for the orientation parameter was not well chosen, and also that the changes in the rugosity parameter was not correctly discernible. Subjects felt that the scale of the transparency should be modified to better fit some of the audio parameters. They also expressed the need of reversing the scale for some mappings. Subjects interestingly reported that they used mainly quick variations of the parameters to identify the correct object, because they perceived them better than slow variations, no matter which graphical parameter was tested.

Discussion

First of all, our results indicated that there was no correlation between the preferences and the performances of the subjects. In other words, the preference of a user for one mapping did not mean that this mapping was the most efficient. This confirmed the need for a user study with objective performance measurements. We almost did not find any significant differences between response times for the different mappings, while we had found some in the preferences experiment. Differences that were found for shininess, orientation, and texture rugosity, seemed to be scale problems. That was indeed confirmed by the subjects comments. Moreover, subjects reported that they followed fast variations, no matter which graphical parameter was used, to identify which object was associated to the sound. This led us to think that the scales of the mappings were finally more important than the map-pings themselves.

Even if the precedent results had shown no significant differences between the mappings, we chose by a process of elimination which graphical parameter we would use for each audio parameter, in relation to the tendencies revealed by the results. So we finally chose the following mappings: Size with Amplitudes, Color with Pitch, Distortion with Spectral Centroid and Transparency with Noisiness. The goal of this last experiment was to test the influence of the number of simultaneous audiovisual mappings on performances. Our hypothesis was that a larger number of audiovisual mappings would improve the identification task performances, because subjects would have several perceptual clues at the same time.

Experimental procedure

The third experiment was made up of two parts, each with four sequences and seven trials by sequence. Subjects again had to choose which of the four cubes displayed was connected to the sound they were hearing by selecting it with the corresponding key of the keyboard, and then give a score for each sequence. For the first part, the sound was a simple sinusoid with only amplitude variations. In the first sequence of this part, only the amplitude-size mapping was activated. In the second, third and fourth sequences, respectively one, two and three additional graphical parameters were randomly animated, to disrupt the subject. For the second part, the sound was a sinusoid with combined variations of the four audio parameters. In the first sequence, only one mapping was activated, size was connected to the amplitude. In the second, third and fourth sequences the following mappings were respectively added: color and pitch, distortion and brightness, transparency and noisiness, as shown in figure 8. In order to avoid a learning effect, the order of the sequences in each part was randomly chosen. This experiment was ran a second time, replacing the color parameter with the distance parameter, to test the interference between size and distance variations.

Results

The response times of the first part of the third experiment are shown in figure 9. An ANOVA indicated a significant effect (F = 9.498 > F(3, 45, 0.05) = 2.812) of the number of disrupting graphical parameters when using the distance parameter. Student-Neumann-Keuls analysis revealed a significant difference between the first sequence and the three other sequences with disruptions.

When using the color parameter, the ANOVA also indicated a significant effect (F = 10.094 > F(3, 45, 0.05) = For the second part of this third experiment, the response times are shown in figure 10. An ANOVA on the results indicated no significant performance difference (F = 0.368 < F(3, 45, 0.05) = 2.812) when activating one, two, three or four mappings at the same time, with color as the second graphical parameter. The same lack of significance (F = 0.550 < F(3, 45, 0.05) = 2.812) was obtained with distance as the second parameter.

Subjects comments

Subjects reported having troubles to perceive size variations when the distance parameter was also used. They also felt there was a problem with distance perception, because of the camera point of view. They found it more difficult to select the correct object when the variation of an audio parameter was heard but not visualised, as in the first sequence of the combination part, than when graphical parameters were randomly variating.

Discussion

One should notice that the response times are much longer than in the previous experiments. This may be explained by the fact that musical sequences were more complex and that it took more time to understand them. Some subjects were also disrupted by the combination of graphical parameters, to which they were not used.

The first part of the experiment revealed that the performances times dropped when a graphical parameter disrupted the perception of another. This was obvious with distance and size, and in the second test with color and transparency. This confirms the importance of the independence of perceptual dimensions, which was pointed out by Giannakis [START_REF] Giannakis | Towards a theoretical framework for sound synthesis based on auditory-visual associations[END_REF].

However, the second part of the experiment did not indicate any significant difference when combining up to four mappings on the objects. This part was different from the first one, because the graphical parameters variations did not always occur at the same time, and because they were connected to audio parameters variations. The results were not what we expected, i.e the combinations do not improve the performances. However, if the graphical parameters do not interfere with each other, these results suggest that may have at least four audio parameters displayed on a single object without performance loss.

CONCLUSION

Our results for the preferences experiment approximatively followed the results of previous user studies on musicians. Significant preferences indeed appeared for the Amplitude / Size and Spectral Centroid / Color Lightness mappings. These results also showed tendencies for the Pitch / Distance and Noisiness / Texture Rugosity mappings. The performances experiment, which was designed to measure the efficiency of the audiovisual mappings, revealed almost no significant differences, except for mappings with wrong scales. These results, combined with subjects comments, may indicate that the choice of the scales of the mappings is actually more important for the efficiency than the choice of the mappings themselves. The results of the combinations experiment did not indicate any significant drop in perfomances when using several mappings on the objects, except when some graphical parameters disrupted the perception of others. This may indicate that musicians can deal with at least four independent audiovisual mappings on a single object without performance loss, if the scales and mappings are correctly chosen. In a 3D environment, where one may navigate in large sets of audiovisual objects, this may bring new musical possibilities.

However, these results may be strongly due to the fact that subjects were musicians, used to specific visualizations. This prevents us from generalizing our conclusions to nonmusicians. The results may also depend on the scales that we used.

Future studies should thus investigate which are the most efficient scales for the audiovisual mappings, taking inspiration from the information visualisation field. Other graphical and audio parameters should also be tested, for example higher level musical parameters like tempo, tonality, rythm and so on. Complex 3D shapes could also be useful to display sound parameters including several dimensions such as sound spectrum.

Figure 1 .

 1 Figure 1. Results of Previous User Studies for AudioVisual Mappings

Figure 2 .

 2 Figure 2. Screenshot of the first experiment

pFigure 3 .

 3 Figure3. First experiment : Preferences -Subjects rates for Amplitude (and average deviations). The significant differences between the mappings from a Wilcoxon analysis are also given.

p

  = 0.0076 p = 0.0033 p = 0.003 p = 0.0125 p = 0.0035 p = 0.0455 p = 0.0047 p = 0.0033 p = 0.003 p = 0.0022 p = 0.0294 p = 0.0211 p = 0.0067 p = 0.0192 p = 0.0178 p = 0.0088 p = 0.0277 p = 0.008 p = 0.0469

Figure 4 .

 4 Figure 4. First experiment : Preferences -Subjects Rates for Pitch (and average deviations). The significant differences between the mappings from a Wilcoxon analysis are also given.

pFigure 5 .

 5 Figure 5. First experiment : Preferences -Subjects rates for Spectral centroid (and average deviations). The significant differences between the mappings from a Wilcoxon analysis are also given.

Figure 6 .

 6 Figure 6. Second experiment screenshots: Amplitude mapped to Shininess and Amplitude mapped to Size

Figure 7 .

 7 Figure 7. Second experiment : Performances -Response times (in ms) (and average deviations) for Amplitude, Pitch, Spectral centroid and Noisiness

3. 7 .

 7 Third experiment: Combined mappings 3.7.1. Goals

Figure 8 .Figure 9 .

 89 Figure 8. Third experiment: Combinations -Screenshots: two and four combined mappings

Figure 10 .

 10 Figure 10. Third experiment: Combinations -Response times (and average deviations) using Color Lightness then Distance for the Pitch parameter

http://www.renoise.com/

http://puredata.info/

http://opensoundcontrol.org/