Arslan, B. * A Brouse

J † Castet

J J ‡ Filatriau

R ‡ Lehembre
email: lehembre@tele.ucl.ac.be

Q ‡ Noirhomme
email: noirhomme@tele.ucl.ac.be

C Simon
email: simon@tele.ucl.ac.be

From Biological Signals to Music

This project proposes to use the analysis of physiological signals, such as the electroencephalogram (EEG), electromyogram (EMG) electrocardiogram (ECG) and electro-oculogram (EOG), to control sound synthesis algorithms in order to build a biologically driven musical instrument. This project took place during the eNTERFACE'05 summer workshop in Mons,

Belgium. Over four weeks specialists from the fields of brain-computer interfaces and sound synthesis worked together to produce biologically controlled musical instruments playable in real-time.

Introduction

Advances in computer science and specifically in Human-Computer Interaction (HCI) have now enabled musical performance using sensor-based instruments in real-time computer synthesis systems [START_REF] Tanaka | Musical Performance Practice on Sensor-based Instruments[END_REF]. Musicians can now use positional, cardiac, muscle and other sensor data to control sound synthesis [START_REF] Nagashima | Bio-Sensing Systems and Bio-Feedback Systems for Interactive Media Arts[END_REF][START_REF] Knapp | Multimodal Interaction in Music Using the Electromyogram and Relative Position Sensing[END_REF].

Simultaneously, advances in Brain-Computer Interface (BCI) research have proven that cerebral patterns can be used as a source of communication and control [START_REF] Wolpaw | Braincomputer interfaces for communication and control[END_REF]. Indeed, cerebral and conventional sensors can be used together with the object of producing a 'bodymusic' controlled according to the musician's cognitive and proprioceptive processes. Research is already being done toward integrating BCI and sound synthesis [START_REF] Brouse | Petit guide de la musique des ondes cérébrales[END_REF][START_REF] Miranda | Toward Direct Brain-Computer Musical Interfaces[END_REF].

One salient approach aims to sonify the data derived from physiological processes by mapping the data directly to sound synthesis parameters [START_REF] Berger | Singing the mind listening[END_REF][START_REF] Potard | Listening to the mind listening : sonification of the coherence matrix and power spectrum of eeg signals[END_REF][START_REF] Dribus | The other ear : a musical sonification of EEG data[END_REF]. Another approach aims to build a musical interface where inference based on complex feature extraction enables the musician to intentionally control sound production [START_REF] Miranda | Toward Direct Brain-Computer Musical Interfaces[END_REF].

In the following, we present: a short history of biologically-controlled instruments; the architecture we designed to acquire, process and play music based on biological signals; strategies for signal acquisition; a discussion of signal processing techniques; the sound synthesis implementation and the instruments we built; and conclude with a presentation of some future directions.

History

Brainwaves are a form of bioelectricity, or electrical phenomena in animals or plants. Human brainwaves were first measured in 1924 by Hans Berger. He termed these electrical measurements the electroencephalogram (EEG), which means literally 'brain electricity writing'. Berger first published his brainwave results in 1929 as " Über das Elektrenkephalogramm des Menschen" [START_REF] Berger | Über das elektrenkephalogramm des menschen[END_REF]. The English translation did not appear until 1969. His results were verified by Adrian and Matthews in 1934 who also attempted to listen to the brainwave signals via an amplified speaker [START_REF] Matthews | The Berger Rhythm: Potential Changes from the Occipital Lobes in Man[END_REF].

This was the first attempt to sonify human brainwaves for auditory display. The first instance of the intentional use of brainwaves to generate music did not occur until 1965, when Alvin Lucier [12], who had begun working with physicist Edmond Dewan, composed a piece of music using brainwaves as the sole generative source. Music for Solo Performer was presented, with encouragement from John Cage, at the Rose Art Museum of Brandeis University in 1965.

In the late 1960s, Richard Teitelbaum was a member of the innovative Rome-based live electronic music group Musica Elettronica Viva (MEV). In performances of Spacecraft (1967) Then, in the late 1960s, another composer, David Rosenboom, began to use EEG signals to generate music. In 1970-71 Rosenboom composed and performed Ecology of the Skin, in which ten live EEG performer-participants interactively generated immersive sonic/visual environments using custom-made electronic circuits. Around the same time, Rosenboom founded the Laboratory of Experimental Aesthetics at York University in Toronto, which encouraged pioneering collaborations between scientists and artists. For the better part of the 1970s, the laboratory undertook experimentation and research into the artistic possibilities of brainwaves and other biological signals in cybernetic biofeedback artistic systems. Many artists and musicians visited and worked at the facility during this time including John Cage, David Behrman, LaMonte Young, and Marian Zazeela. Some of the results of the work at this lab were published in the book "Biofeedback and the Arts" [START_REF] Rosenboom | Biofeedback and the arts : results of early experiments[END_REF]. A more recent monograph by Rosenboom, "Extended Musical Interface with the Human Nervous System" [START_REF] Rosenboom | Extended Musical Interface with the Human Nervous System[END_REF], remains the definitive theoretical aesthetic document in this area.

In France, scientist Roger Lafosse was doing research into brainwave systems and proposed, along with musique concrète pioneer Pierre Henry, a sophisticated live performance system known as Corticalart (art from the cerebral cortex). In a series of free performances done in 1971, along with generated electronic sounds, one saw a television image of Henry in dark sunglasses with electrodes hanging from his head, projected so that the content of his brainwaves changed the colour of the image according to his brainwave patterns.

Starting in the early 1970s, Jacques Vidal, a computer science researcher at UCLA, simultaneously began working to develop the first direct brain-computer interface (BCI) system using a IBM mainframe computer and other custom data acquisition equipment. In 1973, he published "Toward Direct Brain-Computer Communication" [START_REF] Vidal | Toward Direct Brain-Computer Communication[END_REF] based on this work.

In 1990 Jonathan Wolpaw et al [START_REF] Wolpaw | An EEG-based brain-computer interface for cursor control[END_REF] at Albany developed a system to allow a user to exercise rudimentary control over a computer cursor via the alpha band of their EEG spectrum. Around the same time, Christoph Guger and Gert Pfurtscheller also began researching and developing BCI systems along similar lines in Graz, Austria [START_REF] Pfurtscheller | Current trends in Graz Brain-Computer Interface (BCI) research[END_REF].

In the early 1990s two scientists, Benjamin Knapp and Hugh Lusted [START_REF] Knapp | A Bioelectric Controller for Computer Music Applications[END_REF], began working on a humancomputer interface called the BioMuse. It permitted a human to control certain computer functions via bioelectric signals. In 1992, Atau Tanaka [START_REF] Tanaka | Musical Performance Practice on Sensor-based Instruments[END_REF] was commis-sioned by Knapp and Lusted to compose and perform music using the BioMuse as a controller. Tanaka continued to use the BioMuse, primarily as an EMG controller, in live performances throughout the 1990s. In 1996, Knapp and Lusted wrote an article for Scientific American about the BioMuse entitled "Controlling Computers with Neural Signals" [START_REF] Knapp | Controlling Computers with Neural Signals[END_REF].

In 2002, the principal BCI researchers in Albany and Graz published a comprehensive survey of the state of the art in BCI research, "Brain-computer interfaces for communication and control" [START_REF] Wolpaw | Braincomputer interfaces for communication and control[END_REF]. Then, in 2004, an issue dedicated to the broad sweep of current BCI research was published in IEEE Biomedical Transactions [START_REF]BCI special issue[END_REF].

Architecture

Our intention was to build a robust, reusable framework for biosignal capture and processing geared towards musical applications. To maintain flexibility, signal acquisition, processing and sound synthesis are performed on different physical machines linked via ethernet. Data are acquired via custom hardware which is linked to a host computer running a Matlab/Simulink [START_REF]The Mathworks[END_REF] real-time blockset. Data are analysed before being sent -via OpenSoundControl [START_REF] Wright | [END_REF] -to the visualisation, software sound synthesis and spatialisation nodes. The sound synthesis and spatialisation are performed using the Max/MSP [START_REF] Puckette | [END_REF] programming environment.

Software

Matlab and Simulink

We are using various biosignal analysis methods including the wavelet transform and spatial filters. All of the signal processing algorithms are written in Matlab [START_REF]The Mathworks[END_REF]. Because signal acquisition from the EEG cap is done using custom C++ code, we must use a method in C++ to send the data stream to Matlab directly. We implemented our signal processing code as a Simulink [START_REF]The Mathworks[END_REF] blockset using Level-2 M file S-functions with tuneable method parameters. This allows us to dynamically adapt to the incoming signals. Subsequently, we proceed with a real-time, adaptive analysis.

Max/MSP

Max/MSP [START_REF] Puckette | [END_REF] is a software programming environment optimised for flexible real-time control of music systems. It was first developed at IRCAM by Miller Puckette as a simplified front end controller for the 4X series of mainframe music synthesis systems. It was fur-

ENACTIVE05

From Biological Signals to Music ther developed as a commercial product by David Zicarelli [START_REF] Zicarelli | An extensible real-time signal processing environment for Max[END_REF]

Data Exchange

Data transmission between machines is implemented using UDP/IP protocol over ethernet. We chose this for best real-time performance. Reliability of UDP on an ethernet LAN is not an issue from experience. Specific musical messages were encoded using the OpenSound-Control [START_REF] Wright | [END_REF] protocol which sits on top of UDP.

Open Sound Control (OSC)

OSC was conceived as a protocol for the real-time control of computer music synthesisers over modern heterogeneous networks. Its development was informed by shortcomings experienced with the established MIDI standard and the difficulties in developing a more flexible protocol for effective real-time control of expressive music synthesis.

OSC was first proposed by Matthew Wright and Adrian Freed in 1997, since which time it has become very widely implemented in software and hardware designs (although, its use is still not as widespread as MIDI). Although it can function in principle over any appropriate transport/physical layer such as WiFi, serial, USB etc., current implementations of OSC are optimised for UDP/IP transport over Fast Ethernet in a Local Area Network. For our project, we used OSC to transfer data from Matlab (running on a PC with either Linux or Windows OS) to Macintosh computers running Max/MSP.

Data Acquisition

ECG, EMG and EOG were captured on one computer with a multipurpose acquisition system and EEG was acquired on another system specialised for brainwave data capture.

EEG

EEG data are sampled at 64 Hz on 19 channels with a DTI cap. Data are filtered between 0.5 and 30 Hz. Electrodes are positioned following the 10-20 international system and Cz is used as reference. The subject sits in a comfortable chair and is asked to concentrate on different tasks. The recording is done in a normal working place: a noisy room with people working, talking and other ambient sounds. The environment is not free of electrical noise as there are many computers, speakers, monitors, microphones and lights nearby.

EMG, ECG and EOG

To record the EMG and ECG, three Biopac MP100 amplifiers were used. The amplification factor for the EMG was 5000 and the signals were filtered between 0.05-35 Hz. The microphone channel has a gain factor of 200 and DC-300 Hz bandwidth.

Another 2 channel amplifier is used to collect the EOG signals. This amplifer has gain factor of 4000 and 0.4-60Hz passband. For real time capabilities, these amplified signals are fed to a National Instruments DAQPad 6052e analog-digital converter card that uses the IEEE 1394 port.

Disposable ECG electrodes were used for both EOG and EMG recordings. The sounds were captured using the Biopac BSL contact microphone.

BioSignal Processing

We tested various parameter extraction techniques in search of those which could give us the most meaningful results.

We focused mostly on EEG signal processing as it is the richest and most complex bio-signal. The untrained musician normally has less conscious control over brain biosignals as opposed to other biosignals and therefore sophisticated signal processing was reserved for the EEG which needed more processing to produce useful results. The data acquisition program samples blocks of EMG or EOG data in 100 ms frames. Software then calculates the energy for the EOG and EMG channels, and sends this information to the related instruments. The heart sound itself is sent directly to the instruments to provide a rhythmic motif.

Two kinds of EEG analysis are done. The first one attempts to determine the user's intent based on techniques recently developed in the BCI community [START_REF] Wolpaw | Braincomputer interfaces for communication and control[END_REF]. A second approach looks at the origin of the signal and at the activation of different brain areas. The performer has

ENACTIVE05

From Biological Signals to Music less control over results in this case. There are more details on both of these EEG analysis methods at the end of this section.

Detection of Musical Intent

To detect different brain states we measured spatial distribution and temporal rhythms present. Three main rhythms are of interest:

1. Alpha rhythm: usually between 8-12 Hz, this rhythm describes the state of awareness. If we calculate the energy of the signal using the occipital electrodes, we can evaluate the state of awareness of the musician. When he closes his eyes and relaxes the signal increases. When the eyes are open the signal is low.

2. Mu rhythm: This rhythm also ranges from 8 to 12 Hz but can vary from one person to another, sometimes between 12-16 Hz. The mu rhythm corresponds to motor tasks like moving the hands or legs, arms, etc. We use this rhythm to distinguish movements of the left or right hands.

3. Beta rhythm: Comprised of energy between 18-26 Hz. Beta is linked to motor tasks and higher cognitive functions.

The wavelet transform is a technique of timefrequency analysis prefectly suited for task detection. Individual tasks can be detected by looking at specific frequency bands on specific electrodes. This operation, implemented using sub-band filters, provides us with a filter bank tuned to the frequency ranges of interest. We tested our algorithm on two subjects with different kinds of wavelets: Meyer wavelet, 9-7 filters, bi-orthogonal spline wavelet, Symlet 8 and Daubechy 6 wavelets. We finally chose the symlet 8 which gave better overall results.

At the beginning we focused on eye blink detection and α band power detection because both are easily controllable by the musician. We then wanted to try more complex tasks such as those used in the BCI community. These are movements and imaginations of movements, such as hand, foot or tongue movements, 3D spatial imagination or mathematical calculation. The main problem is that each BCI user must be trained to improve his control over the task signal. Therefore we decided to use only right and left hand movements first and not the more complex tasks which would have been harder to detect. Two other techniques were also used: Asymmetry Ratio and Spatial Decomposition.

Eye blinking and α band

Eye blinking is detected on Fp1 and Fp2 electrodes in the 1-8Hz frequency range by looking at increase of the band power. We process the signals from electrodes O1 and O2 -occipital electrodes-to exctract the power of the alpha band.

Asymmetry Ratio

Consider we want to distinguish left from right hand movements. It is known that motor tasks activate the cortex area. Since the brain is divided in two hemispheres that control the two sides of the body it is possible to recognise when a person moves on the left or right side. Let C3 and C4 be the two electrodes positioned on the cortex, the asymmetry ratio can be written as:

Γ F B = P C3,F B -P C4,F B P C3,F B + P C4,F B (1)
where P Cx,F B is the power in a specified frequency band (FB), i.e. the mu frequency band. This ratio has values between 1 and -1. Thus it is positive when the power in the left hemisphere (right hand movements) is higher than the one in the right hemisphere (left hand movements) and vice-versa.

The asymmetry ratio gives good results but is not very flexible and cannot be used to distinguish more than two tasks. This is why it is necessary to search for more sophisticated methods which can process more than just two electrodes simultaneously.

Spatial Decomposition

Two spatial methods have proven to be accurate: The Common Spatial Patterns (CSP) and the Common Spatial Subspace Decomposition (CSSD) [26, ?]. We will shortly describe here the second one (CSSD): This method is based on the decomposition of the covariance matrix grouping two or more different tasks. It is important to highlight the fact that this method needs a learning phase where the user executes two tasks.

The first step is to compute the autocovariance matrix for each task. Given one signal X of dimension N × T for N electrodes and T samples, we decompose X in X A and X B , A and B. By using two different tasks, we can obtain the autocovariance matrix for each task:

R A = X A X T B and R B = X B X T B ( 2
)
We now extract the eigenvectors and eigenvalues from the R matrix that is the sum of R A and R B :

R = R A + R B = U 0 λU T 0 (3)

ENACTIVE05

From Biological Signals to Music -5-

We can now calculate the spatial factors matrix W and the whitening matrix P :

P = λ -1/2 U T 0 and W = U 0 λ 1/2 (4)
If S A = P R A P T and S B = P R B P T , these matrices can be factorised:

S A = U A Σ A U T A S B = U B Σ B U T B ( 5
)
Matrix U A et U B are equals and the sum of their eigenvalue is equal to 1, Σ A + Σ B = I. Σ A et Σ B can be written thus:

Σ A = diag[ 1...1 ma σ 1 ...σ mc mc 0...0 mb ] (6)
Σ B = diag[ 0...0 ma δ 1 ...δ mc mc 1...1 mb ] (7)
Taking the first m a eigenvector from U , we obtain U a and we can now compute the spatial filters and the spatial factors:

SP a = W U a (8)
SF a = U T a P (9)
We proceed identically for the second task, taking care this time with the last mb eigenvectors. Specific signal components of each task can then be extracted easily by multiplying the signal with the corresponding spatial filters and factors. For the task A it gives:

Xa = SP a SF a X (10)
A support vector machine (SVM) with a radial basis function was used as a classifier.

Results

The detection of eye blinking during off-line and realtime analysis was higher than 95%, with a 0.5s time window. For hand movement classification with spatial decomposition, we chose to use a 2s time window. A smaller window significantly decreases the classification accuracy. The CSSD algorithm needs more training data to achieve a good classification rate so we decided to use 200 samples of both right hand and left hand movements, each sample being a 2s time window. Thus, we used an off-line session to train the algorithm. However each time we used the EEG cap for a new session, the electrode locations on the subject's head changed. Performing a training session one time and a test session another time gave poor results so we decided to develop new code in order to do both training and testing in one session. This had to be done quite quickly to ensure the user's comfort.

We achieved an average of 90% good classifications during off-line analysis, and 75% good classifications during real-time recording. Real-time recording accuracy was a bit less than expected. (This was probably due to a less-than-ideal environment -with electrical and other noise -which is not conducive to accurate EEG signal capture and analysis.) The asymmetry ratio gave somewhat poorer results.

Spatial Filters

EEG is a measure of electrical activities of the brain as measured from the external skull area. Different brain processes can activate different areas. Discovering which areas are active is difficult as many different source configurations can lead to the same EEG recording. Noise in the data further complicates this problem.

In the following, we present the methods -based on forward and inverse problems -and the hypothesis we propose to solve the problem in real time.

Forward Problem, head model and solution space

If X is a N x1 vector containing the recorded potential with N representing the number of electrodes. S is an M x1 vector of the true source current with M the unknown number of sources. G is the leadfield matrix which links the source location and orientaion to the electrodes location. G depends of the head model. n is the noise. We can write

X = G S + n (11)
X and S can be extended to more than one dimension to take time into account. S can either represent few dipoles (dipole model) with M ≤ N or represent the full head (image model -one dipole per voxel) with M ≫ N . In the following we will use the latter model.

The forward problem is to try and find the potentials X on the scalp surface knowing the active brain sources S. This approach is far simpler than the inverse approach and its solution is the basis of all Inverse problem solutions.

The leadfield G is based on the Maxwell equations. A finite element model based on the true subject head can be use as lead field but we prefer to use a 4-spheres approximation of the head. It is not subject dependent and less computationally expensive. A simple method consists of seeing the multi-shell model as a composition of single-shells -much as Fourier uses functions as sums

ENACTIVE05

From Biological Signals to Music of sinusoid [START_REF] Berg | A fast method for forward computation of multiple-shell spherical head models[END_REF]. The potential v measured at electrode position r from a dipole q in position r q is v(r, r q , q) ≈ v 1 (r, µ 1 r q , λ 1 q) + v 1 (r, µ 2 r q , λ 2 q) + v 1 (r, µ 3 r q , λ 3 q) (12) λ i and µ i are called Berg's parameters [START_REF] Berg | A fast method for forward computation of multiple-shell spherical head models[END_REF]. They have been empirically computed to approximate three and four-shell head model solution.

When we are looking for the location and orientation of the source, a better approach consists of separating the non-linear search for the location and the linear one for the orientation. The EEG scalar potential can then be seen as a product v(r) = k t (r, r q )q with k(r, r q ) a 3x1 vector. Therefore each single shell potential can be computed as [START_REF] Mosher | EEG and MEG: Forward solutions for inverse methods[END_REF] v 1 (r) = ((c 1c 2 (r.r q ))r q + c 2 r q 2 r).q

with c 1 ≡ 1 4πσ r q 2 2 d.r q d 3 + 1 d - 1 r (13)
c 2 ≡ 1 4πσ r q 2 2 d 3 + d + r r F (r, r q ) (14)
F (r, r q ) = d ( r d + r 2 -(r q .r)) (15)
The brain source space is limited to 361 dipoles located on an half-sphere just below the cortex in a perpendicular orientation to the cortex. This is done because the activity we are looking at is concentrated on the cortex, the activity recorded by the EEG is mainly cortical activity and the limitation of the source space considerably reduces the computation time.

Inverse Problem

The inverse problem can be formulated as a Bayesian inference problem [START_REF] Baillet | Electromagnetic brain mapping[END_REF]

p(S|X) = p(X|S)p(S) p(X) (16)
where p(x) stands for probability distribution of x. We thus look for the sources with the maximum probability. Since p(X) is independent of S it can be considered as an normalizing constant and can be omitted. p(S) is the prior probability distribution of S and represents the prior knowledge we have about the data. This is modified by the data through the posterior probability distribution p(X|S). This probability is linked to the noise. We assume the noise is gaussian, with zero mean and covariance matrix

C n ln p(X|S) = (X -GS) t C -1 n (X -GS) ( 17
)
where t stands for transpose. If the noise is white, we can rewrite equation [START_REF] Knapp | A Bioelectric Controller for Computer Music Applications[END_REF] as

ln p(X|S) = X -GS 2 (18)
In case of zero mean gaussian prior p(S) with variance C S , the problem becomes argmax(ln p(S|X)) = argmax(ln p(X|S)

+ ln p(S)) = argmax((X -GS) t C -1 n (X -GS) + λS t C S S
where the parameter λ gives the influence of the prior information. And the solution is

Ŝ = G t C -1 n (G t C -1 n G + λC -1 S ) -1 X (19)
For a full review of a method to solve the Inverse Problem see [29, ?, 30].

Methods based on different priors were tested. Priors ranged from the simplest -no prior informationto classical prior such as the laplacian and to a specific covariance matrix. The well-know LORETA approach [START_REF] Pascual-Marqui | Review of methods for solving the EEG inverse problem[END_REF] showed the best results on our test set. The LORETA [START_REF] Pascual-Marqui | Review of methods for solving the EEG inverse problem[END_REF] looks for a maximally smooth solution. Therefore a laplacian is used as a prior. In [START_REF] Pfurtscheller | Current trends in Graz Brain-Computer Interface (BCI) research[END_REF] C s is a laplacian on the solution space and C n is the identity matrix.

To enable real time computation, leadfield and prior matrices in [START_REF] Pfurtscheller | Current trends in Graz Brain-Computer Interface (BCI) research[END_REF] are pre-computed. Then we only multiply the pre-computed matrix with the acquired signal. Computation time is less than 0.01s on a typical personal computer.

Results and Application

In the present case of a BCMI, the result can be used for three potential applications: the visualisation process, a pre-filtering step and a processing step.

The current of the 361 dipoles derived using the inverse method is directly used in the visualisation process. The current on every point of the half-sphere is interpolated from the dipole currents. The result is projected on a screen.

6 Sound Synthesis

Introduction

At the end of the workshop, a performance of music was presented with two bio-musicians and various equipment and technicians on stage orchestrating a live bio-music performance before a large audience. The first instrument was a midi instrument based on additive

ENACTIVE05

From Biological Signals to Music synthesis and controlled by the musician's electroencephalogram along with an infrared sensor. The second instrument, driven by electromyograms of the second bio-musician, processed recorded accordion samples using granulation and filtering effects. Furthermore, biological signals managed the spatialized diffusion over eight loudspeakers of the sound produced by two musicians. We here present details of each of these instruments.

Sound Synthesis

Artificial synthesis of sound is the creation, using electronic and/or computational means, of complex waveforms, which, when passed through a sound reproduction system can either mimic a real musical instrument or represent the virtual projection of an imagined musical instrument. This technique was first developed using digital computers in the late 1950s and early 1960s by Max Matthews at Bell Labs. It does have antecedents, however, in the musique concrète experiments of Pierre Schaeffer and Pierre Henry and in the TelHarmonium of Thaddeus Cahill amongst others. The theory and techniques of sound synthesis are now widely developed and are treated in depth in many well-known sources. The chosen software environment, Max/MSP, makes available a wide palette of sound synthesis techniques including: additive, subtractive, frequency modulation, granular etc. With the addition of 3rd party code libraries (externals) Max/MSP can also be used for more sophisticated techniques such as physical modelling synthesis.

Mapping

The commonly used term mapping refers, in the instance of virtual musical instruments, to mathematical transformations which are applied to real-time data received from controllers or sensors so that they may be used as effective control for sound synthesis parameters. This mapping can consist of a number of different mathematical and statistical techniques. To effectively implement a mapping strategy, one must well understand both the ranges and behaviours of the controllers or sensors and the nature of the data stream produced along with the synthesis parameters which are to be controlled.

A useful way of thinking about mapping is to consider its origin in the art of making cartographic maps of the natural world. Mapping thus is forming a flat, virtual representation of a curved, spherical real world which enables that real world to be effectively navigated. Implicit in this is the process of transformation or projection which is necessary to form the virtual representation.

Thus, to effectively perform a musically satisfying mapping, we must understand well the nature of our data sources (sensors and controllers) and the nature of the sounds and music we want to produce (including intrinsic properties and techniques of sound synthesis, sampling, filtering and DSP)

This poses significant problems in the case of biologically controlled instruments in that it is not possible to have an unambiguous interpretation of the meanings of biological signals whether direct or derived. There is some current research in cognitive neuroscience which may indicate directions for understanding and interpreting the musical significance of encephalographic signals, but this is just beginning.

A simple example of a mapping is the alpha rhythm spectral energy to musical intensity. It is well known that strong energy in the frequency band (8-12 Hz) indicates a state of unfocused relaxation without visual attention in the subject. This has commonly been used as a primary controller in EEG-based musical instruments -such as in Alvin Lucier's "Music for Solo Performer" -where strong Alpha EEG directly translate into increased sound intensity and temporal density. If this is not the desired effect then consideration has to be given to how to transform the given data to produce the desired sound or music.

Instrument 1 : an interface between brain and sound

For this instrument, we used the following controls

• right or left body movement (Mu bandwidth)

• eyes open or closed (Alpha bandwidth)

• average brain activity (Alpha bandwidth)

This MAX/MSP patch is based upon these parameters. The sound synthesis is done with a plug-in from Absynth which is software controlled via the MIDI protocol. This patch creates MIDI events which control this synthesis. This synthesis is in particular composed of three oscillators, three Low Frequency Oscillators, and three notch filters. There are two kinds of note trigger:

• a cycle of seven notes

• a trigger of single note Pitch is not controlled continuously.

Regarding the first kind of note trigger, the cycle of notes begins when the artist opens his eyes for the first

ENACTIVE05

From Biological Signals to Music time. Right or left body movements can control the direction of cycle rotation and the panning of the result. The resultant succession of notes is subjected of two randomised variations of the note durations and the delta time between each note.

In the second note trigger, alpha bandwidth is converted to a number between 0 and 3, which is then divided into three parts:

• 0 to 1 : this part is divided into five sections, one note is attributed to each section and the time proprieties are given by the dynamics of the alpha variations

• 1 to 2 : represents the variation of the Low Frequency Oscillator (LFO) frequency

• over 2 : the sound is stopped

The EEG analysis for these controls happens over time.

To have an instantaneous controller, an infrared sensor controller was added. Based on the distance between his hand and the sensor, the artist can control:

• the rotation speed of the cycle, using the right hand

• the frequency of the two other LFO, using the left hand

The performer decides the harmony before playing, which, in the case live performance, has proved to be a good solution.

Results

The aim of this work was to create an instrument controlled by electroencephalogram signals. Musical relationships are usually linked with gestures, yet, here no physical interaction is present. Further, the possibility for complex interactions between a traditional musical instrument, like a guitar, and the performer, retains a great power. To be interesting from an artistic point of view, a musical instrument must provide a large expressive palette to the artist.

The relationship between the artist and the music acts in two directions: the musician interacts with sound production by means of his EEGs but the produced sound also interacts via a feedback influence on the mental state of the musician. Future work could turn toward the biofeedback potential for influencing sound.

Instrument 2 : Real-time granulation and filtering on accordion samples

In the second instrument, sound synthesis is based on the real-time granulation and filtering of recorded accordion samples. During the demonstration, the musician starts his performance by playing and recording a few seconds of accordion which he will then process in realtime. Sound processing was controlled by means of data extracted from electromyograms (EMGs) in measuring muscle contractions in both arms of the musician.

Granulation

Granulation techniques split an original sound into very small acoustic events called grains and reproduce them in high densities of several hundred or thousand grains per second. A lot of transformations on the original sound are made possible with this technique and a large range of very strange timbres can be obtained. In our instrument, three granulation parameters were driven by the performer : the grain size, the pitch shifting, and the pitch shifting variation (that controls the random variations of pitch).

In terms of mapping, the performer selected the synthesis parameter he wanted to vary thanks to an additional midi foot controller and this parameter was then modulated according to the contraction of his arm muscles, measured as electromyograms. The contraction of left arm muscles allowed choosing either to increase or decrease the selected parameter, whereas the variation of the parameters were directly linked to right arm muscle tension.

Flanging

In addition to granulation, a flanging effect was implemented in our instrument. Flanging is created by mixing a signal with a slightly delayed copy of itself, where the length of the delay, less than 10 ms, is constantly changing. The performer had the ability to modulate several flanging parameters (depth, feedback gain) separately via his arm muscle contractions much as was done to control the granulation parameters.

Balance dry/wet sounds

During the performance, the musician had the possibility to control the intensities of dry and wet sounds with the contraction of his left and right arm respectively. This control gave the musician the ability to cross-fade original sound with the processed one by means of very expressive gestures.

Results

Very interesting sonic textures, near or far from original accordion sound, have been created by this instrument. Granulation gave the sensation of clouds of sound,

ENACTIVE05

From Biological Signals to Music whereas, very strange sounds, reinforced by spatialisation effects on eight loudspeakers, were obtained using certain parameter configurations of the flange effect. As with any traditional musical instrument, the first thing going forward will be to practice the instrument in order to properly learn it. These training sessions will aim to improve the mapping between sound parameters and gestures. Data gloves and EMGs measuring muscles contraction in other body parts (legs, shoulders, neck), along with new kinds of sound processing could bring interesting results.

Spatialization and Localization

The human perception of the location of sound sources within a given sound environment are due to a complex series of cues which have evolved according to the physical behaviour of sound in real spaces. These cues can include: intensity, including right-left balance, relative phase, early reflections and reverberation, Doppler shift, timbral shift and many other factors which are actively studied by researchers in auditory perception.

The term 'spatialisation' refers to the creation of a virtual sound space using electronic techniques (analogue or digital) and sound reproduction equipment (amplifiers and speakers) to either mimic the sound-spatial characteristics of some real space or present a virtual representation of an imaginary space reproduced via electronic means. The term 'localisation' refers to the placement of a given sound source within a given spatialised virtual sound environment using the techniques of spatialisation. Given the greatly increased real-time computational power available in todays personal computers, it is now possible to perform complex and subtle spatialisation and localisation of sounds using multiple simultaneous channels of sound reproduction (four or more).

The implementation of a system for the the localisation of individual sound sources and overall spatialisation in this project was based around an 8-channel sound reproduction system. Identical loudspeakers were placed equidistant about a centre point to form a circular pattern around the listening space. All speakers were elevated approximately to ear level.

Sounds were virtually placed within the azimuth of this 360 degree circular sound space by the use of mixing software which approximates an equal-power panning algorithm. The amplitude of each virtual sound source can be individually controlled. Artificial reverb can be added to each sound source individually in order to simulate auditory distance. Finally, each individual sound source can be placed at any azimuth and panned around the circle in any direction and at any speed. Future implementations of this software will take into account some more subtle aspects of auditory localisation including timral adjustments and Doppler effects.

Visualization

In a traditional concert setting, the visual aspect of watching the musicians play is an important part of the overall experience. With an EEG driven musical instrument, the musician must sit very still and immobile. By adding a visual dimension to this, we can enhance the spectator's appreciation of the music.

We studied different ways of visualising the EEG and finally chose to present the signal projected on the brain cortex as explained in section 5.2. While the musician is playing, EEG data are processed once per second using the inverse solution approach and then averaged. A half sphere with the interpolation of the 361 solution is projected on the screen.

Conclusion

During this workshop, two musical instruments based on biological signals were developed. One was based on EEG and the other on EMG. We chose to make an intelligent musical instrument rather than to just sonify the data. The same biosignals were also used to spatialise the sound and visualise the biodata.

We have built an architecture for communication between data acquisition, signal processing and sound synthesis nodes. Our software is based on Matlab and Max/MSP and thus new signal processing and sound synthesis algorithms can be easily implemented.

The present paper reflects the work of seven people over four weeks. This work did not stop at the end of the workshop -it is ongoing and there is much still to be done. The signal processing and musical instrument designs can be improved. The musicians need to achieve better control of their instruments using biological signals. Mapping algorithms need to be improved and the software implementations must be made more robust. Going forward, the members of this team, together and individually, are committed to pursuing the dream of a Music which springs eternal from human biological signals.

  he used various biological signals including brain (EEG) and cardiac (ECG) signals as control sources for electronic synthesisers. Over the next few years, Teitelbaum continued to use EEG and other Proceedings of ENACTIVE05 2nd International Conference on Enactive Interfaces Genoa, Italy, November 17th-18th, 2005 biological signals in his compositions and experiments as triggers for the nascent Moog electronic synthesiser.


Jean-Julien Filatriau

Rémy Lehembre

Quentin Noirhomme
email: noirhomme@tele.ucl.ac.bebuarslan@gsu.edu.tr

Cédric Simon

Burak Arslan

Andrew Brouse
email: andrew.brouse@plymouth.ac.uk

Julien Castet
email: julien.castet@free.fr

Using physiological signals for sound creation

Recent advances in new technologies offer a large range of innovative instruments to design and process sounds. Willing to explore new ways for music creation, specialists from the fields of brain-computer interfaces and sound synthesis worked together during the eNTERFACE05 workshop (Mons, Belgium). The aim of their work was to design an architecture of real-time sound synthesis driven by physiological signals. The following description links natural human signals to sound synthesis algorithms, thus offering rarely used pathways for music exploration. This architecture was tested during a "bio-concert" given at the end of the workshop where two musicians were using their EEG and EMG to perform a musical creation.

Introduction

Advances in computer science and specifically in Human-Computer Interaction (HCI) have now enabled musicians to use sensor-based computer instruments to perform music [START_REF] Tanaka | Musical perfomance practice on sensor-based instruments[END_REF]. Musicians can now use positional, cardiac, muscle and other sensor data to control sound [START_REF] Nagashima | Bio-Sensing systems and bio-feedback systems for interactive media arts[END_REF][START_REF] Knapp | Multimodal Interaction in Music Using the Electromyogram and Relative Position Sensing[END_REF]. Simultaneously, advances in Brain-Computer Interface (BCI) research have shown that cerebral patterns can be used as a source of control [START_REF] Wolpaw | Brain-computer interfaces for communication and control[END_REF]. Indeed, cerebral and conventional sensors can be used together [START_REF] Brouse | Petit guide de la musique des ondes cérébrales[END_REF][START_REF] Miranda | Toward direct Brain-Computer Musical Interfaces[END_REF] with the object of producing a 'body-music' controlled according to the musician?s imagination and proprioception. Research is already being done toward integrating BCI and sound synthesis with two different approaches. The first approach tries to use the sound as a way to better understand the brain activity by mapping the data issued from physiological analysis directly to sound synthesis parameters [START_REF] Berger | Singing the mind listening[END_REF] [8] [START_REF] Dribus | The other ear : a musical sonification of EEG data[END_REF]. This sonification process can be viewed as a translation of biological signals into sound. The second approach aims to build a musical interface where inference based on complex feature extraction enables the musician to intentionally control sound production. This is easy with electromyograms (EMG) or electrooculogram (EOG) but very difficult with electroencephalogram (EEG). In the following, we first present the architecture we developed to acquire, process and play music based on biological signals. Next we go into more detail on signal acquisition part followed by an in-depth discussion of appropriate signal processing techniques. Details of the sound synthesis implementation are then discussed along with the instruments we built. Finally, we conclude and present some future directions.

System architecture overview

Our intention was to build a robust, reusable framework for biological signals capture and processing geared towards musical applications. To maintain flexibility and ensure real-time requirement, signal acquisition, processing and sound synthesis modules were performed on different physical machines linked via Ethernet. Data were acquired via custom hardware linked to a host computer running a Matlab/Simulink real-time blockset [10]. We implemented our signal processing code as a Simulink blockset using Level-2 M file S-functions with tuneable method parameters. This allowed us to dynamically adapt to the incoming signals. For the sound processing methods, we used Max-MSP programming environment dedicated to audio and multimedia real-time processing [START_REF] Max | [END_REF]. Data transmission between machines was implemented using UDP/IP protocol over Ethernet. We chose this for best realtime performance. Messages were encoded thanks to the Open-SoundControl protocol (OSC) [12], which sits on top of UDP. OSC was conceived as a protocol for the real-time control of computer music synthesizers over modern heterogeneous networks. For our project, we used OSC to transfer data from Matlab (running on a PC with either Linux or Windows OS) to Macintosh computers running Max/MSP. Fig. 1 outlines main software and data exchange architecture.

Data Acquisition

Two types of biological signals were considered: electroencephalograms (EEG) and electromyograms (EMG)

Electroencephalograms (EEG)

Electroencephalograms data were recorded at 64 Hz on 19 channels with a DTI cap. Data were filtered between 0.5 and 30 Hz, channels were positioned following the 10-20 international system and Cz was used as reference. The subject sat in a comfortable chair and was asked to concentrate on the different tasks. The recording was done in a normal working place, e.g. a noisy room with people working, speaking and with music. The environment was not free from electrical noise as there are many computers, speakers, screen, microphones and lights around.

Electromyograms (EMG)

To record the electromyograms, three amplifiers of Biopac MP100 system were used. The amplification factor for the EMG was 5000 and the signals were filtered between 0.05-35 Hz. The microphone channel had 200 gain and DC-300Hz bandwidth. For real time capabilities, these amplified signals are fed to the National Instruments DAQPad 6052e analogdigital converter card that uses the IEEE 1394 port. Thus, the data can be acquired, processed and transferred to the musical instruments using Matlab environment and the Data Acquisition toolbox.

BioSignal Processing

We tested various parameter extraction in search of those which could give us the most meaningful results. We focused mostly on EEG signal processing as it is the richest and most complex bio-signal. The untrained musician normally has less conscious control over brain biosignals as opposed to other biosignals and therefore sophisticated signal processing was reserved for the EEG which needed more processing to produce useful results. The data acquisition program samples blocks of EMG data, measuring arm muscles contraction, in 100 ms frames. Software then calculates the energy for EMG channels, and sends this information to the related instruments. Two kinds of EEG analysis are done (Fig. 2). The first one attempts to determine the user?s intent based on techniques recently developed in the BCI community [START_REF] Wolpaw | Brain-computer interfaces for communication and control[END_REF]. A second approach looks at the origin of the signal and at the activation of different brain areas. The performer has less control over results in this case. In the next sections, we present more details on both of these EEG analysis approaches.

Detection of musician's intent

To detect different brain states we used the spatialisation of the activity and the different rhythms present in this activity. Indeed, each part of the brain has a different function and each human being presents specific rhythms at different frequencies.

For example, three main rhythms are of great interest:

1. Alpha rhythm: usually between 8-12 Hz, this rhythm describes the state of awareness. If we calculate the energy of the signal using the occipital electrodes, we can evaluate the awarness state of the musician. When he closes his eyes and relaxes the signal increases. When the eyes are open the signal is low.

Mu rhythm:

This rhythm is also reported to range from 8 to 12 Hz but this band can vary from one person to another, sometimes between 12-16 Hz. The mu rhythm corresponds to motor tasks like moving the hands or legs, arms, etc. We use this rhythm to distinguish left hand movements from right hand movements.

3. Beta rhythm: Ranging between 18-26 Hz, the characteristics of this rhythm are yet to be fully understood but it is believed that it is also linked to motor tasks and higher cognitive function.

The wavelet transform [START_REF] Mallat | A wavelet tour of signal processing[END_REF] is a technique of time-frequency analysis prefectly suited for the task detection. Each task can be detected by looking at specific bandwidth on specific electrodes.

This operation, implemented with sub-band filters, provides us with a filter bank tuned to the frequency ranges of interest. We tested our algorithm on two subjects with different kinds of wavelets: Meyer wavelet, 9-7 filters, bi-orthogonal spline wavelet, Symlet 8 and Daubechy 6 wavelets. We finally chose the symlet 8 which gave better overall results. Once the desired rhythms are obtained, different forms of analysis are possible.

At the beginning we focused on eye blink detection and α band power detection because both are easily controllable by the musician. We then wanted to try more complex tasks such as those used in the BCI community. These are movements and imaginations of movements, such as hand, foot or tongue movements, 3D spatial imagination or mathematical calculation. The main problem is that each BCI user needs a lot of training to improve his control of the task signal. Therefore we decided to use only right and left hand movements first and not the more complex tasks which would have been harder to detect. Since more tasks also means more difficult detection, these are the only tasks used in this project. Two different techniques were used: Asymmetry ratio and spatial decomposition.

Eye blinking and α band

Eye blinking is detected on Fp1 and Fp2 electrodes in the 1-8Hz frequency range by looking at increase of the band power. We process the signals from electrodes O1 and O2 -occipital electrodes-to extract the power of the alpha band.

Asymmetry ratio

Consider we want to distinguish left from right hand movements. It is known that motor tasks activate the cortex area. Since the brain is divided in two hemispheres that control the two sides of the body it is possible to recognize when a person moves on the left or right side. Let C3 and C4 be the two electrodes positioned on the cortex, the asymmetry ratio can be written as:

ΓFB = PC3,F B -PC4,F B PC3,F B + PC4,F B ( 1
)
where PCx,F B is the power in a specified frequency band (FB), i.e. the mu frequency band. This ratio has values between 1 and -1. Thus it is positive when the power in the left hemisphere (right hand movements) is higher than the one in the right hemisphere (left hand movements)and vice-versa. The asymmetry ratio gives good results but is not very flexible and cannot be used to distinguish more than two tasks. This is why it is necessary to search for more sophisticated methods which can process more than just two electrodes as the asymmetry ratio does. Frames of EEG data are processed simultaneously with the different techniques we used. On one hand a wavelet decomposition is done and the frequency bands are sent to different sub-algorithms in order to obtain different parameters. On the other hand, the data is spatially filtered in order to obtain the EEG generators with the inverse solution. This allows us to visualize the brain activity and send parameters representing active regions.

Spatial decomposition

Two spatial methods have proven to be accurate: The Common Spatial Patterns (CSP) and the Common Spatial Subspace Decomposition (CSSD) [START_REF] Wang | Common spatial subspace decomposition applied to analysis of brain responses under multiple task conditions: a simulation study[END_REF][START_REF] Cheng | Mu rhythm-based cursor control: an offlipp[END_REF]. We will shortly describe here the second one (CSSD): This method is based on the decomposition of the covariance matrix grouping two or more different tasks.

Only the simple case of two tasks will be discussed here. It is important to highlight the fact that this method needs a learning phase where the user executes the two tasks. The first step consists in computing the autocovariance matrix for each tasks. Let's take one signal X of dimension N × T for N electrodes and T samples. Decomposing X in XA and XB, A and B being two different tasks, we can obtain the autocovariance matrix for each task:

RA = XAX T B and RB = XB X T B ( 2
)
We now extract the eigenvectors and eigenvalues from the R matrix that is the sum of RA and RB:

R = RA + RB = U0λU T 0 (3)
We can now calculate the spatial factors matrix W and the whitening matrix P :

P = λ -1/2 U T 0 and W = U0λ 1/2 (4)
If SA = P RAP T and SB = P RBP T , these matrices can be factorised:

SA = UAΣAU T A SB = UBΣBU T B ( 5
)
Matrix UA et UB are equals and the sum of their eigenvalue is equal to 1, ΣA + ΣB = I. ΣA et ΣB can thus be written:

ΣA = diag[ 1...1 |{z} ma σ1...σm c | {z } mc 0...0 |{z} mb ] (6)
Taking the first ma eigenvector from U , we obtain Ua and we can now compute the spatial filters F and the spatial factors G:

Fa = W Ua (8) Ga = U T a P (9)
We proceed identically for the second task, but taking this time the last m b eigenvectors. Specific signal components of each task can then be extracted easily by multiplying the signal with the corresponding spatial filters and factors. For the task A it gives:

Xa = FaGaX (10)
A support vector machine (SVM) with a radial basis function was used as a classifier.

Results

The detection of eye blinking during off-line and realtime analysis was higher than 95%, with a 0.5s time window. For hand movement classification with spatial decomposition, we chose to use a 2s time window. A smaller window significantly decreases the classification accuracy. The algorithm CSSD needs more training data to achieve a good classification rate so we decided to use 200 samples of both right hand and left hand movements, each sample being a 2s time window. Thus, we used an off-line session to train the algorithm. However each time we used the EEG cap for a new session, the electrode locations on the subject's head changed. Performing a training session one time and a test session another time gave poor results so we decided to develop new code in order to do both training and testing in one session. This had to be done quite quickly to ensure the user's comfort.

We achieved an average of 90% good classifications during off-line analysis, and 75% good classifications during real-time recording. Real-time recording accuracy was a bit less than expected. (This was probably due to a less-than-ideal environment -with electrical and other noises -which is not conducive to accurate EEG signal capture and analysis.) The asymmetry ratio gave somewhat poorer results.

Spatial Filters

EEG is a measure of electrical activities of the brain as measured on the external skull area. Different brain processes can activate different areas. Thus, knowing which areas are active can give us a helpful clue on the cerebral processes going on. Unfortunately, discovering which areas are active is difficult as many source configurations can lead to the same EEG recording. Noise in the data further complicates the problem. The ill-posedness of the problem leads to many different methods based on different hypotheses to get a unique solution. In the following, we present the methods -based on forward and inverse problems -and the hypothesis we propose to solve the problem in real time.

Forward Problem, head model and solution space

Let X be a N x1 vector containing the recorded potential with N representing the number of electrodes. S is a M x1 vector of the true source current with M the unknown number of sources. G is the leadfield matrix which links the source location and orientation to the electrodes location. G depends of the head model. n is the noise. We can write

X = G S + n ( 11
)
X and S can be extended to more than one dimension to take time into account. S can either represent few dipoles (dipole model) with M ≤ N or represent the full head (image model -one dipole per voxel) with M N . In the following we will use the latter model. The forward problem consists in calculating the potentials X on the scalp surface knowing the active brain sources S. This approach is far simpler than the inverse approach and its solution is the basis of all inverse problem solutions. The leadfield G is based on the Maxwell equations. A finite element model based on the true subject head can be used as the lead field but we prefer to use a 4-spheres approximation of the head. It is not subject dependent and less computationally expensive. A simple method consists of viewing the multi-shell model as a composition of single-shells -much as Fourier uses functions as sums of sinusoid [START_REF] Berg | A fast method for forward computation of multiple-shell spherical head models[END_REF]. The potential v measured at electrode position r from a dipole q in position rq is v(r, rq, q) ≈ v 1 (r, µ1rq, λ1q) + v 1 (r, µ2rq, λ2q) + v 1 (r, µ3rq, λ3q) (12) λi and µi are called Berg's parameters [START_REF] Berg | A fast method for forward computation of multiple-shell spherical head models[END_REF]. They have been empirically computed to approximate three and four-shell head model solution.

When we are looking for the location and orientation of the source, a better approach consists of separating the non-linear search for the location and the linear one for the orientation. The EEG scalar potential can then be seen as a product v(r) = k t (r, rq)q with k(r, rq) a 3x1 vector. Therefore each single shell potential can be computed as [START_REF] Mosher | EEG and MEG: Forward solutions for inverse methods[END_REF] v 1 (r) = ((c1 -c2(r.rq))rq + c2 rq 2 r).q

with c1 ≡ 1 4πσ rq 2 " 2 d.rq d 3 + 1 d - 1 r « (13) c2 ≡ 1 4πσ rq 2 " 2 d 3 + d + r r F (r, rq) « ( 14
)
F (r, rq) = d ( r d + r 2 -(rq.r)) (15)
The brain source space is limited to 361 dipoles located on a half-sphere just below the cortex in a perpendicular orientation to the cortex. This is done because the activity we are looking at is concentrated on the cortex, the activity recorded by the EEG is mainly cortical activity and the limitation of the source space considerably reduces the computation time.

Inverse Problem

The inverse problem can be formulated as a Bayesian inference problem [START_REF] Baillet | Electromagnetic brain mapping[END_REF] p(S|X) = p(X|S)p(S) p(X) [START_REF] Berg | A fast method for forward computation of multiple-shell spherical head models[END_REF] where p(x) stands for probability distribution of x. We thus look for the sources with the maximum probability. Since p(X) is independent of S it can be considered as a normalizing constant and can be omitted. p(S) is the prior probability distribution of S and represents the prior knowledge we have about the data. This is modified by the data through the posterior probability distribution p(X|S). This probability is linked to the noise. If the noise is gaussian -as everybody assumed -with zero mean and covariance matrix Cn

ln p(X|S) = (X -GS) t C -1 n (X -GS) ( 17
)
where t stands for transpose. If the noise is white, we can rewrite equation [START_REF] Mosher | EEG and MEG: Forward solutions for inverse methods[END_REF] as

ln p(X|S) = X -GS 2 (18)
In case of zero mean gaussian prior p(S) with variance CS, the problem becomes

argmax(ln p(S|X)) = argmax(ln p(X|S) + ln p(S))

= argmax((X -GS) t C -1 n (X -GS) + λS t CSS
where the parameter λ gives the influence of the prior information. And the solution is

Ŝ = G t C -1 n (G t C -1 n G + λC -1 S ) -1 X (19)
For a full review of methods for solving the Inverse Problem see [START_REF] Baillet | Electromagnetic brain mapping[END_REF][START_REF] Michel | EEG source imaging[END_REF][START_REF] Pascual-Marqui | Review of methods for solving the EEG inverse problem[END_REF].

Methods based on different priors were tested. Priors ranged from the simplest -no prior information-to classical prior such as the laplacian and to a specific covariance matrix. The well-known LORETA approach [START_REF] Pascual-Marqui | Review of methods for solving the EEG inverse problem[END_REF] showed the best results on our test set. The LORETA looks for a maximally smooth solution. Therefore a laplacian is used as a prior. In equation [START_REF] Michel | EEG source imaging[END_REF], Cs is a laplacian on the solution space and Cn is the identity matrix.

To enable real time computation, leadfield and prior matrices in equation ( 19) are pre-computed. Then we only multiply the pre-computed matrix with the acquired signal. Computation time is less than 0.01s on a typical personal computer.

Musical processing of biological signals

Sound synthesis is the creation, using electronic and/or computational means, of complex waveforms, which, when passed through a sound reproduction system can either mimic a real musical instrument, or represent the virtual projection of an imagined musical instrument. In literature of digital musical instruments [START_REF] Arfib | Mapping strategies between gesture control parameters and synthesis models parameters using perceptual spaces[END_REF], the term mapping refers to the use of realtime data received from controllers and sensors as control parameters that drive sound synthesis processes. During the eN-TERFACE workshop 05, we worked on developing consistent mapping based on physiological signals in order to create biologically-driven musical instruments. For the musical processing of biological signals we chose to use Max/MSP, a widely used software programming environment optimized for flexible real-time control of music systems.

At the end of the workshop, a musical performance was presented with two bio-musicians and various equipment and technicians on stage orchestrating a live bio-music performance before a large audience. The first instrument was a midi instrument based on additive synthesis and controlled by the musicians electroencephalogram along with an infrared sensor. The second instrument, driven by electromyograms of the second bio-musician, processed recorded accordion samples using granulation and filtering effects. Furthermore, electroencephalograms signals managed the spatialized diffusion over eight loudspeakers of the sound produced by two musicians. During the months following the workshop, some experiments were pursued at Communications and Remote Sensing lab (UCL) and another instrument based on electroencephalograms were developed. We here present details of each of these instruments.

Two interfaces brain/sound

In these two instruments, we used three control parameters: right/left body movement (linked to energy in the Mu bandwidth), open/closed eyes and average brain activity (both linked to energy in the Alpha bandwidth). In order to enhance the spectator?s appreciation of the performance, we developed a quiet basic visual representation of brain activity; for this, we chose to present the signal projected on the brain cortex as explained in section 4.2. While the musician was playing, EEG data were processed once per second using the inverse solution approach and then averaged. A half sphere with the interpolation of the 361 solutions was projected on the screen.

First Instrument

In the first instrument, the sound synthesis is done with a plugin from Absynth [22] which is software controlled via the MIDI protocol. The Max/MSP patch interprets the flow of EEG data to create MIDI events that control the synthesis. The synthesis algorithm is composed of three oscillators, three Low Frequency Oscillators, and three notch filters. Sequences of MIDI notes are triggered by the opening of the eyes and permutated according to the right/left body movement. The succession of notes is subject to randomized variations of the notes duration and the delta time between each note. An additional infrared sensor gives an instantaneous control on the frequencies of the LFO.

Second Instrument

In the second instrument, the synthesis is achieved by Quick Time synthesizer driven in the Max-MSP patch. The patch also used the scansynth object developed by Couturier [START_REF] Arfib | Gestural Strategies for specific filtering processes[END_REF] and other externals provided in the Real Time Composition library [START_REF] Essl | An Interactive Realtime Composition for Computer-Controlled Piano[END_REF] allowing to generate harmonized melodies. Alpha and Beta bandwidth are used to both modulate loudness, rhythm and pitch range of the melody and trigger sound events when thresholds are crossed. The opening of the eyes allows the musician to change the Quick Time instrument whereas the panning between left and right speakers is linked to the right/left body movement by the way of the energy in the Mu bandwidth.

Results

The aim of this work was to create an instrument controlled by electroencephalogram signals. While interaction between music and musician usually rely on gestures, there is no physical interaction here. This implies some lack of control in the synthesis process for the musician, though some parameters can be intentionally controlled, with eyes opening especially. By this way we are here at the boundary between two approaches, the sonification approach, where the sound is just a translation of input data without intentional human control, and the musical instrument approach, that relies on a high level of interaction between the musician and the synthesis process. Furthermore, the relationship between the musician and the music acts in two directions: the musician interacts with sound production by means of his EEGs but the produced sound also interacts via a feedback influence on the mental state of the musician. Future work will turn toward the biofeedback potential for influencing sound.

EMG-controlled granular synthesis

In the second instrument, sound synthesis is based on the realtime granulation and filtering of recorded accordion samples. During the demonstration, the musician starts his performance by playing and recording a few seconds of accordion, which he will then process in real-time. Sound processing was controlled by means of data extracted from electromyograms (EMG) measuring muscle contractions in both arms of the musician.

Granulation

Granulation techniques split an original sound into very small acoustic events called grains and reproduce them in high densities of several hundred or thousand grains per second [START_REF] Roads | Microsound[END_REF]. In our instrument, three granulation parameters were driven by the performer: the grain size, the pitch shifting, and the pitch shifting variation. In terms of mapping, the performer selected the synthesis parameter he wanted to vary thanks to an additional midi foot controller and this parameter was then modulated according to the contraction of his arm muscles, measured as electromyograms. The contraction of left arm muscles allowed choosing either to increase or decrease the selected parameter, whereas the variation of the parameters were directly linked to right arm muscle tension.

In addition to granulation, a flanging effect was implemented in our instrument. Flanging is created by mixing a sig-nal with a slightly delayed copy of itself, where the length of the delay, less than 10 ms, is constantly changing. The performer had the ability to both modulate flanging parameters and control the balance between dry and wet sounds via his arm muscle contractions.

Results

Peculiar sounds, near or far from original accordion timbres, have been created by this instrument. Granulation gave the sensation of clouds of sound, whereas very strange sounds, reinforced by spatialisation effects on eight loudspeakers, were obtained using certain filtering parameters configurations. As with any traditional musical instrument, the first thing going forward will be to practice the instrument in order to properly learn it. These training sessions will aim to improve the mapping between sound parameters and gestures. However, a major improvement to enhance the interaction between musician and instrument would be to add electromyograms measuring muscles contraction in other body parts (legs, shoulders, neck) and map these data to new kinds of sound processing.

Conclusion

This paper presents the result of a collaborative work held during the eNTERFACE 05 Summer Workshop (Mons, Belgium), where specialists in biomedical signal processing and sound synthesis fields shared their knowledge over four weeks. The aim of the project was to develop an interface between these two fields. Thus we created several digital musical instruments driven by electroencephalograms and electromyograms signals and a live performance was presented on stage at the end of the workshop. For this, we built an efficient architecture for real-time communication between data acquisition, biomedical signal processing and sound synthesis modules. This modular architecture will enable to easily pursue such experiments in the future : indeed, signal processing methods and mapping strategies could be improved in order to make the system more robust and to enhance the interaction between the musician and the instruments. Developing an advanced visual feedback that translates brain activity also seems to be a very interesting improvement. These are some of the different pathways that we will explore during the second eNTERFACE workshop in Summer 2006 (Dubrovnik, Croatia).

Figure 1 :

 1 Figure 1: System architecture: measured signals from EEG and EMG recorders are sent to simulink via UDP. Data are processed in real-time with simulink and the parameters are sent to the different instruments.

Figure 2 :

 2 Figure2: EEG processing, from recording (left) to play (right). Frames of EEG data are processed simultaneously with the different techniques we used. On one hand a wavelet decomposition is done and the frequency bands are sent to different sub-algorithms in order to obtain different parameters. On the other hand, the data is spatially filtered in order to obtain the EEG generators with the inverse solution. This allows us to visualize the brain activity and send parameters representing active regions.