Arshia Cont
email: arshia.cont@ircam.fr

ON THE CREATIVE USE OF SCORE FOLLOWING AND ITS IMPACT ON RESEARCH

Score following research is one of the active disciplines of sound and music computing since almost 30 years that have haunted both algorithmic and computational development in realtime music information retrieval, as well as artistic applications in interactive computer music. This paper explores the creative use of such technologies and brings attention to new scientific paradigms that emerge out of their artistic use. We show how scientific and artistic goals of score following systems might differ and how the second, continuously helps re-think the first. We focus mostly on the musical goals of score following technologies which brings us to an underestimated field of research, despite its obviousness in creative applications, which is that of synchronous reactive programming and its realization in Antescofo.

INTRODUCTION

Score following is traditionally the automatic and realtime alignment of audio streams from musician(s) on the stage into a symbolic music score. In its artistic use, it allows realtime coordination and synchronization of live electronic programs with human performers for mixed interactive computer music pieces. In the scientific literature, it is also employed in off-line mode for alignment of audio to symbolic music scores as a front-end for music information retrieval applications.

The score following literature is one of the research disciplines in sound and music computing with clear impacts on both research literature and artistic applications in computer music. The number of published articles on score following algorithms are constantly increasing every year, and since a few years, more composers of interactive computer music are employing such technologies into their compositions. Since the inception of score following paradigms in the 1980s, the two fronts have been evolving together and giving birth to a handful of interactive softwares and concepts for computer music. With the advent of robust score following techniques with explicit musical considerations both for composer and performers and the recent flow of composers employing such systems (such as [START_REF] Cont | Antescofo: Anticipatory synchronization and control of interactive parameters in computer music[END_REF][START_REF] Raphael | Music Plus One: A System for Expressive and Flexible Musical Accompaniment[END_REF]), the interaction between artistic use and scientific paradigms of score following is more than apparent.

This paper explores the creative use of score following and its impact on the research. The artistic cases discussed are limited to mixed electronics and instrumental pieces in the computer music repertoire. Specifically, we draw lines between the scientific and artistic goals of score following in general, and attempt to show how the second, often underestimated in the research literature, gives rise to new scientific paradigms to explore. The scientific paradigm exposed here brings the act of composition, as the authorship of time and interaction, close to synchronous programming paradigms in computer science.

The pieces and concepts explored in this paper are taken from the Antescofo1 [START_REF] Cont | Antescofo: Anticipatory synchronization and control of interactive parameters in computer music[END_REF] repertoire, an anticipatory score follower equipped with a synchronous language for realtime computer music composition and performance. Antescofo is probably the first score following system featuring a coupled recognition system and synchronous language. This feature of Antescofo came rather as a necessity from its artistic use than pure scientific endeavor. We begin the paper by some background on the creative use of score following. We then clarify the scientific and musical goals of score following in section 3. Particularly we draw on specific architectures used in most known score following paradigms within an artistic context, and draw conclusions on specific research paradigms that should be considered within this context in section 4. We proceed in section 5 by defining the architecture in Antescofo that addresses these issues, and define the semantics of our synchronous action language in section 6. We finish by exposing some recent examples employing Antescofo in section 7 and demonstrating the discussed points.

HISTORICAL BACKGROUND

Score following research was introduced in [START_REF] Dannenberg | An on-line algorithm for real-time accompaniment[END_REF][START_REF] Vercoe | The synthetic performer in the context of live performance[END_REF] and initially geared towards automatic accompaniment applications in which the computer would synchronously perform and render the accompaniment section with a live performer undertaking the solo part of a given music score. The technical paradigm of score following has passed various stages ever since, evolving from symbolic string-matching techniques, to pitch detection, and probabilistic models. For a historical overview of score following algorithms we refer the curious reader to [START_REF] Cont | A coupled duration-focused architecture for realtime music to score alignment[END_REF] and instead, focus on its artistic employment hereon.

Artistic uses of score following technologies have been made within two trends: Automatic Accompaniment and mixed instrumental and electronics pieces.

Automatic Accompaniment

The goal of automatic accompaniment systems is first to listen to the live performer and extract position and tempo parameters with regards to its music score, and second to perform the accompanying parts synchronous to this live performance. The accompaniment part can be either symbolic data rendered into audio via some synthesis techniques (such as in early versions in [START_REF] Dannenberg | An on-line algorithm for real-time accompaniment[END_REF][START_REF] Vercoe | The synthetic performer in the context of live performance[END_REF]) or employ realtime phase vocoding techniques on an audio recording in the style of Music-Minus-One. Among systems employing the latter is that of Christopher Raphael [START_REF] Raphael | The informatics philharmonic[END_REF], performing automatic accompaniment on a number of classical music repertoire.

Mixed Instrumental and Electronics Repertoire

The consensus for interaction between a live music performance and composed electronics dates back to early experiments of Bruno Maderna2 , Karlheinz Stockhausen and Mario Davidovsky among other composers in 1950s, through tape and instrumental pieces. Synchronization between the instrumental music and electronics was assured either by using click-tracks or active listening. Despite the new possibilities that electronics had brought into the musical language, production means of electronics had introduced enough burden in the process of composition and performance that this realm remained highly experimental up to the 1980s.

Shortly after the advent of score following technologies, several composers recognized the opportunities that such tools could offer both at the compositional and performative levels of computer music. The most evident application would be naturally in synchronizing a pre-written electronic score to a live performance, extending the applications of score following from automatic accompaniment to a mixed repertoire.

Besides the performative comfort in employing score following technologies, some composers immediately recognized and incorporated the new opportunities that score following would bring in authoring interactive electronic scores coupled with realtime capabilities sound generation and transformations. The possibility of creating interactive music systems attracted new artists and researchers, and created one of the most fruitful periods in computer music. Robert Rowe's two volumes [START_REF] Rowe | Machine Musicianship[END_REF][START_REF]Interactive music systems: machine listening and composing[END_REF] demonstrates how such novel paradigms have affected different practices in computer music. Among composers exposed to these new possibilities, Philippe Manoury was one of the earliest composers who integrated interactive music systems into his compositions and as a compositional process both for authoring and live performance. In particular, Manoury's early formalizations of the paradigm in collaboration with Miller Puckette, led to the birth of the Max programming environment 3 , further developed and integrated by other composers such as Boulez, Lippe, and Settle, and since then widely referred to as the realtime school of composition. The most interesting concept brought by Manoury is that of Virtual Scores [START_REF] Manoury | La note et le son[END_REF] developed hereafter.

Virtual Scores

A virtual score is a musical organization in which we know the nature of the parameters that will be processed but not their exact outcome at runtime since they're expressed as a function of the live performance. A virtual score hence consists of electronic programs with fixed or relative values/outcomes to an outside environment. A realtime electronic process is therefore one that exists in a music score, next to the instrumental transcription, and whose outcome is evaluated during live performance and as a function of the instrumental part's interpretation with all its diversity and richness.

The idea of virtual score is thus to bring in both the performative and compositional aspects of computer music within one compositional framework. A score following technology is then responsible for enabling the communication channels between the computer and the musicians according to a score and by allowing complex musical interactions similar to that of human musicians.

The framework of virtual scores is present and at the core of most interactive programming environments in computer music today. Despite its similarity to a traditional framework of composition, it does not limit its practice to traditional norms of music composition and on the contrary it has integrated non-traditional practices of computer music such as interactive composition [START_REF] Chadabe | Interactive composing: An overview[END_REF], hyperinstrument composition [START_REF] Machover | Hyperinstruments: Musically intelligent and interactive performance and creativity systems[END_REF], composed improvisations [START_REF] Chabot | A workstation in live performance: Composed improvisation[END_REF] and more, as employed in Manoury's early realtime pieces among others [START_REF] Puckette | Score Following in Practice[END_REF].

It is worthy to note that the realtime school was subject to constructive and interesting criticisms and debates at its very inception in a 1999 issue of Contemporary Music Review journal. Of particular interest to our work are that of Risset [START_REF] Risset | Composing in real-time?[END_REF] and Stroppa [START_REF] Stroppa | Live electronics or live music? towards a critique of interaction[END_REF] who underlined the lack of compositional and temporal considerations in existing frameworks at the time. We would like to emphasize that interactive pieces in this sense, are not dissociable from automatic accompaniment paradigms in their architecture. Where automatic accompaniment deals with notes or chords in the accompaniment rendering, virtual scores replace them with processes and realtime electronic programs and transformations.

Score Following in Practice

To motivate further discussions, we attempt to provide the architectural design of a typical realtime mixed piece employing score following technologies. We use the "Introduction" part of the piece Anthèmes II composed by Pierre Boulez for violin and live electronics (1997) as demonstrated in figure 1. This music score shows the instrumen- tal (violin) section in parallel to an approximative notation for the realtime electronics accompanying the system. Each system corresponds to a specific electronic process, whether realtime or samplers, by themselves accompanied by spatialization parameters. The sequencing of electronics in this score are either notated as relative to the performance tempo or fixed absolute values if necessary. The circled numbers in the score correspond to synchronization points between the instrumental and the electronics scores. consisting of the electronic score and its sequential execution as a reaction to the received tags from the interactive system. The two components are interactive and reactive due to their implicit nature of time and following [START_REF] Halbwachs | Synchronous Programming of Reactive Systems[END_REF]. The score follower is an interactive systems since it should be considered as part of the physical world (with the musician), and the second reactive since it runs on its own implicit clock and in reaction to an external environment.

SCIENTIFIC AND MUSICAL GOALS OF SCORE FOLLOWING

The score following literature is constantly increasing every year with new algorithmic contributions. It is important to distinguish between the scientific goals and artistic goals of such systems. Our argument here is that while the two goals are non-dissociable, they are however distinct and achieving one does not necessarily entail the other, and hence new research paradigms should be explored.

Scientific Context

Score following, in its scientific context today, deals with correct alignment of audio streams onto a symbolic score and correct extraction of musical parameters of the interpretation in hand. Besides automatic accompaniment systems, score followers and automatic alignment systems are employed as front-ends for many MIR applications such as score-informed editing softwares and source separation.

It is clear that in such applications the alignment and extraction precision is of utmost importance and much algorithmic effort has been dedicated to achieve this. Figure 3 shows a simplified diagram of this aspect of score following where the computer is aware of the symbolic score expected to be performed by the live performer. Most score followers in the literature focus on this aspect of its applications. This is particularly clear by the effort of the community for evaluations of such systems [START_REF] Cont | Evaluation of real-time audio-to-score alignment[END_REF][START_REF] Scofomirex | Score following evaluation proposal[END_REF].

Artistic Context

Despite advances in alignment systems, very few systems have explicitly considered the artistic goals of score following systems and their direct design consequence. The artistic goals of score following as an interactive music system, and their discrepancy with its scientific goals can be discussed within two folds:

1. Realtime Music Performance: Score following is naturally a tool for realtime performance of mixed instrumental and live electronics. The discrepancy between the scientific and artistic goals in this context has much to do with the idea of robustness of the realtime alignment algorithms in score followers, especially in the context of live performance and realtime processing within uncertain environments. An ideal musical performance for an architecture depicted in figure 2 can be naturally achieved if the interactive system in hand has 100% precision in the recognition phase and given any performance.

While recent systems demonstrate high precision and performance in realtime, the architecture in figure 2 is probably a bad choice for the musical finality of an interactive music system. To summarize, the musical goal of such interactive systems require that the musical output is acted upon expected despite any error from the live performer or the recognition system.

Authoring and Composition of Realtime Music Processes:

The ultimate goal of score following as an interactive music system is naturally to express reactive programs that create the electronics part during the composition phase. This task requires a minimum of musical expressivity within the language that describes such interactions and gets naturally close to the idea of virtual scores as discussed earlier. Treating the interactive and reactive phases of an interactive piece separately as depicted in figure 2 most often evades expressing such interactions and most existing systems have found comfort in leaving this very important issue apart.

To demonstrate the above issues, consider again the score in figure 1 as an example: The first issue is clear by differentiating electronic events tagged by ① and ③. Event ① does not necessarily depend on the recognition of the first note in bar 1 (an F ) while event ③ can be considered as having a local scope. If the musician or the recognition system misses the first high F in the violin part, it is evident that ① should not be dismissed. This is however not true for ③. If the chord corresponding to ③ is missed, that event can be consequently dismissed in order to maintain a musical coherence in the output. The second issue has much to do with the authoring of the electronic events presented graphically in figure 1. Parallel lines in this score correspond to concurrent electronic programs which are expected to output synchronously during live performance with their timing notated relative to the live performer's tempo. Most realtime programming environments however do not neither allow such timing expressivity for programming as time is usually expressed in absolute values, nor explicit concurrency in expressing electronic processes.

CONSIDERATIONS FOR CREATIVE USE OF SCORE FOLLOWING

With the above introduction, we draw important requirements for the use of score following as an interactive music system, destined both for composition and performance:

Time is Resource

Explicit modeling of time is of utmost importance for a score following system both at the recognition phase (for realtime performance) and authoring (composition). However, most score following techniques have focused on the event level (pitch, spectrum observation, etc.) and left temporal models approximate or implicit. At the same time, any music score contains important timing information such as tempo, relative durations and timing hierarchies within elements, that can help both phases of score following use. This issue is of extreme importance when such systems are to be employed in realtime (and thus in absence of future information for decoding), and can significantly enhance recognition and also access to temporal elements for computer music composition. The only practical score following systems which consider explicit time models for both phases are Music Plus One [START_REF] Raphael | Music Plus One: A System for Expressive and Flexible Musical Accompaniment[END_REF] and Antescofo [START_REF] Cont | A coupled duration-focused architecture for realtime music to score alignment[END_REF] where tempo and event durations are first-class citizens in the systems and are employed both during recognition and accompaniment. In [START_REF] Raphael | Music Plus One: A System for Expressive and Flexible Musical Accompaniment[END_REF], temporal considerations are taken into account as a secondary pass and cascaded to an event recognition HMM based system, requiring offline learning of time parameters for best performance. In Antescofo an explicit time model is coupled with an audio recognition system through Anticipatory Learning, attempting to reduce complexity of computation and with no requirement for off-line learning. Antescofo in particular makes time and tempo variables explicitly available for programming reactive electronics.

Heterogeneous Models of Time

Any classical piece of music has multiple and heterogeneous models of time, which should be explicitly considered in the conception of any interactive music system. For example, grace notes in classical music are typical of atemporal events which exist spatially in a score but do not contribute to the tempo variations as opposed to temporal events (regular notes and chords). Glissandis whenever the instrumentation allows are also typical of continuous time events as opposed to discrete time events in regular notes and chords. Finally, trills and tremolos in the classical repertoire undergo hierarchical time structures where the global event itself can occupy a discrete or continuous duration while its internal elements can constitute (for example) atemporal elements. Presence of heterogeneous times is more than evident in the contemporary music repertoire and more than essential in expressing electronic processes. An electronic process can contain discrete (relative or absolute time) events as well as continuous controls, or in some cases recursive processes. The point here is that such considerations are neither unique to electronic music, nor to any specific style of music. Western music notation has internalized such temporal structures that are in use by all composers and performers while computer music languages are still behind in terms of expressivity of time and their models. We will come back to this issue later in section 5.

Critical Safetiness

The musical output of an automatic accompaniment or score following system should not solely depend on the recognition system, or even to the live performer at some instances. This is in analogy to human coordination for ensemble performance: A live music performance should be smooth in time, and does not halt in presence of any error in realtime. As discussed in section 3, one of the discrepancies between scientific and artistic goals of a score following system is the issue of live performance and robustness of the recognition in realtime. We showed on a simple example in section 3.2 how a simple specification of electronic processes can save the musical output despite any error from the environment or the interactive recognition system. Interestingly the issue of critical safetiness is the subject of study in most realtime systems [START_REF] Storey | Safety critical computer systems[END_REF] and already employed in the industry. These paradigms should also be adopted for score following systems.

Authoring of Time and Interaction

The most important issue for creative use of score following, is in how such systems would bring live interaction as a first-class citizen in the compositional phase and enable an authoring of time and interaction for artists. While common computer music programming environments enable live interaction with musicians, they are particularly poor for authoring of time and interaction for composed music. The lack of explicit authoring tools of this kind has led to a common division between the performative and compositional aspects of computer music [START_REF] Puckette | Using pd as a score language[END_REF], criticized thoroughly by several pioneers of computer music [START_REF] Risset | Composing in real-time?[END_REF][START_REF] Stroppa | Live electronics or live music? towards a critique of interaction[END_REF], and has been the subject of debate in a recent colloquium on the subject between various artistic disciplines [START_REF] Ircam | Colloque international écritures du temps et de l'interaction[END_REF].

This issue is directly related to domain-specifc computer language design, and in our case for realtime interactive computer music. We believe that this topic should not be treated separately from common technical considerations of score following systems and is directly related to the musical goals of such systems.

ANTESCOFO'S ARCHITECTURE

Antescofo is the latest incarnation of score following technologies at Ircam since 2008 [START_REF] Cont | Antescofo: Anticipatory synchronization and control of interactive parameters in computer music[END_REF]. It is a realtime score following technologies that aims to integrate the points discussed in section 4 within one single environment. To this end, it consists of a state-of-the-art realtime alignment system, capable of aligning complex polyphonic instruments as well as decoding the realtime tempo of live performance. The novelty of the recognition system in Antescofo lies in its coupling of a realtime audio and tempo agent, and capability of handling heterogeneous times during the recognition phase. It is an Anticipatory System with an attempt to predict event positions in the future in order to aid recognition and undertaking of electronic actions. We leave detailed discussions of the recognition algorithm to [START_REF] Cont | A coupled duration-focused architecture for realtime music to score alignment[END_REF] and instead focus on its musical aspects with regards to points discussed in the previous section.

Antescofo is destined for interactive mixed instrumental and electronics pieces and aims at bringing both interactive and reactive components of a typical piece within this repertoire as discussed in section 2.2.2 within one single framework. Figure 4 shows the general diagram of this architecture. Within this architecture, electronic programs are handled within one framework that allows employment of various time scales and coupling of electronic actions to the live tempo if needed. Antescofo can be employed as a traditional score follower, in which case realtime positions and tempo of the performance are obtained as the module's direct outputs during performance. The composer can optionally integrate electronic messages inside the instrumental score, in which case, Antescofo handles their message-passing to host programs that produce the electronics part. In this sense, Antescofo is used both during the compositional phase as an authoring tool for programming synchronous electronic events with regards to the instrumental score, and also in the performance phase by attempting to produce the desired musical output as described in the original score. Antescofo can thus be used as compositional resource during the authorship of both instrumental and electronic score, on top of traditional use of score followers in live performance paradigm. The use of Antescofo as an authoring tool is possible because of the coupling of the recognition paradigm (natural to any score follower) with a synchronous realtime language for computer music composition. The synchronous language aspects of Antescofo has brought into focus the musical goals of such interactive music systems, which has been underestimated so far as a research paradigm itself. In the following section, we define briefly important aspects of this language with regards to points mentioned in section 4.

SEMANTICS OF PERFORMANCE SYNCHRONOUS LANGUAGE IN ANTESCOFO

The musical goal of score following has to deal with both the compositional and performative aspects of the piece of music in question. In the compositional phase, it has to be able to describe electronic processes in parallel to and ordered with regards to instrumental scores, and by employing the rich temporal semantics of musical intellect. The performative phase of such systems is responsible for evaluating electronic processes at a given position and tempo and as a reaction to the live performance. In this respect, an electronic score in a mixed interactive piece, is in close analogy to an orchestral or accompaniment score except that simple notes are replaced by programs with heterogeneous notions of time, and whose outcomes are not known in advance but deterministic in a musical context. Technically speaking, it is a reactive program with the ultimate goal of determinacy (in the computer science term), correct ordering at runtime, and (musical) critical safetiness. Such paradigms have been widely studied in the computer science literature for realtime synchronous languages [START_REF] Halbwachs | Synchronous Programming of Reactive Systems[END_REF] and widely applied in the industry for realtime critical systems. Our goal in this project is to adopt a musical semantics for such languages, whose application paradigms seem to be closely related to the acts of composition and performance. We will not expose the syntax of the language and leave it to curious users, and instead focus on the constructive semantics that allow an authoring of time and interaction in computer music. An important consequence of the architecture discussed earlier is the coexistence of the instrumental score and electronic score within one single score. An Antescofo score contains two semantics: one for describing the music score of the human performer, and another for describing electronic events in an action semantics. Both semantics are capable of describing multiple scales of time (absolute, pulsed, continuous) heterogeneously within one score. The electronic score is a simple message-passing coordination language, where messages are bound to symbols, ordered and grouped as desired to imitate a musical score. These synchronous messages are then scheduled in realtime to be delivered timely to electronic modules. The choice of such semantics is in accordance with the wide practice of interactive music within MaxMSP and PureData programming environments. The goal of the action semantics in Antescofo is to provide expressivity for authoring of time and interaction and the synchronous scheduling of events in realtime. The primitives of this semantic consists of: Discrete Events Message(s) bound to symbols with an optional delay. The delay can be in absolute time or relative to tempo and thus evaluated in runtime and reactive to tempo changes for synchronous output.

Continuous Events Similar to Break-Point Functions (BPF) where output is interpolated between discrete elements and scheduled in relative or absolute time.

Periodic Events A constructive semantic that allows (absolute or relative) periodic discrete or continuous messages, running forever when launched unless killed somewhere in the score.

On top of these primitives, the user can employ the following constructive semantics:

Parallel Groups Primitives above, can be optionally grouped to construct polyphonic phrases in the electronic score using an optional process name. This semantic is there to bring polyphonic authorship as well as independent but relative timing between groups of electronic phrases. This feature also brings a temporal scope for each group within the score during composition which is respected in runtime using the synchronous scheduling relative to tempo and position.

Nested Hierarchies Groups can be nested hierarchically and recursively to allow independent but ordered timings; respected during realtime scheduling.

Macros Evaluated at score load (and non-runtime) in order to provide motivic patterns both in message content and timing for composition of the electronic score.

Dataflow Functionals Mathematical expressions evaluated at runtime, useful to make message content relative to external variables.

Each block of programs in Antescofo accept specific attributes. Among such attributes, composers can specify the scope of each program dealing with their critical safety in case of performance errors, and also the ability to define independent or varying tempo (accelerando or rubato) relative to the performance tempi.

The development of the above primitives and compositionals have been undertaken incrementally and by observing various uses of interactive systems in composition and performance. Antescofo language is currently text-based with graphical support through NoteAbility Pro notation software editor 5 . We believe that a thorough and welldefined semantics can give rise to graphical semantics of programming which is already the case for synchronous languages within the avionics industries [START_REF] Dormoy | Scade 6: a model based solution for safety critical software development[END_REF].

EXAMPLES

In this section, we aim at representing the compositional aspects of the semantics described above. It goes without saying that a thorough presentation of this system is within a performance situation and live coordination of electronic programs with the live performer within the new score following paradigm. Curious readers can refer to Antescofo website6 for online videos and upcoming performances with the system.

Figure 5 shows an excerpt of the Antescofo score of "Otemo" for Vibraphone and live electronics by the composer Vassos Nicolaou [START_REF] Lemouton | Polyphonic audio score following: The otemo case[END_REF] in the NoteAbilityPro score editor. The top staff the Vibraphone score, where as the 11 bottom staves show the grouped primitives, similar to polyphonic lines, for the electronic scores. In realtime performance, each group is launched according to the performer's position and during the entire life of each group (shown here as their length) rescheduled according to detected position and tempo to assure synchronicity. Figure 6 shows an excerpt graphical representation of the electronic part for "Hist Whist" by composer Marco Stroppa for violin and chamber electronics as written in Antescofo. The timeline of the score is on the x-axis, where as the vertical bar demonstrate individual, parallel and nested processes within each block. This excerpt makes extensive use of nested macros written by the composer that control rhythmic progression of harmonization values and their amplitudes running on the realtime audio from the violin. It consists of 8 main blocks corresponding to 8 generated groups, each having hierarchical periodic primitives with periodic kills to imitate a rhythmic progression. Each pair actuates on one harmonizer module (one for transposition values and other on amplitudes), making it a total of four polyphonic realtime processors. The Antescofo score in figure 6 actually consists of 8 lines calling two macros with different arguments. Each macro recursively calls others defined by the composer to create the above patterns. The final outcome consists of more than 100 ordered and concurrent actions that will be synchronized to the performance in realtime.

The visualization in figure 6 is an experimental feature in Antescofo allowing composers to verify visually the contents of the generated scores during composition. It can be seen as hierarchical automata constructing a complex temporal system with resemblance to visual approaches in synchronous languages [START_REF] Harel | StateCharts: a Visual Approach to Complex Systems[END_REF].

CONCLUSIONS

Whereas traditional score following paradigms have put emphasis on high precision in alignment and extraction of symbolic parameters from realtime audio, the creative use of such systems infer other goals which have been underestimated as a research paradigm in the literature. In this paper we attempted to show those missing paradigms that deal with authorship of time and interaction, critical safety of realtime electronic scores during performance, heterogeneous representations of time, and explicit models of time, both for composition and performance of interactive computer music. We showed the close relation between the creative use of score following systems with that of realtime synchronous programming, bridging the gap between compositional and performative aspects of computer music, and bringing the rich expressivity of musical vocabularies into a simple computer language. The emergence of this research paradigm is mostly due to creative uses of score following systems which do not hesitate to rethink our practices and interfaces with computers for making music. We believe that this synergy will create an important momentum between artists and researchers in the years to come and hope that this paper has shed some lights on the importance of this new paradigm in both communities.

Figure 1 .

 1 Figure 1. First two bars of Anthemes II by Pierre Boulez, for violin and live electronics (1997).

Figure 2 Figure 2 .

 22 Figure 2. General diagram of a typical interactive mixed music.

Figure 3 .

 3 Figure 3. General diagram for an online alignment system.

Figure 4 .

 4 Figure 4. Antescofo's general architecture, comprising of both interactive and reactive systems of figure 2 .

Figure 5 .

 5 Figure 5. Antescofo grouped primitives visualization in NoteAbilityPro, excerpt from Vassos Nicolaou's Otemo for Vibraphone and Live electronics (2008)

Figure 6 .

 6 Figure 6. Visualization of electronic processes (excerpt from movement 3) of Hist Whist by Marco Stroppa, for violin and chamber electronics (2009).

http://repmus.ircam.fr/Antescofo

The first mixed music for tape and instrument appears to be "Musica su due dimensioni" by Bruno Maderna for Flute and Tape (1952).

In fact, Manoury's pieces Jupiter and Pluton can be considered as the first historical Max pieces and patches.

See Pd Repertory Project: http://crca.ucsd.edu/~msp/ pdrp/latest/files/doc/

http://debussy.music.ubc.ca/NoteAbility/

http://repmus.ircam.fr/antescofo


Nicola Montecchio

Arshia Cont

A UNIFIED APPROACH TO REAL TIME AUDIO-TO-SCORE AND AUDIO-TO-AUDIO ALIGNMENT USING SEQUENTIAL MONTECARLO INFERENCE TECHNIQUES

Keywords: sequential montecarlo, particle filtering, real time systems, music alignment, score following

We present a methodology for the real time alignment of music signals using sequential Montecarlo inference techniques. The alignment problem is formulated as the state tracking of a dynamical system, and differs from traditional Hidden Markov Model -Dynamic Time Warping based systems in that the hidden state is continuous rather than discrete. The major contribution of this paper is addressing both problems of audio-to-score and audio-to-audio alignment within the same framework in a real time setting. Performances of the proposed methodology on both problems are then evaluated and discussed.

INTRODUCTION

A music alignment system aims at finding a mapping between two music signals, possibly of different nature, that associate all the parts of each signal to music events. In this paper, we focus our attention on the alignment of a streaming audio signal of a music performance, that is a signal fed as input to the system incrementally in real time, against a symbolic score (the digitized version of traditional music notation) or another audio recording.

Scientific research has focused over the years on real time alignment against a symbolic score (usually referred to as score following). Approaches have evolved from pattern matching techniques in the earliest works dating back to the mid-80s to Dynamic Time Warping (DTW) approaches and Hidden Markov Models (HMM); a review of these approaches, which deal mostly with monophonic signals, can be found in [START_REF] Orio | Score following: state of the art and new developments[END_REF]. Similar techniques are also employed in other fields such as gesture following, where HMM map real time motion capture data to reference gestures [START_REF] Bevilacqua | Continuous realtime gesture following and recognition[END_REF]. Recent work on score following, focusing on polyphonic signals, also proposed hybrid graphical models [START_REF] Raphael | Aligning music audio with symbolic scores using a hybrid graphical model[END_REF] and Hidden Hybrid Markov/semi-Markov models [START_REF] Cont | A coupled duration-focused architecture for real-time music-to-score alignment[END_REF] which deal with tempo (i.e., speed of performance) estimation explicitly, and represent the state of the art systems.

Audio to audio alignment received less attention, and has been investigated most notably in [START_REF] Dixon | Match: A music alignment tool chest[END_REF] which adopts a real time version of DTW. In particular, a common trend in many approaches to score following is to synthesize the reference score and to align the incoming audio against what is, effectively, an audio signal.

Our approach presents a unified methodology for the real time alignment of audio to both a symbolic score and an audio reference by exploiting sequential Montecarlo inference techniques, also known as particle filtering [START_REF] Arulampalam | A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking[END_REF]. The advantages of our approach, besides the powerful statistical framework and inherent simplicity of the algorithm, are twofold: unlike most systems, tempo is an explicit parameter within the stochastic framework defined through musical motion equations; moreover, both symbolic and audio alignment problems can be formulated within the same framework by exploiting a continuous representation of the reference media as depicted in Fig. 1 for the symbolic case. discrete score continuous score t Fig. 1. From a discrete (one state per event) to a continuous (one region per event) score representation.

In the following sections we present in detail the methodology used for both score and audio alignment, and provide experimental evaluation on a hand-labeled test collection that is challenging for current state of the art software. We conclude with an overview of future research directions.

METHODOLOGY

Given a music stream and a reference medium, in the form of either a symbolic score or an audio recording, we formulate the alignment problem as a tracking problem, where the current position of the audio stream along the reference is modeled using traditional motion equations.

The system state is modeled as a two-dimensional vector x = (s, t), representing the current position in the reference media and tempo respectively. In the case of audio to audio alignment, s is measured in seconds and t is the playback speed ratio, while in the case of a symbolic score reference, s is measured in musical time from the beginning of the score, and t in quarter notes per second (i.e. bpm/60). The incoming signal processing frontend is based on spectral features extracted from the FFT analysis of an overlapping, windowed signal representation, with hop size ∆T .

In order to use Sequential Montecarlo methods to estimate the hidden variable x k = (s k , t k ) using observation z k at time frame k, we assume that the state evolution Markovian (i.e. depends only on the previous state and the current observation) and define the following quantities:

• p(z k |x k ) is the likelihood of observing an audio frame z k given the current position along the reference s k . We consider a simple spectral similarity measure, defined as the Kullback-Leibler divergence between the power spectrum at frame k and the power spectrum at time s k in the reference audio, or, in the symbolic case, a template spectrum associated to the score event which is active at score position x k . Past experience with realtime score following suggests that the power spectrum is a better feature for similarity computation than chroma, as it preserves octave information.

• p(x k |x k-1 ) is the state transition likelihood; we make use of tempo estimation in the previous frame and assume that tempo is circa equal:

p(x k |x k-1 ) = N ( s k t k | µ, Σ) µ = s k-1 + ∆T t k-1 t k-1 , Σ = σ 2 s ∆T 0 0 σ 2 t ∆T .
Intuitively, this corresponds to a performance where it is expected that the tempo is rather steady but can fluctuate; the parameters σ 2 t and σ 2 s control the variability of tempo and the possibility of local mismatches that do not affect the overall tempo estimate (e.g., a single note played with some delay).

• q(x k |x k-1 , z k ) is the particle sampling function. In our implementation this corresponds to the transition probability density function.

Sequential Montecarlo methods work by recursively approximating the current distribution of the system state using the technique of Sequential Importance Sampling: a random measure {x i k , w i k } Ns i=1 is used to characterize the posterior pdf with a set of points over the state domain and associated weights, and is updated at each time step as in Algorithm 1. An optional resampling step is used to address the degeneracy problem, common to particle filtering approaches; this is discussed in detail in [START_REF] Arulampalam | A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking[END_REF][START_REF] Douc | Comparison of resampling schemes for particle filtering[END_REF]. The decoding of position and tempo is carried out by computing the expected value of the resulting random measure as

Ns i=1 x i k w i k .
Algorithm 1 SIS Particle Filter -Update step

for all i = 1 . . . N s do sample x i k according to q(x i k |x i k-1 , z k-1 ) ŵi k ← w i k-1 p(z k |x i k )p(x i k |x i k-1 ) q(x i k |x i k-1 ,z k ) w i k ← ŵi k j ŵj k end for

EXPERIMENTAL RESULTS

The first set of experiments investigates the ability to adapt to tempo changes by using a synthesized note sequence, while the remaining ones make use of real music recordings, specifically a collection of Chopin's mazurkas annotated with the onset times for the events in the score; in particular, we used a subset of the collection created by C. Sapp for the Mazurka Project 1 consisting of 8 mazurkas for which 4 different performances were annotated. Due to space constraints we omit comparison with other approaches, relegating them to evaluation campaigns such as MIREX 2 , and investigate instead specific situations that benefit from peculiarities of our approach.

Synthetic Data Alignment Evaluation

A random monophonic note sequence was synthesized using a clarinet patch from an orchestral sample library. As the alignments plotted in Fig. 2 show, tempo increases linearly at every quarter note (Fig. 2(a)) or suddenly (Fig. 2(b)), starting from 60 bpm. We measured the average and maximum alignment error for the score events; such error is defined to be the delay or anticipation of the first detection of the event w.r.t. the nominal onset time. Results are summarized in Table 1.

In all the tests a local misalignment can be seen around score position 5; this is due to the fact that two subsequent notes are repeated and no form of onset detection is used. Increasing the number of particles (to 2000) leads to a more robust tempo estimation and consequently avoids this problem. It should be noted that the average error is comparable to the analysis window hop size.

Audio to Score Alignment Evaluation

The symbolic scores were extracted from MIDI files downloaded from the Web, thus increasing the difficulty of the problem because of the many imperfections that are often present (i.e., skipped notes and incorrect notation); another significant issue is related to the intrinsic difficulty in formalizing music aspects like embellishments -which are extensively used in Chopin's music and realized differently by performers -in a format like MIDI. Chopin mazurkas form a suitable test set for two reasons: they are complex polyphonic pieces challenging for state of the art score following softwares, and their executions are characterized by substantial tempo oscillations not explicitly notated in the score [START_REF] Grosche | What makes beat tracking difficult? a case study on Chopin's mazurkas[END_REF].

Results are provided in Table 2, using the same evaluation methodology and parameters that were used with the synthetic dataset; subsequently we treat particular cases in separate paragraphs for the mazurkas marked by an asterisk. A close analysis of the results, obtained inspecting the alignment plots, reveals that most of the times the maximum alignment error occurs on the very last beats of the pieces, characterized by a significant rallentando.

tempo

Handling of Optional Repetitions

Even though recorded interpretations of classical music usually respect the composer's instructions regarding repetitions, performers often choose to skip some of those repetitions, as is the case for one of the recordings of mazurka Op. 7 n. 2. The ability to follow optional repetitions can in principle be extended to the case where the performance can, at certain pre-defined points in the score, skip ahead (or go back) to other sections of the music, thus creating an open form interactive score, a common practice in contemporary music.

The particle filtering scheme is easily adapted to this situation, by allowing the particles to "jump" via a simple modification of the sampling step and transition pdf. Figure 3 compares two performances, where the skip of the repetition is clearly visible.

Tempo Change Issues

The analysis of the alignments of mazurka Op. 68 n. 3 pointed out a situation which is at the moment problematic for our score following system: between bars 31 and 32 there is a tempo change indicated by the composer, followed by 12 repetitions of the same chord that render an estimation of the new tempo not possible without some form of onset detection. As Figure 4 shows, the alignment is correct again when the melody starts, because of the added diversity to the harmonic content.

Audio to Audio Alignment Evaluation

For the audio to audio alignment experiments, the test collection of Section 3.2 was reused. In Table 3 we report the average and maximum alignment error for the score events; such error is defined to be the euclidean distance of the ground truth label from the closest alignment point. A similar parametrization as for the audio to score case was used, however only 200 particles are employed.

An interesting case is that of the repetitions in mazurka Op. 7 n. 2, investigated in Section 3.2.1 on the symbolic score following side. In the audio to audio case it is impossible to define possible jump points automatically, since the score is not known; however in some cases the alignment was nevertheless "correct", i.e., resembling the alignment curve of Figure 3

CONCLUSION AND FUTURE WORK

A system for the alignment of a music audio stream to both symbolic and audio references using a unified methodology was presented and its validity demonstrated by its application to a collection of polyphonic music performances.

As anticipated in the analysis of experimental data, the subject of future research will revolve around the ability to follow open-form interactive scores, a priority that will have immediate application because of the already active collaboration with composers on works that are going to exploit this aspect explicitly in their artistic conception.

Another important aspect, currently under investigation, is the dynamic tuning of the model parameters σ s , σ t using extended Kalman filtering techniques to increase accuracy.

1

  http://www.mazurka.org.uk/ 2 http://www.music-ir.org/mirex/wiki/MIREX_HOME

Fig. 2 .

 2 Fig. 2. Alignment to a synthetic track.

Fig. 3 .

 3 Fig.3. Alignment of a performance with optional repetitions.

Fig. 4 .

 4 Fig. 4. Uncertainty in tempo estimation.

Table 1 .

 1 Audio to score alignment -synthetic data set.

	change	avg. error (s) max. error (s)
	steady	0.12	0.71
	3 bpm linear	0.13	0.63
	6 bpm linear	0.09	0.34
	9 bpm linear	0.09	0.23
	15 bpm sudden	0.12	0.71
	30 bpm sudden	0.11	0.52

Table 2 .

 2 Audio to score alignment -Chopin's Mazurkas.

Table 3 .

 3 (b); this is possible only if the variance of the transition pdf is not too low. Audio to audio alignment -Chopin's Mazurkas.

		avg. error (s) max. error (s)
	Op. 6 n. 4	0.18	0.59
	Op. 7 n. 2 *	0.18	1.06
	Op. 17 n. 4	0.17	2.14
	Op. 24 n. 2	0.29	9.41
	Op. 30 n. 2	0.13	0.58
	Op. 63 n. 3	0.15	1.23
	Op. 67 n. 1	0.16	1.36
	Op. 68 n. 3	0.19	2.12