Cette thèse est le fruit de 4 années de travail éclairées par les nombreuses personnes qui l'ont accompagnée et que je tiens à remercier chaleureusement.

Les premiers remerciements sont pour mes directeurs de thèse, Wendy Mackay, Carlos Agon et Fanis Tsandilas, qui ont accepté de me guider dans cette riche expérience, du début à la fin. Merci ... ...Wendy d'avoir créé ce thème de recherche, de m'avoir accueilli dans ton équipe et de m'avoir tant appris, des bases de l'IHM aux arcanes des études avec les compositeurs en passant par l'écriture scientifique.

...Carlos de m'avoir ouvert cette voie en Master et d'avoir le don d'initier des projets réussis et enrichissants entre science et musique, toujours avec le sourire et le mot pour rire.

...Fanis pour le soutien sans faille et la richesse de notre travail commun. Au delà de ta bonne humeur permanente, j'ai énormément appris grâce à toi et j'ai particulièrement apprécié nos discussions sans fin pour arriver à mieux étudier les compositeurs.

Je remercie très sincèrement les deux rapporteurs de ma thèse: Stéphane Conversy et Simon Holland, pour leur disponibilité, leurs commentaires et leurs suggestions avisées sur cette dissertation. Je remercie également les examinateurs de mon jury Rebecca Fiebrink, Nicolas Roussel et Marc Schoenauer pour leurs questions et remarques qui m'ont permis de prendre un peu plus de recul sur mon travail. Un merci supplémentaire à Marc pour avoir accepté de présider ce grand jury et de l'avoir fait de façon originale et agréable pour tous.

Ce travail doit énormément aux nombreux compositeurs qui ont acceptés de partager leur temps et leur savoir avec moi, Julia Blondeau, Mathieu Bonilla, Karim Haddad et Mikhail Malt en particulier. Un immense merci à Philippe Leroux et Gilbert Nouno pour l'immersion dans le processus de création du croquis au concert.

Merci à Lucie Thomas et Mathieu Chailloux pour l'investissement et l'état d'esprit lors de nos travaux communs. v Ces quatre années de thèse se sont déroulées entre deux équipes, l'équipe In|Situ| du LRI et l'équipe Représentations Musicales de l'IRCAM, qui m'ont offert un cadre de travail des plus agréable avec chacune leur spécificité culturelle et géographique.

Merci aux membres de l'équipe In|Situ| pour votre temps, vos conseils et votre aide précieuse. Une pensée pour les anciens, Clément, Cyprien, Emilien, Julie, Mathieu, Romain... les prochains, les Alex, Can, David, Daniel, Ghita, Justin... et tous ceux de passage..., en espérant vous croiser à nouveaux entre l'enseignement, les conférences et les sorties "touristiques" à Paris. Mention spéciale pour Monireh pour la bonne ambiance permanente dans le bureau.

Merci aux membres de l'IRCAM, en particulier Hugues et Sylvie, pour leur accueil. Merci à Gérard pour m'avoir fournit le gîte et le couvert dans l'équipe RepMus dont je remercie tous les membres, permanents ou pas, mutants ou pas, pour les échanges amicaux, musicaux et (a)temporels entre le local CE et la passerelle. Mention spéciale aux acolytes du bureau A107 : Jean, Jérôme et Hélianthe pour les rigolades et une pensée particulière pour Louis "Tonnetz" Bigo pour la collaboration inoubliable ! Merci à ma famille et à ma grande famille. Maman, Papa, Frérot, les Grenier et les Amigos, merci pour le soutien de près, de loin, de très loin, tout le temps et par tous les temps.

Un dernier merci à Marine et ses sourires sans laquelle la thèse n'aurait surement pas eu la même saveur.

A B S T R A C T

This thesis focuses on the design of interactive paper interfaces for supporting musical creation.

Music composition has been deeply influenced by the computational power brought by computers but despite the use of software to create new sounds or work with symbolic notation, composers still use paper in their creative process. Interactive paper creates new opportunities for combining expression on paper and computation. However, designing for highly individual creative practitioners who use personal musical representations is challenging.

In this thesis, I argue that composers need personal and adaptable structures on paper in which they can express and explore musical ideas. I first present three field studies (Chapter 3) with contemporary composers that examined the use of paper and the computer during the composition process and how linking the two media supports exploration of musical ideas. I then describe a participatory design study that investigates the use of formal musical representations (Chapter 4) for creating new paper interfaces that extend computeraided composition tools. I introduce Paper Substrates (Chapter 5), interactive paper components that provide modular structures for interacting with personal representations of computer-based musical data. I detail tools that we created to develop paper applications with the Paper Substrates approach. Several examples illustrate the creation of personal structures and musical content that can still be interpreted by computer-aided composition software.

I then describe a structured observation study with 12 composers who used Polyphony to compose a short electroacoustic piece (Chapter 6). Polyphony is a unified user interface that integrates interactive paper and electronic user interfaces for composing music. The study allowed us to systematically observe and compare their compositional processes. Finally, I report on a research and creation project with the composer Philippe Leroux during the composition of his piece Quid sit musicus (Chapter 7). Several work sessions with the composer and a musical assistant lead to the design of new paperbased interfaces for generating composition material, synthesizing sounds and controlling the spatialization from handwritten gestures from calligraphic gestures over an old manuscript.

iii R É S U M É Cette thèse porte sur la conception d'interfaces de papier interactif dédiées à la création musicale.

Malgré l'apparition d'outils informatique permettant de créer de nouveaux sons ou d'opérer des transformations sur des représentations symboliques de la musique, les compositeurs utilisent toujours le papier pour composer. Le papier interactif rend théoriquement possible la création d'interfaces combinant l'expression sur le papier et la puissance de calcul de l'ordinateur. Cependant, la conception de telles interfaces reste un problème complexe car chaque compositeur développe son propre processus de création, impliquant de nouveaux concepts et de nouvelles représentations musicales.

Cette thèse propose l'utilisation de structures personnelles et adaptables sur le papier pour permettre aux compositeurs d'exprimer et d'explorer leurs idées musicales. Trois études conduites avec des compositeurs à l'Ircam (Chapitre 3) s'intéressent aux rôles du papier et de l'ordinateur lors du processus de composition et les possibilités du papier interactif pour l'exploration des idées entre ces deux supports. Je décris des sessions de conception participative (Chapitre 4) qui examinent l'utilisation de représentations musicales formelles pour la création de nouvelles interfaces de papier couplées avec les logiciels de composition assistée par ordinateur. Je présente les Paper Substrates (Chapitre 5), des composants papier qui définissent des structures modulaires pour l'interaction avec des représentations personnelles de données musicales issues d'environnements informatique. Je décris une boîte à outil ainsi qu'une application graphique permettant de développer des interfaces utilisant les Paper Substrates. Des exemples illustrent la création de structures personnelles et de données musicales sur le papier qui peuvent être interprétées par les logiciels de composition assistée par ordinateur.

Je présente une étude d'observation structurée conduite avec 12 compositeurs utilisant Polyphony pour composer une pièce électroacoustique (Chapitre 6). Polyphony est une interface qui intègre le papier interactif avec d'autres interfaces pour composer. L'étude a permis d'observer et de comparer les différents processus de composition des participants. Enfin, je détaille un projet de recherche et création mené avec le compositeur Philippe Leroux pour la création de sa pièce Quid sit musicus (Chapitre 7). Nous avons conçu de nouveaux systèmes interactifs pour la création de ressource compositionnelles, la synthèse sonore et le contrôle de la spatialisation à partir de tracés effectués sur une partition médiévale. paper and computer 3. In this dissertation, I explore how contemporary music composers combine paper and computer during their creative process and propose a novel way to express and capture their structured representations of musical data. I have designed and evaluated a variety of interactive paper composition tools that enable composers to create personal interactive paper interfaces taking advantage of the best aspects of paper and software to enhance existing computer-aided composition environments.

Contemporary music composition is a highly creative and disciplined activity that involves the free expression of ideas and requires computer programming skills. During the composition process, composers need to express ideas, explore them and produce final scores. All these tasks usually involve a combination of writing on paper, playing with instruments, either physical or digital and interacting with computer-aided composition software. I am interested in understanding this highly complex process in order to design novel interactive tools that support creativity.

Paper plays a specific role in contemporary music composition. It is often a starting point for making early sketches and also the final medium for the realization of the work as a final score (Letondal et al., 2007). A sketch can represent a complex but incomplete idea to work out the details later [START_REF] Healey | Sketching Musical Compositions[END_REF]. Some sketches are rough and unfinished, others are carefully executed, such as curves that represent amplitude or other real-time processes. Writing is powerful because it enables users to reflect upon an idea and abstract many performance aspects. Hand-drawn sketches are useful for working out a composition structure, and hand-written notes and annotations help the composer remember specific ideas. On paper, composers use traditional notations as well as their own sets of representations dedicated to the task at hand.

Since the fifties, composers have been drawn to computers to analyze audio signals, produce new sounds or generate scores from computer programs. One of the earliest examples of computer generated scores is the Illiac Suite by [START_REF] Hiller | Experimental Music; Composition with an Electronic Computer[END_REF]. They conducted several experiments to generate scores with different sets of musical constraints. Since these first approaches, computer music technology has expanded to cover several aspects of musical creation from sound synthesis in the seventies to current personal digital audio workstations and music score editors [START_REF] Roads | The Computer Music Tutorial[END_REF]. Composers also use graphical programming environments such as Max and Open-Music as means to explore and implement their musical ideas via audio-signal or symbolic data processing.

Previous work by Letondal et al. (2007) and [START_REF] Sellen | The Myth of the Paperless Office[END_REF] show that despite this variety of devices and software, composers still rely on pen and paper to express and explore their musical ideas at different stages of their creative process. In particular, [START_REF] Healey | Sketching Musical Compositions[END_REF] conducted a case study on the evolution of sketches for a specific composition and concluded that sketches provide a "suitably underspecified representation" that helps composers avoid premature commitments. This freedom of expression often leads to a problem when the user shifts to a more constrained computer-based tool to implement these ideas, with an abrupt shift in format and interface.

Most of the computer-aided composition tools feature advanced computational possibilities but are much more limited in terms of interaction. They usually only support mouse and keyboard interactions with buttons and textfields. Music composition is less well developed than fields such as performance [START_REF] Collins | Getting there': Do we need to study how people compose music?[END_REF] and creativity has been neglected in favor of audio processing and final implementation [START_REF] Eaglestone | Composition systems requirements for creativity: what research methodology[END_REF]. The NIME (New Interfaces for Musical Expression) community emphasizes the design and evaluation of novel instruments, interactive installations, and sound controllers [START_REF] Orio | Input devices for musical expression: borrowing tools from HCI[END_REF][START_REF] Magnusson | The acoustic, the digital and the body: A survey on musical instruments[END_REF] for performance over composition itself. Even if playing with new instruments to discover new sounds is part of the composition process [START_REF] Gelineck | Longitudinal Evaluation of the Integration of Digital Musical Instruments into Existing Compositional Work Processes[END_REF], composers also need to express and write ideas. [START_REF] Shneiderman | Creating creativity: user interfaces for supporting innovation[END_REF] argues that designers need to design tools not only for productivity but also to foster innovation. Designing computeraided composition tools not only requires focusing on technical aspects such as enhancing quick input of musical scores or improving sound synthesis algorithms, but it also needs to support the creative aspects of music composition. In his book, [START_REF] Shneiderman | Leonardo's Laptop: Human Needs and the New Computing Technologies[END_REF] suggests that "The old computing is about what computers can do; the new computing is about what people can do."

Interactive paper technology allows to capture notes written with a pen on paper. This technology creates new possibilities for helping composers transition between physical and digital worlds and explicitly combining the benefits of both. For example, as with other tangible interfaces, interaction on paper is space-multiplexed [START_REF] Fitzmaurice | Bricks: laying the foundations for graspable user interfaces[END_REF] rather than time-multiplexed. Users can spread out different, potentially very large, sheets of paper to display and process information. On the contrary, when using a screen or a tablet, users must view everything through a single window. Each time they need to focus on a specific resource, as the previous one is not visually accessible anymore. Since the Digital Desk [START_REF] Wellner | Interacting with paper on the DigitalDesk[END_REF], a seminal work for augmented paper interfaces, previous work stud-ied the use of interactive paper technology to support active reading and annotation of documents [START_REF] Liao | Papiercraft: A gesture-based command system for interactive paper[END_REF], music composition with recognition of personal gestures vocabularies [START_REF] Tsandilas | Musink: composing music through augmented drawing[END_REF] or editing 3D models [START_REF] Song | The ModelCraft framework: Capturing freehand annotations and edits to facilitate the 3D model design process using a digital pen[END_REF]. While several interaction models, techniques and toolkits have been created for pen based computing [START_REF] Steimle | Pen-and-Paper User Interfaces[END_REF], only few focus on creativity support. We still need models and tools tailored for the inherent complexity of music composition that could help composers iteratively design and assess their musical concepts.

We are interested in the design of tools that fully integrate the benefits of paper and software to support music composition. Our goal is to combine the personal representations that the composers create on paper with sophisticated programming offered by computer-aided composition. We also expect that studying such extreme users could help us design for creative experts in general.

thesis statement

In this dissertation, I argue that composers need personal and adaptable paper interfaces that extend their online composition environments, thus providing an additional space for expression and exploration of their musical ideas. Composers not only use paper to sketch rough ideas, they also create evolved and formalized musical representations for the entry, visual evaluation and editing of their ideas. Such structured representations can be computerized to let composers benefit from their computer-aided composition tools. I introduce Paper Substrates, which provide modular structures for creating new paper interfaces with personal representations of computerbased musical data. We used Paper Substrates to build new interactive systems to assist composers' work and to study their composition process.

research approach

Contemporary music composition is an extremely creative field where each composer follows a unique process, thus making the design and evaluation of new interactive systems challenging. To cope with this complexity, I used several user-centered methods, triangulating between observation, design and theory [START_REF] Mackay | HCI, natural science and design[END_REF], to better understand composers, design new technology and evaluate its impact on composers's creative process. Figure 1.1 summarizes the main projects that constitute this thesis following this triangulation framework.

T h e o r y D e s i g n O b s e r v a t i o n

Interviews

Field study

I began by interviewing composers to get insights about the work practice of composers, determine existing problems and potential needs. I observed how composers develop and use formal musical representation on paper to motivate the design of specialized paper components. I also conducted a longitudinal study with one composer during the composition of a piece to understand how the com-poser adapted the technology and how it helped him to explore musical processes.

Participatory design

My primary method of working with composers was participatory design (Greenbaum and Kyng, 1992;[START_REF] Schuler | Participatory design: Principles and practices[END_REF] using techniques developed by [START_REF] Mackay | Using video to support interaction design. DVD Tutorial[END_REF]. The composers are not considered as designers but their expertise and knowledge helped me to focus on their actual concerns. Participatory design is an iterative process encompassing a variety of specific methods to understand users, design tools that match their needs and evaluate the results. Each step of the process informs the next one and evaluation of particular technology is also a mean to gain new insights about users or to generate new research directions.

I used participatory design along with field work with a single composer to understand how he could benefit from interactive paper technology to explore his ideas. I also conducted five participatory design sessions with four composers and a musical assistant to explore the creation of personal interactive paper interfaces linked with computer-based composition tools. The participatory design studies helped to design Paper Substrates to support the creation of personal interactive paper interfaces with formal and structured musical representations. I developed a toolkit and a graphical user interface to design several paper applications. I collaborated with composers to see how they could create personal paper interfaces with Paper Substrates. For example, during my collaboration with Philippe Leroux, we iteratively designed interactive systems to let him create musical material from handwritten calligraphic gestures. The interactive interfaces helped the composer refine his musical concept and processes while interacting on paper.

Technology probes

I developed various interactive paper technology probes [START_REF] Hutchinson | Technology probes: inspiring design for and with families[END_REF] and observed how composers used them. Technology probes serve three interdisciplinary goals: "[...] the social science goal of understanding the needs and desires of users in a real-world setting, the engineering goal of field-testing the technology, and the design goal of inspiring users and researchers to think about new technologies [...]". A technology probe study consists of introducing "[...] simple, flexible, adaptable technologies [...]" in a real world situation to help users evaluate the probe and reflect on their existing practices.

I used a prototype called InkSplorer, a pen-based composition tool that links paper-based and software-based exploration of ideas, as a technology probe to conduct an exploratory evaluation with four composers. The results assessed the potential of linking paper and computer but also highlighted a need to support complex and personal representations of musical data on paper. I designed Polyphony, a user interface integrating pen-based user interface with regular ones to support composition from the early ideation phase to final implementation, as a technology probe to better understand the impact of pen-based interfaces on the composition process.

Structured observation

I used a structured observation method [START_REF] Mackay | Structured Observation to Support Interaction Design[END_REF] to compare the real-world composition processes of experienced composers using my technology. Structured observation combines elements of controlled experiments to facilitate comparisons, with a realistic task to enhance external validity. In contrast to a controlled experiment, the goal is not to assess a clearly defined hypothesis with the data collection. Instead, we want to create highly comparable conditions for observing and comparing common patterns and processes with qualitative and quantitative measures.

contributions

In this dissertation, I discuss how interactive paper can support musical creation by extending computer-aided composition environments with additional physical space on paper. The thesis offers the following empirical, technological and theoretical contributions: empirical findings:

• Despite the extreme variability in the content of their musical ideas, composers use a common strategy: They create a structured representation of the musical idea, and then move back and forth between this structure and the specific details of the composition.

• Composers often abandon their personal representations on paper when they switch to their computer-based environments.

• Linking the two media, paper and computer, improves exploration of musical ideas.

• Using formal and structured representations on paper enables composers create personal paper interfaces that extend their computer-aided composition tools.

• Composers use different strategies to complete a composition task with an interactive tool. Some interact with the tool first, after which they refine and adapt the musical ideas. Others define their ideas first and then use the tool to implement them.

• Interactive paper tools bridge the gap between paper-based representations and computer software, allowing composers to engage in a partnership with the computer to explore and execute their ideas.

innovative and adaptable technologies:

• InkSplorer lets composers interact with handwritten curves on paper to control their musical programs and compare alternative results.

• The Paper Substrates Toolkit and PaperComposer support the creation of personal interactive paper interfaces connected with computer-aided composition environment with Paper Substrates.

• Paper Substrates can be connected, layered or spatially arranged to share parameters between them, create complex structures or synchronize their properties.

• PaperTonnetz lets composers use Tonnetz representations in which strokes become chords and melodies.

• Polyphony integrates a standard mouse-and-keyboard computer user interface with interactive paper, graphics tablet and physical controllers to let composers create an electronic accompaniment to a short instrumental piece.

• Specialized substrates and associated patches in Max and Open-Music support composition with calligraphic gestures written over an ancient manuscript.

theoretical perspectives:

• The concept of substrates captures the method by which composers create complex structures to represent their musical ideas.

Paper Substrates support the entry, editing and refinement of musical ideas by allowing composers to create structured representations that can be transferred to the computer. This underlying framework both captures how composers already generate novel ideas and also provides the architecture for interactive paper-based tools that facilitate experimentation on both paper and the computer.

• The design of the composition task for the structured observation study offers a novel approach for systematic comparison of a real-world composition processes by experienced composers.

thesis overview

Chapter 2 describes background and related work relevant to the design of interactive systems that actively support music composition. It studies computer-aided composition and presents studies about the compositional process. It then reviews several examples of interactive paper interfaces and discusses how they reuse users' expertise on paper and combine it with software.

Chapter 3 reports on three studies conducted with professional composers at Ircam to study how they use paper and software during their composition process. We investigated how expanding their composition tools with interactive paper can support their creative process.

Chapter 4 describes a participatory design study with four professional composers at Ircam. Examples of formal ad hoc representation systems designed by composers on paper motivated the participatory design sessions to explore how such structures could help composers working with their personal data on paper.

Chapter 5 introduces Paper Substrates, interactive paper components that can handle musical data from computer-aided composition applications, interpret pen input and reflect their change into software. It describes their physical and interaction design as well as a toolkit and a graphical user interface to help developers and composers creating personal interactive paper interfaces with Paper Substrates.

Chapter 6 presents a method for comparing the composition processes of professional composers and introduces Polyphony, a user interface that combines interactive paper with common software interfaces. We conducted a structured observation study with twelve composers to systematically observe their creative process and identify how they both adapt and appropriate paper, pen-based interfaces and computer tools.

Chapter 7 reports on a research and creation project with the composer Philippe Leroux for a commissioned piece at Ircam. We collaborated with the composer and a musical assistant to iteratively design new interactive systems that let him reuse the calligraphic gestures from old-manuscript as composition material.

The increasing use of computers in music composition challenges composers' established creative practices. Composers use sophisticated computer-based tools that accomplish specific musical tasks. However, these tools were generally not designed to support early creative exploration of ideas and do not help composers make the transition from early, ill-formed and personal representations on paper to later programs executed on a computer.

This chapter discuss studies that examine the creative process of professional music composers across paper and computer. It then presents computer-aided composition approaches for supporting composers' process. Finally, It gives examples of interactive paper interfaces that integrate both physical and digital practices.

studies of the compositional process

Studies by music theorists and researchers on the compositional process mostly focus on the analysis of recordings, sketches or scores after the completion of a musical piece (though see Section 2.2.3 for some exeptions). Even if composers have emphasized the challenge of creating new music [START_REF] Xenakis | Formalized Music: Thought and Mathematics in Composition[END_REF][START_REF] Stroppa | Un orchestre synthétique : remarques sur une notation personelle[END_REF], only few studies are conducted directly with composers during the composition process. Music composition can be observed from different perspectives. [START_REF] Eaglestone | Composition systems requirements for creativity: what research methodology[END_REF] note that musicologists conduct most of the studies of composers, and interpret them from an educational or music-theoretical perspective rather than technology, software or user interface design aspects. For example, [START_REF] Donin | Theoretical and methodological issues related to long term creative cognition: the case of musical composition[END_REF] studied the entire composition of a piece through interviews and video logs. Although composition involved the use of several computer-aided composition tools, the study focused on long-term cognition rather than how the actual tools supported the creative process.

Musical notation

Music notation appeared initially to aid memorization of music and to transfer musical knowledge. Figure 2.1a shows an example of early notation of music (around the 10 th century) with neumes drawn above the text. Graphical properties of the neumes such as their position indicates approximately their pitches and to which word they apply. After this initial role, notation quickly became a powerful means for composing music as it can represent multiple musical dimensions such as time or pitches in space [START_REF] Bosseur | Du Son au Signe, Histoire de la notation musicale[END_REF]. [START_REF] Dufourt | Musique, pouvoir, écriture[END_REF] argues that "Western music came to be seen as an original act of creation only from the moment it submitted the ear to the control of the gaze1 ". Since the antiquity, music notation had to become more complex in order to capture the various aspects of the music production gestures such as intensity or playing modes in addition to rhythms and pitches [START_REF] Grout | A history of Western music[END_REF]. Professional music composers deal with a large range of music representations both on paper or in the computer [START_REF] Read | Music notation: a manual of modern practice[END_REF][START_REF] Dannenberg | Music Representation Issues, Techniques, and Systems[END_REF].

Co-evolution of composition and notation

Since its earliest form, musical notation evolved over the centuries to encode the increasing number of musical details such as intensities, playing modes or electronic processes. In his thesis, [START_REF] Thiebaut | Sketching Music: Representation and Composition[END_REF] argues that "Music representations have co-evolved with music composition practices and technologies. This co-evolution is illustrated by the initial development of notation as a means to record music, which in turn impacted music composition by providing a formal way to offload memory processes and to develop abstract relationships on the medium itself." He gives the example of geometrical transformations used by Bach on the score to create new material with existing melodies. Another representative example is the composition approach of Iannis Xenakis, one of the most influential post-war composers. Trained as an architect, Xenakis developed graphical representations to explore complex musical forms and models with inspiration from mathematics and physics [START_REF] Xenakis | Formalized Music: Thought and Mathematics in Composition[END_REF]. He regularly used graph paper (Figure 2.1b) and personal graphical templates, which facilitated the drawing of geometrical shapes such as parallel lines and calculation, e. g., approximating line intersections.

The roles of paper

In a study with contemporary composers, Letondal et al. (2007) found that many composers still rely on pencil and paper for sketching partially formed ideas, usually during the earliest stages of the creative process, when the composer first struggles to represent a musical idea. [START_REF] Coughlan | Interaction in creative tasks[END_REF] argues that, when expressing ideas, paper requires a lower cognitive load than software, making paper appropriated for ideation. A sketch can represent a complex, but as yet incomplete idea: the details can be worked out later. [START_REF] Healey | Sketching Musical Compositions[END_REF] conducted a case study on the evolution of sketches for a specific composition. Sketches captured various elements of the composed music, such as sound frequencies, spatial relationships and the temporal evolution of the score. They concluded that sketches provide a "suitably underspecified representation" that helps composers. In his thesis, Thiebaut (2010) goes further and argues that "the use of pen and paper can play a key role in the development and refinement of musical ideas [whereas] existing technology does not support similar functions, but rather encourages a premature commitment to the realization of a piece." [START_REF] Sellen | The Myth of the Paperless Office[END_REF] emphasize that when moving to the computer, specific properties of paper related to his tangible nature such as folding, spatially arranging sheets or carrying are lost.

In addition to sketches, composers also use paper to write their final scores and annotate them. Letondal et al. (2007) explains that paper gets two different roles. First it serves as a flexible medium, successfully supporting sketching and quick capture of ideas, often in a personal manner. Second, it is a static medium where composers fix the final state for a composition.

Composing with computers

In one in-depth study, [START_REF] Eaglestone | Computer support for creativity: help or hindrance[END_REF] noted that an electroacoustic music composer had difficulty keeping track of electronic objects and navigating the various user interfaces. However, they also remarked on the experimental nature of his creative process and found that "errors often produce the most artistically interesting results". Instead of actual errors, which are undesirable, the quote rather refers to non-deterministic behaviors that introduce variability. Interfaces with gestural input instead of textual input could thus help composers avoid too deterministic interactions with the computer-aided composition tools and foster their creativity. Interviews by [START_REF] Gelineck | From Idea to Realization -Understanding the Compositional Processes of Electronic Musicians[END_REF] also support this claim and suggest that creators of electronic music often prefer tools that are "unpredictable" in some way or can be used in unintended ways as "too much control can kill the creative process". A study of composers who created digital instruments [START_REF] Fiebrink | Toward Understanding Human-Computer Interaction in Composing the Instrument[END_REF] showed that they appreciate mappings between gestures and sounds that produce new unexpected results. However, these mappings must also offer a high level of prediction and control.

In a later study, [START_REF] Eaglestone | Are Cognitive Styles an Important Factor in Design of Electroacoustic Music Software[END_REF] also suggest that designers of creativity tools should take into account different cognitive styles that can lead to different composition approaches. According to the results of a study with electroacoustic composers, they conclude that current software is well suited for verbalizers who think more in words rather than imagers who tend to think visually. Imagers would require less textual interfaces. To address these issues, [START_REF] Amitani | Supporting musical composition by externalizing the composer's mental space[END_REF] introduced spatial representations in addition to chronological ones in a music editor to support creativity by triggering new patterns of cognitive processes.

An iterative process

Composing music is a highly iterative task that requires multiple steps to move from an ill-formed idea, usually sketched on paper, to a more formal representation with programs and final scores. Bennett (1976)'s interviews with eight classical composers offer insights into their creative processes. He found that composers usually start with a sketch of a germinal idea, followed by an iteratively refined draft, until the creation of a final musical score. [START_REF] Andrews | How Composers Compose: In Search of the Questions[END_REF] remarked that other studies and writings from composers suggest that composers generally use several strategies and frequently switch between Bennett's stages rather than following them linearly. [START_REF] Buxton | A composer's introduction to computer music[END_REF] identifies two main approaches in music composition software: composing programs and computer-aided composition programs. Composing programs are also frequently called algorithmic composition software as they are programmed to generate music with a certain autonomy. [START_REF] Alpern | Techniques for Algorithmic Composition of Music[END_REF] defined algorithmic composition as "the process of using some formal process to make music with minimal human intervention". Computer-aided composition programs, on the contrary, serve as a tool to help the composer explore and render musical ideas. To differentiate between the two approaches, [START_REF] Buxton | A composer's introduction to computer music[END_REF] argued that the main difference is "the degree of interaction between the composer and the program during the realization of a composition". He suggested that computer-aided composition tools appeared because of the limitations of algorithmic composition programs: "the more knowledge and power that is built into a program, the less general its musical application". This implies that powerful algorithmic programs may not be reusable for other compositions as they will be too specialized. The goal of computer-aided composition software is not to generate music per se but rather to support compositional tasks. According to the computer-aided composition paradigm, the computer is not only a tool capable of generating music on its own, but also a powerful assistant that supports the compositional process. The opening quotation by [START_REF] Laske | Composition Theory in Koenig's Project One and Project Two[END_REF] about composer-program interaction illustrates the expected benefits of computers to generate material but also to assist composers work.

supporting music composition

Computer-aided composition

Computer-aided composition programs focus on the formalization of compositional processes through computational paradigms. According to [START_REF] Assayag | Computer Assisted Composition Today[END_REF], "We conceive such an environment [of computeraided composition ] as a specialized computer language that composers will use to build their own musical universe. [...] This leads us to reflect on the various existing programming models, as well as on the interfaces [...] which make it possible to control this programming, and on the representations of the musical structures, which will be built and transformed using this programming." Such a program should let composers build personal systems in which they can express their ideas or models and solve their own musical problems.

Visual programming

Visual programming plays an important role in computer-aided composition as it allows composers not familiar with conventional computing languages to implement their ideas with advanced graphical user interfaces. Programming environments such as Max and Pure Data are based on data-flow programming. In such languages, functions or "objects" are linked or "patched" together in a graphical environment that models the flow of the control and audio. These two environments are frequently used for building real time audio synthesis engines controlled with various input devices or features extracted from audio input. However, despite several extensions such as the Bach Project by [START_REF] Agostini | Real-Time Computer-Aided Composition with bach[END_REF], these environments provide limited support for classical notation and time-based representations of musical scores.

A different approach of visual programming for composition with symbolic music representations was first introduced by PatchWork and extended by OpenMusic and PWGL [START_REF] Assayag | Computer-Assisted Composition at IRCAM: From Patch-Work to OpenMusic[END_REF]. These environments are based on a demand-driven data-flow paradigm, where objects recursively evaluate their inputs. Figure 2.3 presents an example of patch and a maquette created with OpenMusic. In a maquette, objects evaluate their inputs to compute a result in a patch where their horizontal position determines their onset and duration on a time line. This paradigm provides greater support than dataflow programming for composition as it is an "out-of-time" activity. The OM composers' books [START_REF] Agon | The OM composers[END_REF] illustrates how several composers used OpenMusic to create models, algorithms and scores in their own way. Computer aided composition requires composers to define computational processes and to interact with them through different musical data representations. OpenMusic, as with computer-aided composition environments, provides several musical objects such as scores, waveforms or lists of parameters as input and output to support both signal and symbolic operations over musical content.

Music representations in computer-aided composition

Many computer-based representations are built upon western musical notation, but individual contemporary composers and developers of specialized music software have introduced richer representations and new visualization paradigms. OpenMusic includes sets of objects and functions to control external audio synthesis engines with high and low level representations of sound such as sound descriptors, control curves or synthesis parameters [START_REF] Bresson | Musical Representation of Sound in Computer-Aided Composition: A Visual Programming Framework[END_REF]. Composers can apply symbolic calculation on these representations and better integrate them in their compositional process. While it is possible to extend OpenMusic with additional data representations, functions and visualization, this requires advanced programming skills. To overcome this limitation, the ENP package from Laurson and Kuuskankare ( 2006) lets composers define their personal set of symbols within PWGL and to connect them with sound synthesis.

Sketching interfaces

A number of systems have been designed to take advantage of the power of sketching ideas by linking drawings to music composition. One of the most influential interfaces is the UPIC. Inspired by the approach of Xennakis [START_REF] Lohner | The UPIC system: A user's report[END_REF], UPIC translates drawings on a graphics tablet into sound. Figure 2.4a presents a score sketched on the UPIC with a pen where the horizontal position represents time and the vertical position represent pitch. Other software expanded this approach to desktop applications such as HighC2 or Hyperscore3 where graphical objects can be disposed on a time-line to create a piece. Sonic Sketchpad [START_REF] Coughlan | Interaction in creative tasks[END_REF], Music Sketcher [START_REF] Thiebaut | Drawing electroacoustic music[END_REF] or Different strokes [START_REF] Zadel | Different strokes: a prototype software system for laptop performance and improvisation[END_REF] take a different approach, allowing users to draw graphical forms and to map them with online sound synthesis parameters or sounds. All these systems use a mouse, a graphics tablet or an electronic surface to draw musical forms on a computer screen. et al. (2006) conducted a workshop on creativity support tools. Among the contributions, [START_REF] Resnick | Design Principles for Tools to Support Creative Thinking[END_REF] propose a set of principles to guide their design. They emphasize the need for simple tools that encourage exploration of multiple alternatives and advocate using multiple tools, rather than a single one. [START_REF] Hewett | Creativity support tool evaluation methods and metrics[END_REF], reports on a group discussion about several guidelines for planning such research. Similarly to [START_REF] Shneiderman | Creating creativity: user interfaces for supporting innovation[END_REF], he argues that it is necessary to begin with observation of the creative activities and problems faced by users using field studies, computer logging or participatory design. The researcher should find design requirements to address the real-world identified issues before designing and implementing a solution, often with low fidelity prototypes. Then, several evaluations, from qualitative to more quantitative should compare the new tools versus existing practices. Finally, after iteratively repeating the previous steps, the researcher needs to longitudinally study the use of the new tools in real-world situations. For music and electroacoustic music in particular, [START_REF] Eaglestone | Composition systems requirements for creativity: what research methodology[END_REF] recommend to initially focus on qualitative methods before triangulating between qualitative and quantitative methods.

Designing for creativity support

Shneiderman

A special issue of the International Journal of Human-Computer Studies [START_REF] Edmonds | Computer support for creativity[END_REF] was dedicated to user interfaces that support creativity. In one of these articles, [START_REF] Selker | Fostering motivation and creativity for computer users[END_REF] discusses the need for designing creativity tools that do not interrupt users when they try to express ideas: "Having to start a program, change modes, and go to another part of the screen all take attention. This diversion can disrupt the flow and documentation of ideas." He also recommends that creativity tools should be designed to be integrated into all applications and computer work settings as "people will not always be willing to stop working to start up a creativity-enhancing tool".

Evaluating composition support tools

The creative process is open-ended, which makes the evaluation of composition support tools difficult. Composition interfaces are highly specialized; each is suited to a particular task and not the whole composition process [START_REF] Politis | Computer Music Interface Evaluation[END_REF]. In his survey of computer music interfaces, [START_REF] Pennycook | Computer-music interfaces: a survey[END_REF] argues that: "Unlike text-editing environments, in which measures of productivity can be gathered empirically, in most musical settings, productivity and aesthetic value become hopelessly confused. ... A user interface that satisfies the needs of one musician in an efficient, well-ordered way may be awkward or even counterproductive for another musician".

Prior work on musical instruments evaluation introduced metrics such as degrees of freedom or required expertise [START_REF] Birnbaum | Towards a dimension space for musical devices[END_REF]. Unfortunately, such indicators do not appear to exist for composition support tools. During a workshop on creativity support tools [START_REF] Hewett | Creativity support tool evaluation methods and metrics[END_REF], the participants suggested several possible metrics and techniques to evaluate such systems. Their metrics include the number of alternatives attempted, the value of solutions, the time to come up with a solution or the satisfaction with the solution. They insist on the need to first get clear understanding of domain specific problems using field studies. They argue that designers should use more quantitative studies, possibly with surveys, examining the new tools. They also argue for conducting longitudinal field studies with users to complete the evaluation with real creative tasks. [START_REF] Eaglestone | Composition systems requirements for creativity: what research methodology[END_REF] discuss research methodologies to study creative aspects of composition systems. Common approaches usually include longitudinal studies to collect field data, think aloud protocol and interviews. To increase observations' validity, they argue for triangulation of different data gathering methods such as verbal discussion, video-taping and logging computer data. In a later study [START_REF] Eaglestone | Are Cognitive Styles an Important Factor in Design of Electroacoustic Music Software[END_REF], online questionnaires were used to assess hypotheses on composers's cognitive styles that emerged from qualitative analysis of data from previous studies.

To evaluate and reflect upon their software, Fiebrink et al. ( 2010) conducted weekly meetings with composers. After introducing their software, they used group discussions and questionnaires to evaluate it and to gather feedback on how to improve it. Even if their study was not grounded into realistic composition process, it generated insights about which features of the software composers value such as speed and ease to explore ideas, and led to an improved version of the software. (1998) distinguishes between two approaches for combining the strengths of paper and computers. The first one is to augment paper documents with computing features, such as digitizing the handwriting and converting it to text or diagrams. The second approach is to redesign computers with paper affordances such as mobility or penbased interaction. Mobile phones are examples of computing devices that can be carried and hand-held, usually supporting pen input with stylus.

interactive paper

Mackay

Instead of trying to combine both media in a single one, this thesis focuses on a different approach: extending computer tools with paper interfaces. The goal is to reuse existing pen and paper practices to enhance interaction with existing computer tools.

Augmented reality: reusing users' expertise

An early example that coordinates artifacts between paper and digital worlds is the Digital Desk [START_REF] Wellner | Interacting with paper on the DigitalDesk[END_REF]. Digital Desk (Figure 2.5a) is a physical desk where users can interact with computer programs through physical sheets of paper. A camera and a projector mounted in the ceiling track the user's hand and overlay digital content on the paper sheets. In this work, physical and digital are integrated rather than opposed. [START_REF] Mackay | Video mosaic: laying out time in a physical space[END_REF] used a similar approach in Video Mozaic to control video editing with paper story boards because they remarked that video producers still use such physical artifacts. 2013) conducted a study with air-traffic controllers using Strip'TIC and found that coherence between physical and digital artifacts is a major challenge for such interactive systems.

Anoto technology

The Anoto technology4 , introduced in 2001 captures notes written with a digital pen on paper. Anoto refined Dymetman and Copperman (1998)'s Intelligent Paper approach, where a pen tracks a barely visible 2D barcode printed on the page. Anoto pens integrate a camera embedded in the pen tip to decode a pattern of dots. They then transmit the movements of the pen to the computer (Figure 2.6). The dot pattern can encode an area of approximately 4.6 million squared kilometers (~73 trillion unique sheets of letter-size paper) than can then be divided into unique notebooks and pages. The main applications for this technology is the capture and storage of paper annotations with an emphasis on paper form-filling. Early versions of the digital pen store the strokes on their internal memory and transfer them to a computer via USB. More recent bluetooth versions directly stream pen events to the computer while writing. Much of the recent pen-and-paper research projects and commercial tools are based on this technology as it is mobile, inexpensive and robust.

Capturing and interpreting handwritten input

Paper poses few constraints on writing while the feedback it provides is limited. Therefore, recognizing and interacting with handwritten data can be a difficult technical challenge. Researchers have proposed several interaction techniques to help users input, edit and assign meaning to handwritten content. Some approaches such as Livescribe5 use paper buttons, palettes and menus to facilitate interaction by requiring users to explicitly select modes of interaction and recognition. The Echo smartpen (Figure 2.7) also integrates a processor, a LCD display, a microphone and speakers, providing additional interaction and feedback mechanisms. In particular users can replay the sound recorded while writing by tapping the stroke with the pen tip. This approach is similar to the AudioNotebook introduced by [START_REF] Stifelman | Augmenting real-world objects: a paper-based audio notebook[END_REF] that links handwritten content with audio recordings.

Instead of using buttons and menus to trigger commands, other projects have introduced pen gestures with predefined meaning. The PapierCraft system [START_REF] Liao | Papiercraft: A gesture-based command system for interactive paper[END_REF] proposes a set of editing gestures, such as gestures for copying, pasting, stitching, and deleting printed or handwritten content. For example, users can select and copy an image from a source to a destination document with pen gestures. PaperPoint (Signer and Norrie, 2007) takes a different perspective and enables users to control PowerPoint from a printed version of the slides. Users can select a particular slide by tapping it with the pen tip and add annotations that are displayed on the screen with the slide. [START_REF] Tsandilas | Knotty gestures: subtle traces to support interactive use of paper[END_REF] introduced subtle marks over written strokes that enable users to assign meaning and computational functionality to their notes.

Recognizing handwritten input can be extremely challenging as each user forms letters and symbols differently. For music composition, the vocabulary is vast, including text, numbers, musical scores or graphical scheme. Several recognition algorithms have been adapted or designed for pen input (Signer et al., 2007;[START_REF] Rubine | Specifying gestures by example[END_REF], but their effectiveness is usually limited to specific contexts. To deal with this issue, researchers have explored several solutions where each has its own limitations. PapierCraft [START_REF] Liao | Papiercraft: A gesture-based command system for interactive paper[END_REF] uses only few easy-torecognize gestures separated by pigtails, small loops at the end of the strokes, which serves as ink delimiters. Instead of proposing a predefined vocabulary, Musink [START_REF] Tsandilas | Musink: composing music through augmented drawing[END_REF] helps composers create and evolve their personal notations on paper over time (Figure 2.8). It provides an extensible gesture-based syntax, giving composers significant freedom to create their own composition languages and allows them to link their gestures with music programming software. However, the recognition occurs only once the pen strokes are uploaded to the computer, not when the user is writing. To support the input of music, Tsandilas (2012) uses a mobile phone displaying the results of the recognition and touch-based interactions to help users guide the recognition. Maintaining coherency between the physical and the digital world can be challenging if users need to transition between paper and computer activities or to link content across media. The A-Book [START_REF] Mackay | The missing link: augmenting biology laboratory notebooks[END_REF] is an augmented lab notebook for biologists built with a physical notebook and a PDA. The PDA acts as an information layer that lets users overlay the physical notebook with a table of contents, hyperlinks to web pages or other pages. While using this "Interaction lens" users get direct access to digital information as a layer collocated with the notebook. The PADDS approach by [START_REF] Guimbretière | Paper augmented digital documents[END_REF] aims to reduce the gap between physical and digital documents with the use of paper sheets as proxies of online document versions. Users can work on a document through either its physical or its digital representation. Annotations made on paper appear on the PDF. Users can then print the document to update its content. Musink [START_REF] Tsandilas | Musink: composing music through augmented drawing[END_REF] creates a link between annotations over musical scores and online functions. After printing and annotating a score, composer can use Musink to define the recognition of their gestures and transmit them to OpenMusic. OpenMusic then performs the corresponding musical operations. Both PADDS and Musink create a strong link between the physical and the digital version of the document. In contrast to PADDS, Musink does not maintain identical representations. Instead, composers use personal graphical gestures on paper that are translated as functions or musical elements in the digital version of the score. ButterflyNet [START_REF] Yeh | ButterflyNet: a mobile capture and access system for field biology research[END_REF]) is a mobile notebook for field biologists that captures handwritten notes and photographs. The application uses data from GPS track logs to automatically organize the captured material that can be accessed with a browser. Prism [START_REF] Tabard | From individual to collaborative: the evolution of prism, a hybrid laboratory notebook[END_REF] is an hybrid notebook for biologists that integrates differ-ent streams of information such as handwritten notes, web searches, photos or e-mails. In contrast to ButterflyNet which focuses on capturing various data from the field, Prism integrates information from both the computer and paper notebooks. Paper can be grasped, moved in space, cut or folded. Pens can also be grasped in different ways and combined with other objects such as rulers and compasses to help write. To inform the design of pen and touch interaction techniques, Hinckley et al. ( 2010) conducted a study with a paper notebook. They identified that users "frame" content on paper with the non-dominant hand to focus on a particular element.

They also observed various techniques of physical interaction with paper such as piling of small paper elements, reusing edges of sheets or objects as rulers and bookmarking with the thumb or a finger to keep track of a page while navigating in the notebook. The Anoto technology detects the precise position of the pen on different pages but does not capture physical relationships among pages such as piles or juxtaposed sheets. Stitching gestures [START_REF] Hinckley | Stitching: pen gestures that span multiple displays[END_REF] are pen gestures that span across two different pages to create a link between them. The technique has been implemented with Anoto digital pens to create links between two paper documents [START_REF] Liao | Papiercraft: A gesture-based command system for interactive paper[END_REF] or to add movable bookmarks (Figure 2.10a) [START_REF] Steimle | Designing pen-and-paper user interfaces for interaction with documents[END_REF]. Working with urban architects and designers, [START_REF] Song | The ModelCraft framework: Capturing freehand annotations and edits to facilitate the 3D model design process using a digital pen[END_REF] designed techniques for the creation of 3D models using 3D paper models (Figure 2.10b). Users can print and create paper versions of their 3D models before annotating them with the digital pen. The annotations act as commands and the digital version of the 2D model is modified according to them.

Other work focuses on adding interactions possibilities and context sensing on the pen itself. [START_REF] Song | Grips and gestures on a multi-touch pen[END_REF] proposed a multitouch pen able to detect hand grips and gestures. [START_REF] Sun | Enhancing naturalness of penand-tablet drawing through context sensing[END_REF] used a similar pen to enhance drawing with a pen and a tablet.

Paper remains a static medium and does not provide any visual feedback. Interactive paper applications usually give feedback on the computer as visual information or sound. However, this can force users to switch their focus and distract them work their actual work. [START_REF] Liao | Pen-top feedback for paper-based interfaces[END_REF] explored visual and haptic feedback with a prototype that integrates LEDs and a solenoid mounted on the pen (Figure 2.11b). Livescribe pen provides direct feedback on the pen itself with an embedded screen and speakers. Other approaches augmented paper interaction with small interactive devices, such as PDAs [START_REF] Mackay | The missing link: augmenting biology laboratory notebooks[END_REF] and smartphones [START_REF] Tsandilas | Interpreting strokes on paper with a mobile assistant[END_REF]. Song et al. (2009a) display a visual overlay over architects' plans with a movable mini projector. The projector's position is tracked with two digital pens (Figure 2.11a). The visual projection gives feedback on elements selected with the pen. Users can also articulate commands through marking menus.

Creating paper applications

Creating interactive paper applications form scratch requires significant programming effort. Programmers need to convert raw input from the pen into high-level events and render its output on the screen or send output to external applications [START_REF] Yeh | Designing Interactions that Combine Pen[END_REF]. Anoto and Livescribe have released their own software development kits (SDK). The Anoto SDK includes a plugin for Adobe Acrobat that lets end-users create, print and deploy forms. Despite its simplicity, their solution is limited to simple forms with textfields and tick boxes.

Livescribe extend this approach with an Eclipse plugin that enables developers to place arbitrary shapes on the paper interfaces and link them with pen events. Their tool requires additional Java code to interpret the pen events and create a standalone pen application. Also, both the pen application and the paper interface need to be installed on the pen before using them.

Several frameworks and toolkits have tried to aid the development of more complex paper applications. For example, iPaper [START_REF] Norrie | General framework for the rapid development of interactive paper applications[END_REF] is a framework that supports the rapid development and deployment of interactive paper applications. Programmers define active areas or specialized components and link them to specific services such as sending an mail with handwritten content. Paper Toolkit [START_REF] Yeh | Iterative design and evaluation of an event architecture for pen-and-paper interfaces[END_REF]) is a generic event-based architecture and toolkit for developing paper user interfaces. Letras [START_REF] Heinrichs | Letras: an architecture and framework for ubiquitous pen-and-paper interaction[END_REF] assists the deployment of paper applications where interaction is distributed across different input and output devices. All these programming environments allow developers to create highly specialized applications. Unfortunately, they cannot be used by end-users or musicians to create their own interfaces. For example, existing tools for music composition such as Max or OpenMusic support the creation of personal functions by using higher-level primitives that can be assembled visually. Such tools do not exist yet for paper applications or are restricted to basic interactive components.

summary

This chapter discussed work related to the design of interactive interfaces that mix the strengths of paper and software to support musical creation.

The studies of the composition process showed that computer-aided composition environments let composers implement and explore their musical ideas with real-time processes such as sound synthesis or algorithmic processes before achieving both their final scores and their computer programs for the performance. Paper remains a key medium to compose as it helps composer express ill-formed ideas with rough sketches, create personal musical representations, quickly annotate and archive final scores. Paper also prevents composer from making premature commitments and supports reflection.

Several studies investigated interfaces for sketching music using a mouse or a graphics tablet while others developed frameworks to add complex musical notation possibilities into software. Instead of proposing the replacement of one medium by another, this chapter presented interactive paper interfaces that combine computer tools with paper interfaces. Our goal is to reuse existing pen and paper practices to enhance interaction with existing computer-aided composition environments.

Such technology creates new opportunities to support the work of professional composers as it can capture the written content on paper, interpret it and provide feedback from computing devices or external applications (Letondal et al., 2007;[START_REF] Tsandilas | Musink: composing music through augmented drawing[END_REF][START_REF] Tsandilas | Interpreting strokes on paper with a mobile assistant[END_REF]. However, we still need to explore with composers how to combine sophisticated programming offered by computer-aided composition environments with the personal representations that they create and use on paper.

The next chapter presents three field studies to better understand the composition process of contemporary composers and investigate how interactive paper can support their work.

E X P R E S S I N G A N D E X P L O R I N G M U S I C A L I D E A S A C R O S S PA P E R A N D C O M P U T E R

"Of course, every musical composition involves a great deal of technique: exposition of the themes, their development, variations, polyphonic work (often very automatic), filling in the orchestration, the transitions, et cetera. Today one can compose music with a computer, but the computer always existed in composers' heads. If they had to, composers could write sonatas without a single original idea, just by "cybernetically" expanding on the rules of composition. Janáček's purpose was to destroy this computer!" [START_REF] Kundera | The Art of the Novel[END_REF] This chapter presents three field studies conducted with contemporary music composers at Ircam. First, our interviews with four composers studied how they use both paper and software tools during their creative process. Our design explorations with composers investigated interactive tools for paper expression and exploration. Finally, an exploratory evaluation of InkSplorer, an interactive paper application that links hand-written gestures to OpenMusic and Max, helped us to better understand the composition process and to explore how linking paper and composition tools can support musical creation.

Contemporary music composition is a highly creative activity that requires both musical and technical skills. Composers use software to write musical scores, create new sounds or evaluate elements of a piece via real-time processing. Such tools can serve as a testbed, providing inspiration and the ability to test and assess different musical alternatives [START_REF] Gelineck | From Idea to Realization -Understanding the Compositional Processes of Electronic Musicians[END_REF][START_REF] Fiebrink | Toward Understanding Human-Computer Interaction in Composing the Instrument[END_REF]. However, since composition-support tools mostly focus on the implementation of existing ideas [START_REF] Eaglestone | Composition systems requirements for creativity: what research methodology[END_REF], they can lead to premature commitment [START_REF] Healey | Sketching Musical Compositions[END_REF] or force composers to follow existing approaches instead of developing new ones [START_REF] Kundera | The Art of the Novel[END_REF]. As a means to "destroy this computer", composers still rely on pen and paper during the early creative phases when they need to make aesthetic choices or when they struggle to represent musical ideas (Letondal et al., 2007).

A number of systems, including Xenakis' UPIC [START_REF] Lohner | The UPIC system: A user's report[END_REF], Sonic Sketchpad [START_REF] Coughlan | Interaction in creative tasks[END_REF] or Music Sketcher [START_REF] Thiebaut | Drawing electroacoustic music[END_REF] were designed to take advantage of the power of sketching ideas, by linking drawings to music composition. However, these systems all use a mouse, a graphics tablet or other electronic input devices to draw musical forms on the screen. We want to offer composers the advantages of physical paper, with all its affordances, while also enabling them to benefit from computation.

Previous work by Letondal et al. (2007) offers insights on the roles of paper in the composition process and directions to design new interactive paper for composers. Here, our goals are to further explore the composition process, and focus on pen interactions to enhance composers' work across both media.

This chapter presents three studies conducted with contemporary music composers to understand the roles of paper and computer in their composition process and to explore how interactive paper technology can support musical creation.

study i: interviews

First, we wanted to understand the existing composition process with special emphasis on the roles of paper in the work of professional composers. We were particularly interested in how composers switch between paper and computer tasks. We conducted interviews with four composers at Ircam.

Method

Participants

We interviewed four advanced composition students. All are experienced composers who have won prizes for their compositions. All have studied computer-assisted composition with software tools including OpenMusic, Max and Audiosculpt. They are all male, aged 30-40. We identify them by their initials: NM, AE, EM and MB.

Procedure

We conducted semi-structured interviews with each composer in March and April 2010. As part of their composition program, all participants were finalizing a composition with electronic elements for a soloist. We asked them to bring this piece in addition to their personal computer and other related documents. We began by asking them to describe their current project and discuss how it evolved in both paper and electronic forms. We were particularly interested in learning how the representations that they used changed over time and across different media. We asked Critical Incident-style questions [START_REF] Flanagan | The critical incident technique[END_REF] with recent concrete examples of how they addressed problems, followed by more general open-ended questions. Finally, at the end of each interview, we demonstrated a Livescribe pen which records sound with playback and provides auditory and visual feedback. We asked participants to brainstorm on how such technology could as- Each interview was recorded and lasted approximately one hour. We transcribed and analyzed each interview using a grounded theory [START_REF] Corbin | Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory[END_REF] coding approach, along with photographs and copies of their sketches and scores.

Results

As with every highly creative activity, music composition is characterized by a wide variety of methods, tools and strategies. We observed that each composer had developed a unique work practice involving both paper and software. This section summarizes the main themes that emerged during the field study. It first presents how participants use paper to express ill-formed ideas with sketches and create personal representations. It then studies how participants cycle between paper and computer activities to explore their ideas. It reports on the iterative nature of the composition process in which composers use several musical representations until the final scores. Finally, it reviews their feedback and suggestions on future interactive paper tools.

Expressing ideas on paper

We observed that even though they are experienced users of composition software, all use paper to express their earliest musical ideas. Each has a unique way of working that varies in form and style.

Early sketches

Some begin with blank paper to quickly sketch ideas, either with graphical notation, musical symbols or text. Figure 3.1a shows a sketch made by NM to represent the structure of the whole piece. He uses a personal set of symbols and textual elements to express several phases of the piece. Even if the final piece does not respect exactly this sketch, he used it as a starting point. Instead, AE uses textual NM also uses a notebook with empty staves to record his musical ideas (Figure 3.3a). In contrast, MB uses only blank sheet of paper at the beginning with drawings representing instruments and sound in space (Figure 3.2) and short musical elements for which he adds handwritten staves (Figure 3.3b). He explains that he always uses blank paper to begin a composition, often the verso of previous scores or other documents.

Sketching electronics

To express musical content related to physical instruments, our participants frequently use western music notation. However, when they have to express ideas related to electronic parts, they use additional notations including textual descriptions or graphical representations. The graphical representations can contain various graphical parameters, e. g. scale, color settings, orientation, envelopes, and thickness, which are mapped to musical parameters, often in a vague and not detailed manner. MB combines drawings and musical notation to specify rhythmic properties and sound properties (Figure 3.3b). NM draws bells to represent sound synthesis evoking bell sounds (Figure 3.3a). He adds the symbols directly into the score.

Figure 3.4 shows the graphical representations used by AE to represent the structure of the piece and to detail its electronic process. He uses boxes to represent the parts of the piece and draws lines between them to describe an electronic parameter that will evolve during the piece (Figure 3.4a). AE first draws several alternatives for the structure of the piece with few details. Once he finds an interesting structure, he creates a more detailed sketch that will serve as a reference to implement the electronic processes in Max (Figure 3.4b).

Our participants explained that when working with electronic processes, they also need to focus on spatialization, i. e., the location of the sound in space. Spatialization requires to position sound sources in space and to define the loudspeakers configuration. speakers in the concert room. The composer used that information later to configure the spatialization engine.

Structures and data

Our participants distinguish sketches, which represent specific ideas, from underlying frameworks, which provide structure for their representations. For example, some composers use preprinted musical staves or draw them on their own; others use graph paper or specialized grids to lay out their ideas. This allows both flexibility and control when expressing concepts such as time, duration, pitch and density. Several participants explained that they need to find appropriated structures in which they can express their musical ideas to move from early sketches to their systematic exploration.

Exploring ideas on both paper and in software

After expressing their initial ideas on paper, composers move to an exploration phase, which involves both paper and the computer. AE and EM use OpenMusic and Audiosculpt, combined with hand-written scores to experiment with ideas. NM and MB use Finale and Sibelius (music editors) to produce the final score, after first testing and printing some ideas on paper. They first explore their ideas using Open-Music. Later, they export the results to midi files or to a musical score format to continue their work with Max or with a musical editor like Finale.

Annotating scores

All participants annotate printed or copied scores. AE and EM both handwrite their scores. They annotate and make corrections over previous versions of the score before creating a new cleaner version.

Exploring electronic processes

All of the composers use different electronic processes in addition to the instrumental part of their scores. AE and MB use real-time algorithms to control sound processing. AE, EM and NM use spatialization techniques and EM and MB use real-time synthesis. Regardless of their technical expertise, all move back and forth between paper and computer interfaces, sometimes drawing multiple curves on paper that they next test on the computer, sometimes sketching an idea on paper that was inspired by a computer-generated sound. NM, AE and MB sketch some patches on paper before implementing them with their programming tools. Figure 3.7 shows the paper version of a patch with curves as input for the patch and indications for expected output. It also presents its final implementation in OpenMusic. The online version was significantly different as it evolved and changed several times while the composer implemented the algorithm and explored the resulting musical sequences.

An iterative process between paper and computer

All our participants engaged in an iterative process to achieve their final scores printed on paper and create programs for the electronic processes.

Evolutions of musical representations

During the composition process, the musical representations evolve from early sketches to final scores. For example, Figure 3

Expression, Exploration and Execution

We found that all of the composers engage in three main activities: expressing an initial idea, exploring it, and finally executing it in a composition. This cycle of expression, exploration and execution is highly iterative and occurs on both paper and in software, although paperbased activities occur earlier and end later. Letondal and Mackay (2007) found similar results during their studies with composers and proposed a chronology of paper and computer use (Figure 3.9). In particular, they identified distinct roles of paper. First it serves as a temporary medium to input ideas and supports quick changes. Second, it is a static medium that captures the final state of the score. In contrast to this model, we argue that when switching to computeraided composition environments, composers progressively reduce their paper-based activities. Composers often abandon their personal representations in favor of the existing ones of computer environments. Paper is clearly more flexible than software, demanding fewer constraints when expressing an ill-formed idea. For example, some composers use sketches to represent the structure of the whole piece or, like Marco Stroppa (Letondal et al., 2007), use graph paper to draw extremely precise curves. When they move back to software, some paper-based representations get lost or must be translated into classical notation, which acts as a common language between paper and electronic representations. This runs counter to a Resnick et al.'s (2006) suggestion that "creativity support tools should seamlessly interoperate with other tools". Here, composers must shift between two methods of exploring ideas, which forces them to stay conscious of the medium and distracts them from the idea itself.

Feedback about interactive paper technology

All the composers were interested in automatically transcribing their handwritten scores into musical editors. They were also fascinated by the possibility of listening to parts of a score directly from the pen. EM wished that "the pen could play notes as I am write them on paper". NM explained the he would like to "print scores and make corrections with the pen". MB and NM also proposed using the pen to ease the creation of OpenMusic and Max patches by sketching them on paper similarly to SILK by [START_REF] Landay | Conference companion on Human factors in computing systems common ground -CHI '96[END_REF] which allows users to quickly sketch an interface. All the composers found the Livescribe pen too large and uncomfortable for daily use. They also commented that they frequently use pencils, not pens, and erasers.

The composers also suggested more specific examples related to their personal compositions. For example, AE has been working with a poet for several years and created music from his poems. He envisioned that he could directly relate the handwriting to real-time audio synthesis with a digital pen. In his scenario, he would map the pen gestures to audio parameters such as pitch or amplitude and use the recognized letters to filter the sound. MB suggested that he could reuse the small gestures he sketched on paper to control musical processes in OpenMusic. He would map graphical features of the shapes such as size or orientation with online parameters.

As MB was enthusiastic about the interactive paper technology, we decided to go beyond this initial brainstorming session and conduct participatory design sessions with him. The next section presents our study with MB.

study ii: design explorations with a composer

We conducted an exploratory study with one composer (MB) to investigate how interactive paper could help interactively explore musical ideas on paper. The composer was willing to use interactive paper in his own work. We wanted to identify scenarios grounded in realworld composition activities.

Method

MB just finished his piece described in the previous study (Figure 3.7 & Figure 3.8) and was working on a new piece for saxophone and electronics.

We first conducted a two hour participatory design session in June 2010 followed by four shorter meetings between September and December 2010. We started with a short interview and we created paper prototypes to explore scenarios inspired by examples from his current and previous work. We gave him a Livescribe pen and a notebook paired with an online account to track his use of the pen. We then created a video prototype to explore how to implement some of the ideas suggested by the composer.

During the following meetings, we reflected on several working prototypes designed for the composition of his piece. We based our prototypes on the Livescribe pen as it runs Java ME programs (penlets) and offers a range of functions, including auditory and visual feedback, audio recording and replay, interactive buttons and special areas printed on paper.

Results

MB used the digital pen for the composition of his new electro-acoustic piece. He offered a number of insights on how he switches between expression of rough ideas and experimentation, both on paper and in computer environments.

Free expression on paper

During our first design session, we worked on a scenario for the composition of his current piece. MB begins with sketches and gestures instead of classical musical notation so that he can improvise and work "at the speed of the thought". Figure 3.11 shows his score on paper to define a part of his piece. As we asked about feedback, he explained that providing live feedback from the pen would be too intrusive or distracting during the early stages of his work when he thinks about a new idea. He is, however, interested in automatically translating paper-based gestures into classical notation that can then be interpreted by Finale. For example, the gestures drawn on paper could be first transmitted to OpenMusic and converted into score elements that he could modify before creating a score in Finale. This would save him time and let him focus on expression rather than execution of ideas.

Experimenting and refining musical ideas

After expressing his ideas, MB suggested that he would use the pen to compare several alternatives and refine the most interesting ones. For example, he added delimiters and labels above the initial gestures in his score (Figure 3.11) to be able to refer to them in other parts of the score and quickly compare them in the computer. He created variations by associating new rhythms to existing elements. To create musical sequences involving complex chords for his current piece, MB used the Livescribe pen to record the sound associated with the chords while writing their labels. Figure 3.12a shows the session recorded by the pen. The labels then acted as buttons from which he could listen to the chords with the pen. These interactive elements helped him find interesting chord progressions associated with handwritten rhythms on paper. However, writing the chords, playing the sound files and recording them while writing the chords' labels was fastidious. Therefore, he did it only for four chords out of twenty. To address this issue, we built a dedicated paper interface (Figure 3.12b) to support his experimentations with all the chords and their associated recordings. The boxes contains multiphonic tones for his piece that MB printed from OpenMusic. This prototype lets MB reflect on each sound while working on paper, using a penlet to replay sounds at will.

We found that live interaction with the pen is not recommended for early expressive activities as it can disturb the composer. However, interactive paper could provide the following useful functions during the exploration phase:

1. Record and play sounds by interacting with drawings or printed musical elements (as in Figure 3.12).

2. Evaluate and refine the result of drawings and gestures drawn on paper.

3. Define and modify rhythms and dynamics.

4.

Restructure a piece by indexing different segments of the piece and exploring new structural alternatives.

Video prototype

We used a video prototype1 to concretize how to implement some of these ideas.

scenario: David Inker, a young composer, works on a piece for piano, trombone and electronics. He uses an application combining a Livescribe pen, a dedicated notebook and OpenMusic to compose. He can convert his drawings into musical elements that could be later processed by OpenMusic. The first sheet (see Figure 3.13a) contains a space for the initial expression of ideas (gestures, musical symbols and drawings) and a separate "interaction space" that runs in parallel, along a common timeline. The interactive timeline allows the composer to obtain information about the available gestures on the pen's screen, refine recognition and define rhythms. For example, similarly to knotty gestures [START_REF] Tsandilas | Knotty gestures: subtle traces to support interactive use of paper[END_REF], the composer can draw a dot on the timeline to edit the pitch of the closest gesture. The second sheet (see Figure 3.13b) contains boxes to draw several examples of the gestures to improve their recognition.

study iii: understanding exploration of composition processes with interactive paper

We wanted to understand whether and how interactive paper could assist composers' exploration activities with their computer-aided composition environments, OpenMusic and Max in particular. Our goal was to enhance the computer-based exploration phase by providing additional physical space on paper for reflection, expression, evaluation and refinement of ideas. We designed InkSplorer, an application that lets composers write curves on paper and reuse them as control data in their software. InkSplorer is a palette of tools, not a single prototype. This supports a technology probe [START_REF] Hutchinson | Technology probes: inspiring design for and with families[END_REF] approach, in which our goal is not to validate a particular design solution, but rather to develop tools that composers can easily adapt to meet their individual needs. We hope to both gain new insights about the composition process as well as generate new ideas for designing interactive paper technology that supports the creative process.

InkSplorer

InkSplorer is an application that links hand-written strokes with computer composition tools. More specifically, it allows composer to use drawn curves on paper as input for OpenMusic and Max programs.

InkSplorer uses wireless Anoto ADP-301 pens that detect position and low-precision pressure. Pen data is sent to the computer via Bluetooth. Since drivers were not available for Mac OS X, we redirected pen data from a Windows 7 virtual machine to OpenMusic and Max. We used the OSC [START_REF] Wright | Open Sound Control: A New Protocol for Communicating with Sound Synthesizers[END_REF] communication protocol (fully supported by both Max and OpenMusic) to transmit pen events. We created a library to manage storage and efficient retrieval of data so we can support real-time interaction with strokes on paper. The library was implemented as a Java external for Max. We also implemented patches and libraries in Common LISP for OpenMusic and Java for Max, to facilitate the integration of paper tools into composers' personal workspaces. The user interface in Figure 3.14 is a Max patch that lets users launch paper-aware applications and control the pen configuration. The patch uses Jitter's OpenGL rendering to display incoming pen strokes.

Our implementation allows to use the pen event in real-time and to retrieve previously drawn strokes when the user taps on it with the pen. We created patches that detect and communicate various stroke properties: the x-and-y coordinates of each data point, data point density, pen pressure and time-stamps for each point. We can thus detect writing speed and variability throughout the length of a stroke. From the user's perspective, InkSplorer provides a direct link between stokes on paper and computerized objects. We developed a set of mini-applications of InkSplorer on top of our libraries that integrate interactive paper into Max and OpenMusic:

1. A Max patch that maps the live pen events to a synthesizer inspired by the Theremin. Moving the pen on paper controls two synthesizer' parameters. In our example, we mapped the vertical position of the points to the pitch and their horizontal position to a ratio between two waveforms (see Figure 3.15). user taps on a stroke with the pen, the patch loads the stroke as an envelope, and plays the result. 23. OpenMusic patches that map strokes to BPF and BPC objects (Figure 3.16b). Custom paper templates facilitate drawing and scaling of strokes. Again, each time the user taps on a stroke with the pen, the content of the patch is changed using the selected stroke as input.

4.

OpenMusic patches that convert multi-strokes into musical objects using the maquette representation and custom paper templates. Each stroke created on the page is converted into a temporal object and placed into the maquette according to its position on the page that defines it onset, duration and vertical extent.

5.

Pen-drawing support for bach3 , a Max tool that enhances realtime processing with advanced musical notation. Duration and amplitude profiles of notes can be drawn on paper (Figure 3.17).

Use scenario

The following scenario illustrates a possible use of InkSplorer. The scenario has been inspired by our study with MB. Maria, a composer, is working on a piece for piano and real-time electronics. she has a clear idea of an electronic sound in her mind and captures it on paper in the form of a rough, abstract sketch with some text. She then creates an OpenMusic patch and proceeds to work out how to implement the sound. She inserts a BPF object to control the pitch range and turns to InkSplorer to explore different variations. She draws four curves on paper, singing the sound to herself as she draws. She taps on each curve and listens to the corresponding associated sounds produced by OpenMusic. Maria likes the result of the third curve but decides to change its last segment. She draws several slightly different variation on top of the curve and settles on the second variation. She adds an annotation to remember certain decision details, and circles the curve. OpenMusic stores the circled curve. The composer also saves the original rough sketches and an OpenMusic printout in her notebook.

Method

We wanted to understand how interactive paper could enhance the computer-based exploration phase by providing additional physical space on paper for reflection, expression, evaluation and refinement of ideas. We conducted an exploratory evaluation of InkSplorer with four composers at Ircam in december 2010 to investigate real-world examples where InkSplorer could assist composers experimenting their composition processes with drawn curves. In this study, we used InkSplorer as a technology probe [START_REF] Hutchinson | Technology probes: inspiring design for and with families[END_REF] to help composers reflect on both their practices and on the possible benefits of such technology. Our goals were to evaluate InkSplorer through composers' work, get more insights about their needs and identify new directions for creativity support tools.

Participants

In addition to MB from the previous studies, three professional composers, KH, GL and MM, aged 31-52, agreed to participate in the study. KH, MB and MM were already familiar with the basic Anoto technology. All were expert users of Max and OpenMusic. KH and MM had participated in the development of OpenMusic. GL and MM both teach computer-aided composition at Ircam.

Procedure

Each session lasted approximately two hours. We asked our participants to bring their personal laptop and related documents, including musical scores, drafts of finished or in-progress pieces, and patches in Max and OpenMusic. All sessions were videotaped and later analyzed. Section A.1 presents the study's material, including the cover sheet and the questionnaires.

We first asked each composer about his background, his professional activities, and his experience and frequency of use of different music-composition tools. We then conducted a 30-40 minute semistructured interview, focusing on how they represent and interpret curves and graphical forms, both in software and on paper. We asked for at least three specific examples and asked them to explain in detail how they worked out details, e. g. "Describe the parameters this curve represents." These interviews helped us to understand their work in context and identify concrete scenarios in which drawing curves on paper could be augmented with software functionality.

Next, we explained how to use InkSplorer and the mini-applications described above. Together with the composers, we selected examples from their work and imported their workspaces or parts of them to our laptop computer, where the pen drivers and InkSplorer had been installed. We successfully imported the OpenMusic workspace for three composers but not for KH, due to software version incompatibilities. 4We asked composers to reflect upon how InkSplorer might change how they define, explore or refine musical parameters on paper and in OpenMusic or Max. We encouraged them to draw with the pen and use a "think-aloud" protocol to describe its strengths and weaknesses. At the end of each session, we asked them to give us their reactions to InkSplorer as well as any suggestions they had for future designs.

Results

All four composers use OpenMusic, but only MB and MM use Max for composition. The other two use Max for synthesis and interactive performance. These composers demonstrated diverse uses of curves to control various processes. For example, KH uses short curves to control an individual localized component of an algorithm or a synthesis process. Figure 3.18 (left) shows his use of a short curve to define a synthesis envelope or a pitch variation for granular synthesis. KH made a strong distinction between sound synthesis and music composition: For him, drawing curves to control synthesis, whether on paper or in software, is interesting, but he insisted that he is not a "painter" and does not use curves to compose music. In contrast, MM uses long curves to control global properties of a piece or a section. Such curves are often more complex and more precise than short ones. The composers all chose to explore examples derived from their use of OpenMusic. The following issues concern both interactive paper in general and OpenMusic in particular.

Expressing ideas

Composers varied in how well paper helped them to express musical ideas. For GL, musical ideas reside in computerized patches and InkSplorer is only potentially useful for exploring these ideas faster. In contrast, MM feels that paper is simpler and more intuitive. For him, paper forms an "analog" space that provides more possibilities for expression than the computer, which he finds "digital" and constrained. MB finds the expressive power of both media to be similar, although he enjoys working with the pen more. He treats it as a musical instrument that involves physical movement of the body, a tangible sensation as the curve is drawn on paper: "[I] use this pen just as I do an instrument. Here, I play the pen."

Exploring ideas MB and GL stated that speed is a major strength of interactive paper: it enables them to register multiple ideas and quickly assess their potential. MB feels that the pen saves time and helps him focus on the musical outcome rather than how to implement it. His hand-drawn gestures act as memories of sounds that can be returned to and replayed, even though the actual implementation resides on the computer. He explained that with interactive paper, "The work is not lost in the computer". He also notes that computer screens have limited screen space whereas paper offers almost infinite space to explore ideas and spatially arrange them.

Composers discovered interesting strategies for exploring ideas with InkSplorer. For example, MM drew several long curves on top of each other to evaluate different alternatives in the afore-mentioned composition, each providing incremental corrections (Figure 11,left). He used layers of curves to guide each refinement, explaining, "It's a kind of guide that lets you correct it next time". In Figure 3.20 (right), MB draws variations of a short curve to control a 2-second sound synthesis.

Precision

Composers have different views about the relative amount of precision offered by paper and computers. MM feels that the computer is more precise because it lets him enter exact values, whereas data entry on paper is rougher. In contrast, GL finds that drawing on paper is more precise and lets him produce "more complex results". Finally, MB argues that although paper affords higher precision when drawing curves, it is not necessary for his compositions.

Design issues

The composers agree that integrating paper directly into existing tools, rather than creating a new interface, is the correct approach. However, they also want richer forms of interaction. For example, MM finds it difficult to draw long curves without lifting the pen for a pause. He suggests that we let users easily connect segments together.

Interestingly, only MM feels that capturing pen pressure or drawing characteristics such as pen angle are important, because these are essential in calligraphy. MB suspects that pressure might be useful, but would require practice to be controlled effectively. MM, GL and KH are particularly interested in using special pre-printed paper templates, particularly graph paper and musical sheets. In contrast, MB wants to create his own paper interface. The composers offered various suggestions for improving the pen design, including making them thinner, offering color, and supporting pencils or at least some form of erasure.

discussion

The three studies with composers at Ircam let us better understand composers' creative process and how they cycle between paper and computer-aided composition environments to explore their ideas before achieving the final score. Using participatory design methods, we explored the strengths of interactive paper with several composers. Here, we summarize the main results and implications for design.

Working on paper

Composers frequently start with paper instead of moving directly to the computer. First, paper serves as a flexible personalization tool. It can support very rich representations of musical data, especially spatial and graphical forms that composers frequently use. Second, pen and paper are especially suitable for quickly recording ideas, exploring alternatives and annotating them for future assessment or reference [START_REF] Neiman | Sketches and their functions in early design. A retrospective analysis of a pavilion house[END_REF][START_REF] Sellen | The Myth of the Paperless Office[END_REF].

All our participants use traditional music notation in addition to personal representations on paper. Graphical representations play an important role for the early expression phase as sketches can represent ill-formed ideas, allowing composers to flexibly change their mind or to explore only one aspect at a time. EM explains that "a note written on a staff can sometimes be too suggestive and directly associated to a particular sound". As [START_REF] Healey | Sketching Musical Compositions[END_REF] observe, sketching musical forms on paper helps composers to avoid premature commitments. However, we also observed that music-programming tools have influenced the way composers work on paper (Figure 3.7). Composers often invent new representations to describe data and structures of computerized objects and mix them with their own notations. They can reflect on variations and adapt their representation to their evolving composition process. This is usually hard or impossible to accomplish with existing computerized tools.

Extending computer-aided composition environments

We designed several interactive paper interfaces to support composer's experimentations across paper and computer. During Study II, MB used a pen application to play sounds created with OpenMusic and compare alternatives. Study III, showed that composers could successfully adapt InkSplorer to their composition styles and control their existing patches in Max with curves drawn on paper. Composers emphasized that InkSplorer let them quickly explore their computerbased musical processes on paper, refine and compare several alternatives by replaying and layering existing curves.

Common augmented reality approaches such as PenLight (Song et al., 2009a), focus on how to extend paper with computational features. Here, we examined the opposite direction, i. e., enhancing exploration on computer-aided composition software with pen-based interactions.

Implications for design

The studies with composers highlighted that interactive paper interfaces could support composers' creative process and generated several design opportunities. Here I summarize the key observations that led to our next design efforts: interoperability with composition environments: In all the examples suggested by the composers during the interviews, interactive paper interfaces are connected with composers' computer-aided composition tools such as Max, Finale and Open-Music. For example, NM wanted to print scores from OpenMusic and edit them directly in OpenMusic from pen interactions.

In study III, composers appreciated to use the drawn curves directly in their existing softwares and explained that using a new separated tools would not be appropriated. A complete integration of interactive paper interfaces with computer-aided composition tools should let composers deciding when using which medium is appropriate for a given task [START_REF] Resnick | Design Principles for Tools to Support Creative Thinking[END_REF].

personal interfaces Composers each follow a unique process and use personal representations on paper. During Study II, we participatory designed prototypes for MB that included his own musical data created in OpenMusic. Such interfaces would not be appropriate in their current forms to another composer nor for a different musical task. In Study III, composers wanted to design their own templates to help the entry of their curves. They all wished they could also use additional musical representations on paper, western music notation as well as their personal representations.

summary

In this chapter, we examined how professional composers combine paper and software tools and how they could benefit from interactive paper technology. Initial interviews with four composers helped us understand their composition process, from the early expression of ideas to their systematic exploration and final execution. We found that composers cycle between paper and software to achieve their final scores and musical programs. These activities take place in parallel but they would like to better integrate them. In particular, when they switch to computer exploration, they progressively reduce their use of personal representations on paper, including the more formal ones.

The second study with a composer studied how interactive paper could better support his iterative testing of musical ideas. We found that even if live interaction with the pen is not recommended for early expression, it can assist the composers to explore ideas by recording and playing sounds or interacting with drawn elements to evaluate and modify them.

The third study created InkSplorer, a pen-based composition tool that links paper-based and software-based exploration to facilitate testing and refinements of ideas. We used InkSplorer as a technology probe during an exploratory evaluation with four composers to get insights about their creative process, the technology and to generate new design directions. Composers adapted InkSplorer to their personal composition style. Their gestures on paper served as visual and computational elements that could be quickly revisited, replayed and evaluated, as well as layered and refined with new variations. They were all interested in using addition paper templates as musical sheets or graph paper.

We discussed the need to create personal paper interfaces integrating composers' musical data that could extend their computeraided composition environments with pen-based expression and exploration. The next chapter presents a participatory design study that investigated the design of specialized paper components that can accept composer's musical data.

contributions

1. Refinement of an existing model of paper and computer use during the composition process. We found that composers often abandon their personal representations when they switch to their computer-aided composition environments.

2. Investigation of interactive paper tools for exploration and exploration of musical ideas with a composer using low and high fidelity prototypes.

3. Implementation of InkSplorer, that allows composers to use drawn curves on paper to control musical processes and to compare alternatives in Max and OpenMusic.

4. Implication for the design of personal interactive paper interfaces that augment existing computer-aided composition environments. Free expression on paper should not be interrupted while exploration requires interaction with drawn elements and musical feedback.

PA R T I C I PAT O R Y D E S I G N O F P E R S O N A L S T R U C T U R E S O N PA P E R
"Well in art as in everything else, one can build only upon a resisting foundation: whatever constantly gives way to pressure, constantly renders movement impossible. My freedom thus consists in my moving about within the narrow frame that I have assigned myself for each one of my undertakings." [START_REF] Stravinsky | Poetics of Music in the Form of Six Lessons[END_REF] This chapter presents a participatory design study to explore the creation of personal interactive paper interfaces linked with computerbased composition tools. It describes existing examples of structured musical representation systems created by composers on paper to simplify input, calculation and reflection on their musical ideas. It then reports on a participatory design sessions with four composers and a musical assistant at Ircam to investigate how they would apply such structures in their own work to control their computer-aided composition tools.

The studies presented in Chapter 3 and previous work (Letondal and Mackay, 2007;[START_REF] Healey | Sketching Musical Compositions[END_REF][START_REF] Agon | The OM composers[END_REF] showed that each composer uses personal representation on paper and creates sophisticated programs to experiment with their musical processes. Therefore, a solution that works for one composer may not be suitable for a different one. We need to find abstractions out of the diversity of their individual work processes that will let us design tools that could be easily adapted to their needs. [START_REF] Stokes | Creativity from Constraints: The Psychology of Breakthrough[END_REF] argues that composers, like other creative practitioners, create constraints that allow them to consider new creative directions. For example, Schoenberg's twelve-tone technique1 led to serialism, in which composers use series of values to manipulate musical elements such as pitch, rhythm or intensity. [START_REF] Stravinsky | Poetics of Music in the Form of Six Lessons[END_REF] explains that he builds upon a "resisting foundation" to compose. Such constraints can also be expressed in the form of musical representations systems that integrate musical rules. For example, the five horizontal lines of the staff define a set of constraints to represent music and work with common musical notation symbols.

Although we observed that paper is used for sketching rough and early ideas, several composers use paper to define structured and well formed musical representation systems, other than simple staff, that allow for the visual exploration of musical parts or the definition of complex electronic processes. We found that when composers switch to computer, these personal representations are less used, partly because the transition with software is more difficult. [START_REF] Coughlan | Understanding productive, structural and longitudinal interactions in the design of tools for creative activities[END_REF] argue that creative tools should support structural interaction, letting users create the structure in which a creative outcome can be produced. As these representations are more structured than sketches, they can be computerized and used within interactive paper interfaces to let composers create their own paper interfaces.

This chapter examines two examples of existing structured musical representation systems that combine musical rules and ad hoc representations on paper. A participatory design study with four composers and a musical assistant investigates how composers could create their own interactive paper interfaces, linked with their computeraided composition tools, using such structures.

two examples of musical representation systems

This section describes two examples of formal musical representations systems created by contemporary music composers. In contrast to the examples we observed in the interviews, these systems used has been used and refined over the years by the composers. They became an integral part of their composition processes.

Jean-Marc Chouvel use of Tonnetz

Chouvel's work, entitled Traversée, offers a compelling example of how he used a Tonnetz (German word for "tone-networks") as a tool for exploring composition ideas. A Tonnetz is a two-dimensional network of pitches or chords. Tonnetz layouts are organized with three main directions (diagonal-right, diagonal-left, vertical), each associated with a constant interval. In the neoriemannian Tonnetz (see Fig- ure 4.1), these intervals are the major third, the minor third and the fifth. This Tonnetz is suitable to create and analyze chord progressions.

After creating and printing a Tonnetz on paper, Chouvel drew differently colored paths through the hexagonal cells, each representing a set of pitch sequences. He called the resulting geometrical shapes "constellations" because they look like the line drawings that connect individual stars in a map of the night sky. Once drawn, the composer was able to perform geometrical transformations on the paths and examine the results, represented as notes on a standard musical score. Figure 4.2 illustrates this process. The path that link different pitches can be treated as a sequence, to create a melody or a chord. Chouvel's approach makes it possible to explore a wide variety of ge-ometrical transformations, including translations and rotations. His Tonnetz structure can thus support complex musical operations that are rarely formalized in traditional music theory.

Philippe Schoeller's micro-orchestration

Philippe Schoeller uses a graphical system called micro-orchestration, which is an extended musical staff that captures the whole range of pitches of all the instruments in an orchestra. Figure 4.3 shows the single score he used to sketch the orchestration of one of his pieces.

This graphical system also allows him to allocate parts of the score to specific instruments and then visually assess his choices. At the end of this process, the musical content is transcribed to regular scores for orchestra with multiple staves dedicated to each instrument.

These two examples illustrate how the two composers used graphical representations on paper to visual assess and produce musical sequences that would not have been possible with common music notation, i. e. the musical score. In both cases, we can note that composers lay the musical data over a well-defined structure, the Tonnetz and the micro-orchestration score. These structures simplify input and calculation while they help the composers to think over their piece, explore solutions, and annotate them with the pen. After sketching their ideas within these structures, the composers transcribe the results into regular musical scores.

participatory design study

We wanted to investigate how we can reuse this principle of structured musical representations to design interactive paper interfaces that help contemporary composers work with complex musical data. We conducted a participatory design study with four individual composers and a musical assistant at Ircam between July and September 2011. Our goals were to understand what types of data can be transferred from the computer to paper, how these data can be represented and structured, and how users could interact with them to accomplish challenging tasks as part of their music creation process. We used paper and video prototypes to explore the potential of interactive paper interfaces through real-world scenarios that involve the manipulation of diverse and complex musical data.

Participants

Five participants, all men, aged 25 to 41, agreed to participate in the study. We refer them as: MB, MGV, RD, MK and GB. Two composers were graduate students in a computer-assisted composition program. The first (MB) had completed the first year of the program and worked as a freelance composer. The second (MGV) was enrolled in the second year of the program to create a piece for ensemble produced by Ircam. RD followed a technical degree at a national music academy in audio engineering. MK conducted research on computer-assisted composition in parallel to his composition activity.

Finally, the musical assistant (GB) was a music-literate programmer who helped composers produce the electronic parts of their pieces.

Procedure

We asked participants to bring their personal laptop and scores they were working on. We provided blank, colored and translucent sheets with various dimensions and colors, and millimeter paper. For some participants, we printed score templates that they chose for the purpose of the design session. Several pens, pencils, glue, scissors, postits or tape were available to participants. We met with each participant individually. After a brief introduction about inter active paper technology and our goals, we encouraged participants to identify scenarios from their previous work for which interactive paper might be useful. We began with questions about the kind of elements they would be interested in having on paper. We then asked about how these objects should be represented and used on paper and the computer. Participants described their work process and sketched solutions on paper. We then worked together on the design of an interactive interface for a scenario of their choice and prototyped it with regular paper. At the end of each session, we created a five-minute video prototype [START_REF] Mackay | Video artifacts for design: bridging the Gap between abstraction and detail[END_REF] to capture how the participant would use the interface with his own data. Participants acted out how they would like to interact with the pen. Each session lasted 60 to 100 minutes. 2 Section A.2 presents the cover sheet used during the study.

results

Participants proposed several design ideas and paper prototypes during the design sessions. We collaborated with the composers to create a video prototype for each participants to illustrate how he would like to use an interactive paper interface to address a particular problem in his composition process.

Creating a musical sequence and its orchestration

MB writes music and creates specialization effects for a horn quartet (Figure 4.5). He starts his work on the computer with the definition of a personal score template in Finale, a score editor. He decides to explore his ideas on paper. He uses small, preformatted strips of paper to define each parameter of the music (rhythms, pitches, specialization) independently, where each piece supports a specialized notation, e.g., whole notes for pitches and circular positions for specialization. The composer can place these strips anywhere on his score page. In order to logically associate two strips together, he aligns them on the page and draws a link with the pen. He uses strips of translucent paper to create physical superimposed layers of musical symbols. This allows him to reuse elements of his music, explore alternative solutions, and easily reorganize them in his score. It also lets him create more complex musical structures with multiple layers of musical content.

Exploring alternative orchestrations

MGV works with Orchidée [START_REF] Carpentier | Interacting with Symbol, Sound, and Feature Spaces in Orchidée, a Computer-Aided Orchestration Environment[END_REF], an automatic orchestration tool that generates alternative scores given a sound, a list of instruments, and a collection of sound features (Figure 4.6). He defines the target sound and additional parameters, e.g., the orchestra. Given this configuration, he prints a paper interface with specialized interactive sections with basic information about the configuration and interactive elements such as check boxes to select from options. He interacts with the pen to define a configuration for the software and an orchestration's algorithm. From then, the composer creates new pages that contain the space of generated solutions and the options to filter and evaluate alternatives. The interface also contains empty musical staves where the handwritten musical content is stored but not translated by the computer. The written notes serve as a musical annotation on a selected orchestration solution.

Once the composer has explored a solution, he prints a new version of the interactive page with updated data from the computer. Finally, he stores his paper workspace in a physical folder, keeping track of all the explorations for this part of his work. His goal is to edit the recording directly from the score by modifying the pitch, the rhythm and the volume of certain parts. This is not feasible on the computer as the recording and the score are handled by different software programs. RD prints his interface on paper that groups the score together with the sound waveform. A list of formatted zones let him edit several audio parameters. For example, to alter the pitch or the volume of a musical segment, he draws curves in the corresponding specialized zones. He vertically layers all the editing zones on the same page such that they all share the same timeline. The composer explained that this layout makes it "easier to organize [his work] on pa-per and helps him structure the data". He also uses vertical lines and boxes as visual constraints for his edits. In addition to these elements, the interface contains printed controllers that appear near the specialized zones. The controllers allow him to test values of the associated parameters before making any edits.

Composing with programmable composition modules

MK uses PWGL [START_REF] Laurson | An Overview of PWGL, a Visual Programming Environment for Music[END_REF], a visual programming environment, where he has implemented several composition elements, e.g., composition rules and transformations. His goal is to reuse them on a paper interface (Figure 4.8) to create a new musical piece. He uses several pieces of paper with printed specialized components. Each piece represents a reusable module that groups components by functionality. For example, a module contains areas for drawing graphs, serving as a library of graphical definitions of musical functions. A second module provides a list of composition rules and transformations. An additional module with multiple staves allows him to define intervals. By combining different rules and transformation properties through the modules, he creates the sections of his piece. He also uses a module with staffs to input intervals with common notation. Finally, he composes the overall structure of the piece by specifying the sequence of musical sections over an interactive timeline.

4.3.5

Generating "sound textures" with 2D trajectories GB is a musical assistant who was helping a composer to create sound textures, i. e., temporal evolutions of sounds such as the rain or the forest, with CataRT [START_REF] Schwarz | Real-time corpus-based concatenative synthesis with catart[END_REF]. To create the sound textures, he has to navigate within a multidimensional space of thirty sound descriptors of a database of voice excerpts. Unfortunately, the software does not allow him to keep track of past explorations, which are therefore easily lost, as he has to control a large number of descriptors. GB decides to use preformatted pieces of paper ("a set of pre-printed elements that could be used easily" as he explained) to input trajectories for specific pairs of descriptors (Figure 4.9). By drawing trajectories, he navigates into the sound space and explores alternatives. His computer application visualizes these trajectories and plots their evolution over time. The trace of such explorations remains on paper, linked to their electronic representation. Paper trajectories can be annotated and later reused in different combinations to produce new sounds.

discussion

The participatory design sessions offered compelling examples, which are grounded in their real-world composition process, of paper interfaces that handle their personal data representations and structures.

In summary, we observed the following patterns and functions in their designs.

Specialized components

To represent, input and interact with musical content on paper, participants generated a diverse set of structures and interactive components including: structures for writing musical notes (MB, MGV and MK) or drawing control curves (MGV, RD, MK and GB), lists of compositions rules (MK), lists of parameterized filters (MGV), visualizations of data points representing alternative solutions (MGV), timelines for presenting the sequence of modular musical objects (MK), labels with textual or graphical descriptions (MGV, RD, MK). Each composer designed his personal component for working with notation. RD generated zones of score segments on the computer. In contrast, MB created specialized templates with empty musical staffs. The handwritten musical content in this area was stored but not translated by the computer and served as a musical annotation on a selected orchestration solution. MK created both areas with printed musical content, which represented composition rules or predefined music sequences, and areas with empty staffs dedicated to the entry of note intervals. Finally, MB wrote music by connecting and overlaying paper strips, where each strip handles a different musical variable.

Layering and repositioning

We observed that several structures act as layers that control different parameters of the same musical object. The connection between the layers is either fixed or changes dynamically during the task. In RD's prototype (Figure 4.7), all the components that control a musical parameter are layered vertically. In contrast, MB uses small movable strips of paper that can be placed anywhere on the page. In order to logically associate two strips together, he aligns them physically and uses the pen to draw a link. MB also uses translucent strips to create physical layers of data (Figure 4.5). The decision about whether to use fixed or moveable elements varied across participants. RD and MGV worked with pre-printed data components. The other three participants preferred using movable pieces of paper that could be freely positioned, mixed and reused. Being able to freely reposition musical data on his workspace was particularly important for MB, who commented that he could never make prior decisions about the structure of his work on paper. MK introduced the concept of paper modules, borrowing the modular structure of the composition tool that he used. Modules act as programmable entities that can be reused in different sequences to produce multiple variations of a musical piece.

Archiving and reuse

Composers frequently reuse and refer to elements of a musical score. They expressed the need for keeping a history of their completed work, explaining that they regularly return to their previous pieces to find interesting material. MK commented that several famous composers such as Stravinsky reused their older material in new pieces. To support future reuse, MB, MGV and MK added identifiers such as textual descriptions, numbers, and graphical elements on paper. The mechanism of project folders proposed by MGV goes one step further as it helps users to physically archive their previous work and facilitate reuse in a longer term.

Links with computer-based data

In all the above scenarios, paper unifies interaction, providing a common space for data exploration and manipulation. Data are sometimes generated on the computer and then printed on paper using either an identical representation or more commonly, a modified representation adapted for paper use. In other cases, participants defined specialized templates for entering musical content with the pen. Such templates guide the precision of writing and support its interpretation by both the human and the computer. In addition to structured components dedicated to the representation and entry of musical data, several participants created interactive elements that facilitate interaction with musical software. For example, RD's interface provides play and stop buttons for evaluating partial results and control widgets such as sliders and arrows for testing and adjusting live parameters, e. g., pitch and volume. In addition to controllers of live parameters, MGV's interface contains navigation buttons and tick boxes for making selections in a list of solutions.

summary

We identified and presented two examples of formal structures on paper created by composers: Chouvel's Tonnetz and Schoeller's microorchestration. These structures helped the composers visually evaluating the musical result and allows new musical operation to create alternative musical sequences before translating the final result into regular music scores.

Participatory design sessions with composers helped us to explore how composers could create their own structures on paper to support entry and reflection with their musical ideas while interacting with their computer-aided composition environments. The design sessions resulted in several pen interactions and tangible manipulations of paper components to work with their data: drawing and modifying spe-cialized data over formatted paper, exploring variations by superimposing handwritten data, defining programmable modules, aligning movable components, linking them together, overlaying them, and archiving them into physical folders.

The next chapter introduces Paper Substrates, specialized interactive paper components that are structures to represent musical data and are connected to computer-aided composition environments.

contributions

1. Description of two structured musical representation systems for the entry, visual evaluation and editing musical ideas.

2. Exploration of structures on paper for creating personal interfaces that extend computer-aided composition tools.

C

R E AT I N G P E R S O N A L I N T E R A C T I V E PA P E R I N T E R FA C E S V I A PA P E R S U B S T R AT E S
This chapter introduces Paper Substrates. Paper Substrates are interactive paper components which offer composers the ability to create personal structures and musical content that can still be interpreted by music-programming software. Paper Substrates handle musical data from computer-aided composition applications and provide a structure to visually help data entry and editing interaction with the pen. We present the Paper Substrates Toolkit, which facilitates the development of Paper Substrates, and PaperComposer, a graphical interface builder that lets users create their personal interactive interface on paper and keep track of their work. We have used these tools to create a number of paper applications for creating music, both for non-musicians and professional music composers.

The results of the participatory design study presented in Chapter 4 allowed us to identify concrete scenarios where participants designed their own structures on paper to explore ideas and create music. Some created small movable components to support specific tasks such as writing pitches, organizing elements in sequence or tracing trajectories. Others created whole page interfaces, replicating and adapting their computer-based tools to edit musical recordings or control and explore the generation of orchestration solutions. The prototypes that participants created served as a basis for the design and implementation of Paper Substrates, which are modular user interfaces on interactive paper composed of personal musical structures specialized to the entry and exploration of musical data.

This chapter describes the concept of Paper Substrates and their physical and interaction design. It presents a toolkit and an interface builder that enable developers and musical assistants to create new Paper Substrates and musicians to customize their paper applications and link them to their own computer data. We demonstrate the application of these tools with case studies and participatory design with composers.

paper substrates

Paper Substrates (substrates) are specialized interactive paper components that provide modular structures to help the entry and editing of computer-based musical data on paper. Substrates act as formal representations designed by contemporary composers to explore their musical ideas while retaining great individual freedom of expression. In addition to this role, they are interactive, they can interpret pen and physical interactions to support composers' work with their composition software.

Substrates capture the main properties and roles of the structures and interactive elements designed during our participatory design study (Chapter 4). The traditional music staff is a logical starting point for a substrate as it follows the structure and data approach and is well known by composers. However, participants designed several others structures inspired by their computer-aided composition tools or their personal practices. In some examples, their structures contained preexisting data such as chords, waveforms or curves to edit on paper.

Each substrate is specialized to handle a certain type of data, which can be either printed or handwritten. Substrates contain specific representations of digital data, and are therefore logically connected to computer applications. This means that any data manipulation that takes place on a substrate 's surface also affects its digital representation. For example, in all scenarios, writing with the pen creates or modifies data in their computer tools. Several participants also used specialized components such as paper buttons and thick boxes to control the computer interfaces.

A substrate respects the affordances of pen and paper: it guides and constrains entry of handwritten data, aids recognition, and offers useful information about its roles and functions. Substrates can be printed as background on a primary sheet of paper or as mobile elements that can be positioned anywhere on the page or elsewhere in the working environment.

Specialized substrates can be linked together to support more complex data operations and communicate their data and state to each other. Composers organized their paper interfaces in space, to use a common time line or to group them by functionalities. As with paper flight strips [START_REF] Mackay | Is paper safer? The role of paper flight strips in air traffic control[END_REF], logical groupings can be defined by means of the spatial positioning of fixed or movable substrates. Users can also create explicit connections by drawing links between neighboring substrates. Finally, substrates can be transparent and superimposed to produce physical layers of data and data operations.

In summary, substrates have the following roles and functions:

• Provide templates for working with personal representations of musical data.

• Are responsible for handling their own pen events.

• Support the recognition of specialized pen gestures and handwritten music vocabularies.

• Communicate with music software through typed inputs and outputs.

• Share data and exchange messages with other components through their inputs and outputs.

• Can be fixed on a page or printed on movable piece of paper.

• Are modular and can be linked together by using simple pen gestures.

• Can be spatially linked, e. g., superimposed or juxtaposed.

Physical design and tangible manipulation

Substrates provide a set of structures to guide the entry of pen strokes.

The level of guidance can vary depending on the precision required by a given user task. A template that affords too much input precision can be a burden to creative expression. On the other hand, a template that affords too much freedom can be inappropriate for the entry of precise data or data that await recognition from the system. For example, Figure 5.1 presents three templates for the entry of graphical data, each providing a different level of detail. Substrates can be connected together with tangible manipulation to combine or share musical data. ModelCraft [START_REF] Song | The ModelCraft framework: Capturing freehand annotations and edits to facilitate the 3D model design process using a digital pen[END_REF] proposed similar liking mechanisms that allow designers to stack and join 3D paper-based building blocks together and create more complex structures. Designers can also make use of augmented tools such as rulers and protractors to perform cuts, shape transformations or issue stroke-based commands over the blocks. Here, we are not interested in geometrical structures but focus on the creation of reconfigurable communication links that represent operations over streams of musical data. Getting inspiration from MB's prototype during the participatory design session, two or more substrates can be combined together for creating more complex musical structures with multiple interconnected layers of data. Figure 5.2 shows three physical manipulations that enable the user to connect substrates on the fly. When substrates are connected, they can share data and flows of control. For example, a substrate that defines a musical segment can get spatialization data from a second superimposed substrate. A substrate with sound controllers, such as play and pause, can be joined with a musical sequence to allow for its live evaluation. When connected, substrates can synchronize one or two dimensions of their data according to their horizontal or vertical alignments. In the first example, the spatialization data are applied with respect to the horizontal alignment of the superimposed substrate, which in this case represents time.

To recognize such connections, we use explicit ink gestures, similar to the ones used by [START_REF] Steimle | Designing pen-and-paper user interfaces for interaction with documents[END_REF] for attaching sticky notes to pages, where the pen crosses from one piece of paper to the other. The relative position of the connected substrates is automatically computed by assuming that they are horizontally aligned. We have explored other link mechanisms that do not require horizontal alignment, but interaction becomes more complicated as the user needs to draw more than one stroke. To prevent linking strokes from interfering with content, a substrate can have a margin around its main content boundaries dedicated to writing the linking stroke (see Figure 5.4).

Composers can make use of the above connection mechanisms to reuse their composition material, reorganize it in the physical space and explore musical alternatives. As timelines have an important role in musical writing, in many of the scenarios that we have explored, the spatial positioning of substrates uses time as a reference. As substrates are printed on paper, users can take advantage of common geometric tools such as rulers, protractors and compasses to assist the drawing of their data. Moreover, substrates can take various physical forms by cutting and pasting paper. Different twodimensional shapes could ease the distinction between different substrates or place emphasis on their roles. Cutting and assembling substrates manually can be frustrating and time consuming. However, we envision that future printers will integrate technologies that cut paper with custom shapes (Graphtec, 2013). We also expect that future 3D printers will also be able to print the Anoto dot pattern on the surface of the printed object [START_REF] Zcorp | Zcorp home page[END_REF]. Currently, we use simple rectangular shapes so that substrate can be easily cut, and post-it glue that lets users easily attach substrates to each other.

first prototype

We designed a first interactive prototype1 to test the feasibility of creating substrates, connecting them to computer-aided composition environments, exploring their physical design and concretizing forms of tangible interaction that emerged during our design sessions. The prototype does not capture a real-world scenario but is inspired by two scenarios: MB's scenario where small movable translucent pieces of paper are dynamically connected as additional layers of data.

RD's scenario where separated components share a common time line and offer different areas to write musical data and control the software.

We implemented the prototype with the Anoto technology and ADP-301 digital pens. The pens communicate in real time with a computer through a Bluetooth connection. A Java application registers and interprets events generated as the user interacts with the components and provides direct audio and visual feedback through a Max patch. The communication between the Java application and Max is based on OSC [START_REF] Wright | Open Sound Control: an enabling technology for musical networking[END_REF].

The prototype allows the user to import a musical sequence from music software and print a simplified representation of this sequence on paper as an interactive substrate. The user can also create and print instances of two other substrates: a graph substrate, and a playbar substrate. When connected with a sequence substrate, the graph and playbar substrate obtain the musical sequence's timeline. At run time, users can modify onsets and durations of printed notes by drawing line segments over their rectangular representations. They can draw control curves in the graph substrate and then change them incrementally. Finally, they can use the playbar substrate to select slices in the timeline and replay the corresponding sequence. Each substrate is associated with a different musical object in a Max patch. The state of the online object changes in real time while the user interacts with the data on paper.

We implemented both regular and translucent paper and support both fixed substrates and moveable ones. The latter can be dynamically linked together by drawing simple strokes between their margins. Users can create these links to position or reposition graph and playbar components along the timeline of a musical sequence. Links can later be removed by crossing the trace of the link with a new mark. 5.4 (right) visualizes the structure of the components linked to the active sequence and updates the edited musical score with real time data from the pen. Graphical data sent by graph components are automatically concatenated and applied to the musical object, either to control its volume or of a background sound effect. Users may draw arcs over the playbar component to make selections along its timeline. They can then play the selections and listen to the result by tapping the pen on the drawn arcs.

programming and end-user customization

The first interactive prototype demonstrated the technical feasibility of implementing substrates but the diverse scenarios and interfaces created during the participatory design session cannot be realized with a unique paper application. Instead, we need flexible development tools that minimize programming effort and allow end-users to tailor substrates to their own needs and music-creation tasks. Several research projects, such as iPaper [START_REF] Norrie | General framework for the rapid development of interactive paper applications[END_REF], PaperToolkit [START_REF] Yeh | Iterative design and evaluation of an event architecture for pen-and-paper interfaces[END_REF] and Letras [START_REF] Heinrichs | Letras: an architecture and framework for ubiquitous pen-and-paper interaction[END_REF] developed toolkits to accelerate the development of paper applications, while Anoto and Livescribe released their own software development kits (SDK). All these programming environments focus on document management and applications for taking notes, editing printed documents or filling out forms. In contrast, our focus is on higher-level programming entities that conform well to the properties and behavior of substrates. In addition, we are interested in supporting the communication protocols and data structures of the music applications that professional composers currently use. To this end, we developed the Paper Substrates Toolkit, an application-programming interface (API), and PaperComposer, an interface builder for creating, customizing, and managing interactive paper applications.

Paper Substrates Toolkit

The Paper Substrates Toolkit is a Java development kit that accelerates the development of substrates. The toolkit provides classes for creating and printing customized templates, processing data, storing data and sessions, handling pen events and pen gestures, and receiving or sending messages to external applications. Figure 5.5 summarizes the architecture of the toolkit.

Main API components

The toolkit allows the developer to easily create paper applications by defining instances of existing classes of substrates on paper coordinates of one or more pages. The developer can also create new types of substrates by extending the Substrate class. The toolkit takes care of the management of the Anoto dot pattern, the management of sessions, and the communication of the substrates with external applications. We use Java Reflection to allow for loading new substrates definitions and their instances at run time. Multiple substrates can be grouped together into a single paper application represented by the Document class. A document specifies a set of pages, where each page can contain several substrates. We have developed utilities to store, edit and load a single or multiple documents for live interaction. All the pen gestures created by users at runtime are stored within sessions for later reuse and archiving purposes. In addition, the toolkit allows for recording audio that is synchronized with the captured pen data.

A substrate is a composite paper widget, similar to first-class widgets supported by common user interface toolkits such as Java Swing. It has a shape and position on paper expressed in millimeters and has a visual representation on screen and on paper. More specifically, it provides methods responsible for the painting and printing of its graphical elements and data. These methods are compliant with Java2D, letting developers reuse existing Java2D code. A substrate can be associated with coordinate transforms that map its data from millimeters to application-specific values. It can also register to listeners of specialized pen events such as pen down, pen entered, stroke created, or gesture recognized that happen within its shape.

A substrate can contain one or more interactive components, where each component can have its own listeners. Printed and handwritten data can also be added as child components that listen to pen events. As a printed page or substrates may contain a large number of components that respond to pen events, we developed an event manager that stores and retrieves substrates and data by using efficient spatiotemporal indexing techniques. We use the R-Trees implementation by the SaIL Library [START_REF] Hadjieleftheriou | SaIL: A Spatial Index Library for Efficient Application Integration[END_REF].

Communication between Paper Substrates

To create complex musical data from several layered or connected substrates, we create communication mechanisms between them. Substrates can communicate with each other through static and dynamic links. Static links are created programmatically while dynamic links are created interactively with the pen. Links can only happen between compatible substrates. One of the linked substrates usually acts as a master substrate that defines the reference of data sharing between them. For example, in the first prototype, the substrate with the printed musical sequence in Figure 5.4 defines a reference timeline. This timeline is shared to the other substrates linked to it with the pen. The link mechanism also enables substrates to reuse strokes written on other substrates, re-interpreting, rescaling and transforming them before redirecting them to specialized patches on the computer.

Communication with external music-programming software

Substrates can exchange musical data with external applications. Communication is based on the OSC protocol, as it is well supported by musical software. An OSC server allows registered applications to send and receive updates from substrates. We define an OSC-based data exchange protocol for a variety of data types, from basic lists of numerical values, to more complex sequences of musical notes and chords. Data types are extensible. Developers can add new data types and implement new substrates to handle them. In order to connect substrates with existing applications, we developed libraries for Max and OpenMusic. We chose these tools because there are widely used by composers and can be easily extended programmatically. The Open-Music library is implemented in Common Lisp and supports common musical objects such as graphs for controlling musical parameters and basic score containers. These objects can export their data to interested substrates. The Max library is written in JavaScript as an external object. It can be used within any Max patch to export its window and interface elements such as buttons, graphs and sliders. Developers and users can extend the two libraries to add new data types and functionality.

Support for musical scores

The Paper Substrates Toolkit also supports the use of scores from music notation editors such as Finale and Sibelius that composers frequently use to create scores templates, input music and produce their final scores. The toolkit loads PDF versions of the scores and their formal description in MusicXML files. It associates an area on paper for each graphical element of the score and relates it with its musical content. We based our implementation on the ProxyMusic Library2 . We created a class to play the corresponding midi note if the pen enters a note's area. Substrates can also use the scores' data to create alternatives representations and exports the result into a new MusicXML file that can be processed and rendered by most score editors.

PaperComposer: an interactive interface builder

We wanted to let composers create their own paper interfaces with substrates. To achieve this goal, the Paper Substrates Toolkit alone is not sufficient as it requires additional programming. Moreover, making decisions about the size, the location, the links and data connections of substrates in a paper document at a programming level can be a laborious process, not accessible to the end-user. We developed Paper-Composer, an interactive builder of customized paper interfaces that responds to this need. The tool enables the user to build a document with substrates from predefined substrate classes. The user can customize the layout and properties of the substrates in the document, connect them with the OSC data channels of external applications and print them with Anoto dot pattern. In addition, PaperComposer lets the user test a paper document with the mouse before printing it. It can also be used at run time, providing live feedback as the user interacts with paper and functionalities to reuse drawn strokes. Parameters that can be configured include the color of the substrate, user interaction, OSC output channels, and substrate-specific parameters, such as a range of values and duration. The user can define static links between substrates represented by straight non-printable lines. For example, in our first prototype, the top-left substrate in Figure 5.6 is statically linked to two other substrates, sharing its horizontal axis, which represents time. 4) A virtual page with several substrate including musical staves, a playbar, a curve container, a tonnetz and a sequencer.

Document creation and customization

The substrates can accept data from compatible data sources, both files and external applications. As shown in Figure 5.7, data from OpenMusic can be exported and become available on a dedicated list of PaperComposer. The user can drag a data element from the list and drop it on a compatible substrate to associate them. If the user drops the data on a blank part of the page, a menu appears with possible substrates for these data. When the user selects a substrate, a new instance is created to host the data. Substrates show their data on screen and can be printed on paper through PaperComposer's user interface.

Composers can also import scores created with their musical editors into a specialized score substrate by using the MusicXML file and the associated PDF document. The score substrate creates a new page on the document for each page of the score as well as interactive regions for all the score's elements. Figure 5.8 presents a portion of a score loaded in the substrate with colored rectangles for some of the extracted areas. The staff area (green) can be used to detect musical symbols and annotations. The area under the staff (light blue) can contain crescendos. The rectangle that surrounds a note (blue) allows the composer to listen to the note when the pen enters the area.

Deployment, debugging and session management

PaperComposer can be used to test a paper application at run time. Its interface shows the current page in use and strokes as drawn on paper. Figure 5.7 shows the paper interface at runtime overlaid with the current strokes. All the strokes and audio recordings captured during a session are automatically stored for later reuse. The user can use the session manager from the menu bar to manage past sessions, load them on screen and use a substrate to reinterpret them. Developers can iteratively improve their substrates by reusing recorded strokes as input. PaperComposer allows monitoring network activity with connected clients as well as sent and received messages. The monitor window is accessible through the network icon of the top toolbar.

Summary

To implement and deploy a new substrate, developers need to complete the following steps: step 1: Create a class inheriting the Substrates class or the Attachable-Substrates class.

• Set an icon file and a name.

• Set the coordinate space and set booleans to indicate how axis should be shared when link with another substrate occurs.

• Set the accepted data types.

• Program the paint method with standard Java2D instructions.

• Add one or several pen event listeners and implement how the events modify the data.

step 2: Create a class inheriting the Data class.

• Define an OSC communication protocol.

• Implement the incommingOSCMessage() and submitUpdate() methods.

With these requirements, users can use such a substrate within Pa-perComposer to build their own interface. Compatible data sent by external application can be loaded into the substrate. When users interact with the pen over the substrate, it will trigger data notifications to the application. With our toolkit, our goal is clearly to help developers focus on the design of the visual properties of the substrate and the associated pen input interpretation. Regarding these aspects, we also provide several utilities to simplify strokes with a distance threshold or [START_REF] Douglas | Algorithms for the reduction of the number of points required to represent a digitized line or its caricature[END_REF] algorithms, extract geometrical features, e. g., main orientation, duration, intersections with other strokes. Interpreting pen input remains challenging as each user's way of writing can differ in both time ordering and shape. However, developers can integrate Java libraries and available toolkits such as iGesture (Signer et al., 2007) to recognize gestures or process incoming strokes.

Using the Paper Substrates Toolkit and PaperComposer, composers can create and define substrates on the computer before composing with them on paper. In the next section, I present several examples of paper applications built with the Paper Substrates Toolkit and PaperComposer.

designing new interactive paper interfaces

We used the Paper Substrates Toolkit and PaperComposer to design and implement several paper applications with substrates. This section first presents case studies of paper interfaces designed for both musicians and non-musicians. It then describes the participatory design of new substrates with individual composers to support their composition process.

Case study I: A tangible score for electro-acoustic composition

We wanted to explore the potential of physically positioning substrates to create musical sequences. We developed a tangible score application inspired from graphical representations of electro-acoustic music, e. g., the score made by Rainer Wehinger in 1970 for Gyorgy Ligeti's Artikulation piece. This aural score associates sound objects with shapes that have different graphical properties according to their sonic variation. For example, the size of the shape can control the loudness and the color the timbre of the sound. We designed a simplified version that lets users create musical sequences by assembling several sounds on a paper interface.

Paper application

The application includes a substrate that takes a whole page and represents the space of the score where other small substrates linked with sound files can be added. Each of the small substrates has a color that identifies a sound and a width that defines its duration. It accepts handwritten curves that control the amplitude of the sound over time. The user can stick and link multiple substrates on the score to create a collage of sounds that compose a musical piece. The user can tap the pen on an individual substrate to play the associated sound or tap it on the score to evaluate the complete piece. The paper interface communicates with a Max patch responsible for amplifying and mixing the sounds based on instructions given by the substrates. Our application can be considered as a simplified tangible representation of the maquette interface supported by OpenMusic [START_REF] Bresson | Scores, Programs, and Time Representation : The Sheet Object in OpenMusic[END_REF]. The maquette provides a spatial layout that helps the musician to mix programmable musical objects together and organize them over a timeline.

Public demonstration

We publicly demonstrated the user interface during the "Fête de la Science 2011" event (see Figure 5.9) to encourage participants to explore properties of different sounds and understand the goals and tools of computer-assisted music analysis and composition. Approx- imately 60 to 70 participants, mostly children, interacted with the interface and created collective musical pieces. They all tried to take advantage of the available tangible interactions to edit their score. Several participants tried to compare alternative results by spatially rearranging their sound substrates and by exploring different curves on substrates of the same color. We observed that the tangible nature of the interface is particularly useful for making music creation tools accessible to children in a playful manner.

Case study II: PaperTonnetz, interactive paths in tone networks

Inspired by [START_REF] Chouvel | Traversée du vent et de la lumière. Six remarques pour une phénoménologie de la création musicale[END_REF] composition approach (Section 4.1.1 for details), we iteratively designed PaperTonnetz 3 , an interactive tool that lets musicians explore and compose music with Tonnetz structures by drawing paths on paper. As described in the previous section, a Tonnetz is a tone network in which pitches are organized according to their interval relationships

Initial study

We first conducted an informal study at the Palais de la Découverte in Paris during the MCM'11 conference (Mathematic and Computation in Music, June 2011), to observe how users, both non-musicians and musicians, would respond to an interactive paper Tonnetz interface. The interface lets participants interact with two different Tonnetz created with HexaChord 4 . These Tonnetze provide different musical properties: the former uses large intervals, the latter uses small intervals. With our initial implementation, each time the pen enters a pitch area on paper, a synthesizer plays the corresponding pitch. Most participants began by tapping a particular cell to produce a sound. The real-time sonic feedback encouraged them to explore different patterns and to compare the differences between the two Tonnetze. We observed that non-musicians began by scribbling on paper to "sonify" their gestures, as they could not predict the produced result. In contrast, several experienced musicians started with concrete musical ideas, expressed as scales or chords, and tried to represent them in the Tonnetz space. Even if they enjoyed experimenting with the interface, the musicians identified a number of short-comings. They pointed out the need to express chords and to reuse previously drawn curves, preferring to tap instead of re-drawing the path in order to listen to it again. Based on feedback from the first study, we decided to re-design the PaperTonnetz prototype into a more general composition tool 5 . We designed two substrates: the Tonnetz and the sequencer. The Tonnetz substrate represents pitch networks on paper with various musical and graphical properties, such as relationships among pitches, size and orientation. The user can explore the different auditory relationships among groups of pitches by clicking on cells printed on the Tonnetz and drawing paths between them. The duration of a pitch is defined by the time that the pen stays within its cell. Once a path is drawn on paper, it becomes a reusable interactive element. The Tonnetz translates the path into a melody, i. e., a sequence of notes, based on the time spent in each cell. By tapping the pen on the trace of the path, the user replays the melody created while drawing it with its rhythmic properties preserved. By holding the pen, the user plays the chord that is composed by the pitches defined by the path. Figure 5.10 shows a scenario where a musician uses a sequencer substrate to assemble paths drawn on two Tonnetz substrate into a single musical sequence. Users can define their own paper interfaces using PaperComposer which integrates the Tonnetz and sequencer substrate.

PaperTonnetz

We developed a third version of PaperTonnetz to add octaves. discovery: PaperTonnetz makes each cell and path interactive so they can reveal their musical properties and functionalities. This allows both novices and experts to discover its properties, with real-time auditory feedback.

improvisation: PaperTonnetz lets previously drawn paths to be accessible for future interaction. The user can create individual chords or melodies and then reuse them to explore different musical possibilities.

assembly: PaperTonnetz enables expert users to capture musical sequences based on their improvisations and to export them to other music composition systems. PaperTonnetz supports both MIDI and OSC protocols, so any musical sequence created with it can be imported and edited in any other dedicated application.

Creating paper substrates with composers

We collaborated with two composition students at Ircam to create substrates for their personal composition approaches.

Proportional music notation

We worked with a composer between October and December 2012 to iteratively design a substrate to let him write musical scores. The substrate supports the personal musical notation he uses on paper for his piece for alto and electronics.

We implemented a substrate that recognizes handwritten notes over a staff, allows him to listen to the musical result and transfer it to OpenMusic or Max. Figure 5.12 presents the score written by with his personal notation and the resulting musical sequence in Max According to this notation, notes have an absolute position in time, which is defined by their horizontal position within the staff. A horizontal line connected to a note represents its duration. Our recognizer is based on simple heuristics and supports the following elements: keys (G, F and C alto), notes, intonation symbols (quarter tones, sharps and flats) and duration lines. The composer's original notation does not include stems, i. e., vertical bars that group notes into chords. We added stems to simplify the identification of chords and their onsets. The user can evaluate the musical result by tapping on the rectangular area around the key symbol or by using an additional playbar substrate. To correct a recognition error, the user can hold the pen over the misrecognized symbol and redraw it.

The composer created his own interfaces with PaperComposer that included the notation substrate in combination with graph substrates that describe the dynamics of the musical sequence. He successfully created Max patches that react to the data transmitted from the pen to produce a final sequences played by his own synthesizer. Recognizing handwritten musical notes is a challenging problem [START_REF] Lee | A comparison of HMM, Naïve Bayesian, and Markov model in exploiting knowledge content in digital ink: A case study on handwritten music notation recognition[END_REF]. Previous work has tried to tackle it by using simplified gesture vocabularies [START_REF] Forsberg | The music notepad[END_REF] or interactive devices that allow the user to supervise the recognition process and correct errors [START_REF] Tsandilas | Interpreting strokes on paper with a mobile assistant[END_REF]. Here, we have considerably simplified the recogni- tion problem, as there is a single type of notes and no special notation for rhythms or rests.

Controlling sound synthesis with pen gestures

We collaborated with another composer between November 2012 and January 2013. We designed a substrate for drawing gestures that control a sound synthesis engine developed by the composer. The composer developed a composition framework on Max. According to this framework, the visual space acts as a sequencer where the horizontal axis represents time. The composer organizes several musical elements in time by positioning them in the patch space. He mainly uses text-based descriptions of synthesis parameters but is interested in enriching his expression possibilities with pen input. We started by designing a substrate with minimal functionality that let him explore and develop a synthesis model and, based on that, convert his handwritten gestures into sounds. Having a first prototype as a basis, the composer could better articulate his needs. We met several times with him to refine the functionality and the design of the substrate.

We created a substrate that maps dynamic gestures drawn on paper to the dimensions of computerized objects in a Max patch. The actual properties that form the profile of a gesture are the list of its point coordinates with their timestamps and the associated pen pressure values. We use color-coding to reveal the orientation of a gesture, visual information that the composer uses in the synthesis engine. The beginning of a stroke appears as blue and progressively changes to red. The substrate represents existing objects of the score as rectangles that guide the drawing of synthxesis gestures. After expressing a gesture on paper, the composer can refine it by redrawing only parts of the gesture. In this case, the refining stroke is merged with the ini-tial one. We carefully update the timestamps of the new data points so that they preserve the dynamics of the original gesture. This allows the composer to explore variations of a gesture's shape without changing its original dynamics. He can also edit the position of a gesture in the electronic view and then reprint an updated version of the substrate in order to continue his work on paper. Figure 5.13 explains how the composer uses the substrate as part of his workflow. First, he exports sequences of musical objects from his Max patches. He then uses PaperComposer to associate the exported data streams with a specialized sequence substrate. He prints the substrate on paper to draw his synthesis gestures. The Max patch shows the new gestures immediately after they are drawn on paper.

discussion

The Paper Substrates Toolkit and PaperComposer allowed us to build several of the substrates based on the patterns and principles that emerged throughout our participatory design study presented in Chapter 4. These include data containers for editing musical notation, drawing audio synthesis gestures and exploring graphical data, as well as interactive controllers such as a playbar and a sequencer. We have also been able to reproduce interactive representations of musical structures observed from the work of composers such as the Tonnetz.

Substrates create a bridge between music programming software and paper-based representations by providing a modular, co-adaptative [START_REF] Mackay | Users and Customizable Software: A Co-Adaptive Phenomenon[END_REF] interaction space for writing music on paper. They are not standalone applications so they do not exist on their own. They serve as user interface extensions, co-existing with musical software and computer interfaces. Composers who participated in our studies have argued in favor of this approach, as it allows them to benefit from a new paper interface while they can keep using powerful computer tools assimilated into their work process after years of composition practice.

Substrates help composers to represent and structure musical ideas, visually assess them and explore new musical results. Substrates are also structures which provide guides for the entry of musical content. This content can be directly interpreted and communicated to an external music application to render a musical output. A user can combine multiple substrates together to create more complex musical data or use the spatial positioning of substrates to create sequences. Their tangible nature makes them accessible to non-musicians and children.

We provide both audio and visual feedback through external applications in response to pen events. For certain tasks such as music input, the musical result itself serves as feedback when the user writes a note or an alteration. When performing additional operations such as deleting a note, we provide audio feedback. In such a case, relying only on visual feedback in the graphical user interface could disturb users and force them to frequently switch their visual attention between the paper interface and the screen to verify if the commands have been correctly issued.

summary

This chapter introduced Paper Substrates, interactive paper components that extend music-programming tools with structured paperbased representations of musical data. Substrates support pen and physical interactions to help users write and explore their musical ideas. We developed the Paper Substrates Toolkit that includes an API and PaperComposer, a user interface builder. The API enables developers to define new substrates with personal structures to represent musical data and define associated pen interactions. The interface builder allows both developers and end-users to connect the new representations with data streams of external music software, customize their paper substrates and deploy their interactive paper applications. It also provides mechanisms for session management, debugging, and logging. We described several examples of new paper interfaces built with these tools. Some were designed with composers to support specific phases of their composition process such as early expression with personal musical notation and exploration of sound synthesis algorithms with drawn gestures. PaperTonnetz was inspired by Chouvel's practice and proposes to use Tonnetz as an underlying structure in which strokes become chords and melodies.

These case studies demonstrated the practical use of the Paper Substrates Toolkit and PaperComposer. However, it remains difficult to evaluate and compare such interfaces with other interactive systems. The next chapter presents a method for studying the use of interactive technology in computer-aided composition.

contributions

1. Design of Paper Substrates, interactive paper components connected with computer-aided composition tools that offer paperbased representations of musical data and can interpret pen interactions.

2. Discussion of physical interactions to connect, layer or spatially arrange Paper Substrates to share parameters between them, create complex structures or synchronize their properties. [START_REF] Olsen | Interacting in chaos[END_REF] This chapter presents a technique for structured observation of music composition. We used Polyphony, a unified user interface that integrates interactive paper, tangible controllers and electronic interfaces, for systematically studying the use of interactive technology in computer-aided composition.

Implementation of the

Paper Substrates allowed us to create new interactive systems in which composers could reuse their existing musical representations on paper to interact with their computer-aided composition environments. We designed tools for the exploration phase of the creative process, when composers use formalized music representations. We want to understand how interactive paper-based interfaces for common composition environments could support the phases of music composition, from early sketches to final scores. Unfortunately, we still have little understanding about how composers appropriate interactive systems, what common patterns they use, and how they composition approaches differ with respect to the use of input devices. We must explore diverse strategies not only to understand the process but also to design innovative tools that support it.

Previous work emphasized on the open-ended nature of the composition process [START_REF] Pennycook | Computer-music interfaces: a survey[END_REF][START_REF] Bennett | The Process of Musical Creation: Interviews with Eight Composers[END_REF], which makes the evaluation of computer-aided composition tools difficult. Methods for studying composition tools include collecting questionnaire data [START_REF] Healey | Sketching Musical Compositions[END_REF], talk-aloud protocols and interviews [START_REF] Eaglestone | Composition systems requirements for creativity: what research methodology[END_REF]. Others involve field studies with open-ended explorations of interactive tools [START_REF] Eaglestone | Computer support for creativity: help or hindrance[END_REF][START_REF] Fiebrink | Toward Understanding Human-Computer Interaction in Composing the Instrument[END_REF]. These methods are often conducted in the field with one or few composers to explore a particular use or approach. However, they are not suitable for comparing how different composers work as the highly diverse and personal nature of each composer's work practice makes comparisons difficult. Studying creativity requires seeking the unexpected in noisy, hard to control environments. We need new methods for understanding how expert composers use composition tools.

This chapter presents a technique for structured observation of music composition. It describes our collaboration with a professional composer to create an hour-long composition task for expert composers. It then presents Polyphony, a unified unified user interface that integrates interactive paper and electronic user interfaces for composing music. We use Polyphony to study the composition process in a structured observation study with twelve composers and musicians. The composers used Polyphony to create a complete electronic accompaniment to a Webern's composition. The chapters details our results, both on the composition process and the Polyphony user interface, and discuss how the structured observation method helps us understand real-world, expert-level creative processes.

goals and approach

Our challenge is to observe and evaluate the complete composition process, from early paper sketches to the final electronic score, to inform the design of composition support tools. This requires:

• A structured observation method to compare composition behavior at each stage of the process.

• A short but realistic and complete composition task.

• A technology probe [START_REF] Hutchinson | Technology probes: inspiring design for and with families[END_REF] that integrates a mix of previously studied composition tools, including paper, instruments and composition software, and records their activity throughout the process.

Structured observation method

In order to tease apart individual differences from common composition patterns, we need to observe composers under as similar conditions as possible, as they go through a complete composition cycle. We want to know how and when they use paper, when they explore ideas by playing on instruments or with the computer, and how they transition between rough ideas on paper to formal representations on the computer. Since we are particularly interested in how composers move between expression and implementation of musical ideas, we must investigate how they combine pen-based technologies (graphics tablets or interactive paper) and computer-based composition tools. We use a structured observation method [START_REF] Mackay | Structured Observation to Support Interaction Design[END_REF], similar to a quasi-experiment [START_REF] Cook | Quasi-experimentation: design & analysis issues for field settings[END_REF], that seeks to balance control and external validity. We create a controlled setting in which a relatively large number of expert composers [START_REF] Tsandilas | Musink: composing music through augmented drawing[END_REF] perform the same constrained creative task, with the same set of composition tools, input devices, and software. Composers are free to use their own creative process. We limit the task to one hour to facilitate recruitment of professional composers.

Note that, although this method is similar in form to a controlled experiment, our goal is not to test hypotheses or determine cause and effect, but rather to create highly comparable conditions for comparing measures of qualitative and quantitative behavior. We hope to identify common patterns that emerge, despite the highly individual nature of composition strategies, so we can support fluid transitions between pen-based and existing software composition tools.

Composition task

Creating an appropriate composition task is a major challenge, since the task must be short, yet creative and meaningful to professional composers. We worked with Julia Blondeau, a Ph.D. candidate in music composition, who is a professional composer with in-depth knowledge of music technology. She helped design the task and tested our first prototype. We then pilot tested the task with an experienced, professional composer (C2) who offered additional suggestions about its design.

Creativity stimulus

The starting point for creating a musical piece is an idea, a theme or an internal need that drives the creative process. We replaced this phase by a composition stimulus around which composers develop their piece. We considered two alternatives: a short video clip or abstract graphic animation that acts as inspiration, and an existing musical piece that composers import and reuse in their work. We chose the second alternative. Julia suggested Anton Webern's Bagatelle No. 2 for String Quartet, Op. 9, which is remarkably short, only 20 seconds, yet still considered a complete composition and well known to most contemporary composers.

Instrumentation constraints

A composer usually writes music for a certain combination of physical and digital instruments or a whole orchestra. Contemporary composition often includes transformation of the sounds produced by performers as well as recorded material such as samples or electronic sounds. After several iterations with Julia and C2, we created a task with an audio effect and a synthesizer, explained as follows: Use the effect to create a variation of Webern's 20-second piece and write an accompaniment for the synthesizer. Although this task is not representative of all real-world composition processes, it still requires key composition skills to analyze the given material, explore possibilities offered by the tools and produce an original musical result.

For the audio effect, we implemented a harmonizer that takes the original sound file as input and outputs a transposed version without altering its rhythmic properties. The transposition effect has two continuous parameters: a transposition factor expressed in midi-cents from -1200 to +1200, which is equivalent to minus or plus an octave, and the amplitude of the transposed audio signal from 0 to 1 in linear scale. The synthesizer is a note-controlled polyphonic sine-based synthesizer that accepts a sequence of notes and an amplitude signal from 0 to 1, again in linear scale. Notes are defined by their pitch, onset and duration.

A technology probe

Technology probes [START_REF] Hutchinson | Technology probes: inspiring design for and with families[END_REF] combine three goals: collect data about use, test future technology in situ, and inspire novel design ideas. We wanted to provide a simple, easy-to-learn interface that integrates all phases of the composition process, from the earliest expression of rough ideas to the final implementation and performance of those ideas. Rather than introduce a new, generic composition interface, we created Polyphony, a constrained environment that provided familiar tools, but only those required for the composition task. We deliberately simplified the functionality and musical capabilities so we could concentrate on how interactive tools affect the creative process. Polyphony allows us to study both the composition process and the role played by an interactive system under controlled settings.

polyphony

Polyphony provides a unified user interface for capturing pen-based input (on paper or on a graphics tablet), as well as musical performance on a piano keyboard, and typed or mouse-based input to control established music composition software. 1

Computer interface

The Polyphony interface is implemented in Max and contains three main panels (Figure 6.1). The top panel displays the musical piece that serves as stimulus in two different representations: the waveform of its sound source and its musical score in linear time scale. We use the bach plug-in [START_REF] Agostini | Real-Time Computer-Aided Composition with bach[END_REF] score object to accommodate musical notation in Max. The middle panel includes interactive graphical objects for defining the two parameters of the harmonizer effect: the transposition factor and the amplitude. Finally, the bottom panel is dedicated to the synthesizer. It includes a bach object for entering notes and a graphical object for defining amplitude. To input curves that control the transposition factor of the harmonizer, we used a breakpoint function object, in which the user defines discrete points that are automatically interpolated. In contrast, amplitude curves of both the harmonizer and the synthesizer are continuous functions defined by the drawing. The user interface provides additional interaction mechanisms that complement the editing tools:

Free annotation

Composers can sketch on the original score with different colors or print the current version as it appears on screen and annotate it with a pen or a pencil.

Precision levels

Composers can work at two levels of precision: they can view the whole piece or zoom in on a small part by clicking on the corresponding button on the top panel. The score is divided into five 4-second parts, which C1 and C2 felt was the appropriate level of precision for the task. If a part is selected, each interface element's display is updated, and a green overlay shows the selection on top of the waveform. Composers can move among parts of the piece by dragging the overlay to the target position.

Audio controls

The interface offers several controls for evaluating partial composition results: play and stop buttons, mute switches, gain sliders, and a selector tool. The latter enables the user to select a specific time range to evaluate. Finally, the user can explore the synthesizer and the harmonizer by opening a dedicated window with several live controls: a software piano, buttons and knobs.

Physical controllers

Polyphony uses a small MIDI piano keyboard with buttons and potentiometers to control the synthesizer and audio effect. Figure 6.2 shows the physical representation of the synthesizer and the harmonizer, which allows composers to actually play the synthesizer and the audio effect.

Paper interface

Polyphony paper interface lets composers enter notes and control curves and interactive elements to test sounds and partial results. They can also use free space for sketching and annotations. The printed interface (Figure 6.3) fits on six pairs of A4 pages. One pair contains the global low-precision view of the piece and the five other pairs are dedicated to each individual part of the piece. The first page of each pair contains the transcription of the original piece and free space for writing and sketching, as well as interactive buttons for printing parts of the score and an interactive timeline selector. Composers can draw arcs along the length of the selector to define score ranges and play the associated segments by tapping the line traces. The second page contains a timeline selector for listening to selected segments of the piece and several substrates for writing music for the harmonizer and the synthesizer:

One substrate for drawing control curves for the synthesizer's amplitude;

Two substrates for drawing control curves for the harmonizer, one for pitch transposition, and one for controlling amplitude;

Two staves (G and F key) for entering the synthesizer's pitches and durations, using a simplified notation.

Substrates for entering curves support incremental drawing and refinement. Composers can edit curves and symbols, using as simple mark to erase or modifying them by redrawing some or all of the gesture. Interactive staves automatically recognize most intonation symbols used by composers, e. g., flats, naturals, sharps, quarter sharps and three quartet sharps. All of the above actions are accompanied by audio feedback. The paper interface communicates directly with Max, so data entry on paper is immediately reflected in the computer interface. Composers can refine the score using the mouse and keyboard on the computer or use the digital pen to rework it on a printed page.

We created the Polyphony paper interface with the Paper Substrates approach. We reused and updated existing curves, playbar and handwritten music substrates from our previous work (see see 4). Using those substrates, we created the layout of the paper interface with Pa-perComposer (Figure 6.4). We added additional utilities to log and re- trieve data from partial or whole interactive sessions as well as data coming from the Max patch.

structured observation study

We conducted a structured observation study of twelve expert composers who each composed an electronic accompaniment to Webern's well-known instrumental piece. We focused on: similarities and differences in composition practices, reflections about their own composition processes, and feedback as to the benefits of integrating interactive paper with their usual computer-based composition tools.

Participants

We recruited twelve composers (11 men and one woman), aged 25-70, all right-handed. Ten are professional composers, both composition professors and advanced graduate students. One is a Masters student in acousmatic composition; another is an electronic music controller engineer. Some are highly renowned composers and all have had their music compositions played in public. Participants sit in front of a 24-inch Apple display with a native resolution of 1680 x 1050 pixels. The setup also includes two loudspeakers, headphones, a midi controller (Figure 6.2), a computer keyboard and a mouse. Sessions are recorded with a digital video camera on a tripod placed behind the participant.

Because several composers already use graphics tablets for penbased input (Letondal et al., 2007), we tested two configurations of Polyphony: graphics tablet and interactive paper. Six composers were assigned a Wacom Intuos A5 tablet (Figure 6.5) in which three physical buttons at the top-left corner were labeled with frequent editing functions: play and stop, add a note, and delete a note. The remaining six composers were assigned an interactive paper interface with an Anoto ADP301 digital pen. All other aspects of the task and the suite of composition tools were identical and all participants could sketch on ordinary blank paper, using an assortment of pens, colored pencils and a ruler.

Study design

Participants P1 to P6 used the standard configuration plus interactive paper for pen input. Participants P7 to P12 used the standard configuration plus the graphics tablet. Participants were matched according to age and experience between the two groups.

Procedure

Each session lasted 100 to 120 minutes, in four parts: introduction: We described the goals and motivations of the study.

Participants then answered a short questionnaire about the tools and interfaces that they normally use to compose music. We then described the composition task.

training: We first introduced the Max application, the harmonizer and the synthesizer. We explained how to use the tangible controllers to play with the audio effect and the synthesizer and how to interact with the main computer interface. We then presented a tutorial on how to incorporate pen-based input using the graphics tablet (15 minutes) or the interactive paper (20-30 minutes). We showed how to write musical notes and draw curves, as well as erase and editing functions, and showed how to print the results on paper and iteratively modify their scores.

Participants had three minutes to test the pen-input technology, and then we presented the composition task.

composition: Participants were asked to spend at most 60 minutes to compose their piece. The experimenter acted as a technical assistant and answered questions that arose about the interface.

Participants could practice on the tutorial sheet and print their score at any time. We encouraged them to try the pen-based interfaces but did not enforce their use: Participants were free to use whichever tools best supported their composition process.

debriefing: After the piece was completed, we videotaped a fiveminute interview. We asked participants to comment on their piece and summarize their steps. We asked them to focus on their creative process and how they moved from ideation, to exploration, to execution of their musical ideas. Finally, they completed a questionnaire about the musical interest of the composition task, their opinion of the musical result, their opinion of the user interface, and their suggestions for improvements.

Appendix A.3 presents the study's material, including the cover sheet, the questionnaires and the tutorial sheet.

results

All twelve composers successfully produced a unique musical piece within the 60-minute time limit, except for P11, who managed to finish about 70% of the piece. Their final scores are presented in Appendix B. Below, we examine the tools and the input devices that participants used, analyze their compositional process, and report on their experience and feedback.

Use of input devices, interfaces and controllers

Participants expressed mixed opinions about the different forms of input we provided. Figure 6.6 shows the time each composer spent on each interface. We measure time only for the interface of the composer's primary focus or the one controlled with their dominant hand. Several participants used different hands for different inputs, e.g., writing with the digital pen on paper while playing back the result by pressing the spacebar on the keyboard.

Graphics tablet

P9 was the only composer who completed the piece with the graphics tablet. Although it was his first time using this form of input, he appreciated the gestural control of the pen, especially for drawing the profile of control curves. He commented that he could "focus on morphologies and profiles directly related to performing gestures that I made with the tablet". Figure 6.7 compares the curves that he created with the curves produced by other composers using the mouse or interactive paper. His curves show quick dynamic gestures, a distinctive pattern that cannot be easily reproduced with the other two input devices. P7, P10 and P11 experimented with the tablet but abandoned it quickly after they made a few mistakes. P7 explained that he was already "too trained" working with a mouse. The rest of the composers preferred to use the mouse from the beginning.

Interactive paper

The most senior and most experienced composers (P4 and P6) decided to work directly on the computer and did not use the interactive paper interface. They explained that it would be too complex for them to master and produce a satisfying result. In contrast, P5 used the paper interface almost exclusively (see Figure 6.1), while P1, P2, and P3 alternated between interactive paper, mouse and keyboard. To finalize the score, all the composers used the mouse. P1, P2 and P5 used pen and paper to input pitches for the synthesizer by writing notes on the interactive staves. P2 explained that he found this input method extremely quick. P5, on the other hand, complained that the printed staves were too narrow so it was difficult for him to precisely draw a note at the correct time position. Automatic recognition of intonation symbols did not always work, so some composers used the mouse and keyboard to refine them on the computer. Interestingly, P1 used pencil to sketch input rhythms with conventional notation before transcribing them to the proportional notation supported by the interactive paper interface (Figure 6.8). Composers also used the digital pen to draw control curves. For example, P1 liked the pen because of its gestural control, but used the mouse to fix constant parameters values. P2 used a physical ruler with the digital pen to draw precise control lines (Figure 6.9). Several participants (P1, P2, P3) faced an overload problem when editing hand-written content on interactive paper. For example, after drawing several alternate amplitude curves, P1 used a non-interactive pencil to cross out the old curves and highlight the correct curve, without printing a new version. He asked for a "non-interactive" mode for the digital pen that would allow him to accomplish the same result without having to switch between pens.

P2 and P3 followed a different strategy: each printed new interactive paper pages with their current work states and edited them for more precise results. P3 refined his amplitude curve by drawing more precise curves with the pen, using older traces to guide new refinements (Figure 6.10). P1, P2, P3, and P5 all used this strategy, which is particularly effective, and corresponds to previous research findings presented in Chapter 3.

Tangible controllers

Several participants took advantage of the physical representation of the synthesizer and the harmonizer effect. Before P7 started composing, he explored the pitch controller effect from the harmonizer while playing Webern's original piece. This allowed him to think about the graphical form of the effect before drawing a first, rough version of its curve in the Max interface. P10, P4 and P5 played the MIDI keyboard to try a chord or listen to a particular pitch. P8, P3 and P9 used the harmonizer's buttons to understand its possibilities. Interestingly, P8 could identify the harmonizer's transposition algorithm by playing with its controllers. P11, who had no classical music training, was the only one to play the keyboard along with the audio and then input what he had just played. This composer usually records as he plays piano and edits later.

Summary

All the composers discovered strengths and limitations of each tool they used. P1, P2, P3, P5 and P10 highlighted the benefits of being able to draw "living" curves that inspire them in the early ideation phase. However, when composers wanted to set specific values, e.g., precise transposition values, they preferred using the mouse to control the computer interface. Similarly, while some composers (P1, P2 and P5) used the digital pen to write pitches on paper, they switched to the mouse to define precise onsets. They also turned to the mouse to copy and paste notes, an operation not supported by physical ink.

Observations about the composition process

Exploring the original piece

All composers began by listening to the complete Webern piece; several listened a second time while reading the score. P6 and P12, who each have extensive composition experience, preferred to focus on the sound of the piece rather that its score representation. P12 explained that reading the score would negatively influence his composition.

Early analysis and ideation

All participants said they quickly decided how to segment Webern's piece and came up with the musical concept early in the process. However, several composers said that the task constraints limited their creativity. For example, P2 and P4 realized that their initial idea was impossible to implement. To preserve the esthetics of the original piece, they would have to calculate several musical elements, which was not feasible given the available time and tools. They thus decided to adopt a more spontaneous compositional approach, focusing on the sound itself instead of the score. P2 and P7 said they would probably need a full week to compose something they were truly satisfied with.

P8 and P10 started by sketching on regular paper to note the base series, i.e., the sequence of pitches used to create their piece. P7 and P9 decided to directly annotate the transcribed score in Max. P7 used the mouse to annotate while P9 used the graphics tablet and then printed the annotated score on paper (Figure 6.11) for later use. Interestingly, he did not print until the very end, to check if the result was consistent with his initial ideas. P4 wrote a short text that described the "story" or "path" of the electronics for the four parts that he identified in his piece. All other participants immediately began using the pen-based tool (graphics tablet or interactive paper) or the mouse to implement their ideas without sketching.

Iterative process

All participants completed the composition in several iterations. P7 first added small elements in the whole piece and then iteratively refined each part to improve the result: "I wanted to be sure that pivot notes are added first to guarantee the musical sense. [...] I iterate in order to respect the composition time and the length of the score".

Figure 6.12 illustrates how composers split their time between the harmonizer effect and the synthesizer. P4 said that he carefully organized his task to finish on time. He spent 10 minutes defining his goals and calibrating the tool. He tried different extreme possibilities to adjust the amplitude of the harmonizer and the synthesizer. He then spent 15 minutes composing for the harmonizer effect and 30 minutes for the synthesizer. He reserved five minutes for final edits and improvements. He divided his work into layers, one for the effect and one for the synthesizer, and for each layer, he progressed linearly along the length of the score. He made a few edits on the effect, after working on the synthesizer to improve the musical result. P9 followed a completely different process. He worked on both the synthesizer and the effect in parallel for the whole composition task. He started by working on the last part because he was interested in the end of the piece. He then returned to the beginning of the piece and progressed to the next parts in several steps. He finished by working on the global low-precision view of the whole score to adjust the amplitude and modify some final notes.

Most participants focused on a particular musical sentence or sound object at any moment in time. Once finished, they would zoom out to the global view of the piece and listen to the result before switching to the next element. Most participants listened frequently to all the elements of the composition (original piece, effect, and synthesizer), sometimes switching one of them off. In contrast, P6 isolated and listened to individual elements without the rest of the music.

P8 had an interesting approach involving the use of regular paper and colored pencils. Figure 6.13 shows his annotations directly on the original score. He explained that he started the task by exploring solutions on the computer. He then annotated the score to reflect on his work. After some time, he started by expressing ideas directly on paper and then implemented them on the computer. He iterated about ten times to complete the task. All said they found the task to be interesting and amusing. P1 said that this kind of task "really helps you think about the impact of electronics on the esthetics of a piece." Several others considered the task to be a nice composition exercise that they enjoyed. Regarding the one-hour time limit, most found it reasonable although they would need much more time for a real composition. P6, the most senior composer, felt tired at the end of the experiment and argued that she would require a break if the task were longer. All participants except P6 and P5 were generally satisfied with the result of their composition. P6 would have liked to think more about the musical interest of her piece, but would have required more advanced tools. P5 was disappointed by the sound of the synthesizer. In contrast, P2, P7 and P12 stated that they were satisfied with the result.

Participants appreciated the live feedback provided by the Polyphony interface, particularly the ability to play results directly as they composed. Overall, they liked the harmonizer effect that we provided, but several requested an improved version with several voices, control for independent instruments in the audio file and more extreme transposition values. Many complained about the sinusoidal synthesis that we used for the synthesizer. P4 and P12 said that sinusoidal synthesis is "tiring after several minutes" and P3 added that the synthesis "is not interesting when it is static". Even if they complained about the simplicity of the effect and the synthesizer, all found it easy to understand the musical possibilities.

Finally, participants provided feedback about the functionality and usability of the Polyphony tools. Some requested more precise time alignment of notes. Others wanted copy-and-paste functionality for curves. Finally, P12 wanted more precise drawing capabilities and would have preferred to enter exact numeric parameters for amplitudes and transposition curves with the keyboard.

discussion

The combination of structured observation and an hour-long task resulted in twelve unique musical compositions, each with a comparable snapshot of the composition process. As expected, these expert composers exhibited different musical strategies and choices of composition tools as they moved from initial ideation to the final score. Yet Polyphony successfully supported all of their creative processes.

Appropriation

Professional composers must spend months or years learning how to use music composition tools and integrate the capabilities into their composition process. However, although none can be considered to have 'mastered' Polyphony in the short session time, all managed to successfully produce a composition with tools they had never previously used. These composers each appropriated different aspects of Polyphony to support their unique ways of thinking and many were able to incorporate pen-based input to control the computer-based composition tool Max.

Composition styles

Although we expected to see diverse composition strategies and results, we were particularly interested in shared patterns across most or all of the composers. For example, we noted that the composers focused on enhancing only some segments of Webern's piece, which some referred to the 'relaxing' parts or those with 'less content'. This implies that these professional composers have internalized composition rules as to where it is appropriate to add to the piece.

We identified two main strategies for completing the task: P1, P5, P6, P9, P11 and P12 improvised, refining their ideas and adapting them to the tools, with the tool whereas P2, P3, P4, P7, P8 and P10 first defined their ideas and then moved to implementation. In the debriefing, P1 and P11 said they decided to improvise because they were unfamiliar with the audio effect (P1) or not confident with instrumental music (P11). They found it difficult to plan their compositions without sufficient knowledge of the musical possibilities. P4 also started by drawing extreme curves and exploring low and high pitches to quickly test the available elements before focusing on the composition.

6.5.3

The lack of sketching P1, P4, P8 and P10 all used pencils and paper to sketch or annotate content. Even so, they sketched less on paper than expected, based on previous work [START_REF] Healey | Sketching Musical Compositions[END_REF]Letondal et al., 2007;[START_REF] Tsandilas | Musink: composing music through augmented drawing[END_REF] and our interviews (Chapter 3). The key reasons given by P4, P7 and P12 were that the piece and the allowed time were short enough to remember their ideas. P1, P2, P3 and P11 argued that they did not need to develop a complex esthetic context, which often requires significant sketching activity, as the original score already provides it.

Informal feedback

We gathered informal feedback from several participants about scores from the other participants. All tried to guess who was the composer and gave comments on the compositions. P4, who is an experienced composer, found that several scores did not properly used the amplitude range of the synthesizer. He gave the example of P1, which had to draw really low amplitude curves at the beginning of his piece for high pitches and maximal amplitude when he used low pitches. He explained that he usually teach his students to first explore the range of each parameters.

summary

We introduced a structured observation method that produced twelve comparable snapshots of the composition process, and enabled us to identify how composers both adapt and appropriate paper, penbased interfaces and computer tools. We worked with a contemporary music composer to create and assess an hour-long composition task in which each participant composes an electronic piece with an audio effect and a synthesizer, based on a recording of a 20-second musical piece by Webern. We provided twelve expert composers with Polyphony, our interface to a rich set of existing music composition tools, including pen-based input with either interactive paper or a graphics tablet, as well as a keyboard, mouse, and audio controllers.

Polyphony offers a novel approach for integrating all the phases of the composition process, from early expression of ideas on paper to final implementation on a computer. Given the extreme time constraints, some composers limited their input to they devices they already knew. However, others switched among familiar and nonfamiliar interfaces, exploring their potential. Participants each appropriated Polyphony in their own way. Although initially designed only for the basic composition task, Polyphony proved capable of supporting these expert composers' highly diverse composition strategies. All successfully expressed and implemented their compositions and especially appreciated Polyphony's live feedback and ability to synchronize across input devices. We believe that such "Polyphonic" user interface that integrates several input and out modalities can help cre-ative practitioner to work effectively by using the most appropriate modality for the task at hand.

The next chapter presents a longitudinal study with Philippe Leroux during the composition of a commissioned piece.

contributions

1. Method for structured observation of expert creative behavior.

2. Creation of a one-hour composition task for professional composers.

3. Design and implementation of Polyphony, a multi-faceted tool that integrates regular computer user interface with interactive paper, graphics tablet and physical controllers.

4. Synchronization of multiple input and output devices help to integrate all the phases of the composition process.

5.

Found that each composers appropriate interactive tools in a personal manner and that Polyphony supports multiple composition strategies. Some composers interact with the tool first, after which they refine and adapt the musical ideas. Others define their ideas first and then use the tool to implement them.

Q U I D S I T M U S I C U S : D E S I G N I N G I N T E R A C T I V E PA P E R I N T E R FA C E S F O R A R E A L -W O R L D C O M P O S I T I O N
This chapter presents a research and creation project with Philippe Leroux during the composition his piece Quid sit musicus at Ircam. We collaborated with the composer and a musical assistant to design interactive tools that let the composer reuse the calligraphic gestures from an old-manuscript as composition material.

Illuminated manuscripts of medieval music contain rich decorations in addition to handwritten neumatic notation (Figure 7.2). Our research and creation project with composer Philippe Leroux investigates the use of such handwritten symbols during the composition of his piece Quid sit musicus. We designed several tools that combine computer-aided composition with interactive paper to support Philippe Leroux's composition process. Figure 7.1 gives an overview of the principal working sessions of the project. This chapter presents the musical motivation and background of this work from the composer's point of view. It then describes interactive tools designed for the project and their use by Philippe Leroux. Finally, it discusses how these tools helped the composer to explore and refine his musical processes with pen-based interaction, and presents future perspectives for the project. Philippe Leroux has long been interested in the ideas of writing and gestures in his composition [START_REF] Leroux | phraser le monde: continuit{é}, geste et {é}nergie dans l'oeuvre musicale[END_REF]. For example, his pieces VOI(REX) (2002) and Extended Apocalypsis (2006/2011) use written letters, words or shapes as control data for computer-aided composition processes (P. [START_REF] Leroux | The Model of the Model in VOI(REX)[END_REF]. In these earlier pieces, the composer used a mouse and/or a graphics tablet to input gestures.

An important missing feature for the composer with these interfaces was the lack of traces from the written gestures, like the ink a pen would leave on a sheet of paper. When writing on paper, composers can analyze, react and correct their work depending on the trace of their own gestures. Unfortunately, graphics tablets and other traditional input devices do not provide such visual memory.

For the composition of Quid sit musicus, Philippe Leroux wanted to reinterpret the calligraphic gestures of a manuscript from Guillaume de Machaut, a composer from the fourteenth century. The manuscript, similarly to other medieval "illuminated manuscripts" [START_REF] Hamel | The British Library Guide to Manuscript Illumination: History and Techniques[END_REF], contains rich decorations in addition to handwritten neumatic notation (Figure 7.2). Figure 7.3 shows an excerpt of the manuscript of the piece Ma fin est mon commencement by Machaut that the composer wanted to use.

In his piece Je brûle, dit-elle un jour, à un camarade (1991), Philippe Leroux already used neumatic notation to write the final score. However, he did not use computer-aided composition tools. In the present project, he was interested in giving new meaning to the calligraphic forms in the medieval manuscript by using their graphical and dynamic properties in algorithmic or electro-acoustic processes. Image- based recognition techniques such as scanning can convert the neumatic shapes into musical information [START_REF] Fornés | Primitive segmentation in old handwritten music scores[END_REF], but they cannot retrieve information from the gestural act of writing the symbols on paper. For this reason, the composer decided to trace by himself some excerpts of the manuscript with a digital pen in order to generate and receive information from accurate calligraphic gestures. We designed and implemented interactive tools that connect the calligraphic gestures drawn on paper to the composer's computeraided composition environments for the composition of his piece Quid sit musicus. We analyze the strokes while writing and transmit several of their features to OpenMusic and Max patches that process the incoming data to create musical material such as rhythms, chords, harmonic scales as well as synthesis and spatialization control data. Figure 7.4 shows an example of the whole system in which a handwritten form is converted into a musical sequence with OpenMusic. The substrate detects intersecting strokes to recognize shapes while the user is drawing, and computes features from these shapes. The top substrate contains data written by the composer. The substrate makes each written stroke an "interactive component", allowing further pen interactions. For example, the user can "click" over a stroke with the pen to trigger an action such as computing a sequence or playing the result.

interactive tools

Interactive paper interface

Writing on paper and feature extraction

The composer begins by drawing his own calligraphic forms over the original ones. The substrate groups both intersecting and close strokes into single forms. We used distance and time thresholds defined from a previous strokes session recorded at the beginning of the project. Figure 7.6 shows the composer writing on the paper interface and the visual feedback in PaperComposer. For each detected form, the interface displays data for: the strokes, the centroid of their points, an associated polygon and two main axes. The associated polygon is the convex hull [START_REF] Duckham | Efficient generation of simple polygons for characterizing the shape of a set of points in the plane[END_REF] of the points that define the form. The main axes are two orthogonal segments of the minimal enclosing rectangle of the polygon.

Philippe Leroux wanted to use several features of the calligraphic forms to control his musical processes. The composer defined six categories to match the traditional neumatic forms: punctum, punctum inclinatum, virga, virga inclinatum and porrectus. We used simple heuristics with tests on width and heigh ratios of the drawn shapes to classify them according to these categories. The substrate highlights the forms with different colors depending on their category (see Figure 7.5). In case of recognition errors, the composer can edit the category of a shape with a pop-up menu which appears when he selects a shape on the screen with the mouse. Each symbol is also associated to a single point, its centroid, which is used to estimate a pitch in midi-cents precision depending on its position relative to the lines of the staff. Figure 7.7 shows the computed features sent by the substrate and received in an OpenMusic patch. These features include: the list of strokes, the centroid, the duration, the pitch, the category ("type"), two perimeters, the area, two main axes and thickness envelopes.

Interacting with the pen

Each time the composer writes a new stroke, the substrate updates and transmits features of the current form. It allows the composer to monitor the data coming from the pen while writing, and to visually assess the result of the recognition. The composer can also retrieve the features of a previously drawn shape without modifying it by clicking over an existing form with the pen. We use the drawn strokes and the interior of the associated polygon as an interactive selection area.

Reactive OpenMusic library

Philippe Leroux wanted to reuse his existing library in OpenMusic (P. [START_REF] Leroux | The Model of the Model in VOI(REX)[END_REF] to process the incoming data from the substrates. Unlike real-time musical systems that react to internal clocks, external stimuli or data streams following a "data-driven" approach [START_REF] Puckette | Combining Event and Signal Processing in the MAX Graphical Programming Environment[END_REF], OpenMusic executes programs upon user requests in "defferedtime", following a demand-driven strategy (see Figure 7.8a). This was problematic because we wanted to provide audio-visual feedback in OpenMusic when the composer interacts with the paper interface. We collaborated with Jean Bresson 1 to update the library and create additional tools to take advantage of a new reactive programming extension of OpenMusic. This new library lets the composer create his 1 Jean Bresson is a researcher at Ircam and OpenMusic developer.

own patches that can react to the pen events and calligraphic gestures' data to provide direct audio-visual feedback in the composer's patches.

Reactive Programs in OpenMusic

Bresson and Giavitto (2014) developed a hybrid computational model that combines the demand-driven computational approach with reactive data-driven computations within the OpenMusic visual programing environment. In this model, each node in the program graph has a reactive status. Reactive nodes can transmit data and trigger computations through their output connections upon the occurrence of events (see Figure 7.8b). An event in the reactive model can be the modification of a box value, of an input or of a connection in the graph. It can be produced by user actions (e.g. while creating the program), by internal running processes, or by external incoming data (e.g. received via MIDI or OSC messages). Events propagate through reactive branches of the graph, updating (i.e., re-evaluating) downstreamconnected boxes. This reactive model creates new opportunities for composers' creative activity in OpenMusic as it does not require them to significantly alter their pre-existing patches in order to make them reactive. First, it could ease composers' experimental processes: when input data is manually parametrized, the new values automatically propagate to the downstream components of a patch according to the reactive boxes' status. Second, the reactive model permits OpenMusic visual programs to receive data from external processes and applications such as substrates. Incoming received data can immediately update the connected musical elements in the compositional processes.

Inspired by equivalent tools in real-time musical environments, we developed or extended a number of tools in OpenMusic that fit this reactive framework. Below are the main new OpenMusic boxes that we used in this project:

• osc-receive runs a server thread listening to an open UDP socket to receive OSC messages. Reception of a message changes the value of the box, triggers an event and propagates a notification in the graph.

• route-osc controls the propagation of a notification by testing the address of an incoming OSC message. It also stores in the the latest messages for each correct tests corresponding outlet.

• om-send and om-receive allow to transmit notifications and data between different locations in the visual program(s), bypassing the graph connectivity.

Figure 7.9 shows two OpenMusic patches that use these tools to process incoming OSC messages. Reactive objects are identified with bold, dark red frames. The patch in Figure 7.9a contains an OSCreceive box that transmits the messages to the route-osc objects. The data filtered by route-osc are used to build OpenMusic objects that are in turn "sent" via om-send boxes. In Figure 7.9b, the om-receive boxes receive a notification and update data structures accordingly.

Musical processes in OpenMusic

Philippe Leroux designed several processes to create the harmony of the piece, rhythms, melodic and harmonic gestures from the pen data.

The main harmonic scale comes from the detected pitches of the forms in the manuscript (see Figure 7.9b). For the rhythms, he created talea, i.e., rhythmic motives, from proportions related to the classification of the forms according to their category, duration, area or position in the manuscript. Philippe Leroux also created additional harmonic and melodic content by transforming the main harmonic scale in various ways, using techniques such as frequency modulation, pitch shifting, distortion or simulation of the Doppler effect. All these processes are controlled with the features extracted from calligraphic forms, and the timing of the resulting musical elements in the piece comes from a classification of these forms.

Below, we detail three examples of processes involving different features from the calligraphic forms: the strokes points; the speed of the pen movements and the thickness envelopes. of notes, the duration of the sequence, its ambitus and whether it should be polyphonic or "unfolded" in time (see (P. [START_REF] Leroux | The Model of the Model in VOI(REX)[END_REF]). Another part of the patch, not visible in the figure, filters the sequence with the main harmonic scale pitches.

2. The patch in Figure 7.11 receives the pen stroke points and computes a speed profile for the whole calligraphic shape. The box pdoppler takes a chord and the speed profile to compute a sequence of chords by applying a Doppler transformation.

3. The patch in Figure 7.12 uses the thickness envelopes of a form to distort the frequencies of a chord. The resulting chord-sequence is created from a single repeated chord distorted between the successive pairs of values from the envelopes. In both Figure 7.10 and Figure 7.12, a play box is at the end of the reactive chain: each time a message is received from the pen server, the sequence is updated and rendered through the OM player.

Controlling sound synthesis

Philippe Leroux wanted to use the chords generated with his musical processes to control sound synthesis. In particular, he was interested in creating synthetic sounds following sequences of the chords. Figure 7.13 shows how the composer annotated the symbols that generated the chords with numbers corresponding to their position in an initial sequences. He sorted the chords with various criterion including the position of the symbols in the score, their pitch or their area to create new sequences.

We collaborated with Gilbert Nouno, the musical assistant for the piece, to implement OpenMusic functions to create control data for the synthesis and a new synthesizer in Max. We created function that computes a simplified voice-leading between the chords, i. e., matching each note of a chord to the closest one in the following chord. Figure 7.14 presents a reactive patch in OpenMusic using our voiceleading function to create trajectories between pitches of successive chords. If the pitches are close enough the function connects them in a single trajectory. Otherwise, the previous trajectory vanish and a new one appears. A graphical object displays the resulting trajectories. Using these trajectories, the musical assistant synthesized several sounds that the composer included in the electronic part of the piece.

Controlling spatialization

We also built tools to control the spatialization of both singers and electronic sounds with the pen in real-time. The goal was to use such tools during the performance and to create spatialized sounds to add to the electronic part. For example, Figure 7.15 shows a plugin implemented in Max for Live that controls the position of a sound source in space according to the pen movements streamed via OSC. After drawing a stroke, once the composer raise the pen up, the handwritten stroke become a trajectory followed by the sound source. We implemented several features such as following the trajectory at constant speed and the speed of the writing gesture or following it in reverse mode.

discussion and perspectives

We designed interactive systems that combine interactive paper with a new reactive implementation of his existing patches in OpenMusic and Max patches to support to support input of calligraphic forms and active exploration of the composer's compositional processes with the digital pen.

Interacting with computer-processes on paper

Using the interactive tools, Philippe Leroux was able to interactively visualize recognized shapes and their features in OpenMusic while he was drawing. He assessed and edited the recognition if necessary, either by drawing new strokes or using the graphical user interface. Leroux also frequently used the digital pen as a "pointing" device to select previously drawn shapes and import their features in OpenMusic patches. The composer created reactive patches in which the input data update musical objects, and where the use of play boxes (see for instance Figure 7.10 and Figure 7.12) establishes a direct causality between drawing the gestures and listening to the musical results. By mixing pen interactions with his regular use of the computer-aided composition environment, the composer was able to determine which forms were the most interesting. From these forms, he created a set of chords and note sequences that he used in the piece. He explained that he used some of the patches as if he was "improvising with a piano to search a chord color or a convincing melodic suite."

During the composition of his piece, the composer drew several times over different versions of the paper interface to adjust and edit his calligraphic gestures, while using the feedback in OpenMusic as a reference. Once he was satisfied with a particular result, he could save it, print it and use it as a new support for interaction. In Figure 7.5 for instance, the top component of the interface contains recorded forms from a previous session. Figure 7.16 is the scan of a paper interface used to explore and compare recognized forms. Barely visible dots in the different forms are the traces of pen-pointing interactions. While the current technological set-up offers promising possibilities for the composer, several aspects could still be improved. First, the composer wished he could to use pressure information but the digital pen does not provide an accurate measure of this parameter while writing.

Second, although he appreciated interacting with the ink traces left by the pen, Philippe Leroux explained that he was missing the ability to erase strokes directly on paper, as he would do with a pencil. Instead, he needed to print an updated version of the interface before rewriting the incorrect strokes. Previous work proposed methods to support erasing [START_REF] Olberding | Towards Understanding Erasingbased Interactions: Adding Erasing Capabilities to Anoto Pens[END_REF] but they require building new ink cartridges for the pen.

Finally, Philippe Leroux wanted to freely annotate the results of his explorations on the paper interface. Unfortunately, he could not use the digital pen for that, since it would have interfered with the existing shapes. He thus used a pencil to draw selection of shape and add number corresponding to his OM processes (Figure 7.13). The composer suggested that we could use a non-interactive mode for the pen where the handwritten strokes would not be interpreted by the pen server and just considered as annotations.

Explicit reactive programming in OpenMusic

Our initial implementation of the OpenMusic library had a background client that interpreted incoming OSC messages to update ad-hoc "reactive objects" in OpenMusic. As the exploratory nature of the project required frequent modifications of these, we finally designed the current tools so that visual program reactivity is at the front, in order to let the user determine himself the routing of incoming messages, and change the data interpretation with regular visual programming tools. This approach may lead to more complex patches, but it supports quick change in mappings and parametrization. Figure 7.17 shows the patch that processes all the OSC messages from the substrate and converts them into OpenMusic objects.

Figure 7.17: OpenMusic patch receiving and parsing the data from the substrates.

Ongoing and future work

We are working on the last segment of the project: the use of the digital pen for the live performance. The composer and the musical assistant wanted to use the pen during parts of the performance, when the excerpt of the piece we used on paper will be interpreted by the singers. In particular, the musical assistant would be on stage, as a performer, and would write over the manuscript. His drawings should control live video processing to reveal the score to the audience and control the spatialization. We are working on a Max patch that displays the score in the background and each new stroke drawn with then pen reveals the part of the score underneath. Figure 7.18 shows a screen capture of the video output. We already informally discussed with Philippe Leroux about how these interactive systems influenced his process and asked him to write a short text about it. However, we still need to carefully examine all the data gathered during this project, conduct a more structured interview with the composer and the musical assistant to better understand the impact of such interactive technology.

summary

This chapter presented our research and creation project with Philippe Leroux. We designed interactive systems in close collaboration with the composer Philippe Leroux to let him control musical processes by tracing over calligraphic gestures in an illuminated manuscript.

We showed how the composer adapted and used the tools during the composition of his piece Quid sit musicus. The reactive framework in OpenMusic combined with the pen interactions has proven successful in supporting the exploration of several alternatives as well as refining his musical processes.

We believe that interactive computer-aided composition environments could be fertile for creativity as the composer can engage in a partnership with the computer to iteratively improve his patches and programs while adjusting his handwritten input.

Future work with Philippe Leroux will investigate interactive paper interfaces and tools for the live performance, including real-time spatialization of the singers and video feedback.

contributions

1. Implementation of several substrates to extract features from handwritten calligraphic gestures connected with Max and Open-Music patches to generate musical material and create control data for spatialization and sound synthesis.

2. Found that interactive tools for computer-aided composition help the composer engage in a partnership with the computer too explore his programs and his ideas. In particular, the systematic exploration of the handwritten input with direct audiovisual feedback helps to compare alternatives and refine the musical processes.

"There is no such thing as an empty space or an empty time. There is always something to see, something to hear. In fact, try as we may to make a silence, we cannot." [START_REF] Cage | Silence: Lectures and Writings[END_REF] This dissertation investigates the design of interactive paper interfaces to support musical creation. The main objectives are:

• Observe the creative process of professional composers and understand how linking paper and computer-aided composition environments could support the exploration of their musical ideas.

• Design and implement interactive interfaces that extend existing computer-tools with personal representations of musical data on paper.

• Develop a theoretical framework that captures the diversity of the creative composition process.

To conclude this work, this chapter summarizes the contributions of the thesis and presents future directions for research. This dissertation describes the results of field, design and exploratory studies with composers to understand how linking paper and computers can support their creative process. The studies highlight that composers are purposefully idiosyncratic and they often create new models and personal representations for each piece. Composers use both media in parallel while not being able to truly integrate them. I argue that these concurrent roles often lead composers to abandon their personal representations on paper when they switch to their computer-based environments.

I found that when interacting with drawn curves on paper, the gestures serve as visual and computational elements that could be quickly revisited, replayed and evaluated, as well as refined with new variations. Composers emphasized that integrating interactive paper with their existing composition tools is a promising approach as it gives them an additional physical space on paper to enhance exploration of their ideas. I investigate the creation of personal paper interfaces with formal music representation systems on paper that embed musical constraints and allow composers to visually explore musical ideas and facilitate their entry. I observed that non-musicians began by scribbling on paper to "sonify" their gestures with Tonnetz representations whereas experienced musicians tried to represent their ideas in the Tonnetz space.

I conducted a structured observation study to compare composers's composition process and how they appropriate interactive tools. The study demonstrates that composers can successfully adapt a "Polyphonic" user interface with multiple synchronized input and output modalities to their personal composition styles. Composers used two main strategies to complete the task. Some interact with the tool before refining and adapting their ideas, and others begin with definition of ideas and then implement them.

Finally, we collaborated with a composer and a musical assistant for the composition of the piece "Quid sit musicus". The project emphasizes that expanding existing computer-aided composition environments with interactive technology helps the composer engage in a partnership with these tools to iteratively refine both his input and his musical processes.

Design and implementation of interactive systems

InkSplorer lets composers control computer-based processes in Max and OpenMusic with handwritten curves on paper. I conducted a participatory design study with composers to explore several pen interactions and tangible manipulations of paper components to work with their musical data. I then designed Paper Substrates, interactive paper components that let composers create their own paper interfaces with personal representations of computer-based musical data. Paper Substrates are modular and can be spatially positioned, dynamically connected or overlaid. These physical interactions allow composers to control distinct parameters of a same musical object with several substrates and perform musical operation such as aligning elements along a common time-line.

I created the Paper Substrates Toolkit and PaperComposer to facilitate the creation of personal interfaces with substrates and their use by composers. Both tools are extensible to support new substrates or to be adapted to particular context such as a controlled study to log pen events and communication with external applications. I have used these tools to design several new interactive systems. I worked with composers to create new substrates to write musical scores and control sound synthesis. PaperTonnetz uses Tonnetz representation in which paths are chords or melodies. Polyphony synchronizes interactive pa-per with standard interface and physical controllers to composer a short electroacoustic piece.

In collaboration with a composer and a musical assistant, I implemented interactive paper interfaces to compose with calligraphic gestures written over a medieval manuscript and experiment computerbased composition processes by interacting with the pen.

Separation of structure and musical data

The concept of substrates builds upon the distinction made by composers between the underlying structure in which they write their ideas and the musical content itself. Paper Substrates captures the process by which composers create complex structures to represent their musical ideas. Paper Substrates support the entry, editing and refinement of musical ideas by allowing composers to create structured representations that support computation. This framework both captures how composers already generate novel ideas and also provides the architecture for interactive paper-based tools that facilitate experimentation on both paper and the computer.

discussion

Individual composers, their teachers and researchers who study them can benefit from the insights about both the details of the composition process and, more importantly, the concept of substrates for articulating that process. Similarly, researchers and designers of interactive systems can consider extending interfaces with interactive paper to support creative exploration of computer-based data.

Composers could use the existing substrates to connect their paperbased activities with their computer-aided composition environments. They can build their own interfaces on paper to draw and refine curves, write musical sequences with simplified musical notation or by drawing in Tonnetz representations and control their software with paper buttons, playbar and sliders.

My collaborations with composers emphasize possible applications of these interfaces for music education and music analysis. Using direct feedback of handwritten music could be valuable for composition students, mostly to quickly associate symbolic notation with sounds or to learn complex chords or microtonal scales. Teachers can take advantage of the Tonnetz representations that has been proven useful to learn and analyze harmony (Holland, 1994;[START_REF] Chouvel | Analyser l'harmonie -aux frontières de la tonalité[END_REF]. Interacting with such representations on paper could foster internalization of harmony and support music analysis. Our approach may also encourage composers to further develop their own structured representation systems to use them as substrates.

Tracking the written elements and the pen interactions could help researchers to collect quantitative data that they can correlate with phases of the composition process. In addition to existing methods [START_REF] Eaglestone | Composition systems requirements for creativity: what research methodology[END_REF], the structured observation method and the associated composition task can be adapted to make observations about interactive technology for creativity in controlled settings. Such studies could help to compare behaviors of professional composers and examine how they appropriate the new tools.

For developers of computer music interfaces, interactive paper can make the use of such tools more accessible to beginners. During public demonstrations, we observe that using movable pieces of paper helped children access music creation tools in a playful manner. We also get much interest by musicians and composers that see new perspectives to express and explore musical ideas.

My work with researchers, musical assistants and composers suggests that paper interfaces could support collaboration. In contrast to computer screens, the interactive paper interfaces were usually in the center of the group of people involved, and we frequently exchanged the pen to interact or to add annotations for improvements over the same sheet. Paper interfaces could create opportunities to support interaction in collaborative environments.

Composers can be seen as lead users in the Von Hippel's (2007) sense, i. e., "users whose present strong needs will become general in a marketplace months or years in the future". I believe that the Paper Substrates approach could be valuable in other application domains, even if it would require further investigations. For example, results by [START_REF] Oviatt | The impact of interface affordances on human ideation, problem solving, and inferential reasoning[END_REF] provide evidence about the value of interactive paper on problem-solving tasks. I envision that paper substrates could enhance mathematical programming tools such as MATLAB, R, and Mathematica with interactive paper representations adapted to tasks of individual users. Deciding on the parameters of an equation, approximating solutions through graphical representation of mathematical functions, making calculations over tabular structures are examples of tasks that could benefit from our approach.

Directions for future research

Of course the concept of substrate has limitations. We have established that this phenomenon of separating the structure and the musical content exists in the group of composers we have studied, but we need to conduct more research to verify its generalization, both in music composition and other creative fields. Also, the current implementation of Paper Substrates requires a programmer to design and implement a new substrates. This suggests a number of direction for future research.

Extending and improving Paper Substrates

A main challenge concerns the implementation and definition of new Paper Substrates. We need tools that let composers combine higher level primitives to define the underlying structures of their substrates and specify the interpretation of pen interactions. Instead of systematically defining the whole interface in the computer before printing and using it, I am particularly interested in investigating the interactive definition of templates by drawing them directly on paper before or after writing musical content. For example, the Livescribe pen allows users to draw a piano keyboard made of vertical and horizontal lines anywhere on the page. Once drawn the piano, the pen plays the corresponding note when the pen tip enters a key. I want to offer similar functionalities with musical staff or graph containers in particular. This would also help composers better integrate substrates in their composition process with progressive transitions between ill-formed ideas and more formal ones.

Additional feedback modalities and physical interactions with pen and paper could support the interactive definition of substrates and help composers discover their substrates. We can explore bending gestures [START_REF] Lahey | Paper-Phone: understanding the use of bend gestures in mobile devices with flexible electronic paper displays[END_REF], pressure sensing as well as physical interactions with the pen [START_REF] Song | Grips and gestures on a multi-touch pen[END_REF] to let composers experiment before writing their ideas or switch between pen modes, e. g., interactive and non-interactive modes to freely annotate results.

Currently, Substrates provide audio and visual feedback via external applications in response to events and data transmitted by the substrates. I am interested in exploring additional alternatives such as bimanual interaction techniques based on the use of portable miniprojectors [START_REF] Song | MouseLight: bimanual interactions on digital paper using a pen and a spatially-aware mobile projector[END_REF] and mobile phones [START_REF] Tsandilas | Interpreting strokes on paper with a mobile assistant[END_REF] that bring visual feedback close to the writing workspace. Other feedback modalities such as haptic feedback [START_REF] Liao | Pen-top feedback for paper-based interfaces[END_REF] or auditory feedback are particularly interesting directions, but should not disturb users while they are executing precise drawings or listening to the musical result. Auditory augmentation [START_REF] Bovermann | Auditory augmentation[END_REF] is an interesting approach because it combines the sound of a physical object with a filtered version according to data streams. We can use such method to notify the composer that a recognition problem occurred by slightly filtering the sound made by the pen while writing.

I am interested in extending the Paper Substrates Toolkit and Paper-Composer to enable programmers and users to define or reconfigure audio and visual feedback in response to pen actions or physical interactions at the level of the actual substrate, where visual, auditory and haptic feedback is provided by any of the above methods. This will allow developers to generate richer paper applications that combine both physical and virtual instances of a substrate.

Interactive structures for creativity

Over the long term, I plan to extend the research questions explored in this dissertation to investigate new ways of enhancing the creative composition process. For example, I would like to study the design of interactive systems able to help users become experts, guide the entry of the content and facilitate the exploration as well as the refinement of their musical ideas. Writing on paper is powerful because it enables users to reflect upon an idea and abstract many performance aspects. However, how can we cope with data such as three-dimensional trajectories that cannot be fully represented on paper? How can we help musicians to learn complex digital music instruments? I propose to investigate interactive structures to guide the input with visual, auditory and haptic feedback and feedforward modalities.

For example, to help users learn and perform gestures, we can apply existing research such as Octopocus [START_REF] Bau | OctoPocus: a dynamic guide for learning gesture-based command sets[END_REF]. Octopocus, visually reveals the available gestures and continuously reacts to the user's input to provide updated feedback about the current state of the recognition. In this case, the predefined gestures define the underlying structure that help selecting a command. We could explore the use of haptic feedback to reinforce such structures and make them accessible in non-visual interfaces such as digital music instrument.

The structures could also help for exploration and refinement of musical ideas by using the data itself. [START_REF] Zadel | Haptic Feedback for Different Strokes Using DIMPLE[END_REF] explored the use of haptic feedback to enhance a sketching musical application. The haptic feedback helped users detect when they intersect existing strokes and provide more precise control of the drawing. In addition to such approaches where the goal is to "feel" existing elements, we can also explore haptic feedforward for refining gestures. A possible application could be to edit existing spatialization trajectories with an haptic device. The haptic device could follow the motion of the trajectory and its movements could be altered by the composer to iteratively refine and improve the trajectory.

In conclusion, I have observed real-world composition process with a number of methods and identified a fundamental underlying principle that characterizes the composition creative process. Composers create and use structures in which they express and explore their ideas with their own musical representations. This concept of substrate allowed to create interactive tools that preserve the simplicity of expressing ideas on paper while enabling a more fluid transition to computer-aided composition environments. I have demonstrated diverse interactive systems using interactive paper that support expression, exploration and execution of musical ideas.

A S T U D Y M AT E R I A L

Appendix A presents the studies' material including the cover sheets, interview guides, comparison forms and questionnaires. Section II: Processus de composition. We describe the results of an initial test of the system in a public setting, and how we revised PaperTonnetz to better support three activities: discovering, improvising and assembling musical sequences in a Tonnetz. We conclude with a discussion of directions for future research with respect to creating novel paper-based interactive music representations to support musical composition.

INTRODUCTION

A Tonnetz, or "tone-network" in German, is a two-dimensional representation of the relationships among pitches. Individual pitches are laid out along multiple axes: each intersection is displayed as a cell with a specific pitch. Created by Euler in 1739, Tonnetze were first used to represent just intonation, with equal distances between the pitches. In the mid 1800s, Riemann explored the use of these spaces to chart harmonic motion between chords and modulation among keys. More recently, Tonnetze have been used to analyze classes of pitches, representing neo-Riemannian transformations [1] in which every cell is associated with one of 12 pitches, independently of octave. The vertical axis organizes pitches in cycles of fifths; the two diagonal axes represent minor third and major third cycles. Figure 1 illustrates a region of a neo-Riemannian Tonnetz that extends infinitely in all directions and helps visualize harmonic chord progressions. Creating spatial relationships that follow precise mathematical properties, such as the interval cycles that underlie harmonic progressions, offers opportunities for both analysis and composition. Systems such as Harmony Space [START_REF] Holland | Learning with harmony space: an overview[END_REF] and Isochords [START_REF] Bergstrom | Isochords: visualizing structure in music[END_REF] allow composers to analyze chords, harmony and MIDI sequences using different Tonnetz representations. We are interested in another role for Tonnetze, i.e. to support music composition. We were inspired by Jean-Marc Chouvel's composition process [START_REF] Chouvel | Traversée du vent et de la lumière. six remarques pour une phénoménologie de la création musicale[END_REF], in which he designs his own Tonnetz, prints them on paper, and then generates musical sequences. He draws paths through sets of Tonnetz cells, which provides the inherent order necessary to define musical sequences. After exploring his ideas, he then transforms them into a standard format on a musical score (see Figure 2). Although this works well, we wanted to take his approach one step further to provide a direct link between paths drawn on a paper Tonnetz and on-line composition tools.

This paper presents PaperTonnetz which lets composers create their own Tonnetz representations, print them on interactive paper, and 'play' them with a pen. We use Anoto technology1 to detect the position of a digital pen as it draws ink on paper and produces the corresponding pitch. PaperTonnetz can be used as a simple performance tool, similar to the tablet-based systems described above, but with an emphasis on drawing paths rather than pressing buttons. However, PaperTonnetz can also be used as a sophisticated composition tool, enabling the composer to capture, replay, transform and compare sequences and chords. Our goal is to combine the flexibility of paper and the power of the computer to help composers explore new creative possibilities.

Section 2 describes related work on the use of Tonnetze to support performance as well as related research on interactive paper to support composition. Next, we discuss how Tonnetze can be designed to form a composition space. We then introduce PaperTonnetz and describe our initial study of users, both novices and experienced musicians, as they tried it at a public event. We explain how the results influenced the design of PaperTonnetz to explicitly support discovery, improvisation and assembly of complex musical sequences. We conclude with a discussion of the benefits of using interactive paper and Tonnetze to support music composition as well as directions for future research.

RELATED WORK

Tonnetze may be implemented on a computer, enabling musicians to use them for performances. For example, C-Thru 2 , creates physical Tonnetz keyboard in which each cell is a physical hexagonal button. The user can play individual notes by pressing one button at a time, or play several simultaneously to create a chord.

Tablet-based systems such as Musix and IsoKey offer users a customizable isomorphic layout on the screen. Like C-Thru's products, users can press the hexagonal cells to produce individual pitches, melodies and chord patterns. They also allow users to switch among different Tonnetz layouts during a performance. Maupin et al. [START_REF] Maupin | Isomorphic tessellations for musical keyboards[END_REF] examined the harmonic and melodic characteristics of different square and hexagonal pitch layouts and identified which layouts are better suited to different types of improvisation, within particular harmonic contexts.

Although these systems provide interesting performance opportunities, they were not designed to support composition. Computer-based tools that are specifically intended for composition are usually dedicated to the later stages of the creative process [START_REF] Garcia | Inksplorer: Exploring musical ideas on paper and computer[END_REF], such as assigning sequences in time, editing scores or experimenting with advanced forms of sound synthesis. Most of these tools involve learning a specific language, such as MAX/MSP 3 or OpenMusic [START_REF] Agon | OpenMusic : Un langage visuel pour la composition musicale assistee par ordinateur[END_REF], although a few provide more direct methods, such as Buxton et al.'s [START_REF] Buxton | The use of hierarchy and instance in a data structure for computer music[END_REF] techniques for interpreting hand-drawn musical notations.

Yet even composers with computer skills and access to advanced composition systems prefer to sketch their early creative ideas on paper [START_REF] Thiebaut | Drawing electroacoustic music[END_REF][START_REF] Letondal | Paperoles et musique[END_REF]. Paper helps creative professionals externalize their ideas [START_REF] Sellen | The myth of the paperless office[END_REF] and composers often create their own personal languages to express musical ideas [START_REF] Tsandilas | Musink: composing music through augmented drawing[END_REF], which are difficult to translate into computer terms.

One method for bridging the gap between physical paper and composition software is interactive paper. The most common approach uses Anoto technology [START_REF] Costa-Cunha | Augmented paper and anoto stylus[END_REF], which uses a digital pen to capture the precise location as the pen moves on the paper. The pen contains an integrated video camera in the tip. The paper is printed with tiny, almost invisible dots that create a unique, identifiable pattern. When the user draws on this paper with the pen, the camera detects the precise location of the pen on the page, as well as the time it was written, the pen that was used and even which page. This information can be processed by the pen, for simple task such as playing a sound, or simultaneously transmitted to a more sophisticated application on a computer.

In previous work, we explored several techniques for incorporating interactive paper into the composition process. For example, Musink allows composers to define their own vocabulary of annotations on musical scores, which can then be interpreted as functions in computer-aided music composition software such as OpenMusic. Similarly, Ink-Splorer [START_REF] Garcia | Inksplorer: Exploring musical ideas on paper and computer[END_REF] lets composers experiment with different curves drawn on paper to control computer-based algorithms and then use them as a resource for new compositions. We are interested in allowing composers to use Tonnetz as an aid to the composition process. This requires creating a Tonnetz-based composition space, discussed in section 3, and an interactive paper-based system for letting composers capture and modify musical expressions, described in section 4.

TONNETZ AS A COMPOSITION SPACE

Composition example

Chouvel's work, entitled Traversée [4], offers a compelling example of how a Tonnetz can be used as a tool for exploring composition ideas. After creating and printing a Tonnetz on paper, Chouvel drew differently colored paths through the hexagonal cells, each representing a set of pitch sequences. He called the resulting geometrical shapes 'constellations' because they look like the line drawings that connect individual stars in a map of the night sky. Once drawn, he was able to perform geometrical transformations on the paths and examine the results, represented as notes on a standard musical score. Figure 2 illustrates this process: note that the paths that link different pitches may be treated as a sequence, to create a melody, or in parallel, to create a chord.

Le parcours des constellations est parfaitement arbitraire, mais absolument régulier, et dessine pour chaque lettre une figure différente. Ces parcours exploitent évidemment les propriétés de l'espace hexagonal et opèrent des transformations intervalliques importantes en conservant une sorte de logique qui n'a rien à voir avec la transposition. Le total chromatique permet de reconnaître d'autres parcours imbriqués et donc de définir des suites « négatives » de notes. Voici l'exemple de la lettre L qui est celle du premier temps de la pièce. Chouvel's approach makes it possible to explore a wide variety of geometrical transformations, including translations, rotations and homothetic transformations. His Ton-netz can thus support complex musical operations that are rarely formalized in traditional music theory. We build upon Chouvel's approach to support three aspects of composition:

• expressing musical ideas by drawing paths through a Tonnetz space

• performing geometrical transformations on those ideas

• translating those ideas into playable music.

3.2 Two dimensional pitch layouts

Tonnetz layouts

We focus on pitch-class Tonnetze with three interval classes laid out in a hexagonal pattern. Pitch-class Tonnetze have reached to many algebraic considerations. Works presented in this paper are inspired and belong to recent researches on the use of spatial computing to investigate these groupbased representations [START_REF] Giavitto | Declarative definition of group indexed data structures and approximation of their domains[END_REF]. The three axes (vertical, diagonal left and diagonal right) of hexagonal Tonnetze correspond to the three elementary moves illustrated on Figure 3. Here, every sequence of three steps returns to the starting point. Thus the sum of the interval classes i1, i2 and i3 is always equal to 0 (mod. 12) with respect the hexagonal layout. Figure 3 shows how pitch classes can be represented with a hexagonal cell with six neighbors, each of which can be accessed by adding to the pitch class the three interval classes and their respective inverses. For example, in the Tonnetz Figure 1, one can jump from a fifth by moving up the vertical axis. Moving in the opposite direction adds a fourth to the current pitch class. This offers the composer a one-step method for transposing any melodic sequence or chord.

Catanzaros [START_REF] Catanzaro | Generalized tonnetze[END_REF] lists the 12 possible combinations of interval classes that respect this constraint and can be laid out in a hexagonal pattern. Maupin et al. [5] discuss the advantages and disadvantages of different Tonnetz layouts for melody and harmony, within the context of a performance. These aspects can be discussed from a compositional point of view. The composer can choose a Tonnetz with neighborhood intervals that are similar to the desired chord intervals to represent it as a compact shape. In particular, every three-note chord can be represented with a triangular shape, in one of the 12 hexagonal Tonnetz. For example, T [START_REF] Bergstrom | Isochords: visualizing structure in music[END_REF][START_REF] Chouvel | Traversée du vent et de la lumière. six remarques pour une phénoménologie de la création musicale[END_REF][START_REF] Maupin | Isomorphic tessellations for musical keyboards[END_REF] is neo-Riemannian Tonnetz in which any three adjacent hexagons form a major or a minor chord.

Heptatonic layouts

We have also been exploring more complex problems, such as how to use a hexagonal layout to represent a seven-pitch (heptatonic) scale. Figure 4 shows how seven pitch classes can be laid out in a chromatic circle such that any move between two pitch classes can be designated by one, two or three steps, up or down from the initial pitch class. These six translations can be associated with the six possible moves in the hexagonal space. Although the resulting space noticed in [START_REF] Bigo | Building topological spaces for musical objects[END_REF] is not a perfect Tonnetz since the axes do not represent constant semitone intervals, it still offers a powerful tool for composition.

In summary, Tonnetze can be used to represent interval cycles, chords, and scales, and offer composers a powerful new tool to support composition. The next section describes our work on PaperTonnetz, which offers both a novel form of Tonnetz and the use of interactive paper to link the expression of musical ideas on paper to real-world compositions.

STUDY OF INTERACTIVE PAPER TONNETZ

We chose the MCM '11 conference in Paris (Mathematic and Computation in Music, June 2011), to learn how users, both non-musicians and musicians, would respond to an interactive paper Tonnetz interface. Figure 5 shows two hexagonal Tonnetze: T [START_REF] Bergstrom | Isochords: visualizing structure in music[END_REF][START_REF] Chouvel | Traversée du vent et de la lumière. six remarques pour une phénoménologie de la création musicale[END_REF][START_REF] Maupin | Isomorphic tessellations for musical keyboards[END_REF] and T [1,[START_REF] Holland | Learning with harmony space: an overview[END_REF][START_REF] Thiebaut | Drawing electroacoustic music[END_REF] which were printed with the Anoto dot pattern to create interactive paper. These Tonnetze provide different musical properties: the former uses large intervals, the latter uses small intervals.

Participants used the ADP-301 digital pen to hear pitches in real time, as they draw lines from one hexagonal cell to another. The pitch changes as the pen crosses a boundary and continues for as long as the pen stays in that cell. Participants can thus create melodic sequences and control timing through their gestures.

The Tonnetz does not dictate a particular octave, so we chose an arbitrary set, with the 12 pitches from C4 (60) to B4 (71). Users could choose 'parsimonious mode' to obtain more melodic results: We minimized the interval between subsequent notes by re-calculating each new pitch class. For example, moving from a C to a B in this mode creates a short pitch transition from C4 (60) to B3 (59) instead of from C4 (60) to B4 (71).

Technical details

The first version of PaperTonnetz was written in Java and uses ADP-301 digital pens to communicate via Bluetooth. The Tonnetze are printed with Anoto dot patterns that can distinguish among 20 unique pages. The Open Sound Control Protocol [START_REF] Chouvel | Analyser l'harmonie -aux frontières de la tonalité[END_REF] communicates run-time data to a Max/MSP patch, which plays pitches in real-time or routes them to other MIDI-or OSC-compatible applications.

User study

We set up a table at MCM'11 with copies of both types of Tonnetz. Approximately 30 people participated, including both professional musicians and non-musicians. We began with a quick explanation of Tonnetze and interactive paper technology. We then watched as they tried and compared the two Tonnetz patterns. We answered their questions about interactive paper, computing and music theory and then asked them to tell us their perceptions of the two different layouts and of the interactive paper Tonnetz. We concluded by asking them whether and how such a tool might help them either to perform or to compose music.

Results

Trying the prototype: Most participants began by tapping a particular cell to produce a sound. They would then scribble with the pen, covering a large number of cells and listening to the result. The real-time sonic feedback encouraged them to explore different patterns and to compare the differences between the two Tonnetze.

Sonification and Representation: Participants realized that drawing particular shapes resulted in identifiable sound patterns, which gave them clues about the differing properties of the two Tonnetze. Non-musicians were able to detect differences, but did not understand them. Musicians, however, performed systematic explorations in order to understand the musical relationships among the cells. They attempted to draw specific melodies or scales in order to understand how they were represented in the network.

Comparing Tonnetze: After becoming familiar with how the system works, most participants tried to compare the two Tonnetze. We suggested that they draw straight lines and geometrical shapes such as triangles or polygons in both networks and compared the sound results. Although both novices and musicians were able to hear clear differences between the two Tonnetze, only the musicians were able to explain what they were hearing. Table 1 compares T [START_REF] Bergstrom | Isochords: visualizing structure in music[END_REF][START_REF] Chouvel | Traversée du vent et de la lumière. six remarques pour une phénoménologie de la création musicale[END_REF][START_REF] Maupin | Isomorphic tessellations for musical keyboards[END_REF] T [1,[START_REF] Holland | Learning with harmony space: an overview[END_REF][START_REF] Thiebaut | Drawing electroacoustic music[END_REF] Non-musicians Small cyclic moves on neighbor regions sound well.

Continuous paths can create melodies that can be sung.

Musicians

Produce easily arpeggios.

Difficult to play a scale.

Many scales are easy to perform. Melodies are easy to reproduce.

Table 1. Participants comparison between the two Tonnetz.

the reactions of musicians and non-musicians with respect to each of the two Tonnetze. Their reactions were consistent with the harmonic and melodic properties of hexagonal notes layouts studied in [START_REF] Maupin | Isomorphic tessellations for musical keyboards[END_REF].

Limitations: The musicians identified a number of shortcomings of the first prototype. They pointed out the need to express chords and felt that version 1 would not be sufficient to fully explore a musical idea. They wanted to reuse previously drawn curves, preferring to tap instead of redrawing the path in order to listen to it again. Most musicians asked for additional pitch layouts; some even suggested specific alternatives. A few musicians also asked if they could generate personal networks that contained irregular shapes and neighborhood properties.

PAPERTONNETZ: A COMPOSITION TOOL

Based on feedback from the first study, we decided to redesign the PaperTonnetz prototype into a more general composition tool (see online video4 ). We incorporated a number of ideas from Chouvel, including highlighting the path, defined as a trace of ink over several cells, as the central interaction element. We gave composers the ability to listen to chord and note sequences in real time and provided a link between the paths on paper and notes represented on a musical score. Finally, we allow composers to choose among different Tonnetze or design their own. Once the composer has laid out the desired Tonnetz in the workspace, the document can be printed with any goodquality printer. PaperTonnetz prints the user's design onto an Anoto dot pattern and the user can interact with the resulting page with a digital pen. Figure 6, provides an example of how a composer can use PaperTonnetz to explore ideas for a new composition.

Creating paper interface

In this scenario, the composer, " Hugo" , creates a composition in C minor key with three voices: bass part, lead part and chords part. The interface consists in series of commands on the top, two icons representing sequencers and Tonnetze to the right and a large workspace for assembling the elements of the printed paper page in the center. In this example, Hugo begins by dragging a Tonnetz icon to the workspace. He resizes it and places it to the left of the workspace. Next, he configures it: He chooses a hexagonal pattern and the specific Tonnetz intervals, after which he labels it: " C minor Melody" . This Tonnetz is in heptatonic C minor and requires the most space, since Hugo will use it to create a variety of melodies in that key.

Hugo repeats drags another Tonnetz icon to create the bass part in the upper left corner. He creates a smaller Tonnetz, zoomed in with larger individual cells. He labels it: " T [1,[START_REF] Holland | Learning with harmony space: an overview[END_REF][START_REF] Thiebaut | Drawing electroacoustic music[END_REF] Bass" to help him remember the particular intervals he chose. This one is particularly appropriate for the bass part, since the intervals are short, so drawing paths requires only small movements.

Finally, Hugo adds the chord part, in the lower left hand corner. Here, he chooses to represent pitches as circles rather than hexagons, with pairs of lines between them. If the user touches the double lines, the two interconnected pitches are played together as a chord. If they touch the triangular space within three circles, the three corresponding pitches are played together. He labels it " T [START_REF] Bergstrom | Isochords: visualizing structure in music[END_REF][START_REF] Chouvel | Traversée du vent et de la lumière. six remarques pour une phénoménologie de la création musicale[END_REF][START_REF] Maupin | Isomorphic tessellations for musical keyboards[END_REF] Chords" , a Tonnetz that we have already seen is particularly appropriate for playing major and minor chord patterns, since each can be played by drawing a compact, triangular path.

Assembling Sequences

Hugo next includes a sequencer by dragging the sequencer icon to the space below the " C minor Melody" Tonnetz. The sequencer will allow the composer to create and assemble music sequences that are drawn in the other three Tonnetz spaces. The sequencer consists in a band of rectangles, each of which can contain a track (a set of paths within the Tonnetz). Tracks are stored as MIDI files that can be sent to the MAX/MSP synthesizer to be played or for additional processing.

Supporting Composition

Hugo prints a paper copy of the workspace, which now contains three different Tonnetze (for the melody,bass and chords) as well as a sequencer to capture and assemble individual paths that become standard MIDI tracks. In Figure 7, Hugo has drawn three paths in the " C minor Melody" heptatonic Tonnetz, which correspond to three chords, C minor seventh, C minor and F minor. These are translated into notes on the score at the left, based on the length of time spent in each cell. He can replay any path simply by touching it with the pen: this preserves the rhythmic properties of the original path. Hugo can also play the path as a chord by holding the pen on a path: all notes are played simultaneously. Any created path can be reused many times, either as a sequence or as a chord.

Hugo can also assemble sequences of paths, to create musical tracks. He draws a stroke in one of the boxes in the sequencer. If he clicks on a path, it is added as a sequence of notes; if he holds the path, the notes in the path are added as a chord. He can use the same process to add additional sequences or chords. To complete the track, Hugo draws a second stroke in the sequencer box. This creates a track that is displayed on a score (Figure 7: right). Here, Hugo has created a series of three chords. He clicks on the box to hear the sequence which is rendered from a bach.roll5 object played with a Max/MSP patch). Hugo repeats the process, creating a series of tracks, which can be played in series or at the same time.

Supporting Improvisation

Although the paper Tonnetz is designed primarily for composition, it can also be used for performance. As in our user study, novices and students can explore the different auditory relationships among pitches by clicking on cells and drawing paths through them. The duration of each pitch is defined by the length of time that the pen stays in each cell. More advanced musicians will be able to capture their improvisations and explore musical ideas in real time.

DISCUSSION

Discovering, Improvising and Assembling

We designed PaperTonnetz to accommodate three primary musical activities: discovery, improvisation and assembly, each of which adds requirements to the user interface.

Discovery: PaperTonnetz makes each cell and path interactive so they can reveal their musical properties and functionalities. This allows both novices and experts to discover its properties, with real-time auditory feedback.

Improvisation: PaperTonnetz allows previously drawn paths to be accessible for future interaction, including playing, transformation, and refinement. The user can create individual chords and melodies and then reuse, recombine and modify them to explore different possibilities.

Assembly: PaperTonnetz enables expert users to capture musical sequences based on their improvisations and to export them to other music composition systems. For example, music sequencers can handle the MIDI sequences produced by PaperTonnetz for advanced arranging and editing. PaperTonnetz supports both MIDI and OSC protocols, so any musical sequence created with it can be imported and edited in any other dedicated application. This could be improved by providing a plug-in version of PaperTonnetz that can be hosted by a compatible digital audio workstation.

Interacting with drawn gestures

We use the notion of path as a main interactive element dynamically created by the user. We believe that a trace left on paper by the pen can be a simple and powerful way to get back to previous improvisation. On the other hand, this approach includes some practical limitations, such as the saturation of some regions after many drawings. As we see in Figure 2, the composer reuses the same path with a different orientation and position in space. Then, providing tools and interaction to reuse drawn paths in different networks is a future direction. Using a screen-based tablet can efficiently help such tasks. However, paper affords sketching and writing by using common and personal notation. Then, it supports composers in expressing and exploring their musical ideas [START_REF] Garcia | Inksplorer: Exploring musical ideas on paper and computer[END_REF][START_REF] Thiebaut | Drawing electroacoustic music[END_REF][START_REF] Letondal | Paperoles et musique[END_REF].

We are also interested in using the gestures properties to control musical parameters. Mapping the pen pressure to the musical velocity is a possible future investigation but the technology is not yet sufficiently reliable to control it properly.

Spatial Representations

Our first prototype proposed the use of two Tonnetze, T [START_REF] Bergstrom | Isochords: visualizing structure in music[END_REF][START_REF] Chouvel | Traversée du vent et de la lumière. six remarques pour une phénoménologie de la création musicale[END_REF][START_REF] Maupin | Isomorphic tessellations for musical keyboards[END_REF] and T [1,[START_REF] Holland | Learning with harmony space: an overview[END_REF][START_REF] Thiebaut | Drawing electroacoustic music[END_REF]. The second version of PaperTonnetz goes one step further by supporting a range of different Tonnetze. We have also started exploring how to advance some of the graphical properties of Tonnetze in space by taking advantage of the physical properties of paper. Paper can be cut, folded and physically moved without losing its functionalities.

We envision that composers could cut their preferred paths or areas on paper and then recombine them, keeping and reusing their most interesting material. Moreover, paper can be used to create three-dimensional shapes, e.g., a cube, that represent new continuous spaces to explore. For example, in the case of heptatonic hexagonal spaces, switching from one face to another would result in a tonality modulation (Figure 8). Another possible direction with hexagonal layouts is to represent microtonal pitch spaces. Chouvel [START_REF] Chouvel | Analyser l'harmonie -aux frontières de la tonalité[END_REF] has previously discussed this direction, while recent work [START_REF] Prechtl | A midi sequencer that widens access to the compositional possibilities of novel tunings[END_REF] proposed an application to handle these concepts. This could open new perspectives for music composition.

In this work, we have mostly focused on hexagonal Tonnetz representations but our goal is to enable composers to create their own musical spaces with arbitrary shapes or more complex neighboring relationships between elements. In order to enable continued strokes with the pen, we are interested in the different ways to tessellate the plan. We can imagine both regular (e.g., the hexagonal layout) and irregular tessellations. Such tessellations could include several types of shapes, linked with different neighboring relationships.

CONCLUSION

In this paper, we investigated the use of Tonnetz representations to support computer aided composition. There is a wide range of pitch layouts based on Tonnetze to represent musical properties or to simplify the performance of scales. We proposed new layouts based on heptatonic scales to extend composition possibilities in two-dimensional pitch spaces. We presented PaperTonnetz, a user interface for composing with interactive paper-based Tonnetz representations. A user study helped us redesign PaperTonnetz and integrate handwritten gestures into interaction with paper so that users can create chords and musical sequences. We presented a set of interaction techniques to facilitate the discovery, improvisation and assembly of musical sequences on paper. We also extended our tool to enable composers to build their own interactive paper interfaces.

In future work, we plan to collaborate with composers to extend its control capabilities for rhythm and pitch, especially the use of several octaves. We are also interested in integrating PaperTonnetz into existing musical software such as OpenMusic and common digital audio workstation applications.

1 1 I

 11 Exploratory Evaluation of InkSplorer . . . . . . . . . . a.1.1 Cover sheet . . . . . . . . . . . . . . . . . . . . . a.1.2 Pre questionnaire . . . . . . . . . . . . . . . . . . a.1.3 Interview guide . . . . . . . . . . . . . . . . . . . a.1.4 Comparison form . . . . . . . . . . . . . . . . . . a.1.5 Post questionnaire . . . . . . . . . . . . . . . . . a.2 Participatory design study of Paper Substrates . . . . . a.2.1 Cover Sheet . . . . . . . . . . . . . . . . . . . . . a.3 Structured observation study of Polyphony . . . . . . . a.3.1 Cover Sheet . . . . . . . . . . . . . . . . . . . . . a.3.2 Pre questionnaire . . . . . . . . . . . . . . . . . . a.3.3 Post questionnaire . . . . . . . . . . . . . . . . . b final scores from the composition study c papertonnetz N T R O D U C T I O N

Figure 1 . 1 :

 11 Figure 1.1: Thesis process. My research includes a variety of qualitative and quantitative methods that include observing and working with composers in field and laboratory settings, designing a variety of interactive paperbased technologies to support the creative process, connected with a common theoretical foundation.

Figure 2 . 1 :

 21 Figure 2.1: Musical notations

Figure 2

 2 Figure 2.2: A simple synthesizer implemented in a PureData patch. c Miller Puckette

Figure 2 . 3 :

 23 Figure 2.3: Left: OpenMusic patch to compute a musical sequence. Right: A maquette that represents temporal objects.

Figure 2 .Figure 2 . 4 :

 224 Figure 2.4: Digital scores

Figure 2 . 5 :

 25 Figure 2.5: Augmented desks

Figure 2 . 6 :

 26 Figure 2.6: The Anoto technology c Anoto Group AB

Figure 2

 2 Figure 2.7: Echo smartpen c Livescribe

Figure 2 . 8 :

 28 Figure 2.8: Musink helps composers define their gesture vocabulary on paper and link it with functions in OpenMusic

Figure 2 . 9 :

 29 Figure 2.9: Augmented Notebooks

Figure 2 . 10 :

 210 Figure 2.10: Physical interactions

Figure 2 . 11 :

 211 Figure 2.11: Feedback for pen and paper interfaces

Figure 3 . 1 :

 31 Figure 3.1: Composers' early sketches

Figure 3 . 3 :

 33 Figure 3.3: Composers' sketches

Figure 3 . 2 :

 32 Figure 3.2: MB's early sketches.

  Figure 3.5

Figure 3 . 5 :

 35 Figure 3.5: EM's indications for speakers configuration

Figure 3 . 4 :

 34 Figure 3.4: AE's sketches for electronic processes with indications of durations, pitches, rhythms and parameters. After making several early sketches to explore the structure of the piece (a), he produced a detailed version (b).

Figure 3 . 6 :

 36 Figure 3.6: NM annotations over musical scores generated with OpenMusic.

Figure 3 . 7 :

 37 Figure 3.7: Mixed representations on paper by MB. Left: a sketch of a patch with curves as input and graphical elements showing the expected result. Right: The resulting patch in OpenMusic after several changes and evolutions.

Figure 3 . 8 :

 38 Figure 3.8: Evolution of musical representations on paper (MB)

Figure 3 . 9 :

 39 Figure 3.9: Chronology of the composition process. Figure from Letondal and Mackay (2007).

Figure 3 .Figure 3 . 10 :

 3310 Figure 3.10 illustrates how composers use paper and software in parallel, without being able to truly integrate them.

Figure 3 . 11 :

 311 Figure 3.11: MB's score segment with small gestures combined with rhythms, time signatures, delimiters and labels.

Figure 3 . 12 :

 312 Figure 3.12: Digital strokes and custom paper format from the study

Figure 3 . 13 :Figure 3 .

 3133 Figure 3.13: Video prototype extracts

Figure 3

 3 Figure 3.15: Theremin patch

Figure 3 . 16 :Figure 3 . 17 :

 316317 Figure 3.16: Mapping pen strokes to online graphical objects

Figure 3 . 18 :

 318 Figure 3.18: Hand-drawn curves control diverse processes

Figure 3 .

 3 18 (right) shows how MM uses long curves to control tempo variations in a 15-minute piece he composed for a short film. MM and KH use OpenMusic's maquette for spatial organizations of musical objects, controlled by temporal and graphical parameters. Reflecting on InkSplorer's support for the maquette, the two composers showed examples from their work (Figure3.19) that could be potentially produced by spatio-temporal mappings between paper gestures and the maquette.

Figure 3 .

 3 Figure 3.19: Spatial and temporal (x-axis) organization of musical objects (KH, MM)

Figure 3 .

 3 Figure 3.20: Reusing or refining curves (MM, MB)

Figure 4

 4 Figure 4.1: Neoriemannian Tonnetz pitch organization.

Figure 4 . 2 :

 42 Figure 4.2: Chouvel's work on paper for his piece Traversée. The composer translates drawn shapes in a score and operates geometrical transformations to create variations. Picture from Jean-Marc Chouvel.

Figure 4 . 3 :

 43 Figure 4.3: Philippe Schoeller score to write music for an orchestra. The composer uses additional lines for semitones. The vertical organization corresponds to the perception of pitches across different instruments. Picture from Philippe Schoeller.

  Figure 4.4 gives an overview of our apparatus during our session with MB.

Figure 4 . 4 :

 44 Figure 4.4: MB showing a score on his computer during the design session.

Figure 4 . 5 :

 45 Figure 4.5: MB creates a musical sequence with movable and transparent pieces of paper. Left: He defines pitches on paper strips with printed staves. The pitches are related to the rhythm that he had previously drawn on a different piece of paper. Right: Several linked layers of paper define the final result. Connected elements (linked or stacked) are unique musical objects but share the same timeline.

Figure 4 . 6 :

 46 Figure 4.6: MGV's prototype with data containers and controllers for the Orchidée application. Left: Paper interface for controlling the application and specifying the parameters of an orchestration query. It includes a graph area with curves, tick boxes and a button to trigger the generation of solutions. Right: Second part of the interface dedicated to the exploration of solutions. It includes a map space where solutions are selected by circling parameterized data points, tick boxes to adjust parameters and a score to annotate the chosen solution.

4. 3 . 3

 33 Editing a recording based on the original score RD works on a recording of a violin part (Figure 4.7).

Figure 4 . 7 :

 47 Figure 4.7: RD's prototype for sound editing. Left: Digital version of the recorded sound in Ableton Live software. Right: Paper interface with printed sequences of musical events (notes in a musical score and audio waveforms) aligned in time and horizontal zones for editing audio parameters (time stretching, pitch and amplitude) with the pen.

Figure 4 . 8 :

 48 Figure 4.8: MK's prototype to create a new piece. Left: the interface consists of distinct paper modules with specific functions: areas for drawing curves, predefined scales, empty staves for the entry of notes, a list of composition rules, and a timeline. Right: The resulting musical sequence on his computer tool.

Figure 4 . 9 :

 49 Figure 4.9: GB's prototype for controlling a sound synthesis with multiple temporal curves. Left: Drawing a curve on paper to control parameters in real-time. Right: Curves drawn on different pieces of paper to control the synthesis parameters of a musical sequence.

Figure 5 . 1 :

 51 Figure 5.1: Different Paper Substrates layouts offer different levels of guidance for the entry of graphs with a range of values (a and c) and grids for precisely drawing curves (b and c).

Figure 5 . 2 :

 52 Figure 5.2: Connecting substrate through physical manipulations: Joining two substrate on their sides (a), inserting one substrate into another, (b) and superimposing two substrate (c).

Figure 5 . 3 :

 53 Figure 5.3: Specifying the time position of three substrate (1-2-3) by aligning them in a container substrate (4) that defines a common timeline.

Figure 5 . 4 :

 54 Figure 5.4: Editing a musical sequence with three different components: a musical sequence, a graph and a playbar. Left: Printed interface with a graph dynamically linked to the sequence. Right: Digital versions of the musical data in Max.

Figure 5 . 4

 54 Figure 5.4 (left) shows a user interacting with the different printed components to edit a musical sequence and control the amplitude of a background sound. The graph (top) and the playbar (bottom) substrates are connected to the musical sequence (center). The Max

Figure 5 . 5 :

 55 Figure 5.5:The programming API available for developers to create paper interfaces for working with the data of musical applications. The API helps the developer to design and reuse Paper Substrates, define the types and the representation of the contained data, and specify the user interaction with these data. Blue elements are specific classes that need to be completed by developers.

Figure 5 . 6

 56 Figure 5.6 illustrates the creation of a three-page document in landscape format. The user adds different types of substrate by dragging them from the list at the right side of the window. The user can resize and position a Paper Substrates on any page and then parameterize it through a configuration window activated by right clicking on it. Parameters that can be configured include the color of the substrate, user interaction, OSC output channels, and substrate-specific parameters, such as a range of values and duration. The user can define static links between substrates represented by straight non-printable lines. For example, in our first prototype, the top-left substrate in Figure5.6 is statically linked to two other substrates, sharing its horizontal axis, which represents time.

Figure 5 . 6 :

 56 Figure 5.6: The main interface of PaperComposer. (1) The pages of the current document. (2) Toolbar with basic functions for managing documents and setting up the parameters of applications. (3) Thumbnails of available substrate classes. They can be dragged into the virtual page to create a new instance. (4) A virtual page with several substrate including musical staves, a playbar, a curve container, a tonnetz and a sequencer.

Figure 5 . 7 :

 57 Figure 5.7: Synchronizing data between external music tools and PaperComposer. (1) An OpenMusic patch with a graph and a sequence of chords. Such data can be exported through a custom OpenMusic library. (2) Data exported from external applications appear in a list on PaperComposer. The user can drag data from the list to add them to the appropriate substrate. (3) The graphical user interface shows the substrates, the data, and the strokes as drawn on paper.

Figure 5 . 8 :

 58 Figure 5.8: Importing musical scores from a score editor. The colored overlays show various areas associated with the score's elements including the staff (green), the space under a staff (light blue) and all the notes (dark blue).

Figure 5 . 9 :

 59 Figure 5.9: Creating a score with movable substrate. Left: Drawing a curve on a sound substrate to control its amplitude. Center: Positioning the substrate on the score substrate to define is onset time. Right: Interacting on the score substrate to listen the whole score made of several sound substrates.

Figure 5 . 10 :

 510 Figure 5.10: Using PaperTonnetz to create and listen to a musical sequence. (1) The user taps on a handwritten path to listen to the corresponding melody and add it to the current track. (2) The sequencer assembles existing paths into tracks. (3) The Max patch displaying the current track made with several paths added as melodies and chords.

  Figure 5.11 shows a printed interface with several colors to encode the octave, the online view in PaperComposer and the Max player which produces the sound.

Figure 5 . 11 :

 511 Figure 5.11: Third version of PaperTonnetz interface. The Tonnetz colors encode the octave of the pitches. Photo H. Raguet c Inria

Figure 5 . 12 :

 512 Figure 5.12: Substrate for working with musical notation co-designed with a composer. Top: Handwritten notes representing pitch and duration. A separate playbar substrate lets the user select and evaluate individual score segments. Bottom: The resulting musical score in a Max patch.

Figure 5 . 13 :

 513 Figure 5.13: Paper Substrates for controlling sound synthesis with pen gestures. (1) The original score in a Max patch. (2) The printed score substrate with refinements of handwritten gestures. (3) The resulting score in a Max patch.

4 . 5 .

 45 Paper Substrates Toolkit and PaperComposer to support the creation of personal interactive paper interfaces with substrates by developers and composers. Communication with external applications and integration of musical scores. Implementation of PaperTonnetz with Louis Bigo to study and use Tonnetz representations in which strokes become chords and melodies. Design of substrates with composers to support handwritten music entry and control of sound synthesis with curves. S T R U C T U R E D O B S E R VAT I O N W I T H P O LY P H O N Y: A M U LT I FA C E T E D T O O L F O R S T U D Y I N G M U S I C C O M P O S I T I O N "People will use a variety of computing devices because they serve their informations needs in a variety of situations."

Figure 6 . 1 :

 61 Figure 6.1: Final Polyphony score by P5. The middle panel (3, 4) controls the harmonizer effect. The bottom panel (5, 6) controls the synthesizer.

Figure 6 . 2 :

 62 Figure 6.2: Figure 2: MIDI keyboard for playing with the harmonizer (in red) and the synthesizer (in blue).

Figure 6 . 3 :

 63 Figure 6.3: P5 inputs curves and notes on two pages of interactive paper, aligned along the common timeline. (1) Page with Webern's piece and print buttons. (2) Page with interactive substrates to work on the piece.

Figure 6 . 4 :

 64 Figure 6.4: Polyphony interface in PaperComposer with data from P3. (1) Virtual page with data from Max and digital pen. (2) List of available Max data streams.

Figure 6 . 5 :

 65 Figure 6.5: P9 uses the standard configuration with a screen, keyboard, mouse, and MIDI controller, plus a graphics tablet.

Figure 6 . 6 :Figure 6 . 7 :

 6667 Figure 6.6: Participants' use of available inputs and interfaces

Figure 6 . 8 :

 68 Figure 6.8: P1 wrote pitches and rhythm before using the proportional notation on the interactive paper.

Figure 6 . 9 :

 69 Figure 6.9: P2 uses the ruler to draw a line with the digital pen.

Figure 6 . 10 :

 610 Figure 6.10: P3's edits over a printed amplitude curve (bottom) associated with a sequence of notes (top)

Figure 6 . 11 :

 611 Figure 6.11: Annotations made by P9 with the graphics tablet

Figure 6 . 12 :

 612 Figure 6.12: Composers' composition strategies. Rectangles represent each participant's activity with the synthesizer and the harmonizer effect.

Figure 6 . 13 :

 613 Figure 6.13: P8 annotates the original score with colored pencils to express ideas and keeps track of his work on the computer

Figure 6 . 14 :

 614 Figure 6.14: Box plot summarizing subjective task evaluation. Thick vertical lines are median values, dots show outliers, and asterisks represent extreme values.

Figure 7 . 1 :

 71 Figure 7.1: Global planning of the working sessions during the project.

Figure 7 . 2 :

 72 Figure 7.2: Illuminated manuscripts from the XIV t h century

Figure 7 . 3 :

 73 Figure 7.3: Original illuminated manuscript used by Philippe Leroux.

Figure 7 . 4 :

 74 Figure 7.4: Exploring computer-based musical processes on paper. Pen events on the paper interface trigger musical processes in OpenMusic which generate direct audio-visual feedback. Photo H. Raguet c Inria.

Figure 7 . 5

 75 Figure 7.5 presents the paper interface that we created for Philippe Leroux with PaperComposer (see Section 5.3.2). It shows two instances of a substrate containing the original score as background image.The substrate detects intersecting strokes to recognize shapes while the user is drawing, and computes features from these shapes. The top substrate contains data written by the composer. The substrate makes each written stroke an "interactive component", allowing further pen interactions. For example, the user can "click" over a stroke with the pen to trigger an action such as computing a sequence or playing the result.

Figure 7 . 5 :

 75 Figure 7.5: Creation of the paper interface in PaperComposer. The virtual page on the center displays the interface, digital strokes and visual feedback.

Figure 7 . 6 :

 76 Figure 7.6: Writing on the paper interface. Left: The composer writing with the digital pen. Right: Visualisation on the screen showing the extracted shape (light-blue), its centroid (red) and its main axes (blue and green).

Figure 7 . 7 :

 77 Figure 7.7: OM patch receiving pen features from the server.

Figure 7 . 8 :

 78 Figure 7.8: Computations of an OM visual program represented as a graph. (a) Demand-driven model: the value of box C is requested by the user and triggers the recursive evaluation of the upstream boxes in the graph. (b) Reactive model: the box H has changed and propagates an update in the downstream boxes whose value may be influenced by this change.

2

 2 Figure7.9 shows two OpenMusic patches that use these tools to process incoming OSC messages. Reactive objects are identified with bold, dark red frames. The patch in Figure7.9a contains an OSCreceive box that transmits the messages to the route-osc objects. The data filtered by route-osc are used to build OpenMusic objects that are in turn "sent" via om-send boxes. In Figure7.9b, the om-receive boxes receive a notification and update data structures accordingly.2

Figure 7 . 9 :

 79 Figure 7.9: Reactive parsing and dispatching of OSC messages in an OM patch. Reactive boxes are displayed with a bold red frame.

Figure 7 . 10 :

 710 Figure 7.10: Converting a shape into a polyphonic sequence.

1 .

 1 The patch in Figure7.10 converts the strokes from a form into a sequence of notes by projecting the shape into a score. The patch offers controls for several parameters such as the number

Figure 7 . 11 :

 711 Figure 7.11: Mapping the pen speed with a Doppler effect transformation.

Figure 7 . 12 :

 712 Figure 7.12: Frequency distortion with thickness envelopes.

Figure 7 . 13 :

 713 Figure 7.13: Annotated score.

Figure 7 . 14 :

 714 Figure 7.14: Reusing a set of chords as Synthesis data

Figure 7 . 15 :

 715 Figure 7.15: Controlling spatialization trajectories

Figure 7 .

 7 Figure 7.16: A version of the paper interface used to write and compare the calligraphic forms.

Figure 7 . 18 :

 718 Figure 7.18: Live visualization of the pen activity.

1

 1 Exploration of musical ideas on paper and computer



  to film the whole evaluation with K7, alim, tripod -1 camera to takes some pictures of the documents -1 Livescribe pen with book to take note with paper replay functionality -1 computer with all the workspaces working properly.ChecklistA. IntroductionGive the consent form and have the participant sign it.  Explain the goals of the study  Give the Pre Questionnaire and have the participant complete it  In the AnotoManager patch, chose the demo folder. Use the demo paper  Connect the Bluetooth pen  Connect the OSC server  Max: Show the jitter demo with replay on open paper  Max: Show the envelope control example  Max: Show the pitch control example  Max: Show the bach examples (duration and slots)  OM: Show the bpf, bpc demo  OM: Show the maquette demo  Consider making a short break to take a coffee and let the composer think about the demo a.1.2 Pre questionnaire 1 out of 1Pré QuestionnaireLe but de ce questionnaire est d'évaluer le type de participant que vous êtes et de récupérer des informations concernant votre pratique en tant que compositeur. Toutes les informations obtenues resteront confidentielles.

Figure B. 4 :P5Figure B. 5 :P7Figure B. 7 :P9Figure B. 9 :

 4579 Figure B.3: Final Polyphony score by P3

  Copyright: c 2012 Louis Bigo et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Figure 1 .

 1 Figure 1. The neo-Riemannian Tonnetz frequently used to visualize chord progressions because of its harmonic properties.

Fig. 11 .

 11 Fig. 11. Suite de hauteurs associée à la lettre L. La cohérence harmonique est liée à la logique impliquée par l'espace sous!jacent, qui donne une représentation topologique de la notion de proximité harmonique. Ainsi pour la lettre L du début de la partition, on retrouve les notes sol# au piano, mais qui n'est pas comptée dans la durée, et do au violon 1, la au violon 2, les instruments désignés par le tableau de la figure 9. D'autre part, le fait que le L soit entouré "cf. figure 8# indique que la mélodie de la voix épouse le parcours de la constellation L. Nous venons d'expliquer en quelque sorte le « branchage » de l'arbre. Reste à comprendre ce qui en fait la frondaison. Un autre brouillon donne des indications importantes. Il se rapporte aussi au texte du poème en donnant à chaque paragraphe un rôle singulier, illustré par les signes présents à gauche du texte. D'une certaine manière, la forme de la pièce est résumée dans ces quelques notations qu'il convient de déchiffrer. Les lettres L, M et P sont prises pour Lettre, Mot et Paragraphe. Cela désigne le niveau auquel la

Figure 2 .

 2 Figure 2. Chouvel's work on paper for his piece Traversée. The composer translates drawn shapes in a score and operates geometrical transformations to create variations. Picture from Jean-Marc Chouvel.

Figure 3 .

 3 Figure 3. Tonnetz notation and interval relationships.

Figure 4 .

 4 Figure 4. C Major heptatonic hexagonal space. Every pitch of the space belongs to this scale. Thus any path inside produces a melody in C Major.

Figure 5 .

 5 Figure 5. A participant discovering the two layouts. When the pen enters a region, the corresponding pitch is played.

Figure 6 .

 6 Figure 6. Left: PaperTonnetz main interface representing the virtual page with three Tonnetze and one sequencer. Right: The Max/MSP patch to play and visualize created sequences.

Figure 7 .

 7 Figure 7. Left: Visualization of drawn paths in the Cminor heptatonic Tonnetz. Right: A sequence created with the three paths added as chords.

Figure 8 .

 8 Figure 8. Left: A 3D paper shape. Right: Two planes representing tonalities of C Major and F Major.

1

  Study I: Interviews . . . . . . . . . . . . . . . . . . . . . 3.1.1 Method . . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Results . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Study II: Design explorations with a composer . . . . . 3.2.1 Method . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Study III: Exploratory evaluation of InkSplorer . . . . . 3.3.1 InkSplorer . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Method . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . Working on paper . . . . . . . . . . . . . . . . . 3.4.2 Extending computer-aided composition environments . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Implications for design . . . . . . . . . . . . . . 3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Contributions . . . . . . . . . . . . . . . . . . . . . . . .

	x	contents
		3.4.1
		ix

4. Utilisez vous les supports suivant lors de votre processus de composition ? Précisez le niveau d'utilisation.

	Papier Libre	Jamais [ ]	Occasionnellement [ ]	Fréquemment [ ]
	Papier à musique	Jamais [ ]	Occasionnellement [ ]	Fréquemment [ ]
	Logiciels de gravure Jamais [ ]	Occasionnellement [ ]	Fréquemment [ ]
	Autre : ________	Jamais [ ]	Occasionnellement [ ]	Fréquemment [ ]
	Environnements de composition assistée par ordinateur :
	Max/Msp	Jamais [ ]	Occasionnellement [ ]	Fréquemment [ ]
	OpenMusic	Jamais [ ]	Occasionnellement [ ]	Fréquemment [ ]
	Autre : ________	Jamais [ ]	Occasionnellement [ ]	Fréquemment [ ]
	5.

Parmi les environnements de composition assistée par ordinateur suivant, explicitez votre niveau de maîtrise.

	a.1.3 Interview guide
		3.	l
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
	n	n éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé ' raur…auu l
	/ d '
			i
	v	' lw
	éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé	o	n
	éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé i
	éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
	éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
	éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
	éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
	éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé n i
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé l éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
	OPENMUSIC MAX/MSP AUTRE éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé 0(inconnu) 1(débutant) 2(notions) 3(intermédiaire) 4(avancé) 0(inconnu) 1(débutant) 2(notions) 3(intermédiaire) 4(avancé) 0(inconnu) 1(débutant) 2(notions) 3(intermédiaire) 4(avancé) éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé n éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé i éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé	5(expert) 5(expert) 5(expert)
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
		éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé

Comparer les deux approches au niveau de la précision des données saisies. 4. Utiliseriez vous le papier augmenté dans un cas similaire?

	Jamais [ ]	rarement [ ]	Occasionnellement [ ]	Fréquemment [ ]

Justifiez brièvement : 5. Remarques et comparaisons supplémentaires sur l'exemple et les outils utilises.

  Le but de ce questionnaire est d'évaluer les possibilités du papier augmenté et les outils proposés pour l'intégration dans les environnements de composition assistée par ordinateur. Les questions concernent les tests effectués lors de la rencontre et nous permettrons d'améliorer les possibilités du papier augmenté pour la composition musicale.

	a.1.5 Post questionnaire
	Post Questionnaire P :

  This paper first investigates how properties of Tonnetze can be applied in the composition process, including the two-dimensional organization of pitches, based on a chord or on a scale. We then describe PaperTonnetz, a tool that lets musicians explore and compose music with Tonnetz representations by making gestures on interactive paper. Unlike screen-based interactive Tonnetz systems that treat the notes as playable buttons, PaperTonnetz allows composers to interact with gestures, creating replayable patterns that represent pitch sequences and/or chords.

	PAPERTONNETZ: MUSIC COMPOSITION WITH INTERACTIVE PAPER
	Louis Bigo 1,3	Jérémie Garcia 1,2	Antoine Spicher 3	Wendy E. Mackay 2,4
	bigo@lacl.fr	garcia@lri.fr	aspicher@lacl.fr	makay@lri.fr
	1 IRCAM, CNRS UMR STMS 1 place Igor Stravinsky F-75004 Paris, France	2 INRIA, LRI Univ. Paris Sud, CNRS F-91405 Orsay, France	3 LACL, Univ. Paris-Est Crteil 61 avenue du Gal de Gaulle F-94010 Créteil, France	4 Stanford University Computer Sciences, Gates B-280 Standord, CA
	ABSTRACT
	Tonnetze are space-based musical representations that lay out individual pitches in a regular structure. They are pri-marily used for analysis with visualization tools or on pa-per and for performance with button-based tablet or tangi-ble interfaces.

"La musique occidentale n'est parvenue à se concevoir comme un acte original de création qu'à partir du moment où elle a soumis l'oreille à l'emprise du regard."

2.2 supporting music composition

http://highc.org/

http://hyperscore.wordpress.com/

www.anoto.com

www.livescribe.com

see http://vimeo.com/12853935

see online video https://vimeo.com/91920734

http://www.bachproject.net

see online video https://vimeo.com/91921077

The twelve pitches of the chromatic scale are equally important and appeared in series.

see the online video https://vimeo.com/91913156 for a summary of the video prototypes.

see online demonstration video https://vimeo.com/91913157

https://proxymusic.kenai.com/

see online video http://vimeo.com/40072179

see online video https://vimeo.com/91921536 6.2 polyphony

Note that our "data-flow" description here is just for illustrative purpose: in reality only notifications are propagated, and the actual data is computed and passed from one box to another following the demand-driven evaluation of the terminal downstream boxes.

publications a.3.2 Pre questionnaire

publications publication

publications

http://www.anoto.com

http://vimeo.com/40072179

http://www.bachproject.net

Acknowledgments

We thank all researchers from IRCAM Musical Representations team, LACL and InSitu for useful insights and comments, particularly Olivier Michel (LACL), Jean-Louis Giavitto, Moreno Andreatta, Carlos Agon, Jean Bresson, Gerard Assayag (IRCAM) and Fanis Tsandilas (INRIA) for their constant help and support. We are also grateful to Jean-Marc Chouvel for his feedback on our work and for inspiring us. Finally, we thank our reviewers for their constructive and helpful suggestions.