Sheelagh Carpendale
email: sheelagh@ucalgary.ca

Sean Lynch
email: sglynch@ucalgary.ca

Miguel A Nacenta
email: miguel.nacenta@ucalgary.ca

ToCoPlay: Graphical Multi-touch Interaction for Composing and Playing Music

Keywords: Graphical Multi-touch Interaction for Composing and Playing Music Multi-touch, collaboration, composition, music, musical instrument

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Musical instruments can be considered interfaces between a musician and the acoustic space. It is, therefore, not surprising that music has been one of the main areas of application for the advances in interface science and engineering of the last few decades. The advent of new interface paradigms such as multi-touch and tangible computing have spawned a series of remarkable prototypes and commercial systems that enable new ways of creating and experiencing music, often beyond what is possible with traditional instruments. For example, new interfaces for musical expression currently allow people without music backgrounds to explore new sounds [START_REF] Blaine | The Jam-O-Drum interactive music system: a study in interaction design[END_REF][START_REF] Newton-Dunn | Block jam: a tangible interface for interactive music[END_REF][START_REF] Patten | Audiopad: a tag-based interface for musical performance[END_REF] and groups of people to perform together closely through interactive tables [START_REF] Jordà | Sonigraphical instruments: from FMOL to the reacTable[END_REF][START_REF] Jordà | The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces[END_REF][START_REF] Newton-Dunn | Block jam: a tangible interface for interactive music[END_REF][START_REF] Patten | Audiopad: a tag-based interface for musical performance[END_REF]. Computer and interface technology has changed how music can be performed, how sound can be generated, and also how it can be composed; however, with few exceptions [START_REF] Hochenbaum | Designing Expressive Musical Interfaces for Tabletop Surfaces[END_REF]20,[START_REF] Newton-Dunn | Block jam: a tangible interface for interactive music[END_REF], composition tools do not take advantage of new interaction possibilities, and still are restricted to mouse and keyboard, sometimes with interfaces to electronic versions of traditional instruments [START_REF] Cubase | [END_REF][START_REF] Band | [END_REF].

In this paper we present ToCoPlay (Touch-Compose-Play), a musical interface designed to support playing and composing music. By enabling seamless transition between performing and composing we explore facilitating musical actions that are already part of musicians' activities, since composing music requires listening and changing what is being created, and, conversely, there are many forms of music where composition is part of the performance process (e.g., jazz). Our system is based on a horizontal multi-touch surface, and is intended for people with or without musical backgrounds. During the design of this new kind of musical interface, we made design decisions to enable control, configurability, a straightforward mapping between the visual and acoustic space, and to support a minimal set of interface elements and operations.

The structure of the paper is as follows: in the next section we provide relevant background on interfaces for musical expression. Then we discuss our goals and strategies for the design of the ToCoPlay interface, followed by its detailed description. We present four examples of ToCoPlay in use, followed by a discussion of the results of our exploration, and finish with our conclusions.

Related Work

There exist many applications for musical expression on digital surfaces, and these vary widely in interaction methods and purpose. Most applications support a combination of the following musical activities: playing sounds, synthesizing sounds, sequencing sounds in real-time, and composing. In the following paragraphs we provide a brief summary of prior work, and how it differs from our own. There are a large number of previous systems that focus on creating new sounds through the modification of streams or samples. Some of these are based mainly on tangibles, such as the seminal work by Jordà et al. on the Reactable [START_REF] Jordà | Sonigraphical instruments: from FMOL to the reacTable[END_REF][START_REF] Jordà | The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces[END_REF], Audiopad [START_REF] Patten | Audiopad: a tag-based interface for musical performance[END_REF], Pin & Play & Perform [START_REF] Villar | Pin & Play & Perform: a rearrangeable interface for musical composition and performance[END_REF], and Condio [START_REF] Condio | [END_REF]. These systems rely mostly on physical blocks that represent certain sound synthesizing operations (e.g., filters, tone generators, echo, samples) that are placed on the table and manipulated in real time to produce new sounds and rhythms. Other related work has the same approach, but is based on touch. For example, Akustich [1] is a direct-touch interface in which participants manipulate sound by performing multi-touch gestures on a table, where each gesture changes the sound and the visuals in a different way. In a similar way, performers using Sound Storm [START_REF]Sound Storm[END_REF] stand in front of a vertical display and use touch to modify sound waves that come across the screen. The Synthesis and Control on Large Multi-touch Displays [START_REF] Davidson | Synthesis and control on large scale multi-touch sensing displays[END_REF] demo by Perceptive Pixel provides a multi-touch knob and slider interface for the Synthesis Toolkit (a sound library for C++).

These systems resemble ToCoPlay in that they allow the real-time modification of sound by human performers, which makes them virtual instruments. However, they differ from ToCoPlay in several ways. First, ToCoPlay is based on standard note scales and sounds; therefore it does not focus on the generation of new sounds and rhythms. In this respect, ToCoPlay is simpler. Second, these systems are not designed to support composition: the temporal evolution of the music is difficult to reproduce because sound is mostly dependent on the state of the interface and the specific user actions. ToCoPlay is designed to store, modify, and reproduce created sequences, which supports performance, composition, and all combinations of the two.

A second group of related systems do not support sound synthesis; instead, they focus on supporting music performance that resembles how we play traditional instruments. For example, SurfaceMusic [START_REF] Fyfe | SurfaceMusic: Mapping Virtual Touchbased Instruments to Physical Models[END_REF] enables playing three instruments on an interactive tabletop (a drum, a string instrument, and a wind instrument) by using a specific multi-touch gesture on each (tapping, strumming, and air-pushing). The Roots [START_REF] Hochenbaum | Designing Expressive Musical Interfaces for Tabletop Surfaces[END_REF] application generates ambient sounds when a user touches a particular region on the multi-touch horizontal table (the Bricktable [START_REF] Hochenbaum | Bricktable: A Musical Tangible Multi-Touch Interface[END_REF]) and produces visually pleasing, spiraling trees. The Jam-o-drum [START_REF] Blaine | The Jam-O-Drum interactive music system: a study in interaction design[END_REF] is a series of external controllers that lay on a horizontal table display, where multiple people can play sounds by tapping their own pad; the Jam-o-drum is primarily for playing games. These systems are closer to ToCoPlay in the simplicity of the basic sound elements that are combined, but they do not generally support composition any more than traditional instruments.

A third group of interfaces enable composition. Traditional applications such as Garage Band [START_REF] Band | [END_REF], and Cubase [START_REF] Cubase | [END_REF] are single user, rely on traditional notation, and are meant to be used from a desktop PC. The MusicTable [START_REF] Stavness | The MUSICtable: a map-based ubiquitous system for social interaction with a digital music collection[END_REF] allows people to place cards that represent pitch and instrument on a table. The cards are detected by the system and played in sequence from left to right. BlockJam [START_REF] Newton-Dunn | Block jam: a tangible interface for interactive music[END_REF] lets people combine attachable active physical blocks that have a button to generate a sequence of sounds. The rules of how the sound is produced depend on the type of block and its current state. The resulting sound cycles through the blocks and visual feedback is provided through a monitor. InstantCity [START_REF] Hauert | Instant City[END_REF] is an art music automata that plays different ambient sounds depending on how blocks are distributed on the table. Xenakis [START_REF] Bischof | Xenakis: combining tangible interaction with probability-based musical composition[END_REF] is a non-deterministic system that uses tangible blocks to generate music according to probability distributions based on the distance between blocks and their similarity. In Noteput [20] participants place musical note-shaped physical blocks onto a digital horizontal surface which are played left to right according to their position, creating a physical staff. Spaces [START_REF] Hochenbaum | Designing Expressive Musical Interfaces for Tabletop Surfaces[END_REF] is an application for composing ambient sounds on a multitouch table, where participants touch a region of a table to make that region a warmer or cooler colour, with matching sound. Stereotronic Multi-Synth Orchestra [START_REF]Stereotronic Multi-synth Orchestra[END_REF] is a Microsoft Surface [START_REF]Microsoft Surface[END_REF] application in which participants place notes into pre-defined concentric rings, and notes are triggered when the rotating element of each concentric ring goes over that particular note. These systems resemble ToCoPlay in their ability to easily set up temporal sequences of sounds, but their design does not facilitate the performative component. ToCoPlay, similarly to all of these systems, allows setting up music and then triggering its performance without much human intervention; however, ToCoPlay focuses on supporting seamless transition between configuring the music, changing it, and performing or improvising with the available components.

In summary, ToCoPlay situates itself as being both a composition tool and an instrument, but not a sound synthesizer. ToCoPlay's goal is different from the goals of systems mentioned above, since we set out to achieve seamless transitions between the playing and composing activities. It also has several features that distinguish it from previous systems, most notably a flexible and precise spatial mapping (contrary to the mostly linear ones described above) and its configurability to create configurations that adapt to the person and the situation.

Design Goal and Strategies

As illustrated by the related work section, there is considerable activity in the development of software with musical capabilities. However, most of these focus on strengthening either the performance potential or concentrate on supporting musical composition. Our goal is to, as part of a musical software interface, investigate the integration of these activities: composition and performance. Thus, we intend to both keep interactions as simple possible and to explore ways of using simple interactions to create powerful methods for introducing complexity. In ToCoPlay, we offer an interface that can be played as an instrument, but that also invites composition. To achieve these goals we apply four main design strategies: enabling control, enabling configurability, creating a formative visual-acoustic mapping, and providing a minimal set of interface objects and operations.

Enabling Control

For ToCoPlay we opted for a discrete paradigm, which is more closely related to how a piano is played (by pressing keys) than to the continuous model used by most modern synthesizers, which are commonly controlled by turning knobs and adjusting continuous parameters. In ToCoPlay individual sounds are made by pressing keys. Complex sounds are developed through grouping, sequencing, and nesting. Grouping, sequencing and nesting are all interactively created and visually explicit.

Enabling Configurability

The physical characteristics of traditional physical instruments, such as the arrangement and shape of keys, have evolved into their current forms through many small mutations that often took centuries. Current electronic interfaces have not yet enjoyed the popularity or the time to evolve in this way. For ToCoPlay we introduce a model in which the location of keys and their grouping is fully configurable in real time. It is reconfigurable during the design of the sounds or melodies and it is reconfigurable during play. These freedoms can enhance the creation of a personalized interface that fits one's own hands and, thus, is easier to play. Through reconfigurability it can also be adapted for different people and compositions; for example, one can group notes together so that they can be played without moving one's hand (e.g., by placing them directly under the fingers when they are in a natural posture), or one can move a certain set of notes closer when they are to be used.

Visual-acoustic Mapping

For most people composing music requires at least two basic elements: a way to try out the sounds, melodies, and harmonies as they are being created, and a way to record how these are produced (including their temporal relationships), for later performance. Many of the current music generation systems that we describe in Section 2 are inadequate for composition, not only because they lack features to record how interaction must take place to reproduce a certain sound, but also because the continuous nature of the interaction makes it difficult to create a notation that accurately and effectively relates the almost infinite variety of gestures that can take place in a multi-parametric continuous space to the way they sound.

Therefore we designed a visual-acoustic mapping that provides a straightforward relationship between what is displayed on screen and what is going to be reproduced, so that actions on the interface can be interpreted as changes in the music. Note that this is not a visualization of the sound but a spatialisation of the relationships between sounds. One such possible mapping is traditional music notation; however, this notation did not evolve to facilitate real-time interaction, is not generally accessible to novices and, most importantly, it imposes a rigid spatial mapping of time (along the staff, moving horizontally through time and then down through the scale) that directly contradicts our configurability goal. In other words, using a staff metaphor for our interactive system would not allow us to make an interface that is easily playable; it would encumber collaboration (e.g., due to its implicit orientation), and would not allow us to group elements according to other groupings that are not strictly temporal. Most of these arguments apply as well to other existing sequencing and composing applications (e.g., [START_REF] Bischof | Xenakis: combining tangible interaction with probability-based musical composition[END_REF]). Again, we keep this spatial/visual relationship simple. Sequencing is specified through explicit directed links, where the spatial distance is mapped to temporal spacing.

Minimal Objects and Operations

Making an interface accessible to novices requires simplicity in the number and complexity of object types in the interface and their operations. We set out to create an interface that was based on a minimalistic approach. Unlike many existing musical interfaces that have large lists of objects, filters, types of connections and parameters, our interface needs only a few objects and a few rules about the ways the objects can relate to each other.

It might seem counter intuitive to restrict the design of the system to a small set of primitives in order to enhance expressivity and encourage playfulness. However, there is a wealth of experience in other fields that shows that minimalistic systems can lead to complex and expressive results; for example, a von Neumann machine, the Game of Life [START_REF] Gardner | On cellular automata, self-replication, the Garden of Eden and the game life[END_REF], SWARM intelligence [START_REF] Bonabeau | Swarm Intelligence: From Natural to Artificial Systems[END_REF], or the game of Go.

The ToCoPlay Interface

The interface of ToCoPlay is composed of five atomic elements: keys, key fountains, containers, links, and dummy keys. These elements can be combined to enable the properties outlined by the design section above. The following subsections provide a detailed description of each of the atomic elements, their interactions, and how their design, interactions and combinations support our design objectives.

Keys

The base atomic element in ToCoPlay is the key, a group of which are shown in Figure 1. A key is a circular interface element that plays a specific note of a specific instrument when touched. The note assigned to a key is one of the twelve pitches of the Chromatic Scale, which are represented visually by a change in value of the key's color; that is, lower notes are represented by darker colors than higher notes. The hue of the key indicates the instrument (or timbre) of the note. At this moment, two synthetic instruments, a sine and a square wave generator, are represented by pink and yellow key hues respectively. Keys are circular to facilitate being easily touched while having a small footprint, which allows many keys to be visible simultaneously on the interface. Note that, unlike with most traditional interfaces, ToCoPlay can have many keys for the same note which, as we will see, supports the generative nature of the system. The one-toone mapping between the colour of the keys and the sound that they emit is designed to enable a direct relationship between the visual and acoustic spaces.

Tapping a key instantly plays the corresponding note and makes it transparent, establishing a clear connection between the visual and the acoustic feedback of the system. A key can be moved and repositioned anywhere across the interface. Repositioning keys allows the musician to arrange them in ways that support playing the interface as a customizable instrument; for example, the keys necessary to play different leit-motifs of a piece can be placed so that they correspond to the tips of the fingers when the hand is in different areas of the table. The mobility of keys on the interface also enables clustering of notes in different visual layouts which, as described under 'links', further supports the compositional aspects of the interface. Keys can also be flicked around in the interface to quickly move them away from an area of interest, or to remove them from the interface if flicked out of the visible area of the interactive table.

Key Fountains

New keys are created by dragging them out of key fountains. Key fountains are represented by a large circle surrounded by the smaller, circular keys (see Figure 3). The keys are arranged around the circle clockwise from C to B in half-tone intervals (12 notes). When a key is removed from a fountain, another copy appears in its place, which enables the creation of an unlimited number of keys of each type, as shown in Figure 2. The interface contains a key fountain for each instrument (timbre). Although fountains initially appear at the center of the application, they can be moved around by dragging them from the central circle to make space for other kinds of interaction. It is also possible to rotate and scale fountains through the standard two-point RST gesture [START_REF] Krueger | VIDEOPLACE -An Artificial Reality[END_REF] to accommodate the needs of people located in different areas of the table. Consistent with the key interface elements, a fountain can also be flicked away out of the screen, which clears it from memory. Note fountains can be duplicated by double tapping on their central circle, again to accommodate the needs of multiple people around the table.

Containers

Containers, which are used to hold elements, are regions defined by free-form touchtraces on the table (see Figure 4). Containers are created by starting a touch-trace on an unoccupied place on the table and closing off the trace. A line indicates the outline of a container while it is being drawn, allowing for the precise definition of regions. When the finger is released and the container is completed, it appears as a translucent blue shape, and it automatically contains the objects that the trace enclosed. Containers can group heterogeneous collections of any interface objects except key fountains (see Figure 5). Containers can also contain other containers, which supports the generative nature of composition by allowing nesting. They enable the individuals to work with their own pieces of a composition, or of an instrument, and then to combine them at a later point in time.

Similarly to key fountains, containers can be moved, scaled or deleted (i.e., they can be dragged, flicked or manipulated with two-finger RST). Objects such as keys and other containers that the container currently holds stay with the container as the container is moved or rotated. This allows musicians to manage the screen area efficiently by manipulating meaningful groupings of objects and supports sociallybased use of the space [START_REF] Scott | Territoriality in collaborative tabletop workspaces[END_REF]. An object (e.g., a key) can be added to a container by moving it from outside the container to inside the container, or by moving the container so that it fully contains the object. Containers also have two buttons. The container and all of the elements and structure it contains can be copied or duplicated by tapping on its green button. The copy of the container appears at a small offset from the original (see Figure 6). The copy also oscillates for a brief period of time to highlight that it is a new object. Next to the copy button, there is a red button that is used to control the lock state of the container. A container can be in one of two states: locked or unlocked. An unlocked container can be moved, resized, or rotated and also allows its contents to be moved or removed. When locked, the container prevents accidental movement or transformation of its contents and itself, keeping all elements in place. When unlocked, containers and their contents can be manipulated to find different internal and external configurations. If a container is unlocked, an element can be moved out of a container; once it is fully outside of a container, and is released, it no longer belongs to that parent container. The opacity of the red lock button reflects the current state of a given container: opaque red means locked, translucent means unlocked. Containers can be deleted in one of two ways: either by throwing them off screen like keys or key fountains, or by a gesture that crosses the container boundary twice. Together, duplication and repositioning provide the freedom to design personalized performance arrangements. Container and key groupings can become 'instruments' that can be shaped according to the specific anatomical or musical needs of the performer (e.g., according to the hand shape) and placed by the musician in different areas as required by the performance (e.g., storing instruments in distant regions when not required and bringing them close when being used). Simultaneously, the grouping of keys supported by containers, combined with nesting, enables the natural organization of musical compositions in different hierarchies such as themes, choruses, bridges, sections, etc. Container manipulation also supports reducing and organizing containers in space according to their current relevance (shown in Figure 7).

Links

The elements described thus far support playing notes, grouping keys so that they form 'instruments' of arbitrary shapes, sizes, and orientations, and distributing these groupings in meaningful layouts over the table. However, the nature of the interaction with these elements does not enable composition or playback, since each note still has to be directly played by the musician. In order to support composition it is necessary to provide a mechanism that allows sequencing of notes at different times and intervals. We provide this mechanism through our link interface element.

Links are one-directional connections from one key to another that denote a sequence in how the notes of the corresponding keys are played (see Figure 8). Links are created by tracing a line that connects two keys when the original key is inside a locked container. Providing that the original key is within a locked container, any two keys can be connected, including keys belonging to different containers, keys that do not belong to any container or even keys that belong to contained sub-containers. Links can be deleted by tracing a line that crosses the link's representation within a locked container or in blank space. If a key is tapped and it has an outgoing link to another key, the other keys will automatically be played when the original key has finished playing its note. If the subsequent keys have links, the subsequent notes will be played and so on, until the chain ends, thus providing a way to sequence music events. The direction of a link is indicated by an arrow at the end of the link, indicating which key is the initial key and which is the subsequent key. The length of the link determines the timing between the tapping or signal reception of the initial key and the start of the note of the subsequent key. The initial key's duration depends also on the duration of the following link, since its note will play until the next note in this particular chain starts to play. Timings created with links are not fully continuous and instead snap to multiples of 125ms.

A key can have any number of incoming or outgoing links; every time that a link sends a signal to a key, its entire set of outgoing links will fire signals to the corresponding keys. This branching enables creating chords and counterpoint melodies -common elements of music in most cultures-as well as cyclic patterns and recursive music as we will show in the examples of Section 7.

Dummy Keys

Dummy keys are special keys that do not play any sound by themselves, but serve as proxies to activate other keys. A dummy key's incoming link length is ignored. By placing a dummy key that connects distant keys A and B (in that sequence) close to B, the note of key A will be played for a time proportional to the distance between the dummy note and key B instead of by the distance between keys A and B (see Figure 9). This allows for flexibility in the connection and placing of groups of keys that need to be played close in time but need to remain in non-adjacent locations.

Similarly, a combination of dummy keys can be used to create instruments with keys located in anatomically convenient locations, as we will show in Section 6.1. Dummy keys can be created by dragging them from the center of a key fountain, and their inner colour is grey.

Implementation

We implemented ToCoPlay to work on the Microsoft Surface™, and used Microsoft's WPF™ libraries for Surface development. As a result, every visual object in ToCoPlay receives events when it is touched. For sound, we used Microsoft's MediaPlayer framework, and triggered wav samples. The wav samples were generated to have two distinct timbres: that of a chromatic scale with sinusoidal tones, and of a chromatic scale with square tones. We used a simple tone generator application to create these, and loaded multiple instances of each file so as to have the ability to play a sample more than once at any point in time. For the visual elements of ToCoPlay, prototypes were built using Microsoft Expression Blend™, and the final versions were built using a combination of Microsoft's C# and XAML.

6 Use Scenarios

Chord and Scale Instrument

This use case scenario illustrates the configurability of the system through the use of shaped containers, key layouts, and dummy keys. It corresponds to Example 1 in the video figure. The video shows how to build two instruments. The first one is a simple instrument with the five keys of a blues pentatonic scale, which are arranged in space to correspond to the tips of the fingers of the performer. Keys are made larger to make them easier to play, and adjusted for angle of comfort for the performer.

The second is an ad-hoc chord instrument that can play three basic chords of a simplified blues progression (C7, F7, and G7). Each chord is formed by four keys, which are activated by a dummy key simultaneously. Each of the chord's dummy keys are activated (with no delay) by a dummy key located within their respective containers, to facilitate playing with the left hand (see Figure 10 and video figure).

The performer shown on the video (one of the authors) does not have any piano experience.

Harmonious Loops

This scenario illustrates how simple musical loops can produce a more complex musical texture through straightforward copying of a container, and the real-time rearrangement of the keys. In this example we part from a single loop, then copy it, modify the copy's loop timing slightly, and play both loops simultaneously. Because both loops are similar but not identical in length, the harmonies being played change continuously, in a sequence that repeats itself on a period much longer than the duration of either loop, creating a complex musical texture out of two simple elements.

Canon

In music, a Canon is a musical form that is based on the repetition of the same melody by several instruments or voices, starting at different points in time. Because each instrument or singer has a delay with respect to the others, different notes sound simultaneously, resulting in chords. This example is a more sophisticated evolution of the harmonic loops, with a more sophisticated output. The triangular structure on top (see Figure 12) triggers the start of a full cycle of the melody (bottom), at times that are delayed by exactly one third of the length of the full melody. The cycle ends when one of the links of the triangular loop is cut (see video figure, Example 3).

Making Music with Shapes

This scenario takes advantage of the flexibility of our visual-acoustic mapping and the configurability of the interface to show how visually creative configurations can be created. In this case, two participants are collaborating in creating music to illustrate Antoine de Saint-Exupéry's famous story from the Little Prince. Figure 13 and the video figure shows how two participants can create shapes, modify them, populate them with music, and then combine them to generate a small sound composition. This example illustrates how ToCoPlay can also be playful, and suitable not only for serious composition, but also for musical games. The creation of shapes to embed musical themes might also prove useful to improve recall and identification of musical sections when the piece is complex, even if not used with visual-aesthetical goals. Designing music interfaces based on multi-touch interactive surfaces provides strong advantages; for example, a horizontal surface can enable collaboration, the input and output space coincide and are fully configurable, and a virtually infinite number of parameters and levels can be controlled through touch manipulations, traces and gestures. These manipulations are often of a continuous nature; for example, in the Reactable the rotation of an element is often mapped to a continuous parameter such as pitch or volume. Our system intentionally diverts from this model and instead offers a small discretized set of alternatives: a key can only play one of twelve notes, and time intervals are multiples of 125ms.

We fully recognize the importance of continuous control, especially for the manipulation of a complex, continuous control space such as sound synthesis. However, our design is more discrete because our goals are different from those of synthesizer and computer instruments. We prefer instead to make the results of actions easier to control and predict, and therefore easier to learn for people without a background in music. In exchange for this simplification of interaction, we are reducing how parameters that can be manipulated through our system; for example, it is not possible yet to produce vibrato or tremolo (small variations in pitch or volume) with ToCoPlay.

Homomorphic Relation between Space and Sound

Many of the existing synthesizers and computer-based instruments described in the related work, as well as most VJing interfaces (e.g., [START_REF] Taylor | Turning the tables: an interactive surface for vjing[END_REF]) provide visuals that depend mostly on the sound that is generated at that moment (or shortly before) and/or the input being provided. Since our main goal to support seamless transitions between performing and composing, we provide a tool that supports the broader view of the piece required by composers. We achieve this through our straightforward mapping that relates the visuals of the interface (the position, colour and lengths of the links) to sound. In particular, the mapping between the length of links and the duration and sequence of notes was of great help and impact when creating the examples described above, and it was often understood in demonstrations before it was explained. This relationship between the visual and the sound spaces is also useful in enabling shifts from composition to performance, since looking at the interface could allow performers to intervene and perform modification of a composed element in real time.

Although it would have been possible to create one-to-one relationships between the spatial and the acoustic spaces, we preferred to allow some freedom. Unlike traditional musical notations, which are generally isomorphic, our spatialisation of time allows several patterns to generate the same sequence of notes; for example, by changing the angle between keys without changing their distance or using dummy keys. This makes space for layouts that are not exclusively following musical requirements, but also accommodate interface considerations (e.g., collaboration, shape of the hand as in Examples 1 and 4), and aesthetic or storytelling possibilities, as shown by Example 4. We believe the visual aesthetic elements can be complementary and important for performing and composing, especially for people without musical background and for children; in fact, notations that cross media boundaries in art are not uncommon (e.g., Apollinaire's concrete poetry).

Few Building Blocks Can Go a Long Way

Our current version of ToCoPlay is certainly limited in several aspects, and does not provide many of the expressive channels that other music applications offer. Nevertheless, we found our decision to keep the interface to a few elements and rules useful and rewarding. We believe that the small set of elements will make the interface easier to learn, and we showed through our examples how initially simple combinations of a few elements can result in fairly complex sound textures (e.g., Example 2).

This kind of approach is particularly suited to a multi-touch interface, and generally difficult to apply to tangible interfaces such as the Reactable [START_REF] Jordà | The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces[END_REF] and Audiopad [START_REF] Patten | Audiopad: a tag-based interface for musical performance[END_REF], which would require many physical elements and can be quite expensive. In multitouch systems have advantages in that we are free to replicate elements until the screen is full. However, touch detection and touch feedback do not achieve the same tactile and haptic quality of tangible interfaces.

Limitations

ToCoPlay is an exploration of musical interfaces and, as such, has helped us uncover new challenges and presented new difficulties. Probably most obvious is the need to improve the quality and quantity of samples for instruments to improve the quality of the sound. The sinusoidal-and square-wave generators used as initial instruments were adequate for our initial explorations, but we are aware that new, richer sounds can substantially help increase the musical value of the interface. Adding an external sound generation engine (possibly on a different machine) can help improve the quality of sound and, at the same time, solve some of the problems derived from dealing with an interface that has many elements and momentarily freezes for some operations such as large container creation.

Similarly, we have contemplated adding new controllable parameters to increase the expressivity of the system. For example, changing the volume (dynamics) of individual notes or containers can be valuable for composing. We also do not currently provide a method of creating pauses and delays in a composition. Future changes in this direction must, however, be weighed against the increase of interface complexity that contradicts our strategy of maintaining a simple set of basic elements and interactions.

Some of the aspects of our visual-acoustic mappings also present space for improvement. For example, we have noticed that the mapping from the HSV colour space of keys to their note's pitch is difficult to read and compare, especially if notes are distant from each other. We are already considering other kinds of mappings based on symbols and non-linear mappings of color to pitch, and mappings that would facilitate the inclusion of more than one octave.

Another issue with our spatial mapping is the ability to read complex scores of music. We are interested in finding out whether spatial relationships could become problematic (e.g., with a lot of containers embedded within one another) and how the interface can be modified for large scale compositions.

Finally, our current evaluation of the system has been constrained to just a few people from our department. Although reactions have been positive, we intend to validate our design by placing it into the real world and gather more impressions to produce a new, better version.

This paper presents ToCoPlay, a multi-touch tabletop musical interface that allows multiple people to create music. The main goal of ToCoPlay is to provide an interface that enables the performance of music while also encouraging composition. We applied four main strategies to design a system that allows seamless transitions between the performance and composition activities of participants: enabling control, enabling configurability, creating a simple visual-acoustic mapping, and relying on a minimal set of interface elements and operations. We also provide a set of four examples that demonstrate different aspects of the operation of the interface and its expressive capabilities. Finally, we discuss the implications of our design decisions and how they make ToCoPlay unique.

Figure 1 .

 1 Figure 1. Keys in ToCoPlay, representing the C Chromatic Scale

Figure 2 .

 2 Figure 2. A key is removed from a key fountain, and is replaced by another key.

Figure 3 .

 3 Figure 3. The two different key fountains, each representing a different timbre. Each key fountain consists of a large circle surrounded by keys, representing the notes in the chromatic scale, from C (darkest) to B (lightest)

Figure 4 .

 4 Figure 4. To trace a container start in an unoccupied space and create a closed form.

Figure 5 .

 5 Figure 5. A container with keys placed inside.

Figure 6 .

 6 Figure 6. Pressing the container's green button creates a duplicate at a slight offset.

Figure 7 .

 7 Figure 7. A container is rotated, scaled and translated to fit inside of another container.

Figure 8 .

 8 Figure 8. A key, inside a locked container, is pressed and then connected to another key, forming a link.

Figure 9 .

 9 Figure 9. A dummy key connects two keys at a large distance. As a result, the key with the outgoing link will have a shorter distance to the outgoing link of the dummy key.

Figure 10 .

 10 Figure 10. A chord instrument (left) and a scale instrument (right).

Figure 11 .

 11 Figure 11. A) A simple loop; B) A slightly modified copy of the loop in A); A) and B) played together will form a complex musical texture; C) Both loops can be modified in real time to further evolve the texture created.

Figure 12 .

 12 Figure 12. A Canon composition

Figure 13 .

 13 Figure 13. Two shapes are created with connected links inside: an elephant inside a snake; or is it a hat?

Acknowledgements

We would like to thank Uta Hinrichs for her valuable assistance creating the first version of the video. We are also grateful for comments and discussion from Lawrence Fyfe and many other members of the Interactions Lab and the Innovis group. This work was supported in part by Canada's Natural Science and Engineering Research Council (NSERC), Canadian Foundations for Innovation (CFI), Alberta's Informatics Circle of Research Excellence (iCORE) and by SMART Technologies Inc.


Helena Lindgren
email: helena@cs.umu.se

Patrik J Winnberg
email: patrikw@cs.umu.se

Peter Winnberg

Domain Experts Tailoring Interaction to Users -an Evaluation Study

Keywords: Formative evaluation, interaction design, end-user development and adaptation, knowledge modeling, e-health

This paper presents ACKTUS, a modeling tool for developing knowledge-based systems for the health domain, and an evaluation study of the system. The main purpose of the evaluation was to investigate whether the functionality and interaction design of ACKTUS was sufficiently intuitive for the domain experts to contribute with knowledge and to model the interaction design of the three end users' applications. Another purpose was to evaluate the applicability of the activity assessment protocol AAIMA for analysis. The study design was qualitative and formative, using observations and interviews with users to collect data. Three medical experts and two experts in occupational therapy participated, providing expertise in four different domains. The participants increased their understanding and skills during the evaluation period leading to improved knowledge-based applications. The AAIMA protocol proved to be useful and the results are fed into ongoing development work on developing the adaptive functionality of the ACKTUS systems.

Introduction

Knowledge-based systems are to an increasing extent used by individuals for monitoring health and for supporting the daily work of health care professionals in the provision of care to individuals (e.g., [START_REF] Kaplan | Evaluating informatics application -clinical decision support systems literature review[END_REF]). In both cases it is of great importance to assure that the mediated knowledge has high quality (e.g. based in evidence-based medicine or other reliable sources) and that the interaction design of the system allows for tailored advice and personalized support in addition to transparency, in order for the end user to gain trust and confidence in the system's content and responses. For these purposes, the system ACKTUS (Activity-Centered Knowledge and interaction modeling Tailored to Users) is being developed, making use of Semantic Web-technology to model both the health-related knowledge and the interaction design of personalized knowledge-based systems [START_REF] Lindgren | A Model for Interaction Design of Personalised Knowledge Systems in the Health Domain[END_REF], [START_REF] Lindgren | Evaluation of a Semantic Web Application for Collaborative Knowledge Building in the Dementia Domain[END_REF]. The knowledge engineering of such systems typically requires deep knowledge in the medical domain, knowledge of formal methods and an extensive amount of time to transform informal knowledge into formal structures (e.g., [START_REF] Seyfang | Bridging the Gap between Informal and Formal Guideline Representations[END_REF]). Because of this, the major aim of the ACKTUS project is to facilitate the modeling process while also making the domain professional take the interaction design of the end user application into consideration. For the domain professional to be able to also model in what way the end user will interact with the system, functionality that supports this has been integrated and extended during development. Research focuses on how the design of the modeling environment allows the domain professionals to accomplish creating what they envision being the functionality and interaction design of the future support systems. This paper presents ACKTUS and the results from a formative evaluation study, conducted as part of the development of ACKTUS. The overall research question is to what extent the domain expert is able to grasp the structure of ACKTUS and how well the system supports the development of this understanding. A related research question is how well the system supports the domain expert in designing the interaction for other end users. These questions are addressed by investigating the participants' performance in modeling tasks in authentic work, before and after the interface re-design, which aimed to improve the understanding of the system. This was partly done by adopting and adapting an activity assessment protocol presented in [START_REF] Lindgren | Personalisation of Internet-Mediated Activity Support Systems in the Rehabilitation of Older Adults -A Pilot Study[END_REF]. Therefore, an additional central research question is to what extent the adapted protocol is useful for capturing knowledge and skill development in users.

Tools for Knowledge Representation and Interaction Design

Guidelines for clinical practice are being developed for the purpose of improving care. These typically come in natural language and are not straightforwardly translated into a guideline interpretable by a computer. In order to facilitate this process a number of representation languages have been developed by different research groups specifically for the purpose of representing clinical guidelines and treatment protocols. These languages typically allow for describing "task-network models", where sets of interacting decisions and actions are to be carried out in sequence or in parallel over a period of time. Some examples are Asbru, PROforma, GLIF, EON, GUIDE, and PRODIGY (see comparisons done by Peleg et al. [START_REF] Peleg | Comparing computer-interpretable guideline models: a case-study approach[END_REF] and Isern et al [START_REF] Isern | Computer-based execution of clinical guidelines: A review[END_REF]). These languages have in common that they provide methods for regulating a care process with pre-and post-conditions attached to plans. Tasks can be defined and associated to different points in a plan, where the task may involve data collection to be made by a health care professional. At these points typically forms appear where the required information according to the underlying clinical guideline can be entered.

Evaluations have been made of these languages within applied implementation projects of clinical guidelines, e.g, [START_REF] Shalom | Can physicians structure clinical guidelines? Experiments with a mark-up-process methodology[END_REF], [START_REF] Sutton | Evaluation of PROforma as a language for implementing medical guidelines in a practical context[END_REF], [START_REF] Eccher | Bridging an Asbru Protocol to an Existing Electronic Patient Record[END_REF]. It has been shown that it takes time for the domain expert to learn the underlying structure [START_REF] Shalom | Can physicians structure clinical guidelines? Experiments with a mark-up-process methodology[END_REF]. In some cases, the knowledge acquisition phase has been facilitated by developing simplified graphical user interfaces (GUI) on top of the original GUI such as in the case of using a MediaWiki for modeling Asbru plans [START_REF] Eccher | Bridging an Asbru Protocol to an Existing Electronic Patient Record[END_REF]. Another approach to address this issue is described by Seyfang and coworkers [START_REF] Seyfang | Bridging the Gap between Informal and Formal Guideline Representations[END_REF]. They developed a semi-structured format for simplifying the process on top of Asbru. In another project, a UML activity flow diagram was used when PROforma showed limitations in modeling decision flows in a graphical structure in an easy way [START_REF] Sutton | Evaluation of PROforma as a language for implementing medical guidelines in a practical context[END_REF]. Besides these languages, which are primarily developed for knowledge engineering purposes, we are not familiar with any generic knowledge modeling system that also allows the domain expert tailor the interaction with the end user application in a similar way as the system in focus for our evaluation study.

ACKTUS and the Application Domains in Focus for the Study

ACKTUS is being developed for the purpose of providing a modeling environment for domain experts who also may be potential end users of knowledge-based systems in the health domain. Another purpose of the system is managing knowledge integrated into knowledge-based systems. One goal is to design the system in such a way that end users are able to contribute to the knowledge in interaction with the system. Therefore, it is not expected that the end users or the domain experts participating in the knowledge modeling are familiar with formal models such as logic or formal ontologies. The task-network modeling languages described in the previous section are developed for modeling clinical practice guidelines and treatment protocols, mainly to support a care process within a care providing organization or between organizations. By contrast, in the current application projects, ACKTUS is used for synthesizing several knowledge sources into a support system in domains where the domain knowledge is incomplete and evolving. Another difference from the task-network languages is that the support systems developed using ACKTUS are not necessarily aimed at health care professionals. The end users in the current application projects are, besides health professionals, mining and construction workers and older persons in their homes. Therefore, ACKTUS also incorporates knowledge such as rules-of-thumb and best practice knowledge to be integrated apart from the knowledge expressed in clinical practice guidelines and other evidencebased medical knowledge sources. Preference orders among knowledge sources, level of expertise in knowledge domains and other contextual factors are taken into account in the reasoning about e.g., diagnosis in a patient case, or about adjustments in a work environment for preventing injuries.

The Semantic Web-application ACKTUS consists of a service-oriented architecture and integrates a formal terminology and information model in the form of an RDF/OWL ontology. The ontology captures 1) components used for tailoring interaction with the resulting knowledge applications, 2) components of argumentbased reasoning and 3) components for modeling the user agents as actors in a reasoning process [START_REF] Lindgren | A Model for Interaction Design of Personalised Knowledge Systems in the Health Domain[END_REF]. The ontology components are used by the domain experts for designing the content and interaction flows, and they put constraints on the modeling activity in order to support the domain expert in the modeling tasks. The outcome is a set of instantiated components and their relations, which function as structured tools for the end user in the interaction with the end user application. In addition to the end user application the ACKTUS architecture allows a developer, with sufficient permissions, to write applications that can access and manipulate the information using readily available Internet technologies like HTTP.

The key functionalities of ACKTUS are the following (Figure 1):

1. Integrate and model knowledge sources 2. Translate the content of knowledge sources into semi-formal argumentation schemes 3. Implement the schemes as formal rules 4. Define reasoning contexts to be activated by interaction protocols 5. Define and model interaction protocols 6. Define and model assessment protocols as sets of interaction protocols 7. Create rules for assessment flows between assessment protocols and interaction protocols 8. Apply and evaluate the composed system through end user system interfaces and with fictive patient or user cases The different functionalities are used to different extent in the application domains, due to their different characteristics. In the dementia domain, where care provision is highly dependent on evidence-based clinical practice guidelines, the tasks 1-3 are crucial and the validation of the work needs to be thoroughly done [START_REF] Lindgren | Towards personalised decision support in the dementia domain based on clinical practice guidelines[END_REF]. Since diagnostic reasoning is performed differently by novice and expert physicians [START_REF] Patel | Emerging paradigms of cognition in medical decisionmaking[END_REF] (e.g., in diagnosing dementia diseases) the notion of reasoning context is used (Task 4). The reasoning contexts are defined by the domain expert, and frame the topics to investigate at different levels of activity, and may be activated following the individual end user's need for support. The end user is a health professional responsible for diagnosing patients with a suspected dementia disease.

In the domain of occupational health and environmental medicine, less evidencebased knowledge is available, making the tasks 1-3 less focused. Furthermore, a large portion of the information is embedded in the work environment and the motivations for using parts of the application are rooted in the end user's current life situation. Here a larger emphasis is put on the interaction design and flow for the purpose of generating interest and motivation in the end user for taking work-related health issues seriously (Tasks 5-8). The same applies to the occupational therapy domain, where the therapist designs (tailors) the interaction to be accomplished by the client subjected to rehabilitation assessment and intervention [START_REF] Lindgren | Designing Systems for Health Promotion and Autonomy in Older Adults[END_REF]. The end users of these applications are private persons, with different life situations motivating the use of a tailored support system. These applications may be supplementary and may share knowledge between agents, such as in the case when the occupational therapist's client is a person developing, or is already diagnosed with an early stage of dementia, or is a person living with a person suffering from dementia.

In this evaluation study we focus on the tasks 5-8 since these are about equally important to the participants, who work in different application projects in our study during the evaluation period. The study then supplements an earlier study focusing the modeling of clinical guidelines in the dementia domain using the tasks 1-3 and 8 [START_REF] Lindgren | Evaluation of a Semantic Web Application for Collaborative Knowledge Building in the Dementia Domain[END_REF].

Methods

A qualitative and formative user-centered evaluation of the end-user graphical user interface of the ACKTUS system was performed using a general action research methodology. The evaluation was performed in two phases, representing two iterations of the development of the system as a whole, including the user interface design. In the first phase the methods activity analysis and cognitive walkthrough were used to form a baseline for user observations and interviews. A re-design was performed before the second phase based on the results of the first phase. During the second phase further evaluations were made using observation of and interviews with users. The methods used and the procedure is described in more detail in this section.

The evaluation study followed an action research methodology, which is a collective and participatory research method to achieve change or learning. In accordance with action research methodology, the project was carried out in cycles of planning, acting, observing the results in a natural environment and reflecting upon them, all the while involving those responsible for and affected by the practice to be changed [START_REF] Davidson | Bridging the IT Adoption Gap for Small Physician Practices: An Action Research Study on Electronic Health Records[END_REF].

The system and its context were analyzed using activity theory as base for analysis [START_REF] Davidson | Bridging the IT Adoption Gap for Small Physician Practices: An Action Research Study on Electronic Health Records[END_REF], generating a structured view of the workflow and the actions possible to perform in the system. Tasks were chosen from this set of actions for the users to perform in the user evaluation sessions. The chosen actions did not cover all aspects of the user interface, but instead those that were most relevant to the users in their ongoing modeling work using the system. The results from the user observations were then analyzed using an activity assessment protocol adapted from Lindgren [START_REF] Lindgren | Personalisation of Internet-Mediated Activity Support Systems in the Rehabilitation of Older Adults -A Pilot Study[END_REF], order to identify where in the interaction problems arose and to what degree the user managed to perform independent work with the system. In the following subsections the activity analysis and the user evaluation study are further described.

Activity Analysis

The options and available actions of the system were analyzed using activity theoretical models. The work that can be performed by a user was divided into two overarching activities on an abstract level, which are carried out by actions of different levels of concreteness. The two main activities to be executed in interaction with the system were identified as: 1) Collaborative knowledge building in domain X, 2) Collaborative interaction design of end user application in domain X (Figure 1). Firstly, the activity of collaborative knowledge building describes how medical health professionals input knowledge into the ACKTUS system interpreted into different levels of complexity. Secondly, how the knowledge is presented and interactively mediated to an end-user is modeled in the collaborative interaction design activity.

These activities are achieved through a number of possible actions using the ACKTUS system as a mediating tool. For instance, knowledge building is achieved by defining knowledge sources and structuring the knowledge into schemes and rules, while the interaction design includes actions such as managing scales, questions and assessment protocols.

Activity is performed in communities of actors; in this case, the collaborative nature of building knowledge and structuring the interaction flow for utilizing that knowledge. A single user will rarely build the knowledge from scratch, as they will typically work in relation to others' ongoing and finished work already entered into the system. The interaction will also develop over time as the knowledge in the system and conditions develop. For instance, the interaction will look different for the first users who enter the first pieces of knowledge than for later users. From this set of activities and actions, a task analysis was developed, listing all options and actions that the interface makes available, from using buttons and controls to e.g. reviewing information that is presented. This task analysis was expanded upon for use in the cognitive walkthrough segment of the evaluation.

AAIMA Protocol for Assessment. A modified version of an instrument for assessing ability in performing computer-supported activity (protocol for Assessment of Autonomy in Internet-Mediated Activity -AAIMA) was used [START_REF] Lindgren | Personalisation of Internet-Mediated Activity Support Systems in the Rehabilitation of Older Adults -A Pilot Study[END_REF] (Table 1). The activity analysis was adapted into a protocol based on this model for use in the observations of users in determining what level of autonomy and complexity a user achieves in performing an action with the system as a tool. Each activity and action is described with three levels of complexity according to the model [START_REF] Lindgren | Personalisation of Internet-Mediated Activity Support Systems in the Rehabilitation of Older Adults -A Pilot Study[END_REF].

For actions, the levels are organized as follows. On the lowest, most basic level (denoted Level 0), the user must understand the purpose of the task and be able to find the section of the interface where the task may be performed. Without an understanding of the purpose of the action, it cannot be executed in a goal-driven manner, which means it could not be considered an action (since actions are goalbased). That is, if a user were to randomly click in the interface and happen to perform the goals of the action by chance, he is still not described as having performed or accomplished the action. On a higher level, Level 1, the user is able to edit information that is already entered into the system with an understanding of the object's components (i.e. that a rule needs premises and a conclusion to function). Table 1. AAIMA-protocol [START_REF] Lindgren | Personalisation of Internet-Mediated Activity Support Systems in the Rehabilitation of Older Adults -A Pilot Study[END_REF] adapted to the study. We omit here the columns showing needs and motives and limit the Able to effectively utilize user cases to evaluate the validity of the interaction flows of the system, assessing relevance of assessment protocols and questions to a certain type of user On Level 2, the user is also able to add new knowledge into the system, with an understanding of the knowledge's interaction and relation to other pieces of information in the system (i.e. that an assessment protocol contains a set of interaction protocols or other assessment protocols). For the third and highest level, a complete understanding of the action and its context is required, and being able to review and analyze the relevance and validity of the knowledge that exists in the system.

If a level of any task causes problems (breakdowns), but the user manages to complete them under the guidance of a more experienced peer, the task is said to lie within the user's zone of proximal development (ZPD) [START_REF] Vygotsky | Mind in Society[END_REF]. This means the task has the possibility of being performed independently by the user within the near future through training and development. The notion of breakdown is used to describe the situation when the user is interrupted in completing the task, possibly due to limitations in the user interface design or the individual's understanding of the task.

User Evaluations

User evaluation sessions were carried out with five domain expert users in total. Two were experts in the field of environmental health, a domain in which the ACKTUS system is currently being applied (ACKTUS-EKF). These were chosen as they are a part of the project and have a say in its development, and represent the user group consisting of the medical professionals that will enter and structure the system's knowledge. They are also specialized in the field of the other user group, the end-user construction and mining workers. In addition, they are specialized in two supplementary sub-fields of environmental health, with a corresponding division of responsibility in the knowledge modeling process. A third medical professional, involved in the development of the dementia diagnosis subsystem (ACKTUS-Dementia), was chosen for the same reasons and participated in the second phase of the evaluation. This user had participated earlier by modeling clinical guidelines and rules corresponding to the tasks 1-3 described in previous section [START_REF] Lindgren | Evaluation of a Semantic Web Application for Collaborative Knowledge Building in the Dementia Domain[END_REF]. Two additional users, experts in the occupational therapy (rehabilitation) domain, participated. In their cases the evaluation was part of an introductory demonstration for the purpose of initiating a process of knowledge and interaction modeling using the subsystem ACKTUS-Rehab.

The users were asked to either continue tasks that they were to perform in the system in the course of their work, or perform certain predetermined tasks if they currently had nothing to work on. The areas covered by their work and the tasks were decided from what was determined to be most relevant to their work in the system at the time and included 1) managing interaction protocols, 2) managing assessment protocols and 3) creating/managing assessment flows using rules. The users were also asked questions about their experience with the interface, what was problematic or what they would like to change, etc. The users who tested the Rehab subsystem were given example tasks to finish that demonstrated how the system could be applied to their domain. Following this, they continued to model their own chosen examples with the same purpose, having gained understanding of the purpose of the system. The same procedure was done when the dementia expert was introduced to the tasks.

While the three medical professionals had experience with parts of the system, they were not familiar with all the tasks. The ACKTUS-EKF users were at a beginner level in regards to managing rules, and the ACKTUS-Dementia user had not attempted to define assessment protocols or use rules to define assessment flows before.

During the first phase of evaluation, the parts of the interface in focus for evaluation were in an initial, basic stage of development. Based on the users' performance and the other methods of evaluation, design suggestions were generated and implemented. Significant updates were made to the interfaces of interaction and assessment protocols in particular, and some updates were made to the interface for creating rules. These updated versions of the interface were used by the participants in the second phase, generating further design alterations to be implemented in the future.

The users' interactions with the system and comments during the session were recorded on video. A total of 12 hours of active modeling work was recorded part from recorded interviews. The actions mediated by the interface (clicks, typing, scrolling etc.) were then transcribed and analyzed using the aforementioned activity protocol. The transcription was divided into segments according to the actions of the protocol that the user was trying to achieve at a given time. Each action was then scored for each user by what level of complexity and with what degree of autonomy that an action was completed. Comments and suggestions were also noted and assembled as design suggestions.

Results

The results from the evaluation study involving domain expert users are further described in the following subsections with a focus on learning and developing understanding of the system detected and interpreted using the AAIMA instrument. The outcome of using ACKTUS in terms of developed end-user applications is described and aspects concerning redesign are motivated.

Using the AAIMA Protocol for Detecting Learning and Skill Development in the Use of ACKTUS

A progress in ability to use ACKTUS for their modeling purposes was seen in the users participating in the study. The results from the AAIMA protocol analysis are summarized in Table 2, which provides an overview of the results. In the table each action and activity is marked with the highest level (a number between 0-3) that the user managed to achieve over the course of 1-3 evaluation sessions during each phase. Additionally, for each activity level, how automatized each action was performed (i.e. to what extent the action caused breakdowns) was also noted, but this is excluded from the table for brevity's sake. The ZPD notation in the table designates that the user was able to complete the action at the given level of complexity, but only with some aid or guidance from the evaluator, indicating that the activity may be performed independently within the nearest future (i.e., the next session). The differences in performance between the two evaluation phases can be seen in Table 2.

In the following description we use the term participants to distinguish the professionals participating in the evaluation study from the future end users of the resulting applications. Phase 1. In the first phase of observations, none of the participants reached Level 3 of the overall activity, that is, structuring an interaction flow and framework, since they did not manage to construct rules for instantiating assessment flows. This was also technically limited at the time of evaluation. They did reach activity Level 2, in that they were able to organize components of knowledge and interaction components in the system to create assessment protocols. As such, what was limiting the interaction at this stage seems to have been the understanding of and application of rules. The only action that reached Level 3 in the first phase, full understanding and reviewing contents, was managing assessment protocols (Figure 2). Level 3 caused problems for one user however, mostly concerning understanding what objects would appear when testing the protocol and how to choose which would appear (i.e. adding questions to the protocols). To summarize the first phase, it seems that the problems for the users were in the understanding of certain actions more than their execution. That is to say, the users were able to edit and create new objects (critical questions etc.) but had trouble with distinguishing different kinds of objects, or understanding the objects' role and relationships in interaction flows (e.g. when an interaction object would come into play). Obtaining a full impression or overview of how things relate and function appeared to be problematic at this stage. Therefore, the system's presentation of these aspects was worked on before Phase 2, for instance, in the labeling of controls and objects, titles, explanations of concepts and natural structure, progression and mappings, etc. (some results are shown in Figures 3 and4).

Redesign of ACKTUS.

The different evaluation methods yielded different design issues to be addressed. The results from the first phase of the user observations and activity analysis indicated that obtaining a full impression or overview of how objects relate and function in the system is problematic. The system's presentation of these aspects needed to be addressed, for instance in the labeling of controls and objects, using clear titles, explaining concepts, using a natural structure and suggesting the next step in a workflow, reducing information load to facilitate user decisions, etc. The results also showed that the rule editor requires specific attention (Figure 5). In addition, the cognitive walkthrough, performed concurrently with the first phase of user observations, identified a number of recurring issues, chiefly concerning visibility, recognizability and understandability of controls and objects in the interface.

In Figure 2 an assessment protocol is shown as an example of an object with the previous version of the interface. Other objects followed the same color theme and design. In the case of assessment protocol, the interface did give some hints as to the structure of the content, by displaying the contents of a sub-protocol with indentation. However, in general there were many types of controls available, unsorted by importance or suggested workflow. This was true for other objects in the system as well, and may have contributed to the lower scores in the AAIMA assessments in the first phase of evaluations. These issues were addressed in the re-design following two overarching themes, namely providing objects in the interface with unique, recognizable and recurring appearances, and hiding certain controls and information when they are not needed. Objects (such as rules, critical questions, assessment protocols etc.) were given unique appearances that recur throughout the interface wherever said objects are used or referenced (Figures 3,4). The workflow in the system was improved by removing unnecessary steps and to provide more feedback at certain points in the interaction. The connection between controls and their affected components and between related objects were strengthened, and interface improvements were done to support a natural workflow, weighing content and information visually according to its relative importance to achieving the task at hand. Some of the changes implemented for the second phase of evaluations are exemplified in Figure 3 and Figure 4. Two participants increased their performance in the managing of interaction protocols to Level 3, indicating a higher understanding of the role of the objects in question in the system as a whole. Also, novice users reached high levels of performance and understanding even from only a short introductory session. Changes such as the ones displayed in Figure 3 and 4 may have contributed to this increased understanding. For one, each object now has a separate color theme, which should help the user to identify and separate objects more quickly.

Another change that may have contributed to the users' increased understanding is the edit mode (Figure 3). Hiding editing controls when first viewing an object allows the user to focus on getting an overview and understanding its content before deciding what to do with the object.

Consistent use of colors, along with icons for each object, should help the user get a sense of "what goes where" in the system. Additionally, the spine to the left in the assessment protocol is meant to further emphasize the structure of it, again to assist the user in recognizing the relations between different kinds of objects (Figure 4,5). Phase 2. In the second phase of user observations, the participants did reach Level 3 of the activity, some needing occasional aid from the evaluators (i.e. the activity lay within their ZPD), since they were also able to create rules within the system (Table 2). However, their proper understanding of the rules and their application was limited, and the concepts, terms and workflow used in the rule creation process still caused breakdowns, though these were handled by the individual in most cases. Still, a progress was seen in, e.g., User 1. In Phase 2 this user was able actually to complete rules that govern question presentation and order, with some guidance from the evaluator. This was not accomplished in the first phase of evaluation, when he only managed to edit parts of existing rules but not create his own rules.

The possibility to apply and evaluate the results in user cases was not included in the first phase of the evaluation study. In the second phase the participants were able to evaluate the content they had entered into assessment protocols by viewing it in a prototype of the end user interface, and from there enter responses in the manner of specific types of users, in order to see whether these fictive users would be shown the dedicated tailored content (e.g., Figure 5). This task generally caused no problems for the participants; indeed, a few of them even chose to view the content as the end user would see it, even when editing basic content. Fig. 5. DMSS end user prototype visualizing the short dementia assessment protocol.

Outcome of the Evaluation Sessions in Terms of Developed Applications

During the evaluation sessions, the participants continued with tasks related to their own work and were able to complete tasks yielding authentic results that will be usable in practice in the future.

For instance, in Phase 2 User 1 worked with two assessment protocols that will be used for assessing environmental health risk factors, concerning vibration specifically, for end users of the EKF subsystem. User 1 was able to review an extensive amount of questions, correcting mistakes such as spelling errors, reviewing the order the questions were presented in when answering a specific way, and editing these factors when necessary. For presentation purposes, it was also necessary for the user to create sub-assessment protocols with which to group the questions. All in all, an assessment protocol that had been developed was fine-tuned and made presentable during the participant's evaluation session. Another assessment protocol was reviewed and adjusted, giving information about what remains to be done. What caused the user the most pause was managing the rules used for deciding question order and presentation conditions.

In the case of User 3, this physician had been involved in developing a knowledge base for dementia integrated in a stand-alone prototype, subjected to evaluations with medical professionals in Asia. Following their request of a quick assessment protocol that would fit their workload and resources (preferably to be finished within three minutes, which is, however, in practice impossible), he modeled during a single session a minimal assessment protocol for diagnosing dementia by reusing components integrated in the full assessment protocol. He was able to redesign the assessment flow with some assistance from a knowledge engineer to follow the associated clinical guideline using rules, and modeled critical questions to appear in the user interface for guiding the end user. He also combined the outcome with an additional short protocol for assessing Alzheimer's disease. This was accomplished within 30 minutes. This 3-minutes protocol was seen as a guided entrance into the full protocol in the complex patient cases, and as such it was seen as a major contribution to the development and tailoring of the dementia support system to different user needs (see Figures 345).

In the case of User 4 and User 5, the setting was an initiation of a collaborative project where User 5 modeled content she will have use of in an end user situation involving older adults. User 4 functions as the leader of their common project and had a larger perspective on the use of ACKTUS in their setting, leading to her to envision a diagnostic tool that integrates research-based knowledge sources as well. Following her idea, she modeled general knowledge for this purpose, supplementary to the knowledge modeled by User 4. The modeled content was anticipated to be used by older adults in a dialogue system, where parts could be re-used and possibly adapted by other professionals in the domain.

An example of the interaction designs produced by the participants during the evaluation period is shown in Figure 5. To summarize, the results of the modeling done by the participants are accustomed, personalized interaction flows adapted to given end users' needs and requests. For instance, in the case of DMSS (Figure 5), the minimal assessment protocol for diagnosing dementia gives the end user the option of using this protocol to quickly assess a patient's mental state and only continue to the larger, more detailed protocol if they decide that this is necessary. In the case of EKF, the questions and information shown to the end user is decided by their previous responses to questions, according to the rules defined by the participants. At the end of the dialogue, the end-user receives personalized advice based on their answers.

Discussion

Defining assessment protocols generally caused little problems for the users in Phase 2. While the users did reach Level 3 in both phases, User 2 and User 4 faced breakdown situations during phase 1 and the activity lay within their ZPD. This was not the case in the second phase. One reason for this increase in skills could be the major revisions in the interface that underwent between the two evaluation phases (see Figures 345). This is also the case for the task of creating interaction protocols, which had received improved functionality and presentation since the first phase, most likely reflected in the increased performance for the users 1, 2 and 4.

One notable change is the increased performance in the task of creating rules. User 1 and User 2 increased their score from barely being able to edit rules or not wanting to participate, to being able to create new rules with some guidance in phase 2. The beginner users (User 4 and User 5) also scored high in the second phase. However, since only parts of the intended interface changes for managing rules had been implemented at the time of the second phase of evaluation due to time constraints, this change likely has other causes as well, such as an increase in motivation or learning. The EKF users may for instance have gained insight into potential applications of rules and therefore were more motivated to learn about them before and during the second phase. These two users' initial perception of the modeling tasks was that they would describe the content and a programmer then would implement the content. Their attitude changed when presented with the system and when starting to model the content, though with some delay. Another motivational factor for the two EKF users was probably the approval of, and encouragement for, the inclusion of personal advice for the end users, coming from a project reference group, which had not been approved at the time for Phase 1. They put large interest in their last sessions on modeling personal advice to appear for end users, which were to be demonstrated for the reference group. User 4, who scored 3 when creating rules, immediately saw potential applications for the rules and the system in their line of work and as such was highly motivated to test and learn.

User 4 expected a different procedure for relating the assessment protocols to the rules, making her somewhat unwillingly adjust to the procedure implemented in ACKTUS. It will be investigated whether this alternative procedure should be supported by the system in the future.

Since a convenience sample was used for selecting the participants (i.e., all professionals involved in modeling tasks during the evaluation period) and due to the fact that the sample contains only five participants, the results are only representative for the group. However, the results indicate that it is possible to learn and use the redesigned system after only a short introduction with the correct motivation grounded in a need for tailoring a knowledge system to specific users or user groups.

Adapting and applying the AAIMA protocol yielded a structured baseline for evaluation and for a continuing evaluation during the ongoing development of ACKTUS and the different end user applications. It also gave insight into a user's comprehension and capability in using the system by incorporating elements of activity theory, such as ZPD. The results from using the protocol to score user performance provided a basis for a structured comparison both between different users and between different sessions with the same user. From this, the development and learning of a participant using the system can be extrapolated, providing insight into areas of concern in the interface and how well the system supports learning. The protocol did not eliminate the subjectivity in determining whether, for instance, a user understands the relationship between different kinds of objects, but as long as the activity levels are interpreted and applied consistently according to the same criteria, the subjectivity is reduced. A way to further investigate its inter-rater reliability would be to have several evaluators apply it to the same set of data and compare their results. The protocol was originally used to assess the skills in computer-mediated activity in elderly users [START_REF] Lindgren | Personalisation of Internet-Mediated Activity Support Systems in the Rehabilitation of Older Adults -A Pilot Study[END_REF], and has in this study been used with participants from the medical professional domain. In this study it was shown that the protocol is useful for a significantly different domain, provided the protocol is adapted based on an activity analysis of the application domain. This adaptation need may prevent the application of the protocol in cases where there are limited resources for assessing the use context. Furthermore, the protocol may show less applicability also in the cases where an actual use situation with dedicated end users cannot be arranged for evaluation. The major contribution with the AAIMA protocol is that it supplements other methods in that it supports a structured theory-based interpretation of the data that can be collected e.g., in a study of focus shifts in activity execution [START_REF] Bodker | Applying activity theory to video analysis: How to make sense of video data in HCI[END_REF]. In addition, it allows for detecting changes in knowledge and behavior over a longer period of time.

Conclusions

A pilot evaluation study was conducted of ACKTUS, an early prototype for modeling knowledge and interaction design of knowledge-based support systems to be used in the health domain. The purpose was to investigate how well the system supports knowledge modeling and interaction design for domain experts in authentic use situations, as well as their development of a conceptual understanding of the system. Consequently, we focused on users with minor or no experience from using the application. Five health-care professionals with expertise in four different domains were observed using the system in 1-3 sessions each, and were interviewed. All participants were working on particular applications relevant for their domain as part of three different research projects. The concept of "zone of proximal development" (ZPD) was used to assess the development of skills, following the AAIMA assessment protocol [START_REF] Lindgren | Personalisation of Internet-Mediated Activity Support Systems in the Rehabilitation of Older Adults -A Pilot Study[END_REF] adapted to the result from an activity analysis done of the activity in focus for evaluation. The protocol proved to be a valuable instrument for detecting and describing the development of the users' skills and understanding of the system. This evaluation also demonstrates the applicability of the protocol to a new user group and a new type of system than it was originally designed for.

Between the phases the ACKTUS user interface was re-designed based on the results from the initial phase. The results show that the users who participated in both phases increased their understanding and skills, and that the two new users who entered in Phase 2 were able to reach the same level of understanding as the three who had used the system in Phase 1. In Phase 2 they were able to model components and interaction design and had gained a sufficient overall understanding of the system. The re-design of the user interface was interpreted as the major reason, and the fact that a certain amount of learning took place between sessions was interpreted as a minor factor. A more important factor was the motivational level in the users. These results illustrate the development of understanding, and that the ACKTUS application does not require extensive training and learning to use, in contrast to other approaches for integrating medical knowledge presented in the literature.

The results are fed into further re-design of the user interface, with the aim at providing novice users of the ACKTUS modeling environment a transparent and intuitive interaction design. When this is accomplished, also end users may contribute with knowledge as part of a community of users sharing a common living knowledge repository. The AAIMA protocol will be used as baseline in future evaluation studies, enabling the detection of development of skills and understanding. The protocol is also being integrated into ACKTUS for developing adaptive functionality using automated methods for detection of activity and skills.

Fig. 1 .

 1 Fig. 1. Overview of ACKTUS functionality in terms of major tasks and their subtasks (indicated by arrows). Dotted arrows represent a flow of activity. Tasks focused in our evaluation is marked.

Fig. 2 .

 2 Fig. 2. Initial version of the assessment protocol, used in Phase 1.

Fig. 3 .

 3 Fig. 3. Assessment protocols in normal and edit view created by User 3 in Phase 2.

Fig. 4 .

 4 Fig. 4. Critical question (interaction protocol) view, and the scheme and rule view in Phase 2.

  table to showing the main activity and four sub-actions in focus.

	Act. level	Activity description
	Activity	Collaborative Interaction Design of End User Application in Domain X
	Level	Understanding the purpose of the interaction design but does not participate
	Level	Contributing by defining basic units of information
	Level	Contributing by designing assessment flows
	Level	Structuring interaction flow and framework
	Action	Model Interaction Protocols
	Level	Finding interaction protocols, understanding their relation to items and scales,
		understanding the difference between information and interaction objects and
		critical questions
	Level	Editing existing interaction protocols, reviewing definitions
	Level	Creating new interaction protocols with appropriate scales or questions
	Level	Full understanding of the role of interaction protocols in structuring interaction
		flows, reviewing relevance of scales, etc.
	Action	Model Assessment Protocols
	Level	Navigating to assessment protocols and understanding their structure and
		purpose
	Level	Editing existing assessment protocols, defining concepts
	Level	Creating new assessment protocols
	Level	Reviewing an assessment protocol's structure by testing it
	Action	Instantiate Assessment flows as Rules
	Level	Finding rules, understanding scheme structure and how their content could be
		fully expressed in rules
	Level	Editing existing rules and understanding the purpose of its individual
		components (premises, conclusion, critical questions)
	Level	Creating new rules, specifying their components and relation to a scheme
	Level	Understanding the function of rules and their role in reasoning, evaluating
		whether a set of rules encompass all the content of a scheme
	Action	Apply in User Cases
	Level	Navigating to user cases, understanding their relation to assessment protocols
		and rules
	Level	Understanding the function of characteristics such as role, motive etc.
	Level	Following assessment flows in a purposeful manner according to a specific user
		case
	Level

Table 2 .

 2 Overview of results from analyses using the activity protocol in Phase 1 and 2. U3 and U5 did not participate in the first phase and U5 was new to the system in Phase 2.

	Activity/action	U1 PH	U2 PH	U3 PH	U4 OT	U5 OT
	PHASE 1	<60 min	<60 min		<60 min
	Collaborative	2	2 (ZPD)	N/A	2 (ZPD)	N/A
	Interaction Design of
	End User Application
	in Domain X
	Manage Interaction	2	2	N/A	2 (ZPD)	N/A
	Protocols
	Manage Assessment	3	3 (ZPD)	N/A	3 (ZPD)	N/A
	Protocols
	Instantiating Assessment	1	0	N/A	N/A	N/A
	flows as Rules
	Apply in User Cases	N/A	N/A	N/A	N/A	N/A
	PHASE 2	102+150min 60+63 min 45+44 min 41 min	40 min
	Collaborative	3 (ZPD)	3 (ZPD)	3	3 (ZPD)	3 (ZPD)
	Interaction Design of
	End User Application
	in Domain X
	Manage Interaction	3	2	3	3	2
	Protocols
	Manage Assessment	3	3	3	3 (ZPD)	3
	Protocols
	Instantiating Assessment	2 (ZPD)	2 (ZPD)	3	3 (ZPD)	2 (ZPD)
	flows as Rules
	Apply in User Cases	3	3	3	3	2

Acknowledgments. The authors would like to thank the professionals participating in studies and development work. The project is partly funded by VINNOVA (Sweden's innovation agency) and Emil and Wera Cornell foundation.