Even Entem

Loic Barthe

Marie-Paule Cani

Frederic Cordier

Michiel Van De Panne

Modeling 3D animals from a side-view sketch

Using 2D contour sketches as input is an attractive solution for easing the creation of 3D models. This paper tackles the problem of creating 3D models of animals from a single, side-view sketch. We use the a priori assumptions of smoothness and structural symmetry of the animal about the sagittal plane to inform the 3D reconstruction. Our contributions include methods for identifying and inferring the contours of shape parts from the input sketch, a method for identifying the hierarchy of these structural parts including the detection of approximate symmetric pairs, and a hierarchical algorithm for positioning and blending these parts into a consistent 3D implicit-surface-based model. We validate this pipeline by showing that a number of plausible animal shapes can be automatically constructed from a single sketch.

Introduction

With the spread of 3D virtual environments and of 3D printing technologies, many practitioners would like to author their own 3D shapes. Among them, animal models -including imaginary and fantastic ones -are an important category. Being able to easily create and then animate animals would be an important step for generating more lively virtual worlds. Animals are also among the models that the general public, especially children, would typically like to sculpt and print.

There is currently no fast and easy method for creating 3D models of animals. Unfortunately, getting data from 3D scans is much more difficult for animals than for humans, beginning with the obvious challenge of requiring an animal to stand still. In addition, such reconstructions are also limited to existing animals. Standard 3D modeling software, such as Autodesk's Maya or Blender, as well as digital sculpting software such as Pixologic-Zbrush, can be used for creating animals, but their complexity limits their use to experienced or passionate users. The use of 3D sculpting is possible, but is still difficult: many people are not adept at sculpting animals using real clay and in this case, they are not likely to perform much better in a virtual setting, even with an investment of time in the mastery of digital sculpting interfaces. Sketch-based modeling systems, which only require users to sketch contours in 2D, are probably the most intuitive and accessible class of methods. However, despite these advantages, they either require users to iteratively draw complex shapes part by part, using different viewpoints, or, alternatively, they require an existing data-base of 3D models.

Our work belongs to the category of sketch-based modeling methods and is the first to explore the creation of a 3D animal model from a single, side-view sketch. We are motivated by the belief that many users are capable of drawing a single sketch that depicts the contour lines and the internal silhouettes of an animal, such as shown in the top-left of Figure 1. If needed, users can use a background drawing or a photo as a guide. The process of inferring 3D geometry from the 2D sketch necessitates the use of relevant assumptions in order to be tractable, and in our context of modeling animal forms, we shall assume smoothness of the resulting shape as well as the presence of structural symmetries. Several further moderate constraints include: (a) restricting ourself to non self-overlapping limbs in the sketch; (b) requiring the user to draw both contours for pairs of symmetric limbs; and (c) ignoring the reconstruction of repetitive details scattered on the surface, such as scales. With these assumptions and constraints in place, the method we develop is capable of automatically converting an input vectorized sketch into a 3D model. The total processing time is less than one second, effectively enabling one to create new animal models in only the time required to sketch them.

Note that in this work, we only tackle the creation of the volumetric shape parts of an animal and that we do not consider the surface components that should be used for ears or scales. The ears we reconstruct are also therefore interpreted as volumes. We are not able to reconstruct large flat parts such as wings. In addition, we reconstruct limbs in a symmetric fashion, even when they were drawn in arbitrary postures in the input sketch. The construction of symmetric 3D models is usually desirable, as it ensures that left and right limbs have identical dimensions. To achieve a desired nonsymmetric posture, the 3D model can be deformed, either by using an animation skeleton and the associated skinning weights, or by directly articulating the implicit surface's skeleton.

Our processing pipeline for creating 3D animals from a sketch is summarized in Section 3. It consists of three main steps, which also correspond to our three technical contributions:

1. the identification of the animal's foreground structural parts in the sketch, with completion of the parts that are not explicitly bounded, such as the top of the legs (see the teaser figure) 2. the generation of a hierarchical graph of depths for the structural parts, using the complete results and based on an algorithm for detecting the portions of the sketch that correspond to symmetrical parts of the animal; 3. 3D reconstruction based on a specific choice of implicit surface, scale invariant integral surfaces, which enables us to accurately reconstruct shape parts from their medial axis in the 2D sketch, and to seamlessly blend them into a single animal model.

Related Work

2D sketches only represent the contours, silhouettes and main features of an object. Converting them into a 3D model therefore requires resolving indeterminacies and inferring a large amount of missing data. Four strategies are commonly used to do so, each of which is based on a different level of hypotheses or a priori knowledge of the shape being modeled (see [START_REF] Cook | A survey of sketch-based 3-d modeling techniques[END_REF] and [START_REF] Olsen | Technical section: Sketch-based modeling: A survey[END_REF] for detailed surveys):

Iterative methods enable the user to build general models part by part, by iteratively adding new shape components from different viewpoints, e.g. the Teddy system [START_REF] Igarashi | Teddy: A sketching interface for 3d freeform design[END_REF]. These methods make the hypothesis that the final shape is a combination of parts that all have planar silhouettes from a given viewpoint, and can therefore be inflated from closed planar contours. Animals belong to this category but in practice these methods still require practice and time in order to achieving a convincing result. Other iterative methods have been developed using alternatives to inflation for geometric construction. These include the use of primitives such as generalized cylinders and ellipsoids, as in [START_REF] Gingold | Structured annotations for 2dto-3d modeling[END_REF], and methods based on implicit surfaces, as in [START_REF] Schmidt | Shapeshop: Sketch-based solid modeling with blobtrees[END_REF] or [START_REF] Bernhardt | Matisse: Painting 2d regions for modeling free-form shapes[END_REF].

Shape matching approaches match the user sketch with silhouettes, or parts of silhouettes of predefined 3D models, possibly enabling some deformation. This method was successfully applied to organic shapes such as humans or animals [START_REF] Kraevoy | Modeling from contour drawings[END_REF], as well as for technical models [START_REF] Fonseca | Sketch-based retrieval of complex drawings using hierarchical topology and geometry[END_REF]. However, they require a template example of the given general class of animal which imposes a restriction on the family of sketches that can be used. We did not consider this approach in our case, in order to allow the user to imagine any animal shape without further restrictions on the number of limbs, horns, etc.

Multi-view methods, e.g., [START_REF] Rivers | 3d modeling with silhouettes[END_REF], generate 3D shapes from two or three sketches drawn from orthogonal viewpoints. They require the ability to draw consistent views of the shape to be reconstructed. They are therefore more easily applicable to man-made objects than to organic shapes. Alternatively, a number of man-made objects can be easily built from a network of 3D curves, which can themselves be directly reconstructed from perspective sketches [START_REF] Schmidt | Analytic drawing of 3d scaffolds[END_REF][START_REF] Xu | True2form: 3d curve networks from 2d sketches via selective regularization[END_REF]. However, these methods require more user input than a single contour sketch, and are generally difficult to apply to the modeling of free form, organic shapes. Lastly, direct methods try to infer 3D shapes from a single complex sketch that depicts complex silhouettes with loops, branches, cusps and T-junctions [START_REF] Karpenko | Smoothsketch: 3d free-form shapes from complex sketches[END_REF]. This technique is probably the most appealing, since it imposes few restrictions on the sketch being drawn, and mimics the human ability to see in 3D when provided a single, 2D-only representation. Different levels of a priori knowledge are used for performing this task, spanning the range from very specific context based hypotheses, as in methods for sketching flowers [START_REF] Ijiri | Seamless integration of initial sketching and subsequent detail editing in flower modeling[END_REF], garments [START_REF] Turquin | A sketch-based interface for clothing virtual characters[END_REF], trees [START_REF] Wither | Structure from silhouettes: a new paradigm for fast sketch-based design of trees[END_REF] or blood vessels [START_REF] Pihuit | Sketch-based modeling of vascular systems: A first step towards interactive teaching of anatomy[END_REF] to more general methods, such as the one for reconstructing arbitrary shapes under the hypothesis of exact geometrical symmetry [START_REF] Cordier | Sketching of mirrorsymmetric shapes[END_REF]. Our research belongs to this last category. In the context of working with animal shapes, we leverage the hypotheses of symmetrical subshapes with planar contours, smoothly blending with the main body. In contrast to related prior work exploiting symmetry [START_REF] Cordier | Sketching of mirrorsymmetric shapes[END_REF], we do not demand exact geometrical symmetry to be present in the input sketch, which would be difficult to draw for most users.

a) b) c) d) e) f) g) h) i) j) k) l) z 0

Overview

Our method builds a 3D model from a single 2D sketch. This involves the following steps, which are also summarized in Figure 1.

We start from a sagittal-view vectorized sketch of an animal in the (x, y) plane. The hand-drawn input sketch is transformed into a set of parametric curves using a (a) (b) Figure 2: Two cases of failure: (a) Discontinuous sketch: a large discontinuity, as the one on the left, prevents the detection of the main body part in the drawing. In that case our algorithm fails from its first steps. A small gap in the sketch, as on the left, is filtered and considered as a connection during our first step. The reconstruction will perform adequately. (b) Incomplete sketch: on the left, a structural part is missing in the sketch, here, the left inner-body silhouette of the front foreground leg. Our algorithm will not be able to detect the presence of a limb and the reconstruction of the legs will fail. On the right, both inner-body silhouettes of the rear foreground leg are drawn and the rear legs will therefore be adequately reconstructed.

vectorization algorithm such as the one proposed by Noris et al. [START_REF] Noris | Topology-driven vectorization of clean line drawings[END_REF]. These algorithms are relatively robust to noise and provide the smooth curves we seek. What the user must really pay attention to is the correctness of the input hand-drawn sketch topology, i.e., the provision of continuous closed contours that do not have spurious crossings with other contours. Figure 2 illustrates two failure cases, the first, in 2(a), is due to an incorrectly closed contour and the second, in 2(b), is due to an incomplete input.

The goal of the first step, detailed in Section 4, is to identify the strokes that correspond to the same structural part, such as an eye, leg, or body, and to infer any missing curves, i.e., perform contour completion, if the resulting contour is open. This is done in the following way: We build a counter-clock-wise oriented half-edge graph whose edges are in general bounded parametric curves (see Figure 1(a,b)). In this graph, we iteratively follow successive half-edges, and store cycles in a list (Figure 1(c)). Each cycle is classified as being either an outer-sketch, island, border or other as explained in Section 4.1 and illustrated in Figure 1(d).

We then define as inner-edges those edges where both half-edges belong to the same cycle (the green edges in Figure 1(e)). These lists of inner-edges represent suggestive curves where two shape parts merge (such as a leg merging with the body). This implies the presence of hidden silhouettes or cusps [START_REF] Karpenko | Smoothsketch: 3d free-form shapes from complex sketches[END_REF]. Each suggestive curve has an open endpoint and a closed endpoing, which are depicted in red and blue, respectively, in Figure 1(e). The next step is to pair these suggestive curves and con-nect their extremities in order to infer new cycles representing the different structural parts of the animal (Figure 1(f)). This is the topic of Section 4.2.

The subsequent step, presented in Section 5, is the computation of a depth hierarchy for structural parts, based on the structural symmetry hypothesis. We first identify the main body part located in the saggital plane, i.e., the torso, and detect structural symmetries around it such as pairs of ears or pairs of front legs, cf. Section 5.1 and Figure 1(g)). We then use the pieces of information we already gathered, e.g., island cycles, structural parts, chest, and symmetries, to define a depth hierarchy between structural parts, as detailed in Section 5.2 and Figure 1(h). This leads us to the last step, detailed in Section 6 which consists of the 3D reconstruction of the animal model. For each pair of limbs, only the one in the foreground of the drawing is considered during the reconstruction steps. The 3D reconstruction of the background limbs are added at the end of the process by symmetry. We choose implicit surfaces to represent shape parts, in order to benefit from their blending capabilities. We first compute the medial axis [START_REF] Fortune | A sweepline algorithm for voronoi diagrams[END_REF] of each structural part (Figure 1(i)). Medial axes are then filtered and specifically simplified in order to be be used as skeletons for 3D implicit surface modeling (Section 6.1 and Figure 1(j). Among the variety of existing implicit models, we use scale invariant integral surfaces (SCALIS) [START_REF] Zanni | SCALeinvariant Integral Surfaces[END_REF] to accurately reconstruct shape parts (Section 6.2). Figure 1(j)) shows the 3D reconstruction of each structural shape part in isolation. Structural parts without sym-metries are placed in the sagittal plane of the model. The depth of the foreground structural parts is computed using the thickness of their 3D reconstruction and the thickness of the part to which it is connected in the sagittal plane (Section 6.3 and Figure 1(k)). Symmetric background parts are then added to provide the final 3D model (Figure 1(l)). The final shape is obtained using a simple sum to blend the fields from the different implicit primitives.

The next sections give a detailed presentation of our solutions at each stage of this process.

Structural parts identification and completion

Representing the sketch as a set of cycles

The first step is to decompose the curves of the sketch into a set of cycles. This decomposition is a preliminary step towards identifying the different parts of the animal drawn in the sketch. In particular, this step is required for the hidden contour computation and the contour closure, as described in Section 5.

The input data is a hand-drawn sketch composed of a set of non-crossing curves that are connected to each other (see Figure 1(a)). Each of these curves is represented by a pair of directed half-edges1 of opposite direction (see Figure 1(b)). The curves of the sketch are then represented using a graph whose edges are the halfedges of the curves and whose nodes are the points at which the curves are connected to each other. Next, we decompose this graph into a set of cycles.

Formally, a cycle is defined as a closed sequence of connected half-edges; each half-edge shares its endpoints with at least two other half-edges. We require the half-edges that compose a cycle not to cross each other; we also require the cycles to be topologically equivalent to a disk. Therefore, a cycle divides the sketch into an "interior" region and an "exterior" region. In order to define the interior of a cycle, cycles and their associated half-edges are given a direction which is either clockwise or counter-clockwise; the interior of the region bounded by a cycle lies on the left side of the directed half-edges that compose this cycle.

The first cycle that we identify is the one which is composed of the half-edges located along the outer boundary of the sketch. This cycle is oriented clockwise and is unbounded; it corresponds to the outer region of the sketch. We name this the outer sketch cycle. Next, we identify the border cycles which are cycles that have one or several half-edges belonging to the outer boundary of the sketch. All of these cycles are oriented in a counter-clockwise fashion. The process to construct a border cycle is as follows: we first select a half edge which is located along the outer boundary of the sketch and which has not yet been associated to a cycle. We next find its endpoint by following its direction and select the next half-edge. This next half-edge is chosen such that its direction is the same as the previous halfedge and the clock-wise oriented angle between the two half-edges is the smallest possible, as illustrated in Figure 3. The process of finding the next half-edge is iterated until we reach the first half-edge.

Finally, we identify the island cycles; island cycles are single half-edges that correspond to a closed curve with no self-intersection and which are not connected to any other curves of the sketch. These island cycles are also oriented counter-clockwise, meaning that they do not represent a hole but rather a surface feature of the animal (such as the eye of cat model shown in Figure 1). After processing all the cycles, there may remain some cycles that could not be classified as any of the three types (outer sketch cycles, border cycles and island cycles); we mark these cycles as others.

Contour completion

The goal is now to convert the set of cycles into closed contours associated to each structural part of the shape.

Once the graph cycles are classified as described above, we use the hypothesis that structural parts have at least one edge in contact with the outer part of the sketch, and consider only the border cycles. In these cycles, we extract the sets of neighbor embedded edges, i.e. edges whose half-edges both belong to the same cycle, and then annotate these as being suggestive curves. Each set is an open path in the graph with one extremity connected to the rest of the graph, i.e., a closed extremity, and one extremity without any connection, i.e., an open extremity (see Figure 1(e)). As we focus on organic models, the suggestive curves come from the drawing of the silhouette of structural parts that stop where the limb smoothly merges with the part of the shape over which they are drawn (for instance the cat legs over the body in Figure 1). We require the user to provide two suggestive curves clearly delimiting each merging limb. Parts need to be individually closed to be identified and reconstructed. To do so, we compute a set of cubic Bézier curves linking open extremities and closed extremities pairwise with a C 1 continuous connection with the edge curves while minimizing the normalized sum of the curvature variations ("Gestalt principles"):

E = 1 l 2 i |k i+1 -k i | where k i = ċi × ci ċi 3 , (1)
Here, l is the curve length, k i is the curvature of the curve at sample point i, ċi is the curve first derivative, and ci is the second derivative at parameter value u i . The sum of curvature variations, l 2 E, is used as the energy associated with the curve. For curves connecting closed extremities, these energies are denoted E c and for those connecting open extremities they are denoted E o . We then compute two energies E i (i = 0, 1) for each pair of suggestive curves, one for each way the closed extremities can be connected (dashed curves in Figure 4(a,b)). These energies are computed as follows:

E i = p i 0 .p i 1 E o + E i c , (2)
where E i c is the energy associated to the curve connecting closed extremities (i = 0 for the closure illustrated in Figure 4(a) and i = 1 for the one illustrated in Figure 4(b)), p i 0 and p i 1 are the probabilities that the curve is correctly connected at each extremity respectively. These probabilities are computed as the ratio p i j = min α i j /β j , 1 ( j = 0, 1) in which angles α 0 0 and β 0 are illustrated in Figure 4(c). All other angles are defined in a symmetric fashion. The best-matching pair of suggestive contours is then chosen as the pair having the minimal sum of energies.

Once the suggestive contours are paired and closed, we define their set of half-edges and derive the new cycles. In these cycles, no edge is embedded in the defined part (i.e. no edge has both half-edges in the same cycle). Together with the remaining border cycles and the island cycles, this gives us the set of the animal structural parts drawn in the sketch.

The minimization of curvature variations is known to provide plausible results when completing organic curves. The selected completion curves linking open and closed extremities does not, in general, intersect other contours. However, if this should occur, it will not be expected as it does not faithfully correspond to the intent of the input sketch, although it does not compro-mise the rest of the reconstruction. If necessary, the user can correct the completion by editing the extremities.

Depth hierarchy of structural parts

Body and symmetrical parts

In order to assign appropriate depth values, we create a graph that represents the desired depth relations. This begins with the choice of a reference part as the root node of the graph, for which it is natural to use the body or torso. For some animals, this main body part may be smaller than the limbs. As such, we did not find a robust method to automatically label this in the input sketch. Instead, we request the user to select it or to draw the sketch so as to locate it in the center of the drawing canvas. Once labeled, this part defines the node of reference for the graph. The method described below automatically locates the remaining structural parts with respect to the body, in terms of their relative depth. We begin by detecting the sketch strokes that correspond to pairs of symmetric parts, as explained next.

We already know the parts, F i , that should be located in front of the body and thus that may have a symmetrical part in the background. These are defined by the open contours with suggestive curves we just completed at the previous step. The goal is now to identify the strokes from the sketch corresponding to B i , the structurally symmetric part with respect to F i , but located in the background. Indeed, B i should not be processed any further, since we are going to use symmetric geometry to create the background limbs. In practice, B i may be composed of several distinct cycles, since it is partly occluded. We use a propagation method for fully selecting it, as we now describe.

We initialize B i with all cycles not belonging to the shape part under F i , but that share an edge with F i . We then compute the medial axis of F i as described in Section 6. Let us denote as p 0 the medial axis extremity corresponding to the suggestive silhouette we closed (see Figure 5). We then denote as p 1 the middle of the closed extremities of the suggestive curves and then define v 0 = p 1p 0 . Starting from both closed extremities, we march along both sides (one clock-wise and the other counter-clock wise) of the cycle of the structural part which is located under F i in the sketch, as long as the opposite half-edge belongs to the outside sketch. If we meet an opposite half-edge belonging to a border cycle, we get both extremities of the shared edge and we compute their middle point p 2 . We define v 1 = p 2p 0 and compute the angle α = | v 0 , v 1 |. If α < π/4 we consider that this cycle potentially belongs to B i , since the angle formed by two symmetrical parts is unlikely to be larger than this value in usual standing poses. Note that this criterion could be relaxed to handle a larger set of poses.

We stop when the half-edge we reach is an occluded edge, i.e., when we meet the next foreground part. If selected on multiple occasions, a potentially symmetric cycle is assigned to the B i for which the angle α with F i is minimal. The results of this selection process are depicted in Figure 1(g), where pairs of symmetric parts are depicted using the same colors.

Graph construction

In order to support the 3D reconstruction of the model, we build a graph whose nodes are the structural parts (and their associated cycle) and whose edges represent the relations over, under, adjacent or over-in with respect to the main body. Foreground parts are defined as lying over the part they have been separated from. Symmetric counterparts are located under the part that their foreground is over. All remaining border parts that have not been designated as part of a symmetric pair are defined as being adjacent to parts with which they share an edge. Islands are assigned to the graph by first using a simple 2D ray-cast in the sketch plane in order to identify the part within which the island lies. The island cycles are over and inside a structural part, and this part is also then defined as being under the island. Each edge is further tagged with additional information, such as the presence of suggestive curves, which help determine if the region being over a particular part is an island or a limb.

3D Reconstruction

Medial-axis computation & simplification

Animals have the convenient property that their shape is naturally approximated as a set of generalized cylinders. A generalized cylinder is a surface obtained by moving a circle of varying radius along a 3D arbitrary curve. An advantage of generalized cylinders is that their skeleton and associated radii are easily determined from their projected image. Given a 2D simple closed curve C, the skeleton and the associated radii of the generalized cylinder whose orthogonal projection matches C are obtained by computing the medial axis and the associated radius function of the curve C.

We start the 3D reconstruction step by reconstructing each structural part independently. The parts which were detected as being symmetric to foreground parts are not considered. The contour curve of each part is a) b)

Figure 6: Medial axis computation and simplication. In (a), the medial axes of the sketch components computed using Fortune's sweep-line algorithm [START_REF] Fortune | A sweepline algorithm for voronoi diagrams[END_REF]. In (b), the skeletons after removing the branches using the Scale-Axis Transform [START_REF] Giesen | The scale axis transform[END_REF] and after simplifying the medial axes using the Douglas-Peucker algorithm [START_REF] Douglas | Algorithms for the reduction of the number of points required to represent a digitized line or its caricature[END_REF]. first sampled so as to obtain a polygonal curve. We then compute the medial axis transform of each of the polygonal curves using the Fortune's sweep-line algorithm [START_REF] Fortune | A sweepline algorithm for voronoi diagrams[END_REF]. The medial axes usually contain many branches because of the noise in the polygonal curves. We use the Scale-Axis Transform method by Giesen et al. [START_REF] Giesen | The scale axis transform[END_REF] to remove these branches. In our experiments, the scale factor of the Scale Axis Transform has been set to 1.2.

The last step is to simplify the medial axis; the purpose of this simplification is to lower the computation time for generating the surface of the reconstructed animal. We have implemented a modified version of the Douglas-Peucker algorithm [START_REF] Douglas | Algorithms for the reduction of the number of points required to represent a digitized line or its caricature[END_REF] which is a method to reduce the number of points of polygonal curves. A cost function is defined for each point, which is the maximum distance between the original curve and the curve after removing the point. Points are removed iteratively in increasing order of their cost. The algorithm stops when the cost of the point to remove is above a user threshold. Figure 6 In (a), the polygonal curve (depicted in black) and its medial axis (depicted in red) before the simplification. In (b), the medial axis has been simplified by removing the point p B . The result is new polygonal curve with fewer segments. The cost value of removing the point p B is d; it is the maximum distance between the original curve (depicted in light grey) and the curve after the simplification.

The Douglas-Peucker algorithm has been modified to take into account the radii of the medial axis. This modification consists of a new cost function. Let C be a polygonal curve and p A , p B , p C be three neighboring points of the medial axis of C. If the point p B is removed, the two segments p A p B and p B p C are replaced with the segment p A p C (see Figure 7). Similarly, the curve corresponding to the medial axis is simplified; two of its segments are removed. The new cost function is defined as the maximum distance d between the original curve and the curve after the simplification of the medial axis.

Surface reconstruction of the shape components

Once the skeletons of all the shape components have been computed, the next step is to reconstruct the 3D surface of these shape components. We have implemented the SCALe-invariant Integral Surfaces (SCALIS) method proposed by Zanni et al. [START_REF] Zanni | SCALeinvariant Integral Surfaces[END_REF]. This method generates an implicit surface using a skeleton and its radius function. In Zanni's method, the scalar field is computed with the Homothetic Polynomial kernel of degree 8. In our method, we have implemented the kernel of degree 6 instead of degree 8. Although the quality of the reconstructed surface is slighly lower, the computation time is much smaller, and allows for surface reconstruction at near-interactive rates. The result is a set of implicit surfaces, one for each shape component (see Figure 1(k)).

Placement & embedding of the shape components

The last step of the surface reconstruction is to assemble the implicit surfaces to generate the 3D shape of the animal. At this stage, the skeletons of the implicit surfaces are all located in the same plane which is the sketching plane. The goal is to place these implicit surfaces so that their depth order complies with that of the sketch; the shape components that are drawn in the sketch foreground should be placed in front of the sketching plane and those drawn in the sketch background should be placed behind the sketching plane. We make use of the depth graph that has been previously computed (see Section 5.2).

We start by placing the implicit surface corresponding to the main body component in the sketching plane. Other implicit surfaces are sequentially placed as follows. Using the depth graph, we find a shape component that has not yet been placed and which is adjacent to a shape component which has already been placed.

Using the depth order between these two shape components, we compute the depth position of the one to be placed.

Let I A and I B be two implicit surfaces, with I A being in front of I B . The depth coordinates of I B have been already computed; those of I A have to be computed. Let p A and r A be the skeleton extremity of the implicit surface I A and its radius respectively. We first compute p A,0 the orthogonal projection of p A onto I B . Next, we compute ---→ N A,0 , the surface normal of the implicit surface I B at the point p A,0 . Finally, we compute p A,1 such that ------→ p A,1 p A,0 is equal to r A and such that the two vectors ---→ N A,0 and ------→ p A,1 p A,0 are parallel and have same direction (see Figure 8(a)). The depth coordinate d A,1 of p A,1 is then used as the depth coordinate of all the points of the skeleton of I A (see Figure 8(b)).

Once the depth coordinates of all the implicit surfaces have been computed, the 3D shape of the animal is then generated by blending these implicit surfaces using the well-known Ricci blending operator. We implement our method as a Maya plugin, enabling us to use the existing sketching interface of Maya to design the vectorized sketches we need as input. Processing the 2D sketches into 3D models is then a fully automatic process. The teaser image and Figures 9101112show a set of results from our method. They validate the fact that our method is able to generate plausible rough shapes for various animals from a single sketch. Note that since we only used a structural symmetry hypothesis around the sagittal plane, our animal models may have an arbitrary number of protruding subparts such as limbs, ears (interpreted as volumes) or horns. Moreover, our choice of scale invariant implicit surfaces (SCALIS) as the 3D representation enables us to capture both the smooth-ness of organic shapes and the singularities that may come from the sketch, such as at the tip of the cat ears.

Results and discussion

Pairs of limbs are reconstructed symmetrically, but the user can manually adjust the pose in 1 to 3 minutes by deforming the geometric skeleton (see Figure 1(i) the geometric skeleton of the cat model). Background limbs can be positioned in sketch pose as in Figure 12(c), or the model can be globally deformed as in Figure 12(mantis-d). In Figure 12(elephant), the ears are not reconstructed as they do not represent any of the different cycles considered in the sketch for our reconstruction (as explained in Section 4.1).

The results have been produced on an Intel Core i7 3770K computer, and the code is compiled with ICC 15.0 in release mode. For all our examples, the full sketch-processing and 3D modeling process run in less than one second. This makes the method applicable within an interactive sketching software tool. More precisely, the computational time for the giraffe is 0.35 seconds, for the gazelle is 0.30 seconds and for the cat 0.37 seconds.

To further validate our system we performed a preliminary user study with 4 different users ranging from beginners to a professional computer artist. Their sketches and resulting models are shown in Figure 13. They all found the concept of suggestive curves to delimit the limbs quite easy to understand. When asked how well the 3D model met their expectations, on a scale from 1-9, they gave a score of 7. The time spent to create the input drawings using the Pencil Curve Tool in Autodesk Maya was less than 2 minutes for all the drawings, with the exception of the rabbit, which took 10 minutes to draw.

Limitations:

Our method has a number of remaining limitations. First, we are limited in the range of sketches we are able to handle:

• The sketch cannot include adjacent internal parts (islands in our terminology) such as if the cat had two adjacent eyes. Otherwise, the classification algorithm will fail;

• A shape part cannot self-overlap in the sketch (such as an animal tail looping and self-occluding itself), nor overlap any other shape part (such as a tail occluding the animals body). Our completion algorithm does not handle these cases. Note that we partly avoid this occlusion problem by using symmetry when creating the limbs, but developing a more general solution would be desirable.

• Drawing pairs of suggestive contours is mandatory when two shape parts blend (such as at the top of a front leg, where it blends with the body), and these contours should be such that our completion algorithm keeps the reconstructed part of the contour inside the main body-shape. Although this holds true in most cases, we currently offer no guarantees regarding this point.

A second family of limitations concerns the 3D shape we output:

• All the shape parts we create have a planar medial axis, which we locate in a vertical plane. Although this works quite well for the legs, this method needs to be improved for parts such as the ears or the horns, which should preferably be oriented to lie in a direction that is normal to the surface.

• We only use segment skeletons for generating the implicit shapes, as opposed to surface skeletons, which restricts our construction to generalized cylinders. This prevents us from capturing the flat surfaces that one can readily observe on the flanks of many animals, or flat features such as elephant ears that cannot be reconstructed with our approach (see Figure 12).

• We reconstruct pairs of limbs using symmetric poses. Automatically fitting the reconstructed background parts is non-trivial since we do not reconstruct a rigging skeleton and most of the drawing of the background limbs is missing. Performing this fitting remains a challenging open problem on its own.

Conclusions and future work

We have presented the first method, to the best of our knowledge, for creating 3D animal models from a single sagittal-view sketch. Our method handles complex sketches with open curves, closed curves, and Tjunctions. Once the main body part is selected, the method is fully automatic for reconstructing a symmetric version of the 3D shape. The reconstructed shape is inferred using only two hypotheses: shape smoothness, except at singular points appearing on the sketch contour; and structural symmetry. The resulting 3D model can then be posed using standard software or by directly manipulating its skeleton.

In future work, we would like to address some of the limitations we listed. In particular, we are in the process of extending our stroke classification and completion method to more general families of sketches. Being able to convert any sketch to the vector graphic complexes proposed in [START_REF] Dalstein | Vector graphics complexes[END_REF] would be an excellent intermediate step for our application. In addition, exploiting the 2D detection of partial symmetries in the cycles, using techniques such as the one by Mitra et al. [START_REF] Mitra | Partial and approximate symmetry detection for 3d geometry[END_REF], may be a promising direction of investigation for improving the pairing stage of our approach. This could also help for handling more complex inputs such as sketches in arbitrary views. A last caveat is that the technique we currently use is not able to reconstruct large flat parts such as wings. A direction is to use more complex skeletons including surface parts as was previously done in the context of reconstruction with convolution surfaces [START_REF] Alexe | Shape modelling by sketching using convolution surfaces[END_REF].

Figure 1 :

 1 Figure 1: Overview of algorithmic stages: a) Half-edge graph; b) Detailed view of the resulting bounded curves; c) Cycles extracted from the graph, each one depicted in a different color; d) Classification of cycles: the different colors now represent the cycle type; e) Detection of suggestive contours; f) Completion of contours around shape parts; g) Detection of the main body and of pairs of symmetrical parts (displayed using the same color); h) Extracting a depth ordering; i) Skeletons extracted from the medial axes of shape parts; j) Reconstructed shape parts; k) Front view: inferring depth information for shape parts; l) Final result after implicit blending.

Figure 3 :

 3 Figure 3: Selecting the next half-edge. In (a), the selection of the next half-edge (shown in green) for T-junctions is chosen such that the angle between the next half-edge and the preceding one (shown in red) is the smallest possible. In (b), the selection of the next half-edge for cusps.

Figure 4 :

 4 Figure 4: (a) and (b): Illustration of the two possible completions of a pair of closed extremities. (c) Close-up on a closed extremity illustrating the two angles α 0 0 and β 0 used in the computation of the probability p 0 0 .

Figure 5 :

 5 Figure5: Identification of the cycles belonging to the shape part at the background that is structurally symmetrical to a foreground limb.

  (b) illustrate the result of the cat skeletons simplification.

Figure 7 :

 7 Figure7: Medial axis simplification. In (a), the polygonal curve (depicted in black) and its medial axis (depicted in red) before the simplification. In (b), the medial axis has been simplified by removing the point p B . The result is new polygonal curve with fewer segments. The cost value of removing the point p B is d; it is the maximum distance between the original curve (depicted in light grey) and the curve after the simplification.

Figure 8 :

 8 Figure 8: Computation of the depth coordinates of the implicit surface I A . In (a), the depth coordinate d A,1 of the skeleton extremity p A is computed. In (b), the skeleton (depicted in red) of I A is translated to the depth coordinate d A,1 .

Figure 9 :

 9 Figure 9: Different views of the 3D reconstruction of the cat sketched in the top-left.

Figure 10 :Figure 11 :

 1011 Figure 10: Different views of the 3D reconstruction of the gazelle sketched in the top-left.

Figure 12 :

 12 Figure 12: Reconstruction of a mantis, an elephant and a dinosaur. (a) The input vectorial sketch, (b) the automatically generated result, (c) the model with the foreground limbs manually adjusted to the sketch and (d) different views of the reconstructed models. The mantis is shown with different poses in (d).

Figure 13 :

 13 Figure 13: Reconstruction of imaginary creatures and a rabbit made by four different users ranging from beginners to a professional computer graphics artist (the rabbit model).

Note that each graph edge geometrically correspond to a full curve of the sketch.

Acknowledgements

We also thank the anonymous reviewers for their useful comments and suggestions, as well as Cédric ZANNI for sharing his implementation of SCALIS. This work was funded by the advanced grant EXPRES-SIVE from the European Research Council (ERC-2011-ADG 20110209) and by the project IM&M (ANR-11-JS02-007).


Dominique Fober
email: fober@grame.fr

Yann Orlarey
email: orlarey@grame.fr

Stéphane Letz
email: letz@grame.fr

Augmented Interactive Scores for Music Creation

This article addresses music representation issues in the context of the contemporary music creation and performance. It exposes the main challenges in terms of music notation and representation, in regard of the new forms of music and with an emphasis on interactive music issues. It presents INScore, an environment for the design of augmented, interactive music scores that has been developed in response to current artistic evolutions. It gives an overview of the system with a presentation of its main features and highlights on the main technologies involved. Concrete examples of use with recent music creations, composers' viewpoint and an electro-acoustic piece modelling will also be given.

Up to a few years, the support provided by computer music to digital music scores has remained quite conventional. Regarding the symbolic notation of music, the approach was mimicking the traditional engraving process. Sophisticated software like MuseScore 1 , or Lilypond 2 (Nienhuys H.-W., Nieuwenhuizen J. 2003) -to cite just a few among free software -provide very efficient ways to generate music scores but for traditional usages i.e. similar to paper scores.

Acousmatic and electro-acoustic music genres have lead to new forms of graphic representation of the music, for analytic and musicological purposes, with music scores made a posteriori. This kind of notation has found little support from computer music, apart the approach proposed for years by the Acousmograph [START_REF] Geslin | Sound and musical representation: the acousmographe software[END_REF] or more recently with EAnalysis [START_REF] Couprie | Eanalysis : Aide a l'analyse de la musique electroacoustique[END_REF].

Premises of interactive music are present for a long time. Stockhausen [START_REF] Stockhausen | Refrain for 3 players[END_REF] or Cage with the Variation series [START_REF] Cage | 1966 Variation I[END_REF](Cage -1966)), have explored indeterminacy and performers interaction, with specifically designed music scores. But with the use of computers, interactive music has become a common musical form.

More generally, the music score has to face the new musical forms resulting from the digital tools used at all levels of the music process, from composition to performance, including live coding [START_REF] Magnusson | Algorithms as scores: Coding live music[END_REF]. It has also to face the migration of musical instruments to gestural and mobile platforms, hybridizations with dance, design, and multimedia.

Music notation generated in interaction with live performance exists for more than a decade. As mentioned by Freeman [START_REF] Freeman | Bringing instrumental musicians into interactive music systems through notation[END_REF], numerous approaches exist: selection of pre-determined score excerpts (Kim-Boyle 2005), mixture of symbolic and graphic elements [START_REF] Winkler | The realtime score. A missing link in computer music performance[END_REF]), use of unconventional graphical notation [START_REF] Gutknecht | Goingpublik: using realtime global score synthesis[END_REF], complex staff based notation [START_REF] Didkovsky | Recent compositions and performance instruments realized in the java music specification language[END_REF]).

These works are based on custom tools that are generally specifically suited to a composer approach. Didkovsky used JMSL [START_REF] Didkovsky | Java music specification language, an introduction and overview[END_REF], a programming language for Java applications, to design interactive scores. Baird is using Lilypond for audience interaction [START_REF] Baird | Real-time generation of music notation via audience interaction using python and gnu lilypond[END_REF]. Lilypond can't be considered as a real-time environment for generating music scores, but it works in Baird's context due to relaxed time constraints.

With the recent Bach [START_REF] Agostini | Bach: An environment for computer-aided composition in max[END_REF] or MaxScore [START_REF] Didkovsky | Maxscore: Music notation in max/msp[END_REF] environments, the symbolic dimension of the music notation starts to be accessible to interaction using common tools.

Representation of the interaction process may also help for the design and the performance of interactive music pieces. When a computer is involved in a performance, almost no feedback is provided to the performer regarding the interaction system state. When there is, the system displays information like cues, time, or pedals state on a separate monitor that is presented is parallel to a paper score. Most of the time, the performer has to concentrate on the score, which makes the separate display hard to follow. (François & al.2007) has designed a system for visual feedback in performer-machine interaction that is focusing on musical improvisation. In the improvisation domain, OMax is also proposing a visualization [START_REF] Lévy | Omaxist dialectics : Capturing, visualizing and expanding improvisations[END_REF] under the form of a graphic representation of the Factor Oracle graph.

Mostly, audio feedback remains the main channel to convey the interactions system state, which is far from being sufficient in many cases.

Performance representation may also be valuable for the performer in rehearsal situation or for pedagogic purpose. For the latter and based on a mirror metaphor, experiments have been made to extend the music score in order to provide feedback to students learning and practicing traditional music instruments [START_REF] Fober | Vemus -feedback and groupware technologies for music instrument learning[END_REF]. This approach was based on an extended music score, supporting various annotations, including performance representations based on the audio signal, but the system was limited by a monophonic score centred approach and a static design of the performance representation.

In regard of the issues above and of the contemporary music creation needs, a unified environment, covering symbolic and graphic notation, opened to real-time interaction is missing from the current landscape of solutions for digital scores and music notation. This has been the main motivation for the development of INScore, an environment for the design of augmented interactive music scores (Fober & al. 2012a).

The next sections give an overview of the INScore environment and of the main technologies involved. Next, this article explains how these technologies are involved in the creation of innovative music scores. Examples of use in concrete artistic situations are given with pieces from Richard Hoadley, Sandeep Baghwati, Jean-Baptiste Barrière and a modelling of Turenas from John Chowning.

INScore Overview

INScore is an environment for the design of interactive augmented digital music scores. It extends the traditional music score to arbitrary heterogeneous graphic objects:

• symbolic music notation using the Guido Music Notation format [START_REF] Hoos | The GUIDO Music Notation Format -a Novel Approach for Adequately Representing Score-level Music[END_REF] or the MusicXML format [START_REF] Good | MusicXML for Notation and Analysis[END_REF],

• text (utf8 encoded or html format),

• images (jpeg, tiff, gif, png, bmp),

• vectorial graphics (basic shapes like rectangles, ellipses, bezier curves or SVG),

• video,

• an original performance representation system [START_REF] Fober | Time synchronization in graphic domain -a new paradigm for augmented music scores[END_REF].

Each component of a score has a graphic and a temporal dimension and can be addressed in both the graphic and temporal space.

The graphic properties include common attributes like position, scale, colour, basic transformations like 3-axis rotations, shear, and effects like blur, colorize or shadow.

The time properties include a date and duration.

Time synchronization in the graphic space

A simple formalism is used to describe relations between the graphic and time space and to represent the time relations of any score components in the graphic space on a master/slave basis.

The formalism relies on segments and mathematical relations between segments. Time segmentation constitutes the spine of the system. Each object includes at least one mapping that is a relation between the object graphic and time segmentations. Composition of different objects relations through their time segmentation results in a graphic-to-graphic relation that expresses the objects time relationships in the graphic space.

The scope of this formalism extends beyond the music score context and could be used as well to described relations between gestures and audio in a performance setting. A detailed description is available from [START_REF] Fober | Segments and mapping for scores and signal representations[END_REF].

Figure 1 shows graphic rectangles used as time cursors and synchronized to a symbolic score.

Performance representation

Performance representation proved to be valuable in particular in a pedagogic context [START_REF] Fober | Vemus -feedback and groupware technologies for music instrument learning[END_REF]).

INScore includes a performance representation system based on signals (audio or gestural signals). The system approaches the graphic of a signal as a graphic signal, i.e. as a composite signal made of several signals in parallel:

• a y coordinate signal A composite graphic signal includes all the information required for drawing without additional computation. It constitutes a very flexible system to elaborate dynamic representations of a performance. Figure 3 gives the example of a combination used to represent both pitch and articulations. The graphic signal g is expressed using 3 signals in parallel as follows:

g = S f0 / S rms / k c
where S f0 is the fundamental frequency, S rms is the RMS values and k c is a constant colour signal.

A message based system

INScore is a message driven system that is based on the Open Sound Control [OSC] protocol [START_REF] Wright | Open Sound Control 1.0 Specification[END_REF]).

An OSC message is composed of an address, specified like a Unix path, followed by arbitrary parameters. The parameters type is among integer, floating point and string values. Example:

/mysynth/channel1/vol 0.7

INScore specializes the basic OSC format with a string as first parameter that describes an action or an attribute of the target object of the OSC address (Figure 4). For example, the following message:

/ITL/scene/score x 0.8

addresses an object named score to set its x position attribute to 0.8.

Messages are provided to control the graphic and time attributes of the score components, to design graphic signals, to synchronize objects and to describe interactive behaviours.

The set of supported messages constitutes the application programming interface [API] and the main way to design music scores.

This message-oriented design opens the door to remote control and to interaction using any OSC capable application or device (typically Max/MSP, Pure Data, but also Python, CSound, Super Collider, etc.)

Musical process representation

Representation of musical process may be critical for the performer in case of interactive music. Existing representation systems are coupled to specific environments and disconnected from the music representation itself.

INScore approach to process representation is inspired by [START_REF] Berthaut | Rouages: Revealing the Mechanisms of Digital Musical Instruments to the Audience[END_REF], where the objective was to improve audience experience notably in the case of electronic music. Computer musical processes are characterized in [START_REF] Fober | Representation of musical computer processes[END_REF]). From INScore viewpoint, a process state and activity is denoted by signals. INScore allows connecting a signal to any graphic attribute of a score component (position, rotation, scale, colour…). This way, any object of a score may convey information about interaction processes.

Events based interaction system

INScore provides interaction features provided at score component level by the way of watchable events. These events are typical UI events (like mouse clicks, mouse move, mouse enter, etc.) extended in the time domain (time enter, time leave, etc.). The principle consists in associating a list of messages to an event (Figure 5). When an event occurs, the associated messages are triggered.

OSC address attribute values

Figure 5. Basic mechanism to describe interactions: a set of messages is associated to an event by sending the watch message to an object.

The messages could be any valid INScore message but arbitrary messages as well, which can be send to any external application using an extended address scheme.

Plugins

INScore can be dynamically extended via external plugins, which are loaded when the corresponding objects are created. It supports currently 2 extensions, one in the domain of gesture following with the IRCAM gesture follower [START_REF] Bevilacqua | Continuous realtime gesture following and recognition[END_REF], another one in the domain of signal processing with the FAUST compiler [START_REF] Orlarey | NEW COMPUTATIONAL PARADIGMS FOR COMPUTER MUSIC, chapter FAUST : an Efficient Functional Approach to DSP Programming[END_REF].

With the gesture follower, specific events are available to design gestural interactions.

The FAUST compiler has been included to provide preprocessing of signals for representation purposes (performance of interaction process representation).

Scripting language

A textual version of the OSC messages that describe a score constitutes the INScore storage format. This textual version has been extended as a scripting language with the inclusion of variables, extended OSC addresses to control external applications, and support for embedded JavaScript sections.

INScore script files can be dropped to score windows to send the enclosed messages. It constitutes an original way of programming since these files may represent a score, but interaction features as well. In a given way, one may think of script files as an extensible commands system.

An open source software

INScore is an open source software that is available from SourceForge 3 . It runs on the main operating systems (GNU Linux, Windows and MacOS).

INScore in Music Creation

Since its first public release, INScore has been used in several musical creations, notably to design interactive music scores.

Calder's Violin

Calder's violin has been composed by Richard Hoadley and was premiered in Cambridge on October 2011. The piece is defined as "automatic music for violin and computer". It involves the live presentation of common practice symbolic music notation created through algorithmically generated material (Figure 6). The notation is then performed by a human musician alongside computergenerated diffused sound or other 'real' musicians.

Technologies used include the SuperCollider audio programming environment and INScore with the OSC protocol used to communicate between them. More details can be found in [START_REF] Hoadley | Calder's violin: Real-time notation and performance through musically expressive algorithms[END_REF].

Alien Lands

"Alien Lands" is the title of a concert given in Montreal in February 2011, entirely dedicated to Sandeep Bhagwati. Three compositions were presented: Alien Lands, Monochrom, and Nil Nisi Nive. All the pieces were using IN-Score for presenting the music score to the performers, with uses that fall in 4 different categories:

• automatic traditional music score: the system is used as a traditional music score but with automatic page turning.

• music score with computed choice: in Divide (a movement of Alien Lands), the performer was instructed to change the order of the measures, lines etc. With the INScore version, the choice was made by the computer and the pages simply presented to the musician. OSC address messages event watch ( )

• complex computed music score: in Sentient (last movement of Alien Lands) and Monochrom 1 & 2, the computer was automatically distributing the score pages. Some elements of the score were generated algorithmically in real-time (like instructions, notes, some elements remained unchanged. A counter was displaying the remaining time. Red and grey coloured boxes at the top of the score were indicating when to play (Figure 7).

• complex interactive score: in Nil Nisi Nive, the performer was controlling the musical elements. The score was generated algorithmically but on performer request, who could ask for a new page at any time.

Miroirs distants

Miroirs distants has been composed by Jean-Baptiste Barrière and was premiered in Lyon and New York, in March 2014, during the Musiques en Scène biennale.

It proposes a special musical situation: two flautists are face to face, look and play straight in the eyes while they are in two remote locations (Lyon and New York for the creation), through what appears to be a large electronic mirror, reflecting both their image and that of the distant performer. The resulting enigmatic image must be deciphered and interpreted: it is in fact a music score (Figure 8). The composition of the image has two functions: one plastic, the other musical. Part of the aesthetic challenge of this project was to mix the two without compromising the quality of both functions.

The image is generated in real-time using a combination of Max/MSP-Jitter and INScore for the symbolic music notation.

Turenas

Turenas has been composed by John Chowning in 1972.

It was one of the first electronic compositions to have the illusion of sounds moving in a 360-degree space. A graphic transcription of the piece has been designed by Laurent Pottier [START_REF] Pottier | Analyse de Turenas -John Chowning[END_REF]), which has been turned into an interactive score with INScore (Figure 9). The score makes use of interactive features: mouse clicks trigger time position messages that are sent to the audio player and internally to the cursor as well. Time events are used to temporary display analytic information over the score, as if living only for a given time span.

Conclusion

Computer technologies for music notation start to be mature enough to support new forms of representations and new artistic forms based on digital scores. Today, many interesting approaches exist, but music notation is mostly used as auxiliary to the composition and writing tools.

Although the way we represent the music is central to the composition process and thus to the musical thought, it is also critical for the performer. Indeed, with the preservation role, the transmission function has been one of the primary orientations of the music notation, which influences the composition process as well.

Today, tools for the performance start to emerge, proposing new approaches to music notation. Although it supports many applications, INScore can be focused to performance issues and proposes a graphic space opened to arbitrary representations (from symbolic notation to pure graphics) and features especially designed to take account of the dynamic aspects of the notation explored by interactive music.

As shown by the presented artistic works, INScore is mature, stable and ready to be involved into music creation. However, it remains an on-going project that takes place in a global research on music notation and representation. Future extensions should enforce the scripting approach in order to develop the system programmability. Migration to mobile platforms (e.g. tablets) and to the Web is also planned, and issues like collaborative approaches to music score design, web performance using shared scores, should be part of future research.

Technological note. INScore makes use of the following technologies:

• The GUIDOEngine 4

• The IRCAM Gesture Follower 5

• The FAUST Compiler 6

• The Qt5 cross-platform application and UI framework 7

• The V8 JavaScript Engine 8

Figure 1 .

 1 Figure 1. A multi-voices score with synchronized cursors. The cursors date is '1/4' and their duration varies from 1 to 5 quarter note. The cursors' width is graphically extended to the corresponding score duration.

•

  Figure2illustrates the components of a graphic signal at a time t.

Figure 2 .

 2 Figure 2. A composite signal at a time t. Any vertical graphic slice is defined at a time t by its y coordinate, a thickness h and a color c.

Figure 3 .

 3 Figure 3. Pitch and articulation combined in a single representation using f0 and rms values in parallel.

Figure 4 .

 4 Figure 4. General format of an INScore message.

Figure 6 .

 6 Figure 6. Calder's Violin -a typical output as displayed to the performer.

Figure 7 .

 7 Figure 7. A complex computed music score as presented to the performer.

Figure 8 .

 8 Figure 8. Aesthetic and musical functions of the score in Miroirs distants.

Figure 9 .

 9 Figure 9. Turenas score, as transcribed by Laurent Pottier and as it appears in INScore. A red cursor indicates the current position in the audio recording of the piece. The whole piece modelling makes use of Max/MSP or Pure Data to play the audio file and to send clock messages to INScore, in order to move a cursor to the corresponding time location.The score implementation supports multiple views, which can be switched by simple drag & drop of INScore scripts, even in real-time. Figure10displays a structural representation of the piece, presented on a single line. In this case, the master/slave synchronisation scheme is inverted: master object is the cursor and the graphic representation is slave of the cursor, which has the effect to scroll the score when the cursor moves in time.

Figure 10 .

 10 Figure 10. A structural view of Turenas. The score is presented as a single line. The bottom view gives the whole score overview and can be used to move the current reading position. The blue cursors indicate the same time position in both views.

Proceedings of Korean Electro-Acoustic Music Society's 2014 Annual Conference (KEAMSAC2014)

Proceedings of Korean Electro-Acoustic Music Society's 2014 Annual Conference (KEAMSAC2014)

Acknowledgments

. This work has been made possible by grants from the French National Research Agency (ANR) INEDIT [ANR-12-CORD-0009] and INTERLUDE [ANR-08-CORD-010] projects.


Matias Ezequiel

Vara Larsen
email: varalars@i3s.unice.fr

Frédéric Julien Deantoni
email: julien.deantoni@polytech.unice.fr

Frédéric Mallet
email: frederic.mallet@unice.fr

Framework for Heterogeneous Modeling and Composition

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Framework for Heterogeneous Modeling and

Composition *

Introduction

The development of embedded and cyber-physical systems is becoming more and more complex. Such systems involve the interactions of very different subsystems, e.g., DSP, Sensors, Analog-Digital devices. The requirements of each subsystem are usually captured by using a Domain Specific Modeling Language (DSML). An heterogeneous system is then specified using several DSMLs. To simulate and verify heterogeneous systems, models conforming to different DSMLs have to be coordinated.

Existing approaches [START_REF] Eker | Taming heterogeneity -the Ptolemy approach[END_REF][START_REF] Boulanger | Simulation of Multi-Formalism Models with ModHel'X[END_REF] allow the user to hierarchically coordinate heterogeneous models. These approaches provide a unique abstract syntax and the behavioral semantics is brought by using a specific Model of Computation (MoC). For instance, in [START_REF] Girault | Hierarchical finite state machines with multiple concurrency models[END_REF], Ptolemy [START_REF] Eker | Taming heterogeneity -the Ptolemy approach[END_REF] has been used to embed hierarchical finite state machines within different concurrent models of computations. Although these approaches are good for experimenting, they hide the coordination semantics in the tool. This limits the study and the tuning of the coordination. More importantly, they do not provide any specific language to help in the specification of the coordination.

Differently, coordination languages [START_REF] Papadopoulos | Coordination models and languages[END_REF] allow the user to manually specify the coordination between models. Thus, the user can explicitly define the coordination between two models written in different languages. There are two kinds of coordination languages: endogenous and exogenous. Endogenous languages provide primitives that must be incorporated within a model for its coordination. In exogenous languages such primitives are without a model. In any case, since the coordination depends on the coordinated models, it has to be redone for each pair of models.

There is a need to capitalize the specification of the coordination between models at the language level by defining a language dedicated to the composition of language behavior. Current structural composition languages [START_REF] Fleurey | Models in software engineering. chapter A Generic Approach for Automatic Model Composition[END_REF][START_REF] Kolovos | Merging models with the epsilon merging language (eml[END_REF] allow the user to specify composition rules at the language level. These rules are then applied between models. Such composition rules rely on two major steps: matching and merging. The matching decides what elements of the syntax are selected and when (i.e., under which condition) the merging is applied. The merging specifies how the selected elements are combined. Similarly, a behavioral composition language relies in a matching step that selects behavioral elements of models. However, the merging step is replaced by a coordination step that specifies how the selected elements must be coordinated.

We propose the Behavioral Composition Operator Language (B-COoL) to specify the behavioral composition of languages. B-COoL allows the user to define operators that define composition rules between elements of the semantics of one or more language(s). This specification can be used to automatically generate the coordination between two (or more) models. Currently, B-COoL relies on a behavioral interface of the languages in terms of Events as proposed in [START_REF] Benoit Combemale | Reifying Concurrency for Executable Metamodeling[END_REF]. First experiments of coordination have been based on the use of CCSL as illustrated by the coordination of a Finite State Machine and a fUML model [START_REF] Vara | Railroad Crossing Heterogeneous Model[END_REF].

* This work is partially supported by the ANR INS Project GEMOC (ANR-12-INSE-0011), and the CNRS PICS Project MBSAR.