Quentin Galvane
email: quentin.galvane@inria.fr

I-Sheng Lin

Fernando Argelaguet
email: fernando.argelaguet@inria.fr

Tsai-Yen Li
email: li@nccu.edu.tw

Marc Christie
email: marc.christie@irisa.fr

VR as a Content Creation Tool for Movie Previsualisation

Keywords:

Figure 1: We propose a novel VR authoring tool dedicated to sketching movies in 3D before shooting them (a phase known as previsualisation). Our tool covers the main stages in film-preproduction: crafting storyboards (left), creating 3D animations (center images), and preparing technical aspects of the shooting (right).

INTRODUCTION

Film and animation industries rely on a range of techniques to prepare and prototype movies before shooting them. Hand-drawn storyboards or digital storyboards [START_REF] Toonboom | [END_REF] remain one of the most traditional ways to design the visual and narrative dimensions of a film sequence.

With the advent of realistic real-time rendering techniques, the film and animation industries have been extending storyboards by exploring the use of 3D virtual environments to prototype movies, a technique termed previsualisation (or previs). Previsualisation consists in creating a rough 3D mockup of the scene, laying out the elements, staging the characters, placing the cameras, and creating a very early edit of the sequence. As an addition to the storyboard, the previsualisation stage provides a mean to creatively explore multiple framing or editing choices, and to confirm or confront initial design ideas from early stages with layout, shooting and editing constraints. Most large budget movies nowadays rely on previsualisation technologies as a way to reduce the costs by early anticipation of issues that may occur during the shooting.

Previsualisations of movies or animated sequences are actually crafted by dedicated companies composed of 3D artists, using traditional modelling/animation tools such as Maya, 3DS or Motion Builder. Creatives in film crews (editors, cameramen, directors of photography) however do not master these tools and hence can only intervene by interacting with the 3D artists. While a few tools such as FrameForge3D, MovieStorm or ShotPro [START_REF]FrameForge[END_REF][START_REF]MovieStorm[END_REF][START_REF] Shotpro | [END_REF] or research prototypes have been proposed to ease the creative process [START_REF] Lino | The directors lens: an intelligent assistant for virtual cinematography[END_REF][START_REF] Nguyen | Vremiere: Inheadset virtual reality video editing[END_REF][START_REF] Shin | Ar storyboard: an augmented reality based interactive storyboard authoring tool[END_REF], most tools remain complex to master, outsourcing the control of the content, and hampering the creativity of film crews. In such a context, we propose to fill a gap by designing accessible and dedicated content creation tools at the storyboard, previsualisation, and technical stages so that part of the creative power can be placed back in the hands of film creatives, and contents can be co-created and co-reviewed by creatives and 3D artists.

The founding idea in our approach is to rely on VR interactive technologies to author these animated 3D contents, building on the immersion offered by VR together with the capacity to manipulate contents using 6D input devices.

In this paper we present how we addressed specific issues in previsualisation, and report feedback from expert users. Our approach extends techniques such as the Directors Lens [START_REF] Lino | The directors lens: an intelligent assistant for virtual cinematography[END_REF] and [START_REF] Leite | Virtual marionette[END_REF] to not only manipulate cameras, but also manipulate scene layouts, 3D character poses and animations, as well as complex camera rigs (cranes) or light setups. Through experiments that involve expert users only, we demonstrate the strengths of this approach, namely (i) the capacity to be easily integrated into existing pipelines, (ii) the improved creativity of users, and (iii) the capacity to seamlessly navigate between all the different stages. To the best of our knowledge, this paper represents the first VR tool the covers all the previsualisation previsualisation stages and received very strong interest from film schools and experts. Leading animation (Pixar, BlueSky studios, Disney) and VFX companies (MPC, ILMx, Halon) have just started to explore the possibilities of VR creation in their pipelines, and discussions with these companies during FMX and Siggraph 2018 showed the clear advance our approach has.

In the following sections, we will report progress in authoring tools for the film industry and will also focus on VR/AR as content authoring tools (Section 2). We will then rely on discussions with experts to derive design considerations (Section 3), before describing our approach (Section 4). Experiments are then reported in Section 5 before drawing conclusions (Section 6).

RELATED WORK

We first explore the use of VR as a mean to create contents interactively rather than as a way to interactively experience contents. From there, we report and comment on tools dedicated to storyboarding and previsualisation before discussing the range of techniques that rely on VR and AR technologies for the design of cinematographic contents.

Virtual Reality as a Content Creation Tool

A number of drawing and authoring tools rely on VR technologies to create artistic contents such as TiltBrush [START_REF] Tiltbrush | [END_REF] or Occlus Story Studio [START_REF]Occulus[END_REF]. Tracked controllers are used as brushes to interact and paint in space. Despite their obvious benefit in creating 3D contents, such tools are not suited to the design of storyboards (as reported in [START_REF] Henrikson | Multi-device storyboards for cinematic narratives in vr[END_REF]), nor to the design of previsualisations (due to the amount of work required to redraw characters in different poses throughout the sequence). Given the ease of use in VR manipulations, VR interfaces have been explored for general sketching tasks [START_REF] Hagbi | In-place sketching for augmented reality games[END_REF].

Tools for Storyboarding and Previsualisation

Multiple digital storyboarding tools are used by the film and animation industries (too many to actually provide an exhaustive list). Essentially these tools support artists in creating and editing drawn sketches. For example, Henrikson et al. [START_REF] Henrikson | Storeoboard: Sketching stereoscopic storyboards[END_REF] proposed a system to design stereo storyboarding. However, tools nowadays stretch towards more animated contents (see ToonBoom [START_REF] Toonboom | [END_REF]) and start to integrate 3D scene layouts and camera placements. While intuitive for artists, these tools remain complex for non-specialists, and interaction controls of 3D contents are very similar to those found in most 3D modellers. Systems have also been proposed to rehearse theater plays using virtual environments [START_REF] Wu | A real-time performance system for virtual theater[END_REF].

In reaction to the complexity of 3D modellers and to address the specific needs of the film industry, a collection of high-level design tools have been proposed to create previsualisations ( [START_REF]FrameForge[END_REF][START_REF]MovieStorm[END_REF][START_REF] Shotpro | [END_REF]). While the interfaces have been simplified and specialized, the control of 3D contents (mainly positioning and orienting them) remains a task that requires to switch between different interactors, and switch between camera views to control the spatial relations between entities (nonimmersive systems).

Researchers have also been exploring the use of physical mockups through miniatures [START_REF] Horiuchi | Virtual stage linked with a physical miniature stage to support multiple users in planning theatrical productions[END_REF] -a technique used in the film industry to control complex camera rigs. In contrast, by proposing a VR immersion and tracked controllers, our system provides a straightforward control of entities (objects, characters or cameras).

AR and VR in storyboarding and previsualisation

Within the specific space of design tools to support the creation of cinematographic contents, a number of approaches have been explored. Such approaches can be characterized according to the stage at which they intervene in the production workflow. Storyboarding. The use of VR in storyboarding has mostly been explored with the intent of creating narratives for VR. Shin et al. [START_REF] Shin | Ar storyboard: an augmented reality based interactive storyboard authoring tool[END_REF] proposed an AR system to author contents, mixing real video footage and 3D generated footage. Henrikson et al. [START_REF] Henrikson | Multi-device storyboards for cinematic narratives in vr[END_REF] propose an approach to the creation of VR narratives by mixing traditional devices such as tablets to draw storyboards, with VR to experience and validate narratives. While in essence, the approach provides a means for a closer collaboration between drawing artists and filmmakers, the expertise of professional artists is still necessary. In opposition, our approach consists of designing tools that do not require 3D expertise. Scene layout. Well known tools such as the VR Editors of Unreal and Unity3D propose a range of effective interactors to handle the creation and manipulation of 3D environments (designing, transforming and positioning entities in 3D). To this end, the editors rely on classical interaction metaphors for 3D manipulation [START_REF] Bowman | 3D User Interfaces: Theory and Practice[END_REF]. While we build on these classical interaction metaphors to manipulate the elements composing the scene, we extend the approach by proposing character manipulation using inverse kinematics, camera manipulation and editing. Camera framing in virtual environments. As a mean to regain control over the creative process, a number of filmmakers have pushed the idea of Virtual Camera Systems, i.e. devices tracked in position and orientation that enable the control of a virtual camera in a virtual 3D environment. Such systems provide the filmmakers with means to creatively explore camera angles, scene layouts and character layouts [START_REF]Five steps to avatar: reinventing moviemaking[END_REF]. The use of virtual camera systems has been later extended with the possibility to navigate between a collection of automatically computed views on the scene and create an edit of a filmic sequence. Our approach relies on these ideas, but extend them by using VR immersion and tracked controllers to address a wider range of stages (creating a storyboard, laying out the scene, and controlling character poses).

Researchers have also been exploring questions of editing in VR [START_REF] Nguyen | Vremiere: Inheadset virtual reality video editing[END_REF] for 360 videos. Mixed reality technologies were exploited in the specific case of previsualisation. Ichikari et al. [START_REF] Ichikari | Mixed reality pre-visualization and camera-work authoring in filmmaking[END_REF] designed an AR system to author camera placement in a mixed reality context, by adding virtual contents to real footage. The system relied on a fixed camera with rotational encoders (pan and tilt) which fed rotational information to synchronize the real and virtual camera motions. The system, however, remained limited to the specific task of exploring camera framing and staging virtual contents in a real background.

Later, technologies referred to as on-set previsualisation, enabled the real-time rendering of virtual contents in synchronization with real contents to anticipate the effect of visual effects in scene rehearsal [START_REF] De Goussencourt | A game engine as a generic platform for real-time previz-on-set in cinema visual effects[END_REF]. Typically, by coupling camera tracking techniques with chroma keys, green screens can seamlessly be replaced by virtual contents. Camera tracking technologies rely on visual odometry or single localization and mapping techniques [START_REF] Mouragnon | Real time localization and 3d reconstruction[END_REF], or visual tracking of static targets [START_REF] Dwarakanath | Robustness of 3d point positions to camera baselines in markerless ar systems[END_REF]. These approaches are more dedicated to explore and validate camera angles rather than interact with the scene's virtual contents. Recently the previsualisation tool ShotPro has proposed an augmented reality mode that enables the creation of virtual scene layouts within a real environment, while also using visual odometry to creation camera angles and shots.

Closer to our work, [START_REF] Muender | Empowering creative people: Virtual reality for previsualization[END_REF] and [START_REF] Galvane | Immersive previz: Vr authoring for film previsualisation[END_REF] started to run exploratory experiments to evaluate how both novices and non-technical professionals from the film, animation, and theater domain assess the use of VR for previs. Early results show very positive feedback that encouraged us to push this research topic further in this paper.

Digital technologies have largely impacted the process of creating real or animated movies. Ubiquitous in the later production stages (e.g. for visual effects and compositing), digital technologies are now gaining importance in the pre-production stages, i.e. in preparing the production. Typically, previsualisation is evolving into what will eventually become a mandatory stage in the authoring of movies. Currently required by producers for preparing high budget movies, previsualisation is more and more used to rehearse complex scenes in TV shows and medium budget productions.

In order to have an in-depth understanding of the authoring process in storyboarding and previsualisation, and what would be the requirements for novel tools, we organized informal discussions with two film practitioners who are also teachers in a leading film and animation school, two technical directors and two artists from previsualisation companies. The process is the following. The director first analyses the movie script1 to create a technical breakdown of a sequence. The technical breakdown identifies which type of scene, camera and character layouts will best translate the script intentions. The technical breakdown is then converted into a storyboard, a process by which artists in relation with the film director create a visual representation of the script through drawings or digital tools [START_REF] Toonboom | [END_REF]. The director iterates on the storyboard with artists, and on completion, a detailed version of the storyboard (called clean-up) serves to communicate the directors' intentions to the film crew.

The previsualisation stage then consists in working from the storyboard representations to create a 3D mock-up version of the scene with decors (background), layouts (how the different characters are placed in relation with the scene) and cinematography (deciding on camera angles and camera motions). Previsualisation also helps to create a first edit of the sequence, determining its rhythm and the continuity between the cameras. Finally this previsualisation stage (that is essentially focused on the narrative and aesthetic design of the sequence) is generally followed by a technical stage (called techvis), in which technical information is added, to communicate with the film crew (camera lenses, spatial organisation organisation of the scene, types of camera rigs and light sources). Nowadays, previsualisation is performed through general purpose 3D modelling tools (Maya, MotionBuilder, 3DS) or dedicated tools such as Moviestorm [START_REF]MovieStorm[END_REF], ShotPro [START_REF] Shotpro | [END_REF] or iClone [START_REF]iClone[END_REF]. In all cases, 3D artists are in charge of these stages, working from the storyboard and through iterations with the film crew.

Design considerations

A central issue reported by the film directors we interviewed is that these technologies make them lose control of the created contents at the early and important design stages, which are the most creative ones. All operations are performed by artists using elaborate sketching tools or 3D modellers most of which are not accessible to film directors. The iterative process through which the design is performed through back and forth with artists also limits the creativity [START_REF] Henrikson | Multi-device storyboards for cinematic narratives in vr[END_REF].

From these discussions we derived a number of design considerations to drive our research:

Intuitive: the proposed tool should be easy to use by previs artists as well as members of the film crew (even for those who have limited knowledge of modelling/animation tools). Hence, the scenes can be accessed and modified at any stage by members of the film crew or previs artists. Clearly, a director would not create his whole movie using such a tool, but he should be able to intervene on any sequence, review the proposed solutions, explore and propose alternatives.

Rapid: the tool should enable the rapid creation and review of cinematic sequences, the exploration of multiple possibilities in a short time here being the key of creativity. Seamless: the tool should merge the different stages that are the storyboard, the previsualisation, and the technical preparation of the shooting. Offering the possibility to seamlessly navigate between these stages that are traditionally organized in sequence, and generally involve different digital tools, would offer a mean to easily revise some early choice and explore multiple possibilities. In particular, technical constraints from the shooting stage (eg type of camera rig used) influences strongly the aesthetic choices made in the previsualisation stage. Hence encompassing these constraints early in the process presents a clear benefit. Workflow-compatible: while all of the interviewed people were positive when discussing the idea of a VR tool to address these stages, they insisted in the capacity to be compatible with main tools in the industry, so that at any stage the created contents could be shared between the VR tool and Motion Builder.

DESIGNING AN INTUITIVE VR PREVISUALIZATION TOOL

We designed the tool around three main components, namely (i) the storyboarding mode, (ii) the previs mode (animating character, object and cameras), and (iii) the techvis mode (placing and animating complex camera cranes and light setups). All the technical solutions proposed in our approach to address these aspects rely on traditional 6DOF manipulation techniques, but they integrate constraints specific to the tasks, to guide the user in their interactions. This paper does not innovate in the way interactions are performed, but in how practical issues from a specific domain can be easily addressed using VR technologies through constrained interaction, and how these can impact the users ease of use and creativity.

All three modes are displayed in Figure 2.

The Storyboarding Mode

Storyboarding mode provides a mean to lay out a static scene, take static shots (thumbnail stage), and annotate the shots with elements of the storyboard visual grammar defined the intended motion of the camera (traveling, panoramic, zoom, etc. -see Figure 6) to create a content that can be shared with the film crew (clean-up stage).

Laying out the scene

The first step consists in defining the location unit -i.e. indoor, outoor, city, countryside. The availability of 3D models through asset stores provide elements that may match the storyboard, yet at this stage, storyboarders generally work with grey-box environments, as a number of choices have not been fully confirmed. Different props used in the scenes can be found on the store (this exploration stage actually doesn't require a precise contents).

Users can interact with the 3D environment and our system using two spatially located VR input devices (Vive controllers in this case). We rely on classical selection, manipulation, navigation and application control techniques from the literature [START_REF] Bowman | 3D User Interfaces: Theory and Practice[END_REF]. The right controller is dedicated to scene editing. Any object in the scene can be selected and displaced by placing the right controller inside it, pressing the trigger and dragging it in position and orientation -such classical interactors ensure quick assimilation of the system. Figure 3 shows examples of two user interactions, a raycasting metaphor to place objects newly created, and a virtual hand metaphor to enable object manipulation. The left controller is dedicated to menus and navigation. The user displacement is easily performed through a ray-casting technique, displaying a ray between the left controller and the intersection point with the scene; teleportation is performed when pressing the left trigger. The user can also fly through the environment using the controller to guide the movement; the motion starts when the user presses a dedicated button and stops when releasing the button. This last feature enforces short acceleration and deceleration period to avoid motion sickness. Finally, as illustrated in Figure 4, interactions with the menus can be performed in two ways: (i) using physical buttons on the controllers or (ii) using a point and click metaphor with a laser directly on UI elements in the scene. Menus are used to navigate between modes (i.e. storyboard, previsualisation and techvis) and give access to the specific features of each mode (e.g. add or delete an object, control time, add camera keypoints, etc.)

Controlling the pose of virtual characters

Controlling a character pose (i.e. manipulating joints in a skeleton) is a complex task when using classical modelling tools. Translation and rotation visual widgets are required to assist the user, and specify in which dimension the manipulation occurs and on which axis, due to depth ambiguity. Dedicated non-immersive tools [START_REF]MovieStorm[END_REF][START_REF] Shotpro | [END_REF] provide a set of predefined poses (e.g. standing, sitting, running, etc.), but provide limited means to manipulated poses. This paper proposes a constrained manipulation metaphor to deal with character poses in VR. No visual widgets are required as 6D manipulations are straightforward with 6D controllers and stereo vision resolves depth cues. The nodes of the skeleton can be selected as any other object in the scene. User manipulations are however designed to be constrained by the kinematic chain between the selected node and the root node of the character. Figure 5 shows some example of poses crafted with our tool.

A classical inverse kinematics solver [START_REF] Baerlocher | An inverse kinematics architecture enforcing an arbitrary number of strict priority levels[END_REF] (i) constrains the maximum displacement of node given the distance and angular constraints expressed on the skeleton, and (ii) updates the positions and orientations of the nodes between the selected node and the root skeleton. Camera placement Using 6DoF tracked devices on hands to handle the placement of cameras presents a tremendous advantage compared to traditional tools. While two-dimensional interfaces require to mix 2DoF manipulators coupled with keyboard combinations to control 6 DoF, VR environments offer a simultaneous and natural control of all six DoFs. Once created, a virtual camera can be grabbed like any object in the scene. A preview screen and a slider to adjust the focal length are placed on top of the cameras and are always oriented towards the user as a billboard. Additional features were added from discussions with experts such (i) the possibility to detach and move the preview screen around so as to view the shot from another location, (ii) the possibility to easily switch between the previews of different cameras.

Creating a storyboard

The storyboarder tool relies on (i) a camera manipulator to create snapshots of the scene (ii) a camera motion annotation to specify the primitive motion followed by the camera, and (iii) a monitor used to preview and organize the shots. Figure 6 shows an overview of this Storyboarder. A 3D model of a camera is used to control the viewpoint; it can be handled like any other object in the scene. The user can preview and modify in real-time the current camera view (including the focal length). The type of camera motion associated with a shot is specified through an overlay on the camera rendering. It follows storyboard visual conventions taken from the literature [START_REF] Hart | The Art of the Storyboard: A Filmmaker's Introduction[END_REF]. Each type of camera motion is described with a specific representation and displayed on a side panel. Figure 7 illustrates some of these representations.

Snapshots can be added, moved or deleted in the Storyboarder monitor. Finally, the Storyboarder compiles the set of snapshots together to generate a full storyboard (see Figure 8). The video provided in supplementary materials demonstrate this storyboarding mode.

The Previsualisation Mode

We propose a set of features (cf Figure 9) to assist the user in animating the static scenes designed in the storyboarding mode.

Camera motions If a storyboard was created with our tool, the user can access it and reuse the cameras placed with the Storyboarder as starting points. To create camera motions, the user has the possibility of (i) creating camera rails, (ii) recording camera motions by tracking the VR controllers over time, and (iii) editing the trajectories created in the first two modes. A camera rail can be designed by placing a number of control points on a spline curve representation and manipulating the positions of the control points as well as the tangents to the curve. Tangents are represented in a way that the expected interaction is straightforward leading an easy design of curve bends. The recording of tracked camera motions is a feature traditionally found in Virtual Camera Systems (see Section 2) and improved by the possibility to observe the surrounding while recording the camera motion (e.g. to anticipate character motions). The user can insert and delete keyframes in the motion, and manipulate these keyframes.

In all modes, the camera orientation can be constrained to a given target, maintaining the position of an actor on the screen while the camera is manipulated. Character animations Animations are authored on characters in the same way as cameras with the possibility to (i) create keyframed animations, (ii) record animations using motion capture suits (iii) record the motions of the controller (eg when manipulating an arm) and (iv) edit the motions once created whatever way they were created. In addition, the manipulations can be constrained by the kinematic chain when manipulating parts of the characters only.

An additional panel, attached to each object being animated, displays the list of keyframes to ease their selection and manipulation.

Editing Finally to complete our VR previzualization tool we propose a new editing method dedicated to VR.

Standard editing tools are based on the 3-point editing principle. This method consists in inserting a portion of a camera rush, delimited by in and out timestamps (first two points), within the final edit at a specific time (third point). The method allows great flexibility such as the capacity to create flashbacks or any other type of non-linear editing. However, user interface associated with this principle is relatively complex and requires the manipulation of a wide range of precise 1DoF and 2DoF controllers mainly tailored for a precise interaction device such as a mouse. Early experiments quickly showed that importing such complex UI in the virtual world and handle interactions with a laser pointer is both counter-intuitive and counter-productive (e.g. limited precision in controlling a time sliders).

Thus, we proposed a simplified linear editing tool. While linear editing does not provide as much freedom as 3-point editing, it is sufficient to craft rough edits of a scene. Furthermore, the editing generated in the VR interface can be exported and later refined with traditional 2D editing tools.

The editing interface was crafted following these interaction constraints (cf. Figure 10). It contains a timeline to display a color-coded representation of the edited shots. The main screen displays the final edit and the two bottom screen show the initial frame of the current and next shot. As the user creates a new cut, a mosaic displays all the camera angles designed by the user. The user has the possibility to interactively change the camera angle, change the scene layout or character pose directly while viewing the results on the editing interface, seamlessly navigating between stages that are traditionally split in different tools.

The Technical Mode

At last the technical mode proposes means to control technical shooting devices such as complex camera rigs and light rigs, that help in preparing the real shooting of the scene. Control of camera rigs A camera rig (eg a camera crane) can be abstracted as a kinematic chain defined between the end effector (the camera manipulated by the user), and a root node of the camera rig, typically the position at which the camera rig is in contact with the floor. Our VR implementation considers camera rigs as any other animated entity in the scene, with the additional constraints due to the kinematic chain (limits of articular joints, limits of extensible sections of the chain). Camera rigs can be animated using keyframes. The system supports any specified camera rig such as Technodolly Cranes -arguably the most complex camera rig in terms of kinematic chain as illustrated in Figure 11.

Control of light rigs

A light rig is a structure that holds a light in a studio setup. Our VR implementation proposes the interactive manipulation of area lights through changes in position, orientation, aperture and intensity. Area lights are based on the implementation of Heitz etal. [START_REF] Heitz | Real-time polygonallight shading with linearly transformed cosines[END_REF], and provide a rendering closer to real light staging conditions. An example is displayed in Figure 12.

EVALUATION

This section describes two experiments, which aim at assessing the usefulness and ease of use of our tool. Our first experiment is guided by a task: users had to create an animated sequence from a given paper storyboard representation, hence reproducing realistic use case. The second experiment is not guided, so as to let users creatively design contents in all three stages (storyboarding, previsualisation and technical preparation).

Experiment 1: Previsualisation from a Storyboard

Thus, the first experiment aimed at evaluating how our tool can be used for reproducing a given storyboard with the goal of generating a short video (only camera and editing operations). The evaluation considers that the users do not have to modify the virtual environment nor the virtual actors.

Apparatus and Participants

Participants were immersed in the virtual environment using an HMD (HTC Vive), interaction capabilities were provided using two HTC Vive controllers. The system was developed in Unity and ensured a constant FPS rate. Ten volunteers (all male, age M = 22.4; SD = 2.9), six from a local film school and four from our lab, took part in the experiment. Half of the participants considered themselves experts in cinematography, film editing, and another half novice in terms of 3D modeling and in the use of VR devices (not exclusive).

camera effector

Figure 11: Our system allows the interactive and key-framed manipulation of complex camera rigs (here a Technodolly rig) defined through a list of constrained degrees of freedom (in red in the top image). The camera effector (in blue) is directly controlled by the user.

Experimental Protocol and Design

Upon their arrival, participants were asked to sign a consent form and fill a pre-experiment questionnaire to gather relevant demographic information for the study (age, sex, previous experience with VR devices, level of expertise in movie making or 3D animation, etc.). At first, the experimenter provided a short demonstration of the tool and its different features. Then, participants were equipped with the head-mounted display and immersed in a training virtual environment. During the training session, which lasted 15 minutes, participants were guided through a short tutorial and were allowed to ask questions to the experimenter. After the training, participants were immersed in a new virtual environment. The virtual environment used in the experiment is the 3D animated scene from a public dataset obtained from [START_REF] Galvane | Continuity Editing for 3D Animation[END_REF].

The main experimental task was to reproduce a given storyboard (see Figure 13), which was always accessible through a 3D panel. Participants were asked to produce a video sequence that best matches the storyboard composing of 12 distinct shots with no time limit. The task required participants to place a set of cameras and edit the entire sequence (defining the duration of the shots for each camera). Two shots also required pan camera motions. The entire editing process was done using our tool and the final result was a short film of 79 seconds. At the end of the experiment, they were asked to fill the NASA-TLX questionnaire in order to measure the effort required to use the tool and provide subjective comments regarding the ease of use and the usefulness of the system. In total, the entire experiment (including the training) lasted one hour.

We could not provide any comparison with other VR editing tools as at the moment, only our tool provides a complete film crafting and editing pipeline in VR.

Results and Discussion

All participants except one were able to finish the task in less than 30 minutes, one novice participant required one hour to finish the task. The accompanying supplementary material 2 provides more examples of the results obtained by participants.

All participants agreed that the learning curve of the tool was fast and would require just a few days to perfectly master the tool. This feeling is supported by the results of the NASA-TLX which showed a very low level of frustration (Frustration Level: Median = 2; IQR = 1) and a low level of required effort (Effort: Median = 3.5; IQR = 2). These results show that our tool could be integrated into a film making pipeline without a strong impact in terms of training, i.e. ensuring a good ease of use.

Another key aspect is whether the tool enables to efficiently perform the required task. The results from the NASA-TLX showed 2 https://cinematography.inria.fr/vr-previsualisation/ that participants achieved a strong level of achievement (Overall Performance: Median = 8; IQR = 1). This result can be interpreted as that users were pleased with the final obtained result. When analyzing the comments from expert users, there is a unanimous and strongly positive feeling for the tool, "It allows to create an animation sequence really fast!", "It can be useful to pre-visualize complex scenes" or It is extremely useful in terms of camera placement and to obtain a global view of the final editing." The 3D animators and film school students said they would love to use the tool for their projects whether it is for storyboarding, for previsualization or for creating a first draft to be refined with a more advanced and dedicated tool (i.e. Blender, Maya, ...). Yet, the comments from the expert users shows that the obtained result might still not be enough for professional film-making editing (which was not the goal), but they could integrate the tool in project/animation films, which shows the maturity of the tool. For example, one participant would have liked to have additional edition tools "The editing features are less developed, it would be more useful to have a 3-point editing tool", and another participant complained about the camera jitter ".. the camera could be more stable" due the high tracking quality and tracking rates of the HTC Vive device. Again, the obtained results could be exported to professional 3D modeling and film editing tools for a higher quality result.

Regarding how demanding it was using the system, NASA-TLX ratings shows that the mental demand was relatively low (Median = 3; IQR = 1.75), the physical demand was low (Median = 2; IQR = 1.75) and the temporal demand was relatively low (Median = 3.5; IQR = 2). These results provide a first assessment of the comfort while using the tool, which does not show any particular issue. Finally, regarding simulation sickness, although we did not directly measure it, participants did not show any particular symptom.

As a side note, one interesting observation is that novice and expert users followed a different workflow. Experts performed the tasks sequentially, first setting up all cameras and then performing the editing. In contrast, novice users constantly switched between editing and camera placement mode, i.e. placing a camera, then adding it to the edit and moving on to the next shot. This shows the potential need to provide different adaptations of the user interface to better match user workflows.

Experiment 2: Creative uses

Since our purpose is not to replace existing modelling and animation tools but to offer a complementary approach, it was not relevant to perform direct task-to-task comparisons with criteria such as time, precision or quality. To complete the first experiment, we rather proposed to rely on aspects such as acceptance through the TAM's construct: Technology Acceptance Model, a direct, pragmatic instrument to measure a given technology's degree of acceptance [START_REF] Morris | How user perceptions influence software use[END_REF], that helped us to perceive the user's satisfaction on an approach very novel for the practitioners.

We focused on the three TAM constructs:

• Perceived Usefulness (PU) -the extent to which the user expects the system to improve his job performance within an organizational setting;

• Perceived Ease of Use (PEU) -the degree to which the user believes the system's use will be free of effort

• Attitude toward Using (AU) -the user's desire to use or favorableness feelings towards using the system

Apparatus and Participants

We followed the same apparatus as the first experiment. Only the participants changed. In this experiment, we worked only with expert users, with at least five years of experience each in storyboarding, previsualisation and 3D animation for the movie industry, or technical preparation of movie shooting. None of the experts in this session had participated in the previous experiment. Three experts were recruited. All of them are also teachers in film and animation schools.

Experimental Protocol and Design

Upon their arrival, participants followed the same protocol (questionnaire, consent form, demographic information). Each expert was assigned to test one specific tool: the storyboard artist tested the storyboarding tool, the animator tested the previsualisation tool and the techvis artist worked with the technical tool. At first, the experimenter provided a short demonstration of the tool and its different features. Then, participants were equipped with the head-mounted display and immersed in a training virtual environment. During the training session, which lasted 15 minutes, participants were guided through a short tutorial and were allowed to ask questions to the experimenter. After the training, participants were immersed in a new virtual environment. The virtual environment used in the experiment is a basic reconstruction of a studio setup, with different types of camera rigs, light rigs and characters to animate. During this stage of the experiment, there were no guidelines as to what to create; users had complete freedom to explore the features of the tools, yet they each had to design a simple sequence. At the end of the experiment, they were asked to fill a questionnaire with limited choices and open answers to evaluate the ease of use and usefulness of the tool. In total, the entire experiment (including the training) lasted one hour per users.

Results and Discussion

All users answered the same questionnaire. For the ease of comprehension, we will name them respectively the Storyboard expert, the Animation expert, and the Techvis expert. First, all three experts were satisfied by the results they designed -they evaluated their satisfaction at 8/10 for the storyboarder and 7/10 for the other two. Second, in evaluating the possible gain in terms of time, users were positive with feedback such as (Animation expert) "the tool would enable the faster design of pitchvis previs and techviz sequences" (pitchvis is the task of creating an animated sequence of the movie to convince producers to invest in the movie production).

Users were more dubious on the gain on the estimation quality of the generated contents, commenting that the tool would need an improved UI to better guide the users in their tasks. It was however highlighted that for the purpose of pitchvis and previs, such quality might be sufficient. On the Perceived Usefulness (PU) regarding the storyboarding, the Storyboard expert commented that while the tool would not provide him as much flexibility as a simple drawing, it could prove extremely useful for directors, producers and anyone involved in the creation process that are not used to sketch ideas on paper -a vast majority of them apparently.

The Perceived Ease of Use (PEU) was overall quite good (evaluated through questions such as "would it be easy to use the tool in current productions"), with the limit that more input/output geometry should be implemented to ensure a good degree of interoperability between 3D tools (exporting cameras only, exporting light sets, or Edit Decision Lists, etc). The experts estimated it would only take them between a few hours to a couple of days to master the tool.

The Storyboard expert commented that intentions and emotions were definitely more easily conveyed by drawings, but that this tool could help in preparing actions or dialogue sequences, where setups may be complex and suffer many iterations. One of the benefits listed by the Animation expert was that "the tool provided a better sense of space" than traditional 3D animation packages, hence giving more time to focus on the creative tasks than the manipulation of the camera perspectives.

On the Attitude towards Using (AU), the overall feedback of these experiments was positive, and all the experts agreed on the potential of such system for production purposes and were ready to use it in their production pipelines if technical aspects such as data interoperability were addressed, as well as more precise control of the animations. As teachers, they also appreciated its potential as an educational tool.

DISCUSSION AND CONCLUSION

With the growing maturity of VR technology, the presence of VR will grow in the professional market. This paper brings for the first time the use of VR for film storyboarding, previsualization and technical preparation by providing a tool that eases content creation for the different stages of film pre-production. The main contributions of the paper are the delimitation of the design space for VR storyboard tools, the iterative conception of the system and its evaluation. The interviews with experts have shown the requirement of letting creatives to keep control of the storyboarding process through simple tools. Thus, we have designed the system trading off expressiveness and control. For example, the editing step is simplified to enable fast prototyping at the expense of precise control. We would like to stress that the final result generated by the system could be further refined using other specific tools. Moreover, the tool could be used for content generation in videogames or VR applications. For example, it can be used to easily generate replays or "film" VR experiences.

Results demonstrate the benefits of the tool in rapidly prototyping filmic sequences for novice and professional users. Participants were able to achieve satisfying performances with just a few minutes of training and the results were overwhelming and extremely positive. All expert users were willing to use future versions of the systems and integrate the tool into their projects. In addition, the proposed technique demonstrates the ease with which tasks that are complex to perform with 3D modelling tools are made simpler by providing a seamless integration between different pre-production stages, and high-level interactions that take advantage of VR and 6DoF manipulation devices.

Finally, the user evaluation also helped to identify a number of limitations, such as the lack of stabilization for handheld camera motions and the lack of more complex film editing techniques. Future work will focus on the design and evaluation of more specific interaction techniques to provide higher expressiveness and control.

Figure 2 :

 2 Figure 2: The three stages in the VR tool with the features developed in each stage.

Figure 3 :Figure 4 :

 34 Figure 3: Users can quickly craft a scene layout by (a) adding and (b) moving objects or actors in the scene.

Figure 5 :

 5 Figure 5: We propose a novel interacting metaphor to craft character poses with Inverse Kinematics to ensure the naturalness of the pose. Starting from an initial pose (Left), the user can create new poses through a small set of operations.

Figure 6 :

 6 Figure 6: Our storyboard interface. (Left) A camera manipulator used to define the shot, the camera focal length and specify the type of camera motion. (Right) The 3D panel used to preview and organize the created shots.

Figure 7 :

 7 Figure 7: Our storyboard interface rely on a set conventions to describe different types of camera motions. Examples are given for (Left) a dolly up, (Center) a Pan to the left and (Right) a dolly in.

Figure 8 :

 8 Figure 8: Examples of four different shots generated by the the Storyboarder in a matter of minutes.

Figure 9 :

 9 Figure 9: Example of features proposed to assist users in shooting a sequence. (Left) Trajectory recording tool. (Right) Rail crafting tool.

Figure 10 :

 10 Figure 10: A linear-editing interface design to create shot edits within the virtual environment

Figure 12 :

 12 Figure 12: Users can add, manipulate and control various types of light rigs available in the environment.

Figure 13 :

 13 Figure 13: Images of two sequences of the input storyboard used in the first evaluation. The first sequence being a static sequence and the second involving a camera pan motion. The color arrows determined the motion of the virtual actors.

Figure 14 :

 14 Figure 14: Boxplot of the results on the NASA-TLX questionnaire for the first experiment.

A script is a textual document that describes the scenes without providing technical information as to which camera angles, scene layouts or cut between camera angles to perform.


Marianne Laurent

Philippe Bretier

Ioannis Kanellos

CONSIDERING THE SUBJECTIVITY TO RATIONALISE EVALUATION APPROACHES: THE EXAMPLE OF SPOKEN DIALOGUE SYSTEMS

Keywords: SDS, Evaluation, Context, Practice

We present in this paper a reading grid aiming at helping the evaluator take into account the subjectivity factor when designing evaluation protocols. Actually, however most contributions to Spoken Dialogue Systems evaluation tend to objectify their approach with rationalising purpose, we believe that subjectivity needs to be considered for valuable evaluations. The first section shows how closely evaluation processes are dependant on their contexts and on the evaluators' perspectives. We then present an anthropocentric framework that establishes the evaluator as a mediator between the consideration of contextual elements and a rationalising corpus of evaluation procedures. We finally anticipate the benefits our framework brings at both individual and community levels.

INTRODUCTION

The issue of SDS evaluation is tackled with opposite approaches from academia and industry.

On the one hand, the academics look for one-size-fits-all evaluation solutions, i.e. shared metrics dedicated to scientific communication within the domain. They need benchmark protocols to evaluate complete Spoken Dialogue Systems (SDS) solutions as well as separate technical modules to be integrated in bench platforms. Moreover, they claim for a portable methodology to process commensurability exercises between SDSs [START_REF] Paek | Toward evaluation that leads to best practices: reconciling dialog evaluation in research and industry[END_REF]. As a result, they need well-specified protocols being both portable from a system to another and adequate to the validation criteria of targeted research communities. This quest for objectivity relies on the idea that an impartial methodology could provide an external perspective to arbitrate on the qualities of various solutions.

On the other hand, as pointed out by Tim Paek, the industrials focus more on the design of services and on best practices than on evaluation [START_REF] Paek | Toward evaluation that leads to best practices: reconciling dialog evaluation in research and industry[END_REF]. The various contributors involved in SDS projects tend to accommodate with ad hoc evaluation protocols that support the local decision-making processes. Therefore, more than on the protocol, they insist on the definition and monitoring of local Key Performance Indicators (KPIs) to scrutinise the design of service. In line with their business objectives, they identify and correct design weaknesses and optimise the user experience. For example, the Orange Labs' development suite [START_REF] Laroche | Hybridisation of Expertise and Reinforcement Learning in Dialogue Systems[END_REF] enables to statistically measure the impact of a given prompt, synthesis voice, prosody, etc. on the overall scenario success. Consequently, commensurability is not of great value for them since the design requirements evolve at the same time of the system design itself.

However the sharing of practices and common evaluation protocols is not a priority, the nomadism of these evaluations approaches may lead to counter-productive efforts. As a matter of fact, the isolated evaluators generally develop and maintain custom-made evaluation spreadsheets, defining selfsuitable KPIs, based on personal definitions and calculation methods [START_REF] Laurent | Ad-hoc evaluations along the lifecycle of industrial spoken dialogue systems: heading to harmonisation?[END_REF]. First, this reduces the opportunity and the ease for collaboration between project stakeholders. Second, the lack of convention for communication of results and comparison of systems' performance may either lead to misunderstanding or to audience disinterest. On the contrary, the implementation of shared practices and standardised indicators may lead to timesaving and improved cooperation.

We conceive both points of view and claim for some efforts to bridge the gap between them. Indeed, it would enable both fields to rationalise their approaches to evaluation, by providing a framework to the nomadic evaluation efforts and considering the subjectivity inherent to each evaluation process at the same time. Just as [START_REF] Paek | Toward evaluation that leads to best practices: reconciling dialog evaluation in research and industry[END_REF] and [START_REF] Pieraccini | Where do we go from here? research and commercial spoken dialog systems[END_REF] advocate for the convergence of academic and commercial approaches of SDS design and evaluation, this article presents a framework to combine the value of a rationalisation of procedures with the consideration of the evaluator subjectivity.

In the second section we underline that, even if some research groups are looking for an objectification of evaluation, subjectivity is an inevitable factor to consider when designing an evaluation process design. The third section presents an anthropocentric framework that considers the evaluator as a mediator betweenthe rationalising corpus of evaluation protocols, its community of practice (CoP) and the evaluation situation. The fourth section explains how such an approach could support the individual activity, as well as the evaluation practices shared in both the communities of practice and the communities of interest.

THE EVALUATOR'S POINT(S) OF VIEW

Evaluation is a subjective process

Along the lifecycle of a project, many are the stakeholders involved [START_REF] Laurent | Ad-hoc evaluations along the lifecycle of industrial spoken dialogue systems: heading to harmonisation?[END_REF]; project owners, technical developers, ergonomics, marketing people, customers and end-users are only examples of the various communities of individuals possibly contributing to a SDS project. Of course, all of them are prone to assess their contribution to the overall project by analysing the system from their activity-biased point of view.

Stufflebeam [START_REF] Daniel | L'évaluation en éducation et la prise de décision[END_REF] defines the evaluation as a process through which one defines, obtains and delivers useful pieces of information to settle between the alternative possible decisions. Likewise, we define an evaluator as an individual who, at some point of a SDS project lifecycle, analyses the system's compliance with a set of expectations or compare systems so as to make a decision. This comprises the evaluation planning (choice or design of the protocol), implementation and report. The evaluation is actually a rationalising contribution to the decision process, even if irrational parameters always gain the upper hand. Therefore, it must be considered as an input to its relative decision-making process, not as a context-independent tool. It is part of an encompassing project in which the evaluator is involved. Examples of questions for SDS evaluation may be: Which of these solutions should I deploy considering the users needs? Has this iteration enhanced or worsen the global service quality? Is the application ready to be deployed or does it need further developments? Such questions are of course tainted with the motivations that animate the stakeholder raising the enquiry.

We hence dismiss the idea of an objective evaluation. First, the evaluation's definition itself underlines its necessary link to a subject. To evaluate is to assign a value to a given object, which presupposes a value judgement delivered by a subject. Second, the subjectivity is present all along the evaluation process, as when understanding the problem than when building an appropriate response to the latter. The top-down approach of the V-Model process described in 4.1 illustrates how the evaluator's subjectivity influences the interpretation of the situation.

Evaluation as a goal-driven argumentation discourse

Yarbus' eye-tracking studies [START_REF] Yarbus | Eye Movements and Vision[END_REF] show that the eye movements depend on what the observer aims at displaying. When observing a scene, the look is goal-guided, trying to identify a selection of clues relevant to a leading issue (see Fig. 1).

These findings are of prime relevance for us since an evaluation protocol is designed, or selected, by an evaluator we consider as a situated subject. Consequently, this design/selection is conditioned by both the evaluator's profile (see 3.1) and the situation that establishes the goals for evaluation (see 3.2). To these regards, setting up an evaluation is like building an argumentative discourse: so as to Fig. 1. The task influences the observer behaviour [START_REF] Yarbus | Eye Movements and Vision[END_REF] provide convincing clarifications to the decision-makers, an evaluator selects the most relevant points of view considering the decision to be made. Moreover, such an argumentation is adjusted to both self-conviction and the conviction of an audience. This means that the discourse must be both consistent to the position taken by the evaluator (the conclusion he aims at reaching) and adapted to the interlocutors' sensibility (norms for argumentation acceptance). The choice of evaluation criteria (see 4.1) is consequently a salient point since they act as argumentative markers for the discourse.

We explained that the evaluators' perspective is of major influence for the design of evaluation; even so, evaluators are not the only source of subjectivity. The following section presents how the evaluators must cope with both subjective factors linked to the project in which they are involved and the system of values inherited from their CoP.

THE EVALUATOR IS A MEDIATOR

This section introduces a reading grid that assimilates the previous reflections. Acting as a thinking model, it consists in an anthropocentric framework where the evaluator assumes the mediation between three dimensions that must be understood both as resources (systems of answers) and as constraints (limits to the evaluator liberty) (see Fig. 2).

The community of practice

As mentioned above, SDS projects involve various families of practitioners prone to evaluate the relative solution. A given evaluator is necessarily biased by its belonging CoP that acts as a normative system. First, the community predisposes the evaluator with a repertory of practices, methodologies and Fig. 2. The evaluation ecosystem points of view for analysis. And, second, it promotes interpretation grids and a system of values that bias the way situations are apprehended. Lorino [START_REF] Lorino | Communities of inquiry and knowledge creation in organizations: the process model in management[END_REF] links this phenomenon with the notion of genre, seen as a common repertory of gestures, wordings, tacit meanings, to which the actors of a given job family refer to. However these actors may adapt the common practice according to their personal style, they remain under the influence of their community. This CoP influences both the evaluator's understanding of the situation and its consequent behaviour as regard evaluation.

The situation acts as an operationalisable context

The situation conditions the goals that motivate the evaluation. Its characterisation is consequently of prime importance since it orients the design, choice and the potential parameterisation of the evaluation to be carried out. We devise the situation as a practical version of the context, the latter being itself a too general concept to be usable. Consequently, we empirically restrain its definition to three factors: the moment in a SDS project lifecycle, the scope of the evaluation and the resources/constraints applied to the evaluation.

Moment within a SDS project

We assume that one of the most significant parameters in the evaluation situation is the stage of the project during which the evaluation is carried out. Actually, it strongly infers evaluation goals and influences the designed protocols. Three major types of evaluation [START_REF] Guingouain | Psychologie sociale et évaluation[END_REF] may be distinguished. First, in the early stages of a project, the prognostic evaluations guide orientation and admission decisions. Examples are return on investment, feasibility and distance-to-target evaluations. Second, the formative evaluations support the intern regulation within the course of the project. For example, they may help spotting design issues or measuring improvements between two versions of a solution in the frame of iterative development. And third, the certifying evaluations are involved, at the last stages of a project, to validate the service with respect to the specifications or predefined tolerance thresholds.

These examples show how the stakes and the protocols may vary between evaluations. Additionally, data capture and analysis differ too. For example, testing a prototype with a panel of test users and testing a live solution with real customers supply different types of data.

Scope of the evaluation

The decision-making context strongly conditions the evaluation scope. First, evaluation may focus on either a technical module of the solution (glass-box evaluation) or a global view on the solution (black-box evaluation). Second, it may address various fields of interest. For example, to measure the impact of the vocal recognition performance on the global usability of the system requires appreciating both the vocal recognition and the ergonomics points of view. Therefore, before building or choosing an evaluation methodology, evaluators must stand back on their practice particularities considering other possible methodologies.

Resources and constraints

As mentioned above, resources and constraints applied to the evaluation strongly depend on the phase in the SDS project. They may also result from the development and financial contexts of the project. Wizard-of-Oz (WoZ) simulations, for example, are known as costly, laborious and time-consuming experimental set-ups, and thus will not be applicable in every context. Similarly, the evaluations based on automatically computed log files require specific tools able to compile and parse them.

The corpus of evaluations

The existing corpus of evaluation contributions may act as a source of inspiration for evaluators. This is also a rationalising system that enables to gain in transparency and reflexive knowledge on the various evaluation procedures. So as to facilitate the acquisition of such knowledge of specialist on the corpus, we proposed a meta-model to classify evaluations (see Fig. 3). It is articulated around three levels of analysis, so as to obtain an individualised knowledge on given evaluation contributions. The first level suggests four categories based on the type of data captured for evaluation: performance measures, predictions of user perception, user perception and adhoc mixed evaluations. The second one focuses on the models and calculation procedures used to transform the row data into meaningful KPIs for a couple evaluator/situation. The third one distinguishes the points of view selected for the analysis. This systematic taxonomy enables to describe the particularities of each evaluation, considering the methodologies as well as the critical viewpoints.

It provides the evaluator with thinking categories, which helps for making their practice explicit and sharable. Yet, it does not claim for being exhaustive in the categories it suggests and allows for the framework extension. It (i) facilitates Fig. 3. A three-layer definition of evaluation the communication and the sharing of results within the domain and (ii) permits to position the solution within a map of existing methodologies. However, it does not take any contextual factor into account.

Implementation in an anthropocentric solution

However the design of evaluation is guided by recommendations [START_REF]Subjective Quality Evaluation of Telephone Services Based on Spoken Dialogue Systems[END_REF] [10] and influenced by both contextual factors and existing solutions, it is not a tractable task. As an alternative, we adopt an anthropocentric approach that provides the evaluators with a pivotal role in such ecosystems1 composed of the CoP, the situation and the corpus of evaluations. They are mediators between a rationalising structure (the evaluation description) and less rational factors that are the interpretation of the decision process and the community footprint on their tacit repertory of practices.

We now work on a software implementation of these ideas: a Multi Point Of vieWs Evaluation Refine Studio (MPOWERS). This system aims the rationalisation -not the standardisation -of evaluation. By rationalisation, we refer to the definition of common norms for the description of processes, common thinking models and vocabulary, for evaluators to make their procedures explicit. Our multi-profile evaluation platform facilitates the design, from a unique corpus of parameters, of personalised evaluations adapted to the particular contexts.

Similarly to our framework for SDS evaluations taxonomy (see Fig. 3), our application will be based on three layers: (i) evaluation data extracted from log files and users and third-party questionnaires, (ii) KPIs aggregated from this row data and (iii) evaluations compiled from these indicators. It stands on the assumption that every evaluation may be modelled as a combination of KPIs, the weight given to each indicator depending on both the system of value supported by the evaluator's community and the situation of evaluation. For a start, the ITU-T Rec. P.Sup24 [START_REF] De | Méthodologie Du Recueil D'informations[END_REF] provides definitions and calculations for sets of common indicators. Our model aims at being dynamic and enriched as new needs arise.

Bottom-up trend

Value judgment: the evaluator creates a meaning. By offloading the evaluator user from low value-added and computable tasks (data collection, cleaning, transformation and aggregation), it enables isolated evaluators to concentrate on the creative aspects of evaluation and the ongoing decision-making. Making the methodology explicit thanks to a prescribed framework, they gain on reusability over projects. Serial refinements over evaluation campaigns and new projects should lead to methodological enhancement.

Top-down trend

The application is still under development but the following section anticipates a twofold relevance. At the individual level, it should provide the evaluator-users with the required expertise to go beyond their step-by-step laborious design process. It should also encourage the collaborative work inside both CoPs and communities of interest.

GAIN EXPERTISE TO GAIN AGILITY

Evaluation design as a problem solving process

As suggested in the introduction, designing a SDS evaluation mostly comes down to a problem-solving oriented process. The evaluators generally build their evaluation stageby-stage from their understanding of the problem. Such an approach can be seen as a V-Model development process. As pictured in figure 4, it starts with a top-down approach, during which evaluators interpret the problem deduced from the situation. Then they translates the problem into objectives, criteria, KPIs and data to be captured [START_REF] Möller | A taxonomy of quality of service and quality of experience of multimodal human-machine interaction[END_REF]:

• Objectives: An evaluation is processed as to prepare a decision. Its objectives are thus constrained by the need to deliver appropriate answers to the decision-makers. The first step therefore consists in interpreting the situation into a set of objectives.

• Criteria. Once the objectives identified, evaluators translate them into evaluation criteria. A criterion is a quality expected in the evaluated object, a viewpoint through which the evaluator examines the object. [START_REF] Rasmussen | Skills, rules, knowledge; signals, signs, and symbols, and other distinctions in human performance models[END_REF] proposes an interesting taxonomy of Quality of Service and Quality of Experience aspects. They may be seen as value scales, as for example, the measure of an application's efficiency or effectiveness, its usability or even the more pragmatic performance / cost ratio. Thanks to making the situation and the evaluator profile explicit, the choice of criteria gains in transparency.

• KPIs. A criterion defines a conceptual ideal for the evaluated application. To determine whether a criterion is satisfied, the evaluator considers a set of various indicators of performance. They correspond to real facts or representations, expressed on the evaluated objects (see [START_REF] De | Méthodologie Du Recueil D'informations[END_REF] for a definition of commonly used parameters from which KPIs can be defined).

• Strategy. Last, a strategy has to be defined as regard the data capture (automatic log, user questionnaires, thirdparty annotation, etc.) and their compilation into appropriate KPIs.

Following this work of analysis and translation into methodological traits, evaluators arrange the evaluation protocol following a compositional approach. This is a symmetric phase to the previous top-down one. They gather the data, prepare them into exploitable KPIs, and compare them with the pre-defined criteria framework. Finally, evaluators provide the decision-makers (possibly themselves) with the results, the evaluation being a input sub-process to a decision making process.

Such a problem solving approach corresponds to what Rasmussen [START_REF] Fischer | Communities of Interest: Learning through the Interaction of Multiple Knowledge Systems[END_REF] positions within the knowledge based behaviours in his Skill-Rule-Knowledge model (see Fig. 5). Indeed, the field lacks a systematic hindsight on the ecosystems of evaluation contributions' to be able to venture shortcuts in this stage-by-stage design process. Therefore, since no routine or pre-defined rule is available, evaluators have to improvise from their personal specialised knowledge on the SDS domain, evaluation protocols and KPIs. We believe that the evaluator would strongly benefit from climbing the SRK model ladder by gaining in expertise.

Toward an expert problem resolution process

The reading grid we introduce will enable a shift from an inexperienced arduous SDS evaluation design to a documented and process-supported one. Enabling the evaluator to reach the rule-based behaviour level, this grid will support him in processing pattern matching between the evaluation situation recognised by the evaluator and an appropriate adequate methodology to implement. Moreover, this will make metaevaluation easier thanks to the acquisition of an expertise on the existing evaluation paradigms along with their relative ecosystems. As a consequence, the evaluator will be able to: • detect more easily changes in evaluation situations, thanks to the disposition of tools to make them explicit,

• dynamically measure the adequacy between an evaluation protocol and the relative situation,

• and, if required, adapt the methodology to the situation.

The decision-making process will then come down to the recognition of a prototypical situation to which an appropriate evaluation may be associated. It requires the capability from evaluators to both recognise these situations and to associate them with types of evaluations. They thus require a frame to recognise situations and a documented and critical description of the evaluation corpus. This includes knowledge on the protocols themselves, but also on their implementation context. In the end, evaluators will adjust the chosen evaluation to local specificities (hierarchy of criteria for example).

An impact on the collaborative work

The reading grid we propose go beyond the individual impact on the evaluator. It also targets the cooperative work inside Communities of Practice (CoPs) on the one hand, and Communities of Interest (CoIs) on the other hand.

CoPs "consist of practitioners who work as a community in a certain domain undertaking similar work (although within each community there are individuals with special expertise, such as power users and local developers") [17]. Our framework may support the collaborative work inside these communities in terms of methodology sharing and negotiation of common protocols for evaluation. This could lead to the emergence of an explicit (no longer tacit) knowledge of existing norms within communities. As a consequence, a meta-analysis would permit to identify regularities within the practices of the evaluators, and thus enable a mapping of CoPs based on observed practices.

CoIs "bring together stakeholders from different CoPs to solve a particular (design) problem of common concern [17]. Our reading grid could be used to understand the complementarities between the different groups of actors inside these temporary communities. Our framework advertises for shared knowledge (corpus of KPIs for example) and enhanced cooperation between actors, instead of the potential conflicts and misunderstandings entailed by the divergence between backgrounds, practices and languages.

Instead of CoI, Lorino [START_REF] Lorino | Communities of inquiry and knowledge creation in organizations: the process model in management[END_REF] suggests the idea of "Community of Inquiry" to describe the group of individuals whose cooperation converge around a "conjoint activity", an "activity that is not characterized by similar practices but their heterogeneous complementarities". In our approach, the emergence of such communities is an essential stake for the overall organisation. It helps to render explicit, and thus sharable and modifiable, the collective activity that would otherwise be seen as a constraint by the isolated stakeholders. However the forming of such communities cannot be decided, it may be encouraged by implementing solution similar to the one we present.

CONCLUSION

We realised this work considering the evaluation issue for spoken dialogue systems, however it is extensible to any other human-machine domains. The framework we presented aims at fostering the rationalisation of the evaluation protocols design while taking into account the situation and the situated evaluators' subjectivity. On the one hand, it helps to make some elements of the evaluation context explicit to the evaluator. It clarifies the situation in terms of decision-making process, audience and constraints, and the community of practice through its value systems. On the other hand rationalising the evaluation may stimulate the sharing of practices across the communities. This convergence gives transparency to the implemented evaluation approaches. It enables improved communication, cooperation, productivity and comprehension across stakeholders, which for them are requisite to stand back from each others' practices and to allow real collaboration.

Fig. 4 .

 4 Fig.4. V-Model evaluation design process. Top-Down process inspired from[START_REF] Franc | L'indispensable subjectivité de l'évaluation[END_REF]

Fig. 5 .

 5 Fig.5. SRK Model by Rasmussen[START_REF] Fischer | Communities of Interest: Learning through the Interaction of Multiple Knowledge Systems[END_REF]

We use here the term ecosystem to refer to a dynamical and complex association of interdependent elements that are constantly interacting with each other.


J Y Le Pommellec

• A El Baroudi

A novel generalized dispersion equation to design circumferential wave fluid sensors

Keywords: Circumferential wave, Transversely isotropic material, Viscoelastic fluid

A novel analytical investigation of circumferential (i.e. torsional) wave propagation in long anisotropic cylindrical rod (waveguide), surrounded by a viscoelastic fluid is proposed. The material is transversely isotropic, with its symmetry axis coincident with the axial axis of the cylindrical rod. In particular, a new form of the complex dispersion equation is presented. The aim of this paper is to study the correlation between the rheological properties of the fluid and the wave characteristics (phase and attenuation). The effect of the frequency and the waveguide radius on the wave characteristics are highlighted. The obtained results show that the measurements should be performed at high frequency using small rod radius. Accordingly, the results can be serve as benchmark solutions in design of torsional wave fluid sensors.

Introduction

The concept of circumferential wave dipstick is attractive in industry for fluid characterizations. The idea is that the wave propagation in a solid elastic rod can sense the fluid rheological properties. The circumferential wave which propagates along the waveguide interacts with fluid boundary, it follows that the circumferential wave properties (velocity and attenuation) are highly affected [START_REF] Kim | The effect of an adjacent viscous fluid on the transmission of torsional stress waves in a submerged waveguide[END_REF].

The circumferential elastic waves in dry cylinders has been studied by many authors [START_REF] Kaul | Torsional waves in an axially homogeneous bimetallic cylinder[END_REF][START_REF] Carcione | Torsional oscillations of anisotropic hollow circular cylinders[END_REF][START_REF] Kudlicka | Dispersion of torsional waves in thick-walled transversely isotropic circular cylinder of infinite length[END_REF]. They have investigated the material properties effects on the phase velocity of the circumferential modes. Huiling et al. have developed a theoretical model of guided circumferential waves propagating in double-walled carbon nanotubes [START_REF] Huiling | Guided circumferential waves in double-walled carbon nanotubes[END_REF]. They studied the dispersion curves of the guided circumferential wave propagation. Xu et al. have modeled the torsional wave propagation along a micro-tube with elastic membrane attached to its inner surface. They have presented numerically the dispersion diagram of the lowest-order wave with the membrane surface effect [START_REF] Xu | Torsional wave in a circular microtube with clogging attached to the inner surface[END_REF]. The interaction between the circumferential waves and viscous fluid has been considered by [START_REF] Kim | Interaction between the torsional vibration of a circular rod and an adjacent viscous fluid[END_REF][START_REF] Kim | Effect of a viscous fluid at the end face on the torsional vibration of a rod[END_REF][START_REF] Abassi | Torsional vibrations of fluid-filled multilayered transversely isotropic finite circular cylinder[END_REF][START_REF] Mnassri | Vibrational frequency analysis of finite elastic tube filled with compressible viscous fluid[END_REF][START_REF] Kim | Instrument for simultaneous measurement of density and viscosity[END_REF]. Nevertheless, the interaction modeling between the circumferential wave and a viscoelastic fluid is still lacking. Indeed, in the field of chemistry [START_REF] Lakes | Viscoelastic measurement techniques[END_REF], medical diagnostics [START_REF] Brust | Rheology of human blood plasma: viscoelastic versus Newtonian behavior[END_REF], or industrial monitoring [START_REF] Cegla | Material property measurement using the quasi-Scholte mode: a waveguide sensor[END_REF], it is necessary to characterize the fluid viscoelasticity. One interesting application is the monitoring of polymerization [START_REF] Harrold | Acoustic waveguide monitoring of the cure and structural integrity of composite materials[END_REF]. During the process, the material changes from viscous fluid to elastic solids. This modification affects the propagation of the circumferential waves in a immersed circular rod.

Circumferential waves in elastic waveguide have been widely exploited to measure the fluid viscosity [START_REF] Kim | The effect of an adjacent viscous fluid on the transmission of torsional stress waves in a submerged waveguide[END_REF][START_REF] Kim | Instrument for simultaneous measurement of density and viscosity[END_REF][START_REF] Roth | A new method for continuous viscosity measurement. General theory of the ultra-viscoson[END_REF][START_REF] Lynnworth | New designs for magnetostrictive probes using extensional, torsional and flexural waves[END_REF]. However, in these papers the character viscoelastic of fluid has not been taken into account. A novel approach based on the exact theory, to be able to accurately predict the circumferential wave behavior in elastic waveguide loaded on its surface with a viscoelastic fluid, remains a daunting task and is the purpose of the present paper. The effects of frequency and waveguide radius on the phase velocity and attenuation of circumferential waves are investigated. The obtained curves show that the attenuation is much more sensitive than phase velocity to https://doi.org/10.1007/s42452-019-0824-6 * A. El Baroudi, adil.elbaroudi@ensam.eu | 1 Arts et Métiers ParisTech, 2 Boulevard du Ronceray, 49035 Angers, France.

Vol:.(1234567890) glycerol concentration. Otherwise, the sensitivity is more significant for small waveguide radius and high frequencies. These results can provide interesting information to design sensors: we can use circumferential wave propagation at high frequency (100 kHz) in anisotropic waveguide with small radius (1 mm).

Physical model description

To describe the waveguide structure that guides circumferential waves, we consider a two-layer system consisting of a viscoelastic fluid and an anisotropic cylindrical rod as shown in Fig. 1. The length of the waveguide is much greater than the radius a of the cross-section so that the interaction of the right-end face with the viscoelastic fluid is negligible [START_REF] Kim | Instrument for simultaneous measurement of density and viscosity[END_REF]. Therefore, the current work focuses on the circumferential modes. The circumferential waves exhibit a multimode character, the fundamental mode plays an important role in many application such as NDT and sensors. Accordingly, in this work, the attention is focused on the properties of the fundamental mode of circumferential waves. For this set of modes, the non-zero components of displacement and velocity are u and v . Moreover, due to the axisymmetry of the studied modes, the velocity field, (r, , z) = 0, v , 0 , and displacement field, (r, , z) = 0, u , 0 , are independent of the vari- able and can be displayed in the (r, z) plane. Note that in previous works the circumferential vibration in an isotropic elastic cylinder was used to determine fluid viscosity. In the present work a novel theoretical approach is developed taking into account both the fluid viscoelasticity and anisotropy of the waveguide.

Mathematical formulation of waveguide

Based on the elastodynamic theory, the equation describing the motion of torsional vibrations of the waveguide in the absence of body forces is governed by where s is density, u is displacement components along circumferential directions, and z , r , are the stress components and can be written using generalized Hooke's law as where C 44 and C 66 are the elastic coefficients. Substitut- ing Eq. ( 2) in Eq. ( 1), yields the following partial differential equation Note that in the case of an isotropic waveguide, C 44 = C 66 = , the Lamé constant.

Mathematical formulation of viscoelastic fluid

In this study, we consider that the fluid velocity is small compared to the dimensions of the model, it then follows that all nonlinear convective inertia effects in the Navier-Stokes equation (NSE) can be neglected, therefore the linearized NSE takes the form where v is the velocity vector, t is the time, f is the density and is the shear stress tensor. In this paper Maxwell model is adopted to describe the fluid viscoelasticity. Therefore, the differential equation for the relation between force and deformation can be written as [START_REF] Joseph | Fluid dynamics of viscoelastic liquids[END_REF] where is the relaxation time, is the dynamic viscosity and is the strain rate tensor. Applying the divergence operator to both sides of Eq. ( 5) and taking into account Eq. ( 4), we get the following viscoelastic fluid equation expressed in terms of the circumferential component of the velocity field The Eq. ( 6) is known as the telegraph equation.

(1) r r + 2 r r + z z = s 2 u t 2 (2) z = C 44 u z , r = C 66 u r - u r (3) 2 u r 2 + 1 r u r - u r 2 + C 44 C 66 2 u z 2 = s C 66 2 u t 2 (4) f t = ∇ ⋅ (5) t + = 2 ( ) (6) 2 v r 2 + 1 r v r - v r 2 + 2 v z 2 = f v t + 2 v t 2

General solution of wave equations

For a circumferential harmonic wave propagation in the z-direction, the solution of Eqs. ( 3) and ( 6) (waveguide displacement u and viscoelastic fluid velocity v ) are sought in the form

where k = k 0 + j is the complex wave number. Note that the real part of the wave number k 0 determines the cir- cumferential wave phase velocity, and the imaginary part represents the circumferential wave attenuation in the propagation direction. After substitution of Eq. ( 7) into Eqs.

(3) and ( 6), the radial dependence can be expressed as where A, B and C are arbitrary amplitudes, I 1 and K 1 are modified Bessel functions of the first and second kind, and 2) and ( 5) the shear stress components that will be used in boundary conditions are given by where the radial dependence is defind as

Complex dispersion equation

In this paragraph boundary conditions must be used to determine the constants A, B and C. Assuming wall outer fluid surface, one can write : (i) continuity of velocity and shear stress at the interface between viscoelastic fluid and waveguide

(7) v u (r, z, t) = V (r) U(r) e j(kz-t) V (r) = BI 1 f r + CK 1 f r , U(r) = AI 1 s r f = √ k 2 - 2j d 2 (1 -j ) , s = √ C 44 C 66 k 2 - 2 c 2 s (8) � r � (r, z, t) = ⎧ ⎪ ⎨ ⎪ ⎩ 1 -j � r v - v r � C 66 � r u - u r � ⎫ ⎪ ⎬ ⎪ ⎭ = � 1 -j f (r) C 66 s (r) � e j(kz-t) f (r) = f BI 2 f r + CK 2 f r , s (r) = s AI 2 s r (9)
v + j u r=a = 0 (ii) wall outer surface of viscoelastic fluid is assumed By substituting Eqs. ( 7) and ( 8) into these boundary conditions provides three homogeneous and linear equations for the constants A, B and C. This system of equations has a nontrivial solution if the determinant of the coefficients equals zero. This leads to the following complex dispersion equation Equation ( 12) represents the complex dispersion equation of circumferential waves propagating in waveguide loaded with a viscoelastic fluid. For given dimensions, elastic waveguide constants and fluid properties, Eq. ( 12) constitutes an implicit transcendental function of k. The complex roots k may be computed using Mathematica software for a first mode vibration. After finding the real part k 0 and the imaginary part of the wavenumber, the circumferential wave phase velocity c p = ∕k 0 can be cal- culated. As a remark, for a semi-infinite viscoelastic fluid, the complex dispersion equation ( 12) becomes In the case of a viscous Newtonian fluid, the complex dispersion equation [START_REF] Brust | Rheology of human blood plasma: viscoelastic versus Newtonian behavior[END_REF] takes the following form which was previously obtained by Kim and Bau [START_REF] Kim | Instrument for simultaneous measurement of density and viscosity[END_REF] for an isotropic waveguide.

Results and discussion

The same material properties used in [START_REF] Pelton | Viscoelastic flows in simple liquids generated by vibrating nanostructures[END_REF][START_REF] Galstyan | A note on the breathing mode of an elastic sphere in Newtonian and complex fluids[END_REF] and given in Table 1 for viscoelastic fluid were taken to construct this numerical example. The waveguide parameters were derived from [START_REF] Zeng | Anisotropy of elastic properties in various aluminium-lithium sheet alloys[END_REF] are given in Table 2. In this work, numerical calculations are performed in the frequency range from 1 to 50 (kHz) and for five values of glycerol concentrations 0, 71, 85, 95 and 100% . Table 3 highlighted the influence of glycerol concentration in water on the phase velocity and attenuation of the wave calculated using the dispersion equation [START_REF] Lakes | Viscoelastic measurement techniques[END_REF]. It can be seen from Table 3 that for each frequency, the phase velocity decreases with the glycerol concentration while the attenuation increases. As a (10) r r=a = 0

(11) v | |r=b = 0 (12) j f 1 -j I 1 s a I 2 f a K 1 f b + I 1 f b K 2 f a + C 66 s I 2 s a I 1 f a K 1 f b -I 1 f b K 1 f a = 0 (13) j f 1 -j I 1 s a K 2 f a -C 66 s I 2 s a K 1 f a = 0 (14) j f I 1 s a K 2 f a -s I 2 s a K 1 f a = 0 Vol:.(1234567890)
remark, for a zero glycerol concentration (pure water), the phase velocity is very close to the shear wave velocity in the waveguide c s (see Table 2). When the glycerol concen- tration increases the phase velocity slowly decreases, and the attenuation significantly increases.

The effect of the frequency on the phase velocity and attenuation is depicted on Fig. 2. It is seen from Fig. 2 that the phase velocity increases with frequency and tends towards waveguide shear velocity c s . It is also shown that the attenuation increases with the frequency. Otherwise, for a given frequency, the phase velocity decreases with the glycerol concentration while the attenuation augments. Finally, we can see that the attenuation is much more sensitive than phase velocity to glycerol concentration. The results highlighted in Fig. 2 can be justified by the influence of frequency on the penetration depth of the circumferential wave in the viscoelastic fluid. When the frequency augments the penetration decreases and the influence of the glycerol concentration on the phase velocity decreases. For high frequency this influence is negligible and the phase velocity is approximatively equal to the waveguide shear velocity c s . Otherwise, the decreasing of the penetration depth linked to the frequency generates an increasing of the attenuation. Consequently, this attenuation is much more sensitive to glycerol concentration for high frequencies.

Waveguide radius is an another essential parameter in design of circumferential wave sensors. The influence of the rod radius on the phase velocity and attenuation is shown on Figs. [START_REF] Carcione | Torsional oscillations of anisotropic hollow circular cylinders[END_REF] that the phase velocity increases with the rod radius and reach shear wave velocity in the waveguide c s (Fig. 3). Oth- erwise, the attenuation decreases in a monotonous way (Fig. 4). For each radius value, phase velocity decreases with glycerol concentration while attenuation increases. This effect is more significant for low radius values. The effect of the glycerol mass fraction on the phase velocity and attenuation is shown on Figs. 5 and6. The calculations are performed for frequency range from 1 to 100 kHz and two rod radius 1 and 6 mm. Each curve shows two regions :

and It can be seen from these figures

• For glycerol concentration less than about 40% , the phase velocity (Fig. 5) is approximatively equal to the shear wave velocity c s in the waveguide (Table 2). Oth- erwise, the attenuation (Fig. 6) is negligible. The influence of the frequency value is very low. • For the glycerol mass fraction exceeding 40% , the phase velocity decreases with mass fraction while the attenuation augments. These behaviors are more significant for small rod radius ( a = 1 mm ) and high frequency. We can see that the attenuation is much more sensitive than phase velocity to glycerol concentration.

Conclusions

In this paper a novel analytical approach was used for the study of an anisotropic cylindrical rod (waveguide) surrounded by a viscoelastic fluid (water-glycerol mixture). A new complex dispersion equation was developed. Therefore, the graphs highlighted the influence of the frequency and waveguide radius on the phase velocity and attenuation for different values of glycerol concentrations. The glycerol concentration effect was also depicted for two waveguide radius (1 mm and 6 mm) and frequency ranging from 1 to 100 kHz. The obtained curves show that the attenuation is much more sensitive than phase velocity to glycerol concentration. Otherwise, The sensitivity is more significant for small waveguide radius and high frequency. Consequently, the characterization of a viscoelastic fluid can be performed using torsional wave propagation at high frequency in anisotropic waveguide with small radius.

Compliance with ethical standards

Conflict of interest The authors declare that they have no conflict of interest.

Fig. 1

 1 Fig. 1 The model geometry of anisotropic cylindrical rod waveguide. s , C 44 and C 66 correspond to the density and elastic constants of the cylindrical rod. The surface of the anisotropic cylinder is loaded by a viscoelastic fluid. The boundary between the anisotropic cylinder and viscoelastic fluid is at r = a . For viscoelastic fluid, , and f are, respectively, relaxation time, dynamic viscosity and density

s

  and f are the radial wavenumbers where d = √ 2 ∕( f ) defines the fluid penetration depth [19], c s = √ C 44 ∕ s is the shear wave velocity in the waveguide. Thus, using Eqs. (

Fig. 3 Fig. 4

 34 Fig. 3 Phase velocity versus cylindrical rod radius

Fig. 5 Fig. 6

 56 Fig. 5 Phase velocity versus glycerol mass fraction

Table 1

 1 Material parameters used for water-glycerol mixtures is the concentration of glycerol in water

	(%)	(Pa s)	f kg/m 3	(ps)
	0	0.000894	1000		0.647
	36	0.0027	1090		1.87
	56	0.00527	1140		3.51
	71	0.02	1190		12.7
	80	0.0447	1210		27.1
	85	0.0923	1220		54.2
	95	0.452	1250		243
	100	0.988	1260		500
	Table 2 Material parameters used for anisotropic cylindrical rod
	(waveguide)
		s kg/m 3 C 44 (Pa)	C 66 (Pa)	c s (m/s)
	Anisotropic rod 2800	2.850 × 10 10 2.345 × 10 10 3190.39
	Table 3 Phase velocity c p and attenuation with a = 3 (mm)	(%)	f = 1 (kHz)	f = 10 (kHz)		f = 50 (kHz)
				c p (m/s)	(Np/m)	c p (m/s)	(Np/m)	c p (m/s)	(Np/m)
			0	3183.99	0.004	3188.36	0.013	3189.48	0.028
			71	3157.66	0.021	3179.97	0.065	3185.72	0.145
			85	3120.01	0.047	3167.81	0.143	3180.25	0.316
			95	3036.50	0.111	3140.23	0.326	3167.77	0.715
			100	2966.22	0.172	3116.47	0.492	3156.93	1.069

Fig. 2 Phase velocity and attenuation versus frequency for a = 3 (mm) Vo.:(0123456789)

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.