Rorik Henrikson

Bruno De Araujo

Fanny Chevalier

Karan Singh

Ravin Balakrishnan

Storeoboard: Sketching Stereoscopic Storyboards

Keywords: Stereoscopic, storyboard, sketching, 3D, anaglyph. ACM Classification H5.2 [User interfaces]: Graphical user interfaces, H5.m [Miscellaneous]: Stereoscopic display

Figure 1. Storeoboard augments sketch-based storyboards with stereoscopic 3D planes for a fluid and flexible authoring of stereoscopic storyboards. -Stereo images in this paper are best viewed with red/cyan anaglyph glasses -

INTRODUCTION

Back in 1922, an audience wearing red/green anaglyph glasses discovered "The Power of Love", the earliest stereoscopic movie played in a theater [START_REF] Zone | Stereoscopic cinema & the origins of 3D film[END_REF]. This illusion of 3D, achieved by projecting two superimposed images, separated into each eye by anaglyph, polarized or shutter glasses, is undergoing resurgence through the recent popularization of stereoscopic displays and improved supporting technology. Yet, in spite of these advances [START_REF] Lipton | Foundations of the Stereoscopic Cinema: A Study in Depth[END_REF] and research to reduce visual discomfort [START_REF] Du | A metric of visual comfort for stereoscopic motion[END_REF], filmmakers are still prone to make choices that may cause unpleasant viewing experiences. Tricks and techniques that traditionally work in 2D do not necessarily transfer to 3D because the latter requires a smoother and gentler editing style [START_REF] Liu | 3D cinematography principles and their applications to stereoscopic media processing[END_REF]. Many new factors exclusive to the Stereoscopic 3D (S3D) medium need to be considered, such as plane separation, parallax position, and depth budgets [START_REF] Atkinson | Stereoscopic-3D storytellingthe conventions, grammar and aesthetics of a new medium[END_REF]. And while filmmakers have developed a language to discuss these concepts, textual descriptions remain "a little broad" to capture the essence of a 3D scene [37, p9].

Motivated to foresee difficulties exclusive to stereoscopy, but also to exploit the unique possibilities of this medium, the 3D cinematography community encourages filmmakers to start "thinking in stereo" as early as possible, arguing that "the sooner you think about 3D, the better your movie will be" [33, p91]. Yet, there are very few early stage tools to support the ideation and discussion of a stereoscopic film. Traditional computer graphics solutions for early visual development and design, in current practices, are either strictly 2D or require 3D modeling skills, producing content that is consumed passively by the creative team [START_REF] Patel | Whitepaper. The New Art of Virtual Moviemaking[END_REF].

Storyboards, in contrast, are powerful tools supporting the organic process where the director and artists explore the film space, try ideas and change visual and narrative aspects in real time before committing to a plan. Not only do storyboards allow the director to concretize and refine their vision, they also serve as the main communication medium between the creative team members [START_REF] Hart | The Art of the Storyboard: A Filmmaker's Introduction[END_REF][START_REF] Katz | Film directing shot by shot: visualizing from concept to screen[END_REF] ensuring everyone works towards the same goals-not some approximation of them.

But current digital storyboard tools only support 2D sketching. As a result, many decisions pertaining to stereoscopy are often pushed to a previsualization stage, left to be figured out on-set, or left to be fixed during postproduction. A few exceptions like Toon Boom's Story Board Pro support 3D, but these tools rely on 3D modeling, wiping out realtime expressive possibilities afforded by the sketch-based nature of traditional storyboards. There is thus a strong need for an exploratory tool to drive the evolution of new vocabulary for the stereo-cinematic medium.

To fill the gap in the filmmakers' toolkit, we contribute with Storeoboard, the first stereoscopic storyboard system that allows the director and artists to explore both the stereoscopic space and concepts in real-time. Storeoboard's design is the result of a distillation of S3D cinematographic principles and issues corralled from loose descriptions in cinematic text, interviews with professional filmmakers, and other anecdotal sources. Support for these principles unique to S3D are built into our stereoscopic storyboards by design. We introduce stereoscopic 3D sketch planes, that conceptually build on the fore/middle/back-ground planes used in cinematography, and stacked layers used in celanimation. Our approach addresses key HCI challenges tied to sketching in stereoscopy, by proposing a fluid and flexible enough interface to support creative flow, while providing rich interactions for depth specification. A thorough evaluation including focus groups, user studies and successful deployment on a feature film suggests that Storeoboard responds to a real need in the S3D filmmaking industry.

The remaining of the paper is as follows. After examining the typical workflow for storyboarding, we distill S3D cinematographic principles and issues that should be discussed at the storyboard stage. We describe Storeoboard, the stereoscopic storyboard tool resulting from our field analysis and present how our tool supports stereoscopic principles, enabling discussion of the unique S3D factors. We report on our evaluation with professional filmmakers and finally, review related work, compare with existing tools, and share further research directions that arose from our observations.

DOMAIN ANALYSIS

To better understand the authoring process of storyboards, and their role in the production pipeline, we conducted informal interviews with six professionals 1 . We describe here the artist's typical workflow in various contexts.

Storyboards are essential planning and coordinating tools at all stages of movie production [START_REF] Hart | The Art of the Storyboard: A Filmmaker's Introduction[END_REF]. When a director starts working with a script, they explore the story one scene at a time, and start thinking about how to break the scene into a series of shots. Through this process, they form their own vision of the scene, which needs to be communicated to the rest of the team-a difficult process through words alone. 1 Interviewees included 2 storyboard artists, 2 directors, one film and television instructor and one director of technology. Their experience ranged from 3 to 30+ years in film and/or animation production (2D&3D).

The Storyboard Workflow

The storyboard process starts when the director calls upon a storyboard artist's drawing skills to concretize their vision with illustrations. The two collaborators can either work closely together or discuss the scenes remotely.

When working together, the director and storyboard artist organically explore the director's vision using pen/pencil/ marker and a pad of paper/sticky notes/note cards. They divide the scene into fore/middle/back-ground layers, and try different visual designs, camera angles, and layouts. This process of externalization aims to help the director clarify and refine their vision, and results in a collection of rough thumbnails (or panels), each serving as a visual description of a key moment of the film. The artist then takes these sketches to create a "clean-up" version on paper or digitally (Figure 2), redrawing the panels with better perspective, detail, and shading; often using different grays, colours, or line thickness to convey depth in 2D [START_REF] Block | 3D Storytelling: How Stereoscopic 3D Works and How to Use It[END_REF][START_REF] Mendiburu | 3D Movie Making: Stereoscopic Digital Cinema from Script to Screen[END_REF].

Alternatively, the director and the storyboard artist do not meet in person. The artist is given the script, and may have a discussion with the director on the phone. This distance yields less iterative input into the sketching process. The artist works directly with digital tools, communicating and refining the sketches until the director is satisfied.

The panels resulting from either one of these approaches are put together (mounting them on a cork board, or digitally side by side) to create the storyboard. This provides a visual story of the entire film and acts as a critical communication tool between all production departments. This visual representation ensures that everyone is working towards the exact same goal that the director envisions. It is therefore critical that the information contained in each panel accurately conveys the information needed for that moment.

Design Considerations

Storyboards are both an important visual communication tool, and the primary medium of dialogue between the director and the artistic team. Depending on the artist and their working context, digital tools come into play at different stages of the storyboard creation process. Our interviews revealed that the main reason for using digital tools later in the workflow is because most of the current professional tools do not yet afford the simplicity and convenience of pen and paper while capturing the director's vision. From this analysis, we distilled a set of general design considerations for a storyboarding tool, beyond it being a lightweight communication tool centered on sketching: Non-intrusive: The interface needs to mimic pen and paper, a medium familiar to artists, and be clear of distractors. Quick authoring: Sketches need to be quick and easy to edit, automatically saving content. Instant feedback: The tool needs to give real-time feedback avoiding techniques based on off-line processing. Overview: The tool needs to support an overview of all the drawn panels, and support the ability to navigate quickly through, and rearrange a storyboard sequence.

S3D Artistic Considerations

Though traditional 2D and S3D movies appear similar, the industry recognizes that S3D movies are not just movies with perceived depth2 , but that there is a whole new set of rules, techniques and understandings needed to create compelling stereo movies [START_REF] Atkinson | Stereoscopic-3D storytellingthe conventions, grammar and aesthetics of a new medium[END_REF][START_REF] Block | 3D Storytelling: How Stereoscopic 3D Works and How to Use It[END_REF][START_REF] Mendiburu | 3D Movie Making: Stereoscopic Digital Cinema from Script to Screen[END_REF][START_REF] Mendiburu | 3D TV and 3D Cinema: tools and processes for creative stereoscopy[END_REF][START_REF] Pennington | Exploring 3D: The New Grammar of Stereoscopic Filmmaking[END_REF][START_REF] Robertson | Rethinking Moviemaking[END_REF]. Due to the increased use of stereo, guides such as the VES Handbook for Visual Effects [START_REF] Zwerman | The VES Handbook of Visual Effects: Industry Standard VFX Practices and Procedures[END_REF] include whole sections dealing with stereoscopic concepts. Apart from the above mentioned design considerations, a stereoscopic design tool needs to include support for factors unique to this style. Based on our literature review and interviews with experts, we distilled key concepts unique to S3D movies. We discuss them here:

Object Placement -the location of objects in stereo space.

While the notion of placement exists for 2D storyboards, it has a more profound effect in S3D.

Plane Separation -the amount of space between planes. Plane separation affects how the audience feels: too much space between planes causes an uncomfortable feeling; planes that are too close feel crowded and busy.

Convergence Location -the elements in the scene that appear to be on the surface of the movie screen. To achieve a stereo effect, images are created with a left and right eye component that are horizontally offset from one another. The fusing of these two images in the brain creates the effect of objects protruding or receding into the screen; the further the offset, the greater the distance an object appears from the screen surface. The convergence plane is where there is no separation between the left and right image, and consequently is the easiest for the audience to process. Parallax Position -the position of objects in-front of, on, or behind the screen. Along with what should appear on the screen itself-i.e., the convergence location-the director must decide which objects should be placed in negative parallax-i.e., in front of the screen, where the audience sits-and in positive parallax-i.e., behind the screen. Parallax placement in stereo space has a profound effect on how immersed the audience feels. If done badly, it can disconnect the audience from the film experience, reminding them they are in a theater (e.g. the traditional gimmick where an object flies out of the screen at the audience, is now recognized as one to be avoided). If done well, the use of parallax can further draw the audience into the film, creating a deeper sense of immersion.

Depth Budget -the distance between the front-most and back-most plane of a scene. Depth budget is used to control the sense of immersion and emotional involvement of the audience. The progression of depth budget has the biggest impact in stereo film and is discussed with a depth script. Depth Script -a chart depicting the change of depth and parallax position from one panel to the next-often compared to a musical score showing the "orchestration" or use of the depth budget throughout a scene [START_REF] Pennington | Exploring 3D: The New Grammar of Stereoscopic Filmmaking[END_REF][START_REF] Robertson | Rethinking Moviemaking[END_REF]. It is common for a depth script to contain both the convergence location and the depth budget progression. A depth script is used to ensure consistency and smooth transitions throughout a film, to help minimize discomfort, and to properly plan so the audience's eyes have time to adjust to depth variations. A depth script is an excellent tool for catching mistakes such as sudden jumps in depth, depth that bounces around, and flat, crowded panels.

Floating Window -the impression of the movie screen being in-front of, or behind the physical screen (Figure 3). In stereo films, the border of the screen acts as a window through which the audience looks. Consequently, it acts much like a separate object in the film and can, itself, be manipulated to make the screen appear closer, further away, tilted forward, backwards, bent or even twisted. This can be used both as a special effect to help set the mood of a scene or to correct mistakes such as edge violations (see below). A 2D equivalent of a floating window is to change the perceived aspect ratio of the screen, creating a false border. By projecting onto this surrounding border, an object appears to move "outside" the physical screen. This effect, which is appealing in 2D, can be astounding in S3D, especially if the object moving outside the frame gets closer to the audience. The concept of a floating window is a new tool in the director's toolkit, and therefore not often considered by directors at the early stages of film design. Yet, there is a growing body of knowledge suggesting that aspects of this technique may be useful to set the mood and feeling of some scenes -i.e. a window tilting forward may feel more ominous. Edge Violation -a visual artifact close to the edge of the screen resulting in visual discomfort. This error usually occurs in one of two ways. First, discomfort and uncertainty arise when an object that stereoscopically should appear in front of the screen is occluded by the edge of the screen. Since the edge of the screen is technically behind the object the brain is uncertain where to place the object; stereopsis tells the brain that the object is in front, and occlusion indicates that it is behind. The second way this violation occurs is when an object is visible with one eye, but not the other. As a result, the brain is not sure whether the object exists or not, causing visual discomfort and disorientation. Edge violations are usually corrected using floating windows.

BRINGING STEREOSCOPY TO STORYBOARDS

In light of our field analysis and derived design considerations, we propose bringing stereoscopy to the storyboard authoring workflow, allowing artists to easily augment their sketches with depth cues, and visualize the resulting stereoscopic panel in real time, as they sketch. To achieve this goal, we aim at leveraging and extending the concept of layers, a mechanism already familiar to artists when abstracting cinematographic scenes down into fore/middle/ back-ground layers for compositing. Enabling the artist to seamlessly try different effects and assess their impact on the stereoscopic view in real time will not only help one to foresee difficulties inherent to S3D, it will also bring a full new dimension to the storyboard as a communication tool now supporting spatial thinking related to S3D film staging.

Our fundamental question "how to bring stereoscopy to the storyboard authoring process?" is twofold. First, how does one enrich the sketching process with a depth dimension without increasing the artist's cognitive cost, so as not to distract them from their primary creative task? Second, how should the user interact to control such stereoscopic visualization while creating content? We identified two interlocked key challenges bound to adding stereoscopy to a creative digital sketching activity:

Fluid depth-cued content editing. Sketching in space using 2D input is a difficult problem that researchers are still trying to address (see "Related Work" section). While working with layers has the advantage of limiting the problem of 3D sketching to a 2.5D sketching task-i.e., by adding essential depth cues to strokes instead of creating fully formed volumes-it also adds to the complexity of the problem in that this depth information must be fully integrated to the creative process, so as to maintain fluid and flexible editing during collaborative discussions. UI design necessitates answers to questions such as how does one maintain liveediting performance while navigating, drawing and changing focus in stereoscopic space? (see e.g. [START_REF] Lo | Stereoscopic 3D copy & paste[END_REF]).

Depth-enabled visualization. Another key aspect concerns visual feedback during the creative and collaborative workflow. More specifically, adding depth to a 2D sketch raises the question of perception [START_REF] Kim | Stereoscopic 3D line drawing[END_REF][START_REF] Yunjin Yongjin | Binocular depth perception of stereoscopic 3D line drawings[END_REF]. While displaying threedimensional content on a flat display is bound to reduce the impression of depth, enabling stereoscopic vision during the sketching process introduces unique new challenges from a user interface design perspective. Practical questions include, for example, depth viewing control: what is the default depth, and to what extent should it be interlocked with the current drawing plane? Should stereoscopy always be active? How to cope with visual clutter of close content when focusing at a far distance?

Creating Stereoscopic Sketches with Current Tools

With the above concepts and questions in mind, we examined the ability to create stereoscopic sketches with existing tools. Performing an online search results in many tutorials for creating anaglyph images with mainstream professional software such as Adobe Photoshop [START_REF]How to Create 3D Anaglyph Images in Photoshop[END_REF][START_REF]How to Create Anaglyph 3D Images That Really Work![END_REF][START_REF]Making Anaglyph Images in Adobe Photoshop[END_REF] and Adobe Illustrator [START_REF]Anaglyphic Text Effect in Adobe Illustrator -Tuts+ Design & Illustration Tutorial[END_REF]19]. Following the tutorials, we created a few stereoscopic images using Photoshop-an industry standard for digital image manipulation-and evaluated the process using this tool. Note that tools other than Photoshop rely on a similar procedure.

The authoring of a simple stereoscopic artifact using Photoshop can be achieved through a tedious repetitive process. First, the artist needs to create a new layer, and draw the content they would like on that layer. To create the anaglyph effect, they duplicate the layer, select the properties of the first layer to turn off the red channel. The same operations needs to done on the second layer, this time, turning off the green and blue channels. Depth control is achieved by nudging one of the two layers a certain number of pixels in one horizontal direction, and nudging the other layer the same distance in the opposite direction. The latest operations (select; nudge; select; nudge) are to be repeated until the layers display the image at the desired depth. The artist repeats the entire process for every additional depth layer.

This process violates our design considerations constraining both expressivity and creativity along the storyboard authoring workflow.

Introducing Stereoscopic 3D Planes

To address the outlined challenges, we propose an approach based on pen interactions on a tablet display that maintains the drawing experience as close to sketching on paper as possible, but augmented with stereoscopy.

Since storyboard artists are accustomed to a layered compositing approach using fore/middle/back-ground planes, our approach treats each panel as a 2.5D stereoscopic drawing volume, parallel to the viewport of our tablet display. This stereo volume can be divided into a stack of transparent stereoscopic 3D planes (Figure 4) which can be seen as a natural extension of the (z-ordered) layer notion used for 2D image compositing. Each of these planes acts as a drawing canvas at a specific depth in the stereo volume.

In contrast to existing sketch-based modeling tools [START_REF] Olsen | Sketch-based modeling: A survey[END_REF] (discussed further in the "Related Work" section), our approach strictly constrains the virtual planes to match the physical viewport of the device. This alleviates the need for explicit manipulation techniques for the orientation of objects and 3D planes while drawing. It also does not force the artist to have a pre-conceived notion of the layout of the 3D scene before starting one's sketch as is necessary when drawing in perspective on ground grids.

Given that a panel is meant to be viewed in stereo, the drawing plane must always correspond to the zero-parallax image plane. To add content in front or behind a plane, one can re-assign plane depths in the stereoscopic space. The content of one plane will be perceived at a different depth than that of another thanks to the stereoscopic visualization.

Initial Approach

We developed a simple stereoscopic sketching prototype that we presented to participants to collect initial feedback. We used red-cyan anaglyph glasses to achieve a stereoscopic view-low-cost hardware that is easily obtained.

Influenced by the physical strategy employed in 3D Tractus [START_REF] Lapides | The 3D Tractus: A Three-Dimensional Drawing Board[END_REF] and Napkin Sketch [START_REF] Xin | Napkin sketch: handheld mixed reality 3D sketching[END_REF], we tracked the spatial location of a tablet-using a VICON system-to navigate inside a drawing volume divided into 20 discrete planes. From the user perspective, it resulted in moving the tablet device towards or away from oneself to access a drawing plane closer or farther in space. Sketching on one of these planes was performed by drawing on the tablet placed at the corresponding physical location. To focus on the conceptual approach, we opted for a minimalist user interface: a single panel (represented as a border line within a white canvas) and the strokes drawn on each plane. The depth of the zeroparallax was accommodated to 70 cm, which corresponds to the average distance of a person holding a tablet in their hand while sketching. To help participants familiarize themselves with the stereoscopic effect before they started creating their own content, we also prompted them with a synthetic image generated from a picture.

We had around 40 people from a variety of backgrounds voluntarily try our system during lab visits (a fourth of the participants were artists). Each person used the system, on average, for a couple of minutes each. We collected feedback through observation and informal discussions.

The users adapted quickly to the stacked planes approach and found the stereoscopy extremely compelling for sketching. The feedback regarding stereoscopic visualization was highly positive, demonstrating value in using this display technique for interpreting line drawings. The navigation restriction to 20 discrete planes was, however, limiting. While we initially thought that pre-defined discrete planes would facilitate an artist's needs, the artists expected continuous navigation which would enable more control over the depth instead. Participants also experienced issues when trying to get back to a given plane, as it was difficult for them to recall the depth at which a given stroke resided. Finally, participants found moving the device to navigate among planes to be entertaining, but cumbersome and tiring when used for long periods of time. This preliminary feedback provided valuable insights to proceed in the design of our approach, and validated the opportunity to bring stereoscopy to sketch storyboards.

STOREOBOARD SYSTEM

In light of the insights gained during our initial exploration, we implemented Storeoboard, a stereoscopic sketch-based storyboard authoring tool grounded in a thorough analysis of moviemakers' creative workflow, as well as concepts exclusive to stereoscopic movies (Figure 5).

Usage Overview

Storeoboard is designed to allow an artist to create stereoscopic storyboards by specifying depth information while sketching. As the artist launches the application, they start by creating a new storyboard scene and its first panel. Storeoboard has three main views: the summary view, the sketching pane, and the depth script view.

Summary View

The summary view features a traditional storyboard layout, i.e., a sequence of panels organized in a grid (Figure 6). This view can be invoked at any time using a pinching-out gesture on the depth script window or the main canvas area of the sketching pane. Each individual panel is represented as a thumbnail, along with its corresponding depth budget bar on the left and annotations below it. Tapping on any panel thumbnail automatically leaves the summary view and enters the sketching pane for that panel. Similarly, tapping on any depth budget bar opens the depth script view (expanded upon later). The summary view also allows one to rearrange panels within the scene: all basic operations such as inserting a new panel, copying, merging, reordering and deleting panels are supported through direct manipulation. It is also worth noting that the summary view and all thumbnails used by the application are stereoscopic.

Sketching Pane

The sketching pane is the core component of Storeoboard, where all creative drawing takes place (Figure 5). It consists of two interactive areas: the main canvas, where the artist can sketch new strokes and visualize the stereoscopic panel, and the depth slider, used to control the current stereoscopic focal depth in the panel-the active plane-and can be used to easily rearrange the 3D planes composing the panel.

As soon as the artist sketches a stroke, a new stereoscopic 3D plane is automatically created and a corresponding thumbnail is added to the depth slider at a position relative to its depth. The artist can keep adding strokes to the current plane, or they can move the slider with their finger to a foreground or background location. By starting to draw at a different depth, they again create a new depth plane. As the slider is manipulated, the stereoscopic image is adjusted continuously to place the current plane on the surface of the display by changing the stereoscopic parallax. Existing planes can be easily rearranged in the slider bar through direct drag-and-drop manipulation of the corresponding thumbnails. Finally, once the artist is done editing a panel, they can move to the next panel by tapping on the right side of the interface. Tapping on the left side of the interface will display the previous panel.

To let the artist focus on their main creative drawing task, Storeoboard generally follows a minimalist version of the pen and touch approach in [START_REF] Hinckley | Sensing techniques for tab-let+stylus interaction[END_REF][START_REF] Hinckley | Pen + touch = new tools[END_REF]: pen input is used for sketching and content editing, whereas touch is dedicated to controlling UI elements.

Depth Script View

The depth script view (Figure 7) displays the progression of the depth budget from one panel to the next and allows the artist to manipulate basic values. This is discussed in more detail in the "Depth Budget & Depth Script" section below.

Storeoboard has been carefully designed so as to support quick authoring and effective communication of stereoscopic artifacts. In the following sections, we get back to the artistic considerations specific to S3D outlined earlier, and discuss how Storeoboard handles these new factors.

Object Placement & Plane Separation

Some of the basic stereoscopic considerations our tool needs to support are object placement and plane separation, which we have addressed in the following manner.

Stereoscopic 3D Plane Control

To create an S3D drawing from a set of stereoscopic 3D planes, we propose a mechanism to specify and control the depth of any input stroke sketched on the main canvas. The depth slider displays a laid out miniature version of all planes to foster quick access and easy manipulations. The depth slider is virtually segmented into three logical areas corresponding to the traditional fore/middle/back-ground viewing composite for reference; but continuous navigation within and across these three areas is supported to allow the artist to position planes as needed. Dragging the slider provides a continuous navigation of the overall space and existing planes, whereas tapping on a depth plane thumbnail results in a direct jump to that depth.

As the slider moves, the stereoscopic view in the sketching pane is smoothly updated to guarantee that the convergence of the active plane matches the screen depth, i.e., the active plane is shown with zero-parallax, which best suits sketching. All other planes, located in front or behind, are perceived at the appropriate relative depth thanks to the stereo rendering. This entire rendering happens in real-time.

Editing Stereoscopic content

To change the depth of an existing 3D plane and tweak the separation between the planes, we enable dragging of its corresponding thumbnail in the depth slider by selecting it using a tap gesture. This operation engages a "move mode" which allows one to move all of the sketched content of the selected plane(s) either forward or backward to make depth adjustments (Figure 9). Through direct manipulation and instantaneous visual feedback, the artist can easily experiment with different arrangements within a panel. Note that we maintain relative size of objects, i.e., moving an object further back will cause this object to shrink on the screen. Holding a finger on the canvas while dragging a plane alters this behavior by maintaining the absolute size of elements.

To reinforce depth cues, we also adjust the shade and width of the stroke according to its depth in real time, mimicking the technique used by storyboard artists. A double tap on a thumbnail initiates a multiple plane selection mode. Subsequently tapped thumbnails add to this selection. Beyond moving planes as a group, having multiple selected planes enables one to control plane separation by bringing them closer together in depth or push them further apart using a pinch gesture. Holding a finger on the canvas while pinching, maintains the absolute size of the elements.

We also support the traditional layer concept. These layers are displayed as a horizontal thumbnail sequence when holding a finger on top of a plane thumbnail (see Figure 5).

Convergence Location & Parallax Position

Storeoboard gives the artist the ability to control which plane appears as the convergence plane, which planes are in positive parallax and which are in negative parallax through the quick manipulations of planes as described above, and specification of the convergence location of the panel, materialized as a small arrow in the depth-slider area.

Recall that while drawing, our tool continuously changes the authoring convergence plane of the panel based on the location of the depth-slider. This introduced an issue as the planes are all in static locations positioned around the convergence. To deal with this issue, we introduced the concept of a default convergence depth (indicated by the little arrow in the depth-slider, see Figure 5). When a new plane is created, the default is set at the middle point of the drawing volume. If the artist wishes to change this location, they simply need to draw a stroke with the pen from the depth slider, onto the main canvas. Any plane that appears below this arrow will be drawn in audience space, everything drawn above, will be drawn in screen space.

Through these simple manipulations, the artist can quickly adjust depth characteristics of the panel without being slowed down or having their artistic workflow impeded.

Floating Windows

The artist can manipulate the frame of a window from the sketch pane of Storeoboard. Side and corner handles, revealed when tapping on the border of the frame, allow resizing. The depth of each frame's handle can also be adjusted individually in the depth slider, allowing the artist to change the very shape of the window, create windows that float in front or behind the convergence plane, or even have screens that tilt, taper or twist. The flexibility of floating window manipulations in Storeoboard allows for quick experimentation of effects as well as planning of viable solutions when an edge violation is detected.

Depth Budget & Depth Script

Storeoboard enables the monitoring of how much depth there is between the furthest and closest point in a panel, how this depth is distributed around the convergence plane, and how this depth progresses, in two ways: through the depth budget of a panel, and through the depth script view.

Depth Budget -The depth budget of a panel is visible in the sketching pane due to the placement of the thumbnails on the depth-slider. In the summary view, each panel includes a depth bar that depicts the distribution of the planes (Figure 6), showing the size and distribution of the depth budget, where it is in relation to the convergence plane, and where each plane is located in the overall space. Tapping on any depth budget bar opens the depth script view.

Depth Script View -The director can see in a glance how the depth of a scene progresses from one panel to the next in the depth script view (Figure 7) where each green bar represents the distribution and location of the depth budget for a panel, and how it is centered on the convergence plane. For instance, the director can immediately spot a depth jump between panels 8, 9 and 10, which may better be avoided as it breaks the smooth progression from panels 8 to 16. Dragging the bars up and down changes the location of the depth planes in relation to the convergence. By using the pen and dragging the top or bottom of the bar, the user can "squish" or "stretch" the depth of a scene to make better use of the available space and help with transitions.

Additional Storyboard Support

To better address the specificity of the storyboard workflow, we include the following functions on the main canvas: onion-skinning which allows the artist to see the neighboring panels as a semi-transparent overlay to guide successive panels; and flip-book functionality to play a sequence of panels as an animation.

USER-CENTERED DESIGN & VALIDATION

Storeoboard is the result of an iterative, user-centered design process. On 3 separate occasions, throughout the project, we worked with a professional storyboard artist. Each session was about 2h long, during which he helped us refine our techniques and workflow. Many design decisions were guided by the regular feedback from the artist. For instance, an early prototype "flipped the page" when switching panels. Though a fast animation, this slowed the artist, breaking his pace. He helped us determine such issues, resulting in a more streamlined process. He created 66 panels (featured in Figures 1,[START_REF] Bae | ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models[END_REF]4 & 9) over the course of the project. We will later refer to this expert as S0.

Evaluation protocol

To better understand the potential of our approach as a creative planning and conceptualization tool, and assess professional artist's acceptance, we evaluated Storeoboard through multiple sessions, selecting different groups based on their level of expertise, and the methods in which they use storyboards. We first describe all three validation sessions we conducted, then present a synthesis of the results.

Session 1: Focus group

Our first session consisted of a focus group with a set of expert storyboard artists -the main target users of our prototype. We held a workshop with 9 participants (aged 21-60; 5 female) at a local arts college. The group consisted of 7 students in their 3rd and 4th year of training as storyboard artists and 2 instructorsboth professional storyboard artists. The purpose of this session was to assess the usability of the tool and evaluate the tool's suitability for a director and storyboard artist to discuss stereo aspects of a film (the prototype version at this stage did not include depth budget and depth script functionalities). The group was given a quick tutorial and allowed 1.5h to freely explore the prototype in small groups -we had 3 tablets for the entire group, and anaglyph glasses for everyone. Participants answered a questionnaire before and after using the tool, and participated in a group discussion to conclude the session. This 3h focus group resulted in 36 panels with 6 sequences and 7 individual panels drawn. Figure 8 is an example artifact created in this session. We will later refer to these experts as S11-9.

Session 2: Semi-structured interviews

This second session consisted of semi-structured individual interviews with 3 directors who work / have worked with stereo films, 2 technical directors and 1 stereographer, for whom a storyboard is a critical reference and communication tool. We aimed to assess whether professionals, familiar with S3D film making, would use stereoscopic storyboards, and at what stage(s) in the process. We also gathered feedback regarding our prototype and its use in the production pipeline. Participants were asked to fill out a questionnaire on their current working practices with storyboards and stereoscopic-related tools (i.e. depth budget, depth script, floating window). Then the experimenter demonstrated Storeoboard and its different functionalities (all functionalities were present), after which participants were asked to answer questions on the tool's different aspects. We will later refer to these participants as S21-6.

Session 3: Deployment

Finally, Storeoboard was successfully deployed as the only storyboard tool on the full length feature film "40 Below and Falling" over a 4-month period (Dec. 2014 to Mar. 2015). The director worked with a storyboard artist to explore his ideas and communicate with other departments to ensure a smooth understanding while filming (the stable version deployed at the time did not contain the depth budget, depth script, nor floating window features). Figure 1 and6 display some examples from the film. The director was debriefed through questionnaires and a phone interview following the filming of the movie. We refer to the director as S3 in the following.

Feedback and discussion

We gained valuable insights from the feedback collected during the different sessions conducted with professional experts. We synthetize them in the following.

Usability and usefulness

All interviewees commented on the usefulness of Storeoboard as a communication tool. It was noted that "there really isn't a language to convey ideas in stereo… [Storeoboard is] the first of its kind" (S0) and "Nice to be able to see stereo at the storyboard stage […] it makes a lot of sense" (S24). Many of the artists also commented positively about the simplicity of the interface, in terms like, "I really like how the interface is simple and intuitive […] it reminded me of drawing on note cards" (S15). When asked about how enjoyable and easy to use Storeoboard was on a 7-point Likert scale, 7 out of the 9 artists from the focus group S1 rated their experience with a score of 6 or greater (1 = not enjoyable to 7 = enjoyable; and 1 = hard to use to 7 = easy to use). The film director (S3) also commented on the ease of use and stated that the program allowed him to "work at the speed of thought". Every participant who tried the tool commented on finding the program fun to use. We questioned all groups about Storeoboard's usefulness for planning and design. All agreed that the tool would be useful for some exploration and communication aspects. One participant asserted that he "would use it for 'what ifs' and 'one off' creative discussions" (S24) another observed that it would allow "studio executives… to 'view' images thanks to the anaglyph" (S21) -though it was noted by the stereographer (S21) that he would want to "work with a storyboard artist".

Focus on stereoscopic functionalities

Only participants of session 2 were exposed and interviewed about the depth budget, depth script and floating windows functionalities. All the industry professionals liked the concept of the depth script and the ability to play with it, though the stereographer (S21) did comment that the storyboard artist should not be the one deciding the depth. What we envision is a workflow where the stereographer starts by providing a high level depth script parameter which could "clamp" the storyboard, setting limits for while the artist draws. The director and stereographer could then adjust the depth of the images at a later time. The stereographer also noted that a depth script needs to be flexible as it can change constantly through the production process. He found our implementation accommodates this need for flexibility.

Feedback on the concept of floating windows was interesting as it is a new tool in the film industry and people are still figuring out how to use them. Currently, floating windows are only accessible to people working in post-production, and they are used mainly for correcting mistakes. The artistic use of floating windows is still largely unknown to most directors, and their uses and value are still being discovered. Hence, most professionals felt the ability to play with floating windows in early stage planning, was "somewhat pre-mature" (S24), but, that the ability "should be in the toolbox" (S21). A participant stated that "Any tool that can be part of the live process is useful; [but] if it slows [the process] down, it's no good" (S24).

Issues and limitations

Storeoboard was met with very positive feedback, though the different groups did note some features that were potentially missing or could cause problems. One aspect that was very apparent with the focus group was that each user has a very personalized way of using the tools with which they work, so the ability to customize the interface is a necessity. Another idea suggested by the different groups was the ability to load an image into the background. This was proposed for both providing reference material in the form of an image or an alignment grid. It was also suggested that having the ability to "cut-out" parts of the background image and move these regions to different planes could drastically speed-up some of the planning processes (especially for film conversions).

With a tool such as Storeoboard, one might expect occlusion of one's hand and other layers to cause problems. This, however, has never been reported -likely because strokes on the same plane are visually grouped together. By occupying different visual depth planes, the strokes have no perceived conflict. Further, hand occlusion is likely not an issue for sketching artists, who typically focus on the pen tip while drawing.

While our current design discourages the creation of planes being placed too close together, technically thumbnail crowding is possible. This was not an issue raised during our evaluation. We suspect that because the difference in depth is barely perceptible when planes are too close, artists do not feel the need to create such concurrent canvases. Nevertheless, one could introduce a magnifying lens or similar technology to temporarily displace overlapping plane thumbnails. Similarly, the scale of the slider bar is not currently adjustable, but could be made into a scrollable widget to address this issue.

A concern that has been raised relates to the use of anaglyph glasses. Artists did not mind wearing them for a period of time, and adapted to the system within minutes; yet, they were skeptical about wearing the glasses all day long. One commented that "[they] hurt my eyes at the end" (S17). Participants in the focus group suggested that possibly drawing the image flat, then "popping" it into stereo for discussion might be an alternate workflow. This idea was supported by the fact that some artists were able to draw most images without glasses, and would put the glasses on to "tweak" the final results only at the end. This was the technique used by the artist of the feature film. Anaglyph was used for this project as it was the most accessible and provided potential for a wider audience; but, anaglyph is only one method to achieve stereo, and we believe other stereo techniques would work equally well or even better, and be less abrasive to the artist (anaglyph is the cheapest, and lowest quality technology on the market). Other methods, however, would be less accessible to individuals without the specialized hardware (such as producers).

Deployment on a live feature film

According to the director of the film (S3), our tool was well received; it "worked exactly as expected" and "achieved exactly what was needed". Before obtaining our tool, the team tried the pre-visualization tools FrameForge Pre-viz Studio, which was perceived as too cumbersome and slow during the brainstorming phase. The director estimates that our tool saved the production hundreds of hours and thousands of dollars; that it "allowed things to flow so much easier between departments. In the latter case, not only the creation of 3D content has to be completed on an external 3D modeling tool, the 3D rendering process of the end result is also a time-consuming process making these tools poor candidates for live editing and realtime collaboration. As an alternative, there are techniques such as previsualization (previs) [START_REF] Patel | Whitepaper. The New Art of Virtual Moviemaking[END_REF] -the process of creating crude 3D models to explore a scene. Previs is an amazing tool for determining the technical and physical restrictions of a scene, provides exact measurements and allows a director to "play" with a camera to discover new shots. Despite this fact, due to the length of time and number of people involved in creating previs, it is a poor choice for exploring ideas, getting one's ideas down on "paper", and discussing early stage concepts. Previs is best as a next-stage technique to refine ideas already captured on paper for complicated, challenging or expensive shots.

Beyond its usage by cinematographers and animators, storyboards are also common instruments for quick user interface prototyping [START_REF] Greenberg | Sketching user experiences: the workbook[END_REF] or even characters' interactive behavior [START_REF] Daniel | PaintBoard: prototyping interactive character behaviors by digitally painting storyboards[END_REF]. However, such ease of usage does not easily translate to 3D and new idioms are required to express depth information. Existing research has addressed such issue by providing alternative methods to create stereoscopic content and pushing the limits of digital sketching.

Stereoscopic Authoring

Using physical devices, Broy et [START_REF] Broy | FrameBox and MirrorBox: tools and guidelines to support designers in prototyping interfaces for 3D displays[END_REF] explored stereoscopic design for S3D UI prototyping, where the user could try different depth arrangements of UI elements, rearranging physical layered planes. This works nicely for design; but, it is too slow and not fluid enough for the needs of planning a film. Other techniques for creating stereo images include Kim et al.'s WYSIWYG system [START_REF] Kim | WYSIWYG stereo painting[END_REF] which explores painting in stereo space, layering "paint" on existing underlying structures. Though effective, systems that use "underpinning" [START_REF] Cohen | Harold: a world made of drawings[END_REF] are inappropriate for storyboards as one needs an understanding of the scene a-priori, to create underlying structures before starting. Storyboards are exploratory, and artists often do not know what they will draw ahead of time, requiring storyboard techniques to be spontaneous, fluid and unstructured. Post-hoc techniques that interpret existing drawings to determine layers and depth to stereoscopize cel animations [START_REF] Liu | Stereoscopizing cel animations[END_REF], while inspiring, are not suitable for real-time sketching in stereo. In fact, it has been noted [START_REF] Lo | Stereoscopic 3D copy & paste[END_REF] that tools and techniques used in stereoscopic environments need to be reconsidered; to be more than just an existing technique transported from a 2D environment onto the 3D canvas; that in fact there are nuances and issues (such as scale and occlusion) that result in the stereo canvas being different from its 2D counterpart.

Sketching in 3D

Digital sketching of 3D objects dates back half a century, to Sutherland's seminal sketchpad system [START_REF] Sutherland | Sketchpad: a man-machine graphical communication system[END_REF] (see [START_REF] Olsen | Sketch-based modeling: A survey[END_REF] for survey). A large body of research deals with interpreting a collection of 2D strokes as a 3D object represented using a network of 3D curves. Tools such as ILoveSketch [START_REF] Bae | ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models[END_REF], True2Form [START_REF] Xu | True2Form: 3D curve networks from 2D sketches via selective regularization[END_REF], and analytic 3D drawing [START_REF] Atkinson | Stereoscopic-3D storytellingthe conventions, grammar and aesthetics of a new medium[END_REF], infer 3D from 2D using a mix of regularity constraints that define geometric relationships between the curves in 3D. Curves are often inferred as cleanly drawn 2D strokes projected onto appropriate 3D planes in these approaches. Another thread of research, exemplified by the organic shape modeler Teddy [START_REF] Igarashi | Teddy: a sketching interface for 3D freeform design[END_REF] and its successors [e.g. 10], builds the surface of a 3D object through the sketching process by inflating or extruding sketch strokes. As an ideation stage sketching tool, 3D objects are both overkill and distract from the primary goal of story layout. Our work makes no assumptions on the nature or meaning of the user's 2D strokes (ours is a free-form sketching interface in the spirit of [START_REF] Dorsey | The Mental Canvas: A Tool for Conceptual Architectural Design and Analysis[END_REF]), but are restricted to planar curves parallel to the view plane following the fore/middle/back-ground cinematic concept. Sketching research has also focused on drawing and control of curves explicitly projected onto a virtual 3D plane [START_REF] Tolba | Sketching with projective 2D strokes[END_REF] (e.g. 3D6B Editor [START_REF] Kallio | 3D6B Editor: Projective 3D sketching with line-based rendering[END_REF]). Researchers have explored physical ways of defining the projection plane's position and orientation: Napkin Sketch [START_REF] Xin | Napkin sketch: handheld mixed reality 3D sketching[END_REF] tracks a physical support, the "napkin"; similarly, Hyve3D [START_REF] Dorta | Hyve-3D: a new embodied interface for immersive collaborative 3D sketching[END_REF] is an immersive virtual environment where the drawing plane is mapped to the 3D position and orientation of the tablet held by the artist; 3D Tractus [START_REF] Lapides | The 3D Tractus: A Three-Dimensional Drawing Board[END_REF] mounts the drawing display on a physical rack to constrain the plane's motion to a depth value. We view these techniques as complementary to our work, where controlling depth of a stereoscopic sketch plane, is but one operation of Storeoboard.

Creating strokes directly in space has also been explored via 3D input (e.g. [START_REF] Grossman | An interface for creating and manipulating curves using a high degree-of-freedom curve input device[END_REF][START_REF] Daniel | Drawing on air: input techniques for controlled 3D line illustration[END_REF]). The SANDDE system [41] uses this in-air drawing technique to create compelling stereoscopic drawings which have been used for animations [START_REF]Animation[END_REF]. These techniques, however, in addition to lacking precise control, move the artist away from the familiar sketching on paper paradigm that we strive to achieve in our system. Our goals are in fact similar to 2.5D modeling [START_REF] Liu | Stereoscopizing cel animations[END_REF][START_REF] Rivers | 2.5D cartoon models[END_REF], as we focus on the essential depth cues to create a stereoscopic effect (such as with stereo painting [START_REF] Kim | WYSIWYG stereo painting[END_REF]), rather than fully formed 3D volumes. Most 2.5D techniques decouple the act of sketching from the depth labeling process. We seamlessly integrate depth manipulations into the drawing workflow.

LIMITATIONS AND FUTURE WORK

Outside of various sketching aids such as drawing primitives or stroke neatening missing in the current prototype, two limitations pointed out by users were: first, the lack of an implicit coupling between stereo viewing and the stereo glasses being on/off; second, that objects drawn as shape outlines are "seethrough" which is apparent upon depth manipulation. This can be mitigated by shading the interior of the shape but a more intelligent inference of closed shapes is subject to future work. As this is the first tool of its kind there are many avenues for future work. e.g. animation interpolation for stereo animatics; automatic switching between 3D visualizations and stereoscopy to reduce visual fatigue; and a larger study on how storyboarding tools may be integrated with other aspects of film production, i.e. sound editing, set design and cinematography. Due to the popularity of stereo 3D systems and this being the first early stage stereo communication tool that has been worked into an existing workflow, there are many opportunities for this technique and approach in other areas of film as well as games and product design (e.g. Broy et. al [START_REF] Broy | FrameBox and MirrorBox: tools and guidelines to support designers in prototyping interfaces for 3D displays[END_REF]).

CONCLUSION

We have introduced Storeoboard, the first real-time sketch based tool that supports the interactive creation of stereoscopic storyboards and facilitates the discussion of unique stereoscopic concepts, collected from cinematic text, interviews and other anecdotal sources. Our contribution stems from a principled approach to the design and development of a system for sketching stereo storyboards: we spoke to artists and filmmakers; built upon traditional cinematographic and animation concepts to design a free-form drawing tool that employs stereoscopic 3D sketch planes; leveraged pen-and-touch input for the direct manipulation of depth-cued content; and evaluated our tool with focus groups, individual user studies and the successful deployment on a live feature film. Our user evaluation suggests that, using our tool, artists can fluidly experiment with a variety of stereo depth arrangements, with minimal time and effort, even during collaborative discussions with the creative team. User feedback leads us to believe that Storeoboard could be effectively integrated in film pre-production and has the potential to greatly improve current practice in stereoscopic filmmaking.

Figure 2 .

 2 Figure 2. Traditional storyboard made with pen on paper. Image courtesy of Naseer Pasha.

Figure 3 .

 3 Figure 3. Floating windows example -in front of screen and behind, both stereoscopic and simulated

Figure 4 .

 4 Figure 4. Illustration of our approach: stereoscopic 3D planes are stacked to form the middle image of Figure 1.

Figure 5 .

 5 Figure 5. Storeoboard User Interface: sketching pane depth slider on the left and main canvas on the right.

Fig- ure 8

 8 illustrates this mechanism.

Figure 6 .

 6 Figure 6. Summary view showing storyboard thumbnails, their depth budget and annotations for a given scene.

Figure 7 .

 7 Figure 7. Depth script view showing the depth progression using green bars for each panel of the scene.

Figure 8 .

 8 Figure 8. Changing the viewing stereoscopic depth within a panel using the slider bar.

Figure 9 .

 9 Figure 9. Moving objects by changing depth planes.

  Finally, the director noted that at first, people would ask "the point of [stereoscopic storyboards].

	"
	"It was a big thing, especially for crews that haven't worked
	with 3D and don't really understand, what should [be done]…it
	allowed them to see right away rather than on set". Regarding
	its use, it was used in "preproduction…by all departments…
	sound, lighting, production design." Further, it "allowed de-
	partments to better understand [the scene], especially lighting
	and production design because they knew where the world was
	already going to be." RELATED WORK
	Many computer-assisted sketching programs have been devel-
	oped to facilitate storyboard authoring, editing and sharing.
	However, existing tools only support 2D content (e.g. Power-
	Production's StoryBoard Quick) or integrate 3D in the pipeline
	by enabling the import of virtual environments and 3D objects
	in the 2D sketching editor (e.g. Toon Boom's Storyboard Pro).

And then when they start to understand what you can do with that, it just made sense." When asked about future projects he stated that he would "use [the program] again in a heartbeat".

It has been noted that "When you are looking at a 2D picture, you look at a flat object defined by the edges of the screen. When you are looking at a[stereo]

3D picture, you look at objects… through a window"[33, p79].

Acknowledgementsa special thanks to our participants for their valuable feedback, the reviewers for their insightful suggestions and our colleagues of the DGP lab at the University of Toronto. Further thanks to Naseer Pasha, Sylvain Malacria, and Dylan Pearce.