Santiago E Montesdeoca

Magic Hock

Soon Seah

Amir Semmo

Pierre Bénard

Romain Vergne

Joëlle Thollot

Davide Benvenuti

MNPR: A Framework for Real-Time Expressive Non-Photorealistic Rendering of 3D Computer Graphics

Keywords: Expressive NPR, object-space stylization, filtering, levels of control, real-time, interaction, framework

We propose a framework for expressive non-photorealistic rendering of 3D computer graphics: MNPR. Our work focuses on enabling stylization pipelines with a wide range of control, thereby covering the interaction spectrum with real-time feedback. In addition, we introduce control semantics that allow cross-stylistic art-direction, which is demonstrated through our implemented watercolor, oil and charcoal stylizations. Our generalized control semantics and their style-specific mappings are designed to be extrapolated to other styles, by adhering to the same control scheme. We then share our implementation details by breaking down our framework and elaborating on its inner workings. Finally, we evaluate the usefulness of each level of control through a user study involving 20 experienced artists and engineers in the industry, who have collectively spent over 245 hours using our system. MNPR is implemented in Autodesk ® Maya ® and open-sourced through this publication, to facilitate adoption by artists and further development by the expressive research and development community.

INTRODUCTION

Non-photorealistic rendering (NPR) has been studied since the emergence of computer graphics-either directly, or indirectly as a side-product in the pursuit of photorealism. As such, NPR presents a significant gamut of research, introducing models, algorithms and systems, which have found practical applications in fields that rely on computer graphics [START_REF] Winnemöller | NPR in the Wild[END_REF]]. However, the application of non-photorealistic rendering in 3D computer graphics (e.g., games, animated films) is limited, especially when compared to photorealistic rendering.

With the plethora of 3D content coming out each year, it has become increasingly relevant for productions to differentiate themselves from others. We argue that expressive NPR approaches are a promising alternative for this purpose, which will be of increasing relevance in the years to come. Therefore, the main goal of this paper is to propose and validate a framework towards expressive NPR of 3D computer graphics in a real production software. This framework provides the tools required for art-direction at different levels of control and covering the interaction spectrum [START_REF] Isenberg | Interactive NPAR: What type of tools should we create?[END_REF]] in multiple styles. In the context of this paper, expressive rendering refers to the ability given to the artist to art-direct the rendered imagery that is produced from 3D data, enabling an individual vision in a specific style.

Rendering, be it photorealistic or non-photorealistic, is not expressive by itself. Photorealistic rendering has pushed towards expressiveness since its inception, offering tools to let artists control the lighting in the scene, the material attributes [START_REF] Schmidt | State of the Art in Artistic Editing of Appearance, Lighting and Material[END_REF]] and tweak the overall photoreal look at the compositing stage. These tools have shown considerable success, to the extent where skilled artists often achieve expressive results that simulate a custom reality, indistinguishable from a photograph.

Non-photorealistic rendering also takes advantage of photorealistic tools and provide a further alternative by releasing them from any physical laws. Additionally, depending on the style of the render, special effects need to be included that help emulate a specific look. Correspondingly, these effects require custom tools that enable different levels of control-depending on the intended audience and use. We cannot generalize tools for all NPR approaches found in 3D computer graphics (i.e., filter-based, example-based, mark-based). However, considering our intended real-time application in games and animation, we focus on providing tools for filter-based NPR approaches that use image-processing to generate the stylized outcome.

Recent research that focused on art-directed filter-based approaches in 3D-space has shown promising results [START_REF] Harvill | Effective Toon-Style Rendering Control Using Scalar Fields[END_REF]Montesdeoca et al. 2017a,b]. However, these existing approaches have only been applied to a single style and present only a narrow interaction spectrum, limiting and slowing down their adoption in the industry. There is still extensive work to be done for expressive non-photorealistic rendering to address and enable the range of stylized looks that artists envision.

The research presented in this paper is motivated by the artistic need for an intuitive and generalized approach towards expressive NPR in 3D computer graphics. For this purpose, we have developed MNPR, an expressive rendering framework in a real production software which focuses on enabling art-direction throughout the interaction spectrum-to maximize its adoption by any skill level and optimize the workflow for each use case. MNPR empowers our stylization pipelines in watercolor, oil and charcoal, but can easily be extended by developers due to its cross-stylistic and open-source nature. To summarize, we make the following contributions:

(1) an art-directed approach towards covering the interaction spectrum in expressive NPR of 3D computer graphics (Section 3.1); (2) generalized control semantics to guide cross-stylistic art-direction (Section 3.2), which is exemplified for three styles;

(3) the implementation insights and source code of MNPR (Section 3.3) as part of its public release; (4) a user study with experienced artists and engineers to evaluate the usefulness of each level of control (Section 4).

The remainder of this paper consists of related work (Section 2), the proposed levels of control and generalized control semantics with implementation details (Section 3), evaluations (Section 4), results (Section 5), and drawn conclusions with potential future work (Section 6).

RELATED WORK

In the footsteps of [START_REF] David H Salesin | Non-photorealistic animation & rendering: 7 grand challenges[END_REF] and [START_REF] Gooch | Viewing Progress in Non-photorealistic Rendering Through Heinlein's Lens[END_REF], [START_REF] Isenberg | Interactive NPAR: What type of tools should we create?[END_REF] argued in his recent meta paper that we, NPAR researchers, "should focus-in addition to working on algorithmic contributions-on how to design the interaction with the algorithmic support". Our work can be seen as the first practical answer to Isenberg's call, in the specific context of interactive filter-based stylization of 3D computer graphics.

We present the design of a stylization framework that is generic enough to incorporate different styles, and that allows the programmer to expose the respective expressive functionality to the artist, throughout the interaction spectrum. Our system has proven its merit in the hands of hundreds of artists who successfully produced art-directed results.

Although interaction might have drawn insufficient attention by the NPAR community, a number of computer systems have been developed along the years, both in the academia and the industry, to give artists some control over the stylization process of 3D assets.

Stroke-based stylization methods offer the most natural, but lowest-level, interaction metaphor, most often achieved by onscreen painting with virtual brushes. Its extension to 3D [START_REF] Hanrahan | Direct WYSIWYG Painting and Texturing on 3D Shapes[END_REF] led to painting systems such as Disney's Deep Canvas [START_REF] Daniels | Deep Canvas in Disney's Tarzan[END_REF]] and OverCoat [START_REF] Schmid | OverCoat: An Implicit Canvas for 3D Painting[END_REF]]. Similar low-level interaction techniques have been proposed for line rendering from 3D models [START_REF] Cardona | Hybrid-space Localized Stylization Method for View-dependent Lines Extracted from 3D Models[END_REF][START_REF] Kalnins | WYSIWYG NPR: Drawing Strokes Directly on 3D Models[END_REF][START_REF] Whited | Computer-assisted Animation of Line and Paint in Disney's Paperman[END_REF]. This resulted in systems that facilitate the creation, combination and application of non-photorealistic techniques in 3D such as Jot [START_REF] Kalnins | [END_REF], OpenNPAR [START_REF] Halper | OPENNPAR: a system for developing, programming, and designing non-photorealistic animation and rendering[END_REF][START_REF] Halper | Creating Nonphotorealistic Images the Designer's Way[END_REF], RenderBots [START_REF] Schlechtweg | RenderBots-Multi-Agent Systems for Direct Image Generation[END_REF]] and more recently, Disney's Meander. Unfortunately, these systems see little adoption in the industry as they either remain proprietary or are implemented as standalone software that is not production-ready. A notable exception is Freestyle [START_REF] Grabli | Programmable Rendering of Line Drawing from 3D Scenes[END_REF], which allows a higher-level, programmable control. However this kind of control is so technical that it was eventually supplemented with dials and knobs when integrated into Blender.

At the opposite end of the interaction spectrum, most generic example-based methods (e.g., [START_REF] Bénard | Stylizing Animation By Example[END_REF][START_REF] Fišer | StyLit: Illumination-guided Example-based Stylization of 3D Renderings[END_REF]) lack intuitive local control of the style transfer, as discussed by [START_REF] Semmo | Neural Style Transfer: A Paradigm Shift for Image-based Artistic Rendering?[END_REF]. Only a few methods targeting a specific task or style, such as hatching illustration [START_REF] Gerl | Interactive Example-based Hatching[END_REF], offer tools that let the user guide the underlying synthesis algorithm.

Closest to our work, but limited to 2D content, the filter-based image stylization framework of Semmo et al. [2016] offers three levels of control-default presets, global parameters, and local adjustments through a painting interface-that drive a variety of stylization algorithms (cartoon, watercolor, oil paint, pencil hatching). Each style is independently parameterized, which makes the framework fully modular but prevents cross-stylistic art-direction.

For 3D graphics, the local parameter adjustments can be painted onto the surface of the 3D objects [START_REF] Harvill | Effective Toon-Style Rendering Control Using Scalar Fields[END_REF]Montesdeoca et al. 2017a,b], instead of a 2D canvas, to ensure spatial coherence in motion. Once rasterized into G-buffers, the obtained 2D parameter maps can then be used to control image-space processing. Going a step further towards a production-ready system raised two questions: (1) how to build higher levels of control on top of the locally painted parameters to cover the full interaction spectrum?; (2) how to design sufficiently generic, yet semantically meaningful parameters to allow cross-stylistic art-direction? We present our answers to those questions in the next section.

EXPRESSIVE NPR

Art-direction is key towards expressive rendering, however, there is arguably no single solution or tool that will satisfy the expressive requirements of every artist. Each artist has a unique vision, approach and workflow when creating artwork, requiring a varied set of tools. 3D artists have the same requirements, but are often constrained by the available algorithms and digital tools. Developers put much effort in addressing these demands, but the task becomes increasingly complex when considering all the different styles and technical approaches inherent to NPR. Our research focuses on proposing a generalized framework for filter-based expressive NPR that provides a wide range of tools for artists and facilitates the further development of art-directed styles for developers.

In Section 3.1, we approach our generalized framework by augmenting the interaction spectrum for expressive rendering, granting a wide range of control to the artist. By doing so, we provide expressive tools for multiple levels of control that will facilitate their workflow and the art-direction of the rendered images.

In Section 3.2, we present our stylistic control semantics, which help developers create tools that work predictably with other styles, allowing cross-stylization with the same parameters. Thus, creating stylizations that generalize upon common phenomena and enabling the artist to visualize their expressive results in different styles.

In Section 3.3, we share our thought process and implementation details of this endeavor, which is supplemented by an open-source release to enable any future researcher and developer to incorporate their non-photorealistic pipelines in an expressive rendering framework within Autodesk ® Maya ® : MNPR.

High Level of Control Low Level of Control

Levels of control

Research in NPR commonly focuses on algorithms that attain a specific effect or style, but offer limited tools for artists to interact with these effects and modify the look of the stylized imagery. This often restraints the artistic workflow and potential of the algorithm itself. To avoid this, ideally a set of art-directed tools that address different levels of control in real-time are required.

Art-directed tools take time to develop, but the rewards justify the effort. Practical interactive tools can increase the creative involvement of artists who can validate the quality of the algorithm and highlight potential intrinsic problems. An ideal workflow should enable 3D artists to quickly set up a general style using high levels of control, offer tools at the mid-level of control to accelerate the stylization process and also provide the ability to meticulously tweak localized parameters-to achieve their particular vision within the style. The tools to cover this range of control are addressed in this section, as outlined in Figure 2.

Style presets and global control. At the highest level of control, style presets allow the artist to save or load predefined global control parameters-interactively changing between different stylization pipelines (e.g., oil, watercolor, charcoal) and assigning the global control parameters in the entire scene, as illustrated in Figure 3. Style presets provide a fast rough start towards a specific style, which is refined by modifying each global control. Global control parameters act as an abstraction layer of effect variables within the image-processing algorithms applied in image-space (e.g., effect amounts, substrate attributes, atmospheric perspective).

Material presets and material control. At the mid-level of control, material presets allow the artist to save or load predefined material parameters of 3D objects. A material preset modifies the underlying shader code and attributes of each material and enable the creation of material libraries with varying painterly shading properties (e.g., dilution, specularity, transparency). Additionally, the painterly materials are embedded with the ability to drive different imageprocessing effects using procedural noise parameters N assigned in material-space, as shown in Figure 4. These allow natural-looking fluctuations in effects that might require them. Mapped control. At the lowest level of control, parameters assigned through mapped control M allow the artist to locally assign specific effects in object-space. This level of control provides the most refined control for the detail oriented artist, but it can also be the most time consuming. However, under real-time interactivity, these tools can be designed to assimilate a painting workflow, becoming immediately intuitive and fun for the artist to use [START_REF] Santiago E Montesdeoca | Art-directed Watercolor Rendered Animation[END_REF]. Mapped control possesses significant uses by itself, but a well thought-out combination of effects can also create targeted stylization concepts (e.g., watercolor turbulence using different pigment densities, as seen in Figure 5)-going beyond the specific effect that they were designed to produce.

Proxy control. In parallel to most levels of control is proxy control, which comprises invisible 3D elements that only contribute to the render with localized stylization parameters P. However, by taking advantage of the 3D space they operate in, proxies encompass the assignment of stylization parameters from a high level of control-affecting the whole image by being placed in front of the camera, resembling NPR lenses [START_REF] Neumann | NPR Lenses: Interactive Tools for Non-photorealistic Line Drawings[END_REF]]-, to a low level of control-affecting a specific part in 3D space-, as visualized in Figure 6. This versatility is unique to proxies, since these operate as standalone elements in the scene, which can form arbitrary representations in object-space. Depending on their design, control proxies can support the previously mentioned procedural parameters in material-space and mapped parameters in object-space.

All these levels of control work in unison to produce the final stylized render. First, the 3D-space driven parameters, which include the procedural noise effect parameters N , the mapped effect parameters M and the proxy effect parameters P contribute to the image-space stylization maps M f x . These positive or negative parameters are combined at the rasterization stage M f x = N + M + P, as seen in Figure 7.

Once the rasterization stage is completed, the stylization pipeline modulates the effect parameters found in the stylization maps with the global effect parameters of the style during runtime. These parameters control the image-processing stages, modifying the effect algorithms that are applied to the original rendered image, to finally create the stylized imagery, as shown in Figure 8.

Generalizing control semantics

Artistic visions can be represented in different styles, as seen in Figure 9. These artworks are an excellent example of two paintings using different media, where the art-direction remains mostly the same. Both masterfully portray a dynamic piece with the same subject in a mystical African setting.

The cross-stylistic art-direction found in traditional paintings is desirable in expressive NPR, as well. However, control semantics are necessary and these need to follow a defined control scheme for the expressive styles to work predictably. Based on our observations and discussions working with artists, we argue that stylization control parameters can be generalized into four distinct categories:

• Pigment-based effects • Substrate-based effects • Edge-based effects • Abstraction-based effects
These four groups can be used to directly correlate the stylization control parameters between different styles.

We have the majority of art-directed parameters in the stylization maps M f x , of which each can manage an effect category. Within each effect category, there are semantics for the type of effect that can be modified. These semantics need to be sufficiently generic, yet semantically meaningful to be adapted to different styles and accepted by the NPR development community. Additionally, these effects need to adhere to a control scheme (summarized in Table 1), which defines what semantics goes to which channel in the stylization map-so that these can be interpreted by other styles. That way, stylization pipelines supporting the same effect categories, respecting the semantics and following the control scheme, would enable e.g., to map an art-directed rendering in a watercolor style to an oil and charcoal style (Figure 10).

We have defined our stylization semantics based on the experience and observations that we have gathered while researching and implementing the styles of watercolor [Montesdeoca et al. 2017a,b], oil paint [Semmo et al. 2016] and charcoal. Based on these different stylizations, we defined the following effects semantics: 3.2.1 Pigment-based effects. This category contains parameters that direct the way the pigmentation is rendered within a specific style:

Table 1: Stylization control scheme. Channel Pigment M f x Substrate M f x R Pigment variation Substrate distortion G Pigment application U-inclination B Pigment density V-inclination Channel Edge M f x Abstraction M f x
• Pigment variation. Controls the degree at which the reflected color of a compound pigment varies towards one or another color. E.g., green pigmentation that deviates to a more blue or yellow color in certain parts. • Pigment application. Controls how the pigment is placed over a substrate. This can be interpreted as the amount or pressure at which pigment is applied to achieve an effect. E.g., dry-brush application, thick application. • Pigment density. Controls the concentration of the pigment placed over a substrate. This is especially relevant to transparent and translucent media (i.e., watercolor, ink, colored pencils), but can also influence opaque media. E.g., dilution, lightness, saturation.

3.2.2 Substrate-based effects. This category contains parameters that direct the way the substrate (i.e., canvas, paper, wall) affects a specific style:

• Substrate distortion. Controls the distortion caused by the substrate roughness on the rendered image. This is especially relevant for fluid media (i.e., watercolor, graffiti). • U-& V-inclinations. Control the inclination of the substrate, which generally affects the direction at which patterns or marks from fluid media evolve. However, generalizing upon this, these parameters are used to define the offset of existing patterns or marks in a horizontal or vertical direction. E.g., bleeding direction, cross-hatching direction, stroke direction.

3.2.3

Edge-based effects. This category contains parameters that direct the way the edges of the subject are rendered within a specific style:

• Edge intensity. Controls the edge strength/intensity within the stylized render. E.g., linework darkness, edge darkening. By adhering to these semantics throughout the stylization pipeline, art-directed scenes can predictably change style and, for the most part, keep the intended effects and look of the expressive render. While these semantics are neither final, nor applicable to all styles and effects, they provide a starting point to address cross-stylization paradigms in expressive rendering.

Implementation

To maximize the use of MNPR by artists and developers alike, we implemented our framework in one of the widely adopted digital content creation packages used in the animation and game industries, Autodesk ® Maya ® . Relying on this software allowed us to easily give access to our technology to hundreds of artists worldwide (over 500 registered users to date), some of which are already using our framework for their own productions.

Tools for different levels of control.

Within MNPR, broken down in the schematic of Figure 11, the 3D scene interaction is entirely managed by Maya. However, enabling the different levels of control for our stylization pipeline required us to develop multiple custom tools, which were developed using Python and PySide2.

Proxy control. In 3D space, we handle the creation of proxy controls from any mesh object, by first disabling the casting and receiving of shadows so that these are not present in the shadow map calculation. We then assign a custom proxy material, which will only render the proxy object to the control maps, once the application data is rasterized. A proxy material preset, loaded from our custom material presets tool, automatically performs these changes, so that the user is only concerned with the art-direction of the "invisible" object.

Mapped control. Any mesh object in the 3D scene can be embedded with mapped control, even proxies. We handle mapped control using vertex color sets, which are created and managed automatically using our custom PaintFX tool. The PaintFX tool modifies three RGBA vertex color sets that render out the controls for the four effect categories that we previously defined. By using vertex colors (instead of textures, for instance), we have a sparse control representation that can take advantage of Maya's animation capabilities to keyframe effect parameters.

Material control. Mapped parameters can only be rendered from the application data if the material supports them. Our custom real-time ShaderFX material renders the 3D application data into the different render targets required by our stylization pipeline: the albedo-which includes some painterly shading properties-, diffuse and specular targets, any necessary G-buffers and the four stylization control maps. Maya's ShaderFX is a real-time, nodebased shader editor that gives us the flexibility to recompile the GPU shader on demand, depending on the operating system and the material properties we require. This mechanism saves unnecessary computation time of optional shading requirements-such as specularity, transparency and procedural noise parameters.

The material control of effects is directed by procedural noise parameters that are also embedded into the ShaderFX material. Our custom NoiseFX tool allows the artist to control noise functions from an abstracted interface, avoiding the navigation through each shader graph towards the specific noise parameter in a node. Our materials accumulate the control contributions generated by these noise functions and the previously mapped parameters, found in meshes and proxies, into the stylization control maps-following the scheme found in Table 1.

3D Scene Interaction

Stylization pipeline.

Once all the data is rendered from the 3D scene, a series of image processing operations, driven by shaders (HLSL or GLSL), are defined and run. These operations conform to the stylization pipeline, which is further driven by the global control parameters and any other external data (e.g., substrate). While the exact implementation of each stylization pipeline is under constant evolution and out of the scope of this paper, we encourage the interested reader to check the code available at the project's Git repository (see Section 6).

The stylization pipeline, powered by our custom plugin, uses the Maya C++ API and takes advantage of the Viewport 2.0 API. However, we extended some API classes and methods to design and abstract a modular framework. This facilitates the creation of stylization pipelines for people who are not used to working with the Maya API, enabling a simple adaptation of art-directed NPR stylization pipelines in three steps:

• define and create attributes for global effect parameters;

• outline the custom stylization pipeline;

• define NoiseFX and PaintFX controls (Python).

Additionally, we have embedded debugging utilities into MNPR to help the developer debug passes, change color spaces and visualize individual channels and their negative values.

Lessons learned.

While working with Maya has propelled the set of features and the pool of artists that we can access, developing for this closed-source software has its caveats. We would like to share a brief list of lessons learned, for any developers or researchers looking into using and expanding MNPR.

• Documentation, example code and user-base of the Viewport 2.0 API is limited. • Maya will not return certain shader errors when working with GLSL shaders. This led us to develop in HLSL first and then port to GLSL afterwards. • Keep hardware limitations in mind when rendering to multiple render targets (8 max). • Packing control parameters into 32 bit floating point targets will not support a simple accumulation of parameters from different levels of control. • Maya's internal texture painting tool will not work on custom materials, which drove us to map to vertex colors instead.

USER EVALUATION

To evaluate MNPR and its usability, we concentrate on two aspects:

(1) how effectively developers can implement different styles and

(2) how useful artists find the levels of control we provide.

Developers' evaluation

We adapted the watercolor stylization of Montesdeoca et al. [2017a[START_REF] Santiago E Montesdeoca | Art-directed Watercolor Rendered Animation[END_REF]Montesdeoca et al. [ , 2017b] while developing our framework, but we needed other styles in order to evaluate it. Therefore, we implemented the oil stylization of [START_REF] Semmo | Image Stylization by Oil Paint Filtering Using Color Palettes[END_REF]Semmo et al. [ , 2016] ] (without color quantization) and invited an inexperienced computer science undergraduate student to develop and implement a charcoal stylization as part of a final year project.

Porting over Semmo et al.'s oil paint stylization work was quite straightforward and took approximately 2 weeks, which included re-writing the compute shaders to HLSL and redefining the order of the image processing operations. However, this time did not include the different development challenges that arose from porting a 2D stylization technique into 3D. A particular challenge was porting over the procedural impasto marks in a coherent way under animation. In this pursuit, we extended MNPR to include a way to extract the motion of each vertex by calculating the deltas with their previous position (after deformations) and made use of coherent noise [START_REF] Kass | Coherent Noise for Non-photorealistic Rendering[END_REF] to advect the procedural noise along the object's motion. While the results are promising, calculating per vertex motion of fully deformable objects is slow and only enabled when real-time interaction is not required.

The charcoal stylization took understandably longer to develop, as it was not a full-time effort, there was no prior shader code and it involved a significant learning journey into image processing and shader development from the developer. Nonetheless, our framework managed to abstract the development process of a graphics engine, allowing to concentrate on the image processing operations and the interplay between them.

MNPR successfully supported both development efforts through the example code/documentation to embed the art-directed parameters within the image-processing operations, following the control scheme at Table 1. Once these parameters were incorporated, the expressive control tools were managed by our framework by simply defining the code to manipulate them. Both of these additional stylization pipelines are featured in our results, showcasing the successful adaptation of styles into MNPR, as well as the cross-stylistic support of art-directed effects. In general, the feedback from these development efforts was quite positive and allowed us to iterate on our framework to improve upon it and extend its functionality.

Artists' evaluation

We aimed to grant artists the widest range of control in objectspace and, thereby, evaluate their preferences and uses at each level of control. For this purpose, we conducted a user study involving experienced artists and engineers to evaluate the usefulness of each control and gather feedback on how to improve them.

A total of 65 users registered their interest to take part in our study and received MNPR with a watercolor stylization pipeline. To bring everyone to the same level of proficiency using the framework and its tools, we included tutorials for each level of control and how to use them effectively. After a period of two weeks, 20 participants answered an online questionnaire that helped us gather their usage data and analyze their feedback. The individual responses are available within the supplementary materials of this paper, of which we have consolidated the following.

Prior experience.

To assess their responses, we were initially interested in their prior experience (Figure 12) using traditional watercolor, other NPR solutions and Autodesk ® Maya ® -together with their overall years of experience in a computer graphics related industry. The majority of users had limited to minimal experience using traditional watercolors (I), which is expected as we were targeting 3D artists and engineers. These results resembled their prior experience with other NPR solutions (II), probably reflecting on the scarce dedicated commercial NPR solutions available to date. Nonetheless, our users were mostly experienced with Maya (III) and within a wide spectrum of experience in computer graphics related industriesinvolving students and veterans alike (IV).

User study.

In this study, the participants were asked to commit to at least two hours when testing MNPR and its distinct levels of control. However, their dedication exceeded our expectations, as most participants reported spending substantially more time testing our framework, as observed in Figure 13a. It allowed them to familiarize themselves better with each level of control.

Levels of control (Figure 13b). Style presets and global control (I) were generally perceived as useful, tending towards highly useful in some cases. This result was expected, as these represent the main control over the stylized render and the starting point towards a stylized look. There was a general consensus that the material presets and material controls (II) were useful, with little deviation. We believe this result represents the general awareness of this level of control, especially considering existing photorealistic workflows that rely heavily on material attributes to define the look of objects. Material effect controls (III), with their procedural noise parameters-providing mid-level control for effects-were generally perceived useful. This level of control took the participants the longest to understand, but most found its usefulness after experimenting with it. Mapped control (IV) was considered the most useful level of control by the majority of the participants. This is probably due to its localized nature and intuitive application through painting tools. Finally, proxy control (V), while being completely detached of the rendered subject, was generally found useful. This decoupled approach to art-direct the stylization received the most varied result, with some participants enthusiastically welcoming the addition (see Figure 14), while others not seeing their usefulness.

Overall satisfaction (Figure 13c). Overall, the participants tended to agree that they achieved their stylistic goal using our framework (I). We also asked the participants if they previously used other NPR solutions, for their current workflow; and if they would have achieved their desired stylization faster using our framework. While they were mostly using offline workflows-which involve several stages with different software-their answers were mostly neutral, but with a tendency towards agreeing. We believe these responses were positive, considering that stylistic goals may differ widely between users and that they were constrained to only one stylization pipeline within our framework, with limited learning time. We expect this to change when more stylization pipelines are developed and MNPR matures in the public domain.

Of special importance to our study and our research is that most agree and even strongly agree that the range of control benefited their stylization (II), reinforcing our work and encouraging us to continue our pursuit towards art-directed tools. However, as one participant pointed out, it is important to find a balance within control, as this might distract users from the "bigger picture". Finally, an overwhelming majority of the participants strongly agreed that they would consider using our framework for their future projects (III).

RESULTS AND DISCUSSION

Based on our experiments and reinforced by our user evaluation, we can conclude that the interaction spectrum, given by our different levels of control, was quite useful for artists-assisting the stylization process by enabling a faster and more versatile workflow. Additionally, since the interaction happens in real-time, the art-direction and rendering are performed concurrently, allowing the artists to see their changes immediately. We tested all these levels of control, working in unison, using three distinct stylizations: watercolor, oil paint and charcoal. Our results are presented throughout Figure 16. Considering the experimental nature of these stylization pipelines, we have managed to produce satisfying results.

Finally, by generalizing our control semantics, our framework permitted the art-direction within one style to be visualized in others, as seen in Figure 1, Figure 15 and Figure 17. These stylization pipelines all run within MNPR in real-time and the user can interactively switch between them. The individual performances of each stylization pipeline have been measured in Table 2 with the following configuration: NVIDIA ® GeForce GTX 1080, Intel ® Xeon ® E5-2609 @2.4Ghz and 16Gb of RAM using Maya 2017. Our framework, without any custom stylization pipeline, involves five different passes to render the 3D scene, adjust or load data, provide channel or color-space debugging utilities and perform anti-aliasing. Our most complex pipeline, with 27 different image-processing passes (including 6 separable filter passes) still manages to run at 60 frames per second when rendering Figure 15.

CONCLUSION

We have presented MNPR, a framework for expressive non-photorealistic rendering of 3D computer graphics, covering the interaction spectrum to optimize the workflow of users with different preferences and skill levels. We have validated this by conducting a user study with 20 experienced artists and engineers in the targeted industries, who shared their feedback and perceived usefulness of the developed tools for each level of control. To enable art-direction in multiple styles, we also proposed sufficiently generic, yet meaningful semantics for parameterization, which allows users to interactively visualize their art-direction in manifold styles. This was tested and successfully showcased within our implemented stylization pipelines involving the watercolor, oil and charcoal mediums.

Finally, we have shared some insight towards our implementation in Autodesk ® Maya ® and are hereby releasing the source code to augment this contribution and motivate adoption in the Expressive community-as a practical answer to Isenberg's 2016 call towards designing the interaction with, or for algorithmic support. The source code can be found at our institutional data repository [Montesdeoca 2018] and at the project's Git repository (http://github.com/semontesdeoca/MNPR). With this work, we hope to make Expressive research more accessible for the target users, validating our methods and inspiring new contributions from the research community.

Limitations and future work. While MNPR provides a useful starting point towards bringing NPR research to the hands of artists, it still remains to be tested and refined with other styles and in production. For example, we have not implemented a method to retain the motion coherence of substrate-based effects yet, as this is still an open problem that needs to be convincingly solved [START_REF] Bousseau | Interactive Watercolor Rendering with Temporal Coherence and Abstraction[END_REF][START_REF] Bénard | Dynamic Solid Textures for Real-Time Coherent Stylization[END_REF][START_REF] Bénard | Stylizing Animation By Example[END_REF][START_REF] Hays | Image and Video Based Painterly Animation[END_REF]. A productionready solution would need to take this into account if temporal continuity is desired i.e., when alternating substrate patterns-which happen naturally with traditionally painted animation that present substrate-based effects-are not desired. We have also shared some limitations through the lessons learned during our implementation (please refer to Section 3.3). Some of these limitations pertain to the current available hardware, but others pertain to Autodesk Maya and its closed-source nature, from which we hope to spur support in the future and work together towards having more expressive NPR available to artists and productions alike.

Our cross-stylistic semantics enable art-direction in multiple styles in a predictive manner. However, although we have tested these style control semantics with watercolor, oil paint and charcoal with relative success, their applicability still needs to be further proven in other styles e.g., hatching [START_REF] Suarez | Real-time 3D rendering with hatching[END_REF] or even Cubism [START_REF] Arpa | Perceptual 3D rendering based on principles of analytical cubism[END_REF]]. Additionally, some styles require more than just a change in stylization pipelines and effects for the subject to acquire the intended look. A clear example is given when contrast in a specific style is achieved through color variations and another through lightness values, as is the case with watercolor/oil and charcoal. As illustrated in Figure 18, a simple conversion from an oil paint stylization to a charcoal stylization will lack the contrast that defines the form of the subject. Notwithstanding, in this case, an adjustment to the materials in the scene through an additional tool, can let us modify the diffuse behavior to accommodate the look and bring the necessary contrast back.

Finally, our tools to art-direct the stylization at different levels of control can be subject to further improvement and better alternatives. Based on the usefulness found by artists, we can especially argue that different ways to provide material and proxy control of effects could provide a promising area of future research, due to their rather unexplored nature and variance presented in their usefulness.

Science and Engineering (SCSE) at the Nanyang Technological University, Singapore, and was partly funded by the Federal Ministry of Education and Research (BMBF), Germany, for the AVA project 01IS15041B.

The terms of the Creative Commons Attribution 4.0 International License are found at: https://creativecommons.org/.

Figure 2 :

 2 Figure 2: Schematic overview of the different levels of control that we address in MNPR.

( a )

 a Normal viewport 2.0 (b) Watercolor preset (c) Oil paint preset (d) Charcoal preset

Figure 3 :

 3 Figure 3: Different stylization applied through style presets to the Pony Cartoon model, cb Slava Zhuravlev.

Figure 4 :

 4 Figure 4: Material-space procedural effects applied after the watercolor preset of Figure 3b.

Figure 5 :

 5 Figure 5: Left: Procedural pigment density (turbulence <5 sec.). Right: Mapped pigment density (turbulence 40+ sec.).

Figure 6 :

 6 Figure 6: Stylized result after applying the mapped control; proxies are visualized in blue.

Figure 7 :

 7 Figure 7: Effect parameters within stylization map M f x . Positive values are visualized in blue, negative values in red.

Figure 8 :

 8 Figure 8: Final watercolor rendering leveraging all levels of control.

  , Graphite 4x12". © Dylan Scott Pierce.Life Spring, Watercolor 8x22". © Dylan Scott Pierce.

Figure 9 :

 9 Figure 9: Paintings showing cross-stylization with different types of media.

Figure 10 :

 10 Figure 10: Art-directed Pony Cartoon model visualized in oil paint (left) and charcoal (right) styles.

Figure 11 :

 11 Figure 11: Schematic of MNPR with its different interaction stages within the 3D pipeline.

Figure 12 :

 12 Figure 12: Experience level of users.

Figure 13 :

 13 Figure 13: Results of the user experiment

Figure 14 :

 14 Figure 14: A screenshot of a scene by a test user where proxies (blue, left) are mostly used to drive the stylization (right).

Figure 15 :

 15 Figure 15: Art-directed cross-stylization character example. Miss Seagull model, © Julien Kaspar.

( a )

 a Baba Yaga's hut model, c b Inuciian. (b) Wiz Biz model, c b Tom Robinson. (c) Chicken Friend ;o model, c b MahoSway.(d) SIKA model, © Marine Quiviger.

Figure 16 :

 16 Figure 16: Rendered frames using our framework. Please refer to the accompanying video for more results.

Figure 17 :

 17 Figure 17: Art-directed cross-stylization example. Summer Afternoon model, © Stevie Brown.

Figure 18 :

 18 Figure 18: Cross-stylization: oil paint to charcoal, Steam Cowboy model © Black Spire Studio.

  • Edge width. Controls the edge thickness of the stylized render. E.g., linework width, edge darkening width. • Edge transition. Controls the edge transition of the subject in relation to neighboring elements. E.g., edge softening, gaps and overlaps. 3.2.4 Abstraction-based effects. This category contains parameters that direct the abstraction of the rendered subject within different styles. • Detail. Controls the amount of detail at different parts of the stylized render within the subject. E.g., detail blur. • Shape. Controls the amount of shape abstraction/distortion of the subjects. E.g., hand tremors. • Blending. Controls the color blending at different parts of the stylized render. E.g., smudges, color bleeding.

Table 2 :

 2 Performance for the different styles of Figure 15, 288.800 triangles, screen resolution of 1920 × 1080 pixels.

	Style	Passes Separable FPS (avg)
	Framework	5	0	250
	Watercolor	13	2	100
	Oil paint	27	6	60
	Charcoal	17	6	70

ACKNOWLEDGMENTS

Special thanks go to Yee Xin Chiew and all the participants in our user study for testing MNPR, sharing their feedback and enduring the hiccups during its development process. We also thank Hans-Martin Rall and Jürgen Döllner for their support during this endeavor. Our gratitude is also with our colleagues for their comments and suggestions, together with the anonymous reviewers for their constructive feedback. We would like to thank the Multi-plAtform Game and Innovation Centre (MAGIC) and the Hasso Plattner Institute for Digital Engineering (HPI) for supporting this international collaboration. This research is supported by the Interdisciplinary Graduate School (IGS) and the School of Computer


Brett Ridel

Patrick Reuter

Nadine Couture

Teaching Spatial Augmented Reality: a practical assignment for large audiences

Keywords: CCS Concepts, Social and professional topics → Computer science education, •Computing methodologies → Mixed / augmented reality, Tracking, Camera calibration, 1. Motivation

a) Real-world setup (b) Scene representing the real-world setup (c) Virtual scene Figure 1: (a) Photo of the real-world setup that we simulate in the assignment, consisting of a video projector and a white diffuse styrofoam object. (b) Overall view of the scene representing the real-world setup, consisting of a simulated projector and the non-textured 3D model of the real-world styrofoam object. (c) Overall view of the virtual scene consisting of a virtual camera and the textured 3D model of the real styrofoam object.

Spatial augmented reality, also known as projection mapping, makes it possible to augment the real world with virtual information by means of a light emitter such as a video projector. First introduced by Raskar [RWLB01], one of the major advantages is that the users look at the real world and not on a screen as in seethrough augmented reality, thus making the presence of a computer seamless. Today, the technology is mature and used in applications Hence, it is timely to establish effective methods to teach spatial augmented reality to students of the different domains. Indeed, most teaching efforts in augmented reality address see-through augmented reality where, in terms of equipment for a practical assignment, a computer, tablet, or phone in conjunction with a camera is sufficient. See for example the tutorials and examples on Udemy1 , Instructables2 , and others 3 . They all use Unity3D for representing the scene, Vuforia 4 for the tracking, and some script programming in C sharp. These examples involve only freely available software tools combined with hardware that most people have available anyway, resulting in a widespread use in teaching.

On the other hand, teaching spatial augmented reality necessarily requires the adjustment of a video projector, which is an equipment that is not available for every student in larger classes, thus requiring new teaching methodologies. Indeed, the motivation for writing this present paper came from the preparation and realization of a three-hour lecture for spatial augmented reality within a three-day course about augmented reality in general, targeted to groups of more than 20 students.

In this paper, we present a new methodology that we conceived at this occasion for teaching spatial augmented reality in a practical assignment to large audiences. The major benefit of our approach is that it does not require a specific equipment such as a video projector for every student while teaching the principal concepts and issues involved in spatial augmented reality applications, and especially calibration and tracking. Moreover, like the successful examples for teaching see-through augmented reality, our approach only relies on freely available software tools with no additional hardware, while still being a practical assignment.

The key idea is to set up a 3D scene with a projector that simulates the real-world projector used in the spatial augmented reality application, and a virtual representation of the real-world object to project virtual information on. So in fact, this 3D scene represents a view of the real-world setup. The image to project is calculated with a virtual camera that captures a textured version of the virtual object. We implemented our teaching methodology in Unity3D as an example. This paper is organized as follows. In Section 2, we recall the principles of spatial augmented reality and the involved difficulties. In Section 3, we first show the overall idea and the targeted example of the assignment, before discussing the involved exercises to simulate a complete spatial augmented reality application step-by-step. In Section 4, we provide results of a preliminary evaluation of our practical assignment based on the feedback of the students, before we conclude and discuss potential improvements in Section 5.

Spatial augmented reality and the involved difficulties

The core of spatial augmented reality is the superimposition of a digital image onto a physical, real-world object, by using projected light. To simplify the writing, we refer in the remainder of this paper to a video projector as the light emitter as most commonly used. The digital image to superimpose is computed by deriving a 2D image from the 3D scene of the real-world object to project on. Using the correct viewing parameters to derive this 2D image is essential for a perfect superimposition, which is quite easy to understand: intuitively, the 2D image must be identical to what an observer would see at the position and orientation of the projector.

There are two major steps for setting up a spatial augmented reality application, namely the calibration and tracking, that we briefly explain in the following. For more technical details, we refer the interested reader to Bimber and Raskar [BR05].

Calibration In order to compute the correct image, the video projector has to be calibrated. This means that its intrinsic parameters, such as the optical center and the focal distance, have to be determined. For the reader less familiar with these optical terms, this roughly corresponds to the viewing cone of the projector. Although intrinsic to the projector and independent from the spatial augmented reality application, these parameters are rarely provided by the manufacturers as they change according to the chosen display resolution and thus the involved aspect ratio, as well as the zoom factor if available. Most often, this calibration step involves the use of a camera in addition to the video projector [AO09

The assignment

To circumvent the use of additional material, our assignment is based on a 3D scene with a simulated projector that "projects" content onto a virtual representation of a real-world object, and the elements are organized hierarchically in a scene graph. Our assignment is targeted to Unity3D as it is freely available, and only basic knowledge of this 3D authoring tool is sufficient.

For the sake of pedagogical effectiveness, we conceived a simple yet representative example where both the real-world object and the projector are at fixed positions: the projection of a texture onto an object made of styrofoam with diffuse reflection. This example can then be extended as desired with dynamic elements, for example. We suggest that the lecturer of the assignment brings one single projector and the real-world object in order to render the explanations more tangible (see Figure 1(a)). We chose the BB8 character since an approximate replica can be easily built with two styrofoam balls that are cheap and easy to obtain, and a textured 3D model is freely available. Recall that in spatial augmented reality applications, it is essential to have both the real-world object together with its virtual representation: either the real-world object is acquired to obtain the 3D model, or, starting from the 3D model, a real-world physical object is fabricated, for example by 3D printing.

The guiding principle is to make the students understand that for

T virtual T real

Figure 2: Illustration of the scene graph used in the practical assignment. The tracking consists in making correspond the transformations T virtual and T real .

a correct superimposition of the virtual information over the realworld object, the projected image in the real world has to be computed properly according to the prior calibration and tracking. To this end, we let the students create a scene graph that consists of two related 3D scenes as described below and illustrated in Figure 2.

Scene representing the real-world setup

The first scene, illustrated in Figure 1(b), is a virtual representation of the real-world setup of Figure 1(a), consisting of the virtual counterparts of the real-world video projector Proj real and the real-world object Obj real to augment. The expected result of the assignment, the augmentation, will become visible in this scene as explained further below, and therefore, our practical assignment does not require specific hardware such as a video projector. We use a green color to identify this scene representing the real-world setup throughout this paper. Virtual scene The second scene, the virtual scene, is created in order to compute the image to project over the real-world object for the augmentation (see Figure 1(c)). It consists of a 3D model of the real-world object Obj virtual , textured as desired, and a virtual camera Cam virtual . We use a red color to identify this virtual scene throughout this paper.

For a correct superimposition, first, concerning the calibration, the intrinsic parameters of the virtual camera must correspond to the intrinsic parameters of the projector. In virtual environments and video games, it is common practice to use the field of view (FOV) as the only intrinsic parameter of the virtual camera. As a projector can be considered as an inverse camera [START_REF] Moreno | Simple, accurate, and robust projector-camera calibration[END_REF], in our assignment, we simplify the intrinsic parameters of the virtual camera Cam virtual and the projector Proj real to the FOV. This allows the students to manipulate only this one unique, representative and intuitive parameter for the result of the calibration. Note again that the calibration is usually far more complex, and we simplify it for a better understanding.

Second, concerning the tracking, the transformation (position and orientation) of the virtual camera relative to the 3D model must correspond to the one between the real-world projector and the real-world object. We exploit the hierarchical organization of our two scenes in the scene graph. Indeed, the relative transformations between the scene graph nodes make it possible to simulate the tracking by copy-pasting the transformation T real between the projector Proj real and the representation of the real-world object Obj real to the transformation T virtual between the virtual camera Cam virtual relative to the 3D model Obj virtual (see again Figure 2).

Our assignment consists of two exercises that construct the aforementioned example step-by-step. In addition to distributing a detailed exercise sheet (provided in the additional files for review), we suggest the lecturer to advise the students directly to keep them at a similar pace for a collective experience. The first exercise consists in rendering a view of the 3D object on planar screens of different sizes and aspect ratios. The second exercise consists in creating the actual example, the projection of an augmentation onto a 3D object representing the real-world object.

Exercice 1: Rendering on planar virtual screens

For getting the students on track and familiar with the working environment in Unity, and also for overriding some default parameters that could hinder further comprehension, we designed a first exercise where a view of the 3D object is rendered on a virtual representation of a planar screen. To this end, the students create the virtual scene that first consists of the virtual 3D object Obj virtual , the virtual camera Cam virtual , and a main camera to visualize the scene. Then, the students create the scene representing the realworld setup, and they add a virtual representation of a real squared screen Screen real as a simple quad. This quad is then textured with the acquired image of the virtual camera Cam virtual . Note that for now, the rendering is done on a virtual representation of a planar screen, and there is not yet a virtual representation of the projector and the real-world object. The resulting scene graph of this first exercise is illustrated in Figure 3.

Note also that when rendering to a texture, Unity's default camera has a FOV of 60 degrees with a 1:1 aspect ratio. For this reason, the students start the exercise with a squared quad representing the screen. Then, when they succeeded to render the image to the squared screen (see Figure 4(a)), we ask them to modify the screen to a rectangular one by augmenting only its width without changing its height. This results in a deformation of the displayed image (see Figure 4(b)) which makes the students understand that the aspect ratio of the virtual camera Cam virtual used to generate the im- age of the screen must correspond to the aspect ratio of the screen Screen real . Consequently, we let the students adjust the aspect ratio of the virtual camera Cam virtual (see Figure 4(c)). Recall that the aspect ratio is one part of the so important intrinsic parameters to take into account in spatial augmented reality.

At the end of this first exercise, in order to prepare the second exercise, we ask the students to discard the quad representing the screen, but to maintain the virtual camera Cam virtual . This has the side effect that this virtual camera keeps a different aspect ratio than the default aspect ratios of virtual cameras and projectors in Unity3D which are the same -otherwise the calibration step of the second exercise, represented by making coincide the aspect ratios, would be trivial and the students could neglect its importance.

Exercice 2: Projection to augment a 3D object

The second exercise consists in extending the preceding observations to the projection on a real-world object. To this end, starting from the scene graph established in the first exercise, the students have to add a virtual representation of the object Obj real and the projector Proj real to the scene representing the real-world setup (Figure 5). The simulated projector Proj real projects the image acquired by the virtual camera Cam virtual , which is identical to the one used in the previous exercise. Note that it is essential that both the real-world object Obj real and the virtual object Obj virtual have the same geometry and the same scale. Concerning the material, we suggest to use a textured virtual object Obj virtual in the virtual scene, and an untextured diffuse object Obj real in the scene representing the real-world setup. In this way, in addition to the technical aspects of making understand the principal concepts of spatial augmented reality, this exercise provides a nice and concrete application, namely color restoration. Indeed, in ancient artifacts, color pigmentations may have disappeared due to weathering, and with spatial augmented reality, the original colors can be projected on the real artifact [RRL * 14].

As a first step towards a superimposition, the students are asked to manually position and orient the projector Proj real in Unity so that its projection covers the entire targeted surface of the object Obj real . As in real spatial augmented reality settings, the students see that the manual positioning is difficult and almost always results in an incorrect superimposition. Here, the tracking comes into play: the virtual camera Cam virtual of the virtual scene has to be placed exactly like the projector Proj real of the scene representing the real-world setup, relative to the respective object. Recall that in a scene graph, the transformations of the nodes are specified relative to their parent nodes. Consequently, as the virtual camera Cam virtual is a child node of the virtual object Obj virtual , and as the projector Proj real is a child node of the virtual representation of the real-world object Obj real , the tracking is done by copying the transformation T real to the transformation T virtual .

After this step, the superimposition is still not correct. Indeed, recall that after the first exercise, the virtual camera Cam virtual has a different aspect ratio than the projector, and thus different intrinsic parameters. As the intrinsic parameters have to coincide, the last step takes into account the calibration: it consists in modifying the aspect ratio of the virtual camera Cam virtual so that it corresponds to the one of the projector Proj real . After this, the students see that the superimposition is correct.

Even though the visual result would be identical, it is essential to note that the virtual camera Cam virtual has to be adjusted according to the projector Proj real , both for the calibration and the tracking. Indeed, in a physical spatial augmented reality setup, it is easy to manipulate the parameters of the virtual camera, whereas the projector has fixed intrinsic parameters. Similarly, the positioning of the real-world projector often has strong physical constraints, and so we rather modify the position and orientation of the virtual camera, and not the ones of the projector.

In addition to these two exercices, we suggest that the lecturer shows a demonstration of the real spatial augmented reality setting with the brought along physical projector and real-world BB8 character, including the calibration and tracking steps. In this way, the students can make the link between these practical steps in the real setting and the underlying theoretical concepts that they experienced in the assignment. We also suggest that the lecturer distributes this present paper that can foster further understandings and provide more insight, as we report below.

Preliminary evaluation

We designed the described practical assignment for a three-hour lecture to teach spatial augmented reality. It was part of a three-day elective course about augmented reality in general, called "Augmented Reality Engineer", that we are teaching for the second consecutive year at the ESTIA Institute of Technology. In order to validate the proposed methodology, we conducted a preliminary evaluation for the 24 students of the first time of our intervention. We asked the students to fill out an anonymous online questionnaire at the end of the three-day course. The participants (3F, 21M, age range from 22 to 24 years) were at master-degree level. They are knowledgeable in Computer Science and have all deliberately chosen to participate in the assignment.

In total, the students replied to five questions, consisting of three closed and two open ones. The three closed questions were asked via a Multiple Choice Questionnaire (MCQ) to qualify the truthfulness of the following three statements:

1. I understood why this practical assignment was part of the threeday course (options: not at all/just about/absolutely). 2. I enjoyed this practical assignment (options: not at all/a little bit/quite/absolutely). 3. I have retained some content from this practical assignment (options: yes/no).

For the first statement, among the three options, the students answered 0% "not at all", 12.5% "just about" and 87.5% "absolutely".

For the second statement, among the four options, the students answered 0% "not at all", 4,2% "a little bit", 12,5% "quite", and 79,2% "absolutely". For the third statement, the students answered 100% yes.

The two open questions, where the students could provide free answers in paragraph, were: 4. From my point of view, the main interest of this practical assignment was to ... 5. I propose to improve this practical assignment by ... We used a discourse analysis [GKL95] to analyse the corpus of the answers. It is a specialized method for discourses related to the Multiple Correspondence Analysis (MCA), followed by a hierarchical clustering analysis. For the first statement, we can distinguish seven semantic universes. The number of occurrences relative to each universe are: 8 for "discovering SAR", 6 for "practical assignment", 5 for "AR in a research project", 4 for "projection and mapping", 4 for "exploring cultural domain", 3 for "exploring Unity3D", 2 for "discovering an aspect of AR", and 1 for "Too cool!". Note that "projection and mapping" are always associated with "discovering SAR", and that the two main answers are exactly what the lecturers wanted to transmit: it is a "practical assignment" for "discovering SAR". The result of the discourse analysis is thus satisfying regarding our objectives. For the second statement, we can distinguish four semantic universes. The number of occurrences relative to each universe are: 21 for "nothing to change", 1 for "go faster!", 1 for "make it longer", and 1 for "provide a more useful example for augmented reality". Summing up, the results obtained with this summative assessment are considered as very good when regarding the closed questions. This is confirmed by the semantic universes given by the students to describe the main interest of the lecture.

When we taught the practical assignment for the second time, we also provided to our students a preliminary version of this present paper, and we gathered their remarks. It came out that the reading of the paper was very welcomed by the students, and some of them stated that it clarified even more some points from the assignment. Some students also claimed that they appreciated to see their practical assignment from a pedagogical point of view.

Conclusion and future work

In this paper, we presented a practical assignment to teach spatial augmented reality to large audiences. As we simulate the projection to an object in 3D, the assignment does not require any specific equipment such as a video projector for the students, while teaching the major concepts involved in spatial augmented reality. This is possible thanks to the hierarchical organization of the objects, the camera, and the projector in a scene graph. To our knowledge, our work is the first approach in this direction, and we hope that it will raise a discussion about teaching methods for spatial augmented reality in larger classrooms. Indeed, teaching spatial augmented reality becomes important nowadays: referring to the market and industrial needs, and according to the Emerging Technology Hype Cycle from Gartner5 , it is today timely to train engineers for augmented reality. This "new job" was first introduced in 2012 at the ISMAR Symposium on Mixed and Augmented Reality, and his/her role is to set up, monitor, and sometimes control the augmentations, just as light or sound technicians set-up, monitor and control lighting and sound [CCN * 12]. We presented our assignment as being tightly related to the use of Unity3D because of its widespread use, its graphical user interface combined with a high-level scripting language, and since it is freely available. However, our assignment can be tailored to other environments and languages, according to the desired student profiles, the level of difficulty, and available time. Indeed, we only rely on standard computer graphics bricks such as importing meshes and textures, multiplying transformation matrices, positioning cameras, rendering to textures, and shader programming. Hence, our assignment can also be implemented, for example, in C++ with OpenGL, according to the students' program.

In the future, according to our received feedbacks, we plan to extend our practical assignment to dynamic settings with moving projectors and/or objects. Dynamic settings introduce real-time constraints for the tracking: in our assignment, introducing an artificial latency for the tracking could simulate these constraints and provide maximum bounds for deriving the appropriate tracking solutions. We also plan to integrate interaction techniques so that the students can influence the augmentation. Moreover, we want to propose different surface materials for the real-world object in order to show the importance of reflection. We are convinced that the organization in a scene graph makes it comfortable to extend the assignment for our future plans.

  for industry, art [BCK * 04], cultural heritage [RRL * 14], and others [MZD * 06].

Figure 3 :

 3 Figure3: Scene graph used in the first exercise of the practical assignment for rendering on a virtual representation of a real planar screen. The aspect ratio of the screen must correspond to the aspect ratio of the virtual camera.

Figure 4 :Figure 5 :

 45 Figure 4: Exercice 1: Rendering of a virtual image on a screen represented by a quad. (a,c) The rendering is correct when taking into account the intrinsic parameters. (b) Otherwise, the rendering is deformed.

c 2018 The Author(s) Eurographics Proceedings c 2018 The Eurographics Association.

www.udemy.com, "Build an augmented reality android app in 1 hour"

www.instructables.com, "Augmented Reality Tutorial for Beginners With Vuforia and Unity

3D" 3 www.justapixel.co.uk, "How to make an AR app in 5 minutes with Unity and Vuforia" 4 www.vuforia.com

Technology Research Gartner Inc., www.gartner.com c 2018 The Author(s) Eurographics Proceedings c 2018 The Eurographics Association.

Acknowledgments

We thank the ESTIA students of "Promotion 2017" and "Promotion 2018" that have participated in the assignment. This work was partly supported by the ISAR project ANR-14-CE24-0013.