M Garcia

R Vergne

M.-A Farhat

P Bénard

C Noûs

J Thollot

Coherent Mark-based Stylization of 3D Scenes at the Compositing Stage

Keywords:

Density map

Stippling style

Figure 1: Thanks to our real-time and temporally coherent anchor point generation process, we produce a wide range of styles using different control maps and splats.

Introduction

Stylized rendering is increasingly popular both in video games and animated films. Dedicated non-photorealistic rendering (NPR) frameworks such as BNPR, LOLLIPOPshaders or Artineering are more commonly used by the artistic community and in production pipelines, but mimicking complex artistic styles remains challenging, especially for animated scenes. As stated by Bénard et al. [START_REF] Bénard | State-of-the-art report on temporal coherence for stylized animations[END_REF], perfectly and simultaneously preserving the im-pression that a stylized image is drawn on a flat canvas (i.e., flatness) while ensuring a strong correlation between the apparent motion field of the 3D scene and the motion of its stylized depiction (i.e., motion coherence) and minimizing abrupt changes (i.e., temporal continuity) during an animation is impossible. Previous approaches, either mark-based or texture-based, compromise between these contradictory constraints, and thus manage to cover different ranges of styles from discrete to continuous. Furthermore, these NPR frameworks are usually built on top of existing 3D rensubmitted to EUROGRAPHICS 2021. derers -originally designed to generate photorealistic imagesthat cannot easily be modified to handle a large range of styles.

In this paper, we present a method that takes advantage of both mark and texture-based approaches to significantly widen this range. We propose to use marks that are coherently and dynamically positioned using a procedural, texture-based approach, which makes their density and style easily controllable. Moreover, our method can easily be integrated in any rendering pipeline as it fully operates in screen-space at the compositing stage using only 3D information stored in G-buffers (Figure 1).

To ensure motion coherence, we generate anchor points that follow the object motion field. Similarly to 3D Voronoi cellular noise, we use an implicit grid that associates a set of neighboring local object positions with a single coherent 3D position. The flatness of the style is enforced both by rendering splats in 2D and by using a fractalization mechanism to maintain a quasi-constant screenspace density of anchor points, especially when zooming in or out. Finally, to prevent popping artifacts and to ensure a stronger temporal continuity, we present a splat opacity scheme related to the anchor points distance to the object silhouettes, and we provide a new order-independent transparency blending function that avoids the blending artifacts observed in previous work.

Our full pipeline runs on the GPU, in parallel for all pixels, allowing for real-time to interactive performance. Since our method works as a post-process, it can also be used offline, e.g., to stylize high-quality animations rendered with global illumination. By varying the splats content, controlling their attributes with various G-buffers, and tuning the blending function, we show that we can reproduce most previous mark-based and some texture-based styles, as well as novel ones that depart from previous stylization techniques.

Related Work

The survey of Bénard et al. [START_REF] Bénard | State-of-the-art report on temporal coherence for stylized animations[END_REF] distinguishes between two main families of approaches when stylizing color regions of 3D animations: texture-based and mark-based methods. Our work belongs to the latter but borrows key ideas from the former.

Texture-based approaches. In all these methods, the marks are embedded in an image which is either mapped over the entire screen or onto the 3D scene. In either case, to preserve the flatness of the marks, their size needs to remain quasi-constant on screen. Fully working in screen-space, Cunzi et al. [CTP * 03] introduce an infinite zoom mechanism, called fractalization, which consists in blending multiple versions of the original image with scaling factors and blending weights based on the distance to the camera. For 3D objects, dedicated mip-maps, called art-maps [PHWF01, SBB17] can be used to adapt the scale of the texture according to the distance to the camera, but the maximum depth range is restricted by the resolution of the largest texture. To handle arbitrary motion, Bénard et al. [START_REF] Bénard | Dynamic solid textures for real-time coherent stylization[END_REF] extend the fractalization algorithm in 3D, even though perspective distortion is only partially corrected. Between these full 2D or 3D techniques, patchbased approaches cover the projected 3D objects with a set of texture patches whose motion is restricted to rigid or affine transformations in screen-space [CDH06, BSM * 07, KYYL08] or UV space [CDLK19] while approximating the 3D scene motion. The flatness of the marks is well preserved, but it may result in noticeable sliding artifacts when the patch size grows; yet for small patches, it becomes similar to mark-based techniques.

Mark-based approaches. As originally introduced by Meier [Mei96], these consists in attaching 2D marks (e.g., stipples, hatches, or generic texture sprites) to anchor points distributed onto, or in the vicinity, of a 3D scene. These marks can either be digitally painted by an artist [Dan99, SSGS11], or the anchor point distribution can be automatically computed to cover the 3D objects at a given camera distance [Mei96]. To allow interactive exploration, the density of points further needs to be adapted to the viewing conditions to remain uniform (e.g., Poisson disk), avoiding both holes and mark aggregates.

Such a distribution can be pre-computed in object-space for a fixed number of camera distances and stored in a hierarchical datastructure [KMN * 99, CRL01, PFS03, CL06, USSK11]. At runtime, the appropriate levels are selected for the current viewpoint. By construction, the anchor points are coherent with the 3D scene, but their spacial distribution after projection may not be perfectly uniform, and the zoom range is limited by the depth of the hierarchy. To overcome these limitations, the points can be directly computed in screen-space [VBTS07, LSF10, DSZ17, USS * 18] and advected along the motion field of the 3D scene. Yet, to maintain a constant density, individual points need to be added or removed, leading to temporal discontinuities.

When a large number of marks are blended together, these temporal events are less noticeable, and the quality of the point distribution is also less critical. Drawing inspiration from sparse convolution noises [LLC * 10], a few many-marks methods have leveraged these properties. Kaplan and Cohen [KC05] and Bousseau et al. [START_REF] Bousseau | Interactive watercolor rendering with temporal coherence and abstraction[END_REF] use a pre-computed hierarchical 3D distribution of anchor points to produce a dynamic canvas and watercolor pigments respectively. Bénard et al. [BLV * 10] generate more complex procedural patterns using Gabor kernels as marks, and introduce a level-of-detail mechanism which blends between two power-of-two sets of random points with a blending weight specifically tailored to Gabor noise. This further reduces temporal discontinuities and can be implemented on the GPU in a geometry shader, albeit with a limited number of points per triangle (unless dynamic tessellation is used). Alternatively, Bleron et al. [BVHT18] produce similar patterns by filtering a much simpler 3D Voronoi cellular noise [Wor96] with screen-space spatially-varying filters, based on geometric data stored in a G-buffer. However, since the noise is restricted to the object interior, a complex inflation step is required if the stylization needs to extend outside object boundaries.

Our method generalizes these previous approaches by dynamically generating a motion-coherent 3D anchor point distribution, which is based on implicit Voronoi noise generated at the compositing stage from a G-buffer. It can thus be applied as a post-process regardless of geometric representation or animation technique. The anchor point density can significantly vary -from very dense to very sparse -depending on the intended style, while remaining quasi-constant in screen-space thanks to an extended fractalization algorithm applied to the point distribution.

Duplicated points filtering

Compute Shader Vertex Shader Vertex transform.

Fragment shading

Fragment Shader

Fragments blending

Fragment Shader

G-buffers

3D positions

Anchor points +Instanced geometry control maps processed by any screen-space filter Figure 2: Our two-step pipeline. First, a distribution of anchor points is generated from 3D positions stored in a G-buffer. Then a splat is instantiated for each anchor point, transformed, rasterized and shaded according to G-buffer data (normals, tangents, texture coordinates, etc.). Splat fragments are eventually composited together using a custom blended function.

General Pipeline

Our stylization pipeline is divided into two main steps summarized in Figure 2. Starting from object-space positions stored in a G-buffer, we first generate a distribution of 3D anchors points with associated weights. For each anchor point, we instantiate a 2D splat and store the list of splat fragments covering each screenspace pixel. Eventually, we color each image pixel using an orderindependent transparency algorithm and a controllable blending function.

Our post-processing pipeline makes use of input G-Buffers containing positions, normals, tangents, depths and texture coordinates. This data is used for computing splat anchors points, size, orientation and opacity. Additional maps might be used to give users more control over splat density, color, size and opacity.

Anchor Points Generation

Central to our approach is the generation of 3D points that are used to anchor 2D splats. Similarly to previous mark-based approaches, we make these points follow the motion field of the 3D object to ensure perfect motion coherence. Flatness perception is increased using a fractalization mechanism, and the temporal continuity of the anchor points is controlled by a weighting scheme that may act on the opacity or size of the corresponding splat. Our key contribution is to compute these points implicitly on the GPU at the compositing stage.

Points computation at a single scale

Our method takes as input a G-buffer in which each pixel x stores the 3D local coordinate position p(x) of an object surface. Inspired by 3D Voronoi cellular noise [Wor96], we generate one point per pixel by decomposing the space into an implicit grid, where each cell origins at c(x) = f loor(f . p(x)). The user-defined frequency parameter f modifies the size of the cells to control the number of generated points. We compute the pseudo-random points p i jk (x) in the cell containing p as well as all its direct neighbors, and select the closest one to p:

v(x) = arg min p i jk (x) p(x) -p i jk (x) , with: p i jk (x) = (c(x) + (i, j, k) + rand(c(x) + (i, j, k) ))/ f ,
where rand is a pseudo random function R 3 → [0, 1] 3 and the vector (i, j, k) ∈ ({-1; 0; 1}) 3 iterates over neighborhood cells. Instead of displaying the shortest distances, as in classic Voronoi noise, we store the closest points a(x) = M . v(x) projected from local object coordinates to clip space coordinates using the Model-View-Projection matrix M. Using the above formula, many pixels may potentially generate the same 3D position. We define A as the subset of anchor points positions with pairwise distinct elements:

A = {a(x) | ∀y = x, a(y) = a(x)}.
Surface projection. The generated anchor points may lie outside the object boundaries, especially when the grid frequency is low. This makes the anchor point distribution hard to control, and it is preferable that the points closely stick to the object surface. We thus introduce a simple yet efficient method to project the anchor points onto the surface. For each point ap ∈ A, we store the set of pixels from which it comes from:

Pa p = {x | a(x) = ap}.
The projected anchor point position a p is then computed as the weighted-average of the pixel positions x ∈ Pa p and associated depths z:

a p = ∑ x∈Pa p p(x) -vp (x, z) / ∑ x∈Pa p p(x) -vp ,
with p(x) -vp the distance between the object-space position stored in x and the 3D anchor point position vp.

Area weights. As G-buffers have finite resolution, special attention needs to be paid to avoid popping or flickering artifacts from frame to frame. The generated points may indeed appear or disappear, especially near silhouettes, due to unstable visibility information in the input buffers. To ensure a proper temporal continuity, we thus compute a weight for each anchor point, that will typically be used as an opacity factor to make stylized splats seamlessly appear and disappear during an animation.

Intuitively, regions containing strong depth discontinuities are more prone to popping, since a given anchor point will be generated by few pixels. As we initially generate one point per pixel, the number of pixels #Pa p generating the same anchor point ap ∈ A effectively measures the surface area that contributed to the generation of this point. For each anchor point, we thus compute its submitted to EUROGRAPHICS 2021. weight wa p as the ratio of the area covered by these pixels and the maximum screen-space area that can potentially generates a point with a given frequency f :

wa p = #Pa p d 2 f π , ( 1
)
where d f is the half diagonal length of a grid cell in screen-space.

We finally provide users with further control over wa p using a smooth step function (Hermite interpolation) parametrized by the upper bound value wmax ∈ [0, 1]:

w ap = 3 wa p wmax 2 -2 wa p wmax 3 .
This typically allows users to increase the global opacity of the splats, which is especially useful for styles where splats should not be blended together.

Point distribution fractalization

To increase the flatness of the final style, we need to keep the anchor point density as uniform as possible in screen-space. We compute such a point distribution implicitly using a fractalization mechanism similar in spirit to the algorithm proposed by Bénard et al. [START_REF] Bénard | Dynamic solid textures for real-time coherent stylization[END_REF] for solid textures. However, instead of blending several octaves of a given texture, we compute a user-controllable number of anchor point sets with associated weights for power-oftwo frequencies. In addition, unlike previous approaches, our frac-talization algorithm is driven by the gradient of the surface positions, which is more generic than the usual distance to the camera, since it captures both scaling effects due to camera motion in depth and distortions of deformable objects (Figure 3).

For a given position p at pixel x, we compute its reference fractalization level as:

lr(p(x)) = log 2 ( ∇p(x) ).
The associated frequency fr = f /2 lr thus increases when the gradient is low, which is the case on flat surfaces and when zooming in. According to the user-defined fractalization range n, we then create n fractalization levels on both sides of the reference level:

l k (p) = f loor(lr(p)) + k with k ∈ [-n; n].
For each level l k , we generate an anchor point ap using the algorithm described in Section 4.1 with grid frequency f k = f /2 l k . To ensure smooth transitions from one level to the next, we modify the anchor point weighting scheme. We first compute the Gaussianweighted distance between the current and reference level:

w k (p) = exp - l k (p) -lr(p) 2 2σ 2 ,
with σ empirically set to 0.4n in all our experiments. We then modify Equation 1 in order to include this distance and modulate the weight of the anchor point ap:

wa p = ∑ x∈Pa p w k (p(x)) d 2 f k π ∑ n j=-n w j (p(x))
.

Intuitively, we normalize all {w k (p)} k for a given p and divide each of them by their respective area coverage. Eventually, wa p is equal to the sum of normalized weights for each point p generating ap at level k.

Splat Blending

In Taking inspiration from previous work on commutative blending functions for order-independent transparency [START_REF] Mcguire | Weighted blended orderindependent transparency[END_REF], our method aims at making parts of the splats that are near to each other depthwise more transparent and blend them with respect to their distance to the "nearest" splat, defined below. Let us consider a per pixel list of N overlapping splat fragments [START_REF] Yang | Realtime concurrent linked list construction on the gpu[END_REF], each fragment i ∈ [1, N] is defined by a color u i , a transparency value α i and a normalized depth d i ∈ [0, 1]. Our key idea is to make fragments that are both close to the camera and opaque more visible. To this end, we first compute a reference depth d re f , which corresponds to the smallest alpha-weighted depth of the overlapping splats, and a reference alpha value α re f using the regular over function [START_REF] Porter | Compositing digital images[END_REF]:

     d re f = min i∈[1,N] (α i (d i -1) + 1) α re f = 1 - N ∏ i=1 (1 -α i ).
For each splat fragment, we then compute a visibility factor v i that depends on both the proximity to the reference depth d re f and alpha value α re f :

v i = 1 N 1 -min(max(d i -d re f , 0) (1 + ω α re f ), 1) α i ,
where ω ∈ [0, 1] is a global blending strength factor controlled by the user. This formula makes splat visibility decrease faster when ω increases as illustrated in Figure 6. For instance, when d re f = 0 and ω = 1, all splats with d i ≥ 0.5 will be invisible and will not contribute to the final blending. Notice that ω is weighted by α re f to avoid fully masking the nearest splat when it is semi-transparent.

From this visibility factor we compute new alpha values ensuring that the final transparency matches the reference α re f :

α i = v i ∑ N j=1 v j α re f .
Intuitively, this formula normalizes the visibility factors and assigns to each fragment the corresponding fraction of the reference transparency. The final color and opacity of a given pixel is then obtained with a weighted sum of the new transparency values:

u blend = ∑ n i=1 α i u i / ∑ n i=1 α i α blend = ∑ n
i=1 α i As our blending function does not rely on depth ordering, it allows splats to seamlessly move in front of each other over time. The final pixel color can also easily be blended with a background image using the over operator.

We compare in Figure 5 assuming that all splats belong to a single layer (as defined in the paper) and that the reference depth is the average distance of the splats to the camera. Similarly to McGuire and Bavoil, we cannot fully reproduce the behavior of the over operator, but our results are reasonably similar and we avoid the rigging artifacts noticeable at the boundary of the overlapping splats. The operator of Bruckner et al. does not seem well-suited for splats blending since it produces results that are quite different from the over operator. For low blending factors ω ∈ [0 : 1], the result is similar to an average, whereas for high ω value, splats that are far away from the reference depth tend to be masked, which may even give the impression that splats are swapped as shown in Figure 5 (the blue splat seems closer to the camera than the red splat). Figure 6 further compares our approach with the over operator on a rotating sphere.

Style Examples

We can produce a wide variety of styles depending on the chosen texture, size, orientation and opacity of the splats, and the density of anchor points. Brush splats can produce painterly renderings ranging from pointillism to impressionism, or more blended painting submitted to EUROGRAPHICS 2021. Figure 6: Comparison between our blending scheme and the over operator [START_REF] Porter | Compositing digital images[END_REF]. The left and right columns show two consecutive frames of a rotation around a sphere. Notice the popping artifacts occurring when one splat come across another using the over operator (first row); our depth-aware order-independent scheme prevents such temporal instabilities (second and third rows). In addition the blending strength parameter ω allows the user to control the perception of individual marks.

styles according to user-defined parameters. A very high number of small splats allows us to generate stippling, while several big splats may be combined to produce almost continuous textures. The user can control the rendering via several global parameters (summarized in Table 1), which can be further refined with screen-space control maps stored in G-buffers, as shown in Figure 1. As long as temporally coherent filters are used to generate such auxiliary maps, the anchor points generated by our method ensure temporally coherent results. Nevertheless, special care must be taken to avoid producing aliasing at each step of the pipeline. We usually apply some amount of smoothing (Gaussian blur) to each map to limit this problem.

Our method also applies to animated objects like articulated characters and deformable environments, as long as temporally coherent input positions are provided. In practice, we baked their rest pose positions into a texture and mapped it onto the object surface so that they get deformed during the animation. We include several animated objects in our results: a deforming blob and twisting torus, an articulated dancer and a procedurally generated ocean.

In the following, we present a set of results trying to reproduce standard styles found in the NPR literature, but also more experimental styles leveraging the variety of splats that we can use. Indeed, the splat itself may consist in a simple alpha mask but can also contain color and normal information or even a procedurally generated and animated content. The corresponding videos and additional examples are provided in the supplemental materials.

Painterly. Using a color buffer to define the color of the splats, we can produce painterly results similar to previous stroke-based rendering methods (Figure 7). The brush stroke used as the splat texture can be enhanced with a normal map, allowing for shading the strokes themselves. An orientation map can be used to align the brush strokes along, for instance, the surface normals, tangents or principal directions of curvature.

We eventually show a result combining two painterly styles. We used Blender to render G-buffers with high-quality lighting effects and combined a style with randomly oriented brushes for the cups and a pointillism style for the teapot and floor. All stylization attributes such as density, size and orientation are generated on the object surface as textures to ensure motion coherence.

Pen-and-ink. Figure 8 shows pen-and-ink examples. For the stippling style, we use a simple dot colored using a gray-scale Phong shading of the model as input color map. The splat density is inversely proportional to the intensity of the shading, and further refined using an input diffuse texture whose gradient depicts areas of high details on the model and in which the density is increased. Such a style raises two challenges: (1) it requires a very high number of splats (we used f = 85 with n = 3 fractalization levels, e.g., generating 131 101 points at the first frame of the stippled sphere animation); (2) it is prone to aliasing due to the small size of the dots. We therefore rendered the models at high resolution (1600 × 1600 pixels) to produce our final results.

Using the same input maps but taking advantage of an orientation map, we created a hatching style using a simple black segment as the splat. Cross-hatching is easily obtained by compositing two layers of hatches oriented orthogonally. Here again, a high number of splats is needed to ensure a detailed depiction of the model.

Textures. Using large splats containing a textured patch, we can produce continuous texture effects similar to spot noise [vW91]. We show in Figure 9 the use of a splat containing a triangle pattern. Depending on the size of the splat, we control the size and precision of the resulting texture. To better depict visual details of the underlying 3D model, we use the model color for each fragment of the splat instead of a single color per splat, i.e., the color at the anchor point position. We also show a more abstract style obtained using a splat containing bubbles, with a very low frequency of anchor points, a simple blue shading and a constant orientation, which increases the abstraction effect. (middle) an articulated dancer is stylized coherently by backing its rest pose positions into a texture and mapping it back on its surface; (bottom) the G-buffers can be generated offline to obtain complex lighting effects in the color map.

Procedural splats. Since our pipeline works as a post-process on the GPU, we can directly compute the splat content online using any procedural generation approach. We show in Figure 10 (top) a Gabor noise example producing results inspired by those of Bénard et al. [BLV * 10]. Each splat is a simple Gabor kernel randomly oriented using a 3D gradient noise but still drawn in 2D. For this particular example, we changed the blending function to the average sum of the splats to respect the standard Gabor noise formulation. We also produced a watercolor example following Bénard Finally we show an animated fiber style where each splat is a simple implicit function based on the distance to a segment. Each segment is deformed by a sine wave whose frequency and phase are randomly selected using 3D gradient noises as input. Animating these parameters over time leads to coherent fiber motion as shown in Figure 10 (bottom) and in the supplemental video.

Implementation

Our method fully runs on the GPU using OpenGL 4.3 inside the node-based compositing system "Gratin" [START_REF] Vergne R | Designing gratin, a gpu-tailored nodebased system[END_REF]. The anchor point generation algorithm described in Section 4 is implemented with two Compute shaders. For every pixel of the input G-buffer, the first shader generates 2n + 1 anchor points, with n the number of fractalization levels. Each of these points is stored at its corresponding screen-space position inside a per-pixel linked list, using atomic operations on integer counters. Ideally, we would have liked to avoid storing duplicated points by filtering them by their depth, but current GPU architectures do not provide atomic operations on 32 bits floating point buffers. Alternatively, we use a second shader that processes each linked list in parallel, computing the averaged projected position of the duplicated anchor points and their associated area weights.

The generated points can then be processed with a regular OpenGL pipeline using Vertex and Fragment shaders. The simplest approach consists in emitting as many GL_POINTS as anchor points and drawing each of them as a 2D sprite centered at their projected position. However, it offers restricted stylization options: it only allows global control over the point size and texture. A more generic approach leverages OpenGL instancing capability to draw any 2D geometry per anchor point. Every vertex of each instance can then be freely transformed in the Vertex shader according to its corresponding anchor point (identified by gl_InstanceID), but also any additional information read from the input G-buffer, such as the underlying surface normal. The splat blending method detailed in Section 5 is implemented as a post-processing step using a standard order-independent transparency technique [START_REF] Yang | Realtime concurrent linked list construction on the gpu[END_REF]. Instead of simply compositing the output fragments using OpenGL blending modes, we store their color, alpha and depth in a per-pixel linked list inside an offscreen buffer. During a final screen-space pass, for each pixel in parallel, the reference depth is first determined. The visibility of all fragments for a given pixel is then computed relatively to this reference and their own opacity; all fragments are eventually blended together using this quantity.

As demonstrated in Section 6, our method can be applied to 3D scenes composed of several objects as well as skinned characters (teapot and dancer examples in Figure 7). To handle multiple objects, we segment the scene with an ID Buffer and use one Model-View-Projection matrix per object to generate the anchor points. For skinned characters, we bake their rest pose positions into textures and wrap them around the objects. The anchor points are then generated by sampling those textures (a single matrix is thus needed); their position is computed as the average of the screenspace positions of the pixels generating the same anchor points as described in Section 4.1.

In Figure 11 (top) we show the performance achieved by this implementation on a Nvidia Geforce RTX 2070 graphic card, for three constant point densities and two output image resolutions (1K and 4K), using 2D splats whose size ranges from 10 × 10 to 400 × 400 pixels. In our experimentation, the three point densities were used for different artistic styles: High point densities (yellow curves) submitted to EUROGRAPHICS 2021.

for stippling and hatching styles (see Figure 8), moderate densities (red curves) for painterly styles (see Figure 7) and low point densities (blue curves) for more abstract styles (see Figure 9). The same overall trend is observed for both output image resolutions with an expected ×4 difference factor for low point densities (blue curves) or small splat sizes. When the splat size or density significantly increases, this difference is progressively reduced to below a ×2 factor since many splats overlap in screen-space. Real-time to interactive performances are maintained for 1K outputs as long as reasonable splat sizes and densities are used, which is sufficient for style design and animation preview. Final high-quality rendering at a 2K resolution can take up to a few seconds.

In Figure 11 (bottom), we show the GPU memory footprint of this implementation for a 1K output image resolution. As expected, the memory usage evolves in a quadratic fashion with respect to splat size with a difference factor rising from ×18 to ×200 for high point densities. This trend is due to the linked-lists of splat fragments being stored at each pixel of the image as required by our order-independent transparency blending algorithm (the memory footprint of the anchor points list is constant for a given splat size). For high point densities and large splats (above 100x100 px), the memory footprint exceeds 12 GB, which is above the storage capacity of most modern GPUs. However, in practice, we did not encounter any style that requires such an extreme configuration. Note that the GPU memory of the linked-lists must be pre-allocated anyway; our implementation checks that the available memory is not exceeded. Beyond this limit, points and fragments are omitted and our algorithm cannot ensure that popping artifacts will not occur.

Evaluation

The evaluation of Non-Photorealistic Rendering techniques is a challenge in itself, especially when dealing with animations since no ground truth or perfect solution exists. Please note that we do not aim to evaluate the final "look" of the stylized renders, as this is an overly subjective assessment and largely depends on the artistic expertise of the user who produced those results. Instead, we propose to discuss and clarify the positioning of our method with respect to previous mark-based approaches. To conduct this comparison as fairly as possible, we first present a set of criteria that, in our opinion, every mark-based method should ensure. We then detail how each class of mark-based methods behaves with respect to these criteria and compare them to our approach, as summarized in Table 2. We use the following criteria for this comparison:

1. Flatness: give the impression that the image is drawn on a flat canvas; 2. Motion coherence: ensure a strong correlation between the motion of the marks and the apparent motion of the 3D object; 3. Temporal continuity: minimize abrupt changes of the marks (e.g., popping and flickering); 4. Ease of integration: ensure that the method easily fits into existing rendering pipelines; 5. Range of style: cover a wide range of styles; 6. Dynamic behavior: easily adapt to animated and deformable objects.

The first three criteria, proposed by Bénard et al. [START_REF] Bénard | State-of-the-art report on temporal coherence for stylized animations[END_REF], apply to any stylization method that addresses the temporal coherence 4 0 0 × 4 0 0 1 0 0 × 1 0 0 8 0 × 8 0 6 0 × 6 0 4 0 × 4 0 2 0 × 2 0 1 0 × 1 0 1000 ( f=4.9) 10000 ( f=15.9) 100000 ( f=51. problem. The other three measure the versatility of such a method. As introduced in Section 2, we divide mark-based stylization methods into four categories according to their sampling approach:

• Static object-space: pre-computed 3D distribution,

• Dynamic object-space: dynamic distribution computed over the 3D geometry,

+ ++ +- - +- - Ours + ++ +- + + ++
Static object-space approaches mostly suffer from flatness issues as marks cannot be added or removed when the camera viewpoint changes, producing holes or aggregates. Dynamic objectspace methods better preserve flatness through complex level of detail (LOD) mechanisms, but those require heavy pre-computations. Furthermore, these mechanisms are only computed in a discrete fashion, which leads to temporal instabilities during transitions, and, as these methods rely directly on the geometry to sample anchor points, their integration into existing pipelines is difficult. Dynamic image-space methods can perfectly satisfy the flatness criteria, but they require an accurate motion field of the scene to preserve motion coherence, and suffer from popping artifacts due to the sudden appearance/disappearance of anchor points. Moreover, most of those approaches have difficulty in dealing with a very high number of points, hence limiting the range of style that they can handle. Finally, many-mark methods use continuous LOD mechanisms that require many anchor points to ensure temporal continuity and thus restrict them to specific styles. In addition, they also rely on the 3D geometry for point sampling, which makes them hard to integrate in existing pipelines.

Similarly to all dynamic image-space methods, our approach provides both flatness and motion coherence by construction, and can only easily process the visible geometry, which prevents, for instance, strokes appearing from hidden parts of the 3D scene. However, unlike previous work, our adaptive opacity factor combined with our continuous LOD mechanism smooth the appearance (resp. disappearance) of anchor points at disocclusions (resp. occlusions), and our blending operator further enforces temporal continuity. Please refer to the accompanying video for a comparison with the method of Lu et al. [LSF10] that clearly illustrates the benefits of these new components. Also note that this blending scheme could be used in previous methods to limit the popping artifacts due to the mark drawing order.

Since our method entirely works at the post-processing stage, it can be directly integrated into any existing rendering pipelines. As it is based on G-Buffers and 2D textures, it is sensitive to noise and aliasing present in these inputs, although this can be largely mitigated by pre-filtering. Our anchor point generation technique itself can produce aliasing artifacts at high frequencies, especially when the projected cell area of the virtual grid is lower than the pixel size. This limitation only occurs for very high frequency styles like stippling or hatching. From our tests, we ascertain that these artifacts are resolved by super-sampling, but this solution is hardly practical as it requires to generate (s + 2) 2 anchor points per pixel (with s the super-sampling factor), which quickly overloads the GPU capacity.

Finally our method covers the range of styles of most previous mark-based methods, from sparse stroke-based styles to manymarks continuous styles. All the computation is done at runtime in a procedural fashion allowing for a fully dynamic approach.

Conclusions and Perspectives

We introduced a new stylization technique based on an implicit and temporally coherent generation of 2D marks. Our approach fully operates at the compositing stage, making it suitable for both realtime and offline pipelines, as demonstrated with the teapot global illumination scene. We have presented stylization results where 3D models are rotating or deforming while preserving motion coherence and temporal continuity, thanks to a new order-independent blending function, a default opacity policy based on a novel coverage criteria, and an extended fractalization mechanism.

Thanks to the freedom offered by the choice of the splat content and the control of its attributes via G-buffers, our method can already produce an unprecedented range of different styles. Yet we would like to investigate further the use of procedurally generated splats and their animation. For example, a Geometry shader could be used to generate more complex splat geometry for each anchor point, e.g., to create curved strokes following the direction of maximum curvature of the underlying 3D object. Moreover implicit splats could even make use of 3D content to obtain volumetric effects, if an adapted blending operation is defined. Besides, we would also like to investigate alternative ways to maintain temporal continuity when splats appear and disappear. Instead of changing their opacity, we could for instance modify their size, or explore different blending schemes in other color spaces.

  submitted to EUROGRAPHICS 2021.

Figure 3 :

 3 Figure 3: Illustration of our fractalization mechanism when zooming in (top) or stretching a sphere (bottom). With fractalization (right side of each image), the density of the generated points remains constant in screen-space, preventing the creation of unwanted holes in the resulting stylization.

p with wa p frame t frame t + 1 Figure 4 :

 14 Figure4: Illustration of our anchor point opacity scheme. The left and right columns show two consecutive frames of a rotation around a sphere. When no opacity scheme is applied (first row), splats suddenly appear at the silhouette of the sphere, whereas our area weights smoothen the appearance of newly visible splats improving temporal continuity (second row).

Figure 5 :

 5 Figure 5: Toy blending example consisting in three circular splats at a constant depth with a smooth radial transparency falloff along their edge and the following global parameters: [u 1 = (0, 0, 1), α 1 = 0.8, d 1 = 0.5], [u 2 = (1, 0, 0), α 2 = 0.6, d 2 = 0], [u 3 = (0, 1, 0), α 3 = 0.4, d 3 = 1]. The accompanying video shows interactive variations of these parameters.

  , on a toy example composed of three circular splats, the results obtained from three different approaches and our method. First, we show the asymmetric over operator. Second, we show the commutative blending function of McGuire and Bavoil [MB13] (Equ. 6 & 10 in the paper, where the depth is further adjusted by a factor γ). Third, we show the distance-based blending operator of Bruckner et al. [BRV * 10] (Equ. 1 & 2 in the paper)

  submitted to EUROGRAPHICS 2021.

Figure 7 :

 7 Figure 7: Painterly style: (top) a large variety of styles is achieved by changing the splat texture and using various orientation maps;(middle) an articulated dancer is stylized coherently by backing its rest pose positions into a texture and mapping it back on its surface; (bottom) the G-buffers can be generated offline to obtain complex lighting effects in the color map.

Figure 8 :

 8 Figure 8: Pen-and-ink: (grom top to bottom) stippling, hatching following the principal direction of curvature, cross hatching, and hatching following the bi-tangent of an animated ocean.

Figure 9 :

 9 Figure 9: Large and content-rich splats allow to create complex textures.

Figure 10 :

 10 Figure 10: Procedural splats: (top) dynamic Gabor noise used to generate a temporally coherent watercolor style, (bottom) implicit animated fiber splat.

Figure 11 :

 11 Figure 11: Rendering time (top) and GPU memory footprint (bottom) with respect to the number of anchor points, 2D splat size and output image resolution (logarithmic scale on both axes).

  this work, we assume that splats are, by default, billboard images that always face the camera. Without special treatment, when the input surface or the camera moves, certain splats may suddenly be drawn over others from one frame to the next causing popping artifacts. Since the over blending function [Wal81, PD84] is asymmetric, it does not solve this issue, even when the splats are not fully opaque. Instead, we need to take into account the depth of each splat to ensure temporally coherent compositing, in the same spirit as the blurred depth test ofLuft et al. [LD06] or the hybrid visibility of Bruckner et al. [BRV

* 10]

. We thus propose a new formula for blending splats together which accounts for their relative distance in depth.

Table 1 :

 1 Global parameters of our pipeline. Users can tune five parameters allowing them to create various styles as well as variations of the same style.

	Notation	Range	Description
	f	[0 : +∞]	Sampling frequency
	s d	[0 : 1]	Splat size (scaling factor)
	wmax	[0 : 1]	Anchor point maximum opacity
	ω	[0 : 1]	Blending strength
	n	[[0; +∞]]	Number of fractalization levels

Table 2 :

 2 Mark-Based stylization approaches comparison.

				Motion	Temporal	Ease of	Range Dynamic
			Flatness coherence continuity integration of style behavior
	Static object-space	e.g., [Mei96]	--	++	+-	-	+-	--
	Dynamic object-space e.g., [CRL01]	+	++	+	-	+-	-
	Dynamic image-space e.g., [LSF10]	++	+-	-	++	-	+
	Many-marks	e.g., [KC05]

submitted to EUROGRAPHICS 2021.

Acknowledgements

We would like to thank Maxime Isnel, Laurence Boissieux and Sunrise Wang for their valuable discussions, suggestions, and assistance with this work, and the reviewers for their constructive comments. The following models come from Sketchfab under Common Creative Attribution license: Wiz Biz by tommonster, Shiny Fish by GenEugene, Mech Drone by Willy Decarpentrie.