Pascal Barla

Belen Masia

Tobias Ritschel

Pietro Gori

Saïd Ladjal

Yann Gousseau

Alasdair Newson

Christophe Kervazo

Kiwon Um

Amal Dev Parakkat

Isabelle Bloch

Henri Maître

Michel Roux

Arthur Ouaknine

Robin Kips

Nicolas Gonthier

Xu Yao

Emanuele Dalsasso

Giammarco Mateus Riva

La Barbera

Matthis Maillard

Raphaël Achddou

Nicolas Cherel

Pierrick Chatillon

Antoine Habis

Erwan Dereure

Rebeca Vetil

Rassim Hadjeres

Carlo Alberto

Florence Barbano

Zoé Carton

Berenger

Gilles Laurent

Malik Boughida

Thibaud Lambert

Théo Tonat

Christophe Lino

Adrien Kaiser

Hélène Legrand

Thibault Lescoat

Sylvain Rousseau

Corentin Mercier

Arthur Roullier

Les Lyonnais

Jérémy Levallois

Romain Rou

Rosalie Martin

First, I would like to thank the reviewers,

interpolation scheme as a navigation means. We also propose a novel approach for SVBRDF mipmapping which preserves material appearance under varying view distances and lighting conditions. As a result, we obtain a drop-in replacement for standard material mipmapping, o ering a signi cant improvement in appearance preservation while still boiling down to a single per-pixel mipmap texture fetch. These operators have been experimentally validated on a large dataset of examples.

Overall, our proposed methods allow for interpolating materials in the canonical space of textures as well as along the downscaling pyramid for preserving and exploring appearance.

Résumé

Le rendu inspiré par la physique est devenu une norme pour le rendu de matériaux numériques dans les industries créatives, telles que les jeux vidéo, les e ets spéciaux, la conception de produits et l'architecture. Il permet aux développeurs et aux artistes de créer et de partager des matériaux photoréalistes prêts à l'emploi entre une grande variété d'applications.

Dans ce contexte, les surfaces 3D sont mises en correspondance avec un espace de texture 2D où leurs fonctions de distribution de ré ectance bidirectionnelle variant dans l'espace sont encodées sous la forme d'un ensemble d'images bitmap appelées cartes PBR, lues e cacement au cours du rendu. Ces cartes représentent des quantités physiques interprétables tout en permettant la reproduction d'un large éventail d'apparences de matériaux. Elles peuvent être reconstruites à partir de photographies du monde réel ou générées de manière procédurale.

Malheureusement, ces deux approches de création de matériaux PBR nécessitent des compétences avancées et un temps considérable pour modéliser des matériaux convaincants destinés à être utilisés par des moteurs de rendu photoréalistes. De plus, bien que ces cartes soient encodées dans le même espace bidimensionnel de pixels, celles-ci décrivent des quantités de nature hétérogènes et à des échelles di érentes, et qui sont en partie corrélées. L'information décrite dans ces cartes est de nature géométrique pour la hauteur et la normale, statistique pour la rugosité, ou encore colorimétrique pour l'albédo. La rugosité modélise la distribution de normales des microfacettes, dont le support dépend de la normale dé nissant l'espace tangent, lui-même dé nit par la position dans la carte de hauteur. Cette description du matériau permet des rendus rapides mais empêche l'utilisation d'outils de traitement d'images RGB conjointement sur les cartes, pour des applications d'interpolation ou de ltrage notamment.

Dans cette thèse, nous explorons des opérateurs de morphing et de niveau de détail e caces pour résoudre les di cultés susmentionnées. Nous proposons un nouvel opérateur de morphing permettant de créer de nouveaux matériaux en mélangeant simplement deux matériaux existants tout en préservant leurs caractéristiques dominantes tout au long de l'interpolation. Cet opérateur permet d'explorer l'espace des matériaux possibles en utilisant des exemples comme ancres et notre schéma d'interpolation comme moyen de navigation. Nous proposons également une nouvelle approche pour le mipmapping SVBRDF qui préserve l'apparence des matériaux sous des conditions de vue et d'éclairage variables. Ainsi, nous substituons simplement le mipmapping standard de matériaux en o rant une amélioration signi cative de la préservation de l'apparence, tout en gardant un unique accès texture par pixel. Ces opérateurs ont été validés expérimentalement au travers d'un grand nombre d'exemples.

Globalement, les méthodes que nous proposons permettent d'interpoler les matériaux dans l'espace canonique des textures ainsi que le long de la pyramide de réduction d'échelle pour préserver et explorer l'apparence.

I.1 Modeling appearance

Computer Graphics is a discipline at the foundation of a growing number of elds, including Computer Assisted Design, Biomedical Imaging, Visual E ects for the Movie Industry, Augmented and Virtual Reality, to cite a few. It relies on a virtual scene representation, where computational models allow for the simulation of natural physics phenomena, such as uid simulation or light transport. The latter dictates the rules of the light interacting in the scene, which ultimately reaches the virtual camera sensor. It results in synthesized images which became, with time, indistinguishable from real photographs (see Fig. I.1,left).

The process of synthesizing an image, called rendering, leverages virtual scenes modeled by surfaces (or volumes) commonly described by triangle meshes. Other scene representations exist, such as voxels, implicit surfaces, or point clouds. However, triangles are well-suited to the graphics pipeline, where they are projected into the image plane in a massively parallel fashion. A triangle mesh acts as a compact and nite representation for surfaces where the light interaction with matter is solved inside the triangle's extent. This phenomenon is responsible for the surface appearance, called material, which depends on the lighting conditions and the viewpoint, as well as material properties (such as color or ne geometry). Fig. I.2 illustrates this modelisation. Instead of de ning these properties for each point of the manifold, they are de ned inside bidimensional images which are then mapped onto the surface. It allows considerable memory savings because images o er compact representations, in comparison with storing all attributes at each point of a high-resolution geometry. Storing ne geometric details in images rather than on the mesh also reduces the size of the asset. This process is commonly called normal mapping and is achieved by baking the triangle mesh's details into a texture (see Fig. I.3). In addition, image processing techniques have been studied for decades and can be easily manipulated inside a wide range of softwares. Finally, this decoupling of coarse geometry and appearance allows reproducing di erent shapes with the same look and feel (see Fig. I.1, where each sphere on the right illustrates materials that have been used for rendering an object of the scene).

I.1.1 Materials in Computer Graphics

To achieve photorealism, light transport equations rely on powerful material representations to describe the interaction between light and matter at physical interfaces. Previous models have been relying on simple bidimensional color, resp. normal, textures encoding di use albedo, resp. small scale geometry, at each point On the left, the original mesh is shown, containing as much as 4 million vertices. In the middle, the mesh is simpli ed to 500 triangles which encode the coarse shape. On the right, the normal map is added onto the low resolution mesh to retrieve the details lost during simpli cation. Credits: Wikipedia. of the surface. However, the emergence of physically-based rendering, or PBR, and its adoption in the graphics industry [START_REF] Burley | Practical physically-based shading in lm and game production[END_REF] enabled richer material descriptions and stunning visuals. This description enables modular work ows and eases the work of artists working collaboratively on the same virtual scene. Indeed, decoupling an object's appearance from its coarse geometry allows for iterating on one or the other task independently. Because of its strong inspiration from physics, this material representation behaves well with physical lights and eases the task of lighting artists for camera and light control. Hence the work ow allows artists to collaborate simultaneously on the di erent tasks in the scene rendering pipeline, especially when it comes to modeling, texturing, rigging, animation, lighting, and rendering (see Fig. I.4). The interactive feedback provided to the artists is key for allowing them to e ciently design and explore the appearance of the scene.

I.1.2 Material standards in the industry

Material standards have been proposed to preserve the appearance of materials across multiple applications. A standard material model is equipped with color (through di use albedo and specular color), as well as geometric information at multiple scales (surface roughness, local normal vector, displacement). Other speci c parameters encode visual variations found on car paint (clearcoat color and roughness), brushed metals (anisotropy angle and strength), cloth or fabric (sheen color and roughness), and skin (subsurface color and strength). These attributes represent interpretable physically-based quantities while modeling good and versatile approximations of real-world materials for creative industries. In this thesis, we focus on lightweight material representations which are compatible with real-time applications. However, the techniques and insights presented in the document can be adapted to more complex material models for o ine rendering.

Many industrial standards for PBR materials have been proposed during the past ten years. They can be divided into two main categories: the ones aiming at realtime, interactive applications, and the others for heavier, production renderings. The former describes several material maps to be given as input to existing closedform BRDF formulas, along with the range of their parameters. The latter provides richer tools for modeling appearance by allowing combinations of BRDF layers as well as custom BRDF sampling and evaluation functions.

The material standards for real-time renderings are often principled, in the sense that they should follow artistic considerations rather than being strictly physically based. Some of these principles can be found in existing standards: intuitive parameters rather than physical ones, with values ranging from zero to one, and with robustness regarding physical plausibility. Multiple standards compatible with real-time rendering exist, such as Disney "Principled" BRDF [START_REF] Burley | Practical physically-based shading in lm and game production[END_REF], Unreal Engine 4 [START_REF] Karis | Physically based shading in theory and practice[END_REF], Frostibe PBR [START_REF] Lagarde | Moving frostbite to physically based rendering[END_REF] or glTF PBR (Khronos Group, 2021).

When dealing with heavier work ows requiring complex rendering e ects, standards were proposed to tackle advanced rendering elements description such as materials, lights, displacement, and pattern generation. Open Shading Language (OSL) for instance is a language for programmable shading developed by Sony Pictures Imageworks and is the de facto standard shading language for VFX and animated features. The OSL shader describes radiance closures, which are evalu-ated or sampled in the context of a physically-based renderer that supports ray tracing and global illumination. MaterialX is another open standard developed by Industrial Light & Magic for representing materials and is a hosted project of the Academy Software Foundation. Both OSL and MaterialX have been used in feature lms and real-time experiences. They allow for platform-independent description and exchange across applications and renderers. Since real-time applications can bene t from these descriptions, a conversion process allows complex e ects to be translated to simpler, more e cient real-time implementations, allowing for their use in a real-time context. This is also the case of Nvidia Material De nition Language (MDL) which describes the appearance of scene elements for industrial and real-time rendering engines.

I.2 Challenges

Materials are often described as a set of textures controlling the parameters of a given re ectance model. The advantage of having such texture-based representation resides in its compactness, its e ciency at render time, and the wide availability of softwares for manipulating them. 3D shapes in comparison require handling the mesh connectivity with varying numbers of neighbors and have no default orientations. Textures bene t from a large body of work on image processing techniques. Filtering methods are well suited for textures only if averaging these quantities makes sense. The information is de ned in a canonical domain R 2 and can have useful properties such as tileability. This means that the texture de ned in [0, 1] × [0, 1] in nitely tiles the plane in both directions, allowing it to cover large surfaces with a compact examplar. The model decouples di erent components of the appearance of the material, such as color from geometry. The latter is encoded at multiple scales which allows artists to design them independently, at the macro, meso, and micro-scale levels.

However, the stack of textures used to represent a material cannot simply be dealt with as a generalisation of RGB textures into a higher dimensional space. The multiscale geometry encoded into displacement, normal and roughness maps often exhibits strong correlations. The nature of parameters varies from simple RGB color, to XYZ components, distribution parameters, or even elevation coe cients. Texture morphing for instance becomes non-trivial in higher dimensions than RGB, with heterogeneous content. For geometry encoding, three distinct scales are used, which are only t to a range of distance of observation. Unfortunately, handling this geometry encoding requires the derivation of a mathematical framework for jointly ltering geometric maps. The maps are given as input to a re ectance function which has a highly non-linear relationship with some of the quantities. The maps are only visualized through the re ectance distribution, but never as is.

Material graph in

I.2.1 Material Design and Aquisition

Material design softwares have been developed to ease material creation and manipulation. They rely on classical RGB image processing techniques, such as procedural mask creation, thresholding, or warping for instance. They allow for the creation of detailed materials with controllable parameters, which can be easily adjusted, such as the color, shape, or size of the patterns. Procedural creation relies on a non-destructive work ow, which enables trial-and-error design, and control over the material graph. However, such softwares are far from easy to master for artists. Working on a single procedural material, even if the process leads to many variations of the same base material, can take multiple days of work. An example of a procedural graph that outputs material maps is illustrated on top in Fig. I.5.

Another group of techniques for creating materials rely on sample acquisition (Fig. I.5, bottom). A typical setup involves a distribution of light sources across the hemisphere, and cameras to record the outgoing radiance of a material sample. Robotic arms can replace a xed light or camera rig and require setting the path for material capture. Lightweight techniques allow users to take a single image or a set of images to feed an optimization framework to recover the material maps. These methods always rely on the availability of a material sample, e cient capture and material encoding, as well as precise camera capture and light conditions. Outdoor material capture requires special pieces of equipment (including a camera, lenses, color checker, lters, spotlights, etc.), cannot deal with thin or transparent objects, and can be achieved only if the weather permits it. As a result, photogrammetry pipelines focusing on material capture are not common in digital content creation industries.

Learning-based approaches emerged as an e cient alternative for both material creation and manipulation. SVBRDF maps estimation from photographs has seen a rise in quality, despite the complexity of the ill-posed inverse problem. Exploring latent representations has become possible, but the corresponding materials fail to reach the quality of handcrafted examples. Deep Learning approaches su er from a lack of widely available data, as well as augmentation techniques for dataset ampli cations.

I.3 Technical Background

I.3.1 Radiometric quantities

Modeling appearance starts with a de nition of the physical quantities used for computing light transport simulations. The scienti c eld studying the measurement of electromagnetic radiation, including visible light, is called Radiometry.

When restricted to the study of electromagnetic radiation in the visible spectrum, in terms of its perception by the human visual system, it becomes Photometry. In this section, we describe the necessary radiometric terms required for de ning re ectance.

Photons are responsible for image formation at the level of the virtual camera sensor. A measure of their energy is called Radiant Energy, is measured in Joules [J] and is written Q:

𝑄 = ℎ𝑐 𝜆 [𝐽 ],
where h is Planck's constant, c is the speed of light in a vacuum and 𝜆 is the wavelength of the photon. Because of the photons' speed, we de ne the energy owing through a surface or a volume per unit of time, which is more practical. This energy is called Radiant Power or Flux, is measured in Watts [W] and is indicated by Φ:

Φ = 𝑑𝑄 𝑑𝑡 [J / s = W],
Because we are interested in studying light-matter interactions at the level of interfaces, we also de ne the ux owing per unit of surface area, called Radiant Flux Area Density, measured in [W/𝑚 2 ] and written 𝑢:

𝑢 = 𝑑Φ 𝑑𝐴 [W/𝑚 2 ],
Two di erent terms have been selected to describe the ow of energy either going toward the surface, called Irradiance (E), or leaving the surface, called Radiosity (B). References to these terms are often found in the Computer Graphics literature.

Another appropriate perspective on the ux relies on its quanti cation per unit solid angle (instead of a unit of surface area), called Radiant Intensity 𝐼 , which de nes the radiant energy leaving a point in the direction Φ per unit solid angle, measured in [W/sr]:

𝐼 = 𝑑Φ 𝑑𝜔 [𝑊 /𝑠𝑟 ],
Finally, the radiometric quantity used for most of the light transport computations is called Radiance 𝐿. It allows us to distinguish the directional distribution of power (as opposed to Irradiance or Radiosity) and is de ned as the radiant ux per unit projected area (hence the cosine term) per unit solid angle. It is written:

𝐿 = 𝑑 2 Φ 𝑑𝜔𝑑𝐴𝑐𝑜𝑠𝜃 [W • 𝑚 -2 • sr -1 ]
As mentioned by [START_REF] Pharr | Physically based rendering: From theory to implementation[END_REF], this quantity allows for the derivation of previously de ned ones, by integrating over the right domain. Moreover, it has the nice property of remaining constant along rays through empty space. This is often assumed when dealing with scenes only consisting of surfaces, and no participating media (such as clouds or troubled water).

Last, BRDF stands for Bidirectional Re ectance Distribution Function. It models the behavior of light re ected from the surface of opaque materials. It produces a wide variety of appearances and helps to visually enrich virtual scenes. We illustrate its parameterization in Fig. I.6 and discuss the motivation behind it in the next section. It is formally de ned as the ratio between the re ected radiance of a surface and the irradiance that caused that re ection. It is a function of four variables: the polar angles of incident and outgoing light, has units of inverse steradians, and is written:

𝑓 𝑟 (𝜔 𝑖 , 𝜔 𝑜 ) = 𝑓 𝑟 (𝜃 𝑖 , 𝜙 𝑖 , 𝜃 𝑜 , 𝜙 𝑜 ) = 𝑑𝐿 𝑜 (𝜔 𝑜 ) 𝑑𝐸 𝑖 (𝜔 𝑖 ) [𝑠𝑟 -1 ]

I.3.2 Re ectance Representations

Re ectance characterizes the behavior of light re ected from a surface (re ection, transmission, absorption), and is strongly linked with its nal appearance. In its general form, re ectance is modeled by a function of many variables, which often include the wavelength of the incident ray of light and its direction and the wavelength and the direction of the ray of light leaving the surface at a (possibly) di erent location on the surface. However, it is common in computer graphics to drop some dependencies at the cost of less variety in the material appearance, but for more practicality and e ciency. Please refer to [START_REF] Guarnera | Virtual material acquisition and representation for computer graphics[END_REF] for a discussion on the most general form of re ectance functions. representation lacks the wavelength dependency, which is responsible for complex spectral phenomena. However, it is still able to capture e ects such as translucency, self-shadowing, self occlusions, and inter-re ections. Its high number of dimensions (8D) prevents easy measurements, and simpler representations are often preferred to it.

By assuming the incident ray position is the same as the outgoing ray, we get the Bidirectional Texture Function (BTF). This representation has the advantage to capture small-scale geometric e ects such as self-shadowing, self-occlusions, and inter-re ections. Still, it is expensive to capture as it requires sweeping in both the spatial (plane) and angular (ray directions) domains.

Ignoring the small-scale e ects, we obtain the Spatially Varying Bidirectional Re ectance Distribution Function (SVBRDF), which has the same number of dimensions as the BTF but is often encoded in bidimensional maps which compactly encode the appearance of opaque materials. This representation is further discussed in Section I.3.5. [START_REF] Guarnera | Virtual material acquisition and representation for computer graphics[END_REF] Dropping the spatial heterogeneity from the BSSRDF, we get the Bidirectional Scattering Distribution Function (BSDF). This function is mostly used in path tracing scenarios, as it models scattering e ects for both re ection and transmission, which are harder to compute e ciently in an interactive context.

Finally, separating opaque from transparent materials, we get either the Bidirectional Re ectance Distribution Function (BRDF) or the Bidirectional Transmittance Distribution Function (BTDF). Such distributions are grouped into three main categories: Phenomenological (or Empirical), Physically-based and Data-Driven. The rst group corresponds to the tting of analytical formula to re ectance data, the second is motivated by the laws of Physics and Optics, and the third consists in storing acquired data into e cient data structures using suitable basis functions.

I.3.3 Physically-based BRDF

Most of the re ectance distributions described so far are suitable for many scienti c applications, which require precision and are not sensitive to the speed of execution. However, for Digital Content Creation which focuses on design interaction and visual appearance, the SVBRDF is the de facto standard for modeling re ectance. With time, the choice of re ectance function, or BRDF model, switched from Phenomenological to Physically-based as the latter allowed the reproduction of photorealistic e ects much more e ciently by improving the productivity of artists.

Physically-Based BRDF must enforce the constraints imposed by the laws of physics. The rst constraint is called Helmholtz reciprocity and states that the light source and observation directions may be switched without changing the value of the function. It can be written:

∀𝜔 𝑖 , 𝜔 𝑜 ∈ Ω, 𝑓 𝑟 (𝜔 𝑖 , 𝜔 𝑜 ) = 𝑓 𝑟 (𝜔 𝑜 , 𝜔 𝑖 )
The second constraint is energy conservation, meaning the outgoing energy cannot be greater than the incoming energy. This can be written:

∀𝜔 𝑜 ∈ Ω ∫ Ω 𝑓 𝑟 (𝜔 𝑖 , 𝜔 𝑜 ) (𝑛•𝜔 𝑖 ) 𝑑𝜔 𝑖 ≤ 1
The third constraint is non-negativity:

∀𝜔 𝑖 , 𝜔 𝑜 ∈ Ω, 𝑓 𝑟 (𝜔 𝑖 , 𝜔 𝑜 ) ≥ 0

I.3.4 The Microfacet Model

Most rendering engines rely on microfacet-based BRDF, introduced by Cook-Torrance [START_REF] Cook | A re ectance model for computer graphics[END_REF], both for real-time and production renderings. It is often represented as the sum of two terms, the di use and the specular term. It is illustrated in Fig. I.8 and is written:

𝑓 𝑟 (𝜔 𝑖 , 𝜔 𝑜 ) = 𝑓 𝑑 + 𝑓 𝑠 (𝜔 𝑖 , 𝜔 𝑜 ) = 𝜌 𝜋 + 𝐹 (𝜔 𝑖 , 𝜔 ℎ )𝐺 (𝜔 𝑖 , 𝜔 𝑜 , 𝜔 ℎ )𝐷 (𝜔 ℎ ) 4 (𝑛•𝜔 𝑖 ) (𝑛•𝜔 𝑜 ) (I.1)
The BRDF is parameterized by the light vector 𝜔 𝑖 and the view vector 𝜔 𝑜 . The normal 𝑛 of the point where the function is evaluated is considered, along with the half-angle vector 𝜔 ℎ , corresponding to the orientation of microfacets which re ect light. The latter is computed using 𝜔 𝑖 and 𝜔 𝑜 :

𝜔 ℎ = 𝜔 𝑖 + 𝜔 𝑜 𝜔 𝑖 + 𝜔 𝑜
The di use term of the BRDF, often called Lambertian term, is written 𝑓 𝑑 . It is modulated by a di use color 𝜌 and originates from internal scattering or multiple surface re ections in the model [START_REF] Cook | A re ectance model for computer graphics[END_REF]. The second term 𝑓 𝑠 is called specular and relates to the contributions of mirror-like microfacets. It is composed of the 𝐷, 𝐹 and 𝐺 factors, which we describe in the next paragraphs.

The rst factor 𝐷 relates to the distribution of microfacet normals. It has a strong in uence on the nal appearance as it determines the size and shape of specular highlights. It a ects the perception of surface roughness. Rougher surfaces will create blurry highlights, as opposed to glossier surfaces which produce sharper ones. It is often modeled using either a Gaussian-like or a Cauchy-like distribution.

The rst is called the Beckmann distribution [START_REF] Beckmann | The scattering of electromagnetic waves from rough surfaces[END_REF] and is de ned in Eq. I.2. The second was proposed by [START_REF] Trowbridge | Average irregularity representation of a rough surface for ray re ection[END_REF], often referred to as GGX [START_REF] Walter | Microfacet models for refraction through rough surfaces[END_REF], and is de ned in Eq. I.3. These two distributions can be visualized in Fig. I.9. The GGX distribution is characterized by its long tail and represents metals more faithfully than the Beckmann NDF. Other distributions can be found in the literature, but the ones presented are the most used in practice. They are both parameterized by a roughness value 𝛼 which controls the width of the distribution. Their expressions take the form: where 𝜒 + is the Heaviside step function (0 if the argument is negative, 1 if it is strictly positive). The Fresnel term 𝐹 also plays an important role in the appearance by predicting the fraction of power that is re ected from the surface depending on the view angle. This e ect is illustrated in Fig I .10, where the quantity of re ected and refracted light varies with the angle between the surface normal and the direction of view. Fresnel equations require refractive index values sampled over the visible spectrum, which may be complex numbers. For realtime scenarios, the Schlick's approximation is favored, and is parameterized by a specular color called Fresnel re ectance at normal incidence 𝐹 0 . It is written:

𝐷 𝐵𝑒𝑐𝑘𝑚𝑎𝑛𝑛 = 𝜒 + (𝑛 • 𝜔 ℎ ) 𝜋𝛼 2 (𝑛 • 𝜔 ℎ ) 4 exp (𝑛 • 𝜔 ℎ ) 2 -1 𝛼 2 (𝑛 • 𝜔 ℎ ) 2 (I.2) 𝐷 𝐺𝐺𝑋 = 𝜒 + (𝜔 ℎ • 𝑛) • 𝛼 2 𝜋 ((𝑛 • 𝜔 ℎ ) 2 (𝛼 2 -1) + 1) 2 (I.3)
𝐹 (𝜔 𝑖 , 𝜔 ℎ ) = 𝐹 0 + (1 -𝐹 0 )(1 -(𝜔 𝑖 • 𝜔 ℎ )) 5
Last, the geometry term 𝐺, also named masking-shadowing function or Geometric Attenuation Factor (GAF), takes into account the fraction of microfacets which are either masked from the view direction, or shadowed from others in the light direction. A physically accurate form of this term has been proposed [START_REF] Ross | Detailed analytical approach to the gaussian surface bidirectional re ectance distribution function specular component applied to the sea surface[END_REF], taking into account Height-Correlated Masking and Shadowing, and is the one recommended [START_REF] Heitz | Understanding the masking-shadowing function in microfacet-based brdfs[END_REF]. It takes the form:

𝐺 (𝜔 𝑖 , 𝜔 𝑜 , 𝜔 ℎ ) = 𝜒 + (𝜔 𝑖 • 𝜔 ℎ ) 𝜒 + (𝜔 𝑜 • 𝜔 ℎ ) 1 + Λ(𝜔 𝑖 ) + Λ(𝜔 𝑜 )
The Λ function is derived from the distribution of normals used in the microfacet model (either Beckmann or GGX). It takes the form: Λ 𝐺𝐺𝑋 (𝜔 𝑥 ) = 1 2

Λ 𝐵𝑒𝑐𝑘𝑚𝑎𝑛𝑛 (𝜔 𝑥 ) = erf(𝑎) -1 2 + 1 2𝑎 √ 𝜋 exp(-𝑎 2 ), 𝑎 = (𝑛 • 𝜔 𝑥 ) 2 𝛼 (1 -(𝑛 • 𝜔 𝑥 ) 2 )
-1 + √ 1 + 𝛼 2 (1 -(𝑛 • 𝜔 𝑥 ) 2 ) (𝑛 • 𝜔 𝑥 ) 2
For computing the Beckmann masking-shadowing function, [START_REF] Walter | Microfacet models for refraction through rough surfaces[END_REF] proposed an accurate rational approximation, which gives:

Λ 𝐵𝑒𝑐𝑘𝑚𝑎𝑛𝑛 (𝜔 𝑥 ) 1-1.259𝑎+0.396𝑎 2 3.535𝑎+2.181𝑎 2 if 𝑎 < 1.6 0 otherwise

I.3.5 The SVBRDF model

The SVBRDF encoding relies on a set of bidimensional bitmap images called PBR maps. The metallic term describes the material as conductor (metals) or dielectric (such as plastic). It is used to compute the specular color term 𝐹 0 or the di use term 𝜌 in I.3.4 from the albedo map, such that:

𝜌 = (1 -metallic) • basecolor 𝐹 0 = 0.04 • (1 -metallic) + basecolor • metallic
These maps can sometimes be found separated into di use color and specular color maps, without the need for a metallic map (which is a more compact representation for di use and specular colors).

The three remaining maps encode geometric information at multiple scales. The height map, or displacement map, acts as a height eld on the surface. It describes the quantity of displacement (positive or negative) to apply to a vertex of the mesh in the direction of the local normal. It is often used along with tessellation, which subdivides the geometry to produce smaller triangles and hence ner details. This map acts on the silhouette of the surface, as well as the shadows computed using raytracing, ambient occlusion, or shadow maps.

Decreasing roughness

Decreasing roughness Figure I.12: Anisotropy variations, from [START_REF] Ashikhmin | An anisotropic phong brdf model[END_REF].

At a ner scale, the normal map encodes a perturbation of the local normal in tangent space, which adds details to the rendering, without the need for ner geom-etry. This rendering trick enables ne details to be rendered and enhances realism by adding surface imperfections, but can result in visibility artifacts [START_REF] Schüssler | Microfacet-based normal mapping for robust monte carlo path tracing[END_REF]. As explained previously, normal mapping has been originally developed for baking geometry information inside a texture, allowing for the simpli cation of the triangle mesh.

At the microscopic level, the roughness map dictates the width of the normal distribution function which controls the aspect of re ections on the surface. A rough material will have no visible re ections, whereas a specular, non-rough material will act as a mirror. In between, glossy materials (roughness between zero and one) will have blurry re ections.

While the roughness parameter produces a wide variety of highlights, some appearances such as scratched metals, or woven patterns cannot be properly modeled by a single roughness parameter that describes isotropic surfaces, meaning identical in all directions. To support more e ects, anisotropic models have been proposed. They replace the normal distribution and masking-shadowing functions with an anisotropic variant, parameterized by an angle of anisotropy and two roughness values for the tangent and bitangent vectors. They allow for the reproduction of a wider set of appearances, as shown in Fig. I.12. Henceforth, with a model for re ectance at hand, we can study how light propagates in the whole scene, which is necessary for producing a synthesized image. Such modelisation is achieved using the light transport equation, or rendering equation, proposed by [START_REF] Kajiya | The rendering equation[END_REF]) and aims at computing what is called Global Illumination (GI) in a scene. It can be written as:

I.3.6 The Rendering Equation

𝐿 𝑜 (𝜔 𝑜 ) = 𝐿 𝑒 (𝜔 𝑜 ) + ∫ Ω 𝑓 𝑟 (𝜔 𝑖 , 𝜔 𝑜 )𝐿 𝑖 (𝜔 𝑖 ) (𝑛•𝜔 𝑖 ) 𝑑𝜔 𝑖
This formula states that the exitant radiance 𝐿 𝑜 (𝜔 𝑜 ) must be equal to the emitted radiance 𝐿 𝑒 (𝜔 𝑜 ) plus the scattered radiance represented by the integral over the hemisphere Ω. This integral is given as the product of the BRDF 𝑓 𝑟 (𝜔 𝑖 , 𝜔 𝑜 ) (which we discuss in the next paragraph), the incoming radiance 𝐿 𝑖 (𝜔 𝑖 ), and the weakening factor of outward irradiance 𝑛•𝜔 𝑖 . The latter is due to the light ux being smeared across a surface whose area is larger than the projected area perpendicular to the incident direction. This phenomenon is described by Lambert's cosine law.

Global illumination is computed by recursively casting rays towards points in the scene which ultimately reach an emitter (see Fig I .13).

I.3.7 Rendering Algorithms

The rendering equation has no analytical solution, because of the complex integral over the hemisphere. Nevertheless, there are di erent ways to solve it for synthesizing images, which often fall into two main categories: rasterization and ray tracing. These techniques mainly aim at solving the visibility problem, which relates to the occlusion of objects by other objects in the scene. The choice of one or the other method is application dependent: because rasterization leverages a hardware-accelerated pipeline, it tends to produce images at a much faster rate than ray tracing methods. Interactive applications (such as Virtual and Augmented reality) favor rasterization while "o ine" applications (e.g., Visual E ects for Movies) rely on heavier methods which often generate more realistic and accurate e ects.

Recently, the rise of hardware-accelerated ray tracing modules in graphics cards blurs the boundaries between both techniques, as the two are used concurrently in hybrid rendering pipelines. Rasterization solves visibility by projecting the geometry (usually represented as triangle meshes) into each pixel of the image using discretization (see Fig. I.14,left). It relies on di erent culling mechanisms to prevent the projection of all the elements in the scene, which speed up the computations. Modern pipelines leverage a Z-bu er which keeps track of the current depth (or distance to the camera) of the projected geometry to identify which geometry should be rendered. Rasterization allows to render millions of polygons in a few milliseconds and is well suited for applications such as video games, real-time visualisation softwares, augmented and virtual reality. The computation involving the rendering equation happens at a stage called shading. It considers the per-pixel projected geometry and texture information and evaluates the integral of the rendering equation. For punctual lights, the integral simpli es to closed form solutions which ease the computation. For more complex lighting setups, such as environment lights, many approximations have been proposed to e ciently compute closed-form solutions.

Ray tracing on the other hand refers to a category of techniques that tackle the problem the other way round. The visibility is handled by launching a ray from the virtual camera. The ray travels through the image formation plane to reach the scene's geometry (see Fig. I.14,right). Many variants of this algorithm have been proposed in the past, starting with [START_REF] Whitted | An improved illumination model for shaded display[END_REF] who proposed a tree structure to simulate re ection and refraction e ects. Another seminal work [START_REF] Veach | Robust monte carlo methods for light transport simulation[END_REF] addressed more complex e ects (such as soft shadows and ambient occlusion, glossy and di use re ections, depth-of-eld, and motion blur), which set the foundation for Monte Carlo methods. A common way to accelerate ray-tracing algorithms is to enclose the geometry inside a spatial hierarchical data structure. This prevents a given ray to inspect all triangles in the scene. Path tracing, which is used to compute GI using multiple light bounces, naturally handles the rendering equation because it is recursive in nature: each intersection with the geometry will emit new rays towards other objects (light source or geometry) which again will emit new rays. Note that ray tracing is not speci c to rendering, it is also used for e.g., collision detection or audio simulations.

I.3.8 Learning based approaches

Learning-based approaches rely on simple di erentiable operations and provide a solid optimization framework for storing or processing data by learning from samples from a given data distribution.

The study of deep neural networks for computer vision and image processing tasks (see example in Fig. I.15) naturally led to their use in graphics applications.

They solve complex problems involved in rendering, such as denoising, guided sampling, distributions approximation, and material appearance encoding.

The simplest neural architecture consists of multiple layers of nodes that use nonlinear activation functions and is called a Multilayer Perceptron. This architecture relies on backpropagation for updating its weights between the connected nodes of each layer in a process often called training. This structure, although very simple, allows to e ciently encode complex functions, and has been used in a wide variety of tasks related to Computer Graphics, such as encoding Radiance Fields [START_REF] Mildenhall | Nerf: Representing scenes as neural radiance elds for view synthesis[END_REF], Radiosity [START_REF] Hadadan | Neural radiosity[END_REF] or BRDF [START_REF] Sztrajman | Neural brdf representation and importance sampling[END_REF].

I.3.9 Mipmapping

MIP mapping is a widespread technique for preventing aliasing in a rendered image.

It is based on a pre-computation step that stores progressively lower-resolution images from a high-resolution input into a mipmap pyramid. MIP stands for multum in parvo, meaning "many things in a small place". This method allows to increase rendering speed and reduces aliasing artifacts such as staircase e ects of Moiré patterns (see Fig. I.16). The method has been developed by [START_REF] Williams | Pyramidal Parametrics[END_REF] in his article Pyramidal Parametrics. It is widely used in interactive applications and has a speci c function call in many graphics APIs (such as OpenGL, DirectX, or Vulkan) for computing the pyramid. This method is e cient at ltering colored textures but may fail on other types of data such as SVBRDF maps since they represent quantities that cannot be linearly ltered.

without mipmapping base texture mipmaps with mipmapping

I.4 Outline

The remainder of the manuscript is organized as follows:

Chapter II provides an overview of previous works related to this thesis. Specifically, we focus on RGB Texture Manipulation, Filtering and Level-of-Detail approaches, as well as Di erentiable methods.

Chapter III describes the elaboration of a morphing operator for material design which leverages existing datasets of materials. We aim at providing a structure and appearance-preserving material interpolation operator, in order to design plausible new materials from two inputs. Speci cally, our main contributions are summarized as follows:

• an e cient operator to explore and design-by-interpolation novel SVBRDFs based on an existing collection of materials,

• a transport grid model to guide the morphing process from the dominant mesostructures of the materials,

• novel real-time detail-preserving mechanisms to blend albedo and normal vectors consistently in this context.

In Chapter IV we analyse the e ect of enabling the variation of the observation distance during interactive rendering and its link with the multiple scales of geometry encoded in the SVBRDF maps. We show that using simple Multilayer Perceptrons allows for producing a pre ltered pyramid which compactly encodes normal and roughness information at multiple level-of-details. In this chapter, our contributions relate to an appearance driven optimization framework to produce a mipmap pyramid by learning a downsampling operator. We propose:

• an e cient pipeline for learning mipmapping lters requiring no data preparation for training,

• a neural architecture encoding anisotropic appearance and generalising on unseen materials,

• a tensor-based formulation for anisotropic BRDF distributions which is well-suited for di erentiable pipelines and trilinear interpolation.

Chapter V proposes to extend the discussion by further analysing the previous chapters and proposing various applications of the methods developed in this thesis.

Finally, Chapter VI concludes this thesis work on the domain-speci c content of SVBRDF maps which encode the rich, multiscale geometric structures of the material in a compact yet expressive representation. We propose to discuss future research directions linked with this topic.

I.5 Scienti c Publications

I.5.2 Released source code

As part of this thesis, open-source programs have been released publicly.

MatMorpher: A viewer for material morphing and a transport grid computation tool are regrouped into a single software implementing the contributions of Chapter III, available at: https://github.com/AlbanGauthier/Matmorpher Texture Design Warping and Sharpness Preservation: Code for the comparison with previous work on Texture Morphing presented in Chapter III, available as a branch of the previous git repository.

MIPNet: An implementation of the learning pipeline for computing mipmap pyramids, following the contributions of Chapter IV, available at: https:// github.com/AlbanGauthier/mipnet_neural_mipmap LEADR and SSGT: An implementation of previous work related to SVBRDF mipmapping, for preprocessing and realtime rendering of mipmapped materials, presented in Chapter IV, available at the aforementioned link.

II

Related Work

Our motivation for studying materials represented as SVBRDF relies on the availability of a large body of work on image processing techniques. In particular, image manipulation has been studied for the case of textures which o er speci c statistical properties such as self-similarity and homogeneity, compared to natural images. We review approaches related to RGB texture synthesis, morphing and interpolation in Section II.1.

Furthermore, e ciently designing and rendering materials in an interactive context often relies on precomputations, which trade storage for rendering speed. These precomputations are based on level-of-detail strategies for representing appearance at coarser levels when rendering ne details becomes unnecessary. We identify previous works on ltering and level-of-detail techniques for RGB images as well as materials, represented as maps or directly encoded into a re ectance function in Section II.2.

Finally, we propose to dive into e cient optimization frameworks based on differentiability in Section II.3, which we build one of our operators upon. They have seen a sharp increase in their use in the context of image design and aquisition. These methods solve di cult inverse problems e ciently, while allowing for compact representations in the case of learning-based approaches.

II.1 RGB Texture Manipulation

II.1.1 Exemplar texture synthesis

Texture synthesis is an area of work dedicated to the creation of textures from exemplars. This research eld tackles image manipulation techniques which take user intent into consideration when synthesizing an output. Two stateof-the-art reports [START_REF] Wie | State of the Art in Examplebased Texture Synthesis[END_REF][START_REF] Barnes | A survey of the state-of-the-art in patch-based synthesis[END_REF]) describe means to generate numerous variations of a texture, starting from an exemplar pattern. Speci c variations of input textures have been studied, such as weathering texture simulations [START_REF] Bellini | Time-varying weathering in texture space[END_REF] which can be used to synthesize general tileable, inhomogeneous and directional textures [START_REF] Moritz | Texture Stationarization: Turning Photos into Tileable Textures[END_REF][START_REF] Zhou | Analysis and Controlled Synthesis of Inhomogeneous Textures[END_REF]. Such techniques however are limited to a single input RGB texture. [START_REF] Ray | Material Space Texturing[END_REF] produce a 3D texture tile from two input RGB textures and their feature masks to allow for spatially varying texturing of surfaces. We choose a more lightweight approach for our operator presented in the following Chapter III, as their technique requires both high storage and long precomputation for high resolution textures. Recently, Guel et al. [START_REF] Guehl | Semi-procedural Textures Using Point Process Texture Basis Functions[END_REF] presented a semi-procedural approach that avoids drawbacks of procedural textures and leverages advantages of data-driven methods. This approach allows for material structure interpolation (see Fig. II.1) but requires a binary segmentation of the input material, does not preserve the input features and is spatially but not temporally stable. This method however has been demonstrated on a stack of simple SVBRDF maps. [START_REF] Kabul | An optimal control approach for texture metamorphosis[END_REF]) also focus on RGB texture interpolation, by optimizing for the complete texture metamorphosis, under an optical ow framework. In contrast, our morphing operator optimizes for an advection eld retargeting salient structures, before blending retargeted signals in a secondary phase (similar to (Matusik et [START_REF] Schuster | A three-level approach to texture mapping and synthesis on 3d surfaces[END_REF] extended patch-based texture generation to meshes, which uses a multiscale optimization to minimize the visual artifacts between patches. They propose height-based blending as well as a histogram preserving blending -similar to [START_REF] Heitz | High-Performance By-Example Noise using a Histogram-Preserving Blending Operator[END_REF][START_REF] Burley | On Histogram-preserving Blending for Randomized Texture Tiling[END_REF] -as a mean of interpolating between SVBRDF maps. These methods do not tackle the problem of blending between two distinct materials however.

II.1.2 Image Morphing and Manipulation

Image Retargeting [START_REF] Shamir | Visual media retargeting[END_REF][START_REF] Rubinstein | A comparative study of image retargeting[END_REF] address natural images and video scaling/cropping, including seam-carving [START_REF] Avidan | Seam carving for content-aware image resizing[END_REF], scale-and-stretch (Y.-S. Wang et al., 2008) and distortion-free shape deformation [START_REF] Karni | Energy-Based Image Deformation[END_REF]. [START_REF] Liu | Texture splicing[END_REF] focus on texture-guided re nement, by leveraging the texton appearance from one input texture and the distribution of the other. These techniques apply to RGB images only and do not provide interpolation between two input images, but rather focus on the deformation.

II.1.3 Patch-based methods

Fang et al. [START_REF] Fang | Detail preserving shape deformation in image editing[END_REF] build upon patch-based texture synthesis to deform shapes while avoiding stretch and compression. [START_REF] Barnes | PatchMatch: a randomized correspondence algorithm for structural image editing[END_REF] provide an interactive editing tool based on a quick randomized algorithm for nding approximate nearest neighbor matches between image patches, with applications in retargeting and reshu ing. [START_REF] Darabi | Image melding: combining inconsistent images using patch-based synthesis[END_REF] proposed to synthesize a transition region between two source images leveraging an enriched patch-based search, a screened Poisson equation solver and a sharpness preserving norm. [START_REF] Mechrez | Saliency driven image manipulation[END_REF] base their optimization framework on a patch-based manipulation to enhance natural images. [START_REF] Ruiters | Patch-based texture interpolation[END_REF] is related to our proposed morphing operator by presenting a patch-based method to create spatial interpolations between two RGB textures but di ers from our work in several ways. This method is not t for interactive design and requires a user-supplied distribution mask prior to the patch interpolation (see results in Fig. II.3). In contrast, our method can leverage a blending mask interactively. In their work, manually crafted feature maps are used -instead of the compass operator [START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF], which is deemed unsatisfactory by [START_REF] Ruiters | Patch-based texture interpolation[END_REF]. Last but not least, their method is fundamentally a spatial interpolation method between non-superposed inputs (an in-painting method) whereas ours is a temporal interpolation method between superposed inputs. Image Melding [START_REF] Darabi | Image melding: combining inconsistent images using patch-based synthesis[END_REF] produces similar results than [START_REF] Ruiters | Patch-based texture interpolation[END_REF] with a more lightweight optimization. Mipmapping was rst introduced by Williams [START_REF] Williams | Pyramidal Parametrics[END_REF] and provides an e cient solution to approximate multiple samples at runtime by precomputing a pre ltered image pyramid using box, Gaussian, Lanczos or Kaiser lters [START_REF] Akenine-Möller | Real-time rendering 4th edition[END_REF]. At rendering time, this mipmap pyramid is accessed using trilinear interpolation and often considers the anisotropy of the pixel footprint in texture space [START_REF] Manson | Parameterization-Aware MIP-Mapping[END_REF]. Our ltering operator described in Chapter IV aims at providing an image pyramid for each SVBRDF map by carefully crafting a downsampling kernel. Image downsampling methods optimize kernels based on image features at multiple scales [START_REF] Kopf | Content-adaptive image downscaling[END_REF] (see Fig. II.4) or using the SSIM metric [START_REF] Öztireli | Perceptually based downscaling of images[END_REF]. Later, [START_REF] Weber | Rapid, detail-preserving image downscaling[END_REF] proposed a variant of joint-bilateral lters to compute downscaled images at interactive rates.

II.2 Filtering and

In our framework, downsampling kernels are learned using a neural network which allows for generalisation across multiple materials.

II.2.2 Normal map ltering

Normal distributions can be approximated using a single isotropic or anisotropic lobe which results in e cient renderings at the cost of appearance preservation. Schilling [START_REF] Schilling | Towards real-time photorealistic rendering: challenges and solutions[END_REF] identi ed that roughness information can be derived from the normal map and encodes normal aggregates in a covariance matrix, while [START_REF] Olano | Normal distribution mapping[END_REF]) use a single 3D Gaussian lobe. Later, [START_REF] Toksvig | Mipmapping Normal Maps[END_REF] proposed to compute the width of the NDF based on the accumulated normals' length. Since a single lobe is often not enough, many previous works proposed to encode the NDF into multiple lobes. [START_REF] Han | Frequency domain normal map ltering[END_REF] propose a framework which generalises previous works on the topic [START_REF] Fournier | Filtering normal maps and creating multiple surfaces[END_REF][START_REF] Tan | Multiresolution Re ectance Filtering[END_REF] and encode averaged information into spherical harmonics (SH) or von Mises-Fisher (vMF) distributions. Such techniques require a custom shading while we provide a drop-in replacement for standard mipmapping.

II.2.3 Rendering high resolution normal maps.

E orts have been made to render microstructures such as glints or scratches, since they require higher resolution normal maps which result in highly complex NDFs. O ine as well as real-time [START_REF] Zirr | Real-time rendering of procedural multiscale materials[END_REF][START_REF] Chermain | Procedural Physically based BRDF for Real-Time Rendering of Glints[END_REF][START_REF] Tan | Real-time microstructure rendering with mip-mapped normal map samples[END_REF] techniques have been proposed for this task. Such methods focus on a speci c appearance problem that we are not addressing in this thesis. Please refer to (J. [START_REF] Zhu | Recent Advances in Glinty Appearance Rendering[END_REF] for a complete related work on the topic.

II.2.4 Re ectance ltering

More generally, previous work addressed multiscale appearance preservation by ltering at the shading level. Please refer to Bruneton and Neyret [START_REF] Bruneton | A survey of nonlinear pre ltering methods for e cient and accurate surface shading[END_REF] for a review on the topic. Becker and Nelson (Becker & Max, 1993) proposed a method to smoothly transition between displacement, bump mapping and BRDF in a uni ed framework. Later, e orts were put on the representation of aggregated BRDFs. [START_REF] Claustres | Wavelet encoding of BRDFs for real-time rendering[END_REF] used wavelet encoding, while (P. [START_REF] Tan | Filtering and rendering of resolution-dependent re ectance models[END_REF] generalised the scope of their previous work on Gaussian mixture model. [START_REF] Xu | Real-Time Linear BRDF MIP-Mapping[END_REF] proposed a framework to encode SVBRDFs and normals into BRDF mipmaps, and to lter the latter in real-time. [START_REF] Heitz | Filtering color mapped textures and surfaces[END_REF] proposed to address the non-linear behavior of rendering color textures mapped onto surfaces. More recently, [START_REF] Wu | Accurate appearance preserving pre ltering for rendering displacement-mapped surfaces[END_REF] focused on appearance preservation of displaced surfaces by jointly pre ltering the displacement map and the SVBRDF. These techniques require engine modi cations to be applied in a real-time context. Putting SVBRDFs aside, several works proposed to tackle volume ltering [START_REF] Loubet | Hybrid mesh-volume LoDs for all-scale preltering of complex 3D assets[END_REF][START_REF] Zhao | Downsampling scattering parameters for rendering anisotropic media[END_REF] or study the e ect of ltering when dealing with BTF data [START_REF] Jarabo | E ects of approximate ltering on the appearance of bidirectional texture functions[END_REF].

When rendering materials at di erent scales, aliasing occurs, especially for highly specular materials. Several works [START_REF] Tokuyoshi | Stable Geometric Specular Antialiasing with Projected-Space NDF Filtering[END_REF][START_REF] Chermain | Real-Time Geometric Glint Anti-Aliasing with Normal Map Filtering[END_REF] proposed to tackle Geometric Specular Antialiasing when rendering highly specular materials. These techniques can be applied as a post-process to our method, but do not focus on SVBRDF pre ltering. [START_REF] Goodfellow | Generative adversarial nets[END_REF] for learning to translate an image from a source to a target domain in the absence of paired examples, which was later improved and applied to a dataset of celebrity faces [START_REF] Choi | Stargan: Uni ed generative adversarial networks for multi-domain image-to-image translation[END_REF]. [START_REF] Zhu | Generative visual manipulation on the natural image manifold[END_REF] proposed to learn the natural image manifold directly from data using a GAN, allowing for image interpolation with user control over the object's shapes. E and et al. (E and et al., 2020) propose a multiscale feature space approach incorporating a deep convolutional neural network. Deep learning methods have recently proven their e ciency at synthesizing 2D and 3D textures [START_REF] Sendik | Deep Correlations for Texture synthesis[END_REF][START_REF] Gutierrez | On Demand Solid Texture Synthesis Using Deep 3D Networks[END_REF] as well as interpolating between them [START_REF] Yu | Texture mixer: A network for controllable synthesis and interpolation of texture[END_REF][START_REF] Henzler | Learning a Neural 3D Texture Space From 2D Exemplars[END_REF]. [START_REF] Vacher | Texture interpolation for probing visual perception[END_REF] study texture interpolation and synthesis with deep neural networks, providing insights about the distribution of CNN activations of natural textures. These techniques work well with stochastic textures but the models tend to over t to the trained data, and always fail to preserve salient structures and autosimilarities in the inputs. MaterialGAN (Guo, Smith, Hašan, et al., 2020) requires an inverse rendering optimization to project the SVBRDF in the latent space. The interpolation of the latent variables produces temporally stable interpolation (Fig. II.5) but fails to preserve both input textures.

II.3 Di erentiable methods

II.3.2 Di erentiable rendering

Recent advances in di erentiable rendering pipelines allow for the optimization of rendering data (interpretable or implicit) so that its appearance matches the nal pixel colors. This is done by backpropagating gradients of an image-based loss through the rendering pipeline. Several material acquisition methods have been proposed [START_REF] Deschaintre | Single-image SVBRDF capture with a rendering-aware deep network[END_REF]Guo, Smith, Hašan, et al., 2020;[START_REF] Zhou | Adversarial single-image svbrdf estimation with hybrid training[END_REF] which allow for SVBRDF parameters recovery using a rendering loss. These methods allow for the creation of the SVBRDF map at a single scale, and do not solve the challenge of mipmapping the SVBRDF.

Even though SVBRDF maps can be computed explicitly, [START_REF] Kuznetsov | Neumip: Multi-resolution neural materials[END_REF] propose to encode material appearance in a level-of-detail pyramid of neural textures, which are fed to a decoder network. By using two multilayer perceptrons (a Neural O set Module and a Texture Decoder) along with a Neural Texture Pyramid, they encode complex material information which can be queried in real time. However, this technique is not t for traditional rasterization pipelines. The Neural Texture Pyramid creates a material speci c implicit representation which is tied to the MLP Decoder to output radiance information. [START_REF] Rainer | Neural BTF Compression and Interpolation[END_REF] proposed to learn BTF and BRDF [START_REF] Rainer | Uni ed Neural Encoding of BTFs[END_REF] encodings using fully-connected multilayer perceptrons. Neural radiance elds [START_REF] Mildenhall | Nerf: Representing scenes as neural radiance elds for view synthesis[END_REF] allow for the encoding of spatio-angular radiance information inside a volume using a di erentiable framework. This method has been extended to support mipmapping [START_REF] Barron | Mip-nerf: A multiscale representation for anti-aliasing neural radiance elds[END_REF] and SVBRDF-like parameters [START_REF] Boss | Nerd: Neural re ectance decomposition from image collections[END_REF]. However, NeuMIP and Radiance Field-like solutions do not tackle SVBRDF pre ltering, but rather encoding of shape and appearance into a neural network, possibly at multiple scales. We aim at preserving the structure and content of interpretable SVBRDF maps so that they can directly be used inside current renderers without requiring any engine code modi cation. [START_REF] Hasselgren | Appearance-Driven Automatic 3D Model Simpli cation[END_REF] mates the task of decimating geometry manually. The method allows to transfer geometric details of the mesh in the normal and displacement maps (via joint shapeappearance simpli cation) and is able to construct a per material mipmap pyramid. Since their method only optimizes for a given couple composed of a geometry and a SVBRDF maps stack, it is incapable of any generalisation over a similar dataset. Our work aims at using the learned downsampling kernels for a wide variety of materials beyond the training data to prevent a lengthy optimization.

III A Morphing Operator for SVBRDFs

III.1 Design By Example

As discussed previously, materials o er a rich variety of appearance for 3D scenes, and are a key component to enhance realism in synthesized images. However, designing materials from scratch is a complex and time-consuming task. Still, there already exists large database of existing detailed materials curated by the artists' hard work. These databases have been populated by captured, or procedural materials. The latter allow for exploring local appearance changes but cannot extrapolate outside the scope of the artists-de ned parameters. What if we used material as examplars to navigate the space of appearance by combining, mixing or interpolating between them? These material exemplars could be used as anchors to explore large regions of the space of possible materials, given a minimalistic interaction framework for this task.

Hence, we aim at leveraging existing artist-generated content, and allow for the design of novel and perceptually-valid materials, providing a means of exploring the space of material appearance. Such schemes allow for dataset ampli cation for machine learning applications, by mixing pairs of existing materials, as well as user-driven variations of existing content (see Fig. III.1). Our goal is to develop a tool for interactive material design in mind, using a minimal interaction scheme, to be used by novice and expert users alike.

As mentionned in Section I.2, the stack of SVBRDF maps, although lying in the same bidimensional domain, cannot be interpreted as a generalisation of RGB images into higher dimensions due to the di erent nature of each map. Exploring such high dimensional space could be naively achieved through linear interpolation but would then su er from unwanted overlap of structures, ghosting artifacts and a lack of sharpness in the output (see Fig. III.15,left). The resulting materials fall outside the perceptual space of valid materials.

To alleviate these problems, we take inspiration from the image morphing community [START_REF] Wolberg | Image morphing: A survey[END_REF] and identify coarse geometric structures from the materials to account for the variety of channels of the SVBRDF. We propose specialized operators to account for the various scales of the geometric channels, and interpolate quantities accordingly. Hence, we propose a structure-preserving material interpolation operator which allows to blend between two existing materials while preserving their dominant features all along the interpolation. Note that the SVBRDF is responsible for the appearance only through the BRDF, to which the relation between the quantities and their radiance response is non-linear. This operator is designed to be: (i) fast enough to provide real-time feedback, (ii) easily controlled using an intuitive blending map, (iii) feature-preserving across the multiple material channels. We stress the need of the operator to support interactive editing of high resolution SVBRDF maps up to a resolution of 4K.

The method is based on a transport mechanism which continuously transforms the individual input PBR maps into their destination counterparts in a featuresensitive manner. We rst present works related to this problem in Section III.2. We propose an overview of our pipeline for material morphing in Section III.3. We start by detecting salient edges in the materials and create a mapping in the form of a transport grid in order to preserve important visual features during the interpolation. These operations are detailed in Section III.4. Each PBR map is then carefully interpolated to avoid detail losses and ghosting stemming from the naive interpolation, as described in Section III.5.

More precisely, we propose the following contributions:

• an e cient operator to explore and design-by-interpolation novel SVBRDFs based on an existing collection of materials,

• a transport grid model to guide the morphing process from the dominant mesostructures of the materials,

• novel real-time detail-preserving mechanisms to blend albedo and normal vectors consistently in this context.

III.2 Related work

The work of Matusik et al. [START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF] is closely related to ours. They present a system for designing novel textures starting from an input database, allowing to smoothly interpolate textures in a purposefully created space. Their work applies to RGB textures but not SVBRDF maps, which lie in incompatible spaces (color, spatial components, and distribution coe cients) and do not exhibit a complete spatial redundancy. We provide a thorough comparison to their method, which we reimplemented.

Two successive works [START_REF] Heitz | High-Performance By-Example Noise using a Histogram-Preserving Blending Operator[END_REF][START_REF] Burley | On Histogram-preserving Blending for Randomized Texture Tiling[END_REF] studied the synthesis of in nite textures using a randomized texture tiling of a stochastic input, preserving sharpness along the boundaries of each tile. As we observed that the approach to RGB texture blending presented in [START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF]) falls short with high-resolution images, we take inspiration from the aforementioned methods to preserve the albedo sharpness in our morphing operator.

Material editing methods allow for the direct manipulation of existing BRDFs or SVBRDFs (see [START_REF] Schmidt | State of the art in artistic editing of appearance, lighting and material[END_REF] for a comprehensive review of material design and editing methods). (J. [START_REF] Wang | Appearance manifolds for modeling time-variant appearance of materials[END_REF]) and (R. Wang et al., 2008) navigate through the space of appearance by modeling the time-varying surface response or manipulating homogeneous di use BSSRDFs. [START_REF] Di Renzo | Appim: Linear spaces for image-based appearance editing[END_REF] propose editing spaces for material parameters, providing common editing operations such as scaling, curve tting and interpolation. Our method shares with this work the local frames editing approach. However our method di ers in its ability to morph the mesostructures from one SVBRDF to another to explore the space of possible appearances.

Contour detection. Contours are known to play a major role in human visual perception [START_REF] Papari | Edge and line oriented contour detection: State of the art[END_REF]; the cues they provide inspired a number of methods coming from the vast image processing literature. However, contour detection remains an ill-posed problem, somewhat dual to the image segmentation problem. Recently, Convolutional Neural Networks have proven their e ciency at addressing such problems, detecting contours of foreground objects as opposed to regions in the background [START_REF] Maninis | Convolutional oriented boundaries: From image segmentation to high-level tasks[END_REF]. These methods are trained on natural images, which di er signi cantly in structure from PBR maps, for which the notions of foreground/background are not well de ned. They are also costly to evaluate, making them not ideal for interactive manipulation. The resulting contours appear thicker and blurrier than classical approaches based on local operators, which we adopt in our framework. In particular, Grompone Von Gioi et al. [START_REF] Grompone Von Gioi | Unsupervised Smooth Contour Detection[END_REF] proposed an Unsupervised Smooth Contour Detection. They present an e cient algorithm producing subpixel contour detection, without the need to learn a distribution prior to the detection.

Optimal Transport provides a framework for smooth interpolation and has been used for images [START_REF] Mérigot | A Multiscale Approach to Optimal Transport[END_REF][START_REF] Xia | Synthesizing and Mixing Stationary Gaussian Texture Models[END_REF], BRDFs [START_REF] Bonneel | Displacement interpolation using lagrangian mass transport[END_REF], 3D shapes [START_REF] Lévy | A numerical algorithm for l2 semi-discrete optimal transport in 3d[END_REF][START_REF] Solomon | Convolutional wasserstein distances: E cient optimal transportation on geometric domains[END_REF], point clouds [START_REF] Bonneel | Sliced and Radon Wasserstein Barycenters of Measures[END_REF][START_REF] Bonneel | SPOT: sliced partial optimal transport[END_REF], color histograms [START_REF] Bonneel | Wasserstein barycentric coordinates: histogram regression using optimal transport[END_REF] and textures [START_REF] Rabin | Wasserstein Barycenter and Its Application to Texture Mixing[END_REF]. Recently, Nader et al. [START_REF] Nader | Instant transport maps on 2D grids[END_REF] proposed to quickly compute continuous transport maps between 2D probability densities discretized on uniform grids. Unfortunately, this framework does not aim at preserving structures during interpolation, but rather minimizing the cost of transforming one distribution to the other. Such costs are sensitive to outliers, in the sense that all mass is required to be transported to a target distribution.

Finally, with respect to learning-based methods presented in Chapter II, and in the particular context of SVBRDFs, our method does not require lengthy training and only uses two input materials to produce a continuous, structure preserving interpolation, with no need to learn an a priori distribution. Indeed, our approach can feed these approaches by providing an advanced mechanism to amplify the data set before training, as a substitute to the simple linear interpolations performed by e.g., Deschaintre et al. [START_REF] Deschaintre | Single-image SVBRDF capture with a rendering-aware deep network[END_REF] or Guo et al. (Guo, Smith, Hašan, et al., 2020) to populate their training set.

III.3 Overview

Let us denote an input SVBRDF as a function

𝑀 : [0, 1] 2 → {[0, 1] 3 , 𝑆 2 , [0, 1], [0, 1], [0, 1]}
which associates to each point of the unit square domain Ω := [0, 1] 2 a di use albedo, a normal vector as well as roughness, metallicness and height (or displacement) scalar values. Starting from two input materials 𝑀 1 and 𝑀 2 , our method (Figure III.2) aims at producing a new SVBRDF 𝑀 𝛼 by morphing between the maps of 𝑀 1 and 𝑀 2 with a ratio 𝛼. To do so, we rst compute a transport grid guided by the contours detected out of 𝑀 1 and 𝑀 2 , and use it to morph each individual map in real time. Secondly, we use two novel operators to preserve the appearance of 𝑀 1 and 𝑀 2 during the morphing. More precisely, we use a histogram aware color blending to preserve the sharpness of the input albedo maps and introduce a normal-and-height cross-interpolation mechanism to recompute the normal from the interpolated geometry and preserve details. Propelled by our transport grid, these operators yield the nal interpolated SVBRDF 𝑀 𝛼 . In its most general form, our morphing mechanism lets the user provide a spatially varying morphing ratio 𝛼 given by a scalar eld 𝛼 𝑆 : Ω → [0, 1]. This scalar eld is de ned over the same texture domain as 𝑀 and its range values modulate a per-texel interpolation of 𝑀 𝛼 , performing a spatially-varying non-linear mix between 𝑀 1 and 𝑀 2 .

III.4 Transport grid

Our transport grid models per-texel correspondences between 𝑀 1 and 𝑀 2 . We design it speci cally to avoid the default per-pixel correspondence of linear blending which yields unsatisfactory geometry during the interpolation. The transport grid aims at preserving the structures present in the material as much as possible all along the interpolation. To do so, we rst detect contours from the input materials before optimizing a deformation over the meshed unit square.

III.4.1 Contour detection

We seek feature lines present in the input materials to use them as a super structure driving the morphing process. Such lines typically separate materials regions where texel statistics are close to uniform. In our case, the di erent channels of each SVBRDF provide a rich space that we leverage to detect the dominant mesostructures.

More precisely, we base our contour extraction on the parameter-free algorithm proposed by Grompone et al. [START_REF] Grompone Von Gioi | Unsupervised Smooth Contour Detection[END_REF] which produces locally consistent results across all SVBRDF maps in a matter of seconds.

By default, we run this algorithm on the heightmap of each material 𝑀. Since our method aims at preserving shape features during the interpolation, most heightmaps are su cient to describe the geometry and produce meaningful contours. Optionally, the user can control the process by activating contour detection on a per-channel basis using a set of booleans 𝜃 = {𝜃 𝑎 , 𝜃 𝑟 , 𝜃 𝑚 , 𝜃 𝑛 , 𝜃 ℎ }. This prevents unsatisfactory detection in materials where the heightmap fails to provide sharp features. The contour detection algorithm is fed with greyscale images. For the normal map, we simply retain its z component. The albedo map is converted to greyscale using the luminance channel. Secondly, we sample the resulting polylines in the selected maps before concatenating the resulting 2D point sets into a single contour sampling C = {𝑝 1 , ..., 𝑝 𝑘 } made of 2D points 𝑝 𝑖 ∈ Ω. This yields a global contour extraction operator:

𝐷 (𝑀, 𝜃 ) → C
that we apply on both 𝑀 1 and 𝑀 2 .

III.4.2 Transport Map Generation

Our transport map provides a one-to-one correspondence for each texel between 𝑀 1 and 𝑀 2 . We aim at transporting e ciently points 𝑃 = {𝑝 𝑖 ∈ Ω} 𝑖 sampled from the source material contours C 1 in order to best align them onto a target point set 𝑄 = {𝑞 𝑗 ∈ Ω} 𝑗 sampled from the target material contours C 2 .

Since we cannot make assumptions on the input such as, e.g., perfect sampling or one-to-one structure correspondence, we design our algorithm with the prime intent of being robust to outliers. To do so, we propose to minimize a transport energy modeling a kind of elastic sparse Iterative Closest Point method [START_REF] Bouaziz | Sparse iterative closest point[END_REF]):

E t (𝑓 ) := ∑ 𝑖,𝑗 𝑤 𝑖 𝑗 𝑓 (𝑝 𝑖 ) -𝑞 𝑗 𝑠 + 𝜆E reg (𝑓 ) (III.1) s.t. 𝑓 (𝜕Ω) = 𝜕Ω
𝑤 𝑖 𝑗 being a localization kernel, 𝑠 ∈]0, 2] (𝑠 < 1 resulting in increased robustness to outliers).

Iterative solver. Minimizing this energy is nontrivial for sparsity parameters 𝑠 < 2. We present here an e cient iterative solver for this task. We start with a given con guration 𝑓 (by default: Identity). In practice, at each iteration, we restrict the double summation by taking the 𝑘 nearest neighbors of 𝑓 (𝑝 𝑖 ), and set the weights 𝑤 𝑖 𝑗 to 0 for any 𝑞 𝑗 further away from 𝑓 (𝑝 𝑖 ). Our implementation makes use of a Gaussian kernel with standard deviation 0.01. Further, we enforce harmonic (null Laplacian) deformations to regularize the solution.

We start with a uniform grid with vertices {𝑣 𝑘,𝑙 = (𝑥 𝑘,𝑙 , 𝑦 𝑘,𝑙 ) ∈ Ω; (𝑘, 𝑙) ∈ [0, 𝑁 [ 2 } stacked into the optimization unknowns vector v, together with bilinear interpolation basis functions (𝑓 (𝑝 𝑖 ) = 𝐴 𝑖 • v, 𝐴 𝑖 being the row containing the bilinear coordinates of 𝑝 𝑖 in the input grid-mesh), and the standard four-point approximation of the Laplacian operator onto the grid-mesh (Δ𝑓 (𝑣 𝑘,𝑙 ) := 𝑣 𝑘+1,𝑙 + 𝑣 𝑘 -1,𝑙 + 𝑣 𝑘,𝑙+1 + 𝑣 𝑘,𝑙 -1 -4𝑣 𝑘,𝑙 ). In order to approximate the 𝐿 𝑠 norm, we use a standard reweighting scheme, and nally minimize at each iteration

E t (v) := ∑ 𝑖,𝑗 ∈𝑁 𝑁 (𝑖 ) 𝑤 𝑖 𝑗 𝐴 𝑖 • v -𝑞 𝑗 2 (III.2) + 𝜆 ∑ 0<𝑘,𝑙 <𝑁 -1 Δ𝑓 (𝑣 𝑘,𝑙 ) 2 interior regularity term + E 𝜕 boundary terms
where

𝑤 𝑖 𝑗 = 𝑤 𝑖 𝑗 max(𝜖, 𝐴 𝑖 • v current -𝑞 𝑗 ) 𝑠 -2
is used to approximate the 𝐿 𝑠 norm objective function (𝜖 being a safety parameter here to avoid divisions by 0, 𝜖 = 10 -3 in our implementation).

In practice, we repeat this optimization 𝑇 times using a Cholesky decomposition, and advect the solution with a step 𝛿 = 𝑡/𝑇 at iteration 𝑡 ∈ [1,𝑇 ]. This allows re ning progressively the local matching 𝑁 𝑁 (𝑖) (the 𝑘 points in 𝑄 nearest from point 𝑓 current (𝑝 𝑖 ) = 𝐴 𝑖 • v current ), and typically results in better matchings when structures to be matched are not aligned properly or in presence of outliers.

Boundary conditions So far, we have not constrained the boundaries of the mesh explicitly. While typical boundary conditions such as Neumann or Dirichlet In the more general case, when dealing with dense structures to match (top), the boundary conditions impact the solution only locally. When dealing with structures to match that are very sparsely distributed (below), the boundary conditions strongly impact the solution globally, as most conditions are too loose to enforce strong regularization in these cases. Whether the inputs are tileable or not, we nd that periodic harmonicity provides the best results in all cases.

are used in a variety of transport problems in Computer Graphics, we advocate the use of ad-hoc boundary conditions that are better suited to our problem. We analyze in the following various boundary conditions and discuss their pros and cons.

Constraining the boundary image (i.e., 𝑓 (𝜕Ω) = 𝜕Ω) can be done trivially by removing the corresponding variables from the optimization, or almost equivalently, by adding a large penalty term onto those (e.g., 𝜇|𝑥 0,𝑙 -0| 2 , 𝜇|𝑥 𝑁 -1,𝑙 -1| 2 ). The boundary term takes in this case the form:

E 𝜕 + =𝜇 ∑ 𝑘 |𝑦 𝑘,0 -0| 2 + |𝑦 𝑘,𝑁 -1 -1| 2 (III.3) + 𝜇 ∑ 𝑙 |𝑥 0,𝑙 -0| 2 + |𝑥 𝑁 -1,𝑙 -1| 2
Note that tileability (i.e., 𝑓 (𝜕Ω) ≠ 𝜕Ω and {𝑓 (Ω) + 𝑖 * (1, 0) + 𝑗 * (0, 1); 𝑖, 𝑗 ∈ Z} de nes a partition of R 2 ) can be enforced instead in a trivial manner similarly, e.g., by adding a large penalty such as 𝜇|𝑥 𝑁 -1,𝑙 -𝑥 0,𝑙 -1| 2 .

Boundary smoothness can be enforced in several ways. We have studied unidimensional harmonicity as well as periodic harmonicity.

The former can be enforced by adding the following terms to E 𝜕 :

E 𝜕 + =𝜆 ∑ 0<𝑘<𝑁 -1 |2𝑥 𝑘,0 -𝑥 𝑘 -1,0 -𝑥 𝑘+1,0 | 2 (III.4) + 𝜆 ∑ 0<𝑙 <𝑁 -1 |2𝑦 0,𝑙 -𝑦 0,𝑙 -1 -𝑦 0,𝑙+1 | 2
while the latter is enforced by adding the following terms to E 𝜕 :

E 𝜕 + = 𝜆 ∑ (𝑘,𝑙 ) ∈𝜕[0,𝑁 -1] 2 Δ𝑓 (𝑣 𝑘,𝑙 ) 2
while making sure that o -grid indices point to the correct variables inducing periodicity in the Laplacian operator (e.g. 𝑥 -1,𝑙 , resp. 𝑥 𝑁 ,𝑙 , is replaced 𝑥 𝑁 -2,𝑙 , resp. 𝑥 1,𝑙 ).

We have analyzed the impact of these conditions on various examples, and we found that periodic harmonicity results in increased stability in all cases, even if the input textures are not tileable (see Fig. III.3). Note that in that case, the output interpolation will not be constrained to be tileable, but will merely be constrained to be advected in a smooth manner around the boundaries.

III.5 Material Interpolation

With our transport grid in hand, we can now perform an interpolation that produces visually appealing motions of the input materials mesostructures. The second component of our method is dedicated to the blending applied to each texel during the interpolation.

Our experiments revealed that linearly combining the input color (resp. normal) maps produces unsatisfactory results; therefore, we propose techniques to improve the blending in Section 5.1 (resp. 5.2) before describing our nal SVBRDF morphing method (Section 5.3).

III.5.1 Histogram aware color blending

When interpolating colors linearly, details from each material tend to be lacking and colors appear dull. Matusik et al. [START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF] who link this behavior to the averaging e ect led by the blending suggest extracting high-frequency statistics and enforce these statistics on the interpolated result. This is achieved by capturing high-frequency content using a multiscale frequency decomposition. They use color histogram matching and enforce the histogram of the linearly blended texture decomposition to be an interpolation of the histograms of the input decompositions. We tested their approach on albedo maps, and it appears the e ect of the decomposition vanishes when working with high resolution textures (1K and 4K). Their interpolation focuses on 128x128 textures, which do not require 𝛼 is the interpolation of the inverse CDF of images I 1 and I 2 . Our technique better preserves color sharpness compared to a linear interpolation. a multiscale approach. A simpler way of interpolating texture is done by using histogram matching as in [START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF], but with no prior decomposition of the images.

ℋ 1 ℋ 2 𝐶𝐶𝐶𝐶𝐹𝐹 -1 𝒢𝒢 ∘ 𝐶𝐶𝐶𝐶𝐹𝐹 ℋ (1 -𝑡𝑡) 𝑡𝑡 ℋ 𝛼𝛼 𝐶𝐶𝐶𝐶𝐹𝐹 𝛼𝛼 -1 ∘ 𝐶𝐶𝐶𝐶𝐹𝐹 𝒢𝒢 𝐼𝐼 2 𝐼𝐼 1 𝐼𝐼 2 𝐺𝐺 𝐼𝐼 1 𝐺𝐺 𝐼𝐼 𝛼𝛼 𝐺𝐺 𝐼𝐼 𝛼𝛼
ALGORITHM 1: Histogram aware blending of texels p1, p2

𝑝 𝛼 ← (1 -𝛼) 𝐼 𝐺 1 [(𝑥, 𝑦) 𝑝 1 ] + 𝛼 𝐼 𝐺 2 [(𝑥, 𝑦) 𝑝 2 ] 𝑊 ← √ (1 -𝛼) 2 + 𝛼 2 𝑝 𝛼 ← 𝑆 * [ Ĝ ] (𝑝 𝛼 ,𝑊 ) 𝑝 𝛼 ← 𝐿𝑈𝑇 -1 𝛼 [𝐶𝐷𝐹 𝐺 (𝑝 𝛼 )] with 𝐿𝑈𝑇 -1 𝛼 = (1 -𝛼) 𝐶𝐷𝐹 -1 1 + 𝛼 𝐶𝐷𝐹 -1 2
We take insights from Heitz et al. [START_REF] Heitz | High-Performance By-Example Noise using a Histogram-Preserving Blending Operator[END_REF] who show that blending linearly between two random variables acts as a histogram convolution, resulting in a low variance in the histogram of blended textures. Even though this result is given under the assumptions that the texture's pixels are independent and identically distributed -which breaks in our case -the idea inspired our histogram aware blending operator (see Algorithm 1). We use both the Cumulative Distribution Function of each input, and the truncated gaussian distribution from Burley et al. [START_REF] Burley | On Histogram-preserving Blending for Randomized Texture Tiling[END_REF]. Similarly to Burley, we use gaussianization with a Soft-clipping Contrast Operator 𝑆 *

[ Ĝ ]

to obtain more contrasted interpolation. We gaussianize independently each input and obtain 𝐼 𝐺 1 and 𝐼 𝐺 2 , which we then interpolate linearly before reverting the histogram equalization by applying successively the truncated gaussian CDF (𝐶𝐷𝐹 𝐺 in Algorithm 1), and the interpolated inverse CDF of the input textures through a lookup table (𝐿𝑈𝑇 -1 𝛼 in Algorithm 1). The pipeline is illustrated in Figure III.4. The whole pipeline is executed per channel. In practice, the user can choose to run our histogram aware interpolation in either RBG or YCbCr color space to prevent a slight shift in hue which occurs with one or the other depending on the input material pair.

We compare the results of di erent linearly interpolated texture enhancements (sharpness preservation, histogram matching and gaussianization) in Fig. III.12. We recall that gaussianization consists of applying Burley's approach for interpolating between two di erent textures, rather than on a single one, as proposed in his work.

III.5.2 Height and Normal cross-interpolation III.5.2.1 Normal map preprocessing

For procedural PBR materials, the normal and height maps are often created following these steps: rst, artists create displacement information which corresponds to the base geometry. This information is stored in a greyscale height map. Second, they add shading details -bump mapping -on top of the base geometry and store the resulting normal in pre-displacement tangent space, instead of the local space induced by the displacement of the geometry (post-displacement tangent space). Fetching normals from pre-displacement tangent space simpli es the computations at runtime and avoids the regeneration of the post-displacement tangent space, which does not vary across time in classical scenarios. At runtime, the per-vertex tangent-space basis is provided and helps the reorientation of the normal according to the geometry. However, this results in wrongly oriented normals when modifying the displacement factor during rendering if the tangent frame is not recomputed (Fig. III.5,bottom).

In our context, the interpolation between structures is non-trivial and such an approach fails to produce a valid interpolated normal. Besides, the extra details lie in post-displacement tangent space, and must consequently be blended in this basis. Similarly to [START_REF] Di Renzo | Appim: Linear spaces for image-based appearance editing[END_REF], we use a decomposition of the normal map w.r.t. the geometric normal. Speci cally, we start by reverse-engineering the normal baking process, in order to access these post-displacement details (bump maps). On the rst row, the normals (right hemisphere) are computed from the displaced surface. On the second row, a simple texel fetch in the normal map followed by a change of frame using the TBN matrix, overlayed on the displaced geometry. Displacing the geometry with a varying height factor (each column) creates a "painting e ect" (left hemisphere), where the normal re ections seem to be painted on the surface, no matter the orientation of the tiles.

At runtime, the geometry i.e., the height map information, is linearly interpolated. The details are combined using a spherical linear interpolation, and the resulting detail normals are consistently added back to the geometry.

To further support our argumentation about invalid normals interpolation, we illustrate the need for a joint interpolation of the height and the normal maps in Fig. III.6. When interpolating linearly the geometry between the at solid shape and the dotted shape (Fig. III.6,left), one can see that the angle de ned by the linking element between the two plateaus and the base segment follows the relation 𝜃 (𝛼) = arctan(𝛼ℎ/𝑑) (the blue curve), and not simply the one given by the SLERP interpolation of the normal map (i.e., 𝛼 arctan(ℎ/𝑑), the red curve). This angle also corresponds to the one between the normal of this linking element and the up-axis. One has therefore to distinguish between the macro-structure orientation given by the height map and the meso-structure orientation, which is described by the deviation of the normal from the macro-structure orientation. We base our height-and-normal interpolation operator on this observation. In order to preserve sharp geometric features during the interpolation, we perform a two-scale analysis of the normal map by decomposing it into a coarse and a detail map, recomposed after interpolation. To mix geometric parameters consistenly, we propose to extract the details of the two normal maps n 1 , n 2 by computing normals n ℎ 1 , n ℎ 2 derived from the coarse height maps 𝐻 1 , 𝐻 2 using a gradient operator. By reorienting the original normals into the coarse one, we extract detail maps d 1 , d 2 which are mixed using Spherical Linear Interpolation (SLERP) into a resulting mixed detail map d 𝛼 . The height maps are blended linearly to produce 𝐻 𝛼 , which provides a coarse normal map n ℎ 𝛼 , to which the details from d 𝛼 are added back to obtain the nal normal n 𝛼 . This pipeline is achieved after computing the 1-to-1 texel correspondences with the transport grid.

Following [START_REF] Di Renzo | Appim: Linear spaces for image-based appearance editing[END_REF] and Mikkelsen [START_REF] Mikkelsen | Surface gradient-based bump mapping framework[END_REF] layered approach, we propose a normal map decomposition summarized in Fig. III.7. Basically, we rst compute a detail map for each material, using the regular normal map and a coarse normal map derived from the height map. Then we interpolate linearly the heightmaps, and compute a coarse normal map from the geometry. Lastly, we use a spherical linear interpolation on the detail maps, extract the proper normal from the geometry, and add the details to get the nal interpolated normal map.

ALGORITHM 2: Normal Reorientation algorithm To extract a detail map vector d w.r.t. the original normal and height map (method getDetails in Alg. 2) (i) we de ne the coarse base normal n ℎ from the height map, (ii) we build a transform 𝑅 that rotates n ℎ into the normal map vector n containing the higher frequency signal and (iii) we apply 𝑅 to z, the unit normal of the plane (see Fig. We compute the rotation 𝑅 from the coarse geometry n ℎ to the detailed one n and apply it to a unit vector perpendicular to the plane z to obtain the detail vector d.

n ℎ 1 ← normalize((∇ 𝑥 𝐻 1 , ∇ 𝑦 𝐻 1 , ℎ 1 )) ; // compute normal from height n ℎ 2 ← normalize((∇ 𝑥 𝐻 2 , ∇ 𝑦 𝐻 2 , ℎ 2 )) d 1 ← getDetails(n ℎ 1 , n 1 ) ; // extract
To compute a normal map from each material's height map, we use a Sobel lter which provides smoother results than a simple gradient [START_REF] Weiss | Triplanar Displacement Mapping for Terrain Rendering[END_REF]. We use 16 bits height maps for better precision. However, recovering normals from the height maps with the same orientation and amplitude as the original normal maps requires the knowledge of the global height factor ℎ used to compute it originally. Therefore, we retrieve this factor as follows: let 𝐻 : [0, 1] × [0, 1] → [0, 1] be the height map and ∇ 𝑥 , ∇ 𝑦 the vertical, resp. horizontal Sobel operators, such that:

∇ 𝑥 𝐻 = -1 0 1 -2 0 2 -1 0 1 * 𝐻 and ∇ 𝑦 𝐻 = -1 -2 -1 0 0 0 1 2 1 * 𝐻
To retrieve ℎ, we solve the following optimization problem, where N is the original normal map and 𝑁 ℎ the normal map recovered from the heightmap:

argmin ℎ∈ [0,1] 𝑓 (ℎ) = ∑ (𝑖,𝑗 ) 𝑑 (𝑛 ℎ 𝑖,𝑗 , 𝑛 𝑖,𝑗 ) with 𝑑 (𝑥, 𝑦) = 𝑎𝑐𝑜𝑠 (𝑥 • 𝑦) 1
In practice we compute 128 normal maps from the height map in a compute shader with a varying height factor ℎ ∈ {𝑖/127, 𝑖 ∈ [0, 127]} prior to the rendering. Experimentally we found out that all the computed f functions show a convex behavior, thus allowing to easily recover the ℎ parameter.

III.5.3 Practical Implementation

Our transport map 𝑓 provides a one-to-one correspondence between both materials. As previously stated, the user can provide a scalar eld 𝛼 𝑆 with values between 0 and 1 at each texel to control the interpolation. This eld is parameterized both spatially and temporally and drives the transport function:

𝑓 𝑆 : Ω → Ω, 𝑓 𝑆 (𝑝) = (1 -𝛼 𝑆 (𝑝))𝑝 + 𝛼 𝑆 (𝑝)𝑓 (𝑝)
For a point 𝑝 1 ∈ Ω describing a location in 𝑀 1 and the corresponding interpolation parameter 𝛼 fetched from the 𝛼 𝑆 , we compute the value at the interpolated point 𝑝 𝛼 by linearly interpolating between its source 𝑝 1 and its target 𝑓 (𝑝 1 ) = 𝑝 2 ∈ Ω, where we gather the corresponding SVBRDF parameters in 𝑀 2 :

𝑝 𝛼 = (1 -𝛼)𝑝 1 + 𝛼𝑝 2 , (III.5)
At 𝑝 𝛼 , we map values of 𝑀 1 at 𝑝 1 , and those of 𝑀 2 at 𝑝 2 . For the height and metallic maps, we linearly blend the values to obtain those of the nal maps -note that PBR materials being physically inspired rather than physically correct [START_REF] Burley | Practical physically-based shading in lm and game production[END_REF], linearly interpolating the metallic term is considered the industry de facto standard for interpolation. We observe that the transition from shiny metal to a rusty or worn aspect gives a pleasing result. Regarding roughness, [START_REF] Di Renzo | Appim: Linear spaces for image-based appearance editing[END_REF] proposes to interpolate the cumulated distribution functions of the BRDFs, which can be seen as a form of optimal-transport-based interpolation. While this framework provides satisfying results and is mathematically well-grounded, we chose to simply interpolate the square mapping of the roughness, as advocated in Disney's principled BRDF model [START_REF] Burley | Practical physically-based shading in lm and game production[END_REF], as artists observe empirically that this results in a perceptual linear change of the materials (see Section 5.4: Specular D details). For the albedo and normal, we use our previously introduced operators. Those values are fed to a standard microfacet model with a GGX normal distribution function for rendering.

In practice, we run this process in real time using the programmable rendering pipeline of modern GPUs. We rst rasterize the displaced grid using rest positions as a color attribute. The displacement of each vertex 𝑣 is computed as 𝑓 𝑆 (𝑣) (i.e., using the 𝛼 values at their original position in 𝛼 𝑆 , see Equation III.5), thus resulting e ectively in a coarse linear rendering of 𝑓 -1 𝑆 : Ω ↦ → Ω. At each output texel 𝑞 of the computed image, we rst read the coordinate 𝑝 1 = 𝑓 -1 𝑆 (𝑞) of the texel 𝑚 1 from 𝑀 1 to blend, we then fetch 𝛼 at position 𝑝 1 , and we compute the coordinate 𝑝 2 = 𝑓 (𝑝 1 ) of the texel 𝑝 2 from 𝑀 2 to blend. With these at hand, we then interpolate the materials' texels 𝑚 1 and 𝑚 2 accordingly, and nally shade the output fragment. Being real time, our method allows the user to vary the blending eld 𝛼 𝑆 interactively using a brush metaphor or the parameters of a procedural texture.

III.6 Evaluation and Applications

In Fig. III.14, we present a collection of interpolation results obtained with our method on a set of pairs of SVBRDFs. Note that videos are better suited to visualize temporal interpolation, and that we provide video comparisons in the supplemental materials of the corresponding article.

III.6.1 Ablation study

We evaluate the bene t of individual components of our approach in an ablation study.

Transport Grid. Our transport grid allows for a clean contour registration which morphs the structures of the rst input material into the others. We show in Fig. III.9 that when a contour from one material has no match in the other material, a ridge collapses or appears while preserving its shape. This is the desired transport behavior when outliers occur. We compare our transport grid with [START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF], which fails to produce precise contour registration between maps.

Normal Reorientation allows to preserve the shape features of the geometry along the interpolation. Compared to spherical linear interpolation, which produces wrongly oriented normals, this component ensures proper detail preservation (see Fig.

III.6.2 Comparisons

We reimplemented the method of [START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF] in order to compare their results to ours, in the restricted context of RGB texture interpolation. We provide a comparison in Fig. III.11, and explain the shortcomings of the method when dealing with high resolution SVBRDF maps. Their technique is composed of two main steps: the warp grid computation and the RGB texture interpolation. First, the warp grid computation reveals discontinuity artifacts, which result from the independent per-vertex optimization at each iteration. The method fails to provide a globally smooth warpgrid, and as a result creates discontinuities at the borders of each region inside materials. We also noted that processing a material as a 9D stack of greyscale images fails to create salient feature maps to blend. Therefore, we exhaustively selected the best feature maps combination for each of our example materials, visualizing the resulting halfway morphing between albedo maps. This results in warpgrids which are not able to consistently align features in materials.

Second, the steerable pyramid decomposition followed by histogram matching [START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF] sharpness preservation fails to enhance details on 1K pairs of images and helps to sharpen the result at the pixel scale (require zooming-in). Simpler histogram matching of interpolated textures produces sharp results but has a higher runtime complexity than texture gaussianization, which helps to preserve details after linear blending. Note that no warping is done for this comparison.

falls short when dealing with high resolution textures. We applied the technique on both 128x128 textures (as in their work) and with 1K and 4K textures, and as the resolution increases, the e ect of matching histograms on the ltered images fades progressively.

We noticed that in most cases, simply applying histogram matching [START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF] to the whole image with no predecomposition yields better contrasted results. In Fig. III.12, we provide a comparison between linearly interpolated textures. The advantage of this method over simpler histogram matching is that the histogram of the interpolated texture is known during interpolation, and does not require a pass through the whole image to compute the histogram of the interpolated texture. As explained previously, simpler histogram matching can be used as a way of sharpening the interpolated texture. Unfortunately, this method has a runtime complexity of 𝑂 (𝑁 ) (N being the total number of pixels of each image) compared to the constant 𝑂 (1) complexity of the gaussianization (in that case the histogram matching happens during precomputation), which makes real-time editing possible.

We also compare our technique to MaterialGAN (Guo, Smith, Hašan, et al., 2020), which proposes the exploration of a learned distribution of material appearance by

III.6.3 Performance

The transport grid precomputation depends on the number of contour samples detected in each material. We report the timings of the contour detection for a single map at 4K resolution (which is downsampled at 1K beforehands only for the contour detection phase), along with the transport grid computation in Table III.1. For this experiment, all contour points were provided to the later stage (ranging from 10'000 up to 80'000) but a resampling can be done to achieve better performance. At runtime, the dynamic interpolation cost of our method is roughly twice higher than linear interpolation, still allowing for real time design of the interpolation scalar eld 𝛼 𝑆 . In Table III.1 we compare our method against a simple linear interpolation and report the precomputation timing, all measured on a Ryzen 5 5600X CPU running at 4.6GHz equipped with a Nvidia RTX 2080 Ti GPU. We stress that temporal interpolations are better viewed in videos (provided as supplemental materials).

III.7 Limitations and Future Work

Our method struggles with materials which contain only stochastic structures such as moss or dirt, and is favored when materials exhibit medium to large salient structures. We also assume a rough initial alignment of the two inputs when they are highly periodic, which could be automated but raises the question of preserving tileability if present. Similarly, the transport grid provides good contour matches when the mesostructures of the two inputs are spatially close. Otherwise, the interpolation results in a simple blend for the regions where no structures can be blended with.

Last, even if most of our material examples present a low frequency periodicity in their structures, our technique does not exploit this selfsimilarity, which is an interesting direction for future work.

III.8 Discussion

We presented a novel structure-preserving operator for interpolating two SVBRDFs.

To do so, we introduced a transport grid which guides the interpolation based on the dominant mesostructures detected in the input, and combined it with normal reorientation and histogram-aware color blending to better preserve details and sharp features.

Our operator is fast enough to run in real time, providing a simple mechanism to explore material variations out of a set of exemplars, with a visual quality which is superior to previous methods, for a cost which is close to linear interpolation. Our approach proposes multiple techniques which can be used independently.

For instance, the height factor retrieval from Section III.5.2 can bene t work ows which produce both a geometry and its corresponding normal map.

As opposed to linear interpolation in euclidian space, which produces unplausible In the context of realtime, by-example material design, interactivity is a key component for artistic freedom. Providing users with realtime feedback during the creation process prevents frustration and allows them to rapidly explore multiple design strategies. This aspect is even more crucial as artists tend to deal with assets of dozens of high resolution (4K) textures. This realtime feedback should be given for material design, but also during rendering, because the SVBRDF maps' expressivity is only made possible by the shading process which feed them to the rendering equation. We proposed a tool which enables realtime feedback for the exploration by the user.

However, e ciently rendering materials in constrained real-time scenarios requires material information to be encoded at a given scale for a given distance of observation. This loss of generality prevents e cient and accurate renderings especially when zooming out. Indeed, the pixel-to-texel ratio decreases with the distance, one pixel covering a larger patch of the surface where textures are mapped onto. This phenomenon leads to sampling artifacts characterized by temporal ickering and aliasing e.g., Moiré patterns or staircase e ects. To enable further exploration of the space of material appearance, we propose to tackle the exploration across level-of-details. We aim at artefact-free and interactive renderings of materials, lying in the space of appearance.

IV.1 Problem Analysis

In the previous chapter, we have described how relatively simple operators in image space achieve e cient and structure preserving material morphing between two existing materials 𝑀 1 and 𝑀 2 , following:

𝑀 𝛼 = (1 -𝛼) 𝑀 1 ⊕ 𝛼 𝑀 2 .
Re ectance ltering however lacks simple tools because of the nonlinear behavior of the shading with respect to the SVBRDF maps. Our operator could be used naively to achieve SVBRDF mipmapping of four texels of the same material into one: p

= 1 4 𝑝 1 ⊕ 1 4 𝑝 2 ⊕ 1 4 𝑝 3 1 4 𝑝 4 (see Fig.IV.1, left).
But ltering a material for representing its appearance into a mipmap pyramid requires geometric information transfer across maps such as height, normal and roughness.

Traditional anti-aliasing techniques for color textures ltering rely on mipmapping [START_REF] Williams | Pyramidal Parametrics[END_REF]. The method's underlying assumption is based on a linear relationship between texels in the textures to pre lter and the nal rendered pixel. This pre ltering method is widely available on current graphics API and eliminates most of the typical rendering artifacts mentioned above. Unfortunately, using mipmapping independently and on various texture types, e.g., containing geometric information such as the normal and displacement map, is erroneous. As illustrated in Fig. III.15, the euclidian space in not suitable for mixing or interpolating between SVBRDF maps.

When looking at a material from afar, what were considered as macro-or mesoscale details become microscopic details (with respect to the distance of observation), which yields a new source of roughness of the surface. Several methods [START_REF] Olano | Lean mapping[END_REF][START_REF] Hery | Geometry into shading[END_REF][START_REF] Dupuy | Linear e cient antialiased displacement and re ectance mapping[END_REF] address this issue by transferring bump map (i.e., normal or displacement) information into roughness. However, the shading model used for the transfer is based on the Beckmann normal distribution function (NDF) and assumes a gaussian distribution of slopes at all scales, which is not compatible with the more commonly used [START_REF] Trowbridge | Average irregularity representation of a rough surface for ray re ection[END_REF] (often referred to as GGX [START_REF] Walter | Microfacet models for refraction through rough surfaces[END_REF]) distribution. This hypothesis fails for many PBR materials (e.g., highly structured), which exhibit a wide variety of geometric statistics depending on the distance to the observer. In addition, it has been noticed previously [START_REF] Estevez | A microfacet-based shadowing function to solve the bump terminator problem: High-quality and realtime rendering with dxr and other apis[END_REF] that the GGX lobe, being based on a Cauchy distribution, has unde ned mean and variance. To circumvent this issue, Patry [START_REF] Patry | Physically based shading in theory and practice[END_REF] uses an heuristic based on the SGGX article [START_REF] Heitz | The sggx micro ake distribution[END_REF] to linearly downsample SGGX matrices, of which GGX is a special case. This heuristic fails for many in-the-wild SVBRDFs which exhibit strong patterns is their normal maps.

Our goal is to provide the best possible mipmap to a given integrator compatible with PBR materials description, using no extra space for texture at the shading stage than classical MIP mapping. We propose to replace the simple mipmapping operator with a learned downsampling lter based on a multilayer perceptron cascade, which learns information transfer across level-of-details. To do so, we employ a di erentiable rendering pipeline along with a rendering loss. Each input texture patch is given as input to the neural architecture which concurrently downsamples each channel of the SVBRDF. The result is fed to the renderer and compared to a multisampled groundtruth computed on the y. We focus our e orts on two anisotropic BRDF models to account for appearance changes at varying scales, namely Cook-Torrance with the GGX and Beckmann variants for the normal distribution functions (NDF), and the Ashikhmin-Shirley (2000) (a.k.a. Anisotropic Phong) model. These models grant a wider variety of highlights, especially for isotropic materials which showcase anisotropic behavior when viewed from afar.

The remaining of this chapter is organised as follows: we rst present works related to normal map ltering using anisotropic representations in Section IV.2. Then, we develop a method for mipmapping anisotropic SVBRDF models, which relies on a novel tensor-based formulation for anisotropic roughness, and a learningbased pipeline crafted for this task. We describe these components thoroughly in Section IV.3. An overview of our approach for learning mipmapping lters is provided, as well as implementation details. We show the results of our method in Section IV.4, which provides better appearance preservation compared to previous work. Finally, we o er a discussion about the limitation of our method and conclude by identifying future works in Section IV.5.

Our contributions are (i) an e cient pipeline for learning mipmapping lters requiring no data preparation for training, (ii) a neural architecture encoding anisotropic appearance and generalising on unseen materials, and (iii) a tensorbased formulation for anisotropic BRDF distributions which is well-suited for di erentiable pipelines and trilinear interpolation.

IV.2 Related work

Normal map ltering. The most recent and widely adopted techniques, LEAN [START_REF] Olano | Lean mapping[END_REF], LEADR [START_REF] Dupuy | Linear e cient antialiased displacement and re ectance mapping[END_REF]) and Pixar's Bump to Roughness [START_REF] Hery | Geometry into shading[END_REF], make use of additional maps encoding geometric statistics, namely the mean and/or variance of the normal map or displacement map, which behave well with classic mipmapping. They use an anisotropic Beckmann distribution to represent the appearance at all scales and make the assumption that the small-scale geometry contained in the normal map follows gaussian statistics. This assumption does not hold for many materials and is incompatible with other BRDF distributions such as Phong or GGX. We propose a comparison between our model learnt on the Beckmann model and LEADR, which limitations are further discussed in Sec IV.4.

Patry [START_REF] Patry | Physically based shading in theory and practice[END_REF] recently proposed a SGGX-based ltering, compatible with GGX distribution, to lter anisotropic specular maps. The GGX distribution is shown to be a special case of the SGGX distribution, which is encoded using a tensor-based representation. The original paper [START_REF] Heitz | The sggx micro ake distribution[END_REF] provides a parameter estimation algorithm to convert a given NDF into SGGX matrix parameters. However, they show that linear interpolation of matrix parameters is a good approximation of this algorithm. Hence, Patry propose to linearly lter SGGX matrix parameters extracted from an anisotropic GGX distribution at runtime. The eigenvalues and eigenvectors of the ltered matrix are extracted and fed to a classical anisotropic GGX distribution. We take inspiration from their method, and use a tensor-based representation which behaves well with linear ltering. However we avoid the eigenvalue decomposition which leads to artifacts for extreme tensor value con gurations. We propose a comparison between our model learnt on the GGX model and their work in Sec IV.4, which we call SSGT (Samurai Shading at Ghost of Tsushima).

IV.3 Method

IV.3.1 Training from anisotropic SVBRDF models

To demonstrate our technique, we focus on two commonly used BRDF models: the Cook-Torrance microfacet model along with the anisotropic GGX [START_REF] Walter | Microfacet models for refraction through rough surfaces[END_REF] and Beckmann [START_REF] Beckmann | The scattering of electromagnetic waves from rough surfaces[END_REF] microfacet distributions, using the [START_REF] Schlick | An inexpensive brdf model for physically-based rendering[END_REF] Fresnel term and the uncorrelated Smith shadowing-masking term [START_REF] Heitz | Understanding the masking-shadowing function in microfacet-based brdfs[END_REF]; and the Anisotropic Phong model [START_REF] Ashikhmin | An anisotropic phong brdf model[END_REF]. Our mipmapping framework takes as input the base color, normal, metallic, height (or displacement) and two roughness maps along with an anisotropy angle map. When the input material is isotropic, the two roughness maps are identical and the angle map contains only 0 degree angles. The base color map contains both di use albedo and specular information, and is modulated by the metallic map. This results in a linear relationship between di use albedo, specular color and the resulting radiance. On top of in uencing the shading for viewpoints above the surface, the height channel impacts the coarse geometric aspect of the shape through the de nition of its silhouettes. For this reason, we rely on well-established geometric simpli cation methods for the height channel, and use as baseline the de facto standard linear mipmapping. Thus we focus our e orts on mipmapping the remaining maps that most in uence the shading at di erent scales, namely normal and roughness.

We follow those well-established common good practices:

1. We interpolate (and average trilinearly when rendering from the created mipmaps) the linearly-perceptual roughness 𝛼 instead of the roughness 𝑟 . [START_REF] Burley | Practical physically-based shading in lm and game production[END_REF] suggests indeed that 𝛼 = 𝑟 2 remaps correctly the roughness in [0, 1], resulting in a progressive linear behavior for the remapped roughness.

2. We forbid too small values for the roughness, to avoid numerical instabilities when using GGX or Beckmann distributions with point lights. Following [START_REF] Lagarde | Moving frostbite to physically based rendering[END_REF], we enforce 𝑟 ≥ 𝜖 𝑟 = 0.045, resulting in

𝛼 ≥ 𝜖 𝛼 = 𝜖 2 𝑟 0.002.
3. Because we aim at mipmapping (and averaging) non-axis-aligned roughness values, we use a tensor-based representation. Since the angle-based representation is not unique (adding 𝜋 to the angle results in the same anisotropy, and isotropic tensors have unde ned angles by de nition), averaging 4 of those representations during mipmapping (or rendering using the mipmaps) is an ill-de ned operation, while averaging the tensors is well-de ned. This common observation motivated among others the SGGX-based ltering [START_REF] Patry | Physically based shading in theory and practice[END_REF], to which we compare our technique in this paper.

Following these last observations, we encode the anisotropic linearly-perceptual roughness as a symmetric 2 × 2 tensor:

𝐴 := 𝑅 𝛾 𝛼 𝑏 0 0 𝛼 𝑡 𝑅 𝑇 𝛾 := 𝑎 𝑐 𝑐 𝑏 . (IV.1)
Given this de nition, note that 𝐴 2 has the same eigenvectors as 𝐴, and its eigenvalues are the squared eigenvalues of 𝐴:

𝐴 2 = 𝑅 𝛾 𝛼 2 𝑏 0 0 𝛼 2 𝑡 𝑅 𝑇 𝛾 (IV.2)
We provide in Appendix A the expressions for the anisotropic GGX, Beckmann and Ashikhmin-Shirley models using this representation, which we use in our renderer and learning architecture.

When learning (or mipmapping, or averaging) the channels (𝑎, 𝑏, 𝑐) of 𝐴, we have to ensure that those correspond to physically-plausible roughness values (i.e., the eigenvalues of 𝐴): 𝜖 𝛼 ≤ 𝛼 ≤ 1. We derive the following constraints on 𝐴's trace, determinant, and eigenvalues:

det(𝐴) = 𝑎𝑏 -𝑐 2 = 𝛼 𝑏 𝛼 𝑡 ≥ 𝜖 2 𝛼 trace(𝐴) = 𝑎 + 𝑏 = 𝛼 𝑏 + 𝛼 𝑡 ≥ 2𝜖 𝛼 𝜖 𝛼 ≤ 𝑎 = cos 2 (𝛾)𝛼 𝑏 + sin 2 (𝛾)𝛼 𝑡 ≤ 1 𝜖 𝛼 ≤ 𝑏 = sin 2 (𝛾)𝛼 𝑏 + cos 2 (𝛾)𝛼 𝑡 ≤ 1
Constraining the largest eigenvalue to be less than 1 leads to:

𝑎 + 𝑏 + √ (𝑎 -𝑏) 2 + 4𝑐 2 2 ≤ 1 ⇔ 𝑐 2 ≤ (1 -𝑎)(1 -𝑏)
Similarly, constraining the smallest one to be larger than 𝜖 𝛼 leads to:

𝑎 + 𝑏 - √ (𝑎 -𝑏) 2 + 4𝑐 2 2 ≥ 𝜖 𝛼 ⇔ 𝑐 2 ≤ (𝑎 -𝜖 𝛼 )(𝑏 -𝜖 𝛼 )
In summary, the following constraints need to be met:

𝜖 𝛼 ≤ 𝑎 ≤ 1 (IV.3) 𝜖 𝛼 ≤ 𝑏 ≤ 1 (IV.4) 𝑐 2 ≤ min(𝑎𝑏 -𝜖 2 𝛼 , (1-𝑎)(1-𝑏), (𝑎-𝜖 𝛼 )(𝑏 -𝜖 𝛼 )) =: 𝑐 2 𝑚𝑎𝑥 (IV.5)
To ensure those, we propose a simple projection procedure, which leads to good results in practice in our observations. We rst clamp 𝑎 and 𝑏 within

[𝛼 𝑚𝑖𝑛 , 1], before clamping 𝑐 within - √ 𝑐 2 𝑚𝑎𝑥 , √ 𝑐 2 𝑚𝑎𝑥 .
Note that this projection is not an orthogonal projection onto the space of valid (𝑎, 𝑏, 𝑐) tensor channels, but it is close enough in practice for our application scenarios. Note also that the rst two conditions on (𝑎, 𝑏) ensure that 𝑐 2 𝑚𝑎𝑥 is positive, which makes our procedure valid for any input (𝑎, 𝑏, 𝑐) channels. These bounds ensure physically-plausible roughness values, and result in increased stability of gradient-based optimizations, as they ensure that none of the BRDFs we consider (see Appendix A) take negative or in nite (unde ned) values.

IV.3.2 Mipmapping

We de ne the BRDF as a function 𝑓 : x, 𝜔 𝑖 , 𝜔 𝑜 ∈ R 𝑑 × Ω 2 → R 3 , where x is a vector of 𝑑 material parameters. For a texel at position 𝑝 ∈ R 2 , the 𝑘 𝑡ℎ level-of-detail of a SVBRDF pyramid is expressed as follows:

LoD 𝑘 : 𝑝 ∈ R 2 → LoD 𝑘 (𝑝) ∈ R 𝑑 (IV.6)
Classic mipmapping relies on a simple texel average per channel. This supposes that there is a linear relationship between the SVBRDF maps and the BRDF. However, as described in [START_REF] Bruneton | A survey of nonlinear pre ltering methods for e cient and accurate surface shading[END_REF], it is not the case in general. We aim at nding a translation-invariant kernel H , which computes any texel at LoD 𝑘 from LoD 𝑘 -2 and LoD 𝑘 -1 , such that for any level-of-detail 𝑘, and position 𝑝:

LoD 𝑘 (𝑝) H ({LoD 𝑘 -2 (P 𝑘 -2 (𝑝)) , LoD 𝑘 -1 (P 𝑘 -1 (𝑝))}) (IV.7)
where P 𝑘 (𝑝) ⊂ R 2 covers the footprint of texel 𝑝 in LoD 𝑘 .

We can construct the mipmap by applying the kernel over all patches of the 𝑖 -2 𝑡ℎ and 𝑖 -1 𝑡ℎ levels and get the level 𝑖. To do so, we learn a kernel model to minimize the di erence between rendered values. Given a SVBRDF at base level LoD 0 , we minimize the 𝐿 1 distance over all positions 𝑝 between the ground truth radiance of the material computed from averaged radiance values corresponding to the footprint of 𝑝, called GT, and the rendering of a single sample per pixel radiance at position 𝑝. We de ne the rendering loss 𝐿 total to minimize as follows: H 𝐵 requires a larger capacity than H 𝐴 because of its higher dimensional input. The groundtruth is computed using the full resolution input SVBRDF.

𝐿 total ∑ 𝑘 ≥1 ∑ 𝑝 ∈LoD 𝑘 ∑ 𝜔 𝑖 ,𝜔 𝑜 ∈Ω 𝐿 𝑘 (𝑝, 𝜔 𝑖 , 𝜔 𝑜 ) , s.t. (IV.8) 𝐿 𝑘 (𝑝, 𝜔 𝑖 , 𝜔 𝑜 ) = 𝑓 (LoD 𝑘 (𝑝) , 𝜔 𝑖 , 𝜔 𝑜 ) -GT (𝑝, 𝜔 𝑖 , 𝜔 𝑜 ) 1 , (IV.9) GT (𝑝, 𝜔 𝑖 , 𝜔 𝑜 ) = 1 #{P 0 (𝑝)} ∑ 𝑥 ∈ P 0 (𝑝 ) 𝑓 (LoD 0 (𝑥) , 𝜔 𝑖 , 𝜔 𝑜 ) (IV.10)
For the computation of LoD 1 , only LoD 0 is required. Note that our loss depends on the actual BRDF model used for rendering, but any model which supports di erentiation can be used beyond the ones we chose to illustrate our approach.

IV.3.3 Overview

Neural architecture Our normal-roughness downsampling kernels are implemented as a cascade of multiple fully-connected MLPs (see Fig. IV.2): H 𝐴 and H 𝐵 , which downsample at half and quarter resolution, respectively. In all of our experiments, we use four occurrences of the networks H 𝐴 and H 𝐵 (i.e., the orange block in Fig. IV.2 is repeated four times). To better capture the anisotropic behavior of certain materials, each network is required to process anisotropic data, even when those are missing from the base LoD. We here follow the common assumption that isotropic materials may appear as anisotropic when seen from afar. Compared to a single-level architecture (Sec. IV.3.4), our cascade grants each network visibility over complex multiscale phenomena which only appear progressively in the LoD, while being e cient even if shallow (Sec. IV.3.5). Enforcing network compactnessby limiting the size and the number of the hidden layers of the MLPs to a minimum -leads to better generalisation [START_REF] Heaton | Ian goodfellow, yoshua bengio, and aaron courville: Deep learning[END_REF], prevents material over tting and o ers faster training and inference. Hence, we apply the networks on small local texel footprints to keep the number of weights to a minimum. H 𝐴 processes 2x2 patches of material parameters, where H 𝐵 takes as inputs three patches of size 4x4, 2x2 and 1x1. Also, we noticed faster convergence when learning di erences from the averaged maps. Hence we use the linearly downsampled versions of LoD 0 and LoD 1 in addition with the network output to compute LoD 1 and LoD 2 , respectively.

SVBRDF processing

Our pipeline aims at concurrently downsampling normal and anisotropic roughness maps through our learned kernels. We provide patches of the full resolution maps once to H 𝐴 , which outputs a 2x downsample. This output is fed a second time to H 𝐴 , which results in a 4x downsample. H 𝐵 gets the original, 2x and 4x downsampled maps as input and produces a 4x downsample. The metallic and albedo maps are linearly downsampled at half and quarter resolution and provided to the renderer, along with the downsampled normal and roughness maps from the rst output of H 𝐴 and the ones from H 𝐵 . This allows to render a texture patch at half and quarter of the original resolution. The height is downsampled linearly and is used to compute the light position in the rendering pass. Alternatively, height downsampling can optimize for error quadrics [START_REF] Garland | Surface simpli cation using quadric error metrics[END_REF]Trettner & Kobbelt, 2020).

Physically-based renderer. We implement a di erentiable physically-based renderer in PyTorch. As input, we provide the downsampled material maps, as well as a lighting setup. We render materials on a plane viewed perpendicularly which eases UV (texels)/screen (rendered pixels) mapping. Tonemapping the radiance is common to prevent numerical instabilities for gradient descent optimization methods [START_REF] Deschaintre | Single-image SVBRDF capture with a rendering-aware deep network[END_REF][START_REF] Hasselgren | Appearance-Driven Automatic 3D Model Simpli cation[END_REF][START_REF] Kuznetsov | Neumip: Multi-resolution neural materials[END_REF]. We compute the rendering loss after tonemapping because our aim is to optimize mipmaps according to the perceived appearance, rather than the raw responses.

The operator is interchangeable in the training, and can adapt to the downstream implementation to which the mipmaps will be fed. We tested multiple tonemappers for learning our mipmapping lters, and found out a simple [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] worked better than e.g., a logarithmic tonemapper as used in [START_REF] Kuznetsov | Neumip: Multi-resolution neural materials[END_REF].

Training loss We divide our loss in a sum of terms, described in Eq. IV.8, computed on the tonemapped radiance values of our renders, using Loss 𝑖 corresponding to the computation of level-of-detail LoD 𝑖 . Each loss is computed as the 𝐿 1 perpixel distance between ground truth and mipmapped renderings. To compute the groundtruth, we render materials with a 1:1 pixel to texel ratio at the original SVBRDF resolution. The rendering is then bilinearly downsampled until it reaches the target resolution for the corresponding loss.

Linear Mipmap Network Input

IV.3.4 Single-resolution neural architecture

To warm up, let us rst discuss the use of a single 2x downsampling kernel H 𝐴 (see Fig. IV.3). This pipeline learns LoD 1 from LoD 0 only. The inference for creating the mipmap pyramid is straightforward: each level of detail LoD 𝑖+1 is computed from the previously given level LoD 𝑖 using H 𝐴 .

This model already provides improvements over a simple average of all maps in the SVBRDF mipmap pyramid, even when using a small number of network parameters (2 fully connected hidden layers, both of size 16). The network is able to generalise over unseen materials when trained with a su cient number of materials. Similarly to LEAN and LEADR, we observe a transfer from normal to roughness where the variance among normals over a patch is high. However, this simpli ed model su ers from a number of artifacts. When applied successively at each level of the pyramid (in the manner of Fig. IV.4), the network tends to accumulate errors across previously unseen LoDs. Additionally, this architecture cannot reliably create e ects in the mipmap, which were not present in the input, such as anisotropy. Indeed, such e ets are not present in the training set. This motivates our cascaded architecture, where H 𝐴 learns to handle such data as input on top of outputting it. 2) is designed to make H 𝐴 learn to output complex e ects such as anisotropy as well as treat it robustly as input, even when those e ects are not present in the rst level-of-detail, but emerge progressively from the mesostructures convolved at higher mip levels. Rather than simply learning jointly several levels of the mipmap (which we evaluate in Sec. IV.4), we introduce a two-level network H 𝐵 jointly trained with H 𝐴 (orange dotted group in Fig. Our nal multi-resolution architecture requires a 4×4 footprint for the computation of a single texel in the coarser mipmap levels, which makes the model compact enough to allow for robust generalisation capabilities, and fast enough at mipmap inference (less than a second for the whole pyramid).

Architecture hyper-parameters

In our experiments, we found that setting H 𝐴 to be composed of 2 hidden layers of size 512 and H 𝐵 of 3 layers of size 1024 resulted in high-quality results and a good generalisation behavior, indicating that our model is compact enough to avoid over tting.

IV.3.6 Training setup

Ground truth training data. We train our model with a ground truth computed just-in-time, in the sense that no precomputation is required prior to training. In our implementation, we use the same rendering pipeline for training and computing the ground truth, since di erentiation can simply be disabled for the latter.

Training details We noticed that training the network using only a few light directions leads to an unstable training. To improve convergence, we use up to 32 sample points of the Hammersley sequence distributed on the hemisphere. We use batches of size 16 using 32 × 32 texture patches to compute the loss. We adopt a learning rate of 0.0001 along with an Adam optimizer.

IV.4 Results

In this section we rst compare with previous work, both with methods performing generic on-the-y SVBRDF mipmapping, and methods relying on per-material optimizations. Our main results are illustrated in Figures IV.7, IV.8 and IV.9, on the GGX, Beckmann and Ashikhmin-Shirley models respectively. These gures feature results of the materials from several point of views to showcase the main e ects appearing in the created mipmaps. Note that we provide all these scenes along with our viewer as additional material.

Our method relies on an o ine preprocessing step, which optimizes the weights of our neural networks. The training of this model was performed on 1104 materials using 40 epochs, taking approximately 3h per epoch using a Nvidia V100 GPU, and requires less than 2Go of VRAM. The test set is composed of 100 materials unseen during training. Once trained, our model generates the pre ltered version of any given material with linear complexity in the number of texels. Our (unoptimized) python implementation takes less than a second to generate the full mipmap for 4096 × 4096 SVBRDF maps.

We report quantitative comparisons in Table IV.1 between the baseline, MIPNet and our two competitors for the two previously mentionned BRDF models: Cook-Torrance with GGX and Beckmann NDFs, and the Anisotropic Phong model. Our method consistently outperforms the baseline and our competitors, which sometimes perform poorly compared to the baseline. These methods tend to overblur the normal map or add disproportional roughness o sets, which fails to preserve the appearance from close up views as well as distant ones. Our approach improves on average over the baseline, but some speci c regions might be still better with the baseline. Note that materials which showcase relatively at normal maps do not favor either techniques, since the transfer from normal to anisotropic roughness is limited. In this case, our method outputs a at normal map which does not a ect the roughness ltering. Still, our approach improves signi cantly compared to the baseline on most cases -for instance structured shiny materials BRDF method train set (error ×10 -3 ) test set (error ×10 Table IV.1: Quantitative comparisons between the baseline, our method and the competition on each BRDF variant. The train set is composed of 1104 di erent materials from 14 categories. The test set is composed of 100 materials, identi ed as the most di cult (given by baseline error). The best result for each comparison is highlighted in bold.

featuring anisotropic e ects-and avoids large arti cial specular spots. We provide renders from multiple view and distance con gurations in the supplementary materials of the corresponding article.

IV.4.1 Comparison to generic SVBRDF mipmapping

We compare our method with state-of-the-art techniques which generate materials mipmaps used by standard shading integrators, and use the common 4-to-1 texel averaging as a baseline, implemented by glGenerateMipmap in OpenGL (Khronos Group, 2022). As our method, these techniques can process a material instantaneously. We evaluate the ability of our method to preserve the material appearance at di erent scales, under various point-of-views and lighting directions. We report our ndings in Table IV.1.

GGX: comparison with SSGT We present renderings using our technique for the GGX distribution in Fig. IV.7, in which we compare our results to standard linear mipmapping (the baseline) and with SSGT [START_REF] Patry | Physically based shading in theory and practice[END_REF] which is specialized in GGX distributions mipmapping.

We remind that with SSGT, the anisotropic roughness and the normal are packed into a 3 × 3 symmetric tensor (sharing similarities with our roughness encoding on that point), and this representation is mipmapped. At shading time, the local normal is extracted from the mipmapped linearly-interpolated tensor as its largest eigenvector, and the 2 × 2 roughness tensor as the orthogonal part to the normal.

While we observe in practice a behavior closer to the groundtruth than the baseline on many examples, this technique can extract strongly biased lobes on challenging materials, where the normal map features strong discontinuities (materials III, IV, V in Fig. IV.7), and is often further away from the groundtruth than the baseline on this type of examples.

Our results consistently feature the overall anisotropy e ects appearing when the scene is rendered from further away on these challenging examples. Note that we still observe cases where our approach leads to incorrect results on some viewpoints. We analyze this in the limitations section (section IV.5). More results can be found in the supplementary material of the corresponding article. collection of view and light conditions. We modify their publicly available implementation to better match our context, by adding an anisotropic GGX model and disabling the optimization of the geometry, the albedo map, and the metallic map.

AutoLoD uses an expensive per-object optimization, requiring around 1h and 11GB of VRAM on a Quadro RTX 6000, for a material with resolution of 1024 × 1024 (20k iterations with a learning rate of 0.003). The main drawback of AutoLoD resides in its heavy optimization, which is strongly tied to a single material and does not allow generalisation.

IV.5 Discussion & Future Work

Training strategies Multiple strategies for material mipmapping are made possible with our method. First, we consider over tting the training on a single material SVBRDF. This results in better appearance than standard mipmapping in general but sometimes fails to preserve the appearance for the unseen LoDs of the pyramid, which indices are higher than the number of cascades of the pipeline. Another strategy consists in pretraining the architecture on multiple materials and ne-tuning the weights of the network on each material separately. We observe a consistent improvement over the material over tting which supposes a regularization induced by optimizing over diversi ed examples. The choice of the dataset is critical for a good generalisation, and an alternative to a broad, all material generalisation consists in specializing networks by materials categories.

Choice of Loss

Our experiments demonstrate a better convergence using a per-pixel L1 loss as well as better appearance preservation across LoDs according to the FLIP metric [START_REF] Andersson | FLIP: A Di erence Evaluator for Alternating Images[END_REF]. We tried an optimization based directly on FLIP, but this led to poorer convergence and quality in the nal output.

Ablation While designing the cascaded architecture, we focused on the quality of the visual appearance as well as the loss value during training. Fig. IV.11 showcases the quality of mipmapped materials for a single occurrence and four occurrences of H A and H B blocks in the cascaded architecture which provides better generalisation and visual quality.

Failure cases Fig. IV.12 (right column) showcases two examples that stand out as failure cases of our approach on the GGX model, where we see clearly that the SSGT approach produces better results than ours. These results are probably due to the low number of sample counts in the learned groundtruth for such highly specular materials (roughness smaller than 1e-2 in the original maps), which creates re ies and prevents smooth gradients to be computed.

Dedicated architectures for material models Our cascaded neural model is designed to e ciently mipmap materials based on an anisotropic GGX/Beck- [START_REF] Kuznetsov | Neumip: Multi-resolution neural materials[END_REF], and AutoLoD [START_REF] Hasselgren | Appearance-Driven Automatic 3D Model Simpli cation[END_REF]. For this example, preprocess takes less than a second for our method while requiring 90 minutes for NeuMIP and 25 minutes for AutoLoD (10k iterations). mann NDF or the Ashikhmin-Shirley model. The case of sheen [START_REF] Burley | Practical physically-based shading in lm and game production[END_REF], layered [START_REF] Weier | Rendering layered materials with anisotropic interfaces[END_REF] and iridescent [START_REF] Belcour | A Practical Extension to Microfacet Theory for the Modeling of Varying Iridescence[END_REF] materials would be interesting to study.

Mipmapping the albedo and metallic channels We focused on the mipmapping of the normal and roughness values for several SVBRDF models. While it is commonly accepted in the literature to mipmap the albedo channel separatelythere is no real consensus on the case of the metallic channel -we believe that coupling the mipmapping of all channels might help mimicking more complex behaviors appearing at di erent scales (see Fig. IV.12, top left).

Single lobe vs many lobes As showcased in Fig. IV.12 (bottom left), tting a single lobe can prove insu cient to adequately represent the complex radiance distributions emerging at coarse scale (see also Fig. IV.7,examples III and IV). The use of a single anisotropic lobe is motivated by the choice of remaining compatible to existing rendering system, and by the low storage requirements resulting in high performance renderings. Still, our method is stable under the expressivity power of a single anisotropic lobe complex setup such as cross-shaped microstructures (see the speci c examples cited in the supplementary materials of the corresponding article). We plan to investigate the tting of several distributions (akin to layered materials), similar to (H. [START_REF] Tan | Real-time microstructure rendering with mip-mapped normal map samples[END_REF] who proposed to tackle Bottom left: Some very challenging materials might require several lobes to accurately depict the behavior seen from afar. Right: We observe some failure cases for GGX.

the MIPmapping of normal map-based microstructures in the context of real-time rendering. However, we anticipate that optimizing for the precise number of lobes, for a given material, in a gradient-based optimization framework as well as interpolating between the levels of such representations at runtime will prove particularly challenging.

Multiresolution expression power

We saw in our experiments that restricting the computation footprint to a standard 2 × 2 footprint (i.e., using combination of 4 texels to compute one texel in the next mipmap level) was insu cient to e ciently mipmap complex materials for their use with rich lighting. In our work, we have considered a 2-resolution computation architecture using a 4 × 4 texel footprint to compute the mipmaps after the second LoD. This choice led to e cient mipmap synthesis for large 4K SVBRDFs. It seems that increasing the texel computation footprint and the number of resolutions used does not help mipmapping SVBRDFs more precisely (using for example a 8 × 8 texel footprint, corresponding to a 3resolution computation architecture, or a 16 × 16 texel footprint corresponding to 4-level computation architecture), as shown in Sec. V.3.2. We envision that training a heavy architecture followed by simpli cation of the optimized network could allow for e cient synthesis of the mipmap levels while making use of geometric descriptors optimized at many resolutions.

Conclusion

We presented a cascaded neural model to learn downsampling kernels for computing a SVBRDF mipmap pyramid. Our method generalises over unseen materials and better preserves the appearance of the materials at multiple distances of observation, with the particular behavior of giving rise to anisotropic roughness in the mip levels, which captures salient mesostructures. Overall, our approach is a drop-in replacement for standard MIP mapping, requiring no modi cation to the host rendering engine.

V

Analysis and Applications

V.1 Introduction

In this chapter we propose an analysis of our morphing operator in the context of Deep Learning work ows, and focus on analyses and applications which are derived from the mipmapping project.

We rst propose an analysis of our morphing operator as a database ampli cation tool, by training two tasks over di erent datasets, and providing qualitative and quantitative results. Then we then detail an application of our learned downscaling kernels in the context of asset simpli cation. We also propose to explore two extensions for the architecture of the downscaling operator by implementing 3level and 4-level variants. Finally, we analyse how a bruteforce approach compares to our downscaling operator in terms of quality and computation speed.

V.2 Dataset ampli cation

V.2.1 Background

The learning technique presented in Chapter IV relies on a rich dataset of materials. The latter is composed of more than a thousand di erent materials, for training the model and testing it on unseen data. This dataset showcases a variety of e ects occuring at multiple scales, where the normal to roughness transfer preserves the appearance of the material viewed from a distance. However not all materials provide such detailed normal maps, and this number of material variations may not be su cient for data-hungry learning approaches such as generative models training, or SVBRDF estimation. The lack of data for these tasks often leads to poor generalisation capabilities. Data ampli cation techniques for SVBRDFs [START_REF] Deschaintre | Single-image SVBRDF capture with a rendering-aware deep network[END_REF] have been proposed to tackle the lack of available data. They rely on random perturbations of the important parameters exposed by artists when the procedural graph of the material is available. In addition, they use 𝛼-blending between random pairs of materials, which increases the diversity of detailed shading e ets while keeping plausible material appearance. Lastly, they use random scaling, orientation, and light conditions.

The rst limitation of this set of techniques relies on the availability of many procedural graphs. Most of the graphs designed by artists require payments to be accessed, and the license can prevent their use in learning-based approaches. Secondly, 𝛼-blending between random materials may result in a loss of details and produce invalid material appearance (see Fig.

V.1), as advocated at the end of Chapter III, especially when training generative models.

V.2.2 Data augmentation by material morphing

Our analysis of the ampli cation mechanism provided by our morphing operator relies on multiple hypotheses. First, the transport mechanism is key to preserve the visual appearance of the interpolated materials, but is also assumed, by registering contours together, to provide higher delity materials for learning applications. Second, the albedo and normal operators provide higher complexity mixing of these channels. For the albedo, the interpolation path is non-trivial compared to linear interpolation in RGB space and retains contrast in the colors of both input maps.

For the normal map mixing operator, the use of the height (or displacement) map allows to treat coarse geometry and details consistently. We propose a methodology method train set (error ×10 -3 ) test set (error ×10 to analyse the use of our morphing operator on the learning task presented in Chapter IV.

To augment a database of materials given an initial set, we rst identify materials with coarse geometric structures. We pick materials from di erent categories to account for the variability of appearance and compute transport grids between each pair (𝐴, 𝐵) of materials. Our transport grid computation is non-commutative, so we compute both grids (𝐴 → 𝐵 and 𝐵 → 𝐴) and keep the one with minimal distortion (computed using a simple per-vertex distance to the undeformed grid).

We then produce novel interpolated SVBRDFs with an interpolation parameter 𝛼 = 0.5. Hence, we obtain as much as 105 new materials from a set of 15 originals.

V.2.3 Training mipmapping kernels

We propose to focus on a single material category, to assess the quality of intraclass material morphing compared to linear 𝛼-blending. We choose an initial set of 15 materials from the Metals category, and create a halfway interpolation using both naive linear interpolation and the morphing operator of Chapter III. This brings an additional 105 new materials to use in the training of metal-specialized downsampling kernels, which adds up to 120 materials in total.

We train our downsampling kernels for 30 epochs on the two datasets and report quantitative comparisons in Table V.1, following the same testing conditions as Chapter IV. We use 2 hidden layers of size 64 for H 𝐴 and 3 hidden layers of size 128 for H 𝐵 , which limits the network capacities and prevents over tting on the relatively small dataset. The table shows that when we evaluate our trained kernels on the materials common to both training datasets (the original 15), both methods compare similarly, with a slight advantage for the linear dataset. When using the unseen data of the test set however, the dataset produced using the morphing operator consistently provides better metrics ( FLIP, L1 and MSE) than the one using the linear blending. This experiment gives an insight about the capabilities of the morphing operator when it deals with augmenting variability in material datasets, but a more exhaustive study should be performed to con rm these preliminary results. V.3 Analyses of the mipmapping operator

V.3.1 Asset simpli cation

In contexts where the size of the nal application must be limited, it is common to choose to reduce the asset size. This choice is also motivated by performance improvements, by relying on less draw calls thanks to smaller geometric assets, and fetching smaller (hence, locally coherent) textures from the GPU VRAM. A common application consists in asset simpli cation in a limited memory and performance context such as mobile phones. For SVBRDFs, a solution for obtaining lower quality versions of the maps is to run the procedural graph textures generation at lower resolution. This solution is at the cost of using less detailed pattern generators, and providing a less detailed output. As shown in Fig. V.2, the generated material at 1K resolution does not preserve the details found in both the roughness and normal maps of the same material generated at 4K resolution. Linearly averaging the maps leads to aliasing, loss of details, and also fails to preserve the appearance of the original 4K material. Our technique (MIPNet) however transfers details from the normal to the roughness map, and partially removes aliasing in the normal map by providing better ltering.

V.3.2 Architecture Extension

As explained in the conclusion of Chapter IV, we have considered a 2-resolution computation architecture using a 4 × 4 texel footprint to compute the mipmaps after the second LoD. We now propose to study whether increasing the texel computation footprint and the number of resolutions helps mipmapping SVBRDFs even more precisely. We use either a 8 × 8 texel footprint, corresponding to a 3-resolution computation architecture, or a 16 × 16 texel footprint corresponding to 4-level computation architecture.

As seen in Fig.

V.3, extending the architecture by one or two levels seems to only slighlty improve the results on the materials identi ed as failure cases of our method, and still fails to account for the highly specular shape produced by the detailed normal maps. These results suggest that the computational model has only little in uence on the quality of the mipmaps, but may be enhanced by e.g., better sampling of the groundtruth, or more varied examples in the database.

V.3.3 Bruteforce mipmapping

In this section we analyse a bruteforce approach for mipmapping using an exhaustive parameters search. With the compute capabilities of current GPUs, performing millions of operations in parallel for millions of pixels (i.e., the number of pixels in a 1K texture) seems more feasible than ever. Hence, we study a bruteforce approach consisting in discretizing the normal and roughness values of a given pixel into their respective ranges and nding the best possible t, given a set of SVBRDF maps. We look for the combination of parameters which minimizes the di erence between a rendering from these parameters and the supersampled groundtruth of the corresponding footprint using the original values. We propose a similar approach to the one described in Chapter IV, where we render a material on a plane viewed perpendicularly to ease the UV/screen mapping. The groundtruth GT is computed by averaging radiance values into one reference radiance, over multiple view and light directions. We de ne the BRDF as a function 𝑓 : 𝑛 𝑥 , 𝑛 𝑦 , 𝑟 𝑥 , 𝑟 𝑦 , 𝜃, x, 𝜔 𝑖 , 𝜔 𝑜 ∈ R 9 × Ω 2 → R 3 , where (𝑛 𝑥 , 𝑛 𝑦 ) correspond to the normal coe cients projected onto the plane, (𝑟 𝑥 , 𝑟 𝑦 , 𝜃 ) are the anisotropic roughness coe cients, x is a vector of 4 material parameters (3 for di use and 1 for metallic).

We also de ne the high-resolution SVBRDF as LoD 0 : 𝑝 ∈ R 2 → LoD 0 (𝑝) ∈ R 9 , which provides the material parameters de ned previously. Formally, for a position 𝑝, we aim at nding a tuple (𝑛 𝑥 , 𝑛 𝑦 , 𝑟 𝑥 , 𝑟 𝑦 , 𝜃 ), the normal and anisotropic matrix coe cients, s. where P 0 (𝑝) ⊂ R 2 covers the footprint of texel 𝑝 in LoD 0 , and 𝑓 is a given BRDF model.

We exhaustively search for the set of ve parameters (normal and roughness) which minimize 𝜎 (𝑝) for each position 𝑝. Unfortunately, sampling the ve dimensional domain uniformly quickly becomes unpratical for ne discretizations. In practive, we discretize the roughness parameters (𝑟 𝑥 , 𝑟 𝑦 , 𝜃 ) over their own ranges:

𝜖 𝛼 ≤ 𝑟 2 𝑥 ≤ 1 (V.3) 𝜖 𝛼 ≤ 𝑟 2 𝑦 ≤ 1 (V.

VI Conclusion

We opened this thesis by presenting the challenges linked with the manipulation of a commonly used material model, and the work related to the manipulation of similar types of data. We proposed a novel morphing process, which, using simple yet e ective operators, allows for the preservation of interpolated materials appearance and hence the discovery of the underlying appearance manifold. We then proposed a compact non-linear downsampling kernel for tackling level-ofdetail ltering, to further preserve appearance across mipmaps.

VI.1 Contributions

We proposed an e cient operator to explore and design-by-interpolation novel SVBRDFs based on an existing collection of materials. To that end, a transport grid model is used to guide the morphing process from the dominant mesostructures of the materials. This allows for the creation of novel real-time detail-preserving mechanisms to blend albedo and normal vectors consistently in this context.

Secondly, our contributions relate to an appearance driven optimization framework to produce a mipmap pyramid by learning a downsampling operator. We proposed an e cient pipeline for learning mipmapping lters requiring no data preparation for training. Our neural architecture encodes anisotropic appearance and generalises on unseen materials. Our tensor-based formulation for anisotropic BRDF distributions is well-suited for di erentiable pipelines and trilinear interpolation.

VI.2 Future prospects

Extending manifold navigation Navigating the appearance manifold has been studied through the morphing operator which interpolates between existing data

Level of Detail

Figure VI.1: Future directions include material super-resolution (orange arrows), and local appearance exploration (green arrows), using e.g. semantic guiding, latent space disentanglement or multi-material extrapolation. points and by using a downsampling kernel to compute appearance preserving mipmaps. Some attempts at learning the appearance manifold of materials have been made but are limited to interpolations of feature vectors in the latent space. Artists often need interpretable quantities to deal with when designing materials. Using advances in latent space disentanglement for discovering meaningful directions of variability would prove useful for this task. Also, the rise of text-to-image generative models paves the way for semantic guiding of appearance exploration in the manifold. The morphing operator described in Chapter III would provide a dataset for learning generative models, such as Generative Adversarial Networks or Di usion models, which require many, varied examples.

Material super-resolution One logical follow-up work to carry out would be the exploration of material super-resolution, as a way of navigating through level-of-details from bottom to top levels of the appearance space (see Fig. Extensive work has been conducted on RGB image super-resolution, but as argued previously in this thesis, naively applying them to the stack SVBRDF would fail to produce valid materials. Super-resolution of radiance images however could be studied to leverage o -the-shelf upscaling architectures and use them to optimize SVBRDF maps through a di erentiable rendering pipeline. On the other hand, if super-resolution is applied directly on the maps, geometric information should ow from the roughness to the normal map details and then to displacement, given that they become large enough to a ect the model's silhouettes. The analysis of the geometric level-of-detail encoding conducted in previous chapters would prove bene cial when studying how coarse structures emerge from the height map, and are preserved when the SVBRDF is downsampled. In addition, the concurrent upscaling of di use and specular albedo requires an analysis of the correlations of such channels (along with the geometric ones) to produce visually plausible results. Since the material's details depend on its nature (e.g., metal, grounds, fabrics), by-example or semantic guiding should be considered for the upscaling. The former can be achieved using specialized per-category generative models, while the latter could require approaches like recent text-to-images generative models, making use of a pretrained language encoding model to drive the detail creation. Enforcing the upscaled SVBRDF to match the original input when going through mipmapping would prove useful to constrain the created SVBRDF to retain a valid appearance.

Multi-material interpolation and extrapolation Our morphing operator acts on a pair of materials, but navigating the convex hull formed by the morphing between multiple materials would be bene cial for explorating larger spaces of appearance. Extrapolation has not been studied, but would also help for this exploration, as a way of further extending the appearance space navigation.

Complex material models In this thesis we restricted ourselves to the most common SVBRDF maps used in practice. However there exists a wide variety of maps of di erent natures, e.g. clearcoat, sheen, translucency, scattering and iridescence. They enhance the visual aspect of materials and complexify the appearance. The choice of a single anisotropic lobe for appearance modeling is a requirement for highly e cient renderings. However extending the model to a selective number of multiple lobes, or studying the e ect of multilayered materials with mipmapping remains unsolved.

Neural representations Our material model remains grounded to the SVBRDF representation, but recent works focus more and more on neural representations, such as Radiance Fields, or Neural (SV)BSDF encoding. These approaches allow for relatively compact representations and o er appearance preservation at multiple scales. However it is not clear how deal with highly specular materials, nor if the representation generalises well on unseen appearance.

A.2 Anisotropic Beckmann distributions

We use the following common approximation [START_REF] Walter | Microfacet models for refraction through rough surfaces[END_REF][START_REF] Heitz | Understanding the masking-shadowing function in microfacet-based brdfs[END_REF] (where 𝑥 = 𝑖 or 𝑜 in the following):

Λ 𝐵 𝜔 𝑥 1 -𝐺 1 (𝜔 𝑥 )
𝐺 1 (𝜔 𝑥 )

𝑏

. This equivalence is motivated in [START_REF] Olano | Lean mapping[END_REF] to best match the anisotropy pro les of Phong and Beckmann distributions. The specular term of [START_REF] Ashikhmin | An anisotropic phong brdf model[END_REF] is then given by: Keywords: computer graphics, material appearance modeling, realtime rendering Abstract: The Physically Based Rendering workflow has become a standard for rendering digital materials for the creative industries, such as video games, visual special effects, product design and architecture. It enables developers and artists to create and share readyto-use photorealistic materials among a wide variety of applications. Unfortunately, PBR material authoring require advanced skills and a significant amount of time to model convincing materials to be used by photorealistic renderers. In this thesis, we explore efficient morphing and level-of-detail operators in the context of digital materials. Our proposed methods allow for interpolating materials in the canonical space of textures as well as along the downscaling pyramid for preserving and exploring appearance of these materials.

  Figure I.1: Left: A render of a 3D scene composed of multiple objects. Right: the materials used to model the appearance of these objects. Credits: APARTMENT G1069 by Gokulnath Vasudevan, Behance.

Figure I. 2 :

 2 Figure I.2: Rendering using triangle meshes. Rendering a scene often requires geometry encoded as a triangle mesh. In a typical real-time scenario, triangles are projected on the virtual camera plane and shaded according to the light sources.

Figure I. 3 :

 3 Figure I.3: Normal Mapping.On the left, the original mesh is shown, containing as much as 4 million vertices. In the middle, the mesh is simpli ed to 500 triangles which encode the coarse shape. On the right, the normal map is added onto the low resolution mesh to retrieve the details lost during simpli cation. Credits: Wikipedia.

Figure I. 4 :

 4 Figure I.4: The NVIDIA Omniverse™ platform, showcasing a real-time collaborative work ow. Artists work on the same scene's geometry and rigging (top left), layout (middle left) and texturing (bottom left), while the camera is placed to render the nal shot (right). All softwares must share the same material standard to consistently render material appearance in every viewport. Credits: Nvidia.

  Figure I.5: Multiple means for material design and aquisition.

Figure I. 6 :

 6 Figure I.6: Radiometric and geometric quantities used for the BRDF de nition.

Fig

  Fig. I.7 presents a simple taxonomy of re ectance functions, starting with the Bidirectional Surface Scattering Re ectance Distribution Function (BSSRDF). This

Figure I. 7 :

 7 Figure I.7: Simple taxonomy of re ectance functions. The arrow in the right represents the number of dimensions required to encode the function. From[START_REF] Guarnera | Virtual material acquisition and representation for computer graphics[END_REF]

Figure I. 8 :

 8 Figure I.8: Di use 𝑓 𝑑 and specular 𝑓 𝑠 components of the Cook-Torrance BRDF.

Figure I. 9 :

 9 Figure I.9: Comparison between the Beckmann NDF (left) and the Trowbridge-Reitz, or GGX, NDF (right) using the same low roughness value (𝛼 = 0.1).

Figure I. 10 :

 10 Figure I.10: Perfect re ection at Sand Harbor State Park, Lake Tahoe, by Donny Barnec. This photograph illustrates the Fresnel e ect on a large body of water.

  Figure I.11: Example of a PBR material mapped to various meshes. On top are shown the most common maps used for rendering the meshes in the bottom. Note that the basecolor map encodes both di use albedo and Fresnel re ectance at normal incidence and is modulated by the metallic map.

Figure I. 13 :

 13 Figure I.13: Computing Global Illumination in a scene requires solving recursively the rendering equation by adding up incoming light contributions 𝐿 𝑖 at each light bounce (de ned by a local normal 𝑛 in the hemispheres Ω). The latter are either emitted by primary light sources 𝐿 𝑒 or by secondary light sources such as lit surfaces 𝐿 𝑜 .

Figure I. 14 :

 14 Figure I.14: Rasterization projects triangles onto the image plane (left) while ray tracing techniques casts rays through the image plane towards the scene (right). Image courtesy of Wikipedia and OpenGL.org.

Figure I. 15 :

 15 Figure I.15: Architecture of a typical convolutionnal neural network (CNN) used for image classi cation. An image is given as input (left) and the network outputs a classi cation vector with a probability associated to each class (right). Credits: Wikipedia.

Figure I. 16 :

 16 Figure I.16: MIP mapping aims at pre ltering a high-resolution texture (top left, base texture) into downsampled versions (top right, mipmaps) which are used at render time to prevent aliasing e ects (bottom renderings).

I. 5

 5 .1 Peer-reviewed articles Most of the contributions discussed in this thesis have been presented in international scienti c conferences: • Alban G , Jean-Marc T and Tamy B . 2021. MatMorpher: A Morphing Operator for SVBRDFs. In Eurographics Symposium on Rendering 2021 (Symposium Track) (presented at the EGSR 2021 conference). MIPNet: Neural Normal-to-Anisotropic-Roughness MIP mapping. In ACM Transactions on Graphics, Volume 41, Issue 6, Article No.: 246 (December 2022) (presented at SIGGRAPH Asia 2022).[START_REF] Gauthier | Mipnet: Neural normal-to-anisotropic-roughness mip mapping[END_REF]

Figure II. 1 :

 1 Figure II.1: Semi-procedural texture synthesis with varying PPTBF parameters from (Guehl et al., 2020). For each example, the input is shown on the left, and on the right, a spatially varying result showing the computed binary mask and material output.

Figure II. 2 :

 2 Figure II.2: Examples of spatially varying texture interpolation using manually painted weight maps from[START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF].

Figure II. 3 :

 3 Figure II.3: Texture interpolation examples from (Ruiters et al., 2010). Input textures and feature maps are shown below each interpolation. Each interpolation requires the interpolation mask prior the optimization.

  Figure II.4: Image downscaling examples from[START_REF] Kopf | Content-adaptive image downscaling[END_REF]. Content-adaptive image downscaling preserves sharpness while mitigating sampling artifacts caused by naïve subsampling.

  Figure II.5: Latent material interpolation from(Guo, Smith, Hašan, et al., 2020).

  Figure II.6: Appearance-driven di erentiable pipeline. Modi ed from (Hasselgren et al., 2021).

Figure III. 1 :

 1 Figure III.1: Design by example. An existing set of materials (circled in black) allows for the creation of novel appearances by combining the inputs (shown along the lines).

  Figure III.2: Overview of our method for material morphing.

Figure III. 3 :

 3 Figure III.3: Boundary conditions analysis.In the more general case, when dealing with dense structures to match (top), the boundary conditions impact the solution only locally. When dealing with structures to match that are very sparsely distributed (below), the boundary conditions strongly impact the solution globally, as most conditions are too loose to enforce strong regularization in these cases. Whether the inputs are tileable or not, we nd that periodic harmonicity provides the best results in all cases.

Figure III. 4 :

 4 Figure III.4: Starting with albedos 𝐼 1 and 𝐼 2 , we compute the gaussianized albedos 𝐼 𝐺 1 and 𝐼 𝐺 2 , interpolate them to get 𝐼 𝐺 𝛼 and apply the inverse histogram composition which results in 𝐼 𝛼 . CDF H , resp. CDF G , refers to the cumulative distribution function of image with histogram H , resp. of the gaussian distribution. CDF -1𝛼 is the interpolation of the inverse CDF of images I 1 and I 2 . Our technique better preserves color sharpness compared to a linear interpolation.

Figure III. 5 :

 5 Figure III.5: Normal map and geometric normal mismatch. On the rst row, the normals (right hemisphere) are computed from the displaced surface. On the second row, a simple texel fetch in the normal map followed by a change of frame using the TBN matrix, overlayed on the displaced geometry. Displacing the geometry with a varying height factor (each column) creates a "painting e ect" (left hemisphere), where the normal re ections seem to be painted on the surface, no matter the orientation of the tiles.

Figure III. 6 :

 6 Figure III.6: The linear interpolation of the height map is inconsistent with the spherical linear interpolation (SLERP) of the normal map. The plots were computed for ℎ/𝑑 = 10.

Figure III. 7 :

 7 Figure III.7: Normal reorientation pipeline. To mix geometric parameters consistenly, we propose to extract the details of the two normal maps n 1 , n 2 by computing normals n ℎ 1 , n ℎ 2 derived from the coarse height maps 𝐻 1 , 𝐻 2 using a gradient operator. By reorienting the original normals into the coarse one, we extract detail maps d 1 , d 2 which are mixed using Spherical Linear Interpolation (SLERP) into a resulting mixed detail map d 𝛼 . The height maps are blended linearly to produce 𝐻 𝛼 , which provides a coarse normal map n ℎ 𝛼 , to which the details from d 𝛼 are added back to obtain the nal normal n 𝛼 . This pipeline is achieved after computing the 1-to-1 texel correspondences with the transport grid.

  III.8). The addDetails function in Alg. 2 corresponds to the inverse operation. After extracting the details, we combine them by using a spherical linear interpolation, and we add them to the interpolated coarse normal similarly to[START_REF] Barré-Brisebois | Blending in detail[END_REF].

Figure III. 8 :

 8 Figure III.8: Normal reorientation. We compute the rotation 𝑅 from the coarse geometry n ℎ to the detailed one n and apply it to a unit vector perpendicular to the plane z to obtain the detail vector d.

  Figure III.9: Transport grid ablation. Without our transport grid, i.e. using linear blending (left), undesired overlapping of structures along the edges tend to appear.[START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF] technique (middle) fails to correctly register contours, due to imprecise feature maps. The resulting warpgrid exhibits discontinuities along borders due to the independent per-vertex optimization. Using our transport grid (right) allows for contour registration (circle) and creates a natural morphing even if a contour does not nd a correspondent in the other material (rectangles).

  Figure III.10: Normal reorientation ablation. We show a halfway interpolation between two materials. Our reorientation component better conveys the shape features all along the interpolation by computing a geometric normal with added details. No transport map was used for this example.

Figure III. 11 :

 11 Figure III.11: RGB texture interpolation comparison between the method of (Matusik et al., 2005) (top) and our method (bottom). For each example, we show the features used to compute the transport grid (visualized in red) and ve interpolation steps from 0.0 to 1.0.

Figure III. 12 :

 12 Figure III.12: Comparison of linearly interpolated RGB texture enhancements. Halfway per-pixel linear interpolations between texture A (leftmost column) and B (rightmost column) are shown.[START_REF] Matusik | Texture design using a simplicial complex of morphable textures[END_REF] sharpness preservation fails to enhance details on 1K pairs of images and helps to sharpen the result at the pixel scale (require zooming-in). Simpler histogram matching of interpolated textures produces sharp results but has a higher runtime complexity than texture gaussianization, which helps to preserve details after linear blending. Note that no warping is done for this comparison.

Figure III. 13 :

 13 Figure III.13: Comparison between MaterialGAN and naive linear interpolation. Each set of results illustrate the interpolation using MaterialGAN (top) compared with linear interpolation (bottom).

  Figure III.14: Interpolated materials 𝑀 𝛼 with a linear blending (top row) compared to our technique (bottom row), using a uniform value for 𝛼 ∈ {0, 0.2, 0.4, 0.6, 0.8, 1}.We stress that temporal interpolations are better viewed in videos (provided as supplemental materials).

Figure

  Figure III.15: Left: linear interpolation in euclidian space R ℎ•𝑤•𝑐 de ned by the texel coordinates (ℎ • 𝑤) and the channels 𝑐 produces unplausible materials, falling outside of the space of valid materials appearance. Right: our morphing operator allows the exploring of the appearance manifold de ned by each original (colored crosses) and novel material's positions.

Figure IV. 2 :

 2 Figure IV.2: Overview of our training process. Mipmap levels LoD 𝑖 are generated using a cascaded architecture of multilayer perceptrons. The cascade of networks takes as input a height map ℎ, a normal map (through the 𝑛 𝑥 , 𝑛 𝑦 components), and the roughness map which provides the initial tensor coe cients (𝑎, 𝑏, 𝑐). The height, metallic and baseColor maps are linearly downsampled and used along with the outputs of the cascade to be rendered (using random light -𝜔 𝑖 -and view -𝜔 𝑜directions) and used in the nal loss term. This cascade is composed of successive blocks of H 𝐴 and H 𝐵 (circled in dotted orange) which architectures are detailed in the top right corner. H 𝐵 requires a larger capacity than H 𝐴 because of its higher dimensional input. The groundtruth is computed using the full resolution input SVBRDF.

Figure IV. 3 :

 3 Figure IV.3: Simpli ed, single resolution training pipeline.

Figure IV. 4 :

 4 Figure IV.4: Single resolution inference pipeline. Each level LoD 𝑖 is fed to H 𝐴 and provides the next level-of-detail LoD 𝑖+1

  IV

  .

  2) that learns to best account for two-scale descriptors and o ers a robust cascaded inference mechanism (see Fig. IV.5).

Figure IV. 5 :

 5 Figure IV.5: Multi resolution inference pipeline.

  FigureIV.6: Transfer between maps: From isotropic SVBRDF parameters (left, normal map and roughness from base LoD), our method is able to generate mip levels which reveal information transfer from normal to roughness and anisotropy creation (middle: generated maps from LoD 2 ), responsible for the elongated highlight in the rendering (right, top linear mipmapping, bottom ours) in accordance to the groundtruth (not shown).

Fig

  Fig. IV.6 showcases an example of normal-to-roughness transfer between levelof-details. The base levels (LoD 0 ) are shown on the left, in the middle the tensor coe cients which show anisotropy (a is stronger than b which denotes stronger

Figure IV. 7 :

 7 Figure IV.7: Renderings (GGX) of various materials. Every second row shows the FLIP error compared to the groundtruth and the mipmap LoD used for rendering (with trilinear ltering).

Figure IV. 8 :

 8 Figure IV.8: Renderings (Beckmann) of materials. Every second row shows the FLIP error compared to the groundtruth and the mipmap LoD used for rendering (with trilinear ltering).

Figure IV. 9 :

 9 Figure IV.9: Renderings (Ashikhmin-Shirley) of materials under varying view conditions. Every second row shows the FLIP error compared to the groundtruth, and color image for the mipmap LoD used for rendering (with trilinear ltering).

Figure IV. 10 :

 10 Figure IV.10: Qualitative comparison with o ine material preprocessing methods NeuMIP[START_REF] Kuznetsov | Neumip: Multi-resolution neural materials[END_REF], and AutoLoD[START_REF] Hasselgren | Appearance-Driven Automatic 3D Model Simpli cation[END_REF]. For this example, preprocess takes less than a second for our method while requiring 90 minutes for NeuMIP and 25 minutes for AutoLoD (10k iterations).

Figure

  Figure IV.11: Qualitative comparison of the number of occurrences of H A and H B blocks used during training using the same batch size for the two experiments. With only a single occurrence (i.e., no repetition of the orange block in Fig. IV.2), the network fails to learn anisotropy which appears at various level-of-details (see groundtruth), often after the rst level. With four occurrences (four repetitions of the orange block in Fig. IV.2), anisotropy is better learned, by providing gradients of losses computed with further level-of-details.

Figure IV. 12 :

 12 Figure IV.12: Limitations of our approach. Top left: While we couple the mipmapping of roughness and normals in our method, mipmapping the albedo channel separately does not allow reproducing the base color shift observed in the groundtruth.

Figure V. 1 :

 1 Figure V.1: Three examples from the dataset of (Deschaintre et al., 2018), displaying from left to right, a rendering, and the normal, di use, roughness and specular maps. These examples result from the 𝛼-blending of two existing materials.

Figure V. 2 :

 2 Figure V.2:Comparison between SVBRDF compression techniques. This gure illustrates the use of MIPNet for a simple SVBRDF "compression". Generating the SVBRDF from the procedural graph at lower resolution (1K, middle, left) produces a valid material, which however di ers from the 4K generation (middle, right). Linear average (far left) produces aliasing in the normal map and di ers in appearance when rendered compared to the original 4K material. Our solution (far right) adds extra anisotropic roughness (coe cients 𝑎 and 𝑏) to account for the ne details in the normal map, which are lost due to the lower resolution encoding.

  Figure V.3: Comparison between the original 2-level and the extended (3 and 4 levels) architectures. The mean FLIP error is reported on the top right of the error maps.

  t.: argmin (𝑛 𝑥 ,𝑛 𝑦 ,𝑟 𝑥 ,𝑟 𝑦 ,𝜃 ) 𝜎 (𝑝) = ∑ 𝜔 𝑖 ,𝜔 𝑜 ∈Ω 𝑓 𝑛 𝑥 , 𝑛 𝑦 , 𝑟 𝑥 , 𝑟 𝑦 , 𝜃, 𝜔 𝑖 , 𝜔 𝑜 -GT (𝑝, 𝜔 𝑖 , 𝜔 𝑜 ) 1 , (V.1) GT (𝑝, 𝜔 𝑖 , 𝜔 𝑜 ) = 1 #{P 0 (𝑝)} ∑ 𝑥 ∈ P 0 (𝑝 ) 𝑓 (LoD 0 (𝑥) , 𝜔 𝑖 , 𝜔 𝑜 ) (V.2)

  Figure V.4: Normal and Anisotropic roughness maps resulting from the exhaustive search. The roughness pyarmid encodes the (𝑟𝑥, 𝑟𝑦, 𝜃 ) coe cients. As expected, the green color appearing at higher LoDs comes from the anisotropic e ect produced by geometry inside the normal map. baseline exhaustive search groundtruth LoD

  VI.1).

𝜌 2 (

 2 𝑠 (𝜔 𝑖 , 𝜔 𝑜 ) := √ (𝑠 𝑡 + 1) (𝑠 𝑏 + 1) 8𝜋 (𝑛 • 𝜔 ℎ ) 𝑠 𝑡 (𝑡𝛾 •𝜔 ℎ ) 2 +𝑠 𝑏 (𝑏𝛾 •𝜔 ℎ ) 2 1-(𝑛•𝜔 ℎ ) 2(𝜔 ℎ • 𝜔 𝑖 ) max ((𝑛 • 𝜔 𝑖 ) , (𝑛 • 𝜔 𝑜 )) = √ det(𝐴 2 ) + tr(𝐴 2 ) + 1 8𝜋 det(𝐴) (𝑛 • 𝜔 ℎ ) 𝜔 𝑇 ℎ •𝐵 𝑇 •𝐴 2 •𝐵•𝜔 ℎ (1-cos 2 (𝜃 ℎ ) ) det(𝐴 2 ) (𝜔 ℎ • 𝜔 𝑖 ) max ((𝑛 • 𝜔 𝑖 ) , (𝑛 • 𝜔 𝑜 )) = √ det(𝐴 2 ) + tr(𝐴 2 ) + 1 8𝜋 det(𝐴) (𝑛 • 𝜔 ℎ ) 𝐴•𝐵•𝜔 ℎ sin(𝜃 ℎ ) det(𝐴) 2 𝜔 ℎ • 𝜔 𝑖 ) max ((𝑛 • 𝜔 𝑖 ) , (𝑛 • 𝜔 𝑜 ))Titre: Des opérateurs de morphing et de niveau de détail pour le design et le rendu interactif de matériaux Mots clés: modélisation de matériaux, algorithme temps réel, rendu 3D Résumé: Le rendu inspiré par la physique est devenu une norme pour le rendu de matériaux numériques dans les industries créatives, telles que les jeux vidéo, les effets spéciaux, la conception de produits et l'architecture. Il permet aux développeurs et aux artistes de créer et de partager des matériaux photoréalistes prêts à l'emploi entre une grande variété d'applications. Malheureusement, la création de matériaux PBR nécessite des compétences avancées et un temps considérable pour modéliser des matériaux convaincants destinés à être utilisés par des moteurs de synthèse d'images réalistes. Dans cette thèse, nous explorons la création d'opérateurs efficaces de mélange et de calcul de niveau de détail dans le contexte des matériaux numériques. Les méthodes que nous proposons permettent d'interpoler les matériaux dans l'espace canonique des textures ainsi que le long de la pyramide de niveau de détails pour préserver et explorer l'apparence de ces matériaux.Title: Morphing and Level-of-Detail Operators for Interactive Digital Material Design and Rendering

  details from normalsd 2 ← getDetails(n ℎ 2 , n 2 ) d 𝛼 ← slerp(d 1 , d 2 , 𝛼) ;

	// interpolate details
	n ℎ 𝛼 ← normalize(cross(dFdx(position), dFdy(position)))

n 𝛼 ← addDetails(n ℎ 𝛼 , d 𝛼 ) ; // add details in the coarse normal

  TableIII.1: Timings. We render an interpolation between two materials for a total of about 50 million triangles under a single point light, and under three point lights and Image Based Lighting for the experiment of the last two lines of this table.

		Timings
		Contour Detection	Transport Grid
	4K downsampled to 1K	3.1 ± 2.1 s	18.6 ± 6.7 s
		Gaussianization	Normal Compute
	Ours 4K	350 ms	400 ms
		Single light	3 lights + IBL
	Linear 4K	7.87 ± 0.07 ms	9.89 ± 0.09 ms
	Ours 4K	15.10 ± 0.06 ms	19.41 ± 0.13 ms

  Quantitative comparisons between the trainings on the linear average dataset and the dataset computed using our morphing operator. The train set is composed of the 15 original materials (which both datasets have in common), when the test set contains 100 materials.

							-3 )
		FLIP	L1	MSE	FLIP	L1	MSE
	linear	14.49 3.71	0.33	121.07 30.39	3.5
	morphing 14.54 3.72	0.34	118.91 29.51 3.12
	Table V.1:

  √cos(𝜙 𝑥 ) 2 𝛼 2 𝑡 + sin(𝜙 𝑥 ) 2 𝛼 2 𝐴 • 𝐽 • 𝐵 • 𝜔 𝑥 2 𝜉 𝑥 := 1 𝛼 𝑥 tan(𝜃 𝑥 ) = cos(𝜃 𝑥 ) 𝐴 • 𝐽 • 𝐵 • 𝜔 𝑥 2 𝐷 𝐵 := 𝜒 + (𝜔 ℎ • 𝑛) 𝜋𝛼 𝑡 𝛼 𝑏 cos 4 (𝜃 ℎ ) exptan 2 (𝜃 ℎ ) cos 2 (𝜙 ℎ )𝛼 2 𝑏 + sin 2 (𝜙 ℎ )𝛼 2 𝜒 + (𝜔 ℎ • 𝑛) 𝜋 det(𝐴) cos 4 (𝜃 ℎ ) exp -𝐴 • 𝐵 • 𝜔 ℎ cos(𝜃 ℎ ) det(𝐴)

		1-1.259𝜉 𝑥 +0.396𝜉 𝑥 3.535𝜉 𝑥 +2.181𝜉 𝑥 2	2	if 𝜉 𝑥 < 1.6
		0		otherwise
	𝛼 𝑥 :=	𝑏
	=	1 sin(𝜃 𝑥 )
				𝑡
				𝛼 2 𝑏 𝛼 2 𝑡
	=

Remerciements

IV

Neural Normal-to-Anisotropic-Roughness MIP mapping

Level of Detail

Figure IV.1: Exploring the appearance across level-of-details spaces. Left: exploration among (black arrows) and between (blue and red arrows) level-of-detail appearance manifolds. Right: Traditionnal linear MIPmapping can be considered as a naive linear interpolation between groups of 4 texels average in euclidian space (purple arrows).

Beckmann: comparison with LEADR We present our results on the Beckmann distribution in Fig. IV.8, in which we compare our results to standard linear mipmapping (the baseline) as well as to the LEADR technique [START_REF] Dupuy | Linear e cient antialiased displacement and re ectance mapping[END_REF], which is the current state-of-the-art in Beckmann distributions mipmapping. This method uses two maps encoding the statistics of the normal (or displacement) map. Note that this model is not well-suited for the use of both normal and displacement maps (which often encode complementary signals): the ltering is performed on either one of them, but not both. As can be seen, the baseline features biased anisotropic e ects on most examples (most noticeable on materials II, IV, V), similarly to the LEADR technique (most noticeable on materials I, II, III, VI). As expected, the examples on which LEADR fails to capture the anisotropy (and performs worse than the baseline, which is a simple linear averaging) are the ones where the normal map features geometric statistics that deviate strongly from Gaussian distributions, which is the main hypothesis made by LEADR. More results can be found in the supplementary material of the corresponding article.

Ashikhmin-Shirley

We present our results on the Ashikhmin-Shirley distribution in Fig. IV.9, in which we compare our results to the baseline, which is once again simple linear averaging (glGenerateMipmap). To the best of our knowledge, only the baseline has been used so far with this model. We can observe that both roughness spread (case I, II and III) and anisotropy (case IV) emerge from our MIP pyramid while being not captured by the baseline.

IV.4.2 Comparison to per-material optimization methods

We also compare our method with techniques that compute materials level of details using per-material optimizations. We report in Figure IV.10 qualitative comparisons and timings for processing a single material. With respect to those two methods, we achieve reasonable material appearance preservation while being 3 to 4 orders of magnitude faster for the mipmap generation. More visual comparisons are available in the supplemental material of the corresponding article.

NeuMIP We compare renderings of [START_REF] Kuznetsov | Neumip: Multi-resolution neural materials[END_REF] using the trained weights and rendering code kindly provided by the authors, with the same materials they trained their model on. Note that a quantitative comparison is impractical since their MBTF has been learned on path-traced rendering, and using an unspeci ed shading model. NeuMIP must train a di erent model for each material, taking approximately 45 minutes per material with neural textures of resolution 512 × 512. NeuMIP also requires a neural module at material evaluation time, making its integration in a path-tracing engine not straightforward.

AutoLoD [START_REF] Hasselgren | Appearance-Driven Automatic 3D Model Simpli cation[END_REF] propose a di erentiable framework to modify the geometry and shading models by optimizing a rendering loss for a given

Appendices

A Tensor-based normal distribution functions

In this section, we note (𝑡, 𝑏) the axis-aligned tangent and bitangent vectors (whereas 𝑡 𝛾 , 𝑏 𝛾 denote the rotated tangent and bitangent vectors), 𝑛 the normal, 𝜔 𝑖 the unit vector pointing to the light, 𝜔 𝑜 the unit vector pointing to the camera, 𝜔 ℎ the half vector (𝜔 ℎ = (𝜔 𝑖 + 𝜔 𝑜 )/ 𝜔 𝑖 + 𝜔 𝑜 ), and (𝜃, 𝜙) the spherical coordinates of unit vectors 𝜔 in the frame (𝑛, 𝑡 𝛾 , 𝑏 𝛾 ) (see inset), as well as 𝜒 + (𝑥) = 1 for 𝑥 ≥ 0 and 0 otherwise the Heavyside function.

The height-correlated masking and shadowing function 𝐺 for the GGX and Beckmann distributions is given by

A.1 Anisotropic GGX distributions

The normal distribution 𝐷 𝐺𝐺𝑋 is given by:

where we use det(𝐴) = 𝛼 𝑏 𝛼 𝑡 , and 𝐵 𝑇 = (𝑡, 𝑏) ∈ R 3×2 .

We rewrite the masking-shadowing for the GGX distribution: where 𝑥 = 𝑖 or 𝑜 in the previous equation, and 𝐽 := (0, 1; 1, 0) ∈ R 2×2 swaps both lines of the matrix at its right side.