Quentin Le Lidec

Ivan Laptev

Cordelia Schmid

Justin Carpentier

Differentiable Rendering with Perturbed Optimizers

come   L'archive ouverte pluridisciplinaire

Introduction

Many common tasks in computer vision such as 3D shape modelling [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF][START_REF] Hoiem | Recovering surface layout from an image[END_REF][START_REF] Saxena | Learning 3-d scene structure from a single still image[END_REF][START_REF] Wang | Pixel2mesh: Generating 3d mesh models from single rgb images[END_REF][START_REF] Yan | Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision[END_REF] or 6D pose estimation [START_REF] Labbe | Cosypose: Consistent multi-view multi-object 6d pose estimation[END_REF][START_REF] Li | Deepim: Deep iterative matching for 6d pose estimation[END_REF][START_REF] Lowe | Three-dimensional object recognition from single two-dimensional images[END_REF][START_REF] Rad | Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth[END_REF][START_REF] Roberts | Machine perception of three-dimensional solids[END_REF] aim at inferring 3D information directly from 2D images. Most of the recent approaches rely on (deep) neural networks and thus require large training datasets with 3D shapes along with well-chosen priors on these shapes. Render & compare methods [START_REF] Labbe | Cosypose: Consistent multi-view multi-object 6d pose estimation[END_REF][START_REF] Li | Deepim: Deep iterative matching for 6d pose estimation[END_REF] circumvent the non-differentiability of the rendering process by learning gradients steps from large datasets. Using a more structured strategy would allow to alleviate the need for such a strong supervision. In this respect, differentiable rendering intends to model the effective image generation process to compute a gradient related to the task to solve. This approach has the benefit of containing the prior knowledge of the rendering process while being interpretable. This makes it possible to provide a supervision for neural networks in a weakly-supervised manner [START_REF] Deng | Self-supervised 6d object pose estimation for robot manipulation[END_REF][START_REF] Kato | Neural 3d mesh renderer[END_REF][START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF]. The main challenge of differentiable rendering lies in the non-smoothness of the classical rendering process. In a renderer, both the rasterization steps, which consist in evaluating the pixel values by discretizing the 2D projected color-maps, and the aggregation steps, which merge the color-maps of several objects along the depth dimension by using a Z-buffering operation, are non-differentiable operations (Fig. 1). Intuitively, these steps imply discontinuities in the final rendered image with respect to the 3D positions of the scene objects. For example, if an object moves on a plane parallel to the Figure 1: Top: Overview of the rendering process: both rasterization and aggregation steps induce non-smoothness in the computational flow. Bottom: Illustration of the differentiable perturbed aggregation process. The rasterization step is made differentiable in a similar way. camera, some pixels will immediately change color at the moment the object enters the camera view or becomes unocluded by another object.

In this paper, we propose to exploit randomized smoothing techniques within the context of differentiable rendering to automatically soften the non-smooth rendering operations, making them naturally differentiable. The generality of our approach offers a theoretical understanding of some of the existing differentiable renderers while its flexibility leads to competitive or even state-of-the-art results in practice. We make the following contributions:

→ We formulate the non-smooth operations occurring in the rendering process as solutions of optimization problems and, based on recent work [START_REF] Berthet | Learning with differentiable perturbed optimizers[END_REF], we propose a natural way of smoothing them using random perturbations. We highlight the versatility of this smoothing formulation and show how it offers a theoretical understanding of several existing differentiable renderers. → We propose a general way to use control variate methods to reduce variance when estimating the gradients of the perturbed optimizers, which allows for sharp gradients even in the case of weaker perturbations. → We introduce an adaptive scheme to automatically adjust the smoothing parameters by relying on sensitivity analysis of differentiable perturbed renderers, leading to a robust and adaptive behavior during the optimization process. → We demonstrate through experiments on pose optimization and 3D mesh reconstruction that the resulting gradients combined to the adaptive smoothing provide a strong signal, leading to more accurate solutions when compared to state-of-the-art alternatives based on smooth gradient approximations [START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF].

Background

In this section, we review the fundamental aspects behind image rendering and recall the notion of perturbed optimizers which are at the core of the proposed approach.

Rasterizer and "aggregater" as optimizers. We consider the rendering process of an RGB image of height h I and width w I , from a scene composed with meshes represented by a set of m triangles. We assume that every triangle has already been projected onto the camera plane and their associated color maps

C j ∈ R h I ×w I ×3 , j ∈ [1 .
. m], have been computed using a chosen illumination model (Phong [29], Gouraud [START_REF] Gouraud | Computer Display of Curved Surfaces[END_REF] etc.) and interpolating local properties of the mesh. We denote by I the occupancy map such that I i j is equal to 1 if the center of the i th pixel is inside the 2D projection of the j th triangle on the camera plane and 0 otherwise. By denoting d(i, j) the Euclidean distance from the center of the i th pixel to the projection of the j th triangle, the rasterization step (which actually consists in computing the occupancy map I) can be written as:

I i j = H(d(i, j)), (1)
where H corresponds to the Heaviside function defined by:

H(x) = 0 if x ≤ 0 1 otherwise = argmax 0≤y≤1 y x. (2)
We call R ∈ R h I ×w I ×3 the final rendered image which is obtained by aggregating the color map of each triangle. In classical renderers, this step is done by using a Z-buffer so that only foreground objects are visible. This corresponds to:

R i = j w i j (z)C i j , with w i (z) = argmax y s.t. y 1=1,y≥0, yj =0 if I i j =0 z, y , (3)
where w, z ∈ R m+1 and for 1 ≤ j ≤ m, z j is the inverse depth of the j th triangle and the (m + 1) th coordinate of z and w account for the background. The inverse depth of the background is fixed to z m+1 = z min . From Eq. ( 2) and ( 3), it appears that argmax operations play a central role in renderers and are typically non-differentiable functions with respect to their input arguments, as discussed next.

Perturbed optimizers. In [START_REF] Berthet | Learning with differentiable perturbed optimizers[END_REF], Berthet et al. introduce a generic approach to handle non-smooth problems of the form y * (θ) = argmax y∈C θ, y with C a convex polytope, and make these problems differentiable by randomly perturbing them. More precisely, y * (θ) necessarily lies on a vertex of the convex set C. Thus, when θ is only slightly modified, y * remains on the same vertex, but when the perturbation grows up to a certain level, y * jumps onto another vertex. Concretely, this means that y * (θ) is piece-wise constant with respect to θ: the Jacobian J θ y * is null almost everywhere and undefined otherwise. This is why the rasterization (2) and aggregation (3) steps make renderers non-differentiable. Following [START_REF] Berthet | Learning with differentiable perturbed optimizers[END_REF], y * (θ) can be approximated by the perturbed optimizer:

y * (θ) = E Z [y * (θ + Z)] (4)
= E Z argmax y∈C θ + Z, y , (5)
where Z is a random noise following a distribution of the form µ(z) ∝ exp(-ν(z)). Then, we have y * (θ) -→0 ---→ y * (θ), y * (θ) is differentiable and its gradients are non-null everywhere. Intuitively, one can see from (5) that perturbed optimizers y * are actually a convolved version of the rigid optimizer y * . Moreover, their Jacobian with respect to θ is given by:

J θ y * (θ) = E Z y * (θ + Z) ∇ν(Z) . (6)
It is worth noticing at this stage that, when = 0, one recovers the standard formulation of the rigid optimizer. In general, y * (θ) and J θ y * (θ) do not admit closed-form expressions. To overcome this issue, Monte-Carlo estimators are exploited to compute an estimate of these quantities, as recalled in Sec. 4.

Related work

Our work builds on results in differentiable rendering, differentiable optimization and randomized smoothing.

Differentiable rendering. Some of the earliest differentiable renderers rely on the rigid rasterization-based rendering process recalled in the previous section, while exploiting some gradient approximations of the non-smooth operations in the backward. OpenDR [START_REF] Loper | Opendr: An approximate differentiable renderer[END_REF] uses a first-order Taylor expansion to estimate the gradients of a classical renderer, resulting in gradients concentrated around the edges of the rendered images. In the same vein, NMR [START_REF] Kato | Neural 3d mesh renderer[END_REF] proposes to avoid the issue of local gradients by doing manual interpolation during the backward pass. For both OpenDR and NMR, the discrepancy between the function evaluated in the forward pass and the gradients computed in the backward pass may lead to unknown or undesired behaviour when optimizing over these approximated gradients. More closely related to our work, SoftRas [START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF] directly approximates the forward pass of usual renderers to make it naturally differentiable. Similarly, DIB-R [START_REF] Chen | Learning to predict 3d objects with an interpolation-based differentiable renderer[END_REF] introduces the analytical derivatives of the faces' barycentric coordinates to get a smooth rendering of foreground pixels during the forward pass, and thus, gets gradients naturally relying on local properties of meshes. Additionally, a soft aggregation step is required to backpropagate gradients towards background pixels in a similar way to SoftRas. In a parallel line of work, physically realistic renderers are made differentiable by introducing a stochastic estimation of the derivatives of the ray tracing integral [START_REF] Li | Differentiable monte carlo ray tracing through edge sampling[END_REF][START_REF] Nimier-David | Mitsuba 2: A retargetable forward and inverse renderer[END_REF]. This research direction seems promising as it allows to generate images with global illumination effects. However, this class of renderers remains computationally expensive when compared to rasterization-based algorithms, which makes them currently difficult to exploit within classic computer vision or robotics contexts, where low computation timings matter.

Differentiable optimization and randomized smoothing. More generally, providing ways to differentiate solutions of constrained optimization problems has been part of the recent growing effort to introduce more structured operations in the differentiable programming paradigm [START_REF] Roulet | Differentiable programming à la moreau[END_REF], going beyond classic neural networks. A first approach consists in automatically unrolling the optimization process used to solve the problem [START_REF] Domke | Generic methods for optimization-based modeling[END_REF][START_REF] Maclaurin | Gradient-based hyperparameter optimization through reversible learning[END_REF]. However, because the computational graph grows with the number of optimization steps, implicit differentiation, which relies on the differentiation of optimality conditions, should be preferred when available [START_REF] Agrawal | Differentiating through a cone program[END_REF][START_REF] Amos | Optnet: Differentiable optimization as a layer in neural networks[END_REF][START_REF] Le Lidec | Differentiable simulation for physical system identification[END_REF]. These methods compute exact gradients whenever the solution is differentiable with respect to the parameters of the problem. In this paper, we show how differentiating through a classic renderer relates to computing derivatives of a Linear Programming (LP) problem. In this particular case and as recalled in the Sec. 2, solutions of the optimization problem vary in a heavily non-smooth way as gradients are null almost-everywhere and non-definite otherwise, thus, making them hard to exploit within classic optimization algorithms.

To overcome this issue, a first approach consists in introducing a gradient proxy by approximating the solution with a piecewise-linear interpolation [START_REF] Vlastelica | Differentiation of blackbox combinatorial solvers[END_REF] which can also lead to null gradient. Another approach leverages random perturbations to replace the original problem by a smoother approximation [START_REF] Abernethy | 1 perturbation techniques in online learning and optimization[END_REF][START_REF] Berthet | Learning with differentiable perturbed optimizers[END_REF]. For our differentiable renderer, we use this later method as it requires only little effort to transform non-smooth optimizers into differentiable ones, while guaranteeing non-null gradients everywhere. Moreover, randomized smoothing has been shown to facilitate optimization of non-smooth problems [START_REF] Duchi | Randomized smoothing for stochastic optimization[END_REF]. Solving a smooth approximation leads to improved performance when compared to methods dealing with the original rigid problem. In addition to its smoothing effect, the addition of noise also acts as an implicit regularization during the training process which is valuable for globalization aspects [START_REF] Berthet | Learning with differentiable perturbed optimizers[END_REF][START_REF] Cohen | Certified adversarial robustness via randomized smoothing[END_REF], i.e. converging towards better local optima.

Approximate differentiable rendering via perturbed optimizers

In this section, we detail the main contributions of the paper. We first introduce the reformulation of the rasterization and aggregation steps as perturbed optimizers, and propose a variance reduction mechanism to alleviate the computational burden inherent to these kinds of stochastic estimators. Additionally, we introduce an adaptive scheme to automatically adjust the smoothing parameters inherent to the rendering process.

Perturbed renderer: a general approximate and differentiable renderer

As detailed in Sec. 2, the rasterization step (2) can be directly written as the argmax of an LP.

Similarly, the aggregation (3) step can be slightly modified to reformulate it as an argmax of an LP. Indeed, when y ≥ 0, the hard constraint y j = 0 if I i j = 0 is equivalent to I i j yj > 0, which can be Figure 2: Examples of images obtained from a perturbed differentiable rendering process with a Gaussian noise. With smoothing parameters set to 0, we retrieve the rigid renderer, while adding noise makes pixels from the background appear on the foreground and vice versa.

approximated by adding a logarithmic barrier y j ln I j i in the objective function as done in classical interior point methods [START_REF] Boyd | Convex optimization[END_REF]:

w i α (z) = argmax y s.t. y 1 =1,y≥0 z + 1 α ln(I i ), y , (7)
because ln

I i j I i j - →0 ----→ -∞, enforcing w i j I i j - →0 ----→ 0. α -
→ +∞ approximates the hard constraint and allows to retrieve the classical formulation (3) of w. Using this formulation, it is possible to introduce a differentiable approximation of the rasterization and aggregation steps:

Îi j = H σ (d(i, j)), with H σ (x) = E X [H(x + σX)] = E X [argmax 0≤y≤1 y, x + σX ], (8)
Ri = j w i α,γ j (z)C i j , with w i α,γ (z) = E Z [w α (z + γZ)] (9)
= E Z [ argmax y s.t. y 1 =1,y≥0 y + 1 α ln( Îi ), z + γZ ]. (10)
By proceeding this way, we get a rendering process for which every pixel is influenced by every triangle of meshes (Fig. 1,2), similarly to SoftRas [START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF]. More concretely, as shown in [START_REF] Berthet | Learning with differentiable perturbed optimizers[END_REF], the gradients obtained with the randomized smoothing are guaranteed to be non-null everywhere unlike some existing methods [START_REF] Chen | Learning to predict 3d objects with an interpolation-based differentiable renderer[END_REF][START_REF] Kato | Neural 3d mesh renderer[END_REF][START_REF] Loper | Opendr: An approximate differentiable renderer[END_REF]. This makes it possible to use the resulting gradients as a strong supervision signal for optimizing directly at the pixel level, as shown through the experiments in Sec. 5. At this stage, it is worth noting that the prior distribution on the noise Z used for smoothing is not fixed. One can choose various distributions as a prior thus leading to different smoothing patterns, which makes the approach versatile. Indeed, as a different noise prior induces different smoothing, some specific choices allow to retrieve some of the existing differentiable renderers [START_REF] Kato | Neural 3d mesh renderer[END_REF][START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF][START_REF] Loper | Opendr: An approximate differentiable renderer[END_REF]. In particular, using a Logistic and a Gumbel prior respectively on X and Z leads to SoftRas [START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF]. Further details are included in Appendix A.

More practically, the smoothing parameters σ, γ appearing in differentiable renderers such as SoftRas [START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF] or DIB-R [START_REF] Chen | Learning to predict 3d objects with an interpolation-based differentiable renderer[END_REF] can be hard to set. In contrast, in the proposed approach, the smoothing is naturally interpretable as a noise acting directly on positions of mesh triangles. Thus, the smoothing parameters representing the noise intensity can be scaled automatically according to the object dimensions. In Sec. 4.3, we notably propose a generic approach to automatically adapt them along the optimization process.

Exploiting the noise inside a perturbed renderer

In practice, ( 5) and ( 6) are useful even in the cases when the choice of prior on Z does not induce any analytical expression for y * and J θ y * . Indeed, Monte-Carlo estimators of these two quantities can be directly obtained from these two expressions: where M is the total number of samples used to compute the approximations.

y M (x) = 1 M M i=1 y * (θ + Z (i) ) (11)
J θ y M (z) = 1 M M i=1 y * (θ + Z (i) ) 1 ∇ν(Z (i) ) , (12)
Approximately, we have Var

J θ y M (z) ∝ Var[y * (θ+ Z)∇ν(Z (i) ) ] M 2
. When decreasing , the variance of the Monte-Carlo estimator increases, which can make the gradients very noisy and difficult to exploit in practice. For this reason, we use the control variates method [START_REF] Law | Simulation modeling and analysis[END_REF] in order to reduce the variance of our estimators. Because the quantity y * (θ)∇ν(Z) has a null expectation when Z has a symmetric distribution and is positively correlated with y * (θ + Z)∇ν(Z) , we can rewrite (6) as:

J θ y * (θ) = E Z (y * (θ + Z) -y * (θ)) ∇ν(Z) / , (13)
which naturally leads to a variance-reduced Monte-Carlo estimator of the Jacobian:

J θ y M (θ) = 1 M M i=1 y * (θ + Z (i) ) -y * (θ) 1 ∇ν(Z (i) ) . ( 14
)
The computation of y M requires the solution of M perturbed problems instead of only one in the case of classical rigid optimizer. Fortunately, this computation is naturally parallelizable, leading to a constant computation time at the cost of an increased memory footprint. Using our variancereduced estimator alleviates the need for a high number of Monte Carlo samples, hence reducing the computational burden inherent to perturbed optimizers (Fig. 3). As shown in Sec. 5 (Fig. 5, left), M = 8 samples is already sufficient to get stable and accurate results. In addition, computations from the forward pass can be reused in the backward pass and time and memory complexities to evaluate these passes are comparable to classical differentiable renderers (Tab. 1). Consequently, evaluating gradients does not require any extra computation. It is worth mentioning at this stage that this variance reduction mechanism in fact applies to any perturbed optimizer. This variance reduction technique can be interpreted as estimating a sub-gradient with the finite differences in a random direction and is inspired by the field of random optimization [START_REF] Nesterov | Random gradient-free minimization of convex functions[END_REF].

Making the smoothing adaptive with sensitivity analysis

In a way similar to [START_REF] Boyd | Convex optimization[END_REF], it is possible to formulate the sensitivity of the perturbed optimizer with respect to the smoothing parameter :

J y * (θ) = E Z y * (θ + Z) ∇ν(Z) Z -1 / (15)
and, as previously done, we can build a variance-reduced Monte-Carlo estimator of this quantity:

J y M (θ) = 1 M M i=1 y * (θ + Z (i) ) -y * (θ) ∇ν(Z (i) ) Z (i) -1 (16)
Figure 4: Perturbed differentiable renderer and adaptive smoothing lead to precise 6D pose estimation.

In the case of differentiable rendering, a notable property is the positivity of the sensitivity of any locally convex loss L (which is valid for the RGB loss) with respect to the γ smoothing parameter when approaching the solution. Indeed, in the neighborhood of the solution, we have θ ≈ θ t and a first-order Taylor expansion gives:

∂L ∂γ (θ) ≈ J γ w γ (z) C ∇ 2 L(R(θ t ))CJ γ w γ (z) ≥ 0, (17)
because for a locally convex loss, ∇ 2 L(R(θ t )) is positive definite near a local optimum. We exploit this property to gradually reduce the smoothing when needed. To do so, we track an exponential moving average of the sensitivity during the optimization process:

v t γ = β γ v t-1 γ + (1 -β γ ) ∂L ∂γ (θ t ) (18)
where β γ is a scalar parameter. Whenever v t γ is positive, the smoothing (σ, γ) is decreased at a constant rate (Fig. 4). Note that the adaptive algorithm does not require any further computation as sensitivity can be obtained from backpropagation and also beneficially applies to other differentiable renderers [START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF] (see 5).

Results

In this section, we explore applications of the proposed differentiable rendering pipeline to standard computer vision tasks: single-view 3D pose estimation and 3D mesh reconstruction. Our implementation is based on Pytorch3d [START_REF] Ravi | Accelerating 3d deep learning with pytorch3d[END_REF] and will be publicly released upon publication. It corresponds to a modular differentiable renderer where switching between different type of noise distributions (Gaussian, Cauchy, etc.) is possible. We notably compare our renderer against the open source SoftRas implementation available in Pytorch3d and DIB-R from Kaolin library. The results for these renderers are obtained by running the experiments in our setup as we were not able to get access to the original implementations of the pose optimization and shape reconstruction problems.

Single-view 3D pose estimation is concerned with the retrieving the 3D rotation θ ∈ SO(3) of an object from a reference image. Similarly to [START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF], we first consider the case of a colored cube and fit the rendered image of this cube to a single view of the true pose R(θ t ) (Fig. 4). The problem can be formulated as a regression problem of the form:

min θ L RGB (θ) = 1 2 R(θ t ) -R(θ) 2 2 , ( 19
)
where R is the smooth differentiable rendered image. We aim at analysing the sensitivity of our perturbed differentiable renderer with respect to random initial guesses of various amplitudes from the real pose. Thus, the initial θ is randomly perturbed from the true pose with various intensity for the perturbation (exploiting an angle-axis representation of the rotation, with a randomized rotation axis). The intensity of the angular perturbation is taken between 20 and 80 degrees. Consequently, the value of the average final error has an increased variance and is often perturbed by local minima (Tab. 2). To avoid these caveats, we rely on another metric: the percentage of solved tasks (see Tab. 2 and Fig. 5 and6), accounting for the amount of final errors which are below a given threshold (10 degrees for Tab. 2 and bottom row of Fig. 6). For this task, we compare our method with two different distributions for the smoothing noise (Gaussian and Cauchy priors) to SoftRas. We use Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with parameters β 1 = 0.9 and β 2 = 0.999 and operate with 128×128 RGB images, each optimization problem taking about 1 minute to solve on a Nvidia RTX6000 GPU. We provide results of 100 random perturbations for various magnitude of initial perturbations in Tab. 2 and Fig. 5,6. Error bars provide standard deviations while running experiments with different random seeds. Fig. 5 (left) demonstrates the advantage of our variance reduction method to estimate perturbed optimizers and precisely retrieve the true pose at a lower computational cost. Indeed, the variance-reduced perturbed renderer is able to perform precise optimization with only 2 samples (80% of final errors are under 5°while it is less than 50% without the use of control variates). Additionally, our adaptive smoothing significantly improves the number of solved tasks for both SoftRas and our perturbed renderer (Fig. 5, right).

Additional results in Tab. 2, Fig. 6 and Fig. 11 in Appendix demonstrate that our perturbed renderers with a Gaussian smoothing, outperforms SoftRas and can achieve state-of-the-art results for pose optimization tasks on various objects. As shown in Appendix (Prop. 4), SoftRas is equivalent to using a Gumbel smoothing which is an asymmetric prior. However, no direction should be preferred when smoothing the rendering process which explain the better performances of the Gaussian prior. Bottom row of Fig. 6 shows that our method is robust to the choice of initial values of smoothing parameters. During optimization, the stochasticity of the gradient due to the noise from the perturbed renderer acts as an implicit regularization of the objective function. This helps to avoid sharp local minima or saddle points and converge to more stable minimal regions, leading to better optima. As mentioned earlier, results are limited by the inherent non-convexity of the rendering process and differentiable renderers are not able to recover from a bad initial guess. Increasing the noise intensity to further smoothen the objective function would not help in this case, as it would also lead to a critical loss of information on the rendering process.

Self-supervised 3D mesh reconstruction from a single image. In our second experiment, we demonstrate the ability of our perturbed differentiable renderer to provide a supervisory signal for training a neural network. We use the renderer to self-supervise the reconstruction of a 3D mesh and the corresponding colors from a single image (see Fig. 7). To do so, we use the network architecture proposed in [START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF] and the subset of the Shapenet dataset [START_REF] Chang | Shapenet: An information-rich 3d model repository[END_REF] from [START_REF] Kato | Neural 3d mesh renderer[END_REF]. The network is composed of a convolutional encoder followed by two decoders: one for the mesh reconstruction and another one for color retrieving (more details in Appendix B).

Using the same setup as described in [START_REF] Chen | Learning to predict 3d objects with an interpolation-based differentiable renderer[END_REF][START_REF] Kato | Neural 3d mesh renderer[END_REF][START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF] , the training is done by minimizing the following loss:

L = λ sil L sil + λ RGB L RGB + λ lap L lap , (20)
where L sil is the negative IoU between silhouettes I and Î, L RGB is the 1 RGB loss between R and R, L lap is a Laplacian regularization term penalizing the relative change in position between neighbour vertices (detailed description in Appendix B). For training, we use Adam algorithm with a learning rate of 10 -4 and parameters β 1 = 0.9, β 2 = 0.999. Even if the memory consumption of the perturbed renderer may limit the maximum size of batches, its flexibility makes it possible to switch to a deterministic approximation such as [START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF] to use larger batches in order to finetune the neural network, if needed.

We analyze the quality of obtained 3D reconstruction through a rendering pipeline by providing IoU measured on 13 different classes of the test set (see Tab. 3). We observe that our perturbed renderer obtains state-of-the-art accuracies on this task. Error bars (representing the standard deviation obtained by retraining the model with different random seeds) confirm the stability of our method. Additionally, we illustrate qualitative results for colorized 3D mesh reconstruction from a single image in Fig. 7(right) confirming the ability of perturbed renderers to provide strong visual supervision signals.

Conclusion

In this paper, we introduced a novel approach to differentiable rendering leveraging recent contributions on perturbed optimizers. We demonstrated the flexibility of our approach grounded in its underlying theoretical formulation. The combination of our proposed perturbed renderers with variance-reduction mechanisms and sensitivity analysis enables for robust application of our rendering approach in practice. Our results notably show that perturbed differentiable rendering can reach state-of-the-art performance on pose optimization and is also competitive for self-supervised 3D mesh reconstruction. Additionally, perturbed differentiable rendering can be easily implemented on top of existing implementations without major modifications, while generalizing existing renderers such as SoftRas. As future work, we plan to embed our generic differentiable renderer within a differentiable simulation pipeline in order to learn physical models, by using visual data as strong supervision signals to learn physics.

Appendices A Differentiable perturbed renderer

This section describes some possible choice of priors for the smoothing noise of our differentiable perturbed renderers and discuss the eventual links with already existing renderers. In addition, we provide justification of the variance reduction and adaptive smoothing methods for different priors on the noise.

A.1 Perturbed renderers

The following propositions discuss some possible choices of noise prior for the perturbed renderer, exhibiting the links with already existing differentiable renderers [START_REF] Loper | Opendr: An approximate differentiable renderer[END_REF][START_REF] Kato | Neural 3d mesh renderer[END_REF][START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF]. Proposition 1. The sigmoid function corresponds to the perturbed Heaviside function with a logistic prior on noise:

sigmoid(x) = E[H(x + Z)] (21)
where Z ∼ Logistic(0, 1)

Proof. First, we note that the sigmoid function correspond to the first weight from the softmax applied to the R 2 vector x = x 0 . Indeed, we have :

σ(x) = 1 1 + exp(-x) = exp(x) exp(x) + exp(0) (22)
= sof tmax ( x)

T 1 0 (23)
From the previous proposition, with Z a random variable in R 2 distributed with a Gumbel distribution, we have:

sof tmax( x) = E[ argmax y s.t. y 1=1 y, x + Z ] (24)
= E[ argmax y s.t. y 1=1

y T x + Z 1 Z 2 ] ( 25
) = E[ argmax y s.t. y 1=1 y T x + (Z 1 -Z 2 ) 0 ] (26)
because

x + (Z 1 -Z 2 ) > 0 is equivalent to x + Z 1 > Z 2 .
Finally, we can re-write:

σ(x) = E[ argmax y s.t. y 1 =1 y T x + Z 0 ] T 1 0 (27) = E[argmax 0≤y≤1 y(x + Z)] (28)
= E[H(x + Z)] (29)
where Z follows a logistic law.

Which allows to conclude that the sigmoid corresponds to the Heaviside function perturbed with a logistic noise. We could use the fact that the cumulative distribution function of the logistic distribution is the sigmoid function to make an alternative proof.

Proposition 2. The affine approximation of the step function corresponds to the perturbed Heaviside function with an uniform prior on noise:

H af f (x) = E[H(x + Z)] (30)
where Z ∼ U(-1 2 , 1 2 )

Figure 8: Modifying the prior on the noise leads to different approximations of a non-smooth operator.

The sigmoid and affine approximations are obtained by using respectively a Logistic and a Uniform prior.

Following the previous results, one can observe that OpenDR [START_REF] Loper | Opendr: An approximate differentiable renderer[END_REF] corresponds to a rasterization step where a uniform prior is used for the smoothing noise in order to get gradients during the backward pass. More precisely, the uniform distribution is taken with a support as large as a pixel size which allows to get non null gradients for pixels lying on the border of triangles. This can be linked to antialiasing techniques which are added to classical rendering pipeline to remove high frequencies introduced by the non-smooth rasterization. These techniques make the intensity of a pixel vary linearly with respect to the position of the triangle which can also be seen as an affine approximation of the rasterization. However, one should note that the density of the uniform distribution cannot be written in the form of µ(z) ∝ exp(-ν(z)) so the properties on differentiability from [START_REF] Berthet | Learning with differentiable perturbed optimizers[END_REF] does not apply. In particular, this approximation is non-differentiable at some points (x = 1 or -1) and can have null gradients (for x < -1 or x > 1) which is inducing localized gradients for OpenDR [START_REF] Loper | Opendr: An approximate differentiable renderer[END_REF]. For this reason, the approach of NMR [START_REF] Kato | Neural 3d mesh renderer[END_REF] actually consists in having a variable range for the distribution support in order to get non-null gradients for a wider region. This approach corresponds to the affine interpolation proposed in [START_REF] Vlastelica | Differentiation of blackbox combinatorial solvers[END_REF] and can lead to null gradients. More closely related to our approach, SoftRas smoothen the rasterization by using a sigmoid approximation while Z-buffering is replaced by a softmax during aggregation. As shown by propositions 1 and 4, this approach actually corresponds to a special case of a perturbed renderer where rasterization and aggregation steps were smoothen by using respectively a Logistic and a Gumbel prior for the noise.

In short, this means that some existing renderers [START_REF] Loper | Opendr: An approximate differentiable renderer[END_REF][START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF][START_REF] Kato | Neural 3d mesh renderer[END_REF] can be interpreted in the framework of differentiable perturbed renderers when considering specific priors for the noise used for smoothing.

In addition, the generality of the approach allows to consider a continuous range of novel differentiable renderers in between the already existing ones.

A.2 Variance reduction

As introduced in the paper, control variates methods can be used to reduce the noise of the Monte-Carlo estimators of the Jacobian of a perturbed renderer, without inducing any extra-computation. In particular, we prove this in the case of Gaussian and Cauchy priors on the smoothing noise.

Proposition 5. Jacobian of perturbed renderers can be written as :

J θ y * (θ) = E Z (y * (θ + Z) -y * (θ)) ∇ν(Z) 1 , (46)
when Z follows a Gaussian distribution.

Proof. In a similar way to 5, using the same change of variable, we get:

J θ y * (θ) = ∂ ∂ ∞ -∞ y * (θ + z)µ(z)dz , (59) = ∞ -∞ y * (z ) ∂ ∂ µ z -θ dz , (60) = ∞ -∞ y * (z ) 1 2 ∇ν z -θ z -θ -1 µ z -θ dz ( 61
)
which yields with the inverse change of variable:

J y * (θ) = E Z y * (θ + Z) ∇ν(Z) Z -1 (62)
Then, for Z following a Gaussian distribution, we have:

E Z y * (θ) ∇ν(Z) Z -1 = y * (θ)E Z Z 2 -1 (63) = 0 (64)
for Z following a standard Gaussian noise. Finally, we can write:

J y * (θ) = E Z (y * (θ + Z) -y * (θ)) ∇ν(Z) Z -1 (65)
Proposition 8. The sensitivity of the perturbed optimizer with respect to the smoothing parameter can be expressed as:

J y * (θ) = E Z (y * (θ + Z) -y * (θ)) ∇ν(Z) Z -1 (66)
for Z a smoothing noise following a Cauchy prior.

Proof. We adapt the previous proof to the case of a Cauchy distribution for the noise. We observe that:

J y * (θ) = E Z y * (θ + Z) ∇ν(Z) Z -1 (67)
remains valid. In addition, we have :

E Z y * (θ) ∇ν(Z) Z -1 = y * (θ) 1 E Z 2 Z 2 1 + Z 2 -1 (68)
for Z following a Cauchy noise. Moreover, an integration by parts gives:

∞ -∞ 2x 2 (1 + x 2 ) 2 dx = -x 1 + x 2 ∞ -∞ + ∞ -∞ 1 1 + x 2 dx (70) = [arctan(x)]] ∞ -∞ (71)
so we have :

E Z 2 Z 2 1 + Z 2 -1 = 0 (72)
As previously done in Proposition 7, we finally get:

J y * (θ) = E Z (y * (θ + Z) -y * (θ)) ∇ν(Z) Z -1 (73)
Figure 9: Pose optimization with an initial guess uniformly sampled on the rotation space. Results are reported for our differentiable perturbed renderer with Adaptive Smoothing (AS) and Variance Reduction (VR), and for SoftRas [START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF] with and without Adaptive Smoothing.

B Experimental results

In this section, we provide details on our experimental setup and some additional results. In addition, we provide the code for our differentiable perturbed renderer based on Pytorch3d [START_REF] Ravi | Accelerating 3d deep learning with pytorch3d[END_REF] and the experiments on pose optimization while additional experiments will be made available upon publication.

B.1 Pose optimization

Trying to solve pose optimization from an initial guess taken uniformly in the rotation space leads to very variable results when measuring the average final error (Fig. 9). When the initialization is too far from the initial guess, the optimization process converges towards a local optima. This limitation is inherent to approaches using differentiable rendering to solve pose optimization tasks and could be avoided only by using a combination with other methods, e.g. Render & Compare [START_REF] Labbe | Cosypose: Consistent multi-view multi-object 6d pose estimation[END_REF], to get better initial guess for the pose. For this reason, we prefer to consider the ability of our differentiable renderer to retrieve the true pose with an initial guess obtained by perturbing the target and measure the amount of final errors below a given precision threshold (Fig. [START_REF] Deng | Self-supervised 6d object pose estimation for robot manipulation[END_REF][START_REF] Domke | Generic methods for optimization-based modeling[END_REF].

In addition to the pose optimization task on the cube, we propose to solve the same problem with others objects from Shapenet dataset [START_REF] Chang | Shapenet: An information-rich 3d model repository[END_REF] (Fig. 11).

B.2 Mesh reconstruction

Reconstruction network Network architecture is based on [START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF] and is represented in Fig. 12. The encoder is composed of 3 convolutional layers (each of them is followed by a batch normalization operations) and 3 fully connected layers. We use two different decoders to retrieve respectively the vertices' position and their color. The positional decoder is composed of 3 fully connected layers. The decoder used for colors retrieval is composed of a branch selecting a palette of N p colors on the input image while the second branch mixes these colors to colorize each of the N v vertices.

Training setup We use the same setup as described in [START_REF] Chen | Learning to predict 3d objects with an interpolation-based differentiable renderer[END_REF][START_REF] Kato | Neural 3d mesh renderer[END_REF][START_REF] Liu | Soft rasterizer: A differentiable renderer for image-based 3d reasoning[END_REF]. The training is done by minimizing a loss combining silhouette and color information with a regularization term:

L = λ sil L sil + λ RGB L RGB + λ lap L lap , (74)
and we set λ sil = 1, λ RGB = 1, λ lap = 3.10 -3 during training. In more details, L sil is the negative IoU between silhouettes I and Î which can be expressed as:

L sil = E 1 - I Î 1 I + Î -I Î 1 ( 75
)
L RGB is the 1 RGB loss between R and R:

L RGB = R -R 1 ( 76
)
L lap is a Laplacian regularization term penalizing the relative change in position between a vertices v and its neighbours N (v) which can be written:

L lap = v   v - 1 |N (v)| v ∈N (v) v   2 (77)
Additional results We provide additional qualitative results for colorized mesh reconstruction (Fig. 13).

Figure 3 :

 3 Figure 3: Control variates methods [19] allow to reduce the variance of the gradient of the smooth argmax. Left: Gaussian-perturbed argmax operator, Middle: estimator of gradient Right: variancereduced estimator of gradient.

Figure 5 :

 5 Figure 5: Left: Variance reduction (VR) drastically reduces the number of Monte-Carlo samples required to retrieve the true pose from a 20°perturbation. Right: Adaptive smoothing (AS) improves resolution of precise pose optimization for a 80°perturbation.

Figure 6 :

 6 Figure 6: Top row: perturbed renderer combined with variance reduction and adaptive scheme improves SoftRas results on pose optimization. Bottom row: The method is robust w.r.t. initial smoothing values. Higher is better.

Figure 7 :

 7 Figure 7: Left: A neural network is trained with a self-supervision signal from a differentiable rendering process. Right: Qualitative results from self-supervised 3D mesh reconstruction using a perturbed differentiable renderer. 1 st column: Input image. 2 nd and 3 rd columns: reconstructed mesh viewed from different angles.

Figure 10 :

 10 Figure 10: Pose optimization results for a perturbed renderer with Gaussian and Cauchy priors on the smoothing noise, and for SoftRas.

Figure 11 :

 11 Figure 11: Pose optimization on various objects: a bag (Top) and a mailbox (Bottom)

Figure 12 :

 12 Figure 12: Learning architecture for the task of colorized 3D mesh reconstruction. First row: Overview of reconstruction network. Second row: Encoder network. Third row: Positional decoder. Fourth row: Color decoder.

Figure 13 :

 13 Figure 13: Results from unsupervised 3D mesh and color reconstruction. 1 st and 4 th column: Input image. Other columns: Reconstructed 3D mesh from different angles.

Table 1 :

 1 Time and memory complexity of our perturbed renderer for the forward and backward computation during the pose optimization task[START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF].

	# of samples	1	2	8	32	64	SoftRas Hard renderer
	Forward (ms)	30 (±3) 30 (±3) 31 (±3) 31 (±3) 31 (±3) 29 (±3)	29 (±3)
	Backward (ms)	19 (±1) 19 (±1) 19 (±2) 20 (±1) 22 (±1) 18 (±1)	N/A
	Max memory (Mb)	217.6	217.6	217.6	303.1	440.0	180.3	178.5

Table 2 :

 2 Results of pose optimization from single-view image using perturbed differentiable renderer with variance reduction and adaptive smoothing.

	Initial error		20°50°80°D
	iff. renderer	SoftRas Cauchy	Gaussian	SoftRas Cauchy	Gaussian	SoftRas Cauchy	Gaussian
			smoothing	smoothing		smoothing	smoothing		smoothing	smoothing
	Avg. error (°)	5.5	6.1	3.8	14.6	20.9	11.6	33.5	41.8	25.3
	Std. error (°)	4.9	5.1	4.4	26.6	21.0	22.5	35.3	34.0	34.9
	Task solved (%) 84 ±4.0	82 ±2.3	93 ±0.4	75 ±5.3	71 ±2.5	84 ±1.3	55 ±4.8	55 ±5.5	67 ±2.6

Table 3 :

 3 Results of mesh reconstruction on the ShapeNet dataset[START_REF] Chang | Shapenet: An information-rich 3d model repository[END_REF] reported with 3D IoU (%) ±0.08) (±0.32) (±0.15) (±0.11) (±0.14) (±0.31) (±0.24) (±0.09) (±0.12) (±0.21) (±0.33) (±0.07) (±0.11)

		Airplane	Bench	Dresser	Car	Chair	Display	Lamp	Speaker	Rifle	Sofa	Table	Phone	Vessel	Mean
	NMR [16]	58.5	45.7	74.1	71.3	41.4	55.5	36.7	67.4	55.7	60.2	39.1	76.2	59.4	57.0
	SoftRas [23] 2	62.0	47.55	66.2	69.4	49.4	60.0	43.3	62.6	61.4	60.4	43.6	76.4	59.9	58.6
	DIB-R [8] 2	59.7	50.9	66.2	72.6	52.0	57.9	43.8	63.8	61.0	65.4	50.5	76.3	58.6	59.9
	Ours	63.5	49.4	67.1	72.3	51.6	60.4	44.3	62.8	65.7	66.7	49.2	80.2	60.0	61.0
		(±0.15) (

These numbers were obtained by running the renderer in our own setup (cameras, lightning, training parameters etc.) in order to have comparable results. This may explain the slight difference with the numbers from the original publications.

Acknowledgments and Disclosure of Funding

We warmly thank Francis Bach for useful discussions. This work was supported in part by the HPC resources from GENCI-IDRIS(Grant AD011012215), the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute), and Louis Vuitton ENS Chair on Artificial Intelligence.

Proof. We have :

Proposition 3. The arctan approximation of the step function corresponds to the perturbed Heaviside function with a Cauchy prior on noise:

where Z ∼ Cauchy(0, 1)

Proof. We have :

Proposition 4. The softmax function corresponds to the perturbed maximal coordinates with a Gumbel prior on the noise:

where Z ∼ Gumbel(0, 1)

Proof. The Gumbel-max trick is a classical property of the Gumbel distribution stating: the maximizer of M fixed values z i which have been perturbed by a noise i.i.d Z i following a Gumbel distribution, follows an exponentially weighted distribution [START_REF] Gumbel | Statistical theory of extreme values and some practical applications: a series of lectures[END_REF]. This can be written:

where Z i are i.i.d. and following a Gumbel distribution. Thus, for a noise Z following a Gumbel distribution, we have :

= i e i P(max

Proof. As done in [START_REF] Berthet | Learning with differentiable perturbed optimizers[END_REF][START_REF] Abernethy | 1 perturbation techniques in online learning and optimization[END_REF], with a change of variable z = θ + z we have:

And by doing the inverse change of variable, we get:

In addition, for Z following a standard Gaussian distribution we have

and thus:

and because Z is centered, we get:

Finally we get:

Proposition 6. Jacobian of perturbed renderers can be written as :

when Z follows a Cauchy distribution.

Proof. Proceeding in a similar way to 5, for a Cauchy distribution we have ν(z) = ln(1 + z 2 ), thus

and finally we have:

for Z following a Cauchy distribution

A.3 Adaptive smoothing

In the same way as we proceeded for Jacobians computation, control variates method can be used to reduce the variance of Monte-Carlo estimators of the sensitivity of perturbed optimizers with respect to smoothing parameter . Proposition 7. The sensitivity of the perturbed optimizer with respect to the smoothing parameter can be expressed as:

J y * (θ) = E Z (y * (θ + Z) -y * (θ)) ∇ν(Z) Z -1 (58)

for Z a smoothing noise following a Gaussian prior.