Tom Monnier

Matthew Fisher

Alexei A Efros

Mathieu Aubry

Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency

Keywords:

Given a collection of single-view images from an object category, we learn without additional supervision an autoencoder that explicitly generates shape, texture, pose and background. (right) At inference time, our approach reconstructs highquality textured meshes from raw single-view images.

Abstract. Approaches for single-view reconstruction typically rely on viewpoint annotations, silhouettes, the absence of background, multiple views of the same instance, a template shape, or symmetry. We avoid all such supervision and assumptions by explicitly leveraging the consistency between images of different object instances. As a result, our method can learn from large collections of unlabelled images depicting the same object category. Our main contributions are two ways for leveraging crossinstance consistency: (i) progressive conditioning, a training strategy to gradually specialize the model from category to instances in a curriculum learning fashion; and (ii) neighbor reconstruction, a loss enforcing consistency between instances having similar shape or texture. Also critical to the success of our method are: our structured autoencoding architecture decomposing an image into explicit shape, texture, pose, and background; an adapted formulation of differential rendering; and a new optimization scheme alternating between 3D and pose learning. We compare our approach, UNICORN, both on the diverse synthetic ShapeNet dataset -the classical benchmark for methods requiring multiple views as supervision -and on standard real-image benchmarks (Pascal3D+ Car, CUB) for which most methods require known templates and silhouette annotations. We also showcase applicability to more challenging real-world collections (CompCars, LSUN), where silhouettes are not available and images are not cropped around the object.

Introduction

One of the most magical human perceptual abilities is being able to see the 3D world behind a 2D image -a mathematically impossible task! Indeed, the ancient Greeks were so incredulous at the possibility that humans could be "hallucinating" the third dimension, that they proposed the utterly implausible Emission Theory of Vision [START_REF] Finger | Origins of neuroscience: a history of explorations into brain function[END_REF] (eye emitting light to "sense" the world) to explain it to themselves. In the history of computer vision, single-view reconstruction (SVR) has had an almost cult status as one of the holy grail problems [START_REF] Hoiem | Geometric Context from a Single Image[END_REF][START_REF] Hoiem | Putting Objects in Perspective[END_REF][START_REF] Saxena | Make3D: Learning 3D Scene Structure from a Single Still Image[END_REF]. Recent advancements in deep learning methods have dramatically improved results in this area [START_REF] Choy | 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction[END_REF][START_REF] Mescheder | Occupancy Networks: Learning 3D Reconstruction in Function Space[END_REF]. However, the best methods still require costly supervision at training time, such as multiple views [START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF][START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF]. Despite efforts to remove such requirements, the works with the least supervision still rely on two signals limiting their applicability: (i) silhouettes and (ii) strong assumptions such as symmetries [START_REF] Hu | Self-Supervised 3D Mesh Reconstruction From Single Images[END_REF][START_REF] Kanazawa | Learning Category-Specific Mesh Reconstruction from Image Collections[END_REF], known template shapes [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF][START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF], or the absence of background [START_REF] Wu | Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild[END_REF]. Although crucial to achieve reasonable results, priors like silhouettes and symmetry can also harm the reconstruction quality: silhouette annotations are often coarse [START_REF] Chen | Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer[END_REF] and small symmetry prediction errors can yield unrealistic reconstructions [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF][START_REF] Wu | Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild[END_REF].

In this paper, we propose the most unsupervised approach to single-view reconstruction to date, which we demonstrate to be competitive for diverse datasets. Table 1 summarizes the differences between our approach and representative prior works. More precisely, we learn in an analysis-by-synthesis fashion a network that predicts for each input image: 1) a 3D shape parametrized as a deformation of an ellipsoid, 2) a texture map, 3) a camera viewpoint, and 4) a background image (Fig. 1). Our main insight to remove the supervision and assumptions required by other methods is to leverage the consistency across different instances. First, we design a training procedure, progressive conditioning, which encourages the model to share elements between images by strongly constraining the variability of shape,

Method Supervision Data Output

Pix2Mesh [START_REF] Wang | Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images[END_REF], AtlasNet [START_REF] Groueix | AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation[END_REF], OccNet [START_REF] Mescheder | Occupancy Networks: Learning 3D Reconstruction in Function Space[END_REF] 3D ShapeNet 3D PTN [START_REF] Yan | Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision[END_REF], NMR [START_REF] Kato | Neural 3D Mesh Renderer[END_REF] MV, CK, S ShapeNet 3D DRC [START_REF] Tulsiani | Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency[END_REF], SoftRas [START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF], DVR [START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF] MV, CK, S ShapeNet 3D, T GANverse3D [START_REF] Zhang | Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering[END_REF] MV, CK, S Bird, Car, Horse 3D, T DPC [START_REF] Insafutdinov | Unsupervised Learning of Shape and Pose with Differentiable Point Clouds[END_REF], MVC [START_REF] Tulsiani | Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction[END_REF] MV, S ShapeNet 3D, P Vicente et al. [START_REF] Vicente | Reconstructing PASCAL VOC[END_REF], CSDM [START_REF] Kar | Category-Specific Object Reconstruction from a Single Image[END_REF], DRC [START_REF] Tulsiani | Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency[END_REF] CK, S Pascal3D 3D CMR [START_REF] Kanazawa | Learning Category-Specific Mesh Reconstruction from Image Collections[END_REF] CK, S, A( †) Bird, Car, Plane 3D, T SDF-SRN [START_REF] Lin | SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images[END_REF], TARS [START_REF] Duggal | Topologically-Aware Deformation Fields for Single-View 3D Reconstruction[END_REF] CK, S ShapeNet, Bird, Car, Plane 3D, T TexturedMeshGen [START_REF] Henderson | Leveraging 2D Data to Learn Textured 3D Mesh Generation[END_REF] CK, A( †) ShapeNet, Bird, Car 3D, T UCMR [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF], IMR [START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF] S, A(♢, †) Animal, Car, Moto 3D, T, P UMR [START_REF] Li | Selfsupervised Single-view 3D Reconstruction via Semantic Consistency[END_REF] S, A(↔, †) Animal, Car, Moto 3D, T, P RADAR [START_REF] Wu | De-rendering the World's Revolutionary Artefacts[END_REF] S, A( ‡) Vase 3D, T, P SMR [START_REF] Hu | Self-Supervised 3D Mesh Reconstruction From Single Images[END_REF] S, A( †) ShapeNet, Animal, Moto 3D, T, P Unsup3D [START_REF] Wu | Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild[END_REF] A(⊠, ⋖, †) Face D, T, P Henderson & Ferrari [START_REF] Henderson | Learning single-image 3D reconstruction by generative modelling of shape, pose and shading[END_REF] A(⊠, ∅) ShapeNet 3D, P Ours None ShapeNet, Animal, Car, Moto 3D, T, P texture and background at the beginning of training and progressively allowing for more diversity (Fig. 2a). Second, we introduce a neighbor reconstruction loss, which explicitly enforces neighboring instances from different viewpoints to share the same shape or texture model (Fig. 2b). Note that these simple yet effective techniques are data-driven and not specific to any dataset. Our only remaining assumption is the knowledge of the semantic class of the depicted object.

We also provide two technical insights that we found critical to learn our model without viewpoint and silhouette annotations: (i) a differentiable rendering formulation inspired by layered image models [START_REF] Jojic | Learning Flexible Sprites in Video Layers[END_REF][START_REF] Monnier | Unsupervised Layered Image Decomposition Into Object Prototypes[END_REF] which we found to perform better than the classical SoftRasterizer [START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF], and (ii) a new optimization strategy which alternates between learning a set of pose candidates with associated probabilities and learning all other components using the most likely candidate.

We validate our approach on the standard ShapeNet [START_REF] Chang | ShapeNet: An Information-Rich 3D Model Repository[END_REF] benchmark, real image SVR benchmarks (Pascal3D+ Car [START_REF] Xiang | Beyond PASCAL: A benchmark for 3D object detection in the wild[END_REF], CUB [START_REF] Welinder | Caltech-UCSD Birds 200[END_REF]) as well as more complex real-world datasets (CompCars [START_REF] Yang | A large-scale car dataset for fine-grained categorization and verification[END_REF], LSUN Motorbike and Horse [START_REF] Yu | LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop[END_REF]). In all scenarios, we demonstrate results competitive with the best supervised methods.

Summary. We present UNICORN, a framework leveraging UNsupervised cross-Instance COnsistency for 3D ReconstructioN. Our main contributions are: 1) the most unsupervised SVR system to date, demonstrating state-of-theart textured 3D reconstructions for both generic shapes and real images, and not requiring supervision or restrictive assumptions beyond a categorical image collection; 2) two data-driven techniques to enforce cross-instance consistency, namely progressive conditioning and neighbor reconstruction. Code and video results are available at imagine.enpc.fr/ monniert/UNICORN.

Related work

We first review deep SVR methods and mesh-based differential renderers we build upon. We then discuss works exploring cross-instance consistency and curriculum learning techniques, to which our progressive conditioning is related.

Deep SVR. There is a clear trend to remove supervision from deep SVR pipelines to directly learn 3D from raw 2D images, which we summarize in Table 1.

A first group of methods uses strong supervision, either paired 3D and images or multiple views of the same object. Direct 3D supervision is successfully used to learn voxels [START_REF] Choy | 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction[END_REF], meshes [START_REF] Wang | Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images[END_REF], parametrized surfaces [START_REF] Groueix | AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation[END_REF] and implicit functions [START_REF] Mescheder | Occupancy Networks: Learning 3D Reconstruction in Function Space[END_REF][START_REF] Xu | DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction[END_REF]. The first methods using silhouettes and multiple views initially require camera poses and are also developed for diverse 3D shape representations: [START_REF] Tulsiani | Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency[END_REF][START_REF] Yan | Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision[END_REF] opt for voxels, [START_REF] Chen | Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer[END_REF][START_REF] Kato | Neural 3D Mesh Renderer[END_REF][START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF] introduce mesh renderers, and [START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF] adapts implicit representations. Works like [START_REF] Insafutdinov | Unsupervised Learning of Shape and Pose with Differentiable Point Clouds[END_REF][START_REF] Tulsiani | Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction[END_REF] then introduce techniques to remove the assumption of known poses. Except for [START_REF] Zhang | Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering[END_REF] which leverages GAN-generated images [START_REF] Goodfellow | Generative Adversarial Nets[END_REF][START_REF] Karras | A Style-Based Generator Architecture for Generative Adversarial Networks[END_REF], these works are typically limited to synthetic datasets.

A second group of methods aims at removing the need for 3D and multi-view supervision. This is very challenging and they hence typically focus on learning 3D from images of a single category. Early works [START_REF] Kar | Category-Specific Object Reconstruction from a Single Image[END_REF][START_REF] Tulsiani | Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency[END_REF][START_REF] Vicente | Reconstructing PASCAL VOC[END_REF] estimate camera poses with keypoints and minimize the silhouette reprojection error. The ability to predict textures is first incorporated by CMR [START_REF] Kanazawa | Learning Category-Specific Mesh Reconstruction from Image Collections[END_REF] which, in addition to keypoints and silhouettes, uses symmetry priors. Recent works [START_REF] Duggal | Topologically-Aware Deformation Fields for Single-View 3D Reconstruction[END_REF][START_REF] Lin | SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images[END_REF] replace the mesh representation of CMR with implicit functions that do not require symmetry priors, yet the predicted texture quality is strongly deteriorated. [START_REF] Henderson | Leveraging 2D Data to Learn Textured 3D Mesh Generation[END_REF] improves upon CMR and develops a framework for images with camera annotations that does not rely on silhouettes. Two works managed to further avoid the need for camera estimates but at the cost of additional hypothesis: [START_REF] Henderson | Learning single-image 3D reconstruction by generative modelling of shape, pose and shading[END_REF] shows results with textureless synthetic objects, [START_REF] Wu | Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild[END_REF] models 2.5D objects like faces with limited background and viewpoint variation. Finally, recent works only require object silhouettes but they also make additional assumptions: [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF][START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF] use known template shapes, [START_REF] Li | Selfsupervised Single-view 3D Reconstruction via Semantic Consistency[END_REF] assumes access to an off-the-shelf system predicting part semantics, and [START_REF] Wu | De-rendering the World's Revolutionary Artefacts[END_REF] targets solids of revolution. Other related works [START_REF] Gadelha | 3D Shape Induction from 2D Views of Multiple Objects[END_REF][START_REF] Henzler | Escaping Plato's Cave: 3D Shape From Adversarial Rendering[END_REF][START_REF] Hu | Self-Supervised 3D Mesh Reconstruction From Single Images[END_REF][START_REF] Kato | Learning View Priors for Single-view 3D Reconstruction[END_REF][START_REF] Pavllo | Convolutional Generation of Textured 3D Meshes[END_REF][START_REF] Ye | Shelf-Supervised Mesh Prediction in the Wild[END_REF] leverage in addition generative adversarial techniques to improve the learning.

In this work, we do not use camera estimates, keypoints, silhouettes, nor strong dataset-specific assumptions, and demonstrate results for both diverse shapes and real images. To the best of our knowledge, we present the first generic SVR system learned from raw image collections.

Mesh-based differentiable rendering. We represent 3D models as meshes with parametrized surfaces, as introduced in AtlasNet [START_REF] Groueix | AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation[END_REF] and advocated by [START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF]. We optimize the mesh geometry, texture and camera parameters associated to an image using differentiable rendering. Loper and Black [START_REF] Loper | OpenDR: An Approximate Differentiable Renderer[END_REF] introduce the first generic differentiable renderer by approximating derivatives with local filters, and [START_REF] Kato | Neural 3D Mesh Renderer[END_REF] proposes an alternative approximation more suitable to learning neural networks. Another set of methods instead approximates the rendering function to allow differentiability, including SoftRasterizer [START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF][START_REF] Ravi | Accelerating 3D Deep Learning with PyTorch3D[END_REF] and DIB-R [START_REF] Chen | Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer[END_REF]. We refer the reader to [START_REF] Kato | Differentiable Rendering: A Survey[END_REF] for a comprehensive study. We build upon SoftRasterizer [START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF], but modify the rendering function to learn without silhouette information.

Cross-instance consistency. Although all methods learned on categorical image collections implicitly leverage the consistency across instances, few recent works explicitly explore such a signal. Inspired by [START_REF] Kulkarni | Canonical Surface Mapping via Geometric Cycle Consistency[END_REF], the SVR system of [START_REF] Hu | Self-Supervised 3D Mesh Reconstruction From Single Images[END_REF] is learned by enforcing consistency between the interpolated 3D attributes of two instances and attributes predicted for the associated reconstruction. [START_REF] Yao | Discovering 3D Parts from Image Collections[END_REF] discovers 3D parts using the inconsistency of parts across instances. Closer to our approach, [START_REF] Navaneet | From Image Collections to Point Clouds with Self-supervised Shape and Pose Networks[END_REF] introduces a loss enforcing cross-silhouette consistency. Yet it differs from our work in two ways: (i) the loss operates on silhouettes, whereas our loss is adapted to image reconstruction by modeling background and separating two terms related to shape and texture, and (ii) the loss is used as a refinement on top of two cycle consistency losses for poses and 3D reconstructions, whereas we demonstrate results without additional self-supervised losses.

Curriculum learning. The idea of learning networks by "starting small" dates back to Elman [START_REF] Elman | Learning and development in neural networks: The importance of starting small[END_REF] where two curriculum learning schemes are studied: (i) increasing the difficulty of samples, and (ii) increasing the model complexity. We respectively coin them curriculum sampling and curriculum modeling for differentiation. Known to drastically improve the convergence speed [START_REF] Bengio | Curriculum learning[END_REF], curriculum sampling is widely adopted across various applications [START_REF] Bengio | Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks[END_REF][START_REF] Ilg | FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks[END_REF][START_REF] Schroff | FaceNet: A unified embedding for face recognition and clustering[END_REF]. On the contrary, curriculum modeling is typically less studied although crucial to various methods. For example, [START_REF] Wang | Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images[END_REF] performs SVR in a coarse-to-fine manner by increasing the number of mesh vertices, and [START_REF] Monnier | Deep Transformation-Invariant Clustering[END_REF] clusters images by aligning them with transformations that increase in complexity. We propose a new form of curriculum modeling dubbed progressive conditioning which enables us to avoid bad minima.

Approach

Our goal is to learn a neural network that reconstructs a textured 3D object from a single input image. We assume we have access to a raw collection of images depicting objects from the same category, without any further annotation. We propose to learn in an analysis-by-synthesis fashion by autoencoding images in a structured way (Fig. 3). We first introduce our structured autoencoder (Sec. 3.1). We then present how we learn models consistent across instances (Sec. 3.2). Finally, we discuss one more technical contribution necessary to our system: an alternate optimization strategy for joint 3D and pose estimation (Sec. 3.3).

Notations. We use bold lowercase for vectors (e.g., a), bold uppercase for images (e.g., A), double-struck uppercase for meshes (e.g., A), calligraphic uppercase for the main modules of our system (e.g., A), lowercase indexed with generic parameters θ for networks (e.g., a θ ), and write a 1:N the ordered set {a 1 , . . . , a n }.

Structured autoencoding

Overview. Our approach can be seen as a structured autoencoder: it takes an image as input, computes parameters with an encoder, and decodes them into explicit and interpretable factors that are composed to generate an image. We model images as the rendering of textured meshes on top of background images. For a given image I, our model thus predicts a shape, a texture, a pose and a background which are composed to get the reconstruction Î, as shown in Figure 3. More specifically, the image I is fed to convolutional encoder networks e θ which output parameters e θ (I) = {z sh , z tx , a, z bg } used for the decoding part. a is a 9D vector including the object pose, while the dimension of the latent codes z sh , z tx and z bg will vary during training (see Sec. 3.2). In the following, we describe the decoding modules using these parameters to build the final image by generating a shape, adding texture, positioning it and rendering it over a background. Shape deformation. We follow [START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF] and use the parametrization of AtlasNet [START_REF] Groueix | AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation[END_REF] where different shapes are represented as deformation fields applied to the unit sphere. We apply the deformation to an icosphere slightly stretched into an ellipsoid mesh E using a fixed anisotropic scaling. More specifically, given a 3D vertex x of the ellipsoid, our shape deformation module S z sh is defined by S z sh (x) = x + s θ (x, z sh ), where s θ is a Multi-Layer Perceptron taking as input the concatenation of a 3D point x and a shape code z sh . Applying this displacement to all the ellipsoid vertices enables us to generate a shaped mesh S = S z sh (E). We found that using an ellipsoid instead of a raw icosphere was very effective in encouraging the learning of objects aligned w.r.t. the canonical axes. Learning surface deformations is often preferred to vertex-wise displacements as it enables mapping surfaces, and thus meshes, at any resolution. For us, the mesh resolution is kept fixed and such a representation is a way to regularize the deformations.

Texturing. Following the idea of CMR [START_REF] Kanazawa | Learning Category-Specific Mesh Reconstruction from Image Collections[END_REF], we model textures as an image UVmapped onto the mesh through the reference ellipsoid. More specifically, given a texture code z tx , a convolutional network t θ is used to produce an image t θ (z tx ), which is UV-mapped onto the sphere using spherical coordinates to associate a 2D point to every vertex of the ellipsoid, and thus to each vertex of the shaped mesh. We write T ztx this module generating a textured mesh T = T ztx (S). Affine transformation. To render the textured mesh T, we define its position w.r.t. the camera. In addition, we found it beneficial to explicitly model an anisotropic scaling of the objects. Because predicting poses is difficult, we predict K poses candidates, defined by rotations r 1:K and translations t 1:K , and associated probabilities p 1:K . This involves learning challenges we tackle with a specific optimization procedure described in Sec. 3.3. At inference, we select the pose with highest probability. We combine the scaling and the most likely 6D pose in a single affine transformation module A a . More precisely, A a is parametrized by a = {s, r, t}, where s, r, t ∈ IR 3 respectively correspond to the three scales of an anisotropic scaling, the three Euler angles of a rotation and the three coordinates of a translation. A 3D point x on the mesh is then transformed by A a (x) = rot(r)diag(s)x + t where rot(r) is the rotation matrix associated to r and diag(s) is the diagonal matrix associated to s. Our module is applied to all points of the textured mesh T resulting in a posed mesh P = A a (T).

Rendering with background. The final step of our process is to render the mesh over a background image. The background image is generated from a background code z bg by a convolutional network b θ . A differentiable module B z bg renders the posed mesh P over this background image b θ (z bg ) resulting in a reconstructed image Î = B z bg (P). We perform rendering through soft rasterization of the mesh. Because we observed divergence results when learning geometry from raw photometric comparison with the standard SoftRasterizer [START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF][START_REF] Ravi | Accelerating 3D Deep Learning with PyTorch3D[END_REF], we propose two key changes: a layered aggregation of the projected faces and an alternative occupancy function. We provide details in our supplementary material.

Unsupervised learning with cross-instance consistency

We propose to learn our structured autoencoder without any supervision, by synthesizing 2D images and minimizing a reconstruction loss. Due to the unconstrained nature of the problem, such an approach typically yields degenerate solutions (Fig. 4a and Fig. 4b). While previous works leverage silhouettes and dataset-specific priors to mitigate this issue, we instead propose two unsupervised data-driven techniques, namely progressive conditioning (a training strategy) and neighbor reconstruction (a training loss). We thus optimize the shape, texture and background by minimizing for each image I reconstructed as Î:

L 3D = L rec (I, Î) + λ nbr L nbr + λ reg L reg , (1)
where λ nbr and λ reg are scalar hyperparameters, and L rec , L nbr and L reg are respectively the reconstruction, neighbor reconstruction, and regularization losses, described below. In all experiments, we use λ nbr = 1 and λ reg = 0.01. Note that we optimize pose prediction using a slightly different loss in an alternate optimization scheme described in Section 3.3.

Reconstruction and regularization losses. Our reconstruction loss has two terms, a pixel-wise squared L 2 loss L pix and a perceptual loss [START_REF] Zhang | The Unreasonable Effectiveness of Deep Features as a Perceptual Metric[END_REF] L perc defined as an L 2 loss on the relu3 3 layer of a pre-trained VGG16 [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF], similar to [START_REF] Wu | Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild[END_REF]. While pixel-wise losses are common for autoencoders, we found it crucial to add a perceptual loss to learn textures that are discriminative for the pose estimation.

Our full reconstruction loss can be written L rec (I, Î) = L pix (I, Î) + λ perc L perc (I, Î) and we use λ perc = 10 in all experiments. While our deformation-based surface parametrization naturally regularizes the shape, we sometimes observe bad minima where the surface has folds. Following prior works [START_REF] Chen | Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer[END_REF][START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF][START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF][START_REF] Zhang | NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild[END_REF], we thus add a small regularization term L reg = L norm + L lap consisting of a normal consistency loss [START_REF] Desbrun | Implicit fairing of irregular meshes using diffusion and curvature flow[END_REF] L norm and a Laplacian smoothing loss [START_REF] Nealen | Laplacian mesh optimization[END_REF] L lap .

Progressive conditioning. The goal of progressive conditioning is to encourage the model to share elements (e.g., shape, texture, background) across instances to prevent degenerate solutions. Inspired by the curriculum learning philosophy [START_REF] Elman | Learning and development in neural networks: The importance of starting small[END_REF][START_REF] Monnier | Deep Transformation-Invariant Clustering[END_REF][START_REF] Wang | Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images[END_REF], we propose to do so by gradually increasing the latent space representing the shape, texture and background. Intuitively, restricting the latent space implicitly encourages maximizing the information shared across instances. For example, a latent space of dimension 0 (i.e., no conditioning) amounts to learning a global representation that is the same for all instances, while a latent space of dimension 1 restricts all the generated shapes, textures or backgrounds to lie on a 1-dimensional manifold. Progressively increasing the size of the latent code during training can be interpreted as gradually specializing from category-level to instance-level knowledge. Figure 2a Neighbor reconstruction. The idea behind neighbor reconstruction is to explicitly enforce consistency between different instances. Our key assumption is that neighboring instances with similar shape or texture exist in the dataset. If such neighbors are correctly identified, switching their shape or texture in our generation model should give similar reconstruction results. For a given input image, we hence propose to use its shape or texture attribute in the image formation process of neighboring instances and apply our reconstruction loss on associated renderings. Intuitively, this process can be seen as mimicking a multi-view supervision without actually having access to multi-view images by finding neighboring instances in well-designed latent spaces. Figure 2b illustrates the procedure with an example. More specifically, let {z sh , z tx , a, z bg } be the parameters predicted by our encoder for a given input training image I, let Ω be a memory bank storing the images and parameters of the last M instances processed by the network. We write Ω (m) = {I (m) , z (m) sh , z (m) tx , a (m) , z (m) bg } each of these M instances and associated parameters. We first select the closest instance from the memory bank Ω in the texture (respectively shape) code space using the L 2 distance,

m t = argmin m ∥z tx -z (m)
tx ∥ 2 (respectively m s = argmin m ∥z sh -z (m) sh ∥ 2 ). We then swap the codes and generate the reconstruction Î(mt) tx (respectively Î(ms) sh ) using the parameters {z (mt) sh , z tx , a (mt) , z (mt) bg } (respectively {z sh , z (ms) tx , a (ms) , z (ms) bg }). Finally, we compute the reconstruction loss between the images I (mt) and Î(mt) tx (respectively I (ms) and Î(ms) sh ). Our full loss can thus be written:

L nbr = L rec (I (mt) , Î(mt) tx ) + L rec (I (ms) , Î(ms) sh ). ( 2
)
Note that we recompute the parameters of the selected instances with the current network state, to avoid uncontrolled effects of changes in the network state. Also note that, for computational reasons, we do not use this loss in the first stage where codes are almost the same for all instances.

To prevent latent codes from specializing by viewpoint, we split the viewpoints into V bins w.r.t. the rotation angle, uniformly sample a target bin for each input and look for the nearest instances only in the subset of instances within the target viewpoint range (see the supplementary for details). In all experiments, we use V = 5 and a memory bank of size M = 1024.

Alternate 3D and pose learning

Because predicting 6D poses is hard due to self-occlusions and local minima, we follow prior works [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF][START_REF] Henderson | Learning single-image 3D reconstruction by generative modelling of shape, pose and shading[END_REF][START_REF] Insafutdinov | Unsupervised Learning of Shape and Pose with Differentiable Point Clouds[END_REF][START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF] and predict multiple pose candidates and their likelihood. However, we identify failure modes in their optimization framework (detailed in our supplementary material) and instead propose a new optimization that alternates between 3D and pose learning. More specifically, given an input image I, we predict K pose candidates {(r 1 , t 1 ), . . . , (r K , t K )}, and their associated probabilities p 1:K . We render the model from the different poses, yielding K reconstructions Î1:K . We then alternate the learning between two steps: (i) the 3D-step where shape, texture and background branches of the network are updated by minimizing L 3D using the pose associated to the highest probability, and (ii) the P-step where the branches of the network predicting candidate poses and their associated probabilities are updated by minimizing:

L P = k p k L rec (I, Îk ) + λ uni L uni , (3)
where L rec is the reconstruction loss described in Sec. 3.2, L uni is a regularization loss on the predicted poses and λ uni is a scalar hyperparameter. More precisely, we use L uni = k |p k -1 /K| where pk is the averaged probabilities for candidate k in a particular training batch. Similar to [START_REF] Henderson | Learning single-image 3D reconstruction by generative modelling of shape, pose and shading[END_REF], we found it crucial to introduce this regularization term to encourage the use of all pose candidates. In particular, this prevents a collapse mode where only few pose candidates are used. Note that we do not use the neighbor reconstruction loss nor the mesh regularization loss which are not relevant for viewpoints. In all experiments, we use λ uni = 0.02. Inspired by the camera multiplex of [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF], we parametrize rotations with the classical Euler angles (azimuth, elevation and roll) and rotation candidates correspond to offset angles w.r.t. reference viewpoints. Since in practice elevation has limited variations, our reference viewpoints are uniformly sampled along the azimuth dimension. Note that compared to [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF], we do not directly optimize a set of pose candidates per training image, but instead learn a set of K predictors for the entire dataset. We use K = 6 in all experiments.

Experiments

We validate our approach in two standard setups. It is first quantitatively evaluated on ShapeNet where state-of-the-art methods use multiple views as supervision. Then, we compare it on standard real-image benchmarks and demonstrate its applicability to more complex datasets. Finally, we present an ablation study.

Evaluation on the ShapeNet benchmark

We compare our approach to state-of-the-art methods using multi-views, viewpoints and silhouettes as supervision. Our method is instead learned without supervision. For all compared methods, one model is trained per class. We adhere to community standards [START_REF] Kato | Neural 3D Mesh Renderer[END_REF][START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF][START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF] and use the renderings and splits from [START_REF] Kato | Neural 3D Mesh Renderer[END_REF] of the ShapeNet dataset [START_REF] Chang | ShapeNet: An Information-Rich 3D Model Repository[END_REF]. It corresponds to a subset of 13 classes of 3D objects, each object being rendered into a 64 × 64 image from 24 viewpoints uniformly spaced along the azimuth dimension. We evaluate all methods using the standard Chamfer-L 1 distance [START_REF] Mescheder | Occupancy Networks: Learning 3D Reconstruction in Function Space[END_REF][START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF], where predicted shapes are pre-aligned using our gradient-based version of the Iterative Closest Point (ICP) [START_REF] Besl | A method for registration of 3-D shapes[END_REF] with anisotropic scaling. Indeed, compared to competing methods having access to the groundtruth viewpoint during training, we need to predict it for each input image in addition to the 3D shape. This yields to both shape/pose ambiguities (e.g., a small nearby object or a bigger one far from the camera) and small misalignment errors that dramatically degrade the performances. We provide evaluation details as well as results without ICP in our supplementary.

We report quantitative results and compare to the state of the art in Table 2, where methods using multi-views are visually separated. We evaluate the pretrained weights for SDF-SRN [START_REF] Lin | SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images[END_REF] and train the models from scratch using the official implementation for DVR [START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF]. We tried evaluating SMR [START_REF] Hu | Self-Supervised 3D Mesh Reconstruction From Single Images[END_REF] but could not reproduce the results. We do not compare to TARS [START_REF] Duggal | Topologically-Aware Deformation Fields for Single-View 3D Reconstruction[END_REF] which is based on SDF-SRN and share the same performances. Our approach achieves results that are on average better than the state-of-the-art methods supervised with silhouette We compare to DVR [START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF] and SoftRas [START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF] learned with full supervision (MV, CK, S).

and viewpoint annotations. This is a strong result: while silhouettes are trivial in this benchmark, learning without viewpoint annotations is extremely challenging as it involves solving the pose estimation and shape reconstruction problems simultaneously. For some categories, our performances are even better than DVR supervised with multiple views. This shows that our system learned on raw images generates 3D reconstructions comparable to the ones obtained from methods using geometry cues like silhouettes and multiple views. Note that for the lamp category, our method predicts degenerate 3D shapes; we hypothesize this is due to their rotation invariance which makes the viewpoint estimation ambiguous. We visualize and compare the quality of our 3D reconstructions in Figure 5. The first three examples correspond to examples advertised in DVR [START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF], the last two corresponds to examples we selected. Our method generates textured meshes of high-quality across all these categories. The geometry obtained is sharp and accurate, and the predicted texture mostly corresponds to the input.

Results on real images

Pascal3D+ Car and CUB benchmarks. We compare our approach to stateof-the-art SVR methods on real images, where multiple views are not available. All competing methods use silhouette supervision and output meshes that are symmetric. CMR [START_REF] Kanazawa | Learning Category-Specific Mesh Reconstruction from Image Collections[END_REF] additionally use keypoints, UCMR [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF] and IMR [START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF] starts learning from a given template shape; we do not use any of these and directly learn from raw images. We strictly follow the community standards [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF][START_REF] Kanazawa | Learning Category-Specific Mesh Reconstruction from Image Collections[END_REF][START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF] and use the train/test splits of Pascal3D+ Car [START_REF] Xiang | Beyond PASCAL: A benchmark for 3D object detection in the wild[END_REF] (5000/220 images) and CUB-200-2011 [START_REF] Welinder | Caltech-UCSD Birds 200[END_REF] (5964/2874 images). Images are square-cropped around the object using bounding box annotations and resized to 64 × 64. and CUB (bottom) and compare to CMR [START_REF] Kanazawa | Learning Category-Specific Mesh Reconstruction from Image Collections[END_REF], IMR [START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF], UCMR [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF].

A quantitative comparison is summarized in Table 3, where we report 3D IoU, Chamfer-L 1 (with ICP alignment), Mask IoU for Pascal3D+ Car, and Percentage of Correct Keypoints thresholded at α = 0.1 (PCK@0.1) [START_REF] Kulkarni | Canonical Surface Mapping via Geometric Cycle Consistency[END_REF], Mask IoU for CUB. Our approach yields competitive results across all metrics although it does not rely on any supervision used by other works. On Pascal3D+ Car, we achieve significantly better results than UCMR for Chamfer-L 1 and Mask IoU, which we argue are less biased metrics than the standard 3D IoU [START_REF] Kanazawa | Learning Category-Specific Mesh Reconstruction from Image Collections[END_REF][START_REF] Tulsiani | Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency[END_REF] computed on unaligned shapes (see supplementary). On CUB, our approach achieves reasonable results that are however slightly worse than the state of the art. We hypothesize this is linked to our pose regularization term encouraging the use of all viewpoints whereas these bird images clearly lack back views.

We qualitatively compare our approach to the state of the art in Figure 6. For each input, we show the mesh rendered from two viewpoints. For our car results, we additionally show meshes with synthetic textures to emphasize correspondences. Qualitatively, our approach yields results on par with prior works both in terms of geometric accuracy and overall realism. Although the textures obtained in [START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF] look more accurate, they are modeled as pixel flows, which has a clear limitation when synthesizing unseen texture parts. Note that we do not recover details like the bird legs which are missed by prior works due to coarse silhouette annotations. We hypothesize we also miss them because they are hardly consistent across instances, e.g., legs can be bent in multiple ways.

Real-word datasets. Motivated by 3D-aware image synthesis methods learned in-the-wild [START_REF] Nguyen-Phuoc | HoloGAN: Unsupervised learning of 3D representations from natural images[END_REF][START_REF] Niemeyer | GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields[END_REF], we investigate whether our approach can be applied to realworld datasets where silhouettes are not available and images are not methodically (a) CompCars [START_REF] Yang | A large-scale car dataset for fine-grained categorization and verification[END_REF] (b) LSUN Motorbike and Horse [START_REF] Yu | LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop[END_REF] Fig. 7: Real-world dataset results. From left to right, we show: input and output images, the predicted mask, a correspondence map and the mesh rendered from 3 viewpoints. Note that for LSUN Horse, the geometry quality is low and outlines our approach limitations (see text). Best viewed digitally.

cropped around the object. We adhere to standards from the 3D-aware image synthesis community [START_REF] Nguyen-Phuoc | HoloGAN: Unsupervised learning of 3D representations from natural images[END_REF][START_REF] Niemeyer | GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields[END_REF] and apply our approach to 64 × 64 images of CompCars [START_REF] Yang | A large-scale car dataset for fine-grained categorization and verification[END_REF]. In addition, we provide results for the more difficult scenario of LSUN images [START_REF] Yu | LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop[END_REF] for motorbikes and horses. Because many LSUN images are noise, we filter the datasets as follows: we manually select 16 reference images with different poses, find the nearest neighbors from the first 200k images in a pre-trained ResNet-18 [START_REF] He | Deep Residual Learning for Image Recognition[END_REF] feature space, and keep the top 2k for each reference image. We repeat the procedure with flipped reference images yielding 25k images.

Our results are shown in Figure 7. For each input image, we show from left to right: the output image, the predicted mask, a correspondence map, and the 3D reconstruction rendered from the predicted viewpoint and two other viewpoints. Although our approach is trained to synthesize images, these are all natural by-products. While the quality of our 3D car reconstructions is high, the reconstructions obtained for LSUN images lack some realism and accuracy (especially for horses), thus outlining limitations of our approach. However, our segmentation and correspondence maps emphasize our system ability to accurately localize the object and find correspondences, even when the geometry is coarse.

Limitations. Even if our approach is a strong step towards generic unsupervised SVR, we can outline three main limitations. First, the lack of different views in the data harms the results (e.g., most CUB birds have concave backs); this can be linked to our uniform pose regularization term which is not adequate in these cases. Second, complex textures are not predicted correctly (e.g., motorbikes in LSUN). Although we argue it could be improved by more advanced autoencoders, the neighbor reconstruction term may prevent unique textures to be generated. Finally, despite its applicability to multiple object categories and diverse datasets, our multi-stage progressive training is cumbersome and an automatic way of progressively specializing to instances is much more desirable.

Ablation study

We analyze the influence of our neighbor reconstruction loss L nbr and progressive conditioning (PC) by running experiments without each component. First, we provide quantitative results on ShapeNet in Table 4. When L nbr is removed, the results are worse for almost all categories, outlining that it is important to the predicted geometry accuracy. When PC is removed, results are comparable to the full model for airplane and car but much worse for chair and table. Indeed, they involve more complex shapes and our system falls into a bad minimum with degenerate solutions, a scenario that is avoided with PC.

Second, we perform a visual comparison on ShapeNet and CompCars examples in Figure 8. For each input, we show the mesh rendered from the predicted viewpoint and a different viewpoint. When L nbr is removed, we observe that the reconstruction seen from the predicted viewpoint is correct but it is either wrong for chairs and degraded for cars when seen from the other viewpoint. Indeed, the neighbor reconstruction explicitly enforces the unseen reconstructed parts to be consistent with other instances. When PC is removed, we observe degenerate reconstructions where the object seen from a different viewpoint is not realistic.

Conclusion

We presented UNICORN, an unsupervised SVR method which, in contrast to all prior works, learns from raw images only. We demonstrated it yields high-quality results for diverse shapes as well as challenging real-world image collections. This was enabled by two key contributions aiming at leveraging consistency across different instances: our progressive conditioning training strategy and neighbor reconstruction loss. We believe our work includes both an important step forward for unsupervised SVR and the introduction of a valuable conceptual insight.

Supplementary Material for Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency

In this supplementary document, we first describe our custom differentiable rendering function (Appendix A). Then, we present additional model insights (Appendix B) as well as quantitative evaluation details (Appendix C). Finally, we provide implementation details (Appendix D), including network architectures, design choices and training details.

A Custom differentiable rendering

Our output images correspond to the soft rasterization of a textured mesh on top of a background image. We observe divergence results when learning geometry from raw photometric comparison with the standard SoftRasterizer [START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF] and thus propose two key changes. In the following, given a mesh M and a background B, we describe our rendering function R producing the image Î = R(M, B). We first present SoftRasterizer formulation and its limitations, then introduce our modifications. In the following, we write pixel-wise multiplication with ⊙ and the division of image-sized tensors corresponds to pixel-wise division.

SoftRasterizer formulation. Given a 2D pixel location i, the influence of a face j is modeled by an occupancy function:

O SR (i, j) = sigmoid ν(i, j) σ , (4)
where σ is a temperature, ν(i, j) is the signed Euclidean distance between pixel i and projected face j. Let us call L -1 the maximum number of faces intersecting the ray associated to a pixel and sort, for each pixel, the intersecting faces by increasing depth. Image-sized maps for occupancy O ℓ , color C ℓ and depth D ℓ are built associating to each pixel the ℓ-th intersecting face attributes. Background is modeled as additional maps such that O L = 1, C L = B, and D L = d bg is a constant, far from the camera. The SoftRasterizer's aggregation function C SR merges them to render the final image Î: The formulation introduced in Equation ( 5) has one main limitation: gradients don't flow well through O 1:L obtained by soft rasterization, and thus vertex positions cannot be optimized by raw photometric comparison. The simple case of a single face on a black background gives:

C SR (O 1:L , C 1:L , D 1:L ) = L ℓ=1 O ℓ ⊙ exp(D ′ ℓ /γ) k O k ⊙ exp(D ′ k /γ) ⊙ C ℓ , (5)
Î = O 1 ⊙ e D ′ 1/γ O 1 ⊙ e D ′ 1/γ + e ϵ /γ ⊙ C 1 ≈ O 1 ⊙ e D ′ 1/γ O 1 ⊙ e D ′ 1/γ ⊙ C 1 = C 1 , (6)
for almost all O 1 , D ′ 1 . Indeed, considering x, η > 0, we have xe (ϵ+η) /γ ≫ e ϵ /γ iif x ≫ e -η /γ . Even in the best case where η = ϵ = 10 -3 (i.e., the object is close to d far ), this holds for all x ≫ e -10 ≈ 4 × 10 -5 ! We found that tuning γ was not sufficient to mitigate the issue, one would have to tune γ, d near , d far , d bg simultaneously to enable the optimization of the vertex positions.

Our layered formulation. Inspired by layered image models [START_REF] Jojic | Learning Flexible Sprites in Video Layers[END_REF][START_REF] Monnier | Unsupervised Layered Image Decomposition Into Object Prototypes[END_REF], we propose to model the rendering of a mesh as the layered composition of its projected face attributes. More specifically, given occupancy O 1:L and color C 1:L maps, we render an image Î through the classical recursive alpha compositing:

C(O 1:L , C 1:L ) = L ℓ=1 L k<ℓ (1 -O k ) ⊙ O ℓ ⊙ C ℓ . (7)
This formulation has a clear interpretation where color maps are overlaid on top of each other with a transparency corresponding to their occupancy map. Note that we choose to drop the explicit depth dependency as all 3D coordinates (including depth) of a vertex already receive gradients by 3D-to-2D projection. Our layered aggregation used together with the SoftRasterizer's occupancy function O SR results in face inner-borders that are visually unpleasant. We thus instead use the occupancy function introduced in [5] defined by:

O(i, j) = exp(min(0, ν(i, j) σ )). (8)
Compared to O SR , this function yields constant occupancy of 1 inside the faces.

In addition to its simplicity, our differential renderer has two main advantages compared to SoftRasterizer. First, gradients can directly flow through occupancies O 1:L and the vertex positions can be updated by photometric comparison. Second, our formulation involves only one hyperparameter (σ) instead of five, making it easier to use.

B Model insights

B.1 Progressive conditioning

Figure 9 shows the results obtained on CompCars [START_REF] Yang | A large-scale car dataset for fine-grained categorization and verification[END_REF] at the end of each stage of the training. Given an input image (leftmost column), we show for each training stage the predicted outputs. From left to right, they correspond to a side view of the shape, the texture image and the background image. We can observe that all shape, texture and background models gradually specialize to the instance represented in the input. In particular, this allows us to start with a weak background model to avoid degenerate solutions and to end up with a powerful background model to improve the reconstruction quality. Also note how all the texture images are aligned.

B.2 Neighbor reconstruction

When computing the neighbor reconstructions, we explicitly find neighbors that have a viewpoint different from the predicted viewpoint. More specifically, for a given input, we compute the angle between the predicted rotation matrix and all rotation matrices of the memory bank. Following standard conventions, such an angle lies in [0 • , 180 • ]. Then, we select a target angle range as follows: we split the range of angles [20 • , 180 • ] into a partition of V uniform and continuous bins, and we uniformly sample one of the V angle ranges. Finally, we look for neighbors in the subset of instances having an angle within the selected range.

In all experiments, we use V = 5. We use a total angle range of [20 • , 180 • ] instead of [0 • , 180 • ] to remove instances having a similar pose. Note that we first tried to find neighbors of different poses without further constraint (which amounts to using V = 1) but we found that learned latent codes were specialized by viewpoints, e.g., front / back view images corresponding to a shape mode with unrealistic side views, and side view images corresponding to a shape mode with unrealistic front / back views.

(a) Optimization in [START_REF] Insafutdinov | Unsupervised Learning of Shape and Pose with Differentiable Point Clouds[END_REF] (b) Optimization in [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF][START_REF] Henderson | Learning single-image 3D reconstruction by generative modelling of shape, pose and shading[END_REF][START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF] Fig. 10: Prior optimizations for joint 3D/pose learning.

B.3 Joint 3D and pose learning

We analyze prior works on joint 3D and pose learning, illustrated in Figure 10, and compare them with our proposed optimization scheme, illustrated in Figure 11.

Prior optimization schemes can be split in two groups: (i) learning through the minimal error reconstruction [START_REF] Insafutdinov | Unsupervised Learning of Shape and Pose with Differentiable Point Clouds[END_REF], and (ii) learning through an expected error [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF][START_REF] Henderson | Learning single-image 3D reconstruction by generative modelling of shape, pose and shading[END_REF][START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF].

In [START_REF] Insafutdinov | Unsupervised Learning of Shape and Pose with Differentiable Point Clouds[END_REF], all reconstructions associated to the different pose candidates are computed and both 3D and poses are updated using the reconstruction yielding the minimal error (see Figure 10a). We identified two major issues. First, because the other poses are not updated for a given input, we observed that a typical failure case corresponds to a collapse mode where only a single pose (or a small subset of poses) is used for all inputs. Indeed, there is no particular constraint that encourages the use of all pose candidates. Second, inference is not efficient as the object has to be rendered from all poses to find the correct object pose.

In [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF][START_REF] Henderson | Learning single-image 3D reconstruction by generative modelling of shape, pose and shading[END_REF][START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF], 3D and poses are updated using an expected reconstruction loss (see Figure 10b). While this allows to constrain the use of all pose candidates with a regularization on the predicted probabilities, we identified one major weakness common to these frameworks. Because the 3D receives gradients from all views, we observed a typical failure case where the 3D tries to fit the target input from all pose candidates yielding inaccurate texture and geometry. We argue such behaviour was not observed in previous works as they typically use a symmetry prior which prevents it from happening. Note that CMR [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF] proposes to directly optimize for each training image a set of parameters corresponding to the pose candidates. This procedure not only involves memory issues as the number of parameters scales linearly with the number of training images, but also inference problems for new images. To mitigate the issue, they propose to use the learned poses as ground-truth to train an additional network that performs pose estimation given a new image.

In contrast, our proposed alternate optimization, illustrated in Figure 11, leverages the best of both worlds: (i) 3D receives gradients from the most likely reconstruction, and (ii) all poses are updated using an expected loss. In practice, we alternate the optimization every new batch of inputs, and we define one iteration as either a a 3D-step or a P-step.

C Quantitative evaluation

C.1 ICP alignment for better 3D evaluation

In the main paper, we align shapes using our gradient-based version of the Iterative Closest Point (ICP) [START_REF] Besl | A method for registration of 3-D shapes[END_REF] with anisotropic scaling before evaluating 3D reconstructions. For consistency, we use the same protocol across benchmarks and advocate to do so for future comparisons. First, meshes are centered and normalized so that they exactly fit inside the cube of unit length [-0.5, 0.5] 3 ; this is important to obtain results that are comparable. Second, we sample 100k points on the mesh surfaces. Third, we run our ICP implementation which minimizes by gradient descent the Chamfer-L 2 distance between the point clouds by jointly optimizing 3 translation parameters, 6 rotation parameters [START_REF] Zhou | On the Continuity of Rotation Representations in Neural Networks[END_REF] and 3 scaling parameters. In practice, we use Adam optimizer [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF], a learning rate of 0.01 and 100 iterations. Note that we use this gradient-based version of ICP instead of the classical iterative formulation as we found it to diverge when optimizing scale. We argue that an ICP pre-processing is crucial for an unbiased 3D reconstruction evaluation and provide real examples in Figure 12 to support our claim. Rows correspond to different transformations of the same canonical shape, and for each row, we show: the transformation used, the resulting 3D shape, a rendering example as well as Chamfer-L 1 distance to the canonical shape. We overlay the visuals with green contours representing the canonical shape and the canonical rendering for easier comparisons. We can make two important observations. First, Fig. 12: 3D reconstruction evaluation with and without ICP. Rows correspond to results obtained for transformed versions of a canonical shape and columns correspond to, from left to right, the transformation used, resulting 3D shape, a rendering example and Chamfer-L1 distance to the canonical shape. Green contours represent the shape and rendering from the canonical object for visual comparisons. although all the transformed shapes are excellent 3D reconstructions of the canonical shape, they result in dramatically poor performances. As a comparison, these performances are similar to our ShapeNet results with ICP when the model outputs degenerate reconstructions. Pre-processing the shapes using an ICP with anisotropic scaling mitigates this issue. Second, as shown by the rendering examples, for all these different shapes, we can find a pose that yields almost identical renderings. This hence emphasizes the numerous shape/pose ambiguities that arise from a given rendered image. As a result, it is extremely unlikely that a fully unsupervised SVR system predicting from a single image both the 3D shape and the pose will recover the exact pair of shape/pose used for annotations. In this case, the cameras used for rendering are the same and we do not even consider focal variations, which raises even more ambiguities.

C.2 ShapeNet results without ICP

For completeness, we provide quantitative results obtained without ICP on the ShapeNet benchmark in Table 5. We indicate the supervision used and visually separate methods using multi-view supervision. In addition to methods compared in the main paper, we report (i) results from category-agnostic versions (Cat. agn) of DVR [START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF] and SoftRas [START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF] presented in [START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF] and (ii) divergence results obtained by removing silhouette supervision from DVR and SoftRas.

D Implementation details

D.1 Modeling

Network architecture. We use the same neural network architecture for all experiments. The encoder is composed of 4 CNN backbones followed by separate Multi-Layer Perceptron (MLP) heads predicting a rendering parameter. More specifically, the 4 backbones are respectively used to predict: (i) shape code z sh and scale s; (ii) texture code z tx ; (iii) background code z bg ; (iv) rotations r 1:K , translations t 1:K , and pose probabilities p 1:K . Note that using a shared backbone instead of separated ones also yields great results and is advocated for decreasing the memory footprint and training time; the major benefit from using separated backbones is to produce slightly more detailed textures and background. We follow prior works in SVR [START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF][START_REF] Groueix | AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation[END_REF][START_REF] Mescheder | Occupancy Networks: Learning 3D Reconstruction in Function Space[END_REF][START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF] and use randomly initialized ResNet-18 [START_REF] He | Deep Residual Learning for Image Recognition[END_REF] as backbone. Each MLP head has the same architecture with 3 hidden layers of 128 units and ReLU activations. The last layer of the MLP heads for shape, texture and background codes is initialized to zero to avoid discontinuity when increasing the size of the latent codes. The final activation of the MLP heads for scale, rotation, and translation is a tanh function and the output is scaled and shifted using predefined constants in order to control their range (see Table 6 for selected ranges). The learnable parts of the decoder are the shape deformation network s θ and the two CNN generators t θ and b θ which respectively output 64 × 64 images for texture and background. The MLP modeling the deformations has 3 hidden layers, ReLU activations and 128 units for real images; we use an increased number of units for ShapeNet (512) which provides a small boost in performances. The CNN generators share the same architecture which is identical to the generator used in GIRAFFE [START_REF] Niemeyer | GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields[END_REF]. We refer the reader to [START_REF] Niemeyer | GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields[END_REF] for details. [START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF][START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF] on ShapeNet [START_REF] Chang | ShapeNet: An Information-Rich 3D Model Repository[END_REF], we keep the default rendering values used to generate the images for the focal length f , the distance to the camera tz and the elevation re. For real images, we keep the classical value of 2.732 for the distance to the camera tz and use a field of view (fov) of 30 • . Note that we did not finetune these parameters, they were selected once through visual comparisons on a toy example.

Other design choices. In all experiments, the predefined anisotropic scaling used to deform the icosphere into an ellipsoid is [1, 0.7, 0.7]. In Table 6, we detail other design choices that are specific to all categories of ShapeNet [START_REF] Chang | ShapeNet: An Information-Rich 3D Model Repository[END_REF] (second column) or all real-image datasets (third column). This notably includes a predetermined global scaling of the ellipsoid, a camera defined by a focal length f or a field of view (fov), as well as scaling, translation and rotation ranges.

D.2 Training

In all experiments, we use a batch size of 32 images of size 64 × 64 and Adam optimizer [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF] with a constant learning rate of 10 -4 that is divided by 5 at the very end of the training for a few epochs. The training corresponds to 4 stages where latent code dimensions are increased at the beginning of each stage and the network is then trained until convergence. We use dimensions 0/2/8/64 for the shape code, 2/8/64/512 for the texture code, and 4/8/64/256 for the background code if any. In line with the curriculum modeling of [START_REF] Monnier | Deep Transformation-Invariant Clustering[END_REF], we found it beneficial for the first stage to gradually increase the model complexity: we first learn to position the fixed ellipsoid in the image, then we allow the ellipsoid to be deformed, and finally we allow scale variabilities. In particular, we found this procedure prevents the model to learn prototypical shapes with unrealistic proportions. In the following, we describe other training details specific to ShapeNet [START_REF] Chang | ShapeNet: An Information-Rich 3D Model Repository[END_REF] benchmark and real-image datasets.

ShapeNet dataset. We use the same training strategy for all categories. We train the first stage for 50k iterations, and each of the other stage for 250k iterations, where one iteration corresponds to either a 3D-step or a P-step of our alternate optimization. We do not learn a background model as all images are rendered on top of a white background. However, we found that our system learned in such synthetic setting was prone to a bad local minimum where the predicted textures have white regions that accommodate for wrong shape prediction. Intuitively, this is expected as the system has no particular signal to distinguish white background regions from white object parts. To mitigate the issue, we constrain our texture model as follows: (i) during the first stage, the predicted texture image is averaged to yield a constant texture, and (ii) during the other stages, we occasionally use averaged textures instead of the real ones. More specifically, we sample a Bernoulli variable with probability p = 0.2 at each iteration and average the predicted texture image in case of success. We found this simple procedure to work well to resolve such shape/texture ambiguity.

Real-image datasets. We use the same training strategy for all real-image datasets. We train each stage for roughly 750k iterations, where one iteration either corresponds to a 3D-step or a P-step of our alternate optimization. Learning our structured autoencoder in such real-image scenario, without silhouette nor symmetry constraints, is very challenging. We found our system sometimes falls into a bad local minimum where the texture model is specialized by viewpoints, e.g., dark cars always correspond to a frontal view and light cars always correspond to a back view. To alleviate the issue, we encourage uniform textures by occasionally using averaged textures instead of the real ones during rendering, as done on the ShapeNet benchmark. More specifically, we sample a Bernoulli variable with probability p = 0.2 at each iteration and average the predicted texture image in case of success. We observed that it was very effective in practice, and we also found it helped preventing the object texture from modeling background regions. We do not use such technique in the last stage to increase the texture accuracy.

Fig. 1 :

 1 Fig. 1: Single-View Reconstruction by Cross-Instance Consistency. (left)Given a collection of single-view images from an object category, we learn without additional supervision an autoencoder that explicitly generates shape, texture, pose and background. (right) At inference time, our approach reconstructs highquality textured meshes from raw single-view images.

Fig. 2 :

 2 Fig. 2: Leveraging cross-instance consistency. (a) Progressive conditioning amounts to gradually increasing, in a multi-stage fashion, the size of the conditioning latent spaces, here associated to shape z sh and texture ztx. (b) We explicitly share the shape and texture models across neighboring instances by swapping their characteristics and applying a loss to associated neighbor reconstructions.

Fig. 3 :

 3 Fig. 3: Structured autoencoding. Given an input, we predict parameters that are decoded into 4 factors (shape, texture, pose, background) and composed to generate the output. Progressive conditioning is represented with .

Fig. 4 :

 4 Fig. 4: Degenerate solutions. An SVR system learned by raw image autoencoding is prone to degenerate solutions through (a) the background or (b) the 3D object model. We alleviate the issue with cross-instance consistency.

Fig. 5 :

 5 Fig. 5: Visual comparisons.We compare to DVR[START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF] and SoftRas[START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF] learned with full supervision (MV, CK, S).

Fig. 6 :

 6 Fig.6: Real-image comparisons. We show reconstructions on Pascal3D+ Cars (top) and CUB (bottom) and compare to CMR[START_REF] Kanazawa | Learning Category-Specific Mesh Reconstruction from Image Collections[END_REF], IMR[START_REF] Tulsiani | Implicit Mesh Reconstruction from Unannotated Image Collections[END_REF], UCMR[START_REF] Goel | Shape and Viewpoint without Keypoints[END_REF].

Fig. 8 :

 8 Fig. 8: Ablation visual results. For each input, we show the mesh rendered from two viewpoints.

  where γ is a temperature parameter, D ′ ℓ = d far -D ℓ d far -dnear and d near , d far correspond to near/far cut-off distances. This formulation hence relies on 5 hyperparameters (σ, γ, d near , d far , d bg ) and default values are σ = γ = 10 -4 , d near = 1, d far = 100 and d far -d bg d far -dnear = ϵ = 10 -3 .

Fig. 9 :

 9 Fig. 9: Progressive conditioning on CompCars [61]. Given an input image (leftmost column), we show for each training stage, from left to right, a side view of the predicted shape, the texture image and the background image.

Fig. 11 :

 11 Fig. 11: Our alternate 3D / pose optimization. Compared to prior works, we propose an optimization that alternates between 2 steps. (a) We update the 3D using the most likely pose candidate (3D-step). (b) We update the pose candidates and associated probabilities using the expected loss (P-step).

Table 1 :

 1 Comparison

with selected works. For each method, we outline the supervision and priors used (3D, Multi-Views, Camera or Keypoints, Silhouettes, Assumptions like ♢ template shape, † symmetry, ‡ solid of revolution, ↔ semantic consistency, ⊠ no/limited background, ⋖ frontal view, ∅ no texture), which data it has been applied to and the model output (3D, Texture, Pose, Depth).

  illustrates the procedure with example results where we can observe the progressive specialization to particular instances: reactors gradually appear/disappear, textures get more accurate. Because common neural network implementations have fixed-size inputs, we implement progressive conditioning by masking, stage-by-stage, a decreasing number of values of the latent code. All our experiments share the same 4-stage training strategy where the latent code dimension is increased at the beginning of each stage and the network is then trained until convergence. We use dimensions 0/2/8/64 for the shape code, 2/8/64/512 for the texture code and 4/8/64/256 for the background code. We provide real-image results for each stage in our supplementary material.

Table 2 :

 2 Method Ours SDF-SRN[START_REF] Lin | SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images[END_REF] DVR[START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF] DVR[START_REF] Niemeyer | Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision[END_REF] ShapeNet

	MV				✓
	CK		✓	✓	✓
	S		✓	✓	✓
	airplane 0.110	0.128	0.114	0.111
	bench	0.159	-	0.255	0.176
	cabinet 0.137	-	0.254	0.158
	car	0.168	0.150	0.203	0.153
	chair	0.253	0.262	0.371	0.205
	display	0.220	-	0.257	0.163
	lamp	0.523	-	0.363	0.281
	phone	0.127	-	0.191	0.076
	rifle	0.097	-	0.130	0.083
	sofa	0.192	-	0.321	0.160
	speaker 0.224	-	0.312	0.215
	table	0.243	-	0.303	0.230
	vessel	0.155	-	0.180	0.151
	mean	0.201	-	0.250	0.166

comparison. We report Chamfer-L1 ↓ obtained after ICP, best results are highlighted. Supervisions are: Multi-Views, Camera or Keypoints, Silhouettes.

Table 3 :

 3 Real

-image quantitative comparisons. Supervision corresponds to Camera or Keypoints, Silhouettes, Assumptions (see Tab. 1 for details).

Table 4 :

 4 Ablation results on ShapeNet[START_REF] Chang | ShapeNet: An Information-Rich 3D Model Repository[END_REF].

Table 5 :

 5 ShapeNet comparison without ICP. We report Chamfer-L1 ↓, supervisions are: Multi-Views, Camera or Keypoints, Silhouettes. We separate methods using multiviews and best results are highlighted in each group.

Table 6 :

 6 • , 360 • ] [0 • , 360 • ] re (elevation) 30 • [-10 • , 30 • ] rr (roll) 0 • [-30 • , 30 • ] Design choices. Following standard practices

	Design type	ShapeNet Real-image
	ellipsoid scale	0.4	0.6
	camera	f = 3.732 fov = 30 •
	sx/sy/sz	1 ± 0.5	1 ± 0.3
	tx/ty	0 ± 0.5	0 ± 0.3
	tz (depth)	2.732	2.732 ± 0.3
	ra (azimuth) [0

Acknowledgements. We thank François Darmon for inspiring discussions; Robin Champenois, Romain Loiseau, Elliot Vincent for manuscript feedback; Michael Niemeyer, Shubham Goel for evaluation details. This work was supported in part by ANR project EnHerit ANR-17-CE23-0008, project Rapid Tabasco, gifts from Adobe and HPC resources from GENCI-IDRIS (2021-AD011011697R1).


Phuong T Do

Carlos Alonso-Ramos

Xavier Le Roux

Laurent Vivien

Isabelle Ledoux

Bernard Journet

Eric Cassan
email: eric.cassan@u-psud.fr

Tunable optoelectronic oscillator based on silicon micro-ring resonator

Keywords: Silicon photonic, function integration, silicon microring resonator, optoelectronic oscillator

INTRODUCTION

The generation of broadband and low noise microwave and millimetre wave signals is important for a plethora of applications, including among other, radar, wireless communications, optical signal processing, warfare systems, etc [START_REF] Li | An Optically Tunable Optoelectronic Oscillator[END_REF]. Among various types of microwave and millimetre wave generation approaches, the optoelectronic oscillator (OEO) is a particularly interesting solution due to its capability to provide spectrally pure and wideband tunable signals [START_REF] Li | An Optically Tunable Optoelectronic Oscillator[END_REF]. However, the tunability of the conventional OEO is limited by the tunability of the microwave filter inside the loop, especially for high operation frequency. Microwave photonic filters (MPF) are a promising alternative to overcome this limitation, allowing reconfigurable microwave signal generation in OEO with a wide tuning range [START_REF] Capmany | Microwave photonics combines two worlds[END_REF][START_REF] Zhang | Tunable Optoelectronic Oscillator Incorporating a Single Passband Microwave Photonic Filter[END_REF]. Moreover, the progress of integrated microwave photonics (IMWP) [START_REF] Perez | Integrated microwave photonics[END_REF] provides now a solid framework for the possible full integration of OEOs.

Here, we propose a new approach for the implementation of Si OEO that allows for wideband tunability in the microwave signal generation. The oscillation signal is created by the beating between the optical source and one of the sideband lobes generated by amplitude modulation. This sideband lobe is selected by a silicon add-drop ring resonator (RR) working as an optical bandpass filter. We experimentally show that by tuning the wavelength of the source, the microwave frequency generated by the OEO can be tuned within the range from 5.9 to 18.2 GHz, which is limited by the elements available in the experimental setup.

PRINCIPLE OF OPERATION

The basic schematic and principle of operation of the proposed OEO are depicted in Fig. 1. In the proposed tunable OEO, the optical signal coming from laser source is separated in two branches, one is sent directly to photo-detector (PD) while the other is sent to an intensity modulator (IM). Initially, the modulator signal gets just seeded by white noise existing inside the loop. This signal is then passing through an add-drop RR before being sent to the PD. If the laser wavelength and one of the resonances of RR have the same phase and the distance between them is in close proximity in working range of the loop, then a beating of frequency will be generated. This beating signal is converted as an RF frequency (f1) through PD (point (2)).

EXPERIMENTAL RESULTS.

For the proof-of-concept demonstration of the proposed OEO, we integrate only the micro-ring resonator in the silicon chip. The RR is an add-drop silicon ring resonator, implemented with 450-nm-wide strip waveguide, fabricated from SOI wafer with 220 nm thick Si thin film on top of a 3 µm buried oxide layer. The fabrication result shows in Fig. 1c. Fig. 1d illustrates for optical transmission of the device. We obtained optical transmission with free spectral range FSR at FSRλ = 640 pm, corresponding with FSRfre ≈ 77 GHz. Optical quality factor Qopt of the RR has been investigated by fitting the resonance peak through the Lorentzian function, by doing so, the Qopt at 8.1x10 4 has been obtained. Figure 2 shows the experimental setup used to characterize the OEO. A 90/10 optical splitter is used to separate the light source from tuneable laser. 90 % of the optical power is sent to the Erbium doped amplifier (EDFA) followed by the IM, RR and a second EDFA. Signal at the output of second EDFA is combined with the other 10 % of the optical source (see Fig. 2) using 50/50 optical combiner. In the experiment setup, the polarization controller (PC) has been used in order to proper control the polarization matching between the laser source and the signal at the output of the second EDFA. At the output of the optical combiner, one arm is connected with the optical spectrum analyser (OSA) in order to read the laser or resonance wavelength while the other is connected with the PD. The final setup includes RF amplifier, RF coupler and electrical spectrum analyser (ESA). Fig. 3a illustrates the electrical spectrum of the generated microwave signal at 5.9 GHz with resolution bandwidth of 2.2 kHz and span of 6 MHz. In order to quantify the tunability capability of the OEO, we sweep the wavelength of the laser and collect the microwave signal in the ESA. Fig. 3b shows the measured oscillation spectrum with different laser wavelength, with a tuning range of 12.3 GHz, between 5.9 GHz and 18.2 GHz. Fig. 3c illustrates the linear variation of the oscillation signal as a function of the laser wavelength.

Figure 1 .

 1 Figure 1. Basic schematic (a) and principle of operation (b) of the proposed tunable OEO, SEM of the fabricated ring resonator (c) and the optical transmission of the device (d) (coupling gap: 300nm, coupling length: 4.5 µm). IM: Intensity modulator, RR: Ring resonator, PD: Photo-detector.

Figure 2 .

 2 Figure 2. Experiment setup for tunable OEO. EDFA: Erbium doped amplifier, PC: Polarization controller, OSA: Optical spectrum analyser, G: RF amplifier and ESA: Electric spectrum analyser.

Figure 3 .

 3 Figure 3. Spectrum of signal (a) and oscillation signal change with different laser wavelength (b) oscillation frequency / laser wavelength.

CONCLUSION.

As a conclusion, a novel OEO, based on add-drop Si micro-ring resonator, providing wideband tunability in the frequency of the microwave signal generated. The oscillation signal is created via the beating between a laser source and a sideband lobe produced by the intensity modulation of the same laser. We show experimentally that by changing the wavelength spacing between the optical source and the resonance wavelength, a microwave signal is generated with a tuneable frequency range from 5.9 GHz to 18.2 GHz. These results are an important step towards the implementation of tuneable OEO based on silicon photonics.