Fatima Suleman
email: fatima.suleman@iscte.pr

IREDU Jean-Jacques Paul
email: jean-jacques.paul@u-bourgogne.fr

Diversity of human capital attributes and diversity of remunerations

Introduction

The purpose is to provide some empirical evidence for promoting new insights into the economics of education. Particular attention is paid to the concept of competence and its influence on employee reward. The paper aims at comparing the impact on fixed earnings and flexible pay of the traditional human capital theory variables (education and experience) on the one hand and of specifically identified and assessed competences, on the other hand.

The objective is to test if the HCV (years of schooling, years of labour market experience) and competences substitute or complement each other in the definition of earnings. If they are substitutes, competences may constitute a more explicit vision of what HCV concretely represent. If they are complements, one can assume that they refer to two different dimensions:

-HCV are used in an "anonymous" way to determine the average level of earnings corresponding to given levels of education and experience, in accordance with social rules;

competences indicate the effective use of different knowledge and skills acquired and are used when individual and contingent criteria are required. This paper is made up of four parts. First of all, basic considerations will be presented in order to relate our approach to other researches and to expose some limits of human capital theory (HCT). Section 2 describes how the data have been collected. In a third part, methodological considerations are discussed. The fourth part presents the main results of the empirical analysis.

Basic considerations on limits of the traditional human capital approach

Our approach is close to the one developed by [START_REF] Green | The Value of Skills[END_REF], by Allen and [START_REF] Allen Jim | Educational Mismatches versus Skill Mismatches : Effects on Wages, Job Satisfaction and Job Search[END_REF], by [START_REF] Paul | De quelques interrogations sur nos approches traditionnelles en économie de l'éducation, surgies de l'analyse du comportement des étudiants européens[END_REF] and by Heijke, Meng and Ramaerkers (2002) in trying to find the "value of skills" through hedonic earnings equations. The aforementioned research works represent new approaches of HCT, which propose the use of a checklist of competences to define the individual level of competence.

Our research offers two original features: a) the use of competences assessed by direct supervisors, this is to say an hetero evaluation, and not a self evaluation, as used by the abovementioned authors; b) the use of profit shares benefited by employees and in addition to the earnings, to assess the impact of human capital variables and competences.

Traditional human capital variables and competences

Arguing from the point of view of the emerging "economics of competences", one could enforce the claim that the HCV are not sufficient for defining the level of individual competence and its reward in the labour market [START_REF] Suleman | La production et valorisation des compétences sur le marché du travail. Des approches néoclassiques à l'économie des conventions[END_REF]. Following this line of reasoning, we can formulate the main restrictions of HCT for defining this level of competence.

Firstly, according to the HCT, years of schooling and experience are a proxy for individual competence. The theory has paid little attention to the specific knowledge and skills acquired. Usually, the HCT assumes the stock of human capital homogeneous for a

given amount of years of study, or a given degree. New approaches of HCT suggest that the portfolio of competences could differ for a same level of traditional human capital variable.

A limit left by HCT is the confusion between the process and the product. That is to say, when the number of years of schooling is considered, there is confusion between the source of acquisition of the competence and the competence itself. In addition to this, the HCT does not specify the kinds of knowledge and skills acquired through the investments in schooling and experience.

Thus it is assumed that the individual competence refers to qualification or the resources acquired through HC investments. However, the importance of the contextual use of knowledge and skills play an important role. The job matching theory explicitly takes into account this effective use of individual knowledge and skills.

According to [START_REF] Heijke | The Knowledge and Skills of Economics Graduates and Their Significance on the Labour Market, Maastrich[END_REF], the job matching theory differs from HCT because it does not presuppose that individual knowledge and skills are productive in all available jobs. The main premise of the job matching theory is that jobs and individuals are both heterogeneous. Consequently, if there are differences between jobs, individuals can have comparative advantages in accessing and performing particular activities [START_REF] Heijke | The Knowledge and Skills of Economics Graduates and Their Significance on the Labour Market, Maastrich[END_REF].

For the sake of clarity, this paper puts forward a definition of individual competence, which takes into account the following: the qualification acquired through investments in human capital; the effective use of the knowledge and skills; and the assessment of the knowledge and skills acquired and used [START_REF] Suleman | La production et valorisation des compétences sur le marché du travail. Des approches néoclassiques à l'économie des conventions[END_REF].

1.2. The diversity of human capital rewards HCT usually does not take into account the whole reward system. The theory studies how individuals are rewarded through fixed salary or wage. Nevertheless, someone as

Armstrong refers to fixed salary as the base pay and suggests that there may also be additional payments related to performance, competence, contribution, skills [START_REF] Armstrong | Employee Reward[END_REF]. These represent the contingent component of the pay. In his book "Employee

Reward", [START_REF] Armstrong | Employee Reward[END_REF] defines competence-related remuneration as a method of remunerating individuals according to their ability to perform: "competence-related pay does not confine itself to the acquisition of competence. It is about the effective use of competence to generate added value" (Armstrong, 1999: 294).

However, Armstrong maintains that pay is in fact "related" to competences rather than "based" upon them. Indeed, according to Armstrong, it would seem impossible to base remuneration directly upon competences, since the evaluation of competences remains extremely difficult. Other factors, such as those linked to the market, can also influence remuneration.

In accordance with this concept, we will introduce the human resource management concept of "remuneration" to refer to all cash payments and benefits received by employees.

The remuneration includes the fixed salary and any additional pay, as well the contingent or flexible pay, such as profit shares.

Our main objective is to underline the heterogeneity of human capital reward, which is, to some extent, the result of employer wage policies. The moral hazard problem employers face calls for some reward rules, which can contribute to leading the employees to cooperate. The question is how firms can configure their wage policies to guarantee the effective use of convenient competences, as well as to face institutional constraints.

The data

The data were supplied by an original survey of five large banking companies in Portugal. Six hundred clerks (not in a supervising position) were interviewed regarding their individual characteristics (age, gender, education, experience in the labour market, experience in the company). Their respective supervisors were asked to assess their competences using a list of thirty competences. The list of competences had been previously checked with the help of human resources managers of the main banking companies and some branch managers. In this paper, because of some missing values, only 443 employees will be considered, located in a total of 77 agencies.

There are four main reasons why the banking sector was chosen for the survey: Their average number of school years is 12.7, the number of years of total experience is 17.4 and the number of years of experience within the bank is 11.3.

A proportion of 61% of the population gets a flexible pay, which represents on average about one month pay. The records of the assessment by supervisors for each of the thirty competences were synthesised using a principal component analysis. Four main factors were produced, making it possible to define four groups of competences: cognitive competences, strategic competences, behaviour towards the organisation, general knowledge.

Profit sharing as a proportion of the total annual earnings

The four clusters of competences built on the principal components

Methodological considerations

The objective is to compare to what extent the structure of remuneration differs when either basic pay or profit sharing are considered.

Since only part of the workers receive a flexible salary, we need to test if they differ from the others and if any difference can affect the value of the regression coefficients. The earnings models have to take potential selectivity biases into account.

The second question arises from the hierarchical structure of the data, since workers are grouped under the responsibility of one supervisor, who has been asked to assess their skills and competences. Several problems can arise when such phenomenon and variables are studied: first, since the evaluation is a subjective appreciation of workers by their supervisor, it can be supposed there is some endogeneity. Two factors may cause such an endogeneity: the supervisor may anticipate the consequence of his/her judgement in sub-estimating or super-estimating the mark, if this mark has an impact on the employee's reward. In our survey, the evaluation of skills and competences has been conducted for the specific purpose of the research, without any link with the remuneration strategy. The other factor may be linked to the subjectivity itself of the supervisor. A given worker, with the same level of skill/competences, may be assessed differently by two different supervisors. The method used for building the four dimensions of skills/competences, using a long list of items, may limit this bias, if we consider the supervisors may assess differently the various items. Nevertheless, some supervisors may systematically have high marks, and others low marks. Introducing random effects through multilevel modelling may allow controlling for this endogeneity. An additional method is to consider not the raw values of the competences, but the values centered around the mean of the grades given by each evaluator, as well as the mean of the grades itself.

Let us note the following variables. HC, for human capital variables: number of schooling years, experience, square of experience and gender (for convenience, gender is considered together with the human capital variables) CV, for competence variables: cognitive competences, strategic competences, behaviour towards organization, general knowledge.

Let be FW and PS the fixed wage and the profit share of the individual i.

Actually, three families of models will be tested.

Simple OLS models will be tested in order to assess the proportion of variance of the two types of reward explained by the two families of variables Ln (FW) = f(HC) Ln (PS)= f(HC) Ln (FW) = f(HC, CV) Ln (PS) = f(HC, CV) Then a Heckman model will be considered to assess to what extent the restriction of the modelling to the workers who get a flexible pay can modify the results. For simplicity, only the human capital variables will be considered.

Ln (FW) = f(HC, λ) Ln (PS) = f(HC, λ)
Where λ is the inverse of the Mills ratio, estimated with the following selection equation: Proba (PS>0)= f(HC, type of contract) And finally a multilevel approach is used to take into account the multilevel structure of the data. Two different expressions are considered, one with the individual ratings and one with the ratings centred around the evaluator mean and the evaluator mean itself.

∑ ∑

= = + + = 4 1 4 1 0 m ij mj l ij lj j ij z x Y δ β β
with either the fixed part or the flexible part of the wages of the individual i assessed by the supervisor of the branch j, the HC variables and the competence variables

ij Y ij x ij z ] ) ( [ . . 4 1 4 1 0 j mj j m ij mj l ij lj j ij z z z x Y γ δ β β + - + + = ∑ ∑ = =
The inverse of Mills ratio will also be included in the multilevel models, together with the individuals variables.

Results

In order to get a first sight on the respective influence of the two groups of variables on the two types of remuneration, simple OLS regressions may be run. These regressions allow to undertake a variance analysis and to estimate the significance of the regression coefficient of the different variables taken into account. They consider the individuals who get a flexible income, on top of their fixed income (267 workers). ------------+----------------------------- -------------+------------------------------Adj R-squared = 0.4182 Total | 12.1877988 266 .045818793 Root MSE = .16327 ------------+--------------------------------------------------------------- ------------+----------------------------- ------------+----------------------------- -------------+--------------------------------------------------------------- ------------------------------------------------------------------------------Regression: logarithm (flexible income) = fn (number of school years, experience, experience square, gender/male=1) -------------+--------------------------------------------------------------- ---------------------------------------------------------------------------- ------------+----------------------------- -------------+----------------------------- -------------+---------------------------------------------------------------

Regression

------------------------------------------------------------------------------ Lg (fixed)| Coef. Std. Err. t P>|t| [95% Conf. Interval] -
Source | SS df MS Number of obs = 267 -------------+------------------------------ F( 4, 262) = 4.20 Model | 4.33416229 4 1.08354057 Prob > F = 0.0026 Residual | 67.6171437 262 .258080701 R-squared = 0.0602 -------------+------------------------------ Adj R-squared = 0.0459 Total | 71.951306 266 .270493632 Root MSE = .50802 ------------------------------------------------------------------------------ Lg (
----------------------------------------------------------------------------- Lg(flex) | Coef. Std. Err. t P>|t| [95% Conf. Interval]

  a) it is a sector in which the concept of competence finds widespread use in human resources management; b) following the restructuring process in the sector, there is a need for competences to carry out commercial functions; c) the organisational structure of companies based on branches with small teams and direct supervision by the branch manager; d) the geographical distribution of branches throughout the Portuguese territory.

-

  Regression: logarithm (flexible income) = fn (number of school years, experience, experience square, gender/male=1, cognitive, strategic, behaviour towards organization, general knowledge)

  wage and a flexible part corresponding to the profit sharing. It tried to take into account the complexity of such data, according to selectivity and multilevel issues.The results, whatever the structure of the model used, confirm that the traditional human capital variables explain better the traditional way of remunerating workers, whereas the competence variables explain better the performance based remuneration. Two different patterns of remuneration are related to such findings. Traditional wages are mainly decided through conventional rules, where education and experience are basic ingredients. On the other hand, when performance based remuneration is considered, the effective engagement of the worker, as assessed by his/her supervisor, becomes pre-eminent. The question which then arises is to know to what extent the second way of remunerating workers will expand or not. If yes, education systems will probably more directly questioned about the competences they developed rather than about the format of the sheepskins they provide.

fixed earnings) = fn (number of school years, experience, experience square, gender/male=1, cognitive, strategic, behaviour towards organization, general knowledge)

	Regression: logarithm (Source | SS df	MS		Number of obs =	267
	-
									-
	Sch.years|	.0244576	.0057371	4.26	0.000	.0131609		.0357542
	Exp	|	.0297862	.0041649	7.15	0.000	.0215854		.0379871
	Exp²	| -.0003294	.000085	-3.88	0.000	-.0004966	-.0001621
	Gender	|	.0189675	.0203765	0.93	0.353	-.021155		.05909
	Intercept|	11.5448	.103725	111.30	0.000	11.34056		11.74904

Two main conclusions can be drawn from these results: the human capital variables explain better, as expected, the fixed part of the earnings, whereas the competence variables explain better the flexible part of the earnings. This statement can be expressed in a symmetrical way: the human capital variables hardly explain the flexible part of the earnings, whereas the competences are weakly related to the fixed part of the earnings.

In the models with the fixed earnings, the adjusted R square with only the three human capital variables and gender, but increased only to 47% when the four competences are taken into account. When the models with the flexible part of earnings are considered, the human capital variables contribute only to 5% of the total variance, whereas the introduction of the competences increases the proportion of explained variance to 24% (a proportion multiplied by near 5 times).

These first simple results can be challenged since they don't take into account the characteristics of the data. Workers who get a flexible income on top of the fixed part represent a part of the total of workers: the question of a potential selectivity bias has to be considered. On the other hand, the workers are grouped into branches under the responsibility of a supervisor. That means that the residuals at a same level may not be independent and that the competence grades can depend partly on the subjectivity of the supervisors.

In order to test any selection effect on the coefficient, two Heckman selection models have been run, with the fixed earnings and the variable pay respectively. The main equations include the four previous human capital and gender variables, whereas the selection equations consider the three human capital variables and the type of contract (permanent versus short term). Prob > chi2 = 0.0000 -------------+----------------------------------------------------------------P -------------+--------------------------------------------------------------- ------------+--------------------------------------------------------------- ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- Prob > chi2 = 0.0004 -------------+----------------------------------------------------------------P ----------------------------------------------------------------------------- In the two models, there appears a clear selection effect, since the correlation between the errors of the two equations is significant as well as the lambdas. To get a permanent contract makes more eligible to the flexible pay regime. But this selection does not affect the significance of the coefficients, which remain highly significant in the fixed earnings equation and poorly significant in the flexible pay equation.

Heckman selection model

In the following models, the hierarchical structure of the data will be taken into account, using a multilevel approach (the GLLAMM procedure, under STATA, has been used). In order to take care of potential endogeneity issues, two measures of the competences have been considered. In a first stage, the initial grades of the competences have been introduced into the models. In the second one, the competences have been decomposed into two parts, a measure of each respective competence centred around the mean of the branch and the mean of the branch (i.e. the mean for each supervisor). Such a method allows assessing to what extent an influence of the branch (or of the supervisor) has to be considered. But the limit of such an approach relies on the impossibility of estimating the proportion of the variance explained by the variables of a same level. The Mills ratios have been introduced into the models.

Multilevel models with competence ratings log likelihood = 211.06339 ----------------------------------------------------------------------------- -------------+--------------------------------------------------------------- -----------------------------------------------------------------------------Variance at level 1 - -----------------------------------------------------------------------------.01741961 (.00127149) Variances and covariances of random effects - -----------------------------------------------------------------------------***level 2 (branch) var(1): .01197069 (.00221272) ------------------------------------------------------------------------------Variance at level 1 - -----------------------------------------------------------------------------.1064908 (.01036833)

Variances and covariances of random effects ------------------------------------------------------------------------------***level 2 (branch) var(1): .07784365 (.01487941)

With the two models, the level 2 (the branch) appears to be significant. In the fixed pay model, the level 2 represents 41% of the total variance, and 42% with the flexible pay model. But the main important result regards the significance of the coefficients which is not affected by the new structure of the models. Human capital variables remain more significant in the fixed pay model, whereas the competence variables remain more significant in the flexible pay model.

A different treatment of the competence variables (centred variables and supervisor means) does not affect neither the significance nor the value of the competence variables.

Multilevel models with competence ratings centred around the supervisor mean and with the supervisor mean ------------+--------------------------------------------------------------- ------------------------------------------------------------------------------Variance at level 1 ------------------------------------------------------------------------------.01727457 (.00125411)

Variances and covariances of random effects ------------------------------------------------------------------------------***level 2 (branch) var(1): .01347566 (.00322022) ------------+--------------------------------------------------------------- -----------------------------------------------------------------------------Variance at level 1 - -----------------------------------------------------------------------------.1052111 (.01058723)

Variances and covariances of random effects


Keywords: CCS Concepts, Computing methodologies → Computer graphics, Image-based rendering, Reconstruction

Street-level imagery is now abundant but does not have sufficient capture density to be usable for Image-Based Rendering (IBR) of facades. We present a method that exploits repetitive elements in facades -such as windows -to perform data augmentation, in turn improving camera calibration, reconstructed geometry and overall rendering quality for IBR. The main intuition behind our approach is that a few views of several instances of an element provide similar information to many views of a single instance of that element. We first select similar instances of an element from 3-4 views of a facade and transform them into a common coordinate system, creating a "platonic" element. We use this common space to refine the camera calibration of each view of each instance and to reconstruct a 3D mesh of the element with multi-view stereo, that we regularize to obtain a piecewise-planar mesh aligned with dominant image contours. Observing the same element under multiple views also allows us to identify reflective areas -such as glass panels -which we use at rendering time to generate plausible reflections using an environment map. Our detailed 3D mesh, augmented set of views, and reflection mask enable image-based rendering of much higher quality than results obtained using the input images directly.

Introduction

Recent Image-Based Rendering (IBR) algorithms [CDSHD13, OCDD15, HRDB16, PZ17] allow high-quality free-viewpoint nav-igation of cityscapes, but require relatively high capture density -typically a high-resolution image every meter or so. Citywide street level captures are now widely available (e.g., Google Streetview, Microsoft Bing StreetSide), but are captured much more sparsely. Such low densities hinder all components of IBR: camera calibration, geometric reconstruction, and image interpolation for rendering. In this paper we develop solutions for IBR from sparse street-level capture by leveraging the repetitive nature of facades. The key idea in our work is to extract an idealized or platonic object corresponding to a given repetitive element, and use its multiple instances present in the facade to perform data augmentation allowing us to perform high-quality free-viewpoint IBR from sparse input. In particular, we focus on repetitive windows, which exhibit rich geometric and photometric details.

Authors version

E 1 E 2 E 3 C 1 C 2 (a) Input scene (b) Platonic scene C 1,3 * C 1,2 * C 1,1 * C 2,3 * C 2,2 * C 2,1 * E *
Figure 2: A small number of views of similar -but physically distinct -elements E i can be seen as a larger number of views of a unique element E * if we align them into a common coordinate system.

We start with a small number of views of a given facade (3-4 street-level pictures), and approximate cameras calibrated using Structure-from-Motion (SfM). We then semi-automatically extract cropped instances of the repetitive windows, which we call subviews. Our idea is to consider that all these subviews represent the same platonic window, as illustrated in Fig. 2. We use 3D information in all subviews during Structure-from-Motion to provide finer camera calibration in the common space, while running multi-view stereo in this space gives us a dense, albeit noisy 3D mesh of the platonic window. We refine this mesh by decomposing it into planar polygonal regions aligned with image edges, which typically correspond to the window panels, frames and surrounding bricks.

While the multiple subviews provided by similar instances improve 3D reconstruction, they include color variations that are view-dependent (such as reflections on glass panels) and instance dependent (such as different color or shape of blinds), which produce severe popping artifacts if used directly in IBR. We extract view-dependent variations by analyzing each instance separately, effectively removing reflections from the pictures. Aggregating the resulting reflection layers over all instances gives us a unique reflection mask for the platonic element, which we use to composite environment reflections during rendering. We treat instance-dependent variations by re-projecting each subview of an instance into the subviews of all other instances, only mixing information between different instances in the presence of occlusions.

In summary, we introduce the idea of exploiting repetitions to augment visual data in the context of image-based rendering of facades, and show how this augmentation improves camera calibration, 3D reconstruction, and reflection segmentation. These ele-ments combined allow us to greatly improve visual quality for IBR of facades captured with a small number of pictures.

Related work

Our work targets Image-Based Rendering and the use of repetitions, which are both extensive research areas. We review here closely related work; more complete reviews can be found in surveys or books [LHOK * 10, SCK08].

Image-based rendering. Image-Based Rendering (IBR) algorithms synthesize novel views by interpolating between photographs of the same scene. The Lumigraph [START_REF] Gortler | The lumigraph[END_REF] and Lightfield [START_REF] Levoy | Light field rendering[END_REF] methods generate novel views by interpolating light rays in a 4D light volume, but require the scene to be captured at very high density.

When the input images are sparser, additional information such as a 3D mesh of the scene can improve image interpolation [BBM * 01]. However, the baseline of the input images still needs to be small enough to obtain good angular resolution, and avoid ghosting and other rendering artifacts. Wide baselines also hinder automatic 3D reconstruction of the mesh, which is particularly problematic along object silhouettes where foreground and background should not be mixed. Chaurasia et al. [CDSHD13] and Hedman et al. [HRDB16] address this latter challenge by respectively segmenting the image along strong contours or by aligning the 3D mesh to each input image independently. In contrast, we exploit repetitions present in facades to artificially augment the number of input views, which benefits both the 3D reconstruction and image interpolation steps of Image-Based Rendering.

While dense IBR representations like Lightfields naturally handle view-dependent effects, wide-baseline methods based on 3D meshes often produce strong popping in the presence of reflections. Proper handling of such effects requires a multi-layer representation where the reflected layer lies at a different depth than the reflective surface [SKG * 12, KLS * 13]. While we extract reflection layers from our input images, we cannot recover the depth of the reflected objects because they are rarely visible in multiple images due to our wide baseline. Instead, we combine the information provided by the reflection layers of all instances of a repetitive window to estimate a reflection mask for that window, which we use to render plausible reflections using image-based lighting.

Repetitions. The presence of repetitions in images has been exploited for numerous applications in Computer Graphics. In particular, several methods build on the idea that different instances of a repetitive element can be seen as multiple views of the same platonic element. For example, Xu et From the input images, we first segment and select repetitive windows, followed by camera calibration of these subviews by a modified SfM algorithm. In the second step, we use the calibrated subviews to reconstruct a platonic mesh of the window using multiview stereo, and subsequently refine it to obtain a piecewise planar geometry. In the third step we extract reflections from each subview and deduce a reflection mask of the window. We additionally generate images of all instances of the window from all available viewpoints. We finally re-insert the platonic geometry and images of each instance into the complete facade for improved image-based rendering.

ploit repetitions to segment window elements on a facade. Jiang et al. [JTC11] extract lattice structures from repetitive facades and improve an image-based modeling process. In contrast, we focus on the components required for high-quality image based rendering of facade elements -camera calibration, geometric reconstruction, and image interpolation -rather than their use within a procedural modeling context.

Overview

Figure 3 illustrates the main steps of our approach. Our input is a small set of images (3-4) of a building, taken at street level alongside the facade. The pictures are acquired with a consumer camera, using fixed exposure.

We first automatically detect windows in the facade using a deep classification network, and ask the user to select the windows that are visually similar. We crop the input images around each such instance to obtain a set of images that all represent the same platonic window, under different viewpoints. We will refer to these crops as subviews.

Due to the wide baseline of the input images, existing structurefrom-motion methods only produce an approximate calibration of the input cameras. We use this information to compute an initial guess of the camera of each subview in a common platonic space, which we update by running a structure-from-motion algorithm on all subviews. The initial guess allows us to reject erroneous correspondences that may occur when a subview not only shows its central window, but also part of its neighboring windows. This filtering process allows us to improve "platonic" camera calibration for each of the subviews.

After platonic camera calibration, we run a multiview-stereo algorithm to reconstruct a 3D mesh of the platonic element. All subviews represent a similar window, but they often contain instancespecific details that disturb generic reconstruction algorithms, resulting in noisy surfaces. We regularize this geometry to obtain a are present in different states in the various instances, they can be sorted and animated. We follow a similar strategy as the above approaches by turning spatial repetition into angular information, although we target the different application domain of image-based rendering.

Our approach is also inspired by 3D reconstruction methods for urban scenes. In particular, we build on the work by Wu Detecting repetitions can also be a step towards inference of procedural rules from an existing building. Some methods extract repetitions an fassed variations from a single fronto-parallel image of a facade, as a set of tiles and rules [MRM * 10, DRSVG13]. New facade images can then be generated by following these procedural rules and mixing the different tiles. Starting from a single planar facade image, Muller et al. [MZWVG07] factorize irreducible architecural elements. These tiles are matched to a bank of 3D models (windows, doors, etc) that are inserted in the facade plane and textured. The goal of our method is similar, but does not rely on external 3D models. Given a set of input images and 3D model of a building, Aliaga et al. [ARB07] propose a user-assisted method to construct a procedural model of the building. The procedural grammar enables semantic edits, such as adding new floors, while the input images enable view-dependent texture mapping of each facade element. Jiang et al. [START_REF] Jiang | Symmetric architecture modeling with a single image[END_REF] attain similar results using a unique input image and user-drawn strokes, estimating the camera pose using the scene symmetries. Zhao et al. [ZYZQ12] also ex-piecewise planar mesh composed of polygonal regions aligned with the dominant lines of the window.

The 3D mesh we obtain allows us to re-project any subview into any other one. We use this feature to extract view-dependent variations between subviews of the same instance, and to generate images of each instance as seen from the cameras of all subviews. While the reflections extracted from a single instance may be sparse, combining the information provided by all instances allows us to estimate a reflection mask that indicates the reflective areas of the platonic window.

Given these preprocessing steps, we create a complete scene representation by inserting the 3D mesh of the platonic window at the location of each of its instances over the facade mesh. We composite each view of a given instance to be consistent with the corresponding view of the entire facade, generating an enriched set of subviews by combining views and instances via reprojection. We use the resulting mesh and images in an image-based rendering algorithm, which we enhance with image-based reflections using our reflection mask. An additional use of our method output data is the generation of a multi-view textured mesh.

Terminology. In what follows, the term input scene refers to the original 3-4 photos of the entire facade, along with calibrated cameras and a coarse 3D mesh computed using multiview stereo. The platonic scene refers to the scene containing the calibrated cameras and reconstructed geometry of the platonic element, computed from multiple subviews of that element. All elements of the platonic scene are denoted with a superscript * . We denote the initial images as V i , and the input physical instances of the repetitive element as E j . The cropped subviews V i, j use a double-index notation, indicating the input view i they were extracted from, and the physical instance j they contain. The associated cameras in the common platonic space are denoted C * i, j and the platonic model itself E * . We illustrate these quantities in Fig. 4.

Windows Extraction and Platonic Camera Calibration

The first step of our method consists in cropping repetitive windows in the facade to form a multi-view dataset of the corresponding platonic window. Calibrating the cameras of each crop within a common space then enables 3D reconstruction of the window with higher accuracy than using solely the input images.

Window extraction

We identify the repetitive windows in a semi-automatic manner, where we automatically detect candidate windows and let the user select the ones that should be considered similar (Fig. 5).

We generate the candidate windows by running the rectified images through a deep classification network. We use a U-Net architecture [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF], trained on the CMP Facade Database [RT13], containing 600 rectified images of facades with ground truth labels of architectural elements. We only predict two labels -windows and background. We obtain the rectified facade using the vanishing point method by Wu et al. [WFP10] We then process the classification to extract its connected components, which should correspond

(a) Input Scene (b) Platonic Scene V i C i E j V i,j C i,j E * * Figure 4
: From given input views V i of a facade and their associated cameras C i (a), we extract subviews V i, j of similar windows E j . From these, we can reconstruct a 3D model of the platonic element E * along with platonic cameras C * i, j (b) in a common space.

to individual windows. We compute the bounding box of each component and scale it by a factor of 1.75 to include its surrounding, and re-project these boxes into the input image to be shown to the user.

This user interaction is very easy, and only takes a few seconds per dataset (please see video). Automating this process should be possible but would require similarity metrics robust to differences between instances of the same window while being sensitive to differences between different windows, e.g., top parts of the first and second row of windows in Fig. 5. We leave the exploration of robust similarity detection, for instance based on deep features, to future work.

Platonic Camera Calibration

We now have a list of cropped subviews, each representing an instance of the platonic window seen from a different viewpoint. However, our automatic crops sometimes contain parts of neighboring windows on their sides, as shown in Fig. 6(a). These duplicates challenge structure-from-motion algorithms since points on the central window in one subview may be matched to points in the side window in another subview.

We filter out such erroneous matches by deriving an approximate camera for each subview, which we compute from the known position and size of the corresponding crop in the input image, as well as from the camera pose of that image. This computation also requires knowledge of the facade geometry in order to position the approximate cameras such that they all point to the same element We then expand these boxes to include the surroundings of each window and project them back into the image (f). We ask users to select the boxes corresponding to a group of similar windows (g).

in platonic space. We approximate this geometry as a plane, fit to the point cloud of the facade, computed by running multiviewstereo reconstruction on the input images (Fig. 7). In most cases, a RANSAC fit is sufficient; when this fails we ask the user to specify the plane by selecting three points on the point cloud. Given these cameras, we modify an SfM algorithm [MMMO] to reject matches for which a point, when reprojected into the other subview, lands further away than half the image dimension from its correspondence. This filtering operation greatly improves the quality of platonic camera calibration, as shown in Fig. 8.

Geometry Reconstruction

We first describe how to reconstruct a 3D mesh of the platonic window, before explaining how we repeat it over the entire facade. Figure 8: Camera calibration using standard approach (left), and the result with our approach (right). Note how in the standard approach, the rightmost camera does not see the window, and that three cameras are incorrectly co-located.

Platonic Window

The platonic cameras estimated by our modified SfM algorithm can now be used for dense 3D reconstruction of the platonic window. However, we found that generic multi-view stereo algorithms Our refinement method operates in the subview for which the view direction is the most orthogonal to the facade plane. We segment this subview into convex polygons aligned with image edges using the recent algorithm of Bauchet and Lafarge [START_REF] Lafarge | KIPPI: KInetic Polygonal Partitioning of Images[END_REF], as shown in Fig. 9. In a nutshell, this algorithm detects small line segments in the image and extends them until they intersect other segments to form closed regions. We then formulate an optimization that displaces each region along the facade normal. Our formulation combines a data term, which seeks to keep the region close to the noisy point cloud it covers, and a smoothness term, which encourages neighboring regions to align if they share similar colors. Denoting r a polygonal region, d r its displacement along the facade normal (dr = 0 at the facade plane), d re f the average distance to the facade plane of the reconstructed points covered by r, N (r) the two-ring neighborhood of r, and c r the average color of r, we want to minimize

E(d r) = |dr -d re f | + λ 1 W ∑ n∈N (r) e -cr-cn 2 |dr -dn| (1)
where W = ∑ n∈N(r) e -cr-cn 2 is a normalization factor and λ balances the two terms. We set λ = 15 in all our experiments.

Given the small number of regions, we solve this optimization problem using a simple mean field algorithm [ZC93], where we iteratively update the displacement of each region to minimize Eq. 1 given the displacement of its neighbors. At each iteration, we try to assign to a given region all displacements of its neighbors, as well as a regular sampling of the depth interval of the noisy reconstruction with respect to the facade plane. We initialize this optimization by setting d r = d re f . At the end of the optimization, we connect the displaced polygons by additional faces to handle depth discontinuities between neighbors. Figure 10 illustrates the result of this optimization on two 3D meshes, the first obtained by running multiview stereo on the input images, and the second obtained from all subviews. The best result is achieved with the latter option, where our refinement removes bumps and captures well the flat parts of the window and wall.

Complete Facade

Given the reconstructed geometry of the platonic window, we generate a mesh for the entire facade by replicating the platonic mesh on the facade plane at the positions of all window instances. However, since the facade plane and the platonic mesh have been generated by separate executions of the structure-from-motion algorithm, they are not in the same coordinate system and do not have the same scale. We deduce the 3D rigid transformations from the platonic scene back to the input scene by matching the platonic cameras with the cropped cameras previously estimated in Sec. 4.2, as illustrated in Fig. 11. For each match we compute a candidate transformation and make it independent from the instance position. We average all match transformations to obtain the final one. We provide the detailed computation of this transformation in Appendix A.

(a) Input Scene (b) Platonic Scene

E j P i,j P i,j E C i,j C i C i,j * * * T i,j
Figure 11: For each platonic camera, we estimate the transformation to align it back to the corresponding estimated camera in the input scene. We use this transformation to copy the platonic mesh at each window occurrence in the facade. While the above process results in an improved 3D mesh of the facade (see Fig. 12), it is not perfectly aligned with the input images due to the numerous geometric transformations involved and to the approximate calibration of the input cameras. Such misalignements produce significant ghosting if the original images are used along the refined mesh in an image-based rendering algorithm, as shown in Fig. 13. We next describe how to improve rendering quality by leveraging the more precisely calibrated subviews of the windows. We use these subview cameras, placed back into the input scene, along with the approximate input cameras for the final rendering. The subviews also provide complementary information gathered from different instances of the window, allowing us to better handle reflections than when using only the input images.

Data Factorization and Augmentation

While the cropped subviews all represent the same platonic element, they each contain view-dependent and instance-dependent variations which would yield significant popping artifacts if used directly for image-based rendering. We treat these two sources of variations separately, as view-dependent variations mainly correspond to reflections over the window panels, while instancedependent variations correspond to changes of shape or color of the window frame.

View-Dependent Variations

Each instance of a window is seen in several of our input images. While window frames are mostly diffuse, the panels are typically reflective and exhibit strong variations between these input views. However, the input camera baseline is often too wide to observe any overlap in reflections, which prevents them to be rendered with existing solutions based on 3D reconstruction of the reflected scene [KLS * 13]. Instead, we propose to remove these view-dependent variations to later replace them by plausible reflections from an environment map.

Reflection Separation. Given all the subviews V •, j of a given instance, we remove the reflections in each subview by reprojecting all other subviews onto it, and computing the median gradient of the resulting image stack. We use the 3D mesh computed in Sec. 5 to perform this reprojection. As observed by Weiss in the context of shadow removal [Wei01], the median gradient preserves the visual content shared by the aligned images, while it discards content that only appears in one of the images. Integrating the resulting gradient fields gives images where most reflections have been removed, as shown in Fig. 14. We refer to these images as diffuse layers, and to the removed information as view-dependent layers.

In practice, we only perform reflection separation for pixels identified as windows by the deep classification network (Sec. 4.1), since those are the most likely to be reflective.

We noticed that median filtering is also effective at removing occluders present in one of the input images (trees, sign posts). Assuming that these occluders have a very different color than the scene they occlude, we also run the above process on pixels that exhibit a high variance in hue over the images. Reflection Mask. We are now equipped with one view-dependent layer per subview, which gives us indications of which parts of the platonic window are reflective. We aggregate this information across subviews to generate a unique reflection mask, which we use at rendering time to composite the environment map reflections over the window. Since the extracted reflections often only cover part of the window panels, we again leverage the polygonal segmentation of the cropped subviews to obtain clean, regular masks.

We first reproject all view-dependent layers into the subview from which the polygonal segmentation has been computed (Sec. 5). We then count the number of pixels in each region for which the reprojected layers have non-zero intensities, and we compute the sum of all such pixel intensities. We estimate the average value and variance of these quantities over all regions. We consider that a region is in the reflection mask if both values differ by more than 0.25 standard deviation from their average over all regions. Intuitively, this approach selects regions that are well covered by intense view-dependent effects. The resulting reflection mask (shown in Fig. 14) is then reprojected to all subviews. The entire viewdependent variations extraction process is illustrated in Fig. 15.

E 2 E 1 C 1,1 * C 2,1 * C 3,1 * C 1,2 * C 2,2 * C 3,
Reflected Environment. The reflections present in the input images are too sparse to be used directly for rendering, but they provide some information about the environment surrounding the facade, see Fig. 16. In particular, we use the extracted reflections to construct an incomplete environment map, which we use to manually select a similar environment map from a lightprobe library [START_REF] Ehinger | Recognizing scene viewpoint using panoramic place representation[END_REF].

Figure 16: Input images (left) and the resulting environment map after reprojection of the extracted reflections (top right, zoom as inset). We use this incomplete environment map to select a similar one from a lightprobe dataset (bottom right).

Instance-Dependent Variations

The key idea behind our approach is to consider that a few views of different instances of a window can be seen as multiple views of a single platonic window. While we have shown the potential of this idea to generate a precise platonic mesh and reflection mask, using all subviews V i, j to render a single window yields severe popping of visual content specific to each instance. For instance, the window blinds may change height or color over different instances, producing distracting animations as we rotate around the window. Each instance thus only has a subset of coherent subviews, which correspond to the crops of that instance in the original images.

E 1 E 2 Masks ... ... V 1 1,1 V 1 3,1 V 1 2,1 V 1 1,2 V 1 3,2 V 1 2,2 V 2 1,1 V 2 3,1 V 2 2,1 V 2 1,2 V 2 3,2 V 2 2,2
... Each instance E i of a window is only seen in a subset of subviews (images with colored border). We augment this coherent subset by re-projecting the available subviews in other viewpoints, and fill-in occlusions and missing parts using the corrsponding subviews from another instance (colored areas in images with black border). We also re-project the reflection mask in all subviews.

Our solution consists in augmenting the coherent set of each instance by reprojecting the available subviews into their closest missing subviews. Since we only have a few available subviews per instance, parts of the window may be occluded in other subviews. We fill in these parts using content from another instance for which this particular subview is available. We introduce additional notation to indicate the target instance k: V k i, j . When the physical instance j is equal to the target instance k, all pixels are covered in the sub-view, see Fig. 17. When the target instance k = j some pixels need to be reprojected from other views to complete the cropped image.

Fig. 17 illustrates the resulting data augmentation on two window instances, each seen in three subviews, yielding six coherent subviews for each instance. Note that this overall process is similar in spirit to the re-projection performed at runtime by the imagebased rendering algorithm to generate novel virtual views. However, augmenting the subviews in an offline pre-process allows us ULR using input views ULR using instance specific subviews

ULR using reprojected subviews

Figure 18: IBR comparison on our full scene mesh, using resp. the three input views, each instance original subviews, and the reprojected subviews for each instance.

to employ a costly gradient-domain fusion [PGB03] when combining parts from different instances at occlusions. The color coding in Fig. 17 indicates the source of the pixels in each subview: colored V k i, j with outlined subiews are cases where target and and physical instance are the same, while those in black include reprojection. Note that the colored pixels in the reprojected views correspond to the source instance used; in the general case these can come from several instances. The effect of this augmentation is shown in Fig. 18.

Complete Facade

We now need to create a complete model of the facade, requiring us to make the window-scale subviews compatible with the facade-scale input images. The main challenge we face at this stage is that the facade-scale images are not perfectly aligned with the subviews and refined 3D mesh, which results in visible seams if the two sets of images are combined naively during image-based rendering (see Fig. 19(a)). For each subview, our solution is to project the closest input views into it, and stitch the two images such that the subview is kept over the window, while the input image is used over the surrounding walls. We apply Digital Photomontage [ADA * 04] to achieve a seamless composite, using the blurred bounding box of the subview as the foreground/background unary term (see Fig. 19(b)).

This approach also helps with color differences between the subviews and the facade due to exposure variations in the scene. As a result, subviews are seamlessly composited and we can place the instances at different locations on the facade, creating a novel facade (see Fig. 20).

Implementation and results

The final augmented dataset contains the matrix of augmented views, V k i, j , the corresponding cameras C * i, j and the augmented geometry. In addition, we have the selected environment map that will be used for IBL.

Rendering

We render the scene using an extension of the Unstructured Lumigraph Rendering (ULR) method [BBM * 01], that computes blend- ing weights between input images for each pixel of the desired output. Our expanded set of cameras C * and images V k i j augment the image data available for the regions in the windows. In the remaining areas of the facade plane, it is equivalent to a ULR render using solely the three input images. In the regions of the platonic elements, only the platonic cameras are used, to avoid any ghosting caused by the unprocessed input images.

To restore plausible view-dependent effects that are important to the perceived accuracy and realism of the scene, we overlay a reflection layer over the output image, using the environment map previously selected. For each pixel lying in one of the areas delimited by the reflection masks (Sec. 6.1), we reflect the view-to-point vector with respect to the facade plane normal and use the resulting direction to query the environment map. The resulting color is additively blended with the underlying diffuse color.

Implementation

We implemented our method in our custom C++ framework and render in an OpenGL pipeline. For the individual components we used the implementation of [WFP11] to rectify input images, and a Tensorflow implementation of the U-net for window segmentation. The entire preprocessing pipeline requires between 10 and 30 minutes for all our scenes; camera calibration and reconstruction requires less than 5 minutes both for input and platonic scenes. All timings are reported on an Intel Xeon 12 core 2.6GHz machine with 32Gb of memory. Rendering is real-time.

Results

We validate our results on five scenes, two with facades from London (3 input images each, from Ceylan et al. [START_REF] Ceylan | Coupled structure-from-motion and 3d symmetry detection for urban facades[END_REF]) and three from Paris (3 input images for Paris1 and 4 for the others). Scenes are processed as described in the previous sections. Scenes Paris2 and Paris3 required manual definition of the plane. In Fig. 21, for each scene we show the input augmented geometry and compare our result to the baseline result rendered with ULR rendering on the MVS reconstruction using the input images. The resulting improvement in quality is more clearly visible in the supplemental video, when changing the novel view and for the IBL rendering of reflections. Our contributions could also benefit rendering algorithms based on a diffuse textured mesh. In particular, the refined mesh yields better alignment of the re-projected images, which could benefit multi-view texturing. We present a comparison between textured meshes generated from our data and our image-based rendering approach in Fig. 22. Additional comparisons, illustrating the relative advantages of each part of our method can also be found in the accompanying videos. As shown in Fig. 20, a novel facade can be generated from an existing scene by swapping window instances.

Limitations and future work

Limitations. We currently rely on user intervention for subview selection. A clustering algorithm with a specific similarity metric that allows for minor differences between instances of a group could be used. Our algorithm is sensitive to the precision of the original calibration and reconstruction of the input images. In some cases, even though the MVS reconstruction fails completely we are able to produce a usable result, albeit with some artifacts (Fig. 23). Improved resolution street-side captures (e.g., [Eth17]) should provide sufficient quality that will allow our approach to be used. Eliminating the manual steps of subview selection and plane extraction (when needed) would allow the approach to be used at a much larger scale.

Future Work. In future work, it would be interesting to generalize our approach to sub-blocks of a given platonic element, e.g., a pediment shared by a door and a window. Such a generalization would allow our method to treat elements that are not necessarily repetitive, e.g., a door that shares sufficient sub-elements with repetitive windows. This would also allow our approach to treat elements other than windows. Another interesting avenue of future work would be the usage of our approach as a component for procedural modelling and generation. Our augmented subiews could be used as part of a procedural modelling system, allowing the generation of different combinations of the different instances. Such a method would need to include a way to generate the rest of the facade in a procedural manner, while being consistent with the requirements of IBR. . From left to right: textured mesh reconstructed using the 3 input views, our refined mesh textured using the 3 input views, our refined mesh textured with all our generated subviews, and our IBR solution for the same view. For each platonic camera C * i, j , we estimate an approximate corresponding camera C i, j in the input scene frame. As in Sec. 4.2, we compute it from the input camera C i and the cropped image V i, j location in V i . We compute a transformation from camera C * i, j to C i, j by aligning their positions, and their direction, up and right vectors. This transformation only characterizes a rotation and translation, leaving an unknown scaling factor between the two frames.

To lift this ambiguity, we use the distances between the camera C i,j * E * P i,j *

Figure 24: We cast rays from the borders of the view associated to C * i, j , intersecting the platonic model (red points). These samples can be used to estimate the facade plane (dotted grey line).

towards the mesh. The motivation is that such intersection points will belong to the facade wall. From these points we can estimate the facade plane. We intersect a central ray with it to obtain P * i, j . Then, d * i, j is the distance between P * i, j and C * i, j , d i, j the distance between P i, j and C i, j . We estimate the scaling factor s from the ratio of these distances, taking into account the field of view values of C * i, j and C i, j .

s = d i, j d * i, j
tan(fov Ci, j /2) tan(fov C * i, j /2) In practice, we express the transformation from platonic scene space to C * i, j image space, then estimate the scaling factor; we compose this result with the transformation from C i, j image space to input scene space. The candidate (T i, j ) transformations are centered (from E j to the origin) and averaged to obtain the final transformation T . This transformation is used to place the platonic mesh and cameras back into the input scene frame, before duplicating and translating them at each E j position using P i, j . and the facade, in the platonic and input frames (respectively d i * , j and d i, j ).

The optical center of C i, j intersects the known facade plane (and thus E j ) at P i, j . Similarly the optical center of C i * , j intersects the platonic facade plane at P i, * j (Fig. 24). This point does not necessarily belong to the platonic mesh, due to geometric details such as window recess. We estimate the plane of the facade in the platonic frame by sampling pixels near the borders of V i, j , and casting rays

EurographicsFigure 1 :

 1 Figure 1: Given a small number of pictures of a facade (a), we augment the number of views of the repetitive windows by combining the different views of each instance (b). We place the augmented set of viewpoints into a common space, allowing us to generate finer 3D geometry and to identify reflective areas (c). Compared to an image-based rendering of the input views, our solution produces sharper results and fewer popping artifacts (best seen in the accompanying video).

S

  . Rodriguez & A. Bousseau & F. Durand & G. Drettakis / Exploiting Repetitions for IBR

Figure 3 :

 3 Figure3: Overview. From the input images, we first segment and select repetitive windows, followed by camera calibration of these subviews by a modified SfM algorithm. In the second step, we use the calibrated subviews to reconstruct a platonic mesh of the window using multiview stereo, and subsequently refine it to obtain a piecewise planar geometry. In the third step we extract reflections from each subview and deduce a reflection mask of the window. We additionally generate images of all instances of the window from all available viewpoints. We finally re-insert the platonic geometry and images of each instance into the complete facade for improved image-based rendering.

  et al. [WFP11], who detects repetitive facade elements inside a single image to estimate a depth map, and on the work by Xia et al. [XFT * 08], who regularizes noisy depth maps by decomposing facades into rectangular regions with constant depth. We combine and extend these two ideas to reconstruct a detailed 3D mesh from multiple images of a repetitive element. Our use of repetitions for 3D reconstruction is also related to the work of Zheng et al. [ZSW * 10] and Demir et al. [DAB15], who aggregate information from repetitive pieces of a 3D point clouds to consolidate it. Heinly et al. [HDF14] also use repetitions to improve camera calibration by detecting conflicting observations between viewpoints.

Figure 5 :

 5 Figure5: We rectify each input image (a,b) before feeding it to a deep classification network that detects window pixels (c). We smooth the label map (d) and compute the bounding box of each connected component (e). We then expand these boxes to include the surroundings of each window and project them back into the image (f). We ask users to select the boxes corresponding to a group of similar windows (g).

Figure 6 :

 6 Figure 6: (a) Candidate matches between two views using standard approaches. The two views represent a similar window: the first view contains parts of another window on its side, which are matched to the central window of the second view. (b) We use an approximate camera derived from the input scene to filter these outliers. (c) We reproject each pair of matched points into each other subview using the approximate cameras. If both reprojections land close to their correspondences, the match is kept.

Figure 7 :

 7 Figure 7: The estimated facade plane is displayed (blue) along with the input scene sparse point cloud. Input and crop cameras are shown. The yellow points are the intersection of each crop subview with the plane.

  [Rea18] tend to produce noisy 3D point clouds on such input, possibly due to the per-instance and per-view variations present in the cropped subviews. Inspired by prior work on image-based facade modeling [XFT * 08], we refine this noisy reconstruction by decomposing it into flat polygonal regions parallel to the facade, and aligned with image edges.

Figure 9 :

 9 Figure 9: Left: Input image. Right: Polygonal segmentation.

Figure 10 :

 10 Figure 10: From left to right: noisy surface obtained by applying multi-view stereo on the input images, refinement of this noisy surface, noisy surface obtained by applying multi-view stereo on the cropped subviews, refinement of this noisy surface. The cropped subviews provide additional information that yield a more detailed mesh after refinement.

Figure 12 :

 12 Figure12: Visualization of the generated facade mesh (center) from the camera of an input view (left). We overlay the input image onto the mesh view to assess alignment. Another viewpoint on the same mesh (right).

Figure 13 :

 13 Figure 13: Mis-alignments introduce ghosting when performing image-based rendering using the original images and the refined mesh.

Figure 14 :

 14 Figure 14: From left to right: input cropped image, extracted diffuse layer, extracted variation layer (intensity), reprojected reflection mask.

Figure 15 :

 15 Figure 15: For each instance, we decompose the associated cropped views into diffuse and view-dependent effects layers. All view-dependent layers are then combined to estimate a frontoparallel mask guided by the segmentation. The resulting mask is then reprojected in all views.

Figure 17:

  Figure17: Each instance E i of a window is only seen in a subset of subviews (images with colored border). We augment this coherent subset by re-projecting the available subviews in other viewpoints, and fill-in occlusions and missing parts using the corrsponding subviews from another instance (colored areas in images with black border). We also re-project the reflection mask in all subviews.

Figure 19 :

 19 Figure 19: (a) If the crop is blended naively, we observe visible seams. (b) Unary terms weighting with distance from boundaries. (c) After the graph cut, visible seams are eliminated. (d) Resulting cut.

Figure 20 :

 20 Figure 20: Our method allows for instances of the platonic element to be swapped and reinserted in the final scene in a coherent manner. Top: original facade, bottom: novel facade.

Figure 22 :

 22 Figure 22: Multi-view texturing in this figure was performed using the texturing module of RealityCapture [Rea18]. From left to right: textured mesh reconstructed using the 3 input views, our refined mesh textured using the 3 input views, our refined mesh textured with all our generated subviews, and our IBR solution for the same view.

Figure 23 :

 23 Figure 23: On this scene where standard MVS reconstruction fails (middle row), we obtain a usable result (right).

  al. [XWL * 08] take a single picture of a flock of animals to generate an animation of that animal, effectively turning spatial repetition into temporal information. Aittala et al. [AWL * 15] use repetitions in a flash picture of a material sample to extract patches of the material under dif-

		Element Extraction	User Interaction:	Crops & Plane Selection	Structure From Motion		Diffuse	Variabilities Extraction Mask View-dependent
	Input Images	Platonic Element Extraction Original Cameras Calibration Estimation Platonic Cameras	Geometric Reconstruction Reconstruction Refinement	Layer Extraction and Views augmentation	Image-Based Rendering
		and Camera Calibration	of Platonic Element	View Augmentation
						ferent lighting conditions, recovering rich angular information for
						SVBRDF acquisition. Dekel et al. [DMIF15] extract local differ-
						ences between similar patches to attenuate or accentuate small vari-
						ations in an image. Closer to our context is the work of Alhalawani
						et al. [AYLM13], who rely on user interaction to detect window-
						specific details, such as blinds and shutters. Since these details

Acknowledgements

This research was partially funded by the European Union Horizon 2020 research and innovation programme under grant agreement No. 727188 EMOTIVE, http://www.emotiveproject.eu/, and by generous donations from Technicolor and Adobe. The authors thank J.-P. Bauchet and F. Lafarge for their help with the KIPPI segmentation and J. Philip for his plane fitting implementation.

Appendix A: Transformation from platonic scene to input scene

We need to estimate a transformation T from the platonic frame to the input scene frame, in order to place the platonic model and cameras back into the input scene frame. We evaluate a series of candidate transformations T i, j using each platonic camera separately (see Fig. 11).