Keywords: Image warping, enhancement, shape perception. Concepts:, Computing methodologies → Image manipulation;

We present an interactive method that manipulates perceived object shape from a single input color image thanks to a warping technique implemented on the GPU. The key idea is to give the illusion of shape sharpening or rounding by exaggerating orientation patterns in the image that are strongly correlated to surface curvature. We build on a growing literature in both human and computer vision showing the importance of orientation patterns in the communication of shape, which we complement with mathematical relationships and a statistical image analysis revealing that structure tensors are indeed strongly correlated to surface shape features. We then rely on these correlations to introduce a flow-guided image warping algorithm, which in effect exaggerates orientation patterns involved in shape perception. We evaluate our technique by 1) comparing it to ground truth shape deformations, and 2) performing two perceptual experiments to assess its effects. Our algorithm produces convincing shape manipulation results on synthetic images and photographs, for various materials and lighting environments.

Flow-Guided Warping for Image-Based Shape Manipulation

Romain Vergne 1,2 * Pascal Barla 2 † Georges-Pierre Bonneau 1,2 ‡ Roland W. Fleming 3 § Univ. Grenoble Alpes, CNRS 1 Inria 2 Justus Liebig Universität Gießen 3

Introduction

When presented with the image of an object as in Figure 1a, we effortlessly perceive its shape, material, and the way it is lit by the surrounding environment. This is a formidable achievement. Accomplishing this feat through inverse optics would be extremely challenging as the necessary inferences are severely under-constrained. However, recent work in visual perception suggests an alternative, and perhaps more reasonable, explanation of how we perceive and process object appearance. In particular, [START_REF] Fleming | Specular reflections and the perception of shape[END_REF]2011] propose that image properties are indicative of object shape and material properties, simply because they are sufficiently correlated together. This opens an exciting avenue of research for computer graphics: by identifying the appropriate correlations, one could elicit perceptually compelling changes in object appearance by mere manipulations of its image, without having to solve the difficult inverse optics problems of reconstructing shape, material or lighting.

Figure 2: Perception studies suggest that orientation patterns (here due to the compression and stretching of reflections on a shiny 3D object) provide strong visual cues to surface shape.

In this paper, we identify and exploit correlations between image gradient structure and the corresponding object shape. The key idea is to consider that orientation patterns in the image are mainly due to surface curvature. This is illustrated in Figure 2, where orientation patterns are due to compression and stretching of the reflected environment. Recent work in human and computer vision stresses the importance of such patterns in the perception of shape (e.g., [START_REF] Ben-Shahar | On the perceptual organization of texture and shading flows: from a geometrical model to coherence computation[END_REF][START_REF] Fleming | Specular reflections and the perception of shape[END_REF][START_REF] Fleming | Estimation of 3d shape from image orientations[END_REF]). We complement these perceptual findings with mathematical relationships and a statistical analysis in Section 3.

Equipped with these correlations between image and shape properties, in Section 4 we introduce a novel image deformation algorithm that runs in real-time on the GPU. It works in two stages: first the scale of shape-related features are estimated per pixel; then the image is warped, guided by orientation patterns at their respective scale. As shown in Figure 1b,c, such an image warping produces a vivid impression of shape changes, making the object appear either sharper or rounder depending on warping parameters. We evaluate our algorithm by comparing it to ground truth shape deformations in Section 4.3 and by conducting perceptual validation through a pair of psychophysics experiments in Section 5. We demonstrate that our method works for a wide range of object materials and environment lightings in Section 6.

Related work

Image manipulation Early methods for the manipulation of content in images were based on partial differential equations, such as anisotropic diffusion [START_REF] Perona | Scale-space and edge detection using anisotropic diffusion[END_REF] for edge-preserving smoothing, or shock filtering [START_REF] Osher | Feature-oriented image enhancement using shock filters[END_REF] for detail enhancement. Recent methods often make use of multi-scale decompositions to manipulate image details while preserving edges (e.g., [START_REF] Farbman | Edge-preserving decompositions for multi-scale tone and detail manipulation[END_REF][START_REF] Paris | Local laplacian filters: Edge-aware image processing with a laplacian pyramid[END_REF]). However, the focus of these methods is on the editing and filtering of low-level image features, rather than on modifying the perceived attributes of the depicted objects, such as their shape or material. [START_REF] Fattal | Multiscale shape and detail enhancement from multi-light image collections[END_REF] specifically target shape details by using a multiscale decomposition on multi-light image collections. [START_REF] Boyadzhiev | Band-sifting decomposition for image-based material editing[END_REF] focus instead on material editing in a single input image, which is permitted by a careful perceptual analysis of subbands in the multi-scale decomposition. Techniques that magnify nearly imperceptible details in images also work in a multi-scale fashion. They may either magnify deviations through time [START_REF] Wu | Eulerian video magnification for revealing subtle changes in the world[END_REF][START_REF] Wadhwa | Phase-based video motion processing[END_REF], with respect to ideal geometries [START_REF] Wadhwa | Deviation magnification: Revealing departures from ideal geometries[END_REF] or relative to an ideally repeated pattern [START_REF] Dekel | Revealing and modifying non-local variations in a sin-gle image[END_REF].

A parallel line of work stems from the seminal intrinsic decomposition technique of [START_REF] Barrow | Recovering intrinsic scene characteristics from images[END_REF], which consists in separating an image into its reflectance and shading components. Modern solutions to the intrinsic decomposition problem may require user annotations on a single input image [START_REF] Bousseau | User assisted intrinsic images[END_REF] for increased accuracy. This still assumes that the input image is a simple product between reflectance and illumination components. The work of [START_REF] Carroll | Illumination decomposition for material recoloring with consistent interreflections[END_REF] improves on this limitation by further decomposing illumination into direct and indirect components. Other methods propose user-assisted decompositions that address more complex material properties: [START_REF] Dong | Appgen: Interactive material modeling from a single image[END_REF] propose a workflow for separating diffuse and specular terms in color textures; [START_REF] Yeung | Matting and compositing of transparent and refractive objects[END_REF] present a method to separate a transparent object from its background, and recompose it on a different background. Both methods are user-assisted.

The work of [START_REF] Yeung | Matting and compositing of transparent and refractive objects[END_REF] is most related to ours in that it makes use of image warping to mimic the effect of refraction due to the transparent object on the new background. Our approach instead applies warping directly to the image of a foreground opaque object to modify its perceived shape (without requiring user annotations). It also makes use of a multi-scale analysis to identify the scales of shape features per-pixel.

3D reconstruction An alternative approach to manipulate image content is first to reconstruct 3D information, then modify it and finally re-render (part of) the image.

The technique of [START_REF] Khan | Image-based material editing[END_REF] makes the simplifying assumption that surface depth is inversely proportional to image intensity, and computes normals from this surface via differentiation. An environment lighting is recovered by extracting and duplicating the background image. These approximations prove sufficient for image-based material editing in many practical cases: they permit re-texturing an object or even making it transparent. The method has been extended by Guttierrez et al. [2008] to include caustics by treating the recovered front object surface as a collection of thin lens segments. Once depth and normal buffers become available, other manipulation techniques are possible such as Unsharp Masking [START_REF] Luft | Image enhancement by unsharp masking the depth buffer[END_REF] or Surface Flows [START_REF] Vergne | Surface flows for image-based shading design[END_REF]]. The latter approach specifically extends the method of Khan et al. to directly manipulate highlights in the image, in a way that ensures the deformations conform to the recovered normals.

Approaches based on the recovery of depth or normal buffers suffer from a common limitation: they cannot rotate the object or freely reposition it in space. In contrast, methods that estimate full 3D data [START_REF] Chen | 3sweepp: Extracting editable objects from a single photo[END_REF][START_REF] Kholgade | 3d object manipulation in a single photograph using stock 3d models[END_REF]] must restrict the category of objects they recover. In the work of [START_REF] Chen | 3sweepp: Extracting editable objects from a single photo[END_REF], users assist object extraction by identifying shape components using strokes. Strokes are then automatically adjusted by the algorithm, and used in a sweeping technique to obtain 3D shapes. As a result, recovered objects are restricted to collections of generalized cylinders and cuboids. In the work of [START_REF] Kholgade | 3d object manipulation in a single photograph using stock 3d models[END_REF], 3D shape is recovered with the help of stock 3D models. Recovered objects are thus limited by the availability of similar 3D models, as well as by symmetry priors.

Our approach altogether avoids the reconstruction of 3D information by identifying image properties directly related to shape, which are then manipulated through warping. However, it shares the limitation of methods based on normal or depth buffers as it does not permit 3D object transformations such as rotations.

Shape perception Theories of shape-from-shading (e.g., [START_REF] Horn | Shape from Shading[END_REF]) and shape-from-specularities (e.g., [START_REF] Oren | A theory of specular surface geometry[END_REF][START_REF] Savarese | What do reflections tell us about the shape of a mirror[END_REF]) have for the most part relied on computational models. Shape-from-shading models often make the assumption that there is a main light source direction that must be estimated before shape can be recovered. In shape-fromspecularities, the hypothesis is similar: the way a nearby object is distorted by shape through reflection cannot be exploited if the neighboring object is unknown. These are drastic constraints that are often not verified in real-world scenarios: the environment may contain multiple strong light sources and reflected objects may be out of sight.

An alternative approach consists in relying on orientation patterns that are directly observable in the image. Early perceptual evidence comes from the importance of patterns of isophotes in the perception of shape-from-shading [START_REF] Koenderink | Photometric invariants related to solid shape[END_REF]. This line of research has evolved in the computer vision community as well, such as in the work of Zucker and colleagues (e.g., [START_REF] Ben-Shahar | On the perceptual organization of texture and shading flows: from a geometrical model to coherence computation[END_REF]) who studied the similar concept of "shading flows". Orientation patterns in images play a similar role in the perception of shape-from-specularities [START_REF] Fleming | Specular reflections and the perception of shape[END_REF]. They are due to compression and stretching of reflections caused by surface curvature. Similar types of patterns may be observed with surface texture markings [START_REF] Fleming | Estimation of 3d shape from image orientations[END_REF], which suggests a central role of orientation patterns in the perception of shape from various cues.

Orientation patterns are relevant to shape perception because they are sufficiently correlated to physical shape, assuming generic viewpoint and light source configurations [START_REF] Freeman | The generic viewpoint assumption in a framework for visual perception[END_REF]]. However, in some cases orientation patterns may provide information that departs from physical shape, and perceptual studies show that in these cases perceived shape differs as well. For instance, strong directional lighting may distort the perception of shape from shading [START_REF] Caniard | Distortion in 3d shape estimation with changes in illumination[END_REF]. The choice of material may also modulate the perception of 3D shape [START_REF] Wijntjes | Relative flattening between velvet and matte 3d shapes: Evidence for similar shape-from-shading computations[END_REF][START_REF] Mooney | Specular image structure modulates the perception of three-dimensional shape[END_REF].

We take inspiration from modern models of shape perception and study the correlation between shape and image properties in Section 3. We then use these correlations to design an image deformation algorithm that gives the illusion of shape changes in images. However, our goal is not to manipulate physical, but perceived shape, which we expect to vary with lighting and materials.

Image analysis

We begin with a mathematical analysis of the orientation patterns produced by the rendering of a 3D object. Our goal here is to provide a quantitative assessment of the relationships between surface shape and orientation patterns in the image. It is intended to complement the vision literature on the subject, as well as to provide a justification for the warping algorithm of Section 4.

Definitions

We consider a simplified model of image formation in the neighborhood P of a pixel of interest p (see Figure 3 left). We assume P corresponds to surface points that lie on the same 3D object (i.e., no occluding contour crosses P). The image intensity (for a single color channel) of a neighboring pixel q ∈ P is then defined by:

I : P → [0, 1] q → M (Lr(q, ωo)) ,
where M is a tone mapping operator and Lr is the radiance reflected from the object surface according to [START_REF] Kajiya | The rendering equation[END_REF]]:

Lr(q, ωo) = Ωn ρn(q, ωi, ωo)Li(q, ωi) max(0, ωi • n(q)) dωi, with ωo and ωi the outgoing view and incoming light directions, n the surface normal at q, Ωn the upper hemisphere, ρn the material (Bidirectional Reflectance Distribution Function) and Li the incoming radiance.

We further assume that spatial variations of ωo, ρn and Li are negligible inside P. In particular, this means that we consider a locally orthographic camera, a homogeneous material and a distant illumination. As a result, the only variable that depends on q is the normal n(q), which allows us to write (see Figure 3 right):

I(q) ≈ S (m(q)) , (1)
where

m = [m0 m1] T = [ -n 0 n 2 -n 1
n 2 ] T is the surface slope corresponding to the normal n = [n0 n1 n2] T expressed in screen-space; and S is a shading function that encapsulates both reflected radiance and tone mapping. We use surface slopes since they have an exact correspondence with front-facing unit normals, while being easier to relate to surface shape, as explained later on.

Equation 1 thus expresses image intensity I inside P by a local shading function S applied to surface slopes m. In the following, we visualize such shading functions using Lit Spheres [START_REF] Sloan | The lit sphere: A model for capturing npr shading from art[END_REF]; they are not equivalent though, as they are related by a normal-to-slope transform. In Figure 4 we have rendered a single object with several Lit Spheres exhibiting various shading directions and frequencies. In this case, Equation 1 is exact and the same shading function is applied everywhere. More realistic rendering scenarios using global illumination are considered in Section 3.3.

Figure 3:

The image intensity I for pixels q in a neighborhood P around a pixel of interest p (left) is locally approximated by a shading function S applied to surface slopes m (right), with surface slopes being directly related to screen-space normals n.

Mathematical relationships

Recent findings in human and computer vision show that orientation patterns in the image constitute visual cues to object shape [START_REF] Ben-Shahar | On the perceptual organization of texture and shading flows: from a geometrical model to coherence computation[END_REF][START_REF] Fleming | Specular reflections and the perception of shape[END_REF][START_REF] Fleming | Estimation of 3d shape from image orientations[END_REF]]. Our goal is to draw explicit relationships between such patterns and object shape in the context of Equation 1. A common approach to identify orientation patterns is to compute structure tensors [START_REF] Bigun | Optimal orientation detection of linear symmetry[END_REF][START_REF] Brox | Adaptive structure tensors and their applications[END_REF]]:

TI (p) = P w( p -q )∇I(q)∇I(q) T dq, ( 2
)
where w is usually a Gaussian kernel truncated on P. An important property of the structure tensor is its positive semi-definitiveness. In particular, the maximum eigenvector identifies the dominant local image orientation irrespective of its direction.

The image gradient inside Equation 2 is obtained by differentiating Equation 1, yielding ∇I(q) = ∇m(q)∇S(m). The surface slope gradient ∇m = [∇m0 ∇m1] is by definition equivalent to the hessian of the surface depth. It is related to surface curvature, but also depends on the positioning of the surface with respect to the viewpoint, as already pointed out by [START_REF] Vergne | Surface flows for image-based shading design[END_REF]. We call shape features the properties of ∇m across the image. The shading gradient ∇S incorporates angular variations due to both lighting and material. We may thus expect its magnitude to be higher for specular materials and sharp lighting environments.

Replacing ∇I in Equation 2 by its local approximation yields:

TI (p) = P w( p -q )∇m(q)∇S(m)∇S(m) T ∇m(q) T dq.
(3) In order to relate TI to shape features, we also compute the structure tensor Tm on surface slopes:

Tm(p) = P w( p -q )∇m(q)∇m(q) T dq. ( 4
)
The difference between the two tensors lies in the ∇S∇S T term.

By performing an eigen decomposition, we obtain:

∇S(m)∇S(m) T = ∇S(m) 2 R φ S P0R -φ S , (5)
where R is a 2D rotation matrix, φS is the angle between ∇S(m) and e0 = [1, 0] T and P0 = I -e1e T 1 is the projection on e0. Inserting Equation 5 into Equation 3 and comparing it to Equation 4 sheds light on two important differences between TI and Tm: the ∇S(m) 2 term modulates each surface slope gradient based on the shading gradient magnitude, while the R φ S P0R -φ S term projects each surface slope gradient in the shading gradient direction. When combined through integration, the projected and modulated surface slope gradients yield a tensor TI that is strikingly similar to Tm, as shown in the third and fourth rows of Figure 4. This suggests that TI contains reliable visual cues to surface shape, across a wide range of material and lighting. Nevertheless, TI and Tm differ in a few subtle, yet important, details that appear in Figure 4. We describe them below.

(a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k)

Masking

In regions corresponding to near-uniform portions of S (such as with an overcast sky or a dim diffuse shading), ∇S ≈ 0 for all m. As a result, TI ≈ 0 and shape features cannot appear in these image regions. This is called masking in the vision literature, and may be due to either lighting or material [START_REF] Wijntjes | Relative flattening between velvet and matte 3d shapes: Evidence for similar shape-from-shading computations[END_REF].

Directional masking When the shading gradient is strongly directional (as with a collimated lighting), all surface slope gradients are projected in roughly the same direction. For instance, when φS ≈ 0 for all m, we get TI ≈ Tm 0 (i.e., the tensor due to surface slopes in the horizontal direction). This leads to directional masking effects [START_REF] Fleming | Specular reflections and the perception of shape[END_REF]] in places where ∇m0 ≈ 0.

Exaggeration and distortion.

If instead shading gradients of strong magnitude are roughly aligned with surface slope gradients, shape will tend to be exaggerated. For instance it will happen when most surface variations are due to ∇m0 and φS ≈ 0. This exaggeration effect is usually stronger for specular materials [START_REF] Mooney | Specular image structure modulates the perception of three-dimensional shape[END_REF]. It may also lead to distortion effects [START_REF] Caniard | Distortion in 3d shape estimation with changes in illumination[END_REF] when the distribution of shading gradient magnitudes is shifted from the origin in S.

Statistical analysis

Even though we are mostly interested in perceived shape, it remains to be shown that the latter is sufficiently similar to physical shape in more general cases than the one in Figure 4. For this purpose, we conducted a statistical analysis on a collection of objects rendered with global illumination using Mitsuba [START_REF] Jakob | Mitsuba renderer[END_REF]]. We used four blobby objects of increasing shape complexity, each made of four types of material and rendered in four different environment lightings. Figure 5 shows a subset of these data; the full dataset is pre-sented in a supplementary document.

For each object, we computed its surface-based tensor along with image-based tensors for each condition. Next, we assembled a pair of similarity matrices relating all possible conditions, yielding two 17 × 17 symmetric matrices per object. The first matrix measures orientation similarity averaged over all pixels. It is computed as the dot product between the two maximum eigen vectors of each tensor. The second matrix measures the cross-correlation of the two maximum eigen values (i.e., the best correlation among shifted versions of the image), again averaged over all pixels. We use it instead of correlation to account for the occurrence of the distortion effects, which tend to shift shading patterns. Similarity matrix pairs for individual objects are provided in supplementary material.

Figure 6 summarizes our results using a pair of matrices where similarities have been averaged over all four objects. The average similarity of orientations across all objects and conditions equals 0.87, with a standard deviation of 0.22; while the average crosscorrelation of magnitudes across all objects and conditions equals 0.84, with a standard deviation of 0.24 (with a p-value ≪ 0.05). These statistics demonstrate that image-based tensors are significantly correlated not only to the surface-based tensor (first row of each matrix), but also to each other (all other rows).

Image deformation

We have shown in the previous section that image-based structure tensors are significantly correlated to shape features, and likely constitute visual cues to perceived shape. Our goal now is to make use of these correlations to deform an image so as to give the illusion of shape changes.

Until now we have implicitly assumed that shape features occur at a single global scale in the image. However, when considering images of complex objects, shape features may occur at different scales in different locations in the image. We thus begin in Section 4.1 by explaining how we identify pertinent scales at which image-based structure tensors will be computed. We next describe our warping algorithm in Section 4.2, compare it to surface-based deformation and provide a geometric interpretation of its effects in Section 4.3. Our GPU implementation is detailed in Section 4.4.

Scale selection

The process of identifying pertinent scales per pixel is called automatic scale selection in the scale space literature [START_REF] Lindeberg | Feature detection with automatic scale selection[END_REF]].

We use a simple scale selection mechanism that identifies a single scale σI per pixel p using a weighted combination of the form:

σI (p, γ) = σ 2γ βI (p, σ) dσ βI (p, σ) dσ , (6)
where σ ∈ (0, +∞) stands for the scale dimension, γ ∈ R * is a user-controlled parameter and βI is a scale-space image-based weighting function. The γ parameter directly comes from Lindeberg et al. [1998]: it permits biasing the selected scale toward the smallest scale when it is close to 0 (or towards the biggest scale when greater than 1). We choose γ = 0.5 as a default value, since Equation 6 then boils down to a linear combination of weights. For the weight, we use βI (p, σ) = Tr (TI (p)) where the structure tensor TI from Equation 2 is implicitly computed within a Gaussian window wσ of standard deviation σ, truncated to P(σ) = {q | pq < 3σ}. The same approach is easily applied to images of surface slopes: the trace of the tensor Tm from Equation 4 is then used to build a scale-space weight function βm, yielding σm.

The two resulting scale maps σm and σI are compared in Figure 7 for various shading functions. As before, we use various Lit Spheres for rendering the entire object. It is easily shown that for an affine shading function S, we have σI = σm. For more complex combinations of materials and lighting environments, the two scale maps depart from each other. In particular, high-frequency shading functions exhibit more pronounced differences at low values of γ.

Nevertheless, we have found this simple scale selection mechanism to provide reasonable results in practice, as shown in Section 6.

Deformation algorithm

The general idea of our image deformation algorithm is to exaggerate the orientation patterns found in an input image through warping. For instance, when such patterns are due to either compression or stretching of reflections as in Figure 2, warping should exaggerate compression and stretching so as to give the illusion of shape change. In the following, we use the maximum eigenvalue λ max I of TI to guide warping since it provides the magnitude of compression or stretching along the dominant local orientation.

Our image deformation procedure is summarized in Algorithm 1 and illustrated in Figure 8. It takes as input an image I, a scale bias γ and a warping parameter α, and returns a warped image. We optionally take a gray-scale mask as input to control warping locally. We describe our algorithm step-by-step below.

Structure tensor

The algorithm starts (lines 2-3) by computing a scale map σI using Equation 6, then a structure tensor image TI at the selected scales using Equation 2. However, as discussed in the previous section, both shape and shading features act on TI . Taking inspiration from Ben Shahar et al. [2001], we enforce orientation continuity (line 4). This is done by smoothing TI in the direction of the minimum eigen vector, assuming variations along these directions are mainly due to shading. An illustration is given in the inset figure, where streamlines (in grey) follow minimum eigen vector directions and the smoothing kernel (in orange) is an elliptic Gaussian.

Coherence enforcement

The approach is similar in practice to the image stylization method of [START_REF] Kyprianidis | Image abstraction by structure adaptive filtering[END_REF].

We use a filtering kernel of standard deviation σI and anisotropy Tensor fields before and after continuation enforcement are visualized in Figure 8b.

Guiding flow

The next three steps are devoted to computing a unit gradient field that will serve as a guiding flow for warping. We start (line 5) by extracting the maximum eigen value λ max I from TI . The purpose of local adaptation (line 6) is to make sure all shape features get exaggerated irrespective of their scale. To this end, we center and rescale λ max I at a pixel p based on neighbor eigen values using

λ max I -E[λ max I ] Std[λ max I ]
, where E and Std denote the expectation and standard deviation inside P(σI ). We call this process local adaptation because of its similarity with lightness adaptation in vision. The non-adapted and adapted maximum eigen value images are visualized in Figure 8c. The guiding flow is then computed (line 7) by convolving λ max I with a 1st-order Gaussian derivative of scale σI and normalizing the resulting gradient field.

Image warping Now that we are equipped with a guiding flow, we are ready to compute an image W of warped pixel coordinates (lines 8-9). The warping function is detailed in Algorithm 2. It takes an initial pixel position p, a flow ∇λ, a scale map σ and a warping parameter α. The main idea is to walk along ∇λ pixel-bypixel until a distance d (in pixels) has been traversed. The direction depends on the sign of α, while d is determined both by the magnitude of α and the local scale σ. Using W directly for image warping creates image discontinuities for large α values, which are due to regions where many adjacent pixels are warped to the same coordinate. We avoid such artifacts (line 10) by smoothing out the warped coordinates of W , convolving it with a Gaussian weight function of standard deviation σI . We compare non-smoothed vs smoothed coordinates in Figure 8d. The final warped image is obtained by a composition of functions (line 11) and is shown in Figure 8e. From a user perspective, our image deformation algorithm offers two controls: the warping amplitude α determines both the magnitude and direction of warping (α < 0 leads to sharpening, α > 0 to rounding and α = 0 has no effect), while the (optional) scale bias γ controls the selected scale as explained in Section 4.1.

Geometric interpretation

We now give a geometric interpretation of our image deformation algorithm to provide a more intuitive understanding of its effect. We apply two different types of warping on rendered images: one based on the image I, the other based on surface slopes m. Surfacebased warping is easily obtained by replacing I, σI , TI and λ max We first compare image-and surface-based warping in Figure 9 using height fields rendered with Lit Spheres from Figure 4(e,f,h,j). The two types of warping yield qualitatively similar results for various shading functions and warping parameters.

Moreover, since the shading function is applied to the whole image in these examples, warping the image based on surface slopes is strictly equivalent to warping surface slopes and re-rendering. Consequently, we use Poisson reconstruction [Pérez et al. 2003] guided by both initial and warped surface slopes to obtain initial and warped surface depths. As shown in Figure 10, comparing height fields before and after applying warping provides an intuitive geometric interpretation of our algorithm for different signs of α.

GPU implementation

We have implemented Algorithm 1 in Gratin [Vergne and Barla 2015], using height passes on the GPU. Scale selection requires two passes: one to compute the tensor at multiple scales, and another one to apply Equation 6. In practice, we build a Gaussian pyramid for TI using a custom mip map texture, which improves efficiency without sacrificing accuracy. The remainder of Algorithm 1 requires six passes since lines {5, 6} and {10, 11} can each be implemented in a single pass.

The performance of our warping algorithm is mainly determined by the scale map σI : it not only controls the neighborhood size in lines {3, 4, 6, 7} of Algorithm 1 but also the traversal distance d in Algorithm 2. However, note that the α parameter is first used at line 9 in Algorithm 1; hence the first steps of the algorithm only have to be recomputed if the γ parameter is modified. Our technique works at interactive rates when modifying γ in most practical cases. This is demonstrated in table 1 where we use a single global scale σ to measure performance independently of the input image.

We give timings as a function of the diameter of P; hence our smallest neighborhood corresponds to a scale of 1/3. Note however that when only α is modified, our method runs in real-time (at more than 30fps in all our examples). In practice, for large-scale deformations, we use a low-resolution image as input to lines 2 to 10 in Algorithm 1 to retain interactive performances. The high-resolution image is then used in line 11 to fetch warped coordinates and obtain a high quality result. Table 1: Performance (in ms) for lines 2 to 7 in Algorithm 1, for various neighborhood sizes (⊘(P) = 6σ + 1 in our implementation) at two resolutions, using a Nvidia Quadro M6000.

Perceptual evaluation

Since our image deformation technique is intended to manipulate perceived shape in images, its effects must be validated through perceptual experimentation. We have conducted two experiments: one based on the synthetic images used in our Supplemental Material (see also Section 3.3), and another based on photographs. We wrote our experiments in Octave, using the Psychophysics Toolbox extensions [START_REF] Brainard | The psychophysics toolbox[END_REF]].

Experiment 1: synthetic images

Observers 12 subjects (excluding the authors) with normal or corrected-to-normal vision participated in this experiment. All were students in Computer Graphics or Human Vision and were naive with respect to the purpose of the experiment. In particular, no mention of image warping was made.

Procedure Subjects were first shown a simple example of shape sharpening and rounding (see Supplemental material). They were then presented with a pair of images in each trial and asked to pick the one where object shape was sharpest. Each pair contained an original (unmodified) image of a rendered 3D object, and a warped image, their position on the screen (left or right) being randomized on each trial. We rendered a subset of the data used for correlation analysis: 4 objects made of 2 materials in 2 lighting environments.

We applied both image-and surface-based warping on all conditions with α ∈ {-1.0, -0.5, 0.5, 1.0} at a fixed scale σ = 4 for diffuse materials and σ = 5 for glossy ones (same for both warpings). All combinations were repeated twice and the order of trials randomized, leading to a total of 256 trials. Participants usually completed the experiment in less than 30 minutes.

Results

We consider data coming from image-based and surfacebased warping separately. A judgment is considered correct when it matches the sign of α. Participants provided correct answers in 86.6% of the trials involving image-based warping (with a std of 26.5%), and in 80.0% of the trials involving surface-based warping (with a std of 32.3%). We plot these percentages as a function of α for all subjects in Figure 12-left.

Experiment 2: photographs

Observers 12 subjects (excluding authors) participated in experiment 2, of whom 7 had already participated in experiment 1. Procedure Subjects were first shown a simple example, this time using a warped photograph (see Supplemental material). Then, on each trial, they were asked to identify in which of two images the object shape appeared sharper. The difference with experiment 1 is that input images were photographs; hence only image-based warping was applied in this case. We used 9 photographs showing objects of different shapes and made of different materials. We kept the same parameter values for α and we adapted σ in each image to match their relative feature scales. We repeated all combinations twice, leading to a total of 72 trials. Participants usually completed the experiment in less than 10 minutes.

Results

Participants provided correct answers in 85.8% of the trials (with a std of 28%). We plot these percentages as a function of α for all subjects in Figure 12-right.

Discussion

Experiment 1 shows that subjects could readily perceive the effect of both image-based and surface-based warping on synthetic input. Surprisingly, image-based warping gave slightly higher percentage correct results, which might be partly explained by exaggeration effects (see Section 3.2). Experiment 2 shows that subjects were equally good at perceiving the effect of image-based warping on input photographs. Figure 12 shows that in both experiments, a warping amplitude as small as α = ±0.5 is enough to elicit changes in shape perception. We also observe a general decrease in performance for α = 1.0. This might be explained by the effect of our warping algorithm, which tends to "flatten" perceived shape in some image regions for high positive values of α.

These experiments confirm that our flow-guided warping algorithm is effective at manipulating perceived surface shape in color images. Figure 12 hints that image-based warping may even produce a stronger effect compared to surface-based warping, although the difference is not statistically significant, so a more refined analysis is required. If this observation turns out to be significant, one explanation might be that shape perception is essentially based on directly available image properties, which includes masking, exaggeration and distortion effects not present in surface-based warping results.

Results

We now present warping results showing the robustness of our technique to different types of material and lighting, along with detailed parameter control and comparisons with related work. The supplemental video shows results with animated warping parameters, which vividly conveys the effect of our warping technique.

Material and lighting

Our method works for a variety of matte to shiny materials in natural as well as artificial lighting. Figure 11a demonstrates warping of small shape features on a lizard in studio lighting. The spaces between the scales that occur at various sizes are either reduced or enlarged. Figure 11b applies warping to a shiny metallic helmet in artificial indoor lighting. Observe in particular how the intricate details of the front part are either sharpened or rounded. Results on aged metallic objects are shown in Figures 1 and14. Our method works equally well on materials exhibiting a mix of diffuse and specular effects, as shown in Figure 11c. The image shows the wooden statue of a face painted in red, black and white, photographed partly indoors and outdoors. Our image warping method makes surface creases either sharper or rounder depending on chosen parameters. It is applied to another wooden statue in Figure 11d, shot in natural outdoor lighting. Observe how the holes in the statue appear either reduced or enlarged, in a way similar to Figure 11a. Our method also works on translucent (natural or manmade) objects. Figure 11e shows a strawberry shot in collimated lighting. Image warping makes it look more or less 'inflated'. Figure 11f shows a macro photograph of a glazed ceramic frog, exhibiting a mix of specular reflection and subtle translucency. The effect of our method is best seen on the mouth and nostrils, which become appropriately reduced or enlarged. Additional results with varying materials and lighting properties are shown in Figure 18 and in the supplemental document. Warping control Until now, we have focused on the effect of the warping amplitude α on the final result. Figure 13 shows how our local scale selection process makes it possible to adapt warping to the size of shape features on a synthetic example. In our experience, photographs typically contain features at various scales, and considering a single global scale will inevitably distort some of these features as shown at the bottom right of the figure. Figure 14 further demonstrates the effect of the scale bias γ on image warping: it provides an effective control over the scale of exaggerated shape features. We have found that when pushed toward high scales (i.e., γ > 1), our warping technique produces compelling caricatures by strongly exaggerating face features, as seen in Figure 15. We show these examples in animation in the supplemental video, along with a brush-based interface that we use to control α and γ locally.

Comparisons Our method follows a different approach compared to related work: instead of modifying pixel intensities, it modifies pixel locations. This is shown in Figure 16, where we compare our method to two classic image manipulations: shock filtering [START_REF] Osher | Feature-oriented image enhancement using shock filters[END_REF]] and unsharp masking [START_REF] Luft | Image enhancement by unsharp masking the depth buffer[END_REF]. It is clearly apparent that the effects on the final image are different; in particular, neither the material or lighting appear to change in our results, as opposed to previous work. We believe that the combination of methods that modify pixel intensities and locations is an interesting direction of future work, as discussed next.

Limitations and Future Work

Our warping algorithm presents a few technical limitations. The scale selection mechanism considers a single scale per pixel. However, objects with complex shapes may exhibit multiple scales at a single image location. Applying warping with a bias for large scales might distort small shape details, as can be seen by zooming in Figure 8e for instance. We thus plan to extend our technique to work directly in scale space to manipulate such complex surface structures. Warping may also be restricted by the limited information available in images. For instance, shape features may appear faintly because of a locally smooth shading. In such cases, warping will still work but produce rather subtle effects, as seen in Figure 11e for example. Another limitation occurs with extremely thin highlights, which when rounded lead to flat regions of uniform intensity. Addressing these issues will likely require incorporating additional filtering operators to our approach.

From a more general perspective, our method is restricted by the α = -1 input image α = 1 hypothesis made in Equation 1. In particular, images containing many occluding contours (e.g., natural landscapes), optical effects (depth of field) or high-frequency variations due to textures or shadows usually lead to artifacts, as shown in Figure 17. In our current system, we rely on a user-provided mask to apply warping only to a foreground, in-focus and not strongly textured object, hence avoiding these artifacts. A challenging issue will be to deal with the occurrence of multiple layered orientation patterns due to textures, shadows or transparency (ideally without requiring user annotations). As shown in Figure 17, our current approach already produces reasonable results with transparency, but tends to create artifacts in the presence of textures. Another potential improvement would be to analyze and manipulate object shape with a more global approach. This is necessary not only to disambiguate concave shape features from convex ones, but also to preserve object symmetries through manipulation. Finally, our method is not temporally coherent as shown in the supplemental video. In future work, we would like to explicitly take into account the additional shape cues available in motion for the exaggeration of shape in videos.

Conclusions

We have presented a real-time image warping method that gives compelling illusion of shape changes, as demonstrated by our perceptual evaluation. It relies on correlations between surface shape features and image orientations, which provide additional evidence for the central role played by orientation patterns in the perception of shape. Similarly to recent work [START_REF] Wu | Eulerian video magnification for revealing subtle changes in the world[END_REF][START_REF] Wadhwa | Phase-based video motion processing[END_REF][START_REF] Dekel | Revealing and modifying non-local variations in a sin-gle image[END_REF][START_REF] Boyadzhiev | Band-sifting decomposition for image-based material editing[END_REF], our method specifically targets one scene property (surface shape) in one family of input images (objects with homogeneous materials). An exciting direction of research will be to combine and improve on these separate approaches to deal with arbitrary images and scene properties.

  (a) Input image -c Expertissim (b) Shape sharpening (c) Shape rounding

Figure 1 :

 1 Figure 1: Our warping technique takes as input (a) a single image (Jules Bennes, after Barye: "walking lion") and modifies its perceived surface shape, either making it sharper in (b) or rounder in (c).

Figure 4 :

 4 Figure 4: Comparisons between surface-based (a) and image-based (b-k) structure tensors. Rendered images (second row) make use of various Lit Spheres (first row). Structure tensors (third row) are visualized using a combination of two techniques. Colors are expressed in HSV space, with their hue angle corresponding to the angle made by the maximum eigen vector with the horizontal axis, and their opacity corresponding to the maximum eigen value. Directional patterns are obtained by applying a line integral convolution[START_REF] Stalling | Fast and resolution independent line integral convolution[END_REF] to a noise texture in the direction of the maximum eigen vector. Maximum eigen values along a horizontal scanline (bottom row) help compare surface (red curve) versus image features (blue curves). Masking effects occur due to near-constant regions of the Lit Spheres: in (e) due to the brown base shading; in (g) due to the wide highlight; in (h) due to the "sky" region. Directional masking affects the top and bottom of (b) since shading mostly exhibits horizontal variations. Exaggerations are mostly apparent in(h,i,j,k) where shading exhibits hard edges and high frequencies, while distortions are clearly seen in (h,i) (see the shift of the central peak in the bottom row).

Figure 5 :Figure 6 :

 56 Figure 5: A small subset of our statistical analysis data. Top row: object normals and four rendered images using different materials in the same illumination environment. Bottom row: the corresponding surface-based tensor and four image-based tensors. The full dataset contains sixteen rendered images (four materials, four illumination environments) per object and four objects in total.

Figure 7 :

 7 Figure 7: Top: a 3D object rendered with 3 Lit Spheres. Bottom: surface-and image-based scale maps obtained with γ = {0.1|0.7} (the darker the shade, the smaller the selected scale).

Algorithm 1

 1 Image deformation 1: procedure DEFORM(I, α, γ)

Figure 8 :

 8 Figure 8: We illustrate our warping algorithm starting from an input image I in (a). Structure tensors TI is visualized in (b) before and after continuation. Their maximum eigen values λ max I are shown in (c) before and after local adaptation. The warped coordinates W (using α = -1 and guided by ∇λ max I ) are visualized using a distorted grid in (d). They are used to produce the final warped image I • W in (e).

  and minimum eigen values of TI .

Algorithm 2

 2 Warping function 1: procedure WARP(p, ∇λ, σ, α) 2: d ← |α| σ(p) 3: while d ≥ 1 do 4: p ← p + sign(α)∇λ 5: d ← min(d -1, |α| σ(p)) 6: return p + sign(α)∇λ d

IFigure 9 :

 9 Figure 9: Comparisons between surface-based warping (left halves) and image-based warping (right halves). Middle row: an input height field rendered with 4 different Lit Spheres. Top tow: Sharpening of surface shape features (α = -1). Bottom row: Rounding of surface shape features (α = 1). The two types of warping yield qualitatively similar results overall. Their slight differences become more pronounced with higher-frequency shading.

Figure 10 :

 10 Figure 10: We apply surface-based warping to an input height field (b), either sharpening it (using α = -1) in (a) or rounding it (using α = 1) in (c). Its effect on surface normals is shown in the top row. A geometric interpretation is given in the bottom row by reconstructing the height field and taking a horizontal image slice. Height values are shown in black and curvature magnitudes in red.

Figure 12 :

 12 Figure 12: Percentage correct results as a function of the warping amplitude α. Error bars show standard errors of the mean according to the number of participants. In experiment 1 (left), we plot results for image-and surface-based warping separately. In experiment 2 (right), only image-warping results are available. The dashed horizontal line indicates chance level. Subject judgements are well above chance in all conditions in both experiments, which confirms that our warping algorithm produces perceivable shape changes in images.

Figure 11 :

 11 Figure 11: Image-based warping applied to photographs exhibiting different materials and lighting conditions. Each result shows the initial image in the middle, a sharpening result (α = -1) on the left and a rounded result (α = 1) on the right. Warping is only applied to foreground objects. (a) Macro of a lizzard with an intricate scale pattern shot in studio lighting. (b) Metallic helmet photographed indoors. (c) Painted wooden statue shot at the entrance of a budhist temple (partially indoor and outdoor). (d) Wooden statue photograph taken in natural outdoor lighting. (e) Strawberry photographed in studio (collimated) lighting. (f) Macro of a glazed ceramic frog in natural lighting.

Figure 13 :

 13 Figure 13: Comparison between local scale selection (left) and a single global scale (right) on a synthetic example. The guiding flow (top row) better adapts to the size variation of features with our local scale. This has a significant impact on the warped image (bottom row) when compared to the input image (shown above the red line): using a single global scale tends to distort shape features.

Figure 14 :

 14 Figure 14: Variations of the scale bias γ on the photograph of a bronze statue of a woman. The top image shows the input image with its mask. Each column shows rounding (α = 1) at two different scales (γ = {0.1, 0.6}).

Figure 15 :

 15 Figure 15: With large scale bias values (here γ = 1.0), our warping method produces striking face caricatures.

Figure 16 :

 16 Figure 16: Comparisons of our approach with previous work. (a) Input image; (b) shock filter; (c) unsharp masking; (d) our method (sharpening, α = -1): (d) our method (rounding, α = 1).

Figure 17 :

 17 Figure 17: Two example photographs that violate our hypothesis. Top: a transparent object. Bottom: a textured object.

Figure 18 :

 18 Figure18: More results with varying materials, and scale choices (see supplemental document).

Acknowledgements

We would like to thank Pierre Bénard and the anonymous reviewers for their insightful comments on the paper. We are also grateful to Jean-Pierre and Marion Vergne for providing the photographs used in Figures 15, 16 and 17 (top). We thank Paul Debevec for his environment maps used in Figure 4. The photograph used in Figure 1 is courtesy of c Expertissim. Images of Figures 11 (a,e,f), 14, 17 (bottom) and 18 come from pixabay.com (released under Creative Commons CC0). This work was partly supported by the EU Marie Curie Initial Training Network "PRISM" (FP7-PEOPLE-2012-ITN, Grant Agreement: 316746) and by the DFG SFB-TRR-135 "Cardinal Mechanisms of Perception".


Juan-Manuel Pérez-Rúa

Tomas Crivelli

Patrick Bouthemy

Patrick Pérez

Determining occlusions from space and time image reconstructions

The problem of localizing occlusions between consecutive frames of a video is important but rarely tackled on its own. In most works, it is tightly interleaved with the computation of accurate optical flows, which leads to a delicate chicken-and-egg problem. With this in mind, we propose a novel approach to occlusion detection where visibility or not of a point in next frame is formulated in terms of visual reconstruction. The key issue is now to determine how well a pixel in the first image can be "reconstructed" from co-located colors in the next image. We first exploit this reasoning at the pixel level with a new detection criterion. Contrary to the ubiquitous displaced-framedifference and forward-backward flow vector matching, the proposed alternative does not critically depend on a precomputed, dense displacement field, while being shown to be more effective. We then leverage this local modeling within an energy-minimization framework that delivers occlusion maps. An easy-to-obtain collection of parametric motion models is exploited within the energy to provide the required level of motion information. Our approach outperforms state-of-the-art detection methods on the challenging MPI Sintel dataset.

Introduction

Detecting occluded areas at each instant of a video sequence is of utmost interest for many computer vision applications. In fact, even though occlusion detection is mostly associated with the problem of computing inter-image correspondences (optical flow for monocular vision, or disparity map in stereo vision), it is very informative on its own. Among other applications, occlusion-based reasoning has been applied to contour and object tracking [START_REF] Pan | Robust occlusion handling in object tracking[END_REF][START_REF] Yilmaz | Contour-based object tracking with occlusion handling in video acquired using mobile cameras[END_REF], segmentation of multiple objects [START_REF] Wu | Segmentation of multiple, partially occluded objects by grouping, merging, assigning part detection responses[END_REF], action recognition [START_REF] Weinland | Making action recognition robust to occlusions and viewpoint changes[END_REF], pose estimation [START_REF] Wang | Multiple tree models for occlusion and spatial constraints in human pose estimation[END_REF], and depth ordering [START_REF] Smith | Layered motion segmentation and depth ordering by tracking edges[END_REF].

In spite of the usual association between motion field estimation and occlusion detection, it is worth noting that physical motion within the scene by itself does not deter-mine if an element is hidden at a given instant. An additional factor is needed in the equation, that is, the observer point of view, or in other words, the observed 2D visual representation of the real 3D world, i.e., the image. This is a well-known fact and limitation of the optical flow as a representation of physical motion [START_REF] Verri | Motion field and optical flow: Qualitative properties[END_REF]. When working with a succession of discrete-in-time and discrete-in-space 2D images, one can define that a point of a given image is occluded in the next (or other) image of the sequence, if it is not visible by the observer in the latter.

Many state-of-the-art approaches tackle occlusion detection based on the following question: Does an image point have a correspondent in the other image that can be confidently identified as physically identical? In practice, this is indeed evidenced, either implicitly or explicitly, by a wide range of formulations that consider the problem of occlusion detection as inseparable of displacement estimation [START_REF] Alvarez | Symmetrical dense optical flow estimation with occlusions detection[END_REF][START_REF] Fortun | Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow[END_REF][START_REF] Kolmogorov | Computing visual correspondence with occlusions using graph cuts[END_REF][START_REF] Sun | Symmetric stereo matching for occlusion handling[END_REF][START_REF] Xu | A segmentation based variational model for accurate optical flow estimation[END_REF]. This simple question leads however to obstacles in formulating the problem. First, true dense correspondences between images are not easily obtained, especially on occluded pixels where the optical flow (or disparity, likewise) is not well defined. Even in non-occluded areas, rapid changes in appearance and scale make the definition and the estimation of unique correspondences difficult.

We strive for an alternative, more direct, approach to occlusion detection. We adopt an image reconstruction viewpoint that frees us, to a large extent, from the need of jointly estimating an accurate, dense motion field. Instead, we only make use of "plausible" motions simply extracted from the image pair.

The main idea is to assess whether or not pixel appearance can be equally well explained by its spatial neighborhood in the same frame and by suitable co-located pixels in the other frame. If not, this point is likely to be occluded in the next image. In this way, occlusion detection can be sought as independent of the knowledge of an accurate, univocal motion field between the two images. It suffices to exploit loosely the spatiotemporal coherency in order to select a suitable spatiotemporal neighborhood. The paper is organized as follows. In Section 2 we discuss relevant literature and motivate the need for a different approach to occlusion detection. We then devote Section 3 to introducing the ideas behind our novel approach, whose formulation is given in Section 4. We report experimental results, including extensive comparisons to recent occlusion detection methods, in Section 5. We provide concluding remarks in Section 6.

On true motion and occlusion models

Seeing occlusion detection as only part of a joint motionocclusion problem requires (1) to model accurately visual motion and (2) to model occlusions in the light of this motion. As we will show, even if the true motion field is known, approaches of this type might fail to produce accurate occlusion maps. As a consequence, we argue that motion should remain an auxiliary variable and not the main object of interest. This idea deeply contrasts with the reasonings one can find in the literature.

A popular idea is that an occlusion is a violation of the optical flow constraint: "Occlusions generally become apparent when integrated over time because violations of the brightness-constancy constraint of optical flow accumulate in occluded areas" [START_REF] Estellers | Detecting occlusions as an inverse problem[END_REF].

Other authors make similar claims, without referring to optical flow integration but stating instead that, under Lambertian reflections and the constant illumination assumption, a brightness change between corresponding points indicates an occlusion of the point in the second image [START_REF] Ayvaci | Sparse occlusion detection with optical flow[END_REF][START_REF] Xiao | Bilateral filtering-based optical flow estimation with occlusion detection[END_REF].

Another assertion is that flow errors occur in occlusion areas or that an occlusion is an explanation of motion mismatching: ". . . the most probable reasons for such a situation [flow mismatching] is an occlusion problem or an error in the estimated matching" [START_REF] Alvarez | Symmetrical dense optical flow estimation with occlusions detection[END_REF]. This notion has also been exploited by other works [START_REF] Fortun | Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow[END_REF][START_REF] Proesmans | Determination of optical flow and its discontinuities using non-linear diffusion[END_REF][START_REF] Yang | Dense, accurate optical flow estimation with piecewise parametric model[END_REF] where forwardbackward flow inconsistency is used to detect occlusions.

Similarly to the brightness conservation constraint, some authors have proposed that a point is occluded if it switches from one motion layer or segment to another between consecutive frames: "To consistently reason about occlusions, we examine the layer assignments [of two points in consec-utive frames] at locations corresponded by the underlying flow fields" [START_REF] Sun | Layered image motion with explicit occlusions, temporal consistency, and depth ordering[END_REF]. In our opinion, this argument falls short for non-planar motions and self-occlusions even when optical flow is allowed to deviate from the assumed parametric motion as in [START_REF] Sun | Layered image motion with explicit occlusions, temporal consistency, and depth ordering[END_REF]. Ambiguities on occlusion estimation from layer assignment are alleviated by enforcing temporal layer consistency [START_REF] Sun | Layered segmentation and optical flow estimation over time[END_REF].

Relying on a joint motion-occlusion estimation leads to a chicken-and-egg situation, which is handled in alternation: "[The algorithm] iterates between estimating values for [occlusion map], and optimizing the current optical flow estimates by differential techniques" [START_REF] Strecha | A probabilistic approach to large displacement optical flow and occlusion detection[END_REF].

Other approaches include the uniqueness criterion [START_REF] Brown | Advances in computational stereo[END_REF], which is known for producing a large amount of false positives [START_REF] Xu | Motion detail preserving optical flow estimation[END_REF], and combinations of the criteria explained above. For instance, [START_REF] Ince | Occlusion-aware optical flow estimation[END_REF] considers flow symmetry in combination with the violations of the optical flow brightness constancy constraint. Another example is [START_REF] Sun | Symmetric stereo matching for occlusion handling[END_REF] which enforces disparity consistency by labeling points that cannot be matched with another point in the second image as occluded, while proposing a sequential approach that computes occlusion and disparity iteratively. Different views on the related problem of finding occlusion borders, but without determining exactly the occlusion regions are found in [START_REF] Apostoloff | Learning spatiotemporal t-junctions for occlusion detection[END_REF][START_REF] Jacobson | An online learning approach to occlusion boundary detection[END_REF][START_REF] Raza | Finding temporally consistent occlusion boundaries in videos using geometric context[END_REF][START_REF] Sargin | Probabilistic occlusion boundary detection on spatiotemporal lattices[END_REF][START_REF] Stein | Occlusion boundaries from motion: Low-level detection and mid-level reasoning[END_REF][START_REF] Sundberg | Occlusion boundary detection and figure/ground assignment from optical flow[END_REF]. Doubtless, the underlying motion is indeed helpful to find occlusions. This is further confirmed by the work of Humayun et al. [START_REF] Humayun | Learning to find occlusion regions[END_REF] where a large amount of features were used to train a random forest, giving a high importance to motion-based features. In the same way, knowing the occlusion labeling of image pixels clearly helps motion estimation. Specifically, it better guides regularization and smoothing. Many optical flow methods try to deal with occlusions by embedding a discrete state into a continuous numerical scheme [START_REF] Ballester | A TV-L1 optical flow method with occlusion detection[END_REF], or an aggregation framework [START_REF] Fortun | Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow[END_REF], ending with complex, usually joint formulations that improve motion estimation. Similarly, stereo vision approaches that are formulated as discrete label-selection problems can be naturally extended to handle occlusions by adding a label for the occlusion state [START_REF] Kang | Handling occlusions in dense multi-view stereo[END_REF][START_REF] Kolmogorov | Computing visual correspondence with occlusions using graph cuts[END_REF], relying on efficient discrete energy optimization techniques. In order to pin down our claims, let us assume that we know the true motion field for an image pair 1 and let us analyze the most common underlying reasonings for occlusion detection. Figure 1 shows results for the occlusion detection by finding violations of the color constancy assumption along this motion, that is by thresholding the so-called displaced frame difference (DFD). If x and x are locations on image grid Ω of two truly corresponding pixels in color images I 1 and I 2 respectively, x is declared occluded if

I 1 (x) -I 2 (x ) > ε c , (1)
where ε c is a threshold, which is proposed with variants in [START_REF] Ayvaci | Sparse occlusion detection with optical flow[END_REF][START_REF] Xiao | Bilateral filtering-based optical flow estimation with occlusion detection[END_REF]. Even though the image sequence in this example does not contain significant illumination changes nor extra post-processing effects like mist or motion blur, the color constancy criterion is not robust enough to detect occlusions even if the true motion is known. This is easily verified visually by looking at regions where the norm of the color difference does not have large enough values even across occluded areas (Fig. 1).

A similar experiment can demonstrate that surprisingly, even when the true optical flow is available, the forwardbackward flow consistency criteria might fail to accurately capture the real occlusion map (Fig. 2). This criterion assumes that given corresponding points x and x and their associated forward and backward flows, [START_REF] Alvarez | Symmetrical dense optical flow estimation with occlusions detection[END_REF][START_REF] Ben-Ari | Variational stereo vision with sharp discontinuities and occlusion handling[END_REF][START_REF] Fortun | Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow[END_REF]. Implementation-wise, the backward flow at x is obtained by bilinear interpolation, since x is generally not an integer grid position. Such a detail by itself generates erroneous flow mismatching which may be enlarged in different situations such as motion discontinuities or zooming. Even if ground-truth motion is available at image grid points, w b (x ) may introduce a position drift while going backwards to the first image. Figure 2 shows occlusion maps obtained by this criterion for a syn- 1 In the sense of the projection of the true physical motion onto the image plane. thetic zoom. The errors in the occlusion map can be explained by the grid discretization of the flows, making the forward-backward flow difference grow in several zones of the non-occluded area (whiter pixels in Fig. 2), and leading to a large number of false positives.

w f (x) and w b (x ), x is declared occluded if w f (x) + w b (x ) > ε f , where ε f is a threshold
As the aforementioned two criteria are the most commonly used [START_REF] Alvarez | Symmetrical dense optical flow estimation with occlusions detection[END_REF][START_REF] Ayvaci | Sparse occlusion detection with optical flow[END_REF][START_REF] Ben-Ari | Variational stereo vision with sharp discontinuities and occlusion handling[END_REF][START_REF] Fortun | Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow[END_REF][START_REF] Xiao | Bilateral filtering-based optical flow estimation with occlusion detection[END_REF], the amount of errors they can lead to should not be neglected. Reducing the dependency of occlusion detection on flow quality, as we propose, is one answer to this problem.

From image reconstructions to occlusion

We consider that the property of being occluded is intrinsic to each one of the points of an image. This means that detecting occlusions may be posed as an independent problem, and not necessarily strongly attached to a per-pixel estimation of motion. We start from the standard concept of an occluded point, represented in the image space through the simplified concept of a pixel, as one that is visible in a first image and not visible in a second image.

A visible to non-visible transition implies a loss of information between the two images. This means that there is a pixel in the first image that cannot be explained using the second image. At a larger scale, to pin things down, suppose a well-defined object present at one instant. The question we ask is: Can the visual information carried by the object be plausibly explained or "reconstructed" by visual data from the second image? Failing to perform this reconstruction implies an absence of information and thus, a disappearing object. Occlusion detection can then be defined as a spatiotemporal reconstruction problem.

For this problem to be well-posed there are two main issues. First, quality of the visual reconstruction has to be assessed with respect to a reference information. Incidentally, the true reconstruction is available here, and is precisely the same first image! Note that we reason directly on the quality of the reconstruction (with known reference) rather than indirectly on the quality of motion (unknown field). Sec-ond, the reconstruction should be plausible, meaning that the way we pull information from the second image to reconstruct the first, must be consistent with apparent scene changes. Without the latter condition, one can get away with physically improbable information flows.

We start by focusing first aspect of the problem, that is, how to construct an occlusion criterion able to reason on the basis of image reconstruction. The criterion itself assumes that the reconstruction is plausible and consistent. A plausible reconstruction from a pair of images can be generated given a plausible correspondence motion field between them. This defines a natural way of pulling information from one image towards the other. A non-plausible reconstruction would be one that propagates color information between points that do not physically relate. Such a reconstruction is not necessarily useless, as many problems in image processing are not interested in the interpretation of the correspondence itself, like motion-compensated image compression, nearest neighbor search (e.g. [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF]) or video denoising.

This puts forward the fact that motion indeed intervenes, leading us to the second aspect of the problem. We propose a complete framework for occlusion map estimation which exploits dynamic idiosyncrasies of the scene of interest. Indeed, the plausibility of the reconstruction (loosely, how probable it is) does not demand accuracy in motion estimation nor a hard decision on which is the optimal correspondence vector.

Proposed occlusion detection

We start by defining a reconstruction-based criterion for independently detecting occluded pixels (Section 4.1), assuming the knowledge of a correspondence map. In Section 4.2, we then leverage this new local model within an image-wise formulation of occlusion detection, where instrumental correspondences are obtained from a collection of suitable parametric motion models. No accurate optical flow is thus required while searching the best binary occlusion labeling over the image.

A reconstruction-based criterion

Let us define two functions ζ(I) and η(I ; I, w) that provide two different reconstructions of the same image I, either from itself (intra-image reconstruction) or from another related image I , a correspondence field between the two, w, being given (inter-image reconstruction under correspondence guidance). Given a pair (I 1 , I 2 ) of successive video frames and some correspondence field w from the first image to the second one, we shall denote in short ζ 1 = ζ(I 1 ) and η 1,2 = η(I 2 ; I 1 , w).

In essence, η 1,2 = η 1,2 (x) x∈Ω conveys the appearance of image I 1 that is retained by the second image. As such, under the true motion field, η 1,2 is expected to deviate towards the appearance of I 2 for all the occluded points. This particular behavior is clearly visible in Fig. 3, where segments with largest motions appear doubled in a stroboscopic-like effect. On the other hand, ζ 1 captures the intrinsic appearance of I 1 revisited from its own perspective for every x on the image grid Ω.

This means that, if the two functions are defined in a suitable way, one could deduct whether a pixel at location x in the first image of the pair is visible or not in the second one by comparing ζ 1 and η 1,2 around this location. Experiments showed that this comparison is better conducted in an asymmetric way whereby a local color model g (defined later in Eq. 6) is fitted to ζ 1 in the neighborhood of x and used to assess the likelihood of η 1,2 (x) under visibility hypothesis.

Reasoning independently at the pixel level for now, point x will be considered as not visible in the second image if:

-ln g η 1,2 (x) > ε v , (2)
where ε v is a conveniently chosen threshold, and g an exponential density function.

The function ζ aims mostly at "simplifying" the first input image I 1 such that robust comparisons can be conducted later on. Yet, it is important to preserve the structure of the input image. A natural choice for this function is the classic bilateral filter [START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF]

]: 2 ζ 1 (x) = 1 Z(x; I 1 ) y∈Nx α(x, y; I 1 )I 1 (y), (3)
where N x is a square window centered at x, Z(x; I 1 ) = y∈Nx α(x, y; I 1 ) is a normalization factor and the weighting function α depends on both appearance and spatial proximity within pixel pair:

α(x, y; I 1 ) = f a ( I 1 (y) -I 1 (x) )f s ( y -x ), (4)
with f a and f s being Gaussian kernels.

In a similar fashion, η 1,2 will form a structure-preserving reconstruction of the first image I 1 but, this time, using colors from the second image I 2 under the guidance of correspondence map w:

η 1,2 (x) = 1 Z(x; I 1 ) y∈Nx α(x, y; I 1 )I 2 y + w(y) . (5)
This can be seen as a "displaced cross-bilateral filter", that is, the cross-bilateral filtering [START_REF] Kopf | Joint bilateral upsampling[END_REF] of a warped image. Note that, as previously stated, we make use of a correspondence map only as a tool to find a valid reconstruction of I 1 .

It is important that the two reconstruction functions share the same filter weights. 3 This way, η 1,2 captures as much as possible the local structure of I 1 and both reconstructions are comparable pixel-wise. Intentionally, this structure-mimicking behavior is not favorable for reconstructing points that are visible only in the first image, i.e., occluded points. The next step in the procedure consists in assessing whether a point x is occluded or not based on the criterion defined in [START_REF] Alvarez | Symmetrical dense optical flow estimation with occlusions detection[END_REF]. In order to conduct this step in practice, we propose here to over-segment the first reconstructed image into homogeneous segments where meaningful local color models can be estimated. In our experiments, these homogeneous segments are SLIC superpixels [START_REF] Achanta | SLIC superpixels compared to state-of-the-art superpixel methods[END_REF], and a Gaussian Mixture Model (GMM) of color is extracted for each of them, defining the density g in (2).

Image ζ 1 is segmented into J super-pixels and the j-th one, S j ⊂ Ω, is equipped with the mixture:

g j = Kj k=1 π j k G(µ j k , Σ j k ), (6)
where π j k , µ j k and Σ j k are respectively the weight, the mean and the covariance matrix of the k-th component of the mixture. These GMMs provide good local models of ζ 1 in the sense that one can assume that

∀x ∈ Ω, ζ 1 (x) ∼ g s(x) , (7)
where s(x) ∈ 1, J is the index of super-pixel containing x. This should also hold for η 1,2 (x) provided the point is not occluded in second image. The novel reconstructionbased test for occlusion detection at the pixel level (2) finally reads:

x occluded if -ln g s(x) η 1,2 (x) > ε v . (8)
Contrary to DFD-based test (1), this one is not based on a point-to-point comparison but on a point-to-local-model one. We shall demonstrate experimentally that it is a more powerful alternative. Yet, as DFD-based test, it still assumes that a correspondence map is available to produce the temporal image reconstruction η 1,2 = η(I 2 ; I 1 , w). Next, we explain how to use this novel modeling over the whole image without depending on a single, accurate motion field.

From motion models to occlusions without stopping by optical flow

We propose a method for detecting occlusions which uses the image-reconstruction reasoning explained above. In order to be agnostic to optical flow computation, we propose to rely on a collection of motion models that spans the various dynamics of the scene and thus enables plausible image reconstructions. As classically exploited in video segmentation and analysis, the apparent motion at work in natural dynamic scenes can often be decomposed into a set of low-complexity models, typically region-wise affine models. Such a paradigm recently proved useful also to estimate dense optical flows [START_REF] Yang | Dense, accurate optical flow estimation with piecewise parametric model[END_REF]. In our case, such models will provide candidate correspondences at each pixel, leading to a discrete labeling problem intertwined with the main one of occlusion detection.

We start by computing a set W = {w k , k = 1 • • • K} of K parametric motion models that are relevant to different sub-regions of the scene. We extract a large number K of overlapping windows of different sizes, starting with a window encompassing the full image support, and subsequently reducing the size by a half and changing position of the windows with a fixed overlap factor of 50%, covering the whole image for every window size. For the image size of the Sintel dataset, with four levels of window sizes we obtain, for instance, K = 115 windows.

For each window, we robustly estimate a parametric warp that captures at best the motion of corresponding scene fragment. Several classic techniques can be used to this end. In our experiments, we combine semi-dense matching, to handle large displacements, with robust affine motion estimation. We first extract point matches between images I 1 and I 2 with DeepMatching4 [START_REF] Revaud | DeepMatching: Hierarchical deformable dense matching[END_REF] and fit an initial affine motion to the matches originated from the window of interest. These models are then refined with Motion2D5 , an M-estimator relying on all support intensities [START_REF] Odobez | Robust multiresolution estimation of parametric motion models[END_REF]. This multi-window motion estimation approach arguably provides a partially redundant collection of motion models, but it is simple and it circumvents in particular the intricate problem of motion segmentation. From the set W of K parametric motion models thus obtained, we want to exploit the most plausible at each pixel to achieve occlusion detection. Observe that with this procedure, pixel-wise occlusion modeling is not tied to a single, accurate, dense optical flow, but rather to a region-wise characterization of the scene dynamics.

The task to solve is now the one of jointly selecting a motion model and deciding on visibility for each pixel. We pose it as an energy minimization problem with respect to a motion model labeling M = m(x) x∈Ω ∈ 1, K Ω and an occlusion map O = o(x) x∈Ω ∈ {0, 1} Ω , where 1 means occluded point. For pixel location x, label m(x) indicates which one of the available parametric motion models is relevant, while o(x) establishes if there is an occlusion or not. For m(x) = k, the associated inter-image reconstruction ( 5) is denoted η k 1,2 (x). The joint energy to minimize is defined as:

E(O, M ) = x∈Ω φ x (o(x), m(x)) + DL(M )+ (9) x∼y ψ m x,y (m(x), m(y)) + ψ o x,y (o(x), o(y))
where the second sum is taken over all pairs of neighboring pixels. The unary potential reads

φ x (0, k) = -ln g s(x) (η k 1,2 (x)), φ x (1, k) = α v , (10
) where α v > 0 is the cost of labeling a single pixel as occluded. It is related to ε v in pixel-wise test [START_REF] Boykov | Fast approximate energy minimization via graph cuts[END_REF]. This data term is thus not based on point-wise displaced frame differences as classically done, but on reconstruction-based local modeling. Although this modeling effectively penalizes motions that do not preserve color information up to the precision of the local model, this data-term, as seen from the unknown motion point-of-view, would not be suitable to yield an accurate pixel-wise motion estimation. Again, this is not the intention anyway, the sole aim being to reason locally on as plausible as possible inter-image reconstructions. This is further analyzed in Section 5.

Since each motion label k corresponds to a specific image window, one could restrict the labeling of pixel x according to the windows it belongs to. We found, however, that this restriction can cause block-like artifacts in the label assignment, with subsequent damage to final occlusion labeling. To capture motion model locality in a less drastic way, we propose instead to double φ x (0, k) (unary potential for visible points) for motion models k stemming from windows x does not belong to.

The binary potentials share a similar form of contrast-sensitive smoothing:

ψ a x,y (k, k ) = λ a exp (-β a I 1 (x) -I 1 (y) ) [k = k ] (11)
with a ∈ {"o", "m"}, where [•] is Iverson bracket and λ o , λ m are positive parameters.

Finally, the global motion label cost DL(M ) penalizes the complexity of the labeling through its "description length", i.e., the number of labels effectively used:

DL(M ) = λ c {k : ∃x ∈ Ω, m(x) = k}. (12)
The occlusion map O and, as a by-product, the motionmodel label map M , are obtained by minimizing E(O, M ) (Eq. 9) with a block coordinate descent in an alternate way. Given occlusion assignment O, minimizing E(O, M ) w.r.t. M only amounts to minimizing

x∈Ω φ x (o(x), m(x)) + x∼y ψ m
x,y (m(x), m(y)) + DL(M ).

(13) This can be done approximately by using α-expansions with the method of [START_REF] Delong | Fast approximate energy minimization with label costs[END_REF] in order to handle the global label cost term. Subsequently, for a given motion model label map, the occlusion map can be recovered by minimizing w.r.t. O the following function:

x∈Ω φ x (o(x), m(x)) + x∼y ψ o x,y (o(x), o(y)), (14)
with graph-cuts [START_REF] Boykov | Fast approximate energy minimization via graph cuts[END_REF]. The occlusion and label maps are alternatively updated for a small number of iterations. Recall that this process is not oriented at optical flow recovery, but at selecting plausible motion models.

Experimental results

For the quantitative evaluations reported in this section, we rely on the MPI Sintel dataset6 [START_REF] Butler | A naturalistic open source movie for optical flow evaluation[END_REF]. This dataset comprises 69 sequences from the open-source CGI movie Sintel 7 for which ground-truth optical flows and occlusion maps have been computed from the known 3D dynamic structure of the scenes. For all the experiments, we fixed the number of super-pixels J to 700. We found experimentally that having only 2 components for each GMM provides good results in general. Furthermore, the standard deviation of the Gaussian kernels was set to 1.0, with a window size of 5 pixels. These parameters are related to the local color model variability within the superpixel supports.

Evaluation of the occlusion criterion

Comparisons are conducted against DFD-based detection, which is at the heart of approaches based on analyzing violations of the brightness constancy assumption [START_REF] Ayvaci | Sparse occlusion detection with optical flow[END_REF][START_REF] Xiao | Bilateral filtering-based optical flow estimation with occlusion detection[END_REF]. For eight sequences of the dataset, we report in Fig. 4 occlusion detection ROC curves (true-positive-rate (TPR) vs. false-positive-rate (FPR)) and area under the ROC curves (AUC), both varying the threshold ε c in the rule (1) and varying ε v in our own framework [START_REF] Boykov | Fast approximate energy minimization via graph cuts[END_REF]. It can be seen that our reconstruction-based criterion has more discriminative power than classic DFD criterion in the prospect of occlusion detection. In all regimes and on all the sequences the former outperforms the latter, often by a substantial margin. More interestingly, for most of the sequences, i.e., "alley 1", "alley 2", "ambush 4", "ambush 5", "ambush 7" and "bamboo 1", the decrease in occlusion detection performance due to flow inaccuracies (with noisy ground-truth or DeepFlow estimate) is less significant for our method. Remarkably, when used together with DeepFlow, our criterion outperforms on several sequences the DFD criterion based on true motion field.

Full system evaluation

We now turn to the complete minimization-based method introduced in Section 4.2. We compare it on MPI Sintel sequences to several recent approaches: (1) a learning-based occlusion detection algorithm, which uses a non trivial ensemble of hand-designed features [START_REF] Humayun | Learning to find occlusion regions[END_REF], including forward-backward flow inconsistencies for several optical flow methods; (2) a method based on layer assignment and depth ordering [START_REF] Sun | Layered image motion with explicit occlusions, temporal consistency, and depth ordering[END_REF]; (3) a method that leverages reasoning on local layers relationships [START_REF] Sun | Local layering for joint motion estimation and occlusion detection[END_REF]; (4) a sparse occlusion detection method that relies on departures from the optical flow color constancy assumption [START_REF] Ayvaci | Sparse occlusion detection with optical flow[END_REF].

In the experiments, we limit the number of iterations of our alternate minimization method to 2, as it converges Table 1 summarizes results by the average F-score computed over all 69 ground-truth MPI Sintel sequences, using two ways of setting the main occlusion parameter for each method, e.g., α v for our method. In a first set of experiments ("Global 69" column), it is manually set at once for all sequences. For [START_REF] Sun | Local layering for joint motion estimation and occlusion detection[END_REF] and [START_REF] Humayun | Learning to find occlusion regions[END_REF], the corresponding parameters are set to 0.5, as reported by [START_REF] Sun | Local layering for joint motion estimation and occlusion detection[END_REF]; for our method α v is set to 10; finally, for [START_REF] Ayvaci | Sparse occlusion detection with optical flow[END_REF] 9 and [30]10 we set the parameters to the values proposed by the respective authors. In a second round ("Oracle 69" column), we tuned the key parameter of each method so as to maximize the F-scores. As it can be appreciated, our reconstruction-based method outperforms all the other reported methods in both settings. Finally, we also present some results (as available) by tuning the parameters to maximize the F-scores only in the 23 training sequences on the final rendering pass of the Sintel dataset ("Oracle Final" column).

We provide in Fig. 5 several samples of our results for visual inspection. Despite the complexity of some of these scenes, occlusion maps of good quality are obtained. In particular, extended occlusions produced by large displacements are very well captured, while retaining some details of smaller occluded regions. It is also interesting to examine associated motion model labelings M = m(x) x∈Ω and associated motion flows w m(x) (x) x∈Ω (Fig. 5 ef). While motion labels define segments that relate well to actual moving regions (e.g. the arm of the woman in the fourth sequence or the whole person in the first one), the flows are not very accurate for all the sequences. This is not surprising since source motion models have been estimated beforehand over arbitrary image windows. Motion model selection with no further processing cannot produce accurate optical flows. This reveals, as already highlighted in Section 4.2, that the distinctive nature of our framework Table 1. Quantitative comparisons on MPI Sintel dataset. For each method, the average F-score (the higher, the better) is computed with two different ways of setting the main detection parameter over all the available training sequences and rendering passes. We also present results on the Final rendering pass.

Detection method Oracle 69 Global 69 Oracle Final

Learning [START_REF] Humayun | Learning to find occlusion regions[END_REF] 0.535 0.448 -Depth order [START_REF] Sun | Layered image motion with explicit occlusions, temporal consistency, and depth ordering[END_REF] is to focus on good reconstruction-based modeling of occlusion, with inference requiring only plausible correspondences, not accurate per-pixel optical flow. The local color modeling, however, carries a few problems that somewhat limit the performance of our method. For example, GMMs easily overfit in textureless regions, causing false positives under slight temporal illumination variations. Regarding execution times, for a sample image pair of size 436×1024, our full method takes about 1212 s (29 s for the multi-window motion estimation and 1183 s for the energy minimization) on an Intel i7-3540M CPU @ 3.00GHz. In comparison, [START_REF] Ayvaci | Sparse occlusion detection with optical flow[END_REF] takes 1601 s, while [START_REF] Sun | Layered image motion with explicit occlusions, temporal consistency, and depth ordering[END_REF] takes 2201 s on the same machine.

Finally, we show in Fig. 6 additional comparative results on real-world image sequences. Qualitative assessment of these results further confirm the merit of our approach: more accurate occlusion maps are produced compared to [START_REF] Ayvaci | Sparse occlusion detection with optical flow[END_REF] and [START_REF] Sun | Layered image motion with explicit occlusions, temporal consistency, and depth ordering[END_REF].

Concluding remarks

We have introduced a new approach to detect occlusions that occur from one frame to another in a video sequence. Departing from classic approaches that tightly link this task to the accurate computation of optical flow, we propose a local spatio-temporal reconstruction model that only requires the knowledge of a plausible motion. Given a collection of parametric motion models simply extracted from the scene at hand, we show how this local reconstruction model can be harnessed within a global energy function to deliver high quality occlusion maps.

Quantitative experiments yielded two important findings: (1) Proposed reconstruction-based modeling provides a detection criterion at the pixel level that is a powerful alternative to classic criterion based on intensity inconsistency along the flow. Even when exploiting the true flow, later criteria perform less well that ours with the output of an off-the-shelf flow estimator. (2) Our complete energybased framework consistently outperforms state-of-the-art approaches on MPI Sintel dataset. More generally, we qualitatively observed the ability of our framework to produce good quality occlusion detection maps, even in scenes comprising large occluded regions and complex motions.

Figure 1 .

 1 Figure 1. Occlusions from color inconsistencies along true flow. (a-b) Two successive frames of the clean ambush 2 sequence from the MPI Sintel dataset [10]. (c) Occlusion ground truth with occlusions shown in white; (d) Norm of the color difference between points matched by the true flow (2D projection of the known 3D motions); (e)-(h) Occlusion maps obtained by thresholding at 0.015, 0.050, 0.100 and 0.250 respectively, the min-max normalized (between 0 and 1) color differences along the flow. They are all unsatisfactory (low precision and/or low recall).

Figure 2 .

 2 Figure 2. Fictitious occlusions from forward-backward inconsistencies of divergent flows. (a)-(b) Forward-backward divergent flows associated to a zoom-in (classic hue-saturation color coding at the pixel level, and subset of motion vectors superimposed for better visualization). (c) True occlusion map, devoid of occlusions; (d) Norm of the forward-backward flow difference between points matched by the true flow. Because of interpolations required to evaluate backward flows at non-pixel positions, differences are space dependent (e)-(h) Occlusion maps obtained by thresholding at 0.100, 0.333, 0.500 and 0.900, respectively, the normalized forward-backward differences along the flow. For better visualization of the error pattern, we zoom-in the resulting images (bottom-left corner). False positives occur all over the image grid.

Figure 3 .

 3 Figure 3. Pipeline for proposed occlusion detection criterion. Given successive images I1 and I2, functions ζ and η generate two "reconstructions" ζ1 and η1,2 of I1. The second reconstruction is obtained from I2, under the guidance of a given motion field and of I1 (to preserve the structures of it). An arbitrary window of an occluded zone is zoomed-in for inspection. The likelihood of the color at each pixel of η1,2 is evaluated under the corresponding local model extracted from ζ1 at the super-pixel level (point-to-local-color-model comparison). This provides a soft-occlusion map that can be either thresholded pixel-wise to obtain a binary occlusion map, or embedded in the unaries of a joint labeling cost function (not shown here).

  Let us first demonstrate the value of the reconstructionbased criterion introduced in Section 4.1 by performing pixel-wise occlusion detection under the guidance of three different correspondence fields: (1) The true optical flow, as accessible in MPI Sintel sequences; (2) The true flow contaminated by independent additive Gaussian noise of standard deviation 2.5; (3) The flow estimated with Deep-Flow 8 [38], a state-of-art optic flow estimator which does not handle occlusions but solves for long displacements.

Figure 4 .

 4 Figure 4. Quantitative comparison of proposed reconstruction-based criterion and classic DFD-based criterion. Occlusion detection ROC curves and associated AUC on eight sequences of the MPI Sintel dataset, obtained by varying the threshold of proposed criterion (solid lines) and of DFD criterion (dashed lines). Colors indicate the origin of the motion field: true optical flow (green), true optical flow contaminated by Gaussian noise (blue), DeepFlow estimate (red).

Figure 5 .

 5 Figure 5. Qualitative results of our occlusion detection method on several MPI Sintel scenes. From left to right: Average of the two input frames; Occlusion ground-truth; Occlusion map with proposed method; Final motion model labeling, where each color loosely represents the size and position of the window linked to the selected motion model through a jet-map colorization (Big to small and from upper-left to bottom-right); Optical flow obtained by evaluating the selected motion model at each pixel (with classic hue-saturation color coding); And true optical flow with same color coding.

Figure 6 .

 6 Figure 6. Qualitative comparison with state-of-the-art on real images. From left to right: Average of the two input frames, final occlusion map with proposed method, results of [4], and of [30]. quickly. Furthermore, we set λ o = 20.0, λ m = 33.3, β o = 0.1, β m = 0.2, and λ c = 10 3 by hand-tuning. Setting critical parameter α v is discussed below.Table1summarizes results by the average F-score computed over all 69 ground-truth MPI Sintel sequences, using two ways of setting the main occlusion parameter for each method, e.g., α v for our method. In a first set of experiments ("Global 69" column), it is manually set at once for all sequences. For[START_REF] Sun | Local layering for joint motion estimation and occlusion detection[END_REF] and[START_REF] Humayun | Learning to find occlusion regions[END_REF], the corresponding parameters are set to 0.5, as reported by[START_REF] Sun | Local layering for joint motion estimation and occlusion detection[END_REF]; for our method α v is set to 10; finally, for[START_REF] Ayvaci | Sparse occlusion detection with optical flow[END_REF] 9 and [30] 10 we set the parameters to the values proposed by the respective authors. In a second round ("Oracle 69" column), we tuned the key parameter of each method so as to maximize the F-scores. As it can be appreciated, our reconstruction-based method outperforms all the other reported methods in both settings. Finally, we also present some results (as available) by tuning the parameters to maximize the F-scores only in the 23 training sequences on the final rendering pass of the Sintel dataset ("Oracle Final" column).We provide in Fig.5several samples of our results for visual inspection. Despite the complexity of some of these scenes, occlusion maps of good quality are obtained. In particular, extended occlusions produced by large displacements are very well captured, while retaining some details of smaller occluded regions. It is also interesting to examine associated motion model labelings M = m(x) x∈Ω and associated motion flows w m(x) (x) x∈Ω (Fig.5 ef). While motion labels define segments that relate well to actual moving regions (e.g. the arm of the woman in the fourth sequence or the whole person in the first one), the flows are not very accurate for all the sequences. This is not surprising since source motion models have been estimated beforehand over arbitrary image windows. Motion model selection with no further processing cannot produce accurate optical flows. This reveals, as already highlighted in Section 4.2, that the distinctive nature of our framework

Any other discontinuity-preserving image filter could be used, provided it possesses a guided version.

[START_REF] Apostoloff | Learning spatiotemporal t-junctions for occlusion detection[END_REF] In particular, we have the desirable property ζ(I) = η(I; I, 0).

http://lear.inrialpes.fr/src/deepmatching/

http://www.irisa.fr/vista/Motion2D/

http://sintel.is.tue.mpg.de/downloads

https://durian.blender.org/download/

http://lear.inrialpes.fr/src/deepflow/

http://vision.ucla.edu/ ˜ayvaci/spaocc.html

We thank authors for providing us with their source code.