Sergi Pujades

Frédéric Devernay

Laurent Boiron

Rémi Ronfard

The Stereoscopic Zoom

We study camera models to generate stereoscopic zoom shots, i.e. using very long focal length lenses. Stereoscopic images are usually generated with two cameras. However, we show that two cameras are unable to create compelling stereoscopic images for extreme focal length lenses. Inspired by the practitioners' use of the long focal length lenses we propose two different configurations: we "get closer" to the scene, or we create "perspective deformations". Both configurations are build upon state-of-the-art image-based rendering methods allowing the formal deduction of precise parameters of the cameras depending on the scene to be acquired. We present a proof of concept with the acquisition of a representative simplified scene. We discuss the advantages and drawbacks of each configuration.

Introduction

Now that technical progress has made 3D cinema and television a reality, artists should be able to explore new narratives, which take advantage of the optical illusion of depth from stereopsis in the storytelling.

"Zooming" means the change of a lens' focal length. Lenses are either "prime" (fixed focal length) or "zoom". Most lenses equipped with a long focal length are zooms, because the cameraman needs to adjust the focal length to create the desired image frame. Thus in the paper we (ab)use the word "zoom" to refer to a long focal length lens.

Zoom is one of the main limitations when shooting stereoscopic footage [25,9]. The limitation arises from the incompatibility between the configuration avoiding ocular divergence, known to be a major cause of visual fatigue [40,37], and the configuration avoiding the "cardboard effect", known to create a poor viewing experience [START_REF] Yamanoue | Geometrical analysis of puppet-theater and cardboard effects in stereoscopic HDTV images[END_REF]25].

Yet zooms provide two opportunities to 2D cinematographers. The first is to get closer to the scene. Sometimes it is not possible to place the camera at the desired position, like for instance, when filming a polar bear in the wild. The second is aesthetic: it is well known that different focal lengths distort the perspective, and directors take advantage of these distortions to convey emotions. One of the most famous example in 2D is the vertigo effect, created by Alfred Hitchcock in 1958 in his feature film Vertigo. In stereoscopic movies too, directors should be given the opportunity to play with the perspective distortions at will to create new narratives yet to be invented.

Overview. In Sec.2 we introduce the notation and concepts related to stereoscopic filming. We illustrate the problems arising when filming with long focal lenses. In Sec. 3 we review the limitations of the existing state of the art methods. In Sec. [START_REF] Buyssens | Depth-aware patch-based image disocclusion for virtual view synthesis[END_REF] we present the simplified scene we use and in Sec. 5 and 6 we contribute two new camera models. One to get closer to the scene and one to add perspective deformations to the scene. In Sec. 7 we present a proof of concept for each camera model and in Sec. 8 we compare both configurations and conclude.

Problem Statement

Perceived Depth from Stereopsis. In Fig. 1 we introduce the parameters characterizing at the same time an acquisition stereo system and a projection stereo system [START_REF] Devernay | Stereoscopic cinema[END_REF]. In the acquisition setup, the distance between the optical centers of the camera b is the baseline. The cameras' convergence distance is H, and the plane parallel to the images at distance H is the convergence plane. The intersection of the camera visibility frustums with the convergence plane defines the convergence window. Its width is W . In the projection setup, the distance between the eyes of the spectator is b . The distance between the spectator and the screen is H and the width of the screen is W . We assume b, b , H, H , W and W are > 0. Given the acquisition parameters b, H, W, we can compute the normalized disparity d of an element at depth z, and given d and the projection 1 parameters b , H , W , we can compute the perceived depth from stereopsis z :

H z C l C r W P b M l M r W × d H z C l C r W P b M l M r W × d Symbol Acquisition Projection C l ,
d(z) = b W (z -H) z and z (d) = b H b -W d . (1)
Combining both terms we can compute the relationship between the true depth in the 3D scene z and the perceived depth from stereopsis

z (z) = zb H W z(b W -bW ) + bHW . (2)
Ocular Divergence Limits. Ocular divergence happens when both eyes look at the screen with a negative angle between them 1 . Both viewing rays intersect behind the spectator, i.e. z < 0. The numerator of Eq. 2 can not be negative because z, b , W, H are all positive. The denominator bHW + z(b W -bW ) is negative at z = +∞ iif b W -bW ≤ 0. The equality establishes the biggest nondivergence baseline:

b div = b W W . (3)
Note that if b W -W b < 0, only elements at depth z > -bHW (b W -bW ) cause eye divergence in the projection 1 A human can perform ocular divergence within a small range (0.5 -1 • ) [37]. For the sake of simplicity we consider the limit to be 0 • . room. Thus in a controlled acquisition setting, b div could be slightly bigger.

Roundness Factor. In 1952 the shape ratio was defined as the ratio between depth magnification ( ∂z ∂z ) and width magnification of the perceived depth ( ∂x ∂x or ∂y ∂y ) [40]. Later on, the term roundness factor has been used [25,[START_REF] Devernay | Stereoscopic cinema[END_REF]. Deriving Eq. 2 we obtain the expression of the roundness factor of an element at depth z

ρ(z) = bH W z(b W -bW ) + bHW . (4)
If the roundness factor of a scene element is smaller than 0.3, it is perceived in depth, but it appears itself as flat, as if it was drawn on a cutout cardboard [25,7]. Thus the roundness factor is important, as it allows to quantify the "cardboard effect". Establishing a target roundness for a specific depth (e.g. z = H), allows to compute the corresponding baseline:

b ρ(H) = b ρ(H) H H . (5)
Long focal length lenses: ocular divergence vs. roundness. Let f = H W be the normalized acquisition focal length, and f = H W the normalized projection focal length. The ratio between both focal lengths is

f f = b ρ(H) b div ρ(H). (6)
Several studies study the best values for f , i.e. the optimal viewing distance with respect to the screen size [START_REF] Devernay | New view synthesis for stereo cinema by hybrid disparity remapping[END_REF][START_REF] Pitié | Depthartist: a stereoscopic 3D conversion tool for CG animation[END_REF]. We now study how a disparity mapping function affects the perceived depth from stereopsis. The disparity mapping function φ(d) is generally assumed to be increasing monotonic, to avoid mapping farther objects of the scene in front of nearer ones. Using φ(d) instead of d in Eq. 1 we obtain the modified perceived depth from stereopsis

z (z) = b H b -W φ b W (z-H) z . (7)
In order to avoid ocular divergence

φ(d) ≤ b W ∀d ∈ R, (8)
which establishes the mapped ocular divergence limits constraint on φ(d). The mapped roundness factor is

ρ(z) = bH z φ (d(z)) (b -W φ(d(z))) , (9)
where φ denotes the derivative of φ w.r.t. d. Note that ∀d, φ(d) needs to be differentiable.

Related Work

We review the domains of free-viewpoint video, blending multiple views, disparity mapping and multi-rigging.

Free-Viewpoint Video [START_REF] Chen | Multidimensional characterization of quality of experience of stereoscopic 3D TV[END_REF][START_REF] Smolic | Development of MPEG standards for 3D and free viewpoint video[END_REF] . The main idea of freeviewpoint video is that multiple images of a scene can be projected onto a geometric proxy in order to generate new realistic view-dependent images [18,[START_REF] Moezzi | Virtual view generation for 3D digital video[END_REF][START_REF] Buehler | Unstructured lumigraph rendering[END_REF][START_REF] Carranza | Free-viewpoint video of human actors[END_REF][START_REF] Tanimoto | FTV (free-viewpoint television[END_REF]. Most contributions in this domain target the productions of live events [START_REF] Germann | Novel-view synthesis of outdoor sport events using an adaptive view-dependent geometry[END_REF]. Stereoscopic images can be rendered from a standard camera configuration used for a 2D broadcast [15], i.e. combining their many cameras (up to 26). In that work authors do not explicitly address the long focal length shots, and our approach to the stereoscopic zoom could be integrated in such a framework.

Blending Multiple Views. Image-based rendering (IBR) is a plenoptic sampling problem [START_REF] Mcmillan | Plenoptic modeling: An imagebased rendering system[END_REF]. One needs to reconstruct an optical ray from the available sampled rays. Unstructured Lumigraph Rendering [3] established the now prevailing [16,8,[START_REF] Kopf | First-person hyperlapse videos[END_REF] seven desirable properties that all IBR methods should fulfill. Among them, the resolution sensitivity property states: "In reality, image pixels are not really measures of a single ray, but instead an integral over a set of rays subtending a small solid angle. This angular extent should ideally be accounted for by the rendering algorithm" while the minimal angular deviation property states: "source images rays with similar angles to the desired ray should be used when possible". Both can be formally deduced from the uncertainty of the 3D geometry [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF]. The minimal angular deviation was also empirically devised to avoid visual artifacts [START_REF] Vangorp | Perception of visual artifacts in imagebased rendering of fac ¸ades[END_REF]. In our work, we place the cameras using the resolution sensitivity and minimal angular deviation constraints.

Disparity Mapping Methods. As we saw in Sec. 2, a clever modification of disparity mapping function φ(d) can reduce the distortions in the 3D transformation between the acquired and perceived scenes, for instance avoiding ocular divergence or adding roundness. Disparity mapping functions can be linear [19], non linear [START_REF] Devernay | New view synthesis for stereo cinema by hybrid disparity remapping[END_REF] or a combination of disparity mapping operators [21,[START_REF] Pitié | Depthartist: a stereoscopic 3D conversion tool for CG animation[END_REF]. They are usually the same for all pixels in the image, but could be locally adapted to preserve details [42]. Once the disparity is modified, the novel view synthesis problem basically reduces to a view interpolation problem. We can classify the methods in two groups. First, dense disparity maps warps or Depth-Image-Based Rendering (DIBR) methods, generate a virtual viewpoint relying on a disparity map [START_REF] Scharstein | A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[END_REF] using texture and depth information of the original images [START_REF] Zinger | With. Free-viewpoint depth image based rendering[END_REF]. Second, content aware warps treat the novel view synthesis problem as a 2D mesh deformation problem, extensively studied in the field of media retargeting [START_REF] Wang | Optimized scale-and-stretch for image resizing[END_REF][START_REF] Chang | Content-aware display adaptation and interactive editing for stereoscopic images[END_REF]14]. The idea is to consider the image as a regular grid, and compute the grid transformation preserving a set of constraints. In addition to stereoscopic, temporal and saliency constraints [21], constraints preserving lines and planes [47] can be used, as well as manually defined constraints [6,22].

Content aware warps have two main limitations. First they only allows moderate modifications of the initial disparity -e.g. ×2or × 3 expansion -in order to avoid visible stretch artifacts in the final images. Second it is unclear how to blend multiple images generated with these techniques, whereas blending is addressed in DIBR literature.

Multi-Rigging Techniques. In our approach we propose to use different cameras, each acquiring the scene with a different baseline, and then combine the images into the final shot (Sec. 5 and 6). Capturing one scene with several configurations is called multi-rigging [25, [START_REF] Devernay | Stereoscopic cinema[END_REF][START_REF] Dsouza | Think in 3D: Food For Thought for Directors, Cinematographers and Stereographers[END_REF]. The space is divided into depth regions, each acquired with a different configuration. Then the shots are composed depending on the depth of the elements. Special care is needed in the transition between configurations as visible artifacts could appear [START_REF] Pinskiy | Stereo compositing accelerated by quadtree structures in piecewise linear and curvilinear spaces[END_REF]. In live action stereoscopic 3D films, green screens are used to help with the depth composition [START_REF] Dsouza | Think in 3D: Food For Thought for Directors, Cinematographers and Stereographers[END_REF], involving important human efforts. In CGI (computer-generated imagery) films, an "empty safe area" with no scene objects around the compositing depths allows avoiding visual artifacts [START_REF] Pinskiy | Stereo compositing accelerated by quadtree structures in piecewise linear and curvilinear spaces[END_REF].

Non-linear viewing rays or bent rays [START_REF] Pinskiy | Stereo compositing accelerated by quadtree structures in piecewise linear and curvilinear spaces[END_REF] can be used to smoothly transition between parts of the scene captured with different baselines. In fact, a multi-rig configuration can be associated with a disparity mapping function φ(d).

In Sec. 6 we propose a multi-rig system and its associated φ(d). We propose to compute a world transformation Φ based on φ, so that we can handle the multi-rig problem as an IBR problem in a world where the optical rays are not straight.

Simplified Scene

To demonstrate the different possibilities to create a stereoscopic zoom, we focus on a simplified layout of the scene. It consists of a main subject and a background. It is a classic scenario where zooms are used in 2D, for instance to create a closeup of a soccer player focusing before a penalty kick. The physical cameras cannot disturb the performance, thus in our simplified scene we assume that the cameras cannot be "on the field". Figure 2 the distance between the cameras and the subject of interest, and the distance between the cameras and the background.

Our goal is to establish camera positions c i and parameters to render the stereoscopic images following the director's mise-en-scene. We assume the virtual and actual cameras have the same image resolution.

Being On the Field

The Mise-en-Scene. The first stereoscopic mise-enscene is the unconstrained placement of the cameras if it were possible: they would be on the field. The director freely defines a virtual filming configuration (b v , H v , W v ) in order to obtain the desired perceived depth z with the projection parameters (b , H , W ). In Fig. 3 we illustrate a virtual configuration producing a linear depth mapping. The goal now is how to place the actual acquisition cameras in order to best render the virtual images.

The Quadri-Rig. To place the actual cameras, we use the minimal angular deviation and resolution sensitivity proposed by [3] and formalized by [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF]. We place the actual cameras one by one and proceed in two stages: we first choose the focal length and then the camera position. Because the scene can be roughly decomposed in two layers, we propose to use two actual cameras to generate each virtual view. To generate the left virtual view we propose to use a camera to acquire the subject of interest and another one to acquire the background. Symmetrically, we use two cameras to generate the right virtual view. We name the resulting camera model the "Quadri-Rig".

Choosing the Focal Length. To obtain an image of a flat element with an equivalent resolution at a distance z r with a focal length f r and at a distance z v with a focal length f v , the relation between the focal lengths is f r = f v zr zv . Note that this computation is only valid for a flat element at a single depth. Elements in front (or behind) this depth have an increasing (or decreasing) image size, depending on the distances z r and z v as shown in Fig. 4. By symmetry, both cameras acquiring the subject of interest have the same focal length f s , and both cameras acquiring the background have the same focal length f b (see Fig. 5):

f s = f v z s H v and f b = f v z b z b -z v . ( 10
)
Choosing the camera positions. Our camera position goal for the actual cameras can be stated as the position minimizing the angular deviation between the optical rays of the virtual camera and the actual camera capturing the scene element. To avoid the need of a geometric estimate of the acquired scene elements, we assume them to be punctual. Then the actual camera position minimizing the angular deviation is the one fulfilling the epipolar consistency: the camera is aligned with the optical center of the virtual camera and the position of the element to render (see Fig. 6).

The remaining question is how to choose the point representing the acquired scene element. A natural choice seems to select the center of the subject as its simplified 3D position. The center of the subject of interest can be easily approximated as the center of gravity of the 3D subject's points seen by the camera. Similarly, the center of the background can be chosen as the center of the background seen by both images.

Distort the World!

We now propose a novel camera model inspired by the 2D-to-3D conversion methods, where the director establishes multiple depth and roundness constraints on an initial 2D frame (see Fig. 7). We study i) how they translate into multiple acquisition settings, and ii) how to combine the acquired images into the final stereoscopic effect.

The Mise-en-Scene. First the director chooses the 2D frame of the image, by placing a camera on the possible location area and adjusts the focal length to frame the subject of interest. Then, for each relevant element of the scene at depth z e , the director specifies the expected perceived depth in the projection room z (z e ) = z e . In addition, the director may also specify the expected roundness factor of each element, i.e. ρ(z e ) = ρ e . Although at the time of the stereoscopic mise-en-scene the director is most probably unaware of the actual depth of the scene, the provided depth description establishes constraints on the perceived depth function z (z) (Eq. 2) and the roundness factor function ρ(z) (Eq. 4).

Our goal now is to translate the depth and roundness factor constraints into a (potentially) multi-view acquisition device. The 6 parameters of z (z) from Eq. 2 are (b, H, W ) and (b , H , W ). Assuming the projection parameters fixed, then z (z) has only three degrees of freedom left. Specifying more than three constraints creates an over-determined system, i.e. one acquisition setting is not enough.

The first constraint on our setting is the focal length of the camera, chosen to create the 2D frame: f = W H . Then, one depth constraint z (z e ) = z e together with a roundness constraint ρ(z e ) = ρ e fully constrain the acquisition setup. Similarly, if the convergence distance is kept constant for all constraints (z (H) = H ) then only a depth or roundness constraint fixes the acquisition baseline. We may thus need as many acquisition cameras as constraints.

The Tri-Rig. For a generic scene containing many elements, it becomes unreasonable to use as many cameras as depth constraints. But a scene with a low number of constraints can be acquired with a small number of cameras. In our simplified scene layout with two elements, a constraint for the depth and roundness of the subject of interest, and a depth constraint on the background, result in a three camera configuration: the Tri-Rig.

We choose the camera set by the director to establish the 2D frame as the leftmost camera. Then, the second camera is chosen so that the subject of interest is perceived at the desired depth with the desired roundness (z (z s ) = z s , ρ(z s ) = ρ s ). For example, choosing z s = H, z (H) = H and ρ s = 1, defines the baseline between the left most camera and the second one b round = b H H . This configuration also establishes the convergence window width and distance (W, H) that we will keep fix. Then the third camera is placed so that the perceived depth of the background is the desired one (z (z b ) = z b ). For example, using z (∞) = ∞ to avoid ocular divergence gives b div = b W W . Reference baseline and disparity mapping. We have now acquired three images and we need to compose them into a stereoscopic pair. The initial director's image is used as the leftmost image. Now we need to render a right image following the director's constraints. Notice that while in the "Being on the field" approach we had virtual cameras establishing the cameras to be rendered, in this case we do not know where the camera to be rendered is. Furthermore, the target image cannot be obtained with a standard pinhole camera. We thus describe the image formation process of the target view with the composition of two functions. First, a function d(z) transforms scene depth values z into disparity values d, and then a disparity mapping function φ(d) transforms the acquired disparities d into the desired ones. To define the depth to disparity function d(z) we need a reference acquisition setup (b r , H r , W r ), which can be arbitrarily chosen among any director's constraint. Then, each supplementary constrain is taken into account by defining a control point on the φ(d) function. A depth constraint z (z e ) = z e constrains φ(d), whereas a roundness constraint ρ(z e ) = ρ e constrains φ(d) and φ (d). Once all control points set, the final continuous function φ(d) can be computed with any interpolation technique exactly interpolating the control points and its derivatives [START_REF] Pitié | Depthartist: a stereoscopic 3D conversion tool for CG animation[END_REF]21]. φ(d(z)) must be differentiable, otherwise the roundness factor is not be defined. The shape of φ(d) varies depending on the reference baseline. In Fig. 8 we show two functions obtained using either b div or b round as reference.

Blending multiple images and the world distortion. Once φ is defined we can theoretically compute the warps from the input images into the target image and use [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF] to render the images. Even though the blending weights of [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF] can be computed as they only rely on the image transformation, it is unclear how to compute the weights for the method proposed by [3], as they rely on angles between optical rays which are affected by the φ function.

Moreover, the actual implementation of the warps between the input and target images needs special attention, as the occlusion handling in this process is also affected by φ. Two geometric elements p 1 and p 2 projected at two different image locations u 1 and u 2 , may have the same image coordinates after the disparity mapping warp. The visibility test, also known as z-buffering, should use the final disparity mapped values d instead of the depth of the element to the camera z. The element with a lower disparity value oc- cludes the element with a higher disparity value. Similarly, when computing the inverse warp, two image points u 1 and u 2 , may be warped into points u 1 and u 2 having the same x-and y-coordinates. Thus, to first warp the image and then reconstruct the 3D points can not be done by storing the warped values in a classical single planar buffer. Some elements of the buffer will be empty, whereas other elements will have multiple assignments. A special pipeline should be implemented. However, as the occlusion handling in the render engines such as OpenGL [45] has been optimized over the years for standard pinhole camera projections, we would like to take advantage of the actual rendering techniques. Hence we propose not to apply the disparity mapping φ(d) in the images, but to distort the world accordingly with a function Φ : R 3 → R 3 before the pinhole camera projection. With this pipeline we can compute the desired image warps with a classic depth occlusion handling, and, we will also be capable to compute angles between the viewing rays, as required by [3].

Originally the disparity mapping function was introduced to operate in image space, as the main applications targetted post-production [21,[START_REF] Devernay | New view synthesis for stereo cinema by hybrid disparity remapping[END_REF]. However, if we are at the acquisition stage, we can introduce a world deformation Φ based on φ. In Fig. 9 we illustrate the construction of the proposed world distortion. Let us consider two cameras, the left at c l and the right at c r , defining a reference setup (b r , H r , W r ). A point p = (x, y, z) in the original scene is projected into the right image point u = (u, v, d(z)), where d(z) is obtained by using (b r , H r , W r ) in Eq. 1. The image point u is then mapped into u = (u , v, φ(d(z))) using the predefined φ(d). Then a 3D point p = (x , y , z ) can be reconstructed as follows. It's z-coordinate is obtained as z (φ(d(z))) using (b r , H r , W r ) in Eq. 1. Then we still need to define x and y to obtain the 3D distortion of the world Φ. As the frame of the left camera is chosen by the director, a natural constraint on Φ(p) is that both p and p project on the same point on the left camera. This way, the image of the left camera is unaffected by the geometry distortion, and Φ(p) = p is fully defined. Let us point out that if φ(d) is strictly increasing and differentiable, then Φ(p) -1 exists and is well defined.

With the proposed world distortion, angles between the desired optical ray and the input camera ray can be computed as illustrated in Fig. 10. Given a point u on the reference image, its 3D point p in the distorted world can be computed. Then the desired viewing ray in the distorted world can be undistorted using the DΦ -1 evaluated at the depth of p . The undistorted ray can now be compared to the ray between p and c i , where p is the undistorted version of p , and c i the optical center of the input view. The weights of the method proposed by [3] can now be computed.

With the proposed world distortion we extend the disparity mapping problem into a more general image-based rendering problem. In practice Φ(x) can be efficiently implemented with a vertex shader. While our approach distorts the world to obtain straight viewing rays to properly handle occlusions, the distorted world should not be used to compute any geometric values, such as angles or distances, as for instance, illumination techniques relying on them would lead erroneous results.

Proofs of Concept

We present proofs of concept of the proposed approaches on a synthetic dataset, blender lego, where the exact camera parameters and the exact geometry of the scene are known.

Quadri-Rig. To demonstrate the quadri-rig we rendered 6 images (see Fig. 11). The left and right virtual images were rendered at the desired virtual positions v l and v r . These images were used as ground truth for comparison with the rendered images. The other four images of the dataset correspond to the "Quadri-Rig" configuration: two images of the cameras acquiring the subject and two images of the cameras acquiring the background.

In Table 1 we present the PSNR and DSSIM computed Table 1. Numerical results for the synthetic dataset. We compare [3] and [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF]. The first value is PSNR (bigger is better), the second value is DSSIM in units of 10 -4 (smaller is better). The best value is highlighted in bold.

Figure 12. First row: stereoscopic pair obtained from the Quadri-Rig with [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF]. Second row: closeups of the rendered views. As the background is not a plane, large areas in black are not acquired by any actual camera (compare with Fig. 15). values between the ground truth virtual images and the rendered ones using [3] and [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF]. Because of the low number of images, all methods yield very similar results. This result is coherent with the results obtained in [START_REF] Pujades | Viewpoint interpolation: Direct and variational methods[END_REF]. The difference in the blending weights has no significant impact on the rendered images when few images are used. The high PSNR and low DDSIM values obtained with the synthetic dataset show that the rendered images at visible locations are accurate. In the first row of Fig. 12 we reproduce the stereoscopic pair rendered with [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF]. Because the background of the scene is purely flat, large regions visible in the virtual views are not acquired by any of the four actual cameras. The closeups in Fig. 12 show these occlusions.

Tri-Rig. To demonstrate the "Tri-Rig" we rendered three views of the blender lego dataset (see Fig. illustrates the world distortion created by Φ(x). The subject's depth is preserved, while the background elements are pulled forward to decrease their disparity values on the final image. In the first row of Fig. 15 we show the final rendered images using [3] and [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF], which, visually, do not present any noticeable difference. In the second row of Fig. 15 we show closeups of the regions in the rendered images, which are not acquired by any source image. These regions are very small compared to the large black areas obtained with the "Quadri-Rig" approach shown in Fig. 12. Moreover, let us recall that the left image of the target stereoscopic pair is the original one, i.e. it does not suffer at all of occluded regions. As the target image does not correspond to a perspective camera, we do not have a reference image to compare with, and thus we can not numerically evaluate the obtained results. Our approach distorts the world to obtain straight viewing rays and properly handle occlusions. However, the distorted world should not be used to compute any metric values, such as angles or distances. For instance, rendering techniques such as illumination, relying on metric values, should not be computed in the distorted world as they would lead to erroneous results.

Discussion and Conclusion

In this paper we addressed the problem of generating stereoscopic images with long focal lenses. We have illustrated why multiple cameras are needed and deduced two different camera models, the "Quadri-Rig" and "Tri-Rig". Each model is inspired by the intention of the director, either to get closer to the scene or to add aesthetic perspective deformation to the scene. Although the mise-en-scene of each method is very different, we compare both camera models to highlight their advantages and flaws.

A key advantage of the "Tri-Rig" with respect to the "Quadri-Rig" is that one of the images, i.e. the left one, is acquired by a source camera. Indeed, the perceived quality of a stereoscopic pair of images is close (and sometimes equal) to the quality of the best of both images [35]. Thus having the raw output of the camera as the left view provides the highest quality possible. Moreover, as illustrated in Fig. 12, large areas needed by the target images of the "Quadri-Rig" may not be acquired by any of the four actual cameras. Although these regions could be filled in using an inpainting method [29, 17, 27, 4], these occlusion regions are smaller in the "Tri-Rig" setup, as shown in Fig. 15.

In this work the camera models target a scene with a simple layout. Nevertheless the proposed camera models are generic and can be extended to scenes with more elements. For each new relevant element in the scene, a new pair of cameras are needed. However, as in the "Tri-Rig" all cameras have the same focal length, all the left cameras of each configuration can be the same. Thus, given a scene with N ≥ 2 relevant elements, the "Quadri-Rig"s complexity is 2N , whereas the "Tri-Rig"s complexity is N + 1.

Because of the advantages of the "Tri-Rig" with respect to the "Quadri-Rig", the director could also use the "Tri-Rig" with the intention to get closer to the scene. In this case, similarly to the use of a zoom in 2D, the perspective distortions would be a consequence, not the intention.

Future work should address the validation of the images generated by the "Tri-Rig". As we do not have a reference image to compare with, we could not assess the relevance of the proposed camera model. We believe that a subjective evaluation of the obtained results should be conducted in the future to assess the proposed approach. One possibility to evaluate if the "Tri-Rig" is capable to create compelling stereoscopic images would be to conduct a user study. The observers would be presented with images generated with the "Tri-Rig" and images generated with disparity mapping methods using only two images [21,47,[START_REF] Devernay | New view synthesis for stereo cinema by hybrid disparity remapping[END_REF]. The observer could then choose if one is preferred, or equal preference.

Figure 1 .

 1 Figure 1. Parameters describing the shooting and projection geometries (reproduced from [10]). (l and r indicate left and right).

Figure 2 .

 2 Figure 2. Simple scene with a subject of interest and a background. Actual cameras can only be placed outside the field. The distance between the actual camera location and the subject of interest and the background are zs and z b respectively.

Figure 3 .

 3 Figure3. Acquisition and projection of the simplified scene. We illustrate a possible mise-en-scene. The director freely chooses the virtual cameras (left), so that the perceived depth from stereopsis in the projection room presents no distortions (right).

Figure 4 .Figure 5 .

 45 Figure 4. Two cameras with different focal lengths acquire a green object with the same resolution. The object has the same size on both images. A red object farther away has a bigger image size in the camera with a longer focal length.

Figure 6 .

 6 Figure 6. Diagram of the obtained camera positions for the "Quadri-Rig". Left: the cameras acquiring the subject c s * are aligned with the position of the subject of interest s and the virtual cameras c v * . The cameras acquiring the background c b * are aligned with the center of the background p and the virtual cameras c v * . Right: When s, p and v l are aligned, c b l and c s l are equal.

Figure 7 .

 7 Figure 7. The original image (left) is annotated with the expected disparity in the final shot (right). Images reproduced from [28].

Figure 8 .

 8 Figure 8. Disparity mapping function φ(d) examples. Left: φ(d) expanding the disparity range near the zero disparity, i.e. the subject of interest depth. Disparity values after the expansion are compressed and the disparity values of the background are preserved. Right: φ(d) compressing the disparity range for the background values. Disparity values at the convergence depth are preserved.

Figure 9 .

 9 Figure 9. The proposed world distortion Φ as the composition of: the projection of p to u, the image warp on u defined by φ(d), and the backprojection of u into p . Left: Top view scheme of the scene. Right: function composition graph.

Figure 10 .

 10 Figure 10. Computing angles between viewing rays to render the image point u with the camera ci. The direction of the viewing ray defined by u and p is undistorted using DΦ -1 . The angle α between the undistorted ray and p and ci can be computed.

Figure 11 .

 11 Figure 11. The "Quadri-Rig" blender lego dataset images. a) and b) is the virtual stereoscopic pair. c) and d) are the images acquired with the background cameras. e) and f) are the images acquired with the subject of interest cameras.left image right image [3] 33.93 288 33.94 290 [33] 34.00 287 34.02 288

Figure 13 .

 13 Figure 13. The "Tri-Rig" images of the blender lego dataset. a) and b): left and right images acquiring the background. a) and c): left and right images acquiring the subject of interest.

Figure 14 .

 14 Figure 14. The world distortion for the "Tri-Rig" blender lego dataset. Left: original 3D scene. Right: Distorted 3D scene. The background depth is compressed, so that further elements create a smaller disparity in the final rendered image.

Figure 15 .

 15 Figure 15. First row: Right images rendered from the Tri-Rig with [3] (left) and[START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF] (right). No noticeable difference is visible between the images. Second Row: Occluded regions in the "Tri-Rig". Few pixels around depth discontinuities are not acquired by any camera (compare with Fig.12).

For instance the "Angenieux Optimo 28-340 cinema lens"[START_REF] Angenieux | Online angenieux portefolio[END_REF]


Sergi Pujades

Frédéric Devernay

Bayesian View Synthesis and Image-Based Rendering Principles

, formalizing some of the previously proposed heuristics, pointing out which physical phenomena could lie behind each. However, some important heuristics were still not taken into account and lack proper formalization.

We contribute a new physics-based generative model and the corresponding Maximum a Posteriori estimate, providing the desired unification between heuristics-based methods and a Bayesian formulation. The key point is to systematically consider the error induced by the uncertainty in the geometric proxy. We provide an extensive discussion, analyzing how the obtained equations explain the heuristics developed in previous methods. Furthermore, we show that our novel Bayesian model significantly improves the quality of novel views, in particular if the scene geometry estimate is inaccurate.

Introduction

We address the problem of novel view synthesis in the domain of Image-Based Rendering (IBR) [START_REF] Shum | Image-based rendering[END_REF], where the aim is to synthesize views from different viewpoints using a set of input views in arbitrary configuration. Most of the methods from the state of the art use heuristics to define energies or target functions to minimize, achieving excellent results. A major breakthrough in IBR was the inspiring work of Buehler et al. [4]. They define the "desirable properties" which any IBR algorithm should have. Those A depth distribution along an optical ray of camera vi will propagate differently depending on the viewing angle of the rendered camera u or u ′ . The bigger the angle, the bigger the projected uncertainty will be.

directives still prevail throughout the current state of the art.

Recently, however, the use of the Bayesian formalism has been introduced in IBR techniques, with the work proposed by Wanner and Goldluecke [START_REF] Cho | Image restoration by matching gradient distributions[END_REF]. They provide the first Bayesian framework for novel view synthesis, describing the image formation process with a physics-based generative model and deriving its Maximum a Posteriori (MAP) estimate. Moreover, their variational method does not only address the problem of novel view synthesis. It directly addresses the synthesis of new super-resolved images, and provides a solid framework for other related problems, namely image denoising or image deblurring.

Interestingly, although [4] and [START_REF] Cho | Image restoration by matching gradient distributions[END_REF] have addressed the same problem, their theoretical results do not converge into a unified framework. On the one hand, the guidelines dictated by Buehler et al. in [4] have proven to be very effective, but lack a formal reasoning supporting them. Moreover, it is unclear how the balance between some of the de-

D C 1 C 2 P α 2 α 1 D C 1 C 4 P α 1 C 2 C 3 α 2 α 3 α 4 Figure 2.
View D is generated from cameras Ci using [START_REF] Cho | Image restoration by matching gradient distributions[END_REF]. Left: camera C2 will be favored over camera C1 because of the foreshortening effect. However, the angular distance of the viewing rays between D and C1 is much smaller than D and C2. Right: configuration with a flat scene. All cameras will have the same contribution, despite the different viewing angles.

sirable properties should be handled. An illustrative example is the tradeoff between epipole consistency and resolution sensitivity. The former notes that "when a desired ray passes through the center of projection of a source camera it can be trivially reconstructed", while the latter observes that "in reality, image pixels are not really measures of a single ray, but instead an integral over a set of rays subtending a small solid angle. This angular extent should ideally be accounted for by the rendering algorithm." The epipole consistency is enforced with an angular deviation term, while the resolution sensitivity is driven by the Jacobian of the planar homography relating the views. Both heuristics seem reasonable, but which one should dominate? The choice of the weights between the properties is user-tuned, and in their experiments, parameters have to be adjusted differently depending on the scene. On the other hand, the existing Bayesian model [START_REF] Cho | Image restoration by matching gradient distributions[END_REF] is able to explain some of the heuristics, but still violates others which seem evident and have proven to work effectively. For example, we do find an analytic deduction of the influence of the foreshortening effects due to the scene geometry in the energy. The findings confirm the heuristic proposed by Buehler et al. in [4]: it is driven by the Jacobian of the transformation relating the views. However, when carefully analyzing the final equations in [START_REF] Cho | Image restoration by matching gradient distributions[END_REF], an important desirable property proposed in [4] is still missing: the minimal angular deviation of the viewing rays is not enforced and even violated in some cases, as illustrated in Fig. 2.

The differences between state of the art generative models and the energies proposed by generally accepted heuristics is what has motivated the present work. Our goal is to retain the advantage of the intrinsically parameter-free energies arising from the Bayesian formalism, while pushing the image formation model boundaries of [START_REF] Cho | Image restoration by matching gradient distributions[END_REF] and provide a new model which is capable of explaining most of the currently accepted intuitions of the state of the art in IBR.

Contributions.

The key theoretical contribution of the proposed method is the systematic modeling of the error introduced in the lambertian image formation process via the inaccuracy in the estimates of the geometric proxy. We call this inaccuracy depth uncertainty, referring to the depth estimates from the input images. In addition to this error, we also consider the image sensor noise, commonly modeled as Gaussian. We extensively analyze the theoretical implications of the obtained energy, discussing the formal deduction of the state of the art heuristics from our model. This work provides the first Bayesian formulation explicitly deriving the heuristics of [4].

From a practical point of view, we numerically evaluate the performance of our method comparing it to the best existing method within the Bayesian framework. Experimental results show that we achieve state of the art results with regard to objective measures on public datasets. Moreover, we are also capable of addressing super-resolution, capitalizing on the general framework established in [START_REF] Cho | Image restoration by matching gradient distributions[END_REF]. The new model is not without a price, since its optimization is less straightforward. However, existing methods allow us to overcome this difficulty. Source code is publicly available at http://sf.net/projects/cocolib/.

Related work

Since the early work on plenoptic modeling [START_REF] Mcmillan | Plenoptic modeling: An imagebased rendering system[END_REF] proposed by McMillan and Bishop, many IBR techniques have been developed for several purposes, e.g. free-viewpoint rendering [24], image morphing [START_REF] Wolberg | Image morphing: a survey[END_REF] or image view interpolation [START_REF] Smolic | Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems[END_REF] among others. The taxonomy done by Shum et al. [START_REF] Shum | Image-based rendering[END_REF] shows that most IBR methods use a geometric proxy, and they classify them in an "IBR Continuum" depending on how much geometry they use. On one end of this continuum we have methods which do not use any but rely on a large collection of input images, like light field rendering [START_REF] Levoy | Light field rendering[END_REF], and concentric mosaics [START_REF] Shum | Rendering with concentric mosaics[END_REF]. On the opposite end, we have rendering techniques relying on explicit geometry, using accurate geometric models but few images, such as layered depth images [START_REF] Shade | Layered depth images[END_REF]6] and view-dependent texture mapping [9]. In between, we find methods using an implicit representation of the geometry, such as view interpolation techniques [7,[START_REF] Chen | View interpolation for image synthesis[END_REF] relying generally on optical flow, transfer methods [START_REF] Laveau | 3-d scene representation as a collection of images[END_REF] establishing correspondences along the viewing rays using epipolar geometry, and the Lumigraph [11], which uses an approximate explicit geometry and a relatively dense set of images.

When Buehler et al. introduced Unstructured Lumigraph Rendering [4], they established the seven "desirable properties" that all IBR methods should fulfill: use of geometric proxies, unstructured input, epipole consistency, minimal angular deviation, continuity, resolution sensitivity, equivalent ray consistency, and real-time. This work has been of major importance in the community, and most IBR methods follow these guidelines.

Although Bayesian formalisms are a common way to deal with spatial super-resolution in the multi-view and light field setting [3, 10], they have only recently been introduced to IBR with the work by Wanner and Goldluecke [START_REF] Cho | Image restoration by matching gradient distributions[END_REF].

While their work provides a physical explanation for the resolution sensitivity property, the minimal angular deviation can be violated in their final equations. Most interestingly, Vangorp et al. [26] empirically verify which properties in IBR methods are prone to create visual artifacts, and one of their main results identifies angular deviation as a key property to be taken into account to avoid visual artifacts.

Even if the performance achieved by state of the art 3D reconstruction methods in estimating geometric proxies is phenomenal, considering them as perfect seems too strong of an assumption: even the best ones have an uncertainty in their final estimates. Naturally, novel view synthesis is prone to producing visual artifacts in regions with a poor (implicit or explicit) reconstruction. One way to address this problem is to improve the acquisition setting, as done by Zitnick et al. [START_REF] Zitnick | High-quality video view interpolation using a layered representation[END_REF]. They achieve a good enough reconstruction, leading to impressive novel view synthesis. However, their setting is heavily constrained.

In [22], Takahashi studies the theoretical impact of errors in the geometric proxy when rendering a new view from 2 images. We improve [22] by addressing more general camera configurations and providing an efficient method to find the solution, both explicitly left as future work. In [START_REF] Takahashi | Super-resolved freeviewpoint image synthesis using semi-global depth estimation and depth-reliability-based regularization[END_REF] Takahashi and Naemura use the depth uncertainty information to leverage the regularizer term (prior). But this consideration does still not take into account the minimal angular deviation, because distinct contributions for each camera are not allowed. We solve this issue in this work.

Novel view synthesis generative model

Our goal is to synthesize a (possibly super-resolved) view u : Γ → R from a novel viewpoint c using a set of images v i : Ω i → R captured from general positions c i . We assume we have an estimate of a geometric proxy which is sufficient to establish correspondence between the views. More formally, the geometric proxy induces a backwards warp map τ i : Ω i → Γ from each input image to the novel view, as well as a binary occlusion mask m i : Ω i → {0, 1}, which takes the value one if and only if a point in Ω i is visible in Γ. If we restrict τ i to the set of visible points V i ⊂ Ω i , it is injective and its left inverse

β i : τ i (V i ) → Ω i is well defined, see Fig. 3.
Ideal image formation model. In order to consider the loss of resolution from super-resolved novel view to input view, we model the subsampling process by applying a blur kernel b in the image formation process of v i . It corresponds to the point spread function (PSF) of camera i. Each pixel of v i stores the integrated intensities from a collection of rays from the scene, and the novel view u will always be considered as having a higher resolution than the input views.

x x ′ σ zi (x ′ ) σ zi (x) β i τ i τ i (x) Ω i Γ c c i
Let us discard the effects of visibility for a moment, supposing all points are visible. Also suppose we have a perfect backward warp map τ * i from Ω i to Γ, and perfect input images v * i . Assuming the lambertian image formation model, the idealized exact relationship between novel view and input views is

v * i = b * (u • τ * i ), (1)
being • the function composition operator. However, the observed images v i and geometry τ i are not perfect, and we need to consider these factors in the image formation model.

Sensor error and image error. First, we consider Gaussian sensor noise on all cameras with variance σ 2 s . While the sensor noise variance σ 2 s and the subsampling kernel b could be different among views, for the sake of simplicity of notation, we will assume them to be identical.

Second, we consider the error in the geometry estimate, which implies that the corresponding backwards warp τ i is different from the ideal map τ * i . This induces an intensity error ǫ gi in the image formation process,

ǫ gi = b * (u • τ * i ) -b * (u • τ i ). (2)
The uncertainty related to the intensity error ǫ gi is denoted by σ gi : Ω i → R. Note that both have intensity units.

Taking into account the above errors, the image formation model becomes:

v i = b * (u • τ i ) + ǫ gi + ǫ s .
(3) While we make the common assumption that ǫ s follows a Gaussian distribution, the distribution of ǫ gi is yet unknown to us. What we know is that ǫ gi is strongly related to the geometric error. In the next section, we study the relationship between their distributions. Dependency of image error on geometric error. The geometric proxy yields for each pixel x in Ω i a depth measure z i which is associated with an uncertainty σ zi , giving us a distribution of depth along the viewing ray, as illustrated in Fig. 3. We now consider the error ǫ zi in the estimation of the geometric proxy, expressed in world units. The previous image error ǫ gi is dependent on the underlying geometric error. Note that the image error has intensity units and must not be confused with ǫ zi having geometric units. In contrast to the blur kernel and the sensor noise, we allow these errors to be different for each view and for each pixel in each view, as made explicit in the notation.

We assume that the error distribution for the depth estimates is normal, ǫ zi ∼ N (0, σ 2 zi ). The goal is now to derive how this distribution generates a color error distribution in the image formation process. Propagating a distribution with an arbitrary function is not straightforward, even if in our case, this depth error distribution is assumed to be Gaussian, and will only be propagated along the epipolar geometry between the views. Instead of computing the full color distribution along the viewing ray, we linearize and consider the first order Taylor expansion of v i with respect to z i . This implies that the resulting color distribution is also Gaussian, with mean u • τ i and standard deviation

σ gi = σ zi ∂v i ∂z i . (4)
Using Eq. ( 1) and the chain rule, we find that

σ gi = σ zi b * ∂(u • τ i ) ∂z i = σ zi b * (∇u • τ i )• ∂τ i ∂z i . (5)
MAP estimate and energy. In the Bayesian formulation, the MAP estimate of the novel view can be found as the image u minimizing the energy

E(u) = E data (u) + λE prior (u), (6)
where the data term E data (u) is deduced from the generative model, and E prior (u) is a smoothing term which is detailed afterwards. λ > 0 is the only parameter of our method, and it controls the smoothness of the solution.

Let us consider the two error sources as independent, additive and Gaussian. Then their sum is also a normal distribution with zero mean and variance σ 2 s + σ 2 gi . The data term computed from the generative model of Eq. ( 3) is given by:

E data (u) = n i=1 1 2 Ωi ω i (u) m i (b * (u • τ i ) -v i ) 2 dx, (7)
with

ω i (u) = σ 2 s + σ 2 gi -1 . (8)
This data term is similar to the one found in the previous model from [START_REF] Cho | Image restoration by matching gradient distributions[END_REF], except for the factor ω i (u), which can be seen as a weight that depends both on the depth uncertainty and on the latent image u being computed. If there were no depth uncertainty, this term would reduce to σ 2 s , which gives exactly the energy found in [START_REF] Cho | Image restoration by matching gradient distributions[END_REF].

From Eq. ( 5), we can observe that the term σ 2 gi in ω i (u) becomes smaller if the length of the vector ∂τ i /∂z i decreases. The derivative ∂τ i /∂z i denotes how much the reprojection of a point x i from the original view v i onto the novel view u varies when its depth z i (x i ) changes. This vector points towards the direction of the epipolar line on u issued from the point x i of v i , and its magnitude decreases with the angle between the optical ray issued from the original view v i and the optical ray from the novel view u. As illustrated in Fig. 1, the term σ 2 gi thus accounts for the minimal angular deviation "desirable property" from [4], which was not accounted for in [START_REF] Cho | Image restoration by matching gradient distributions[END_REF].

Let us analyze more precisely under which circumstances the weight ω i (u) reaches its maximal value 1/σ 2 s , which is the value found in the previous model. There are three situations in which this occurs. The first one is if ∂τ i /∂z i = 0, i.e. the depth of a point in v i has no influence on its reprojection onto u. This can only happen if the two optical rays are identical, which corresponds to the epipole consistency property from [4]. The second one is if ∇u = 0, i.e. the rendered image has no gradient or texture at the considered point: in this case, an error on the depth estimate has no effect on the rendered view. The last situation is if ∇u at the rendered point is orthogonal to the direction of the epipolar line from camera i passing through the rendered point: a small error on the depth estimate in camera i does not have an effect on the rendered view because the direction of influence of this error is tangent to a contour.

Choosing the prior. The prior is introduced in the Bayesian formulation to restrain the possible configurations of the target image. Usually, it is used to overcome the ill-posedness of the problem: in the analysis of superresolution by Baker and Kanade [START_REF] Baker | Limits on super-resolution and how to break them[END_REF], they show that the dimension of the null-space of the matrix system increases with an increase of the super-resolution factor. Furthermore, in novel view synthesis, some parts of the image may not be seen by any contributing view, thus a regularization prior allows to fill the gaps with plausible information. Thus, the choice of the prior will have significant influence on the final result.

Very interesting priors have been developed in order to overcome specific issues in super-resolution [START_REF] Shan | High-quality motion deblurring from a single image[END_REF]. There are also techniques allowing to learn generic image priors from a collection of images [START_REF] Roth | Fields of experts: A framework for learning image priors[END_REF]. However, in this work we focus on the generative model, and we use basic total variation as a regularizer,

E prior (u) = Γ |Du| , (9)
which is convex and has been extensively studied in the context of image analysis problems [START_REF] Chambolle | An algorithm for total variation minimization and applications[END_REF]. The search for optimal priors will be a topic of future work.

Optimization. The energy from Eq. ( 6) is hard to opti-mize because the weights ω i (u) in Eq. ( 7) are a nonlinear function of the latent image u. Similarly to [8], we propose a re-weighted iterative method. We use an estimate ũ of u, set at ũ = 1 n v i • β i in the first iteration. We consider then ω i (ũ) constant during each iteration, making the simplified energy convex. Furthermore, with arguments similar to [START_REF] Cho | Image restoration by matching gradient distributions[END_REF], we can show that the functional derivative of the simplified data term is

dE i data (u) = ω i (ũ) |detDβ i | m i b * (b * (u • τ i )-v i ) •β i , (10)
where b(x) = b(-x) is the adjoint kernel. This functional derivative is Lipschitz-continuous, which allows to minimize the energy via the fast iterative shrinkage and thresholding algorithm (FISTA) [2]. With the solution of this simplified problem, we update ũ, thus obtaining new weights, and a new energy. We solve it again with FISTA, and iterate. Although the minimization problem to be solved within each iteration is convex, in general we cannot hope to find the global minimum of Eq. ( 6).

Relation to the principles of IBR

As we see in Eq. ( 10), the weighting factor for each view is composed of two terms. The term |detDβ i | is the same as in [START_REF] Cho | Image restoration by matching gradient distributions[END_REF] and corresponds to a measure of image deformation: it is the surface of a pixel from u projected to v i . We can formulate the intuition behind it as how much does the observed scene change when the viewpoint changes?

The term ω i (u) corresponds to the depth uncertainty, as was explained in the previous section. The intuition behind this is: how much does the observed scene change if the measured depth changes?

Let us now carefully establish the links of the proposed energy with the "desirable properties" of IBR stated in [4].

Use of geometric proxy & unstructured input. The geometric proxy is incorporated via the backward warp maps τ i , and the input can be unstructured (i.e. a random set of views in generic position).

Epipole Consistency. As explained previously, the weighting factor ω i (u) is maximal as soon as the optical rays from x i and x are identical, so that if a camera has its epipole at x, then the contribution of this camera at x via the ω i (u) term is higher. Epipole Consistency is thus satisfied.

Minimal angular deviation. This heuristic is provided by σ gi from Eq. ( 5): if all other dimensions are kept constant (resolution, distance to the scene, etc.), then the magnitude of the vector ∂τ i /∂z i in Eq. ( 5) is exactly proportional to the sine of angle between the optical rays from both cameras to the same scene point.

Resolution sensitivity. This heuristic is followed by the term |detDβ i |, which measures the surface of a pixel from u projected to v i . The larger the resolution of camera i, the bigger this surface, so that resolution sensitivity is properly handled.

Equivalent ray consistency. "Through any empty region of space, the ray along a given line-of-sight should be reconstructed consistently, regardless of the viewpoint position (unless dictated by other goals ...)" [4]

. This is trivially satisfied by our framework, since the weights are varying continuously with the camera parameters (through the continuous variation of the backward warp maps τ i ). Moving the novel view camera along an optical ray (which is the situation used to describe this property in [4]) is just a special case.

Continuity. The continuity principle in IBR demands that the final rendered image is varying continuously with the camera parameters of the original views. This implies that there are no seams at visibility boundaries between cameras, which may happen near the borders of the intersection of the field of view of each camera with the scene, or at depth discontinuities seen from each camera. The typical heuristic to enforce this form of continuity is to lower the contribution of a camera near a visibility boundary or the boundary of its field-of-view [START_REF] Raskar | Blending multiple views[END_REF]4]. Our equations do not satisfy this property and the obtained weights do not fall to zero when approaching a visibility boundary. This could easily be enforced by smoothing the visibility maps m i near the depth and visibility discontinuities, without changing the zero set of these functions. However, since we claim to have a completely physics-based Bayesian formulation, any operation on the visibility map should be sustained by a physical explanation, which we are still missing, and this is part of our future work.

Note that the prior term in the energy reduces the problems, most notably visual artifacts, which are due to not handling the continuity properly. However, a prior on the novel views cannot completely solve the continuity problem, which depends on the scene and camera geometry.

Real-Time. The final "desirable property" is for the method to be real-time. Our method is not yet real-time, mainly because of the computational complexity of the MAP estimate: 2 to 3 seconds are necessary to render a 768×768 image from 8 source images. However, both the resolution algorithms and the hardware architectures are evolving quickly, and much better performance can be expected in the next few years.

If super-resolution is not important, instead of solving the full MAP problem, it seems reasonable to use real-time regularization in the form of inpainting methods to obtain an acceptable result.

Balance between properties. One of the advantage of our method with respect to [4] is that the balance between the different properties is not handled by user-defined parameters, but implied from a formal deduction. Imagine a configuration with two cameras: one with low minimal angular distance but high resolution sensitivity change, and another with high minimal angular distance but low resolution sensitivity change. Which one should contribute more to the final image? In [4], the angular distance is preferred to the resolution sensitivity by a ratio of 1/0.05 = 20 (Hallway dataset). In our equations, these variations are completely physics-based. An angular deviation of ∆α between views is penalized proportionally to 1 sin 2 ∆α , due to the change in σ 2 gi . A foreshortening effect or resolution difference causing an image scale factor s is penalized proportionally to 1 s 2 , due to the change in |det Dβ i |. The balance between these factors is properly handled by taking into account the sensor noise σ 2

s . An exception is the weight λ, used in the prior term. Note that this is common in all work on image analysis based on Bayesian principles: since there is currently no meaningful way to obtain a prior distribution on the space of images , one needs to work with regularization by objective priors. Of course one could also use existing methods [START_REF] Roth | Fields of experts: A framework for learning image priors[END_REF] allowing to estimate this prior directly from the input images, thus obtaining a completely parameter-free model.

Experiments

Simplified camera configuration. Although we are addressing a generic case of novel view synthesis, in order to simplify the implementation of the optimization procedure, in the experiments we suppose that our cameras have a simplified configuration. Specifically, all viewpoints are in a common plane, which is parallel to all image planes, i.e. we are dealing with a 4D light field in the Lumigraph parametrization [11]. The novel view is also synthesized in the same image plane, which means that τ i is simply given by a translation proportional to the normalized disparity d i ,

τ i (x) = x + d i (x)(c -c i ). ( 11
)
Normalized disparity is expressed in pixels per world units, and is together with its associated uncertainty related to depth via:

d i (x) = f i z i (x) and σ di (x) = σ zi (x) f i z i (x) 2 , ( 12
)
where f i is the camera focal length expressed in pixels. Plugging [START_REF] Laveau | 3-d scene representation as a collection of images[END_REF] and ( 11) into (5), we derive the link between the geometric error and its associated image error as:

σ gi = σ di |(b * ((∇u • τ i ) • (c -c i )))| , (13)
where σ di models the disparity noise. Finally, the deformation term in Eq. ( 10) is:

|detDβ i | = |detDτ i | -1 = |1 + ∇d i • (c -c i )| -1 . ( 14
)
Datasets. To validate the theoretical contribution, we compare results on two light field datasets: The HCI Light Field Database [START_REF] Wanner | Datasets and benchmarks for densely sampled 4D light fields[END_REF], and the Stanford Light Field Archive [25]. These datasets provide a wide collection of challenging synthetic and real-world scenes.

In a first set of experiments, we render an existing view from the dataset at the same resolution, without using the respective view as an input to the algorithm. We consider two different qualities of geometric proxy: an approximate one from estimated disparity maps, and an extremely poor one represented by an infinite flat fronto-parallel plane in the estimated center of the scene. We adapt σ di accordingly, i.e. when using the estimated disparity, we use a value corresponding to the expected accuracy of the reconstruction method: σ di = dmax-dmin nbLayers , where nbLayers is the number of disparities considered by the method. When a bare plane in the middle of the scene is used, we instead use σ di = dmax-dmin

4

. In all cases, σ s = 1/255. A second set of experiments is performed by rendering a 3×3 super-resolved image from a set of 5×5 input views. Although super-resolution is not the main purpose of the paper, we also provide a comparison with the state of the art. As super-resolution relies on sub-pixel disparity values, we only show the results obtained with the estimated disparity maps.

In Tab. 1, we show the numerical results obtained by our method, and compare it to the ones achieved with [START_REF] Cho | Image restoration by matching gradient distributions[END_REF]. We measure the PSNR and DSSIM between the actual and generated images. Although our method visibly performs better, numerical values should be interpreted carefully. In Fig. 4, we show detailed closeups illustrating the benefits or our method. As high resolution images are not available for most of the datasets, PSNR and DSSIM values for the super-resolved images are computed by subsampling the input images, generating the novel super-resolved view and comparing it with the original one.

When rendering with precise geometry, both methods are roughly equivalent with respect to PSNR and DSSIM values (first and last two rows of Tab. 1). When the quality of the proxy degrades (third and fourth rows of Tab. 1), our method clearly outperforms previous work, taking advantage of the explicit modeling of depth uncertainty. As shown in the closeups of Fig. 4, our method better reconstructs color edges in all configurations. Full-resolution images are provided in the supplemental material. Computation time when rendering at target resolution 768 × 768 with 8 input images is on the order of 2 to 3 seconds. Computation time for super-resolved view synthesis with a factor of 3 × 3 and 24 input images is around 2 to 3 minutes. All experiments used an nVidia GTX Titan GPU.

Discussion and conclusion

The main contribution of this paper is to establish the first formal link between the heuristics proposed in the recent decades for novel view synthesis, and the energy de- 1. Numerical results for synthetic and real-world light fields from two different online archives. We compare our method to Wanner and Goldluecke [START_REF] Cho | Image restoration by matching gradient distributions[END_REF] with respect to same-resolution view synthesis for estimated disparity and a flat plane proxy, as well as super-resolved view synthesis. For each light field, the first value is the PSNR (bigger is better), the second value is DSSIM in units of 10 -4 (smaller is better). The better value is highlighted in bold. See text for a detailed description of the experiments. This model can be used to solve the generic problem which consists in generating a novel view from a heterogeneous set of input images, and a geometric description of the scene (called a geometric proxy), which can be either explicit (i.e. the estimated geometry of the 3D scene) or implicit (i.e. a set of correspondence maps between original views and the novel view).

Part of our contribution is the analysis of how the proposed model fulfills almost all the guidelines established by Buehler et al. [4]. The proposed generative model pro-vides a formal description of the intuitive heuristics behind these guidelines. The key element to this unification is to take into account the error in the estimated geometric proxy when rendering a new image. We have extensively discussed how our physics-based model explains the reasons why some important heuristics were picked up in the first place. The theoretical benefits of the model outperform state of the art by overcoming its limitations. Moreover, the experiments conducted on synthetic and real images show that our method improves state of the art performance in terms of rendered image quality.

Future work should better handle the visibility term in the model. In this work, visibility is computed from depth, but depth itself contains errors, which should propagate onto the visibility maps. This could be a key solution to incorporate the last missing Continuity heuristic into this physics-based Bayesian framework. Also extending the model to non-Lambertian scenes is crucial but quite hard. One would need to include general BRDF and lighting information to correctly model the transformation between input and novel views.

An important observation is that if the 3D reconstruction method or the 2D-2D image correspondence method provides not only depth estimates, but also the associated depth uncertainty, the image-based rendering method can benefit from this information to create better novel views. This should thus be a goal when developing new (implicit or explicit) reconstruction methods aimed at IBR.

Figure 1 .

 1 Figure1. A depth distribution along an optical ray of camera vi will propagate differently depending on the viewing angle of the rendered camera u or u ′ . The bigger the angle, the bigger the projected uncertainty will be.

Figure 3 .

 3 Figure 3. Transfer map τi from image plane Ωi into target image plane Γ. The depth uncertainty σz i may be different among pixels.

  tarot (CD) couple (CD) buddha (PD) maria (PD) still life (SR) truck (SR) Original Wanner et al.

Figure 4 .

 4 Figure 4. Visual comparison of novel views obtained for different light fields.From top to bottom, the rows present closeups of the ground truth images, the results obtained by[START_REF] Cho | Image restoration by matching gradient distributions[END_REF], and our results. CD stands for computed disparity, PD for planar disparity and SR for superresolution, see text for details. Full resolution images can be found in the additional material. The results obtained by the proposed method are visibly sharper, in particular along color edges.