Jinglei Shi

Xiaoran Jiang

IEEE Christine Guillemot Fellow

Deep Residual Architecture Using Pixel and Feature Cues for View Synthesis and Temporal Interpolation

Keywords: Light field, view synthesis, pixel and feature cues, residual connection, video frame interpolation

In this paper, we propose a deep residual architecture that can be used both for synthesizing high quality angular views in light fields and temporal frames in classical videos. The proposed framework consists of an optical flow estimator optimized for view synthesis, a trainable feature extractor and a residual convolutional network for pixel and feature-based view reconstruction. Among these modules, the fine-tuning of the optical flow estimator specifically for the view synthesis task yields scene depth or motion information that is well optimized for the targeted problem. In cooperation with the end-to-end trainable encoder, the synthesis block employs both pixel-based and feature-based synthesis with residual connection blocks, and the two synthesized views are fused with the help of a learned soft mask to obtain the final reconstructed view. Experimental results with various datasets show that our method performs favorably against other state-of-the-art (SOTA) methods with a large gain for light field view synthesis. Furthermore, with a little modification, our method can also be used for video frame interpolation, generating high quality frames compared with SOTA interpolation methods.

I. INTRODUCTION

S CENE sampling rate plays a key role in determining the perception quality of visual content like videos or immersive content such as light fields (LF). However, the video or light field sampling rates can be limited due to technological limitations, e.g. the sensor readout speed, its resolution, the storage space or transmission bandwidth [START_REF] Wilburn | High performance imaging using large camera arrays[END_REF], [START_REF] Ng | Light field photography with a hand-held plenoptic camera[END_REF]. A small video frame rate can lead to jerky motion while undersampled angular light field views have large parallax and can yield perspective discontinuities when navigating within the scene. Video frame rate conversion as well as light field view synthesis have therefore become widely investigated problems with specific solutions.

Existing video frame interpolation (VFI) approaches consist of synthesizing intermediate frame from given input frames and they can be roughly classified into three categories, i.e. optical flow [START_REF] Liu | Video frame synthesis using deep voxel flow[END_REF]- [START_REF]Softmax splatting for video frame interpolation[END_REF] or feature flow-based schemes [START_REF] Gui | Featureflow: Robust video interpolation via structure-to-texture generation[END_REF], kernelbased schemes [START_REF] Niklaus | Video frame interpolation via adaptive convolution[END_REF]- [START_REF] Bao | Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement[END_REF] and phase-based ones [START_REF] Meyer | Phase-based frame interpolation for video[END_REF], [START_REF] Meyer | Phasenet for video frame interpolation[END_REF].

Light field view synthesis (LFVS) methods can similarly be roughly categorized as depth-dependent methods and depthindependent ones. Depth-dependent schemes [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF]- [START_REF] Mildenhall | Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[END_REF] consist in first estimating scene depth which is then used for image warping. Kalantari et al. [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] use two cascaded networks respectively dedicated to depth prior estimation and color synthesis. Penner et al. [START_REF] Penner | Soft 3D reconstruction for view synthesis[END_REF] propose instead a soft 3D reconstruction method in which a confidence measure of depth helps for better handling boundary artifacts during the synthesis. Mildenhall et al. [START_REF] Mildenhall | Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[END_REF] slice the scene into Multi-Plane Images (MPIs) at different depths, then warp and fuse these planes according to the corresponding depth clues. Another method in [START_REF] Wu | Learning sheared epi structure for light field reconstruction[END_REF] relies on Epipolar-Plane Images (EPI) shearing leading to a stack of sheared EPI which can be seen as a plane sweep volume as used in [START_REF] Penner | Soft 3D reconstruction for view synthesis[END_REF] and [START_REF] Mildenhall | Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[END_REF]. The above methods, that we refer here as depth-dependent methods, rely on geometry estimation for correctly warping images from one viewpoint to another. Other methods, that we refer here as depth-independent methods, do not require a prior geometry estimation. They instead retrieve views by exploiting signal priors such as sparsity [START_REF] Shi | Light field reconstruction using sparsity in the continuous fourier domain[END_REF] or smoothness of EPIs [START_REF] Vagharshakyan | Light field reconstruction using shearlet transform[END_REF], [START_REF] Wang | End-to-end view synthesis for light field imaging with pseudo 4DCNN[END_REF]. The Neural Radiance Field (NeRF) concept has been introduced in [START_REF] Mildenhall | Nerf: Representing scenes as neural radiance fields for view synthesis[END_REF] to model the mapping between the 5D spatial and angular coordinates of light rays emitted by the scene into its three RGB colour components and a volume density measure. This mapping is learned for each scene to be processed from densely captured images, using a multi-layer perceptron and assuming that the camera pose parameters are known. The model can then be efficiently used for synthesizing novel light field views. The neural radiance field learning has been further optimized in [START_REF] Wang | NeRF--: Neural radiance fields without known camera parameters[END_REF] to avoid prior computation of camera pose parameters, in [START_REF] Chen | MVSNeRF: Fast generalizable radiance field reconstruction from multiview stereo[END_REF] and [START_REF] Chibane | Stereo radiance fields (SRF): Learning view synthesis for sparse views of novel scenes[END_REF] to enable faster inference as well as using a sparser set of input views, and to enable generalization to new scenes.

In this paper, we describe a deep residual network architecture that can be used for both light field view synthesis (LFVS) and temporal video frame interpolation (VFI). For the light field view synthesis problem, the proposed architecture takes a sparse subset of light field views and output a whole light field at any desired angular sampling rate. The proposed architecture builds upon the FPFR solution we proposed in [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF]. Both approaches adopt the idea of merging a pixel-based view reconstruction and a feature-based reconstruction. However, the architecture in [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF] suffers from limitations in terms of performance, slow convergence and training instability. We propose in this paper very effective improvements both in terms of architecture design and training schedule in order to address each of these limitations, as follows:

• We introduce a light-weight feature extractor (1.9M vs 2.4M parameters) which consumes less memory and can be trained end-to-end to extract deep features that are better suited for the reconstruction problem at hand. This improves both the performance and the convergence of the entire network. • We replace the convolutional layers of PixRNet and FeatRNet in [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF] by residual convolutional blocks. On one hand, the residual blocks deepen PixRNet and FeatR-Net, enhancing their synthesis capability. On the other hand, the residual connections in the convolutional blocks guarantee convergence speed of the networks (by avoiding gradient vanishing). These residual blocks hence also improve performance and network convergence. • The weights of the optical flow estimator are optimized for the view synthesis problem or the temporal frame interpolation problem, in an un-supervised manner, i.e. without using ground truth optical flows or disparity maps. This operation is essential for the VFI task, since a flow estimator trained on synthetic data (for which ground truth motion is available), when applied to real-world frames, may lead to network collapse. The estimator optimized for view synthesis improves the training stability.

Note that there is a strong similarity between the LFVS and VFI tasks. Light field views can indeed be seen as frames of a video, in the special case of a static scene captured with a moving camera. Therefore, we further show how the proposed architecture can be adapted to the temporal video interpolation problem, where the scenes can be dynamic with moving objects. We show that this novel architecture gives synthesized views of a higher quality compared with recent reference methods, such as DeepVS [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF], Soft3D [START_REF] Penner | Soft 3D reconstruction for view synthesis[END_REF], LLFF [START_REF] Mildenhall | Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[END_REF], EPI [START_REF] Wu | Learning sheared epi structure for light field reconstruction[END_REF], and FPFR [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF], for both dense and sparse light fields. Our comparative analysis with the NeRF-based view synthesis method [START_REF] Mildenhall | Nerf: Representing scenes as neural radiance fields for view synthesis[END_REF], for different input view configurations, show that the NeRF method performs quite well, provided the radiance field is learned from a sufficient number of input views, but that the proposed method gives better synthesis results for a small number of input views. This advantage is further analysed in light of storage or compression implications. We indeed show the PSNR-rate benefits of the proposed method compared with NeRF and with several compression methods. Finally, we show that the proposed architecture can also be used for video frame rate conversion with high quality which is quite competitive compared with the one obtained with reference methods, e.g., [START_REF]Video frame interpolation via adaptive separable convolution[END_REF], [START_REF] Jiang | Super slomo: High quality estimation of multiple intermediate frames for video interpolation[END_REF], [START_REF] Bao | Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement[END_REF], [START_REF] Gui | Featureflow: Robust video interpolation via structure-to-texture generation[END_REF] and [START_REF]Softmax splatting for video frame interpolation[END_REF], specifically dedicated to the temporal frame interpolation task.

II. RELATED WORK

Light field view synthesis and video frame temporal interpolation methods follow very similar principles. Although developed in parallel with methods specific to each problem, they can be similarly classified in two categories. One category of methods first estimates disparity maps or optical flows which are then used to warp input images in either the angular or temporal dimensions. A second category of methods does not rely on explicit depth or motion information but instead replies on signal priors that are specific to the input data characteristics.

A. Light field view synthesis 1) Depth-dependent schemes: Depth-dependent schemes follow the principles of image based rendering methods, as e.g. in [START_REF] Penner | Soft 3D reconstruction for view synthesis[END_REF], [START_REF] Gortler | The lumigraph[END_REF]- [START_REF] Zhang | Light field from micro-baseline image pair[END_REF], which have been prevailing for many years in the field of view synthesis. Along the same lines, but using deep neural networks for both depth estimation and view synthesis, one finds the approach of Kalantari et al. [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] in which the authors sequentially connect two convolutional neural networks (CNNs) dedicated respectively to depth estimation and color fusion. The depth estimation network exploits cues that are well suited for dense light fields with small parallax. Hence, the method, mostly designed for dense structured light fields, fails in occluded regions, especially for sparse light fields having large baselines. Due to depth imprecision, the synthesized views also suffer from blurriness, tearing and ghosting effects. The method in [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF] first estimate disparity using a learned optical flow estimator which is then used to warp not only the input views but also feature representations of these views. This leads to pixelbased and feature-based synthesized views which then merged using a learned soft mask. This method gives state-of-the-art results.

To cope with imprecision of disparity estimators, the concept of Multi-Plane Image (MPI) representation has been introduced in [START_REF] Zhou | Stereo magnification: Learning view synthesis using multiplane images[END_REF] for stereo views, and then used in [START_REF] Mildenhall | Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[END_REF] to render novel views from irregularly sampled views in unstructured light fields. In [START_REF] Mildenhall | Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[END_REF], an MPI representation is learned for each source view, from plane sweep volumes (PSVs), and the source MPIs are then warped and merged to synthesize the target view. MPIs exploit notions of visibility or transparency with alpha blending maps [START_REF] Zhou | Stereo magnification: Learning view synthesis using multiplane images[END_REF], [START_REF] Flynn | Deepstereo: Learning to predict new views from the world's imagery[END_REF]. In the same vein, Wu et al. [START_REF] Wu | Learning sheared epi structure for light field reconstruction[END_REF] construct a stack of sheared EPIs, that can be seen as a PSV, which are then fused to reconstruct light fields. They train a CNN to give a score which is then used for the fusion of sheared EPIs.

2) Depth-independent schemes: The use of various signal priors has also been investigated for view synthesis, often regarding the problem as a problem of angular super-resolution. For example, the authors in [START_REF] Shi | Light field reconstruction using sparsity in the continuous fourier domain[END_REF] reconstruct dense light field views from a subset of samples by exploiting sparsity of the continuous Fourier spectrum. The authors in [START_REF] Vagharshakyan | Light field reconstruction using shearlet transform[END_REF] apply a shearlet transform-based inpainting technique on EPIs to generate densely sampled light fields, EPI per EPI. Variational methods imposing some smoothness contraints on EPI have also been considered for view synthesis [START_REF] Wanner | Variational light field analysis for disparity estimation and super-resolution[END_REF]. Similarly, the authors in [START_REF] Wang | End-to-end view synthesis for light field imaging with pseudo 4DCNN[END_REF] use a pseudo 4DCNN to upsample the input light field in the angular dimension. Boundary artifacts appear when the target views are far from the source views. The above schemes are effective on narrow-baseline light fields, but usually fail with those having large baselines. The authors in [START_REF] Meng | High-dimensional dense residual convolutional neural network for light field reconstruction[END_REF] describe a spatio-angular restoration network using 4D convolutions and high-resolution residual blocks, in order to extract spatial features that preserve geometrical properties. To further exploit scene geometry, and better handle occlusions and non lambertian scenes, the method in [START_REF] Mildenhall | Nerf: Representing scenes as neural radiance fields for view synthesis[END_REF] optimizes a 5D neural radiance field representation, with volume density and view-dependent color at any location of a scene, from a set of input images. The authors then use volume rendering techniques to sample the scene representation along rays, and render the scene from any viewpoint. This seminal work gave rise to several variants of NeRF learning methods, such as the so-called MVSNeRF [START_REF] Chen | MVSNeRF: Fast generalizable radiance field reconstruction from multiview stereo[END_REF] that aims at reconstructing radiance fields from a sparse set of views and via faster inference, or such as NeRF- [START_REF] Wang | NeRF--: Neural radiance fields without known camera parameters[END_REF] that avoids a prior estimation of camera pose parameters. Stereo radiance fields (SRF) are also introduced in [START_REF] Chibane | Stereo radiance fields (SRF): Learning view synthesis for sparse views of novel scenes[END_REF] that can generalize the learning of the radiance field to new scenes.

B. Video frame interpolation

1) Flow-based interpolation: Flow-based methods consists in first estimating optical flows between given frames, and then in interpolating or extrapolating target frames along the motion vectors. This is the case of the Deep Voxel Flow (DVF) in [START_REF] Liu | Video frame synthesis using deep voxel flow[END_REF] in which a network is trained to estimate the optical flows and produce a temporal mask for trilinear interpolation of the input frames. The authors in [START_REF] Jiang | Super slomo: High quality estimation of multiple intermediate frames for video interpolation[END_REF] use a first U-Net network to predict bi-directional optical flows which are then combined to approximate the intermediate optical flows. A second U-Net refines the intermediate optical flows as well as predicts soft visibility maps in order to warp and linearly merge the input frames. In [START_REF] Niklaus | Context-aware synthesis for video frame interpolation[END_REF], [START_REF]Softmax splatting for video frame interpolation[END_REF], the authors instead employ the stateof-the-art PWC-Net [START_REF] Sun | PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume[END_REF] for optical flow estimation. They then warp input frames and corresponding deep features to generate the interpolated frames. The problem of occlusion handling is further addressed in [START_REF] Bao | Depth-aware video frame interpolation[END_REF] by using depth information in addition to optical flows, together with local interpolation kernels. A recent method in [START_REF] Gui | Featureflow: Robust video interpolation via structure-to-texture generation[END_REF] uses a multi-flow multiattention generator to estimate feature flows instead of optical flows. The feature flows are then used to warp and interpolate feature maps of the input frames into the target position. Instead of considering only two input frames for interpolating an intermediate frame, the authors in [START_REF] Choi | High-quality frame interpolation via tridirectional inference[END_REF] propose a framework utilizing three frames to remove tearing and ghosting artifacts of moving objects.

2) Flow-free interpolation: A second category of methods can be referred to as flow-free interpolation methods. This is the case of kernel-based methods [START_REF] Niklaus | Video frame interpolation via adaptive convolution[END_REF], [START_REF]Video frame interpolation via adaptive separable convolution[END_REF] which, instead of estimating optical flows, estimate a series of spatially-adaptive interpolation kernels used to convolve the input frames in order to produce the interpolated ones. Besides the effort for improving the estimation of kernels reported in [START_REF] Niklaus | Video frame interpolation via adaptive convolution[END_REF], [START_REF]Video frame interpolation via adaptive separable convolution[END_REF], the authors in [START_REF] Niklaus | Revisiting adaptive convolutions for video frame interpolation[END_REF] also propose a set of techniques that can be applied to kernel-based methods to further improve the performance. Kernel-based methods are computationally expensive and do not incorporate explicit mechanisms for occlusion handling. Please note that such spatially-adaptive filters and optical flows can nevertheless be combined as in [START_REF] Bao | Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement[END_REF]. Phase-based methods [START_REF] Meyer | Phase-based frame interpolation for video[END_REF], [START_REF] Meyer | Phasenet for video frame interpolation[END_REF] represent the motion as a per-pixel shift and operate phase modification for each pixel. However, these methods can not reach the same level of details as flow-based methods.

III. BACKGROUND AND NOTATIONS

In the following we consider both video sequences and light fields. Considering a video sequence, we denote a frame captured at instant t as I t in the following sections. A light field is represented by a 4D function L(x, y, u, v) [START_REF] Levoy | Light field rendering[END_REF], where (x, y) ∈ 1, X × 1, Y and (u, v) ∈ 1, U × 1, V are respectively the light field spatial and angular coordinates. A light field can be interpreted as an array of views observed from different viewpoints, and we denote the view L(x, y, u 0 , v 0 ) observed from a certain viewpoint i = (u 0 , v 0 ) as I i (or I u0,v0 ) for sake of notation simplicity.

IV. METHODOLOGY A. Architecture overview

The proposed architecture aims at reconstructing a densely sampled light field from a subset of input views, thus performing angular view synthesis, and we show that it can also be used to reconstruct a video sequence from a subset of input frames. The overall architecture is depicted in Fig 1 .  For both problems, it is composed of the same three modules. The first module Flow estimation (blue) which will compute disparity maps for the angular view synthesis problem and optical flows for the frame temporal interpolation problem. The flow estimation module is followed by a pixel-based view synthesis module, PixRNet (orange), and by a featurebased view synthesis module, called FeatRNet (violet). The outputs of these two synthesis modules are then merged using a learned soft fusion mask generated by a network called FusNet (green).

Depending on the targeted problem, the application of the above architecture varies only in terms of input images and in the semantic of the estimated flows. In the case of light field angular view synthesis, the inputs of the architecture are 2 × 2 light field views. These four inputs, according to their relative positions, are respectively denoted as I tl (top left), I tr (top right), I bl (bottom left) and I br (bottom right) for convenience. In the case of video frame temporal interpolation, the inputs of the architecture are two temporally adjacent frames I 0 and I 1 and the architecture produces temporally intermediate frames. Depending on the targeted problem, the flows estimated by the Flow estimation module are disparity maps in the case of light field angular view synthesis, whereas are they are bi-directional optical flows in the case of temporal frame interpolation.

B. Flow estimation

The Flow estimation module predicts disparity maps for angular view synthesis or bidirectional motions for temporal interpolation.

1) Disparity maps: The notion of 'disparity' refers to the horizontal and vertical displacement of points in the image plane when moving from one viewpoint to the other one [START_REF] Mayer | A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[END_REF], [START_REF] Liang | Learning for disparity estimation through feature constancy[END_REF].

Given four corner views I tl , I tr , I bl , I br , we can actually estimate two disparity maps for each view, one is from the horizontal view pair and the other is from the vertical view pair. Taking I tl as an example, two corresponding disparity maps d 1 , d 2 are obtained from two image pairs as follows: where FNet is a flow-estimation network, the subscript h refers to the horizontal component of the output optical flow. R and R -1 are respectively counterclockwise and clockwise rotation of 90 • , which permits us to convert vertical disparities to horizontal ones. In our work, we employ a state-of-the-art flow estimation network PWC-Net [START_REF] Sun | PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume[END_REF] as FNet, with a novel finetuning strategy that optimizes its weights for the targeted view synthesis or temporal interpolation tasks without using any ground truth optical flows. The training strategy hence differs from the one in [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF], where we finetune PWC-Net with ground truth disparity maps between light field image pair, More details will be presented in IV-C2. Let us note that any other flow estimation network can likewise be used in our architecture.

d 1 = FNet(I tl , I tr ) h , (1)
d 2 = R -1 • FNet(R(I tl ), R(I bl )) h , (2)
Both

e n = i∈{tr,bl,br} RGB (I tl -Ĩn i→tl ), n = 1, 2, (3)
where Ĩn i→tl represents image warped from current position i to top left corner using disparity d n . The fusion of d 1 and d 2 is performed via value selection for each pixel p:

n = arg min n e n (p), d tl (p) = d n (p). (4)
Similarly, we can obtain disparity maps d tr , d bl and d br for the other three views.

2) Bidirectional optical flows: In the case of temporal interpolation task, the flow estimation module takes two input frames I 0 and I 1 and output a forward flow (from instant 0 to 1) and a backward flow (from instant 1 to 0):

f low 0→1 = FNet(I 0 , I 1 ), (5)
f low 1→0 = FNet(I 1 , I 0 ). (6)
The estimated optical flows have horizontal and vertical components. Moreover, as there is only one estimated motion for each view, further refinement by fusion is not required. Both flows are used for forward and backward warping when performing the temporal interpolation, as explained later in Section IV-C1.

C. View warping and Fusion

Based on the estimated optical flows or disparity maps, the input views and their feature maps are warped onto the target view position j via a differentiable forward warping operation while computing at the same time binary occlusion masks. The warped images and feature maps are respectively fed into the PixRNet and FeatRNet synthesis modules along with the occlusion masks to synthesize two views Îpix j and Îfeat j . The image Îpix j obtained through a pixel-wise synthesis in PixRNet, is accurate in homogeneous regions but tend to be blurred in highly textured regions and at object boundaries. This can be explained by the fact that pixels to be fused are not fully aligned in such regions due to disparity value inconsistencies among the input views. In the FeatRNet module, deep features covering different receptive fields are warped to the desired position and, thanks to a pyramidal reconstruction, the retrieved Îfeat j is more accurate in textured regions. Finally, the FusNet module learns a soft mask M to merge Îpix j and Îfeat j into an image Îj which well reconstructs both homogeneous and textured regions.

1) Image forward warping: Backward warping (BW) has been a widely used operation for various image processing tasks, e.g. image classification [START_REF] Jaderberg | Spatial transformer networks[END_REF], depth image based rendering (DIBR) [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF], [START_REF] Mildenhall | Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[END_REF] etc.It consists in projecting pixels from one source view to a target one, using the optical flow associated to the target view, hence requires the optical flow at the target position. We use instead a forward warping (FW) operation, which does not require flow estimation at the target position, and can moreover better handle occlusions via disparity-dependent interpolation. This FW operation proceeds as follows.

Let us assume that we would like to project a pixel p = (x p , y p ) from a source view to a target view. In the FW procedure, given a source viewpoint i, a target viewpoint j and a disparity map d i in the case of light field view synthesis, or given optical flow f low 0→1 between source frames I 0 , I 1 and the target intermediate instant t in the case of video frame interpolation, the non-integer coordinates of the projected pixel p = (x p, y p) can be computed as:

p = p + (j -i)d i (p), for LFVS p + t × f low 0→1 (p), for VFI (7)
where the term t × f low 0→1 approximates the flow f low 0→t between the frame I 0 and the frame at time instant t, which is then used to warp pixels of frame I 0 . The frame I 1 is similarly warped using

f low 1→t = (1 -t) × f low 1→0 .
The pixel value Ĩi→j (q) (or Ĩ0→t (q)) at integer coordinates q = (x q , y q ) of the warped image will be interpolated as a weighted sum of nearby source pixels I i (p) (or I 0 (p)):

I i→j (q) = p I i (p)W (p, q) p W (p, q) + , (8)
where W (p, q) is an interpolation weight term, and is a very small value for numeric stability. Three properties are considered during its design: 1-). this term should be totally differentiable for end-to-end learning. 2-) the further a pixel is from the target position, the smaller weight it will have for interpolation. 3-) pixel overlaps must be handled in the occluded regions. Based on these considerations, this term is finally proposed as the product of a distance term w D and an overlap handling term w O :

W (p, q) = w D (p, q)w O (p), (9)
where

w D (p, q) = l(x p , x q )l(y p , y q ), (10)
with

l(x 1 , x 2 ) = (1 -|x 1 -x 2 |), if |x 1 -x 2 |< 1; 0, otherwise. (11)
The term w O is used for handling overlapping warped pixels, since pixels from foreground and background are often overlaid in occluded regions. In the light field view synthesis problem, we use the disparity maps d i of each input view to compute this weight. However, disparity maps are not available for the video frame interpolation problem, and since scene depth is difficult to estimate from a monocular video, we use the warping error for handling overlapping pixels.

The exponential function gives more importance to pixels having smaller disparity value or brightness error and less importance to those with larger disparity value or brightness error. The brightness error e 0 of I 0 is calculated as

e 0 = RBG |I 0 -BW(I 1 , f low 0→1 )|, (13)
where I 0 and I 1 are the two input frames and BW denotes the backward warping operation.

Non occupied pixel positions on the target view after pixel warping are indicated as disoccluded positions via a binary mask M . However, to detect disoccluded pixels in a differentiable manner, and to handle interpolation in disoccluded positions, we also warp the weight map w O . For each warped position, we thus have RGB and weight values. If there is no pixels in the neighbourhood, then w D is 0 in all neighbourhood positions, indicating, in a differentiable manner, that the corresponding pixel position is a disoccluded pixel. The warped weights are also used to compute the interpolation weights.

In summary, by taking a source image along with its disparity map or optical flow and brightness error, the FW operation outputs a warped image and a binary disocclusion mask as follows:

Ĩi→j , Mi→j = FW(I i , d i , i, j) (14)
for LFVS, or

Ĩ0→t , M0→t = FW(I 0 , f low 0→t , e 0 ) (15)
for VFI. Both disparity and brightness error are effective cues in handling pixel overlaps, as the larger is the brightness error of a pixel, or the larger is its disparity value, the more likely it will be overlaid after warping. Similar brightness error-based image warping operation, called 'softmax splatting', can be found in [START_REF]Softmax splatting for video frame interpolation[END_REF]. And forward splatting operation is also applied in [START_REF] Tulsiani | Layer-structured 3D scene inference via view synthesis[END_REF] to infer a layer-structured 3D representation of a scene from a single image.

2) Finetuning the flow estimation module: Depthdependent LFVS approaches such as in [START_REF] Penner | Soft 3D reconstruction for view synthesis[END_REF], [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF] and flow-based VFI schemes such as in [START_REF] Bao | Depth-aware video frame interpolation[END_REF]- [START_REF]Softmax splatting for video frame interpolation[END_REF], using an offthe-shelf disparity or optical flow estimation module, both rely on a good initialization of that module whether or not end-to-end finetuning is conducted later. However, most of these concerned networks are trained on synthetic datasets with ground truth disparities [START_REF] Mayer | A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[END_REF], [START_REF] Ilg | Occlusions, motion and depth boundaries with a generic network for disparity, optical flow or scene flow estimation[END_REF] or optical flows [START_REF] Butler | A naturalistic open source movie for optical flow evaluation[END_REF], [START_REF] Dosovitskiy | Flownet: Learning optical flow with convolutional networks[END_REF], and are not fully adapted to the view synthesis problem. Here, based on our aforementioned FW operation, we propose instead a simple but effective finetuning procedure that enables us to finetune the initial flow estimation module under view synthesis or frame-interpolation supervision, i.e. without any ground truth disparity maps or optical flows.

More specifically, in the case of LFVS, given four input views I tl , I tr , I bl , I br and corresponding disparity maps d tl ,

|( Ĩi→j -I j ) Mi→j | 1 ( 16
)
where θ is the set of parameters of the FNet module, and Ĩi→j , Mi→j are obtained by the FW operation in Eq. 14.

In the case of VFI, given the input frames I 0 , I 1 and the associated estimated flows f low 0→1 , f low 1→0 , we interpolate a frame at an intermediate instant t, by optimizing

argmin θ t ∈{0,1} |( Ĩt →t -I t ) Mt →t | 1 , (17)
where Ĩt →t , Mt →t are obtained via Eq. 15, with f low 0→t = t × f low 0→1 and f low 1→t = (1 -t) × f low 1→0 . Fig. 2 illustrates the warped images obtained before and after finetuning. Before finetuning, we can observe severe deformations in regions like bear's ear and licence plate that are challenging for the synthesis process. These artifacts are corrected thanks to the finetuning procedure.

3) Pixel and feature-based reconstruction: Thanks to the finetuned Flow estimation module, we can obtain disparity maps or optical flows optimized for synthesis. The synthesis proceeds in parallel in both pixel and deep feature domains.

Pixel-based reconstruction is performed by the PixRNet module, where, in the case of LFVS, using estimated disparity maps and the forward warping operation, four corner views are first projected to the target position j to generate the warped images { Ĩtl→j , Ĩtr→j , Ĩbl→j , Ĩbr→j } and the corresponding occlusion masks { Mtl→j , Mtr→j , Mbl→j , Mbr→j }. The warped images and occlusion masks are then concatenated to be fed into the PixRNet module. Similarly, in the case of VFI, warped frames { Ĩ0→t , Ĩ1→t } and masks { M0→t , M1→t } are instead concatenated and fed into the PixRNet module to learn the reconstructed Îpix t . Disoccluded pixels are clearly indicated by the occlusion masks, which hence makes it easier for PixRNet to inpaint these disoccluded regions while fusing different views into the target view Îpix j . Different from the LFVS approach in [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF] that cascades four simple convolutional layers, we replace the first three layers with novel residual convolutional block (RCB) architecture illustrated in Fig 3 . A RCB is composed of three convolutional layers, where the first layers with kernel size 1 × 1 convert C1 input channels to C2 channels. Then two cascaded convolutional layers with kernel size 3 × 3 learn a residual map acting on the input. Moreover, in comparison with the original design in [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF], we quadruple the number of channels in RCB to further enhance its learning ability. The usage of RCB improves the reconstruction efficiency. More details will be investigated in Sec. VI-C.

Due to inconsistency between disparities or optical flows of input views, those warped views or frames are not fully aligned and the fusion may yield blurriness especially in highly textured regions and object contours. To overcome this limitation, we propose using an additional feature-based reconstruction module named FeatRNet, in which a trainable encoder is employed to flexibly extract deep features at three different resolutions later used in a pyramidal reconstruction step.

In comparison with the VGG19 encoder with fixed weights used in [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF], our trainable encoder is 1-) more lightweight than the VGG19 encoder (1.9M parameters vs 2.3M parameters). 2-) more flexible to extract features suitable for synthesis. The feature maps of input views are extracted at three different resolutions as:

{f 1 i , f 2 i , f 3 i } = F eatExt(I i ), ∀i ∈ {tl, tr, bl, br} (18)
where F eatExt denotes the feature extraction operation and the extracted feature maps f s+1 i are at half resolution compared with f s i . Like in the PixRNet module, these feature maps are then warped to the desired position j using forward warping as

f s i→j , ms i→j = FW(f s i , d i , i, j). (19)
Warped features and masks at lower levels are first concatenated and fused by RCBs, and then 2× upsampled by a deconvolutional layer to be fed to the next level. Finally, at the highest level, the warped feature maps and the occlusion masks along with upsampled features are merged to get the desired view Îfeat j . In case of VFI, the FeatRNet module will take two frames as inputs and follow the same reconstruction process to obtain Îfeat t . Let us note that in [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF], feature maps of the ground truth image are extracted using a fixed VGG19 encoder to supervise the feature reconstruction at each level of the FeatRNet module. With a trainable encoder, the extracted feature maps of the ground truth image vary with the learned encoder weights, and supervising the feature reconstruction with varying ground truth feature maps results in model performance oscillations during training. We therefore no longer supervise feature reconstruction and directly focus on the final reconstruction of Îfeat ) have complementary properties in homogeneous and highly textured regions, we want the synthesis to benefit from the advantages of both. A FusNet module is therefore employed to merge two reconstructed views into the final target view Îj (or Ît ) with a soft mask M having value between 0 and 1 (forced by sigmoid activation) as follows

Îj = M Îpix j + (1 -M ) Îfeat j . ( 20
)
To train the entire architecture, we propose a loss function composed of three reconstructed views as follows

L = γ 1 L l ( Îpix j , I j ) + γ 2 L l ( Îfeat j , I j ) + γ 3 L l ( Îj , I j ), (21)
where L l represents a Laplacian loss [START_REF] Bojanowski | Optimizing the latent space of generative networks[END_REF] with 3 levels.

V. IMPLEMENTATION DETAILS A. Training data preparation

In the case of LFVS, we employ 78 synthetic light fields proposed in [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF] and 16 light fields in 'Additional' category of [START_REF] Honauer | A dataset and evaluation methodology for depth estimation on 4D light fields[END_REF] to form our synthetic training set. All light fields are of resolution 512 × 512 × 9 × 9. We also use 100 scenes proposed in [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF], which are captured by a Lytro Illum camera with resolution 376 × 541 × 14 × 14, by extracting the 8 × 8 central views of each scene to form our real-world training set. Since real-world data contains complicated lighting conditions and artifacts caused by hardware limitations, making it distinct from synthetic data, we therefore prepare two models, one trained on synthetic data and the other further finetuned on real-world data. The finetuning of FNet and the training of the whole pipeline are conducted using the same dataset. As light fields from DLFD and SLFD in [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF] and those in [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] captured by a Lytro Illum camera, are with different baselines, we therefore adopt different configurations when extracting views: for light fields in DLFD that have narrow baseline, four corner views are selected by an angular distance corresponding to view index difference l ∈ {5, 6, 7, 8}. For light fields in SLFD having wide baseline, corner views are extracted with l ∈ {2, 3}. When using the real-world training set,the corresponding angular distance is set to be l ∈ {6, 7}. The ground truth view is randomly selected within the square defined by corner views. We use randomly cropped patches of size 160 × 160 and a batch size of 8 in the training step.

For VFI, we adopt the widely used Vimeo90K dataset in [START_REF] Xue | Video enhancement with task-oriented flow[END_REF] for training. This training set contains 51,313 video triplets with resolution 256 × 448. We use the intermediate frame as ground truth and other two frames as inputs, and randomly crop patches in size 160 × 160 during training, the batch size is set to 18.

B. Data augmentation

We also apply geometrical and chromatic transformations to increase the variety of the training data. The chromatic transformation is carried out by changing the contrast, color balance, brightness and gamma value. The contrast is sampled within the interval [-0.8, 0.4], the color balance is adjusted with a multiplicative factor sampled within [0.5, 2], the brightness is modified by adding a value drawn according to a Gaussian distribution with σ = 0.2, and the gamma value is sampled within the interval [0.7, 1.5]. For the geometrical transform, we randomly rotate the image by 0 • , 90 • , 180 • or 270 • counterclockwise and flip it on the left and right.

C. Training schedule

The training schedule of the model can be divided into an initialization step followed by an end-to-end optimization step. In the initialization step, we fix the weights in the FNet module and set γ 1 = γ 2 = γ 3 = 1 in the loss function. The training will enable both PixRNet, FeatRNet and FusNet to generate views that approximate the ground truth view. We set the learning rate to 0.0001 for this step, and the whole pipeline is trained for 3000 epoches in the case of VSLF and for 30 epoches in the case of VFI.

Then, by setting γ 1 = γ 2 = 0 and γ 3 = 1 in the loss function, we end-to-end train the framework to optimize the final synthesized view, which makes PixRNet and FeatRNet to retrieve views with complementary properties. The learning rate in this stage is set to 0.00001 and we stop the training when there is no performance amelioration within 2000 epoches for LFVS or 20 epoches for VFI. The training takes about 4 days for LFVS and 5 days for VFI on a Nvidia Tesla V100 GPU with 32GB GRAM, the pipeline is implemented using the tensorflow framework. Our method has about 18.2M parameters (where 9.3M parameters are from PWC-Net), and it takes about 0.9s to synthesize an image of size 512×512×3 (36.4M FLOPS).

VI. EXPERIMENTAL RESULTS

In this section, we evaluate the effectiveness of our proposed method for both LFVS and VFI tasks using various datasets.

A. Light field view synthesis 1) Synthetic LFs: We first validate the effectiveness of the proposed architecture using public synthetic light field datasets:

• densely sampled light fields stilllife, buddha, butterfly, monasRoom with resolutions 768 × 768 × 9 × 9 from the old HCI dataset [START_REF] Wanner | Datasets and benchmarks for densely sampled 4D light fields[END_REF], and boxes, cotton, dino, sideboard with resolutions 512 × 512 × 9 × 9 from new HCI dataset [START_REF] Honauer | A dataset and evaluation methodology for depth estimation on 4D light fields[END_REF],

• sparsely sampled light fields Toy bricks, Elec devices, Two vases, Sculpture, Bear with resolutions 512 × 512 × 9 × 9 from the INRIA synth dataset [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF] We focus on the central 7×7 and 3×3 angular views in densely and sparsely sampled light fields respectively, synthesizing all views, given the four corner views.

Our approach is compared with state-of-the-art approaches, e.g. learning-based DIBR pipeline [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] (DeepVS), EPI-based view synthesis [START_REF] Wu | Learning sheared epi structure for light field reconstruction[END_REF] (EPI), synthesis using multi-layer scene representation with learning [START_REF] Mildenhall | Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[END_REF] (LLFF) and without learning [START_REF] Penner | Soft 3D reconstruction for view synthesis[END_REF] (Soft3D), and with our previous pixel-feature based reconstruction scheme [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF] (FPFR). During the test, all involved methods take four views as inputs, except LLFF, which take an extra fifth view as its released model requires at least five input views. We thus employ the horizontal immediate neighbour of I tl as the fifth view. Furthermore, we directly use ground truth camera pose instead of estimated ones in LLFF to make sure the reconstruction quality only depends on the synthesis process. Given that training data plays a critical role in determining the performance of learning-based methods, we further finetuned pre-trained models of compared methods using the same training data as for our method for having a fair comparison.

Table I gives a quantitative evaluation of the synthesized central view in terms of PSNR, with all tested light fields arranged from dense to sparse according to the disparity range. In the last two columns, we show the performance of the vanilla version (Ours) and test-time augmented version (Ours*) of our method. The test-time augmentation is achieved by averaging eight reconstructed views, whose corresponding input views are rotated and flipped, after inverse rotation and flip processing. We can observe from the table that our novel pipeline outperforms the state-of-the-art methods with most scenes, and in average gives about 0.8dB gain against the best reference method. Besides testing on the central view that is the most distant from corner views, we further trace the PSNR curves when varying the target position for the different methods, Fig 6(a) shows the synthesis quality evolution in terms of viewpoints. We can observe that our method outperforms reference ones with a very large margin. Furthermore, in comparison with the FPFR network, our novel architecture gains in average 1dB for views distant from the corner views, and even more for views that are near the source input views.

2) Real-world LFs: We have also tested our method using real-world scenes from the testset of [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF]. We feed four views to the tested methods except for LLFF, which takes a fifth view. As the ground truth camera pose is unavailable for realworld data, we hence convert the camera pose-based view transform to a disparity-based transform when constructing the PSV in LLFF. Table II shows the quantitative measures obtained for the central reconstructed views. A gain of 0.5dB is observed against FPFR, the best reference method.

Fig. 5 shows reconstruction error maps for both synthetic and real-world light fields, we can observe that our method reconstructs views with less errors, especially on the object contours and subtle structures in highly textured regions. More experimental results are available in our project homepage: http://clim.inria.fr/research/TCI-VS-VI/index.html.

3) View extrapolation in LFs: Extrapolating views beyond the original field of view of the light field is a very challenging task, as less information on the target view is available in the input views. We evaluate the extrapolation capability of our method in comparison with two reference methods, namely the FDL approach in [START_REF] Pendu | A fourier disparity layer representation for light fields[END_REF], the LLFF method [START_REF] Mildenhall | Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[END_REF] and our previous pipeline FPFR [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF]. All tested methods take four corner views of the central 3 × 3 subset of light field views (red slashes in Fig. 6(b)) as inputs, except for the LLFF method which additionally takes the central view as a fifth input view (grey slashes in Fig. 6(b)), to synthesize a 9 × 9 light field. Fig. 6(c) shows the performances for each method. We can observe that our method yields better results than the reference methods. Wider is the baseline, more important are the gains with our approach. Let us note that all learning-based methods used in this test have been trained for the interpolation task, hence here we assess the inherent extrapolation capability of each method without any further finetuning.

4) Comparative evaluation with Neural Radiance Fields: In recent years, view synthesis methods have been proposed based on the concept of Neural Radiance Fields (NeRF). NeRFs represent a mapping between 5D spatial ((x, y, z) Light field view DeepVS [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] Soft3D [START_REF] Penner | Soft 3D reconstruction for view synthesis[END_REF] LLFF [START_REF] Mildenhall | Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[END_REF] EPI [START_REF] Wu | Learning sheared epi structure for light field reconstruction[END_REF] FPFR [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF] Ours View indices for interpolation (top) and extrapolation (bottom). 4 input views (red slash) are used for DeepVS [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF], EPI [START_REF] Wu | Learning sheared epi structure for light field reconstruction[END_REF], Soft3D [START_REF] Penner | Soft 3D reconstruction for view synthesis[END_REF], FPFR [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF], FDL [START_REF] Pendu | A fourier disparity layer representation for light fields[END_REF] and our method, whereas 5 input views (grey) are used for LLFF [START_REF] Mildenhall | Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[END_REF]. the network weights can be seen as a representation of the scene radiance. More views are used to train the network (a multi-layer perceptron), more precise and complete is the Radiance Field, and better are the rendering results. We therefore performed a comparative evaluation with three types of input view configurations: 1) four corner views (Fig. 7(a)). 2) eight border views (Fig. 7(b)). 3) nine views (Fig. 7(c)). The performance measures are averaged on all the other light field views that are synthesized. Table III shows the measured PSNR on 4 LFs from the benchmark considered in [START_REF] Honauer | A dataset and evaluation methodology for depth estimation on 4D light fields[END_REF]. The quality of the views rendered from the learned NeRF model increases with the number of input views. However, when taking only the four corner views, and the corresponding ground truth camera poses, the quality of the views rendered using NeRF is inferior to the views synthesized with our method, except for one light field. Set aside the per-scene training complexity, this shows the limit of the approach when the number of input views is limited. The advantage of our approach in that case is particularly interesting in a compression context, since it leads to higher compression efficiency for the light field, as we show in the next section. The perscene training complexity of NeRF has later been addressed in the state-of-the-art work of MVSNeRF [START_REF] Chen | MVSNeRF: Fast generalizable radiance field reconstruction from multiview stereo[END_REF] by proposing a method in which the trained network can be applied to novel scenes. However, as shown in Table III per-scene finetuning can further improve the performance of MVSNeRF (PSNR in the parentheses), our method likewise achieves better PSNR after the same finetuning, which proves the superiority of our method against MVSNeRF.

5) Compression performance analysis:

We consider a compression set-up in which the reference views, according to the configurations of Figure 7(a), are first encoded using the HEVC-inter coding standard (HM 16.10). The LFVS methods are then used for recovering the entire light field from the compressed/decompressed reference views. This LFVS-based compression pipeline is summarized in Fig. 8, and we use FPFR [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF], NeRF [START_REF] Mildenhall | Nerf: Representing scenes as neural radiance fields for view synthesis[END_REF], MVSNeRF [START_REF] Chen | MVSNeRF: Fast generalizable radiance field reconstruction from multiview stereo[END_REF] and our scheme as the LFVS method. Besides the LFVS-based pipeline, we also apply deep learning-based video sequence compression methods (HLVC [START_REF] Yang | Learning for video compression with hierarchical quality and recurrent enhancement[END_REF], RLVC [START_REF]Learning for video compression with recurrent auto-encoder and recurrent probability model[END_REF]) and the classical HEVC inter coding method to compress light fields. Fig. 9 shows the obtained PSNR-rate curves with the different LF compression strategies. Our method achieves better compression efficiency than other referenced methods, which means that our method Fig. 8. LF compression pipeline based on LFVS method. We use FPFR [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF], NeRF [START_REF] Mildenhall | Nerf: Representing scenes as neural radiance fields for view synthesis[END_REF], MVSNeRF [START_REF] Chen | MVSNeRF: Fast generalizable radiance field reconstruction from multiview stereo[END_REF] and our proposed method as LFVS method in the pipeline. Fig. 9. Rate-distortion curves of scenes 'cotton' and 'dino' using different LF compression methods. Both methods FPFR [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF], MVSNeRF [START_REF] Chen | MVSNeRF: Fast generalizable radiance field reconstruction from multiview stereo[END_REF] and ours are tested without per-scene finetuning, while NeRF [START_REF] Mildenhall | Nerf: Representing scenes as neural radiance fields for view synthesis[END_REF] is trained for the test scene. is more suitable for compression applications. Let us note that, although radiance field-based methods NeRF and MVSNeRF work well for view synthesis task itself, they either work poorly without per-scene finetuning (MVSNeRF) or require a long-time training (NeRF) for the compression task, which makes them less practical for the compression task.

B. Video Frame Interpolation

Besides angular view synthesis in light field, we also assess our pipeline for video frame temporal interpolation, using two datasets:

• The Vimeo90K datatset [START_REF] Xue | Video enhancement with task-oriented flow[END_REF] that contains 3,782 video triplets with resolution 256×448, covering a large variety of scenes and actions. We interpolate the intermediate frames using the two adjacent frames as inputs. • The Adobe240fps dataset [START_REF] Su | Deep video deblurring for hand-held cameras[END_REF] that is a real-world video set captured with a handheld camera, it is more challenging than Vimeo90K as it contains more motion blur. We extracted 777 frame triplets (about 10% of the total number of frames) with resolution 360 × 640 and synthesized the intermediate frames. We compare the quality of the interpolated frames with the one obtained with state-of-the-art VFI methods, i.e. with MEMC-Net [START_REF] Bao | Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement[END_REF], which merges kernel and flow-based synthesis in its pipeline; FeFlow [START_REF] Gui | Featureflow: Robust video interpolation via structure-to-texture generation[END_REF], which exploits a flow of features instead of pixels for motion estimation; and Softmax Splatting (SMSP) [START_REF]Softmax splatting for video frame interpolation[END_REF], a flow-based framework that employs pixels and features in the synthesis. For the MEMC-Net and FeFlow methods, We use the official authors source code and their pre-trained model. However, as the code and model are not available for SMSP, we re-implemented this method and trained it using the Vimeo90K training set following the instructions in the paper. We could not reproduce the results of the paper, therefore, we give in Table IV both results, the ones taken from the paper [START_REF]Softmax splatting for video frame interpolation[END_REF] (noted as SMSP*), and those obtained with our implementation.

Table IV gives the PSNR and SSIM performances for all tested methods, we can notice that our proposed architecture, although originally designed for angular view synthesis in light field, is well suited for temporal video frame interpolation. It can generate intermediate frames of a quality that is comparable to the one of the top-rank official SMSP method. The table shows that the proposed method outperforms state-ofthe-art methods by a large margin. Fig 10 shows interpolated frames obtained with different methods. Compared with other methods, our architecture is able to retrieve high quality frames with subtle details. Let us note that when the input frames contain some blur, like in the 1st row, the reference methods may suffer from deformation or may fail to synthesize intermediate views, while our method is more robust and can still generate plausible results. Let us note that the proposed VFI method can be used to compress a video. The approach can indeed be applied in a similar way as the inter-prediction method based on deep frame interpolation networks in [START_REF] Bégaint | Deep frame interpolation for video compression[END_REF]. For each frame that can be coded with bi-directional interprediction from past and future frames, an additional reference frame can be provided by interpolating the original reference frames.

C. Ablation study

In this section, we carry out an ablation study to analyze the performance gain brought by the new architecture when compared with [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF].

1) Residual convolutional block (RCB): We assess the performance gain obtained by using the RCB instead of simple convolutional layers. We consider the PixRNet module and fix the weights in the FNet module so that the quality of the synthesized views only depends on the synthesis block. We compare the three variants shown in Fig 11, where Fig 11(a) (a) MEMC-Net [START_REF] Bao | Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement[END_REF] (b) FeFlow [START_REF] Gui | Featureflow: Robust video interpolation via structure-to-texture generation[END_REF] (c) SMSP [START_REF]Softmax splatting for video frame interpolation[END_REF] Ours GT Fig. 10. Video frame interpolation results in comparison with (a) MEMC-Net [START_REF] Bao | Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement[END_REF], (b) FeFlow [START_REF] Gui | Featureflow: Robust video interpolation via structure-to-texture generation[END_REF], (c) SMSP [START_REF]Softmax splatting for video frame interpolation[END_REF]. The upper two rows use frames from [START_REF] Su | Deep video deblurring for hand-held cameras[END_REF], the lower two rows use frames from [START_REF] Xue | Video enhancement with task-oriented flow[END_REF]. We use official outputs of SMSP for the last two rows. shows the fundamental architecture adopted in [START_REF] Shi | Learning fused pixel and featurebased view reconstructions for light fields[END_REF], , with all curves have been obtained using four HCI light field scenes in [START_REF] Honauer | A dataset and evaluation methodology for depth estimation on 4D light fields[END_REF]. One can notice that adding kernels and layers like in Fig. 11(c) allows improving the network performance. Residual connections further ameliorate the synthesis quality.

2) Trainable encoder: Another improvement in our pipeline is the use of a trainable encoder, instead of a fixed VGG19 encoder. A lightweight trainable encoder that flexibly extracts feature maps in cooperation with the synthesis step effectively accelerate the network convergence. By fixing weights in the FNet module, we trace the learning curves of the FeatRNet module using two different encoders (see Fig. 12(b)). We can note that with a trainable encoder, the FeatRNet module quickly converges to a high PSNR value, which means that the extracted features are suitable for synthesis. While the features extracted with the VGG19 encoder originally designed for the classification task are less suitable for the reconstruction task, and yields slower network convergence.

3) Global architecture: Finally, Fig. 12(c) shows that the training of the proposed architecture, when compared with FPFR, converges faster. Tables I and II show that this novel architecture is quite efficient for both light field view synthesis and video frame temporal interpolation.

VII. CONCLUSION

In this paper, we have presented a learning-based architecture for both angular view synthesis for light field and video frame temporal interpolation. The proposed scheme employs a flow estimation network to predict flows between input images. The synthesis is then conducted in parallel in pixel-based and feature-based branches followed by a soft mask fusion to obtain the synthesized view. Compared with the FPFR solution originally designed for light field view synthesis, the novel architecture adopts residual convolutional blocks (RCBs) and a lighter weight trainable encoder that allows improving both training efficiency and synthesis performance for light field view synthesis. The architecture is further generalized to handle video frame temporal interpolation. The effectiveness of the proposed solution is demonstrated by comparing with state-of-the-art methods using both light field synthesis and video interpolation datasets. Experimental results show that our method tackles well both tasks with high quality results.

Fig. 1 .

 1 Fig. 1. Overview of the proposed deep architecture for both light field view synthesis and video frame temporal interpolation. In the case of the light field view synthesis problem, the input views are 4 sparse light field views (e.g. corner views) with i = {tl, tr, bl, br} being the index of the source view positions, and j the index of the target view position. In the case of video frame temporal interpolation, the input views are temporally adjacent views, with i = {0, 1} the index of source frame time instants, and j = t the target frame instant.

  d 1 and d 2 may contain estimation errors due to color inconsistencies or occlusions. Inspired by the work in [39], we further improve disparity accuracy by merging two disparity maps d 1 and d 2 into a single one d tl . More specifically, based on each disparity map, other three views I tr , I bl and I br are projected to the position of I tl and the corresponding warping errors e 1 (for d 1 ) and e 2 (for d 2 ) are computed as:

Fig. 2 .

 2 Fig.2. Visual comparison of views before & after finetuning (FT). The views in the 1st row are from the light field dataset in[START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF], while those in the 2nd row are from the video frame interpolation dataset in[START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF] d tr , d bl , d br estimated by Flow estimation module, we aim at synthesizing the view at j. We finetune the estimated flows, without using ground truth flows, by optimizing argmin θ

Fig. 3 .

 3 Fig. 3. Architecture of Residual Convolutional Block (RCB) with input tensor of dimension [B, H, W, C1] and output tensor of dimension [B, H, W, C2].

  FeatRNet module. That also helps simplifying the training schedule as detailed later in Sec. V-C.As the reconstructed views Îpix j

{γ 1 , γ 2 , γ 3 }

 123 are weight hyperparameters. Fig 4 shows the synthesized central views Îpix 5,5 , Îfeat 5,5 , Î5,5 and their Fourier transforms in the case of LFVS. And we can observe from Fig 4(a) and Fig 4(b) that pixel and feature-based reconstructions have complementary properties: the pixelbased reconstruction is more blurred and the energy in the Fourier domain is mainly concentrated in low frequencies, while the feature-based reconstruction is better in textured regions, and most of its energy resides in high frequencies. After fusing two views with a mask, we can see in Fig 4(c) that the final reconstructed view inherits the advantages of both pixel and feature-based reconstruction, and is close to the ground truth view in Fig 4(d) in both the color and Fourier domains.

Fig. 4 .

 4 Fig. 4. Visual comparison in both pixel and frequency domains between Îpix 5,5 , Îfeat 5,5 , Î5,5 and I 5,5 in the case of LFVS.

Fig. 5 .Fig. 6 .

 56 Fig. 5. Reconstruction error maps of view Î5,5 for different synthesis methods, with higher error value in red and lower error value in blue.

Fig. 7 .

 7 Fig. 7. Three types of input view configurations for NeRF [21], blue squares are input views for training and gray squares are views to be inferred.

Fig. 11 .

 11 Fig. 11. Three variants for PixRNet, with (a) fundamental architecture used in [25], (b) PixRNet based on RCB, (c) PixRNet based on CB (without residual connection).

  Fig 11(c) an enhanced architecture with more kernels and layers, which can also be considered as the non-residual connection version of Fig 11(b), and where Fig 11(b) is our current PixRNet architecture based on RCB. Their corresponding learning curves are compared in Fig 12(a)

Fig. 12 .

 12 Fig. 12. Learning curves of (left) different architectures (PixRNet in [25], PixRNet using RCB, PixRNet using CB); (middle) different encoders (FeatRNet using VGG19 encoder, and FeatRNet using a trainable encoder); (right) FPFR and our new architecture

.

TABLE I QUANTITATIVE

 I RESULTS (PSNR) FOR THE RECONSTRUCTED CENTRAL VIEW OF THE SYNTHETIC TEST LIGHT FIELDS. THE CORRESPONDING DATASETS ARE INDICATED BY SYMBOLS:[START_REF] Honauer | A dataset and evaluation methodology for depth estimation on 4D light fields[END_REF],[START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF] AND †[START_REF] Wanner | Datasets and benchmarks for densely sampled 4D light fields[END_REF]. THE BEST AND THE SECOND BEST PERFORMANCES ARE SHOWN IN BOLD AND UNDERLINED.

	LFs	Disparity range	DeepVS [14]	Soft3D [15]	LLFF [16]	EPI [17]	FPFR [25]	Ours	Ours*
	mona †	[-5,5] (10)	38.90	40.92	41.20	37.54	42.47	43.26	43.57
	butterfly †	[-6,8] (14)	40.68	42.75	41.35	39.61	42.69	43.52	43.79
	buddha †	[-10,6] (16)	41.08	41.86	40.66	40.05	42.78	43.50	43.72
	cotton	[-9,9] (18)	47.24	48.95	47.07	47.97	48.58	50.48	50.59
	boxes	[-7,13] (20)	33.64	32.14	34.97	31.65	33.86	33.41	34.29
	dino	[-10,10] (20)	38.41	41.69	41.26	38.44	42.66	43.69	44.09
	sideboard	[-10,12] (22)	30.91	30.23	32.33	27.30	31.85	32.37	32.75
	Toy bricks	[-1,22] (23)	28.90	36.58	37.98	31.46	38.84	40.44	40.89
	Elec devices	[-10,17] (27)	34.09	36.24	36.76	31.55	37.63	39.25	39.50
	stilllife †	[-16,16] (32)	26.29	34.73	32.73	32.02	36.39	37.20	37.74
	Lion	[-5,29] (34)	28.05	35.18	35.22	33.91	35.47	35.66	35.75
	Two vases	[-5,39] (44)	25.65	32.49	35.82	29.09	35.56	35.54	36.26
	Sculpture	[-26,34] (60)	22.31	29.15	29.68	26.22	30.09	30.02	30.12
	Bear	[-38,53] (91)	18.36	28.00	33.22	23.40	31.87	33.41	34.22
	Average	-	32.49	36.49	36.43	33.59	37.92	38.70	39.09

TABLE II QUANTITATIVE

 II RESULTS (PSNR) FOR THE RECONSTRUCTED VIEW (5,5) WITH REAL-WORLD DATA (8 × 8 VIEWS) [14]. THE BEST PERFORMANCES ARE SHOWN IN BOLD AND THE SECOND BEST ONES ARE UNDERLINED.

	LFs	DeepVS	Soft3D	LLFF	EPI	FPFR	Ours
	Cars	31.53	27.68	29.06	28.17	32.05	32.86
	Flower1	33.13	30.29	30.00	30.44	34.19	34.63
	Flower2	31.95	30.52	28.90	29.26	33.80	34.12
	Rock	34.32	32.67	32.60	32.46	36.48	37.13
	Leaves	27.97	27.34	27.74	26.48	32.27	32.65
	Seahorse	32.03	30.41	28.50	26.62	34.68	34.95
	Average	31.82	29.82	29.47	28.90	33.91	34.39

TABLE III QUANTITATIVE

 III EVALUATIONS (PSNR) FOR THE SYNTHESIZED VIEWS. METHOD NERF [21] IS TRAINED WITH DIFFERENT NUMBERS OF VIEWS (4,8,9, SHOWN IN FIG.7). WE SHOW IN THE PARENTHESES THE PSNR AFTER SCENE-WISE FINETUNING FOR OUR METHOD AND MVSNERF[START_REF] Chen | MVSNeRF: Fast generalizable radiance field reconstruction from multiview stereo[END_REF]

	LFs	Ours-4	MVSNeRF-4	NeRF-4	NeRF-8	NeRF-9
	boxes	33.89 (38.84)	31.10 (35.32)	36.88	39.22	39.30
	cotton	50.39 (50.84)	38.14 (47.94)	46.67	47.51	47.74
	dino	44.04 (45.56)	26.94 (42.95)	40.29	43.63	43.83
	sideboard	32.49 (35.38)	22.52 (34.25)	32.15	36.19	36.88
	Average	40.20 (42.66)	29.68 (40.12)	38.99	41.64	41.93

TABLE IV QUANTITATIVE

 IV EVALUATIONS (PSNR&SSIM) FOR THE SYNTHESIZED INTERMEDIATE FRAMES. THE BEST PERFORMANCES ARE SHOWN IN BOLDAND THE SECOND BEST ONES ARE UNDERLINED.

	Methods	Vimeo90K [49] PSNR SSIM	Adobe240fps [54] PSNR SSIM
	SuperSloMo [4]	30.92	0.932	28.44	0.897
	SepConv [10]	33.80	0.956	31.16	0.923
	MEMC-Net [11]	34.43	0.963	31.54	0.927
	FeFlow [8]	35.09	0.963	31.50	0.925
	SMSP [7]	35.51	0.967	31.60	0.927
	SMSP* [7]	36.10	0.970	-	-
	Ours	35.96	0.970	31.72	0.928

This project has been supported by the EU H2020 Research and Innovation Programme under grant agreement No 694122 (ERC advanced grant CLIM). The authors are with INRIA,