Yan Li

Qiong Wang

Lu Zhang

Gauthier Lafruit

A Lightweight Depth Estimation Network for Wide-baseline Light Fields

Keywords: Light field, depth estimation, convolutional neural network, lightweight, wide-baseline, narrow-baseline, synthetic dataset

Existing traditional and ConvNet-based methods for light field depth estimation mainly work on the narrow-baseline scenario. This paper explores the feasibility and capability of ConvNets to estimate depth in another promising scenario: wide-baseline light fields. Due to the deficiency of training samples, a large-scale and diverse synthetic wide-baseline dataset with labelled data is introduced for depth prediction tasks. Considering the practical goal for real-world applications, we design an end-to-end trained lightweight convolutional network to infer depths from light fields, called LLF-Net. The proposed LLF-Net is built by incorporating a cost volume which allows variable angular light field inputs and an attention module that enables to recover details at occlusion areas. Evaluations are made on the synthetic and real-world wide-baseline light fields, and experimental results show that the proposed network achieves the best performance when compared to recent stateof-the-art methods. We also evaluate our LLF-Net on narrowbaseline datasets, and it consequently improves the performance of previous methods.

I. INTRODUCTION

I N practical research areas such as 3D reconstruction, view synthesis and autonomous driving, accurate depth estimation is crucially needed. The light field, referred to as 4D computational photography technology, has been active in reconstructing depth in the real-world scenes. Different from 2D photography, light fields record the radiance of the lights from diverse directions, enhancing possibilities of perceiving depth. In this work, we focus on depth estimation for light fields.

Existing light field datasets can be divided into narrowbaseline and wide-baseline. Narrow-baseline light fields are typically captured by a plenoptic camera where a grid of micro-lenses are placed between the main lens and the image sensor, e.g., Lytro ILLUM camera [START_REF] Ng | Lytro redefines photography with light field cameras[END_REF]. A light field image from the camera is usually separated into the so-called subaperture images, and the baseline between sub-aperture images is very narrow. To date, traditional [START_REF] Kim | Scene reconstruction from high spatio-angular resolution light fields[END_REF][START_REF] Wang | Occlusion-aware depth estimation using light-field cameras[END_REF][START_REF] Jeon | Accurate depth map estimation from a lenslet light field camera[END_REF][START_REF] Si | Dense depth-map estimation and geometry inference from light fields via global optimization[END_REF][START_REF] Zhu | Occlusion-model guided antiocclusion depth estimation in light field[END_REF][START_REF] Strecke | Accurate depth and normal maps from occlusion-aware focal stack symmetry[END_REF][START_REF] Park | Robust light field depth estimation using occlusion-noise aware data costs[END_REF][START_REF] Navarro | Robust and dense depth estimation for light field images[END_REF][START_REF] Li | Scalable light field disparity estimation with occlusion detection[END_REF][START_REF] Chen | Accurate light field depth estimation with superpixel regularization over partially occluded regions[END_REF][START_REF] Schilling | Trust your model: Light field depth estimation with inline occlusion handling[END_REF][START_REF] Huang | Empirical bayesian light-field stereo matching by robust pseudo random field modeling[END_REF] and ConvNet-based [START_REF] Heber | Convolutional networks for shape from light field[END_REF][START_REF] Luo | Epipatch based convolutional neural network for depth estimation on 4d light field[END_REF][START_REF] Heber | Neural epi-volume networks for shape from light field[END_REF][START_REF] Feng | Benchmark data set and method for depth estimation from light field images[END_REF][START_REF] Zhou | Scale and orientation aware epi-patch learning for light field depth estimation[END_REF][START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF][START_REF] Zhou | Learning depth cues from focal stack for light field depth estimation[END_REF] methods have been well studied for high performance in narrow-baseline light fields, and achieved a low percentage of errors, e.g., EPINET [START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF]. For wide-baseline light fields, they are usually captured by a camera array or gantry (i.e., Manuscript received 2020. The paper is supported by the China Scholarship Council funding and Van Buuren-Jaumotte-Demoulin doctoral funding. Yan Li and Gauthier Lafruit are with the LISA Department, Université Libre De Bruxelles, Brussels, Belgium. Lu Zhang is with the IETR lab, INSA Rennes, Rennes, France. Qiong Wang is with Zhejiang University of Technology, Hangzhou, China (e-mail: wangqiong819@gmail.com). Fig. 1. An example of the proposed wide-baseline light fields. Our network LLF-Net has the fewer parameters (1.8 million) but is capable of performing more accurate depth estimation when compared to LBDE-E [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF] with 198.8 million parameters and EPI-Shift [START_REF] Leistner | Learning to think outside the box: Wide-baseline light field depth estimation with epi-shift[END_REF] with 31.6 million parameters. a conventional camera is placed onto a gantry, and then uniformly moved by a motor in a plane). The baseline between the recorded wide-baseline light-field images is large and the spatial resolution of images is usually high. Compared with the narrow-baseline scenario, the wide-baseline is more capable of improving depth accuracy due to its large baseline [START_REF] Yang | Hierarchical deep stereo matching on high-resolution images[END_REF]. Up to now, considerable efforts have been also made by traditional methods [START_REF] Wang | Occlusion-aware depth estimation using light-field cameras[END_REF][START_REF] Zhu | Occlusion-model guided antiocclusion depth estimation in light field[END_REF][START_REF] Li | Scalable light field disparity estimation with occlusion detection[END_REF][START_REF] Huang | Empirical bayesian light-field stereo matching by robust pseudo random field modeling[END_REF][START_REF] Dabała | Efficient multiimage correspondences for on-line light field video processing[END_REF][START_REF] Sabater | Dataset and pipeline for multi-view light-field video[END_REF][START_REF] Chuchvara | Fast and accurate depth estimation from sparse light fields[END_REF] to solve the problem of depth estimation in the wide-baseline scenario. However, ConvNet-based approaches are rarely studied in this scenario. Our objective is to explore and apply ConvNets into depth estimation for wide-baseline light fields.

Training ConvNets requires a large amount of labelled data. Unfortunately, there were no large-scale public wide-baseline light field datasets to the extent that limits the development of ConvNet-based methods. For a new dataset creation, a straightforward way is to collect real data and label them through physical depth sensing devices (e.g., structure light sensor or LiDAR). However, it is difficult, tedious or even unsuitable: structured light sensor is cheap but usually produces inaccurate depth which may cause performance degradation in ConvNets models, while LiDAR offers accurate depth but is unaffordable. Similar to the narrow-baseline scenario, we put effort into creating a synthetic wide-baseline dataset with accurate (ground truth) depths, aiming at training and evaluating ConvNet models, inferring depth for real-world datasets, and serving to the research community for future promising researches. We use a 3D computer graphics software to create a large-scale, wide-baseline synthetic dataset with diversities (around 0.4K light fields), called WLF. It consists of two subsets: photo-realistic subset (Hand-designed), and nearly photo-realistic subset (Flying-objects). Fig. 1 shows an example scene that contains light fields and ground truth.

With respect to the ConvNet model for wide-baseline scenario, we consider to develop a lightweight model toward more practical goals, e.g., applications in mobile devices. To date, Shi et al. [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF] present a divide-and-train deep model and Leistner et al. [START_REF] Leistner | Learning to think outside the box: Wide-baseline light field depth estimation with epi-shift[END_REF] present an end-to-end trained deep model for wide-baseline light field depth estimation. However, the two models have more than 198 million parameters and 31 million parameters respectively, which are so heavyweight that they are less suitable for practical applications. Since EPINET [START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF] achieves the state-of-the-art performance in the narrow-baseline scenario with much fewer parameters (5.1 million), we made an attempt to test and re-train (denoted as EPINET T) the proposed WLF dataset, but the performances are too poor (cf. Fig. 10). Thus, we put forward a novel end-to-end trained lightweight and effective network LLF-Net by taking knowledge from stereo-based ConvNet models. As shown in Fig. 1, our model is lightweight and the quality of our reconstructed depth map is superior to the state-of-the-art methods in the wide-baseline scenario.

In our network, features are extracted for each view of the horizontal and vertical stream of light field views, and then the cost volume is obtained by a sequence of operations, i.e., shift-interpolation, cost calculation and fusion operations. With respect to the fusion, a divide-concatenate-sum operation is adopted, allowing flexible light field inputs while maintaining depth accuracy. An attention mechanism is proposed used in the cost aggregation module to adaptively assign weights to the feature maps of image views in each stream (view attention) and to the two streams (stream attention), which helps depth estimation at occlusion regions and preserves depth discontinuities. We make evaluations of the proposed network on the WLF test set and real-world datasets, and experimental results show that our network outperforms state-of-the-art methods in both quantitative and qualitative evaluations.

The main contributions of our work are summarized as follows:

1) We introduce a large-scale and diverse synthetic dataset with ground truth labels in the wide-baseline scenario for the first time (to the best of our knowledge), offering possibilities to further exploit and validate the learning-based methods.

2) We design a novel end-to-end trainable lightweight network for light field depth estimation.

3) Our network is built with the novel cost volume module that allows flexible light field inputs, and the attention module for better handling occlusions and preserving depth discontinuities.

4) The performance of the proposed network outperforms state-of-the-art light field depth estimation methods in the wide-baseline scenario, and even the narrow-baseline scenario.

We have released the training sets, comprising of part I at https://zenodo.org/record/3931237.X1 B RT7SaF and part II at https://zenodo.org/record/3934712.X1 CoxT7SaF. The rest of the proposed WLF dataset, the trained models and the source code of the proposed model are released at https://sites.google.com/site/yanliresearch/llf-net after publication, which can be used for reproducing our results, comparisons and further improvements.

II. RELATED WORK

A. Deep learning-based methods

Recently, deep learning methods have gained much attention in estimating depth from light fields. Heber et al. [START_REF] Heber | Convolutional networks for shape from light field[END_REF], Luo et al. [START_REF] Luo | Epipatch based convolutional neural network for depth estimation on 4d light field[END_REF], Heber et al. [START_REF] Heber | Neural epi-volume networks for shape from light field[END_REF] and Feng et al. [START_REF] Feng | Benchmark data set and method for depth estimation from light field images[END_REF] feed the input of Epipolar Plane Image (EPI) to the ConvNet where the network learns the proportional relation between the slope of the epipolar line and depth. However, this relation is hard to learn in wide-baseline light fields due to the absence of the epipolar line on the EPI. Shin et al. [START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF] present the EPINET where the geometric characteristic of epipolar images is used for depth estimation. EPINET achieves top-performing performance for the narrow-baseline scenario, but poor performance for the wide-baseline scenario even with re-training on wide-baseline datasets. Shi et al. [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF][START_REF] Jiang | Depth estimation with occlusion handling from a sparse set of light field views[END_REF] propose to fine-tune FlowNet 2.0 [START_REF] Ilg | Flownet 2.0: Evolution of optical flow estimation with deep networks[END_REF] for the initial depth prediction, followed by a refinement network to improve depth quality. Leistner et al. [START_REF] Leistner | Learning to think outside the box: Wide-baseline light field depth estimation with epi-shift[END_REF] present an EPI-Shift network, where light field stacks are shifted from wide-baseline to narrow-baseline, and then used to predict depths by trained models from narrowbaseline datasets. Though these two methods can work onto wide-baseline light fields, both models are heavyweight and not suitable for practical applications. In contrast, our proposed network is a lightweight network, and able to perform well on the wide-baseline light field inputs.

B. Light Field Datasets

We review the publicly available synthetic light field dataset, which are frequently used for training or comparing the performance of competing ConvNet-based methods in depth estimation.

Most of these available datasets belong to the narrowbaseline, which are composed of a grid of 9x9 light field image views and with the small disparity range [-4, 4] (HCI [START_REF] Wanner | Datasets and benchmarks for densely sampled 4d light fields[END_REF], CVIA-HCI [START_REF] Honauer | A dataset and evaluation methodology for depth estimation on 4d light fields[END_REF], and DLFD [START_REF] Jiang | A learning based depth estimation framework for 4d densely and sparsely sampled light fields[END_REF]). HCI [START_REF] Wanner | Datasets and benchmarks for densely sampled 4d light fields[END_REF] and CVIA-HCI [START_REF] Honauer | A dataset and evaluation methodology for depth estimation on 4d light fields[END_REF] are the two most frequently-used datasets for the narrow-baseline scenario in literature. The HCI includes 7 frames/scenes with available full ground truth depths, which are usually used for evaluation. The CVIA-HCI includes 16 frames with available ground truth depths that are provided for training, and 8 frames with available ground truth provided for evaluating the methods. Nevertheless, models trained on the CVIA-HCI and/or even HCI and DLFD datasets are not able to infer depth well on wide-baseline datasets due to the mismatch between the source and target disparity range. Hence we propose to build a large-scale light field dataset to train the deep models and stimulate more future developments for the wide-baseline scenario.

III. METHODOLOGY

The goal of the proposed method is to estimate the dense depth map for the center view from light fields. The light field inputs are parameterized by the two-plane parametrization L(x, y, u, v), where (u, v) are the camera plane coordinates and (x, y) are the image plane coordinates. The relationship between the other views and center view are represented as:

L(x, y, u, v) = L(x + (u * -u)d(x, y), y + (v * -v)d(x, y), u * , v * ), (u ∈ [1, N ], v ∈ [1, N ]) (1)
where N denotes the number of (angular) views along the horizontal and vertical directions in light fields, (u * , v * ) represent the coordinates of the central view, and d(x, y) is the disparity of the pixel (x, y) in the central view (i.e. the offset between the central view and its adjacent right view). Estimating the disparities (or depths) of the central view corresponds to searching the offset of corresponding points in other views. An overview of the proposed network architecture is illustrated in Fig. 2 and detailed in Table I. The cross-hair of light field images are used and fed into the proposed network. To make image correspondence features distinguishable, deep feature descriptors are extracted for each view from crosshair views in the Feature Extraction (Section III-A). Next, the discriminative cost volume [START_REF] Wang | Occlusion-aware depth estimation using light-field cameras[END_REF][START_REF] Jeon | Accurate depth map estimation from a lenslet light field camera[END_REF] is constructed by operating all extracted features in the Cost Volume Generation (Section III-B). Afterwards, an attention mechanism is introduced to remove disparity errors caused by the occlusion, and a 3D encoder-decoder network is applied to regularize the disparity space in the Cost Aggregation (Section III-C). Finally, the disparity map is produced in Disparity Regression (Section III-D), and the smooth L1 loss (Section III-E) is used for training our network.

A. Feature Extraction

Our network takes as input the horizontal and vertical streams of image views with the dimension H × W × N from light fields, where H and W represent the height and width of image (spatial resolution). We apply a 2D plane convolution network to extract distinct features. It is firstly constructed by one convolution layer and one Conv-Bn-Relu

∈ S 0 • ∪ S 90 • ) Conv K2S2 H/2 × W/2 × C X C1 1 ConvBnR K2S1 H/2 × W/2 × C C1 1 C1 2 Conv K2S2 H/4 × W/4 × C C1 2 C2 1 ConvBnR K2S1 H/4 × W/4 × C C2 1 C2 2 Cost volume (C2 2S 0 • = {C2 2} N 1 , C2 2S 90 • = {C2 2} N 1 ) Shift Cost L/4×H/4×W/4× NC C2 2S 0 • SIC1 Shift Cost L/4×H/4×W/4× NC C2 2S 90 • SIC12 Div Concat L/4×H/4×W/4× 6C SIC1, SIC2 {DC} 3(N -1)/2 1 Sum L/4×H/4×W/4× 6C {DC} 3(N -1)/2 1 [CV 0 • , CV 90 • ] View and Stream attention View Attention 0 • L/4 × H/4 × W/4 × 3C CV 0 • CV v1 View Attention 90 • L/4 × H/4 × W/4 × 3C CV 90 • CV v2 Stream Attention L/4 × H/4 × W/4 × 6C CV v1, CV v2 CV s Cost regularization 3DConvBnR K3S1 L/4 × H/4 × W/4 × 2C CV s 3Cbr1 3DConvBnR K3S1 L/4 × H/4 × W/4 × 2C 3Cbr1 3Cbr2 3DConvBnR K3S2 L/8 × H/8 × W/8 × 4C 3Cbr2 3Cbr3 3DeConvBnR K3S2 L/4 × H/4 × W/4 × 2C 3Cbr3 3DCbr1 3DeConvBnR K3S1 L/4 × H/4 × W/4 × 1 3DCbr1 3DCbr2 Upsampling L × H × W × 1 3DCbr2 Up1 SoftArg H × W × 1 Up1 D
block (a convolution layer followed by a batch normalization layer, and a ReLU layer), in which the stride of the former convolution layer is set to 2 for down-sampling inputs and the latter is set to 1. Then the same structure is repeated to produce sub-scale features. Finally, the output feature maps are downsized to a quarter spatial resolution. The kernel of the convolution filters is 2x2. We adopt the shared 2D network on both streams of views since we found sharing parameters is better than the non-sharing case in terms of disparity accuracy and efficiency.

B. Cost Volume Generation

Given two streams of feature maps, a sequence of operations, i.e., shift-interpolation, cost calculation and fusion are used to generate the cost volume. Note that building the cost volume does not introduce any parameters to train.

1) Shift-Interpolation: The feature maps of the central view F r are regarded as the reference, and the others along the stream are the target feature maps F t . The feature maps of target views F t are shifted toward the reference view by each hypothesis disparity d within the disparity range (see Eq. 2). Then the bilinear interpolation is employed to calculate appropriate values for each pixel at sub-pixel position.

d = d min + n(d max -d min )/L, (n ∈ {0, 1, ..., L -1}) (2)
where d min , d max and L represent the minimum and maximum disparity in the range and the number of labels (or discrete disparities), respectively. The dimension of (warped) feature maps for each target view and the feature maps for the reference view is herein (L × H l × W l × C l ), where l denotes the scale level and C indicates the channels.

2) Cost Calculation: After obtaining the warped feature maps from streams of target views at the scale level l, we calculate the matching cost by using the concatenation [START_REF] Kendall | End-to-end learning of geometry and context for deep stereo regression[END_REF] between the reference and target feature maps to form a 4D cost volume, as shown in Eq. ( 2) and Eq. (3).

C l ( d, y, x, c) = C{F l (y + d(v * -v), x + d(u * -u), c i ) , (i = 1, ..., N )} (3)
Herein, c i denotes all the feature channels of view i.

3) Fusion: At this step, we make a fusion of calculated costs across views and streams such that neighboring views or streams enhance capabilities of solving ambiguity problems in correspondence matching. Actually, there exists different strategies to perform cost fusions. From the perspective of input sizes, strategies vary from fixed, to non-fixed or nearfixed inputs. Hereafter we discuss fusion variants in details, as are demonstrated in Fig. 3.

Concat is employed to concatenate all reference-target pairs of costs or horizontal and vertical groups of costs along the channel dimension, where the stacked size of the former is

L l × H l × W l × 4N C, and the latter is L l × H l × W l × 2N C.
Since the number of stacked feature channels is equal to that of convolution input filters, the networks then require the fixed inputs (i.e. the number of input views during the test should be the same with that during the training).

Sum computes the sum of all reference-target costs in which each cost is calculated by the absolute difference between the reference and target view. The sum fusion produces the fixedlength output L l × H l × W l × C regardless of the input size.

Divide, Concat, Sum (DCS) is designed to fuse costs across multiple-baseline cost volumes. The DCS fusion is presented as the combination of the Concat fusion and the Sum fusion. Given the cross-hair light fields, all the target views are positioned along the four directions (i.e. left, right, top, and bottom direction) of the reference view, and there exists a large set of target views along each direction. Actually, the target views in each direction are located in different camera baselines, ranging from 1 to N-1; thus there might exist redundancy among these baselines. An intuitive idea for eliminating this redundancy is to make fusions across the baselines. Specifically, we propose to divide the cross-hair light field inputs into (N-1)/2 groups with the same baseline of views; and each group contains five views, including one reference view and its four directional neighboring target views. For each group, all feature maps are firstly concatenated across the channel dimension (Concat fusion). Then we take the sum over all groups (Sum fusion). Note that the vertical views are not rotated by 90 degrees. Compared with the pure Concat fusion, this fusion requires any number of groups of the above-mentioned five input views, which is not limited to the fixed number of input views. Compared with the Sum fusion, this fusion maintains absolute directional information, which might contribute to high depth accuracy. The fused output, comprised of the horizontal part CV 0 • and the vertical part CV 90 • (see Fig. 3), has the dimension L l × H l × W l × 6C. DCS is flexible in the number of input angular views N . Note that when N is set to 3, DCS will be the same with the Concat fusion.

C. Cost Aggregation

Cost aggregation is leveraged to refine the fused cost volume since we did not take into account the potential occlusion issues before. With respect to the occlusion, we know that the same 3D real-world point that is visible in the reference view might be occluded by foreground objects in the target view, which leads to difficulties in finding correspondences. This issue is alleviated in our input cross-hair light fields since points might be visible in some angular views. Moreover, we are aware that points are occluded in the horizontal stream of views but might be less or not occluded in the vertical stream of views, and vice versa (cf. Fig. 8). Likewise, for any stream that consists of multiple views, the point is not seen in some target views, but might be visible in other views. Thus, if all views are equally used in searching the correspondence, the occlusion issue appearing in some views or stream can cause trouble to find the correct disparity due to the mismatches. This implicitly indicates that the cost features calculated from the Cost Volume Generation are not equally important. Note that 2D attention modules (e.g. used to change the weights of the features on each stage of the ResNet to enhance the consistency [START_REF] Yu | Learning a discriminative feature network for semantic segmentation[END_REF]) are increasingly used in deep neural networks to focus on what we want. To make our model focus on more informative features (e.g. that are more helpful for alleviating the occlusion issue and preserving depth discontinuity), we extend the 2D attention network to two types of 3D attention networks (view attention and stream attention). The view attention module is designed to exploit the interdependencies among feature channels with the same angular direction (0 • or 90 • ), while the stream attention module is designed to exploit the interdependencies among feature channels with different angular directions (cross-direction). Thus, the input of "View Attention 0 • " is the horizontal part of the output of "Sum" (CV 0 • ) and the input of "View Attention 90 • " is the vertical part of the output of "Sum" (CV 90 • ), cf. Table I or Fig. 3; and the input of the "Stream attention" is the concatenation of the output CV v1 from "View Attention 0 • " and the output CV v2 from "View Attention 90 • " along the channel dimension. For the attention block, cf. Fig. 4, the global average pooling is applied to capture the channel-wise statistics of the cost volume, followed by a simple gating mechanism capturing the channel-wise dependencies of the cost volume. Within this gating mechanism, the softmax operator in the view attention block is responsible for the output of the attention map, containing multiple different weights for the features in each stream; and the sigmoid operator in the stream attention block assigns the two weights to the features from the two streams respectively. The interdependent features of the cost volume are thus assigned to a large weight, and vice versa.

Followed by the attention network, a 3D encoder-decoder network is used to regularize the output of the attention network across the disparity dimension. This naturally involves large context information, which enforces the smoothness at low texture regions. This network is built by three 3D convolutions and two transposed convolutions. Last, the bilinear interpolation is used to resize back to the same spatial resolution of inputs, and the output has the dimension L×H ×W ×1.

D. Disparity Regression

The differential soft argmin operator proposed by [START_REF] Kendall | End-to-end learning of geometry and context for deep stereo regression[END_REF] is employed to obtain the final disparity map. The soft argmin operator regresses continuous disparities D by calculating the expectation of weighted disparities, as given in Eq. 4,

D = dmax d=dmin d * P ( d) (4)
where P ( d) is the weight probability of the pixel at disparity d.

E. Training Loss

We use the smooth L1 loss for the training process, which is computed between the predicted disparity d and the ground truth g in patch p as in Eq. 5 and Eq. 6,

L = i∈p Smooth L1 ( di -g i ) (5) Smooth L1 (x) = 0.5x 2 |x| ≤ 1 |x| -0.5 otherwise (6)
IV. WLF DATASET

We have designed a large-scale, wide-baseline synthetic light field camera array dataset WLF. The quantity of the light fields is 381, which is around 14 times larger than that of the popularly-used dataset CVIA-HCI. Each light field provides 9x9 angular (RGB) images and ground truth disparities as similar to the CVIA-HCI dataset. The light fields cover high resolution (1920x1080) and low resolution (512x512) images.

A. Dataset Construction

To enrich the dataset diversity, the WLF dataset is constructed in two scenarios: Hand-designed and Flying-objects. The scenes in Hand-designed and Flying-objects scenarios are rendered by the Cycle engine in the open source software Blender1 . Fig. 5 shows the rendered samples from these two scenarios, and the statistics of the WLF dataset are given in Table II.

Hand-designed Scenario: We carefully collect free 3D models from different websites2 with free licenses and elaborately assemble them to create physically plausible and meaningful scenes. Each scene contains more than two challenges in depth estimation: fine structure, repetitive pattern, occlusion, shading, and/or glossy appearance. Hand-designed scenario counts the aesthetic impression, but the manual design of 3D scenes is tedious and expensive, which causes difficulties to generate a large size dataset. This subset includes 36 scenes, and is split into 24 training scenes and 12 test scenes.

Flying-objects Scenario The richness of the dataset content is significant, therefore we attempt to render new scenes with flying objects in a faster way, which is inspired by recent advances of synthetic scenes with flying objects [START_REF] Dosovitskiy | Flownet: Learning optical flow with convolutional networks[END_REF][START_REF] Mayer | A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[END_REF][START_REF] Tremblay | Training deep networks with synthetic data: Bridging the reality gap by domain randomization[END_REF] in deep learning methods. Specifically, we carefully collect a large number of 3D models from the websites 2 and [START_REF] Chang | Shapenet: An information-rich 3d model repository[END_REF], and collect the texture images and environmental maps from Google Image. We then make a 3D cube in 3D space of Blender software, and the surfaces of cube are randomly textured. Next, a number of objects, which vary from 2 to 20, are randomly and automatically put in the cube, including 1-15 static objects and 1-5 random moving objects. The objects are randomly scaled, rotated and translated. Moreover, the light intensity is random, and the light field cameras are randomly and slightly translated. This subset includes 345 scenes, and is provided for training models.

Hand-designed

Flying-objects

B. Evaluation metrics

To measure the accuracy of reconstructed disparities from wide-baseline light fields, we adopt the widely-used metrics in depth estimation [START_REF] Honauer | A dataset and evaluation methodology for depth estimation on 4d light fields[END_REF][START_REF] Scharstein | High-resolution stereo datasets with subpixel-accurate ground truth[END_REF][START_REF] Menze | Object scene flow for autonomous vehicles[END_REF], i.e., mean square error (MSE) and bad pixels. The bad pixels are computed as the percentage of errors between the predicted disparity and ground truth disparity that are larger than a threshold. Considering thresholds in [START_REF] Honauer | A dataset and evaluation methodology for depth estimation on 4d light fields[END_REF][START_REF] Scharstein | High-resolution stereo datasets with subpixel-accurate ground truth[END_REF][START_REF] Menze | Object scene flow for autonomous vehicles[END_REF], the thresholds of the bad pixels are set to 0.15, 0.3, 0.6 and 1 in the paper. Note that a lower MSE or bad pixel percentage means a better performance.

V. EXPERIMENTS A. Implementation details

We use Tensorflow [START_REF] Abadi | Tensorflow: A system for large-scale machine learning[END_REF] to implement the proposed network.

The training and inference are both run on a Windows PC equipped with a Nvidia GTX 1080Ti GPU with 11GB memory and Intel i7 3.6Ghz CPU with 32GB memory. We use randomly cropped patches of size 128x128 for wide-baseline training set WLF and a smaller size 64x64 for narrow-baseline training set CVIA-HCI (due to its smaller quantity). Color scaling, 90, 180 and 270 degree rotation, etc are used for increasing the number of the data samples to the order of millions. We use the rmsprop optimizer [START_REF] Tieleman | Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF], and start at the learning rate 1e-4, and then divide it by two after 80k iterations for WLF and after 150k iterations for CVIA-HCI. For each iteration the mini-batch size is 8 for WLF and 16 for CVIA-HCI respectively. The d min and d max in Eq. ( 2) are set to 0 and 50 for WLF, -4 and 4 for CVIA-HCI respectively. The number of labels L is set to 128. III shows the evaluation results, and compares their adaptive ability of testing various angular resolutions without retraining the new angular resolution inputs. The proposed DCS fusion gets the best performance by bad-0.6 metric with considerable parameters. Besides, it is not limited by fixed angular resolution inputs.

Fig. 6 compares the performance results between two fusion ways (Sum and DCS) from testing variable angular resolutions. The DCS fusion always produces more accurate depths than the Sum fusion. When limiting the angular resolution, the DCS achieves much better performance, which means that it is more adaptive to limited input views. Fig. 7 illustrates visual comparison results from these two fusions. The DCS fusion undergoes a degradation in performance, but this is much less than that from the Sum fusion where artifacts occur in the disparity map.

2) Attention networks in Cost aggregation: To test the necessity of the proposed attention networks, Table IV compares the quantitative evaluation results with and without using them. We find that using the attention networks considerably improves the quality of estimated disparity maps.

Fig. 8 shows a visual comparison of disparity estimation without and with using attention networks. For a pixel in the selected patch P (see Fig. 8 (b)), it is occluded in all horizontal views, but it is visible in all vertical views. With the attention networks for selecting more meaningful views, the disparities of pixels around occlusion regions are better estimated and the sharp boundaries at depth discontinuities are better preserved, as shown in Fig. 8 (c-e). VI. We see that the weight sharing case has two advantages. Firstly, the weight sharing helps to reduce the number of parameters of the model (about 0.2 million less parameters compared with the non-sharing case). Secondly, the weight sharing helps to improve the depth estimation performance (the bad pixels are approximately reduced by half). The weight sharing scheme, employed at the low-level layers, learns a general feature representation model for all input views, whereas the non-sharing scheme learns distinct feature representation models for different input views. The latter may lead to a situation where the corresponding pixels are performed by different mathematical operations so that the extracted features of all input views may not be well-matched, which is not beneficial to the subsequent correspondence searching.

5) Training patch size:

The size of patches used in the patch-wise training may influence the performance of Con-vNets [START_REF] Hamwood | Effect of patch size and network architecture on a convolutional neural network approach for automatic segmentation of oct retinal layers[END_REF]. We thus investigated this influence on the performance of the proposed LLF-Net. Specifically, four different sizes, i.e. 96x96, 128x128, 160x160 and 192x192, were selected for the experiments. From Fig. 9, we empirically observe that the model with patch size 128x128 has the lowest bad-0.3 and bad-0.6 errors. The patch size smaller than this size will degrade the depth accuracy, whereas the patch size larger than this size will make little difference, but requires a larger GPU memory footprint.

C. Performance on wide-baseline datasets

To verify the effectiveness of our network LLF-Net on widebaseline light field datasets, we conducted experiments on the synthetic WLF dataset and real-world Google [START_REF] Penner | Soft 3d reconstruction for view synthesis[END_REF] and ULB [START_REF] Bonatto | Ulb high density 2d/3d camera array data set, version 2[END_REF] datasets. The proposed LLF-Net is compared with recent state-of-the-art depth estimation methods, comprising of traditional light field depth estimation methods LF OCC [START_REF] Wang | Occlusion-aware depth estimation using light-field cameras[END_REF] and RPRF [START_REF] Huang | Empirical bayesian light-field stereo matching by robust pseudo random field modeling[END_REF], ConvNet-based methods HSM [START_REF] Yang | Hierarchical deep stereo matching on high-resolution images[END_REF], EPINET [START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF] and LBDE-E [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF]. With respect to ConvNet-based methods, HSM [START_REF] Yang | Hierarchical deep stereo matching on high-resolution images[END_REF] is designed for stereo-based depth estimation, EPINET [START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF] achieves top performance among published methods in narrow-baseline light field datasets and LBDE-E [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF] achieves considerable performance on both narrowand wide-baseline light field datasets. Note that EPINET [START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF] is originally trained on narrow-baseline datasets and fails to infer depth in wide-baseline datasets, therefore we re-trained it using their public source code, and denote it as EPINET T.

1) Synthetic dataset: Table VII shows the quantitative comparisons on the four exemplar scenes from Hand-designed test set of the WLF dataset. Comparing to state-of-the-art methods, our proposed LLF-Net achieves the lowest bad pixel errors in all four scenes. In Fig. 10, visual comparisons of these scenes are given, in which the first column for each scene displays the ground truth or estimated disparity map and the second column shows the central view and bad pixel error map. It is clear that our estimated disparity maps are all closer to the ground truth and have the fewest number of bad pixels. In contrast, the estimated disparity maps from LF OCC are noisy, those from RPRF look over-smoothed and have quantification errors, both EPINET and EPINET T fail to predict disparities, HSM [START_REF] Yang | Hierarchical deep stereo matching on high-resolution images[END_REF] is disturbed by the ambiguous background, EPI-Shift [START_REF] Leistner | Learning to think outside the box: Wide-baseline light field depth estimation with epi-shift[END_REF] and LBDE-E [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF] both seem not able to handle the foreground well.

Moreover, we use all test scenes (12 in total) of WLF for further comparisons by mse, bad-0.15, bad-0.3, bad-0.6 and bad-1 metrics. As shown in Table VIII, our end-to-end trained model produces the lowest average errors in all metrics even with the fewest parameters (110 times fewer than LBDE-E [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF]). We finally compare our performance to the only two ConvNet based methods that allow adaptive angular light field inputs (9x9, 7x7 and 5x5 light fields) during inference. Fig. 11 shows that when the angular resolution of light fields is lower, the performance of our model that is trained from 9x9 light field inputs gradually degrades but is still much better than HSM [START_REF] Yang | Hierarchical deep stereo matching on high-resolution images[END_REF] and LBDE-E [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF].

2) Real-world dataset: Fig. 12 demonstrates visual comparisons on Google (5x5 light fields) and ULB (9x9 light fields) test scenes respectively. We exclude [START_REF] Leistner | Learning to think outside the box: Wide-baseline light field depth estimation with epi-shift[END_REF] for this comparison since the models that it provided only allowed 9x9 light field inputs. Though the proposed model is a lightweight model, it is capable of producing the more accurate disparity maps in real-world scenes when compared to the other ConvNet-based methods. Specifically, for both scenes, EPINET [START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF] is still not able to predict disparities, similar to their results from synthetic datasets. Ours have few noticeable artifacts in the foreground than that in LBDE-E [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF], and in both foreground and background than that in HSM [START_REF] Huang | Empirical bayesian light-field stereo matching by robust pseudo random field modeling[END_REF]. When compared to the traditional methods, ours have fewer artifacts than LF OCC [START_REF] Wang | Occlusion-aware depth estimation using light-field cameras[END_REF] and have fewer over-smoothness issues at occlusion regions in RPRF [START_REF] Huang | Empirical bayesian light-field stereo matching by robust pseudo random field modeling[END_REF]. Besides we can clearly see from the background of the "Path" scene, our model is able to capture more details than the other methods, e.g., more details are recovered on persons.

D. Performance on narrow-baseline datasets

Though the focus of our work is on wide-baseline datasets, we also evaluate the performance of the proposed model on narrow-baseline datasets. The quantitative and qualitative comparison are further made between the proposed model and the recent state-of-the-art methods.

1) Quantitative evaluation: For evaluations, we test frequently-used datasets used in previous works, i.e. seven HCI and eight CVIA-HCI test scenes, and assess by metrics (mse, bad-0.1 and bad-0.07) defined in [START_REF] Wanner | Datasets and benchmarks for densely sampled 4d light fields[END_REF][START_REF] Honauer | A dataset and evaluation methodology for depth estimation on 4d light fields[END_REF][START_REF] Wanner | Variational light field analysis for disparity estimation and super-resolution[END_REF]. We compare our method with state-of-the-art traditional and ConvNet-based methods, and the results are summarized in Table IX. Our model improves the mse and makes a decrease by a large percentage in bad-0.1, when compared with the best published model EPINET [START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF]. Our model achieves similar accuracy with EPINET in bad-0.07 metric, however our model is endto-end trained in less than two days on the same training set with EPINET that is trained more than five days. Furthermore, our model has the smallest capacity among ConvNet models, and runs the fastest (less than 0.5s per frame) among all topperforming methods in the narrow-baseline scenario.

2) Qualitative evaluation: Fig. 13 shows visual comparison results of our method with top-performing methods in Table IX on the scenes from synthetic datasets (HCI and CVIA-HCI) and real-world dataset (EPFL [START_REF] Rerabek | New light field image dataset[END_REF]). Clearly, the proposed method achieves the highest quality of disparity maps, especially on the challenging real-world scenes. Our model not only can recover disparities of the smooth surface with less noise, but is capable of capturing more details at occlusion areas, e.g., chain link fences or empty circles in real-world scenes.

E. Limitations and future work

The performance of the proposed LLF-Net is limited to some challenging issues, i.e. large textureless regions, heavily occluded (textureless or view-dependent) regions and non-Lambertian regions. In fact, it is very difficult to find the correspondences in the large textureless region since there are

Path

Central view LF OCC [START_REF] Yang | Hierarchical deep stereo matching on high-resolution images[END_REF] RPRF [START_REF] Huang | Empirical bayesian light-field stereo matching by robust pseudo random field modeling[END_REF] HSM [START_REF] Yang | Hierarchical deep stereo matching on high-resolution images[END_REF] EPINET [START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF] LBDE-E [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF] Ours Unicorn Central view LF OCC [START_REF] Yang | Hierarchical deep stereo matching on high-resolution images[END_REF] RPRF [START_REF] Huang | Empirical bayesian light-field stereo matching by robust pseudo random field modeling[END_REF] HSM [START_REF] Yang | Hierarchical deep stereo matching on high-resolution images[END_REF] EPINET [START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF] LBDE-E [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF] Ours Central view RPRF [START_REF] Huang | Empirical bayesian light-field stereo matching by robust pseudo random field modeling[END_REF] EPINET [START_REF] Shin | EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images[END_REF] LBDE-E [START_REF] Shi | A framework for learning depth from a flexible subset of dense and sparse light field views[END_REF] Ours a large number of similar pixels or features in the search range. One example (Luxembourg scene) shown in Fig. 13, where the large textureless region, i.e. sky, is in the background.

The correct disparity should be in dark color, which is almost not found in the disparity maps of our method and compared methods. This may be tackled by utilizing the segmentation technique [START_REF] Kim | Scene reconstruction from high spatio-angular resolution light fields[END_REF] to partition the image into the textureless and texture regions beforehand. The heavily occluded (textureless or view-dependent) region is also a tough issue to handle, as shown in the Sideboards scene of Fig. 10 and the Path scene of Fig. 12. This issue is considered in our future work, in which the ground truth occlusion data samples are created and used in supervision. Finally, similar to most of the existing ConvNetbased methods, the proposed LLF-Net does not pay attention to the non-lambertian issue [START_REF] Alperovich | Light field intrinsics with a deep encoder-decoder network[END_REF] either, such as the illumination changes, specular reflection and transparency, while this exists in the real-world scenes (e.g. Luxembourg scene in Fig. 13).

Thus designing a new ConvNet that precisely estimates depth of both lambertian and non-lamertian surfaces is a promising future work.

VI. CONCLUSION

We introduce a large-scale wide-baseline synthetic light field dataset, which can be used for training or comparing competing methods in wide-baseline light field scenarios. We present a novel end-to-end trainable lightweight network LLF-Net based on the cost volume and attention modules for wide-baseline light field depth estimation, allowing flexible angular inputs. Experimental results show that the LLF-Net not only significantly improves the state-of-the-art performance in estimating depth in wide-baseline scenarios, but also narrowbaseline scenarios. Compared to the ConvNet-based methods on flexible angular inputs, our LLF-Net is adaptive and achieves better depth accuracy.

Fig. 2 .

 2 Fig. 2. Overview of the proposed network architecture.

Fig. 3 .

 3 Fig. 3. Variants for cost fusion (best viewed in color).

Fig. 4 .

 4 Fig. 4. Epipolar view and stream residual attention. Global max-pooling is used in pooling to downsize inputs, and each first 1x1x1 convolution is followed by a ReLU activation.

Fig. 5 .

 5 Fig. 5. Examples of WLF dataset: the central view and colored ground truth disparity map are shown.

Fig. 6 .

 6 Fig. 6. Comparisons of the DCS fusion and Sum fusion on flexible angular inputs.

Fig. 7 .

 7 Fig. 7. Visual comparisons of the DCS fusion and Sum fusion on flexible angular inputs.

Fig. 8 .

 8 Fig. 8. Visual comparisons of depth estimation results without and with attention block. (a) central view with a selected patch P in pink bounding box, (b) the patch P (the intersection) and the corresponding patches in the horizontal and vertical views, where the red point indicates the pixel in the central view is occluded in the current view, and the green point means visible in the current view, (c) the ground truth disparity of patch P , (d) the estimated disparity map without attention block and (e) the estimated disparity map with attention block (best viewed in color).

3 )

 3 Dataset scheduling: To check the necessity of the Flying-Object subset in the proposed WLF dataset, we performed ablation experiments under two different training set scheduling schemes. As shown in Table V, the qualitative performance with Flying-objects (with large-scale training frames) in training is improved by a large margin. 4) Weight sharing in Feature extraction: We trained the LLF-Net with and without the use of the weight sharing, respectively. The two cases are compared in terms of the number of parameters and the bad pixels with two threshold values 0.3 and 0.6, cf. Table

Fig. 9 .

 9 Fig. 9. Comparisons of the depth accuracy using different size patches during training.

Fig. 10 .

 10 Fig. 10. Visual comparison results of the scenes from the WLF dataset: the central view and ground truth disparity map are shown in the first row, and the other rows show the predicted disparity maps and bad pixel error maps from state-of-the-arts respectively (Best viewed in color).

Fig. 11 .

 11 Fig. 11. Performance comparisons results from testing the various angular light field inputs.

Fig. 12 .

 12 Fig. 12. Visual comparison results of wide-baseline real-world datasets: the central view and colored disparity map are shown (best viewed in color).

Fig. 13 .

 13 Fig. 13. Visual comparisons of synthetic and real-world datasets: the central view and colored disparity map are shown (best viewed in color).

TABLE I THE

 I DETAILS OF THE PROPOSED NETWORK ARCHITECTURE.

	Layers	Output size	Input layer	Output layer
		Feature extraction (for each X

TABLE II DATASETS

 II STATICS OF WLF

	Dataset	#train #test spatial resolution disparity range
	Hand-designed 24	12	1920 ×1080	[0, 50]
	Flying-objects 345		512 ×512	[0, 50]

TABLE III COMPARISONS

 III OF THE BAD-0.3, BAD-0.6 AND PARAMETERS FOR THREE FUSIONS IN Cost volume generation. THE BEST PERFORMANCE IS IN BOLD.

	Fusion	Parameters	Adaptive	Hand-designed bad-0.3 bad-0.6
	Concat	2.5M		7.00	3.37
	Sum	1.5M		12.72	5.33
	DCS	1.8M		7.01	3.04

TABLE IV COMPARISONS

 IV OF THE DEPTH ACCURACY AND PARAMETERS WITH AND WITHOUT ATTENTION NETWORKS IN Cost aggregation.

	Dataset	Parameters	Hand-designed bad-0.3 bad-0.6
	w/o Attention	1.79 M	7.01	3.04
	Attention	1.82 M	6.25	2.72

TABLE V TRAINING

 V DATASET SCHEDULING.

	Dataset	Hand-designed bad-0.3 bad-0.6
	Hand-designed	10.18	5.53
	Hand-designed+Flying-objects	6.25	2.72

TABLE VI COMPARISONS

 VI OF THE DEPTH ACCURACY AND PARAMETERS WITH AND WITHOUT WEIGHT SHARING IN Feature extraction.

	Dataset	Parameters	Hand-designed bad-0.3 bad-0.6
	No sharing	2.02 M	11.25	5.12
	Sharing	1.82 M	6.25	2.72

TABLE VII BAD

 VII PIXEL ERROR PERCENTAGES OF THE FOUR EXEMPLAR SCENES OF THE WLF DATASET AGAINST THE GROUND TRUTH.

	Scene	Buddha2 bad-0.3 bad-0.6	Furniture2 bad-0.3 bad-0.6	Perikles bad-0.3 bad-0.6	Sideboards bad-0.3 bad-0.6
	LF OCC [3]	98.41	88.64	97.96	49.75	98.49	92.93	97.01	73.67
	RPRF [13]	11.10	0.86	17.11	1.16	14.82	1.26	31.15	25.46
	HSM [21]	32.32	7.27	21.98	7.96	19.09	3.48	53.27	39.90
	EPINET [19]	100	100	100	100	100	100	100	100
	EPINET T [19]	97.65	92.05	96.40	88.27	97.39	94.79	95.79	91.63
	EPI-Shift [26]	22.22	4.51	22.09	6.57	48.81	9.36	50.67	38.16
	LBDE-E [25]	14.01	7.72	16.92	8.18	46.20	32.78	45.89	39.23
	Ours	1.44	0.74	1.93	1.01	3.30	0.58	21.17

TABLE VIII PERFORMANCE

 VIII COMPARISON RESULTS ON THE WLF TEST SET. THE AVERAGE ERRORS OF ALL FRAMES ARE LISTED AND THE BEST PERFORMANCE IS IN BOLD. THE QUANTITY OF PARAMETERS OF CONVNET-BASED METHODS IS IN MILLION (M).

	Method	Parameters	End-to-end trained	mse	Hand-designed bad-0.15 bad-0.3	bad-0.6	bad-1
	LF OCC [3]	-	-	13.56	98.86	97.54	78.63	40.86
	RPRF [13]	-	-	1.70	40.43	16.01	5.43	4.70
	HSM [21]	3.1		1.58	62.22	36.08	14.22	8.52
	EPINET [19]	5.1		458.13	100	100	100	100
	EPINET T [19]	5.1		86.89	98.56	97.10	94.11	89.92
	EPI-Shift [26]	31.6		20.76	61.55	35.95	14.65	12.59
	LBDE-E [25]	198.8		11.12	36.86	29.02	20.86	16.09
	Ours	1.8		0.93	15.04	7.05	3.95	2.80

TABLE IX PERFORMANCE

 IX COMPARISON RESULTS ON NARROW-BASELINE LIGHT FIELD DATASETS. Method Parameters (M) End-to-end trained mse bad-0.1 bad-0.07 Training (days) Inference time (s)

	LF OCC [3]	-	-	3.89	17.89	30.16	-	1.05e4
	LF [4]	-	-	5.61	10.74	16.20	-	1.01e4
	RPRF [13]	-	-	3.37	10.32	15.93	-	34.53
	EPINET [19]	5.1		2.68	9.06	10.54	5-6	1.98
	EPI-Shift [26]	31.6		11.41	10.89	14.84	4	22.57
	LBDE-E [25]	198.8		3.86	9.86	13.61	≈ 2	1.92
	Ours	1.8		2.13	6.60	10.66	≈ 1.6	0.46
	Buddha
	Cotton
	Luxembourg
	Metal 1
	Mort
	Swans 2

https://www.blender.org/

https://chocofur.com, https://sketchfab.com, https://free3d.com


Hinostroza Sáenz
email: israel.hinostroza@centralesupelec.fr

CentraleSupélec Israel D Sondra

Thierry Letertre
email: thierry.letertre@centralesupelec.fr

Ren

Chengfang
email: chengfang.ren@centralesupelec.fr

Analysis of airplane signatures using passive VHF radar for recognition perspectives

Keywords: RCS, bistatic, passive radar, VHF, full-wave simulation, aircraft

Radio beacons, at around 115 MHz (VHF, VOR), are used as opportunity donors in a passive radar configuration. Full-wave simulation of three categories (according to size) of aircrafts shows that their bistatic RCS are statistically comparable without any additional information but perform differently in time (while flying). This discrepancy can be exploited to recognize the aircrafts among the categories. Measurements confirm these ideas and indicate that it may be possible to differentiate the aircrafts among the same category. Encouraging initial results were obtained using Convolutional Neural Networks to classify airplane categories combining simulated bistatic RCS results and real trajectories (collected from ADS-B data).

I. INTRODUCTION

To facilitate the flight navigation, radio beacons, as VORs (VHF Omnidirectional Range) since 1940's, have been installed around the world. The main advantage of VORs is that they transmit an omnidirectional signal, horizontal polarization, in the VHF band (108 -118 MHz) [START_REF] Azarian | Revisiting VOR transmitters: MIMO passive radar using legacy infrastructure[END_REF] [2] making it ideal, as opportunity sources, for passive radar applications [START_REF] Saillant | Using VHF navigation aid to estimate radar cross section of airliners by inverting the radar equation[END_REF] [START_REF] Pisane | Automatic real-time collection of RCS of airplanes in a real bistatic low-frequency configuration using a software defined passive radar based on illuminators of opportunity[END_REF]. In this band the wavelength is about 2.6 m which means that the size of commercial airplanes is on the order of tens of wavelengths. By exploiting the Automatic Dependent Surveillance-Broadcast (ADS-B) data broadcasted (L band) by the airplanes, the position of the airplane can be known and its radar cross section (RCS) can be obtained, as in [START_REF] Saillant | Using VHF navigation aid to estimate radar cross section of airliners by inverting the radar equation[END_REF], and it is of the order of tens of dBsm. Work has also been done to classify, according to size (small, medium, large), the airplanes by exploiting an aggregation of RCS of the airplanes [START_REF] Pisane | Automatic real-time collection of RCS of airplanes in a real bistatic low-frequency configuration using a software defined passive radar based on illuminators of opportunity[END_REF] [START_REF] Pisane | Automatic Target Recognition for Passive Radar[END_REF]. Contrary to optical recognition, characterization of an airplane by its RCS signature is challenging in VHF band, but independent of daytime and weather.

In this paper, we highlight, through simulation and measurements, discriminant RCS patterns for different sizes of planes. First, we present some simulation results of RCS of three airplanes (small, medium, large) in order to understand better their similarities and differences. The second part presents the measurement setup and collected signals, showing a good agreement with the simulation results. The third part focus on the classification of airplanes using convolutional neural networks (CNN) with 2D images (RCS values vs. aspect and bistatic angle). This representation seems to be relevant based on the obtained results. Finally, some conclusions and perspectives are presented.

II. MODELLING AND SIMULATION OF AIRPLANES

A. Airplane models

The dimensions and number of motors of each airplane model can be seen in the Tab. I (see Fig. 1 for a scale). The dimensions of the large, medium and small models, respectively. Perfect electrical conductor (PEC) was considered for the models.

B. Simulation results -RCS

A full-wave simulation using method of moments (MoM) was carried out in FEKO. See Fig. 2 for the simulation setup and spherical coordinates used for the simulation. An incident horizontally polarized (along φ axis) plane wave was considered with incident direction (θi, φi) ∈ [90, 180] × [0, 180] (due to the symmetry of the planes, incident angles φi larger than 180 are obtained from the original simulations) and scattered angles (θs, φs) ∈ [0, 180] × [0, 360], in both cases with angular step of 1deg. Fig. 3 shows the histograms of bistatic RCS of the airplanes models (small, medium and large) for HH (φ-φ) and HV (φ-θ) cases and for all scattered and incident angles considered lines above. We can see that most of the RCS values are concentrated between 0 dBsm and 40 dBsm (in accordance with [START_REF] Saillant | Using VHF navigation aid to estimate radar cross section of airliners by inverting the radar equation[END_REF], Airbus 321 is comparable to the model of medium size used lines above). Most importantly, in global, RCS values between the airplanes are very similar, making the discrimination difficult from RCS only. Additionally, there is no much difference, in global, between RCS values for the HH and HV cases. Since the VOR transmitter uses H polarization, a reception using V polarization helps to strongly reduce the direct path.

C. Simulation results -Power time series

Using the simulation results and ADS-B data from the airplanes, we can also synthesize the variation of the bistatic RCS of the planes for the trajectories of the planes. For this case, we consider a passive radar configuration where the transmitter (Tx) is the RBT VOR (close to Rambouillet), and the receiver (Rx) is near CentraleSupélec -Gif. Fig. 4 shows the position of the Tx, Rx, the plane's trajectory (obtained from real ADS-B data for about 6 minutes) and a "zoom" version of the trajectory (between 3.5 and 5.5 minutes), all of them in Lambert 93 coordinates (spatial units in meters). The plane moves from left to right (approaching landing at Orly airport). Using this "single" trajectory, the received power (considering a transmitting power of 40 dBW) versus time (in min, 2Hz of sampling) is shown for the three planes and polarizations (HH, HV) in Fig. 5.

From Fig. 5, we can see that the ranges of values of received power, per polarization, are similar (as expected from the histograms, cf. Fig. 3) besides a peak of 5 dB (around 3.85 min) above the rest of the other cases, which corresponds to the large airplane. The RCS (not shown) for all the cases is around 30 dBsm for HH polarization and 15 dBsm for HV polarization.

What is different, globally, is the frequency of fluctuation of the power (see Fig. 5), which is different according to the size of the airplane: high frequency for large plane, medium frequency for medium plane and low frequency for small plane, and for both polarizations. This is somewhat expected since, due to the movement of plane, the space of bistatic RCS of the planes is swept with the same rate. The frequency of fluctuations indicates, in this case, the angular frequency due to sweeping throughout the peaks (width and number) of the scattered power due to the illumination of the object: large objects will induce more and finer peaks compared to smaller objects.

We can also see that the received power for the HH polarization is about 15 dB higher than the HV polarization (this might not always be the case, see histograms, Fig. 3). We note that in all the cases we have used a single trajectory and single movement dynamics (same speed, orientation), hence the frequency of fluctuations of RCS (or received power) in real life will depend on the speed of the planes. In order to avoid this problem an estimation of the position of the airplane may be used to transform the frequency of fluctuations versus time into angular frequency.

III. MEASUREMENTS

A. Measurement setup

To collect the scattered power from the airplane, a receiver was installed close to CentraleSupélec-Gif. A VHF antenna with horizontal polarization was used. A SDR receiver was also used to collect complex signals, centered at VOR frequency (114.7 MHz), and connected to a PC (commanded by a Python script). An ADS-B decoder was also used to retrieve the trajectory of the airplane and type of the airplane. Note that the trajectories of the planes (for this data collection) are very similar: approaching Orly-airport.

B. Examples of Collected data

Fig. 6 shows the evolution of relative received power (after pre-processing) in dB for 4 airplanes and two categories: small and medium. The planes are approaching Orly-airport, and, hence, following a similar flight path (although not exactly same path, speed, and attitude). The received power in Fig. 6 contains the reflected signal on the airplane (from RBT VOR) and the "direct" path but strongly attenuated (level of about -66 dB) due to losses in the propagation.

From Fig. 6 we can appreciate the differences on the power evolution between categories: in amplitude in general, as well as in relative level between the main peak (highest level) and secondary ones (close to the main peak). At this stage, we recall that we cannot compare the frequency of fluctuations on the received power due to the different speeds of the plane. It can also be seen that there are clearly differences between the two airplanes of the category small. Note as well that the power evolutions related to the medium category (Boeing 737) are very similar since they have similar flight path and attitude. It is interesting to note that, for the case of Boeing 737 from Fig. 6 which is comparable to the category medium (red line in Fig. 5, see also Tab. I) for the simulations, the difference of level in dB between the peak (-62 dB, relative) at the left side of the green circle and the maximum peak (-57 dB, relative) is about 7 dB, which is similar to the difference of levels for the category medium (red line) in Fig. 5 (see peak at 3.2 min, -80 dBsm and peak at 3.8 min, -72 dBsm).

Convolutional layers

Conv (6x6 (x32 channels)), ReLu MaxPool (3x3) and dropout (0.5) Conv (6x6 (x64 channels)), ReLu MaxPool (3x3) and dropout (0.5) Conv (6x6 (x128 channels)), ReLu MaxPool (3x3) and dropout (0.5)

Fully connected layers

Fully connected (256), ReLu

Fully connected (3), Softmax

Output

IV. RESULTS OF CLASSIFICATION WITH CNN Similar to section II.C, we collected 269 trajectories (using ADS-B data) over which the bistatic RCS are synthesized for each trajectory, 3 categories (or classes) of plane sizes, and 11 VORs (around Paris). A natural representation of these data is considering images of RCS values vs. bistatic and aspect angles [START_REF] Pisane | Automatic real-time collection of RCS of airplanes in a real bistatic low-frequency configuration using a software defined passive radar based on illuminators of opportunity[END_REF] [START_REF] Pisane | Automatic Target Recognition for Passive Radar[END_REF]. Since CNN has usually shown good results for image classification (AlexNet [START_REF] Krizhevsky | ImageNet classification with deep convolutional neural networks[END_REF], VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], GoogleNet [START_REF] Szegedy | Going deeper with convolutions[END_REF], ResNet [START_REF] He | Deep residual learning for image recognition[END_REF]), we propose a similar architecture (see Tab. II for details). In order to train the CNN, a 2D image (bistatic and aspect angle, 181 x 361 pixels, 1 pixel per angle) was constructed using the RCS variations for each trajectory and for the 11 VORs (see Fig. 7) where the colors represent the bistatic RCS values in dBsm (for VV polarization) for each coordinate. Preliminary results considering a single VOR at a time gave very low successful rates of classification due to the high sparsity of the images.

The blue zones in Fig. 7 correspond to the absence of ADS-B data (hence, no RCS is attributed to those coordinates). In this way a total of 807 images (269 trajectories times 3 categories) was fed to the CNN. The dataset was randomly splitted in 3 parts: training dataset (70%), validation dataset (15%) and test dataset (15%). Tab. III shows the confusion matrix for the classification of validation and data set (242 samples). From this table, recognition of small class of airplanes seems better than medium and large class. V. CONCLUSIONS AND PERSPECTIVES It has been shown, through simulations, that, globally, the bistatic RCS of airplanes of different sizes (large, medium and small) are similar, which makes difficult their discrimination on the sole basis of RCS value. It has been also shown that it is more sensible to look for the angular frequency (extraction of this information can be done as in [START_REF] Saillant | Using VHF navigation aid to estimate radar cross section of airliners by inverting the radar equation[END_REF]) of the bistatic RCS when the airplane is moving in order to discriminate the planes by sizes (larger planes will scatter power with finer and more peaks). These simulation results, can be used to synthesize bistatic RCS evolution for arbitrary trajectories (for example based on real trajectories) and include many VORs and different positions of receiver, in order to consider multiple scenarios based on real data to train neural networks (which need a large collection of data that many times are not available by measurement) on the recognition of airplanes. Measurements have been carried out in a passive radar configuration (Tx RBT VOR, Rx near CentraleSupélec -Gif-sur-Yvette) to collect received (scattered) power from the airplanes that are landing at Orlyairport. It has been shown that, effectively, there are enough differences on the received power from the airplanes to differentiate them. It might possible to differentiate among planes of the same category. Finally, some results were obtained using a CNN architecture to classify the airplanes by sizes using the simulation results (considering 269 trajectories, 3 classes and 11 VORs). The results provide a good, and encouraging, rate of classification.

Fig. 1 .

 1 Fig. 1. From top to bottom: large, medium and small airplane models used in FEKO.

Fig. 2 .

 2 Fig. 2. Simulation setup of airplane models, PEC, in FEKO and definition of spherical coordinates used in the simulations.

Fig. 3 .

 3 Fig. 3. Histograms of bistatic RCS (dBsm) obtained by simulation, HH (φφ, top) and HV (φ-θ, bottom), for small, medium and large airplane models.

Fig. 4 .

 4 Fig. 4. Position of Tx (RBT VOR) in green, Rx (near CentraleSupélec-Gif) in red, plane's trajectory in blue (ADS-B data for 6.3 minutes) and "zoom" of trajectory in thick red line (between timestamps 3 and 5 min). The plane is moving from left to right. Lambert 93 coordinates (meters)

Fig. 6 .

 6 Fig.6. Received relative level (dB) of power, HH, after pre-processing of reflected signal on airplanes approaching Orly-airport. Two categories are present: top with small (jet CRJ701) and medium size (Boeing 737) airplane; bottom with medium (Boeing 737) and small size airplane (airplane with propeller ATR-42 500).

Fig. 7 .

 7 Fig. 7. 2D image of bistatic RCS variations (in color) for bistatic and aspect angles, 181 x 361 pixels (similar to [5]), for a single trajectory and 11 VORs (hence, 11 different illumination angles of the airplane).

TABLE I .

 I AIRPLANE MODELS DIMENSIONS

TABLE II

 II

	.	CNN ARCHITECTURE
	Input Size	(181 x 361)

TABLE III

 III

		.	CONFUSION MATRIX
	Predicted		Actual Class
	class	Small	Medium	Large
	Small	96.2%	1.2%	0%
	Medium	3.8%	84.5%	8.7%
	Large	0%	14.3%	91.3%


Naga Devarapalli

Israel Hinostroza
email: israel.hinostroza@centralesupelec.fr

Regis Guinvarcrh
email: regis.guinvarch@centralesupelec.fr

Randy L Haupt
email: rhaupt@ieee.org

Régis Guinvarc'h Sondra

Resonances in Receiving Two-Arm Spiral Antennas

Keywords: planar array, spiral antennas, resonances, transmission line model

whether they are published or not. The documents may come

Resonances in Receiving Two-Arm Spiral Antennas Naga R. K. Devarapalli Mahindra École Centrale Hyderabad, India nrdevara@gmail.com

I. INTRODUCTION

At certain frequencies, a planar array of symmetrical square spiral antennas over a ground plane is not effective for scanning off-broadside [START_REF] Steyskal | Spiral elements for broadband phased arrays[END_REF]. These frequencies are termed as resonances and the current distribution on the arms of the spirals forms a strong standing wave pattern. A spiral antenna becomes resonant at a frequency when the lengths of its arms are multiples of half the resonant wavelength. It has been observed that mutual coupling between the array elements contributes to these resonances [START_REF] Steyskal | Spiral elements for broadband phased arrays[END_REF]. The idea of a "spoiled" spiral (arms with a slight asymmetry in terms of length) was suggested in [START_REF] Steyskal | Spiral elements for broadband phased arrays[END_REF] for suppressing resonances. A lossless transmission line model was then proposed in [START_REF] West | Analysis and feeding of a spiral element used in a planar array[END_REF] to explain the resonances in a spiral antenna in an array with a ground plane. The model was also used to verify that a perfect standing wave may not be set up on a spoiled spiral.

We show here that even a single symmetrical spiral antenna without a ground plane, in the receiving mode, may become resonant when the lengths of its arms are multiples of half the corresponding wavelength. Therefore, there does not seem to be a fundamental requirement for the presence of a ground plane or for a spiral to be in an array configuration.

In fact, resonances have also been observed in spirals of various shapes. There is a standing wave current distribution setup on resonant spiral antennas, with the incident wave offbroadside, impeding the receiving capability of the antennas.

II. RESONANCES ON A SINGLE SPIRAL ANTENNA WITHOUT GROUND PLANE

Fig. 1 shows a single two-arm symmetrical square spiral antenna in the receiving mode. The spiral is terminated in a load ZL at the center. If a linearly polarized plane wave illuminates the antenna along the plane of the spiral a current distribution is induced along the arms of the antenna. The spiral shown in the configuration in Fig. 1 becomes resonant when the incident fields satisfy any of the criteria in (1) or (2).

E1≠0 & E2=0

(1)

E1=0 & E2≠0 (2)
E1 and k1 (E2 and k2) correspond to electric-field and propagation vectors of an incident plane wave.

Taking the center of a spiral as a reference, the plots in Fig. 2 show the amplitude of surface current along the length of an arm on two different resonant two-arm symmetrical spiral antennas. Simulations show that the spiral becomes resonant when the impinging plane wave is off-broadside and when the lengths of the arms are multiples of half the corresponding wavelength associated with the incident frequency. Moreover, the resonances are strongest when the wave's propagation and electric-field vectors are parallel to the plane of the spiral.

Table I shows geometrical dimensions and resonant frequencies considering two types of resonant two-arm symmetrical spiral antennas without a ground plane, where f is a resonant frequency in GHz, l is the length of each spiral arm in mm, lλ is the spiral arm length in multiples of the freespace wavelength (λ) corresponding to the resonant frequency and w is the width of each spiral arm in mm. The load impedance at the feed-terminals of the spirals is 220Ω. Note that the resonance phenomenon in Fig. 2, for a single spiral antenna without a ground plane, is very similar to the ones reported in [START_REF] Steyskal | Spiral elements for broadband phased arrays[END_REF] and [START_REF] West | Analysis and feeding of a spiral element used in a planar array[END_REF] for planar arrays of symmetrical square spiral antennas over a ground plane.

To model the resonances, we have considered a double transmission line model, which has the same geometry as in [START_REF] West | Analysis and feeding of a spiral element used in a planar array[END_REF]. If the arms of the spiral of Fig. 1 are unwrapped, considering the proximity between the arms, we obtain a double transmission line model without the need of an actual ground plane (see Fig. 3). Blue and red colors represent the arms 1 and 2 and the load ZL at the center is in green. However, the transmission lines in our model are considered to be lossy. Even after using the lossy transmission line model, we have obtained the same criteria required for the spiral to become resonant.

III. CONCLUSIONS

Our analysis is helpful in explaining the resonances that appear in receiving spiral antennas even without a ground plane, when the incident field is off-broadside. Resonances are observed in two kinds of individual spirals and are a fundamental characteristic of the element itself. Our analysis is also consistent with previous works, which show that a spoiled spiral is helpful in suppressing strong resonant behavior.

Fig. 1 .Fig. 2 .

 12 Fig. 1. Symmetrical square spiral illuminated by two plane waves.

  Fig. 2a corresponds to the square spiral antenna in Fig. 1 resonating at 1.15 GHz. It was simulated

Fig. 3 .

 3 Fig. 3. Double transmission line model of spiral antenna of Fig. 1. Blue and red colors correspond to arm 1 and 2, respectively. L represents the arm lengths of the spiral.

TABLE I

 I

	.	GEOMETRICAL DIMENSIONS AND RESONANT FREQUENCIES
	CONSIDERED FOR VARIOUS SYMMETRICAL SPIRAL ANTENNAS
	Spiral	Square	Archimedean
	f (GHz)	1.15	0.73
	l (mm)	653.2	1426.6
	lλ	2.5	3.5
	w (mm)	3.4	2.7