Mathieu Labussière
email: mathieu.labu@gmail.com

Céline Teulière
email: celine.teuliere@uca.fr

Frédéric Bernardin
email: frederic.bernardin@cerema.fr

Omar Ait-Aider
email: aideromar.ait-aider@uca.fr

B Mathieu

Omar Ait

Leveraging Blur Information for Plenoptic Camera Calibration

Keywords: Plenoptic camera, Calibration, Multi-focus, Relative blur, Blur circle Pre-calibration Calibration Camera Features Detection Relative blur Applications Blur profile Optimized camera parameters Depth Estim

This paper presents a novel calibration algorithm for plenoptic cameras, especially the multi-focus configuration, where several types of micro-lenses are used, using raw images only. Current calibration methods rely on simplified projection models, use features from reconstructed images, or require separated calibrations for each type of micro-lens. In the multifocus configuration, the same part of a scene will demonstrate different amounts of blur according to the micro-lens focal length. Usually, only micro-images with the smallest amount of blur are used. In order to exploit all available data, we propose to explicitly model the defocus blur in a new camera model with the help of our newly introduced Blur Aware Plenoptic (BAP) feature. First, it is used in a pre-calibration step that retrieves initial camera parameters, and second, to express a new cost function to be minimized in our single optimization process. Third, it is exploited to calibrate the relative blur between micro-images. It links the geometric blur, i.e., the blur circle, to the physical blur, i.e., the point spread function. Finally, we use the resulting blur profile to characterize the camera's depth of field. Quantitative evaluations in controlled environment on real-world data demonstrate the effectiveness of our calibrations.

Introduction

From Lumigraph [START_REF] Lippmann | Integral photography. Academy of the Sciences Lumsdaine[END_REF] to commercial plenoptic cameras [START_REF] Ng | Light field photography with a hand-held plenoptic camera[END_REF][START_REF] Perwaß | SPIE human vision and electronic imaging XVII. Single Lens 3D-Camera with Extended Depth-of[END_REF]), several designs have been proposed to capture information that cannot be captured by conventional cameras. Said cameras capture only one point of view of a scene, whereas a plenoptic camera is a device that allows to retrieve spatial as well as angular information. A same point from a scene is projected into multiple observations on the sensor. For instance, this redundant information can be used for digitally refocusing and rendering [START_REF] Bishop | The light field camera: Extended depth of field, aliasing, and superresolution[END_REF] or for depth estimation [START_REF] Johannsen | A Taxonomy and evaluation of dense light field depth estimation algorithms[END_REF].

This paper focuses on plenoptic cameras based on a microlenses array (MLA) placed between a main lens and a sensor as illustrated in Fig. 3. The specific design of such a camera allows to multiplex both types of information onto the sensor in the form of a micro-images array (MIA), as shown in Fig. 1, but implies a trade-off between the angular and spatial resolutions [START_REF] Georgiev | Spatio-angular resolution tradeoff in integral photography[END_REF][START_REF] Levin | Understanding camera trade-offs through a Bayesian analysis of light field projections[END_REF]Georgiev and Lumsdaine 2009). It is balanced according to the (MLA) position with respect to the main lens focal plane and the sensor plane, corresponding to unfocused [START_REF] Ng | Light field photography with a hand-held plenoptic camera[END_REF] or focused [START_REF] Perwaß | SPIE human vision and electronic imaging XVII. Single Lens 3D-Camera with Extended Depth-of[END_REF][START_REF] Georgiev | The multifocus plenoptic camera[END_REF] configurations.

To further extend the depth of field (DoF) of the plenoptic camera, a multi-focus configuration has been proposed by [START_REF] Perwaß | SPIE human vision and electronic imaging XVII. Single Lens 3D-Camera with Extended Depth-of[END_REF]; [START_REF] Georgiev | The multifocus plenoptic camera[END_REF]. In this setup, the MLA is composed of several micro-lenses with different focal lengths. The same part of a scene will be more or less focused according to the micro-lens' type.

(b) (c) (a)

Fig. 1 The Raytrix R12 multi-focus plenoptic camera used in our experimental setup (a), along with a raw image of a checkerboard calibration target (b). The image is composed of several micro-images with different amounts of blur, arranged in a hexagonal grid. In each micro-image, our Blur Aware Plenoptic (BAP) feature is illustrated by its center and its blur circle (c).

Usually, only micro-images with the smallest amount of blur are used. Alternatively, specific patterns are used to exploit the information [START_REF] Palmieri | Optimizing the lens selection process for multi-focus plenoptic cameras and numerical evaluation[END_REF]. If one were able to relate the camera parameters to the amount of blur in the image, all information could be used simultaneously, without distinction between types of micro-lenses. As a first step in that direction, we propose a calibration method that takes advantage of blur information.

Calibration is an initial step for applications using plenoptic imaging. Conventional cameras are usually modeled as pinhole or thin lens. Due to the complexity of plenoptic cameras' design, the developed models are generally high dimensional. Specific calibration methods have to be proposed to retrieve the intrinsic parameters of these models.

Related Work

Unfocused plenoptic camera calibration: In the unfocused configuration, the main lens is focused at the MLA plane and the sensor plane is placed at the MLA focal plane. The MLA is therefore focused at infinity, thus calling this configuration unfocused. The calibration of such plenoptic cameras [START_REF] Ng | Light field photography with a hand-held plenoptic camera[END_REF] has been widely studied in the literature. Most approaches rely on a thin-lens model for the main lens and an array of pinholes for the micro-lenses. [START_REF] Dansereau | Decoding, calibration and rectification for lenselet-based plenoptic cameras[END_REF] introduced a model to decode the pixels into rays, drawing inspiration from [START_REF] Grossberg | The raxel imaging model and ray-based calibration[END_REF], for the Lytro plenoptic camera [START_REF] Ng | Light field photography with a hand-held plenoptic camera[END_REF]). Their model is not directly associated with physical parameters and is based on corner detection in reconstructed sub-aperture images (SAIs). [START_REF] Zhou | Elsevier Ltd 190-196 a two-step calibration method of lenslet-based light field cameras[END_REF] proposed a practical two-step calibration method for unfocused plenoptic cameras. Their model describes the camera physical parameters but still requires feature points extracted in reconstructed SAIs. Yunsu et al. (2014) formulated a geometric projection model to estimate intrinsic and extrinsic parameters by utilizing raw images directly to avoid errors from reconstruction steps. Their method includes analytical solution and non-linear optimization of the reprojection error of a novel line feature to overcome the difficulties in finding checkerboard corners. [START_REF] Shi | Elsevier 70-88 parametric study on light field volumetric particle image velocimetry[END_REF] proposed a detailed model of a plenoptic camera in the context of particle image velocimetry (PIV). Based on linear optics, they derived a model based on ray-tracing: contrarily to previous methods, they modeled the main lens and each micro-lens as thin-lenses. [START_REF] Hahne | Baseline and triangulation geometry in a standard plenoptic camera[END_REF] developed a ray model by ray-tracing from the sensor side to the object space. They consider only the chief ray, connecting micromage centers (MICs) to the exit pupil center. [START_REF] O'brien | Calibrating light-field cameras using plenoptic disc features[END_REF] introduced a projection model used for their calibration method suited both for unfocused and focused plenoptic cameras. They present a new feature called plenoptic disc, similar in nature to the circle of confusion (CoC) and defined by its center and its radius. Their feature parametrization is in 3D and is in one-to-one correspondence with point positions in the camera frame, as it is detected in reconstructed image. [START_REF] Zhao | Metric calibration of unfocused plenoptic cameras for three-dimensional shape measurement[END_REF] recently presented a metric calibration method for unfocused plenoptic camera only also based on the plenoptic disc but directly from raw image.

In summary, most of the above methods require reconstructed images (SAIs) to extract features, and limit their model to the unfocused configuration, i.e., setting the sensor plane at the micro-lens focal plane. Therefore those models cannot be directly extended to the focused or multi-focus plenoptic camera.

Focused plenoptic camera calibration: With the arrival of commercial focused plenoptic cameras (Lumsdaine and Georgiev 2009;[START_REF] Perwaß | SPIE human vision and electronic imaging XVII. Single Lens 3D-Camera with Extended Depth-of[END_REF], new calibration methods have been proposed. In this configuration, the micro-lenses focus on an intermediate image plane. [START_REF] Johannsen | On the calibration of focused plenoptic cameras[END_REF] formulated a general reprojection model in terms of the physical parameters of a Raytrix camera [START_REF] Perwaß | SPIE human vision and electronic imaging XVII. Single Lens 3D-Camera with Extended Depth-of[END_REF]. They proposed a metric calibration and distortions correction using a grid of circular patterns. This work considered a relatively simple model of lens distortion and required careful initialization of the optimization to converge due to high sensibility to local minima. [START_REF] Heinze | Automated Robust metric calibration algorithm for multifocus plenoptic cameras[END_REF] improved the previous model by considering more sophisticated models of the main lens distortions. They introduced new parameters including the tilt and shift for the main lens. They are able to distinguish each micro-lens type, calibrating then the distance between the MLA and the sensor for each one but in separated calibration processes. The projection model and the metric calibration procedure are incorporated in the 123 Content courtesy of Springer Nature, terms of use apply. Rights reserved.

RxLive software of Raytrix GmbH. [START_REF] Strobl | Stepwise calibration of focused plenoptic cameras[END_REF] presented a step-wise calibration approach to overcome the fragility of the initialization which hinders the final optimization. They first determined main lens parameters, then estimated MLA parameters. However, their calibration framework relied on reconstructed total focus images. [START_REF] Zeller | Calibration and accuracy analysis of a focused plenoptic camera[END_REF] introduced two new methods to calibrate a focused plenoptic camera and depth images obtained from it. In further works (Zeller et al. 2016), they improved the camera projection model by modeling the main lens as a thin lens instead of a pinhole. The calibration process uses the reconstructed total focus image and virtual depth map to compute 3D observations.

All previous methods rely on reconstructed images (SAIs), which can lead to the introduction of errors in the reconstruction step as well as in the calibration process. Usually, computation of reconstructed images requires camera parameters and/or depth information to avoid artifacts and reconstruction error. To overcome this chicken and egg problem, several calibration methods focus on using only raw plenoptic images. [START_REF] Zhang | Unconstrained two-parallel-plane model for focused plenoptic[END_REF] proposed a calibration method based directly on observations from raw images. They used a parallel bi-planar checkerboard to have a depth-scale prior. They considered a detailed model of the MLA geometry that accounts for non-planarity of the array. [START_REF] Zhang | A generic multi-projection-center model and calibration method for light field cameras[END_REF] presented a multi-projection-center model based on the two planes parametrization [START_REF] Levoy | ACM Press Proceedings of the 23rd annual conference on Computer graphics and interactive techniques-SIGGRAPH '96. Light field rendering[END_REF]. They derived a calibration algorithm based on this model and projective transformation, suitable for both unfocused and focused plenoptic cameras. [START_REF] Noury | Light-field camera calibration from raw images DICTA[END_REF] presented a more complete geometrical model than the previous works. This model relates 3D points to their corresponding image projections, working directly with raw images. They developed a new detector to find checkerboard corners with sub-pixel accuracy in each micro-image. They introduced a new cost function based on reprojection errors of both checkerboard corners and micro-lens centers in raw image space. This enforces projected micro-lens centers to get closer to their corresponding MICs, and makes their method robust to wrong parameters initialization especially concerning those of the MLA. However, their method does not consider different types of micro-lenses and forces them to act as pinholes.

Several methods can account for the multi-focus setting. [START_REF] Bok | Geometric calibration of micro-lens-based light field cameras using line features 39[END_REF] extended their previous model (Yunsu et al. 2014) to work with the focused plenoptic camera. They did not explicitly model the micro-lens focal lengths but introduced two additional intrinsic parameters that account for the MLA setting. Each setting -one for each type of microlenses -, models a different distance between the MLA and the sensor. Their method can retrieve different intrinsics by running the optimization for each type separately. [START_REF] Nousias | Corner-based geometric calibration of multi-focus plenoptic cameras[END_REF] considered the geometric calibration of multi-focus plenoptic cameras. Their method allows to identify the micro-lens types and their spatial arrangement. It operates on checkerboard corners retrieved by a custom micro-image corner detector. Then, they applied their method on each type of micro-lens independently to retrieve specific intrinsic and extrinsic parameters for each configuration. Latter researches [START_REF] Bok | Geometric calibration of micro-lens-based light field cameras using line features 39[END_REF][START_REF] Nousias | Corner-based geometric calibration of multi-focus plenoptic cameras[END_REF][START_REF] Noury | Light-field camera calibration from raw images DICTA[END_REF]) have achieved improved performance through automation and accurate identification of feature correspondences in raw images. More recently, [START_REF] Wang | Virtual image points based geometrical parameters' calibration for focused light field camera[END_REF] proposed a geometric calibration method for focused plenoptic cameras based on virtual image points, establishing the mapping from object points behind the main lens and the MLA to image points on the sensor. Their method can be extended to calibrate multi-focus cameras by considering each type of micro-lenses individually.

In conclusion, most of these methods rely on simplified models for optic elements: the MLA misalignment is not considered, and the micro-lenses are modeled as pinholes thus not modeling their apertures. Some do not consider distortions of the main lens or restrict themselves to the focused case. Finally, few have considered the multi-focus case [START_REF] Heinze | Automated Robust metric calibration algorithm for multifocus plenoptic cameras[END_REF][START_REF] Bok | Geometric calibration of micro-lens-based light field cameras using line features 39[END_REF][START_REF] Nousias | Corner-based geometric calibration of multi-focus plenoptic cameras[END_REF][START_REF] Wang | Virtual image points based geometrical parameters' calibration for focused light field camera[END_REF]) but dealt with it in separate processes, leading to intrinsic and extrinsic parameters that vary depending on the type of micro-lens.

Contributions

We present a new calibration method for plenoptic cameras. To the best of our knowledge, it is the first to allow to calibrate the multi-focus plenoptic camera within a single process taking into account all types of micro-lenses simultaneously. To exploit all available information, we propose to explicitly include the defocus blur in a new camera model. Thus, we introduce a new Blur Aware Plenoptic (BAP) feature defined in raw image space that enables us to handle the multi-focus case. We present a new pre-calibration step using BAP features from white images to provide a robust initial estimation of camera parameters. We use our BAP features in a single optimization process that retrieves intrinsic and extrinsic parameters of a multi-focus plenoptic camera directly from raw plenoptic images of a checkerboard target.

This paper extends our previous work [START_REF] Labussière | IEEE 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR). Blur Aware Calibration of Multi-Focus Plenoptic Camera[END_REF]. In addition to our former contributions, we present here an ablation study of the camera parameters and add further comparisons with state-of-the-art calibration methods. A new camera setup has also been tested to validate the generalization of our method, and a simulation setup is proposed to evaluate our method on Lytro-like configuration. Moreover, we take advantage of our BAP features to develop a new relative blur calibration process to link the geometric blur to the physical blur, i.e., the circle of confusion (CoC) Fig. 2 Overview of our proposed method: first, the pre-calibration step retrieves initial camera parameters from white raw images at different apertures; then followed by the detection of BAP features that are used by the camera calibration process and calibration ot the relative blur; finally, once the camera is calibrated, it can be used, as addressed here, for profiling the camera, i.e., to characterize the working range of the camera. Other applications can be considered, such as metric depth estimation.

Fig. 3

Focused plenoptic camera model in Galilean configuration with the notations used in this paper. Object points are projected by the main lens behind the MLA into a virtual intermediate space, and then re-imaged by each micro-lens onto the sensor.

to the point-spread function (PSF). This enables us to fully take advantage of blur in image space. Finally, we propose to use the blur to profile the plenoptic camera in terms of depth of field (DoF).

Paper Organization

An overview of our method is given in Fig. 2. The remainder of this paper is organized as follows. First, we present the camera model and how we model blur with our BAP feature in Sect. 2. Second, we explain in Sect. 3 how we leverage raw white images in the proposed pre-calibration step to initialize camera parameters. Then, we detail the feature detection in Sect. 4 and the calibration processes in Sect. 5, i.e., the camera calibration and the relative blur calibration. Our experimental setup is presented in Sect. 6. Finally, our results are given and discussed in Sect. 7. The notations used in this paper are shown in Fig. 3. Pixel counterparts of metric values are denoted in lower-case Greek letters. Bold font denotes vectors and matrices.

Camera and Blur Models

The (Multi-Focus) Plenoptic Camera

We consider the focused plenoptic camera, especially the multi-focus case as described by [START_REF] Georgiev | The multifocus plenoptic camera[END_REF]; [START_REF] Perwaß | SPIE human vision and electronic imaging XVII. Single Lens 3D-Camera with Extended Depth-of[END_REF]. The camera is composed of a main lens and photosensitive sensor with a micro-lenses array (MLA) in between, as illustrated in Fig. 3. The multi-focus configuration implies that the micro-lenses array consists of I different types of lenses. The setup corresponds to the multifocus system described by [START_REF] Perwaß | SPIE human vision and electronic imaging XVII. Single Lens 3D-Camera with Extended Depth-of[END_REF] with I = 3. Note that our model can be applied to the single-focus plenoptic camera as well, corresponding then to the case where I = 1. Finally, the unfocused configuration is a special case of our model where the micro-lens focal length is equal to the distance between the MLA and the sensor, i.e., f = d.

Main Lens

The main lens is modeled as a thin-lens and maps an object point to a virtual point in an intermediate space called the virtual space. An object at distance a is then projected at a distance b given the focal length F according to the thin-lens equation

1 F = 1 a + 1 b . ( 1
)
The main lens principal point is expressed as u 0 v 0 in image space. We model the main lens as parallel to the sensor plane. Deviations from this hypothesis will be compensated for by tangential distortion parameters. Furthermore, we define our camera reference frame as the main lens frame, with O being the origin, the z-axis coinciding with the optical axis and pointing outside the camera, and the y-axis pointing downwards. Distances are signed according to the following convention: F is positive when the lens is convergent; distances are positive when the point is real, and negative when virtual.

Distortions

We consider distortions of the main lens. Distortions represent deviations from the theoretical thin lens projection model. To correct those errors, we model the radial and tangential components of the lateral distortions using the model of Brown-Conrady (Brown 1966;[START_REF] Conrady | Decentered lens-systems[END_REF]). Depth distortions have also been studied by [START_REF] Heinze | Automated Robust metric calibration algorithm for multifocus plenoptic cameras[END_REF]; Zeller et al. (2016), but [START_REF] Zeller | From the calibration of a lightfield camera to direct plenoptic odometry[END_REF]; [START_REF] Noury | Etalonnage de caméra plénoptique et estimation de profondeur à partir des données brutes[END_REF] both empirically observed that the effects of depth distortions, for large focal length and for large object distance, can be neglected compared to stochastic noise of the depth estimation process. Therefore, we do not include depth distortion in our model.

A distorted point p = x y z 1 expressed in the main lens frame after projection (i.e., in the virtual intermediate space) is thus transformed into p u = φ(p) = x u y u z 1 and is computed as

⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ x u = x 1 + Q 1 ς 2 + Q 2 ς 4 + Q 3 ς 6
[radial]

+ P 1 ς 2 + 2x 2 + 2P 2 x y [tangential] y u = y 1 + Q 1 ς 2 + Q 2 ς 4 + Q 3 ς 6 [radial] + P 2 ς 2 + 2y 2 + 2P 1 x y [tangential]
(2)

where ς 2 = x 2 + y 2 . The three coefficients for the radial component are given by {Q 1 , Q 2 , Q 3 }, and the two coefficients for the tangential by {P 1 , P 2 }.

Micro-Lenses Array

We also model the micro-lenses as thin-lenses allowing to take into account blur in the micro-image. The MLA consists then of I different lens types with focal lengths f (i) where i ∈ [1 . . I ] which are focused on I different planes.

We make the hypothesis that all micro-lenses lie on the same plane. The MLA is approximately centered around the optic axis. We define the farthest micro-lens along the (-x)axis and the (-y)-axis as the origin of the MLA frame, i.e., the center of the upper-left micro-lens. The coordinates axes are orientated the same way as the ones of the main lens. The structural organization of the lenses can be an orthogonal or hexagonal arrangement. The MLA origin is at a distance D from the main lens and at a distance d from the sensor. Theoretically, the distances d(k, l) and D(k, l) are different for each micro-lens (k, l) due to the MLA tilt. However, using the approximation of a constant distance d provides adequate results thanks to the low order of the MLA rotation angles' as shown in Sect. 7, while allowing simpler derivation of the equations.

Furthermore, a detected micro-image center (MIC) usually does not coincide with the optical center of the considered micro-lens. We take into account this deviation in opposition to orthographic projection of MICs which causes inaccuracy in decoded light field. Therefore, the principal point c k,l 0 of the micro-lens indexed by (k, l) is given by

c k,l 0 = u k,l 0 v k,l 0 = d D + d u 0 v 0 -c k,l + c k,l , (3)
where c k,l is the center of the micro-image (k, l) expressed in pixel, as illustrated in Fig. 3 .

Micro-Images Array

Finally, each micro-lens produces a micro-image (MI) onto the sensor. The set of these micro-images has the same structural organization as the MLA. The data can therefore be interpreted as an array of micro-images, called by analogy the micro-images array (MIA). The MIA coordinates are expressed in image space. Let δ i be the pixel distance between two arbitrary consecutive micro-images centers c k,l . With s the metric size of a pixel, let Δ i = sδ i be its metric value, and Δ μ be the metric distance between the two corresponding micro-lens centers C k,l . From similar triangles, the ratio λ between them is given by

λ D d + D = Δ μ Δ i ⇐⇒ Δ μ = λΔ i = D d + D • Δ i . ( 4
)
We make the hypothesis that Δ μ is equal to the micro-lens aperture.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Camera Configuration

When the camera is in the unfocused configuration, the distance separating the sensor and the MLA is equal to the focal length of the micro-lenses, i.e., d = f . Dealing with the focused plenoptic camera, we usually consider two possible configurations as presented by Georgiev and Lumsdaine (2009): 1) Galilean, when objects are projected behind the image sensor; and 2) Keplerian, when objects are projected in front of the image sensor. When considering micro-lenses as thin-lenses, we have to take into account their focal lengths to configure the camera. In practice, considering an object projected at distance b by the main lens, four cases are possible but only two are able to produce an exploitable image, i.e., with acceptable amount of blur, onto the sensor: We introduce then the definition of the internal configuration according to the micro-lens focal length as

f < d ⇒ Keplerian internal configuration, f > d ⇒ Galilean internal configuration.
(5)

Modeling Blur Within the Plenoptic Camera

From optics geometry, the image of a point from a circular lens not focused on the sensor can be modeled by the circle of confusion (CoC). Using a camera with a circular aperture, the blurred image is also circular in shape and is called the blur circle. From similar triangles and from the thin-lens equation (eq. 1), the signed blur radius of the image of a point at a distance a from the lens is expressed as

⎧ ⎪ ⎨ ⎪ ⎩ r = A d 2 1 f - 1 a - 1 d [metric] ρ = r /s [pixel] (6)
with s being the size of a pixel, and A the aperture of this lens.

In continuous domain, the response of an imaging system to a not in-focus point, i.e., the blur, can be expressed by the pointspread function (PSF). Let I (x, y) be the observed blurred image of an object at a constant distance. The image can be computed as the convolution of the PSF noted h(x, y), with the in-focus image, I * (x, y), such as

I (x, y) = h * I * (x, y) , ( 7
)
where * denotes the convolution operator. If the lens aperture is circular and the level of blur low, the PSF h(x, y) can be efficiently modeled by a two-dimensional Gaussian given by h(x, y) = 1 2πσ 2 exp -

x 2 + y 2 2σ 2 , ( 8
)
where the spread parameter σ is proportional to the blur circle radius ρ. Therefore, we can write

σ ∝ ρ ⇔ σ = κ • ρ (9)
where κ is a camera constant that should be determined by calibration [START_REF] Pentland | A new sense for depth of field PAMI-9[END_REF][START_REF] Subbarao | Determining distance from defocused[END_REF]. Note that the spatially-variant spread parameter σ thus depends on the object distance a.

The blur radius ρ appears at several levels within the camera projection: in the blur introduced by the thin-lens model of the micro-lenses and in the formation of the micro-images while taking a white image. Each micro-lens (k, l) projects virtual points onto the sensor at a position (u, v), with a blur radius ρ depending on the distance to the point and the microlens type.

BAP Features and Projection Model

To leverage this blur information, we introduce a new Blur Aware Plenoptic (BAP) feature characterized by its center and its radius, noted p = u v ρ 1 . The BAP feature are visualized in Fig. 1. Therefore, our complete plenoptic camera model allows us to link a scene point p w = x y z 1 to our new BAP feature p in homogeneous coordinates through each micro-lens (k, l) such as

⎡ ⎢ ⎢ ⎣ u v ρ 1 ⎤ ⎥ ⎥ ⎦ ∝ P (i, k, l) • T μ (k, l) • φ (K (F) • T c • p w ) , ( 10
)
where P (i, k, l) is the blur aware plenoptic projection matrix through the micro-lens (k, l) of type i, and computed as

P (i, k, l) = P(k, l) • K f (i) = ⎡ ⎢ ⎢ ⎢ ⎣ d/s 0 u k,l 0 0 0 d/s v k,l 0 0 0 0 Δ μ 2s - Δ μ 2s d 0 0 -1 0 ⎤ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎣ 1 0 0 0 0 1 0 0 0 0 1 0 0 0 -1/ f (i) 1 ⎤ ⎥ ⎥ ⎦ . ( 11
)
P (k, l) is a matrix that projects the 3D virtual point onto the sensor and taking into account the blur radius. K ( f ) is the thin-lens projection matrix for the given focal length. T c is the pose of the main lens with respect to the world frame and T μ (k, l) is the pose of the micro-lens (k, l) expressed 123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

in the camera frame. The function φ(•) models the lateral distortions.

Finally, the projection model from eq. 10 consists of a set of (16 + I ) intrinsic parameters to be optimized, including: the main lens focal length F, expressed in K (F), and its five lateral distortion coefficients Q 1 , Q 2 , Q 3 , P 1 , and P 2 , expressed in φ(•); the sensor translations, encoded in d and (u 0 , v 0 ) through eq. 3, from P(k, l); the MLA pose, including its three rotations θ x , θ y , θ z and three translations t x , t y , D , and the micro-lens pitch Δ μ , expressed in T μ (k, l); and, the I micro-lens focal lengths f (i) , in K( f (i) ).

Profiling the Depth of Field of the Plenoptic Camera

From calibrated camera parameters, we can compute the depth of field (DoF) of each micro-lens type and the blur profile -the blur radii as function of the object distance -, in order to profile the plenoptic camera. The analysis can be done with respect to the MLA pose, and then extended to object space by back-projection. A point at a distance a from MLA is projected back into object space at a distance a according to the thin-lens equation through the main lens, such as

a = (D -a) • F (D -a) -F . ( 1 2 )
Let r 0 be the minimal acceptable radius of the CoC. The smallest diffraction-limited spot resolved by a lens in wave optics, i.e., the radius of the first null of the Airy disc, is r * = 1.22•ν • N * , where ν is the considered light wavelength, and N * = d/A is the working f -number of the lens. The minimal acceptable radius is the maximum between this limit and half the size of a pixel, such as r 0 = max (r * , s/2). For a a micro-lens of type (i), the focus plane distance is given by

a (i) 0 = 1 f (i) - 1 d -1 = d f (i) d -f (i) . ( 13
)
Let A be the micro-lens aperture, we derive then the far a + and near a -focus planes distances:

⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ a (i) + = d A • a (i) 0 A f (i) -2r 0 a (i) 0 -f (i)
[far]

a (i) -= d A • a (i) 0 A f (i) + 2r 0 a (i) 0 -f (i) [near]. ( 14
)
The DoF of a micro-lens of type (i) is computed as the distance between the near and far focus planes, such as

DOF (i) = a (i) + -a (i) -= A f (i) • a (i) 0 • 2r 0 a (i) 0 -f (i) A f (i) 2 -4r 2 0 a (i) 0 -f (i)
2 .

(15)

Note that to fully exploit the combined extended DoF without gaps, the micro-lenses DoF should either just touch or slightly overlap [START_REF] Perwaß | SPIE human vision and electronic imaging XVII. Single Lens 3D-Camera with Extended Depth-of[END_REF]. Finally, under this consideration, the total DoF of the plenoptic camera in MLA space is computed using the micro-lenses DoF as

DOF = max i a (i) + -min i a (i) - . ( 16
)
We can finally plot the blur profile of the camera, along with the focal planes and the total DoF as illustrated by the Fig. 10.

Pre-calibration Using Raw White Images

The goal of the pre-calibration step is to provide a strong initial estimate of the camera parameters. Inspired from depth from defocus theory [START_REF] Subbarao | Depth from defocus: a spatial domain approach[END_REF], we leverage blur information to estimate our blur radius by varying the main lens aperture and using the different micro-lenses focal lengths, in combination with parameters from the image space. This is achieved by using raw white images acquired with a light diffuser mounted on the main objective, and taken at different apertures. We then show how the blur radii are linked to camera parameters, thus enabling their initialization.

Micro-Images Array Calibration

First, the micro-images array (MIA) is calibrated using raw white images. We compute the micro-image centers c k,l by the intensity centroid method with sub-pixel accuracy [START_REF] Thomason | Calibration of a microlens array for a plenoptic camera[END_REF][START_REF] Noury | Light-field camera calibration from raw images DICTA[END_REF]Suligaet al. 2018).

The distance between two micro-image centers δ i is then computed as the optimized edge-length of a fitted 2D regular grid mesh. The optimization is conducted by non-linear minimization of the distances between the grid vertices and the corresponding detected MICs. The pixel translation offset in image coordinates, (τ x , τ y ), and the rotation around the (-z)axis, ϑ z , are also determined during the optimization process.

Deriving the Micro-Image Radius

In white images taken with a light diffuser and a controlled aperture, each type of micro-lens produces a micro-image (MI) with a specific size and intensity. This provides a mean to distinguish between them (Fig. 5). The process of capturing a white image is equivalent for the micro-lenses to 123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Fig. 4

Formation of a micro-image with its radius R through a microlens while taking a white image using a light diffuser, at an aperture A, in Keplerian internal configuration. The point V is the vertex of the cone passing by the main lens and the considered micro-lens. V is the image of V by the micro-lens and R is the radius of its blur circle.

imaging a white uniform object of diameter A at a distance D. The imaging process is schematized in Fig. 4. Using optics geometry, the image of this object, i.e., the resulting MI, corresponds to the image of an imaginary point V constructed as the vertex of the cone passing through the main lens and the considered micro-lens. Let a be the signed distance of this point from the MLA plane, expressed from similar triangles and eq. 4 as

a = -D Δ μ A -Δ μ = -D A d + D D • 1 Δ i -1 -1 , ( 17
)
with A being the main lens aperture. Note the minus sign is added because the vertex is always formed behind the MLA plane, and thus considered as a virtual object for the microlenses. Geometrically, the MI formed is the blur circle of this imaginary point V . Therefore, injecting the latter expression in eq. 6, the metric MI radius R is given by

R = Δ μ 2 d 1 f - 1 a - 1 d = Δ i • D d + D • d 2 • 1 f + A d + D D • 1 Δ i -1 1 D - 1 d = A • d 2D + Δ i • D d + D • d 2 • 1 f - 1 D - 1 d . ( 18
)
From the above equation, the MI radius R depends linearly on the aperture of the main lens. However, the main lens aperture cannot be measured directly whereas we have access to the fnumber value. Recall that the f -number of an optical system is the ratio of the system's focal length F to the aperture, A, given by N = F/A. Finally, we can express the MI radius for each micro-lens focal length type i as

R i N -1 = m • N -1 + q i (19) with m = d F 2D and q i = 1 f (i) • Δ i • D d + D • d 2 - Δ i 2 . ( 20
)
We thus relate the MI radius to the plenoptic camera parameters. It is a function of fixed parameters (d, D, F), measured parameters (Δ i = s • δ i ) and variable parameters (N and f (i) with i ∈ [1 . . I ]).

Let Ω be the set of parameters m, q 1 , . . . , q I , where q i is the value obtained by

q i = 1 f (i) • Δ i • D d + D • d 2 = q i + Δ i 2 . ( 21
)
They are used to compute the radius part of the BAP feature and to initialize the camera parameters.

Micro-image radii estimation: From raw white images, we measure each MI radius = |R| /s in pixel based on image moments fitting. We use the second order central moments of the micro-image to construct a covariance matrix. The radius is proportional to the computed standard deviation σ . Recall that raw moments and centroid are given by M i j =

x,y x i y j I (x, y) and { x, ȳ} = M 10 M 00 , M 01 M 00 , and the central moments by

μ pq = x,y (x -x) p (y -ȳ) q I (x, y) . ( 2 2 )
The covariance matrix is then computed as

cov [I (x, y)] = 1 μ 00 μ 20 μ 11 μ 11 μ 02 = σ x x σ xy σ yx σ yy . ( 23
)
We define σ as the square root of the greatest eigenvalue of the covariance matrix, i.e.,

σ 2 = σ x x + σ yy 2 + 4σ 2 xy + σ x x -σ yy 2 2 . ( 2 4 )
The estimation is robust to noise, works under asymmetrical distribution and is easy to use, but requires a parameter α to convert the standard deviation σ into a pixel radius = α • σ . The parameter α is determined so that at least 98% of the distribution is taken into account. According to the standard normal distribution Z -score [2.33, 2.37]. In our experiments, we set α = 2.357 as it best fits our measurements.

Recall that the pixel MI radius is given by = |R| /s. The metric radius is either positive if formed after the rays inversion, as in Fig. 4, or negative if before, and thus depends on the internal configuration such as

R = • s [Keplerian internal configuration], -• s [Galilean internal configuration]. ( 25
)
Coefficients estimation: Given several raw white images taken at different apertures, we estimate the parameters Ω, i.e., the coefficients of eq. 19, for each type of micro-image. Note that the standard full-stop f -number conventionally indicated on the lens differs from the real f -number. We use then the f -number calculated from the aperture value AV by N = √ 2 AV . The coefficient m is a function of fixed physical parameters independent of the micro-lens focal lengths and the main lens aperture. Therefore, we obtain a set of linear equations, sharing the same slope, but with different y-intercepts. With X = m q 1 . . . q I , the set of equations can be linearly rewritten as

AX = B, and then X

= A A -1

A B

where the matrix A, containing the f -numbers and a selector of the corresponding y-intercept coefficient, and the vector B, containing the radii measurements, are constructed by arranging the terms given the focal length at which they have been calculated. Finally, we compute X with a least-square estimation. Figure 5 shows an example of radii distributions from our experiments computed from white images taken at several f -numbers, and the estimated linear functions. In practice, at least two aperture configurations are required. More can be used to improve the estimation but at the condi-tion that radii measurement distributions are distinguishable from each others, with small overlap.

Camera Parameters Initialization

First, the pixel size s is set according to the manufacturer values. The main lens focal length F is also initialized from them. Given the parameters Ω and the focus distance h, the parameters d and D are initialized as

d ←- 2m H F + ξ • 4m and D ←-H -ξ • 2d, (26)
with ξ = 1 (resp., ξ = -1) in Galilean (resp., Keplerian) internal configuration, and where H is given by Eq. ( 17) of [START_REF] Perwaß | SPIE human vision and electronic imaging XVII. Single Lens 3D-Camera with Extended Depth-of[END_REF],

H = h 2 1 -1 -4 F h . ( 27
)
For completeness, note that the unfocused configuration can be initialized with d ← 2m and D ← F. In a second step, all distortions coefficients are set to zero. The principal point is set as the center of the image. The sensor plane is thus set parallel to the main lens plane, with no rotation, at a distance -(D + d). Seemingly, the MLA plane is initially set parallel to the main lens plane at a distance -D. From the pre-computed MIA parameters, the MLA translation takes into account the (x, y)-offsets -sτ x , -sτ y and the rotation around the z-axis is initialized with -ϑ z . The micro-lenses pitch Δ μ is set according to eq. 4, where the ratio λ is computed using eq. 26 such as

λ ←- F F + 2m . ( 28
)
123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Finally, the initial micro-lenses' focal lengths are also computed from the parameters Ω as follows

f (i) ←- d 2 • q i • Δ μ . ( 29
)
Experiments will show that the initial model is close to the optimized model.

BAP Features Detection in Raw Images

At this point, the MIA is calibrated and micro-images centers are extracted. The raw images are devignetted by dividing them by a white raw image taken with the same aperture. We based our method on a checkerboard calibration pattern. The detection process is divided into two steps: 1) checkerboard images are processed to extract corners at position (u, v); and 2) with the set of parameters Ω and the associated virtual depth estimate for each corner, the corresponding BAP feature is computed in image space.

Computing Blur Radius Through Micro-Lens

To respect the f -number matching principle [START_REF] Perwaß | SPIE human vision and electronic imaging XVII. Single Lens 3D-Camera with Extended Depth-of[END_REF], we configure the main lens f -number such that the micro-images fully tile the sensor without overlap. In this configuration the working f -number of the main imaging system and the micro-lens imaging system should match. We consider the general case of measuring an object p at a distance a from the main lens. First, p is projected through the main lens according to the thin lens equation, 1/F = 1/a + 1/b, resulting in a point p at a distance b behind the main lens, i.e., at a distance a = D -b from the MLA. From eq. 6, the metric radius of the blur circle r of a point p at distance a through a micro-lens of type (i) is expressed as

r = Δ i D d + D • d 2 • 1 f (i) - 1 a - 1 d = Δ i • D d + D • d 2 • 1 f (i) =q i [eq.21] - Δ i • D d + D • d 2 • 1 d =Δ μ /2[eq.4] - Δ i • D d + D =Δ μ [eq.4] • d 2 • 1 a = -Δ μ • d 2 • 1 a + q i - Δ μ 2 . ( 3 0 )
In practice, a and d cannot be measured in raw image space, but the virtual depth can, as it will be shown in the next subsection. Virtual depth refers to relative depth value obtained from disparity. It is defined as the ratio between the signed object distance a and the sensor distance d:

υ = - a d . ( 31
)
The sign convention is reversed for virtual depth computation. Distances are negative in front of the MLA plane. If we re-inject the virtual depth in eq. 30, taking caution of the sign, and using eq. 4, we can derive the radius of the blur circle of a point p at a distance a from the MLA by

r = λΔ i 2 • υ -1 + q i - λΔ i 2 . ( 3 2 )
This equation allows to express the pixel radius of the blur circle ρ = r /s associated to each point having a virtual depth without explicitly evaluating the physical parameters A, D, d, F and f (i) of the camera, directly in image space.

Features Extraction

First, we detect corners in raw images using the detector introduced by Noury et al. ( 2017) with sub-pixel accuracy in each micro-image. With a plenoptic camera, contrarily to a classic camera, a same point in object space is projected into multiple observations onto the sensor. The checkerboard is designed and positioned so that the sets of observations are sufficiently far from each others to be clustered. We use the DBSCAN algorithm [START_REF] Ester | A density-based algorithm for discovering clusters in large spatial databases with noise[END_REF] to identify the clusters. We then associate each point with its cluster of observations. Secondly, once each cluster is identified, we compute the virtual depth υ from the disparity. Let ΔC 1-2 be the distance between the centers of the micro-lenses C 1 and C 2 , i.e., the baseline. Let Δp = | p 1p 2 | be the Euclidean distance between images of the same point in corresponding microimages. The virtual depth υ is calculated with the intercept theorem:

υ = ΔC 1-2 ΔC 1-2 -Δp = η • Δ μ η • Δ μ -Δp = η • λΔ i η • λΔ i -Δp . ( 33
)
If we consider two adjacent micro-lenses, the baseline ΔC 1-2 is just the diameter of a micro-lens, i.e., Δ μ = λΔ i and η = 1. For further apart micro-lenses the baseline is a multiple of that diameter, where η is not necessarily an integer. To handle noise in corner detection, we use a median estimator to compute the virtual depth of the cluster, taking into account all combinations of point pairs in the disparity estimation. Finally, we compute the BAP features from eq. 32, using the set of parameters Ω and the available virtual depth υ. In each frame n, for each micro-image (k, l) of type (i) containing a corner at position (u, v) in the image, the feature p n k,l is given by

p n k,l = u v ρ 1 , with ρ = r /s. ( 3 4 )
123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

In the end, our observations are composed of a set of microimages centers c k,l and a set of BAP features p n k,l allowing us to introduce two reprojection error functions corresponding to each set of features as explains in the next section.

Camera and Relative Blur Calibration

To retrieve the parameters of our camera model (eq. 10), we use a calibration process based on non-linear minimization of reprojection errors. The camera calibration process is divided into three phases: 1) the initial intrinsics are provided by the pre-calibration step; 2) the initial extrinsics are estimated from the raw checkerboard images; and 3) the parameters are refined with a non-linear optimization leveraging our new BAP features. In parallel, using our BAP features, the blur proportionality coefficient of eq. 9 is calibrated, by minimizing the relative blur in a new reprojection error with a non-linear optimization.

Camera Model Initialization

Iterative optimization of non-linear cost functions are sensitive to initial parameters setting. To ensure convergence and to avoid falling into local minima during the process, the parameters must be carefully initialized close to the solution.

Our pre-calibration step provides a strong initial solution for the optimization. Intrinsic parameters are initialized as explained in Sect. 3.3 using only raw white images.

The camera poses T n c , i.e., the extrinsic parameters, are initialized using the same method as by [START_REF] Noury | Light-field camera calibration from raw images DICTA[END_REF]). For each cluster of observations, the barycenter is computed, as illustrated by Fig. 6. Those barycenters can been seen as the projections of the checkerboard corners through the main lens using a standard pinhole model. For each frame, the pose is then estimated using the PnP algorithm [START_REF] Kneip | A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orientation[END_REF], like in classic pinhole imaging system. To associate 3D-2D correspondences, we reproject checkerboard corners based on the estimated pose in image space and link them to their nearest cluster of observations.

Optimizing the Camera Parameters

By introducing blur in our model, we can optimize all parameters within one single optimization process. We propose a new cost function Θ taking into account the blur information of our new BAP feature. The cost is composed of two main terms both expressing errors in the image space: 1) the blur aware plenoptic reprojection error and 2) the main lens center reprojection error.

In the first term, for each frame n, each checkerboard corner p n w is reprojected into the image space through each micro-lens (k, l) of type (i) according to the projection model of eq. 10 and compared to its observations p n k,l . In the second term, the main lens center O is reprojected according to a pinhole model in the image space through each micro-lens (k, l) and compared to its detected micro-image center c k,l . Let S = , T n c be the set of intrinsic and extrinsic T n c parameters to be optimized. The cost function Θ(S) is expressed as

Θ(S) = p n k,l -k,l p n w 2 + c k,l -k,l (O) 2 . ( 35
)
The optimization is conducted using the Levenberg-Marquardt algorithm.

Relative Blur Calibration Using BAP Features

Relative blur estimation has been studied by [START_REF] Ens | An investigation of methods for determining depth from focus[END_REF]; Mannan and Langer (2016). Up to our knowledge, it has never been studied in context of plenoptic camera. As a new contribution, we leverage the relative blur between different micro-images and our BAP features to calibrate the blur proportionality coefficient κ of eq. 9.

Relative blur model: A point imaged by two different micro-lenses of type (i) and ( j) will have different blur amount, i.e., the resulting images will have different spread parameters for the PSF model, such as

I (i) (x, y) = h (i) * I * (x, y) I ( j) (x, y) = h ( j) * I * (x, y) , ( 36
)
where I * (x, y) is the latent in-focus image. We approximate the PSF with a 2D Gaussian as in eq. 8, where the diameter of the blur kernel h (i) is σ (i) . To compare two views with different amount of blur, we use the relative blur model in spatial domain [START_REF] Pentland | A new sense for depth of field PAMI-9[END_REF][START_REF] Subbarao | Parallel depth recovery by changing[END_REF][START_REF] Subbarao | Depth from defocus: a spatial domain approach[END_REF][START_REF] Ens | An investigation of methods for determining depth from focus[END_REF]. As stated by Mannan and Langer (2016), the Gaussian relative blur approximation works well mainly for small relative blurs (up to ρ ≈ 5 pixels) and when the aperture has a simple shape, which is the case with the plenoptic camera. We then use the equally-

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

defocused representation by applying additional blur to the relatively in-focus micro-image, hence,

I (i) (x, y) h r * I ( j) (x, y) if σ (i) ≥ σ ( j) h r * I (i) (x, y) I ( j) (x, y) if σ (i) ≤ σ ( j) . ( 37
)
Note that h r is the relative blur kernel applied to either one of the views such that both views are equally-defocused. The diameter of the relative blur kernel h r is approximated as

σ r (i, j) |σ 2 (i) -σ 2 ( j) |. ( 38
)
This approximation is exact when the PSF is a Gaussian. Since the radius of the relative blur kernel σ r cannot indicate whether the (i) or the ( j) view is more in-focus than the other, we define the relative blur similarly to Chen et al. ( 2015), as

Δσ 2 (i, j) σ 2 (i) -σ 2 ( j) , ( 39
)
where Δσ 2 (i, j) > 0 indicates that a pixel in the ( j)-microimage is more in-focus than its corresponding pixel in the (i)micro-image. Symmetrically, Δσ 2 (i, j) < 0 indicates that the (i)-micro-image is more in-focus. In a similar fashion, we define the relative blur radius as

ρ r (i, j) Δρ 2 (i, j) = |ρ 2 (i) -ρ 2 ( j) | (40)
with σ r = κ • ρ r , and where ρ (i) , ρ ( j) are the blur radii of the BAP features through a micro-lens of type (i) and ( j).

Blur proportionality coefficient calibration:

To calibrate κ, we use our BAP features and the relative blur model applied on micro-images of different types. BAP features { p i } from a same cluster C represent the same point in object space p w . We extract two windows W around the BAP features p i , p j ∈ C ( p w ) of different types, and express them using the equally-defocused representation (eq. 37). As the relative blur radius does not exceed 2.5pix, windows W of size 9×9 are extracted at (u, v) with sub-pixel precision, and represent therefore the same part of the scene in both microimages. Additional blur is applied using a Gaussian kernel of spread parameter σ r . The spread parameter is computed from the ρ part of the BAP features and the parameter κ to be optimized, with initial value κ = 1. Let Θ(κ) be the cost function to be minimized. It is expressed as

Θ(κ) = n p n i , p n j ∈C( p n w ) W p n j -h r * W p n i 2 2 , ( 41
)
given ρ (i) < ρ ( j) and where h r is the PSF with spread

parameter σ r = κ • |ρ 2 (i) -ρ 2 ( j) |.
The optimization is conducted using the Levenberg-Marquardt algorithm.

Experimental Setup

To validate our camera model, we evaluate our method on real-world data obtained with a multi-focus plenoptic camera in a controlled environment. Our experimental setup is illustrated in Fig. 1. The camera is mounted on a linear motion table with micro-metric precision. The target plane is orthogonal to the translation axis, and the camera optical axis is aligned with this axis. The approximate absolute distances at which the images have been taken with the corresponding step lengths are reported in Table 1.

Hardware Environment

For our experiments we used a Raytrix R12 color 3Dlight-field-camera, with a MLA of F/2.4 aperture. The camera is in Galilean internal configuration. We used two different mounted lens, a Nikon AF Nikkor F/1.8D with a 50 mm focal length for comparison with state-of-the-art, and a Nikon AF DC-Nikkor F/2D with a 135 mm focal length to validate the generalization of our model. The MLA organization is hexagonal row-aligned, and composed of 176 × 152 (width × height) micro-lenses with I = 3 different types. The sensor is a Basler beA4000-62KC with a pixel size of s = 0.0055 mm. The raw image resolution is 4080 × 3068 pixel. We calibrate our camera for four focus distance configurations, with h ∈ {450, 1000, ∞} mm for the 50 mm lens, and with h = 1500 mm for the 135 mm lens. Note that when changing the focus setting, the main lens moves with respect to the block MLA-sensor.

Software Environment

All images have been acquired using the MultiCamStudio free software (v6.15.1.3573) of the Euresys company. We set the shutter speed to 5 ms. While taking white images for the pre-calibration step, we set the gain to its maximum value. For Raytrix data, we use their proprietary software RxLive (v4.0.50.2) to calibrate the camera, and compute the depth maps used in the evaluation. Our source code has 123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

been made publicly available: https://github.com/comseeresearch/libpleno, and https://github.com/comsee-research/ compote.

Datasets

We build four datasets with different focus distance h: for the 50 mm lens, R12-A for h = 450 mm, R12-B for h = 1000 mm, and R12-C for h = ∞; for the 135 mm lens, R12-D for h = 1500 mm. Each dataset is composed of:

-white raw plenoptic images acquired at different apertures (N ∈ {4, 5.66, 8, 11.31, 16}) using a light diffuser mounted on the main objective for pre-calibration, -free-hand calibration target images acquired at various poses (in distance and orientation), separated into two subsets, one for the calibration process (16 images) and the other for reprojection error evaluation (15 images), -a white raw plenoptic image acquired in the same luminosity condition and with the same aperture as in the calibration targets acquisition for devignetting, -and, calibration targets acquired with a controlled translation motion for quantitative evaluation, along with the depth maps computed by the RxLive software.

Examples of calibration targets acquired for the R12-B dataset are given in Fig. 7 along with their 3D poses. A summary for each dataset is given in Table 1, indicating checkerboard information and the distances at which the targets have been acquired for calibration and for the controlled evaluation. Our datasets have been made publicly available, and can be downloaded from our public repository at https:// github.com/comsee-research/plenoptic-datasets.

Simulation Environment

In order to validate our model on Lytro-like plenoptic camera configuration, i.e., unfocused plenoptic camera (UPC), we propose to evaluate our model in a simulation environment. We built our own simulator based on raytracing to generate images. Similar to the real-world dataset, we generated a dataset, named UPC-S, composed of several white images taken at different apertures (with N ∈ {2, 4, 5.6}), various checkerboard poses for calibration and validation, and for evaluation, checkerboard images with known translation along the z-axis. Details are also given in Table 1. We used the Lytro Illum intrinsic parameters reported in Table 4 of [START_REF] Bok | Geometric calibration of micro-lens-based light field cameras using line features 39[END_REF] as baseline for the simulation. They have been converted into our parameters and reported in Table 3. The MLA organization is hexagonal row-aligned, and composed of 541 × 434 (width × height) micro-lenses of the same type (I = 1). The raw image resolution is 7728 × 5368 pixel, with a pixel size of s = 0.0014 mm and with micro-image of radius 7.172 pixel.

Results and Discussions

Our evaluation process follows the steps given in the overview (Fig. 2). First, we present the pre-calibration results, where white raw plenoptic images are used for computing micro-image centers, and for estimating initial camera parameters. Second, from the set of devignetted calibration target images, BAP features are extracted, and camera intrinsic and extrinsic parameters are then computed using our non-linear optimization process. In parallel, the same BAP features are also used to calibrate the relative blur proportionality coefficient. Third, we evaluate our model quantitatively, firstly, using the reprojection error as a metric, and secondly, using the relative translation error in a controlled environment. Then, we propose an ablation study of the camera parameters. Finally, we illustrate how to characterize the plenoptic camera extended DoF using the blur profile.

Pre-calibration

To estimate the parameters Ω, we set α = 2.357, and since the camera is in Galilean internal configuration, we use R = -• s, following eq. 25. Fig. 5 shows the microimage radii as function of the inverse f -number with the estimated lines for dataset R12-B. Their distributions are represented by the violin-boxes. For N = 5.66, we can see that radii distributions overlap, and that radii values are slightly overestimated as they do not fit exactly the borders of the micro-images. In practice, we only use white images that present distinguishable radii distributions in the estimation process, usually corresponding to small apertures. In case of R12-B, only white images at N = 11.31 and N = 8 are used. The corresponding coefficients for all datasets are summarized in Table 2. As expected, the parameter m is different for each dataset, since D and Δ i vary with the focus distance h, whereas the q i values are close for all datasets, even for different camera setup (R12-D).

Free-Hand Camera Calibration

Comparison with state-of-the-art: Since our model is close to the one of [START_REF] Noury | Light-field camera calibration from raw images DICTA[END_REF], we compare our intrinsics with the ones obtained under their pinhole assumption using only corner reprojection error and with the same initial parameters. In addition, we evaluate against the method of Nousias et al. ( 2017), which provides a set of intrinsics and extrinsics for each micro-lens type. The equivalence of our parameters and their parameters is given by

F = ( f x + f y ) 2 • s, D = -F • K 1 K 2 • F + 1 -1 , d (i) = D - K 2 D D + K 2 , u 0 = c x and v 0 = c y , ( 42
)
where K 1 and K 2 are the two additional intrinsic parameters that account for the MLA setting in their model. The equivalence also stands for the parameters of [START_REF] Bok | Geometric calibration of micro-lens-based light field cameras using line features 39[END_REF]. The provided detector from [START_REF] Nousias | Corner-based geometric calibration of multi-focus plenoptic cameras[END_REF] was not able to detect corner observations on our datasets. Therefore, we used the same observations for our method (noted BAP in Table 3), Noury et al. ( 2017) method (NOUR), and Nousias et al. ( 2017) method for each type (NOUS1, NOUS2, and NOUS3), which allowed us to focus the comparison on the camera model only. Finally, we provide the calibration parameters obtained from the RxLive software (RTRX) corresponding to the model of [START_REF] Heinze | Automated Robust metric calibration algorithm for multifocus plenoptic cameras[END_REF], and compare our depth measurements to their depth maps.

Initialization: We initialize λ from eq. 28. Its value for each dataset is reported in Table 2. The difference between the initial value of λ and its value computed from optimized camera parameter is less than 0.024%, which validates the use of the initial value from eq. 28 when computing our BAP features. The initial camera parameters reported in Table 3 are computed using the methodology presented in Sect. 3.3. They are used for the BAP and NOUR methods. The camera internal configuration is set to Galilean. When h decreases, D increases. Yet when the main lens focus distance is at infinity, the main lens should focus on the plane υ = 2, which implies that D tends to F -2d as lower bound, as H tends to F. In most cases (here, for R12-A,B,D), we will still have F < D, which usually can describe the camera in Keplerian configuration. In Keplerian internal configuration, the condition F < D stands regardless of the focus distance, as D lower bound is F + 2d.

When using the linear initialization from NOUS, the initial parameters of some configurations corresponded to impossible physical setup or were too far from the solution, hindering the convergence of the optimization. Therefore, in order to continue comparison, we manually set the initial parameters close enough to a solution. In contrast, we can see that the optimized parameters for BAP and NOUR are close to initial values, which shows that our pre-calibration step provides a strong initial solution for the optimization process.

Intrinsic camera parameters: Optimized intrinsic parameters are reported for each dataset and for all the evaluated 123 Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Table 3

Initial intrinsic parameters for each dataset along with the optimized parameters obtained by our method (BAP) and with the methods of [START_REF] Noury | Light-field camera calibration from raw images DICTA[END_REF] (NOUR), of [START_REF] Nousias | Corner-based geometric calibration of multi-focus plenoptic cameras[END_REF] for each micro-lens type (NOUS1, NOUS2, NOUS3) and the parameters obtained from RxLive software (RTRX) methods in Table 3. First, BAP, NOUR and NOUS all verify the condition F ≈ D + 2d when the focus is set at infinity (R12-C). Second, the focal lengths obtained from NOUR, NOUS and RTRX change significantly given the focus distance, and the ones obtained from NOUS even vary according to the micro-lens types. In contrast, only BAP shows stable parameters across all three R12-A,B,C datasets. Shared parameters across datasets (i.e., the focal lengths and the distance between the MLA and the sensor) are close enough to indicate that our model successfully generalizes to different focus configurations. Furthermore, the parameters obtained by our method with an other main lens, i.e., R12-D, are coherent with the previously obtained parameters, stressing out that our model can be applied to a different camera setting. Finally, our method is the only one providing the micro-lenses focal lengths in a single unified model. The other methods calibrate either several MLA-sensor distances (RTRX), or several models, one for each type (NOUS).

R12-A (F = 50mm, h = 450mm) R12-B (F = 50mm, h = 1000mm) Init.
Note that distortion coefficients and MLA rotations are close to zero. The influence of these parameters will be analyzed in the proposed ablation study of the camera model in Sect. 7.4.

On simulated data: First, pre-calibration has been performed using the white raw images. The resulting parameters Ω are coherent with the simulation parameters. With parameters m = -23.639μm and q = -0.146μm, we have d ≈ f , which describes the unfocused configuration. Reference and initial intrinsic parameters are reported in Table 3, along with the optimized parameters. Second, calibration has been performed. The obtained intrinsic parameters are close enough to the references parameters, indicating that our method is able to generalize to the unfocused plenoptic camera.

For completeness, we also quantitatively evaluated the optimized parameters, by estimating the relative displacement between checkerboard with known motion along the z-axis. It results a translation error ε z = 1.64%, which validates the model.

Quantitative Evaluations of the Camera Model

Reprojection error: In the absence of ground truth, we first evaluated the intrinsic parameters by estimating the reprojection error using the previously computed intrinsics. We consider only free-hand calibration target images which are not used in the calibration process. We use the RMSE as a metric to evaluate the reprojection error on the corner part of the features, for each dataset. For the BAP method, the corner reprojection part is reported in Table 4, as well as the radius reprojection part within parentheses. Regarding the NOUS methods, the original error is expressed using the mean reprojection error (MRE). We converted the final error to the RMSE metric for comparison. Note that the latter method operates separately on each type of micro-lens, meaning that the number of features is not the same as with NOUR and BAP. First, the reprojection error is less than 1 pixel for all methods, for each dataset, demonstrating that the computed intrinsics lead to an accurate reprojection model and can be generalized to images which are not from the calibration set. Second, even though the NOUS method provides the lowest RMSE, it shows a significant discrepancy according to the considered type. The error obtained by our method is sightly higher than the error from NOUR, but this can be explained by the fact that our optimization does not aim at minimizing only the corner reprojection error, but the radius reprojection as well. Note that the positional error ε u,v predominates in the total cost by two orders of magnitude compared to the blur radius error ε ρ , but the latter still helps to constrain our model as shown by the relatively close intrinsics between the datasets.

Controlled environment poses evaluation: With our experimental setup, we acquired several images with known relative translation between each frame. We compare the estimated displacements along the z-axis from the extrinsic parameters to the ground truth. The extrinsics are computed with the models estimated from the free-hand calibration. In the case of the RTRX method, we use the filtered depth maps obtained with the proprietary software RxLive to estimate the displacements.

The translation errors along the z-axis with respect to the ground truth displacement from the closest frame are reported in Fig. 8 for datasets R12-A (a), R12-B (b) and R12-C (c). The relative error ε z for a known displacement δ z is computed as the mean absolute relative difference between the estimated displacement δz and the ground truth, for each pair of frames T i , T j separated by a distance δ z , i.e.,

ε z (δ z ) = η -1 (Ti ,T j )|zi -z j =δ z δ z -δz /δ z , ( 43
)
where δz = ẑi -ẑ j , and η is a normalization constant corresponding to the number of frames pair.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved. The error ε z is expressed in percentage of the estimated distances, and truncated to 7% to ease the readability and the comparison. The mean error with its confidence interval across all datasets for our method (BAP), [START_REF] Noury | Light-field camera calibration from raw images DICTA[END_REF] )method (NOUR), [START_REF] Nousias | Corner-based geometric calibration of multi-focus plenoptic cameras[END_REF]) method for each type (NOUS1, NOUS2, NOUS3), and for the proprietary software RxLive (RTRX) are reported in (d). Please refer to the color version for better visualization.

The mean error with its standard deviation across all datasets for BAP, NOUR, NOUS, and RTRX are reported in (d).

Firstly, the mean error across R12-A,B,C datasets are of the same order for the evaluated methods around 3%: for BAP, ε z = 2.92 ± 0.73%; for NOUR, ε z = 3.50 ± 3.08%; for NOUS1, ε z = 1.68±1.53%; for NOUS2, ε z = 3.40±2.19%; for NOUS3, ε z = 3.30 ± 3.35%; and, for RTRX, ε z = 4.96 ± 4.44%. This is also the case for the dataset R12-D where our model has a mean translation error of ε z = 3.37%. Note that all evaluated methods outperform RTRX as the depth maps computation might not be as precise as the optimization of extrinsic parameters. Our method ranks second in terms of relative mean error. Even though lowest error is obtained by the method NOUS for type (1), it presents a large standard deviation and the errors for the other two types are significantly higher. In real application context, there is no way to know in advance which type will produce the smallest error. [START_REF] Nousias | Corner-based geometric calibration of multi-focus plenoptic cameras[END_REF] suggest that when extrinsics are sufficiently close, we can use representative extrinsics that are calculated by averaging the extrinsics from the individual types. Our results do not match this observation as the estimated extrinsics are significantly different for each type. As shown, only the first type gives satisfactory results whereas the other two present a larger error with a significant standard deviation. Averaging the extrinsics from all types will therefore minimize the difference between poses but will not provide the best possible estimation.

Secondly, the standard deviation can be seen as an indicator of the estimation precision across the datasets, and thus indicates whether the model can generalize to several configurations or not. Our model presents the lowest standard deviation as illustrated in Fig. 8 (d). This indicates a low discrepancy between datasets and thus that the model is precise and consistent for all configurations.

Thirdly, we analyze the behavior of each method for each dataset across different distances. None of the methods suffered from a constant bias, as we do not observe a decreasing relative error as the distance increases. BAP and NOUR present a stable relative error for all distances, i.e., with approximately 0.3 % of standard deviation. This indicates that the estimation suffered only from a scale error. One could thus re-scale the poses to provide a precise and accurate estimation. We cannot draw any conclusion for the other methods since the variations do not follow any obvious pattern.

Finally, our model differs from the model of [START_REF] Noury | Light-field camera calibration from raw images DICTA[END_REF] ing those two models, the mean error as well as the standard deviation is smaller with our method. The inclusion of the micro-lens focal lengths in the camera model improves the estimation precision and accuracy, and enables to generalize to several configurations. Dealing with different intrinsics which produce different extrinsics is not satisfactory when using the multi-focus plenoptic camera. In contrast, our model is able to manage all micro-lens types simultaneously, and proves to be stable across various configurations and working distances.

Ablation Study of Camera Parameters

To evaluate the influence of each parameter of the camera model, we present an ablation study of some of them. We focus the analysis on distortion coefficients (Q 1 , Q 2 , Q 3 , P 1 , and P 2 ), on some degrees of freedom of the MLA, especially its tilt with respect to the sensor (θ x , θ y ), and the pitch between micro-lenses (Δ μ ). All combinations of the parameters have been tested, resulting in eight configurations. For each configuration and on each dataset of R12-A,B,C: first, we calibrate the camera intrinsic parameters; second, we evaluate the model using the RMSE of the reprojection error; and finally, we quantitatively estimate the relative translation error on the evaluation dataset. Each configuration has been initialized with the same intrinsic parameters, and used the same observations for all processes. Results are reported in Table 5. The first column is the configuration number.

The Tilt column indicates if we keep ( ) or remove (×) the parameters θ x and θ y . The Pitch column stands for the parameter Δ μ , and the column Dist for the distortion parameters Q 1 , Q 2 , Q 3 , P 1 , and P 2 . The reprojection error ε all is given by its RMSE, and the relative translation error ε z is expressed in percent with respect to the ground truth displacement.

The configuration 1 is our reference, corresponding to the complete model. The optimized parameters are close to the ones from Table 3, i.e., with less than 1% of variation, for all converging configurations and for all datasets.

First, the distortions do not impact the reprojection error of the model. Considering the pairs of configurations (1, 2), (3, 4), and (5, 6), the errors are similar with or without distortions, indicating that our camera does not suffer from lateral distortions. This is due to the relatively large main lens focal length. Nevertheless, distortions may have a role to play in case of shorter focal length.

Second, removing the rotations of the MLA does not improve nor worsen the reprojection error and the pose estimation. When keeping the tilt but freezing the pitch, the model is able to converge. The tilt, in combination with other factors (such as a slight decrease of the main lens focal length), compensates for the error introduced by the approximate value of the pitch. In contrast, configurations 7 and 8 do not converge to a solution, showing that when removing both the tilt and the pitch of the MLA, the model is not constrained enough, and the reprojection error cannot be minimized, resulting in a failure.

Finally, when freezing the pitch to its initial value, the positional part of the reprojection error increases. It is especially the case for dataset R12-A, where the reported errors in Table 5 are the highest of all configurations. This confirms our previous observation that the deviation of the micro-image centers and their optical centers does not satisfy an orthographic projection between the MIA and the MLA. The pitch should be taken into account, on one hand to improve the precision of the model, and on the other hand not to hinder the optimization process.

Relative Blur Calibration

We calibrate the blur proportionality coefficient κ for the three datasets using our BAP features. Figure 9 is shown before, (b), and after, (c), blur addition. The estimated PSF of the relative blur is given in (d).

The optimized blur proportionality coefficients κ are reported in Table 3. Theoretically, the parameter should be the same for all three datasets. Empirically this observation is validated for R12-A and R12-B. Estimated κ for R12-C is lower. This is because the micro-lenses focal lengths in R12-C are slightly shorter than in R12-A and R12-B. Analytically, this difference generates a higher amount of relative blur, and thus a shorter estimate of κ to match the observed blur in image space. In other words, κ compensates for the slight differences in f (i) estimates. Therefore, κ should be calibrated for each dataset.

Profiling the Plenoptic Camera

Using the parameters from our calibration process, we plot the blur profile of the camera, i.e., the evolution of the blur radius with respect to depth for each micro-lens type along with its corresponding DoF. Figure 10 shows the blur profiles obtained for our three focus distance configurations, with their DoFs expressed in mm. The blur radius is expressed in pixel and is given for each type, in red for type (1), in green for type (2) and in blue for type (3). Distances are given in object space in mm with their corresponding virtual depth on a secondary x-axis, spanning from υ = 1 to 15, except for the configuration h = ∞ where we cropped just after the farthest focal plane. In MLA space, the profiles have the same behavior for all focus distances, as it only depends on the MLA parameters.

First, the horizontal dashed line represents the radius of the minimal acceptable e circle of confusion r 0 . In our case, at a wavelength of 750nm, the radius of the smallest diffractionlimited spot is r * = 2.4μm which is less than half the pixel size. We then choose r 0 = s/2. Despite not illustrated in the figure, the blur radius grows exponentially when getting closer to the plane υ = 0. Once this limit is exceeded, the blur decreases and converges to a constant value of approximately 6 pixel. This happens for more distant objects when points are projected in front of MLA implying a negative virtual depth. This is the case for h = 450 and h = 1000 mm, but not for h = ∞, as the points were never projected closer than υ = 2. In the working distance range, the blur does not exceed 5 pixel and grows when points are closer to the camera.

Secondly, we can use the DoF to select the range of working distances where the blur is not noticeable. The DoF increases in object space as the focus distance increases. As reported on the figures: for R12-A, the DoF is of 14.44 mm; for R12-B of 120 mm; and finally, for R12-C, the total DoF is of 223 m. In MLA space the total DoF is constant and spans from υ = 2.15 to 3.45. As expected, the DoF overlap. In particular, the DoF of the type (3) micro-lens is entirely included in the other two, whereas the DoF of the 123 Content courtesy of Springer Nature, terms of use apply. Rights reserved.

type (1) and (2) just touch. Within the total DoF, a point can then be seen focused in two micro-images of different types simultaneously, which eases the matching problem between views.

Finally, we can easily identify the distance limits at which the point will not be in the DoF anymore nor be projected on multiple micro-images, i.e., corresponding to virtual distances |υ| < 2. At these distances, disparity cannot be computed in image space, and no depth estimation can be performed. Such estimation can also be hindered by the resolution in virtual space compared to the resolution in object space as disparity is inversely proportional to virtual depth. For instance, for close objects, points will be projected on more micro-images but with a low disparity. So the profiles can be used to efficiently characterize the range of distances according to the desired application. Furthermore, once the MLA parameters are available, we can simulate an approximate blur profile for the desired focus distance h with the desired main lens focal length F by updating the value of D using eq. 26 and eq. 27.

Conclusion

To calibrate a plenoptic camera, state-of-the-art methods rely on simplifying hypotheses, on reconstructed data or require separate calibration processes to take into account the multifocus configuration. Taking advantage of blur information we propose: 1) a more complete plenoptic camera model with the introduction of a new BAP feature that explicitly models the defocus blur; this new feature is exploited in our calibration process based on non-linear optimization of reprojection errors; 2) a new relative blur calibration to fill the gap between the physical and geometric blur, which enables us to fully exploit blur in image space; and 3) a way to profile the plenoptic camera and its extended depth of field (DoF).

Our camera model is applicable to the multi-focus plenoptic camera (both in Galilean and Keplerian configuration), as well as to the single-focus and unfocused plenoptic camera. In case of the Raytrix multi-focus camera, our ablation study shows that main lens distortions and MLA tilt can be omitted without hindering the calibration process nor the pose estimation. The study also indicates that explicitly including the pitch of the micro-lenses in the model improves the results. In addition, our calibration methods are validated by quantitative evaluations in controlled environment on real-world data. Our method provides strong initial intrinsics during the pre-calibration step, and coherent optimized camera parameters for all evaluated configurations. It shows a low and stable relative translation error across all the datasets.

In the future, we plan to use blur information in complement to disparity to improve metric depth estimation.

1.

2.

3.

4.

5.

6.

Terms and Conditions

Springer Nature journal content, brought to you courtesy of Springer Nature Customer Service Center GmbH ("Springer Nature"). Springer Nature supports a reasonable amount of sharing of research papers by authors, subscribers and authorised users ("Users"), for smallscale personal, non-commercial use provided that all copyright, trade and service marks and other proprietary notices are maintained. By accessing, sharing, receiving or otherwise using the Springer Nature journal content you agree to these terms of use ("Terms"). For these purposes, Springer Nature considers academic use (by researchers and students) to be non-commercial. These Terms are supplementary and will apply in addition to any applicable website terms and conditions, a relevant site licence or a personal subscription. These Terms will prevail over any conflict or ambiguity with regards to the relevant terms, a site licence or a personal subscription (to the extent of the conflict or ambiguity only). For Creative Commons-licensed articles, the terms of the Creative Commons license used will apply. We collect and use personal data to provide access to the Springer Nature journal content. We may also use these personal data internally within ResearchGate and Springer Nature and as agreed share it, in an anonymised way, for purposes of tracking, analysis and reporting. We will not otherwise disclose your personal data outside the ResearchGate or the Springer Nature group of companies unless we have your permission as detailed in the Privacy Policy. While Users may use the Springer Nature journal content for small scale, personal non-commercial use, it is important to note that Users may not: use such content for the purpose of providing other users with access on a regular or large scale basis or as a means to circumvent access control; use such content where to do so would be considered a criminal or statutory offence in any jurisdiction, or gives rise to civil liability, or is otherwise unlawful; falsely or misleadingly imply or suggest endorsement, approval , sponsorship, or association unless explicitly agreed to by Springer Nature in writing; use bots or other automated methods to access the content or redirect messages override any security feature or exclusionary protocol; or share the content in order to create substitute for Springer Nature products or services or a systematic database of Springer Nature journal content.

In line with the restriction against commercial use, Springer Nature does not permit the creation of a product or service that creates revenue, royalties, rent or income from our content or its inclusion as part of a paid for service or for other commercial gain. Springer Nature journal content cannot be used for inter-library loans and librarians may not upload Springer Nature journal content on a large scale into their, or any other, institutional repository. These terms of use are reviewed regularly and may be amended at any time. Springer Nature is not obligated to publish any information or content on this website and may remove it or features or functionality at our sole discretion, at any time with or without notice. Springer Nature may revoke this licence to you at any time and remove access to any copies of the Springer Nature journal content which have been saved. To the fullest extent permitted by law, Springer Nature makes no warranties, representations or guarantees to Users, either express or implied with respect to the Springer nature journal content and all parties disclaim and waive any implied warranties or warranties imposed by law, including merchantability or fitness for any particular purpose. Please note that these rights do not automatically extend to content, data or other material published by Springer Nature that may be licensed from third parties. If you would like to use or distribute our Springer Nature journal content to a wider audience or on a regular basis or in any other manner not expressly permitted by these Terms, please contact Springer Nature at onlineservice@springernature.com

  b < D and f < d in Keplerian; and, b > D and f > d in Galilean. The condition b > D can be achieved both when F > D and F < D. The mode of operation is then constrained by the focal length of the micro-lenses, as suggested by Mignard-Debiseet al. (2017).

Fig. 5 a

 5 Fig. 5 a Micro-image radii as function of the inverse f -number (in magenta), with their distributions represented by the violin-boxes, for our camera consisting of I = 3 different types. b Each type of micro-

Fig. 6

 6 Fig. 6 Checkerboard raw image with: a clusters of observations; b their barycenter used as approximation for extrinsics initialization.

Fig. 7

 7 Fig. 7 Example of calibration targets acquired for distances between 775 and 400 mm from the checkerboard used in the dataset R12-B, and their respective poses in 3D.

  Springer Nature, terms of use apply. Rights reserved.

  intrinsic parameters for the simulated Lytro dataset UPC-S along with the optimized parameters obtained by our method (BAP) 123 Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Fig. 8

 8 Fig.8error along z-axis with respect to the ground truth displacement from the closest frame, for datasets R12-A (a), R12-B (b) and R12-C (c). The error ε z is expressed in percentage of the estimated distances, and truncated to 7% to ease the readability and the comparison. The mean error with its confidence interval across all datasets for

  presents two windows extracted around BAP features of different types from the same cluster, showing different amount of blur. The target image to be equally-defocused according to our model 123 Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Fig. 9 a

 9 Fig. 9 a Reference image with highest amount of blur. b Target image to be equally-defocused. c Target image with additional blur. d Estimated PSF.

Fig. 10

 10 Fig. 10 Blur profile, including each micro-lens type, in object space, at different focus distances: a h = 450 mm; b h = 1000 mm; and c h = ∞ mm. Focal planes and depth of fields are illustrated for each type. The blur radius is expressed in pixel as function of the object distance to the camera in mm. Corresponding virtual depth is reported on the secondary x-axis.

Table 1

 1 Summary of R12-A,B,C,D, and UPC-S datasets contents

	h	Target		Calib. dist.	Eval. dist.
		Size	Scale Min Max	Min Max Step
	A 450	9 × 5 10	175 400	265 385	10
	B 1000	8 × 5 20	400 775	450 900	50
	C ∞	6 × 4 30	500 2500	400 1250 50
	D 1500	5 × 3 20	850 1300	750 1200 50
	S hyperf. 9 × 6 26.25 250 800	200 500	50
	All distances are given in mm. Scale refers to checkerboard square size.
	Evaluation distances refer to the linear motion table setup

Table 2

 2 Set of parameters Ω (in [μm]) computed during the precalibration step for each dataset, along with the calibrated relative blur proportionality coefficient.

		R12-A	R12-B	R12-C	R12-D
	Δ i	128.222	128.293	128.333	127.851
	λ	0.99441	0.99358	0.99380	0.99746
	m	-140.596	-159.562	-155.975	-171.288
	q 1	35.135	36.489	35.443	38.599
	q 2	40.268	42.075	41.278	43.129
	q 3	36.822	38.807	37.858	40.788
	κ	0.6988	0.6989	0.6531	0.8824

Table 3

 3

Table 4

 4 Corner reprojection error for each evaluation dataset (i.e., freehand calibration target images not part of the calibration dataset) using the RMSE metric

		BAP	NOUR	NOUS1	NOUS2	NOUS3
	R12-A	0.856 (0.083)	0.713	0.773	0.667	0.958
	R12-B	0.674 (0.183)	0.618	0.538	0.519	0.593
	R12-C	0.738(0.041)	0.713	1.287	0.681	0.411
	Bold values indicate the lowest errors
	For the BAP method, reprojection error of the radius part is indicated
	within parentheses

  by modeling the micro-lens focal lengths. Compar-123 Content courtesy of Springer Nature, terms use apply. Rights reserved.

Table 5

 5 Ablation study of some camera parameters. For each dataset, the reprojection error ε all , computed using the RMSE along with the relative translation error ε z , expressed in %, are reported

		Tilt	Pitch	Dist	R12-A		R12-B		R12-C
					ε all	ε z	ε all	ε z	ε all	ε z
	1				0.860	3.23	0.698	3.15	0.739	2.31
	2			×	0.866	3.34	0.700	3.20	0.737	3.18
	3		×		0.884	3.88	0.755	3.21	0.770	2.00
	4		×	×	0.891	3.98	0.752	3.24	0.773	3.13
	5	×			0.865	3.48	0.784	3.15	0.760	2.89
	6	×		×	0.864	3.58	0.716	3.16	0.749	3.04
	7	×	×		-	-	-	-	-	-
	8	×	×	×	-	-	-	-	-	-
	Bold values indicate the lowest errors
	The symbol (resp., ×) indicates if we keep (resp., remove) the considered parameters

Acknowledgements

We thank Charles-Antoine Noury and Adrien Coly for their insightful discussions and their help during the acquisitions.

Communicated by Zuzana Kukelova. This work was supported by the AURA Region and the European Union (FEDER) through the MMII project of CPER 2015-2020 MMaSyF challenge.

Data Availability Statement

The code source has been made opensource and publicly available in the following repositories: https:// github.com/comsee-research/libpleno, and https://github.com/comseeresearch/compote.

Code Availibility

The datasets analyzed for this study can be found in the following repository:https://github.com/comsee-research/plenopticdatasets. All data supporting the results of this paper will be provided upon request.

Declarations

Conflict of interest The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/.