Georgios D Evangelidis

Dionyssos Kounades-Bastian

Radu Horaud

Emmanouil Z Psarakis

A Generative Model for the Joint Registration of Multiple Point Sets ⋆

Keywords: point set registration, joint registration, expectation maximization, Gaussian mixture model

This paper describes a probabilistic generative model and its associated algorithm to jointly register multiple point sets. The vast majority of state-of-the-art registration techniques select one of the sets as the "model" and perform pairwise alignments between the other sets and this set. The main drawback of this mode of operation is that there is no guarantee that the model-set is free of noise and outliers, which contaminates the estimation of the registration parameters. Unlike previous work, the proposed method treats all the point sets on an equal footing: they are realizations of a Gaussian mixture (GMM) and the registration is cast into a clustering problem. We formally derive an EM algorithm that estimates both the GMM parameters and the rotations and translations that map each individual set onto the "central" model. The mixture means play the role of the registered set of points while the variances provide rich information about the quality of the registration. We thoroughly validate the proposed method with challenging datasets, we compare it with several state-of-the-art methods, and we show its potential for fusing real depth data.

Introduction

Registration of point sets is an essential methodology in computer vision, computer graphics, robotics, and medical image analysis. To date, while the vast majority of techniques deal with two sets, e.g., [START_REF] Besl | A method for registration of 3-D shapes[END_REF][START_REF] Fitzgibbon | Robust registration of 2D and 3D point sets[END_REF][START_REF] Tsin | A correlation-based approach to robust point set registration[END_REF][START_REF] Myronenko | Point-set registration: Coherent point drift[END_REF][START_REF] Horaud | Rigid and articulated point registration with expectation conditional maximization[END_REF][START_REF] Jian | Robust point set registration using gaussian mixture models[END_REF], the multipleset registration problem has comparatively received less attention, e.g., [START_REF] Williams | Simultaneous registration of multiple corresponding point sets[END_REF][START_REF] Wang | Simultaneous nonrigid registration of multiple point sets and atlas construction[END_REF]. There are many practical situations when multiple-set registration is needed, nevertheless the problem is generally solved by applying pairwise registration repeatedly, either sequentially [START_REF] Blais | Registering multiview range data to create 3d computer objects[END_REF][START_REF] Masuda | A robust method for registration and segmentation of multiple range images[END_REF][START_REF] Izadi | Kinectfusion: Real-time 3d reconstruction and interaction using a moving depth camera[END_REF], or via a one-versus-all strategy [START_REF] Bergevin | Towards a general multiview registration technique[END_REF][START_REF] Castellani | Registration of multiple acoustic range views for underwater scene reconstruction[END_REF][START_REF] Huber | Fully automatic registration of multiple 3d data sets[END_REF].

Regardless of the particular pairwise registration algorithm that is being used, their use for multiple-set registration has limited performance. On the one Fig. 1. The proposed generative model for joint registration of multiple point clouds (left) and the associated graphical model (right). Unlike pairwise registration strategies, the proposed model simultaneously registers an arbitrary number of point clouds with partial or total overlap and optimally estimates both the GMM and registration parameters. Hence, the solution is not biased towards a particular cloud.

hand, sequential register-then-integrate strategies suffer from error propagation while they are optimal only locally, i.e., between point-set pairs. On the other hand, one-versus-all registration apparently leads to a biased estimator since one of the sets governs the registration and the solution is optimal only for this reference set. Therefore, an unbiased solution that evenly distributes the errors across all point sets is particularly desirable.

In this paper we propose a generative approach to the problem of joint registration of multiple 3D point sets. An arbitrary number of point sets are assumed to be generated from the same Gaussian mixture model (GMM). More precisely, an observed point i from set j, once rotated (R j ) and translated (t j ), is generated from the k-th component of a GMM, e.g., Fig. 1. Therefore, the GMM parameters are conditioned by the registration parameters (rotations and translations). This can be cast into a maximum likelihood formulation that is efficiently solved via an expectation conditional maximization (ECM) algorithm that jointly and optimally estimates all the GMM and registration parameters.

Unlike existing approaches to point registration that constrain the GMM means to coincide with the points of one set, the parameters of the proposed mixture model are not tight to a particular set. Existing approaches have the danger that noise and outliers, inherently present in the point set chosen to be the GMM means, contaminates the solution in an irrevocable way. It is well known that noisy data and outliers can be very robustly handled with GMMs by including a uniform component [START_REF] Banfield | Model-based Gaussian and non-Gaussian clustering[END_REF]. This has already been proposed in the framework of pairwise registration [START_REF] Myronenko | Point-set registration: Coherent point drift[END_REF][START_REF] Horaud | Rigid and articulated point registration with expectation conditional maximization[END_REF], in which case one set is supposed to be "bad" while the other one is supposed to be "perfect". In the proposed model all the sets are treated similarly and the GMM means are obtained by averaging over several transformed points belonging to different sets. Therefore, the proposed approach puts all the data on an equal footing and hence it is more robust. This is particularly beneficial when the task is to align a large number of point clouds, e.g., gathered with a depth camera.

The remainder of this paper is organized as follows: Section 2 discusses the related work. Section 3 formulates the problem in a generative probabilistic framework while Section 4 presents the proposed formulation and the associated algorithm. Experiments are presented in Section 5 and Section 6 concludes the paper.

Related work

Modern point registration methods adopt soft assignment strategies, thus generalizing ICP [START_REF] Besl | A method for registration of 3-D shapes[END_REF]. In all these methods one set is the "model" and the other set is the "data" [START_REF] Wells | Statistical approaches to feature-based object recognition[END_REF][START_REF] Granger | Multi-scale EM-ICP: A fast and robust approach for surface registration[END_REF][START_REF] Chui | A new point matching algorithm for non-rigid registration[END_REF][START_REF] Myronenko | Point-set registration: Coherent point drift[END_REF][START_REF] Horaud | Rigid and articulated point registration with expectation conditional maximization[END_REF], to cite just a few. This non-symmetric treatment leads to biased solutions. Alternatively, [START_REF] Jian | Robust point set registration using gaussian mixture models[END_REF][START_REF] Hermans | Robust point set registration using em-icp with information-theoretically optimal outlier handling[END_REF] consider two GMMs, one for each point set and the rigid transformation is applied to one of these mixtures. This leads to a non-linear optimization problem, hence requiring proper initialization. Moreover, outliers are not explicitly taken into account by these methods.

Multiple point-set registration is often solved using a sequential pairwise registration strategy [START_REF] Blais | Registering multiview range data to create 3d computer objects[END_REF][START_REF] Masuda | A robust method for registration and segmentation of multiple range images[END_REF][START_REF] Izadi | Kinectfusion: Real-time 3d reconstruction and interaction using a moving depth camera[END_REF][START_REF] Newcombe | Kinectfusion: Real-time dense surface mapping and tracking[END_REF][START_REF] Chen | Object modelling by registration of multiple range images[END_REF]. Whenever an additional set is available, the model parameters are updated using either ICP or a probabilistic scheme. In addition to the drawbacks associated with pairwise registration, this incremental mode of operation is subject to error propagation. Another possible strategy is to register pairs of views and subsequently to refine the registration parameters by solving an optimization problem that updates the parameters with respect to a reference set [START_REF] Bergevin | Towards a general multiview registration technique[END_REF]. [START_REF] Huber | Fully automatic registration of multiple 3d data sets[END_REF] starts with pairwise registrations to build a connected graph of overlapping sets, while a global optimization step over this graph representation eliminates matches that are not globally consistent. Similarly and more efficiently, [START_REF] Castellani | Registration of multiple acoustic range views for underwater scene reconstruction[END_REF] and [START_REF] Torsello | Multiview registration via graph diffusion of dual quaternions[END_REF] globally refine the registration between overlapping sets by working only in the transformation space. Despite the global refinement step, these methods except [START_REF] Torsello | Multiview registration via graph diffusion of dual quaternions[END_REF] suffer from the same limitation, namely one of the point sets is chosen as a reference set and hence the final parameters are biased.

Multiple point-set registration was also addressed in [START_REF] Williams | Simultaneous registration of multiple corresponding point sets[END_REF][START_REF] Wang | Simultaneous nonrigid registration of multiple point sets and atlas construction[END_REF]. Both these methods estimate a transformation for each point set, such that the transformed sets can be jointly registered. Starting from some known point correspondences between the sets, [START_REF] Williams | Simultaneous registration of multiple corresponding point sets[END_REF] estimates the transformation parameters through the minimization of a criterion that relates any two overlapping sets, and optionally integrates confidence of points. Since point correspondences are provided by pairwise ICP, this approach is referred to as multi-set ICP. As with pairwise ICP, one-to-one correspondences lead to the aforementioned limitations. Notice that the same formulation but with a different optimization method is proposed in [START_REF] Krishnan | Optimisation-on-a-manifold for global registration of multiple 3d point sets[END_REF], and was recently extended in [START_REF] Mateo | Bayesian perspective for the registration of multiple 3d views[END_REF] to deal with unknown correspondences. As in [START_REF] Williams | Simultaneous registration of multiple corresponding point sets[END_REF], [START_REF] Bartoli | Stratified generalized procrustes analysis[END_REF] registers matched shapes by estimating transformations (one per shape) of an unknown reference shape. [START_REF] Wang | Simultaneous nonrigid registration of multiple point sets and atlas construction[END_REF] shares a lot of similarities with [START_REF] Jian | Robust point set registration using gaussian mixture models[END_REF] in the sense that it represents each point set as a GMM and the transformations are applied to these mixtures rather than to individual points. The model parameters are estimated by minimizing the Jensen-Shannon divergence. A by-product of the algorithm is a probabilistic atlas defined by a convex combination of the mixtures. To the best of our knowledge, this is the only method that achieves joint multiple-set registration without recourse to a pairwise strategy. However the GMM representation of a point set inherently encapsulates the set's noisy and outlier observations, and hence the registration of point sets with different amounts of noise and of outliers is problematic, as well as sets with large non overlapping regions.

Problem formulation

Let V j = [v j1 . . . v ji . . . v jNj ] be a R 3×Nj matrix of N j points associated with the j-th point set and let M be the number of sets. We denote with V = {V j } M j=1 the union of all the data points. It is assumed that there is a rigid transformation φ j : R 3 → R 3 that maps a point set j onto a scene-centered model. The objective is to estimate the set-to-scene transformations under the constraint that the sets are jointly registered. It is assumed that the point sets are rigidly-transformed realizations of an unknown "central" GMM. Hence, one can write

P (v ji ) = K k=1 p k N φ j (v ji )|x k , Σ k + p K+1 U (a -b), (1)
where φ j (v ji ) = R j v ji + t j (a 3 × 3 rotation matrix R j and a 3 × 1 translation vector t j ), p k are the mixing coefficients

K+1 k=1 p k = 1,
x k and Σ k are the means and covariance matrices, and U is the uniform distribution parameterized by ab. Here we take ab = h, where h is the volume of the 3D convex hull encompassing the data [START_REF] Horaud | Rigid and articulated point registration with expectation conditional maximization[END_REF]. We now define γ as the ratio between outliers and inliers, that is,

p K+1 = γ K k=1 p k . (2)
This allows to balance the outlier/inlier proportion by choosing γ. To summarize, the model parameters are

Θ = {p k , x k , Σ k } K k=1 , {R j , t j } M j=1 . (3)
We stress that the deterministic nature of φ j does not affect the statistical properties of the mixture model. Fig. 1 shows a graphical representation of the proposed model. This problem can be solved in the framework of expectation-maximization. In particular, we define hidden variables Z = {z ji |j = 1 . . . M , i = 1 . . . N j } such that z ji = k assigns observation φ j (v ji ) to the k-th component of the mixture model, and we aim to maximize the expected complete-data log-likelihood

E(Θ|V, Z) = E Z [logP (V, Z; Θ)|V] = Z P (Z|V, Θ) log(P (V, Z; Θ)) (4)
in order to estimate the parameters Θ.

Joint Multiple-Set Registration

Assuming that the observed data V are independent and identically distributed, it is straightforward to write (4) as

E(Θ|V, Z) = j,i,k α jik log p k + log P (φ j (v ji )|z ji = k; Θ) (5)
where α jik = P (z ji = k|v ji ; Θ) are the posteriors. By replacing the standard expressions of the likelihoods [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF] and by ignoring constant terms, (5) can be written as an objective function of the form

f (Θ) = - 1 2 j,i,k α jik φ j (v ji ) -x k ) 2 Σ k + log |Σ k | -2 log p k + log p K+1 j,i α ji(K+1) (6)
where | • | denotes the determinant and y 2 A = y ⊤ A -1 y. Therefore, one has to solve the following constrained optimization problem:

max Θ f (Θ) s.t. R ⊤ j R j = I, |R j | = 1, ∀j = 1 . . . M. (7)
Direct optimization of f (Θ) via a closed-form solution is difficult owing to the induced non-linearities. Therefore, we adopt an expectation conditional maximization (ECM) scheme to solve [START_REF] Castellani | Registration of multiple acoustic range views for underwater scene reconstruction[END_REF]. ECM is more broadly applicable than EM, while it is well suited for our problem owing to the extended parameter set. Notice that ECM replaces the M-step of EM with a series of conditional maximization (CM) steps, that is, an M-substep for each parameter. We will refer to this algorithm as joint registration of multiple point clouds (JR-MPC); its outline is given in Algorithm 1. JR-MPC maximizes f (Θ), and hence E(Θ|V, Z), sequentially with respect to each parameter, by clamping the remaining ones to their current values. Commonly, such an iterative process leads to a stepwise maximization of the observed-data likelihood as well [START_REF] Meng | Maximum likelihood estimation via the ECM algorithm: a general framework[END_REF]. At each iteration, we first estimate the transformation parameters, given the current GMM parameters, and then we estimate the new GMM parameters, given the new transformation parameters. It is of course possible to adopt a reverse order, in particular when a rough alignment of the point sets is provided. However, we consider no prior information on the rigid transformations, so that the pre-estimation of the registration parameters favors the estimation of the GMM means, x k , that should be well distributed in space. Now that our objective function is specified, we are going to present in detail each step of JR-MPC. We restrict the model to isotropic covariances, i.e., Σ k = σ 2 k I, since it leads to a more efficient algorithm, while experiments with non-isotropic covariance [START_REF] Horaud | Rigid and articulated point registration with expectation conditional maximization[END_REF] showed that there is no significant accuracy gain.

Algorithm 1 Joint Registration of Multiple Point Clouds (JR-MPC)

Require: Initial parameter set Θ 0 1: q ← 1 2: repeat 3:

E-step: Use Θ q-1 to estimate posterior probabilities α q jik = P (zji = k|vji; Θ q-1 )

4:

CM-step-A: Use α q jik , x q-1 k and Σ q-1 k to estimate R q j and t q j .

5:

CM-step-B : Use α q jik , R q j and t q j to estimate the means x q k .

6:

CM-step-C : Use α q jik , R q j , t q j and x q k to estimate the covariances Σ q k .

7:

CM-step-D: Use α q jik to estimate the priors p q k .

8:

q ← q + 1 9: until Convergence 10: return Θ q E-step: By using the definitions for the likelihood and prior terms, and the decomposition of the marginal distribution, P (φ j (v ji )) = K+1 s=1 p s P (φ j (v ji )|z ji = s), the posterior probability α jik of v ij to be an inlier can be computed by

α jik = p k σ -3 k exp -1 2σ 2 k φ j (v ji ) -x k 2 K s=1 p s σ -3 s exp -1 2σ 2 s φ j (v ji ) -x s 2 + β , k = 1, . . . , K, (8)
where β = γ/h(γ + 1) accounts for the outlier term,while the posterior to be an outlier is simply given by α ji(K+1) = 1 -K k=1 α jik . As shown in Alg. 1, the posterior probability at the q-th iteration, α q jik , is computed from (8) using the parameter set Θ q-1 .

CM-step-A:

This step estimates the transformations φ j that maximize f (Θ), given current values for α jik , x k , Σ k . Notice that this estimation can be carried out independently for each set j, since φ j associates each point set with the common set of GMM means.

By setting the GMM parameters to their current values, we reformulate the problem to estimate the roto-translations. It can be easily shown that the maximizers R * j , t * j of f (Θ) coincide with the minimizers of the following constrained optimization problems

   min Rj ,tj (R j W j + t j e ⊤ -X)Λ j 2 F s.t. R ⊤ j R j = I, |R j | = 1 (9)
where Λ j is a K × K diagonal matrix with elements λ jk = 1

σ k Nj i=1 α jik , X = [x 1 , . . . , x K ],
e is a vector of ones, • F denotes the Frobenius norm, and W j = [w j1 , . . . , w jK ], with w jk , is a virtual 3D point given by

w jk = Nj i=1 α jik v ji Nj i=1 α jik , ( 10
)
or the weighted average of points in point set j, being the weights proportional to the posterior probabilities in terms of the k-th component. The above problem is an extension of the problem solved in [START_REF] Umeyama | Least-squares estimation of transformation parameters between two point patterns[END_REF] as here we end up with a weighted case due to Λ j . The problem still has an analytic solution. In specific, the optimal rotation is given by R

* j = U L j S j U R j ⊤ , (11)
where U L j and U R j are the left and right matrices obtained from the singular value decomposition of matrix X j Λ j P j Λ j W ⊤ , with P j = I -Λj e(Λj e) ⊤ (Λj e) ⊤ Λj e being a projection matrix, and

S j = diag(1, 1, |U L j ||U R j |).
Once the optimum rotation is known, the optimum translation is computed by

t * j = 1 tr(Λ 2 j ) (X -R * j W j )Λ 2 j e. ( 12
)
Note that φ j aligns the GMM means {x k } K k=1 with the virtual points {w jk } K k=1 . Hence, our method deals with point sets of different cardinalities and the number of components K in the GMM can be chosen independently of the cardinality of the point sets. CM-step-B and CM-step-C: These steps estimate the GMM means and variances given the current estimates of the rigid transformations and of the posteriors. By setting ∂f /∂x k = 0, k = 1 . . . , K, we easily obtain the optimal means. Then, we replace these values and obtain optimal variances by setting ∂f /∂σ k = 0. This leads to the following formulas for the means and the variances

x * k = M j=1 Nj i=1 α jik (R * j v ji + t * j ) M j=1 Nj i=1 α jik , σ * 2 k = M j=1 Nj i=1 α jik R * j v ji + t * j -x * k 2 2 3 M j=1 Nj i=1 α jik + ǫ 2 , (13)
with ǫ 2 being a very low positive value to efficiently avoid singularities [START_REF] Horaud | Rigid and articulated point registration with expectation conditional maximization[END_REF].

CM-step-D:

This step estimates the priors p k . From (2) we obtain

K k=1 p k = 1/(1 + γ)
. By neglecting the terms in (6) that do not depend on the priors and by using a Lagrange multiplier, the dual objective function becomes

f L (p 1 , . . . , p K , µ) = K k=1   log p k i,j α jik   + µ K k=1 p k - 1 1 + γ . ( 14
)
Setting ∂f L /∂p k = 0 yields the following optimal priors

p * k = j,i α jik µ , k = 1 . . . K and p * K+1 = 1 - K k=1 p * k , (15)
with µ = (γ + 1)(Nj,i α ji(K+1) ) and N = j N j being the cardinality of V.

Note that if γ → 0, which means that there is no uniform component in the mixture, then µ → N , which is in agreement with [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]. Based on the pseudocode of Alg. 1, the above steps are iterated until a convergence criterion is met, e.g., a sufficient number of iterations or a bound on the improvement of f (Θ).

Experiments with Synthetic and Real Data

For quantitative evaluation, we experiment with the 3D models "Bunny", "Lucy" and "Armadillo" from the Stanford 3D scanning repository 1 . We use fully viewed models in order to synthesize multiple point sets, as follows. The model point coordinates are shifted at the origin, the points are downsampled and then rotated in the xz-plane; points with negative z coordinates are rejected. This way, only a part of the object is viewed in each set, the point sets do not fully overlap, and the extent of the overlap depends on the rotation angle, as in real scenarios. It is important to note that the downsampling is different for each set, such different points are present in each set and the sets have different cardinalities (between 1000 and 2000 points). We add Gaussian noise to point coordinates based on a predefined signal-to-noise ratio (SNR), and more importantly, we add outliers to each set which are uniformly distributed around five randomly chosen points of the set.

For comparison, we consider the 3D rigid registration algorithms ICP [START_REF] Besl | A method for registration of 3-D shapes[END_REF], CPD [START_REF] Myronenko | Point-set registration: Coherent point drift[END_REF], ECMPR [START_REF] Horaud | Rigid and articulated point registration with expectation conditional maximization[END_REF], GMMReg [START_REF] Jian | Robust point set registration using gaussian mixture models[END_REF] and the simultaneous registration algorithm of [START_REF] Williams | Simultaneous registration of multiple corresponding point sets[END_REF] abbreviated here as SimReg. Note that CPD is exactly equivalent to ECMPR when it comes to rigid registration and that SimReg internally uses an ICP framework. Other than SimReg, the rest are pairwise registration schemes that register the first point set with each of the rest. Sequential ICP (seqICP) does the known register-then-integrate cycle. Although GMMReg is the version of [START_REF] Wang | Simultaneous nonrigid registration of multiple point sets and atlas construction[END_REF] for two point sets, the authors provide the code only for the pairwise case. We choose GMMReg for comparison since, as showed in [START_REF] Jian | Robust point set registration using gaussian mixture models[END_REF], Levenberg-Marquardt ICP [START_REF] Fitzgibbon | Robust registration of 2D and 3D point sets[END_REF] performs similarly with GMMReg, while [START_REF] Wang | Simultaneous nonrigid registration of multiple point sets and atlas construction[END_REF] shows that GMMReg is superior to Kernel Correlation [START_REF] Tsin | A correlation-based approach to robust point set registration[END_REF].

As far as the registration error is concerned, we use the root-mean-square error (RMSE) of rotation parameters since translation estimation is not challenging. For all algorithms, we implicitly initialise the translations by transferring the centroids of the point clouds into the same point, while identity matrices initialize the rotations. The only exception is the SimReg algorithm which fails without a good starting point, thus the transformations are initialized by pairwise ICP. GMMReg is kind of favored in the comparison, since it uses a two-level optimization process and the first level helps the algorithm to initialize itself. Notice that both SimReg and the proposed method provide rigid transformations for every point set, while ground rotations are typically expressed in terms 1 https://graphics.stanford.edu/data/3Dscanrep/3Dscanrep.html of the first set. Hence, the product of estimations R⊤

1 Rj is compared with the ground rotation R j , and the error is R⊤ 1 Rj -R j F . We consider a tractable case of jointly registering four point sets, the angle between the first set and the other sets being 10 o , 20 o and 30 o respectively. Since JR-MPC starts from a completely unknown GMM, the initial means x k are distributed on a sphere that spans the convex hull of the sets. The variances σ k are initialized with the median distance between x k and all the points in V.

For our experiments, we found that updating priors do not drastically improve the registration, thus we fix the priors equal to 1/(K + 1) and γ = 1/K, while h is chosen to be the volume of a sphere whose radius is 0.5; the latter is not an arbitrary choice since the point coordinates are normalized by the maximum distance between points of the convex hull of V. CPD and ECMPR deal with the outliers in the same way. The number of the components, K, is here equal to 60% of the mean cardinality. We use 100 iterations for all algorithms excepting GMMReg, whose implementation performs 10 and 100 function evaluations for the first and second optimization levels respectively. Fig. 2 shows the final log-RMSE averaged over 100 realisations and all views as a function of outlier percentage for each 3D model. Apparently, ICP and SimReg are more affected by the presence of outliers owing to one-to-one correspondences. CPD and GMMReg are affected in the sense that the former assigns outliers to any of the GMM components, while the latter clusters together outliers. The proposed method is more robust to outliers and the registration is successful even with densely present outliers. The behavior of the proposed algorithm in terms of the outliers is discussed in detail below and showed on Fig. 4. To visualize the convergence rate of the algorithms, we show curves for a typical setting (SN R = 10dB and 20% outliers). Regarding GMMReg, we just plot a line that shows the error in steady state. There is a performance variation as the model changes. "Lucy" is more asymmetric than "Bunny" and "Armadillo", thus a lower floor is achieved. Unlike the competitors, JR-MPC may show a minor perturbation in the first iterations owing to the joint solution and the random initialization of the means x k . However, the estimation of each transformation benefits from the proposed joint solution, in particular when the point sets contain outliers, and JR-MPC attains the lowest floor.

It is also important to show the estimation error between non overlapping sets. This also shows how biased each algorithm is. Based on the above experiment (SNR=10db, 20% outliers), Table 1 reports the average rotation error for the pairs (V 2 , V 3 ) and (V 3 , V 4 ), as well as the standard deviation of these two errors as a measure of bias. All but seqICP do not estimate these direct mappings. The proposed scheme, not only provides the lowest error, but it also offers the most symmetric solution.

A second experiment evaluates the robustness of the algorithms in terms of rotation angle between two point sets, hence the degree of overlap. This also allows us to show how the proposed algorithm deals with the simple case of two point sets. Recall that JR-MPC does not reduce to CPD/ECMPR in the twoset case, but rather it computes the poses of the two sets with respect to the Bunny Lucy Armadillo ICP [START_REF] Besl | A method for registration of 3-D shapes[END_REF] 0.329 0.423 0.047 0.315 0.297 0.009 0.263 0.373 0.055 GMMReg [START_REF] Jian | Robust point set registration using gaussian mixture models[END_REF] 0.364 0.303 0.030 0.129 0.110 0.009 0.228 0.167 0.031 CPD [START_REF] Myronenko | Point-set registration: Coherent point drift[END_REF], ECMPR [START_REF] Horaud | Rigid and articulated point registration with expectation conditional maximization[END_REF] 0.214 0.242 0.014 0.144 0.109 0.017 0.222 0.204 0.009 SimReg [START_REF] Williams | Simultaneous registration of multiple corresponding point sets[END_REF] 0.333 0.415 0.041 0.354 0.245 0.055 0.269 0.301 0.016 JR-MPC 0.181 0.165 0.008 0.068 0.060 0.004 0.147 0.147 0.000 Table 1. Registration error of indirect mappings. For each model, the two first columns show the rotation error of V2 → V3 and V3 → V4 respectively, while the third column shows the standard deviation of these two errors (SN R = 10db, 30% outliers).

"central" GMM. Fig. 3 plots the average RMSE over 50 realizations of "Lucy" and "Armadillo", when the relative rotation angle varies from -90 o to 90 o . As for an acceptable registration error, the proposed scheme achieves the widest and shallowest basin for "Lucy", and competes with GMMReg for "Armadillo". Since "Armadillo" consists of smooth and concave surface parts, the performance of the proposed scheme is better with multiple point sets than the two-set case, hence the difference with GMMReg. The wide basin of GMMReg is also due to its initialization.

As mentioned, a by-product of the proposed method is the reconstruction of an outlier-free model. In addition, we are able to detect the majority of the outlying points based on the variance of the component they most likely belong to. To show this effect, we use the results of one realization of the first experiment with 30% outliers. the first iteration are very low and make the means x k shrink into a very small cell. While the two point sets are around the points (0, 0, 0) and (40, 40, 40), we build the scene model around the point [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF][START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF][START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]. The distribution of the final deviations σ k is shown in (c). We get the same distribution with any model and any outlier percentage, as well as when registering real data. Although one can fit a pdf here, e.g., Rayleigh, it is convenient enough to split the components using the threshold T σ = 2 × median(S), where S = {σ k |k = 1, . . . K}. Accordingly, we build the scene model and we visualize the binary classification of points x k . Apparently, whenever components attract outliers, even not far from the object surface, they tend to spread their hull by increasing their scale. Based on the above thresholding, we can detect such components and reject points that are assigned with high probability to them, as shown in (g). Despite the introduction of the uniform component that prevents the algorithm from building clusters away from the object surface, locally dense outliers are likely to create components outside the surface. In this example, most of the point sets contain outliers above the shoulders, and the algorithm builds components with outliers only, that are post-detected by their variance. The integrated surface is shown in (h) and (i) when "bad" points were automatically removed. Of course, the surface can be post-processed, e.g., smoothing, for a more accurate representation, but this is beyond of our goal. We report here CPU times obtained with unoptimized Matlab implementations of the algorithms. ICP, CPD (ECMPR), SimReg, and JR-MPC require 14.7s, 40.6s, 24.6s, and 20.9s respectively to register four point sets of 1200 points, on an average. The C++ implementation of GMMReg requires 6.7s. JR-MPC runs faster than repeating CPD(ECMPR) since only one GMM is needed and the number of components is less than the number of points. Of course, ICP is the most efficient solution. However, SimReg needs more time as it enables every pair of overlapping point sets.

We also tested our method with real depth data captured from a time-of-flight (TOF) camera that is rigidly attached to two color cameras. Once calibrated [START_REF] Hansard | Projective alignment of range and parallax data[END_REF][START_REF] Hansard | Automatic detection of calibration grids in time-of-flight images[END_REF], this sensor provides 3D point clouds with associated color information. We gathered ten point clouds by manually moving the sensor in front of a scene, e.g., Fig. 5. Multiple-set registration was performed with all the above methods. While only depth information is used for the registration, the use of color information helps the final assessment and also shows the potential for fusing RGB-D data. Fig. 5 shows the results obtained with JR-MPC before (a) and after (b) rejecting outliers, seqICP (c) and SimReg (d). The proposed method successfully register the point clouds, while it automatically removes most of the jump-edge errors contained in range images. SimReg registers the majority of point sets, but it fails to register a few sets that appear flipped in the integrated view. Using the 5-th set as a reference for symmetry reasons, CDP/ECMPR and ICP also fail to register all the clouds while GMMReg yields low performance with too many misalignments. SeqICP causes weak misalignments, since it estimates weak geometric deformations between successive captures. However, the registration is not very accurate and further processing may be necessary, e.g., [START_REF] Izadi | Kinectfusion: Real-time 3d reconstruction and interaction using a moving depth camera[END_REF]. We refer the reader to the supplementary material for the integrated set of all the algorithms, viewed by several viewpoitns.

Conclusions

We presented a probabilistic generative model and its associated algorithm to jointly register multiple point sets. The vast majority of state-of-the-art techniques select one of the sets as the model and attempt to align the other sets onto this model. However, there is no guarantee that the model set is free of noise and outliers and this contaminates the estimation of the registration parameters. Unlike previous work, the proposed method treats all the point sets on an equal footing: they are realizations of a GMM and the registration is cast into a clustering problem. We formally derive an expectation-maximization algorithm that estimates the GMM parameters as well as the rotations and translations between each individual set and a "central" model. In this model the GMM means play the role of the registered points and the variances provide rich information about the quality of the registration. We thoroughly validated the proposed method with challenging datasets, we compared it with several state-of-the-art methods, and we showed its potential for fusing real depth data.

Fig. 2 .

 2 Fig. 2. Top: log-RMSE as a function of outlier percentage when SNR=10dB. Bottom: The learning curve of algorithms for a range of 100 iterations when the models are disturbed by SNR=10dB and 20% outliers. (a) "Lucy", (b) "Bunny" (c) "Armadillo".

  Fig 4 shows in (a) and (b) two out of four point sets, thereby one verifies the distortion of the point sets, as well as how different the sets may be, e.g., the right hand is missing in the first set. The progress of x k estimation is shown in (d-f). Apparently, the algorithm starts by reconstructing the scene model (observe the presence of the right hand). Notice the size increment of the hull of the points x k , during the progress. This is because the posteriors in

Fig. 3 .

 3 Fig. 3. RMSE as a function of the overlap (rotation angle) when two point sets are registered (SNR=20dB, 30% outliers) (a),(b) "Lucy" (c), (d) "Armadillo"

Fig. 4 .

 4 Fig. 4. (a),(b) Two point sets (out of four) with outliers; (c) distribution of estimated variances; instances of GMM means after (d) 5, (e) 15, and (f) 30 iterations; (g) the splitting of model points into inliers and outliers; joint-registration of four point sets (h) before and (i) after removing "bad" points (best viewed on-screen).

Fig. 5 .

 5 Fig. 5. The integrated point clouds from the joint registration of 10 TOF images that record a static scene. Top: color images that roughly show the scene content of each range image (occlusions due to cameras baseline cause some texture artefacts). Bottom: top-view of joint registration obtained from (a) JR-MPC, (b) JR-MPC+outlier rejection, (c) sequential ICP and (d) SimReg.

⋆ This work has received funding from Agence Nationale de la Recherche under the MIXCAM project number ANR-13-BS02-0010-01.

Supplementary material. Datasets, code and videos are publicly available at https://team.inria.fr/perception/research/jrmpc/


Emmanuelle Chapoulie

Rachid Guerchouche

Pierre-David Peit

Gaurav Chaurasia

Philippe Robert

George Drettakis
email: george.drettakis@inria.fr

Pierre-David Petit

Inria

Reminiscence Therapy using Image-Based Rendering in VR

Keywords: Reminiscence Therapy, Image-Based Rendering, Virtual Reality, Immersive Environments, Autobiographical Fluency I.3.7 [COMPUTER GRAPHICS]: Three-Dimensional Graphics and Realism-Virtual reality / Applications

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Autobiographical memory (AM) is an essential component of human memory. In [START_REF] Conway | Memory and the self[END_REF] it is defined as the mental representation of events from one's past (i.e., episodic AM) together with semantic information about the self. AM is one of the underlying processes of reminiscence therapy (RT). RT involves the discussion of past activities, events and experiences with another person or group of people, usually with the aid of tangible props such as photographs, household and other familiar items from the past. There is evidence of some improvement of functional ability in people with dementia using RT [START_REF] Woods | Reminiscence therapy for dementia[END_REF]. RT is one of the most popular psychosocial interventions in dementia care, and is highly rated by staff and participants. It is thus important to develop ways to improve RT using new technologies. In this work, we will focus on immersive Virtual Reality (VR), which we define here as a setup with a large-screen, stereo 3D display, including head (and optionally hand/finger) tracking. There has been promising previous work in VR for memory treatments (e.g., [START_REF] Brooks | The use of virtual reality in memory rehabilitation: current findings and future directions[END_REF][START_REF] Gonneaud | Assessing prospective memory in young healthy adults using virtual reality[END_REF]) In contrast to previous work however, VR support for RT in particular requires the presentation of environments which are familiar to the patient. Creating a realistic 3D model e.g., of a patient's neighborhood or landmarks of their city, is far too expensive using traditional manual modeling. In contrast, image-based modeling and rendering [START_REF] Shum | Image-Based Rendering[END_REF], provides a much easier way to achieve this goal, since only a small set of photographs is required as input. It is however only very recently [START_REF] Chaurasia | Depth synthesis and local warps for plausible image-based navigation[END_REF] that imagebased rendering (IBR) has achieved a sufficient level of quality and efficiency to be suitable for free-viewpoint navigation, and thus for VR. The high level of realism of [START_REF] Chaurasia | Depth synthesis and local warps for plausible image-based navigation[END_REF] can be seen in Fig. 1.

In this paper, we introduce an immersive VR system designed for RT, developed jointly by a group of memory clinicians and computer scientists. Based on the RT requirements listed above our system provides easy capture and display of familiar environments, by using IBR, and supports gesture-based navigation in the environment with finger tracking. We refer to the system as IVIRAGE -Image-based VIrtual ReAlity with GEstures -from now on.

The potential clinical implications of IVIRAGE -if successful -are numerous. For example, the VEs provided by IVIRAGE are ecologically valid and generalizable, which is crucial for memory evaluation [START_REF] Kibby | Ecological validity of neuropsychological tests: focus on the California Verbal Learning test and the Wisconsin card sorting test[END_REF]. The realism and interactivity of IVIRAGE, also hold great promise for motivation and engagement [START_REF] Cohen-Mansfield | Engagement in persons with dementia: the concept and its measurement[END_REF], which are also key components of successul RT.

To evaluate the effectiveness and utility of IVIRAGE in RT, we performed a study with healthy elderly participants (mean age 66.84 years). The study investigates whether IVIRAGE can be useful for RT, by adapting a verbal Autobiographical Fluency test [START_REF] Dritschel | Autobiographical fluency: A method for the study of personal memory[END_REF] to VR. In this test, elderly participants are asked to generate memories based on the images they see. We compare IVIRAGE for an unknown and a familiar environment to determine whether our approach can convey familiarity, and thus increase the number of gen-erated memories. We also investigate whether IVIRAGE is as effective in generating memories as current standard clinical practice.

Our contributions can thus be summarized as follows:

• We introduce the IVIRAGE system, demonstrating the first usage of free-viewpoint IBR in an immersive setting for RT.

• Our study shows that the number of autobiographical memories generated during the exposition to a familiar environment presented in IVIRAGE is higher than that for an unknown environment. This indicates that IBR can convey familiarity of a scene -an essential requirement for the use of VR in RT.

• Our study also shows that the IVIRAGE can be useful for RT, since acceptability scores indicate that the system is well tolerated by elderly patients, and by results that show that our system generates memories as effectively as traditional RT.

PREVIOUS WORK

We first provide some additional background on autobiographical memory, and then briefly overview work on rehabilitation and VR, and in particular work concerning memory and the elderly. We also discuss related work on Image-Based Rendering (IBR).

Autobiographical Memory

The retrieval of AM is a complex process, involving mental traces of past events and related semantic knowledge. These differ e.g., in content, imagery or emotional intensity. A fundamental dimension of an AM is its level of detail, which is thought to be a function of a person's database of memory traces, reconstructive processes, mental schemata, self goals and different retrieval strategies [START_REF] Conway | The construction of autobiographical memories in the self-memory system[END_REF]. Episodic AM, especially the richness of details, is impaired early in the course of Alzheimer's disease or even in the preclinical phase, while semantic memories are spared until moderate stages, indicating a dissociation between both memory systems [START_REF] Berna | Episodic autobiographical memory in normal aging and mild cognitive impairment: A population-based study[END_REF][START_REF] Seidl | Autobiographical memory deficits in alzheimer's disease[END_REF]. AM is used to develop a coherent sense of self, emotions and future plans [START_REF] Bluck | Autobiographical memory: Exploring its functions in everyday life[END_REF]. It thus has a rich role in motivating behavior and in daily life. Many different methods have been used to study AM [START_REF] Griffith | Current psychometric and methodological issues in the measurement of overgeneral autobiographical memory[END_REF]; one approach is to stimulate memories using word, visual or other sensorial cues. We concentrate on verbal Autobiographical Fluency (AF) tests [START_REF] Dritschel | Autobiographical fluency: A method for the study of personal memory[END_REF]. Such verbal fluency tests are widely used to assess language abilities and executive functioning [START_REF] Miller | Verbal fluency as a function of a measure of verbal intelligence and in relation to different types of cerebral pathology[END_REF]. These tasks involve rapid associative exploration and retrieval of words based on phonemic (e.g., starting with a specified letter of the alphabet) or semantic/categorical (e.g., animals) criteria over a brief timed interval.

VR and Neuropsychiatric Disturbances Therapy Early work in VR for cognitive assessment and rehabilitation is described in [START_REF] Riva | Virtual reality in neuro-psycho-physiology: Cognitive, clinical and methodological issues in assessment and rehabilitation[END_REF]; various theoretical and practical issues were discussed in [START_REF] Rizzo | Basic issues in the application of virtual reality for the assessment and rehabilitation of cognitive impairments and functional disabilities[END_REF]. The latter study underlines the importance of having graphics indistinguishable from the real world, demonstrating the utility of VR in this context. Treating phobias using VR has been investigated in different studies, e.g., [START_REF] Bruce | A virtual reality claustrophobia therapy system-implementation and test[END_REF]. VR was also used to treat Post Traumatic Stress Disorder [START_REF] Pair | A virtual reality exposure therapy application for iraq war post traumatic stress disorder[END_REF]. VR has been used extensively for memory rehabilitation; the review in [START_REF] Brooks | The use of virtual reality in memory rehabilitation: current findings and future directions[END_REF] demonstrates the positive effect of this technology. Recently, VR has been used to assess prospective memory using a driving simulator [START_REF] Gonneaud | Assessing prospective memory in young healthy adults using virtual reality[END_REF] with a large screen. Similarly, age effect on episodic memory was studied in a driving simulator [START_REF] Plancher | Age effect on components of episodic memory and feature binding: A virtual reality study[END_REF]. Use of VR with the elderly imposes specific considerations. These have been studied early on (e.g., [START_REF] Mcgee | Issues for the assessment of visuospatial skills in older adults using virtual environment technology[END_REF]), using VEs with stereo display and head/hand tracking. More recent results show that memory function can be improved with immersive audio-visual VR for the elderly [START_REF] Optale | Controlling memory impairment in elderly adults using virtual reality memory training: a randomized controlled pilot study[END_REF]. Our approach follows this line of work, in proposing novel tools to improve RT for the elderly.

Image-Based Rendering View-dependent texture mapping [START_REF] Debevec | Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach[END_REF] and the Unstructured Lumigraph [START_REF] Buehler | Unstructured lumigraph rendering[END_REF] are early examples of IBR methods which allow interactive viewing. However, both suffer from disturbing visual artifacts if the reconstructed 3D geometry is not perfect. There have been many more recent methods which improve various aspects of IBR (see [START_REF] Shum | Image-Based Rendering[END_REF] for an overview).

However, true interactivity with quality at a level acceptable for VR has remained elusive, despite advances in the last few years (e.g., [START_REF] Eisemann | Floating textures[END_REF]). Most IBR systems can be classified into two types -proxy based [START_REF] Buehler | Unstructured lumigraph rendering[END_REF][START_REF] Eisemann | Floating textures[END_REF] and image warping approaches [START_REF] Chaurasia | Silhouette-aware warping for image-based rendering[END_REF][START_REF] Chaurasia | Depth synthesis and local warps for plausible image-based navigation[END_REF]. The former apply a view-dependent texture on the 3D model of the scene while the latter warp input images and blend the warped images to synthesize the final result. Proxy-based approaches require an accurate 3D model which is essentially impossible to obtain using state of the art 3D reconstruction [START_REF] Furukawa | Accurate, dense, and robust multi-view stereopsis[END_REF]. The very recent work on depth synthesis and warp-based rendering [START_REF] Chaurasia | Depth synthesis and local warps for plausible image-based navigation[END_REF], does provide rendering quality which is acceptable for some level of VR display. The quality of the rendered images -given specific restrictions which we develop below -is definitely acceptable for the type of immersive VR we will use in this study, despite some residual visual artifacts. The resulting realism is very high, as can be seen in Fig. 1 and the accompanying video, even in views away from the input photographs used to reconstruct the scene. We summarize this algorithm in Sec. 4.2. since it is central to our study.

GOALS AND HYPOTHESES

Our approach is to use IBR for episodic memory stimulation in an autobiographical fluency protocol. Navigating in a highly realistic Image-Based Virtual Environment (IBVE) opens up new possibilities for RT. In particular, our approach allows easy access to highly realistic representations of familiar and personalized environments, and provides a high level of interactivity.

We investigate whether our immersive system is useful in stimulating autobiographical memories in healthy elderly adults and whether the highly realistic environments provided by IVIRAGE are powerful enough to convey a sense of familiarity to participants. To achieve this goal we designed and ran an experiment in which we compare the number of AF responses obtained in a familiar IBVE to those obtained in an unknown IBVE. We hypothesize that the immersion in the familiar scene will result in a higher number of responses: an indication that IVIRAGE can convey familiarity.

We also investigate whether IVIRAGE is as effective as traditional RT, by presenting a static photograph of a familiar environment, since photographs of familiar sites are used extensively in RT. We test whether IVIRAGE can generate a similar number of memories as the standard approach. However, there is a significant distinction between the two: to allow in-place navigation, the photographs used for the IBVEs have a narrow field-of-view (FOV) to provide a scale 1 immersive experience. In contrast, photographs used in traditional AF usually have large FOVs (e.g., a panoramic view), with the potential to elicit many memories. This gives an advantage to the traditional RT condition compared to the IVIRAGE condition with the familiar environment; we thus expect the number of memories generated by the familiar IBVE to be less than or at best equal to the familiar photo.

To be successful, we need to satisfy two fundamental requirements. First, we need to provide a working system allowing navigation using IBR and second, we need to demonstrate that our system and protocols are well tolerated by elderly participants and can be as effective as traditional RT. In the following, we present our system and experimental results that satisfy these requirements.

IVIRAGE: AN INTERACTIVE SYSTEM BASED ON IBR FOR REMINISCENCE THERAPY

The main goal for our interactive VR system for RT is to provide easy capture of familiar and personalized environments, while allowing realistic interactive display and navigation. The target hardware setup is a large screen with stereo display, and a high-end tracking system, which we describe next. We then describe the IBR approach of [START_REF] Chaurasia | Depth synthesis and local warps for plausible image-based navigation[END_REF] we integrated and adapted to our system.

Hardware Setup

The IBR algorithm [START_REF] Chaurasia | Depth synthesis and local warps for plausible image-based navigation[END_REF] we use in our system is limited to a single screen (see supplemental material for more details); consequently, we only use the front wall of a BARCO iSpace for our experiments (see inset). The size of the screen is 320cm × 240cm (resolution 1600 × 1200). The projectors use passive Infitec stereo via glasses, which are tracked using the ART tracking system (http: //www.ar-tracking.com). In addition, we use the ART wireless finger-tracking of the orientation of the hand and the position of the fingers. We use the three-finger (thumb-index-middle finger) version.

The fact that our participants are elderly (over 60 years old) implies specific precautions in the hardware setup. To avoid any risk of unsteadiness or falling, participants sat on a chair installed in front of the display screen, at a distance of 1 m, during the experiments. A small stool is placed next to the chair for the experimenter.

To enrich the sensation of immersion in the VR environments, we added ambient spatialized 3D sound rendering over a 7.1 loudspeaker setup in the cube. We typically play sounds such as street, car and ambient noise, crowds walking and talking etc.

Image-Based Capture, Rendering and Navigation

To achieve the best possible visual quality, we adapted the approach of [START_REF] Chaurasia | Depth synthesis and local warps for plausible image-based navigation[END_REF]. This approach is designed for free-viewpoint navigation in the scene, making it suitable for interactive VR.

The captured photographs are processed in order to synthesizes plausible depth, and thus compensate for lack of 3D geometry. The positions and orientations of the cameras which captured the images used for one of the scenes in this study are shown in Fig. 2. The approach in [START_REF] Chaurasia | Depth synthesis and local warps for plausible image-based navigation[END_REF] was developed for desktop applications where the user navigates using a mouse. In our immersive space setup, the novel camera position is provided by the head tracker in real time. A description of the algorithmic changes required to adapt [START_REF] Chaurasia | Depth synthesis and local warps for plausible image-based navigation[END_REF] to the immserive display are in the supplemental material. While our IVIRAGE system provides very realistic images, like any image-based method it is restricted to representing content which actually exists in the input images. When the user leaves this region, visual artifacts appear. To avoid this, and to ensure that all participants navigate in the same regions, we limit navigation to the zone where artifacts are very small.

Our system supports gesture-based navigation in the immersive experimental conditions. The majority of participants had no previous VR or gaming expertise. For this reason, we added a training session in order to habituate the participants to the use of the fingertracking device. It consists of a session of grabbing and releasing virtual objects in the IBR environments; more details can be found in the supplemental material.

To navigate inside the IBVEs, participants have to point in the direction they want to follow with the index finger and perform a pinch gesture of the thumb and middle finger. Direction is specified by the palm orientation so that the pointing can be approximate, but we instruct participants to point for clarity and simplicity. The gesture is independent of hand location and thus participants can place their hand on their knees to avoid arm muscular fatigue. This is particularly important for our elderly participants. Visual feedback of their thumb, index and middle fingers is presented as red, green and blue cylinders respectively (see supplemental material).

EXPERIMENTAL METHODOLOGY

We next describe the assessment methodology procedure, which is an adaptation of traditional Autobiographical Fluency [START_REF] Dritschel | Autobiographical fluency: A method for the study of personal memory[END_REF].

Population

The participants were 13 healthy elderly adults (5 Female and 8 Male, mean age = 66.84, SD = 4.33) recruited by the clinical Memory Center research group. Each participant underwent a cognitive behavioral and motor assessment. Subjects not living in the city of experimentation, with a clinical history of neuropsychiatric disease or dementia, with a Mini Mental Score [START_REF] Folstein | Mini-mental state": a practical method for grading the cognitive state of patients for the clinician[END_REF] lower than 28 (see Sec. 6.1) were excluded as well as subjects with dizziness or motor instability. All participants gave their informed consent before beginning the study. Ethical approval was received from the competent Ethics Committee.

Autobiographical memory assessment

In the AF test adapted for the experiment, participants had 2 minutes to generate as many memories as possible related to the environment they see, using short sentences. At the end of this time period the experimenter assessed the quality of the recollection using the Remember/Know procedure [START_REF] Gardiner | Experiences of remembering, knowing, and guessing[END_REF]. The subject was asked to indicate for each of his/her recollection if it is a Remember, i.e., conscious recollection of many vivid contextual details, such as "when" and "how" the information was learned or a Know, i.e., the sensation that the memory/event/environment has been seen before, but not being able to pin down the reason why.

The subjects verbal output was recorded during the experiment. The audio recordings were analyzed by a speech linguist independent from the design of our study. Derived scores include the Total number of memories and the number of Remember and Know recollections. In addition to the AF, subjects also had to complete questionnaires assessing the acceptability and the quality of the experiences during the different conditions and a standard Presence questionnaire (see Sec. 6.3).

Experimental Conditions

We have four different experimental conditions. First, we have two baseline conditions: a control condition which is a grey-colored blank screen and a photograph of a familiar location, which corresponds to the use of photos for traditional AF. We then have the two conditions with IVIRAGE, namely an unknown and a familiar environment. The four conditions are shown in Fig. 3, (a)-(d).

Baseline Conditions. The first condition (a), which we refer to as Grey consists of blank, medium grey screens. The second condition (b), referred to as Familiar Photo (FamPhoto), is a wellknown location in the city of the experiment. Baseline conditions are displayed as simple images, without stereovision nor tracking. IVIRAGE Conditions. Both IVIRAGE conditions are constructed and displayed with the method of [START_REF] Chaurasia | Depth synthesis and local warps for plausible image-based navigation[END_REF]. We refer to the third condition (c) as Familiar IBVE (FamIBVE). It consists of an imagebased representation of a known landmark square in the city of the experimentation. The fourth condition (d) is the Unknown IBVE (UnknoIBVE) which is captured in a public housing neighborhood, and contains buildings without recognizable features. Capture of IBVEs. We took 90 and 35 photographs for the FamIBVE and UnknoIBVE scenes respectively. The placement of the cameras used for the first environment are shown in Fig. 2. The total time for photography, reconstruction and preprocessing is between 2-4 hours, while rendering is real-time.

EXPERIMENTAL PROCEDURE

The experimental procedure has three main steps: inclusion, preparation and exposure to the different conditions.

Clinical inclusion

Clinical inclusion aims at verifying if the subject is a healthy adult and not a patient suffering from a memory disease. Before starting the experiments in the immersive space, each participant spends about 45 minutes with a clinician in a separate room.

After a motor clinical examination, the following tests are performed: Mini Mental Score [START_REF] Folstein | Mini-mental state": a practical method for grading the cognitive state of patients for the clinician[END_REF], Short Cognitive Battery [START_REF] Robert | Screening for alzheimers disease with the short cognitive evaluation battery[END_REF], Frontal Assessment Battery [START_REF] Dubois | The FAB a frontal assessment battery at bedside[END_REF], and the Apathy Inventory [START_REF] Robert | The apathy inventory: assessment of apathy and awareness in alzheimer's disease, parkinson's disease and mild cognitive impairment[END_REF].

This ensures that the participant is physically and mentally fit to participate in the experiment. If successful, the experimenter accompanies the participant to the immersive space. All participants succeeded the inclusion and took part in the study.

Before starting the experiment, we verify if participants are not stereo-blind by displaying a random-dot stereogram, and we perform a calibration procedure for finger tracking to adapt to each individuals' hand and finger size. The latter takes 2 minutes.

Exposure to the environments

The four environments are presented one after the other to each participant. "Grey" is always presented second since we do not expect the order of its presentation to influence performance. The other 3 environments are randomized in 6 different groups.

For the baseline conditions, the participant is presented with the photograph or blank screen for 15 seconds, and then the verbal fluency session begins.

For the IVIRAGE conditions, the session starts with the fingertracking system training session (see supplemental material for more details). The participant is then invited to navigate using the pointing gesture in the environment for about 45 seconds. At the end of the navigation, the verbal fluency test is performed.

We encourage the reader to view the accompanying video which illustrates the compelling experience provided by IVIRAGE.

Post Exposure

After the autobiographical fluency test each participant completed a set of questionnaires. (i) General questionnaire. A questionnaire was presented on a tablet to each participant at the end of each condition. The participant was asked whether the environment (except Grey) was recognized (YES/NO), and a continuous scale from 0 to 10 was used to rate the Familiarity with the environment (except Grey) and Emotion, Anxiety, Motivation, Security and Fatigue for all conditions. (ii) CyberSickness and Presence. For the IVI-RAGE conditions (UnknoIBVE and FamIBVE ), we tested cybersickness with the 22-item questionnaire of [START_REF] Viaud-Delmon | Adaptation as a sensorial profile in trait anxiety: a study with virtual reality[END_REF], and Presence using the questionnaire (http://www.igroup.org/pq/ipq/ index.php) presented in [START_REF] Schubert | The experience of presence: Factor analytic insights[END_REF].

RESULTS

We present the results of our quantitative measurements as well as qualitative evaluation elements.

Quantitative Evaluation

Autobiographical Fluency Responses. The speech linguist recorded the number of responses and determined those that are conscious responses (Remember) or more vague recollections (Know) (Sec. 5.2). Only Remember values are relevant for our study. The median values for Grey, UnknoIBVE, FamIBVE and FamPhoto are respectively: 1, 3, 5 and 5.

Since the number of Remember responses is the most significant measure in the present study, Fig. 4 illustrates the number of such answers for each participant for the IVIRAGE conditions. The Remember responses for the 4 conditions are shown in Fig. 5. We performed the non-parametric Friedman test [START_REF] Cunningham | Experimental design: From user studies to psychophysics[END_REF] on this data. A significant main effect of condition was found for Remember (χ 2 = 18.22, p = 0.0004). No effect of the group order was found (χ 2 = 6.23, p = 0.82).

We then ran a Wilcoxon Signed Ranks Test [START_REF] Cunningham | Experimental design: From user studies to psychophysics[END_REF] to determine whether the differences between conditions are significant. For Remember, Grey is significantly different from FamIBVE (p = 0.0002) and from FamPhoto (p = 0.00056); UnknoIBVE is significantly different from FamIBVE (p = 0.004) and FamPhoto (p = 0.01). No other significant differences were found.

General Questionnaire. All the participants indicated that they recognized the FamIBVE but not the UnknoIBVE (except one participant who knows both environments (see Sec. 7.2)). The results from the familiarity question show a high score for the FamIBVE (Mean=7.92, SD=1.86) and for the FamPhoto (Mean=8.74, SD=1.44) and a low score for the UnknoIBVE (Mean=3.28, SD=3.02). The results for the Emotion, Motivation and Security are shown in Tab. 1. We first ran a Levene normality test [START_REF] Cunningham | Experimental design: From user studies to psychophysics[END_REF] for the condition data for Emotion, Motivation and Security. We found that only Motivation follows a normal distribution. We thus ran a two-way ANOVA test [START_REF] Cunningham | Experimental design: From user studies to psychophysics[END_REF] on the Motivation data and a Friedman test on the Emotion and Security data. The results are as follows: there is a significant main effect of condition for Emotion (χ 2 = 13.64, p = 0.0034) and Motivation (F = 5.56, d f = [START_REF] Brooks | The use of virtual reality in memory rehabilitation: current findings and future directions[END_REF][START_REF] Debevec | Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach[END_REF], p = 0.0030); there is no effect for Security (χ 2 = 4.00, p = 0.26). There is no significant effect of group for all these variables.

The Wilcoxon Signed Ranks Test showed significant differences for Emotion between Grey and FamIBVE (p = 0.01); Grey and FamPhoto (p = 0.013); UnknoIBVE and FamIBVE (p = 0.003); UnknoIBVE and FamPhoto (p = 0.0012). Using the same statistical test, Motivation is significantly higher for FamIBVE compared to Grey (p = 0.018), FamPhoto compared to Grey (p = 0.0073) and UnknoIBVE compared to FamPhoto (p = 0.034).

Presence Questionnaire. The results of the Presence questionnaires have high overall scores (Median=5; scale 0-6) for General Presence and Spatial Presence [START_REF] Schubert | The experience of presence: Factor analytic insights[END_REF]. The Experienced Realism and Involvement have medium scores (Median=3; scale 0-6) for both UnknoIBVE and FamIBVE. The Friedman test showed no significant main effect of condition (χ 2 = 1.32, p = 0.87). Finally, participants did not experience sickness, fatigue or anxiety during the entire experiment across all conditions. This is shown by the data from the CyberSickness questionnaire (Mean=0.46, SD=0.1 for UnknoIBVE and Mean=1.42, SD=0.3 for FamIBVE on a scale of 0 to 88) as well as the Anxiety (Mean=2.3, SD=0.15 on a scale of 0 to 10) and Fatigue (Mean=0.87, SD=0.27 on a scale of 0 to 10) measures from the general questionnaire.

Informal Qualitative Evaluation

We did not perform a formal qualitative evaluation, but we report informal feedback and anecdotes from the experimental session.

All participants found the experience provided by IVIRAGE to be very interesting and stimulating; they expressed their satisfaction with many positive comments at the end of the sessions, both for interaction and navigation. One participant (number 3 in Fig. 4) had worked in the construction industry, and had actually worked on the construction of our unknown environment. Evidently, the un-known environment was actually familiar for this participant, which is reflected in the results of his fluency test.

In our familiar environment, there used to be a newsagent store which was recently closed, replaced by a fashion store with a much younger clientele. This newsagent had significant emotional value for the elderly population, and the vast majority of our participants commented on this issue, stating that they found this to be "a disgrace", "really a pity" etc.

Several participants commented that, despite the high level of realism, they found that the scene was "deserted". We provided the (true) explanation that the scenes are situated in the very early morning, and thus no people are present.

DISCUSSION

From a clinical perspective the results of our study underline two main points which hold promise for the future: 1) It is possible to use IVIRAGE with elderly subjects and 2) that the system can stimulate conscious recollections of autobiographical memory.

Concerning the first point, the physical setup used (a chair or a bench) shows that this approach adapts well to populations e.g., with limited mobility. The responses to the questionnaires on Emotion, Motivation and Security, show that the technological setup was well tolerated by our participants. The fact that IVIRAGE can adapt the VE to the personal history and surroundings of participants holds particular promise from a clinical perspective, since it will be able to cognitively stimulate low mobility subjects.

Concerning the second point, previous work has already demonstrated the utility of VR for cognitive/memory processes (e.g., [START_REF] Brooks | The use of virtual reality in memory rehabilitation: current findings and future directions[END_REF]). The increased number of memories with IVIRAGE for familiar vs. unknown scenes, shows that our system can help generate conscious autobiographical memories. This has several clinical implications. The goal of cognitive impairment treatment is to provide information on the subjects long term functional status.

In this field, the ecological validity or generalizability of neuropsychological test results, beyond the clinical environment, is a crucial component of the evaluative process [START_REF] Kibby | Ecological validity of neuropsychological tests: focus on the California Verbal Learning test and the Wisconsin card sorting test[END_REF]. The highly realistic environments of IVIRAGE are both ecological and generalizable, and give the additional opportunity to create an environment familiar to the patient and therefore closer to his/her real life.

Our study also suggests that it is possible to use VR in a therapeutic context. Our results indicate a positive effect of IVIRAGE on autobiographical memory which is a core component of RT. Our immediate future objective is to use our IVIRAGE system with patients in a clinical setting.

One of the great challenges in RT is whether different technologies maintain users motivation and engagement when confronting them with a repetitive series of training challenges, either cognitive or physical. This is of particular interest given that apathy i.e., the disorder of motivation, is the most frequent behavioral disturbance in elderly subjects with Alzheimer disease and related disorders. The interactivity and realism of IVIRAGE can improve motivation for training processes, with many positive consequences for patients. Engagement, defined as the act of being occupied or involved with an external stimulus [START_REF] Cohen-Mansfield | Engagement in persons with dementia: the concept and its measurement[END_REF] is the most challenging task for professional carers. Understanding individual interests may guide intervention parameters to improve engagement and potential benefits: the tailored environments we provide should thus be beneficial.

From a technical standpoint, the higher number of generated memories for FamIBVE compared to UnknoIBVE, is a very strong indication that IBR does convey familiarity in our immersive setting. The strong Presence scores also underline the ability of IBR to convey a sense of "being there". Despite the advantage (see Sec. 3) of the wide panoramic view of FamPhoto compared to the narrow FOV of FamIBVE, the number of responses is the same for both conditions, confirming the effectiveness of IVIRAGE for RT. We expect that the performance of FamIBVE will increase when the so-lution for multiple screens is available, since the FOV for the IBVE will be much larger.

Algorithmically, a new IBR algorithm for multiple screens is required, thus providing full immersion. While challenging, we are working on solutions to this. The addition of virtual humans to populate the scenes is another interesting challenge that should render the approach even more engaging and useful.

CONCLUSIONS

The results of our study shows that image-based techniques offer great promise for RT, and for VR in general. Our IVIRAGE system has numerous advantages over traditional 3D assets used in VR. The fact that only a few casual photographs are required to create a scene that can be used for VR is an advantage with very significant consequences. The level of realism obtained by the imagery, despite some residual artifacts, is at least as good that produced at great cost with traditional means e.g., manual modeling.

In conclusion, our study has given strong first evidence of the utility of IBR and close-to-natural gesture interaction for immersive VR systems, in the context of memory treatments and in particular RT. We are confident that our work will open the way to a much wider adoption of such technologies. The ease of creation of the IBVEs has the potential to not only greatly impact the use of VR for therapy, but also in other applications, since personalized assets can now be created and realistically displayed at low cost.

  Figure 1: (a) Image-based rendering with novel camera coincident with the input camera (red in inset); the result is identical to the input image. The user can move around freely in the area covered by the cameras. In (b)-(d) the method of [6] generates plausible novel views (red in inset) using the 4 views in blue.

Figure 2 :

 2 Figure 2: Top view of camera positions for the capture of the FamIBVE (see Sec. 5.3). This set of cameras allows navigation in a significantly large portion of the scene (75 × 30m).

Figure 3 :

 3 Figure 3: The four conditions used in our experiment.

Figure 4 :Figure 5 :

 45 Figure 4: UnknoIBVE vs. FamIBVE: Remember answers.

Table 1 :

 1 Acceptability and qualities of the experiences (Emotion (E), Motivation (M), Security (S)) during different conditions. Mn and SD respectively refer to mean and standard deviation.

	/		Grey		UnknoIBVE		FamIBVE	FamPhoto
	/	E	M	S	E	M	S	E	M	S	E	M	S
	Mn 4.29 4.84 7.86 3.82 6.44 7.46 7.23 8.59 9.4	7.30	8.64	9.43
	SD 3.54 3.14 3.00 3.27 3.41 3.28 1.56 1.3	0.47	1.98	0.99	0.38
	Emotion is highest in FamPhoto (Mean=7.30, SD=1.98)
	and FamIBVE (Mean=7.23, 1.56), while it was low for Grey
	(Mean=4.29, SD=3.54) and UnknoIBVE (Mean=3.82, SD=3.27).
	Mean Motivation is highest in familiar conditions (FamPhoto /
	FamIBVE) (Mean=8.64/8.59, SD=0.99/1.3) while mean Security
	for FamIBVE (Mean=9.7, SD=0.47) is on a par with FamPhoto
	(Mean=9.43, SD=0.38).

ACKNOWLEDGEMENTS

We acknowledge funding by the EU IP project VERVE (www.verveconsortium.eu) and the Region of PACA. We thank Autodesk for research and software donations, V. Lafont and A. Bouysset for help in the experiments, M. Marchal, A. Lécuyer and I. Viaud-Delmon for advice and J-S. Roo for photos.