Pierre-Henri Conze
email: pierre-henri.conze@technicolor.com

Philippe Robert
email: philippe.robert@technicolor.com

Tomás Crivelli
email: tomas.crivelli@technicolor.com

Luce Morin
email: luce.morin@insa-rennes.fr

Dense long-term motion estimation via Statistical Multi-Step Flow

Keywords: long-term motion estimation, dense point matching, statistical analysis, long-term trajectories, video editing

We present statistical multi-step flow, a new approach for dense motion estimation in long video sequences. Towards this goal, we propose a two-step framework including an initial dense motion candidates generation and a new iterative motion refinement stage. The first step performs a combinatorial integration of elementary optical flows combined with a statistical candidate displacement fields selection and focuses especially on reducing motion inconsistency. In the second step, the initial estimates are iteratively refined considering several motion candidates including candidates obtained from neighboring frames. For this refinement task, we introduce a new energy formulation which relies on strong temporal smoothness constraints. Experiments compare the proposed statistical multi-step flow approach to state-of-the-art methods through both quantitative assessment using the Flag benchmark dataset and qualitative assessment in the context of video editing.

INTRODUCTION

Dense motion estimation has known significant improvements since early works but deals mainly with matching consecutive frames. Resulting dense motion fields, called optical flows, can straightforwardly be concatenated to describe the trajectories of each pixel along the sequence [START_REF] Corpetti | Dense estimation of fluid flows[END_REF][START_REF] Brox | Object segmentation by long term analysis of point trajectories[END_REF][START_REF] Sundaram | Dense point trajectories by GPU-accelerated large displacement optical flow[END_REF]. However, both estimation and accumulation errors result in dense trajectories which can rapidly diverge and become inconsistent, especially for complex scenes including non-rigid deformations, large motion, zooming, poorly textured areas, illumination changes... Moreover, concatenating motion fields computed between consecutive frames does not allow to recover trajectories after temporary occlusions.

Recent works have contributed to the purpose of dense long-term motion estimation. Multi-frame optical flow formulations [START_REF] Salgado | Temporal constraints in large optical flow estimation[END_REF][START_REF] Papadakis | Dynamically consistent optical flow estimation[END_REF][START_REF] Werlberger | Anisotropic Huber-L1 optical flow[END_REF][START_REF] Volz | Modeling temporal coherence for optical flow[END_REF] have been presented but their temporal smoothness constraints are generally limited to a small number of frames. [START_REF] Sand | Particle video: Long-range motion estimation using point trajectories[END_REF] proposes a sophisticated framework to compute semidense trajectories using a particle representation but the full density is not achieved. To overcome these issues, Garg et al. describe in (Garg et al., 2013) a variational approach with subspace constraints to generate trajectories starting from a reference frame in a non-rigid context. They assume that the sequence of displacement of any point can be expressed as a linear combination of a low-rank motion basis. Therefore, trajectories are estimated assuming that they must lie close to this low dimensional subspace which implicitly acts as a long-term regularization. However, strong a-priori assumptions on scene contents must be provided and dense tracking of multiple objects is possible only if the reference frame is segmented.

The alternative concept of multi-step flow [START_REF] Crivelli | From optical flow to dense long term correspondences[END_REF][START_REF] Crivelli | Multi-Step Flow Fusion: Towards accurate and dense correspondences in long video shots[END_REF] focuses on how to construct dense fields of correspondences over extended time periods using multi-step optical flows (optical flows computed between consecutive frames or with larger inter-frame distances). Multi-step flow sequentially merges a set of displacement fields at each intermediate frame, up to the target frame. This set is obtained via concatenation of multi-step optical flows with displacement vectors already computed for neighbouring frames. Multi-step estimations can handle temporary occlusions since they can jump occluding objects. Contrary to [START_REF] Garg | A variational approach to video registration with subspace constraints[END_REF], multi-step flow considers both trajectory estimation between a reference frame and all the images of the sequence (from-the-reference) and motion estimation to match each image to the reference frame (to-the-reference).

Despite its ability to handle both scenarios, multistep flow has two main drawbacks. First, it performs the selection of displacement fields by relying only on classical optical flow assumptions that can sometimes fail between distant frames. Second, the candidate displacement fields are based on previous estimations. It ensures a certain temporal consistency but can also propagate estimation errors along the following frames of the sequence, until a new available step gives a chance to match with a correct location again.

These limitations can be resolved by extending to the whole sequence the combinatorial multi-step integration and the statistical selection described in [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF] for dense motion estimation between a pair of distant frames. The underlying idea is to first consider a large set composed of combinations of multi-step optical flows and then to study the spatial redundancy of the resulting candidates through a statistical selection to finally select the best matches.

Toward our goal of dense motion estimation in long video shots, we present the statistical multi-step flow two-step framework. First, it extends [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF] to generate several initial dense correspondences between the reference frame and each of the subsequent images independently. Second, we propose to provide an accurate final dense matching by applying a new iterative motion refinement which involves strong temporal smoothness constraints.

STATISTICAL MULTI-STEP FLOW

Let us consider a sequence of N + 1 RGB images {I n } n∈ [[0,...,N]] including I re f considered as a reference frame. In this work, we focus on dense motion estimation between the reference frame I re f and each frame I n of the sequence and we aim at computing from-the-reference and to-the-reference displacement fields. From-the-reference displacement fields link the reference frame I re f to the other frames I n and therefore describe the trajectory of each pixel of I re f along the sequence. To-the-reference displacement fields connect each pixel of I n to locations into I re f .

The proposed statistical multi-step flow performs two main stages. The generation of several initial dense motion correspondences for each pair of frames {I re f , I n } independently is described in Section 2.1. Section 2.2 presents the iterative motion refinement through strong temporal consistency constraints.

Initial motion candidates generation

The goal of the initial motion candidates generation is to compute for each pixel x re f (resp. x n ) of I re f (resp. I n ) K candidate positions in I n (resp. I re f ). Each pair of frames {I re f , I n } is processed independently. Our explanations focus on the estimation of from-thereference displacement fields. In the following, we describe the input data and recall the baseline method [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF] before focusing on how it has been improved and extended to the whole sequence.

Input optical flows fields

As inputs, our method considers a set of optical flow fields estimated from each frame of the sequence including I re f . These optical flows are previously estimated between consecutive frames or with larger This combinatorial integration [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF] is done independently for each pair {I re f , I n } which limits the correlation between candidates selected for neighbouring frames.

steps [START_REF] Crivelli | From optical flow to dense long term correspondences[END_REF], i.e. larger inter-frame distances. Let S n = {s 1 , s 2 , . . . , s Q n } ⊂ {1, . . . , N -n} be the set of Q n possible steps at instant n. The following set of optical flow fields starting from I n is therefore available: {v n,n+s 1 , v n,n+s 2 , . . . , v n,n+s Qn }.

Input optical flow fields are provided with attached occlusion and inconsistency masks. For the pair {I n , I n+s i } with s i ∈ {1, . . . , N -n}, the occlusion mask attached to the optical flow field v n,n+s i indicates the visibility of each pixel of I n in I n+s i . The inconsistency mask attached to v n,n+s i distinguishes consistent and inconsistent optical flow vectors among the ones starting from pixels marked as visible [START_REF] Robert | Disparity-compensated view synthesis for s3D content correction[END_REF]. This feature follows the idea that the backward flow should be the exact opposite of the forward flow.

Baseline method (Conze et al., 2013)

The combinatorial multi-step integration and the statistical selection on which we rely on work as follows.

For the current pair {I re f , I n }, the combinatorial multi-step integration consists in first of all considering all the possible from-the-reference motion paths which start from each pixel x re f , run through the sequence and end in I n . These motion paths are built by concatenating all the possible sequences of un-occluded input multi-step optical flow vectors between I re f and I n . A reasonable number of N s motion paths are then selected through limitations in terms of number of concatenations N c and via a guidedrandom selection. Each remaining motion path leads to a candidate position in I n (Fig. 1

top). Finally, we obtain a set T re f ,n (x re f ) = {x i n } i∈[[0,...,K x re f -1]] of K x re f candidate positions in I n for each pixel x re f of I re f .
A statistical-based selection stage then selects the optimal candidate position among T re f ,n (x re f ). This procedure involves: 1) a statistical criterion which pre-selects a small set of candidates based on spatial density and intrinsic inconsistency values; 2) a global optimization which fuses these candidates to obtain the optimal one while including spatial regularization.

Improvements

The combinatorial multi-step integration and the statistical selection we briefly reviewed has been improved to provide further focus to inconsistency reduction between from/to-the-reference vectors. First, we use only multi-step optical flow vectors considered as consistent according to their inconsistency masks to generate motion paths between I re f and I n . Second, we introduce an outlier removal step before the statistical selection which orders the candidates of T re f ,n (x re f ) with respect to their inconsistency values. A percentage R % of bad candidates is removed and the selection is performed on the remaining ones. Third, at the end of the combinatorial integration and the selection procedure between I re f and I n , the optimal displacement field is incorporated into the processing between I n and I re f which aims at enforcing the motion consistency between from/to-the-reference fields.

Compared to [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF], our displacement fields selection procedure combines differently statistical selection and global optimization. For each x re f ∈ I re f , we select among T re f ,n (x re f ) K sp = 2 × K candidates through statistical selection, with K sp < K x re f . Then, we randomly group by pairs these K sp candidates and choose the K best ones x k n ∀k ∈ [[0, . . . , K -1]] by pair-wise fusing them following a global flow fusion approach. Finally, this same global optimization method fuses these K best candidates to obtain an optimal one: x * n . In other words, these two last steps give a set of candidate displacement fields d k re f ,n and finally d * re f ,n , the optimal one. For pairs of frames relatively close or in case of temporary occlusions, the statistical selection is not adapted due to the small amount of candidates. Therefore, between K + 1 and K sp candidates, we use only the global optimization up to obtain the K best ones.

Our approach is applied bi-directionally. An exactly similar processing between I n and I re f leads to K initial to-the-reference candidate displacement fields.

Extention to the whole sequence

This improved version of the combinatorial integration and the statistical selection of [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF] processes independently all the pairs {I re f , I n }. Only N c , the maximum number of concatenations, changes with respect to the temporal distance between frames. In practice, N c is computed using Eq. ( 1) which leads to a good compromise between a too large number of concatenations which would lead to large propagation errors and the opposite situation which would limit the effectiveness of the statistical processing due to an insufficient number of candidates.

N c (n) = | n -re f | if | n -re f |≤ 5 α 0 . log 10(α 1 .|n -re f |) otherwise (1)
The guided-random selection [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF]) which selects for each pair of frames {I re f , I n } one part of all the possible motion paths limits the correlation between candidates respectively estimated for neighbouring frames. This avoids the situation in which a single estimation error is propagated and therefore badly influences the whole trajectory. The example Fig. 1 shows the motion paths selected by the guided-random selection for pairs {I re f , I n } and {I re f , I n+1 }. We notice that motion paths between I re f and I n+1 are not highly correlated with those between I re f and I n . Indeed, the sets of optical flow vectors involved in both cases are not the same except for v re f ,re f +1 and v re f ,n-1 which are then concatenated with different vectors. v n-2,n contributes for both cases but the considered vectors do not start from the same position. These considerations about the statistical independence of the resulting displacement fields are not addressed by existing methods for which a strong temporal correlation is inescapable.

Iterative motion refinement

The previous stage guarantees a low correlation between the initial motion candidates respectively estimated for pairs {I re f , I n }. Without losing this key characteristic, this second stage aims at iteratively refining the initial estimates while enforcing the temporal smoothness along the sequence.

We propose to question the matching between each pixel x re f (resp. x n ) of I re f (resp. I n ) and the selected position x * n (resp. x * re f ) in I n (resp. I re f ) established during the previous iteration (or the initial motion candidates generation stage if the current iteration is the first one). For this task, we generate several competing candidates which are compared to x * n (resp. x * re f ) through a global optimization approach.

Competing candidates

The competing candidates used to question x * n (resp. x * re f ) are illustrated in Fig. 2 and deals with:

• the K initial candidate positions x k n (resp. x k re f ) ∀k ∈ [[0, . . . , K -1]] (

Global optimization approach

We perform a global optimization method in order to fuse the previously described competing candidates into a single optimal displacement field.

In the from-the-reference case, we introduce L = {l x re f } as a labeling of pixels x re f where each label indicates x l x re f n , one of the candidates listed above. Let d l x re f re f ,n be the corresponding motion vector. We define the energy in Eq. ( 2) and minimize it with respect to L using fusion moves [START_REF] Lempitsky | Fusion moves for Markov random field optimization[END_REF]:

E re f ,n (L) = E d re f ,n (L) + E r re f ,n (L) = ∑ x re f ρ d (ε d re f ,n ) + ∑ x re f ,y re f α x re f ,y re f ρ r ( d l x re f re f ,n (x re f ) -d l y re f re f ,n (y re f ) 1 ) (2)
The data term E d re f ,n , described with more details in Eq. ( 3), involves both matching cost and inconsistency value with respect to d l x re f re f ,n [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF].

In addition, we propose to introduce strong temporal smoothness constraints into the energy formulation:

ε d re f ,n = C(x re f , d l x re f re f ,n (x re f )) + Inc(x re f , d l x re f re f ,n (x re f )) + n+ w 2 ∑ m=n-w 2 m =n C(x l x re f n , x * m -x l x re f n ) + ed m,n + ed n,m (3)
The temporal smoothness constraints translate into three new terms which are computed with respect to each neighbouring candidate x * m defined for the frames inside the temporal window W . These terms are illustrated in Fig. 3 and deal more precisely with:

• the matching cost between x

l x re f n ∈ I n and x * m of I m ,
• the euclidean distance ed m,n between x l x re f n and the ending point of the optical flow v m,n starting from x * m (see Eq. ( 4)). ed m,n encourages the selection of x m n , the candidate coming from I m via the optical flow field v m,n and therefore tends to strengthen the temporal smoothness. Indeed, for x m n , the euclidean distance ed m,n is equal to 0.

ed m,n = (x re f + d l x re f re f ,n ) -(x re f + d * re f ,m + v m,n ) 2 (4)
• the euclidean distance ed n,m between x * m and the ending point of the optical flow vector v n,m starting from x l x re f n (see Eq. ( 5)). If v m,n is consistent, i.e. v m,n ≈ -v n,m , ed n,m is approximately equal to 0 for x m n , the candidate coming from I m , whose selection is again promoted.

ed n,m = (x re f + d * re f ,m ) -(x re f + d l x re f re f ,n + v n,m ) 2 (5)
The regularization term E r re f ,n involves motion similarities with neighbouring positions, as shown in Eq. (2). α x re f ,y re f accounts for local color similarities in the reference frame I re f . The robust functions ρ d and ρ r are respectively the negative log of a Student-t distribution and the Geman-McClure function.

The refinement of to-the-reference displacement fields with our approach is straightforward except that the data term involves neither the matching cost between the current candidate and the temporal neighbouring one nor the euclidean distance ed m,n due to trajectories which can not be handled in this direction.

The global optimization method fuses the displacement fields by pairs and finally chooses to update or not the previous estimations with one of the previously described candidates. The motion refinement phase consists in applying this technique for each pair of frames {I re f , I n } in from-the-reference and to-the-reference directions. The pairs {I re f , I n } are processed in a random order in order to encourage temporal smoothness without introducing a sequential correlation between the resulting displacement fields.

This motion refinement phase is repeated iteratively N it times where one iteration corresponds to the processing of all the pairs {I re f , I n }. The proposed statistical multi-step flow is done once the initial motion candidates generation and the N it iterations of motion refinement have been performed.

EXPERIMENTS

Our experiments focus on the following sequences: MPI S1 [START_REF] Granados | MPI-S1[END_REF] Fig. 4 and 6a-h, Hope Fig. 6i-p, Newspaper Fig. 6q-t, Walking Couple Fig. 7 and Flag [START_REF] Garg | A variational approach to video registration with subspace constraints[END_REF]) Fig. 8. The proposed statistical multi-step flow is referred to as StatFlow in the following. For the experiments, the following parameters have been used:

N c = 7, N s = 100, R % = 50%, K = 3, α 0 = 3, α 1 = 15, w = 5.
The set of steps and input optical flow estimators will be specified for each experiment and each sequence.

Experiments have been conducted as follows. In Section 3.1, we evaluate the performance of our extended version of the combinatorial integration and the statistical selection [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF] through registration and PSNR assessment. The effects of the iterative motion refinement are also studied. Then, we compare StatFlow to state-of-the-art methods through quantitative assessment using the Flag dataset [START_REF] Garg | A variational approach to video registration with subspace constraints[END_REF]) (Section 3.2) and qualitative assessment via texture propagation and tracking (Section 3.3).

Registration and PSNR assessment

The first experiment aims at showing how the improvements we made with respect to [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF] impacts the quality of the displacement fields. We focus on frames pairs taken from MPI S1 and Newspaper (NP). The sets of steps are 1 -5, 10 (NP), 15 (MPI S1), 20 (NP) and 30 (NP). The algorithms are performed taking input multi-step optical flows computed with a version of the disparity estimator described in [START_REF] Robert | Disparity-compensated view synthesis for s3D content correction[END_REF], referred to as 2D-DE.

We compare the optimal displacement fields obtained in output of our initial motion estimates generation (Section 2.1) with those resulting from [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF]. The comparison is done through registration and PSNR assessment. For a given pair {I re f , I n }, the final fields are used to reconstruct I re f from I n through motion compensation and color PSNR scores are computed between I re f and the registered frame for non-occluded pixels. Tables 1 and2 show the PSNR scores for various distances between I re f and I n respectively on the kiosk of MPI S1 (Fig. 4) and on whole images of Newspaper (Fig. 6q-t). Results on MPI S1 show that the initial phase of StatFlow outperforms the combinatorial integration and the statistical selection of [START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF] for all pairs. An example of registration of the kiosk for a distance of 20 frames is given Fig. 4. Multi-step estimations deal satisfactorily with the temporary occlusion. Experiments on Newspaper reveal the same finding: the novelty in terms of inconsistency reduction improves the displacement fields quality. Moreover, the iterative motion refinement stage (N it = 9) allows to obtain better PSNR scores for all pairs compared to the initial stage of StatFlow.

Comparisons with Flag dataset

Quantitative results have been obtained using the dense ground-truth optical flow data provided by the Flag dataset [START_REF] Garg | A variational approach to video registration with subspace constraints[END_REF] for the Flag sequence (Fig. 8). Experiments focus on:

• direct estimation between each pair {I re f , I n } using LDOF [START_REF] Brox | Large displacement optical flow: descriptor matching in variational motion estimation[END_REF], ITV-L1 [START_REF] Wedel | An improved algorithm for TV-L1 optical flow[END_REF] and the keypoint-based nonrigid registration of [START_REF] Pizarro | Feature-based deformable surface detection with self-occlusion reasoning[END_REF]), • concatenation of optical flows computed between consecutive frames using LDOF (LDOF acc), ITV-L1 direct [START_REF] Wedel | An improved algorithm for TV-L1 optical flow[END_REF] 1.43

Table 3: RMS endpoint errors for different methods on the Flag benchmark dataset [START_REF] Garg | A variational approach to video registration with subspace constraints[END_REF].

• multi-frame subspace flow (MFSF) [START_REF] Garg | A variational approach to video registration with subspace constraints[END_REF] using PCA or DCT basis,

• multi-step flow fusion (MSF) [START_REF] Crivelli | Multi-Step Flow Fusion: Towards accurate and dense correspondences in long video shots[END_REF] with LDOF multi-step optical flows,

• StatFlow (N it = 3) with LDOF optical flows.

For the comparison task, Tab. 3 gives for all the previously described methods the RMS (root mean square) endpoint errors between the respective obtained displacement fields and the ground-truth data. RMS errors are estimated for all the foreground pixels and for all the pairs of frames {I re f , I n } together. RMS errors computed for each pair of frames are shown in Fig. 5 for all the methods based on LDOF: LDOF direct, LDOF acc, MSF (LDOF) and StatFlow (LDOF).

The last two multi-step strategies have considered as inputs steps 1 -5, 8, 10, 15, 20, 25, 30, 40 and 50. We can firstly observe that LDOF acc rapidly diverge. This is due to both estimation errors which are propagated along trajectories and accumulation errors inherent to the interpolation process. Moreover, the results obtained through direct motion estimation are reasonably good, especially for [START_REF] Pizarro | Feature-based deformable surface detection with self-occlusion reasoning[END_REF]. LDOF direct gives a lower RMS endpoint error than LDOF acc (1.74 against 4). However, it is not possible to draw conclusions in the light of the Flag sequence because the flag comes back approximately to its initial position at the end of the sequence (Fig. 8a,g). Motion estimation for complex scenes cannot generally rely only on direct estimation and combining optical flow accumulations and direct matching is clearly a more suitable strategy. Tab. 3 and Fig. 5 prove that with the same optical flows as inputs, StatFlow shows a clear improvement compared to MSF (0.69 against 1.41). Although both methods achieve the same quality for first pairs or for some pairs which coincide with existing steps, other displacement fields are computed with a better accuracy using StatFlow. Moreover, StatFlow(LDOF) reaches the same RMS error with respect to MFSF-PCA, the best one of the MFSF approaches, with 0.69. This proves that StatFlow is competitive compared to challenging state-of-the-art methods.

Texture propagation and tracking

We aim now at showing that our method provides satisfying results in a wide set of complex scenes. Moreover, we focus on the comparison between StatFlow (N it = 9) and MSF [START_REF] Crivelli | Multi-Step Flow Fusion: Towards accurate and dense correspondences in long video shots[END_REF] to prove that StatFlow performs a more efficient integration and selection procedure compared to MSF using the same optical flows as inputs. Experiments have been firstly conducted in the context of video editing: we evaluate the accuracy of both methods by motion compensating in I n ∀n textures/logos manually inserted in I re f . In chosen for its good results for video editing tasks. The steps involved are: 1 -5, 8 (Hope), 10, 15 (except for NP), 20 (Hope, NP), 30 (MPI S1, NP).

Given these results, it appears that MSF sometimes distorts structures (bottom left zoom Fig. 6ce, Fig. 6l,m), makes shadow textures appear (bottom right zoom Fig. 6c-e) and does not estimate motion with accuracy (top right zoom Fig. 6e, Fig. 6l,m). Visual results with StatFlow reveal a better longterm propagation (see also Fig. 6r-t). Fig. 7 compares StatFlow(2D-DE) and MSF(2D-DE) with LDOF acc. We observe that LDOF acc badly performs motion estimation for periodic structures. MSF encounters also matching issues (Fig. 7h) whereas StatFlow performs propagation without any visible artifacts.

Finally, StatFlow and MSF are assessed through point tracking. In Fig. 9, the bottom right part of the woman face is tracked from I 115 to I 138 (MPI S1). The 2D+t visualization indicates that some trajectories drift to the background with MSF. This illustrates the inherent issue of MSF which propagates estimation errors due to the sequential processing. Conversely, StatFlow provides accurate fields while limiting the temporal correlation between displacement fields respectively estimated for neighbouring frames.

CONCLUSION

We present statistical multi-step flow, a two-step framework which performs dense long-term motion estimation. Our method starts by generating initial dense correspondences with a focus on inconsistency reduction. For this task, we perform a combinatorial integration of consistent optical flows followed by an efficient statistical selection. This procedure is applied independently between a reference frame and each frame of the sequence. It guarantees a low temporal correlation between the resulting correspondences respectively estimated for each of these pairs. We propose then to enforce temporal smoothness through a new iterative motion refinement. It considers several motion candidates including candidates from neighboring frames and involves a new energy formulation with temporal smoothness constraints. Experiments evaluate the effectiveness of our approach compared to state-of-the-art methods through quantitative assessment using dense groundtruth data and qualitative assessment via texture propagation and tracking for a wide set of complex scenes.

Figure 1 :

 1 Figure 1: Multiple motion candidates are generated via a guided-random selection among all possible motion paths.This combinatorial integration[START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF] is done independently for each pair {I re f , I n } which limits the correlation between candidates selected for neighbouring frames.

Figure 2 :

 2 Figure 2: The displacement field d * re f ,n is questionned by generating for each pixel x re f competing candidates in I n .

Figure 3 :

 3 Figure 3: Matching cost and Euclidean distances ed n,m and ed m,n defined with respect to each temporal neighboring candidate x * m and involved in the proposed energy. These three terms act as strong temporal smoothness constraints.

Figure 4 :

 4 Figure 4: Source frames of the MPI S1 sequence (Granados et al., 2012) and reconstruction of the kiosk of I 25 from I 45 with: e) the combinatorial integration and the statistical selection introduced in (Conze et al., 2013), f) the proposed extended version described in Section 2.1 (initial phase of StatFlow). Black boxes focus on differences between both methods.Method RMS endpoint error (pixels) StatFlow (LDOF) 0.69 MSF (Crivelli et al., 2012a) (LDOF) 1.41 LDOF direct (Brox and Malik, 2011) 1.74 LDOF acc (Brox and Malik, 2011) 4 MFSF-PCA (Garg et al., 2013) 0.69 MFSF-DCT (Garg et al., 2013) 0.80 (Pizarro and Bartoli, 2012) direct 1.24

Figure 5 :

 5 Figure 5: RMS endpoint errors for each pair {I re f , I n } along Flag sequence (Fig. 8) with different methods.

  Figure 6: Texture/logo insertion in I 115 (resp. I 5036 and I 230 ) and propagation along the MPI-S1 (resp. Hope and Newspaper) sequence up to I 137 (resp. I 5063 and I 170 ) using: 1) multi-step flow fusion (MSF) (Crivelli et al., 2012a) with multi-step optical flow fields from (Robert et al., 2012) (2D-DE): MSF(2D-DE); 2) the proposed statistical multi-step flow (StatFlow) with 2D-DE multi-step optical flow fields: StatFlow (2D-DE).

Figure 7 :

 7 Figure 7: Texture insertion in I 0 and propagation up to I 40 (Walking Couple sequence). We compare: d-f) concatenation of LDOF (Brox and Malik, 2011) optical flow fields computed between consecutive frames (LFOF acc); g-i) multi-step flow fusion (MSF) (Crivelli et al., 2012a) using multi-step optical flow fields from (Robert et al., 2012) (2D-DE); j-l) the proposed statistical multi-step flow (StatFlow) using 2D-DE multi-step optical flow fields.

Figure 9 :

 9 Figure 9: Point tracking from I 115 up to I 138 , MPI-S1 sequence (Granados et al., 2012). We compare: b) multi-step flow fusion (MSF) (Crivelli et al., 2012a) using multi-step optical flow fields from (Robert et al., 2012) (2D-DE); c) the proposed statistical multi-step flow (StatFlow) method using 2D-DE multi-step optical flow fields.

  obtained Section 2.1), • candidates from neighbouring frames to enforce temporal smoothing. Let W be the temporal window of width w centered around I n . Between I re f and I n , we use the optical flow fields v m,n between I

	• a candidate position coming from the previous es-timation of d * n,re f (resp. d * re f ,n ) which is inverted to obtain x r n (resp. x r re f ), as illustrated in Fig. 2,

m and I n with m ∈ [[n -w 2 , . . . , n + w 2 ]] and m = n to obtain from x * m ∈ I m the new candidate x m n in I n .

Table 1 :

 1 Registration and PSNR assessment with the combinatorial integration and the statistical selection introduced in[START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF] and the proposed extended version described in Section 2.1 (initial phase of StatFlow). PSNR scores are computed on the kiosk of MPI S1 (Fig.4).

	Frame pairs	{25,45} {25,46} {25,47} {25,48}
	(Conze et al., 2013)	21.83	24.98	25.56	25.83
	StatFlow initial phase	29.02	28.4	27.27	27.23
	Frame pairs	{25,49} {25,50} {25,51} {25,52}
	(Conze et al., 2013)	25.04	24.83	24.48	24.3
	StatFlow initial phase	26.84	26.33	26.1	25.69
	Frame pairs	{160,180} {160,190} {160,200}
	(Conze et al., 2013)	22.50	21.21		18.59
	StatFlow initial phase	22.70	21.39		19.28
	StatFlow	22.93	22.18		20.25
	Frame pairs	{160,210} {160,220} {160,230}
	(Conze et al., 2013)	17.12	15.87		15.76
	StatFlow initial phase	18.21	17.12		16.58
	StatFlow	18.68	17.40		16.81

Table 2 :

 2 Registration and PSNR assessment with: 1) combinatorial integration and statistical selection introduced in[START_REF] Conze | Dense motion estimation between distant frames: combinatorial multi-step integration and statistical selection[END_REF], 2) proposed extended version (Stat-Flow init. phase), 3) whole StatFlow method. PSNR scores are computed on whole images of Newspaper (Fig.6q-t).