Yizhen Lao

Omar Ait-Aider

O Ait

Rolling Shutter Homography and its Applications

Keywords: Rolling shutter, Homography, Relative Pose estimation, Image stitching. !

In this article we study the adaptation of the concept of homography to Rolling Shutter (RS) images. This extension has never been clearly adressed despite the many roles played by the homography matrix in multi-view geometry. We first show that a direct point-to-point relationship on a RS pair can be expressed as a set of 3 to 8 atomic 3x3 matrices depending on the kinematic model used for the instantaneous-motion during image acquisition. We call this group of matrices the RS Homography. We then propose linear solvers for the computation of these matrices using point correspondences. Finally, we derive linear and closed form solutions for two famous problems in computer vision in the case of RS images: image stitching and plane-based relative pose computation. Extensive experiments with both synthetic and real data from public benchmarks show that the proposed methods outperform state-of-art techniques.

INTRODUCTION

A large number of consumer CMOS cameras are equipped with RS sensors because of their low-cost, low-energy-consumption, high frame rate and less-background-noise [START_REF] Linkemann | Global shutter, rolling shutter-functionality and characteristics of two exposure methods (shutter variants)[END_REF]. In this acquisition mode, image pixels are exposed row by row commonly from the top to the bottom. Thus, when the camera is moving during the acquisition, the exposure delay between the lines leads to visual distortions called RS effects, such as on Fig. 2(a).

Ignoring the effects of RS in computer vision applications results in performance degradation or even failure [START_REF] Albl | Rolling shutter camera absolute pose[END_REF], [START_REF] Dai | Rolling shutter camera relative pose: generalized epipolar geometry[END_REF]. Over the last decade, several works have revisited 3D computer vision by taking RS into account such as RS effects removal [START_REF] Forssén | Rectifying rolling shutter video from hand-held devices[END_REF], [START_REF] Rengarajan | From bows to arrows: Rolling shutter rectification of urban scenes[END_REF], [START_REF] Purkait | Rolling shutter correction in manhattan world[END_REF], [START_REF] Lao | A robust method for strong rolling shutter effects correction using lines with automatic feature selection[END_REF], absolute pose estimation [START_REF] Albl | Rolling shutter camera absolute pose[END_REF], [START_REF] Saurer | A minimal solution to the rolling shutter pose estimation problem[END_REF], [START_REF] Ait-Aider | Simultaneous object pose and velocity computation using a single view from a rolling shutter camera[END_REF], [START_REF] Lao | Rolling shutter pose and egomotion estimation using shape-from-template[END_REF], epipolar geometry [START_REF] Dai | Rolling shutter camera relative pose: generalized epipolar geometry[END_REF], [START_REF] Saurer | Rolling Shutter Stereo[END_REF] and Structure from motion (SfM) [START_REF] Saurer | Sparse to dense 3d reconstruction from rolling shutter images[END_REF], [START_REF] Ait-Aider | Structure and kinematics triangulation with a rolling shutter stereo rig[END_REF], [START_REF] Hedborg | Rolling shutter bundle adjustment[END_REF], [START_REF] Albl | Degeneracies in rolling shutter sfm[END_REF], [START_REF] Zhuang | Rolling-shutter-aware differential sfm and image rectification[END_REF], [START_REF] Lao | Robustified structure from motion with rolling-shutter camera using straightness constraint[END_REF].

Related Works and Motivation

Estimating the camera motion by using points is one of the most studied minimal problems in computer vision. For example, with Global Shutter (GS), at least 3 point matches are needed to estimate the absolute pose [START_REF] Gao | Complete solution classification for the perspective-three-point problem[END_REF], while at least 5 are needed to recover the relative pose between two calibrated GS views [START_REF] Nistér | An efficient solution to the five-point relative pose problem[END_REF]. Given the higher complexity of RS projection model [START_REF] Meingast | Geometric models of rollingshutter cameras[END_REF], more points are commonly needed. Methods for structure and motion estimation with RS images can be grouped into two categories: optical flow and epipolar geometry.

Optical flow methods: In [START_REF] Zhuang | Rolling-shutter-aware differential sfm and image rectification[END_REF], 8pt and 9pt linear solvers were developed to recover the relative pose of a RS camera that undergoes constant velocity or constant acceleration motion. Unfortunately, as shown in Fig. 1, consistency between the camera poses and their motion only holds with high-frame rates and smooth movements. In addition to the resulting high computation load, unordered image sets can not be processed. Besides, inter-frame delay has to be exactly pre-calibrated. Fig. 1: (a) Video-based (frame-by-frame processing) methods assume smooth [START_REF] Forssén | Rectifying rolling shutter video from hand-held devices[END_REF], [START_REF] Grundmann | Calibration-free rolling shutter removal[END_REF] or even constant velocity [START_REF] Zhuang | Rolling-shutter-aware differential sfm and image rectification[END_REF] between each two consecutive frames. However, for a general unordered set of images (b), it is hard or impossible to enforce the relative poses and their instantaneous-motions basing on these assumptions.

Epipolar geometry: In multi-view reconstruction, many common configurations become critical with RS cameras and lead to reconstruction ambiguities. Authors in [START_REF] Albl | Degeneracies in rolling shutter sfm[END_REF] provide mathematical analysis for one, two and some multiview configurations. They provide practical recipes on how to photograph with RS cameras to avoid reconstruction errors. This method can be used to unblock some situations but it is not a solution to the standard SfM problem. Authors in [START_REF] Dai | Rolling shutter camera relative pose: generalized epipolar geometry[END_REF] introduce 20pt and 44pt linear solvers for pure translational and uniform motion models respectively. However, the pure translational motion assumption is not feasible to model the camera motion in most of practical applications. Although more general, the 44-point solution requires too many correspondences and is therefore not suitable for use with RANSAC (Random Sample Consensus).

Parametrised homographies: Authors in [START_REF] Forssén | Rectifying rolling shutter video from hand-held devices[END_REF] addressed the rectification and stabilization problem of RS videos by using a sequence of parametrised homographies (one for each image row pair from two consecutive frames). Camera poses are estimated for last Fig. 2: RS images (a). Stitching results obtained with well-known commercial stitching applications such as AutoStitch [START_REF] Brown | Automatic panoramic image stitching using invariant features[END_REF] (b) Microsoft Image Composite Editor (ICE) [23] (c) Adobe Photoshop [24] (d) state-of-the-art multiple homographies stitching method APAP [START_REF] Zaragoza | As-projective-aspossible image stitching with moving dlt[END_REF] (e) and AANAP [START_REF] Lin | Adaptive as-natural-as-possible image stitching[END_REF] (f). The stitching results and the correction of the RS effects obtained with the the proposed method are shown in (g) and (h). rows of each frame. Intermediate translations and rotations are interpolated from these. The camera motion is parametrised as a continuous curve. Curve parameters are solved for using nonlinear least squares over inter-frame correspondences obtained by feature tracking. Thus, for general unordered set of images with large baselines, it is impossible for this method to enforce relationships between the camera poses and their motion (Fig. 1(b)). Besides, as in [START_REF] Zhuang | Rolling-shutter-aware differential sfm and image rectification[END_REF], pre-calibration of the inter-frame delay is required. Similarly, authors in [START_REF] Vasu | Occlusion-aware rolling shutter rectification of 3d scenes[END_REF] try to solve the RS correction problem by building multiple independent homographies between each of image rows. However, this method also requires multiple successive and smooth-motion frames as input since the use of a key-row interpolation approach [START_REF] Forssén | Rectifying rolling shutter video from hand-held devices[END_REF]. Besides, only iterative nonlinear solution is provided for motion estimation.

One of the closest work to ours is [START_REF] Grundmann | Calibration-free rolling shutter removal[END_REF], which employs a homography mixture and proposes a linear solution to estimate inter-frame motion. However, The major issue of this method is that points on a given row (row-block) in the first image have to be matched with points which also belong to a row (row-block) in the second image. This obviously limits the number of matches among unordered images even with small inter-frame motion.

In summary, as shown in Fig. 1, estimating relative poses from general unordered RS image sets remains an open problem and there is a need for new methods which require less input data (i.e. number of matches) and which work for various camera configurations. With some acceptable constraints on the scene structure or on the camera motion, homography could be used instead of epipolar geometry to recover the relative pose with less point matches [START_REF] Hartley | Multiple view geometry in computer vision[END_REF], [START_REF] Zhou | Robust plane-based structure from motion[END_REF], [START_REF] Saurer | Homography based egomotion estimation with a common direction[END_REF]. It has many applications such as image registration and plane-based camera relative pose estimation. However, homography-based methods have not been fully studied and adapted to the RS case.

In this paper, we show that the relationship between points on a RS pair can be expressed as a set of 3 to 7 atomic 3 × 3 matrices, depending on the kinematic model used for the instantaneousmotion during image acquisition. These matrices establishes a direct point-to-point transformation similarly to the homography matrix in the GS case. We call this set of matrices RS Homography. We also propose two linear solvers for the computation of this RS Homography using point correspondences. We first propose a theoretical 36pt linear solution and then derive a practical 13.5pt linear solver that gives good estimates of the homography between two RS views. We also investigate the use of the proposed method for two major computer vision applications:

(1) Plane-based Relative Pose Estimation: Although the RS relative pose problem has been addressed before [START_REF] Dai | Rolling shutter camera relative pose: generalized epipolar geometry[END_REF], [START_REF] Zhuang | Rolling-shutter-aware differential sfm and image rectification[END_REF], a more efficient and robust solution which handles planar scenes is proposed by using RS homography in this paper.

(2) Image Stitching: Users nowadays frequently create panoramas by rotating RS cameras, e.g. 'Pano' mode in iPhone. Besides, 360 VR images such as Street View could also be taken with RS cameras installed on a moving platform such as the Google car [START_REF] Klingner | Street view motion-fromstructure-from-motion[END_REF]. As shown in Fig. 2(b-e), the most well-known commercial stitching software or state-of-the-art methods, which are based on the GS model, lead to poor results in presence of RS effects. Therefore, designing a RS stitching method by using RS Homography will be of valuable significance.

Paper Organization and Contributions

In this paper, we first introduce the RS instantaneous-motion and projection models in section 2. Then we derive the full and the simplified RS homography matrices in section 3 followed by the solutions to estimate these matrices from point matches in section 4. Next we show how to estimate the RS relative pose based on a planar scene in section 5 and also RS image stitching by using RS homography in section 6. Finally, we present experimental results of the proposed methods in section 8.

The contributions of this paper can be summarized as follows:

• This is the first work to investigate the theoretical RS homography matrices which can be expressed as a unique set of 3 to 8 atomic 3 × 3 matrices, depending on the kinematic model used for instantaneous-motion during image acquisition.

•

We develop practical linear solvers for RS Homography matrices computation from two RS views with possibly large baseline (no constraint is done on the interfframe motion, the kinematic model concerns only the instantaneous-motion).

• By using the RS Homography, we develop a linear solution for the plane-based relative pose estimation problem. We also introduce a solution to RS image stitching which not only can correctly align RS images (Fig. 2(g)) but also removes the RS effects at the same time (Fig. 2(h)).

ROLLING SHUTTER PROJECTION MODEL

RS Projection Model. Let a 3D point P i = [x, y, z] (w.r.t to the world coordinate system) be projected into a calibrated RS camera as an image point q i = [u i , v i , 1] :

s i K -1 q i = R i P + t i (1)
where, s i is the scene depth of the corresponding camera. [R i , t i ] is the camera pose when scanline (row) v i is taken. K is the camera calibration matrix. For the readability, image points q are pre-multiplied by K -1 in the rest of the paper. We use the linearized uniform model to constrain the instantaneous-motion of a RS camera during the frame acquisition, which has been justified in previous works [START_REF] Dai | Rolling shutter camera relative pose: generalized epipolar geometry[END_REF], [START_REF] Albl | Degeneracies in rolling shutter sfm[END_REF], [START_REF] Zhuang | Rolling-shutter-aware differential sfm and image rectification[END_REF]. Thus, given the angular velocity ω = [ω x , ω y , ω z ] and the translational velocity d = [d x , d y , d z ] , we can express the camera pose [R i |t i ] at row v i as:

R vi = (I + [ω] × v i )R 0 t vi = t 0 + dv i ( 2
)
where I is the 3 × 3 identity matrix, [ω] × is a skew-symmetric matrix produced by ω, and [R 0 |t 0 ] is the camera pose of the first row.

Rolling Shutter Relative Pose. Let us consider n 3D points

P i , i ∈ [1, n] imaged from two RS views with the camera poses [R vi |t vi ] and [R v i |t v i ] as q i = [u i , v i , 1] and q i = [u i , v i , 1]
in the two images respectively. [I|0] and [R 0 |t 0 ] are the camera poses of the first row of each image. We assume that P i in world coordinates can also be expressed as P vi and P v i in the two camera coordinate systems. The transformations are written as:

P vi = R vi P + t vi (3)
and

P v i = R v i P + t v i (4)
By substituting Eq. 3 into Eq. 4 and eliminating P, we obtain the transformation between P vi and P v i as:

P v i = R v i P + t v i = R v i (R vi (P vi -t vi )) + t v i = R v i R vi Ri P vi + t v i -R v i R vi t vi ti (5)
Thus, the rotation R i and the translation t i between the the row v i in the first image and the row v i in the second image are:

R i = (I + [ω 2 ] × v i )R 0 (I -[ω 1 ] × v i ) t i = t 0 + d 2 v i -(I + [ω 2 ] × v i )R 0 (I -[ω 1 ] × v i )d 1 v i (6)
where {ω 1 , d 1 } and {ω 2 , d 2 } are instantaneous-motion parameters of the two RS views.

RS HOMOGRAPHY

GS Homography Matrix

Let us assume that a planar object (a plane) is observed from two GS cameras with the poses [I|0] and [R 0 |t 0 ]. The transformation between the two corresponding image points q i and q i can be written as:

α i q i = H GS q i = (R 0 - t 0 n d )q i , α i = z /z (7)
where α i is a scale factor that depends on the depth of P i in each camera. H GS is the GS homography matrix, n is the normal vector of the observed plane and d is the distance from the first camera to the plane under the constraint n P i + d = 0.

RS Homography Matrix

Expressions of the normal vector and the distance to plane in an RS frame. When instantaneous-motion occurs during the acquisition, H between two RS cameras varies with different row combinations. The relative pose between the row v i in the first image and the row v i in the second image is defined in Eq. ( 6). However, due to the instantaneous-motion of the first view, the normal vector of the plane and the distance to the plane are also changing dynamically with different scanlines (rows). Firstly, we consider an RS camera is at the pose of its first row, then the plane constraint is:

n 0 P i + d 0 = 0 (8)
By substituting the transform from P i to P vi in Eq. (3) into Eq. ( 8), we obtain:

n 0 (R vi (P vi -t vi )) + d 0 = n 0 R vi n v i P vi + d 0 -n 0 R vi t vi dv i = 0 (9)
Then by substituting linear instantaneous-motion model Eq. ( 2) into Eq. ( 9), we can finally express the normal vector of the plane and distance to the plane w.r.t row v i as:

n i = n 0 (I -[ω 1 ] × v i ) d i = d 0 -n 0 (I -[ω 1 ] × v i )d 1 v i ( 10
)
where n 0 and d 0 are the normal vector and the distance for the first row.

Derivation of RS Homography matrices. For convenience, we denote R i , t i in Eq. ( 6) and n i , d i in Eq. ( 10) as:

R i = R 0 + R 1 v i + R 2 v i + R 3 v i v i t i = t 0 + t 1 v i + t 2 v i + t 3 v 2 i + t 4 v i v i + t 5 v 2 i v i - n i d i ≈ - n 0 -n 0 [ω 1 ] × v i d 0 = N 0 + N 1 v i (11)
where, the auxiliary variables are defined as:

       R 0 = R 0 R 1 = -R 0 [ω 1 ] × R 2 = [ω 2 ] × R 0 R 3 = -[ω 2 ] × R 0 [ω 1 ] ×                t 0 = t 0 t 1 = -R 0 d 1 t 2 = d 2 t 3 = R 0 [ω 1 ] × d 1 t 4 = -[ω 2 ] × R 0 d 1 t 5 = [ω 2 ] × R 0 [ω 1 ] × d 1    N 0 = - n 0 d0 N 1 = n 0 [ω1]× d0
Then, by substituting Eq. ( 11) into Eq. ( 7), we can obtain:

H RS,i = R i - t i n i d i = (R 0 + R 1 v i + R 2 v i + R 3 v i v i ) + (t 0 + t 1 v i + t 2 v i + t 3 v 2 i + t 4 v i v i + t 5 v 2 i v i ) (N 0 + N 1 v i ) = (R 0 + t 0 N 0 ) H GS + (R 1 + t 1 N 0 + t 0 N 1 ) H1 v i + (R 2 + t 2 N 0 ) H2 v i + (R 3 + t 4 N 0 + t 2 N 1 ) H3 v i v i + (t 3 N 0 + t 1 N 1 ) H4 v 2 i + (t 5 N 0 + t 4 N 1 ) H5 v 2 i v i + (t 3 N 1 ) H6 v 3 i + (t 5 N 1 ) H7 v 3 i v i (12
) thus, we can rewrite Eq. ( 12) in a simplified form by using eight 3 × 3 atomic matrices H GS , H 1 ...H 7 as:

H RS,i =H GS + H 1 v i + H 2 v i + H 3 v i v i + H 4 v 2 i + H 5 v 2 i v i + H 6 v 3 i + H 7 v 3 i v i (13)
We draw attention to the fact that despite the presence of row indexes v i and v i in the expression of H RS , matrices H GS and H 1 to H 7 do not depend on these indexes and are unique for all the point matches of any image pair.

Simplified RS Homography Matrix

Approximation of RS relative pose. Under the small rotation assumption, the second order and higher terms in Eq. ( 6) can be ignored. This simplification method is also used in [START_REF] Albl | Rolling shutter camera absolute pose[END_REF], [START_REF] Lao | Robustified structure from motion with rolling-shutter camera using straightness constraint[END_REF], [START_REF] Ito | Self-calibration-based approach to critical motion sequences of rolling-shutter structure from motion[END_REF], [START_REF] Purkait | Minimal solvers for monocular rolling shutter compensation under ackermann motion[END_REF]. This approximation can be justified in that we force the translational speed vectors d 1 and d 2 to be constant in the world coordinate system, which is physically coherent with the constant velocity kinematic model. Therefore, we obtain an approximate expression of RS relative pose: Approximation of the plane pose. In practice, since the translation during acquisition is commonly much smaller than the distance from the camera to the scene plane, we can ignore the terms affected by translational velocities. In addition, we drop the second order terms, and obtain the approximate expressions:

R i = R 0 -R 0 [ω 1 ] × v i + [ω 2 ] × R 0 v i t i = t 0 + d 2 v i -R 0 d 1 v i (14)
n i = n 0 (I -[ω 1 ] × v i ) d i = d 0 -n 0 d 1 v i ≈ d 0 (15)
Using both approximations in Eq. ( 14) and ( 15), the RS homography matrix H RS between the row v i and the row v i in the two images can be simplified:

H RS,i = H GS + A 1 v i + A 2 v i where, A 1 = -R 0 [ω 1 ] × + R 0 d 1 n 0 d 0 + t 0 n 0 [ω 1 ] × d 0 A 2 = [ω 2 ] × R 0 - d 2 n 0 d 0 (16)
where A 1 and A 2 are two atomic matrices. Note that RS Homography consists of the GS homography matrix H GS defined in Eq. ( 7) and of the two matrices A 1 , A 2 which contain the instantaneous-motion parameters.

RS HOMOGRAPHY ESTIMATION

Now, we show how to linearly estimate the RS homographies defined in Eq. ( 13) and ( 16).

4pt GS Homography Estimation

The GS homography matrix is usually computed using the Direct Linear Transform algorithm (DLT). Based on Eq. ( 7), the cross product is given by q vi × H GS q i = 0. This gives two linearly independent equations:

0 -q i v i q i q i 0 -u i q i    H GS,(1) H GS,(2) H GS,(3)    = L i h GS = 0 ( 17
)
where 0 = [0, 0, 0]. Given n point correspondences (n 4), we obtain a system in the form Lh GS where L is a 2n × 9 matrix. The solution is then the singular vector associated to the smallest singular value of L.

36pt Full RS Homography Matrix Estimation

In the RS case, the computation is less straightforward. By substituting Eq. ( 13) into h GS , Eq. ( 17) can be rewritten as follows:

M RS,i h RS =                                              0 q i -q i 0 v i q i -u i q i 0 v i q i -v i q i 0 v i v i q i -v i u i q i 0 v i q i -v i q i 0 v i 2 q i -u i v i q i 0 v i v i q i -v i v i q i 0 v i v i 2 q -u i v i v i q i 0 v i 2 q i -v i 2 q i 0 v i 2 v i q i -u i v i 2 q i 0 v i 2 v i q -v i 2 v i q i 0 v i 2 v i 2 q i -u i v i 2 v i q i 0 v i 3 q i -v i 3 q i 0 v i 3 v i q -u i v i 3 q i 0 v i 3 v i q i -v i 3 v i q i 0 v i 3 v i 2 q i -u i v i 3 v i q i                                                                                               H GS,(1) H GS,(2) H GS,(3) H 1, (1
                                                 = 0 0 (18)
where H (i) , A 1,(i) and A 2,(i) are the i th rows of H GS , A 1 and A 2 respectively. M RS,i is a 2 × 72 matrix, which consists of the terms of u i , v i and q i . h RS is a 72 × 1 vector that contains the components of H GS , H 1 . . . H 7 .

Each point correspondence gives two constraints such as Eq. [START_REF] Gao | Complete solution classification for the perspective-three-point problem[END_REF]. Given n point correspondences (n 36), we obtain a system in the form M RS h RS = [0, . . . , 0] where M is a 2n × 72 matrix. This system is solved using SVD.

13.5pt Simplified RS Homography Estimation

In the simplified case of section. 3.3, by substituting Eq. ( 16) into h GS , Eq. ( 17) can be rewritten as:

              0 q i -q i 0 v i q i -u i q i 0 v i q i -v i q i 0 v i v i q i -v i u i q i 0 v i q i -v i q i 0 v i 2 q i -u i v i q i                               H GS,(1) H GS,(2) H GS,(3) A 1,(1) A 1,(2) A 1,(3) A 2,(1) A 2,(2) A 2,(3)                 = M RS,i h RS = 0 0 (19)
where M RS,i reduces to a 2 × 27 matrix and h RS,i is a 27 × 1 vector with 27 unknowns. Thus, with at least 14 point correspondences, we can estimate h RS linearly by using SVD. In order to obtain stable results, we perform a normalization of M RS in the way explained in [START_REF] Dai | Rolling shutter camera relative pose: generalized epipolar geometry[END_REF], [START_REF] Hartley | Multiple view geometry in computer vision[END_REF].

We extend the standard RANSAC pipeline [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF] for homography verification [START_REF] Brown | Automatic panoramic image stitching using invariant features[END_REF] with the proposed 13.5pt linear solver to obtain a robust estimate of the RS homography matrix and matching inliers.

PLANE-BASED RS RELATIVE POSE AND INSTANTANEOUS-MOTIONS ESTIMATION

The 36pt solver requires a large amount of point matches and therefore it is not a feasible solution for RANSAC. However, using the simplified RS homography brings inaccuracy.

Thus, we suggest the following pipeline: 1) Solve the RS homography matrix by estimating the atomic matrices (H GS , A 1 and A 2 ) with the 13.5pt solver. 2) Linearly extract the relative pose and instantaneous-motions. 3) Refine all the parameters in a nonlinear optimization based on the full RS Homography.

Computing Relative Pose and Instantaneousmotions

Relative pose [R 0 |t 0 ] and plane normal vector n 0 . Once H GS is known, it can be decomposed into R 0 , t 0 and n 0 by using SVD. d 0 is set as 1 and absorbed by t 0 . Generally, this decomposition yields four solutions, where only at most two are physically valid. The positive depth constraint can be used to find the final solution [START_REF] Ma | An invitation to 3-d vision: from images to geometric models[END_REF], [START_REF] Malis | Deeper understanding of the homography decomposition for vision-based control[END_REF].

Instantaneous-motions. We can further retrieve instantaneousmotion parameters thanks to two linear equation systems derived from matrices A 1 and A 2 :

(1) First we compute ω 1 = {ω x 1 , ω y 1 , ω z 1 } and d 1 = {d x 1 , d y 1 , d z 1 } by using the previously computed values of [R 0 |t 0 ] and n 0 in matrix A 1 (6 unknowns with 9 equations): Based on the definition of A 1 in Eq. ( 16), we obtain the following linear system in ω 1 and d 1 (detailed derivation in supplemental materials):

              0 -G 13 G 12 n x 0 R 0,(1) G 13 0 -G 11 n y 0 R 0,(1) -G 12 G 11 0 n z 0 R 0,(1) 0 -G 23 G 22 n x 0 R 0,(2) G 23 0 -G 21 n y 0 R 0,(2) -G 22 G 21 0 n z 0 R 0,(2) 0 -G 33 G 32 n x 0 R 0,(3) G 33 0 -G 31 n y 0 R 0,(3) -G 32 G 31 0 n z 0 R 0,(3)                       ω x 1 ω y 1 ω z 1 d x 1 d y 1 d z 1         =               A 1,11 A 1,12 A 1,13 A 1,21 A 1,22 A 1,23 A 1,31 A 1,32 A 1,33               (20
) where the auxiliary matrix G is defined as G = R 0,(i) + t 0 n 0 and R 0,(i) is the i th row of R 0 . As a result, 6 unknowns in ω 1 and d 1 can be obtained by solving Eq. ( 20) linearly.

(2) Then we extract

ω 2 = {ω x 2 , ω y 2 , ω z 2 } and d 2 = {d x 2 , d y 2 , d z 2 } from A 2 (
6 unknowns with 9 equations): Based on the definition of A 2 in Eq. ( 16), we obtain the following linear system in ω 2 and d 2 (detailed derivation in supplemental materials):

        0 -R 0,(3) R 0,(2) R 0,(3) 0 R 0,(1) -R 0,(2) R 0,(1) 0 -n 0 0 0 0 -n 0 0 0 0 -n 0                 ω x 2 ω y 2 ω z 2 d x 2 d y 2 d z 2         =               A 2,11 A 2,12 A 2,13 A 2,21 A 2,22 A 2,23 A 2,31 A 2,32 A 2,33               (21)
Thus ω 2 and d 2 can be obtained by solving Eq. ( 21) linearly.

Nonlinear Refinement.

The final step consists in a nonlinear refinement of pose and instantaneous-motion parameters with n pairs of point matches which are the inliers from the 13.5pt-RANSAC. This is achieved by minimizing the following cost function where the full homography matrix is used:

arg min R0,t0,n0,d0,ω1,ω2,d1,d2 = n i=1 (M RS,i h RS ) 2 (22)
where M RS,i and h RS are defined in Eq. ( 18).

RS IMAGE STITCHING

Working Assumptions

The goal of image stitching is to create a very wide angle image (or a panorama) from a set of images. After finding the homography matrix that aligns each pair of neighboring cameras, all the images are transformed so that they are mapped into the same projective space.

For that purpose, the cameras are assumed to have rotated about (approximately) the same center of projection. Thus, the RS homography matrix is further simplified by setting t i , d 1 and d 2 to 0, which leads to

H GS = R 0 , A 1 = -R 0 [ω 1 ] × and A 2 = [ω 2 ] × R 0 .

Stitching Pipeline

RS homography estimation with 13.5pt solver

The RS homography matrix of coinciding optical centres has the same structure as the simplified RS homography matrix in Eq. ( 16). Thus, the 13.5pt method (section 4.3) is also feasible here.

RS image alignment

When aligning two GS images all image points can be directly mapped to new locations by applying q i = H GS q i . For two RS images, Eq. ( 16) shows that any pair of feature correspondence, a 2D homography relationship (H RS ) however still holds locally by the controls of row indexes in the two images. However, in stitching case, we want to map all pixels in the first image to the second image without prior about the row indexes of their correspondents. Thus, based on Eq. ( 16), we will develop a global mapping function which align the two images.

Global RS homography mapping. Notice that the row index v i is present in both sides of Eq. ( 13). However, by transformation and substitution, the local homography relationship becomes a globally parametrized non-linear mapping η(q i ) defined as follows (detailed derivation in supplemental materials):

u i v i = η(u i , v i ) = α(u i , v i ) β(u i , v i )
where,

α(u i , v i ) = d e and β(u i , v i ) = -b ± √ b 2 -4ac 2a with,   a b c   =      H 2,(3) H GS,(3) -H GS,(2) -H GS,(2) H 3,(3) H 1,(3) -H 2,( 2
         q i q i v i q i v 2 i q i v 3 i     d e =               H GS,(1) H GS,(3) H 1,(1) v i H 1,(3) v i H 2,(1) β(u i , v i ) H 2,(3) β(u i , v i ) H 3,(1) v i β(u i , v i ) H 3,(3) v i β(u i , v i ) H 4,(1) v 2 i H 4,(3) v 2 i H 5,(1) v 2 i β(u i , v i ) H 5,(3) v 2 i β(u i , v i ) H 6,(1) v 3 i H 6,(3) v 3 i H 7,(1) v 3 i β(u i , v i ) H 7,(3) v 3 i β(u i , v i )                    q i q i . . . q i      (23
) where H GS,(i) is the i th row of H GS . There are two feasible solutions for the mapping but only one is correct in practice. The strategy of selecting the correct solution will be discussed later.

Global RS homography mapping for simplified case. The Full RS homography matrix Eq. ( 13) yields global mapping Eq. ( 23). Similarly, simplified RS homography matrix Eq. ( 16) yields a new globally non-linear mapping and thus updates η(q i ) as follows (detailed derivation in supplemental materials):

u i v i = η(u i , v i ) = α(u i , v i ) β(u i , v i )
where,

α(u i , v i ) = d e and β(u i , v i ) = -b ± √ b 2 -4ac 2a with, a = A 2,(3) q i b = H GS,(3) q i + A 1,(3) q i v i -A 2,(2) q i c = -H GS,(2) q i -A 1,(2) q i v i d = (H GS,(1) + A 1,(1) v i + A 2,(1) β(u i , v i ))q i e = (H GS,(3) + A 1,(3) v i + A 2,(3) β(u i , v i ))q i (24)
Similar to Eq. ( 23), it also leads to two feasible solutions.

It is important to realize that previous works [START_REF] Forssén | Rectifying rolling shutter video from hand-held devices[END_REF], [START_REF] Grundmann | Calibration-free rolling shutter removal[END_REF] use multiple local homographies, that are controlled by both row indexes in the two images. However, as shown in Fig. 4, η is determined only by the shared parameters H GS , H 1 . . . H 7 for all point matches. Thus, it describes a global and unique 2D mapping from q i to q i directly.

Note that η is a nonlinear 2D mapping while the classical GS homography H GS is a linear one. But since η plays the same role as H GS , thus, we keep calling η as RS Homography mapping.

Since we choose 13.5pt solver to estimate the RS Homography, thus, for consistency, global RS homography mapping for simplified case Eq. (24) will be used for image alignment.

Selecting the Correct Solution

In Eq. ( 24), there are two feasible solutions for each pixel in original image. The solution with the smaller Euclidean distance to original point before warping is chosen as the correct one. This is usually the solution which maintains the consistency of the image registration. We will show through experiments that this is indeed effective.

Blending

To seamlessly blend the images, a multi-band blending strategy [START_REF] Burt | A multiresolution spline with application to image mosaics[END_REF], [START_REF] Brown | Recognising panoramas[END_REF] is used.

Correction of Stitched RS Image

After determining the instantaneous-motion parameters by means of the method described in section 5, an inverse mapping is applied to the aligned image points in order to remove RS distortions by compensating camera instantaneous-motion as follows [START_REF] Lao | A robust method for strong rolling shutter effects correction using lines with automatic feature selection[END_REF]:

q correct = K 2 (I -[ω 2 ] × )q (25)
Note that we correct the RS image after stitching. Another solution is perform the pre-rectification to the both RS images after RS homography estimation using ω 1 and ω 2 respectively, followed by classical GS image stitching. In our experiments, we found the choice of 'stitching first' to work better than the other and therefore present only those results.

Uncalibrated Image Stitching

Direct RS image alignment: In image stitching applications, it is common that the input images are uncalibrated. Considering that points q i and q i are the image measurements in pixels (i.e [u i , v i , 1]) instead of normalized points, Eq. ( 19) gives the uncalibrated H image RS,i which is defined as follows:

H image GS = K 2 H GS K 1 -1 , A image 1 = K 2 A 1 K 1 -1 and A image 2 = K 2 A 2 K 1 -1
. K 1 and K 2 are the calibration matrices of the first and second camera respectively.

Thus if normalized image points are replaced by the image measurements in Eq. ( 19) and Eq. ( 24), we can estimate H image RS,i by using the 13.5pt method (section 4.3) and align the images directly in the image space. This means that we can stitch two RS images without prior knowledge of the calibration matrices K 1 and K 2 . RS correction with non-linear refinement: However, the determination of the instantaneous-motion parameters from A image 1 and A image 2 is different from the decomposition method of A 1 and A 2 described in section 5 since the calibration matrices are unknown.

Thus, based on the pin-hole camera model, we assume that principle point is located in the centre of the image. Thus only the focal length f remains unknown. Now, the problem is to estimate the focal length f and instantaneous-motions ω 1 and ω 2 given H image GS , A image 1 and A image 2 . We first set the focal length as 0.9 times of the maximal dimension of each corresponding image [START_REF] Purkait | Minimal solvers for monocular rolling shutter compensation under ackermann motion[END_REF]. By using the direct relative pose and instantaneousmotion algorithm in section 5, we can roughly estimate the rotation between the two images and angular velocities. Finally, we perform an iterative refinement to estimate the focal lengths, the rotation between cameras and instantaneous-motions as follows:

arg min f1,f2,R0,n0ω1,ω2 = n i=1 (M RS,i h RS -b i ) 2 (26)
With the estimated parameters, an inverse mapping similar to Eq. ( 25) is applied to the stitched image directly, as follows:

q correct = K 2 (I -[ω 2 ] × )K 2 -1 q (27)

DISCUSSIONS

The strategy which consists in combining 13.5pt and 36pt solvers offers an efficient and numerically stable solution for the RS Homography estimation problem. It enables us to compute the full set of matrices that play, in the case of RS images, the same role than the classical GS homography matrix. Relative pose and instantaneous motion of both cameras can then be retrieved by decomposing these matrices without a need for any initial guess to be provided by users. Nevertheless, the proposed solver is not a minimal one since it needs more correspondences than specified by the chosen formalism based on constant velocity model which is generally adopted in the literature being a good compromise between simplicity and precision. Indeed, 13.5 point correspondences are used instead of 10.

In the last decade, methods based on Grobner bases have been used successfully to solve several minimal problems in computer vision [START_REF] Stewénius | Gröbner basis methods for minimal problems in computer vision[END_REF]. This was favored by the use of automatic generators of polynomial solvers such as [START_REF] Kukelova | Automatic generator of minimal problem solvers[END_REF] and more recently [START_REF] Larsson | Efficient solvers for minimal problems by syzygy-based reduction[END_REF]. However, as the number of variables and equations grows the process may become intractable in practice. Although we were not able to generate a convenient solver directly from our equation system using [START_REF] Kukelova | Automatic generator of minimal problem solvers[END_REF], a more handcrafted system may potentially lead to a more minimal solver.

EXPERIMENTS

Both the RS homography-based pose estimation (RSH) and image stitching method presented in this paper were evaluated on synthetic and real data. Experiment settings. We generated a plane scene with 60 feature points, which was imaged by two RS cameras with 480 × 640 image resolution. We set the distance from the plane to the optical centre of the first camera as 1 unit, and located the second camera randomly on a sphere around the centre of the plane with 1 unit length radius.

Relative Pose Estimation

Synthetic Data Experiments

Since the ground truth of the relative pose is known, we calculated the relative pose error as follow:

• Rotation error: e rot = arccos((tr(RR GT ) -1)/2)

• Translation error: e trans = arccos(t t GT /( t t GT ))

The results are obtained after averaging the errors over 50 trials (the default setting is 1p noise, 0% outlier rate, 10 degs/frame and 0.04 units/frame for the rotational and translational speed).

Competing methods. We denote the proposed linear RS homography solvers using simplified matrix model with RANSAC verification as 13.5pt-RSH and full matrix model as 36pt-RSH. The nonlinear refinement described in Eq. ( 22) is denoted as NR.

We compared our method with two other methods:

• GSH: Classical GS homography estimation with RANSAC and decomposition 1 .

• GSH + GSNR: GS homography RANSAC followed by nonlinear refinement using GS homography model GSNR (GS plane-based bundle adjustment described but with known calibration matrix [START_REF] Zhou | Robust plane-based structure from motion[END_REF]).

• GSH + RSNR: GS homography RANSAC followed by nonlinear refinement using GS homography model RSNR (Eq. ( 22)).

• 44pt-RSE: RS essential matrix estimation with RANSAC and decomposition [START_REF] Dai | Rolling shutter camera relative pose: generalized epipolar geometry[END_REF].

• 44pt-RSE + NR: RS essential matrix estimation with RANSAC followed by nonlinear refinement NR (minimizing Sampson error).

(1) Stability vs Pixel noise: We first tested the stability of the proposed method in the presence of image noise. We increased the random Gaussian image noise from 0 to 2p. Results in Fig. 5 show that both 36pt-RSH and 44pt-RSE present numerical instabilities with increasing noise level. In contrast, 13.5pt-RSH provides stable and accurate estimations. Note that the performance of 44pt-RSE is significantly improved after performing NR but remains worse than 36pt-RSH + NR.

(2) Stability vs Outlier rate: We also tested the impact of outlier rate on the stability of the GS-based method, the epipolargeometry-based method and the proposed method. The outlier correspondences are created by setting the coordinates of these outliers randomly within the range of RS images. As shown in Fig. 6, with increasing outlier rate, 44pt-RSE fails in finding the correct inlier matches and provides wrong estimations which even can not be refined by performing NR. Similarly, neither of GSH + GSNR and GSH + RSNR can refine the rough estimations provided by GSH. We interpret this observation as GSH fails in filtering outliers. Conversely, the proposed 13.5pt-RSH method obtains significantly more accurate results and shows good outlier rejection performance in RANSAC.

(3) Accuracy vs Instantaneous velocity: We also evaluated the performances by increasing the rotational speed from 0 to 20 degs/frame and the translational speed from 0 to 0.08 units/frame receptively. As shown in Fig. 7, 44pt-RSE + NR, GSH + GSNR and GSH + RSNR provide large errors with the increasing instantaneous speed. In contrast, our method obtains obvious improvements comparing to all the competing methods.

Besides, we investigated the influence of the RS instantaneousmotions on RANSAC-based determination of inliers. As shown in Fig. 8, with the increase of the camera instantaneous-motion, the inlier detection rate of GSH, 44pt-RSE and 36pt-RSH decreases dramatically. In contrast, the proposed 13.5pt-RSH maintains its good performance in the determination.

Real Data Experiment

For readability, in the following sections, we denote the 13.5pt RS homography method with RANSAC and non-linear refinement as RSH (camera poses are final refine by RS bundle adjustment [START_REF] Lao | Robustified structure from motion with rolling-shutter camera using straightness constraint[END_REF]) and GS-based homography with RANSAC and non-linear refinement [START_REF] Zhou | Robust plane-based structure from motion[END_REF] as GSH. (1) Plane-based trajectory estimation. For this experiment we used sequence '01' and '22' from [START_REF] Hedborg | Rolling shutter bundle adjustment[END_REF] which was captured by a camera rig consisted of an iPhone4 camera (RS) and a Canon S95 camera (GS) simultaneously. However, contrarily to [START_REF] Hedborg | Rolling shutter bundle adjustment[END_REF] and [START_REF] Zhuang | Rolling-shutter-aware differential sfm and image rectification[END_REF] which require smooth instantaneous-motion input video, the proposed method can handle large baselines. Thus we just selected non-successive frames with 9 frames interval as an input sequence. Since the ground truth can be computed by using the corresponding frames from GS sequence, we used the method described in [START_REF] Hedborg | Rolling shutter bundle adjustment[END_REF] to calculate the pose error. The visual and quantitative evaluations summarized in Fig. 9 and Table 1 show that the proposed RSH method performs significantly better than GSH in both sequences.

(2) Plane-based SfM. We evaluated both GSH and RSH on two more challenging RS image sequences from [START_REF] Lao | Rolling shutter pose and egomotion estimation using shape-from-template[END_REF]: 'trans' and 'rot' which are taken with mainly translational and rotational velocities respectively. Two RS images with the integrated visible chessboard are chosen from the same sequence to estimate the relative pose with GSH and RSH. Then we perform a triangulation to reconstruct the chessboard (note that the pose of a row of RS image is obtained by using Eq. ( 2)). Since the ground-truth of the 1. http://www.robots.ox.ac.uk/∼vgg/hzbook/code/ poses are unknown, we do the quantitative comparisons using the following two methods: 1) Mapping error: average homography mapping errors of the feature points in the chessboard from the first image to the second. For each point correspondence, the mapping error can be computed as: e i = q i -τ (q i ) , where τ is the 2D homogrpahy mapping. Thus, we set τ (q i ) = H GS q i for GSH while τ (q i ) = η(q i ) for RSH.

2) The reconstruction errors of the chessboard (each reconstructed 3D point is spatially aligned with the groundtruth, by minimizing the sum of all squared point-to-point distances).

The results are presented in Fig. 10 and Table 2 and show that RSH obtains significantly better results compared to GSH in both sequences.

Image stitching

Compared Methods

The image stitching experiments were conducted under the following settings:

1) The input images are uncalibrated (calibrated matrices unavailable). 2) The input images are from unordered set where the sequence order are unavailable (large baselines).

Note that these two settings are common in the real applications [START_REF] Brown | Automatic panoramic image stitching using invariant features[END_REF]. Thus, the compared methods should able to align two uncalibrated and unordered images.

The compared methods can be divided into five categories: 1) GS-based homography method. The proposed methods RSH were compared to GS homography methods such as GSH (Au-toStitch) [START_REF] Brown | Automatic panoramic image stitching using invariant features[END_REF].

2) Commercial softwares. We also compared our methods to well-known commercial image editing applications such as ICE [23] and Photoshop [24].

3) RS video methods. Video stabilization techniques using homography transformation by assuming scene is approximately in one plane or at infinity, are similar to the working of image stitching. Therefore, video stabilization methods [START_REF] Forssén | Rectifying rolling shutter video from hand-held devices[END_REF], [START_REF] Grundmann | Calibration-free rolling shutter removal[END_REF], [START_REF] Liu | Bundled camera paths for video stabilization[END_REF] for RS cameras are potentially suitable for image stitching application. However, [START_REF] Forssén | Rectifying rolling shutter video from hand-held devices[END_REF] requires prior calibration and ordered video frames as input. Although [START_REF] Grundmann | Calibration-free rolling shutter removal[END_REF], [START_REF] Liu | Bundled camera paths for video stabilization[END_REF] are calibration-free methods, but they are both video-based method which violate the unordered images setting. Similarly, method in [START_REF] Cho | Cis video panoramic image[END_REF] generates panoramas from RS videos by assuming that the global motion of each two consecutive frames is affine. Therefore, RS video methods are unable to handle our stitching experiments.

4) Pre-rectified stitching. One solution to solve the RS stitching problem consists in performing a GS-based image stitching of prerectified images using RS rectification methods are such as [START_REF] Rengarajan | From bows to arrows: Rolling shutter rectification of urban scenes[END_REF],

[6], [START_REF] Lao | A robust method for strong rolling shutter effects correction using lines with automatic feature selection[END_REF], [START_REF] Zhuang | Rolling-shutter-aware differential sfm and image rectification[END_REF], [START_REF] Vasu | Occlusion-aware rolling shutter rectification of 3d scenes[END_REF], [START_REF] Rengarajan | Unrolling the shutter: Cnn to correct motion distortions[END_REF]. Unfortunately, none of these methods can satisfy both above-mentioned conditions, thus can not be compared in our image stitching experiments. 5) Spatially-variant homography warping. Recent image warping techniques apply spatially-variant homography model to handle the parallax in stitching. As shown in Eq. ( 13), since RS homography can be explained as multiple GS homographies, image stitching methods such as APAP [START_REF] Zaragoza | As-projective-aspossible image stitching with moving dlt[END_REF] and AANAP [START_REF] Lin | Adaptive as-natural-as-possible image stitching[END_REF] using piecewise homographies are supposed to be able to handle the RS stitching case.

Through the discussions above, we compared the proposed method Fig. 14: Two examples of stitching real RS images from [START_REF] Jia | Probabilistic 3-d motion estimation for rolling shutter video rectification from visual and inertial measurements[END_REF].

RSH to GSH, ICE, Photoshop [24] and APAP in RS stitching experiments.

Synthetic RS Images

(1) Quantitative comparison. We first compared the performances of GSH, APAP, AANAP and RSH on two synthetic RS image sequences with pure rotation from [START_REF] Forssén | Rectifying rolling shutter video from hand-held devices[END_REF]: 'rot1'(camera aim changing) and 'rot2'(both changing camera aim, and in-plane rotation). In order to evaluate the stability of all the methods with different instantaneous-motions, we chose the first frame of each sequence as a reference, then we transformed and stitched all the other frames to it. We kept the number of input feature matches the same for all the three methods and calculated the average transformation errors. The results in Fig. 11 show that in the sequence 'rot' which was taken with only aim changed, all the three methods obtain similar performances while RSH is slightly better than GSH, APAP and AANAP in all the groups. However, in the sequencer 'rot2' which was taken with in-plane rotation, the transformation errors of GSH, APAP and AANAP increase dramatically, while RSH performs obviously better.

(2) Visual comparison. We visually compared the stitching results of Autostitch, APAP, AANAP, ICE, Photoshop and RSH on 'rot2' sequence by randomly choosing two frames. Fig. 12 shows that for two images with similar instantaneous-motions (first row), all methods provide visually good results. Nevertheless, for the pairs with distinct instantaneous-motions (second and third row in Fig. 12), obvious stitching defects appear (e.g. ghosting and misalignment) except with the proposed method RSH. Although strong RS effect remain after stitching, our method can further remove the distortion.

Real RS Images

(1) Real RS image sets. The first input image pair is from a RS image sequence 'indicator' [START_REF] Forssén | Rectifying rolling shutter video from hand-held devices[END_REF] taken by an iPhone4. The second input is from a self-capture dataset 'facade' with strong RS effects. In Fig. 13, we can observe that AutoStitch produces blur on stitched images while the results from APAP are slightly better.

The result of ICE in 'indicator' dataset is visually acceptable although significant misalignments can be observed along the pole. For the 'facade' dataset, ICE gives a dramatically mismatched result. Photoshop performances are visually good in both datasets, however, some wrong alignments are still present on the overlap region such as the pole in the 'indicator' dataset and the eave in the 'facade' dataset. In contrast, RSH achieves the best results.

The third image pair is from [START_REF] Jia | Probabilistic 3-d motion estimation for rolling shutter video rectification from visual and inertial measurements[END_REF] which captures a urban scene under fast rotation. The results in Fig. 14 show that AutoStitch fails in image alignment. APAP and AANAP provide blur in regions with lack of point-matches. ICE and Photoshop provide visually pleasant results, however, geometry inconsistencies or 'object's fracture' are present along the stitching seams. In contrast, the proposed method RSH obtains the best results.

As shown in Fig. 13 and 14, after stitching, our method can remove the distortions and offers a much more visually pleasant stitching images as final outputs.

(2) Effect of number and distribution of point matches on stitching quality. One of the main disadvantage of multiple homograpies methods (APAP and ANNAP) is a sufficient number of point-matches that are uniformly distributed on image plan.

In this experiment, we first evaluated the stitching quality with varying number of point-matches by decreasing the number of point correspondences. We conducted this evaluations on 'facade' dataset. Results in Fig. 15 show that the multiple local homographies (spatially-varying warping) methods such as APAP and AANAP are sensitive to the number of input point-matches. With the decreasing of input matches, the quality of stitching results with APAP and AANAP declines dramatically. In contrast, the global methods GSH and RSH show a relative high stability.

Besides, we evaluated all the stitching methods by using two real RS images from [START_REF] Zhuang | Rolling-shutter-aware differential sfm and image rectification[END_REF], which have a large overlap region. As shown in Fig. 16, the inliers between the two images distribute densely in the right part of the two images while being limited to the white facade on the left part. The stitching results show that APAP and AANAP suffer from this unbalanced point-matches distribution and provide significant distortions on the stitched regions of the 'white facade'. Slight geometrical inconsistencies are present along the stitching seams of ICE and Photoshop's results. In contrast, the proposed method RSH obtains the best result.

Running Time

The experiments were conducted on an i5 CPU at 2.8GHz with 4G RAM. On average, it took around 3.35s for GSH, 13.5s for APAP and 6.5s for RSH(0.05s for 13.5pt solver running per time, 0.16s for the non-linear refinement, 5.9s for the image warping, blending and RS correction). Since the proposed method was implemented in MATLAB, a significant improvement can be expected when using C++.

CONCLUSION

The present work is the first to address the homography for the RS case. We first defined a theoretical RS Homography matrix and proposed a 36pt solver to retrieve it from an image pair. Then we derived a practical homography matrix and the associated 13.5pt linear solver which is more suited for RANSAC based applications. The RS homography was used in two well-known homography-based applications: relative pose estimation and image stitching. The experiment results show that the proposed method is superior to the state-of-the-art techniques and some well-known commercial image editing applications.

Fig. 3 :

 3 Fig. 3: Examples illustrating the general RS Homography case (a) and the rotate-only case (b).

Fig. 4 :

 4 Fig. 4: Comparison of GS homography mapping and RS homography mapping.

Fig. 5 :Fig. 6 :

 56 Fig. 5: Errors of relative pose estimation with increasing image noise.

Fig. 7 :

 7 Fig. 7: Errors of relative pose estimation with increasing rotational speed (a) and translational speed (b).

Fig. 8 :

 8 Fig. 8: Rate of inliers with increasing rotational speed and translational speed.

Fig. 9 :

 9 Fig. 9: Comparison of trajectory estimation (right sides) by using GSH and RSH on two RS image sequences (examples of input RS images are shown on the left side).

Fig. 10 :

 10 Fig. 10: Mapping and 3D reconstruction errors by using GSH and RSH on sequence 'trans' and 'rot' respectively.

Fig. 11 :

 11 Fig. 11: Mapping errors by using GSH, APAP, AANAP and RSH on sequence 'rot1' and 'rot2' respectively.

Fig. 12 :

 12 Fig. 12: Results of synthetic RS images stitching with different methods.

Fig. 13 :

 13 Fig. 13: Results of real uncalibrated RS images stitching with different methods.

Fig. 15 :

 15 Fig. 15: Evaluation of stitching qualities by decreasing the number of input point-matches from 100% inlier to 20%.

Fig. 16 :

 16 Fig. 16: (a) Two real RS images from [16] as input. (b) For fair comparisons, APAP, AANAP and RSH all using the same point-matches produced by GS homography RANSAC (inliers in pink, outliers in green). (c) Stitching results.

TABLE 1 :

 1 Error of the poses estimated from rolling shutter sequences against to ground truth.

	sequence seq01 [unit] seq22 [unit]m
	GSH	0.903	3.620
	RSH	0.199	1.668

TABLE 2 :

 2 Errors

		of the trajectory estimation, 2D transformation
	and 3D Reconstruction
	seq		GSH	RSH
	'trans'	Mapping errors [pixel]	2.824 1.162
		Reconstruction errors [unit] 1.833 0.317
	'rot'	Mapping errors [pixel]	4.131 2.005
		Reconstruction errors [unit] 2.519 0.397

ACKNOWLEDGMENTS

This work has been sponsored by the French government research program "Investissements d'Avenir" through the IDEX-ISITE initiative 16-IDEX-0001 (CAP 20-25), the IMobS3 Laboratory of Excellence (ANR-10-LABX-16-01) and the RobotEx Equipment of Excellence (ANR-10-EQPX-44). This research was also financed by the European Union through the Regional Competitiveness and Employment program -2014-2020-(ERDF -AURA region) and by the AURA region.