A I Comport
email: andrew.comport@i3s.unice.fr

R Mahony

F Spindler
email: fabien.spindler@inria.fr

A Visual Servoing Model for Generalised Cameras: Case study of non-overlapping cameras

This paper proposes an adaptation of classical image-based visual servoing to a generalised imaging model

where cameras are modelled as sets of 3D viewing rays. This new model leads to a generalised visual servoing control formalism that can be applied to any type of imaging system whether it be multi-camera, catadioptric, non-central, etc. In this paper the generalised 3D viewing cones are parameterised geometrically via Pl ücker line coordinates. The new visual servoing model is tested on an a-typical stereo-camera system with non-overlapping cameras. In this case no 3D information is available from triangulation and the system is comparable to a 2D visual servoing system with non-central ray-based control law.

I. INTRODUCTION

The use of camera sensors to control robotic systems is an established problem that has been studied in detail for the last two decades [START_REF] Weiss | Dynamic sensor-based control of robots with visual feedback[END_REF], [START_REF] Feddema | Vision-guided servoing with featurebased trajectory generation[END_REF], [START_REF] Espiau | A new approach to visual servoing in robotics[END_REF]. In the past various cameras including perspective, stereo [START_REF] Hager | Robot feedback control based on stereo vision: Towards calibration-free hand-eye coordination[END_REF], [START_REF] Malis | Robot control from disparate multiple sensors[END_REF], catadioptric and omnidirectional cameras [START_REF] Mezouar | Visual servoing from 3d straight lines with central catadioptric cameras[END_REF], [START_REF] Tahri | Generic decoupled image-based visual servoing for cameras obeying the unified projection model[END_REF] have been used to perform visual servoing [START_REF] Chaumette | Visual servo control, part ii: Advanced approaches[END_REF]. The unified projection model, proposed in [START_REF] Geyer | A unifying theory for central panoramic systems[END_REF], [START_REF]Catadioptric projective geometry[END_REF], takes a first step towards a generalised camera model by classifying cameras, that view the scene through mirrors and lenses, to be grouped together with classic lens based perspective models by a two step mapping via the sphere. In this case, the perspective camera is modelled with a planar mirror. Recently advantages of performing visual servoing from spheres has also been shown in [START_REF] Corke | Spherical image-based visual servo and structure estimation[END_REF].

Although the unified camera model encompasses a wide class of camera models, it is limited to the class of central catadioptric sensors which requires a single effective viewpoint whereby all viewing rays pass through the same nodal point. The aim of this paper is to relax this constraint and extend the visual servoing model to encompass both central and non-central projection cameras, including multicamera systems, in a general framework. This allows one to use a much wider class of sensors including modelling many cameras as one [START_REF] Pless | Using many cameras as one[END_REF] or even conical [START_REF] Cauchois | 3d localization with conical vision[END_REF] or spherical mirrors. Such sensors are particularly important in applications requiring wide angle panoramic views of the world. Furthermore, this will allow greater freedom in the design of systems because it will allow the construction of systems that don't need to be carefully designed and calibrated to ensure that the sensor(s) share a common centre of projection.

The idea of a more general camera model has been around for quite some time (note perhaps its first introduction in [START_REF] Gremban | Geometric camera calibration using systems of linear equations[END_REF]). In the computer vision literature, the generalised camera model [START_REF] Grossberg | A general imaging model and a method for finding its parameters[END_REF] has recently attracted much attention (see [START_REF] Li | A linear approach to motion estimation using generalized camera models[END_REF] and references therein). This model, which will be investigated in detail later, defines the relationship between different types of image measurements so as to unify the wide variety of camera designs. There exists a hierarchy of camera models ranging from x-slit cameras, multi-camera systems and non-central catadioptric cameras [START_REF] Sturm | Multi-view geometry for general camera models[END_REF]. The classical perspective imaging model defines a camera as a bearing only sensor, however, when more than one camera is available or when cameras do not project centrally, different pixels sense bearings from different positions in space. In the generalised model all cameras are unified into a single sensor by modelling each pixel as sensing a cone in 3D space.

The model proposed in [START_REF] Grossberg | A general imaging model and a method for finding its parameters[END_REF] defines the imaging cones in terms of raxels. A raxel is defined in terms of a 3D line defined by a starting point (X, Y, Z) and a direction (φ, θ). The imaging sensor captures light from a cone with this line as it's axis. The cone is defined by its aspect ratio and orientation (f a , f b , Γ). Although this model allows the radiometric quantities to vary for each pixel, this study will focus on the geometric properties and each pixel will be approximated by it's ray. An important paper in this respect is given by Pless [START_REF] Pless | Using many cameras as one[END_REF] who introduced an in-depth formalisation based on defining viewing rays via Plücker lines to derive the structure from motion equations from generalised cameras. This study resulted in a comparison of the information provided by different multi-camera configurations.

A pertinent example of a generalised camera, composed of non-central projections, is a multi-camera system. Whilst the study of multi-camera systems has mainly been focused on systems with large overlap, few works have exploited the advantages of non-overlapping systems. Of course, in the former case it is possible to match features and perform 3D triangulation, however, the field of view is heavily constrained and subsequently has limited information to constrain the six degrees of freedom of the camera motion. As shown through the simulations performed in [START_REF] Pless | Using many cameras as one[END_REF], the most informative configuration, in terms of the Fisher Information, is that of opposite facing cameras. One initial work on nonoverlapping cameras is [START_REF] Caspi | Aligning non-overlapping sequences[END_REF] but it was assumed that optical centers were co-located and the study thus concentrates on the case of cameras lying close together. This was extended in [START_REF] Dornaika | Stereo geometry from 3d ego-motion streams[END_REF] to cameras with non-zero baseline. More recently, The incoming light rays are projected through the imaging system onto the pixels m. The generalised imaging system is modelled as a set of Plücker lines (or more generally raxels [START_REF] Grossberg | A general imaging model and a method for finding its parameters[END_REF] including radiometric and optical parameters).

the case of non-overlapping cameras has also been studied in [START_REF] Mouragnon | Generic and real time structure from motion[END_REF], [START_REF] Clipp | Robust 6dof motion estimation for non-overlapping, multi-camera systems[END_REF] to perform structure and motion estimation. There is also a conceptual relation to earlier studies on hand-eye calibration such as [START_REF] Tsai | A new technique for fully autonomous and efficient 3d robotics hand/eye calibration[END_REF] (as will be used in the experiments) since they allow to determine the relationship between non-overlapping cameras via known movement.

The remainder of the paper is presented as follows. Section II-A defines the Plücker coordinates of viewing rays necessary for the remainder of the paper. Section III outlines the visual servoing approach and integrates the use of generalised viewing rays into a control scheme. Section IV looks at the case study of the generalised camera model applied to two non-overlapping cameras.

II. GENERALISED CAMERA MODEL

The generalised camera model is depicted in Figure 1. A point P ∈ R 3 is imaged at a pixel m = (u, v) that is found by following the ray through the physical system. In the generalised model, each ray leaving the viewing surface can be modeled by a 3D line. Several groups have calibrated these cameras through non-conventional models such that each pixel is individually mapped to a particular viewing ray through lookup tables. The interested reader may refer to [START_REF] Grossberg | A general imaging model and a method for finding its parameters[END_REF], [START_REF] Sturm | A generic concept for camera calibration[END_REF].

A. 3D Viewing Rays and Plücker lines

In order to formalise a generalised camera model, a parameterisation for the viewing ray of each pixel will be given. A line can be defined by the join of two points or the intersection of two planes. Many parameterisations exist ranging from the null space and span representation, to Plücker matrices and Plücker vectors (see [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF] for a brief survey). In this work Plücker vectors will be employed as in [START_REF] Pless | Using many cameras as one[END_REF] and a short summary is given here.

Let P 1 and P 2 ∈ R 3 be two 3D points defining a line and 3 be their homogeneous coordinates. The line joining the two points is given as the Plücker matrix [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF]:

P 1 = (X 1 , Y 1 , Z 1 , W 1 ) ⊤ and P 2 = (X 2 , Y 2 , Z 2 , W 2 ) ⊤ ∈ RP
L = P 1 P ⊤ 2 -P 2 P ⊤ 1 . (1)
The Plücker coordinates are subsequently the 6 non-zero independent elements of this 4 × 4 skew-symmetric matrix

q P q ′ P c q L q ′ P ⊥
Fig. 2. A generalised camera viewing ray. The line is defined by Plücker coordinates L = (q, q ′ ) where q ′ goes into the page. P ⊥ is the point on the line closest the origin. P c is a point on the line that is rigidly linked to the generalised camera. P is a point in the 3D scene.

given as:

L =         L 41 L 42 L 43 L 32 -L 31 L 21         =         W 1 X 2 -X 1 W 2 W 1 Y 2 -Y 1 W 2 W 1 Z 2 -Z 1 W 2 Z 1 Y 2 -Y 1 Z 2 X 1 Z 2 -Z 1 X 2 Y 1 X 2 -X 1 Y 2         ∈ RP 5 , (2)
giving the Plücker coordinates as L = (q, q ′ ) ⊤ . With the normalised homogeneous coordinate W 1 = W 2 = 1, (2) simplifies to give q = P 2 -P 1 as the direction vector (of any length) and q ′ = P 1 × P 2 is the moment vector of the line. Geometrically, the moment vector is perpendicular to the plane containing L and the origin, and its magnitude is twice the area of the triangle formed by the two points and the origin. Substituting for P 1 in q ′ , it can be easily seen that the moment vector is orthogonal to any point on the line P and the vector q so that:

q ′ = q × P. (3)
See Figure 2 for a visualisation.

The line L has 4 degrees of freedom which can be determined as follows. The Plücker line defined in (2) has 6 parameters, however, they are only unique up to scale (only the 5 ratios are significant). Secondly, by definition the moment vector is perpendicular to every displacement along the line, so:

q T q ′ = 0, (4)
which is quadratic in the Plücker line coordinates and can be obtained by evaluating det(L) = 0. This leaves a minimal representation of 4 parameters. In this paper the direction vector, q, is normalised to be a unit vector so as to define the scale of the remaining homogeneous parameters:

q = q |q| , q ∈ S 2 . ( 5
)
In this case, the point P = q × q ′ is the point on the line closest to the origin which is denoted here P ⊥ .

III. VISUAL SERVOING

A. Visual Servoing Model

In classic image-based control [START_REF] Weiss | Dynamic sensor-based control of robots with visual feedback[END_REF], [START_REF] Feddema | Vision-guided servoing with featurebased trajectory generation[END_REF], many different types of features s have been used to perform visual servoing. Interesting features range from sets of points to lines, ellipses, cylinders, distances to 3D models and moments using various types of optical systems. In all these cases the 2D image-sensor measurements m belong to rays passing through a common nodal point.

In general, a vision based control scheme aims to define a task function for the robot as [START_REF] Chaumette | Visual servo control, part ii: Advanced approaches[END_REF]:

e(t) = s(m(t), a) -s * , (6)
where the vector m(t) is usually a set of image measurements in pixel coordinates, a is a vector containing any a priori knowledge about the system such as intrinsic or extrinsic camera calibration parameters or a 3D object model and where s(m(t), a) is a vector of n visual features that has been constructed from the image measurements and may take into account the prior knowledge such as pixel to metric conversion. s * is a corresponding vector of features containing their positions in the desired image.

The design of the control scheme then requires relating the movement of the visual measurements in the image to the movement of the robot. A first step is to determine the relationship between the time variation of s and the camera. The instantaneous spatial velocity of the camera x ∈ R 6 is parameterised by the Lie algebra se(3) as:

x = (υ, ω) ⊤ ∈ se(3), (7)
where υ and ω are the instantaneous linear and angular velocities respectively. This velocity twist is related to a displacement via integration as:

T(x) = exp [ω] × υ 0 0 , ( 8
)
where exp is the exponential map, T = (R, t) ∈ SE(3) is the homogeneous matrix of describing the camera displacement, R ∈ SO(3) is a rotation matrix such that R ⊤ R = I and det(R) = 1, t ∈ R 3 is the translation vector, [.] × represents the skew symmetric matrix operator and where the integration period is taken as ∆t = 1.

The kinematics of the task function ( 6) for a eye-in-hand system are then given as:

ė = Lx, (9)
where L is the interaction matrix related to s.

If the robot is controlled by velocity then x can be taken as the input to the robot controller. If an exponential decrease of the error is imposed then ė = -λe and the control law is given by:

x = -λ L + e, (10)
where the hat operator signifies that the interaction matrix is computed from uncertain measurements and therefore it is only an approximation. L + = (L ⊤ L)L ⊤ ∈ R 6×n is the pseudo inverse of L which must be of full rank 6.

In summary, the classic model is quite general in that any type of feature s may be constructed from both m and the apriori parameters a. It is also worth noting that multi-camera systems have been classically modelled by forming the task error directly in each camera's reference frame and not via generalised coordinates as will be shown in the next section.

B. Generalised Visual Servoing

In this paper, the generalised camera model is used to construct a visual servoing control law in which the sensor measures a set of 3D viewing rays L = (L ⊤ 1 , . . . , L ⊤ n ) ⊤ with n the number of rays (and not just 2D pixel coordinates m as does a classic camera sensor). In this way (2) defines a broad group of possible physical measurements and represents a general class of cameras including non-central projection cameras. This encompasses non-central projection cameras including multi-camera systems. An important result of this is that it amounts to considering that the sensor measure's noisy lines in space whereby the line coordinates encompass all measurement errors including both classical 2D measurement and extrinsic calibration errors.

Given this general model, the rays can be constructed from any type of sensor and the visual servoing task [START_REF] Mezouar | Visual servoing from 3d straight lines with central catadioptric cameras[END_REF] becomes:

e(t) = s(L(a, t), b) -s * , (11)
and where a is now redefined as a vector of general camera parameters and b is a vector containing any a priori knowledge about the system such as a 3D object model, etc. 1) Perspective camera: In this case the viewing rays of the general model can be shown to depend only on a classic pinhole model as L(m(t), ξ), where ξ is the vector of the intrinsic camera parameters required to determine the normalised rays. Under the generalised model a pixel coordinate forms a 3D ray L that passes through the image plane at the point m and intersects with the optical centre of the camera. If the pinhole camera is calibrated with the calibration matrix K(ξ) ∈ R 3×3 , then a normalised point p = (x, y, 1) may be determined from the image coordinates. In this case the moment vector is q ′ = 0 since the origin, P = (0, 0, 0), belongs to the ray. The 3D ray is subsequently defined by the Plücker vector L = ( Km |Km| , 0). 2) Multi-camera system: On the other hand, if the camera has multiple centres of projection, the moment vector is non-null. For example, take a multi-camera system as a generalised non-central projection camera. In this case the rays depend on each camera's pixels M = (m 1 , . . . , m n ) and the cameras' intrinsic and extrinsic parameters a = (ξ c , x c ). In this case each camera c has a moment vector obtained from (15) as q ′ = q × P c , where P c = -R e⊤ t e is the origin of the camera seen in the generalised coordinate system with (R e , t e ) being the extrinsic camera rotation and translation parameters. The direction vector is q = R e⊤ q = R e⊤ Km |Km| . 3) Generalised-camera system: Consider now, a generalised-camera system that directly measures a set of viewing rays L(q(t), q ′ (t)) defined in a generalised reference frame. It is attached to an eye-in-hand visual servoing system that moves wrt. an inertial coordinate system. Viewing rays are directly used to define the task function (higher order features s(L) are not considered here):

e(t) = L(t) -L * , ( 12
)
5685

The set of all viewing rays are related to movement of the camera via the analytical relationship given in section II-A. Each direction vector is related to a 3D world point as:

q i = P w i (t) -P c i , (13)
where P w i is the coordinate of a point in the inertial world coordinate system and P c i is the point which is rigidly fixed to the generalised camera (for a perspective camera it is the optical center). To simplify the discussion, a-priori knowledge of P c is considered available, however, it will be shown in Subsection III-B.4 that is not required in a generalised camera system.

From ( 13), the normalised unit direction vector is:

q(t) = q(t) |q(t)| = P w (t) -P c |q(t)| , ( 14
)
where the ray index is assumed implicit. The moment vector defining each view ray is subsequently:

q ′ (t) = q(t) × P c . ( 15
)
Taking first the time derivative of the direction vector equation ( 14) gives:

˙ q = |q| -1 q + d dt (|q| -1 )q, (16)
where q = Ṗw giving well known motion for a point:

q = -[ω] × q -(υ + [ω] × P c ) (17)
where both υ and ω form the velocity twist of the generalised reference frame wrt. the world frame.

The second term of ( 16) is developed and simplified as:

d dt |q| -1 = d dt q ⊤ q -1 2 , = -1 2 d dt q ⊤ q q ⊤ q -3 2 , = -1 2 q⊤ q q ⊤ q -3 2 -1 2 q ⊤ q q ⊤ q -3 2 , = -|q| 3 -1 q ⊤ q .
(18) Substituting ( 17) and ( 18) into [START_REF] Li | A linear approach to motion estimation using generalized camera models[END_REF] gives the dynamic equation for the unit direction vector as:

˙ q = 1 |q| q -1 |q| 3 q ⊤ qq, = -1 |q| [ω] × q -1 |q| π q (υ + [ω] × P c ) , (19)
where π q = (Iq q ⊤ ) is an orthogonal projection π q : R 3 → T q S 2 onto the tangent space of the sphere S 2 at the point q ∈ S 2 . Equation ( 19) is equivalent to the adjoint transformation of the spherical projection of a point in a classic perspective camera in which the unknown P w can be eliminated from [START_REF] Dornaika | Stereo geometry from 3d ego-motion streams[END_REF] as:

˙ q = [ q] × ω + 1 |q| π q (υ + [ω] × P c ) , (20)
where the skew symmetric property [a] × b = -[b] × a has been used. Taking then, the time derivative of equation ( 15) gives simply: A) which is 'calibrated' such that a sphere encloses all viewing rays. This is simpler since there is only one "base-line" parameter (the sphere radius).

q′ = ˙ q × P c (21)

4) Generalised parametrisation:

Here it is assumed that a mapping from sensor space to the generalised coordinate space has been achieved by a general calibration model [START_REF] Grossberg | A general imaging model and a method for finding its parameters[END_REF], [START_REF] Sturm | A generic concept for camera calibration[END_REF] whereby the viewing rays contain all necessary information. In that case no other information is required such as calibration parameters. This means that the center of projection P c used previously is unknown. This system can be defined simply by a sphere that englobes all the viewing rays (see Figure 3).

From each q and q ′ it is possible to determine the point of the line that is closest to the origin of the generalised camera system as:

P ⊥ (t) = q(t) × q ′ (t), (22)
The variation of Ṗ⊥ is then approximated as a function of its gradient:

Ṗ⊥ (t) ≈ P(t) ⊥ -P ⊥ * (23)
which is determined directly from the current L(t) and desired L * viewing rays. It's maximum variation is bounded by P c and it is assumed that the control law will be chosen to maintain this assumption.

In that case (13) is:

q ⊥ = P w (t) -P ⊥ (t). (24)
Following through the same derivation in the previous section leads to the dynamic equation for the unit direction vector as:

˙ q = 1 |q ⊥ | π q ( Ṗw + Ṗ⊥ ). (25)
Since equation [START_REF] Sturm | Multi-view geometry for general camera models[END_REF] requires P w , it can be substituted using ( 13) and ( 14) by:

P w = |q ⊥ | q + P ⊥ , (26)
into [START_REF] Michel | Singularities in the determination of the situation of a robot effector from the perspective view of three points[END_REF] to give:

˙ q = [ q] × ω - 1 |q ⊥ | π q υ + [[ q] × q ′ ] × ω + Ṗ⊥ . (27)
If the moment vector is evaluated with the current definition using [START_REF] Tsai | A new technique for fully autonomous and efficient 3d robotics hand/eye calibration[END_REF] to give:

q′ = q × Ṗ⊥ -[[q] × q ′ ] × ˙ q. ( 28
)
L L = -1 |q ⊥ | π q [ q] × + 1 |q ⊥ | π q [[ q] × q ′ ] × 1 |q ⊥ | [[ q] × q ′ ] × π q -[[ q] × q ′ ] × [ q] × + 1 |q ⊥ | π q [[ q] × q ′ ] × (28)
The resulting equations ( 27) and (28) relate the velocity of the 3D point to the spatial velocity of a viewing ray as:

L = L L x + B, (29)
where x is the kinematic screw between the generalised camera frame and the world and L L is given in (28). B =

-1 |q ⊥ | π q Ṗ⊥ , q × Ṗ⊥ ⊤
is the bias term related due to P ⊥ . Even if the interaction matrix is rank 2 (since q′ is a linear combination of q), using both measurement vectors provides increased robustness with respect to measurement noise. Notably, if there is more noise in q than q ′ then the moment vector helps minimise this (see Figure 4(b)). Similarly to classical visual servoing, a rough estimate of the depth |q ⊥ | of the 3D point to P ⊥ and is required in practice.

In order to have full rank 6 to control 6dof it is necessary to have at least three non-degenerate and non-singular points in a single view configuration [START_REF] Michel | Singularities in the determination of the situation of a robot effector from the perspective view of three points[END_REF]. In a stereo-perspective case it is necessary to avoid that the 3D point lies on the baseline [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF]. Here the control law is built by stacking multiple viewing ray interaction matrices with more than three non-coplanar points in an appropriate configuration to avoid singularities or ambiguous global minima.

The stability of the system can be analysed by considering the l 2norm candidate Lyapunov function V = | n i=1 e i | which is derived as:

V = e B ⊤ ėB = -λe B ⊤ L e L + e e B , (31)
where e B = e -B and from which the global asymptotic stability is ensured if the following condition is respected:

L e L + e > 0. (32)
IV. RESULTS

In order to test this model a series of simulations were performed followed by validation on a real robotic platform. An eye-in-hand visual servoing experiment is considered.

A. Simulation

Several simulations were perfomed to test the generalised camera visual servoing configuration (see Figure 3). In the simulation an arbitrary number of centers of projection and an arbitrary number of points were generated. In the results shown in Figure 4, 200 projection centers were considered with 10 lines projecting onto each center. The initial general camera pose was set to (0.5, 0.1, 2.5, 280 o , 52 o , 0 o ) and the final to (0, 0, 1, 0 o , 0 o , 0 o ). Noise was added onto both q and q ′ and the effect is compared and as expected the added task vector for q ′ improves when there is a large amount of noise on q. Even with a large rotation of 180 o around the x-axis, the control performs quite nicely. Of course this assumes that the points are visibile over a large domain which is not easily the case in practice as will be seen in the next experiment.

B. Non-overlapping visual servoing

Since both cameras are not viewing the same 3D points, conventional extrinsic camera calibration was not possible. It was necessary to calibrate each camera individually with respect to a common coordinate system. In this case the cameras were mounted on the robot (see Figure 5) and the robot was used as the common coordinate frame. The robot makes a series of automatically planned movements with a camera rigidly mounted at the gripper. At the end of each move, feature coordinates are extracted and camera extrinsic calibration is performed using the technique given in [START_REF] Tsai | A new technique for fully autonomous and efficient 3d robotics hand/eye calibration[END_REF]. To test the limits of the system a task involving a near 90deg rotation around the z axis, along with a considerable translation was considered. The initial and desired images are shown in Figure 6. It can be seen that two points have been used in each image marked by blue crosses. The desired positions, marked in red, have been determined from a desired robot pose and model-based pose estimation has been used to determine the initial position. As can be seen in Figure 7 (a) and (b), an exponential decrease in the overall error is obtained and smooth velocities are used to control the robot. Furthermore a smooth trajectory in both rotation and translation is obtained in 3D shown in Figure 5(b).

V. CONCLUSIONS AND FUTURE WORKS

In conclusion, this paper presented an alternative visual servoing model based on the concept of a generalised camera. This was achieved through the use of Plücker line coordinates to geometrically model general viewing rays in 3D. This has allowed to develop a visual servoing control law that has been tested on a non-overlapping stereo configuration and shown to give very satisfactory results. Future work will be dedicated to the investigation of the use of this general formalism to develop interesting higher level control features.

Fig. 1 .

 1 photosensitve detectors

Fig. 3 .

 3 Fig.3. (a) A simulated generalised camera (used in Section IV-A) which is 'calibrated' such that a sphere encloses all viewing rays. This is simpler since there is only one "base-line" parameter (the sphere radius).

Fig. 4 .Fig. 5 .

 45 Fig.4. A visual servoing simulation for a generalised camera configuration using 200 randomly generated centers of projection on a sphere with radius 1m with 10 random points in each. A large rotation of 180 o around the x axis was made. Red is q, blue is L and yellow q ′ . (a) The error norm with σ = 0.05 Gaussian noise on q ,(b) The error norm with the same noise on q ′ . (c) The trajectory in 3D with the red the desired position, blue the trajectory, and the blue star the initial camera position.

Fig. 6 .Fig. 7 .

 67 Fig. 6. A visual servoing experiment for a non-overlapping stereo eye-inhand configuration using 2 viewing rays in each camera. The red crosses display the desired image positions and the blue crosses the current point match. (a) Camera 1's initial image. (b) Camera 2's initial image. (c) Camera 1's 149th image. (d) Camera 2's 149th image.

VI. ACKNOWLEDGEMENTS

The authors would like to thank the INRIA Lagadic team for use of their Afma 6 robot and software [26]. This research was supported by the CNRS and by the Australian Research Council through discovery grant DP0880509, "Image-based teleoperation of semi-autonomous robotic vehicles".


Laurent Coutard

Franc ¸ois Chaumette

Visual detection and 3D model-based tracking for landing on an aircraft carrier

A challenging task of airborne operations remains the landing on the carrier deck, which limits the carrier operational efficiency during rough sea. In this paper, a method of carrier visual detection and tracking is described. With the help of the aircraft sensors, the carrier is first detected in the image using a warped patch of a reference image. This provides an initialization to a real time 3D model-based tracker estimating the camera pose during the sequence. This method is demonstrated and evaluated using a simulator with high-fidelity visualization and on real video.

I. INTRODUCTION

A. Aircraft carrier and landing

Since its origin during WWI, the aircraft carrier (AC) remains a major tool of sovereignty allowing the force projection across the world. Nowadays, the French Navy operates one carrier: the Charles de Gaulle (R91) powered by a nuclear reactor. It embeds a group of 40 aircrafts and helicopters.

Landing on an AC is usually considered by pilots as a difficult task. Indeed, it consists in landing on a small deck in motion in almost all weather and visibility conditions. The approach trajectory mainly depends of the visibility as presented in Figure 1: • In case of good visibility, the pilot begins a visual approach, which consists in flying over the carrier and turning (point B of Figure 1) to align its aircraft to the runway deck axis (point C) while he begins its descent. • In case of low-visibility or night approach, the procedure begins at 18 km (10 nautical miles) of the AC (point A) and consists in the alignment to the deck axis with a constant glide slope of 4 • at 7.5 km (point B). In french procedures, the pilot remains in the control loop with the help of aiding systems. The Optical Landing System provides the pilot a visual beam, which helps him follow an optimal trajectory until the touchdown. This system is actively controlled by the Landing Carrier Officer, located aboard the carrier. The pilot can land with the passive help of marks on the deck that allow him to control the aircraft glide slope and alignment.

B. Hypothesis and delimitation of the study

This study evaluates the possibility of using the vision sensor with other sensors on the aircraft to achieve later an autonomous landing by visual servoing. The study range is firstly defined between 10 nm (18 km) to 0.5 nm (900 m) to

The authors are with INRIA Rennes-Bretagne Atlantique and IRISA, Campus de Beaulieu, 35042 Rennes cedex, France {Laurent.Coutard, Francois.Chaumette}@inria.fr Imaging sensors available on french aircraft fighters with characteristics compatible with the application are the Front Sector Optronics (FSO) and the Damocles multifunction-pod from Thales [START_REF]DAMOCLES, 3rd Generation Multi-function Targeting pod[END_REF]. The FSO was discarded because its location prevents it from seeing the carrier during approach. The Damocles is located under the aircraft and has two fields-ofview of 1 and 4 • . Carrier and deck occupations in the image are presented in Figure 2 with the two optics of the Damocles. This imaging sensor seems well suited for this application if the detection and tracking are effected in wide space.

Other sensors embedded on the aircraft can be used (e.g. Inertial Measurement Unit-IMU, radio-altimeter and TACAN). TACAN system is composed of an emitter on the AC and a receiver on the aircraft [START_REF]Federal Radionavigation Systems[END_REF]. Receiver provides relative distance and azimuth between them to help the detection (see Figure 4a). Moreover, the carrier heading is known. Finally, the method does not rely on GPS in order to be autonomous.

Image tracking algorithm needs an initialization but TACAN distance and angular components are not accurate enough to provide a correct initialization. Hence, the carrier must be first detected in the image.

C. State of the art

In this application, detection localizes the carrier in the image. Then, the objective is to estimate the pose between the camera and the carrier using only the current image, and finally to track it along the sequence using the previous estimation. 1) Detection: In our application, detection consists in finding the carrier in the image and providing an initial pose to the tracking algorithm. The task is complex because variability of sea and weather conditions affects the appearance of AC which is designed to be the less visible as possible. In [START_REF] Santhalia | A Method to Extract Future Warships in Complex Sea-Sky Background which May Be Virtually Invisible[END_REF], low-level image processing techniques, as edge detection and multi-histogram matching, are used to segment the sky and the sea to finally extract the ship and its position in the image. This method is not invariant enough to visibility and does not provide an accurate localization in the image. Detection of the ship could be processed using motion segmentation as in [START_REF] Sull | Model-based obstacle detection from image sequences[END_REF]. A model of optic flow (OF) is computed using internal sensors and compared to the OF extracted from the image. Difference between these fields leads to obstacle localization. In this method, errors are mainly introduced by the OF extraction, which is often noisy. Moreover, in our application, waves constitute another source of noise.

2) Matching and pose estimation: Several papers present landing techniques on ship deck but mainly concern helicopters, where the landing deck is defined by visible landmarks and the approach is vertical. In [START_REF] Sharp | A vision system for landing an unmanned aerial vehicle[END_REF], some markers are segmented with histogram computation, and features used for localization are the corners of the shape, whose world coordinates are known using linear and nonlinear optimization. In the case of runways, detection can be performed using a model and some a priori data as approximate position. In [START_REF] Soni | Modelling issues in vision based aircraft navigation during landing[END_REF], an exhaustive search is performed to find the best match and to determine the camera pose, which is quite expensive and does not ensure error minimization. Concerning the landing on carrier, the method presented in [START_REF] Guilloux | Approche et appontage assistés par traitement d'image embarqué sur aéronef[END_REF] consists in edge extraction and matching with a model. Segmentation and matching problems can be simplified using infrared emitting spots which are installed on the deck. In [START_REF] Yakimenko | Unmanned aircraft navigation for shipboard landing using infrared vision[END_REF], only three points are used to compute the pose during a test flight. This method uses other sensors to improve and to filter the result and is subject to singularities and inaccuracy.

3) Tracking: Tracking methods use previous positions of (2D or 3D) features to estimate the new position. In [START_REF] Barat | A corner tracker snake approach to segment irregular object shape in video image[END_REF], a corner snake method is used to segment a runway from a rough initial position and to track it in the image without estimating the pose. Dense information of the image is used to directly estimate the projective transformation between the current and a reference image representing runway and large part of the tarmac in [START_REF] Gonc ¸alves | Vision-based autonomous approach and landing for an aircraft using a direct visual tracking method[END_REF] and [START_REF] Dame | Accurate real-time tracking using mutual information[END_REF]. The pose with respect to the airport is estimated using the decomposition of the homography and the knowledge of the distance between runway and camera. [START_REF] Dame | Accurate real-time tracking using mutual information[END_REF] uses Mutual Information criterion which allows a multimodality tracking between airborne infrared images and a satellite reference image. These dense methods are robust to luminance variation but are computationally expensive with large images. Moreover, a runway is a very large and free surface whereas a carrier deck is small and may be occupied in part by aircrafts and superstructures.

In this paper, the relative pose between an aircraft equipped with a roll-tilt camera and an aircraft carrier is estimated during approach and landing trajectories using a robust 3D model-based tracker [START_REF] Comport | Real-time markerless tracking for augmented reality: the virtual visual servoing framework[END_REF] that uses edges as visual features. Tracker initialization consists in detecting the carrier by localizing the maximum correlation coefficient of a patch representing the AC with a region of the current image and then estimating the pose. This detection method is robust to luminance changes (e.g. due to weather) and benefits from dense image information to deal with limited occlusions and patch modifications due to presence of aircrafts on the deck. This paper is organized as follows: in Section II, involved frames are defined. In Section III a detection method is proposed using aircraft sensors to define an image region where the carrier is searched to provide an initial pose to the tracker presented in Section IV. In Section V, detection and tracking algorithms are evaluated on synthetic and real images.

II. DEFINITIONS

Figure 3 represents the different frames involved. Assuming that the Earth is flat (which is true regarding the Earth radius and the study range), the x-axis of the plane frame F 0 is NED (North-East-Down) oriented.

• Aircraft Body and AC frames (respectively F b and F ac ) are conventionally with the z-axis oriented down. • The frame F d corresponds to the origin of the deck, translated and rotated about its z-axis from the carrier frame. It is expressed by the constant known matrix ac M d .

• The frame F t corresponds to the location of the TACAN beacon.

• The pose c M b between the camera frame F c and the aircraft body frame F b is supposed to be known along the image sequence. • Frame F bo corresponds to the frame F b without pitch and roll angles (see Figure 4a). The detection consists in locating the aircraft in the first image to provide an initial pose to the tracker. Ideally, the tracker initialization could be computed with perfect aircraft sensors as IMU, radio-altimeter and TACAN beacon. Nevertheless, sensors inaccuracy involves uncertainties of AC position in the image which are not compatible with the tracker. That is why it is necessary to locate the AC and compute the initial pose. The detection is split in different parts:

• A search area of the image, where the deck origin should be, is computed using available sensors and their accuracies. • The carrier is located in the image by matching a template with the current image using the maximum Zeromean Normalized Cross Correlation (ZNCC) between the template and the image. The ZNCC detection is robust to luminance variation and to presence of aircrafts on the deck. To improve the results, the template is warped using the initial pose given by available sensors. • Finally, the pose is computed with deck corners.

A. Search area in the image

The Damocles pod uses its degrees of freedom to focus the carrier in the image using TACAN information. If TACAN was perfect, deck origin would be the red dot of Figure 4b. Due to TACAN 2σ accuracies (respectively 185m, 1 • ), the deck origin is within the red area of Figure 4b. This search area is used to reduce the search space of the carrier in the image. Sensors used are the aircraft IMU (providing ψ b , θ b , φ b ), radioaltimeter (supposed to be ideal), TACAN beacon and carrier yaw ψ ac (others angles are neglected).

The origin of the deck, defined by its location d X 0 in F d is projected in the image using the perspective projection model pr ξ ( c M d , d X 0 ) where c M d is the deck pose in the camera frame and ξ are the camera intrinsic parameters. c M d can be decomposed in the following expression:

c M d = c M b b M t t M d (1)
• t M d is the deck pose (known) in the TACAN frame.

o Z t -o Z b 0 0 0 1     (2) with      dt = (d m t + δd t ) 2 -( o Z t -o Z b ) 2 ψr = ψ m r + δψ r ψs = ψ ac -ψ b + δψ r
where d m t and ψ m r are the distance and angle between the aircraft and the beacon provided by TACAN (see Figure 4a). These measures are statistically defined by normal laws: N (d t , σ 2 dt ) and N (ψ r , σ 2 ψr ) where d t and ψ r are true values with their respective variances σ 2 dt and σ 2 ψr . Aircraft altitude o Z b is measured by radio-altimeter, whereas the beacon altitude o Z t is a priori known. The search area of Figure 4b is defined by its four corners computed by the maximum and minimum of the TACAN distance and angular 2σ accuracies, δd t and δψ r . Moreover, with the 1 • optics, search area exceeds the width of the image due to TACAN angular inaccuracy. In this case, detection is first applied with the 4 • optics and then improved with the 1 • optics.

B. Warping

To improve the carrier detection in the image by maximizing the ZNCC, the reference image I r is warped. I r is taken at a known pose cr M d with given camera parameters K r . The transformation c M cr between the reference and the current pose (using TACAN) is computed as: c M cr = c M d cr M -1 d . The homography c H cr is built using the plane P( cr n, cr d) defined by the deck where c R cr and c t cr are extracted from c M cr .

c H cr = c R cr + c t cr cr d cr n ⊤ (3)
and the warping matrix c W cr linking pixels of reference and current image is given by:

c W cr = K c H cr K -1 r (4)
The image I w used for ZNCC detection is equal to I r ( c W cr ) where the template of the deck I t is extracted as presented in Figure 5 using deck corners coordinates.

C. Detection by correlation

This step of the method consists in localizing the carrier in the image by computing a similarity criterion between the patch I t previously defined and the window of the image over all the search area defined in Section III-A. The ZNCC formula can be found in [START_REF] Gonzalez | Digital image processing[END_REF]. The maximum of the ZNCC corresponds to a possible location of the carrier. With the template I t of Figure 5c whose histogram covers a wide part of the grey value range [0;255] with the white parts of the deck, an efficient detection is obtained. Carrier is located in the image and deck corners position in the template image is known, and then in the current image. The initial pose of the tracker c M d is then computed with a classical pose estimation method using Virtual Visual Servoing (VVS) [START_REF] Marchand | Virtual visual servoing: a framework for real-time augmented reality[END_REF] from four points (deck corners).

IV. 3D ROBUST MODEL-BASED TRACKING

The method used to track the AC in the image sequence is fully described in [START_REF] Comport | Real-time markerless tracking for augmented reality: the virtual visual servoing framework[END_REF]. In summary, this algorithm consists in estimating the camera pose by minimizing the error between the observed data s * and the position s of the same features determined by the projecting model according to the current pose. This method needs an initial pose c M 0 o and a model of the visible edges to project them in the image. Each projected segment is sampled. For each point, a search for the highest gradient is performed at the normal of the edge. Once this step is realized for all points, an error is defined and minimized by VVS which directly provides the estimated pose between the camera and the AC. This algorithm has already been applied to tracking elements in space applications with stereo camera [START_REF] Dionnet | Robust stereo tracking for space robotic applications[END_REF] and in microassembly of Micro Electro Mechanical Systems [START_REF] Tamadazte | Real-time vision-based microassembly of 3d mems[END_REF].

V. RESULTS

The proposed method was tested both with a high-fidelity simulator and on real images. Each sequence represents a part of the two kind of approaches presented in Figure 1. Sequences and results are presented in the enclosed video.

A. Results on synthetic images

In order to evaluate method efficiency, we developed a simulator based on the XPlane game as a high-fidelity visualization and the Nimitz AC model. The software, developed with the ViSP library [START_REF] Marchand | Visp for visual servoing: a generic software platform with a wide class of robot control skills[END_REF], is interfaced to XPlane via UDP socket using the Xplane Software Development Kit (SDK) [18]. The SDK provides access to many variables as camera, plane and aircraft pose, visibility, weather and sea state at 25Hz.

In this simulation, the aircraft is making a flying over the aircraft carrier at 200m above sea level. Trajectory and attitude are recorded from a real flight. This kind of trajectory is representative of the first part of the approach when the aircraft turns and realigns to the deck (see Figure 1). The carrier makes a straight line with a calm sea with no attitude movement. The "roll-tilt" camera Damocles focuses the carrier in the image center during all the flying over. Moreover, the image is artificially degraded by addition of blur and noise.

In the following image sequence, tracking is initialized using the detection method described in Section III with the warped patch of Figure 6a. To demonstrate the method robustness, the patch was degraded by adding aircrafts on the deck and smoke from catapults. The tracking is performed from about 2950 to 400 meters of the carrier with the same 4 • optics, which corresponds to a AC size in the image of 30% to more than 100% according to Figure 2. Evolution of the carrier in the image is presented on Figures 6b to 6e. The external contour and the 3D superstructure of the carrier stabilize the estimation, especially when the carrier is small and the deck appears very oriented in the image. The estimated pose ac M b between the aircraft and the carrier is compared to the real one in Figure 6f and 6g. As shown on Figure 6g, estimation results regarding the distance toward the AC are good, even during the first part of the sequence, when the deck is very oriented with respect to the camera (average errors of 6 meters and 0.2 degree on the x-axis and for φ angle) as presented on Figures 6b and6c. Furthermore, in the second part of the sequence where the deck is more visible, the position and orientation average errors remain respectively under 0.72 meter and 0.02 degree.

Table I sums up the statistical results on each axis of ac M b . The sensibility of the pose computation is closely dependent on the distance between the camera and the object. That is why error and standard-deviation of the distance are presented in percentage whereas angles are directly presented as error in degrees because these values are small. Pose estimation results are low regarding the distance with respect to AC and obtained without any filtering or fusion. The algorithm time computation mean is about 25 ms with a standard deviation of 8.5 ms over the sequence of 1024×768 images with a 3.06 GHz processor.

B. Results on real images

Since videos recorded from large distance with narrow FOV camera are not available, and good quality videos with embedded camera are rare, a video taken from a TV documentary on French Naval Aviation was used [19]. The camera is located in the aircraft cockpit, involving distortions and reflections into the glass. Video is very blurry due to precision and resolution of the camera (unknown) and internet compression. The aircraft is in the final approach, near point B of the visual approach of Figure 1. Due to the wide camera FOV (40 • ), detection efficiency is proved on limited range, but can be extended to larger range using narrow FOV.

X (%) Y (%) Z (%) ψ ( • ) θ ( • ) φ ( • ) µ 0.
1) Detection: Template image warping cannot be used because no data are available for this video. In this case, detection consists only in finding an AC patch in the image as presented in Section III-C and robustness of the method relies on coefficient invariance with respect to luminance and warp. Figure 7f presents the maximum ZNCC along the sequence for the five patches of Figures 7a to e. We can see that this coefficient remains high over frames. For instance, 170th patch coefficient is superior to 0.8 from the 140-th to the 205-th frame. That is why detection can be validated using a threshold on the ZNCC value for a given patch. Search of the maximum coefficient allows an efficient detection and tracking initialization. Using a database of these five different patches, detection is achieved from the 75-th to the 262-th frame (Figure 7f), corresponding to a distance wrt the AC from 700 to 250 meters. Even without any search area computation (it is the entire image) or warping, the detection by correlation allows a robust tracking initialization.

2) Tracking: The tracker is initialized at the 125-th frame using the 170-th frame patch (Figure 7d). Because the patch is not warped, pose estimation (Figure 8e) and model projection during the first frames (Figure 8a) are not perfect. But after 30 frames, the tracker succeeds to localize the carrier (Figure 8b) and to track it until its disappearing by the aircraft radome occlusion 175 frames later (Figure 8d). Tracking is robust to luminance variations due to the sun reflections and artifacts in the cockpit. The algorithm time computation average is 14 ms with a standard deviation of 7 ms.

VI. CONCLUSION AND FUTURE PERSPECTIVES

A method using a correlation detection and a robust 3D model-based tracker has been applied to the localization of an aircraft carrier. This problematic is defined by its wide range of working and some on-board sensors allow to simplify the initialization of the tracking. The method has been demonstrated on simulation with blurred images and with a low-quality video on a representative part of the study range. To improve the application range, fusion with others sensors as IMU or aircraft and carrier internal models will be used.

A future objective is to close the control loop and evaluate visual servoing for this application, where the aircraft is by nature a non-holonomic under-actuated vehicle.

Figure 1 .

 1 Figure 1. Visual and low-visibility approaches

Figure 2 .

 2 Figure 2. Carrier and deck size in the image in function of the distance

Figure 3 .

 3 Figure 3. Different frames of the problem

•b

  M t = bo M -1 b bo M t where bo M b contains the rotation part o R b (θ b , φ b ) of aircraft pose and bo M t is the part of the equation where TACAN distance and angular 2σ accuracies, δd t and δψ r , are integrated: bo M t =     cos( ψs )sin( ψs ) 0 dt cos( ψr ) sin( ψs ) cos( ψs ) 0 dt sin( ψr ) 0 0 1

Figure 4 .

 4 Figure 4. (a) TACAN measures ψr and dt. (b) Image taken at 2500 m with respect to the carrier with a 4 • FOV; Deck origin (red dot) estimated with an inaccurate TACAN; Search area (in red) due to TACAN 2σ inaccuracy

Figure 5 .

 5 Figure 5. (a) Reference Ir, (b) warped image Iw, (c) patch It

VII. ACKNOWLEDGMENTS

This work is supported by DGA and Dassault Aviation under contribution to Laurent Coutard's Ph.D. grant. The authors would like to thank Nicolas Depinoy, Denis Deshayes and Paul Wilczynski from DGA Flight Testing.


Franc ¸ois Chaumette Laurent Coutard

Jean-Michel Pflimlin

Automatic landing on aircraft carrier by visual servoing

The landing on carrier is a very difficult task even for trained pilots. This paper presents a method to land automatically using aircraft sensors and three visual features inspired by visual cues used by pilots. These features whose link with the aircraft state is established, are introduced in a control scheme using a linearized aircraft model. The control law demonstrates a large convergence domain using simulated visual features and a 3D tracker applied on synthetic images.

I. INTRODUCTION

Landing on an aircraft carrier is usually considered by pilots as one of the most difficult exercise, complicated by visibility conditions, carrier dynamics and small landing area. In function of visibility conditions, several approaches toward the carrier are used, as presented in [START_REF] Coutard | Visual detection and 3d model-based tracking for landing on aircraft carrier[END_REF]. In our case, the studied trajectory consists in beginning the descent at 7.5 km from the carrier and in putting the hook on a desired descent glide. To ensure landing accuracy, no flare is performed. Method can be sum up by keeping the descent rate and angle of attack constant to maintain the aircraft stability and to prevent it from stall.

Control of landing on carrier is not a new problem. It was studied using classical sensors as radar or relative GPS [START_REF] Sousa | Test Results of an F/A-18 Automatic Carrier Landing Using Shipboard Relative Global Positioning System[END_REF], which determine the error wrt a reference trajectory and correct it using a control law that can be optimal [START_REF] Vu | Integration of flight and carrier landing aid systems for shipboard operations[END_REF] or robust [START_REF] Subrahmanyam | H infinity Design of F/A-18A Automatic Carrier Landing System[END_REF]. Some prediction model of the carrier dynamics is implemented in [START_REF] Vu | Integration of flight and carrier landing aid systems for shipboard operations[END_REF] to improve the control.

Visual features used during the landing by the pilot were studied for several decades for cognitive and safety aspects. The aim was to understand what features are used by pilots and to determine their sensibility [START_REF] Galanis | Glide-Path Control Information from Runway Shape[END_REF] in order to model the human reactions and improve the pilot training. [START_REF] Entzinger | Visual Cues in Manual Landing of Airplanes[END_REF] presents a quite complete state of the art of visual features used to control the aircraft during the alignment, the approach and the landing. For example, the distance between the vanishing and the impact points allows pilots to follow the descent glide. The link between relative pose and visual features is established considering low angle assumption in [START_REF] Naish | Control information in visual flight[END_REF] and [START_REF] Wewerinke | A theoretical and experimental analysis of the outside world perception process[END_REF]. The landing on carrier was mainly studied under the scope of aiding systems dealing with the visibility of the optical landing systems. One of the methods used by naval pilots to land on carrier is to control its aircraft in order to focus the glide slope vector present on the Head-Up-Display (HUD) to a triangle mark on the deck as presented in Fig. 1a Jean-Michel.Pflimlin@dassault-aviation.com cue is the angle between the drop line and the deck line that is used by pilots to align wrt the landing deck (see Fig. 1b). This feature provides nice decoupling properties because the segments forming this angle are orthogonal in 3D. Visual servoing allows the control of a dynamic system using visual features as inputs. Features can be the estimated pose by vision (3D features) or point coordinates or line angles in the image (2D features). Classical visual servoing is well defined for six DOF robot controlled in velocity [START_REF] Chaumette | Visual servo control, part i: Basic approaches[END_REF]. This is not the case of aircrafts, which are non-holonomic and under-actuated systems and are not directly controllable in velocity. Visual servoing was used in previous works to land on static runway [START_REF] Rives | Linear structures following by an airship using vanishing point and horizon line in a visual servoing scheme[END_REF][11] [START_REF] Gonc ¸alves | Homography-based visual servoing of an aircraft for automatic approach and landing[END_REF]. At the authors knowledge, it was never used for landing on a moving deck.

In order to design a control law, most of the works model the aircraft as linearized in the space state during the approach or alignment phase of landing. The difference between these works relies on the choice of the features and on the kind of controller. [START_REF] Rives | Linear structures following by an airship using vanishing point and horizon line in a visual servoing scheme[END_REF][13] [START_REF] Bourquardez | Visual servoing of an airplane for alignment with respect to a runway[END_REF] and [START_REF] Azinheira | Image-Based Visual Servoing for Vanishing Features and Ground Lines Tracking: Application to a UAV Automatic Landing[END_REF] use as image features the vanishing point and the composition of angles of the projected runway lines. This set of features presents good decoupling properties. [START_REF] Gonc ¸alves | Homography-based visual servoing of an aircraft for automatic approach and landing[END_REF] uses as visual feature an homography matrix that is the transformation between two images representing the same planar scene. Non linear control scheme using aircraft sensors (Inertial Measurement Unit, Pitot tubes) and camera is studied in [START_REF] Bras | Nonlinear Image-Based Visual Servo Controller for Automatic Landing Guidance of a Fixed-Wing Aircraft[END_REF]. Features are expressed in Plücker coordinates and used to build a centroid vector that is used as input of the control law. In order to improve aircraft behavior, [START_REF] Bourquardez | Visual servoing of an airplane for alignment with respect to a runway[END_REF] takes into account aircraft dynamics by modifying the desired features along the aircraft trajectory. In order to be robust to the wind, classical proportional integral scheme is applied in [START_REF] Rives | Linear structures following by an airship using vanishing point and horizon line in a visual servoing scheme[END_REF] [START_REF] Azinheira | Image-Based Visual Servoing for Vanishing Features and Ground Lines Tracking: Application to a UAV Automatic Landing[END_REF][12] and a wind estimator is built in [START_REF] Bras | Nonlinear Image-Based Visual Servo Controller for Automatic Landing Guidance of a Fixed-Wing Aircraft[END_REF]. This paper presents a new set of three visual features for the landing on aircraft carrier: the x-coordinate of the vanishing point, the distance in the image between the vanishing and impact points, and the angle between deck axis and its width. The second and third features have never been used before. A classical control scheme based on a linearized aircraft model of the Rafale allows the carrier landing even when it moves. Whereas some desired features used in [START_REF] Bourquardez | Visual servoing of an airplane for alignment with respect to a runway[END_REF][14] [START_REF] Gonc ¸alves | Homography-based visual servoing of an aircraft for automatic approach and landing[END_REF] are varying along the glide and need some external information to be computed, our desired features remain constant during the task and they are defined in function of the desired glide. Images are provided by the Damocles designation and recognition pod from Thales Optronics [START_REF] Coutard | Visual detection and 3d model-based tracking for landing on aircraft carrier[END_REF]. It is located under the aircraft on its rightside and has two fields-of-view (1 and 4 • ) and has Roll-Tilt degrees of freedom. This imaging sensor is well suited for this application because the detection and tracking has to be effected in wide space [START_REF] Coutard | Visual detection and 3d model-based tracking for landing on aircraft carrier[END_REF]. Other sensors embedded on the aircraft can be used as Inertial Measurement Unit (IMU). Moreover, the method is independent of GPS and landing aids systems presented in [START_REF] Coutard | Visual detection and 3d model-based tracking for landing on aircraft carrier[END_REF].

The paper is organized as follow: modeling of the aircraft is described in Section II, visual features are investigated in Section III. Using the control law defined in Section IV, simulation results are presented in Section V.

II. MODELING

A. Aircraft model

This section presents the different frames involved, notations and the aircraft dynamics linearized model. F o , F b and F c are respectively a fixed reference, aircraft and camera frames, as presented in Fig. 2. F d corresponds to the deck frame, along which the impact frame F i is located on the x-axis with the same orientation than F d . Pitch and roll orientation of the carrier are not taken into account in this study. The pose vector of the aircraft in F i is defined by P = (p, Φ) = (X, Y, Z, φ, θ, ψ), where the orientation Φ is represented using Euler angles [START_REF] Stevens | Aircraft control and simulation[END_REF]. The aircraft velocity screw in the aircraft frame F b is defined by v b = (υ b , ω b ) = (u, v, w, p, q, r), whose three first and last components are respectively linear and rotational parts. Both velocity screw v b and orientation Φ are provided by the Inertial Measurement Unit. V and α are respectively the aerodynamics velocity and the angle of attack and are provided by pitot tubes. Y , Z and ψ can be related to visual information. Assuming flat-earth hypothesis (neglecting the accelerations produced by the Earth rate), some kinematics relations can be derived using the Direct Cosine Matrix B B and the attitude propagation matrix ξ [START_REF] Stevens | Aircraft control and simulation[END_REF], respectively defined by:

Φ = ξ(Φ) ω b ṗ = B T B υ b (1)
Considering classical small φ and ψ angles, and low movement assumption, which is respected for the flight phase, the model is linearized at an equilibrium flight configuration:

B B =   cθ 0 0 -sθ 0 0 1 0 sθ 0 0 cθ 0   ; ξ -1 =   1 0 -sθ 0 0 1 0 0 0 cθ 0   (2)
where θ 0 is the trim value of the aircraft pitch for this flight configuration. cθ 0 = cos θ 0 and sθ 0 = sin θ 0 . Aircraft dynamics is defined by a non-linear model [START_REF] Stevens | Aircraft control and simulation[END_REF]. A classical method to study a system in order to design a control scheme is to consider a linearization point. For this trim point, longitudinal and lateral motion (respectively defined with lg and lt subscripts) are supposed decoupled:

ẋlg ẋlt = A lg 0 0 A lgt xlg xlt + B lg 0 0 B lgt ũlg ũlt (3)
where xlg = Ṽ , α, θ, q, Z and xlt = p, r, φ, ψ, Ỹ . x and ũ express the variations of the state x and the inputs u around the linearization point defined by x 0 and u 0 as x = xx 0 and ũ = uu 0 .

As many modern aircrafts, fly-by-wire augmentation stability systems control the aircraft. Thus, longitudinal and lateral control inputs, u lg and u lt are respectively the commanded normal acceleration a zc , the throttle τ and the commanded roll-rate p c .

B. Camera

The orientation of the camera is fixed in the aircraft frame and is equal to:

c R b =   0 1 0 0 0 1 1 0 0     cθ c 0 -sθ c 0 1 0 sθ c 0 cθ c   (4)
with θ c = -θ 0 + δ 0 is the tilt angle of the camera, where δ 0 = -tan -1 Z0 X0 is the glide slope angle to the impact point (δ 0 is about -4 • ). X 0 and Z 0 are longitudinal and altitude distances in F i at the linearization point. This camera orientation allows to implicitly center the impact point of the carrier in the image all along the desired landing trajectory.

III. VISUAL FEATURES

As presented in Section I, visual features choose as inputs of the control scheme for the autonomous landing, rely on visual cues used by pilots. Proposed visual features s are represented on Fig. 3. To control the longitudinal motion, the distance l y is well adapted because it is representative of the angle δ between impact point and horizon line as presented on Fig. 4. l y corresponds to the distance between the vanishing and the impact points along the y-axis of the image. In order to maintain a constant glide slope, l y In order to align to the deck, the x-axis position of the vanishing point x f (commonly used in [START_REF] Rives | Linear structures following by an airship using vanishing point and horizon line in a visual servoing scheme[END_REF][13] [START_REF] Bourquardez | Visual servoing of an airplane for alignment with respect to a runway[END_REF]) and the angle θ r are used. The angle θ r is defined by two orthogonal segments belonging to the deck plane. Both provide information related to the cap and lateral position between carrier and aircraft. For example, configuration of Fig. 3 lets think that the camera is placed at the left of the deck alignment line and is not oriented in the deck axis.

At the authors knowledge, we recall that features l y and θ r were never used before in visual servoing to land on static or moving runway, while we will see they allow obtaining satisfactory results.

A. Visual features modeling

The relation between the camera velocity v c (expressed in the camera frame F c ) and visual features s is defined by the interaction matrix L s [START_REF] Chaumette | Visual servo control, part i: Basic approaches[END_REF]:

ṡ = L s v c (5)
with the well-known form for an image point:

L x = L x L y = -1 Z 0 x Z xy -(1 + x 2 ) y 0 -1 Z y Z 1 + y 2 -xy -x (6)
where x, y, Z are respectively the x and y coordinates of the point in the image and the depth of the 3D point in the camera frame. The interaction matrix of the vanishing point corresponds to equation ( 6) considering Z tends to infinity, leading to invariance wrt any translational motion.

In the general case, the interaction matrix of a segment orientation θ = tan -1 ( y1-y2 x1-x2 ) (using coordinates of points x 1 and x 2 ) is defined by:

L θ =         D l sθ -D l cθ -D(x m sθ -y m cθ)/l -x m s 2 θ + ym 2 s2θ xm 2 s2θ -y m c 2 θ -1         T (7)
where D = 1 Z1 -1 Z2 , l is the length of the segment and x m and y m are the coordinates of its middle point.

The three visual features are defined by:

s = l y , x f , θ r = y f -y i , x f , θ d -θ p (8)
where θ d and θ p are the orientations of projected deck alignment line and its orthogonal line belonging to the deck plane at the impact point. From (6), we directly obtain:

L ly = 0 1 Zi -yi Zi y 2 f -y 2 i -x f y f + x i y i -x f + x i (9) L x f = 0 0 0 x f y f -(1 + x 2 f ) y f ( 10
)
The interaction matrix of the angle θ r is obtained from equation ( 7) for the two segments respectively defined by couples of points (i,f ) and (l,r), they are respectively referenced by subscripts d and p :

L θr =        D d l d sθ d - Dp lp sθp -D d l d cθ d + Dp lp cθp -D d (x dm sθ d -y dm cθ d ) l d + Dp(xp m sθp-yp m cθp) lp -x dm s 2 θ d + y dm 2 s2θ d +xp m s 2 θp- yp m 2 s2θp x dm 2 s2θ d -y dm c 2 θ d - xp m 2 s2θp+yp m c 2 θp 0        T (11) Note that D d = 1
Zi since the point f is the vanishing point. For any configuration of the camera, rotation around the zaxis has no effect on the feature θ r . For other components, some simplifications appear at the desired configuration.

B. Interaction matrix at the desired configuration

At the desired configuration, the aircraft is located along its ideal glide slope as presented in Fig. 4 and the camera is oriented as explained in Section II-B. For this desired configuration, visual features are given by x i = 0, y i = 0, x f = 0, θ d = π/2, θ r = 0, D p = 0, x dm = 0, y pm = 0 and l d = -y f . 11), the interaction matrix L s * at desired configuration is given by:

L s * =   0 1 Zi 0 y 2 f 0 0 0 0 0 0 -1 y f -1 |y f | Zi 0 0 0 0 0   (12)
where Z i and y f are different of zero.

The interaction matrix L s * highlights why these features are interesting to land. Indeed, l y is closely linked to longitudinal motions υ y and ω x whereas x f and θ r allow to control respectively lateral velocity υ x and relative cap ω y .

C. From pose to visual features

We now present the link between the aircraft pose and visual features in order to determine the relation between them at desired position. The carrier velocity is not taken into account in this modeling. Using [START_REF] Entzinger | Visual Cues in Manual Landing of Airplanes[END_REF], the link between aircraft velocity v b and time derivative of visual features s is established by:

ṡ = L s c W b v b (13)
where c W b is the velocity screw transformation matrix:

c W b = c R b [ c t b ] × c R b 0 c R b (14)
with c R b and c t b are the pose of the aircraft in the camera frame F c and [ ] × is the skew matrix expression. For simplification purpose, c t b is equal to zero (we consider the camera at the cog of the aircraft).

In order to express the relation with pose derivative, (13) becomes:

ṡ = L s c W b b W i Ṗ (15)
where b W i is the transformation matrix obtained from (1):

b W i = B B 0 0 ξ -1 (16)
Instantiating [START_REF] Bras | Nonlinear Image-Based Visual Servo Controller for Automatic Landing Guidance of a Fixed-Wing Aircraft[END_REF] around the trim point defined in Section II-A, it leads to:

l y = y 2 f θ - s2δ 0 2 X 0 Ẋ - c 2 δ 0 X 0 Ż (17) ẋf = (y f cθ c -sθ c ) φ -(cδ 0 + y f sδ 0 ) ψ (18) θr = cδ 0 |y f | X 0 Ẏ ( 19
)
where cδ 0 = cos δ 0 and sδ 0 = sin δ 0 . The link between the derivative of pose and the derivative of visual features is rewritten as:

ṡ = J s Ṗ (20)
with, considering small angle assumption (cδ 0 = 1, sδ 0 = 0):

J s ≈ 0 0 -1 X 0 0 y 2 f 0 0 0 0 y f cθc-sθc 0 -1 0 1 |y f | X 0 0 0 0 0 (21)
Assuming small displacement around a given pose, the integration of relation (20) leads to:

s = J s P (22)
This constitutes the relation between the pose and the visual features that will be used in the control law. Visual features l y , x f and θ r are respectively related to altitude, relative cap and lateral position of the aircraft with respect to the impact frame.

IV. CONTROL LAW

The objective of the control is to make landing the aircraft using its internal state measured with the IMU and visual features provided by the camera.

A. System outputs

States of the aircraft, presented in equation ( 3) are observable using aircraft sensors and visual measurements. Thus, longitudinal and lateral outputs are composed by:

y lg = V, α, θ, q, l y (23) y lt = p, r, φ, x f , θ r (24)
and linked to the states by ỹlg = C lg xlg and ỹlt = C lt xlt where matrix C lg and C lt are easily obtained from ( 22):

C lg = I 4×4 0 4×1 0 0 y 2 f 0 -1 X0 (25) C lt =   I 3×3 0 3×2 0 0 y f cθ c -sθ c -1 0 0 0 0 0 1 |y f | X0   (26)
I r×c and 0 r×c are respectively Identity and Zeros matrix, with r and c the number of rows and columns.

B. State feedback control law

As in previous works ( [13][11][12][10]), state feedback is used as control law. As the state is not directly available, state is expressed using visual features contained in ỹ. Inputs are computed as follow:

ũlg ũlt = - K lg 0 0 K lt xlg xlt = - K lg C -1 lg 0 0 K lt C -1 lt ỹlg ỹlt ( 27
)
The system is regulated around the point y * lg = (V * , α * , θ * , 0, l * y ) and y * lt = (0, 0, 0, 0, -π/2). It has to be emphasized that desired state remains constant all along the descent, which is not the case with previous methods.

State feedback gains K lg and K lt are computed using optimal LQ method with weighting matrix defined in [START_REF] Stevens | Aircraft control and simulation[END_REF]:

Q = diag(1/x 2 imax ) (28) R = diag(1/ũ 2 imax ) (29)
In order to simplify the control and demonstrate its robustness, matrix C lg and C lt are computed for a given X 0 corresponding to the mean of the longitudinal difference. Elements θ * c and y * f do not vary along the desired descent.

V. RESULTS

This section presents results of the control law proposed in the previous section. To demonstrate the control law validity, it was tested with a simulation software provided by Dassault Aviation, which contains a realistic non-linear model of the stabilized aircraft. We consider a moving carrier at 20 kts (about 10.3 meters per second) with no attitude movement (this case will be further investigated). The wind is not considered. As usual, the deck axis of the carrier is oriented with respect to the carrier axis of an angle of about 7 • . For the two simulations, desired outputs V * , α * , θ * and l * y are respectively equal to 67 meters per second, 14 degrees, 10.3 degrees and -0.07 meters.

A. Simulated visual features

Some assumptions are led in this part whose purpose is to validate the control law. Visual features values are considered available, there is no image processing step. The aircraft is initialized near the usual position at about 7 km from the carrier. Vertical and lateral shift with respect to the ideal trajectory are respectively equal to 100 and 240 meters; moreover, a cap difference of 5 degrees is also introduced.

The alignment and landing tasks are performed with a good behavior as presented on Fig. 5, 6 and 7 which respectively present longitudinal and lateral trajectories, positioning errors between aircraft and carrier, aircraft angles and visual features. Considering a far initial position and the carrier movement, longitudinal and lateral errors (Fig. 6b) decrease smoothly with angles remaining acceptable for the aircraft. Indeed, the angle of attack, which is closely linked to the aircraft stall and stability, is varying very few, and velocity variation is inferior to 5 m/s. Moreover, lateral angles φ and ψ remain in the aircraft capabilities, even with a large initial position error. In the image, the time evolution of features is smooth. The small jump in the vertical trajectory at the beginning is due to the fact that, at this moment, the pitch angle is increasing (Fig. 6a). In the image (Fig. 7), the segment [if ] evolves far from the desired position, l y is not invariant to pitch variation as presented in equation [START_REF] Gonc ¸alves | Homography-based visual servoing of an aircraft for automatic approach and landing[END_REF]. Moreover, the lateral dynamics has an impact on l y since we consider only the y component. Nevertheless, these couplings have not a significant effect since the landing is performed well. Considering that the carrier velocity is not taken into account, a small tracking error appears (more visible for lateral error on Fig. 6b). But this error is reduced as the distance between aircraft and carrier is reduced. Usually, tracking error remains constant at constant velocity, but using our visual features, the tracking error converges to zero at the landing (at the end of the simulation lateral error is equal to 11 cm). The lateral control is also performed without an important coupling with longitudinal control. It remains a small difference in the relative cap ψ r of about 1 • (Fig. 6a) due to the carrier velocity. In the image, it leads to a small difference for x f and θ r (Fig. 7) wrt the desired values.

Considering that the initial position is far from the carrier, and there is no trajectory following, the control scheme and the visual features allow to perform alignment and landing tasks with a compatibility of the aircraft dynamics.

B. Control using a 3D-model based tracker

The control law was tested with a 3D-model based tracker (presented in [START_REF] Coutard | Visual detection and 3d model-based tracking for landing on aircraft carrier[END_REF]) in a simulation environment based on a high fidelity visualization software. The sampling time of the loop is set to 50 ms. The tracker provides directly the estimated pose c

Md between the deck and the camera. In this simulation, camera used is the Damocles pod presented in Section I. Using the estimated pose, the pod tilt and roll are controlled to center the carrier in the image. Knowing the Damocles camera orientation and position c M b , the pose between the aircraft and the deck b M d can be computed. We then consider a virtual static camera c v whose orientation is Md ). To summarize the method, the visual tracking is done with the swiveling Damocles camera c whereas the visual control of the aircraft is performed with a virtual fixed camera c v . Results using the 3D tracker are presented in Figs. 9 and 8 and in the enclosed video. To take into account the model based tracker and the camera with a FOV of 4 • , the task is performed from 3500 meters from the carrier to near the touch point, and gains of the control law were reduced. In order to be representative of the operational conditions, the aircraft is initially shifted of 20 and 35 meters in vertical and lateral positions. The landing phase is performed by taking into account all the loop: dynamics, sensors, perception and control law. As in previous results, the position errors (Fig. 8c) converge to zero with a damped behavior without exceeding limitation of angles (Fig. 8b). Of course, measures of the visual features (Fig. 9c) are noisier due to the image processing step and it has to be highlighted that no filter is implemented. Moreover, it has to be emphasized that the 3D-model based tracker succeeds to track the carrier over a wide variation of the carrier size in the image (see Figs. 9a and9b).

VI. CONCLUSION AND FUTURE PERSPECTIVES

This paper has presented a new set of visual features to perform automatic landing on aircraft carrier. These features were linked to the aircraft pose wrt the carrier and a control scheme was built using a linearized aircraft model. Features and control law were first tested using simulated visual features, and then on synthetic images using a 3D model based tracker. As perspectives, control schemes taking into account the wind and the carrier velocity will be investigated. Other projection models, such as spherical projection, will be also studied.

Fig. 1 .

 1 Fig. 1. (a) Pilots try to put the HUD velocity vector at the basis of the triangle mark in order to follow the glide slope; (b) The angle between the drop and the alignment lines (yellow and dash) provides lateral information

FiFig. 2 .

 2 Fig. 2. Frames and velocities

Fig. 3 .

 3 Fig. 3. Features ly, x f and θr in a general configuration

Fig. 4 .

 4 Fig. 4. Desired configuration for landing

Fig. 5 .Fig. 6 .

 56 Fig.5. Lateral and vertical trajectories of aircraft and impact point of carrier (meters); Black dash line corresponds to the glide desired; Blue dash lines are the orientation of the deck at the beginning and the end of simulation

Fig. 7 .Fig. 8 .

 78 Fig. 7. Visual features simulated: Time evolution of visual features x f , ly (in meters) and θr (in degrees); Vanishing and impact point trajectories (f and i) in the image plane (in pixels)

Fig. 9 .

 9 Fig. 9. First (a) and last (b) images of the simulation, green is the model projection; (c) Time evolution of visual features (in meters for x f and ly, in degrees for θr) and image trajectories of the features (pixels)

  . Another Laurent Coutard and Franc ¸ois Chaumette are with INRIA Rennes-Bretagne Atlantique and IRISA, Campus de Beaulieu,

	35042	Rennes	cedex,		France	{Laurent.Coutard,
	Francois.Chaumette}@inria.fr
	Jean-Michel	Pflimlin	is	with	Dassault	Aviation	78,
	Quai Marcel Dassault -92552 Saint Cloud Cedex 300

VII. ACKNOWLEDGMENTS This work is supported by DGA and Dassault Aviation under contribution to Laurent Coutard's Ph.D. grant. The authors would like to thank Eva Cruck from DGA and Bruno Patin from Dassault Aviation.