Guido Gioioso
email: gioioso@dii.unisi.it

Markus Ryll
email: [markus.ryll@tuebingen.mpg.de

Domenico Prattichizzo
email: prattichizzo]@dii.unisi.it

Heinrich H Bülthoff

Antonio Franchi
email: antonio.franchi@laas.fr

Turning a Near-hovering Controlled Quadrotor into a 3D Force Effector

In this paper the problem of a quadrotor that physically interacts with the surrounding environment through a rigid tool is considered. We present a theoretical design that allows to exert an arbitrary 3D force by using a standard nearhovering controller that was originally developed for contactfree flight control. This is achieved by analytically solving the nonlinear system that relates the quadrotor state, the force exerted by the rigid tool on the environment, and the nearhovering controller action at the equilibrium points, during any generic contact. Stability of the equilibria for the most relevant actions (pushing, releasing, lifting, dropping, and leftright shifting) are proven by means of numerical analysis using the indirect Lyapunov method. An experimental platform, including a suitable tool design, has been developed and used to validate the theory with preliminary experiments.

I. INTRODUCTION

Over the past decade, Unmanned Aerial Vehicles (UAVs) attracted an increasing interest in the robotics community, due to their versatility and to the many potential indoor and outdoor applications, including, e.g., landscape survey, surveillance and reconnaissance, manufacturing and logistics automation. In this context UAVs have been mainly considered as flying sensors rather than flying actuators, and the possibility of using flying robots for mobile manipulation, i.e., enabling physical interaction with the surrounding environment, has not been sufficiently investigated yet. In fact, designing novel flying robotic systems able to perform this kind of interaction would open a much wider range of possible applications in the aerial robotics field.

Among the UAV platforms, the quadrotor gained a lot of popularity in the robotic community, due to its mechanical simplicity and robustness. Being able to hover and perform vertical take-off and landing, it represents also an ideal platform for developing and testing many new aerial manipulation concepts. In [START_REF] Lee | Mechanics and control of quadrotors for tool operation[END_REF] a control law for a quadrotor equipped with a rigid tool is proposed and tested in simulation. The controller is based on feedback linearization thus requiring for a rather precise knowledge of the system mechanical parameters and state estimation that is sometime hard to find in real application scenarios. Furthermore, some caution is needed when dealing with a possible unstable zero dynamics. A passivity-based controller is presented in [START_REF] Fumagalli | Modeling and control of a flying robot for contact inspection[END_REF] for a quadrotor endowed with a delta-shaped manipulator. Using a decoupled approach the actuation of the manipulator is employed to apply forces on a wall relying on the fact that the flight controller is able to keep steady the UAV position during contact. The authors in [START_REF] Bellens | A hybrid pose/wrench control framework for quadrotor helicopters[END_REF] present a hybrid framework with the goal of letting the quadrotor follow a desired trajectory while exerting a force on the environment. The approach relies on the empirical identification of a map that relates the exerted measured wrench with the thrust and torque inputs of the used quadrotor platform. In [START_REF] Srikanth | Controlled manipulation with multiple quadrotors[END_REF] a control framework is presented that allows the one-dimensional manipulation of a cart performed by one or two quadrotors through a rigid tool. In [START_REF] Marconi | Modeling and control of a flying robot interacting with the environment[END_REF] a framework for robust control of a UAV for docking-undocking operation on a vertical surface based on a path following strategy is developed and tested in simulation. A recent work [START_REF] Yüksel | Reshaping the physical properties of a quadrotor through ida-pbc and its application to aerial physical interaction[END_REF] proposes an interaction and damping assignment passivity-based controller (IDA-PBC) that is able to change the apparent dynamical parameters of a quadrotor. The authors have shown that this controller can be successfully used to shape the dynamical behavior of a quadrotor when in contact with a physical environment. Finally, studying the physical interaction with the environment is essential for applications in which UAVs are used as mobile manipulating platforms, see, e.g., [START_REF] Orsag | Modeling and control of MM-UAV: Mobile manipulating unmanned aerial vehicle[END_REF], [START_REF] Lippiello | Exploiting redundancy in cartesian impedance control of UAVs equipped with a robotic arm[END_REF].

In this paper we theoretically formulate the problem of controlling a quadrotor UAV endowed with a rigid tool whose tip has to be used in order to exert a prescribed 3D force on the environment. This is a basic task whose study may be instrumental for more complex manipulation behav-ior performed by a single or multiple UAVs. The advantage of this approach is to obtain a structure that is light-weight, mechanically simple and with a larger time autonomy than a single bulky UAV equipped with a manipulator with several DOFs. Moreover, a system with multiple UAVs endowed with simple tools might result even more effective and versatile as shown, e.g., in [START_REF] Gioioso | Hand driven UAV formation for cooperative grasping and transportation: the flying hand[END_REF] where it is presented the control of a flying hand constituted by multiple simple UAVs each one acting as a single finger.

The model of a quadrotor endowed with a rigid tool is introduced in Sec. II along with a control law suitable for force exertion. The control scheme is based on the nearhovering linearizing control approach that is commonly used for contact-free flight on many quadrotor UAV platforms. The adoption of a widely used control law is the main strength of this approach, which brings many advantageous consequences: i) our solution can be easily applied by other research labs by slightly adapting the built-in controller that is already present on their platform; ii) the same controller is used both for the free flight phase and for the contact phase thus being exempt from the need for exact contact detection and switching-related problems; iii) despite being conceived for a near-hovering contact-free flight, during contact the controller can be used with high tilt angles; furthermore, iv) the control law is based neither on full cancellation, as feedback linearization, nor on sliding surfaces, thus it is expected to have a robust behavior in real world conditions.

By virtue of the analytical study carried-out in Sec. III on the nonlinear equilibria of the controlled system, the controller can be effectively used to apply, at steady state, any given 3D contact force at the contact point. Stability of the controlled system at the equilibria corresponding to the given forces is numerically proven in Sec. IV for the most important interaction cases, such as pushing, releasing, lifting, dropping, and left/right shifting.

The implementation of an experimental platform which includes the design of a tool that can be easily mounted on a quadrotor to establish contact with the environment, is presented in Sec. V where a preliminary experiment in the dynamic case (i.e., time-varying force profile) is also shown. Section VI concludes the paper and presents future developments of this work.

II. MODEL AND STANDARD NH CONTROLLER

In this section we present the notation used throughout the paper while briefly introducing the standard quadrotor dynamical model. Refer to Fig. 1 for an illustration. An inertial world frame W : {O W , w 1 , w 2 , w 3 } and a quadrotor body frame B : {O B , , b 1 , b 2 , b 3 } are considered. The origin of the body frame is coincident with the center of mass of the quadrotor which is also the geometric center of the four propellers. We indicate with p = (p 1 , p 2 , p 3 ) T ∈ IR 3 the position of O B in W and with v = (v 1 , v 2 , v 3 ) T ∈ IR 3 the corresponding linear velocity. The scalar m ∈ IR denotes the mass of the quadrotor and J ∈ IR 3×3 its inertia matrix with respect to the body frame. Finally, let R ∈ IR 3×3 represent the rotation matrix from the body frame to the inertial world frame and ω ∈ IR 3 be the angular velocity vector that represents the rotation of B with respect to W , expressed in B. The orientation of the quadrotor is represented by η := (φ , θ , ψ) T ∈ IR 3 , with φ , θ , ψ being respectively the roll, pitch and yaw angles (RPY). The rotation matrix can be written using RPY as R(η) = R z (ψ)R y (θ )R x (φ ), where R x , R y , R z are the elementary rotation matrices about the three coordinate axis.

As common practice in literature we assume as control inputs the total thrust acting on the -b 3 direction, denoted with λ ∈ IR + , and the total moment expressed in B, denoted with τ = (τ 1 , τ 2 , τ 3 ) T ∈ IR 3 . Being the four inputs less than six (the number of configuration parameters p, η) the quadrotor is an underactuated system. Nevertheless its actuation is enough to exert any 3D force on the environment.

We assume that the quadrotor is equipped with a tool whose tip position in body frame is expressed by the constant vector d = (d x , d y , d z ) T ∈ IR 3 . Without loss of generality we take b 2 aligned with the projection of the tool on the plane spanned by b 1 and b 2 , thus resulting in d y = 0. Let y ∈ IR 3 be the position of the tool-tip in the world frame:

y = p + Rd. (1)
The environment interacts with the quadrotor by exerting a force f e ∈ IR 3 on the tool-tip, which is expressed in the world inertial frame. The interaction moment at the tool-tip is assumed negligible. This assumption is widely used in literature and it is referred to as hard finger contact model [START_REF] Prattichizzo | Grasping[END_REF]. Under these assumptions, the equations of motion of the quadrotor can be written as:

ṗ = v (2) mv = -λ Rẑ + mgẑ + f e (3) η = T(η)ω (4)
J ω = -ω × Jω + τ + d × R T f e , (5)
where ẑ = (0, 0, 1) T and T(η) ∈ IR 3×3 transforms ω to the RPY rates η.

In the following we adopt the common assumption that the quadrotor is never in a singular configuration (i.e., φ = θ = 0), in fact if this happens, the total thrust λ cannot be used to counterbalance the gravity force, thus consisting in a useless case for our purposes.

A. Standard Near-hovering Controller

In this section we revise a control law that has been widely used in literature (see, e.g., [START_REF] Michael | The GRASP multiple micro-UAV testbed[END_REF], [START_REF] Franchi | Bilateral teleoperation of groups of mobile robots with time-varying topology[END_REF], [START_REF] Lee | Semi-autonomous haptic teleoperation control architecture of multiple unmanned aerial vehicles[END_REF]) to perform near-hovering and contact-free trajectory tracking. Even though this controller is well known in literature, it is essential to recall its main structure and notation for the following Sec. III where we will show how this controller can be effectively used to perform a contact force on the environment even in far-hovering conditions.

An inner-loop PID controller tracks some desired RPY angles 3 , relying on the fact that the rotational dynamics is fully actuated. In fact, differentiating (4) and substituting (5), we get

η d = (φ d , θ d , ψ d ) T ∈ IR
η = T(η)J -1 (-ω × Jω + τ + d × R T f e ) + Ṫ(η)ω.
The PID attitude regulation controller is thus defined as:

τ = JT -1 (η)K A η e (6)
where:

K T A =            k φ DA 0 0 k φ PA 0 0 k φ IA 0 0 0 k θ DA 0 0 k θ PA 0 0 k θ IA 0 0 0 k ψ DA 0 0 k ψ PA 0 0 k ψ IA            , η e =          - φ φ d -φ (φ d -φ ) dt - θ θ d -θ (θ d -θ ) dt - ψ sin(ψ d -ψ) sin(ψ d -ψ) dt          (7)
that leads to the closed-loop dynamics:

η + K A η e = TJ -1 (-ω × Jω + d × R T f e ) + Ṫω.
It has been shown that, in contact-free flight and under suitable conditions of the initial state and gain selection, (η e , ηe ) is locally exponentially stable.

Since in contact-free flight this controller is meant to work in near hovering T -1 (η) can be approximated by a 3 × 3 identity matrix in [START_REF] Yüksel | Reshaping the physical properties of a quadrotor through ida-pbc and its application to aerial physical interaction[END_REF]. Furthermore, assuming that the J is diagonal, as common practice, we can hide J in the gain matrix K A thus obtaining a simpler expression for τ:

τ = K A η e . (8)
Note that even if the previous ones seem crude approximations, they have been proven to work very well in reality, see the previously cited work, when the task is to fly in near hovering. Furthermore, we are not interested in having excellent performances in free flight since our focus is in the contact case. An outer-loop controller is then designed by choosing the thrust such as to actuate the vertical dynamics

λ = - m cos φ cos θ (-g -b z P v 3 + k z PP e 3 ) (9)
where b z P and k z PP are two positive scalar gains and e 3 is an additional control input. Secondly, in order to act on the horizontal dynamics, the desired roll φ d and pitch θ d are set such that [START_REF] Prattichizzo | Grasping[END_REF] where e 1 , e 2 are two additional control inputs, and s • and c • denote sin(•) and cos(•) respectively. The interested reader is referred to [START_REF] Lee | Semi-autonomous haptic teleoperation control architecture of multiple unmanned aerial vehicles[END_REF] for the analytical derivation of [START_REF] Prattichizzo | Grasping[END_REF]. We gather the three additional control inputs in the vector e = (e 1 , e 2 , e 3 ) T ∈ IR 3 .

sin φ d sin θ d = m λ -s ψ c ψ -c ψ /c φ -s ψ /c φ -b P v 1 + k PP e 1 -b P v 2 + k PP e 2
Remark 1: (Behavior in contact-free flight) In the literature the controller (8),( 9), [START_REF] Prattichizzo | Grasping[END_REF] has been used to steer the robot during contact-free (i.e., f e = 0) motion. In fact, simply speaking, if we plug [START_REF] Gioioso | Hand driven UAV formation for cooperative grasping and transportation: the flying hand[END_REF] in the third row of (3) we obtain

m v3 = -b z P v 3 + k z PP e 3 , (11)
furthermore, if the attitude controller is fast enough, i.e., s.t. θ ≃ θ d and φ ≃ φ d , the first two rows of (3) become (see [START_REF] Lee | Semi-autonomous haptic teleoperation control architecture of multiple unmanned aerial vehicles[END_REF])

m v1 v2 ≃ -b P v 1 + k PP e 1 -b P v 2 + k PP e 2
Therefore, one can choose e = p dp in order to exponentially regulate p to a desired constant position p d .

III. NONLINEAR EQUILIBRIA DURING CONTACT

Departing from the assumption that is commonly done in the literature when dealing with such controller (see Remark 1) in this paper we consider the case when f e = 0 and possibly θ = θ d and φ = φ d . To this aim in this section we analytically derive the equilibria of the closed loop system (2)-( 5) subject to control (8),( 9), [START_REF] Prattichizzo | Grasping[END_REF]. In particular we are interested in the static nonlinear relationship between -f e , i.e., the force exerted at the equilibrium by the quadrotor on the environment through the tool-tip, and e, i.e., the 3D control input that is left free by the controller described in Sec. II. The goal of the following analysis will be to find e given a desired -f e . The equations to be solved in order to find the sought relation can be obtained by writing (3),( 5),( 8),( 9), [START_REF] Prattichizzo | Grasping[END_REF] at the equilibrium (i.e., v = v = ω = ω = 0). This results in the following nonlinear system of 12 scalar equations A. Total thrust λ and e 3 Equation ( 12) can be rewritten as:

-λ Rẑ + mgẑ + f e = 0 (12) τ + d × R ⊤ f e = 0 (13) τ -K A η e = 0 (14) λ c θ c φ -mg + k PP e 3 = 0 ( 15
)
s φ d s θ d - mk PP λ -s ψ c ψ -c ψ /c φ -s ψ /c φ
f e 1 = λ (c ψ c φ s θ + s ψ s φ ) (17
)

f e 2 = λ (c φ s ψ s θ -s φ c ψ ) (18
)

f e 3 + mg = λ c θ c φ (19)
from which it descends, after some straightforward algebra, that the total thrust must be:

λ = ( f e 1 ) 2 + ( f e 2 ) 2 + ( f e 3 + mg) 2 . ( 20
)
Moreover, using ( 19) and (15) the unknown e 3 can immediately be obtained as

e 3 = - f e 3 k PP . ( 21
)
Preprint version, final version at http://ieeexplore.ieee.org/

x = c φ s θ y = s φ , (22)
(17) and ( 18) can be rewritten as a linear system:

c ψ s ψ s ψ -c ψ x y = 1 λ f e 1 f e 2 ( 23
)
that can be easily solved thus obtaining the following expressions for s φ and s θ

s φ = 1 λ ( f e 1 s ψ -f e 2 c ψ ) (24)
s θ = 1 λ c φ ( f e 1 c ψ + f e 2 s ψ ). ( 25
)
Given the admissible values for φ , θ , it must be c φ > 0 and c θ > 0, therefore we have

c φ = + 1 -s 2 φ ( 26
)
c θ = + 1 -s 2 θ . (27)
Substituting τ in ( 13) from ( 14), we obtain:

-K A η e = d × R ⊤ f e . (28)
Moreover, from equation ( 16) it is straightforward to express s φ d and s θ d as

s φ d = mk PP λ (-s ψ e 1 + c ψ e 2 ) (29)
s θ d = mk PP λ c φ (-c ψ e 1 -s ψ e 2 ) (30)
The system composed by (24),( 25),(28),(29),(30) is thus made by 7 equations with 7 unknowns φ , θ , ψ, φ d , θ d , e 2 , e 3 .

From this point the solution changes depending on the values of the integral gains in K A .

1) No Integral Action (k

φ IA = k θ IA = k ψ IA = 0
): This is the most complex case. The third row of (28) is

K PA s ψ d -ψ = -x[ f 1 (c ψ s θ s φ -s ψ c φ ) + f 2 (s ψ s θ s φ + c ψ c φ ) + f 3 s φ c θ ] (31)
If d x = 0 then ψ = ψ d , and the solution for this case is given in Sec. III-B.2.

In the case of d x = 0 we first show how φ , θ , ψ can be separately found solving the subsystem formed by ( 24),(25),(31). In fact, plugging (24),( 25),( 26),( 27) into (31), after some algebra, we obtain a trigonometric equation in which the only unknowns are s ψ and c ψ , i.e., the yaw ψ. Using the tangent half-angle substitution (t ψ 2 = tan( ψ 2 )) the trigonometric equation that can be written as a 8-th order polynomial equation:

(α (2) +α (7) )t 8 ψ 2 -2(α (4) +α (8) )t 7 ψ 2 -4(α (2) -α (5) -α (6) )t 6 ψ 2 -2(4α (3) -3α (4) +α (8) )t 5 ψ 2 +2(8α (1) +3α (2) -4α (5) +4α (6) -α (7) )t 4 ψ 2 + 2(4α (3) -3α (4) +α (8) )t 3 ψ 2 -4(α (2) -α (5) -α (6) )t 2 ψ 2 + 2(α (4) +α (8) )t ψ 2 +α (2) +α ( 7) =0 (32) where the coefficients are defined as follows:

α (1) = (k α f e 1 c ψ d ) 2 ; α (2) = (k α f e 2 s ψ d ) 2 α (3) = -2k 2 α f e 1 c ψ d ( f e 1 s ψ d + f e 2 c ψ d ) α (4) = -2k 2 α f e 2 s ψ d ( f e 1 s ψ d + f e 2 c ψ d ) α (5) = k 2 α [( f e 1 s ψ d ) 2 + ( f e 2 c ψ d ) 2 + 4 f e 1 f e 2 s ψ d c ψ d ] α (6) = a 2 α -(k α λ c ψ d ) 2 ; α (7) = b 2 α -(k α λ s ψ d ) 2 α (8) = 2a α b α + 2(k α λ ) 2 s ψ d c ψ d and a α = f e3 1 + f e 1 f e2 2 -f e 1 λ 2 + f e 1 f e2 3 + f e 1 f e 3 mg b α = -f e3 2 -f e2 1 f e 2 + f e 2 λ 2 + f e 2 f e2 3 + f e 2 f e 3 mg, k α = -λ k PA d x .
The real solutions of (32), that are efficiently found numerically, represent only candidate solutions for ψ, since some of them hold for negative values of c θ and c φ , which are not allowed in our setting. In order to prune the candidates it is then sufficient to plug them in (24),( 25),( 26),( 27) and validate the found values by substitution in (31).

The variables φ d and θ d can be then computed from the first two rows of (28) and from the knowledge of φ , θ , ψ:

φ d = φ + d z k φ PA [ f e 1 (c ψ s θ s φ -c φ s ψ )+ f e 2 (s ψ s θ s φ +c φ c ψ )+ f e 3 s ψ c θ ] (33)
θ d = θ - d z k θ PA [ f e 1 (c θ c ψ ) + f e 2 (s ψ c θ ) -f e 3 s θ ]+ d x k θ PA [ f e 1 (c ψ c φ s θ + s θ s ψ ) + f e 2 (s ψ c φ s θ -s φ c ψ ) + f e 3 c θ c φ ] (34)
Finally, from (29),(30) it is easy to obtain

e 1 = λ mk PP (-s ψ s φ d -c ψ s θ d c φ ) (35)
e 2 = λ mk PP (c ψ s φ d -s ψ s θ d c φ ). (36)
Therefore φ , θ , φ d , θ d can be computed directly by plugging (37) in ( 24),( 25),( 26),( 29),(30). Finally, as in the previous case, e 1 , e 2 are obtained from ( 35) and (36), respectively.

3) Integral Action on φ and θ (k and θ = θ d . Therefore, equating the rhs of ( 24) with the rhs of ( 29) and the rhs of (25) with the rhs of (30) we obtain:

φ IA > 0, k θ IA > 0,
e 1 = - f e 1 mk PP e 2 = - f e 2 mk PP . (38)

C. Torques τ

In any case can be retrieved by using [START_REF] Slotine | Applied nonlinear control[END_REF], i.e.,

τ = -d × R ⊤ f e . ( 39
)

D. Short Discussion on the Equilibria

Let's observe the expressions of e as function of the desired force -f e and the desired yaw ψ d that we found in the three different cases considered in Sec. III-B.1, III-B.2 and III-B.3. First it can be observed that an analytical expression in closed-form can be written only when at least some integral actions are considered, while it must be numerically computed when

k φ IA = k θ IA = k ψ IA = 0.
Moreover, in all the cases the value of e 3 only depends on the value of f e 3 while this decoupling property holds for e 1 and e 2 only in the third case (where

k φ IA > 0, k θ IA > 0, k ψ IA any).
In this last case it can be also observed that the value of e does not depend on the position d of the tool-tip in body frame.

IV. STABILITY OF THE EQUILIBRIUM POINTS

This section presents a numerical stability analysis for the equilibrium points defined in Sec. III using the indirect Lyapunov method for the case of 'no integral action'. We plan to address the remaining cases in a future work. The dynamical system defined by ( 3),( 4),( 5) subject to control ( 8),( 9), [START_REF] Prattichizzo | Grasping[END_REF] can be interpreted as nonlinear system ẋ = f (x, u) with state x = (p, v, η, ω) and inputs u = (e, ψ d ). In Sec. III we found how to efficiently compute the equilibrium (x e , u e ) for any given external force f e . It is therefore possible to compute the Jacobian matrix ∂ f ∂ x in (x e , u e ) and efficiently check its eigenvalues. Negativeness of the largest real part of all the eigenvalues implies local asymptotical stability of the equilibrium (x e , u e ) for the nonlinear system [START_REF] Slotine | Applied nonlinear control[END_REF].

To this aim, we numerically computed the eigenvalues of the Jacobian matrix considering reasonable values for the system parameters and letting f e , ψ d and d vary over the domains of interest. In particular, we set m = 1 kg,

J = diag{0.13, 0.13, 0.22} kg•m 2 , k PP = 10 b P = 9, k z PP = 10, b z P = 9, k φ PA = k θ PA = k ψ PA = 12, k φ DA = k θ DA = k ψ DA = 7.
All the integral gains of the position and the attitude controller were set to zero. In all these cases we also set ψ d = 0. and consider all the possible positions of the tool-tip d according to the design described in Sec. V-A, i.e., d x ∈ {0.25, 0.27, 0.29, 0.31, 0.33, 0.35} and d z ∈ {-0.14, -0.11, -0.08, -0.05, 0.5, 0.8, 0.11, 0.14} (we recall that we assumed, without loss of generality, d y = 0). The forces are sampled considering five clusters that cover typical interaction cases such as pure pushing/releasing, lifting/dropping while pushing with low and high pushing force, and shifting left/right while pushing with low and high pushing force. Each of the five plots of Fig. 3 shows the evolution of the largest real part within each of the five clusters for every considered value of d x and d z . It is possible to see that in all the cases in exam the largest real part is negative thus resulting in a Hurwitz ∂ f ∂ x and, consequently, in a locally asymptotically stable equilibria for the nonlinear system. We now analyze more in detail each considered case.

1) Pushing/Releasing: In this cluster (first plot in Fig. 3) f e 1 has been regularly sampled from 0 N to -5 N with a sampling step 0.01 N while f e 2 and f e 3 have been keep equal to zero. Since ψ d = 0, this corresponds to the case in which the quadrotor is performing a frontal pushing on the environment with pushing forcef e 1 . In every case a smaller d x corresponds to a larger absolute value of the real part. This behavior is more evident when d z > 0, i.e., the tool-tip is placed below the Center of Mass (CoM). The absolute value is constant w.r.t. d z when d z > 0 and monotonically decreasing w.r.t. d z when d z < 0.

2) Lifting/Dropping while Pushing: In these two clusters (second and third plots of Fig. 3) f e 3 is sampled between 0 N to 2 N with sampling step 0.01 N and two different (constant) values are considered for f e 1 , corresponding to a low (-2.5 N) and a high (-5 N) pushing force, respectively. The component f 3 e is kept equal to zero. The trends w.r.t. to d x and d z are qualitatively comparable to the pure pushing case.

3) Shifting Left/Right while Pushing: In these two last clusters (fourth and fifth plots of Fig. 3) f e 2 is regularly sampled from -2 N to 2 N with a sampling step 0.01 N. The frontal pushing force f e 1 takes, as in the previous case, two different constant values (soft and hard pushing) and f 3 e is kept equal to zero. Since ψ d = 0 this corresponds to the case in which the quadrotor has to perform a lateral shift of the tooltip. It is interesting to note that in the case of low pushing force, basically d z does not affect the largest real part. Dependency from the distance d x is in accordance with the previous analyzed cases. Concerning the trend when f e 1 = -5 N we see that dependency from d x becomes relevant only when d z > 0 (tooltip below the CoM).

V. PRELIMINARY EXPERIMENTS

To further validate the theoretical and numerical analysis we developed an experimental setup that includes a in-house developed quadrotor (described in Sec. V-A), a Six-Axis force torque sensor 'Gamma' by ATI and an external motion capture system aimed at tracking the position and orientation of the quadrotor (see Fig. 1 for the CAD model and Fig. 5 for its implementation). In this Sec. V-B we report the outcome of a preparatory experiment that is designed to approximately perform a pushing and releasing maneuver.

A. Quadrotor and Tool Design

The quadrotor used to perform the experiments is based on the MK-Quadro MikroKopter platform. The overall mass including the tool is 1.2 kg. To increase the agility and decrease the size of the quadrotor we decided to use 20 cm diameter propellers. The overall diameter between two opposing propeller axis is l = 0.314 m. The distance vector between the tool-tip and the quadrotor body frame in performed experiment is d = (0.334, 0.0, 0.056) m. A copy of the tool was mounted on the opposite side in order to balance the weight of the one actually used to apply force on the sensor (see Fig. 1). The tool tip end effector (see Fig. 4) includes a gimbal bearing offering degrees of freedom in all the rotation axis (thus allowing the implementation of a hard finger contact). As it was mounted on our quadrotor, a big degree of freedom (±45 • ) was offered around the pitch axis. To stabilize this degree of freedom a spring system is keeping the end effector tool-tip in the zero position when not in contact. Minor degrees of freedom (±15 • ) are offered around the roll and yaw axis. The end effector tip additionally includes a micro switch that detects the contact.

B. Preliminary Result

In Fig. 5 a sequence of snapshots of the performed experiment is shown. The quadrotor equipped with the tool was remotely driven by a human operator to establish a contact with the surface. During the contact-free phase the operator commands a desired velocity to the UAV. During the contact phase the operator directly commands e 1 , by means of a joystick, to let the pushing force applied by the robot on the surface follow approximately a trapezoidal profile (relying on the quasi-linear relation of Fig. 2 top). The results obtained during the experiments are plotted in Fig. 6. The plots refer only to the contact phase. It is easy to observe how the force applied by the quadrotor and read by the sensor follows the shape of the desired e. Moreover the fingertip position remains practically constant during the experiment (no slippage occurs at the contact point). Finally in Fig. 7 the linear relationship between the applied pushing force and the signal e 1 commanded by the operator is plotted. The expected hysteresis is due to the fact that it is almost impossible for a human operator to fulfill the quasi-staticity of the motion due to the coarse control over e 1 allowed by a small joystick. For the same reason, the increasing and the decreasing phases are clearly asymmetric in Fig. 6. The reader is encouraged to watch the video attached to this submission in which the preliminary experiment is shown.

VI. CONCLUSIONS AND FUTURE WORK

In this paper we presented a control design that allows a quadrotor to exert a desired 3D contact force on a surface through a tool rigidly attached on it. We showed how the same near-hovering controller used for the free flight can be used in contact with the environment to generate a desired contact force. The stability of the system has been then studied and its dependence on the desired 3D force and the Preprint version, final version at http://ieeexplore.ieee.org/ 6

2014 IEEE ICRA position of the tool-tip in body frame has been investigated. The presented theory has been validated by preliminary experimental results in which a human operator manually increased and decreased the value of the error component e 1 , emulating a "Push and Release" task.

As future work we want to thoroughly study the dynamic case and perform full experiments with an autonomous robot in order to validate the approach in the different interaction cases considered in the paper. The desired interaction forces will be given as input to the system thus fully exploiting the theoretical results derived in Section III.

Fig. 1 :

 1 Fig. 1: CAD model of the quadrotor setup developed and used for our experiments and main frames and symbols used to model it.

  unknowns are the sought e 1 , e 2 , e 3 , together with φ , θ , ψ, φ d , θ d , λ , τ 1 , τ 2 , τ 3 . The admissible values of the variables are φ , θ ∈ (-π 2 , π 2 ), ψ ∈ (-π, π), and λ > 0. The known parameters are the external force components f e 1 , f e 2 , f e 3 , and m, g, d, K A , k PP and ψ d .

Fig. 2

 2 Fig.2shows the evolution of e 1 , e 2 , e 3 in this case, depending on f 1 , f 2 , f 3 for typical values of the mechanical and control parameters and ψ d = 0.2) Integral Action only on ψ (kφ IA = k θ IA = 0, k ψ IA > 0):In this case, because of the integral action, at the equilibrium we have that ψ = ψ d .(37)

Fig. 2 :

 2 Fig. 2: Evolution of e 1 , e 2 , e 3 depending on f 1 , f 2 , f 3 [N] in the case of 'no integral action'.

Fig. 3 :

 3 Fig. 3: Stability of the linearized systems. Each square in the plots represents the largest real part of the eigenvalues of ∂ f ∂ x when f e is sampled along relevant interaction force trajectories (clusters) for certain values of d z and d x .

Fig. 4 :

 4 Fig. 4: Left: CAD picture of tool tip: 1. Gimbal bearing, 2. Springs to stabilize gimbal, 3. End effector plate, 4. Micro switch, 5. Connection to quadrotor, 6. Springs to bring back the plate in rest position when not in contact; Right: Schematic side view of tool tip presenting the degrees of freedom of the tool tip.

Fig. 5 :

 5 Fig. 5: Snapshots of time series while quadrotor is performing the experiment.

Fig. 6 :Fig. 7 :

 67 Fig. 6: The main quantities recorded during the performed experiment.

  desired roll/pitch φ d , θ d , and e 1 .e 2 Operating the following change of variables:

3 2014 IEEE ICRA B. RPY Angles φ , θ , ψ,

  Preprint version, final version at http://ieeexplore.ieee.org/

				Pushing
	-0.52
	-0.54	dx=0.25	dx=0.27	dx=0.29	dx=0.31	dx=0.33		dx=0.35
	-0.56
	-0.58
	-0.6
	-0.62	-0.14 -0.11 -0.08 -0.05	0	0.05	0.08	0.11	0.14
			Lift/Dropping while Pushing-Low Pushing Force
	-0.52
	-0.54	dx=0.25	dx=0.27	dx=0.29	dx=0.31	dx=0.33		dx=0.35
	-0.56
	-0.58
	-0.6
	-0.62	-0.14 -0.11 -0.08 -0.05	0	0.05	0.08	0.11	0.14
			Lift/Dropping while Pushing-High Pushing Force
	-0.52
	-0.54	dx=0.25	dx=0.27	dx=0.29	dx=0.31	dx=0.33		dx=0.35
	-0.56
	-0.58
	-0.6
	-0.62	-0.14 -0.11 -0.08 -0.05	0	0.05	0.08	0.11	0.14
			Shifting Left/Right while Pushing-Low Pushing Force
	-0.52
	-0.54	dx=0.25	dx=0.27	dx=0.29	dx=0.31	dx=0.33		dx=0.35
	-0.56
	-0.58
	-0.6
	-0.62	-0.14 -0.11 -0.08 -0.05	0	0.05	0.08	0.11	0.14
			Shifting Left/Right while Pushing-High Pushing Force
	-0.52
	-0.54	dx=0.25	dx=0.27	dx=0.29	dx=0.31	dx=0.33		dx=0.35
	-0.56
	-0.58
	-0.6
	-0.62	-0.14 -0.11 -0.08 -0.05	0	0.05	0.08	0.11	0.14
				dz
								5	2014 IEEE ICRA

Preprint version, final version at http://ieeexplore.ieee.org/ 2

ACKNOWLEDGMENTS

The research leading to these results has received funding from the European Union Seventh Framework Programme FP7/2007-2013 under grant agreement n 601165 of the project WEARHAP WEARable HAPtics for humans and robots.


He ´le `ne Pillet
email: helene.pillet@ensam.eu

Morgan Sangeux

Je ´ro ˆme Hausselle

El Rami

Rachkidi

Wafa Skalli

A reference method for the evaluation of femoral head joint center location technique based on external markers

Introduction

An accurate method of joint center localization is essential for kinematic and kinetic calculations in movement analysis [START_REF] Stagni | Effects of hip joint centre mislocation on gait analysis results[END_REF]. To estimate the location of the hip joint center (HJC), previous studies have used predictive [START_REF] Bell | A comparison of the accuracy of several hip center location prediction methods[END_REF][START_REF] Davis | A gait analysis data collection and reduction technique[END_REF], functional [START_REF] Leardini | Validation of a functional method for the estimation of hip joint centre location[END_REF][START_REF] Ehrig | A survey of formal methods for determining functional joint axes[END_REF] or medical image based [START_REF] Otake | Fourdimensional model of the lower extremity after total hip arthroplasty[END_REF][START_REF] Kawakami | Gait analysis system for assessment of dynamic loading axis of the knee[END_REF][START_REF] Ohnishi | Threedimensional motion study of femur, tibia, and patella at the knee joint from biplane fluoroscopy and CT images[END_REF][START_REF] Sholukha | Double-step registration of in vivo stereophotogrammetry with both in vitro 6-DOFs electrogoniometry and CT medical imaging[END_REF] methods. Predictive and functional methods offer easy integration into existing clinical assessment but their accuracy needs to be established before implementation. Medical image based techniques may provide accurate 3D position of joint centers in the coordinate system of the imaging device. To be useful, these coordinates need to be registered to the pelvis coordinate system defined for movement analysis from external skin markers. This critical step raises two issues. First, when external markers were not available on the medical images, researchers have to assume that external markers were exactly positioned over bony landmarks that can be localized on the medical images. Any discrepancy between external markers positions and bony landmarks affects the accuracy of the HJCs coordinates in the pelvic coordinate system. Second, when external markers were available on the medical images, the subjects' posture in the imaging device may be different to the static calibration posture for movement analysis. For example, when the subjects are lying supine [START_REF] Otake | Fourdimensional model of the lower extremity after total hip arthroplasty[END_REF] major bias may be expected from soft tissue artifact between the lying supine and standing up postures.

In this context, a new system, EOS 1 (EOS Imaging -France), based on low-dose bi-planar X-rays taken in a standing position [START_REF] Dubousset | EOS: A new imaging system with low dose radiation in standing position for spine and bone & joint disorders[END_REF] together with external motion capture markers may serve as a reference. Images from this system are used to reconstruct 3D bones models which can be registered to external markers and in the appropriate subject's posture for static calibration. It is then possible to assess the relationship between the external markers based coordinate systems and the bones models. In clinical routine, the systematic use of X-rays, even low dose (6-9 times less than conventional X-rays), remains problematic. However, depending on the accuracy, it may be suitable to validate alternative methods of HJCs localization. The aim of the present study was to evaluate the accuracy of hip joint center (HJC) localization using the EOS 1 system.

Material and method

Subjects

Seventeen volunteers (11 males and 6 females) were considered after informed consent and ethical committee approval. Subjects had a mean weight of 75.8 kg (range 54.3-102 kg), mean height 1.74 m (range 1.52-1.92 m) and Body Mass Indexes (BMI) ranged from 17 to 33.

Keywords: Biplanar X-rays EOS Gait analysis Biomechanics Joint center A B S T R A C T
Accurate localization of joint centers is essential in movement analysis. However, joint centers cannot be directly palpated and alternative methods must be used. To assess the relative merits of these methods, a medical image based reference should be used. The EOS 1 system, a new low dose bi-planar X-rays imaging technique may be considered. The aim of this study was to evaluate the accuracy of hip joint center (HJC) localization using the EOS 1 system. Seventeen healthy young adults participated in the study. Femoral heads and pelvic external markers were localized using the EOS 1 system and the HJCs were expressed in the movement analysis coordinate system. Results showed that external marker localization was reliable within 0.15 mm for trained assessors. Mean accuracy for HJC localization was 2.9 mm (SD: 1.3, max: 6.2). The EOS based method therefore appeared reliable and may be used for femoral head localization or as a reference to assess the accuracy of other methods for HJC localization.

Protocol

Volunteers were equipped with four 14 mm reflective markers placed on the anterior and posterior iliac spine landmarks of the pelvis. For each subject, two acquisitions were taken with the EOS 1 (Fig. 1) in a standard standing posture described by Chaibi et al. [START_REF] Chaibi | Fast 3D reconstruction of the lower limb using a parametric model and statistical inferences and clinical measurements calculation from biplanar X-rays[END_REF]. Each acquisition consisted of two simultaneous X-ray images in the sagittal and coronal plane. Acquisitions were performed 5 min apart to allow the system to cool down. Three-dimensional reconstruction of the femur was performed as described by Chaibi et al. [START_REF] Chaibi | Fast 3D reconstruction of the lower limb using a parametric model and statistical inferences and clinical measurements calculation from biplanar X-rays[END_REF]. A sphere was fitted in the least square sense to the femoral head region of the model to localize the femoral head center in the EOS 1 coordinate system. Three dimensional positions of the pelvic markers were also computed: the projected images of a 14 mm diameter sphere model were manually adjusted on the two sets of bi-planar X-rays to match reflective markers contours.

Repeatability of markers identification

Data from 4 of the subjects were used (1 male and 1 female with a BMI > 25, 1 male and 1 female with a BMI < 25). Markers from the first pair of images were considered. Two trained operators (bioengineers) independently placed three times the entire set of markers. The method described by Glu ¨er et al. [START_REF] Glu ¨er | Accurate assessment of precision errors: how to measure the reproducibility of bone densitometry techniques[END_REF] was used to calculate the inter-operator repeatability of marker localization on the images.

Accuracy of femoral head registration (Fig. 2)

The right and left femoral heads were localized for each acquisition (FH1 EOS1 and FH2 EOS2 ). We calculated the transformation to map the pelvic markers from the first acquisition to the second acquisition in the least square sense [START_REF] So ¨derkvist | Determining the movements of the skeleton using well-configured markers[END_REF]. The transformation was then applied to the position of the femoral head from the first acquisition to map it onto the second acquisition which gives FH1 EOS2 . Accuracy of the femoral head registration was calculated as D FH , the distance between FH1 EOS2 and FH2 EOS2 . It should be noted that D FH includes accuracy of femoral head localization and repeatability of pelvis markers localization.

Results and discussion

Inter-operator variability in markers localization was 0.15 mm. Uncertainty was mainly related to visibility of some of the markers due to bone or marker superposition, and sometimes to the lack of sphericity of the markers on the images. The distance D FH was homogeneous for left and right HJC and 2.9 mm in average (SD: 1.3). There was only one outlier (6.2 mm, Table 1, S8 left) over 34 measurements.

The EOS method appears more reliable than other published methods. In a recent study describing a 3D ultrasound method [START_REF] Peters | Validation of 3-D freehand ultrasound for the determination of the hip joint centre[END_REF], authors reported an accuracy of 4 mm for the distance between the left and right femoral heads with respect to magnetic resonance imaging (MRI). However, since MRI images were taken lying supine and external motion capture markers cannot be seen on the images, the authors were unable to assess the accuracy of the femoral head localization with respect to the pelvic coordinate system. Leardini et al. used a similar technique based X-ray (called RSA) to compare the position of the HJC output by predictive and functional methods [START_REF] Leardini | Validation of a functional method for the estimation of hip joint centre location[END_REF]. The authors reported a comparable accuracy for the marker localization (0.2 mm) from the RSA technique but did not report the accuracy of the femoral head localization. However, the technique described in Leardini et al. study involves levels of ionizing radiation that are too high to be used in extensive data collection and necessitated the use of tantalum balls instead of external markers. It is only since the EOS 1 system, with low dose radiation, that such stereoradiography techniques may be envisaged for extensive studies.

Typical accuracy of non-image based methods to locate the hip joint center range from 13 mm to 30 mm [START_REF] Sangeux | Hip joint centre localization: evaluation on normal subjects in the context of gait analysis[END_REF]. The current technique, with an average and directly comparable accuracy of 2.9 mm in average seems appropriate to serve as a reference. A great advantage of this approach, i.e. simultaneous visualization of femoral head and external markers, is that it is not affected by subject specificities such as BMI or unusual anatomy. It therefore constitutes a strong base either for direct measurement of the hip joint center or for evaluation of alternative methods of estimation.

Fig. 1 .

 1 Fig. 1. Two pairs of radiographies for one subject (the first pair centered on the right lower limb and the second pair centered on the left lower limb). Pelvis markers are visible on both X-rays.

Fig. 2 .

 2 Fig. 2. Schematic of the process to obtain accuracy data.

Table 1

 1 Accuracy of pelvis markers based registration of the femoral head for 17 subjects: distance (mm) between the femoral head registered from pelvis markers and the femoral head computed from the femur model for right and left sides.

	Subjects	Distance for the	Distance for the left
		right femoral head (mm)	femoral head (mm)
	1	2.0	3.2
	2	3.7	3.6
	3	3.2	1.8
	4	3.3	2.7
	5	0.7	1.5
	6	2.8	1.7
	7	2.1	1.5
	8	3.0	6.2
	9	1.0	1.8
	10	4.5	4.4
	11	0.9	2.5
	12	3.8	2.8
	13	4.7	4.5
	14	4.2	2.1
	15	4.1	3.7
	16	1.7	3.0
	17	3.0	3.6
	Mean	2.9	3.0
	Max	4.7	6.2

Acknowledgments

The authors are grateful to VICON (OMG-UK) for the loan of a motion capture system necessary for the overall study.

Conflict of interest statement

The authors do not have any financial or personal relationships with other people or organizations that could inappropriately influence the manuscript.


Minh Tien Phan
email: minh.phan@hds.utc.fr

Indira Thouvenin
email: indira.thouvenin@utc.fr

Vincent Fremont
email: vincent.fremont@utc.fr

Véronique Cherfaoui
email: veronique.cherfaoui@utc.fr

Estimating Driver Unawareness of Pedestrian Based On Visual Behaviors and Driving Behaviors

Every year, more than 1.24 million people die in traffic accidents around the world, half of which are among pedestrians and cyclists, and 22 percent of which are children (WHO, 2013). Thus, preventing pedestrians from motor accidents is an important problem in achieving a safe and secure society. A lot of research in the advanced driver assistance is able to detect the pedestrian with the in-vehicle sensors and inform the driver of their presence. However, most of these alert systems do not adapt to the driver, they become distracting, annoying the driver and will be often ignored or deactivated. Therefore, taking into account the driver's state and understanding fully the situation are the challenges for these systems.

This study is the first step of a project named MINARDA (Mobile and Interactive Augmented Reality for Driving Assistance). This system is strongly connected to the user. It aims to use a suitable visualization metaphor in augmented reality that allows the driver to avoid dangerous situations. Particularly, we focus on the situation in which the pedestrian is in danger. This paper will be structured as follows. In sections 2 and 3, we underline our objectives of research and define the problem. In section 4, we review some related works. In Section 5, we describe the proposed methodology to treat this problem. Section 6 describes our expected outcome, conclusion and perspectives.

OUTLINE OF OBJECTIVES

As mentioned above, the MINARDA system needs to fully understand the situation to be able to provide the appropriate cues to the user. In the context of pedestrian safety, our first objective is to find out what the features that characterize the driver's awareness or unawareness of pedestrian are. Once we can identify these features, our second objective is to build a model of the awareness and unawareness a driver has of pedestrians appearing on the road ahead. At last, we will envisage estimating the different levels of danger to pedestrian. The more precisely the danger levels are classified, the more suitably the assistance cues can adapt to the driver.

RESEARCH PROBLEM

In a previous work, [START_REF] Engel | Predicting human perception performance with machine vision[END_REF], 2012), Wakayama, (2012) proposed to estimate how easy it is for the driver to see and notice pedestrians. The studies included the estimate of driver's visual attention and pedestrian detectability in an image. In this study, we aim to provide a new and more direct approach. It can be assumed that the driver's awareness or unawareness of pedestrian could be estimated by observing his visual behaviors and driving behaviors. More precisely, our hypothesis is whenever the driver is aware of a pedestrian ahead, he has to direct his gaze to pedestrian and does some reactions on vehicle, (brake, deceleration, wheel angle change, etc.) what is not in the case of unawareness.

This preliminary suggestion leads to two related questions:

1. Is there any difference in visual behaviors and driving behaviors between two states driver's awareness of and unawareness of something on road (pedestrian in this case)?

2. Is it possible to extract and use those patterns to help identify or learn information about the danger to pedestrian?

In order to answer these questions, we have to envisage different experiments and consider different methods to measure visual behaviors and driving behaviors. Because of their reliability and flexibility, in this work, we use a vision-based method to observe the visual behaviors and various in-vehicle sensors to measure the driving behaviors.

STATE OF THE ART

In the state of the art includes the related research that helps to understand the danger to pedestrian, particularly those are related to the driver. Thus, we talk about the problem of pedestrian detectability, we review in detail the studies of driver inattention detection, and the studies which are directly related to the driver's awareness of the pedestrian. We will then identify the strength and weakness of each method that should be considered in our works.

Pedestrian Detectability

The studies on the detectability of the pedestrian aim to estimate how easy it is for the driver to see and notice pedestrians. In the work of [START_REF] Engel | Predicting human perception performance with machine vision[END_REF], the authors did the experiments to measure the time to recognizing the pedestrian on image. Then, they determined the features of pedestrians that correlated with this time. Thus, the detectability of pedestrian was modeled. The features of pedestrian on image could be for example, the position of pedestrian, size, color, light, etc. This study was also extended to the dynamic scene concept. Actually, instead of analyzing the pedestrian's detectability on images, it is possible to do the experiments on the sequence video [START_REF] Engel | Detectability prediction in dynamic scenes for enhanced environment perception[END_REF]. Another extension was proposed by Wakayama (2012) who considered the motion of pedestrian to be one of the features that characterize the pedestrian detectability. However, as before, because the fact that the pedestrians can be seen easily does not mean that the driver is cognitively aware of them [START_REF] Fukagawa | Estimating Driver Awareness of Pedestrians from Driving Behavior based on a Probabilistic Model[END_REF].

Driver Inattention Detection

The lack of attention by drivers is the biggest single cause of serious road accidents. A great amount of work have been done on the inattention state of the driver. However, the inattention of the driver was defined in different ways. In this part (4.2), the inattention means the abnormal states of the driver such as fatigue, drowsiness or distraction, etc. Otherwise, in the rest of our research, driver's unawareness of pedestrian is considered to be a kind of inattention based on this definition: "Driver Inattention means insufficient or no attention to activities critical for safe driving" (Regan et al, 2011). According to this, unawareness of the pedestrian means insufficient or no attention to this pedestrian for safe driving. The driver's behaviors and his inattention state are analyzed and detected under different techniques, the majority of which are as follows.

Physiological Measures Observation

The physiological signals such as heart rate, respiration or EEG and are considered to be the most reliable signals to analyze the driver behaviors and to detect inattentiveness.

Heart rate is easily determined through Electrocardiogram (ECG) signal. The mental stress while driving for example, increases blood pressure, heart rate and activate the sympathetic nervous system. (Moriguchi et al, 1992). There are significance impacts in the heart rate when a driver is in a state of cognitive inattention (thinking while driving for example) [START_REF] Akin | Estimating vigilance level by using EEG and EMG signals[END_REF].

Moreover, the EEG (Electroencephalogram) signal has various frequency bands which can signify different states of the driver. For example, there is evidence that the increase of the alpha band (8-13Hz) corresponds to relaxation. Increase of the delta band (0.5-4Hz) and the theta band (4-8Hz) correspond to sleep activity and drowsiness which signify a potential failure of attention and behaviors. (Östlund, 2004), (Santana, 2002). In another experiment, twelve types of energy parameters computed from three bands alpha, beta, and delta were chosen as the evaluation index of driver fatigue (Li, 2012) However, these techniques are obtrusive, since it is necessary to attach some electrodes on the drivers, causing annoyance to them. Consequently, it is difficult to be implemented in the real-life applications.

Visual Behaviors Observation

Driver behaviors could be somehow observed from the changes in their facial features like eyes, head and face. In this context, several methods have been proposed. This measurement has the advantage of being unobtrusive, since they could be collected with remote eye-head trackers. Moreover, they are quite reliable.

Measurements related to head movement such as nodding frequency and those related to the eye region such as eyelid distance changes, eye close duration, blinking frequency, and the recently developed parameter PERCLOS (percentage of time in a minute that the eye is 80% closed) (Lin,2012) were widely used in research. After extracting these features, [START_REF] Bergasa | Visual Monitoring of Driver Inattention[END_REF] and Mohamad-Hoseyn Sigari (2013) built a fuzzy model or finite state machine to estimate the inattentive, distracted or fatigue state of driver.

Another cue is the size of the pupils. The pupils is the part of the iris that allows light to enter the retina. Besides light, the pupil dilates when mental or cognitive effort is given. It was observed that the pupil was dilated and the diameter of average pupil size increased by 15% when the driver was cognitively distracted [START_REF] Akin | Estimating vigilance level by using EEG and EMG signals[END_REF], [START_REF] Benedetto | Driver workload and eye blink duration[END_REF].

Gaze behaviors are another cues that can be used as a metric to find if the concentration of the driver was on driving or not. It was noticed that, when the driver was cognitively distracted, glancing at instruments and mirrors decreased significantly [START_REF] Harbluk | An on-road assessment of cognitive distraction: Impacts on driver's visual behavior and braking performance[END_REF]. A simple method to detect the driver's attention was proposed by [START_REF] Fletcher | Driver Inattention Detection based on Eye Gaze-Road Event Correlation[END_REF]. They used driver's gaze vector and road scene event correlation to estimate if the driver has seen the event or not. Due to the "looking but not seeing" problem, this work was not able to reliably determine if a certain road event (pedestrian for example) is perceived by the driver. However it could identify if the driver has not perceived a road event.

Deeper in this context, [START_REF] Doshi | Head and gaze dynamics in visual attention and context learning[END_REF] provided a result on the observation of the dynamic of overt visual attentions shifts. They found that there are various interactions between head and eye movements that are useful in detecting the driver distractions, as well as the driver intent. Indeed, their results validated the differences existing between goal-oriented and stimulus-oriented gaze shift. Moreover, this feature could be observed in dynamics of eye and head movements. They also investigated the problem of detecting the intent of the driver in changing lane which was considered to be a goal-oriented attention shift. [START_REF] Doshi | Head and eye gaze dynamics during visual attention shifts in complex environments[END_REF]. This result showed that the driver attentiveness or awareness of something should be observed through the dynamic of many parameters.

Driving Behaviors Observation

Another method on the driver's inattention research is to detect it indirectly through driving behaviors such as vehicle speed, steering wheel movement, lateral position, and break or acceleration pedal states. (Imamura et al, 2008), (Ueno et al, 1994).

The projects conducted by HASTE (Human Machine and the Safety of Traffic in Europe) demonstrated that the steering measures were significantly affected by the visual task, when a subject had to perform the visual task, the steering effort was higher than in the baseline condition. (Östlund et al., 2004).

The IVDRs (In-vehicle Data Recorder) offer valuable information on a driver's behavior through the analysis of automobile-operating information. Jensen and Wagner (2011) also proposed a combination of three analysis methods to evaluate the driver performance: data threshold violations, phase plane analysis with limits and a recurrence plot with outlier limits. These methods were based on the measures of vehicle speed, engine speed, vehicle latitude and longitude coordinates, and lateral accelerations.

Although these techniques are not obtrusive, they are subject to several limitations such as vehicle type, driver experience, geometric characteristics, state of road, etc. [START_REF] Bergasa | Visual Monitoring of Driver Inattention[END_REF].

Driver Unawareness of Pedestrian

To the best of our knowledge, there are no studies that are directly related to the estimation of the driver unawareness of pedestrian or other traffic events. The most closely related works are those of [START_REF] Fukagawa | Estimating Driver Awareness of Pedestrians from Driving Behavior based on a Probabilistic Model[END_REF] who proposed a hypothesis that is likely linked to our work. Their study was based on the driver's operational data such as pressure on the accelerator pedal, pressure on the brake pedal (called acceleration reaction), steering angle and vehicle behavior data such as vehicle speed to estimate the driver's awareness of pedestrian.

Their hypothesis was that whenever a pedestrian appears on the road, if the driver noticed it, he had to do the acceleration reaction somehow. They used the driving behavior data collected by Research Institute of Human Engineering for Quality of Life [START_REF] Akamatsu | Modeling of Driving Behavior When Approaching an Intersection Based on Measured Behavioral Data on An Actual Road[END_REF]. From that, they proposed to calculate the probability of acceleration reaction being observed at a distance given that response to driver's aware of the pedestrian. This probability was assumed to be a log-normal distribution. They also proposed to calculate the probability of acceleration reaction being observed at a distance given that not in response to driver's aware of the pedestrian and this probability was considered to be a uniform distribution. Hence, using Bayes theorem, they calculated the probability that one acceleration reaction was caused in response to driver's awareness of pedestrian.

However, there is a couple of weaknesses in this study. Firstly, in the data collected in real and actual road condition, it was supposed that whenever a pedestrian appeared on road, the driver had to notice them, this is hard to verify. Secondly, the driver could totally accelerate if he had been aware of the pedestrians and identify that they were not in danger. On the other hand, this probability model is not reliable because of the use of a specific distribution law. Finally, this study can't determine if the driver has not noticed the pedestrian or has been unaware of them.

METHODOLOGY

In this work, we have defined the driver's unawareness and awareness of pedestrian. We use the camera-based system to observe his visual behavior and the in-vehicle sensors to measure the driving behaviors. Moreover, in the experiment, we have discussed two most important issues as follow. Hence, the methodology of research is determined

Driving Simulation vs. Naturalistic Driving Data Collection

In our work, both driving simulations and real conditions driving are expected out. Firstly, a driving simulation is used to collect all information from different sensors. From that, the feature vectors representing the driver's awareness and unawareness of pedestrian will be extracted. The simulation environment is useful because we can control the situation and can propose different scenario that fit well to our research problem. For example, we can propose two scenarios. First, a scenario where the driver is asked to recognize the gender of a pedestrian while driving. On the other hand, a scenario of driver's unawareness in which the driver is demanded to do a second visual task while driving.

Naturalistic driving data help to verify the hypothesis in real conditions. Many experiments have been conducted in simulated environment and the results have been discussed. However, in real driving conditions the result might be drastically lower as a moving vehicle presents new challenges like variable lighting, changing background and vibration, etc. [START_REF] Arun | Driver inattention detection methods: A review[END_REF]. The data collected in real conditions will be extracted and annotated whenever a pedestrian appears on road. Then, we will compare the driver visual reaction and acceleration reaction with the ones that have been extracted in simulation driving.

We use the help of MINARDA system embedded on CARMEN platform to collect the information.

Figure 1: MINARDA system and CARMEN car-Ready for simulation driving as well as naturalistic driving data collection

Hybrid Measures

Each previously mentioned method get their own advantages and limitations. Nevertheless, in the advanced driving assistance systems, the environment, the vehicle and the driver have to be considered to be an overall driving system (Trivedi et al. 2007). Proposed study is to consolidate various measures which could lead to a good detection system. Indeed, beside the pedestrian detection system that provides all properties of pedestrian on road (position, distance, time to collision, etc.), we propose to use the visual behaviors, specially the looking vector which is provided by SMI Glass and head motion which is provided by the Facelab System. In the other hand, the driving behaviors such as the driver acceleration reaction are measured by the IVDR. These cues will be collected in high frequency to analyze the driver's awareness or unawareness of pedestrian.

Our hypothesis is that, this kind of driver inattention should be observed through the change of the looking reaction with respect to a pedestrian or the change in his driving acceleration. Or it may be characterized by the interaction of these both reactions, for example, by a reaction of looking first and followed by an acceleration reaction. This hypothesis may be complete the study of Yuuki Fukugawa (2013) since we are able to observe the case when the driver is aware of pedestrian and does no reaction acceleration and the case when the driver does the acceleration reaction but he is unawareness of a pedestrian.

EXPECTED OUTCOME

At the end of this study, the reliable patterns that characterize the driver's states of unawareness and awareness of pedestrian are expected to be extracted out. Then, we intend to develop a new learning algorithm based on these patterns in order to model this driver's behavior and the danger level to pedestrian. The whole algorithm will be implemented on the MINARDA system.

In this paper, we talked about the danger to pedestrian. We provided a review on the methods that help to analyze the driver behaviors and to detect his inattention state. We propose a novel approach directly related to pedestrian safety, the driver's awareness and unawareness of pedestrian. This study could be extended to other type of road events such as traffic light or other vehicles detection and help us to understand better the driving context.

Prevention, 39, 372 -379. Imamura, T.; Yamashita, H.; bin Othman, R.; Zhang, Z. and Miyake, T. 2008. "Driving behavior classification river sensing based on vehicle steering wheel operations". SICE Annual Conference, 2008, 2714-2718. Jensen, M. and Wagner, J. 2011. "Analysis of In-Vehicle Driver Behaviour Data for Improved Safety". International Journal of Vehicle Safety. 197-212. Li, W.; chang He, Q.; min Fan, X. and min Fei, Z.2012. "Evaluation of driver fatigue on two channels of EEG data". Neuroscience Letters, 506, 235 -239. Lin, S. T.; Tan, Y. Y.; Chua, P. Y.; Tey, L. K. & Ang, C. H. 2012 "PERCLOS Threshold for Drowsiness Detection during Real Driving" Journal of Vision, 12, 546. Mohamad-Hoseyn Sigari, Mahmood Fathy and Soryani, M. 2013. "A Driver Face Monitoring System for Fatigue and Distraction Detection". International Journal of Vehicular Technology, 11. Östlund, J; and al. 2004. "Delivrable 2-HMI and safety related driver performance". Human Machine Interface and the Safety of Traffic in Europe (HASTE) project. Regan, M. A.; Hallett, C. and Gordon, C. P. 2011. "Driver distraction and driver inattention: Definition, relationship and taxonomy". Accident Analysis & Prevention, 43, 1771 -1781. Santana Diaz, A.; Jammes, B.; Esteve, D. and Gonzalez Mendoza, M. 2002. "Driver hypovigilance diagnosis using wavelets and statistical analysis". 2002. Intelligent Transportation Systems. Proceedings. The IEEE 5th International Conference on, 162-167 Trivedi, M. M.; G, T. and Mccall, J. 2007. "Looking-in and looking-out of a vehicle: Computer-vision-based enhanced vehicle safety". IEEE Trans. Intell. Transp. Syst, 8, 108-120. Ueno, H.; Kaneda, M. and Tsukino, M. 1994. "Development of drowsiness detection system". Vehicle Navigation and Information Systems Conference, 1994. Proceedings. 1994, 15-20. Wakayama, M.; Deguchi, D.; Doman, K.; Ide, I.; Murase, H. and Tamatsu, Y.2012. "Estimation of the human performance for pedestrian detectability based on visual search and motion features".Pattern Recognition (ICPR), 2012 21st International

 Conference on, 1940-1943. WHO, World Health Organization, 2013. "Global StatusReport on Road Safety Supporting a decade of action".