Brahim Boufoussi

Marco Dozzi

Renaud Marty

Local time and Tanaka formula for a Volterra-type multifractional Gaussian process

The stochastic calculus for Gaussian processes is applied to obtain a Tanaka formula for a Volterra-type multifractional Gaussian process. The existence and the regularity properties of the local time of this process is obtained by means of Berman's Fourier analytic approach.

Introduction

Several types of multifractional Gaussian processes have been studied, including the processes introduced by Lévy-Véhel and Peltier [START_REF] Véhel | Multifractional Brownian motion: definition and preliminary results[END_REF] and by Benassi, Jaffard and Roux [START_REF] Benassi | Elliptic Gaussian random processes[END_REF], known as the moving average and the harmonisable versions of multifractional Brownian motion, and the processes introduced by Benassi, Cohen and Istas [START_REF] Benassi | Identifying the multifractional function of a Gaussian process[END_REF] and, based on the calculus of (multi-)fractional differentiation and integration, by Surgailis [17]. These processes are usually defined by replacing, in certain representations of fractional Brownian motion (fbm), the Hurst parameter H by a Hurst function h, i.e. a real function of the time parameter with values in (0,1). Generalisations to stable processes have been studied too [START_REF] Lacaux | Real harmonizable multifractional Lévy motions[END_REF][START_REF] Stoev | Path properties of the linear multifractional stable motion[END_REF][START_REF] Stoev | Stochastic properties of the linear multifractional stable motion[END_REF]. The stochastic properties and the regularity of the trajectories of these processes can be characterised in the same terms as for fbm, in particular by the property of local asymptotic self-similarity or by the pointwise Hölder exponent, which is constant and equal to H for fbm, but varies in time following h in the multifractional case.

The aim of this article is to study a Volterra type multifractional Gaussian process B h which fits into the framework of stochastic calculus, and its local time. In fact, the stochastic calculus, including the Itô formula and stochastic differential equations, is now well established for fbm, where stochastic integrals have been defined in the Malliavin sense or by means of Wick products. The stochastic calculus for multifractional Gaussian processes has not yet been developed explicitely, but important elements, including a divergence integral and an Itô formula, have been proven in [START_REF] Alòs | Stochastic calculus with respect to Gaussian processes[END_REF] for a class of Gaussian processes admitting a kernel representation with respect to Brownian motion. The process B h we study in this article is defined by allowing for a Hurst function in the kernel representation of fbm, and it belongs to the class of Gaussian processes studied in [START_REF] Alòs | Stochastic calculus with respect to Gaussian processes[END_REF]. The Itô formula in [START_REF] Alòs | Stochastic calculus with respect to Gaussian processes[END_REF] is applied to get a representation of the local time for which, as it is usually the case in Tanaka-like formulas, the occupation measure is not the Lebesgue measure, but the quadratic variation process. However, contrary to fbm, the interpretation in our case is more delicate because the quadratic variation process is not necessarily increasing (but of bounded variation). We compare this local time to the local time with respect to the Lebesgue measure, which we obtain, together with the regularity properties of its trajectories, by the Fourier analytic approach initiated by S. Berman [START_REF] Berman | Local times and sample function properties of stationary Gaussian process[END_REF] and which is based on the notion of local nondeterminism (LND). Sufficient conditions for the LND property to hold are given in [START_REF] Boufoussi | Path properties of a class of locally asymptotically self similar processes[END_REF], and we show that these conditions are implied by the stochastic properties, in particular the lass property, of B h .

It is well known that the cases where the Hurst parameter or the Hurst function is < 1/2 resp. > 1/2 have to be treated separately (see [START_REF] Alòs | Stochastic calculus with respect to Gaussian processes[END_REF] and [START_REF] Nualart | The Malliavin calculus and related topics[END_REF]). We are interested in the case where the Hurst function h takes values in (1/2,1). In fact, the Volterra-type process we study here is defined only for this case, which is appropriate for long memory applications [START_REF] Samorodnitsky | Stable non-Gaussian random processes[END_REF]. The article is organised as follows : In Section 2 stochastic properties and the regularity of the trajectories of B h are proven. The form of the covariance function (Proposition 2) shows in particular that B h differs from the harmonisable and moving average multifractional Brownian motions. The continuity of the trajectories of B h is obtained by classical criteria from estimates for the second order moments of the increments (Proposition 3). The lass property is proven in Proposition 5; it implies that the pointwise Hölder exponent of B h is equal to h (Proposition 6). In Section 3 B h is shown to satisfy the property named (K4) in [START_REF] Alòs | Stochastic calculus with respect to Gaussian processes[END_REF] and that the quadratic variation of B h is of bounded variation. Therefore the divergence integral and the Itô formula developed in [START_REF] Alòs | Stochastic calculus with respect to Gaussian processes[END_REF] (recalled briefly in this section) hold for B h . Section 4 is devoted to the local time of B h . Its existence and square integrability with respect to the space variable follows from a classical criterion due to S. Berman ([3]) (Proposition 11). Then a Tanaka-type formula with the divergence integral of Section 3 is given (Theorem 12), where the occupation measure of the local time is the quadratic variation of B h . Finally the local time with respect to the Lebesgue measure and its regularity in the space and time variables are given in Theorem 16. The proofs are based on the LND property, which is shown to hold for B h in Proposition 14. The sufficient conditions, named A and A m are in fact closely related to the lass property, which is the main ingredient for the proof of Proposition 14. Proposition 15 shows that the condition A holds in fact for a much larger class of Gaussian processes than B h .

A Volterra-type multifractional Gaussian process

It is well-known that the fractional Brownian motion B H with (fixed) Hurst parameter H ∈ (1/2, 1) can be represented for any t ≥ 0 as

B H (t) = t 0 K H (t, u)W (du),
where

K H (t, u) = c H u 1/2-H t u (y -u) H-3/2 y H-1/2 dy, with c H = πH(2H -1) Γ(2 -2H)Γ(H + 1/2) 2 sin(π(H -1/2)) 1/2 (H -1/2)
and W (dy) is a Gaussian measure.

Let a and b be two real numbers satisfying 1/2 < a < b < 1. Throughout the paper we consider a function h : R → [a, b]. We assume that this function is β-Hölder with sup h < β. Define the centered Gaussian process

B h = {B h (t), t ≥ 0} by B h (t) = B h(t) (t) = t 0 K h(t) (t, u)W (du),
where

K h(t) (t, u) = u 1/2-h(t) t u (y -u) h(t)-3/2 y h(t)-1/2 dy. (1)
If h(•) = H is a constant, B H is a fractional Brownian motion up to the multiplicative constant c H . Before establishing properties of B h we give a lemma regarding an estimate on K H that we use throughout the paper.

Lemma 1 For every T > 0 there exists a function

Φ T ∈ L 2 ((0, T ], R + ) such that for every s ∈ (0, T ] sup λ∈[a,b],t∈(0,T ] ∂ ∂λ K λ (t, s) ≤ Φ T (s).
Proof. We have

∂ ∂λ K λ (t, s) = (-log s)s 1/2-λ t s (y -s) λ-3/2 y λ-1/2 dy +s 1/2-λ t s (y -s) λ-3/2 y λ-1/2 (log(y -s) + log y)dy.
Then for every T > 0 there exists a constant C a,b,T such that for every λ ∈ [a, b] and s ∈ (0, T ]

∂ ∂λ K λ (t, s) ≤ C a,b (1 ∨ | log s|)s 1/2-b =: Φ T (s)
that concludes the proof.

The following proposition gives the covariance of this process.

Proposition 2 Let X = {X(t, λ), t ≥ 0, λ ∈ (1/2, 1)} be the two-parameter process given by X(t, λ)

= t 0 K λ (t, u)W (du). Then E[X(t, λ)X(s, λ ′ )] = t 0 dy s 0 dz β(y, z, λ, λ ′ )|y -z| λ+λ ′ -2 y z λ-λ ′ where β(y, z, λ, λ ′ ) = β(2 -λ -λ ′ , λ ′ -1/2)1 {y>z} + β(2 -λ -λ ′ , λ -1/2)1 {y<z} and β(a, b) (a, b > 0) is the Beta function. In particular, E[X(t, λ) 2 ] = t 0 dy t 0 dz β(2 -2λ, λ -1/2)|y -z| 2λ-2 = β(2 -2λ, λ -1/2) λ(2λ -1) t 2λ .
Proof. We have

E[X(t, λ)X(s, λ ′ )] = t∧s 0 K λ (t, u)K λ ′ (s, u)du = t∧s 0 u 1-λ-λ ′ t u (y -u) λ-3/2 y λ-1/2 dy × s u (z -u) λ ′ -3/2 z λ ′ -1/2 dz du = t 0 dy s 0 dz y λ-1/2 z λ ′ -1/2 × y∧z 0 u 1-λ-λ ′ (y -u) λ-3/2 (z -u) λ ′ -3/2 du.
We fix y > z and calculate the following integral by making the successive substitutions u = vz, w = (y -vz)/(1 -v), t = w/y and s = 1/t:

z 0 u 1-λ-λ ′ (y -u) λ-3/2 (z -u) λ ′ -3/2 du = z 1/2-λ 1 0 λ ′ (y -vz) λ-3/2 (1 -v) λ ′ -3/2 dv = z 1/2-λ (y -z) λ+λ ′ -2 +∞ y (w -y) 1-λ-λ ′ w λ-3/2 (y -z) λ+λ ′ -2 dw = z 1/2-λ y 1/2-λ ′ (y -z) λ+λ ′ -2 +∞ 1 (t -1) 1-λ-λ ′ t λ-3/2 dt = z 1/2-λ y 1/2-λ ′ (y -z) λ+λ ′ -2 β(2 -λ -λ ′ , λ ′ -1/2).
In the same way we get for y < z :

y 0 u 1-λ-λ ′ (y -u) λ-3/2 (z -u) λ ′ -3/2 du = z 1/2-λ y 1 2 -λ ′ (z -y) λ+λ ′ -2 β(2 -λ -λ ′ , λ -1/2).
This concludes the proof.

In the sequel we need the estimates we establish in the following proposition.

Proposition 3

The process X satisfies the following estimates. a) For all s and t ≥ 0

E[(X(t, λ) -X(s, λ)) 2 ] = c -2 λ |t -s| 2λ (2)
b) For every T > 0, there exists a constant C T > 0 such that for every t ∈ [0, T ] and every λ and

λ ′ ∈ [a, b] E[(X(t, λ) -X(t, λ ′ )) 2 ] ≤ C T |λ -λ ′ | 2 (3)
Proof.

Proof of a). For every λ ∈ [a, b] the process X(•, λ) : t → X(t, λ) is a fractional Brownian motion with variance c -2 λ so we deduce (2). Proof of b). We have

E[(X(t, λ) -X(t, λ ′ )) 2 ] = t 0 (K λ (t, u) -K λ ′ (t, u)) 2 du. There exists ξ = ξ(λ, λ ′ ) ∈ [min{λ, λ ′ }, max{λ, λ ′ }] such that K λ (t, u) -K λ ′ (t, u) = (λ -λ ′ ) ∂ ∂λ K λ (t, u) λ=ξ=ξ(λ,λ ′ )
.

Then, thanks to Lemma 1 we get for every t, λ and λ

′ E[(X(t, λ) -X(t, λ ′ )) 2 ] ≤ |λ -λ ′ | 2 T 0 (Φ T (u)) 2 du
that ends the proof.

From the last proposition we can deduce the continuity of B h .

Corollary 4

The process B h defined above has continuous trajectories.

Proof. We deduce from Proposition 3 that for every s and

t in a compact interval [0, T ] such that |t -s| < 1 E[(B h (t) -B h (s)) 2 ] = E[(X(t, h(t)) -X(s, h(s))) 2 ] ≤ 2E[(X(t, h(t)) -X(t, h(s))) 2 ] +2E[(X(t, h(s)) -X(s, h(s))) 2 ] ≤ 2C T |h(t) -h(s)| 2 + 2c -2 h(s) |t -s| 2h(s) ≤ 2C T |t -s| 2β + 2 sup λ∈[a,b] (c -2 λ )|t -s| 2a . Then E[(B h (t) -B h (s)) 2 ]/|t -s| 2a
is bounded and since B h is Gaussian we can deduce from [START_REF] Billingsley | Convergence of Probability Measures[END_REF] its continuity. Now we deal with the local self-similarity property of B h .

Proposition 5

The process B h is locally self-similar. More precisely, for every t, we have the following convergence in distribution:

lim ε→0 B h (t + εu) -B h (t) ε h(t) u≥0 = c -1 h(t) B h(t) (u) u≥0
where lim ε→0 stands for the limit in distribution in the space of continuous functions endowed with the uniform norm on every compact set.

Proof. Let us start by proving the convergence of the finite dimensional distribution. Because B h is Gaussian, it suffices to show the convergence of the second-order moments.

We then can write for every u and v:

1 ε 2h(t) E [(B h (t + εu) -B h (t)) (B h (t + εv) -B h (t))] = 1 ε 2h(t) (I 1 (ε) + I 2 (ε) + I 3 (ε) + I 4 (ε))
where

I 1 (ε) = E[(X(t + εu, h(t)) -X(t, h(t))) × (X(t + εv, h(t)) -X(t, h(t)))], I 2 (ε) = E[(X(t + εu, h(t + εu)) -X(t + εu, h(t))) × (X(t + εv, h(t)) -X(t, h(t)))], I 3 (ε) = E[(X(t + εu, h(t)) -X(t, h(t))) × (X(t + εv, h(t + εv)) -X(t + εv, h(t)))], I 4 (ε) = E[(X(t + εu, h(t + εu)) -X(t + εu, h(t))) × (X(t + εv, h(t + εv)) -X(t + εv, h(t)))].
Thanks to the selfsimilarity of the fractional Brownian motion and by stationarity of its increments we get

lim ε→0 1 ε 2h(t) I 1 (ε) = 1 2c 2 h(t) (|u| 2h(t) + |v| 2h(t) -|u -v| 2h(t) ). (4)
Then, because of Cauchy-Schwartz inequality, it is enough to prove

lim ε→0 1 ε 2h(t) E (X(t + εu, h(t + εu)) -X(t + εu, h(t))) 2 = 0 (5) to get lim ε→0 1 ε 2h(t) (I 2 (ε) + I 3 (ε) + I 4 (ε)) = 0. (6)
So, using Lemma 1 we get

1 ε 2h(t) E (X(t + εu, h(t + εu)) -X(t + εu, h(t))) 2 = 1 ε 2h(t) t+εu 0 K h(t+εu) (t + εu, s) -K h(t) (t + εu, s) 2 ds ≤ (h(t + εu) -h(t)) 2 ε 2h(t) t+εu 0 sup H ∂ ∂H K H (t + εu, s) 2 ds ≤ Cε 2β-2h(t) t+u 0 |Φ t+u (s)| 2 ds -→ ε→0 0.
Now it remains to prove the tightness in the space of continuous functions endowed by the uniform norm. We also consider T > 0 such that t, t + εu and t + εv ∈ [0, T ] for all ε. Making similar calculations as in the proof of Corollary 4 we get that there exist

C T > 0 such that E B h (t + εu) -B h (t) ε h(t) - B h (t + εv) -B h (t) ε h(t) 2 = 1 ε 2h(t) E (B h (t + εu) -B h (t + εv)) 2 ≤ C T ε 2h(t) |εu -εv| 2h(t+εv) = C T ε 2(h(t+εv)-h(t)) |u -v| 2h(t+εv) Since h is β-Hölder then ε 2(h(t+εv)-h(t)) is uniformly bounded. Moreover h(t + εv) ≥ a, thus E B h (t + εu) -B h (t) ε h(t) - B h (t + εv) -B h (t) ε h(t) 2 ≤ C T |u -v| 2a
that ends the proof.

It is classical to deduce pointwise Hölder continuity from local self-similarity [START_REF] Benassi | Identifying the multifractional function of a Gaussian process[END_REF]. We recall that the pointwise Hölder continuity of a function f is characterized by the pointwise Hölder exponent α f (t 0 ) defined at each point t 0 as

α f (t 0 ) = sup α > 0 : lim t→t 0 |f (t) -(t 0 )| |t -t 0 | α = 0 . Proposition 6 For every t 0 ∈ R + the pointwise Hölder exponent α B h (t 0 ) of B h is almost surely equal to h(t 0 ).
Proof. We deduce from Proposition 5 that α B h (t 0 ) ≤ h(t 0 ). Now we prove that α B h (t 0 ) ≥ h(t 0 ). Let ε > 0. For every s and t ∈ [t 0 -ε, t 0 + ε] such that |t -s| < 1, from Proposition 3 we have

E[(B h (t) -B h (s)) 2 ] ≤ C|t -s| 2 inf [t 0 -ε,t 0 +ε] h .
By the fact that B h is Gaussian and applying Kolmogorov theorem [START_REF] Billingsley | Convergence of Probability Measures[END_REF], we get that

lim t→t 0 |f (t) -f (t 0 )|/|t -t 0 | α = 0 for every α < inf [t 0 -ε,t 0 +ε] h.
This holds for every ε > 0, so by continuity of h we get lim

t→t 0 |f (t) -f (t 0 )|/|t -t 0 | α = 0 for every α < h(t 0 ).
We can deduce that α B h (t 0 ) ≥ h(t 0 ), and hence α B h (t 0 ) = h(t 0 )

Stochastic calculus for B h

The aim of this section is to apply the stochastic calculus developed by Alòs, Mazet and Nualart in [START_REF] Alòs | Stochastic calculus with respect to Gaussian processes[END_REF] to get a stochastic integral for B h and an Itô formula. We recall that in [START_REF] Alòs | Stochastic calculus with respect to Gaussian processes[END_REF] the following hypothesis, called (K4), appears for regular kernels:

• Hypothesis (K4). For all s ∈ [0, T ), K(•, s) has bounded variation on the interval (s, T ], and

T 0 |K|((s, T ], s) 2 ds < ∞.
Lemma 7 Suppose that h is of bounded variation on (s, T ] for all s ∈ [0, T ). Then (K4) holds for (t, s) → K h(t) (t, s) defined by [START_REF] Alòs | Stochastic calculus with respect to Gaussian processes[END_REF].

Proof. Let Var n (s,T ] (•, s) = sup t 0 =s<t 1 <...<tn=T n i=1 |K h(t i ) (t i , s) -K h(t i-1 ) (t i-1 , s)|,
and suppose without restriction of generality that T = 1. Then

|K h(t i ) (t i , s) -K h(t i-1 ) (t i-1 , s)| ≤ I 1 (i) + I 2 (i),
where

I 1 (i) = |K h(t i ) (t i , s) -K h(t i ) (t i-1 , s)| and I 2 (i) = |K h(t i ) (t i-1 , s) -K h(t i-1 ) (t i-1 , s)|
We have

I 1 (i) = s 1/2-h(t i ) t i t i-1 (y -s) h(t i )-3/2 y h(t i )-1/2 dy ≤ s 1/2-b t i t i-1
(y -s) a-3/2 y a-1/2 dy.

Therefore

n i=1 |K h(t i ) (t i , s) -K h(t i ) (t i-1 , s)| ≤ s 1/2-b 1 s (y -s) a-3/2 y a-1/2 dy ≤ C(a)s 1/2-b . (7)
Regarding I 2 we have

n i=1 K h(t i ) (t i-1 , s) -K h(t i-1 ) (t i-1 , s) ≤ n i=1 |h(t i ) -h(t i-1 )| sup s≤t≤1,a≤λ≤b ∂ ∂λ K λ (t, s) . The proof of Lemma 1 implies that Var n (s,T ] (•, s) ≤ C(1∨ | log s |)s 1/2-b
, where the constant C > 0 depends on h (but not on n). This implies that (K4) holds for K h .

Remark 8 Since h is also supposed β-Hölder continuous for some β ≤ 1, we will suppose from now on that h is Lipschitz-continuous. Notice that lim tցu K h(t) (t, u) = 0 for all u > 0.

For simplicity we write K instead of K h but keep in mind that (t, s) → K(t, s) means (t, s) → K h(t) (t, s) and that differentials with respect to t act also via h.

In the sequel we need the following proposition regarding the variance of B h .

Proposition 9 The variance s → R s = E[B h (s) 2
] is of bounded variation on (0, T ].

Proof. We have This concludes the proof.

n i=1 R s i+1 -R s i = n i=1 s i+1 0 K(s i+1 , r) 2 dr - s i 0 K(s i , r) 2 dr ≤ n i=1 s i+1 0 K(s i+1 , r) 2 dr + n i=1 s i 0 [K(s i+1 , r) 2 -K(s i , r) 2 ]dr .
For any f ∈ L 2 ([0, T ]) let Kf be defined by (Kf )(t) = t 0 K(t, s)f (s)ds. Let E be the set of step functions on [0, T ], and let the operator K * be defined on E by (K * ϕ)(s) = T s ϕ(t)K(dt, s). Then K * is the adjoint of K. In fact, for a i ∈ R, 0 = s 1 < s 2 < ... < s n+1 = T , ϕ = n i=1 a i 1 (s i ,s i+1 ] (s) and f ∈ L 2 ([0, T ]), we write

(K * ϕ)(s) = n i=1 1 (s i ,s i+1 ] (s) n j=i a j (K(s j+1 , s) -K(s j , s))
and

T 0 (K * ϕ)(s)f (s)ds = n j=1 a j T 0 j i=1 1 (s i ,s i+1 ] (s)(K(s j+1 , s) -K(s j , s))f (s)ds = n j=1 a j [(Kf )(s j+1 ) -(Kf )(s j )] = T 0 ϕ(t)(Kf )(dt). ( 8
)
As usual the Reproducing Kernel Hilbert Space (RKHS) H is defined as the closure of the linear span of the indicator functions {1 [0,t] , t ∈ [0, T ]} with respect to the scalar product 1

[0,t] , 1 [0,s] H = E[B h(t) (t)B h(s) (s)] ≡ R(t, s)
. By replacing f (s)ds in (8) by W (ds), we have by ( 1)

B h (ϕ) ≡ T 0 ϕ(t)B h (dt) = T 0 (K * ϕ)(s)W (ds).
Therefore

ϕ 2 H = E[B h (ϕ) 2 ] = K * ϕ 2 L 2 ([0,T ]) ≤ T 0 T 0 |ϕ(t)||K|(dt, s) 2 ds =: ϕ 2 K .
Let us denote by H K the completion of E with respect to the • K -norm. Then H K is continuously embedded in H.

In order to define the stochastic integral with respect to B h , let us denote by S the set of smooth cylindrical random variables of the form

F = f (B h (ϕ 1 ), B h (ϕ 2 ), ..., B h (ϕ n )), where n ≧ 1, f ∈ C ∞ b (R n ) (f
and all its derivatives are bounded) and ϕ 1 , ϕ 2 , ..., ϕ n ∈ H. Let us also denote by

D 1,2 (H K ) the closure of {F ∈ S : F ∈ L 2 (Ω, H K ), DF ∈ L 2 (Ω × H K , H K )}. Then D 1,2 (H K ) is included in the domain Dom(δ B h ) of the divergence operator of B h , and the integral of u ∈ Dom(δ B h ) with respect to B h is given by δ B h (u) ≡ T 0 u δB h = T 0 (K * u) δW ≡ T 0 T s u(r)K(dr, s) δW (s),
where the last two integrals are the divergence integrals with respect to Brownian motion. Let us recall that the integral δ B h (u) is defined, for any u ∈ L 2 (Ω, H), as the unique element in L 2 (Ω) which satisfies the duality relationship E(δ B h (u)F ) = E DF, u H for all F ∈ S.

The following Itô formula, due to Alòs, Mazet and Nualart [START_REF] Alòs | Stochastic calculus with respect to Gaussian processes[END_REF], will be applied in the next section. Let F be a function of class C 2 (R) satisfying the condition

max{|F (x)|, |F ′ (x)|, |F ′′ (x)|} ≤ ce λ|x| 2 ,
where c and λ are positive constants such that λ < 1 4 sup 0≦t≤T R t -1

. This implies that the process F ′ (B t ) belongs to D 1,2 (H K ). The integral t 0 F ′ (B h (s))δB h (s) is therefore well defined for all t ∈ [0, T ], and the following Itô-type formula holds ([1], Theorem 2): This definition implies that the local time of B h satisfies the following occupation density formula C g(t, B h (t))dt = C×R g(t, x)L(dt, x)dx [START_REF] Lacaux | Real harmonizable multifractional Lévy motions[END_REF] for all continuous functions with compact support g : C × R → R. If g does not depend explicitely on t, we get the more classical occupation density formula where the right side of ( 10) is replaced by C g(x)L(C, x)dx.

F (B h (t)) = F (0) + t 0 F ′ (B h (s))δB h (s) + 1 2 t 0 F ′′ (B h (s))dR(s). ( 9

Proposition 11

The local time of B h exists P -a.s. on any interval [0, T ] and is a square integrable function of x.

Proof. By [START_REF] Berman | Local times and sample function properties of stationary Gaussian process[END_REF], for any continuous and zero mean Gaussian process {X t , t ∈ [0, T ]} with bounded covariance function, the condition

T 0 T 0 ds dt E[|X t -X s | 2 ] < +∞ (11)
is sufficient for the local time of X to exist and to be a square integrable function of x. If (s, t) is away from the diagonal we write for s < t

E[|B h (t) -B h (s)| 2 ] = s 0 (K h(t) (t, u) -K h(s) (s, u)) 2 du + t s K h(t) (t, u) 2 du
and deduce from (1) that the second term stays strictly positive as (s, t)

varies in [0, t - ε] × [0, T ]. If (s, t) is close to the diagonal, say 0 ≤ t -s < ε, by Proposition 5, E[|B h (t) -B h (s)| 2 ] ∼ c -2 h(t) (t -s) 2h(t)
as s ր t, and a direct calculation shows that (11) is satisfied.

Let us now derive a Tanaka-type formula for B h . Since the last term in the Itô formula ( 9) is an integral with respect to the variance function R and since R is not in general increasing, but only of finite variation, the trajectorial representation of the local time is more delicate than for Brownian motion or for fractional Brownian motion. On the time intervals where R is (strictly) inceasing or decreasing, this formula gives in fact an occupation density L related to L.

Theorem 12 Suppose that h is continuously differentiable. Then, for all a ∈ R, a) and L is the local time with respect to the Lebesgue measure of B h . On the time intervals [s, t] where R is strictly inceasing (resp. strictly decreasing), L (resp. -L) is the (positive) occupation density of B h with respect to the measure induced by R.

|B h (t) -a| -|B h (s) -a| = t s sign(B h (u) -a)dB h (u) + L([s, t], a) P -a.s., where L([s, t], a) = t s R ′ (u)L(du,
Remark 13 a) No information on the local time of B h can be drawn from the above formula if

R ′ (u) = 0 on the interval [s, t]. In fact, R ′ (u) = d du E[B 2 h(u) (u)] = d du (u 2h(u) ) = 2 h ′ (u) log u + h(u) u u 2h(u) ,
and R ′ (u) = 0 if h(u) = 1/ log u ∈ (1/2, 1) on an interval (s, t). In this case L([s, t], a) = 0 for all a. b) Tanaka formulas for fractional Brownian motion have been shown by several authors. We refer to the survey [START_REF] Coutin | An introduction to (stochastic) calculus with respect to fractional Brownian motion[END_REF] for references.

Proof. For ε > 0 let p ε (x) = (2πε) -1/2 exp(-x 2 /(2ε)). We apply the Itô formula of the previous section to the process

F ε (x) = t 0 F ′ ε (y)dy where F ′ ε (x) = 2 x -∞ p ε (y)dy -1.
Then F ′ ε (x) converges to sign(x) and F ε (x) converges to |x| as ε → 0. By [START_REF] Coutin | Tanaka formula for the fractional Brownian motion[END_REF], for ε > 0 fixed,

F ε (B h (t) -a) = F ε (-a) + t 0 t s F ′ ε (B h (r) -a)K(dr, s)δW s + t 0 p ǫ (B h (s) -a)dR(s). (12)
Notice that by (K4) the process {F ′ ε (B h (r) -a), r ∈ [0, t]} belongs to L 2 (Ω, H K ) and belongs therefore to Dom(δ B h ). Or, equivalently, the process

{ t s F ′ ε (B h (r)-a)K h (dr, s), s ∈ [0, t]} belongs to Dom(δ W ). Clearly F ε (B h (t) -a) converges to |B h (t) -a| in L 2 (Ω) and F ǫ (-a) converges to |a| if ε → 0. Moreover, the process {F ′ ε (B h (r) -a), r ∈ [0, t]} converges, as ε → 0, to {sign(B h (r) -a), r ∈ [0, t]} in L 2 (Ω, H K ). In fact, by (K4), E t 0 t s |F ′ ε (B h (r) -a) -sign(B h (r) -a)||K h |(dr, s) 2 ds
is upper bounded independently of ε, and we may apply the dominated convergence theorem (Lemma 1 of [START_REF] Coutin | Tanaka formula for the fractional Brownian motion[END_REF]). Therefore the last term in [START_REF] Nualart | The Malliavin calculus and related topics[END_REF] converges in L 2 (Ω). Let us denote the limit by Λ a t . Therefore, for any continous function f with compact support in R, t 0 p ε (B h (s)) -a)dR(s) f (a)da [START_REF] Pitt | Local time of Gaussian vector fields[END_REF] converges in L 1 (Ω) to Λ a t f (a)da. In fact, the dominated convergence theorem applies, because

t 0 E[p ε (B h (s)) -a)]dR(s) = t 0 p R(s)+ε (a)dR(s) ≦ t 0 s -b R ′ (s)ds < ∞.
But (13) converges also to

t 0 f (B h (s))dR(s) = t 0 f (B h (s))R ′ (s)ds, where we use the fact that R is differentiable if h is differentiable. Hence t 0 f (B h (s))R ′ (s)ds = Λ a t f (a)da.
By the occupation density formula [START_REF] Lacaux | Real harmonizable multifractional Lévy motions[END_REF] applied to g(s, x) = f (x)R ′ (s) we get Λ a t = t 0 R ′ (s)L(ds, a) = L(t, a) for λ-a.e. a, P -a.s. We can extend to all a ∈ R since there exists a jointly continous version of L and therefore of L, as will be shown next by Berman's methods which are independent of the Tanaka formula.

We state now regularity properties in time and space of the trajectories of L. The regularity properties of L follow easily since L(t, x) = t 0 R ′ (s)L(ds, x). In order to show the existence of a jointly continuous version of L, we recall the hypotheses introduced in [START_REF] Boufoussi | Path properties of a class of locally asymptotically self similar processes[END_REF] and show that they are satisfied for B h . We recall them in terms of any real valued separable random process {X(t), t ∈ [0, T ]} with Borel sample functions.

• Hypothesis (A). There exist numbers ρ 0 > 0 and H ∈ (0, 1) and a positive function ψ ∈ L 1 (R) such that for all λ ∈ R and t, s ∈ [0, T ], 0 < |t -s| < ρ 0 we have

E exp iλ X(t) -X(s) |t -s| H ≤ ψ(λ).
• Hypothesis (A m ). There exist positive constants δ and c (both eventually depending on m ≥ 2) such that for all t 1 , t 2 , • • • , t m with 0 =:

t 0 < t 1 < • • • < t m ≤ T and |t m -t 1 | < δ we have E   exp   i m j=1 u j (X(t j ) -X(t j-1 ))     ≤ m j=1 |E[exp(icu j (X(t j ) -X(t j-1 )))]| for all u 1 , u 2 , • • • , u m ∈ R.
Hypothesis (A) is evidently satisfied for selfsimilar processes with stationary increments: here ψ(λ) = |E[e iλX (1) ]|. Hypothesis (A) is also closely related to asymptotic selfsimilarity, and it holds in fact for a fairly large class of processes (see Proposition 15). If X has independent increments, (A m ) is trivially true for all m ≥ 2. When the left and the right side of the inequality is applied to the characteristic function of a Gaussian process X, we get the condition which is known in the litterature under the name of local nondeterminism (LND). Loosely speaking, LND says how much dependence is allowed for the increments of the process if the local time should have certain regularity properties. As a general rule, the trajectories of local time get more regular if the trajectories of the process get less regular.

Proposition 14 For all T > 0, the process {B h (t), t ∈ [0, T ]} satisfies Hypotheses (A) and (A m ) for all m ≥ 2 with H = max 0≤t≤T h(t).

Proof. Let us start by showing (A). By similar calculations as in the proof of Proposition 5, for every t ∈ [0, T ]

E[ε -2h(t) (B h (t + ε) -B h (t)) 2 ] - 1 c 2 h(t) ≤ ε β-sup [0,T ] h T +1 0 (Φ T +1 (s)) 2 ds.
We can now conclude by applying Proposition 15. Now we prove (A m ) for all m ≥ 2. We show that B h satisfies the LND property as it has been introduced by S. Berman for Gaussian processes. For simplicity we write B instead of B h in this proof. Let t 1 < t 2 < • • • < t m , and let V m be the relative conditioning error given by

V m = Var[B(t m ) -B(t m-1 )|B(t 1 ), • • • , B(t m-1 )] Var[B(t m ) -B(t m-1 )] .
The Gaussian process is said to be LND if

lim inf cց0 + 0<tm-t 1 ≤c V m > 0. (14)
This condition means that a small increment of the process is not completely predictable on the basis of a finite number of observations from the immediate past. For Gaussian processes S. Berman [START_REF] Berman | Local nondeterminism and local times of Gaussian processes[END_REF] has proved that ( 14) implies (A m ). More precisely he has shown that if X satisfies ( 14) for all m ≥ 2, then there exist constants C m > 0 and δ m > 0 such that, for all

u 1 , u 2 , • • • , u m ∈ R, Var   m j=1 u j [B(t j ) -B(t j-1 )]   ≥ C m m j=1 u 2 j Var [B(t j ) -B(t j-1 )] ,
where t 0 = 0 and t 1 < • • • < t m are different and lie in an interval of length at most δ m . This implies (A m ). In order to prove that B verifies [START_REF] Samorodnitsky | Stable non-Gaussian random processes[END_REF], fix m ≥ 2 and let t < t 1 < t where the last inequality follows from Lemma 7.1 of [START_REF] Pitt | Local time of Gaussian vector fields[END_REF]. The last term is strictly positive since the 0 < u 1 < • • • < u m < t.

The beginning of the proof of Proposition 14 shows that the hypothesis (A) holds for a much larger class of processes. In fact we have only used the assumption of the following proposition.

Proposition 15 Let {X t , t ∈ [0, T ]}be a centered Gaussian process. Suppose that for some positive continuous functions f : [0, T ] → (0, 1) and g : [0, T ] → (0, ∞)

E[ε -2f (t) (X(t + ε) -X(t)) 2 ] -→ ε→0 g(t) (15)
uniformly in t. Then Hypothesis (A) holds.

Proof. Let us fix H > sup f =: f * . Because X is Gaussian and centered, we have

E exp iλ X(t + ε) -X(t) ε H = exp - λ 2 2 E X(t + ε) -X(t) ε H 2 . ( 16
)
Because of ( 15), there exists ε 0 such that for every ε satisfying |ε| < ε 0 and for every t we have

E X(t + ε) -X(t) ε f (t) 2 ≥ c 2 (17)
where c = inf [0,T ] g. Besides,

ε 2f (t)-2H ≥ ε 2f * -2H ≥ 1, (18)
and thus

E X(t + ε) -X(t) ε H 2 = ε 2f (t)-2H E X(t + ε) -X(t) ε f (t) 2 ≥ c 2 . ( 19
)
Then, combining ( 16) and (19) we get for every λ, and t and s satisfying |t -s| < ε 0 ,

E exp iλ X(t + ε) -X(t) ε H ≤ exp - λ 2 c 4 .
Then we choose ψ(λ) = exp -λ 2 c/4 to conclude the proof.

Let us now state a regularity result for the trajectories of L. The following theorem says that L(t, x) is Hölder continuous in t of order 1 -H and Hölder continuous in x of order 1-H 2H , where H is the constant appearing in (A). For the proofs we refer to [START_REF] Boufoussi | Path properties of a class of locally asymptotically self similar processes[END_REF], where it is shown that these regularities hold for any process starting at zero and satisfying (A) and (A m ) for all m ≥ 2.

Theorem 16 Suppose that ψ in (A) satisfies |u|≥1 |u| (1-H)/H ψ(u)du < ∞.

Then {B h (t), t ∈ [0, T ]} has a jointly continuous local time {L(t, x), (t, x) ∈ [0, T ] × R}. Moreover, for any compact set K ⊂ R and any interval I ⊂ [0, T ] with length less than ρ 0 (the constant appearing in (A)), (i) if 0 < ξ < (1 -H)/2H, then |L(I, x) -L(I, y)| ≤ η|x -y| ξ for all x, y ∈ K (ii) if 0 < δ < 1 -H, then sup x∈K L(I, x) ≤ η|I| δ , where η is a random variable, almost surely positive and finite and |I| is the length of I.

2 T 0 k

 20 The functions |K(s, r)1 [0,s] (r)| are bounded by the square integrable function k(r) = |K(T, r)| + |K|((r, T ], r). Hence the first term above is upper bounded by T 0 k(r) 2 dr. The second term is upper bounded by s i , s i+1 ], r) |K(s i+1 , r) + K(s i , r)| dr ≤ (r)|K|((r, T ], r)dr < ∞.

) 4

 4 Local time and Tanaka formula for B hFirst we prove by means of a criterion for Gaussian processes due to S. Berman that B h has a local time with respect to the Lebesgue measure. Then we derive a Tanaka-type formula from the Itô formula of Section 3 and show that B h satisfies the LND property. This implies continuity and Hölder regularities in space and in time of the trajectories of the local time.

Definition 10 For any Borel set

C ⊂ R + the occupation measure m C of B h on C is defined, for all Borel sets A ⊂ R, by m C (A) = λ{t ∈ C, B h(t) (t) ∈ A},

where λ is the Lebesgue measure on R + . If m C is absolutely continous with respect to the Lebesgue measure on R, the local time (or occupation density) of B h on C is defined as the Radon-Nikodym derivative of m C and will be denoted by {L(C, x), x ∈ R}. Sometimes we write L(t, x) instead of L([0, t], x).

where C h(t) is a constant depending on t. Therefore lim δ→0

Var[B(t m ) -B(t m-1 )|B(t 1 ), • • • , B(t m-1 )]

Var[B(t m ) -B(t m-1 )]

Moreover, by adding B(t) to the conditional set,

Then 0 < u i < t, and when δ → 0 the t i = t δ i are chosen in such a way that the u i do not change. Therefore

.

By


Juan Vazquez-Rodriguez

Grégoire Lefebvre

Julien Cumin

James L Crowley

EMOTION RECOGNITION WITH PRE-TRAINED TRANSFORMERS USING MULTIMODAL SIGNALS AUTHOR VERSION

Keywords: Affective Computing, Multimodal Emotion Recognition, Machine Learning

In this paper, we address the problem of multimodal emotion recognition from multiple physiological signals. We demonstrate that a Transformer-based approach is suitable for this task. In addition, we present how such models may be pretrained in a multimodal scenario to improve emotion recognition performances. We evaluate the benefits of using multimodal inputs and pre-training with our approach on a state-ofthe-art dataset.

Introduction

The increasing availability of mass-market wearable devices equipped with sensors for physiological signals provides new possibilities for monitoring the emotional health and wellbeing of users [START_REF] Abdat | Human-Computer Interaction Using Emotion Recognition from Facial Expression[END_REF]. Although less reliable than medical-grade sensors, signals from wearable sensors like electrocardiograms (ECG) and electroencephalograms (EEG) can be combined to provide estimates of the emotional state of users.

In this work, we report on experiments with a Transformerbased approach for interpreting emotional state from different physiological signals obtained from wearable devices. We explore the estimation of emotional state from individual sensor modalities, including EEG and ECG, and show that fusing the two modalities leads to better results indicating that these modalities convey complementary information. Furthermore, we demonstrate that a Transformer-based approach can be used to provide reliable estimates of emotional state from such signals.

We center our work on recognizing emotions from ECG and EEG signals. There are other works that address multimodal emotion recognition [START_REF] Rahman | Integrating Multimodal Information in Large Pretrained Transformers[END_REF][START_REF] Siriwardhana | Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion[END_REF][START_REF] Siriwardhana | Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition[END_REF][START_REF] Khare | Self-Supervised Learning with Cross-Modal Transformers for Emotion Recognition[END_REF], but the majority use signals such as images, sound and text, and not physiological signals. Although some authors have explored the use of physiological signals for emotion recognition [START_REF] Miranda Correa | AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups[END_REF][START_REF] Siddharth | Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing[END_REF][START_REF] Ross | Unsupervised multi-modal representation learning for affective computing with multi-corpus wearable data[END_REF], such signals have received less attention than other sensing modalities.

A common problem when addressing the task of emotion recognition is the lack of labeled data to effectively train deeplearning models [START_REF] Miranda Correa | AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups[END_REF]. A possible approach to address this problem is the use of unsupervised pre-training techniques [START_REF] Erhan | Why Does Unsupervised Pre-training Help Deep Learning?[END_REF]. However, pre-training with multiple signal modalities raises additional challenges. In this work, we investigate the use of a late-fusion approach, where we pre-train and fine-tune different single-modality models, and then combine the outputs of the individual models to obtain a fused feature that can be used to perform emotion prediction.

We use a Transformer [START_REF] Vaswani | Attention is All you Need[END_REF] to process physiological signals.

The Transformer was originally developed for Natural Language Processing (NLP) tasks, with the intent of processing sequences of words. Given that physiological signals are sequences of values, the Transformer can be adapted for physiological signal processing [START_REF] Yan | Fusing Transformer Model with Temporal Features for ECG Heartbeat Classification[END_REF]. Transformers employ a learned attention mechanism to dynamically score the relevance of different parts of an input according to context. Attentionbased processing is appropriate for processing physiological signals, as some parts of a signal may convey more information than other parts depending on the task and context. Another advantage of using a Transformer is that we can benefit from a very successful pre-training technique described in BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] and developed for NLP tasks, which we can adapt to our needs. This pre-training strategy has been successfully adapted to other domains like Computer Vision [START_REF] Sun | VideoBERT: A Joint Model for Video and Language Representation Learning[END_REF], Speech Processing [START_REF] Huang | Speech Recognition by Simply Fine-Tuning Bert[END_REF] and Affective Computing [START_REF] Vazquez-Rodriguez | Transformer-Based Self-Supervised Learning for Emotion Recognition[END_REF].

The main contributions of this paper are:

1. We present a technique for recognizing emotions from multimodal physiological signals using a Transformer.

2. We describe a method to pre-train the Transformer for recognizing emotions from multimodal physiological signals.

3. We provide results from experiments that show that a multimodal pre-training strategy is effective for improving emotion recognition performances.

Contrary to traditional techniques like Gaussian naive Bayes [START_REF] Gjoreski | An Inter-domain Study for Arousal Recognition from Physiological Signals[END_REF], k-Nearest Neighbours [START_REF] Shu | Wearable Emotion Recognition Using Heart Rate Data from a Smart Bracelet[END_REF] and Support Vector Machines [START_REF] Miranda Correa | AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups[END_REF], deep-learning may be used to recognize emotions directly from sensor signals without a need to design feature descriptors. This is particularly useful for the recognition of emotions from physiological signals where there are no well-established feature descriptors for signal encoding.

An example of a deep-learning approach is provided by the work of Santamaria et al. [START_REF] Santamaria-Granados | Using Deep Convolutional Neural Network for Emotion Detection on a Physiological Signals Dataset (AMIGOS)[END_REF], where they employ models based on Convolutional Neural Networks (CNN) to perform emotion recognition. Another example is the work of Harper and Southern [START_REF] Harper | A Bayesian Deep Learning Framework for End-To-End Prediction of Emotion from Heartbeat[END_REF], who use a combination of Recurrent Neural Networks (RNN) and CNNs. The Transformer [START_REF] Vaswani | Attention is All you Need[END_REF], which uses stacked layers of self-attention, has recently emerged as a powerful alternative to Convolutional and Recurrent Networks. In this work, we are interested in whether a Transformer architecture can be an effective tool to recognize emotions from multiple physiological signals.

A variety of authors have explored deep-learning models for emotion recognition using multimodal signals. Most of these works use images, audio and/or text as inputs [2-5, 20, 21]. In a few cases, physiological signals have been used to improve recognition from image, audio and text [START_REF] Choi | Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals[END_REF][START_REF] Xing | Exploiting EEG Signals and Audiovisual Feature Fusion for Video Emotion Recognition[END_REF][START_REF] Matsuda | EmoTour: Multimodal Emotion Recognition using Physiological and Audio-Visual Features[END_REF]. A few authors have described the use of multiple physiological signal modalities [START_REF] Miranda Correa | AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups[END_REF][START_REF] Siddharth | Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing[END_REF][START_REF] Ross | Unsupervised multi-modal representation learning for affective computing with multi-corpus wearable data[END_REF]. These works consistently demonstrate the benefits of exploiting multiple signal modalities to improve the performance of emotion recognition.

Some multimodal approaches employ pre-training techniques to improve their results. The authors of [START_REF] Rahman | Integrating Multimodal Information in Large Pretrained Transformers[END_REF][START_REF] Siriwardhana | Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion[END_REF][START_REF] Siriwardhana | Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition[END_REF][START_REF] Khare | Self-Supervised Learning with Cross-Modal Transformers for Emotion Recognition[END_REF] develop models based on Transformers, using images, audio and text to recognize emotions. Rahman et al. [START_REF] Rahman | Integrating Multimodal Information in Large Pretrained Transformers[END_REF] report on the use of BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF], a Transformer-based model pre-trained for NLP tasks, to process text, incorporating visual and audio modalities in a middle-fusion process. Siriwardhana et al. [START_REF] Siriwardhana | Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion[END_REF][START_REF] Siriwardhana | Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition[END_REF] describes the use of pre-trained models to extract features from visual, audio and text modalities, and followed by a cross-modal Transformer [START_REF] Tsai | Multimodal Transformer for Unaligned Multimodal Language Sequences[END_REF] to combine these different features. Khare et al.

[5] use a BERT-like approach, masking some words in the input text, along with the audio and visual parts that correspond to those words, and then pre-training the model by predicting the masked words. 3 Approach

Multimodal Emotion Recognition

When performing a classification task using multiple modalities, an important problem is to determine the processing level at which different signal modalities should be combined or fused. One option is early-fusion, where the combination of input features is used as input for the model. A second option is to do late-fusion, where the outputs of single-modality models are combined and a second-level model is trained to perform the classification. A third option is a compromise between these two extremes: middle-fusion that combines features from intermediate layers of the models.

Early-fusion can be provided by simply concatenating the input signals in the temporal dimension, to form a single (longer) sequence. The problem with this approach is that the computational complexity of the Transformer is O(n 2 ), where n is the length of the input. Therefore, a late-fusion approach has the advantage of simplifying the training process: several singlemodality models can be trained one by one on less powerful hardware than the one required to train a single, more computationally expensive multimodal model. Then, if the singlemodality models are frozen, the second-level model of the latefusion approach can also be easily trained without using very powerful hardware resources. A similar reasoning can be applied to see the advantages of late-fusion over middle-fusion in this scenario.

Another major difficulty with early-fusion is that it hinders the benefits of pre-training techniques. With pre-training, we seek to use many different datasets, not necessarily related to the task of emotion recognition, to obtain a more general representation of the information from the different modalities. However, early-fusion limits pre-training to datasets that include all of the targeted modalities, thus severely limiting the availability of datasets that can be used. With late-fusion, one can pre-train each single-modality model independently from one another, with potentially different datasets.

In our case, we are interested in a multimodal approach that allows us to use pre-training techniques to improve the per-formance of the model. In particular, we are interested in attention-based models such as the Transformer [START_REF] Vaswani | Attention is All you Need[END_REF], and in pre-training techniques similar to the ones used in BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF].

In this work, we explore the use of a late fusion approach. This allows self-supervised pre-training for individual sensor modalities by reconstructing masked values in the input signal, similar to what is done in BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF]. We refer to this as Masked Value Prediction or simply MVP.

With this approach, recognition training is performed in two steps. In the first step, we train two single-modality models: one to recognize emotions from electrocardiogram (ECG) signals, and one to recognize emotions from electroencephalogram (EEG) signals. Both models are trained separately, using MVP pre-training. In the second step, we concatenate the outputs of the single-modality models and use this combined representation to train a Fully-Connected Network (FCN) to recognize emotions. While other fusion approaches may be possible like max or average pooling, or majority voting, concatenation has been found to provide a simple and effective technique for use with Transformers.

In the rest of this section, we provide details about the ECG and EEG single-modality models, and the fused model that combines both signals.

ECG Single-Modality Emotion Recognition

For our ECG emotion recognizer, we employ the approach described in [START_REF] Vazquez-Rodriguez | Transformer-Based Self-Supervised Learning for Emotion Recognition[END_REF] and depicted in Figure 1. In this approach, a pre-training step is first used prior to fine-tuning the model to improve its performance. We provide a brief description of this approach in the remainder of this subsection. We refer the reader to the original paper for further details.

The ECG single-modality emotion recognition model from [START_REF] Vazquez-Rodriguez | Transformer-Based Self-Supervised Learning for Emotion Recognition[END_REF] is based on the Transformer [START_REF] Vaswani | Attention is All you Need[END_REF]. The Transformer is an architecture capable of incorporating contextualized information thanks to its self-attention mechanisms. As shown in Figure 1, the model first encodes ECG signals using a 1D Convolutional Neural Network (1D-CNN) to obtain a sequence of features that represent the input signal. Then, similar to BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF], a classification token named CLS is added to the beginning of the sequence of features. In the next step, the feature sequence appended with the CLS token is fed into a Transformer, which produces contextualized representations of the signal. Then, the process follows one of two paths, depending on if we are in the pre-training phase or the fine-tuning phase.

Pre-Training Phase

The pre-training task consists in using MVP, that is, masking some points from the original signal, and then predicting those masked points with the help of a Fully-Connected Network (FCN) placed on top of the Transformer, as can be seen in path A of Figure 1. Since this task is self-supervised, we do not need labeled data for this phase.

Fine-Tuning Phase

After the model has been pre-trained, it is fine-tuned to recognize emotions. As depicted in path B of Figure 1, the vector CLS e , which is the representation of the CLS token, is used as input of an FCN that functions as a classifier to predict the emotion. Starting with the same pre-trained weights, the model is fine-tuned twice: one to predict arousal and another to predict valence. Thus, Predicted Emotion in Figure 1 refers to either arousal or valence. During this phase, all the parameters of the model, including the Transformer and the 1D-CNN parameters, are fine-tuned. This phase is supervised, therefore labeled data is employed.

EEG Single-Modality Emotion Recognition

To build our EEG single-modality emotion recognition model, we adapt the ECG model from [START_REF] Vazquez-Rodriguez | Transformer-Based Self-Supervised Learning for Emotion Recognition[END_REF] and described in the previous subsection, to accept EEG signals.

First, we need to take into account that the ECG signal has only one channel, while the EEG signal is typically multichannel. Therefore, the 1D-CNN that encodes the raw signal has to be changed from having one-channel input to having as many inputs as the number of channels present in the EEG signal, while the rest of the structure of the 1D-CNN remains the same. The shape of the output from the 1D-CNN encoder remains similar to the original ECG model, and thus we can keep the same Transformer architecture.

Secondly, during the pre-training of the model, we mask the same temporal segments of the EEG signal across all the different channels. The size of the output layer of the FCN that works as masked-point predictor has to be the same as the number of input channels. This way, each predicted output value corresponds to the (masked) value of each channel.

Aside from these changes, no other adaptation is needed for fine-tuning the EEG model, as we employ the CLS e vector as input for the FCN used as a classifier to predict the emotion, as it is done in the ECG model.

Fused-Signals Emotion Recognition Model

We use late-fusion to fuse the ECG and EEG signals, using the outputs from each of the trained single-modality emotion recognizers. As depicted in Figure 2, we take the output of the last hidden layer of the FCN (not from the output layer) of each single-modality model, and we concatenate those outputs to form our combined features. Although other methods might be considered to do this fusion, for example using average pooling of the outputs of the single-modality recognizers, we employ concatenation because it allows us to have singlemodality models with outputs of different sizes. This is convenient because this way we can choose without constraints the output sizes that make the fused model perform the best.

To perform the emotion prediction from the fused modalities, we use another FCN that we refer to as Multimodal Emotion Classifier in Figure 2. For this step, we freeze the weights of the single-modality emotion recognizers, and only train the

Experiments

We evaluate our model on the task of binary emotion prediction, that is to predict high and low levels of arousal and valence, from multimodal physiological signals.

In this section, we describe our experimental setup, giving details about the datasets and the hyper-parameters for both EEG and fused-based models.

Datasets

To train and evaluate our models, we used the AMIGOS dataset [START_REF] Miranda Correa | AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups[END_REF]. This dataset includes data from 40 subjects, where emotions were induced by making the subjects watch emotional videos. A total of 37 subjects watched 20 videos, while 3 subjects watched only 16. After watching each video, the subjects filled out a self-assessment form where they rated on a scale of 1 to 9 their levels of arousal and valence. We use the results of this self-assessment as labels in our experiments. Since we are interested in binary emotion classification, we use the average value as a threshold to obtain high and low classes of arousal and valence. In total, there are around 65 hours of data in the AMIGOS dataset.

The AMIGOS dataset includes both ECG and EEG signals.

We treat ECG as a single-channel signal, and we use the signal taken from the left arm. For EEG, we use 10 channels: F7, F3, T7, P7, O1, O2, P8, T8, F4, F8. We chose those 10 channels from the 14 available because these channels are also present in the datasets that we use for pre-training, which we will describe shortly. We use the provided 128Hz down-sampled signals. ECG signals are filtered with a low-pass filter with a cut-off frequency of 60Hz, and EEG signals are filtered with a band-pass filter with frequencies between 4.0 and 45.0Hz.

Data for pre-training

As described in Section 3.2, the first step in training a singlemodality emotion recognizer is pre-training. To pre-train the EEG emotion recognizer, we gathered EEG data that does not necessarily include labels of emotion. We use the following datasets: WAY-EEG-GAL [START_REF] Luciw | Multi-channel EEG recordings during 3,936 grasp and lift trials with varying weight and friction[END_REF], BCI2000 [START_REF] Schalk | BCI2000: A general-purpose brain-computer interface (BCI) system[END_REF][START_REF] Goldberger | PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals[END_REF], and Large-EEG-BCI [START_REF] Kaya | A large electroencephalographic motor imagery dataset for electroencephalographic brain computer interfaces[END_REF]. These datasets were gathered to develop Brain-Computer Interfaces, so they do not include any labels related to emotions. We also use parts of AMIGOS in the pre-training step, taking care of not using the same samples to pre-train and evaluate the model, for each fold of crossvalidation. The quantity of data that we gathered to pre-train the EEG model is comparable to the data used to pre-train the ECG model in [START_REF] Vazquez-Rodriguez | Transformer-Based Self-Supervised Learning for Emotion Recognition[END_REF]: in total there are around 195 hours of EEG data available for pre-training, while the ECG model was pre-trained with around 230 hours of data.

ECG Signal EEG Signal

ECG Emotion Recognizer EEG Emotion Recognizer

Last hiddenlayer outputs

Concatenation

Mulitmodal Emotion Classifier [FCN]

Predicted Emotion Figure 2: Fused Model. Late-fusion is used to combine the ECG and EEG signals. The outputs of the last layer from both single-modality models are concatenated, and then used as input to an FCN that performs the emotion prediction.

Signal pre-processing

Much like [START_REF] Vazquez-Rodriguez | Transformer-Based Self-Supervised Learning for Emotion Recognition[END_REF] for ECG data, we filter EEG signals using an 8 th order Butterworth band-pass filter, with cut-off frequencies of 0.8Hz and 50Hz. We also downsample signals to 128Hz.

In addition, we normalize signals with zero-mean and unitvariance for each subject. Finally, we split each signal into 10-second segments. Each segment is used as a sample in our experiments, as in other state-of-the-art works [START_REF] Miranda Correa | AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups[END_REF][START_REF] Ross | Unsupervised multi-modal representation learning for affective computing with multi-corpus wearable data[END_REF][START_REF] Sarkar | Self-Supervised Learning for ECG-Based Emotion Recognition[END_REF], [START_REF] Vazquez-Rodriguez | Transformer-Based Self-Supervised Learning for Emotion Recognition[END_REF].

ECG Emotion Recognizer

The ECG emotion recognition model follows the architecture presented by Vazquez-Rodriguez et al. in [START_REF] Vazquez-Rodriguez | Transformer-Based Self-Supervised Learning for Emotion Recognition[END_REF]. This model was fine-tuned and evaluated on AMIGOS as presented in Section 4.1. This model was parameterized and pre-trained as described in [START_REF] Vazquez-Rodriguez | Transformer-Based Self-Supervised Learning for Emotion Recognition[END_REF].

EEG Emotion Recognizer

The EEG emotion recognition model follows the architecture described in Section 3.3. The 1D-CNN is composed of three layers with kernel sizes (65, [START_REF] Liaw | Tune: A Research Platform for Distributed Model Selection and Training[END_REF][START_REF] Shu | Wearable Emotion Recognition Using Heart Rate Data from a Smart Bracelet[END_REF], with the number of channels equal to (64, 128, 256), with stride 1 in all the layers, and using the ReLU activation function. With this configuration, the size of the receptive field is 113 input points, which is equivalent to 0.88s. Based on preliminary studies, we believe that this receptive field size is suitable for EEG signals. For the Transformer, the number of layers is 2 and the number of heads is also 2, with a hidden size of 256.

For pre-training, the FCN used to predict masked points has one hidden layer with size 128 and ReLU activation, and an output layer that gives 10 outputs values, where each output value corresponds to the predicted value of each masked EEG channel. The mean squared error between the real and the predicted values is used as loss. We pre-train the model for 500 epochs, warming up the learning rate during the first 30 epochs up to 0.0005, and then use linear decay. We use Adam optimization with β 1 = 0.9, β 2 = 0.999 and L 2 weight decay of 0.005. A dropout value of 0.1 is used in the Transformer.

For fine-tuning, the FCN that predicts binary emotions has one hidden layer with a size of 64 and ReLU activation functions. An output layer is used to project the output to a single value, that corresponds to the prediction of the emotion. Two different networks are fine-tuned, one to predict arousal and another to predict valence. The models are fine-tuned using binary cross-entropy loss for 100 epochs, starting with a learning rate of 0.0001 and decreasing by a factor of 0.65 every 45 epochs. Adam optimization is used, with β 1 = 0.9, β 2 = 0.999 and L 2 weight decay of 0.00001. A dropout value of 0.6 is used in the FCN that predicts emotions.

These hyper-parameters were optimized using the Ray Tune toolkit [START_REF] Liaw | Tune: A Research Platform for Distributed Model Selection and Training[END_REF] on validation data extracted from AMIGOS, for each fold of cross-validation.

Multimodal Emotion Recognizer

The multimodal emotion recognition model follows the architecture described in Section 3.4. The FCN has two hidden layers of size 64 and 32, and an output layer that projects the result to a single value used to predict the binary emotion. As we did for the single-modality models, we train one model to predict arousal and another to predict valence. The activation function used is ReLU. This network is trained for 52 epochs, starting with a learning rate of 0.00001 and decaying it every 20 epochs with a factor of 0.65. A dropout value of 0.1 is used during the training of this network. We employ Adam optimization with β 1 = 0.9, β 2 = 0.999 and L 2 weight decay of 0.00001. We also use the Ray Tune toolkit to optimize these hyper-parameters, as we did for the EEG emotion recognizer.

Results

In this section, we discuss the results of the different experiments we performed to evaluate our model for emotion recognition on the AMIGOS dataset. We use the mean accuracy and F1 score between the two predicted classes as metrics, averaged across 10 folds of cross-validation. We also report a twosided 95% confidence interval, calculated with a t-distribution with 9 degrees of freedom.

EEG single-modality Emotion Recognition

We first report in Table 1 the performances of our singlemodality emotion recognizer on EEG signals, using the pretraining strategy described in section 3.2.1, and without using pre-training. We see that the pre-training strategy improves emotion recognition performances for all metrics, for both arousal and valence. For example, the F1 score for valence goes from 0.7±6.8e -3 when not using pre-training, to

Valence F1

ECG [START_REF] Vazquez-Rodriguez | Transformer-Based Self-Supervised Learning for Emotion Recognition[END_REF]0.88±5.4e -3 0.87±5.4e -3 0.83±7.8e -3 0.83±7.4e -3 EEG 0.81±10.7e -3 0.80±9.4e -3 0.77±9.3e -3 0.77±9.1e -3 Fused 0.89±5.0e -3 0.89±5.0e -3 0.85±3.8e -3 0.85±3.9e -3 0.77±9.1e -3 when pre-training is used. This confirms that the pre-training strategy we employ is useful when processing EEG signals, and helps the Transformer learn better representations that in turn produce better results when predicting emotion. The comparisons of all metrics shown in Table 1 have a two-tailed P value less than 1e -5 , thus the difference between them can be considered to be statistically significant.

Fused Model Results

In Table 2 we present the performances of our fused model, along with the performances of the single-modality models. Pre-training was used in the single-modality models, and the fused model employs those single-modality models as part of the fusion strategy, as described in Section 3.4. We see that our late-fusion approach improves performance over singlemodality models. For example, we obtain a valence accuracy in the fused model of 0.85±3.8e -3 compared to 0.83±7.8e -3 and 0.77±9.3e -3 when using only EEC and EEG signals respectively. Moreover, when comparing the results of the single-modality models with the fused strategy, the two-tailed P values are less than 1e -3 , thus the difference is statistically significant.

Figures 3 and4 show two different samples used in our experiments. Figure 3 For the sample of Figure 3, the ECG model predicts the wrong class, while the EEG model predicts the correct class. We can see that the fused model is helpful when the ECG modality makes a wrong prediction, relying on the information from the EEG signal to do the classification correctly. In Figure 4, we present the same channels as in the previous example, but for a sample that is classified correctly by the ECG model and misclassified by the EEG model. Now the fused model is capable of relying on the information from the ECG signal to classify correctly this sample. Therefore, our fusion model is capable of paying attention to the right modality when one is informative and the other leads to incorrect predictions. Looking at the signals for these 2 samples, it is not obvious why misclassifications occur for one modality or

Valence F1

No 0.86±4.9e -3 0.85±5.1e -3 0.82±6.5e -3 0.81±6.8e -3 Yes 0.89±5.0e -3 0.89±5.0e -3 0.85±3.8e -3 0.85±3.9e -3 the other, compared to other signals. This showcases that our model is capable of extracting meaningful hidden features in both modalities.

Effectiveness of Pre-training in the Fused Model

In Table 3, we compare the performances of our fused model, depending on whether it uses pre-trained single-modality models, or single-modality models with no pre-training. We see that the pre-trained fused model achieves superior performance compared to the fused model with no pre-training. For example, the arousal F1 score improves from 0.85±5.1e -3 to 0.89±5.0e -3 . The results shown in Table 2 have two-tailed P values less than 1e -4 , thus the difference between them is extremely statistically significant. These results indicate that the benefits obtained from pre-training single-modality models are carried over when combining them with our late-fusion strategy. Fused: EEG+ECG+GRS [START_REF] Miranda Correa | AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups[END_REF] 0.564 0.560 Fused: EEG+ECG (ours) 0.89±5.0e -3 0.85±3.9e -3

Comparison with some baselines on AMIGOS dataset

We report in Table 4 the performance of our models and the results reported by the authors of the AMIGOS dataset [START_REF] Miranda Correa | AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups[END_REF], that we consider as a baseline. We should note that the experimental protocol used in the baseline is different than our protocol, for example the length of their segments is 20s, thus their and our results are not directly comparable. In any case, we present those results to showcase results obtained by other works, and to see if our approach has acceptable performance. We see that our performances are much higher than the baseline, both using only EEG and also in the fused approach. The fused approach in the baseline uses Galvanic Skin Response (GSR) in addition to EEG and ECG. Our approach is performing at much higher F1 scores with one less modality, which further validates that our approach is promising.

Conclusion and Perspectives

In this work, we presented a new Transformer-based architecture with pre-training for emotion recognition on multimodal physiological signals. We experimentally showed, using the AMIGOS dataset, that our approach can predict valence and arousal with significant accuracy. In addition, we demonstrated that our late-fusion multimodal approach improves performances over single-modality. Finally, we compared the benefits of our pre-training strategy for multimodal situations. Overall, our architecture is capable of reaching state-of-the-art performance for emotion recognition.

Future works include investigating new ways to perform pretraining in multimodal situations: instead of pre-training individual modalities in independent Transformers, we can expect that using a single multimodal Transformer could lead to better performances. Indeed, this way, the model could start to incorporate information from different modalities as early as in the pre-training phase. In general, new ways of combining different modalities in a pre-trainable Transformer architecture, be it early-fusion, middle-fusion, or late-fusion, are valuable avenues of research to improve emotion recognition performances of such models. Moreover, the combination of physiological signals with more traditional modalities such as images and audio may help to better understand how pre-training and Transformer-based models behave for multimodal emotion recognition.

  Raw Physiological Signal (Values masked during pre-training)

Figure 1 :

 1 Figure 1: Single-Modality Emotion Recognizer: The raw signal is encoded by a 1D-CNN and processed with a Transformer. First, the model is pre-trained by masking some values of the unlabeled input signal and then predicting those masked values (Part A). Then, labeled data is used to fine-tune the model in a supervised way (Part B).

  Figures3 and 4show two different samples used in our experiments. Figure3(a) and Figure 4(a) show the ECG signals, and Figure 3(b) and 4(b) show channels F7 and F3 of the EEG signal.For the sample of Figure3, the ECG model predicts the wrong class, while the EEG model predicts the correct class. We can see that the fused model is helpful when the ECG modality makes a wrong prediction, relying on the information from the EEG signal to do the classification correctly. In Figure4, we present the same channels as in the previous example, but for a sample that is classified correctly by the ECG model and misclassified by the EEG model. Now the fused model is capable of relying on the information from the ECG signal to classify correctly this sample. Therefore, our fusion model is capable of paying attention to the right modality when one is informative and the other leads to incorrect predictions. Looking at the signals for these 2 samples, it is not obvious why misclassifications occur for one modality or

Figure 3 :

 3 Figure 3: Sample correctly classified by the Fused Model and the EEG model, but incorrectly classified by the ECG model.Figure (a) shows the ECG signal, and Figure (b) shows channels F7 and F3 of the EEG signal.

Figure 4 :

 4 Figure 4: Sample correctly classified by the Fused Model and the ECG model, but incorrectly classified by the EEG model. The ECG signal is shown in Figure (a) and channels F7 and F3 of the EEG signal are shown in Figure (b).

Table 1 :

 1 Emotion recognition performances for the EEG model.

	Pre-	Arousal	Arousal	Valence	Valence
	train	Acc.	F1	Acc.	F1
	No	0.76±7.3e -3 0.75±8.3e -3 0.7±6.4e -3 0.7±6.8e -3
	Yes	0.81±10.7e -3 0.80±9.4e -3 0.77±9.3e -3 0.77±9.1e -3

Table 2 :

 2 Emotion recognition performances of single-modality models and of the fused model.

	Arousal	Arousal	Valence
	Acc.	F1	Acc.

Table 3 :

 3 Fused Model: Pre-Training vs No Pre-Training

	Pre-	Arousal	Arousal	Valence
	train	Acc.	F1	Acc.

Table 4 :

 4 Comparison with the AMIGOS Dataset Baseline

		Arousal F1 Valence F1
	EEG [6]	0.577	0.564
	EEG (ours)	0.80±9.4e -3 0.77±9.1e -3

Acknowledgements: This work has been partially supported by the MIAI Multidisciplinary AI Institute at the Univ. Grenoble Alpes: (MIAI@Grenoble Alpes -ANR-19-P3IA-0003).