Aurélien Duret

Meng Wang
email: m.wang@tudelft.nl

Ludovic Leclercq
email: ludovic.leclercq@ifsttar.fr

Aurelien Duret
email: aurelien.duret@ifsttar.fr

Duret

LECLERCQ Wang

Truck platooning strategy

HAL is

INTRODUCTION

Vehicle automation is seen as one of the promising solutions to today's traffic problems since it will revolutionize the way we operate our vehicles today [START_REF] Shladover | Automated vehicles for highway operations (automated highway systems)[END_REF]. However, it is also acknowledged that individual automation can hardly bring benefits to collective traffic systems, whiles their connected automated vehicles (CAV) pose more potential in improving traffic operations [START_REF] Shladover | Cooperative adaptive cruise control[END_REF].

Vehicle platooning is a highway CAV application characterized by a string of vehicles keeping short spacing with Vehicle-to-Vehicle (V2V) communication [START_REF] Nowakowski | Operational concepts for truck maneuvers with cooperative adaptive cruise control[END_REF][START_REF] Halle | A decentralized approach to collaborative driving coordination[END_REF][START_REF] Wang | Cooperative car-following control: Distributed algorithm and impact on moving jam features[END_REF]. Vehicle platooning is not a new concept. In the U.S., PATH started the research on automated platoons more than two decades ago, with a public demonstration of vehicle platooning on a special track in San Diego in 1997 [START_REF] Rajamani | Demonstration of integrated longitudinal and lateral control for the operation of automated vehicles in platoons[END_REF]. From there onward, PATH continued this line of research and recently experimented a string Cooperative Adaptive Cruise Control vehicles on public highway in 2013 [START_REF] Milanés | Cooperative adaptive cruise control in real traffic situations[END_REF].

As a subclass of vehicle platooning problem, truck platooning has received much attention recently. Technically, truck platooning is more challenging compared to car platooning due to the complex truck dynamics. From implementation perspective, truck platooning is more likely to happen before platooning of individual cars due to the pronounced benefits of fuel saving and more mature business models [START_REF] Nowakowski | Operational concepts for truck maneuvers with cooperative adaptive cruise control[END_REF]. In Europe, several projects and initiatives were conducted that aimed at putting truck/vehicle platooning on pubic roads, including the Safe Road Trains for the Environment (SARTRE) project [START_REF] Bergenhem | Challenges of platooning on public motorways[END_REF], the COMPANION project [START_REF] Alam | Heavy-duty vehicle platooning for sustainable freight transportation: A cooperative method to enhance safety and efficiency[END_REF], the Grand Cooperative Driving Challenge (GCDC) [START_REF] Van Nunen | Cooperative competition for future mobility[END_REF], and the Truck Platooning Challenge [START_REF]European truck platooning challenge 2016: Lessons learnt[END_REF]. Truck platooning initiatives have been reported in the U.S. and Japan [START_REF] Tsugawa | A review of truck platooning projects for energy savings[END_REF].

One of the lessons learned from the existing truck platooning projects is problems that large platoons may bring at freeway entrances and exits [START_REF] Nowakowski | Operational concepts for truck maneuvers with cooperative adaptive cruise control[END_REF][START_REF]European truck platooning challenge 2016: Lessons learnt[END_REF]. How to handle vehicles merging while long platoons are passing the merging area is an intriguing question for both researchers and practitioners. This work deals with the design of truck platooning strategy at merges to facilitate smooth merging.

Relevant work in literature

There are many technical challenges with truck/vehicle platooning, including environment perception, reliable communication, platooning coordination and vehicle control [START_REF] Shladover | Automated vehicles for highway operations (automated highway systems)[END_REF][START_REF] Varaiya | Smart cars on smart roads: problems of control[END_REF]. In view of platooning coordination and control, several studies discussed the system control architecture involving automated vehicle platoons, most of which pertain to a hierarchical setting [START_REF] Varaiya | Smart cars on smart roads: problems of control[END_REF][START_REF] Halle | A decentralized approach to collaborative driving coordination[END_REF]. Often a coordination layer is placed between the traffic control layer and the vehicle control layer, dealing with trajectory planning and coordination among platoons [START_REF] Varaiya | Smart cars on smart roads: problems of control[END_REF][START_REF] Halle | A decentralized approach to collaborative driving coordination[END_REF].

Literature has been focusing on vehicle control layer design for platoons. Different control approaches have been used to design vehicle longitudinal control systems. String stability and collision avoidance have attracted considerable attention, addressing the problems of how the controller can attenuate disturbance propagation along the platoon [START_REF] Ploeg | L p string stability of cascaded systems: Application to vehicle platooning[END_REF] and how to combine vehicle following with collision avoidance mechanism [START_REF] Wang | Cooperative car-following control: Distributed algorithm and impact on moving jam features[END_REF].

A few researchers have designed maneuver strategies at the coordination layer. A set of protocols for highway maneuvers of platoons was proposed in [START_REF] Hsu | Protocol design for an automated highway system[END_REF], including platoon merge, platoon split, and lane change. The design of the protocol is based on a finite state description of the platoon maneuvers, giving the commands from the upper level traffic control layer. While

3

DURET, WANG, LECLERCQ acknowledging the pioneering role of this work, the design was not aimed at generating decisions of when and where to split a platoon at highway entrance and was based on fully automated vehicle environment.

In [START_REF] Godbole | Design of entry and exit maneuvers for ivhs[END_REF], entry and exit maneuvers of platoons on highway were discussed. Strategies were proposed that were applicable in highways with dedicated automated vehicle lanes and transitional lanes near entry and exit. The design was based on the assumption that the merging vehicle is a CAV and it required changes of infrastructure, e.g. a parallel transition lane or dedicated ramps, raising concerns over its applicability in reality.

In [START_REF] Halle | A decentralized approach to collaborative driving coordination[END_REF], two basic platoon maneuver strategies, merge and split, were proposed to facilitate another automated vehicle joining an existing platoon and a platoon member leaving the platoon respectively. The split strategy was further elaborated by sub-tasks of initiating split request, creating safe gap and changing lane. However, when is optimal to start the gap creation process and how the transient maneuver looks like were not formulated.

More recently, a finite state machine approach was also reported in [17] for platoon maneuver protocols. The protocols were combined with CACC control logic was used to represent longitudinal behavior. An interesting approach was presented in [18], where a spring-massdamper analogy was adopted to describe platooning dynamics. This modeling approach allows derivation of efficient platooning strategies at entry and exit by controlling the spring constant and damping coefficient. It can capture different scenarios where an automated vehicle join or leave platoons.

Notice that another body of literature focused on platoon formation strategies [19,[START_REF] Saeednia | Analysis of strategies for truck platooning[END_REF].

We restrict the discussion on platoon formation because this paper concentrates on how to split a platoon rather than forming a platoon.

Literature shows that quite some effort in defining platoon maneuver protocols at highway entry and exit. These studies focus on dynamics and interactions between platoons or between a platoon and an individual CAV, which implies communication between interacting platoons/vehicles. Some even requires additional changes in the infrastructure, which may impede the near-term application of the strategies. In addition, there is a gap between finite state description of the platoon maneuvers and the detailed operational truck platooning strategies for the transition between states. In case of truck platooning, it is likely that the platoon has to be detached to facilitate merging vehicles at on-ramp sections. A decision-making strategy to support when and where to split the truck platoon is yet to be explored.

To this end, the paper proposes an original approach to split platoon approaching merging areas. The split strategy aims to provide safe and comfortable merging maneuvers while restricting the disturbance for the platoon. The problem is posed, based on reasonable physical and behavioral assumptions about string-stable vehicle following dynamics. A heuristic-based solution and optimization-based solution are proposed based on simplified car-following model for platoons, which enables us deriving the vehicle index and time to yield a gap analytically.

The performance of the strategies is verified with simulation.

In the remainder of the paper, we first present the vehicle-following model for platoons, followed by the formulation of the decision-making strategy and its optimality verification. After that, we apply the strategy to derive some insights into the relations between model parameters.

We summarize the findings and future research in the conclusion section.

DURET, WANG, LECLERCQ

MODEL FOR PLATOON DYNAMICS NEAR MERGE Vehicle-following model

Vehicle trajectories are described by a Vehicle-Following model, also referred to as Car-Following (CF) model. In the paper, we assume the design of longitudinal vehicle control algorithm guarantees string stability of the truck platoon and the Newell model is considered [START_REF] Newell | A simplified car-following theory: a lower order model[END_REF] for the CF model. It consists in a piece-wise linear speed-headway relationship, which is consistent with the macroscopic kinematic wave model originaly proposed in [START_REF] Lighthill | On kinematic waves. ii. a theory of traffic flow on long crowded roads[END_REF][START_REF] Richards | Shock waves on the highway[END_REF]. Several arguments are in favor of this model for the study : simple but not simplistic representation of first-order platoon dynamics; limited number (three) and observability of model parameters; ability to describe car-following steady states; numerous prior works on vehicle trajectory-based calibration and validation. In the paper, the model is supposed to describe the platoons steady states, i.e. when the platoon is formed with no disturbance to the trucks time-gap policy. It assumes that under normal car-following situations, the speed of the following truck is adjusted with respect the distance to its leader. The speed-headway relationship is defined based on three parameters: u the free-flow speed, w the maximum wave speed and κ.

v t+1 i = min u, (s t i - 1 κ ) • w • κ (1)
Notice that the minimum time headway h is determined as follows :

h = 1 κ . w + u w • u (2)
Also notice that from a physical perspective :

κ is directly related to the minimum spacing at stop s x = 1/κ (including the truck length).

-r=1/(w • κ) is the reaction time, i.e. a speed variation of the leader will impact the follower trajectory with a time shift r.

The car-following behavior can be completed with two additionnal models. First, an acceleraton model which bounds the acceleration with a maximum value a x [START_REF] Leclercq | Bounded acceleration close to fixed and moving bottlenecks[END_REF]. Second, a relaxation model [START_REF] Leclercq | Relaxation phenomenon after lane changing: Experimental validation with ngsim data set[END_REF][START_REF] Laval | Microscopic modeling of the relaxation phenomenon using a macroscopic lane-changing model[END_REF] which describes the speed difference ε that a follower is willing to accept when its spacing is lower that its equilibrium value. The complete CF model can adress most of the car-following situations with a minimal set of five parameters.

Parameters for platooning under constant-time-gap policy

All the five parameters have a physically meaning and default values can easily be proposed.

The literature provides many references to refine their calibration, mostly based on empirical trajectories [START_REF] Duret | Estimating individual speed-spacing relationship and assessing ability of newell's car-following model to reproduce trajectories[END_REF][START_REF] Duret | Passing rates to measure relaxation and impact of lanechanging in congestion[END_REF]. To the best of authors knowledge, the proposed CF-model has never been calibrated for truck platooning operations. Here we propose basic formulas to calibrate the carfollowing parameters according to the platooning strategy. Classically, a Truck-Platooning (TP) strategies consider a target gap g, which can be expressed in time (g t ) (e.g. the constant time gap policy) or in space (g s ) (e.g. the constant spacing policy). The gap corresponds to the time or space difference between the front bumper of the following current truck and the rear bumber of its leader. In the remaining we consider the time gap, but identical results can be drawn by considering the space gap instead. The speed-spacing function of the CF model has to be calibrated with respect to this gap. In the case of the Newell CF model, only 3 parameters are concerned: u; w and κ. However, u is classically related to the maximum authorized speed on the section, and κ is constrained by the length of the vehicle L. Consequently, the maximum wave speed w is the parameter to be adjusted according to the TP strategy.

Considering the length of a truck L and the minimum time gap g t , the minimum time headway during truck platooning operation writes:

h p = g t + L/u (3)
the corresponding space headway is s p = u • g t + L. Thus, the maximum wave speed adjusted for TP strategy becomes :

w p = u u • κ • (g t + L/u) -1 (4)
Notice that this equation holds if

u • κ • (g t + L/u) > 1.
Consequently the minimum gap g has to respect the following constraint:

g t > 1 u • 1 κ -L (5)
This condition imposes a minimum gap between trucks according to their car-following properties.

Remark It should be noted that under platooning operation, the reaction time

r p = 1/(w p • κ)
of truck is (or should be) highly decreased by the raising of its maximum wave speed.

DURET, WANG, LECLERCQ

Platoon dynamics due to merge

When a vehicle is about to merge from the on-ramp, a truck will decrease its speed to open a gap for the merging vehicle. We propose to process the speed of the complying truck considering relaxation. Thus, the complying truck will accept a speed lower than its leader, such that the gap will progressively increase. It provides the advantage of simplicity, it accounts for the speed of the leading truck and it rely on a two physical parameters:

• ε, the speed difference with the predecessor during the relaxation period

• a x , the maximum acceleration rate at the end of the relaxation period.

. The CF model is modified accordingly and the different model formulations from [START_REF] Leclercq | Bounded acceleration close to fixed and moving bottlenecks[END_REF][START_REF] Leclercq | Relaxation phenomenon after lane changing: Experimental validation with ngsim data set[END_REF][START_REF] Laval | Microscopic modeling of the relaxation phenomenon using a macroscopic lane-changing model[END_REF][START_REF] Duret | Passing rates to measure relaxation and impact of lanechanging in congestion[END_REF] can be combined as follow:

v t+1 i = min v t+1 i,dem , v t+1 i,sup (6)
where

         v t+1 i,dem = max(u, v t i + a x • dt) v t+1 i,sup =    v t i-1 -ε if relaxation is active (s t i - 1 κ ) • w • κ if not (7)
Note that the time step must be set as : δt = 1/w • κ. The process is illustrated in Figure 2. On the left side, the platoon is composed of 10 trucks crosses the stretch of road at the free flow speed u = 20m/s with a constant time gap g p = 0.5 s between every trucks. On the right side on the figure, the fifth truck (I n = 5) decreases its speed between t = 30s and t = 60s, with a speed difference ε = 1m/s. Then the truck re-accelerate at the rate a x to retrieve the free-flow speed. During this period, a gap is formed and the final spacing headway between the forth and fifth trucks reaches s = 60m.

At this stage, some issues are raised. How to choose the truck that must comply by decreasing its speed? How to estimate the starting time of the complying maneuver? These problems are posed and analytical (heuristic) solutions are proposed in the next section.

DESIGN OF PLATOON SPLIT STRATEGY

Without anticipation of merging maneuvers, the presence of platoons involves safety issues and traffic disturbances. In Figure 2, a platoon of 10 trucks crosses a road stretch (3km). A vehicle merges at the time-position t = 60s, x = 1000m (trajectory not represented for a better readability). On the left picture, the platoon does not anticipate the merging maneuver: the vehicle merges in a short space, which raises comfort and safety issues, and immediate following truck behind the merging vehicle may over-react and trigger disturbance for the surrounding traffic.

On the right picture, the platoon anticipates the merging maneuver : the seventh truck decreases its speed far upstream the merge, so that a gap is created. The vehicle merges in comfortable and safety conditions, and the disturbance for surrounding traffic on the carriageway is smoothed.

The main objective of the platoon strategy is to determine the truck index in the platoon that should yield a gap for the merging vehicle, denoted by i yield and the time instant that the gap creation process starts, denoted by t yield in real time, so that merging vehicles can merge smoothly onto the main carriageway. Thus the decision variables are: d = (i yield , T a ).

Operational assumptions

The design of the platooning strategy at merges is based on several assumptions. In particular, we assume that:

• trucks in the platoon follow their leader with a constant time gap g t , corresponding to a time headway h p = g t + L/u;

• conventional vehicles from on-ramp merge at the same speed of the platoon, denoted u;

• the truck platoon leader has the knowledge of the time t merge that the merging vehicles start to move to the main carriageway from the acceleration lane. This can be assisted by a roadside system that detects the position and speed of the merging vehicle;

• the truck platoon leader decides which truck in the platoon should open a gap to ac-8 DURET, WANG, LECLERCQ commodate the merging vehicles and when it should do so. Trucks in the platoon follow the decisions of the truck platoon leader;

• the acceleration for trucks is bounded, denoted a x ;

• the merging is performed in a relatively smooth way, e.g. no emergency braking will be needed for the trucks and merging cars. The merging process model is described in the previous section.

Heuristics-based strategy

The problem is posed in Figure 3, and a heuristic solution is proposed to estimation the estimating the yielding truck index i yield , the anticipation time T a and the speed difference ε. Recipes for each variable to estimate are presented in paragraphs below.

Solution for i yield

We start with the heuristic method that based on the first-in-first-out (FIFO) principle, i.e. for all vehicles involved in the merging process, the passing sequence at the merging point is determined by the predicted arrival time t arrive i at the fixed merging point X m . The arrival time is determined by the constant speed heuristics, i.e. vehicles will maintain their current speed in the near future, thus we have:

t arrive i (t) = X m -x i (t) v i (t) (8)
Thus we can determine the yielding truck index (increasing in the upstream direction) as:

i yield = i ∈ t arrive i-1 (t) < T m < t arrive i (t) (9)
Since the truck index in the platoon is an ordered set, the uniqueness of the solution ( 9) is guaranteed.

Solution for T a

After determining the yielding vehicle index, the time that the truck starts to yield must be determined. Some empirical evidence shows that it takes around 20 to 40 seconds for a truck to split from a platoon [START_REF] Bergenhem | Challenges of platooning on public motorways[END_REF]. This is consistent with the duration of relaxation process after merge [START_REF] Duret | Passing rates to measure relaxation and impact of lanechanging in congestion[END_REF].

Thus we choose to command the designated truck starting to yield gap at the time T yield before the merging vehicle moves onto the mainline:

T yield = T m -T a ( 10
)
where T a is the anticipation time before the merging maneuvers. The objective is to formulate anticipation time with respect the to decision variables, i.e. the time that a truck needs to decrease

their speed to open a gap [T yield ,t 1 ], and then to accelerate to come back to its initial speed [t 1 , T m ]. Recipe for t 1 During the acceleration period, the following truck accelerates from the speed uε to reach the free-flow speed u. Assuming a constant acceleration rate a x , it gives:

t 1 = T m - ε a x (11)
Recipe for T yield From Figure 3 u

• (g t + T a ) = (u -ε) • (t 1 -T yield ) + T m t 1 T m t 1 a x • dt + S c (12
T a = S c -h p • u ε + ε 2 • a x (13)
Solution for ε

In the previous section, the optimal anticipation time is the decision variable to set, assuming a given ε. However, in reality, the platoon maybe be informed of the merging time from a road side unit, where the only decision variable is the speed difference ε. Consequently, the problem is re-formulated to find the optimal ε for a given anticipation time T a = T m -T yield . To solve this problem, Equation 13gives:

ε 2 • a x -T a + S c -h p • u ε = 0 ⇔ ε 2 2 • a x -T a • ε +(S c -h p • u) = 0 (14)
A positive solution exists for Equation14 if (and only if)

T a > 2•(S c -h p •u) a x
. The solution is:

ε + = a x • T a • 1 -1 - 2 • (S c -h p • u) T 2 a • a x (15)
Equation 15 is easy to implement, but the importance of each parameter is not explicit. We propose to derive a second order Taylor expansion to bring out high (first order) and low (second order) importance parameter.

ε = S c -h p • u T a + 1 2 • (S c -h p • u) 2 T 3 a • a x = S c -h p • u T a 1 + 1 2 • S c -h p • u T 2 a • a x (16)
Equation 16 demonstrate that the anticipation time T a plays a major role while the acceleration rate a x has a minor impact on ε.

It should be noted here that S is a critical spacing composed of S c m (v), critical space headways for the merging vehicle, and S c f (v), critical spacing for the following truck. The solution can be easily adapted for multiple merging maneuvers, when a platoon of N vehicles needs to merge on the carriageway. In this case, The solution basically considers the critical spacing for a platoon

of N vehicles S c (v, N) = N • S c (v).
Then the critical spacing S c becomes:

S c = S c n (v, N) + S c f (v).

Equivalence to an optimization-based strategy

In this section, we show that the heuristic-based solution is equivalent to an optimization-based strategy. Recall the general objective of the platoon split strategy: opening a sufficient gap within a closely-spaced truck platoon to facilitate comfortable merging of conventional vehicles from 11 DURET, WANG, LECLERCQ on-ramp. To this end, we formulate the problem as the following mathematical programme, with objective function:

max i yield ,t yield N ∑ j=1 q T m j ≡ max i yield ,t yield N ∑ j=1 v T m j s T m j + L ( 17
)
where N is the total number of trucks in the platoon and q i (T m ) is the local flux for vehicle i.

The optimal decisions are subject to some physical and behavioral constraints, including:

• The car-following dynamics in normal conditions, for truck platoons and during merging, see Section 2.

• Safe merge: at the time of merge, the yielding truck should keep a spacing with respect to the merging vehicle no less than the normal vehicle-following spacing:

s T m i yield ≥ s t (18)
• Bounds on speed: the vehicle speeds are bounded by the zero speed and free speed u;

0 ≤ v i (t) ≤ u (19)
• Limited acceleration capability;

vi (t) ≤ a x (20)
The formulation entails that the optimal maneuver strategy aims at maximizing the flux at the time of merging while respecting physical and behavioral constraints.

Notice that the objective function can be rewritten as:

N ∑ j=1 v T m i s T m i + L = i yield ∑ j=1 1 g t + L/u + v T m i yield s T m i yield + L + N ∑ j=i yield 1 g t + L/v j (21)
The first term in the right hand side (RHS) of Equation 21is at its maximum flux since v j ≤ u and we would like as many truck passing the merging vehicle as possible. But the safety constraint restricts the number to the first truck in the platoon that would arrive at the merging point later than the merging vehicle at the maximum speed u. This gives the same solution for i yield as in Equation 9. The third term in the RHS of 21 is a decreasing function of v j and achieves the maximum when v j = u.

The middle term in the RHS of 21 depends on the yielding time t yield . Let tyield = T yield + δ , where T yield is determined by the heuristic method. If δ > 0, using the same derivation process as in the heuristic-based method, the spacing of the yielding truck at the time of merge will become s yield = s f + u • δ , while the speed of the yielding truck and those behind the yielding truck at the time of merge is the same as u. Thus the third term of the objective function achieves its maximum but the middle term is smaller than that from the heuristic solution: e.g.

u i s f +uδ +L < u s f +L .
12 DURET, WANG, LECLERCQ TRUCK PLATOONING STRATEGY NEAR MERGE When δ < 0, we can only achieve s yield = s f when the acceleration adopted by the yielding truck i yield is less than a x . This implies the speed of the yielding truck at the time of merge is lower than free speed: v T m i yield < u, and the second and third term becomes:

v T m i yield s f +L + ∑ N j=i yield 1 g t +L/v i yield < u s f +L + ∑ N j=i yield 1 g t +L/u ,
which is smaller than that with the heuristic solution. Consequently, we can conclude that the heuristics-based solution is the optimal solution that maximize the flux at the time of merging conditioned at providing sufficient gap for the conventional vehicle to merge.

APPLICATION

Illustrative example

Let consider the following situation:

• a platoon is composed of 10 trucks, moving with a speed u = 20m/s with a time gap policy is g t = 1s.

• a merging maneuver has the following parameters : V m = u = 20m/s and the merging time T m = 60s.

• in case of merging, the yielding truck opens the gap with speed difference of ε = 2m/s and then with acceleration at a x = 1m/s 2 .

Figure 4a illustrates vehicle trajectories with no anticipation (left) and with the optimal anticipation maneuver (right). According to the heuristic-based strategy proposed in the paper, the optimal solution is : i yield = 8, T a = 48s. On the right picture, the eighth truck in the platoon starts to decrease its speed at the time t = T yield = 12s from 20m/s to 18m/s. It keeps this speed to open the gap until the time t 1 = 58s, and it accelerates at the rate a x = 1m/s 2 to return to its free-flow speed. The variation of the space headway with respect to time is illustrated in Figure 4c, where the space headway is gradually increased from h = 40m to h = 125m during the yielding maneuver. Finally, at the time t=60s, the space headway has reached h = 125m, i.e. 105m excluding the truck length.

Decision support chart

The heuristic-based solution can be the basis for designing decision support diagrams. Let assume that a platoon of trucks received the following merging information from a road site unit : "a vehicle will merge in 50s from the next on-ramp". With the help of Figure 5(a), the truck will start its maneuvers immediately with a speed difference ε = 2m/s. The decision can be adjusted according to the number of vehicle to merge (See Figure 5) and the maximum acceleration rate a x of the truck.

Impact on traffic

It should be noted that the anticipation maneuver smooths the impact of merging maneuvers inside platoon of truck. It should also be noted that the anticipation limits the impact on traffic flow. Indeed, total travel time or total delay for trucks does not depend on the transitional yielding parameters, but it only depends on the number of vehicle(s) that merge, which causes a short delay to the trucks that follows the merging vehicle(s).

On the contrary, if the merging maneuvers are not anticipated by platoons, one can expect [START_REF] Tsugawa | A review of truck platooning projects for energy savings[END_REF] DURET, WANG, LECLERCQ

CONCLUSION Main contribution

The paper proposes an original solution to split platoon of trucks approaching merging areas.

The splitting aims to provide a safe and comfortable merging maneuvers while restricting the disturbance on the carriageway. The problem is posed, based on reasonable physical and behavioral assumptions. A heuristic-based solution is proposed, which optimizes the flux at the time of merging while supplying safe gap for the merging vehicles. The proposed solution can be the basis for designing efficient and safe splitting strategies for platoons. The paper also provides decision support charts that can be the basis for scaling the anticipation time or the speed difference of trucks with respect to the number of vehicle(s) to merge.

DURET, WANG, LECLERCQ

Figure 1 :

 1 Figure 1: Fundamental diagram

Figure 2 :

 2 Figure 2: Platoon trajectories without splitting (left) and with splitting (right)

Figure 3 Figure 3 :

 33 Figure3shows that two time periods have to be considered separately.

  ) 10 DURET, WANG, LECLERCQ where S c = S c m (v) + S c f (v). It follows from Equations 11 and 12 that:

Figure 4 :

 4 Figure 4: Platoon trajectories without anticipation (left) and with the heuristic-based decision (right)

Figure 5 :

 5 Figure 5: Decision support charts for adapting (a) anticipation time T a = f (ε, N) , (b) speed difference ε = f (T a , N) or (c) number of vehicle(s) to merge N = f (N, T a )

Further research

In the paper, the solution is adapted to platoons of vehicles from the on-ramp merges into a platoon of trucks on the main line. Further research is needed to extend the proposed solution to address several independent merging vehicles. Moreover, it is assumed that vehicles merge at the speed of the platoon, which could be extended for low speed merging maneuvers. And finally, the proposition can be the basis for advanced solutions, where the anticipation time can be combined with a merging control, for instance with ramp-metering strategies. The ramp metering algorithm would control the rate and the speed of the merging maneuvers, and would provide more flexibility framework to improve global safety and comfort indicators, for both trucks and light vehicles. [START_REF] Godbole | Design of entry and exit maneuvers for ivhs[END_REF] DURET, WANG, LECLERCQ


Concetta Fazio
email: concetta.fazio@ec.europa.eu

Karl-Fredrik Nilsson

Dario Manara

Arjan Plompen

Andrea Bucalossi

Stéphane Bourg

Rik-Wouter Bosch

Jean-Claude Bouchter

Walter Ambrosini

Rosa Lo Frano

Leon Cizelj

Pedro Dieguez

Infrastructure and international cooperation in research and knowledge transfer: supporting access to key infrastructures and pan-European research -lessons learned

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Pooling and integration of research infrastructures as well as their access had the objective to promote in Europe the development of networks with high quality research infrastructures and their optimum use on a European scale based on the needs expressed by the research community. The infrastructure access scheme called integrated infrastructure initiative (I-3) has the objective to promote access to infrastructure for European researchers or research teams for their research needs, irrespective of the location of the infrastructure. I-3 projects have in general three components, which are (i) networking, (ii) transnational access to infrastructure and (iii) joint research initiatives. Past I-3 projects have been conducted around specific topics as e.g. the FP6 EURATOM supported projects VELLA (Virtual European Lead Laboratory), EFNUDAT (European Facilities for Nuclear Data Measurements) and NUDAME (Neutron data measurements at IRMM) and ACTINET-I3 (Actinide Network) as well as the FP7 EURATOM project TALISMAN. These five projects as indicated by their acronyms were focused on three major topics: liquid lead technology to support the development of lead-cooled reactor systems (VELLA); nuclear data measurement for nuclear reactor physics and basic science applications (EFNUDAT and NUDAME) and actinides science (ACTINET-I3 and TALISMAN). Furthermore, the European Commission has supported along the past EURA-TOM framework programs several projects and initiatives related to Education and Training, aimed at attracting young talented students and professionals to the nuclear field. These projects and initiatives were more people centred and crosscutting different nuclear topics. The components of the Education and Training projects are the development and execution of specific classroom or (i) online courses for students, (ii) training for professional development and (iii) mobility schemes. Examples of projects with such type of scheme that are completed or ongoing are GENTLE, ANNETTE (without mobility grants) and ENEN-plus (more focussed on mobility and dissemination). Finally, mobility grants are also part of projects that are built around specific topics as for instance NUGENIA-plus and ENSII-plus. In the following chapters, an overview of the above listed projects will be given and more in particular the outcome of the mobility grants implemented in the projects TALISMAN, GENTLE and NUGENIA-plus will be discussed in terms of organisation of the access to the infrastructures and achievements. Finally, this manuscript includes also the recent initiative of the Joint Research Centre to grant access to its research infrastructures.

Transnational access to research infrastructure

As already mentioned in the introduction, the access to research infrastructure has been organised in Europe along three different schemes: integrated infrastructure initiatives; mobility grants within topical projects; mobility grants within education and training projects.

In all three cases, access to research infrastructure is granted to researcher, research teams or students. However, the Integrated Infrastructure Initiative had the aim to pool specialised infrastructures around specific topical areas (e.g. liquid metal technology, nuclear data, actinide science), while the mobility grants were more people centred and focussed on promoting mobility into different research infrastructures. Hereafter, a brief overview is given for TALISMAN, NUGENIA+ and GENTLE corresponding to the three different schemes, respectively, and where appropriate extension and examples from the other projects are included.

TALISMAN

The TALISMAN project was established as a follow-up of the previously successfully concluded Network of Excellence ACTINET-6 and Integrated Infrastructure Initiative ACTINET-I3. The importance to establish a network of competences and infrastructure for actinide science is due to the fact that, actinides of interest for nuclear energy are radioactive elements and their study requires specific tools, facilities and licences that are available only to few European academia and research organisations. Therefore, it is strategic to coordinate the European actinide infrastructures and to strengthen its scientific community in view of performing excellence research and developing excellent professionals in the field. In this context, TALISMAN had the objective to establish a network of Actinide facilities and infrastructures across the EU to structure and foster their joint development in terms of capacity and performance. TALISMAN supported Transnational Access to these facilities through the organization of periodic calls for Joint Research Projects (JRP) and conducted a set of Joint Research Activities (JRA) involving member organisations, with the objective to improve the performance of infrastructures by developing new relevant instrumentations and/or data of common interest.

TALISMAN has also promoted training and education actions through the organisation of summer schools, networking meetings for trained young scientists, attributing travel grants to students attending international conferences on actinides sciences.

The facilities pooled in TALISMAN were hot laboratories belonging to CEA, JRC, KIT, NNL, Chalmers University and HZDR; as well as beam lines belonging to KIT, PSI and HZDR (see Fig. 1).

The selection of the transnational access to be funded was organized through calls for proposals (two times per year over three years). At the end of each call, the proposals were sent to the Project Scientific Advisory Committee that received a list of ranking criteria, established by the Executive Committee of the project. These criteria were related to (i) the originality of the subject and its compatibility to the TALISMAN portfolio, (ii) the skills of the teams (both visitor and pooled facility), (iii) the relevance of the choice of the pooled facility and that all results had to be publishable.

Within the TALISMAN project, 6 calls were published and in total 107 proposals for infrastructure access were received. From the 107 proposals 96 were granted and 91 were concluded (5 proposals were cancelled due to issues encountered by the visiting teams).

The distribution of the access over the seven involved infrastructure is shown in Figures 2 and3 summarises the access over the three broad scopes defined within the TALISMAN project, i.e. scope 1 = separations, scope 2 = environmental actinide chemistry and scope 3 = irradiated materials.

The teams hosted at the pooled infrastructures through the TALISMAN grants were either researchers/scientists and/or students.

In Figure 4, the countries of origin of the different research teams asking for accessing the pooled infrastructures are reported. As shown in this figure, TALISMAN was not restricted to only European research teams but research teams from France, Germany and UK were the most numerous.

NUGENIA-PLUS

The objective of the FP7 EURATOM NUGENIA-PLUS project was to support the NUGENIA Association in its role to coordinate and integrate European research on safety of the Gen II and III nuclear installations in order to better ensure their safe long-term operation, integrating private and public efforts, and initiating international collaboration that will create benefit in its activity fields. 1 The project was a combination of Coordination and Support Action and a Collaborative Project. The Coordination and Support Action was aimed at establishing a management structure to carry out the planning and management of R&D including project calls, proposal evaluation, project follow-up dissemination and valorisation of R&D results in the area of safety of existing Gen II and future Gen III nuclear installations. The part dedicated to collaborative project was based on thematic calls for research proposals organized among the NUGE-NIA technical areas, i.e. plant safety and risk assessment, severe accident prevention and management, core and reactor performance, integrity assessment of systems, structures and components, innovative Generation III design and harmonisation of procedures and methods.

Within NUGENIA-PLUS also mobility grants were offered with the scope to allow young and senior professionals to visit selected key NUGENIA infrastructures (including experimental facilities and modelling and simulation platforms). The overall objective of this action was to enhance the relationships between European R&D facilities and NUGENIA end users. Two type of mobility grants were defined, namely, short training periods for post-doc students and researchers (typically less than 1 month) and long training visits for more experienced staff (from 1 to 3 months). As far as the rules for application were concerned, it was established that the grants were limited to members of NUGENIA-PLUS consortium in terms of hosting organisation and in terms of applicants, but exemptions from this rule were also foreseen.

Within NUGENIA a "Resource Map" which included a database of infrastructures (experimental facilities and modelling platforms) was established that allowed the applicants to select the most suitable infrastructure and related contact person for its grant application. A team established within the work package addressing the mobility grants evaluated the proposals. The criteria for evaluating the proposals were: (i) topic within the NUGENIA roadmap; (ii) training related to infrastructures; (iii) quality of application and requested funding within the budgetary framework.

During the one year of continuous call (there were no deadline for applications), 18 mobility grants have been assigned. As shown in Figure 5, the applicants were from 9 different EU countries with the majority belonging to research organisations and universities. The organisations 1 https://cordis.europa.eu/project/rcn/110017/factsheet/en hosting the grant holders were belonging to 8 different EU countries as shown in Figure 6. The geographical distribution of applicants and hosting organisation is quite interesting since one can identify a rough pattern from Central and Eastern Europe towards Western Europe. This pattern might be due to the communication effort performed for the NUGENIA grants. A further explanation could be that some infrastructures are not available in these European regions.

Among the 18 grants three were long-term visits and fifteen short term visits. The topical distribution of the grants was quite diversified, although the majorities of the topics were within the areas of (i) severe accidents and (ii) integrity assessment of system and structures. The distribution is shown in Figure 7.

The NUGENA-PLUS responsible for the grant assignment did also a critical review of the process and defined the following conclusions and recommendations: a more efficient communication of the availability of the grants and its open call without deadline would have improved the number of applications (indeed, the budget allocated to the grants were not fully exploited); the administrative part concerning the coordination and transfer of the grants can be simplified. The coordination (organisation and payment of the grants) should be with one organisation, whereas in NUGENIA it was split over two different project partners. Also, the payment can be simplified moving from real costs to lump sum; the distribution over the technical areas was not even.

Indeed, two topical areas get more interest with respect to the others, but no further assessment was done with this respect; a further recommendation that was formulated on the basis of the experience gathered during the calls for access to infrastructure was that the members of the evaluation committee should be well defined and the number of participants to this committee should be in the order of 4-5.

Within the ESNII-PLUS project a similar approach as for NUGENIA-PLUS was adopted. The first step of ESNII-PLUS was the identification of available research facilities associated with the research needs for the different reactor concepts [START_REF] Agostini | Experimental Facilities Dedicated to ESNII concept[END_REF]. A "Research Facility map" resulted from this analysis and within ESFR-SMART a mobility grant program for SFR was launched. The call for the grants were organized similar to the I-3 approach, however, results of the outcome of the grant assignment were not made available at the time of writing this manuscript since the project is still ongoing. An essential tool to achieve the training objective of GENTLE was the SREs. The SREs could last between 1 and 24 months at the participating research establishments of the GENTLE consortia and applicants could come from any European academic institutions. The SRE proposals were defined as common research between the applicant and the hosting research institution and were focused on the understanding of basic phenomena related to material behaviour or process technology, the development of analytical methods, or measurement and modelling of fundamental properties.

The selection was based on a written proposal, directly submitted by the student, which was then examined by the GENTLE SRE evaluation committee. Scientific quality, availability of equipment, staff and materials at the hosting institution, training benefit to the applicant, and impact on the field were the main selection criteria.

Within the GENTLE project particular attention was devoted to the rules that are summarised hereafter [START_REF] Manara | Deliverable D2.1 Periodic report with extended scientific summaries of SREs and list of publications[END_REF]: applicants had to fill in a dedicated form stating the main objectives of the research proposal, as well as a reasonably detailed work description, indicating a suitable host institution (beneficiary) and local supervisors for their SREs; the minimum stay of students within GENTLE SREs shall be 1 month, maximum 24 months, but can be subject to local rules at the hosting organization; candidates belonging to partner as well as non-partner

European academic or research institutions can apply; students must be enrolled in an EU academic or research institution but must not necessarily have a European citizenship;

agreement on local grant rules (€/month) of the hosting institution shall be applied. The recommended grant is on the order of 1000 €/month; the grant could not be used to extend PhD studies at the institution where the PhD is performed. Only one application plus one extension were allowed (with a maximum total duration of 24 months); during the complete SRE, the students had to be enrolled at the university; -SREs within the same town/region were allowed, but will not be supported financially.

Moreover it was established that all members of the GENTLE consortium could recruit students within SRE projects approved by the evaluation committee and the costs had to be claimed by the beneficiaries (either host or sending institution, provided the latter is also a GENTLE partner) who will recruit the student. A suitable administrative and financial framework for the reimbursement of SRE costs was defined by each partner separately, due to the different legal conditions to which each GENTLE participant was bound. Some of the partners had already defined such framework, while others had to define and implement it.

At the end of the GENTLE project, a final report on SRE was published, where statistical analysis of this training tool was done [START_REF] Manara | Deliverable D2.3 Periodic report with extended scientific summaries of SREs for[END_REF]. What follows is a summary of this analysis.

A total of 84 SREs were granted during the GENTLE project duration (2013-2016), corresponding to 10-20 SREs per year (depending on the single SRE duration). Forty-seven SRE applications were received for 2016, while during the two years 2014 and 2015 in total thirty seven applications were received. This, more than double number of applications for 2016, was the result of important efforts done to advertise GENTLE to EU students and most probably also due to a sort of "word-of-mouth chain reaction", which has increased the popularity of the GENTLE SRE initiative among EU students in nuclearrelated subjects.

In the next figures, statistics about the accepted SRE projects over the whole duration of the project (2013-2016) are shown. Figure 8 shows the origin of the academic institutions at which the students were enrolled. Sixteen EU countries and most of the main countries having nuclear education and training programs were represented and most students were from universities located in Spain, France and Italy.

Figure 9 shows the number of SREs per GENTLE beneficiary. It can be noticed here that the majority of GENTLE partners hosted SREs. It is worth pointing out that the main experimental facilities available at GENTLE partners (namely at SCK-CEN, KIT, PSI and JRC) have been largely used for SREs and JRC infrastructures hosted more SREs with respect to the other partners.

Figure 10 schematically shows a rather well-balanced distribution of the accepted SREs among various research and engineering topics. In Figure 11, one can see that more student-months were devoted to experimental work rather than computational modelling activities. This is rather understandable, considering that experimental work in nuclear-related topics often require complex facilities that are not available at universities. Therefore, students willing to perform experimental work in these fields are more easily motivated to seek external internships in research centres offering access to such facilities.

In summary, the GENTLE SRE program was highly successful and very popular. This large popularity increase of the project GENTLE over the years shows the success of the initiative as a whole, and specifically of an intense advertisement activity, including the regularly updated website, https://ec.europa.eu/jrc/en/page/gentle-projectgraduate-and-executive-nuclear-training-lifelong-education. Many students and supervisors have shared their disappointment about the fact that this project was ending, which should be encouraging about the launch of further similar international projects supporting the mobility of students.

The quantitative statistics and qualitative feedback from students and hosts paint a very positive picture of this activity: a large number (seventeen) of EU countries sending students for GENTLE SREs and a very broad spectrum of nuclear-related subjects were covered by the SREs. In conclusion, GENTLE Student Research Experiences have been an effective and highly successful tool for supporting student mobility across EU nuclearrelated facilities.

The ENEN+ project (Attract, Retain and Develop New Nuclear Talents Beyond Academic Curricula) can be considered as a follow-up of the GENTLE SRE experience. Indeed, ENEN+ proposes cost-effective actions to attract, develop and retain new talents in nuclear professions with the objective to preserve, maintain and further develop the valuable nuclear knowledge for todays and future generations. The ENEN+ project focuses on learners and careers in nuclear reactor engineering and safety, waste management and geological disposal, radiation protection and medical applications. The most notable action of the ENEN+ project is mobility funding for learners at different stages of the early career. The mobility grants are accessible through the web application and selection system (http://plus.enen.eu) to the individuals aiming at starting or improving their careers in nuclear.

Conclusion and further/future initiatives

In the above paragraphs the experience gathered during the implementation of different transnational access to infrastructure funding schemes have been summarised. As described before, over the past years there have been different approaches to grant access to the infrastructures. The approaches have been either infrastructure and topic oriented or people oriented. In all cases, successful accomplishments of the projects have been reported. The important lessons learned within the different projects are related to the definition of rules (administrative, financial and scientific) to grant access and to the advertisement of the mobility opportunity and reaching out to the European nuclear community.

Moreover, it could be relevant to elaborate a blended approach for the transnational access to infrastructures through mobility grants at pooled facilities within the different topical areas as done within ACTINET, NUGE-NIA, ESNII, VELLA etc. and also people oriented as done within GENTLE and ENEN+. Ideal would be if such type of initiative would be coordinated centrally taking care of all organisational and administrative issues in order to aim at a harmonised access scheme as well as coordinating the different topical/pooled facilities. This centralised entity could be for instance ENEN. In support to this approach, ENEN has already started to create a database of infrastructure as documented in the report [START_REF] Bouchter | Infrastructure and international cooperation in research and knowledge transfer: supporting access to key infrastructures and pan-European research À lessons learned[END_REF].

Within the European Commission there is a further initiative initiated over the last year and that concerns the access to all Joint Research Centres Infrastructures including the nuclear one, with the objective to exploit their full potential. The JRC open access has the aim to promote innovative research and development; dissemination of knowledge; improvement related methods and skills; training of researchers and technicians and collaboration at European level. More information on open access opportunity can be found at the JRC science hub link https://ec.europa.eu/jrc/en/research-facility/ open-access. In combination of this JRC initiative there will be within the Horizon 2020 framework a further action in collaboration with RTD in order to make available mobility funds to the European Community dedicated to European research teams, students and SMEs to support their access to the nuclear JRC infrastructures.

Fig. 1 .

 1 Fig. 1. Infrastructures pooled within the TALISMAN project. Courtesy S. Bourg, CEA. Details on the facilities can be gathered at the link: https://cordis.europa.eu/project/rcn/93856/factsheet/en

Fig. 5 .

 5 Fig. 5. Countries of origin of applicants. Data taken from [2].

Fig. 2 .

 2 Fig. 2. Frequency access to the infrastructures pooled within the TALISMAN project. Courtesy S. Bourg, CEA.

Fig. 4 .

 4 Fig. 4. Countries of origin of the research teams asking for access to infrastructures. Data taken from [1].

Fig. 3 .

 3 Fig. 3. Access to infrastructure distribution over the three broad TALISMAN scopes: scope 1 = separations, scope 2 = environmental actinide chemistry and scope 3 = irradiated materials. Courtesy S. Bourg, CEA.

Fig.Fig. 7 .

 7 Fig. Countries of origin of organisations. Data taken from [2].

Fig.

  Fig. Distribution of SREs over the beneficiaries.

Fig. 11 .

 11 Fig. 11. Share between experimental and modelling SREs.

C. Fazio et al.: EPJ Nuclear Sci. Technol. 6, 27 (2020)

C. Fazio et al.: EPJ Nuclear Sci. Technol.6, 27 (2020)

the 6th EURATOM Framework Programme the instrument introduced by the European Commission were the Integrated Infrastructure Initiatives (I-3). Moreover, funding schemes to support Education and Training for students and professional developments were defined also. The main difference between these two funding schemes is that I-3 are topic-driven projects with access to infrastructure components, while the Education and Training related projects have a mobility component that is applied for the different research topics. The outcome of projects as TALISMAN (I-3), EFNUDAT/NUDAME (I-3), GENTLE (mobility), ENEN-plus (mobility), NUGENIA-plus (mobility within TA of NUGENIA) and ESNIIplus (I-3 similar) will be shortly presented as well as the future European Commission plans in the field of access to research infrastructure.


Irina Illina
email: irina.illina@loria.fr

Dominique Fohr
email: dominique.fohr@loria.fr

DNN-based semantic rescoring models for speech recognition

Keywords: Automatic speech recognition, semantics, embeddings, BERT

In this work, we address the problem of improving an automatic speech recognition (ASR) system. We want to efficiently model long-term semantic relations between words and introduce this information through a semantic model. We propose neural network (NN) semantic models for rescoring the N-best hypothesis list. These models use two types of representations as part of DNN input features: static word embeddings (from word2vec) and dynamic contextual embeddings (from BERT). Semantic information is computed thanks to these representations and used in the hypothesis pair comparison mode. We perform experiments on the publicly available dataset TED-LIUM. Clean speech and speech mixed with real noise are experimented, according to our industrial project context. The proposed BERT-based rescoring approach gives a significant improvement of the word error rate (WER) over the ASR system without rescoring semantic models under all experimented conditions and with n-gram and recurrent NN language model (Long Short-Term model, LSTM).

INTRODUCTION

The performance of ASR is determined by the precision with which spoken words are modeled. Using acoustic and linguistic knowledge, an ASR system generates the best hypothesis corresponding to the recognized sentence. Our work is performed in the context of an industrial project. Due to the constraints of this project, we chose to study only the N-best list rescoring approaches to improve recognition accuracy.

State of the art ASR systems only take into account acoustic (acoustic model), lexical, and syntactic information (local n-gram language models (LM)). It may be of interest to incorporate additional knowledge into the decoding process to help ASR tackle not only clean conditions but also mismatched conditions, noisy environments, conditions specific to a particular application, etc. Some studies have attempted to include such information in an ASR. In [START_REF] Fuchun | Search results based N-best hypothesis rescoring with maximum entropy classification[END_REF], recognizer score, linguistic analysis, grammar construction, semantic discrimination score are used to rescore the N-best list. [START_REF] Jinyu | A Study on Knowledge Source Integration for Candidate Rescoring in Automatic Speech Recognition[END_REF] indicate that articulation can provide additional information in rescoring. The use of external knowledge sources such as knowledge graph is proposed in [START_REF] Kumar | Use of Knowledge Graph in Rescoring the N-best List in Automatic Speech Recognition[END_REF]. The authors proposed to utilize the DBpedia knowledge graph in form of a connected graph. An N-best rescoring based on a Statistical Language Model or Dynamic Semantic Model is designed in [START_REF] Verhasselt | N-best list rescoring in speech recognition[END_REF].

In this article, we want to introduce semantic information in the ASR system via the N-best rescoring. Previous studies have shown that this information can be useful for ASR rescoring. The integration of semantic frames and target words in the recurrent neural network LM [START_REF] Bayer | Semantic Language Models for Automatic Speech Recognition[END_REF], the use of an in-domain LM and a semantic parser [START_REF] Corona | Improving Black-box Speech Recognition using Semantic Parsing[END_REF], the introduction of the semantic grammars with ambiguous context information [START_REF] Gaspers | Semantic parsing of speech using grammars learned with weak supervision[END_REF] improve the accuracy of the transcriptions. Several techniques including subword units, adaptive softmax, and knowledge distillation with a large-scale model to train Transformer LMs are proposed in [START_REF] Huang | An Empirical Study of Efficient ASR rescoring with Transformers[END_REF]. The authors have shown that the combination of all these techniques can significantly reduce the size of the model and improve the ASR accuracy with N-best rescoring. [START_REF] Ogawa | Rescoring N-Best Speech Recognition List Based on One-on-One Hypothesis Comparison Using Encoder-Classifier Model[END_REF] introduce a deep duel model composed of an LSTM-based encoder followed by fully-connected linear layer and binary classifier. In [START_REF] Ogawa | Improved Deep Duel Model for Rescoring N-best Speech Recognition List Using Backward LSTMLM and Ensemble Encoders[END_REF], this approach is improved by employing ensemble encoders, which have powerful encoding capability. [START_REF] Shin | Effective Sentence Scoring Method Using BERT for Speech Recognition[END_REF] adapt BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Wang | BERT has a mouth, and it must speak: BERT as a Markov random field language model[END_REF] to sentence scoring, and the left and right representations are mixed with a bidirectional language model.

In our work, we aim to add long-range semantic information to ASR by reevaluating the list of ASR N-best hypotheses. This research work has been carried out in the framework of an industrial project that aimed to perform the ASR in noisy conditions (fighter aircrafts). We are interested in two types of experimental conditions: clean conditions, and the context of noisy test data. These conditions are very common in real applications. We believe that some ASR errors can be corrected by taking into account distant contextual dependencies. In noisy conditions, the acoustic information is less reliable. We hope that in noisy parts of speech, the semantic model might help to remove acoustic ambiguities. The main points of the proposed rescoring approaches are: (a) rescoring the ASR N-best list using two types of continuous semantic models applied to each hypothesis: static word-based word2vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] and dynamic sentence-based BERT; (b) using a deep NN (DNN) framework on these semantic representations; (c) comparing hypotheses two per two; (d) combining semantic information with the ASR scores of each hypothesis (acoustic and linguistic).

Compared to [START_REF] Shin | Effective Sentence Scoring Method Using BERT for Speech Recognition[END_REF], where only one sentence is taken at inference and masked word prediction is performed with BERT, we use hypothesis pairs and the sentence prediction capability of BERT. Compared to our previous work [START_REF] Level | Introduction of semantic model to help speech recognition[END_REF], we employ a more powerful model (BERT) and train a DNN network. Compared to [START_REF] Ogawa | Rescoring N-Best Speech Recognition List Based on One-on-One Hypothesis Comparison Using Encoder-Classifier Model[END_REF], we use an efficient transformer model (BERT) to compare hypotheses. Introduction For each of the hypothesized word w of the sentence to recognize, an ASR system provides an acoustic score Pac (w) and a linguistic score Plm (w). The best sentence hypothesis is the one that maximizes the likelihood of the word sequence:

𝑊 ̂= 𝑎𝑟𝑔𝑚𝑎𝑥 ℎ 𝑖 𝜖𝐻 ∏ 𝑃 𝑎𝑐 (𝑤) 𝛼 * 𝑃 𝑙𝑚 (𝑤) 𝛽 𝑤𝜖ℎ 𝑖 ( 1
)
𝑊 ̂ is the recognized sentence (the end result); w is a hypothesized word; H is the set of N-best hypotheses; hi is the i-th sentence hypothesis; α and β represent the weights of the acoustic and the language models.

To take into account the semantic information, one powerful solution can be to reevaluate (rescore) the best hypotheses of the ASR system. In [START_REF] Level | Introduction of semantic model to help speech recognition[END_REF] we proposed to introduce the semantic probability for each hypothesis Psem(h) to take into account the semantic context of the sentence. This was performed through a definition of context part and possibility zones. In this rescoring approach, Pac (h), Plm (h), and the semantic score Psem(h) are computed separately and combined using specific weights α, β and γ (for Psem(h)) for each hypothesis:

𝑊 ̂= 𝑎𝑟𝑔𝑚𝑎𝑥 ℎ 𝑖 𝜖𝐻 𝑃 𝑎𝑐 (ℎ 𝑖 ) α * 𝑃 𝑙𝑚 (ℎ 𝑖 ) β * 𝑃 𝑠𝑒𝑚 (ℎ 𝑖 ) γ (2) In the current work, we propose a DNN-based rescoring models that rescore a pair of ASR hypotheses, one at a time. We use hypothesis pairs to get a tractable size of the DNN input vectors. Each of these pairs is represented by acoustic, linguistic, and semantic information. In our current approach, semantic information is introduced using two types of semantic representations: word2vec or BERT.

DNN-based rescoring models

The main idea behind our rescoring approach is: (a) to train DNN-based rescoring models with input features extracted from the ASR N-best list of training data; (b) to apply these models to each hypothesis pair of N-best list of a sentence to be recognized and recompute the hypothesis scores; (c) to select as the recognized sentence the hypothesis with the best recomputed score.

As mentioned before, our DNN-based rescoring models rescore pairs of ASR hypotheses. For each pair of hypotheses (hi, hj), the expected DNN output is: 1, if WER of hi is lower than WER of hj; otherwise, 0.

The global algorithm of the N-best list rescoring is as follows. From the N-best list of a sentence to recognize, for each hypothesis hi we want to compute the cumulated score scoresem(hi). To perform this, for each hypothesis pair (hi, hj) in the N-best list of this sentence:

 We apply the DNN rescoring model and obtain the output value vij (between 0 and 1). A value vij greather than 0.5 means hi is better than hj.  We update the scores of both hypotheses as: scoresem(hi) += vij; scoresem(hj) += 1-vij. (3) After dealing with all the hypothesis pairs, for each hypothesis hi, we obtain the cumulated score scoresem(hi) and employ it as a pseudo probability Psem(hi), combined with the acoustic and linguistic likelihoods according to the equation (2).

word2vec-based rescoring approach

For this method, we define the contextual part and the possibility zones of the Nbest list [START_REF] Level | Introduction of semantic model to help speech recognition[END_REF]. A context part consists of the words common to all the N-best hypotheses generated by the ASR for one sentence. We assume that this part captures the semantic information of the topic context of the sentence. We represent the contextual part with the average of the word2vec embedding vectors of the words of the contextual part:

𝑉 𝑐𝑜𝑛𝑡𝑒𝑥𝑡 = ∑ 𝑉 𝑤𝑜𝑟𝑑2𝑣𝑒𝑐 (𝑤) / 𝑛𝑏𝑟𝑤 𝑐𝑜𝑛𝑡𝑒𝑥𝑡 𝑤 ∈ 𝑐𝑜𝑛𝑡𝑒𝑥𝑡 ( 4
)
where nbrwcontext is the number of words in the context part, and Vword2vec(w) corresponds to a word2vec embedding vector w of the contextual part.

The possibility zones of a hypothesis are the set of words that do not belong to the contextual part. Possibility zones correspond to the area where we want to find the words to be corrected. We represent the possibility zones of each hypothesis by the average of the word2vec embedding vectors of the words of the possibility zones:

𝑉 ℎ𝑖 = ( ∑ 𝑉 𝑤𝑜𝑟𝑑2𝑉𝑒𝑐 (𝑤)) / 𝑛𝑏𝑟𝑤 𝑝𝑜𝑠𝑠 𝑤 ∈ ℎ𝑖 𝑤 ∉𝑐𝑜𝑛𝑡𝑒𝑥𝑡 (5)
where nbrw poss is the number of words in the possibility zones.

For a pair of hypotheses (hi, hj), the input vector for DNN network of the proposed word2vec-based rescoring model could contain the following features:

 context part vector Vcontext;  possibility part vector Vhi for hypothesis hi;  possibility part vector Vhj for hypothesis hj;  cosine distance between Vcontext and Vhi;  cosine distance between Vcontext and Vhj;  acoustic score of hi:

𝑃 𝑎𝑐 (ℎ 𝑖 ) = ∏ 𝑃 𝑎𝑐 (w) 𝑤𝜖ℎ 𝑖 ;  acoustic score of hj: 𝑃 𝑎𝑐 (ℎ 𝑗 ) = ∏ 𝑃 𝑎𝑐 (w) 𝑤𝜖ℎ 𝑗 ;  linguistic score of hi: 𝑃 𝑙𝑚 (ℎ 𝑖 ) = ∏ 𝑃 𝑙𝑚 (w) 𝑤𝜖ℎ 𝑗 ;  linguistic score of hj: 𝑃 𝑙𝑚 (ℎ 𝑗 ) = ∏ 𝑃 𝑙𝑚 (w) 𝑤𝜖ℎ 𝑗 .
During training, the DNN output is set to 1 (or 0) if the first (or the second) hypothesis of the hypothesis pair achieved the lowest WER. . The main advantage of the proposed approach is that acoustic, linguistic, and semantic information are trained together thanks to NN-based framework. Then, according to equation (3), we obtain the cumulated score scoresem(hi). This cumulated score is used as Psem(hi) with an appropriate weighting factor γ for combination according to equation [START_REF] Corona | Improving Black-box Speech Recognition using Semantic Parsing[END_REF]. The hypothesis which obtains the greatest combined score is chosen as the recognized sentence. The proposed DNN configuration for the word2vec-based rescoring model is presented in the left side of figure 1, and corresponds to a neural network with 3 fully connected layers. Fullyconnected layers are used to process the hypothesis pair-level representations, presented previously, and a sigmoid activation is used at the last layer to give vij in output. We call this rescoring model word2vecsem.

BERT-based rescoring approach

BERT is a multi-layer bidirectional transformer encoder that achieves state-of-theart performance for multiples natural language tasks. The pre-trained BERT model can be fine-tuned using task-specific data [START_REF] Turc | Well-Read Students Learn Better: On the Importance of Pre-training Compact Models[END_REF].

As the cosine distance is not meaningful for BERT semantic model [24, 25], we cannot use it to compare the hypotheses, as we did with the word2vec model. So, we only compute the semantic information at the sentence level, as described below.

In our approach, we propose to take a pre-trained BERT model and fine-tune it using application-specific data. Two methods can be used to fine-tune the BERT: masked LM and next sentence prediction. We are basing our BERT fine-tuning on a task similar to the last one. We fine-tune BERT using only embeddings of CLS tokens (see Figure 1, right side). We enter a hypothesis pair (hi, hj), that we want to compare, to a BERT model. The output is set to 1 (or 0) if the first (or the second) hypothesis achieved the lowest WER. For each hypothesis hi, we obtain the cumulated score scoresem(hi) (see equation ( 3)) and use it as a pseudo probability Psem(hi). As for the word2vec-based rescoring model, this semantic probability is combined with the acoustic and linguistic likelihoods according to equation ( 2) with an appropriate weighting factor γ (to be optimized). In the end, the hypothesis that obtains the highest combined score is chosen as the recognized sentence. We call this rescoring model BERTsem.

EXPERIMENTAL CONDITIONS

Corpus description

TED-LIUM corpus [START_REF] Fernandez | TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation[END_REF], containing recordings from TED conferences, is used. This corpus is publicly available. Each conference is focused on a particular subject, so the corpus is well suited to our study of exploring the semantic information. The train, development and test partitions provided within the corpus, are employed: 452 hours for training, 8 conferences for development, and 11 conferences for test (see Table 1). This research work was carried out as part of an industrial project. The project concerns the recognition of speech in noisy conditions, more precisely in a fighter aircraft. To get closer to real aircraft conditions, we add noise to the development and test sets: noise added at 5 dB and 10 dB Signal-to-Noise Ratio (SNR) of an F16 from the NOISEX-92 corpus [START_REF] Varga | Assessment for automatic speech recognition II. NOISEX-92: A database and an experiment to study the effect of additive noise on speech recognition systems[END_REF]. F-16 Fighting Falcon is a single-engine multirole fighter aircraft. The noise is not added to the training part. In addition to that, the proposed approaches are evaluated in clean conditions (development and testing).

Data

Nbr

Recognition system

The recognition system based on the Kaldi voice recognition toolbox [START_REF] Povey | The Kaldi Speech Recognition Toolkit[END_REF] is employed. TDNN (Time Delay Neural Network) [START_REF] Waibel | Phoneme Recognition Using Time-Delay Neural Networks[END_REF][START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF] triphone acoustic models are trained on the training part (without added noise) of TED-LIUM. We perform Statelevel Minimum Bayes Risk training. The lexicon and LM were provided in the TED-LIUM distribution. The lexicon contains 150k words. We perform recognition using the 4-grams and RNNLM (LSTM) models [START_REF] Level | Introduction of semantic model to help speech recognition[END_REF]. We want to explore if using more powerful LM, the proposed rescoring models can improve the ASR. In all experiments, during rescoring, the LM (4-grams or RNNLM) is not modified. The 4-grams LM has 2 million grams. 4-grams and RNNLM were estimated from a textual corpus of 250 million words.

As usual, we employ the development set to choose the best parameter configuration and the test set to evaluate the proposed methods with this best configuration. We compute the WER to measure the performance. It is not possible to calculate the perplexity of our models, because the proposed models only compare two hypotheses. Therefore, in this article, we will not be providing any results related to perplexity. According to our previous work on the semantic model [START_REF] Level | Introduction of semantic model to help speech recognition[END_REF], we chose to employ an N-best list of 20 hypotheses. This size of the N-best list is reasonable to generate the pairs of hypotheses and to have a tractable computational load during the training of rescoring models.

Rescoring models

During DNN rescoring model training, the hypothesis pairs that get the same WER are not used. During evaluation (with development and test sets), all hypothesis pairs are considered. For all experiments, combination weights are: α=1, β is between 8 and 10. γ is between 80 and 100.

word2vec-based rescoring model. We train the word2vec model on a text corpus of one billion words extracted from the OpenWebText corpus. The size of the generated embedding vector is 300 and the embedding models 700k words. DNN configuration for word2vec-based rescoring model is a neural network with 3 fully connected layers (see figure 1, left part). The dropout is 30 %, the activation function is ReLU. For the last layer, the activation function is a sigmoid. The loss function is a binary cross-entropy. We use Adam optimizer. Mini-batch size is 64 samples.

BERT-based rescoring model. We download the pre-trained BERT models provided by Google [START_REF] Turc | Well-Read Students Learn Better: On the Importance of Pre-training Compact Models[END_REF]. We perform the experiments using models with 4, 8, or 12 transformer layers and the size of the hidden layers is 128, 256, or 512 neurons. In the figures, we note these models as LxxHyyy. For instance, L8H256 means the BERT model with 8 transformer layers and 256 as the size of each hidden layer. Three epochs of fine-tuning are performed with mini-batch size of 32 samples.

EXPERIMENTAL RESULTS

Impact of hyperparameters

In this section, we investigate the different hyperparameters of the proposed models. As our task concerns noisy conditions, we decided to perform this study on speech in noisy conditions. The hyperparameters are studied on the development set of TED-LIUM and the best values were applied for the final evaluation on the test set. We 4-grams LM for recognition. During rescoring the LM is not modified.

word2vec-based rescoring model

Impact of training corpus size. We utilize three different sizes of the training data: 1 million pairs of hypotheses (corresponding to 100 TED-LIUM talks of the training set), 6.6 million pairs of hypotheses (500 TED-LIUM talks of the training set), and 13.2 million pairs of hypotheses (1000 TED-LIUM talks of the training set). We observe the similar performance of the word2vec-based model for all data sizes. For lack of space, we do not give these results in the article and will use 500 training talks.

Impact of different DNN input features. We evaluate three configurations (config1, config2, config3 in Figure 2): in config1, the DNN input contains only acoustic scores differences, linguistic scores differences and cosine differences for each hypothesis pair (3 features); in config2, we utilize the acoustic, linguistic scores, and cosine distances (6 features); config3 implements all input features, presented in Section 2.2.1 (906 features). Figure 2 shows that config1 achieves the best performance and config3 is less efficient than config1. Then, embedding features provide no benefit. It is possible, that the relevant acoustic and linguistic data are diluted because the size of the embedding features (900) tends to dominate the size of the acoustic and linguistic features (4). In the following experiments, a word2vec rescoring model based on 500 training talks and config1 will be used.

BERT-based rescoring model

Acoustic and LM probabilities combination (see eq. ( 2)) is not used in these experiments. They will be used in the overall evaluation. Impact of the training corpus size. Figure 3 presents the results on the development corpus using L8H128 BERTsem rescoring model with different amounts of data, i.e. pairs of N-best hypotheses, for fine-tuning. These results show that increasing the size of the fine-tuning data has a significant effect on the WER: more fine-tuning data is profitable to obtain an efficient BERT-based semantic model up to 1000 talks, beyond a degradation is observed.

Impact of the number of hidden layers.

Figure 4 shows the recognition performances as a function of the number of layers of the BERTsem model. The size of the hidden layers is 128 and the size of the fine-tuning data is 1000 talks. Using 12 layers gives the best performance for the two SNR levels. We observe that this parameter plays an important role. Impact of the hidden layers size. Figure 5 reports the importance of the hidden layers size. We use the L12Hyyy BERT model fine-tuned on 1000 training talks. We may observe a variation according to the size of the hidden layers. The best performance is obtained for a size of 256.

In conclusion, we can say that for the BERT-based rescoring model, it is important to utilize a large enough corpus for fine-tuning the model and to choose a model with many transformer layers. The size of 256 for hidden layers, 12 layers and 1000 talks for fine-tuning seems to be a good compromise. These values will be used in the following.

Global results

To further analyze the impact of proposed rescoring models, Table 2 and 3 report the WER for the development and the test sets of TED-LIUM with noise conditions of 10 and 5 dB and with clean speech. In the tables, method Random corresponds to the random selection of the recognition result from the N-best list, without the proposed rescoring models. Method Baseline corresponds to not using the rescoring models (standard ASR). Method Oracle represents the maximum performance that can be obtained by searching in the N-best hypotheses: we select the hypothesis which minimizes the WER for each sentence. The other lines of the table display the performance of the proposed approaches. For all experiments, the N-best list of 20 is used.

For the proposed rescoring models, we study 3 configurations: (1) Rescoring using only the scores scoresem(h) computed with rescoring models as presented in section 2.2 (denoted Xsem in Tables). In this case, in equation (2) α=0, β=0, and γ=1.

(2) Rescoring using a combination of the score scoresem(h), and the acoustic score Pac (h) (denoted Xsem comb. with ac. scores in Tables). In this case, scoresem(h) is used as a pseudo probability and multiplied to the acoustic likelihood with a proper weighting factor γ (to optimize). In this case, Plm (h) is not used, namely, in equation (2) β=0.

(3) Rescoring using a combination of the score scoresem(h), the acoustic Pac (h), and the linguistic score Plm (h) (Xsem comb. with ac./ling. scores in Tables).

We present the results only for the best BERT-based rescoring model L12H256 finetuned using 1000 training talks.

From Table 2, we can observe that word2vecsem rescoring model gives a small but significant improvement compared to the baseline system (confidence interval is computed according to the matched-pairs test [START_REF] Gillick | Some Statistical Issues in the Comparison of Speech Recognition Algorithms[END_REF], used for deciding whether the difference in error-rates between two algorithms tested on the same data set is statistically significant). Unsurprisingly, the proposed BERT-based rescoring model outperforms the word2vec-based model. It is important to note that in the word2vec, the word embed- Adding the acoustic score to the rescoring models (Xsem comb. with ac. scores in Tables) improves the performance. Indeed, the acoustic score is an important feature and should be taken into account. On the other hand, adding the linguistic score during rescoring gives no improvement compared to the Xsem model. We do not present this result in the tables. Using the linguistic and acoustic scores in the BERT rescoring model (BERTsem comb. with ac./4-grams scores) brings only small improvement compared to BERTsem comb. with ac. score: Google's BERT model, trained on billions of sentences, probably captures the linguistic structure of the language better than an n-gram LM trained on a much smaller corpus.

Methods

For BERT-based rescoring results, all improvements are significant compared to the baseline system. On the test set, BERTsem comb. with ac./4-grams scores obtains an absolute improvement of 2.4 % for 5 dB (37.9 % versus 40.3 %), 1.7 % for 10 dB (19.4 % WER versus 21.1 % WER), and 0.4 % for clean speech (8.5 % versus 8.9 %) compared to the baseline system. This corresponds to about of 6 % (for 5 dB), 8 % (for 10 dB), and 4 % (for clean speech) of relative WER improvement.

To better model long-range dependencies of LM, we perform the ASR experiments using a more powerful RNNLM (LSTM). In this case, the RNNLM is used. RNNLM is applied on the ASR word lattices and employed to generate the N-best list. Table 3 reports the results for the same set of experiments but using RNNLM. We can observe that the proposed rescoring methods give consistent and significant improvements, except for clean speech. In clean conditions, only BERTsem comb. with ac./RNNLM scores give an improvement compared to the baseline system. Finally, the best system (BERTsem comb. with ac./RNNLM scores) on the test set gives relative improvement of about 4.6 % for 5 dB (35.4 % versus 37.1 %), 4.5 % for 10 dB (16.9 % versus 17.7 %), and 8.3 % for clean conditions (6.6 % versus 7.2 %) compared to the baseline system. These improvements are significant. We observe also, that in the case of RNNLM, for some cases, the improvements are smaller compared to the 4-grams case. It is possible that RNNLM may reduce the effect of semantic rescoring because RNNLM takes better into account the long-range context dependences.

Methods

CONCLUSION

The goal of this article is to improve the ASR using a rescoring of ASR N-best hypotheses. The main idea of the two proposed approaches is to model the semantic characteristics of words and their contexts. Two approaches are proposed: word2vec-based and BERT-based rescoring models. The information, extracted thanks to these representations, is learned using DNN-based training. Acoustic and linguistic information is integrated too. To evaluate our methodology, the corpus of TED-LIUM conferences is used. The best rescoring system BERT, combined with acoustic and linguistic scores, brings between 4 % and 8 % of relative WER improvement compared to the baseline system, using 4-grams or RNNLMs, and evaluated in clean and noisy conditions. These improvements are statistically significant.

6

Fig. 1 .

 1 Fig. 1. Architecture of the proposed DNN networks (inference stage): (left) word2vec-based rescoring DNN network; (right) BERT-based rescoring DNN network.

Fig. 2 .

 2 Fig. 2. ASR WER (%) on the TED-LIUM development set for different word2vec model configurations (different DNN input features). SNR of 5 and 10 dB. 4-grams LM, training using 100 talks.

Fig. 3 .

 3 Fig. 3. ASR WER (%) on the TED-LIUM development corpus as function of the amount of BERT fine-tuning data. SNR of 5 dB and 10 dB, 4-grams LM, L8H128 BERTsem model.

Fig. 4 .

 4 Fig. 4. ASR WER (%) on the TED-LIUM development corpus according to the number of layers for the BERT model. SNR of 5 dB and 10 dB, 4-grams LM, LxH128 BERTsem model fine-tuned using 1000 training talks.

Fig. 5 .

 5 Fig. 5. ASR WER (%) on the TED-LIUM development corpus as function of the size of hidden layer of BERT. SNR of 5 and 10 dB, 4-grams LM, L12Hyyy BERTsem fine-tuned on 1000 talks.

Table 1 .

 1 The statistics of the TED-LIUM dataset.

		. of talks	Nbr. of words	Duration	Nbr. of segments
	Train	2,351	4,778,000	452h	268,000
	Development	8	17,783	1h36	507
	Test	11	27,500	2h37	1,155

Table 2 .

 2 ASR WER (%) on the TED-LIUM development and test sets, SNR of 10 and 5 dB, and no added noise. N-best hypotheses list of 20 hypotheses, 4-grams LM. L12H256 BERT model fine-tuned on 1000 training talks.dings are static and a word with multiple meanings is conflated into a single representation. In the BERT model, the word embeddings are dynamic and more powerful, because one word can have several embeddings in the function of the context words.

	/systems	SNR 5 dB Dev. Test	SNR 10 dB Dev. Test	no added noise Dev. Test
	Random system	33.5	41.3	16.9	22.9	10.6	12.1
	Baseline system	32.7	40.3	15.7	21.1	8.7	8.9
	word2vecsem	32.1	39.2	15.3	20.6	8.5	8.8
	word2vecsem comb with ac. scores	31.8	39.2	15.2	20.5	8.5	8.8
	word2vecsemcomb.with.ac./4grams.sc.	31.5	38.8	15.2	20.4	8.5	8.8
	BERTsem	31.1	38.7	14.4	19.8	8.0	8.7
	BERTsem comb with ac. scores	30.6	37.9	14.2	19.4	7.9	8.6
	BERTsem.comb with ac./4grams sc.	30.6	37.9	14.1	19.4	7.8	8.5
	Oracle	27.5	33.2	11.2	15.0	5.2	4.7

Table 3 .

 3 ASR WER (%). N-best hypotheses list of 20 hypotheses. TED-LIUM development and test sets, SNR of 10 and 5 dB, and no added noise. RNNLM (LSTM). L12H256 BERTsem model fine-tuned on 1000 talks.

	/systems	SNR 5 dB Dev Test	SNR 10 dB Dev Test	no added noise Dev Test
	Random system	29.2	38.4	13.9 20.2	8.9	10.8
	Baseline system	28.2	37.1	12.3 17.7	6.6	7.2
	word2vecsem	27.4	36.3	12.0 17.5	6.6	7.2
	word2vecsem comb with ac. scores	27.3	35.6	12.1 17.5	6.8	7.2
	word2vecsem comb with ac./RNNLM sc.	27.3	35.5	12.0 17.4	6.6	7.2
	BERTsem	27.0	35.9	12.0 17.4	7.1	8.1
	BERTsem comb with ac. scores	26.6	35.3	11.6 17.1	6.9	7.1
	BERTsem comb with ac./RNNLM sc.	26.5	35.4	11.5 16.9	6.0	6.6
	Oracle	23.1	30.2	8.3	12.1	3.8	3.5

Acknowledgments

The authors thank the DGA (Direction Générale de l'Armement, part of the French Ministry of Defence), Thales AVS and Dassault Aviation who are supporting the funding of this study and the "Man-Machine Teaming" scientific program.