Stavros Nousias

Christos Tselios

Dimitris Bitzas
email: bintzas@ece.upatras.gr

Olivier Orfila
email: olivier.orfila@ifsttar.fr

Samantha Jamson
email: s.l.jamson@its.leeds.ac.uk

Pablo Mejuto
email: pablo.mejuto@ctag.com

Dimitrios Amaxilatis
email: d.amaxilatis@sparkworks.net

Orestis Akrivopoulos

Ioannis Chatzigiannakis

Aris S Lalos
email: aris.lalos@ece.upatras.gr

Konstantinos Moustakas
email: moustakas@ece.upatras.gr

Managing nonuniformities and uncertainties in vehicle-oriented sensor data over next generation networks

Keywords: Graph Matrix Completion, V2X, MEC, Sensor Data

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I. Introduction

As global telecommunication market shifts towards the 5G era and the fascinating technological advances it proclaims [START_REF] Bianchi | Superfluidity: a flexible functional architecture for 5g networks[END_REF], it becomes obvious that new applications and verticals with a tremendous affect in our everyday lives are approaching. The deployment of a highly heterogeneous, always available, brisk and agile network, offering inherent support for billions of interconnected devices with less than 1 millisecond end-to-end latency [START_REF] Tselios | On QoE-awareness through Virtualized Probes in 5G Networks[END_REF], will transform almost every application from simple daily entertainment to autonomous private transportation and the automotive domain in general, inside Smart Cities which will provide the proper underlying infrastructure.

The automotive domain is somehow divided into two separate yet highly consolidating tracks, autonomous driving vehicles (ADV) and vehicle-to-everything (V2X) communication [START_REF] Vlachos | Autonomous driving in 5g: Mitigating interference in ofdm-based vehicular communications[END_REF], with V2X act as a key technology enabler for ADV by allowing moving vehicles to approach each other more safely, thus enabling traffic flow optimization techniques and increased situational awareness.

Vehicle-oriented notifications and alerts are propagated to nearby infrastructure, properly equipped pedestrians or other vehicles, rendering every involving entity capable of reacting fast and efficiently regardless of the situation [START_REF] Hobert | Enhancements of v2x communication in support of cooperative autonomous driving[END_REF] for instance, to the now hazardous cases of blind intersections, closed curves, lane switching and overtaking which often cause fatal car accidents worldwide. Smart Cities all around the globe are deploying dedicated infrastructure for facilitating V2X communication in a seamless manner, since citizen security and effective public/private transportation are considered top priorities.

Information retrieval, transfer and manipulation in realtime, requires a rapidly operating network with decreased end-to-end propagation delay, large throughput and low latency. In addition, for efficient and secure V2X communication applications, the same network should include proper interfaces for establishing robust connections with the heterogeneous devices involved, being adequately fault tolerant and able to deliver informative messages even with fragmented data.

In this paper, motivated by the necessity for fault tolerant systems that will extract information from incomplete datasets, as well as the omnipresent demand of limited data transfer through the rather congested Smart City networks that will be forced to operate partially based on the resource savvy MEC architecture, we present a method of managing nonuniformities and uncertainties in vehicle-related data. This solution will allow us to retrieve adequately detailed information from smaller or compromised datasets through an algebraic interpolation process that intends to effectively fill the missing information gaps without changing the overall essence of the actual dataset. Depending on the use case, the proposed solution could be used for analytics retrieved from artificially reconstructed data or simply as a method of lowering the threshold of the necessary amount of data needed for a specific decision.

The rest of this paper is organized as follows: Section II identifies our motivation and previous work on the topic. Section III describes the mathematical method used for the vehicle-oriented data reconstruction process. Section IV presents the testbed architecture while the necessary results are included in Section V. Finally, Section VI concludes the paper.

II. Motivation and Previous Work

Over the last few years, cloud computing evolved to a highly disruptive architectural paradigm which clearly dominated the global market of on-demand computational resources, allowing end users to efficiently lease them according their needs on any given time [START_REF] Tselios | Cloud Computing: A Great Revenue Opportunity for Telecommunication Industry[END_REF], [START_REF] Tselios | Cloud for multimedia applications and services over heterogeneous networks ensuring QoE[END_REF]. However, most service providers deploy massive geographically isolated datacenters, leading to a large round trip delay, network congestion or service quality degradation, all highly compromising factors for real-time, latencysensitive service requests and applications.

An initial approach for tackling issues such as increased delay and elevated latency was eliminating distance between the data source and the corresponding computational resources that handle them as proposed by [START_REF] Garcia Lopez | Edge-centric computing: Vision and challenges[END_REF]. This work introduced the notion of data processing in the network edge through dedicated nodes referred to as "edge devices". Alas, despite its efficiency, this approach does not comply with the common deployment paradigms of cloud computing followed by every vendor, therefore an alternative solution was needed, however with respect to the notion of edge computing which was proved to be a step towards the right direction. The most recent architectural model for providing cloud computing capabilities paired with an IT service environment at the edge of the mobile network within the Radio Access Network (RAN) and in proximity to the service subscribers is Multi-access Edge Computing formerly known as Mobile Edge Computing (MEC) [START_REF] Mach | Mobile edge computing: A survey on architecture and computation offloading[END_REF]. This novel framework is characterized by ultra-low latency, high bandwidth, real-time access to radio network and context information, location awareness, efficient network operation and service delivery, thus ensuring high quality of experience (QoE) for all interconnected users . According to the European 5G Public Private Partnership (5G-PPP)1 MEC facilitates the actual transformation of the mobile broadband network into a programmable ecosystem and paves the way towards meeting the original standards of 5G in terms of expected throughput, latency, scalability and automation.

To further clarify the abilities of the proposed architecture, European Telecommunications Standards Institute (ETSI) created an industry specification group (ISG) which in [START_REF] Etsi Gs Mec | Mobile Edge Computing (MEC): Service Scenarios[END_REF] describes a few service scenarios under consideration that fully take advantage of MEC towards increasing performance compared providing such services through the cloud or core network servers. As expected, the Smart City-relevant, Vehicleto-Infrastructure/Pedestrian/Vehicle (V2X) use case was among the dominant ones, since it uses MEC to extend the connected car cloud into the highly distributed mobile base station environment, and enable data and applications to be housed close to the vehicles. This can reduce the round trip time of data and enable a layer of abstraction from both the core network and applications provided over the internet. MEC applications can run on corresponding MEC servers which are deployed at the base station site to provide roadside functionality. The MEC applications are designed to receive local messages directly from vehicle and roadside sensors, analyze them and then propagate (with extremely low latency) hazard warnings and other latency-sensitive messages to all interconnected devices, as depicted in Figure 1. This enables for instance a nearby car to receive data in a matter of milliseconds, allowing the driver to immediately react.

However a serious issue still remains; what happens when, regardless the reason, the obtained vehicle-oriented sensor datasets are not adequate for proper analysis? Or similarly, which is the baseline above which problematic situations can be identified, taking into consideration that network conditions may compromise the transmission of the whole set of sensor information from the vehicles. This thorny issue could be efficiently tackled by employing Matrix Completion techniques [START_REF] Keshavan | Matrix completion from a few entries[END_REF] as analyzed in the following section.

III. Low rank approximations in the presence of missing data

A. Preliminaries on Matrix Completion

Assuming that we are given a low-rank incomplete matrix V with dimensions 𝑚×𝑛, then Matrix Completion (MC) refers to the problem of reconstructing the values of the data matrix when only a small subset of its entries is available. More specifically, given V ∈ ℝ 𝑚×𝑛 with rank 𝑅 and a set of known entries 𝐾 , then the solution of the optimization problem

minimize 𝜏 ‖Y‖ * s.t. Y 𝑖𝑗 = V 𝑖𝑗 , 𝑖, 𝑗 ∈ 𝐾 (1)
where 𝜏 is a weighting parameter and ‖Y‖ * is the nuclear norm of Y in (1) can precisely recover the content of matrix V . The nuclear norm is formulated as

‖Y‖ * = ∑ 𝑘 𝜎 𝑘 (Y) (2)
where 𝜎 𝑘 (Y) is the k-th singular value of 𝑌 . As the authors in [START_REF] Candès | Exact matrix completion via convex optimization[END_REF] state, for a matrix sampled with a random process with 𝑛 entries, so that 𝑛 ≥ 𝑐 1 𝑁 5/4 𝑅𝑙𝑜𝑔(𝑁 ) then with a probability 1 -𝑐 2 𝑁 -3 𝑙𝑜𝑔𝑁 the solution of ( 1) is equal to V where 𝑐 1 and 𝑐 2 are constants. To perform the minimization of (1) the singular value thresholding (SVT) algorithm [START_REF] Cai | A singular value thresholding algorithm for matrix completion[END_REF] is employed. The SVT operator 𝐷 𝜏 (Z) is presented in [START_REF] Cai | A singular value thresholding algorithm for matrix completion[END_REF]. Given Z the dual variable of the Lagrangian, the minimizer of ( 1) is evaluated via the iterative execution of the following equations:

Y 𝑙 = 𝒟 𝜏 (Z 𝑙-1 ) (3)
Z 𝑙 = Z 𝑙-1 + Ω ∘ (𝑉 -𝑌 ) (4)
for 𝑙 = 0, … , 𝑙 𝑚𝑎𝑥 . Moreover, matrix Ω is defined as

Ω = { 1 𝑖, 𝑗 ∈ 𝐾 0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

B. Application to one dimensional signals

To be able to apply the aforementioned approach in the case of time-series, we initially rearrange the one dimensional data into square or rectangular matrices. To be more specific, we initially construct matrix columns by concatenating the measurements 𝑣 1 , 𝑣 2 , ⋯ , 𝑣 𝑛⋅𝑚 as presented in Figure 2,

V = ⎡ ⎢ ⎢ ⎣ 𝑣 1 𝑣 𝑚+1 … 𝑣 (𝑛-1)𝑚+1 𝑣 2 𝑣 𝑚+2 … 𝑣 (𝑛-1)𝑚+2 ⋮ ⋮ ⋱ ⋮ 𝑣 𝑚 𝑣 2𝑚 … 𝑣 𝑛⋅𝑚 ⎤ ⎥ ⎥ ⎦ (5)
To be able to evaluate the reconstruction ability of the aforementioned MC approach to the current dataset, we need to identify the required number of singular values 𝑘 that minimize the nuclear norm of

V 𝑘 ∑ 1 𝜎 𝑘 (V) ≈ 𝑚𝑖𝑛(𝑛,𝑚) ∑ 1 𝜎 𝑘 (V) (6)
so that Fig. 2: Matrix formulation

∑ 𝑘 1 𝜎 𝑘 (V) ∑ 𝑚𝑖𝑛(𝑛,𝑚) 1 𝜎 𝑘 (V) ≥ 0.98 (7)
In order to utilize more effectively the temporal correlation of the vehicle sensor data measurements in the time domain we further impose Laplacian constraints in the initial optimization problem in order to move the missing sample 𝑣 𝑖 with index 𝑖 to the weighted average of its known neighbors:

𝑥 𝑖 - 𝑤 𝑟 𝑤 𝑙 + 𝑤 𝑟 𝑥 𝑖-𝑤 𝑙 - 𝑤 𝑙 𝑤 𝑙 + 𝑤 𝑟 𝑥 𝑖+𝑤 𝑟 (8)
where the weight 𝑤 𝑟 is equal to the temporal distance between 𝑥 𝑖 and the next available entry and correspondingly 𝑤 𝑙 denotes the temporal distance between 𝑥 𝑖 and the previous available entry. The aforementioned operation can be expressed in matrix form by formulating the Laplacian matrix L of dimensions 𝑛𝑚 × 𝑛𝑚 defined as

L = ⎧ { { ⎨ { { ⎩ 1 𝑖 ∉ 𝐾, 𝑗 = 𝑖 -𝑤 𝑟 𝑤 𝑙 +𝑤 𝑟 𝑖 ∉ 𝐾, 𝑗 = 𝑖 -𝑤 𝑙 -𝑤 𝑙 𝑤 𝑙 +𝑤 𝑟 𝑖 ∉ 𝐾, 𝑗 = 𝑖 + 𝑤 𝑟 0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
To minimize the distance between the i-th component of the reconstructed time series 𝑥 𝑖 and the weighted mean 𝑦 𝑖 of the available nearest neighbours we need to minimize the following expression:

‖L𝑣𝑒𝑐(Y)‖ 2 2 (9)

C. Matrix completion with Laplacian constraints

Therefore, the nuclear norm minimization problem after the addition of the Laplacian constraint can be written as follows: [START_REF] Keshavan | Matrix completion from a few entries[END_REF] where the first term minimizes the error between the known values and the estimated, the second term imposes the low rank constraint to the recovered matrix and the last term moves the estimated value to the weighted average of the nearest available neighbours. Y is the optimization variable, the parameter 𝜇 is the regularization parameter for the Laplacian constraint and ‖.‖ 𝐹 corresponds to the Frobenius norm.

min Y ‖Ω ∘ (Y -V)‖ 𝐹 + 𝜏 ‖Y‖ * + 𝜇 ‖L𝑣𝑒𝑐(Y)‖ 2 2
The Lagrangian of the splitting version of the optimization problem in [START_REF] Keshavan | Matrix completion from a few entries[END_REF] can be written as

ℒ(Y, U, Z) = 1 2 ‖Ω ∘ (U -V)‖ 2 𝐹 + 𝜏 ‖Y‖ * + + 𝜇 ‖L𝑣𝑒𝑐(U)‖ 2 2 + 𝜌 2 ‖Y -U‖ 2 𝐹 ( 11
)
where U, Z are the dual variables and 𝜌 is the penalty parameter. The solution of ( 11) can be efficiently obtained after using the Alternating Direction Method of Multipliers (ADMM) [START_REF] Boyd | Distributed optimization and statistical learning via the alternating direction method of multipliers[END_REF] which can be summarized into the following steps:

Y(𝑙 + 1) =𝑎𝑟𝑔 min Y ℒ(Y, U(𝑙), Z(𝑙)) (12
For equation ( 12) the minimizer is given by the SVT operator Y(𝑙 + 1) = 𝐷 𝜏 (U(𝑙) -𝜌 -1 Z(𝑙)).

The minimizer of equation ( 13), it can be easily shown that is given by the following formulation:

Ω ∘ (U -V) + 𝜇(L 𝑇 L)U + 𝜌(U -Y -𝜌 -1 Z) = 0 (15)
The solution of (15) is found after executing iteratively for 𝑙 = 0, … , 𝑙 𝑚𝑎𝑥 the following steps:

1) Equation ( 15) is formulated as

[𝐷(𝑣𝑒𝑐(Ω)) + 𝜇(L 𝑇 L) + 𝜌I] 𝑢 = 𝑣𝑒𝑐(Ω ∘ V + 𝜌Y + Z) (16) 2) Compute Y(𝑙 + 1) = 𝐷 𝜏 (U(𝑙) -𝜌 -1 Z(𝑙)) (17)
3) Employ over-relaxation parameter 𝛾 so that

W = 𝛾Y(𝑙 + 1) -(1 -𝛾)U(𝑙)) (18) 4) Solve
IV. System Architecture For properly evaluating the aforementioned method through real data from vehicle sensors the system architecture of Figure 3 was deployed. Vehicle-oriented data collected by the embedded sensors of the car were obtained using an On-Board Diagnostics (OBD) module, supported by every major manufacturer following specific EU regulatory guidelines. Without the loss of generality, only data regarding vehicle speed, engine roundsper-minute (RPM) and throttle position were collected for creating matrices stored in a per-trip fashion. The OBD module was connected over Bluetooth protocol to an Android smartphone, on which a specialized, tailormade application was running2 . This application acted as a data aggregator that accumulated sensor values, added a timestamp and created a .CSV file. After each trip, the application transmitted the .CSV file through throttled 802.11g/4G LTE connections for emulating network congestion into a specially designed [START_REF] Akrivopoulos | On the deployment of healthcare applications over fog computing infrastructure[END_REF] online repository for further process. The online data undergo the process described in the previous section for accurately evaluating the effect of Matrix Completion method.

V. Experimental Results

This section provides the performance analysis for the proposed method described in section III. Experiments were carried out after using real-world data of vehicle speed, engine RPM and throttle position. The matrices were filled with data corresponding to six driving sessions of the same driver over the same route. The time series of all driving sessions synchronized according to the time the vehicle started, while the duration of each driving session was approximately 26 minutes.

In order to execute matrix completion, the time series of data were reshaped in a matrix of size 138 × 138 for all three types of sensor data namely speed, engine RPM and throttle position. Each column of this matrix corresponds to approximately 1 minute of driving. The matrices with missing entries were obtained from the initial matrices of data after the removal of values at random. The missing entries percentage started from 5% and reached 60% with 5% step. For each percentage of missing values the proposed approaches are executed in 100 different permutations. The final result is the mean value of all permutations results per missing percentage value.

The reconstruction accuracy was evaluated using the Normalized Mean Square Error and the Normalized Root Mean Square Error for the different number of missing entries for the three aforementioned cases.

The first case study, related to the RPM measurements showed that the matrix of engine RPM data has high correlation thus resulting in a low rank matrix as can be easily seen by inspecting the eigenvalue distribution in Figure 4. This attribute explains also the accurate reconstruction of the matrix with even 60% missing entries, justified by the relatively small NMSE that is presented in Figure 5. The matrix completion approaches provide better reconstruction results in the case of vehicle speed data. The main reason is again the low rank of the vehicle speed matrix Figure 6. It is worth noting that the NMSE is close to zero even with 60% missing entries as shown in Figure 7.

The rank of the throttle position matrix is not as low as in the previous two cases of engine RPM and vehicle speed Figure 8. The results in this case are worse, but the difference between the regularized and the conventional MC approach remains obvious Figure 9.

VI. Conclusions

This paper introduced the Laplacian Matrix Completion method and demonstrated is superior performance compared to MC in three cases of vehicle-related sensor data. At 60% missing entries the Laplacian Matrix Completion reconstructs the matrices with half NMSE of conventional Matrix Completion.

By capitalizing on the low rank property of the sensor data retrieved by the OBD module, we permeat benefits

Fig. 1 :

 1 Fig. 1: V2X communications over Multi-access Edge Computing Infrastructure in a Smart City

  )U(𝑙 + 1) =𝑎𝑟𝑔 min U ℒ(Y(𝑙 + 1), U, Z(𝑙 + 1))(13)Z(𝑙 + 1) =Z(𝑙) + 𝜌(Y(𝑙 + 1) -U(𝑙 + 1))

[

  𝐷(𝑣𝑒𝑐(Ω)) + 𝜇(L 𝑇 L) + 𝜌I] 𝑢 = 𝑣𝑒𝑐(Ω ∘ V + 𝜌W + Z(𝑙)) (19)

5) Reshape u to matrix U 6 )

 6 Set Z equal to Z(𝑙) + 𝜌(Y(𝑙 + 1) -U(𝑙 + 1))

Fig. 3 :

 3 Fig. 3: System Architecture

Fig. 4 :

 4 Fig. 4: Eigenvectors importance for matrix of engine RPM data.

Fig. 5 :Fig. 6 :

 56 Fig. 5: Comparison of proposed Laplacian based MC with conventional MC at various percentages of known entries for engine RPM dataset:(a) NRMSE (b) NMSE

Fig. 7 :Fig. 8 :

 78 Fig. 7: Comparison of proposed Laplacian method with conventional MC at various percentages of known entries for vehicle speed dataset:(a) NRMSE (b) NMSE

Fig. 9 :

 9 Fig. 9: Comparison of proposed Laplacian with conventional MC at various percentages of known entries using the throttle position dataset:(a) NRMSE (b) NMSE

https://5g-ppp.eu/

https://play.google.com/store/apps/details?id=app.gamecar. sparkworks.net.gamecardatalogger

Acknowledgement

This work was partly funded by European Union's Horizon 2020 research and innovation programme GamECAR under grant agreement No 732068


Alexis Poignant

Mathilde Legrand

Nathanael Jarrassé

Guillaume Morel

come

Computing the positioning error of an upper-arm robotic prosthesis from the observation of its wearer's posture

Computing the positioning error of an upper-arm robotic prosthesis from the observation of its wearer's posture.

Alexis Poignant, Mathilde Legrand, Nathanaël Jarrassé and Guillaume Morel

Abstract-When the arm prosthesis worn by an amputated Human being is not adequately configured with respect to the end-effector task, body compensations are often observed. Namely, to compensate for a wrong joint positioning on the robotic distal side, a subject trying to reach a desired position/orientation of his/her hand mobilizes his/her proximal joints, thus exploiting the redundancy of the human+robot kinematic chain.

In this paper, we explore the possibility of exploiting this well-known behavior to reverse the causality: if we observe the posture of an amputated subject wearing a prosthesis during a hand positioning task, to what extent can we infer the positioning error of the prosthesis?

To answer this question, we make the assumption that the adequate, or natural posture for a given task is one that optimizes a postural score. The proposed approach then consists in i) measuring the joint posture of the subject fitted with the prosthesis; ii) search for an alternative posture that optimizes a postural score within the null space of the human+robot kinematic chain and iii) compute the position error for the robot joints between the initial and the optimized posture.

An experimental evaluation is provided with non amputated subjects who emulate erratic positioning of their distal joints during hand positioning tasks. Results show that joint errors are estimated with a precision that seems compatible with the implementation of a real time control algorithm.

I. BACKGROUND

Mechanical achievements, such as the augmentation of the number of DoF and the weight reduction [START_REF] Cordella | Literature review on needs of upper limb prosthesis users[END_REF], have paved the way for a new generation of prosthetic arms, both more functional and comfortable. These electrical prosthesis should, with time, match the benefits of body-powered prosthesis (precision and design for heavy work [START_REF] Millstein | Prosthetic use in adult upper limb amputees: a comparison of the body powered and electrically powered prostheses[END_REF]) while being more comfortable and providing a greater range of functions [START_REF] Farina | The extraction of neural information from the surface emg for the control of upper-limb prostheses: emerging avenues and challenges[END_REF]).

Commercially available upper-limb electric prostheses are controlled by their wearer through voluntarily generated auxiliary signals, most often electromyograhic signals (EMG). Though, surface EMG measurement highly depends on skin surface factors [START_REF] Farina | The extraction of neural information from the surface emg for the control of upper-limb prostheses: emerging avenues and challenges[END_REF], considerably reducing the control robustness, while requiring extended learning phases from users, especially when dealing with the control of multiple active DoF. Meanwhile, the use of auxiliary signals generates two parallel control loops for the users: the body natural sensorimotor close-loop and the auxiliary prosthetic open-loop. The management of these two parallel loops is complex and non-natural. Therefore, users tend to underuse the prosthetic Fig. 1: A subject equipped with a robotized elbow, and having his wrist and hand locked, is asked to position a distal point at a given location materialized by a robot manipulator. On the left, a correct elbow positioning (135 • between the human humerus and the robotic ulna) results in a global posture that seems natural. On the right, with an erratic elbow joint position (90 • between humerus and ulna) the subject has to lean his trunk to achieve the task. This is called a body compensation.

joints to the benefit of residual functional joints with body compensations [START_REF] Cordella | Literature review on needs of upper limb prosthesis users[END_REF].

In the present paper, we aim at exploiting this natural behavior to evaluate whether a robotic prosthesis is correctly positioned during a manipulation task. The general idea, when looking at Fig. 1, consists in trying to compute the natural posture adopted on the left image from the measurement of the erratic posture adopted on the right image. Given a task, computing a natural posture, or, reciprocally, characterizing an ergonomically erratic posture, is a problem that is widely studied in the domain of industrial workstation optimization, and more specifically in the specific context of human-robot cooperation in the factories.

A first approach is based on dynamic simulation of a virtual manikin, using energy and forces based criteria mixed with task criteria, [START_REF] Maurice | Assessment of physical exposure to musculoskeletal risks in collaborative robotics using dynamic simulation[END_REF], [START_REF] De Magistris | Dynamic virtual manikin control design for the assessment of the workstation ergonomy[END_REF], [START_REF] Ore | Virtual evaluation of industrial human-robot cooperation: An automotive case study[END_REF]. These methods have proven to be efficient but the transposition from the virtual to the real world strongly depends on the human model accuracy, leading to the identification of subject specific parameters for real-world applications [START_REF] Peternel | Towards ergonomic control of human-robot co-manipulation and handover[END_REF]. Besides, these methods return trajectories, and relies on quadratic programming, requiring heavy computation power, as well as complex sensors (force sensors for example) that may difficulty be integrated to light and movable cobots or prostheses for real-world assessments.

A second approach relies on lighter kinematic models. This can be done while accounting for ergonomic criteria [START_REF] Shafti | Real-time robot-assisted ergonomics*[END_REF] and task specific criteria [START_REF] Busch | Postural optimization for an ergonomic human-robot interaction[END_REF] to continuously control the behavior of the cobot. These methods optimize the global RULA-score (which evaluates the exposure to risk factors associated with work related upper limb disorders [START_REF] Mcatamney | RULA: a survey method for the investigation of world-related upper limb disorders[END_REF]), either discretely [START_REF] Shafti | Real-time robot-assisted ergonomics*[END_REF] -and therefore controlling sequentially the postural objective -or by continuously fitting polynomial functions, requiring a large set of pre-recorded postures to do such interpolation [START_REF] Busch | Planning ergonomic sequences of actions in human-robot interaction[END_REF]. Further, these approaches propose to correct the user posture by moving the manipulated tool or instrument in the task space. Rather, in a prosthetic approach, there is no possible control on the task. In fact, the task is a priori unknown. For this reason, we propose to assume that the task is correctly realized by the user thanks to compensation, and therefore to search for a possibly better joint configuration inside the null-space of the redundant human+prosthesis kinematic chain.

To this aim, generalized weighted inverse [START_REF] Lura | The creation of a robotics based human upper body model for predictive simulation of prostheses performance[END_REF][13] can be used to link the end-effector velocity to the adequate prosthesis' motion. These models are computationally efficient but their weights are calculated by minimizing a criteria on previously recorded data, which requires important dataset from prosthesis users. These weights are also task specific, yet could be extended to multiple tasks by adding neural networks for classification [START_REF] Lura | The creation of a robotics based human upper body model for predictive simulation of prostheses performance[END_REF], [START_REF] Xia | A dual neural network for kinematic control of redundant robot manipulators[END_REF] but, again, the required dataset increases. Moreover, the weights do not ensure that the prosthesis movement will be ergonomically adequate, and the generalization from one user to another is complex.

To use ergonomic criteria without modifying the nullspace, we can exploit a projector [START_REF] Liegeois | Automatic supervisory control of the configuration and behavior of multibody mechanisms[END_REF], [START_REF] Siciliano | Kinematic control of redundant robot manipulators: A tutorial[END_REF]. Commonly employed for the control of redundant robotic arms with a high number of DoF, null-space control can here be transposed to human + prosthesis kinematic model. Instead of looking for the appropriate weights to minimize a criteria on recorded data, the null-space allows to minimize an ergonomic criteria projected on the solution space. Moreover, this criteria is neither task specific nor user specific, making its extrapolation easier from one user to another without any records. This is why this approach was chosen.

The rest of the paper is organized as follows. We first detail the proposed optimization approach (Section II). Then we present the experimental protocol and setup that was used to gather data in order to evaluate the performance of our method (Section III). Experimental results are then provided (Section IV) and discussed (section V).

II. PROPOSED APPROACH

We parameterize the human+prosthesis kinematic chain by a joint angular position : q = (q i ), i ∈ {1 • • • n} with n representing the total number of DoFs, including those of the prosthesis and those used to model the human kinematic chain. We suppose that a sensory device provides a measure q m of the joint posture q at any time t.

We aim at computing q, an optimized joint vector that results in the same end-effector position and orientation as q m , while minimizing a cost function C. Inspiring from ergonomic scores such as RULA table, widely used to evaluate the goodness of a given posture defined by joint angles, [START_REF] Ergo-Plus | A step-by-step guide to the rula assessment tool[END_REF], we consider that the optimal joint position corresponds to an upper body at rest, characterized by a desired position q d . The computed optimal position is defined by:

q = argmin q C(q, q d ) s.t f (q) = f (q m )
where f (.) denotes the kinematic model of the hu-man+prosthesis chain, mapping the joint position q into the the hand position and orientation. This problem is iteratively solved by computing a null-space reconfiguration:

q ← q init δ ← 2ε while δ > ε, δ ← J † (q) [f (q m ) -f (q)] -I -J † (q)J(q)) grad (C(q, q d )) q ← q + δ end while q ← q (1)
where J(q) ∈ R n×6 denotes the Jacobian matrix for a given joint position angle q; † the generalized inverse operator.

The step δ is formed of two terms: the task-space step J(q) † [f (q m ) -f (q)], which ensures the respect of the constraint f (q m ) = f (q) and the null-space step I -J † (q)J(q)) grad (C(q, q d )) that locally minimizes the cost function C. The convergence is obtained when the step δ is as small as a tunable positive threshold ε.

Finally, the postural error, between the natural and optimised model, is computed as the difference between q m and q the joint angle at convergence.

For the cost function C we chose:

C(q -q d ) = M Z(q -q d ) 2 2 , (2)
where M ∈ R n×n + is a diagonal matrix of positive weights and Z denotes a dead zone along each component of q, i.e.,

Z i (q -q d ) =      q i -q d,i + z i if q i -q d,i > +z i q i -q d,i -z i if q i -q d,i < -z i 0 otherwise
Therefore the i-th component of the gradient function g = grad(C) writes:

g i (q -q d ) =      2M i (q i -q d,i + z i ) if q i -q d,i > +z i 2M i (q i -q d,i -z i ) if q i -q d,i < -z i 0 otherwise

III. MATERIAL AND METHODS

A. Kinematic model

We consider the theoretical case of a person amputated at the humerus level, equipped with a 2 DoF prosthesis: elbow flexion + wrist prono-supination. The person is sat on a stool (thus legs are not involved in body compensations at this stage). The task consists in positioning/orienting the hand at a desired location.

The global kinematic model involves 9 revolute joints :

• 7 human actuated joints (consisting in a 3 DoF spherical pelvis joint, a 1 DoF scapula elevation joint and a 3 DoF spherical shoulder joint; • 2 robotic joints (1 DoF elbow and a 1 DoF wrist). The Dennavit and Hartenberg parameters of this model are described in Table I, where the length of the skeleton bones

α π/2 -π/2 -π/2 0 π/2 -π/2 π/2 π/2 0 θ of f set 0 0 0 0 0 -π/2 0 π 0 a 0 0 0 l scapula 0 0 0 0 0 d 0 0 l torso 0 0 0 l a 0 l f a
are noted l torso , l scapula , l a , l f a , for respectively the torso, the scapulae, the arm and the forearm. Such a kinematic model is the one provided by the skeleton tracking functionality of a Microsoft Kinect V2 sensor which is used to measure q m . Figure 2 shows a photo of a subject observed by the Kinect together with the superimposed extracted skeleton.

B. RULA-inspired criteria

Since the RULA score is a discrete score, it cannot be directly used as an optimization criteria. We rather use the ergonomically ideal joint angles to define a continuous cost function. Some joints, such as the trunk, are defined with the ideal position of 0 • , whereas some joints are defined ergonomically good when within a given range. For example, the elbow is ideally position when its angle fits within 80 • ± 20 • . The ideal ergonomic posture is thus defined by the joint vector q d with associated deadzone vector z.

This criteria relies on the hypothesis that compensatory movements always degrade ergonomics, and that the desired posture q d remains the same, no matter the performed task.

To avoid having a cost function being weighted the same for all joints (which would make the trunk as ergonomically significant as the shoulder), the weighting matrix M was designed in order to reflect the specificity of the RULA score for the different joints. Those weights were adjusted the following way: the higher a joint angle denotes a compensatory movement, the higher the associated weight should be, while a purely functional joint is not weighted. In order to avoid over-fitting, the weights were tuned by hand (Table II). The shoulder and wrist weights are set to 0 • as they are purely functional joints during the performed tasks. The elbow being highly functional (closely related to the distance of the target if the trunk is straight), its weight is also low.

More complex criteria can be derived from discrete scores, for example by fitting polynomial functions on sets of postures [START_REF] Busch | Postural optimization for an ergonomic human-robot interaction[END_REF] but this requires to learn from a dataset of postures, which is difficult to obtain especially for specific population such as prosthesis users. Rather, the chosen cost function can be easily configured with a very small dataset.

C. Protocol

An experiment was conducted on 15 non-amputated subjects from 20 to 55 years old, from 1.60 to 1.90m tall, 6 women and 9 men. Those had to perform hand positioning tasks at a given location. The positioning was first done in a natural way (without constraints). Second, subjects were asked to adopt an erratic positioning of their distal joints so as to mimic the case of an amputated subject with a prosthesis exhibiting an erroneous posture, and to reach the same given location for the hand. They were sat on a stool in order to limit the displacement of the pelvis and to compare more accurately the different posture with and without compensation. They were wearing a wrist splint to block their wrist flexion and abduction, to mimic a 2 DOF prosthesis configuration. Each subject performed 18 reaching tasks with their right arm, with and without (provoked) compensations. The 18 tasks consist of 6 positions: 2 distances (80% and 60% maximum reach) × 3 horizontal positions (in front of the right shoulder position and ± the shoulder width) × 1 height ( sternum height), across 3 different hand orientation: palm vertical, palm horizontal, palm at a 45 • to the ground. These 18 couples position/orientation represents 18 targets normalized to the subjects' size.

Each task goes as follow: I) The subjects are asked to reach the target naturally, and this "Natural Posture" is recorded. Then, while maintaining hand position and orientation, they are instructed to move their distal joints (wrist and elbow) according to pseudo-randomly drawn instructions : flexion or extension of the elbow and pronation or supination of the wrist (4 total compensations, each being accounting for 25% of the total compensated posture set). II) While keeping their distal joints blocked, they were asked again to reach the target. To do so, a visual feedback is provided on a screen. A "Compensated Posture" is then recorded. III) Offline, the optimization algorithm is run and return an "Optimal Posture". The total dataset is constituted of 15 * 18 = 270 sets of postures, each set being composed of 1 natural posture (Nat), 1 compensated posture (Comp), and 1 optimal posture (Opt) returned by the algorithm. Fig. 3: Examples of recorded kinematics for one participant reaching two different targets and seen from two different angles. On the left the subject compensated by bending the trunk forward, while on the right, he compensated backward (because of imposed elbow extension). The computed optimized postures appear to be close to the natural ones in both cases. The data is centered on the pelvis' natural posture.

D. Joint offset and jitter from the Kinect sensor

The Kinect v2 Camera's joint detection faces important offsets and jitter [START_REF] Shu | Application of extended kalman filter for improving the accuracy and smoothness of kinect skeletonjoint estimates[END_REF] [START_REF] Wang | Evaluation of pose tracking accuracy in the first and second generations of microsoft kinect[END_REF]. In our protocol, we estimated, at 5 feet, the maximum jitter to be about 2.5 cm. Consequently, we estimated the elbow angle's noise to be around ±5 • for a 35 cm elbow-hand bone.

Similarly, the measure of the wrist pronosupination might suffer of equivalent noise, as well as the fact that subjects, between the compensated and natural posture, do not exactly orient their hand the same way; to solve this, the end-effector constraint f qm is set as the measurement of the natural posture. The same observation occurs for the bone's length that varies between measures; therefore, during optimization, it is set as the natural posture lengths. And, finally, the base of the kinematic model being the position of the pelvic, for the same reason, we set the base of the optimized model to be the base of the natural kinematic model.

Joint offsets are strongly undesired for the pelvic joint which defines the base of our kinematic model, and strongly impact the back's leaning angle, which is the most weighted criteria of our optimization. In sitting position, the pelvis measured position is always overshot by a few centimeters, and consequently the trunk angle is over-estimated. Even when leaning backwards, the trunk angle is almost always positive. To compensate that offset, we do not set the trunk leaning angle desired q d,2 at 0 • but it is instead set as the mean angle of the natural posture for each subject (out of the 18 natural postures). That mean trunk angle q nat,T is, depending on the subject, between 5 and 18 • . The same thing is performed for the scapula angle q nat,B as the natural scapula elevation detected by the Kinect strongly depends on the subject physiognomy and clothing. The desired angles and the dead zones Z chosen are summed up in Table III

q d 0 q nat,T q nat,B 80 • Z 0 0 0 ±20 •

IV. EXPERIMENTAL RESULTS

A. Postural errors between conditions

The RMSE of the absolute errors between the "Natural" and "Compensated" measured postures (Nat-Comp) and between the "Natural" and the computed "Optimum" postures (Nat-Opt) are shown in the Table IV, which indicates that, globally, the Optimum postures are closer to the Natural ones, than the Compensated ones do.

B. Distribution of errors

Figure 4 shows the histograms of errors, comparing the error between the "Natural" and "Compensated" measured postures (in orange) and the one between the "Natural" and the computed "Optimum" postures (in blue) for the trunk, scapula elevation, elbow joint angle and wrist joint angle. The histogram indicates that the distribution of the errors between Natural and Optimum postures are unimodal, centered on zeros and with a limited spread, clearly showing that the Optimum computed postures are closer to the Natural ones than the compensated ones. Looking at the Natural-Compensated error (orange), it can be noticed that for the elbow, the distribution looks like the sum of two Gaussian distributions, one representing the flexion and one the extension of the elbow that the experimenter asked the subjects to perform. Similar observations should be seen for the wrist, with pronation and supination, but with a flatter distribution as subjects tend to unintentionally move their wrist while trying to reach the targets. This also explains why no errors were sometimes observed between Natural and Compensated postures while a compensatory movement was asked. Finally, the trunk distribution shows that subjects prefer to lean forward than backward.

C. Relative evaluation of errors

Figure 5 presents the error between Natural and Optimum postures (Opt-Nat) as a function of the error between Natural and Compensated (Comp-Nat) ones. Oppositely to the histograms, these figures show paired input and output errors Fig. 4: From left to right: Histograms of the elbow, wrist, trunk and scapula angle error, before optimization (in orange, the error between natural and compensated) and after optimization (in blue, the error between natural and optimum) of the optimisation, as we want each posture to be improved, and not just a general improvement of the mean error. On the proposed figure, a postural improvement for a given posture is defined by a point placed below the axis defined by y = x (dashed black lines). The green zones denotes an improvement of the joint position for the given joint. The red zone denotes a deterioration. We also plotted margins of ±5 • (black lines) that represent the estimated noise due to the Kinect. Therefore, points inside these margins may be considered as neither improved nor deteriorated. Finally, Figure 6 presents the same data but computed for the sum of the absolute errors of the 4 considered angles : trunk, scapula, elbow and wrist. This figure can be seen as a global classifier for detecting compensatory motions. Indeed, when a subject does not compensate, the Natural-Compensated and Natural-Optimised errors are equal and close to 0 • (less than 5 or 10 • for example, depending on the chosen threshold), while otherwise there is an important difference.

Out of the 270 postures, only 4 postures were misclassified (in the red zone Figure 5. for the elbow. These misclassified points have a null error Natural-Compensated regarding the elbow angle, meaning the algorithm identified high compensations on the other angles and cancelled them while increasing the elbow error. While only 2 postures were globally misclassified with a 5 • threshold (Figure 6), these postures have a global error Natural-Compensated close to 0 • , meaning the subject actually did not compensated while asked. Finally, it can be seen that every postures with a global compensated-natural error of more than 20 • were correctly classified, meaning the global detection of compensatory motions was effective.

V. DISCUSSION

The experimental results indicates that the proposed nullspace optimization approach with its RULA-inspired score was able, for a given task (i.e. hand positioning and orienting), to compute, from a typical erroneous posture, a postural solution close to the natural and comfortable strategy exhibited by users. For all angles, the global RMSE was highly improved comparing before and after optimization. The method appeared to be effective on every tested posture and not only with a limited set of them as shown in Fig. 5.

A. Extension to more complex models

This results were obtained with a 9 DoF kinematic model, and a 6 DoF task, thus within a 3 DoF redundancy problem to optimize. As seen, we used important weights for the 3 DoF pelvic joint, and, in this configuration, converging to the 3 correct trunk angles should be sufficient to have a null error between the optimum and the natural position. This means that, in this specific case, there is only one viable strategy to reach the target without compensations. If we were to increase the complexity of the kinematic model with more DoF, the choice of the weight would be more critical to converge towards the natural posture within a larger set of solutions. For example, the weights of the optimization cost-function should probably be tuned more carefully for a 4-DoF prosthesis (with a spherical wrist), as the redundancy of the kinematic model would require 5 angles to be correctly estimated through optimization instead of 3 (for example, the flexion and abduction of the wrist should be close to 0 • ) but we think that this method could correctly be implemented on a 4-DoF prosthesis.

B. Kinect limitations

It should also be mentioned that optimum that is computed strongly depends on the end-effector position and orientation measured f (q m ). The noise of the wrist, palm and thumb position strongly impacting f (q m ) and therefore considering the global convergence for a spherical wrist, the Kinect might not be precise enough. However, for a 1 DoF wrist model, the Kinect has proven to be precise enough in our experiment. Similarly, it is important to highlight that the Kinect noise affects the measured lengths of the bones from one posture to another (as previously mentioned in II.E). These lengths variations impact the determined joint position q. The natural and optimized joint positions are calculated for the same bone lengths, while the bone lengths of the non-natural posture vary by a couple of centimeters. Consequently, because of such difference, the error improvement should be nuanced. However, the results still remain significant and the visual postural improvement of the kinematic model is flagrant, as shown in Figure 3. Nevertheless, RGB-D cameras might be viable for medical rehabilitation applications, as it is quite cheap and could Fig. 5: The relative error (from left to right and top to bottom: elbow, wrist, trunk, scapula) after optimization as a function of the error before optimization. The dark dashed-dotted line represents the y = x and y = -x axis, and the thin black lines the ±5 • margin of our estimated noise. An improve point prediction is located in the green zone, between the y = x and y = -x axis. The closer it is to the horizontal axis, the better the prediction is. easily be set at patients' home, as well as industrial context in front of a worker's bench. The Kinect noise might even be reduced for real-time applications by using an Extended Kalman Filter [START_REF] Shu | Application of extended kalman filter for improving the accuracy and smoothness of kinect skeletonjoint estimates[END_REF]. However, for daily-life prosthetic implementations, the algorithm should be transposed to alternative technologies, such as Inertial Measurement Units to estimate the subjects postures for a use outside of a laboratory environment.

C. Computational power

The optimization algorithm being linear, it does not require high computational power: on a generic Intel core i5 laptop running non-optimized Matlab code, the optimization timing was about 10ms (30 iterations) for a 0.1 • of precision error set as stopping criteria ε. Additionally in a real-time application to a prosthesis control such as in [START_REF] Legrand | Controlling robotic assistive devices with natural body compensations[END_REF], since the optimization initial guess would be set to the posture obtained for the previous computation step, the error to cancel would be reduced and so would be the number of optimization iterations.

VI. CONCLUSION

In this study we presented a simple approach to search for an alternative posture that optimizes an ergonomic postural score within the null space of a human kinematic chain. The results of a preliminary experimental campaign on a group of non-amputated subjects (emulating compensatory movements similar to the ones exhibited by amputated users with inadequate prosthesis position) indicates that this method is effective and able to precisely determine more ergonomic and natural solutions for the distal joints (wrist and elbow) angles as well as for the trunk and scapula. Unlike existing approaches, this was achieved with a simple extension of a discrete ergonomic score which is not task nor user specific, do not require any training data, and is computationally much more efficient that dynamic manikin based simulations as well as easier to parameterize.

While further work has to be conducted for this method to be implemented in realistic scenarios exploiting wearable motion sensors to reconstruct postural data, the proposed approach paves the way for multiple DoF prosthesis control. Indeed, it could be used as a joint error computation stage within a movement-based compensation cancellation control approach described in [START_REF] Legrand | Controlling robotic assistive devices with natural body compensations[END_REF]. Further, this method could also be used to quantify compensatory motions in industrial or medical context, possibly extending existing discrete ergonomic scores.

Fig. 2 :

 2 Fig. 2: View of the experimental setup used: a subject (with his measured skeleton added in red), performing a compensatory movement (while maintaining his hand position and orientation) in front of the Kinect V2; A visual interface indicates the hand position/orientation to reach and displays the subject's skeletal model, from different perspectives

Fig. 6 :

 6 Fig. 6: Absolute error after optimization as a function of the error before optimization for the sum of the trunk, scapula, elbow and wrist angle. The dark dashed-dotted line represents the y = x axis, and the thin black lines the ±5 • margin of our estimated noise.

TABLE I :

 I Denavit-Hartenberg parameters of the torso and right arm kinematic model Denavit-Hartenberg parameters Joint Trunk 1 Trunk 2 Trunk 3 Scapula Shoulder 1 Shoulder 2 Shoulder 3 Elbow Wrist

TABLE II :

 II Chosen optimization weights

			Optimization weights for each joint
	Joint	Trunk 1,3 Trunk 2	Scapula Shoulder 1,2,3 Elbow Wrist
	M	100	300	100	0	10	0

TABLE III :

 III Chosen Desired angle in •

		Desired angles for each joint
	Joint	Trunk 1,3 Trunk 2	Scapula Elbow

TABLE IV :

 IV RMSE of the absolute errors RMSE of the absolute errors in •

	Joint	Trunk 2	Scapula	Elbow	Wrist
	Nat-Comp	10.3354	8.4725	26.4834	12.8714
	Nat-Opt	2.6812	3.7261	4.6764	3.4311

ACKNOWLEDGMENT

The authors would like to thank all the participants of the experiment. This work was carried out in accordance with the recommendations of the University Paris Descartes ethic committee CERES.

*This work was supported by ANR BYCEPS, ANR-18-CE19-0004.


Dominique Fohr
email: dominique.fohr@loria.fr

Irina Illina
email: irina.illina@loria.fr

BERT-based Semantic Model for Rescoring N-best Speech Recognition List

Keywords: automatic speech recognition, semantic context, embeddings, BERT

This work aims to improve automatic speech recognition (ASR) by modeling long-term semantic relations. We propose to perform this through rescoring the ASR N-best hypotheses list.

To achieve this, we propose two deep neural network (DNN) models and combine semantic, acoustic, and linguistic information. Our DNN rescoring models are aimed at selecting hypotheses that have better semantic consistency and therefore lower WER. We investigate a powerful representation as part of input features to our DNN model: dynamic contextual embeddings from Transformer-based BERT. Acoustic and linguistic features are also included. We perform experiments on the publicly available dataset TED-LIUM. We evaluate in clean and in noisy conditions, with n-gram and Recurrent Neural Network Language Model (RNNLM), more precisely Long Short-Term Memory (LSTM) model. The proposed rescoring approaches give significant WER improvements over the ASR system without rescoring models. Furthermore, the combination of rescoring methods based on BERT and GPT-2 scores achieves the best results.

Introduction

ASR systems have made significant progress in recent years. Classical ASR systems only take into account acoustic, lexical, and syntactic information (local n-gram language models (LM)). When conditions between training and testing differ, like noisy environments, the audio signal is distorted, and the acoustic model may not be able to compensate for this variability. Even if noise compensation methods work well [START_REF] Li | An Overview of Noise-robust Automatic Speech Recognition[END_REF], it is of interest to incorporate semantic knowledge into the decoding process to help the ASR better account for the longterm semantic context and furthermore to combat adverse conditions. This improvement should also be useful when training and testing conditions match, since semantic information is important for ASR systems.

Some studies have tried to include this information into ASR. The authors of [START_REF] Sheikh | Modelling Semantic Context of OOV Words in Large Vocabulary Continuous Speech Recognition[END_REF] use a semantic context for recovering proper names missed in the ASR process. [START_REF] Bayer | Semantic Language Models for Automatic Speech Recognition[END_REF] integrate semantic frames and target words into recurrent neural network LM. In [START_REF] Corona | Improving Black-box Speech Recognition using Semantic Parsing[END_REF] the re-ranking of the ASR hypotheses using an in-domain LM and a semantic parser significantly improves the accuracy of the transcription and semantic understanding. Furthermore, [START_REF] Gaspers | Semantic Parsing of Speech using Grammars Learned with Weak Supervision[END_REF] introduce semantic grammars applicable for ASR and understanding using ambiguous context information.

Studies have shown that rescoring the ASR N-best hypotheses list can be an efficient solution to incorporate longrange semantic information. [START_REF] Song | L2RS: a Learning-to-rescore Mechanism for Automatic Speech Recognition[END_REF] formalize the N-best list rescoring as a learning problem and use a wide range of features with automatically optimized weights. [START_REF] Ogawa | Rescoring Nbest Speech Recognition List Based on One-on-One Hypothesis Comparison Using Encoder-Classifier Model[END_REF] [START_REF] Ogawa | Improved Deep Duel Model for Rescoring N-best Speech Recognition List Using Backward LSTMLM and Ensemble Encoders[END_REF] introduce Nbest rescoring through an LSTM-based encoder network followed by a fully-connected feed-forward NN-based binaryclass classifier. [START_REF] Shin | Effective Sentence Scoring Method Using BERT for Speech Recognition[END_REF] propose a bi-directional LM for rescoring, and utilize the word prediction capability of the BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] [START_REF] Wang | BERT has a Mouth, and it Must Speak: BERT as a Markov Random Field Language Model[END_REF].

In this work, we aim to add long-range semantic information to ASR through rescoring the N-best hypotheses list. We believe that some ASR errors can be corrected by taking into account distant contextual dependencies, which is important for noisy conditions. We theorize that in noisy parts of speech, the semantic model (SM) will help remove acoustic ambiguities. The core ideas of the proposed rescoring approaches are as follows. First, we use a continuous SM to represent each hypothesis: BERT model. Different semantic properties and efficiencies of BERT motivated us to explore it for our task. Second, we compare ASR hypotheses two per two and propose two BERT-based models. Finally, we propose efficient DNN architecture to train together semantic, acoustic, and linguistic information. The obtained score is combined with the ASR scores attached to each hypothesis (acoustic and linguistic) and used to rescore the ASR N-best list. Regarding [START_REF] Kumar | Use of Knowledge Graph in Rescoring the N-best List in Automatic Speech Recognition[END_REF], where semantic relations between entities are extracted from DBpedia and used as features for rescoring, we use transformer BERT model that are pre-trained on large and diverse text corpora. Compared to [START_REF] Ogawa | Rescoring Nbest Speech Recognition List Based on One-on-One Hypothesis Comparison Using Encoder-Classifier Model[END_REF][14], we use a more powerful model with several transformer layers. Compared to [START_REF] Shin | Effective Sentence Scoring Method Using BERT for Speech Recognition[END_REF], where masked word prediction is performed for BERT, we use the sentence prediction capability of the BERT model. Compared to our previous work [START_REF] Level | Introduction of Semantic Model to Help Speech Recognition[END_REF], we use the BERT SM to represent the hypotheses at the sentence level, and train hypotheses representations by a DNN. In experiments using a publicly available speech corpus, we systematically explore the effectiveness of the proposed features and their combinations. The proposed approaches steadily outperform the baseline ASR system in clean and all noisy conditions. The proposed approaches are competitive compared to GPT-2 rescoring. This research work was carried out as part of an industrial project.

Proposed methodology 2.1. Introduction

A classical speech recognition system provides an acoustic score Pac (w) and a linguistic score Plm (w) for each of the hypothesized words w of the utterance to recognize. The best sentence hypothesis is the one that maximizes the likelihood of the word sequence:

𝑊 ̂= 𝑎𝑟𝑔𝑚𝑎𝑥 ℎ 𝑖 𝜖𝐻 ∏ 𝑃 𝑎𝑐 (𝑤) 𝛼 * 𝑃 𝑙𝑚 (𝑤) 𝛽 𝑤𝜖ℎ 𝑖 (1)
𝑊 ̂ is the recognized sentence (the end result); H is the set of Nbest hypotheses; hi is the i-th sentence hypothesis; w is a hypothesized word. α and β represent the weights of the acoustic and language models.

An efficient way to take into account semantic information is to re-evaluate (rescore) the best hypotheses of the ASR system. We propose to introduce for each hypothesis hi the semantic probability Psem(hi) to take into account the semantic context of the sentence. In our rescoring approach, Pac (hi), Plm (hi), and the semantic score Psem(hi) are computed and combined using specific weights α, β and γ (for Psem(hi)) for each hypothesis:

𝑊 ̂= 𝑎𝑟𝑔𝑚𝑎𝑥 ℎ 𝑖 𝜖𝐻 𝑃 𝑎𝑐 (ℎ 𝑖 ) α * 𝑃 𝑙𝑚 (ℎ 𝑖 ) β * 𝑃 𝑠𝑒𝑚 (ℎ 𝑖 ) γ (2)
We propose to rescore using a pair of ASR hypotheses, one at a time. We use hypothesis pairs to get a tractable size of the rescoring DNN input vectors. Each hypothesis of each pair is represented by semantic information, produced by proposed DNN rescoring models based on BERT representation. We also explored the word2vec model [START_REF] Mikolov | Distributed Representations of Words and Phrases and their Compositionality[END_REF] but given its poor performance, we have not included its results in this article.

Furthermore, we propose to go beyond a simple score combination, like in eq. ( 2). We propose two DNN-based rescoring models producing Psem(hi): (a) the first model, called BERTsem, is purely semantic and only uses textual information as input; (b) the second model, called BERTalsem, takes acoustic, linguistic, and textual information as the input. We believe that the acoustic and linguistic information should be trained together with the semantic information to give an accurate rescoring model.

DNN-based rescoring models

Our proposed DNN models use a pair of hypotheses. For each hypothesis pair (hi, hj), the expected DNN output v is: (a) 1, if the WER of hi is lower than the WER of hj ; (b) otherwise, 0.

The overall algorithm of the N-best list rescoring is as follows. For a given sentence, for each hypothesis hi we want to compute the cumulated score scoresem(hi). To perform this, for each hypothesis pair (hi, hj) of the N-best list of this sentence:

 we apply the DNN model and obtain the output value vij (between 0 and 1). A value vijs close to 1 means that hi is better than hj. We use this value to compute the scores for these hypotheses.  we update the scores of both hypotheses as:

scoresem(hi) += vij; scoresem(hj) += 1-vij

The obtained cumulated score scoresem(hi) is used as a pseudo probability Psem(hi) and combined with the acoustic and linguistic likelihoods with a proper weighting factor (to be optimized) according to eq. ( 2). In the end, the hypothesis that obtains the best score is chosen as the recognized sentence.

Our two proposed DNN-based rescoring models producing Psem(hi) are based on BERT, which is a multi-layer bidirectional transformer encoder that achieves state-of-the-art performances for various natural language tasks (NLP). The pre-trained BERT model can be fine-tuned using task-specific data [START_REF] Turc | Well-Read Students Learn Better: On the Importance of Pre-training Compact Models[END_REF]. Since the cosine distance is not meaningful for BERT SMs [25][26], we compute the semantic information at the sentence level, as described below.

In our approach, we employ a pre-trained BERT model. Two methods can be used to fine-tune BERT using applicationspecific data: masked LM and next sentence prediction. We base our BERT fine-tuning on a task similar to the latter.

The first proposed model, BERTsem,, consists of performing the fine-tuning of BERT (with a fully connected layer at the top of BERT) using only embeddings of CLS tokens (the first token of a sentence, used for sentence classification). During the training, we input a hypothesis pair (hi, hj), that we want to compare, and the output is set to 1 (or 0) if the first (or the second) hypothesis achieves the lowest WER.

The second proposed model, BERTalsem, takes input as feature vectors which include acoustic, linguistic, and textual information. Figure 1 shows the architecture of this model: the text of the hypothesis pair is given to the BERT model. Then, the embedding token of BERT, representing this pair, is given to the tra, followed by max pooling and average pooling, and then by a fully connected layer (FC) with a ReLU (Rectified Linear Unit) activation function. Finally, the output of this FC is concatenated with the acoustic and linguistic information of the hypothesis pair and passed through the second FC layer followed by a sigmoid activation function (to obtain a value between 0 and 1). In the end, the output vij is obtained. In this setting, we use a one-layer Bi-LSTM and two layers of FC. More complex DNN architectures can also be considered. Advantage of this model is that the weights of acoustic, linguistic, and semantic information are learned together to provide a more powerful model.

Experimental conditions

Corpus description

For this study, we use the publicly available TED-LIUM corpus [START_REF] Fernandez | TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation[END_REF], which contains recordings from TED conferences. Each conference, within the corpus, is focused on a particular subject, so the data is well suited to our study. We use the train, development, and test partitions provided within the TED-LIUM corpus: 452 hours for training (268k segments), 8 conferences (507 segments) for development, and 11 conferences (1155 segments) for the test set (see Table 1). As usual, we apply the development set to choose the best parameter configuration, and the test set to evaluate the proposed methods with the best configuration. We compute the WER to measure the performance. Since our model only compares two hypotheses and cannot estimate the word probabilities, it is not possible to calculate the perplexity of our model. Therefore, in this article, we will not be providing any results related to perplexity.

This research work was carried out as part of an industrial project, studying the recognition of speech in noisy conditions, more precisely in fighter aircrafts. Then, we add noise to the development and test sets to get closer to the actual conditions of an aircraft: noise added at 10 dB and 5dB SNR (noise of an F16 from the NOISEX-92 corpus [START_REF] Varga | Assessment for automatic speech recognition II. NOISEX-92: A Database and an Experiment to Study the Effect of Additive Noise on Speech Recognition Systems[END_REF]). The noise is not added to the training set. Furthermore, we evaluate the proposed approaches in clean conditions (training and testing). We also perform N-best list generation using the RNNLM model (LSTM) [11] [21]. Our objective for this is to verify if, by using a more powerful LM, the proposed rescoring models can improve the ASR. In all experiments, during N-best rescoring, the LM (4-gram or RNNLM) is not modified.

Rescoring models

According to our previous work on semantic models [START_REF] Level | Introduction of Semantic Model to Help Speech Recognition[END_REF], the use of 5 or 10 hypotheses of the N-best list is not for efficient rescoring. Using more than 25 hypotheses shows no further improvement. In this study, we chose to use an N-best list of 20 hypotheses in all our experiments. Moreover, this size of N-best lists seems to be reasonable to generate the pairs of hypotheses and to have a tractable computational load during the training. In the case that a larger sized N-best list was to be required, a different pair comparison strategy would have been used [START_REF] Ogawa | Rescoring Nbest Speech Recognition List Based on One-on-One Hypothesis Comparison Using Encoder-Classifier Model[END_REF]. During the training, the hypothesis pairs that get the same WER are not used. During evaluation (with development and test sets), hypothesis pairs are considered, because we don't have the word error rate for these hypotheses.

For all experiments, combination weights are: α=1, β is between 8 and 10, and γ is between 80 and 100. For each model, the weight values performing the best N-best rescoring performance for the development data were selected as the optimal value for the test data.

For BERTx models, we downloaded the pre-trained BERT model provided by Google with 110M parameters, 12 layers, and the size of the hidden layers at 768 [START_REF] Turc | Well-Read Students Learn Better: On the Importance of Pre-training Compact Models[END_REF]. For the BERTsem fine-tuning three epochs are performed. For BERTalsem we use Adam optimizer and binary cross-entropy loss function. We iterate the training for four epochs: during the first two epochs the weights of BERTsem are frozen, during the last two epochs all weights are updated. The dropout is 30 %. We train the proposed models using all training set.

As the Generative Pre-Training Transformer 2 model (GPT-2) showed good performance in several NLP tasks [START_REF] Radford | Language Models are Unsupervised Multitask Learners[END_REF], we applied this model in our experiments. The pre-trained GPT-2 LM was downloaded from the Hugging Face site. This model contains 117M parameters and was trained by OpenAI on 40GB of Internet text. This model is used in our experiments as the LM during the N-best rescoring (instead of n-gram). We also performed similar experiments using Masked Language Model (MLM) [START_REF] Salazar | Masked Language Model Scoring[END_REF]. The results are less good than those of GPT-2 and are not presented here for lack of space.

Experimental results

We investigated the different hyperparameters of the proposed models. We can say that for the BERT-based rescoring models, it is important to use a large corpus of training (millions of pairs of hypotheses) and to choose a model with many hidden layers (we tested 4, 8, and 12 layers). For lack of space, we do not give these results in this article.

We report the WER for the development and the test sets of TED-LIUM with clean speech and in noise conditions of 10 and 5 dB. In Tables 2 and3, the first line of results (method Random), corresponds to the random selection of the recognition result from the N-best hypotheses without the use of the proposed rescoring models. The second line of the Tables (method Baseline), corresponds to WER performance without using the rescoring models (standard ASR). The last line of the Tables (method Oracle) represents the maximum performance that can be obtained by searching in the N-best hypotheses: we select the hypothesis, which minimizes the WER for each sentence. The other lines of the Tables give the performance of the proposed approaches.

To fairly compare the proposed transformer-based models to other state-of-the-art transformer-based models introducing long-range context dependencies, we experiment with a rescoring based on the GPT-2 model. It corresponds to the rescoring of N-best hypotheses according to two configurations: (a) using eq. ( 1) with Plm (h) given by the GPT-2, instead of n-gram, while the SM is not used (GPT2 comb. with ac. scores in Tables); (b) using eq. ( 2) with Psem (h) given by the BERTalsem model and Plm (h) given by the GPT-2, instead of the n-gram (BERTalsem comb. with ac./GPT-2 scores in Tables).

For our rescoring models, we study three configurations:  Rescoring using only the scores scoresem(h) computed with BERT-based rescoring methods (denoted BERTx in Tables). In this case, in eq. ( 2) α=0, β=0, and γ=1.  Rescoring using a combination of the BERT-based scoresem(h), and the acoustic score Pac (h) (BERTx comb. with ac. scores in Tables). In this case, scoresem(h) is used as a pseudo probability and multiplied to the acoustic likelihood with a proper weighting factor γ. Plm (hi) is not used in this combination, namely in eq. ( 2) β=0.  Rescoring using a combination of the BERT-based score, the acoustic score Pac (h) and the linguistic score Plm (h) (BERTx comb. with ac./x scores, in Tables). For the most efficient BERTalsem model, we also use the GPT-2 score as a linguistic score to combine as described above. From Table 2 we can observe that for all conditions and all evaluated rescoring models, the proposed rescoring models outperform the baseline system. This shows that the proposed Transformer-based rescoring models are efficient at capturing a significant proportion of the semantic information. Combining the acoustic score with the BERTsem model (BERTsem comb. with ac. scores in Tables) improves the performance. Indeed, the acoustic score is an important feature and should be taken into account. On the other hand, combining the linguistic score alone with the BERT rescoring gives no improvement compared to the BERT model. We do not present this result in the Tables. Google's BERT model, trained on billions of sentences, probably captures the linguistic structure of the language better than a simple n-gram LM trained on a much smaller corpus. Using the linguistic and acoustic scores with the BERT rescoring model (BERTsem comb. with ac./4gram scores) brings small additional improvement. From these results, for BERTalsem we decided to combine the acoustic and linguistic scores. We observe that in all cases, BERTalsem, trained with acoustic and linguistic scores, provides further large WER reductions compared to BERTsem.. BERTalsem combined with the GPT-2 model, a widely-used robust model, brings a significant improvement compared to the rescoring using GPT-2 comb. with ac. scores (denoted by "*" in Tables). This indicates that BERTalsem can efficiently incorporate semantic and acoustic information, while remaining competitive, and bring complementary information compared to the GPT-2 model. For BERT-based results, all improvements are significant compared to the baseline system (confidence interval at 5% significance level is computed according to the matched-pairs test [START_REF] Gillick | Some Statistical Issues in the Comparison of Speech Recognition Algorithms[END_REF]). On the test set, BERTalsem comb. with ac./4-gram scores achieves 8 % (for 5 dB), 13 % (for 10 dB), and 21 % (for clean) relative WER compared to the baseline system. Compared to the GPT-2 comb. with ac. scores, BERTalsem comb. with ac./GPT-2 scores allow us to obtain additional significant improvements.

As n-gram LM is limited in its ability to model long-range dependencies, we performed the ASR experiments using the more powerful RNNLM (LSTM). Table 3 reports the results for the same set of experiments, but, instead of n-gram, the RNNLM (LSTM) is used to generate the N-best hypotheses. The proposed rescoring BERT-based methods give consistent improvements compared to the n-gram LM results. All previous observations are valid for RNNLM-based experiments. The best system (BERTalsem comb. with ac./RNNLM scores) gives between 7 % and 14 % of relative improvement on the test set compared to the baseline system. These improvements are also significant. Compared to the 4-gram results, the improvements of RNNLM are smaller, but this is due to the fact that RNNLM can take into account more distant dependencies than n-gram.

Conclusions

In this article, we focus on the task of improving automatic speech recognition in clean and noisy conditions. Our methodology is based on taking into account semantics through powerful representations that capture the long-term relations of words and their contexts. The semantic information of the utterance is taken into account through a rescoring module on ASR N-best hypotheses. We proposed two effective DNN approaches based on the BERT model: one approach uses BERT-fine-tuning and represents a purely SM. The second approach uses a DNN and BERT models trained using semantic, acoustic, and linguistic information. On the corpus of TED-LIUM conferences, the system BERTalsem with ac./LM scores achieves between 7% and 21% of relative improvement compared to the baseline system. These improvements are statistically significant for all evaluated (clean and noisy) conditions and the two LMs: n-gram and RNNLM (LSTM). BERTalsem remains competitive compared to GPT-2 rescoring, and the best performance is obtained by BERTalsem with ac./GPT-2 scores. Future work will include the introduction of an attention mechanism and context information beyond the utterance level [START_REF] Futami | Distilling the Knowledge of BERT for Sequence-to-Sequence ASR[END_REF].

Figure 1 :

 1 Figure 1: Proposed BERTalsem rescoring model.

Table 1 :

 1 The statistics of the TED-LIUM dataset.We use a recognition system based on the Kaldi voice recognition toolbox[START_REF] Povey | The Kaldi Speech Recognition Toolkit[END_REF]. TDNN triphone acoustic models are trained on the training part (without noise) of TED-LIUM using sMBR training (State-level Minimum Bayes Risk). The lexicon and LM were provided in the TED-LIUM distribution. The lexicon contains 150k words. The LM has 2 million 4-grams and was estimated from a textual corpus of 250 million words.

	Data	Nbr. of talks	Nbr. of words Duration
	Train	2,351	4.8M	452h
	Development	8	17,783	1h36
	Test	11	27,500	2h37
	3.2. Recognition system description

Table 2 :

 2 ASR WER (%) on the TED-LIUM development and test sets, SNR of 10 and 5 dB, 20-best hypotheses, 4-gram LM. "*" denotes significantly different result compared to "GPT-2 comb. with ac. scores" configuration.

	Methods/systems	SNR 5 dB Dev. Test	SNR 10 dB Dev. Test	no added noise Dev. Test
	Random system	33.5	41.3	16.9	22.9	10.6	12.1
	Baseline system	32.7	40.3	15.7	21.1	8.7	8.9
	GPT-2 comb. with ac. scores	30.0	37.1	13.1	17.9	6.8	7.3
	BERTsem	31.1	38.7	14.4	19.8	8.0	8.7
	BERTsem comb. with ac. scores	30.6	37.9	14.2	19.4	7.9	8.6
	BERTsem comb. with ac./4-gram scores	30.6	37.9	14.1	19.4	7.8	8.5
	BERTalsem	30.4	37.5	13.5	18.6	6.9	7.3
	BERTalsem comb. with ac./4-gram scores	30.2	36.9	13.4	18.3	6.8	7.0
	BERTalsem comb. with ac./GPT-2 scores	29.7*	36.6*	12.8*	17.5*	6.4*	6.6*
	Oracle	27.5	33.2	11.2	15.0	5.2	4.7

Table 3 :

 3 ASR WER (%) on the TED-LIUM development and test sets, SNR of 10 and 5 dB, 20-best hypotheses, RNNLM (LSTM). "*"denotes significantly different result compared to "GPT-2 comb. with ac. scores" configuration.

	Methods/systems	SNR 5 dB Dev Test	SNR 10 dB Dev Test	no added noise Dev Test
	Random system	29.2	38.4	13.9	20.2	8.9	10.8
	Baseline system	28.2	37.1	12.3	17.7	6.6	7.2
	GPT-2 comb. with ac. scores	26.2	34.9	11.0	15.9	6.1	6.7
	BERTsem	27.0	35.9	12.0	17.4	7.1	8.1
	BERTsem comb. with ac. scores	26.6	35.3	11.6	17.1	6.9	7.1
	BERTsem comb. with ac./RNNLM scores	26.5	35.4	11.5	16.9	6.0	6.6
	BERTalsem	26.1	35.2	11.0	16.4	5.9	6.6
	BERTalsem comb. with ac./RNNLM scores	25.9*	34.5*	10.8*	15.9	5.6*	6.1*
	BERTalsem comb. with ac./GPT-2 scores	25.5*	34.4*	10.4*	15.4*	5.4*	5.7*
	Oracle	23.1	30.2	8.3	12.1	3.8	3.5

Acknowledgments

The authors thank the DGA (Direction Générale de l'Armement, part of the French Ministry of Defence), Thales AVS and Dassault Aviation who are supporting the funding of this study and the "Man-Machine Teaming" scientific program in which this project is taking place.