Loulou Kosmala
email: loulou.kosmala@sorbonne-nouvelle.fr

Prismes Ea

On the Multifunctionality and Multimodality of Silent Pauses in Native and Non-native Interactions

This paper explores the multifunctional uses of silent pauses, adopting a multimodal perspective. Pauses are said to serve numerous functions in speech, and this small comparative study looks at their distribution across L1 and L2 speakers in face-to-face interactions. While overall results showed a higher rate of pauses in L2 than in L1, a lot of individual differences were also found. The visual-gestural features of discourse (gesture, gaze, facial expressions) were found to be essential when analyzing pauses as they can reflect their multifunctionality in a different modality, which allows for a deeper understanding of their internal processes in L1 and L2.

INTRODUCTION

When we speak spontaneously, we constantly need to produce pauses for a variety of reasons. From a strictly physiological point of view, pausing is necessary to allow us to breathe and inhale between cycles of speech; but pauses can also serve several functions in discourse, such as planning and structuring speech, marking a word, holding the floor, or allowing other people to take the turn. This paper focuses on the use of silent pauses, i.e. "silent periods between vocalizations" [START_REF] Cenoz | Pauses and Communication Strategies in Second Language Speech[END_REF].

Silent pauses have been studied from several perspectives: they can be viewed as hesitation or disfluency markers [START_REF] Goldman-Eisler | The predictability of words in context and the length of pauses in speech[END_REF][START_REF] Maclay | Hesitation phenomena in spontaneous English speech[END_REF] which suspend the speech flow and occur when speakers are uncertain or thinking what to say next. In non-native speech, L2 speakers tend to produce more pauses of longer duration and in mid-clause position before low-frequency words [START_REF] Tavakoli | Pausing patterns: Differences between L2 learners and native speakers[END_REF][START_REF] De | Predicting pauses in L1 and L2 speech: The effects of utterance boundaries and word frequency[END_REF] which can be explained by their limited proficiency of the language [START_REF] Cenoz | Pauses and Communication Strategies in Second Language Speech[END_REF]. However, pauses can also be viewed as important components of interaction, as they serve several communicative functions. In Conversation Analysis, a delay in speech can be interpreted as meaningful as it can be a sign of a dispreferred answer in assessments [START_REF] Roberts | The interaction of inter-turn silence with prosodic cues in listener perceptions of "trouble" in conversation[END_REF], and pauses can also be used to manage turn-taking [START_REF] Schegloff | A simplest systematics for the organization of turn-taking for conversation[END_REF]. The use of pauses is also affected by several factors, such as individual variation, speaking style, and the type of speech produced [START_REF] Duez | Silent and non-silent pauses in three speech styles[END_REF].

Grounded in a functional, dynamic, and interactional approach to grammar [START_REF] Mondada | Pour une linguistique interactionnelle[END_REF], this paper aims to explore the multifunctionality of silent pauses in native and nonnative interactions of French and English, following a multimodal perspective [START_REF] Kendon | Gesture: Visible action as utterance[END_REF]. A few studies have looked at the production of gestures during pauses in native and non-native conditions [START_REF] Tellier | Gesturing while pausing in conversation: Self-oriented or partneroriented?[END_REF][START_REF] Stam | The sound of silence. Why Gesture?: How the Hands Function in Speaking[END_REF], but it is still an aspect that has not been widely explored in pausing phenomena. Since pauses are known to serve a variety of functions, the visual-gestural modality of discourse (eye gaze, facial expression, manual gestures) can provide a deeper understanding of their different internal processes. The main hypotheses for this study are that (1) silent pauses occur more frequently in L2 speech than in L1, but they appear in similar functional contexts, (2) L2 speakers will make use of more gestures during pauses than L1 speakers and these gestures can reflect their multifunctionality.

METHOD

This small study of pauses was conducted on a sample of the SITAF Corpus [START_REF] Horgues | Why some things are better done in tandem[END_REF] which includes 12 video recordings of dyadic interactions between French and American students (undergraduate level) engaged in semidirected conversations (6 pairs). The speakers interacted respectively in English and in French, alternating from L1 to L2. The methodology used for this analysis is adapted from previous studies conducted on the use of (dis)fluencies [START_REF] Kosmala | A preliminary study of hesitation phenomena in L1 and L2 productions: A multimodal approach[END_REF][START_REF] Kosmala | Synchronization of (Dis)fluent Speech and Gesture: A Multimodal Approach to[END_REF] which looked at the distribution of several (dis)fluency markers (filled pauses, silent pauses, repetitions, repairs etc.). The silent pauses were extracted from the data and were coded manually according to their (a) duration in ms (400ms minimum duration threshold, following [START_REF] Tavakoli | Pausing patterns: Differences between L2 learners and native speakers[END_REF]) (b) level of combination, whether they appeared isolated, or combined with other (dis)fluency markers, and their (c) functional contexts. The term "function" here is replaced by "context" as it is difficult to tell whether a single silent pause serves one attributed function as it can always be affected by its co-occurring markers, its context of use, and the accompanying manual gestures. Four functional contexts were defined: (1) structuring-contexts in which the speaker is currently planning, structuring, emphasizing parts of their speech, which can be indicated by the presence of discourse markers (and, but, or); (2) uncertainty-contexts in which the speaker shows signs of uncertainty (e.g. frown); (3) interactive-contexts in which the speaker is engaged in the interaction and expresses their stance (e.g. gaze towards the interlocutor, interactional gesture), (4) undefined. Since pauses can also be used for physiological reasons or other reasons that are unknown or very difficult to detect, this category was used when pauses occurred in contexts which did not apply to those described above. Lastly, the gestures which occurred during pauses were also annotated, by looking at the "gesture phrase" [START_REF] Kendon | Gesture: Visible action as utterance[END_REF] whether the gestures remained in rest position (on the lap), were held in the same position, or if they were fully produced. The functional types of gestures were also annotated, mainly (1) referential gestures, which are related to the meaning conveyed in the content of speech, (2) deictic-anaphoric gestures, which place referents in time and space through pointing, (3) parsing gestures, used for emphasis or marking speech segments (production-oriented), (4) thinking gestures, metapragmatic gestures produced during communication breakdown, (5) interactional gestures, which enact a communicative action (speech act, interactional moveinteraction-oriented).

RESULTS

A total of 468 silent pauses were found in the data. On average, L2 speakers produced significantly more silent pauses than L1 speakers (p=0.0004). They produced 6,3 silent pauses per hundred words (N=288) as opposed to L1 speakers who produced 3,7 (N=180), which corroborates previous findings. Less significant differences were found in the duration of the silent pauses, as they lasted on average 739ms in L1 and 795ms in L2 (STDV 428 and 458) but a lot of individual differences were found. For example, one American speaker produced pauses of an average duration of 1197ms in his L1 and of 846ms in his L2, which does not support previous predictions [START_REF] De | Predicting pauses in L1 and L2 speech: The effects of utterance boundaries and word frequency[END_REF], so no clear-cut conclusions can be made at this point. Silent pauses were also found to occur more frequently in co-occurrence with other (dis)fluency markers in L2 (54% N=156) than in L1 (38% N=69), which shows that silent pauses often combine with other markers, such as filled pauses [START_REF] Betz | Fill the silence! Basics for modeling hesitation[END_REF][START_REF] Grosjean | Analyse des variables temporelles du français spontané[END_REF][START_REF] Merlo | Descriptive discourse: Topic familiarity and disfluencies[END_REF], but the fact that L2 speakers produced more clusters of pauses and (dis)fluency markers may indicate that they resort to different stalling mechanisms and need to fill the silence with other vocal markers.

Results also show that the pauses behaved very similarly in L1 and L2 as no significant differences (p=0.08) were found in their distribution according to the functional contexts: 44% for undefined (N=79/180 in L1; N=128/288 in L2), 7% for uncertainty (N=11/180 in L1; N=19/288 in L2), 9% for interactive (N=16/180 in L1; N=21/288 in L2), and 40% for structuring (N=71/180 in L1; N=114/288 in L2). This finding shows that a lot of pauses tend to be produced in a semi-automatic way for no clear pragmatic reasons other than just buying time in speech (44%) and for structuring and segmenting speech (40%), but it does not seem to be affected by language proficiency as the native and non-native conditions show similar behavioral patterns. But once again, a lot of individual differences were found, as some speakers produced much more pauses in contexts of uncertainty in their L2 than in their L1. Additionally, more gestures occurred with pauses in L2 speech (56% N=163) than in L1 speech (36% N=65) (p=0.01), which partially confirms the view that L2 speakers produce more gestures than L1 speakers [START_REF] Merlo | Descriptive discourse: Topic familiarity and disfluencies[END_REF], but this needs to be confirmed by looking at all the gestures in the data. L2 speakers also produced more thinking gestures in pauses (27% of the completed gestures) than L1 speakers (17%) while L1 speakers produced more parsing gestures in pauses (46% of their completed gestures, as opposed to 38% for L1 speakers). These findings point out the gestural activity found during pauses, and this will be illustrated in the following qualitative analyses.

Qualitative analyses: example from a pair

As it has often been mentioned in the literature, the use of (dis)fluency markers varies strongly per speakers [START_REF] Betz | Are we all disfluent in our own special way and should dialogue systems also be? Elektronische Sprachsignalverarbeitung[END_REF], so it is essential to look at the individual profiles of speakers when investigating the use of silent pauses. This paper will now focus on two qualitative analyses taken from one pair of the corpus. In these examples, the two participants are interacting in French and are discussing the differences between being a traveler and being a tourist.

In the first example, the non-native speaker (NNS) is talking about travelers who get to stay in a foreign country for a longer time, but she mispronounces the noun "longer" (plus longtemps). She quickly realizes her mistake, and as Fig. 1 shows, she then produces a silent pause of 580 ms, but she also holds her hands in the same position, and with her gaze fixed on her interlocutor she slightly moves her head in the direction of her partner, to elicit help. The native speaker (NS) quickly understands her partner's request, and gives her the right pronunciation of the word; then NNS repeats the target word, this time with the right pronunciation, and finishes her sentence. This example shows that NNS did not verbally seek help from her interlocutor, but she instead relied on multimodal resources (pausing, holding her hands, and gazing) to ask for the right pronunciation and to give her partner the floor, which stresses out the interactional dimension of pauses.

In the second example, NS is talking about how tourists tend to see trips as 'to-do lists' as they aim to go to one place and see different monuments. When she starts saying that tourists like to go to one place, she produces a silent pause before the noun "place" and produces a vertical movement with her right hand towards her left open palm (Fig. 2). She then repeats the same gesture after saying the word 'place' and repeats it again as she produces a lexical repetition: "this monument, this monument, this monument". The type of gesture produced in the pause is different from the one in Fig. 1 as in this case she is rather "marking the words" (see [START_REF] Stam | The sound of silence. Why Gesture?: How the Hands Function in Speaking[END_REF]), in other words, she is emphasizing key words in her utterance, as to draw attention to them. When she first initiated this gesture, it was during the silent pause, which stresses out its pragmatic use. She may have produced it to make her speech clearer for her partner (interaction-oriented) or to mark distinct speech segments (production-oriented). In any case, the gesture accompanying the silent pause was found to be informative of the ongoing processes associated with her pausing.

CONCLUSION

To conclude, this small comparative study has stressed out the multifunctionality and multimodality of silent pauses in native and non-native speech. Silent pauses can be used as strategies to resolve speech difficulties, to emphasize parts of speech, to make an interactional move, or simply to buy time. While some of them appear to be semiautomatic, it is important to take individual differences into account, as speakers make use of them in different ways to achieve different things. The multimodal approach used in this study was found to be essential when analyzing the interactional dimension of pauses in the qualitative analyses. In this view, pauses are no longer 'silent' or 'unfilled' in the multimodal sense as they can be filled with other rich semiotic resources.

Fig 1 .

 1 Fig 1. Multimodal activity during a silent pause: implicitly seeking help from the interlocutor

Fig 2 .

 2 Fig 2. Initiation of a parsing gesture during a pause: marking the words


Chia-Ju Peng

Yi-Chun Chen

Chun-Chuan Chen

Shih-Jui Chen

Barthélemy Cagneau

Luc Chassagne

An EEG-based attentiveness recognition system using Hilbert-Huang transform and support vector machine

Hilbert-Huang transform (HHT) is useful for the analysis of nonlinear or nonstationary biosignals including brainwaves. In this work, a method is proposed for the characterization of the levels of attentiveness by using electroencephalogram (EEG) signals and HHT analysis.

Single channel EEG signals over the frontal area were acquired from participants at different levels of attentiveness and were decomposed into a set of intrinsic mode functions (IMF) by empirical mode decomposition (EMD). Hilbert transform analysis was applied to each IMF to obtain the marginal frequency spectra, and then the band powers and spectral entropies (SE) were selected as the attributes entered a support vector machine (SVM) for the two-class classification. Compared with the predictive models of approximate entropy (ApEn) and fast Fourier transform (FFT), the results showed that the band powers extracted from IMF2 to IMF5 of 𝛼 and 𝛽 waves and their SE can best discriminate between attentive and relaxed states with the classification average accuracy of 84.80%. In conclusion, this integrated signal processing method is capable of attentiveness recognition and may be used in a clinical setting for the detection of attention deficit.

Introduction

Attention is an important feature that reflects the mental state of the brain and can be measured by using electroencephalography (EEG). The measurement of the degree of attention is mainly associated with α and β waves [START_REF] Klimesch | Induced alpha band power changes in the human EEG and attention[END_REF]. In particular, α waves between 8 and 13 Hz with amplitudes from 30 to 50 μV are evident on the EEG of a relaxed participant with closed eyes. The β oscillations between 14 and 30 Hz with amplitudes from 5 to 20 μV are evident during active attention. Therefore, quantifying these frequency-specific features using EEG can be used to probe the level of attentiveness [START_REF] Lubar | Evaluation of the effectiveness of EEG neurofeedback training for ADHD in a clinical setting as measured by changes in TOVA scores, behavioral ratings, and WISC-R performance[END_REF][START_REF] Hillard | Neurofeedback Training Aimed to Improve Focused Attention and Alertness in Children With ADHD: A Study of Relative Power of EEG Rhythms Using Custom-Made Software Application[END_REF].

Previous studies have shown that for EEG attentiveness recognition, using a k-nearest neighbor classifier based on the self-assessment manikin model can yield an average accuracy of 57.03% [START_REF] Li | A real-time EEG-based BCI system for attention recognition in ubiquitous environment[END_REF], and using support vector machine (SVM) model of power spectral density resulted in an average accuracy of 76.82% [START_REF] Liu | Recognizing the degree of human attention using EEG signals from mobile sensors[END_REF]. In addition, the accuracy can be increased to 81% when taking into account approximate entropy using fuzzy entropy [START_REF] Xu | Characterization and classification of EEG attention based on fuzzy entropy[END_REF].

On a single subject level, the accuracy is up to 89.4% when using the integration of common spatial pattern filtering and nonlinear mutual information method [START_REF] Hamadicharef | Learning EEG-based spectral-spatial patterns for attention level measurement[END_REF]. Taken together, frequency-specific and nonlinear features extracted from EEG are essential for attentiveness recognition. Therefore, in this study, we proposed a method for the characterization of the levels of attentiveness based on Hilbert-Huang transform (HHT) and SVM. HHT has been used to process nonlinear and nonstationary brainwave signals [START_REF] Lin | Hilbert-Huang transformation-based time-frequency analysis methods in biomedical signal applications[END_REF][START_REF] Wolpaw | Brain-computer interface technology: A review of the first international meeting[END_REF] in EEG analysis and clinical applications, such as motor imaginary [START_REF] Panoulas | Spectrum as a New Field for the Identification of EEG Event Related De-/Synchronization for BCI applications[END_REF], seizure detection [START_REF] Li | Feature extraction and recognition of ictal EEG using EMD and SVM[END_REF][START_REF] Oweis | Seizure classification in EEG signals utilizing Hilbert-Huang transform[END_REF][START_REF] Rehman | Application of Multivariate empirical mode decomposition for seizure detection in EEG signals[END_REF], and anesthesia monitoring [START_REF] Shalbaf | Using the Hilbert-Huang transform to measure the electroencephalographic effect of propofol[END_REF][START_REF] Chen | Comparison of FFT and marginal spectra of EEG using empirical mode decomposition to monitor anesthesia[END_REF]. SVM, a machine learning technique, gradually becomes a popular translation method for classification with high accuracy [START_REF] Blankertz | The BCI competition III: Validating alternative approaches to actual BCI problems[END_REF][START_REF] Garrett | Comparison of linear, nonlinear, and feature selection methods for EEG signal classification[END_REF][START_REF] Zhang | Comparing the Performance of Random Forest, SVM and Their Variants for ECG Quality Assessment Combined with Nonlinear Features[END_REF]. Given a set of training samples, SVM can build a predictive model of the specific EEG features by the supervised learning algorithm, performing a classifier for attentiveness.

In this study, we measured the brain activity over the frontal area with one channel EEG device during participants solving some puzzles shown on the screen as well as during a resting period. EEG signals were first decomposed to intrinsic mode functions (IMF) using empirical mode decomposition (EMD), after which the instantaneous frequencies of IMFs were obtained using HHT. The resulting marginal spectra (MS) of specific frequency bands and spectral entropy (SE) entered an SVM as the feature attributes for characterization of attentiveness.

Materials and methods

2-1. Data collection

EEG data were measured by using a commercial mobile EEG monitor (MindWave, NeuroSky) at a sampling rate of 512 Hz [START_REF] Sałabun | Processing and spectral analysis of the raw EEG signal from the MindWave[END_REF]. The unipolar recording device had a fixed channel position on the scalp surface of the forehead (Fp1), according to the International 10/20 system [START_REF] Jasper | Report of the committee on methods of clinical examination in electroencephalography[END_REF]. An ear clip (A1) of the device was used to provide a ground reference to filter out the electrical noise.

This study, numbered 201812EM027, was approved by the Research Ethics Committee of National Taiwan University, Taiwan. Twenty participants were recruited: eight males and twelve females, whose ages ranged from 20 to 26 (average age = 21.9). The participants were instructed to sit still in a quiet room, wearing the EEG monitor. Two tasks were conducted by the participants: (1) pay their attention on solving designated spot-thedifference puzzles shown on a screen for 5 minutes; (2) then relax with their eyes opened and fixated at a blank screen for 5 minutes. Figure 1 shows the EEG of a representative participant during the two tasks. The dashed lines indicate the onset of the rest task. The continuous EEG signal was epoched as a collection of time-locked trials with the time length of 1 second. The middle 200 epochs of each task were analyzed to build a classifying model.

As shown in Figure 1, the red block is considered as attention data (200 epochs), and the green block is relaxation data (200 epochs). Therefore, 200 attention and 200 relaxation epochs of twenty participants were analyzed to build the individualized classifiers. Besides, both of the last 50 epochs were used as the testing data to validate the models. The flowchart of this study is shown in Figure 2. The selected features of EEG data extracted by Hilbert-Huang analysis (EMD, HT, and MS) were entered the SVM for attentiveness recognition.

2-2. Hilbert-Huang Transform

HHT [START_REF] Huang | The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis[END_REF] is a time-frequency-energy method for the analysis of nonlinear or nonstationary data sets, the process of which can be divided into two parts: EMD and the Hilbert transform (HT) [START_REF] Boashash | Estimating and Interpreting the Instantaneous Frequency of a Signal-Part I: Fundamentals[END_REF]. For dealing with nonlinear and non-stationary signals and extracting the basis for the HT, EMD has been proposed to generate finite component sets empirically from original data. The repetitive extraction of EMD is based on the oscillatory modes and waveforms of signals in the time domain, assuming that any data comprises different simple intrinsic oscillatory modes. An IMF of a signal is a function with (1) the same number of zero-crossings and extrema, and with (2) symmetric envelopes defined by local maxima and minima. Meeting these conditions, IMFs form an orthogonal basis for the original signal.

The definition guarantees the HT of IMF has well behavior to analyze the instantaneous frequency benefited from its simple oscillation. The procedure of extracting an IMF is called the sifting process of EMD, which is used to remove riding waves and symmetrize the wave profiles. The sifting process should be repeated as many times as necessary to convert the extracted signal into an IMF. Thus, a signal, 𝑥(𝑡), can be represented as

𝑥(𝑡) = ∑ 𝑐 𝑖 (𝑡) + 𝑟 𝑛 (𝑡) 𝑛 𝑖=1 , (1)
where 𝑐 𝑖 (𝑡) and 𝑟 𝑛 (𝑡) are the i th IMF and the residue, respectively [START_REF] Panoulas | Spectrum as a New Field for the Identification of EEG Event Related De-/Synchronization for BCI applications[END_REF][START_REF] Huang | The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis[END_REF]. All raw EEG epochs in this study were decomposed into 8 IMFs and a residue, the number of which was determined by sample length and stopping criteria of the sifting process.

After the EMD process, the HT of the i th IMF can be calculated as [START_REF] Bedrosian | A product theorem for Hilbert transforms[END_REF] 𝑦

𝑖 (𝑡) = 1 𝜋 𝑃 ∫ 𝑐 𝑖 (𝜏) 𝑡-𝜏 𝑑𝜏 ∞ -∞ , ( 2
)
where 𝑃 is the Cauchy principal value. By arranging 𝑐 𝑖 (𝑡) and 𝑦 𝑖 (𝑡) into a complex pair, an analytic signal 𝑧 𝑖 (𝑡) can be formed as

𝑧 𝑖 (𝑡) = 𝑎 𝑖 (𝑡)e 𝑗∅ 𝑖 (𝑡) = 𝑐 𝑖 (𝑡) + 𝑗𝑦 𝑖 (𝑡), (3)
where 𝑎 𝑖 (𝑡) is defined as the instantaneous amplitude, and ∅ 𝑖 (𝑡) is defined as the instantaneous phase. Hence all HT of IMFs constitute the HHT spectrum 𝐻(𝜔, 𝑡) of whole signal 𝑥(𝑡), presenting time-frequency-energy information as a 3D spectrum:

𝐻(𝜔, 𝑡) = 𝐻𝐻𝑇{𝑥(𝑡)} = ∑ 𝑎 𝑖 (𝑡)𝑒 𝑗 ∫ 𝜔 𝑖 (𝑡)𝑑𝑡 𝑛 𝑖=1 , (4)
where 𝜔 𝑖 (𝑡) is defined as instantaneous angular frequency as 𝑑∅ 𝑖 (𝑡) 𝑑𝑡 ⁄ , and the residue 𝑟 𝑛 (𝑡) is omitted. Finally, the MS, representing the accumulated energy over the entire data span from the contribution of each frequency value, can be defined as:

ℎ(𝜔) = ∫ 𝐻(𝜔, 𝑡)𝑑𝑡 𝑇 0 . (5)
Moreover, SE was further used to measure the quantities of signal disorder in the frequency domain [START_REF] Powell | A spectral entropy method for distinguishing regular and irregular motion of Hamiltonian systems[END_REF], and SE can be evaluated by the normalized powers of frequencies:

SE = - ∑ (ℎ ̂(𝑓) log 2 ℎ ̂(𝑓)) 𝑓 log 2 𝑚 , (6)
where ℎ ̂(𝑓) = ℎ(𝑓) ∑ ℎ(𝑓) is the normalized frequency component and 𝑚 is the number of frequency components. The normalized entropy values were between 0 (complete regularity) and 1 (maximum irregularity), showing the concentration of frequency distribution [START_REF] Humeau-Heurtier | Refined multiscale Hilbert-Huang spectral entropy and its application to central and peripheral cardiovascular data[END_REF].

2-3. Feature selection and Support vector machine

Having extracted the frequency-specific power from HHT and computed the SE, we employed the linear forward selection method to reduce the number of attributes that enter SVM.

The SVM was developed from statistical learning theory to analyze a data set for the classification of multi-classes [START_REF] Lotte | A review of classification algorithms for EEG-based brain-computer interfaces[END_REF][START_REF] Chang | LIBSVM: a library for support vector machines[END_REF]. A data set is trained to acquire a mathematical model, which is used to discriminate a testing data set. For binary classification, an SVM model constructs a hyperplane that optimally separates data sets into one of two classes, and the distance from the hyperplane to the nearest data points on each side is maximized.

Assuming the testing data set is linearly separable, a general form of hyperplane can be defined by 𝐰 T 𝐱 + b = 0, where 𝐰 is the normal vector to the hyperplane, and b is the bias term, and a classifier, 𝑑 = sgn(𝐰 T 𝐱 + b), can be selected. For each data point 𝐱 𝑖 , the following equation must be satisfied:

𝑑 𝑖 (𝐰 T 𝐱 𝑖 + b) ≥ 1, for 1 ≤ 𝑖 ≤ 𝑛. (7)
𝐰 and b are then optimized to set an optimal separating hyperplane, and the margin between the two classes is maximized [START_REF] Lotte | A review of classification algorithms for EEG-based brain-computer interfaces[END_REF].

If the testing data set is not linearly separable, slack variables ξ 𝑖 are introduced to measure the degree of misclassification, and the primal problem is modified to [START_REF] Chang | LIBSVM: a library for support vector machines[END_REF] minimize:

‖𝐰‖ 𝟐 /2 + 𝐶 ∑ ξ 𝑖 subject to: 𝑑 𝑖 (𝐰 T 𝚽(𝐱 𝑖 ) + b) ≥ 1 -ξ 𝑖 , for 1 ≤ 𝑖 ≤ 𝑛, (8)
where 𝐶 is the regularization parameter, which controls the punishment for misclassified data points and 𝚽(𝐱 𝑖 ) maps 𝐱 𝑖 into a higher dimensional space to make the separation in that space easier. To reduce the computational load, the Representer Theorem [START_REF] Kimeldorf | Some results on Tchebycheffian spline functions[END_REF] shows that 𝐰 with large dimensionality can be written as a linear combination of the training data, 𝐰 = ∑ 𝛼 𝑖 𝑑 𝑖 𝚽(𝐱 𝑖 ). Therefore, we can optimize 𝛼 𝑖 instead of 𝐰, and the decision function becomes

𝑓(𝐱) = ∑ 𝛼 𝑖 𝑑 𝑖 𝐾(𝐱 𝑖 , 𝐱) + b, (9)
where 𝐾(𝐱 𝑖 , 𝐱) = 𝚽(𝐱 𝑖 )

In this study, a Gaussian radial basis function (RBF) kernel, 𝐾(𝐱, 𝐱′) = exp(-𝛾‖𝐱 -𝐱′‖ 2 ), was used. Both 𝐶 and 𝛾 are carefully chosen to obtain optimal results.

A typical procedure of LIBSVM [START_REF] Chang | LIBSVM: a library for support vector machines[END_REF] involves several steps: (1) the input of attributes of a data set with pre-classified indices, [START_REF] Lubar | Evaluation of the effectiveness of EEG neurofeedback training for ADHD in a clinical setting as measured by changes in TOVA scores, behavioral ratings, and WISC-R performance[END_REF] training the data to build a model, and

predicting the classification or information of a test data set from the model. In the c-support vector classification in this study, the attribute vectors of attention epochs were labeled as class 1 in advance, while those of relaxed epochs were labeled as class -1. These attribute vectors with two-task labels comprised an input matrix for SVM training. After building the models, the classification of new epochs can be predicted using these models. The marginal spectrum of an IMF presents the time integration of its 3D spectrum and describes the distribution of power contained in the IMF as a function of frequency. Figure 5 shows the marginal spectra of IMF2 to IMF5, which were used to derive the features of frequency. Table I also indicates that using more features as the attributes did not help to obtain better accuracy. The result of feature selection of machine learning further confirmed that IMF1 and IMF6~IMF8 were not important features for classification. In other words, the most important EEG features that can reflect the state of attentiveness in this study are the

Results and discussions

attributes #6: α-IMF2, α-IMF3, α-IMF4, α-IMF5, β-IMF2, β-IMF3, β-IMF4, β-IMF5, α-SE,
and β-SE.

Having established the individualized predictive models of SVM, we tested the predictive model to classify the attentiveness state of each participant as listed in Table II.

The mean self-accuracy of 84.80% with 10-fold cross validation shows that the selected features can offer efficient differentiation for the assessment of attentiveness. To validate the individualized models that we built, additional 50 attention and 50 relaxation epochs of each participant were treated as the independent test data. The models classified the test data and outputted the predictions of the attentiveness, the accuracy of which are listed in Table III. The results show the agreement of the predictive models.

To compare with the proposed method that generated the time-frequency-domain and nonlinear features, we also implemented two analytic methods: the approximate entropy (ApEn) of original EEG signal as a time-domain and nonlinear feature, and the power spectral density of original EEG signal estimated by fast Fourier transform (FFT) as a frequency-domain and linear feature. These features were then respectively input into SVM as the attributes to build the predictive models for comparison. Figure 6 presents the boxplots of the above three methods, and the one-way ANOVA indicates the statistically significant differences in their results (p < 0.01). The SVM model of ApEn had a mean accuracy 73.94% with the smallest standard deviation compared to the others. The SVM model of FFT had a mean accuracy 77.80%, however, its large standard deviation indicated that there was huge between-subject variability in FFT features. According to the results, the proposed method of SVM combining with HHT can best discriminate between attentive and relaxed states.

Moreover, in this method, we used only one channel data to build the predictive models and yielded good accuracy up to 96.00% with an average of twenty participants being 83.90%. EEG recording using more channels may help improve the system; nevertheless, the single channel EEG monitor is inexpensive, convenient and portable for publics. We believe that this convenient method has the potential to be used in clinical settings for the detection of attention deficit.

Conclusion

A method of feature extraction and characterization of EEG signals using HHT frequency analysis and SVM has been presented. Raw EEG data have been analyzed by HHT to obtain marginal spectra for nonlinear and nonstationary frequency information. The α and β band powers of IMF2~5 and their spectral entropies were selected as the attributes of SVM to obtain the mean accuracy of 84.80%. We conclude that the proposed method can offer efficient differentiation for the assessment of attentiveness, showing promise in applications of attention deficit detection or biofeedback training.

Figure 3

 3 Figure 3 presents the full HHT 3D spectrum of the signal in Figure 1, providing the

Figure 1 .

 1 Figure 1. A continuous raw EEG signal of a representative participant while conducting two tasks. The participant paid attention for 5 minutes (until the dotted line) and then took a break with eyes opened for another 5 minutes. The epochs in the red block were collected as attention data, and the epochs in the green block were collected as relaxation data for further analysis.

Figure 2 .

 2 Figure 2. The flowchart of the proposed method. The EEG data were treated with Hilbert-Huang analysis to obtain the time-frequency information. The extracted features were the input of the support vector machine to build a proper classifier for attentiveness recognition.

Figure 3 .

 3 Figure 3. The HHT spectrum of the representative participant while conducting two tasks.The colorbar shows the magnitude of the energy distribution. The conducted task was switched at the 300th second. The time-frequency-energy information of attentive and relaxed tasks are illustrated in red and green blocks, respectively.

Figure 4 .

 4 Figure 4. (a) IMFs of an EEG epoch when the representative participant was paying attention to solve a puzzle. (b) IMFs of an EEG epoch when the representative participant had been instructed to relax and take a rest.

Figure 5 .

 5 Figure 5. Marginal spectra of IMF2~5 when the representative participant was (a) paying attention and (b) taking a rest.

Figure 6 .

 6 Figure 6. The boxplots of accuracy of SVM models using HHT marginal power density, approximate entropy, and Fourier power density as the attributes.

7 α-

 7 IMF1, α-IMF2, α-IMF3, α-IMF4, α-IMF5, β-IMF1, β-IMF2, β-IMF3, β-IMF4, β-IMF5, α-SE, β-SE 91.25% 8 α-IMF2, α-IMF3, α-IMF4, α-IMF5, α-IMF6, β-IMF2, β-IMF3, β-IMF4, β-IMF5, β-IMF6, α-SE, β-SE91.75%9 α-IMF1, α-IMF2, α-IMF3, α-IMF4, α-IMF5, α-IMF6, β-IMF1, β-IMF2, β-IMF3, β-IMF4, β-IMF5, β-IMF6, α-SE, β-SE 91.75% 10 α-IMF1, α-IMF2, α-IMF3, α-IMF4, α-IMF5, α-IMF6, α-IMF7, α-IMF8, β-IMF1, β-IMF2, β-IMF3, β-IMF4, β-IMF5, β-IMF6, β-IMF7, β-IMF8, α-SE, β-SE 91.75%

  T 𝚽(𝐱) is the kernel function. The new dual problem is modified

	to [29]:
	maximize: ∑ 𝛼 𝑖 -1/2 × ∑ 𝛼 𝑗 𝛼 𝑘 𝑑 𝑗 𝑑 𝑘 𝑗𝑘 𝑖	𝐾(𝐱 𝑗 , 𝐱 𝑘 )
	subject to: 0 ≤ 𝛼 𝑖 ≤ 𝐶, 𝑎𝑛𝑑 ∑ 𝛼 𝑖 𝑑 𝑖 = 0

𝑖

.

Table I

 I lists the attention assessment results of a representative participant using SVM with several attribute vectors. Regarding the impact of attributes on the accuracy, the features comprising of 𝛼 and 𝛽 powers estimated from {IMF2~IMF5 and SE} #6

obtained the best classification result of 93.25% accuracy. Frequency-specific power attributes of the original EEG signal after HHT {𝛼 -original, 𝛽 -original} #1 resulted in 85.25% accuracy, suggesting an essential role in reflecting the attentiveness. The result that {SE} #4 feature alone could obtain the prediction accuracy of 86.25% suggested that the nonlinear characteristics of brain dynamics were important indicators of the mental states.

TABLE I .

 I THE IMPACT OF DIFFERENT ATTRIBUTE VECTORS ON THE ACCURACY OF SVM

		MODELS OF A REPRESENTATIVE PARTICIPANT
	#	Attributes	Accuracy
	1	α-original, β-original	85.25%
	2	α-IMF4, β-IMF4	82.00%
	3	α-IMF2, α-IMF3, α-IMF4, α-IMF5, β-IMF2, β-IMF3, β-IMF4, β-IMF5	88.00%
	4	α-SE, β-SE	86.25%
	5	α-original, β-original, α-SE, β-SE	87.50%
		α-IMF2, α-IMF3, α-IMF4, α-IMF5,
	6	β-IMF2, β-IMF3, β-IMF4, β-IMF5,	93.25%
		α-SE, β-SE

TABLE II .

 II THE ACCURACY OF SVM MODELS OF 20 PARTICIPANTS USING ATTRIBUTES #6

	Participant	Accuracy	Participant	Accuracy
	1	77.75%	11	81.50%
	2	86.75%	12	84.00%
	3	81.00%	13	83.25%
	4	85.50%	14	88.25%
	5	89.75%	15	89.00%
	6	77.25%	16	72.25%
	7	77.00%	17	90.25%
	8	93.50%	18	82.00%
	9	89.75%	19	93.25%
	10	81.25%	20	92.75%
			Average	84.80%
	TABLE III. THE ACCURACY OF PREDICTIONS WITH INDEPENDENT TEST DATA
	Participant	Accuracy	Participant	Accuracy
	1	82.00%	11	78.00%
	2	91.00%	12	80.00%
	3	92.00%	13	79.00%
	4	66.00%	14	91.00%
	5	89.00%	15	96.00%
	6	76.00%	16	77.00%
	7	73.00%	17	86.00%
	8	92.00%	18	82.00%
	9	89.00%	19	92.00%
	10	76.00%	20	91.00%
			Average	83.90%

Acknowledgments

This material is based upon work supported by the Ministry of Science and Technology, Taiwan, under contract no. MOST 106-2221-E-008-042, MOST 106-2221-E-008-066-MY2 and MOST 105-2221-E-008-038.

Funding: Ministry of Science and Technology, Taiwan

Ethical approval: This study, numbered 201812EM027, was approved by the Research

Competing interests: The authors declare that they have no conflict of interest.

Ethics Committee of National Taiwan University, Taiwan.


est destinée au dépôt