Fabien Lotte

Camille Jeunet

Defining and Quantifying Users' Mental Imagery-based BCI skills: a first step

Objective: While promising for many applications, Electroencephalography (EEG)-based Brain-Computer Interfaces (BCIs) are still scarcely used outside laboratories, due to a poor reliability. It is thus necessary to study and fix this reliability issue. Doing so requires the use of appropriate reliability metrics to quantify both the classification algorithm and the BCI user's performances. So far, Classification Accuracy (CA) is the typical metric used for both aspects. However, we argue in this paper that CA is a poor metric to study BCI users' skills. Here, we propose a definition and new metrics to quantify such BCI skills for Mental Imagery (MI) BCIs, independently of any classification algorithm.

Approach: We first show in this paper that CA is notably unspecific, discrete, training data and classifier dependent, and as such may not always reflect successful self-modulation of EEG patterns by the user. We then propose a definition of MI-BCI skills that reflects how well the user can self-modulate EEG patterns, and thus how well he could control an MI-BCI. Finally, we propose new performance metrics, classDis, restDist and classStab that specifically measure how distinct and stable the EEG patterns produced by the user are, independently of any classifier.

Main results: By re-analyzing EEG data sets with such new metrics, we indeed confirmed that CA may hide some increase in MI-BCI skills or hide the user inability to self-modulate a given EEG pattern. On the other hand, our new metrics could reveal such skill improvements as well as identify when a mental task performed by a user was no different than rest EEG.

Significance: Our results showed that when studying MI-BCI users' skills, CA should be used with care, and complemented with metrics such as the new ones proposed. Our results also stressed the need to redefine BCI user training by considering the different BCI subskills and their measures. To promote the complementary use of our new metrics, we provide the Matlab code to compute them for free and open-source.

Introduction

While they are very promising for numerous applications, such as assistive technology or gaming, Electroencephalography (EEG)-based Brain-Computer Interfaces (BCIs) are still scarcely used outside laboratories [START_REF] Chavarriaga | Heading for new shores! overcoming pitfalls in BCI design[END_REF]. This is mostly due to their poor reliability, as they often recognize erroneous mental commands from the user. One of the main current challenges for the community is thus to improve BCI reliability [START_REF] Chavarriaga | Heading for new shores! overcoming pitfalls in BCI design[END_REF]. This is currently addressed at different levels, such as trying to design more robust EEG signal processing algorithms, or trying to improve BCI user training approaches, which have been shown to be inappropriate and a major cause of poor performances, both in theory and in practice [START_REF] Chavarriaga | Heading for new shores! overcoming pitfalls in BCI design[END_REF][START_REF] Jeunet | Why standard brain-computer interface (BCI) training protocols should be changed: An experimental study[END_REF][START_REF] Lotte | Towards improved BCI based on human learning principles[END_REF][START_REF] Lotte | Flaws in current human training protocols for spontaneous brain-computer interfaces: Lessons learned from instructional design[END_REF]. Improving these different aspects requires metrics to measure BCIs reliability and thus their performances. Indeed such performance metrics could identify what are the limitations of a given algorithm or training approach, which is a necessary first step towards fixing these limitations [START_REF] Chavarriaga | Heading for new shores! overcoming pitfalls in BCI design[END_REF].

User performance metrics are particularly useful for studying and improving Mental Imagery (MI) BCI user skills acquisition. Appropriate performance metrics could indeed help to understand what users have successfully learned or still need to improve, which can then be used to guide them, i.e., to provide them with appropriate training tasks and feedback. In EEG-based BCI, the most used metric is online Classification Accuracy (CA), i.e., the percentage of mental commands that were correctly recognized by the BCI during online use [START_REF] Schlögl | Towards Brain-Computer Interfacing, chapter Evaluation criteria in BCI research[END_REF][START_REF] Thomas | An analysis of performance evaluation for motor-imagery based BCI[END_REF][START_REF] Thompson | Performance measurement for brain-computer or brain-machine interfaces: a tutorial[END_REF]. Online CA, together with other machine learning evaluation metrics [START_REF] Schlögl | Towards Brain-Computer Interfacing, chapter Evaluation criteria in BCI research[END_REF][START_REF] Thomas | An analysis of performance evaluation for motor-imagery based BCI[END_REF][START_REF] Thompson | Performance measurement for brain-computer or brain-machine interfaces: a tutorial[END_REF], have been successfully used to quantify the decoding performance of the BCI, i.e., how well the BCI recognizes the users' commands. However, CA is also used to study BCI users' performance and learning, i.e., how well users can modulate/self-regulate their EEG signals to control the BCI, and how much they learn to do so. For instance, CA is typically used to study how different kinds of feedback influence BCI users' training [START_REF] Kaufmann | Visually multimodal vs. classic unimodal feedback approach for smr-bcis: a comparison study[END_REF][START_REF] Mladenović | The impact of flow in an EEG-based brain computer interface[END_REF][START_REF] Pillette | PEANUT: Personalised Emotional Agent for Neurotechnology User-Training[END_REF], or how different psychological factors influence BCI users' learning and performances [START_REF] Jeunet | Advances in user-training for mentalimagery-based BCI control: Psychological and cognitive factors and their neural correlates[END_REF].

In this paper, we argue and demonstrate that CA alone, as used in online MI-BCI, is not enough to study user performance and thus their MI-BCI skills. Indeed, this metric is notably discrete -an input data is either correctly or incorrectly classified -as well as classifier and training data dependent, since changing the training data or classifier will change the resulting performance, independently of the actual users' BCI skills, i.e., independently of how well they can modulate their EEG signals using MI. Overall, CA can tell us how well the machine can recognize the EEG patterns from the user, but was not designed to tell us how well the user is able to produce clear, stable and distinct EEG patterns. As such, CA might not provide us with this information, and we actually show in this paper that CA can actually fail to do so in practice. In order to fully understand BCI user skill acquisition, alternative or additional metrics are thus necessary. Moreover, to be able to quantify MI-BCI skills using such metrics, we need to define what those skills are. Therefore, in this paper, we propose a first definition of MI-BCI skills and propose simple and computationally efficient metrics to quantify them. We then compare them with the classically used online CA 1 . We show that using online (or simulated online) CA as metric may actually hide several relevant aspects of BCI skill acquisition. In particular, online CA may miss users' MI-BCI skills increase overtime or fail to identify that a mental task performed is actually no different than rest EEG. Our new metrics can overcome these limitations. Since performance metrics were also used in several papers to identified psychological and neurophysiological factors influencing BCI performance, we also studied whether our new metrics could confirm the previously identified influence of one of them: spatial abilities [START_REF] Jeunet | Advances in user-training for mentalimagery-based BCI control: Psychological and cognitive factors and their neural correlates[END_REF]. Our results showed that indeed some of our new metrics are also significantly correlated to spatial abilities, hence further confirming the importance of this factor.

This paper is organized as follows: The following section presents a brief survey of the common performance metrics used in BCI, notably those related to the classification accuracy, as well as their limitations. The next section introduces the materials and methods, notably the MI-BCI skills definition and the new metrics we propose. It also presents the data sets on which these measures are compared. Then the Results section compares the performances estimated with all metrics, which are then discussed in the Discussion section. The last section concludes the paper.

2 Performance metrics for BCI

Current common performance metrics

As indicated before, Classification Accuracy (CA) is the most used metric to quantify BCI performances. Typically, the classifier is trained on the EEG signals from the trials of the first BCI run(s) (calibration run(s)) and applied to classify the users' EEG signals from the trials of the subsequent runs. CA is defined as the percentage of these EEG trials that were correctly classified [START_REF] Thomas | An analysis of performance evaluation for motor-imagery based BCI[END_REF]. From the classification results, we can also obtain a more detailed information on the performances from the Confusion Matrix (CM), which informs about how many trials from each class were estimated to be from each one of the possible classes. The CM is defined as follows for a two class problem: Here, the number in row i, column j is the number of trials from class i that was classified as belonging to class j. Thus, a and d correspond to correct classifications (the real and estimated classes are the same), and c and b to erroneous classifications. CA (in %) can thus be computed as follows:

CA = a + d a + b + c + d × 100 (1)
or more generally, for any number of classes, if CM ij is the (i, j) th element of the confusion matrix CM , then:

CA = i CM ii i,j CM ij × 100 (2)
From there we can also estimate the CA of each class, e.g., CMii j CMij × 100 is the percentage of trials from class i that were correctly classified.

In addition to the CA, other metrics have been proposed based on the CM, notably the Kappa coefficient [START_REF] Schlögl | Towards Brain-Computer Interfacing, chapter Evaluation criteria in BCI research[END_REF][START_REF] Thomas | An analysis of performance evaluation for motor-imagery based BCI[END_REF][START_REF] Thompson | Performance measurement for brain-computer or brain-machine interfaces: a tutorial[END_REF], or the extended CM to support non-control states [START_REF] Bianchi | Performances evaluation and optimization of brain computer interface systems in a copy spelling task[END_REF].

When the BCI is used to select items (e.g., letters to spell or a direction with a wheelchair), metrics that also take into account the selection time have been proposed, such as the widely used Information Transfer Rate (ITR) [START_REF] Wolpaw | EEG-based communication: improved accuracy by response verification[END_REF], the BCI utility metric [START_REF] Seno | The utility metric: a novel method to assess the overall performance of discrete brain-computer interfaces[END_REF], the correct number of spelled letters per minute (for spellers) [START_REF] Spüler | A high-speed brain-computer interface (BCI) using dry EEG electrodes[END_REF] or the Rate of Information Gain (RIG) [START_REF] Hill | A general method for assessing braincomputer interface performance and its limitations[END_REF].

Metrics have also been proposed to study the BCI performances specifically for ERP-BCI applications such as ERP-Spellers, with the projected accuracy [START_REF] Colwell | Projected accuracy metric for the p300 speller[END_REF] or the classifier-based latency estimation (CBLE) [START_REF] Thompson | Classifier-based latency estimation: a novel way to estimate and predict bci accuracy[END_REF]. For BCI-based cursor control, Fitt's law and its variants have also been used to estimate BCI performances [START_REF] Felton | Evaluation of a modified fitts law brain-computer interface target acquisition task in able and motor disabled individuals[END_REF].

Interestingly enough, Hill and colleagues proposed to quantify BCI user training and performances using an adaptive stair-case task, i.e., a game with adaptive difficulty [START_REF] Hill | A general method for assessing braincomputer interface performance and its limitations[END_REF]. The median difficulty level reached by the BCI user in this game has been shown to reflect well BCI control performances, and to be highly correlated to the RIG mentioned above.

Bauer et al also proposed to quantify the participant opportunity for learning in BCI according to the concepts of the Zone of Proximal Development (ZPD). They estimated the ZPD as the difference between the True Positive Rate (TPR) and the False Positive Rate (FPR) for various thresholds, in a binary BCI with a positive (active) and negative (rest) class [START_REF] Bauer | Estimating cognitive load during self-regulation of brain activity and neurofeedback with therapeutic brain-computer interfaces[END_REF].

It should be mentioned that to study user learning in BCI and in NF applications, the neurophysiological variations of EEG over time have been studied. Typically the power of the EEG from a given channel and frequency band is computed, and compared after and before training, or is correlated to the BCI class labels (commands), see, e.g., [START_REF] Contreras-Vidal | Identifying engineering, clinical and patient's metrics for evaluating and performance of brain-machine interface (bmi) systems[END_REF][START_REF] Gruzelier | EEG-neurofeedback for optimising performance. i: a review of cognitive and affective outcome in healthy participants[END_REF][START_REF] Vidaurre | Co-adaptive calibration to improve bci efficiency[END_REF]. This can provide spatial and/or spectral topographies of the main changes in EEG due to training. No global metric of neurophysiological change due to BCI/NF training is available though.

Among all these metrics, the CA and possibly the ITR (when taking into account a specific application) are by far the most used performance metrics for BCI. Typically CA is the metric used in most papers to quantify BCI user performance and learning [START_REF] Jeunet | Advances in user-training for mentalimagery-based BCI control: Psychological and cognitive factors and their neural correlates[END_REF]. Unfortunately, as we will see below, this metric suffers from several limitations for this task. The other metrics mentioned here also share most of these limitations.

Limitations

CA and the other metrics mentioned above are very useful to quantify the decoding performance of a BCI and/or the performance with the applications controlled using the BCI (e.g., spelling performance with an ERP-speller) [START_REF] Contreras-Vidal | Identifying engineering, clinical and patient's metrics for evaluating and performance of brain-machine interface (bmi) systems[END_REF][START_REF] Thompson | Performance measurement for brain-computer or brain-machine interfaces: a tutorial[END_REF][START_REF] Thomas | An analysis of performance evaluation for motor-imagery based BCI[END_REF][START_REF] Schlögl | Towards Brain-Computer Interfacing, chapter Evaluation criteria in BCI research[END_REF]. However, when it comes to studying how well users can voluntarily modulate their EEG signals to control the BCI, we argue that such metrics actually suffer from many limitations, and that dedicated metrics are needed.

First, these metrics are unspecific: they only provide the global performance, but not what is correctly classified or not, nor why it is so. Then, these metrics are typically discrete measures: a trial is either correctly classified or not, there is no middle ground. As such, even if the user produces a stronger EEG modulation than before, but not strong enough to make the trial correctly classified, metrics such as CA will not change.

All the metrics above measure the global performance of the BCI or of the BCIcontrolled application. As such the obtained performance metrics depend on both the EEG signal processing tools, the user skill at BCI control and the application interface (e.g., different speller configurations or interaction techniques), among others. Thus, such metrics cannot unambiguously inform us about the users' MI-BCI skills and learning curve. It should be mentioned though that Kübler et al. proposed to evaluate BCI systems not only according to machine performance but also according to the user experience. They notably proposed to evaluate the overall usability of BCIs, by considering the system efficiency and effectiveness, as well as the satisfaction it gave users [START_REF] Kübler | The user-centered design as novel perspective for evaluating the usability of bci-controlled applications[END_REF]. Note that the satisfaction is a subjective metric, and that both efficiency and effectiveness depend also heavily on the EEG signal processing tools and interface design. Metrics dedicated to quantify the users' skills at BCI control are thus still lacking.

The currently used performance metrics and notably CA are also strongly classifier and training data dependent. Changing the classifier type, its parameters, or the amount and/or quality of the training data will change the metric, independently of how well users can modulate their EEG activity using MI. Therefore, variations of these metrics might not always reflect users' proficiency at BCI control. In particular, such metrics and notably CA reflect the user's performance if and only if the classifier is a good classifier, i.e., a classifier that can classify successfully the EEG patterns that the user can modulate using MI to control the BCI. However in practice it is rarely the case, due to many factors. Classifiers are indeed sensitive to non-stationarities, and thus would lead to poor CA when applied on EEG data from a different distribution than that of the calibration run. This is likely to happen if users are trying out various strategies or are learning. This can also happen if their MI were of poor quality during the calibration runs, which is also likely to happen for naive users, unfamiliar with such MI. To take an extreme case, let us consider a hypothetical user who can produce very strong and very localized EEG desynchronization during motor imagery. With a properly trained classifier, such a user could reach 100% classification accuracy. However, if we give that user a dysfunctional, random classifier, trained on pure noise, then this user will reach a CA at chance level. In that last case, using CA as the metric to quantify the user's BCI skills would have made us conclude this user was unable to use an MI-BCI at all. Yet, using another classifier, adapted to the EEG signals that the user can modulate using MI, would have revealed how skilled this user is, and how well he could control an MI-BCI. This example highlights the need for metrics to quantify users' BCI skills independently of a single given classifier, as classifier-dependent metrics such as CA can be highly misleading. Finally, when based on a discriminative classifier such as Linear Discriminant Analysis (LDA), the most used classifiers for BCI [START_REF] Chavarriaga | Heading for new shores! overcoming pitfalls in BCI design[END_REF], CA does not reflect how well a given mental command can be recognized but rather how distinct the mental commands are from each other. Therefore, if users are unable to modulate their EEG signals for one class (e.g., left hand MI), they may still obtain very high CA as long they can modulate their EEG for the other class (e.g., right hand MI), since the EEG signals from the two classes would still be distinct. This leads to a last limitation: in MI-BCI, CA and other metrics usually consider the MI EEG signals only, but not the rest EEG signals. As illustrated just before, this prevents us from identifying whether the user's EEG patterns during MI are actually any different from rest EEG.

For all these reasons, CA and other existing metrics may not be able to reveal some important aspects of MI-BCI user performance and learning. In other words, CA and related metrics are appropriate measures to study how well the BCI can decode the users' mental commands, but on their own, they are not enough to study how well users can modulate their EEG patterns using MI in order to control an MI-BCI and how well they are learning to do so. This thus calls for a definition of what MI-BCI skills can be and for new and specific metrics to quantify these skills. This is what we propose in this paper. We present our definition of MI-BCI skills and describe our new metrics in the following sections.

Materials and methods

Defining MI-BCI Skills

So far, whereas it is clear that MI-BCI control requires and involves learning [START_REF] Neuper | Brain-Computer Interfaces, chapter Neurofeedback Training for BCI Control[END_REF][START_REF] Lotte | Flaws in current human training protocols for spontaneous brain-computer interfaces: Lessons learned from instructional design[END_REF][START_REF] Jeunet | Human Learning for Brain-Computer Interfaces[END_REF] there is no formal or agreed upon definition of the skills to be learned. There is thus a need to go towards a definition and quantification of such skills that we denote here as MI-BCI skills. In order to try to conceptualize MI-BCI skills, let us first consider a simple analogy in which a human user also employs a given human-computer interface (HCI) to achieve a goal: Formula 1 racing. With Formula 1 racing, a human userthe pilot -has to drive a Formula 1 car in order to complete a race as fast as possible. The outcome of the race, which is the score of the pilot, is how fast they completed the race. This outcome is compared to that of other pilots to designate the winner. It is important to note that this outcome of the race depends on both the driver, and notably his driving skills, and on the Formula 1 car used (e.g., how fast it can go). Therefore, a poor race outcome can be due to either a bad (slow) car, that is thus unlikely to win irrespectively of how skilled the driver is, or can be due to a bad driver, which can also fail despite being provided with a fast car. The best outcome is to be expected with both a skilled driver and a fast car. The driver should thus train in order to acquire good Formula 1 racing skills. Note that this skill is not dependent on a single Formula 1. Naturally, knowing well the used Formula 1 will increase the likelihood of success for the driver. However, good Formula 1 drivers are not good with a single Formula 1, they are good drivers in general. For instance, Michael Schumacher won many races with many different types of cars, with different engines, tires, chassis and from different constructors 2 .

In this paper, we argue that quantifying MI-BCI performance should follow a similar logic. In order words, in our analogy, the pilot should be replaced by the MI-BCI user, and the Formula 1 car by the classifier. This would mean that CA, i.e, the BCI use outcome, depends both on the users' skills at MI-BCI control and on the quality of the classifier used. As such, CA does not reflect BCI users' skills only. While training to use a given classifier is likely to improve CA, a skilled MI-BCI user should be able to reach high MI-BCI control with various classifiers, and not just with a single one. Note by the way that during MI-BCI use, classifiers are regularly retrained, or updated, see, e.g., [START_REF] Vidaurre | Co-adaptive calibration to improve bci efficiency[END_REF][START_REF] Pfurtscheller | Motor imagery and direct brain-computer communication[END_REF], which means that a good BCI user cannot be good with a single classifier, but has to be good with various classifiers. This ensures generalizable skills, and prevent overfitting to a specific classifier. If the BCI user is able to produce clear and stable brain activity patterns, then multiple types of classifiers would be able to recognize those patterns and could consequently be used to control the BCI. Therefore, for the user, controlling an EEG-based MI-BCI means producing EEG patterns that can be reliably translated into commands for an application using MI. Accordingly, we propose the following definition for the skills required to do so, i.e., MI-BCI skills:

"MI-BCI skills correspond to the ability of the user to voluntarily produce brain activity patterns that are distinct between mental tasks, and stable within mental tasks, so that they can be translated reliably and consistently into control commands. The more stable and distinct the brain activity patterns, the higher the MI-BCI skills."

Note that this definition being that of a skill, it is naturally focused on the user, and not on the machine (i.e., not on the classifier or other acquisition and processing hardware and software). From this definition, it follows that classification accuracy, which is the most commonly used metric to quantify MI-BCI performance and user learning [START_REF] Jeunet | Advances in user-training for mentalimagery-based BCI control: Psychological and cognitive factors and their neural correlates[END_REF][START_REF] Kübler | How much learning is involved in BCI-control?[END_REF][START_REF] Neuper | Brain-Computer Interfaces, chapter Neurofeedback Training for BCI Control[END_REF], could successfully measure MI-BCI skill in some conditions, but may fail to do so properly in some other conditions. In particular, classification accuracy would reflect MI-BCI skills if and only if the classifier3 is able to exploit and recognize the brain activity patterns mentioned above, and perfectly translate them into control commands. If the classifier exploits other brain activity patterns, that the user struggles to make distinct and stable, then classification accuracy will not reflect the true user MI-BCI skills. In other words, while high MI-BCI skills can mean high classification accuracy, low classification accuracy may not always mean low MI-BCI skills, but can rather mean an inappropriate classifier. There is no bijection between these two measures. We aim at demonstrating this point experimentally in the present paper, and at proposing metrics to quantify such MI-BCI skills, in order to provide new tools to study MI-BCI user training.

New Performance metrics 1: Run-Wise Cross-Validation

To address some of the limitations mentioned above, a possible approach (not new in itself but typically not used to study MI-BCI user training) would be to perform Run-Wise Cross-Validation (RWCV). The idea is to use CV to estimate offline the CA of each run. With RWCV, the trials from the current run are divided into K parts, K-1 parts being used to train the classifier, and the last part to test the latter, the process being repeated K times, and the obtained CA averaged over the K testing parts. This also provides a run-wise confusion matrix and class-specific CV accuracies, as done with the standard CA. We will assess this approach in this paper. Interestingly enough, in [START_REF] Mcfarland | Should the parameters of a bci translation algorithm be continually adapted?[END_REF], McFarland et al. explored classifier adaptation based on CV on each run, which proved to be the most efficient offline adaptation strategy among those tested for motorimagery BCI. It suggests that EEG signals vary at the run scale, and thus that RWCV might also be a relevant tool to study varying performances over time.

Since training and testing are performed on each run, and for different parts of each run, it makes RWCV CA much less sensitive to training data and to non-stationarities. This metric remains non-specific and discrete though along with being classifier-specific. In addition, it still ignores the rest EEG. It is also computationally expensive. As such its use can be impractical, or even impossible if we target online performance evaluation for instance.

New Performance metrics 2: Stability and distinctiveness

To further improve on the metric mentioned above, there is thus a need for metrics that are also specific, continuous, that consider rest EEG signals, that are classifier independent and computationally cheap and that actually measure MI-BCI skills as we defined in Section 3.1. In other words, we need metrics that measure how distinct and stable the EEG patterns produced by the users with MI are, in a continuous and classifier independent way. A stable pattern would be a pattern that is not changing dramatically between trials, and thus with a small variance. A distinct EEG pattern would be both 1) an EEG pattern that is distinct from the rest EEG pattern, i.e., there is a specific signature to that pattern and 2) a pattern that is distinct from the EEG patterns of the other MI tasks, so that each can be associated to a distinct command. As such, it would make sense to design a set of metrics dedicated to estimating how stable and distinct the EEG patterns for each MI task actually are.

Interestingly enough, metrics quantifying these various properties can be defined using distances in a Riemannian geometry framework. Indeed, Riemannian geometry offers an efficient and simple way to measure distances between covariance matrices, such matrices being increasingly used to represent EEG patterns [START_REF] Congedo | Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review[END_REF][START_REF] Yger | Riemannian approaches in brain-computer interfaces: a review[END_REF].

Riemannian geometry in brief

Let us first consider matrix X i ∈ R Nc×Ns of EEG signals from trial i, with N c the number of channels and N s the number of samples per trial. This can be a matrix of preprocessed EEG signals, for instance a matrix of EEG signals band-pass filtered in 8-30Hz for a motor imagery-based BCI experiment. The spatial covariance matrix C i of this trial is defined as C i = 1 Ns X i X T i , with T being transpose. Therefore, the diagonal elements of C i represent the EEG band power for each channel (in the band in which the signals were band-pass filtered, e.g., 8-30Hz in our example above), and the off-diagonal elements, their covariations. Such spatial covariance matrices are used -implicitly or explicitly -to represent EEG signals in numerous MI-BCI designs, notably those based on the Common Spatial Patterns (CSP) algorithm, and many others [START_REF] Congedo | Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review[END_REF][START_REF] Roijendijk | Classifying regularized sensor covariance matrices: An alternative to CSP[END_REF][START_REF] Tomioka | A regularized discriminative framework for eeg analysis with application to brain-computer interface[END_REF][START_REF] Yger | Riemannian approaches in brain-computer interfaces: a review[END_REF]. The Riemannian distance δ R (C i , C j ) between covariance matrices C i and C j can be defined as:

δ R (C i , C j ) = [ n i=1 log(λ i ) 2 ] 1/2 (3)
where the λ i are the eigen values of C -1 i C j . This Riemannian distance is particularly interesting since it is affine invariant: it is invariant to full rank linear transformations, i.e., to variations such as normalization or channel displacement [START_REF] Congedo | Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review[END_REF][START_REF] Yger | Riemannian approaches in brain-computer interfaces: a review[END_REF]. As such, the Riemannian distance has been used successfully for robust EEG signal decoding, in various kinds of BCIs [START_REF] Congedo | Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review[END_REF][START_REF] Yger | Riemannian approaches in brain-computer interfaces: a review[END_REF]. In this paper, we show that this distance can also be a very relevant tool to quantify how distinct and stable the EEG patterns produced by a BCI user are.

Basic stability and distinctiveness metrics

How distinct the EEG patterns produced during two MI tasks are from each other could be quantified using the Riemannian distance between the average covariance matrices for each MI task. Then, the stability of a given EEG pattern can be defined using the average distance between each trial covariance matrix and the average covariance matrix for this task, which is a form of Riemannian standard deviation [START_REF] Yger | Riemannian approaches in brain-computer interfaces: a review[END_REF]. More formally, let us first define the Riemannian mean C of a set of covariance matrices C i [START_REF] Yger | Riemannian approaches in brain-computer interfaces: a review[END_REF] as:

C = argmin C N i=1 δ 2 R (C i , C) (4)
Note that there are efficient implementations to obtain such mean matrices [START_REF] Yger | Riemannian approaches in brain-computer interfaces: a review[END_REF]. We can also define the mean absolute deviation (i.e. a form of dispersion measure) σ C of a set of covariance matrices C i as:

σ C = 1 N N i=1 δ R (C i , C) (5)
Distinctiveness From the definitions above, we propose to define the distinctiveness classDis of the EEG patterns from two MI classes A and B as:

classDis(A, B) = δ R ( CA , CB ) 1 2 (σ C A + σ C B ) (6)
where CK and σ C K are respectively the mean and mean absolute deviation of the covariance matrices from MI class K. This equation can be seen as an extension of the Fisher criterion (see, e.g., [START_REF] Fukunaga | Statistical Pattern Recognition[END_REF]) to covariance matrices: the further apart from each other are the average EEG patterns from each class (measured by δ R ( CA , CB )), with respect to their variance (represented by 1 2 (σ C A + σ C B )), the higher their distinctiveness.

However such a metric only works for 2 classes. Thus, still following the analogy with the Fisher criterion and its multiclass extensions, we can define a similar metric for the multiclass case. For a BCI with N c MI commands/classes (N c > 2), this metric would be defined as the ratio of the between class variance to the within class variance, as for the multiclass Fisher criterion [START_REF] Fukunaga | Statistical Pattern Recognition[END_REF]:

classDis({A i }) = N c i=1 δ R ( CAi , CA ) N c i=1 σ C A i (7)
where CA is the average of the mean covariance matrices from all classes.

Similarly, we propose to define the distinctiveness restDis between the EEG patterns from one class and those from the rest state as:

restDis(A) = δ R ( CA , Crest ) 1 2 (σ C A + σ C rest ) (8)
where Crest and σ C rest are respectively the mean and mean absolute deviation of the covariance matrices of the rest EEG.

Stability Finally, we can define the stability of the EEG patterns from one MI task as being inversely proportional to the mean absolute deviation of the covariance matrices from that task:

classStab(A) = 1 1 + σ C A (9)
As such, the lower the mean absolute deviation of the EEG patterns, as represented by the spatial covariance matrix, the higher the stability.

Considering more than the spatial EEG patterns

The metrics above only consider the spatial covariance matrices, i.e., the spatial EEG features, in a given frequency band (the band in which the EEG signals were band-pass filtered). While this is the most common way to represent EEG patterns, it could be interesting to study the EEG patterns more finely. In particular, if we assume that MI-BCI skills or learning could differ in different EEG spectral features, it would seem relevant to have metrics that measure users' MI-BCI skills in both spatial and spectral EEG patterns.

This could be achieved for instance by using large covariance matrices containing both the spatial (EEG channels) and spectral (EEG frequency bands) information as rows/columns. However, such large matrices may lead to numerical instabilities and statistical estimation issues (see, e.g., [START_REF] Yger | Averaging covariance matrices for eeg signal classification based on the csp: an empirical study[END_REF]). Thus, to avoid such problems, we propose the following simple metrics which have the same matrix dimensionality as with the purely spatial EEG patterns.

First we can estimate the covariance matrices C

i,Fj = 1 Ns X Fj i (X Fj i )
T for various frequency bands, where X Fj i is the matrix of EEG signals from trial i, filtered in frequency band F j . We can then compute the distinctiveness classDist Fj (A, B) using covariance matrices C i,Fj in equations 6 or 7 (for binary or multiclass problems respectively), for each frequency band F j . Then we can compute the overall spatio-spectral distinctiveness as:

spaspecClassDis(A, B) = N f j=1 classDist Fj (A, B) (10)
where N f is the number of frequency bands considered. In a similar way, we can define the spatio-spectral class stability and rest distinctiveness by summing the values obtained by each of these metrics over different frequency bands F j . This thus give us spaspecClassStab and spaspecRestDis, see also Table 2.

Summary of the new metrics

Most of the metrics presented above are intuitive and computationally efficient metrics to quantify some aspects of users' skills at MI-BCI control. They are also training data and classifier independent, as well as robust to some non-stationarities given the affine invariance of δ R . All these metrics are summarized in the Tables below. In particular, the core, basic metrics classDist, restDist and classStab are presented in Table 1, while their spatio-spectral variants are presented in Table 2.

= δ R ( CA , CB ) 1 2 (σ C A +σ C B ) Class distinctiveness (multiclass) classDis({A i }) = N c i=1 δ R ( C A i , CA ) N c i=1 σ C A i Rest distinctiveness restDis(A) = δ R ( CA , Crest ) 1 2 (σ C A +σ C rest ) Stability classStab(A) = 1 1+σ C A
Regarding the distinctiveness metrics, it is important to realize that they do not depend nor use any classifier, and as such that what they measure may be very different from CA. Indeed, an unadapted classifier, i.e., a classifier trained on outdated data, may lead to poor CA, while an adapted one may lead to high CA, if class distinctiveness is high and if the classifier can exploit these EEG patterns that are highly distinct from each other. Class distinctiveness is thus a measure of MI-BCI skills that consider the potential BCI control that the user can get, provided that the BCI classifier is adapted to this user, i.e., uses the EEG patterns produced with MI that are distinct from each other. In the following, we compare the various metrics offline with CA and RWCV CA.

N f j=1 classDis Fj (A, B) spatio-spectral rest distinctiveness spaspecRestDis(A) = N f j=1 restDis Fj (A) spatio-spectral stability spaspecClassStab(A) = N f j=1 classStab Fj (A)

Data set and evaluation

To compare the performance metrics, we used two MI-BCI data sets, from our previous experiments. More specifically, we used 1) a 2-class motor imagery data set, in which N=20 users trained for a single session (i.e., one day) to use the BCI [START_REF] Jeunet | Why standard brain-computer interface (BCI) training protocols should be changed: An experimental study[END_REF], and 2) a 3class mental imagery BCI data set, in which 17 users trained for 6 sessions (i.e., 6 days) to use the BCI [START_REF] Jeunet | Predicting Mental Imagery-Based BCI Performance from Personality, Cognitive Profile and Neurophysiological Patterns[END_REF]. These two data sets enable us to study the various metrics with both short term (single session) and longer term (6 sessions) user training, as well as for binary and multiclass problems. They also enable us to see how the metrics behave for motor imagery as well as non-motor imagery tasks. It should be noted here that the BCI users in these two experiments were trained to control the BCI using as feedback a classical bar feedback, i.e., a bar extending towards the recognized class according to the classifier output. More precisely, the feedback bar had a length and a direction updated 16 times per second, according to the distance of the features, extracted from the last second of EEG data, to the separating hyperplane of a Linear Discriminant Analysis classifier (see [START_REF] Jeunet | Why standard brain-computer interface (BCI) training protocols should be changed: An experimental study[END_REF] and [START_REF] Jeunet | Predicting Mental Imagery-Based BCI Performance from Personality, Cognitive Profile and Neurophysiological Patterns[END_REF] for details). Thus, users were trained to control the BCI using a feedback somehow related to the online classification accuracy. The two data sets are described here after.

Data set 1: Single session, 2-class motor imagery data set

This data set comprises the EEG signals of 20 BCI-naive participants, who had to learn to do 2 MI-tasks, namely imagining left-and right-hand movements. Participants first had to complete a calibration run, without feedback, followed by 4 feedback runs.

Each run was composed of 20 trials for each of the two MI tasks (displayed in a random order). At the beginning of each trial, a fixation cross was displayed. Then, after 2s, a beep sound occurred. Then, at t = 3s, the instruction appeared as an arrow the direction of which indicates the MI task to be performed, i.e., an arrow pointing left indicated a left hand MI and an arrow pointing right a right hand MI. From t = 3.250s, a feedback was provided for 4s in the shape of a bar the direction of which indicating the mental task that had been recognized and the length of which representing the classifier output. More details about this data set can be found in [START_REF] Jeunet | Why standard brain-computer interface (BCI) training protocols should be changed: An experimental study[END_REF].

Data set 2: Multi-sessions, 3-class mental imagery data set

Seventeen BCI-naive participants took part in this study, for 6 different sessions each (each on a different day). The three mental imagery-tasks that participants had to learn to perform were 1) left-hand motor imagery, 2) mental rotation of a 3D geometric figure and 3) mental subtraction of a 2 digit number from a 3-digit number (both displayed on screen). Each session comprised 5 runs. During each run, participants had to perform 45 trials (15 trials per task), each trial lasting 8s. At t=0s, an arrow was displayed with a left hand pictogram on its left (left hand MI task), the subtraction to be performed on top (mental subtraction task) and a 3D shape on its right (mental rotation task). At t=2s, a "beep" announced the coming instruction and one second later, at t=3s, a red arrow was displayed for 1.250s. The direction of the arrow informed the participant which task to perform, e.g., an arrow pointing to the left meant the user had to perform a left hand MI task. Finally, at t=4.250s, for 4s, a visual feedback was provided in the shape of a blue bar, the length of which varied according to the classifier output. Only positive feedback was displayed, i.e., the feedback was provided only when there was a match between the instruction and the recognized task. More details about this data set can be found in [START_REF] Jeunet | Predicting Mental Imagery-Based BCI Performance from Personality, Cognitive Profile and Neurophysiological Patterns[END_REF].

The impact of spatial abilities on BCI performances

Interestingly enough, both data sets come from experiments which aimed, inter alia, to identify cognitive and personality profiles that correlated to BCI performances [START_REF] Jeunet | Why standard brain-computer interface (BCI) training protocols should be changed: An experimental study[END_REF][START_REF] Jeunet | Predicting Mental Imagery-Based BCI Performance from Personality, Cognitive Profile and Neurophysiological Patterns[END_REF]. The typical online classification accuracy was used as a measure of BCI performance in these works. In both studies, we observed a strong and significant correlation between classification accuracy and user's spatial abilities, the latter being measured using a mental rotation test [START_REF] Vandenberg | Mental rotations, a group test of threedimensional spatial visualization[END_REF], see [START_REF] Jeunet | Why standard brain-computer interface (BCI) training protocols should be changed: An experimental study[END_REF][START_REF] Jeunet | Predicting Mental Imagery-Based BCI Performance from Personality, Cognitive Profile and Neurophysiological Patterns[END_REF] for details. We will thus also study in the present work whether a correlation between spatial abilities and our new metrics of MI-BCI skills can be observed as well.

EEG signal recording and processing

For both data sets, EEG signals were recorded using 30 active scalp electrodes (F3, Fz, F4, FT7,FC5, FC3, FCz, FC4, FC6, FT8, C5, C3, C1, Cz, C2, C4, C6, CP3, CPz, CP4, P5, P3, P1, Pz, P2, P4, P6, PO7, PO8, 10-20 system) using a g.USBAmp (g.tec, Austria). EEG data were sampled at 256 Hz.

For both data sets, EEG data was band-pass filtered in 8-30 Hz. For each trial, the MI EEG segment used was the 2s long segment starting 0.5s after the cue (left, right or up arrow), i.e., from second 3.5 to 5.5 of each trial. For the rest EEG signals, we used the 2s long segment immediately before the cue, i.e., from second 1 to 3 of each trial.

Regarding the CA and RWCV CA metrics, for both data sets we used Common Spatial Patterns (CSP) spatial filters [START_REF] Ramoser | Optimal spatial filtering of single trial EEG during imagined hand movement[END_REF], the spatially filtered signals band power as features, and a Linear Discriminant Analysis (LDA) [START_REF] Lotte | A review of classification algorithms for EEG-based brain-computer interfaces[END_REF] as classifier. More precisely, for the 2-class data set, we used 3 pairs of Common Spatial Pattern (CSP) spatial filters and a LDA classifier, as in [START_REF] Jeunet | Why standard brain-computer interface (BCI) training protocols should be changed: An experimental study[END_REF]. For the multi-class data set, we used 3 sets of CSP filters, each optimized to discriminate EEG signals for a given class from those for the other two classes. We optimized 2 pairs of spatial filters for each class, thus leading to 12 CSP filters. The resulting 12 band power features were classified using a multi-class LDA, built by combining three LDA in a one-versus-the-rest scheme, as in [START_REF] Jeunet | Predicting Mental Imagery-Based BCI Performance from Personality, Cognitive Profile and Neurophysiological Patterns[END_REF].

For the standard (here simulated online) CA, we trained the CSP and LDA classifiers on the EEG data from the calibration run and used it to classify the EEG data from the 4 subsequent runs, as done online in [START_REF] Jeunet | Why standard brain-computer interface (BCI) training protocols should be changed: An experimental study[END_REF][START_REF] Jeunet | Predicting Mental Imagery-Based BCI Performance from Personality, Cognitive Profile and Neurophysiological Patterns[END_REF]. For the multi-session data set (data set 2), to account for some of the between session-variability, the LDA classifiers' biases were re-calculated after the first run of sessions 2 to 6, based on the data from this first run, as done in [START_REF] Friedrich | Whatever works: A systematic usercentered training protocol to optimize brain-computer interfacing individually[END_REF], and as done during the online experiments that resulted in this data set [START_REF] Jeunet | Predicting Mental Imagery-Based BCI Performance from Personality, Cognitive Profile and Neurophysiological Patterns[END_REF].

For the spatio-spectral metrics (spaspecClassDis, spaspecRestDis, spaspecClassStab) we used the frequency bands 8-10Hz, 10-12Hz, 12-18Hz and 18-30Hz, which corresponds to low-alpha, high-alpha, low-beta and high-beta frequency bands.

Regarding the RWCV CA, we used 4-fold CV on each run. For classDis, restDis, classStab, spaspecClassDis, spaspecRestDis, and spaspecClassStab the trial covariance matrices were estimated using automatic shrinkage with the algorithm proposed in [START_REF] Ledoit | A well-conditioned estimator for large-dimensional covariance matrices[END_REF].

Results

We first present the results of the various metrics averaged over all subjects of each data set, to provide an overview of what they measure. Then, for each data set we focus on some specific subjects, to reveal more specific behaviors of each metric. Note that when presenting the different metrics on the same Figure, we used Z-scores as these metrics use different units (percentage for CA and RWCV CA, arbitrary distance units for classDis, restDis and classStab).

Average results

Data set 1: Single session, 2-class motor imagery data set

Figure 1 shows the average measures of distinctiveness between classes (MI tasks), i.e., CA, RWCV CA, classDis and spaspecClassDis, for each run of data set 1 (2-class motor imagery). CA displays some oscillations in performance, but no global increase in performance over runs. On the other hand, RWCV CA, classDis and spaspecClassDis reveal a clear continuous increase in distinctiveness between classes over runs. The 2way ANOVA Metric*Run (Metric: CA, RWCV CA, classDis, spaspecClassDis -transformed to z-score to enable comparisons; Run: 2 to 5) for repeated measures showed a trend towards a metric*run interaction [F(1,19) = 3.081; p = 0.095; η 2 = 0.140], see also Figure 2.

Figures 3a and3b show the class-specific performance metrics, i.e., class-wise CA, class-wise RWCA, restDis and spaspecRestDis. Here as well, CA does not show any obvious increase in performance over runs, while both RWCV CA, restDis and spaspecRestDis show some performance increase over runs, notably for class 2 (right hand motor imagery). The 3-way ANOVA Metric*Class*Run (Metric: CWCA, CW

Data set 2: Multi-sessions, 3-class mental imagery data set

As further described here-after, it is interesting to observe that, on this data set, spatiospectral metrics revealed different performance variation dynamics than those revealed by the purely spatial metrics.

In particular, regarding the measures of distinctiveness between classes (MI tasks), i.e., CA, RWCV CA, classDis and spaspecClassDis (See Figure 5), spaspecClassDis revealed a continuous increase in performances over all sessions except the last one, whereas classDis rather revealed an increase of performance over the first 3 runs followed by a decrease in performance for the subsequent runs. In contrast, CA showed a continuous decrease of performance over sessions. The 2-way ANOVA Metric*Session (Metric: CA, RWCV CA, classDis, spaspecClassDis (z-scores); Session: 1 to 6) for repeated measures showed a significant metric*run interaction [F(1,17) = 4.502; While it was not the case with the previous data set, here significant differences are As can be seen on Figure 9, different performance variations over sessions seem to occur, on average, in the different bands. In particular, with classDis, there is a continuous positive increase in performance, which may be due to learning, that can be observed across sessions around the alpha band (8-10Hz, 10-12 Hz) and low beta one , whereas in the high beta band (18-30Hz), after a sharp performance increase in the first session, performances tended to decrease over the subsequent sessions This could explain the inverted U-shape performance obtained with spatial class-Dis, with performances first increasing over the first 3 sessions, and then decreasing, which could be the result of a simultaneous increase in performance in alpha and decrease in performance in high beta. Similarly, when looking at classStab, we can see a positive increase in stability over sessions in alpha, no obvious change in low beta, and a decrease in performances in high-beta.

Altogether these results seem worth considering: they suggest that even when using the classifier output in a broad EEG band (here 8-30Hz) as feedback, users can increase their performance in a more specific band only, which was mostly in alpha for this protocol and subjects. This then stresses the need to monitor the progresses in each band, in order to 1) be able to observe such performance increase that may reflect learning, if any and 2) possibly restrain the feedback/training tasks or the EEG features to focus on that frequency band in which there may seem to be a more efficient learning. The metrics we propose in this paper enable us to do exactly that kind of monitoring.

Some subject specific results

As stated earlier, we observed a high inter-subject variability, therefore it is interesting to further investigate the different patterns observed in terms of metrics' evolution across the runs, for individual subjects. It will enable the analysis of the behavior of the different metrics and provide insights on their pros and cons.

Data set 1: Single session, 2-class motor imagery data set

In this data set, for instance, all the distinctiveness measures for subject S4 could reveal a clear performance improvement over time, possibly due to learning. However, the same metrics for subject S5 did not show any performance improvement over time with the online CA, whereas both RWCV CA, classDis and spaspecClassDis revealed a clear performance increase over runs, which might me due to some form of learning (see Fig. 11). Metrics for subject S9 (Fig. 12) revealed another interesting phe-Figure 5: The average measures of distinctiveness between classes, across runs and sessions, for data set 2. Different sessions (i.e., days) are displayed in different colors, and the consecutive runs from the same session are linked to each other. nomenon. While both CA and RWCV CA did not show any performance increase, classDis and spaspecClassDis did. However, both restDis and spaspecRestDis revealed that class 1 (left hand motor imagery) actually became increasingly more similar to rest EEG over the runs (restDis and spaspecRestDis for class 1 sharply decreased from run 2), and thus that the increased classDis was probably due to the BCI discriminating rest vs right-hand MI rather than left-vs right-hand MI. CA cannot identify such a phenomenon since it ignores rest EEG.

Finally, analyses of Subject S19's data (Fig. 13) showed decreasing class discriminability with CA, RWCV CA, classDis and spaspecClassDis. However, the data revealed some continuous performance increase over runs with both restDis and spaspecRest-Dis, for both classes. One possible hypothesis to explain this phenomenon could be that this subject learned to modulate their EEG signals so that they differ from rest EEG, but may have more trouble generating consistently distinct patterns between both the MI tasks. Such a phenomenon has also been observed with simultaneous EEG-fMRI in [START_REF] Zich | Real-time EEG feedback during simultaneous EEG-fMRI identifies the cortical signature of motor imagery[END_REF], in which some subjects showed modulations of brain activity during MI with respect to rest signals, but no lateralization of the patterns. The restDis and/or and spaspecRestDis metrics could thus be a cheap and easy way to identify this phenomenon in EEG.

Data set 2: Multi-sessions, 3-class mental imagery data set

Regarding the multi-sessions data set, relevant individual results include for instance those of subject s11. With this subject, all metrics of class distinctiveness measured a continuous increase in performance over sessions, see Figure 14.

However, the same metrics did not measure similar performance dynamics on the data of subject s9, see Figure 15. On this subject, the CA metric suggested that the distinctiveness between classes actually decreased continuously across runs and sessions, whereas both RWCV-CA, classDis and spaspecClassDis actually measured the exact opposite: a continuous increase in performances across runs and sessions. Note here that, again, CA does not actually measure how distinct the EEG patterns from each class are, but how well the classifier trained on the data from the calibration run can distinguish these patterns. Contrary to CA, both RWCV-CA, classDis and spaspec-ClassDis are training data independent, and actually measure how distinct the class EEG patterns from each run are. As such, they can reveal what could be interpreted here as a strong learning effect (see the Discussion in Section 5 for more details about this).

A last interesting case is that of subject s3, presented in Figure 16. With this sub- ject, the distinctiveness between classes actually did not really show a specific performance increase over sessions. However, looking at the class-specific distinctiveness does reveal a gradual performance increase, that may suggest some learning: the EEG patterns from each class actually gradually became more distinct from rest EEG patterns over runs and sessions, as measured by restDis and spaspecRestDis. Only restDis and spaspecRestDis can reveal such performance increase since only them actually consider rest EEG data, which other metrics based on classification accuracy usually ignore.

Do the new metrics confirm the influence of spatial abilities on MI-BCI performances?

As indicated earlier (see Section 3.5.3), for both data set 1 and data set 2, the original online experiments revealed a significant correlation between online CA (average peak CA over the runs for data set 1 and average mean CA over the sessions for data set 2) and spatial abilities [START_REF] Jeunet | Why standard brain-computer interface (BCI) training protocols should be changed: An experimental study[END_REF][START_REF] Jeunet | Predicting Mental Imagery-Based BCI Performance from Personality, Cognitive Profile and Neurophysiological Patterns[END_REF]. Participants' spatial abilities were assessed from the mental rotation score they obtained after completing the Mental Rotation Test of Vandenberg [START_REF] Vandenberg | Mental rotations, a group test of threedimensional spatial visualization[END_REF]. In the present paper, we thus also computed these correlations with our new metrics. We performed partial correlations in order to control for the gender effect associated with mental rotation scores [START_REF] Vandenberg | Mental rotations, a group test of threedimensional spatial visualization[END_REF]. Regarding data set 1, a significant correlation was revealed between mental rotation scores and average RWCV CA

Discussion

Globally, average results showed either a trend towards significance (data set 1) or a significant (data set 2) metric*run interaction. This suggested that some metrics (here RWCV CA, classDis and spaspecClassDis) revealed continuous increase in performance while another (CA) did not. Such continuous increase in performance may arguably be due to learning, with subjects MI-BCI skills gradually improving with practice. In the BCI literature, studies usually consider that learning occurred when CA gradually increased over runs and/or sessions [START_REF] Kübler | How much learning is involved in BCI-control?[END_REF][START_REF] Neuper | Brain-Computer Interfaces, chapter Neurofeedback Training for BCI Control[END_REF][START_REF] Jeunet | Advances in user-training for mentalimagery-based BCI control: Psychological and cognitive factors and their neural correlates[END_REF]. It is worth noting, however, that a continuous increase of performance over time, whatever the performance Figure 12: Subject S9, for which class 1 (left hand motor imagery) became like rest metrics (i.e., both CA, RWCV CA and classDis), might also be due to some alternative factors other than learning. For instance, electrode impedance might decrease over time due to sweat or due to the gel moving and making better contact with the scalp. This would in turn increase the EEG signal-to-noise ratio and thus possibly increase CA or classDis, independently of any learning from the subject. However, we would like to stress that on data set 2, which was based on six sessions, over six different days, we also observed increase in class distinctiveness over sessions (i.e., over days), both on several individual subjects and on average over all subjects. Observing a continu-Figure 13: Subject S19 produced EEG patterns increasingly more different than rest, but not distinct from each other.

ous reduction of impedance over several separate days would seem extremely unlikely. Therefore, while we cannot rule out for sure alternative explanations and prove this is learning, learning remains the most likely explanation for a continuous increase of a performance metric (either classDis or restDis) over days. Thus, in the following discussion, we consider that such continuous increase in performance reflect learning. This is indeed the most likely hypothesis, but we keep in mind that this is not the only one. Overall, RWCV CA, classDis and spaspecClassDis seemed to reveal some form of user learning that CA seemed to have missed. This is all the more interesting given the fact that the feedback was based on the CA metric. Indeed, participants were asked to make the bar feedback, that is proportional to the classifier output and thus related to the online CA, as long as possible in the correct direction. Despite such feedback being based on a possibly incomplete metric (as it may miss some learning), most of the participants demonstrated to be able to modulate their EEG patterns increasingly better, sometimes leading to the improvement of the performance metrics. This result is promising for the future as it suggests that with a better feedback, e.g., a feedback directly related to our new metrics, the ability of the participants to learn to modulate efficiently their EEG patterns, in order to improve their BCI control, may be enhanced.

This result also raises the question of whether different aspects of "self-regulation learning" might have been involved. Indeed, two aspects may have taken place here. The first aspect would be voluntary learning, when users learned from BCI feedback how to self-regulate their EEG patterns by identifying the best mental strategies. This learning aspect thus involved cognitive processes underlain by specific neurophysiological activities, and would be the most typical aspect of learning that is targeted by feedback training in BCI. An additional aspect of learning may have been involved here though. This aspect of learning might have occurred in parallel, due to habituation or repeated practice of the mental imagery tasks, which may have induced neuronal plasticity. Indeed, the feedback provided to users was not directly related to our new met-Figure 15: Subject s9, data set 2, for which CA suggested a continuous decrease in distinctiveness between classes whereas both RWCV-CA, classDis and spaspecClassDis -being training data independent -actually measured a continuous increase in distinctiveness across sessions. Different sessions (i.e., days) are displayed in different colors, and the consecutive runs from the same session are linked to each other. rics of performance, yet users managed to improve such metrics over time. This might suggest a form of learning that occurred without a dedicated feedback. In the future, it would be interesting to study to which extent each of these two aspects of learning is involved. Note that our new metrics are nonetheless not completely orthogonal to the online classifier output. As such, the online feedback might still carry some useful information that users might have used to increase their performances as measured by our metrics.

On the other hand, these results also suggested that different performance metrics can reveal different aspects of MI-BCI users' skills and learning. Notably, they first showed that CA may sometimes be unable to measure that users can modulate their EEG patterns using MI increasingly well, whereas metrics such as RWCV CA, class-Dis and spaspecClassDis can reveal performance increase over runs and sessions, most likely related to user learning. They even revealed fast performance increase, and thus possibly fast learning effects, in several subjects, with continuous progress over runs, over a single day of training in data set 1. This can have profound implications for the study of BCI user training. For instance, the present results may explain why in [START_REF] Kübler | How much learning is involved in BCI-control?[END_REF], it has been concluded that most BCI studies -and notably those based on machine learning -do not actually involve human learning (defined as continuous CA increase). Indeed, in most of the studies surveyed in [START_REF] Kübler | How much learning is involved in BCI-control?[END_REF], CA was used as the performance metric. As such, human learning might have occurred, but CA might not have been able to measure it. It thus seems necessary to re-analyze EEG data from previous studies with complementary performance metrics such as the ones proposed here, to obtain complementary measures that may reflect whether human learning could have occurred.

The performance increase over runs and sessions, and thus possible learning effects, that were revealed by the new metrics also stress the need for co-adaptive BCI systems, and explain the success of these approaches, see, e.g., [START_REF] Faller | A co-adaptive brain-computer interface for end users with severe motor impairment[END_REF][START_REF] Vidaurre | Machine-learningbased coadaptive calibration for brain-computer interfaces[END_REF]. On a related note, it would be relevant to compare the new metrics proposed in this paper with online classification accuracies obtained with adaptive classifiers. Indeed, adaptive classifiers being continuously updated, they are much less sensitive to the initial training data. The classification accuracy they provide is thus more likely to provide a more faithful account of the BCI user's skills. In the data sets studied here, subjects were trained online using a fixed classifier though, as in most BCI studies.

When comparing the spatial metrics to the spatio-spectral ones, it appeared that the latter were more likely to reveal performance increases and thus possible learning effects, due to different users' performance variations in different frequency bands. We would thus advocate for reporting these spatio-spectral metrics, or at least to report the spatial ones in various frequency bands.

The restDis and spaspecRestDis metrics also highlighted the need to consider rest EEG when evaluating MI-BCI users' skills. Not doing so may prevent us from realizing that the user is not able to perform one of the MI tasks. If the target BCI application actually requires the user to perform real MI, e.g., for stroke rehabilitation, this aspect should be monitored. On the other hand, as seen on the multi-sessions data set (data set 2), restDis/spaspecRestDis also showed that some subjects actually improved restDis and spaspecRestDis across sessions but not classDis nor spaspecClassDis, which can only be observed using such metrics. For such types of subjects, this means that it might be more efficient to use one of the mental tasks as a brain switch 4 , since that mental task leads to EEG patterns that are distinct from those of rest EEG, but not so much from those related to the other mental tasks. Alternatively, this might mean that it might be worth specifically training the user to make the different patterns more distinct from each other.

It should be mentioned that our new metrics are relative metrics and not absolute ones. Indeed, the distance between matrices depends on the dimension of these matrices. Matrices with larger dimensions would tend to be further away from each other, the same way as vectors with larger dimensions would tend to be further away from each other [START_REF] Friedman | On bias, variance, 0/1-loss, and the curse-of-dimensionality[END_REF]. Thus, although our metrics can be very useful to compare training procedures or feedbacks between each other, when using the same EEG cap and channels, they cannot be used to compare experiments with different number of channels between each other. Thus, our metrics do not aim at replacing classification accuracy, but rather at complementing it. Indeed using our metrics should maximize the chance to measure and compare MI-BCI skills improvement over time (i.e., possible learning effects), which classification accuracy, as used online, is likely to miss, as well as to understand in a more refine way what skills the user has learned or not. For instance, what mental commands the user is mastering increasingly well, or how stable their EEG patterns are should be studied with the metrics proposed here, i.e., classDis, rest-Dis, classStab, and/or spaspecClassDis, spaspecRestDist, spaspecClassStab. In the future, it might be worth considering using such metrics as dedicated feedback in initial training tasks for BCI, to train explicitly distinctiveness and stability. Then, once the users managed to produce stable and distinct EEG patterns, as measured using such metrics, then can be trained for BCI control using a classifier, as currently done.

It is interesting to note that for the data sets analyzed, RWCV CA and class-Dis/spaspecClassDis generally measured consistent performance dynamics: they usually both measured a positive performance increase at the same time, when CA may not have. This seems to stress the importance of metrics that are training data independent to measure MI-BCI users' skills and learning. Indeed, contrary to CA, neither RWCV CA nor classDis/spaspecClassDis depend on the training data from the calibration run used to obtain the online classifier. As such, if users managed to improve the distinctiveness of their EEG patterns, possibly in a direction that the online classifier will miss, both RWCA CV and classDis/spaspecClassDis will be able to measure it. Such metrics should be thus be preferred to study user MI-BCI skills and learning. Again, CA tells us how well the classifier can recognize the EEG patterns produced by the user, but not necessarily how well the user can produce these patterns and make them as stable and distinct as possible. While RWCV CA is thus also very useful to study the distinctiveness between the EEG patterns produced by the user, it cannot tell us how such patterns differ from rest EEG patterns nor how stable they are. On the other hand, that is something that our new metrics restDis and classStab can measure, and were designed to do. Thus, our new metrics not only address the limitations of current measures such as CA, but also enable us to look at other aspects of MI-BCI users' skills that current metrics cannot see.

When studying possible correlations between our new metrics and spatial abilities, as measured using mental rotation test scores, significant correlations were obtained, in particular for data set 2. This result seems to further confirm the relevance of spatial abilities as a major predictor of performance for MI-BCIs [START_REF] Jeunet | Advances in user-training for mentalimagery-based BCI control: Psychological and cognitive factors and their neural correlates[END_REF]. As argued in [START_REF] Jeunet | Advances in user-training for mentalimagery-based BCI control: Psychological and cognitive factors and their neural correlates[END_REF], spatial abilities corresponding to the ability of producing, manipulating and transforming mental images [START_REF] Poltrock | Individual differences in visual imagery and spatial ability[END_REF], it makes sense that they influence MI-BCI performance.

Altogether, these new metrics led to new ways to look at the data and at MI-BCI users' skills. They suggested that MI-BCI control skills are multidimensional, and cannot be summarized by using only CA. In particular, our metrics suggested that MI-BCI user training did not only influence the distinctiveness between classes, but also the distinctiveness between each class and rest EEG signals, or the stability of each class EEG patterns, possibly in a different way in different frequency bands. All those metrics thus seem to reflect different aspects/components of MI-BCI control skills, which we denote as "subskills" in the following. Our results with the new metrics also suggested that different users might acquire these various MI-BCI control subskills in different ways. For instance, some users managed to improve the distinctiveness of each class with rest EEG patterns over time but not the distinctiveness between classes, whether it was the opposite for some other users. In turns, this calls for new ways to analyze and improve MI-BCI user training, by taking into account such MI-BCI subskills. For instance, different neuropsychological factors might be needed to predict performances for each of the different BCI subskills (restDis, classDis, classStabpossibly in different frequency bands). Along the same lines, since MI-BCI skills seem to gather several subskills, it might mean that BCI training tasks and feedbacks should also target and reflect such subskills. For instance, it might be worth designing and studying training tasks dedicated to improve each one of the subskills. Additionally, it might be necessary to provide BCI users with a dedicated feedback for each of the subskills, to inform them about their progress in each of them. These results thus pave the way for many promising ways to refine and improve MI-BCI user training.

Naturally, the proposed metrics are not solving all performance evaluation issues of MI-BCI, and can still be improved. First, contrary to metrics such as CA, there is no closed-form solution to estimate the chance level of these metrics. If one is interesting by chance-level performance, they should resort to permutation tests. These new metrics also implicitly assume that the covariance matrix variability (as measured by the mean absolute deviation) around the mean is uniform (same variability in all direction). If that is not the case in practice, it may distort the distinctiveness and stability metrics. Finally, these metrics do not consider the time dimension. It would be relevant in the future to consider metrics estimating how fast the user can produce the EEG patterns using MI, and how long they can maintain them. There are thus still a lot of room for improvement and/or extensions of these metrics. Nonetheless, our results show that these new metrics still provide some new and essential insights into MI-BCI users' skills and learning, and highlighted major limitations of current metrics for this topic. As such, we advocate that such new metrics should be considered in addition to existing ones when studying MI-BCI users' skills and learning in the future.

Conclusion

In this paper, we argued that CA (online or simulated online), the most used metric to quantify BCI performance, should not be used alone to study MI-BCI users' skills and learning. We indeed identified many limitations of CA for this purpose. We proposed a first definition of MI-BCI skills and proposed new metrics, based on Riemannian distance, to quantify them. Our new metrics can measure how distinct the EEG patterns produced by the user are between each class, how each of them differ from rest EEG patterns, and how stable they are. These metrics all are classifier and training data independent. This makes them theoretically more appropriate to study MI-BCI user skills than CA, which heavily depends on the classifier and training data used -whereas MI-BCI users' skills should be measured independently from them.

An evaluation and comparison of these metrics indeed confirmed that in practice online CA may hide some continuous performance increase that may reflect learning effects and cannot identify how different an MI class is from rest EEG. They also revealed that different users seem to learn MI-BCI skills in different ways, thus highlighting various MI-BCI subskills. We therefore conclude that, when studying MI-BCI users' skills and learning, CA should be used with care, and should be complemented with metrics such as the ones proposed. Our results also stress the need to redefine MI-BCI user training by considering the different MI-BCI subskills and their measures. In order to ease the adoption of such metrics by the community, we provide their Matlab code for free and open-source on the following webpage: http://sites. google.com/site/fabienlotte/code-and-softwares/metricsofperformance

Naturally, this study would benefit from being replicated on other data sets, with different protocols and classifiers, to further confirm and validate its outcome. Nonetheless, this study and metrics open many promising perspectives. In particular it would be interesting to re-analyze the relationship between users' profile, notably neurophysiological, personality and cognitive profile, and these new performance metrics (so far done by looking for correlation with online CA only, see [START_REF] Jeunet | Advances in user-training for mentalimagery-based BCI control: Psychological and cognitive factors and their neural correlates[END_REF] for a review), which could reveal new predictors of performance, and thus new ways of improving BCI user training. These metrics could also be used as the basis to design new feedbacks, and in particular explanatory feedbacks [START_REF] Schumacher | Towards explanatory feedback for user training in brain-computer interfaces[END_REF]. Indeed, these metrics being based on simple distance measures, they could be computed online, using incrementally estimated average covariance matrices. In contrast, the RWCV CA metric cannot be used online, notably due to its computational cost. The classDis, restDis and classStab metrics could thus be provided as online feedback, to tell users whether they should improve the distinctiveness with rest, with another class, or the stability of their patterns, for instance. These concepts being abstract and unusual for BCI users, a considerable work is needed in terms of user-centered design and human-computer interaction to find out the most consistent, intuitive and pleasant ways to provide such an explanatory feedback. To do so, such features might be combined with engaging and rich tools for real-time visualization of EEG patterns, such as those presented in [START_REF] Mercier-Ganady | The mind-mirror: See your brain in action in your head using EEG and augmented reality[END_REF][START_REF] Frey | Teegi: Tangible EEG interface[END_REF][START_REF] Frey | Scientific outreach with Teegi, a tangible EEG interface to talk about neurotechnologies[END_REF]. These metrics revealing what seemed like fast learning effects, they could also be used as a cheap, possibly online way (faster and more convenient than CV) to identify when to update and retrain classifiers. Finally, it would be relevant to further refine these metrics, for instance by defining sub-metrics, for subsets of EEG channels, over specific brain areas, to study brain area specific learning processes. Overall, we are convinced that MI-BCI user training should be further studied, and we hope these new metrics could be a new way to look at it.

Figure 1 :

 1 Figure 1: The average measures of distinctiveness between classes, across runs, for data set 1.

Figure 2 :

 2 Figure 2: Z-scores for each of the metrics of distinctiveness between classes (CA, RWCV CA, classDis and spaspecClassDis) as a function of the run for the 2-classes data set (data set 1). This graph reveals improvements across runs for all the metrics but CA.

Figure 3 :

 3 Figure 3: Z-scores for each of the metrics of class-specific distinctiveness (CA, RWCV CA, restDis and spaspecRestDis) as a function of the run and of the class (a: left-hand MI (class 1), b: right-hand MI (class 2)) for the 2-classes data set (data set 1).

Figure 4 :

 4 Figure 4: Z-scores for each of the metrics (classStab and spaspecClassStab) as a function of the run and of the class (a: left-hand MI, b: right-hand MI, c: rest) for the 2-classes data set (data set 1).

Figure 6 :

 6 Figure 6: Z-scores for each of the metrics of distinctiveness between classes (CA, RWCV CA, classDis and spaspecClassDis) as a function of the run for the 3-classes data set (data set 2).

Figure 7 :

 7 Figure 7: Z-scores for each of the metrics of class-specific distinctiveness (CA, RWCV CA, restDis and spaspecRestDis) as a function of the run and of the class (a: left-hand MI, b: mental subtraction, c: mental rotation) for the 3-classes data set.

Figure 8 :

 8 Figure 8: Z-scores for each of the stability metrics (classStab and spaspecClassStab) as a function of the run and of the class (a: left-hand MI, b: mental subtraction, c: mental rotation, d: rest) for the 3-classes data set.

  [r = 0.500, p < 0.05], but not with the other metrics: CA [r = 0.373, p = 0.116], classDis [r = 0.125, p = 0.609], spaSpecClassDis [r = 0.069, p = 0.780]. On the other hand, regarding data set 2, significant correlations were revealed between mental rotation scores and all the metrics averaged over sessions: CA [r = 0.519, p < 0.05], RWCV CA [r = 0.535, p < 0.05], classDis [r = 0.609, p < 0.05] and spaSpecClassDis [r = 0.557, p < 0.05].

Figure 9 :

 9 Figure 9: grand average spatial classDis in various frequency bands, over sessions and runs, for data set 2. Different sessions (i.e., days) are displayed in different colors, and the consecutive runs from the same session are linked to each other.

Figure 10 :

 10 Figure 10: grand average spatial classStab in various frequency bands, over sessions and runs, for data set 2. Different sessions (i.e., days) are displayed in different colors, and the consecutive runs from the same session are linked to each other.

Figure 11 :

 11 Figure 11: Examples of 2 subjects for which, either CA measured a learning effect like the other metrics (top -subject S5), or did not whereas the other metrics did (bottomsubject S4)

Figure 14 :

 14 Figure 14: Subject s11, data set 2, for which all 3 metrics (CA, RWCV-CA, classDis) all measured a continuous increase in performance across sessions. Different sessions (i.e., days) are displayed in different colors, and the consecutive runs from the same session are linked to each other.

Figure 16 :

 16 Figure 16: Subject s3, data set 2, for which there was no increase in distinctiveness between classes (as seen by all metrics -bottom figure), but for which the distinctiveness with the rest EEG patterns (restDis and spaspecRestDis -top Figure) increased across sessions, which only restDis and/or spaspecRestDis could measure. Different sessions (i.e., days) are displayed in different colors, and the consecutive runs from the same session are linked to each other.

Table 1 :

 1 Confusion matrix for two classes

		Estimated class
		Class 1	Class 2
	Real Class 1	a	b
	Class Class 2	c	d

Table 1 :

 1 Basic distinctiveness and stability metrics

		Basic metric
	Class distinctiveness (2 classes)	classDis(A, B)

Table 2 :

 2 Spatio-spectral variants of the core metrics, considering N f frequency bands F j

		spatio-spectral metric
	spatio-spectral class distinctiveness	spaspecClassDis(A, B) =

Preliminary results with a subset of the metrics proposed here, and on a single data set, with only two classes and a single session per subject, have been published as a conference paper in[START_REF] Lotte | Online classification accuracy is a poor metric to study mental imagery-based BCI user learning: an experimental demonstration and new metrics[END_REF]

http://www.statsf1.com/en/michael-schumacher/victoire.aspx

In this discussion, we use the word classifier in a broad sense, i.e., including the feature extraction and filtering steps. In other words, the classifier refers here to all the processing pipeline translating EEG signals into a control command.

A brain switch is a single-class BCI, which uses a single mental task to send a command when the EEG patterns evoked by this task becomes different than that of rest EEG signals, see, e.g.,[START_REF] Müller-Putz | Fast set-up asynchronous brain-switch based on detection of foot motor imagery in 1-channel EEG[END_REF]

Acknowledgments

This work was supported by the French National Research Agency with the REBEL project (grant ANR-15-CE23-0013-01), the European Research Council with the Brain-Conquest project (grant ERC-2016-STG-714567) as well as by the EPFL/Inria International Lab.