Quang-Vinh Dang
email: quang-vinh.dang@inria.fr

Jérôme Franc
email: jerome.francois@inria.fr

Utilizing attack enumerations to study SDN/NFV vulnerabilities

Several cybersecurity attack enumerations area available today. These enumerations present lists of known attack patterns (CAPEC), security weaknesses (CWE) or cybersecurity vulnerabilities (CVE). These enumerations are being developed separately and manually.

In this paper, we present the efforts in determining the relations between enumerations automatically. We rely on text-based, graph-based and recommendation-based approaches. Then we present of using the prediction in recommending related attacks to SDN/NFV security issues.

Experimental results showed that we can actually infer real relations. Furthermore, the results gave some insights into how the enumerations are created and linked, and some suggestions to improve the process in the future.

I. INTRODUCTION

Network softwarization with SDN and NFV attracts a lot of attention from academia and industry in recent years. Indeed, while the use of softwarized networks is increased, more vulnerabilities will be found and used by attackers. In order to handle efficiently security threats, security experts share their knowledge. Hence, most important and common attack patterns and vulnerabilities are collected and organized in attack enumerations. Recent years have seen attack descriptions related to softwarized networks.

Using standard knowledge representations and enumerations is a popular approach in dealing with security threats [START_REF] Martin | Making security measurable and manageable[END_REF]. Several well-known security enumerations are developed and maintained that provide to the community a well-annotated knowledge repository on cyber-security issues. CAPEC (Common Attack Pattern Enumeration and Classification), CWE (Common Weakness Enumeration) and CVE (Common Vulnerabilities and Exposures) are among them [START_REF] Wang | Ovm: an ontology for vulnerability management[END_REF]. The enumerations provide open-source lists of security issues that can be used by both researchers and practitioners.

Indeed, there are relations between these enumerations. By presenting relations between them, we can extend our knowledge regarding a particular problem. For instance, when a reader checks a particular CVE item, she can benefit by checking other related CAPEC or CWE items. Nevertheless the relations are being created manually that cannot cope with the large scale of these enumerations.

However, few research exists on finding the relations between these enumerations automatically, as well as suggesting possible related items. In this paper we present several approaches for relation inference. Then we apply these approaches to find attack patterns and software weaknesses that are related to SDN/NFV software vulnerabilities.

A. Enumerations

MITRE, which is a not-for-profit company 1 , maintains three security enumerations that are investigated in this paper.

• CAPEC (Common Attack Pattern Enumeration and Classification) [START_REF] Mitre | Common attack pattern enumeration and classification (CAPEC)[END_REF] is a public catalog of common attack patterns classified intuitively. • CWE (Common Weakness Enumeration) [START_REF]Common weakness enumeration (CWE)[END_REF] is a formal list of software weakness types. • CVE (Common Vulnerabilities and Exposures) [START_REF]Common vulnerabilities and exposures (CVE)[END_REF] is a list of common identifiers for publicly known cybersecurity vulnerabilities. Ranking by abstraction level, we could say that CAPEC provides abstract information about attack pattern, CWE provides a list of practical weaknesses that attackers can exploit, and CVE provides the practical methods that attackers can use.

Definitely there are relations between the enumerations. For instance, MITRE defined that the CAPEC item with ID "1" (Accessing Functionality Not Properly Constrained by ACLs) is related to several CWE items with the following IDs: 285 (Improper Authorization), 732 (Incorrect Permission Assignment for Critical Resource) etc. However these relations have been defined manually. It raises several problems:

• It is difficult to cope with large enumerations manually.

• It is difficult to define relationships through multiple enumerations. For instance, there is no direct CAPEC-CVE relations defined. It will be more difficult if we include more enumerations in the future. • It is highly probable to miss a potential relation, or create a not-related relation. In this paper, we address two problems:

• How to infer a relation between items that belong to different enumerations? In other words, how to reconstruct the links established by human experts? We refer the problem as link prediction.

• Given an item, how to recommend related items that belong to other enumerations? In other words, we want to find potential links that human experts might miss. We refer the problem as link recommendation.

As a result, we provide the security practitioners with more useful information to understand and handle a security incidents, i.e. indicating other threats which seems very similar by nature and for which she can thus apply a similar corrective action.

In this paper, we present the enumerations as a graph with nodes as items and edges as relations between them. Hence the goal is to analyze and predict missing links. By using graph-, text-and matrix-based features, we can effectively predict the links that are established manually and recommend new relations that did not exist in the data. We particularly apply our techniques to threats and vulnerabilities related to softwarized networks.

The paper is organized as follows. In Section II we present the related studies. We describe our methods in details in Section III. The experimental results are discussed in Section IV. We conclude our paper and draw some potential future works in Section V.

II. RELATED WORK

Since its beginning in 1999 [START_REF] Martin | Making security measurable and manageable[END_REF], CAPEC, CWE and CVE have been used as important enumerations in security analysis [START_REF] Navarro | Huma: A multi-layer framework for threat analysis in a heterogeneous log environment[END_REF], attack modelling [START_REF] Kotenko | The CAPEC based generator of attack scenarios for network security evaluation[END_REF] or risk assessment [START_REF] Madria | Offline risk assessment of cloud service providers[END_REF].

To the best of our knowledge, there is not yet a comprehensive study on relations between different enumerations. In fact, several studies focused on analyzing a single enumeration and its internal dependencies, or analyzing established relations that are created manually by researchers.

Several research works focus only on CAPEC data. The author of [START_REF] Noel | Interactive visualization and text mining for the capec cyber attack catalog[END_REF] presented a work on taxonomy analysis and visualization of CAPEC. The study analyzed the relations within CAPEC items in different visualized approaches such as clustering or hierarchical analysis. However, the study does not analyze the CAPEC items in the relation with external enumerations like CVE. Furthermore, the study focuses on established relations rather than discovering the new ones. Related to [START_REF] Noel | Interactive visualization and text mining for the capec cyber attack catalog[END_REF], the authors of [START_REF] Wang | Ovm: an ontology for vulnerability management[END_REF] analyzed the established relations to build an ontology rather than studying the missing links between enumerations.

Regarding CVE analysis there exist the works of [START_REF] Chen | A categorization framework for common computer vulnerabilities and exposures[END_REF], [START_REF] Toloudis | Associating the severity of vulnerabilities with their description[END_REF] or [START_REF] Neuhaus | Security trend analysis with CVE topic models[END_REF]. The authors analyzed the internal dependencies inside the CVE enumeration but do not extend the analysis to external knowledge repositories.

In other further analyses and usage of ontology based on open attack enumerations [START_REF] Wang | Ranking attacks based on vulnerability analysis[END_REF] there is no inference on finding new relations between CAPEC, CWE and CVE.

The authors of [START_REF] Scarabeo | Mining known attack patterns from security-related events[END_REF] presented a similar approach with us by using TF-IDF (Term Frequency -Inverse Document Frequency) to measure the similarity of CAPEC items and event-log. However, the authors do not utilize graph and matrix-based information. Furthermore, the authors focused on a different objective: they want to match a CAPEC item with an event-log rather than a CWE or CVE item.

Partly the work of [START_REF] Navarro | Huma: A multi-layer framework for threat analysis in a heterogeneous log environment[END_REF] shared a similar goal with our study. However, similar to [START_REF] Scarabeo | Mining known attack patterns from security-related events[END_REF], the authors of [START_REF] Navarro | Huma: A multi-layer framework for threat analysis in a heterogeneous log environment[END_REF] do not utilize the graph information. Furthermore, the authors used word2vec [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF] that usually requires a large training dataset, and the result of word2vec is not clear and easy to interpret as the result of TF-IDF. On the other hand, akin to [START_REF] Scarabeo | Mining known attack patterns from security-related events[END_REF], the authors of [START_REF] Navarro | Huma: A multi-layer framework for threat analysis in a heterogeneous log environment[END_REF] utilized an unsupervised learning without an explicit evaluation. Hence, it is difficult to evaluate the performance of the matching process. In our study we evaluate the recommendation by both manual and automated evaluations.

In the PhD thesis of [START_REF] Lyons | A recommender system in the cyber defense domain[END_REF] the author elaborated hybridrecommendation techniques for defender counteractions. However, as we will show in the Section IV, the traditional recommendation techniques are not quite useful in our scenario due to the sparsity of the graph.

In this paper, we analyze the external relations over multiple enumerations. We aim to not only analyzing and reconstructing established relations but also discovering and recommending the new relations that did not exist yet.

III. METHODOLOGY

1) Graph-based approach: By representing enumerations and relations between them as a graph, we turn the problem into link-prediction problem in graphs [START_REF] Martínez | A survey of link prediction in complex networks[END_REF]. The nodes of the graph are items in enumerations, and links between them are relations defined in the enumerations. We consider only nondirected graphs, because the relations are bi-directional.

Intuitively, if two nodes are more similar, it is more likely that they are related. Therefore we need to define a similarity function.

We leveraged different similarity-based link-prediction algorithms: s 1 , s 2 and s 3 . We use Γ x to denote the list of neighbors of node x in the graph.

• Adamic-Adar index [START_REF] Adamic | Friends and neighbors on the web[END_REF] is a similarity measure that calculates the similarity score between two nodes as:

s 1 (x, y) = z∈Γx∩Γy 1 log|Γ z | (1)
• Resource allocation [START_REF] Zhou | Predicting missing links via local information[END_REF] is motivated by the resource allocation process in complex graphs.

s 2 (x, y) = z∈Γx∩Γy 1 |Γ z | (2)
Adamic-Adar and Resoure Allocation metrics perform well in social networks [START_REF] Martínez | A survey of link prediction in complex networks[END_REF] due to the fact that human might actively seek to new connections. However, in an extremely sparse graph, the existing relations might not enough to verify the similarity of the nodes. Hence we use the third metric that is Preferential Attachment. • Preferential attachment is motivated from the Barabaśi-Albert graph model [START_REF] Barabási | Emergence of scaling in random networks[END_REF]. The main idea of the model is "the rich get richer", i.e. if a node has a high connection degree, it is more likely that it will be connected in the future. More precisely, in Barabaśi-Albert graph model nodes are added one by one. When a new node is added, it will connect with each existing nodes with a probability that is proportional to the number of links that the existing nodes already have. Formally, suppose a new node x is added to the graph G =< V, E >, the probability that x will connect to the node i is:

p i = |Γ i | k∈V |Γ k | (3)
The preferential attachment similarity score is calculated as:

s 3 (x, y) = |Γ x ||Γ y | (4)
2) Text-based approach: Because items are text-based description, comparing the text would be helpful to detect those which are related to each other. Text similarity calculation is an active research trend today. Multiple approaches have been defined in last several decades. However, in this study we focus on using Term Frequency (TF) -Inverse Document Frequency (IDF) [START_REF] Jones | A statistical interpretation of term specificity and its application in retrieval[END_REF] technique due to several reasons:

• TF-IDF does not require a huge training dataset.

• Based on our observations, the relations between two items usually can be determined by keywords. For instance, if two items both mention some keywords like buffer over-read or ACL misconfiguration, it is likely that the two items are related. Using TF-IDF we can effectively extract keywords from contents of enumeration items.

• The items of the security enumerations are written in semi-formal language, i.e. they follow some specific writing styles for scientific articles. Therefore we do not need to worry about the semantic difference. TF of a term t in a document d is calculated as count(t, d)/|d| and IDF of a term t for a document d in a set of document D is calculate as log[n/df (t)] + 1. Here, we notate count(t, d) as the number of occurrences of term t in the document d, and df (t) is the number of documents that contains the term t. The similarity score is calculated by using cosine distance function. Therefore, a keyword specific to a document is a word which is frequent in this document but very rare in other ones.

In order to apply TF-IDF in an efficient manner, considered terms for calculation are limited. Stop-words and terms with a frequency higher than 95% are removed since they are not representative and so will have a very IDF factor. Also, the terms with a frequency lower than 5% are also removed as they are not representative of the document (a low TF value in that case).

3) Recommendation-based approach: If we present the enumerations and the relations as a matrix, the task of relation prediction turns to be matrix completion [START_REF] Pech | Link prediction via matrix completion[END_REF]. We applied non-negative matrix factorization (NMF) technique [START_REF] Lee | Algorithms for non-negative matrix factorization[END_REF] from recommendation systems to fill out the missing values.

In order to doing so, we present the relations between enumerations as a matrix. For instance, we can present CAPEC-CWE relations as a matrix D with rows are CAPEC items and The task is done by finding a factorization of a matrix D such as D ≈ P × Q in such P presents CAPEC items and Q presents CWE items. The predicting value of D(i, j) = P i .Q T j is calculated by inner product. We visualize the process in Figure 1.

We note that all above methods are used for the link prediction task, but only text-based method is used for link recommendation task.

IV. EVALUATION A. Datasets

In this work, three public datasets are used:

• CAPEC2 version 2.11 that contains 545 definitions.

• CWE3 version 3.0 that contains 714 definitions.

• CVE4 data created on 06-Feb-2018 that contains 122, 866 definitions. We display the relations between three datasets in Figure 2. The links represent the relations between items in corresponding datasets. For instance, one CAPEC item can have relations with multiple CAPEC items as well as multiple CWE items.

B. Metrics

Because we address two problems (prediction and recommendation), two sets of metrics have been defined.

a) Term Definitions: To recall, if we represent the predicted labels and true labels as in Table I.

We calculate the accuracy, recall (or True Positive Rate), precision and FPR (False Positive Rate) as follows. Accuracy measures the ratio of correct prediction over the total number of prediction. Precision measures the ratio of correct positive prediction over the total number of positive prediction. Recall measures the ratio of positive instances that the model can retrieve from the predictive dataset (testing set).

accuracy = T P + T N /(T P +

F P + F N + T N ) (5)
recall = T P /(T P + F N )

precision = T P /(T P + F P )

F P R = F P /(F P + T N ) (7)
The AUC score is calculated as the area under the ROC curve as visualized in Figure 3. ROC curve is formed by two axes: TPR (i.e, precision) and FPR, i.e. it presents the performance of a model regarding a metric (TPR).

1) Link Prediction: We consider the problem of relation prediction as a classification, i.e. given two items, one is a CAPEC item and one is a CWE item, or one is a CWE item and one is a CVE item, we need to predict if there a relation between them exists or not. Therefore, the problem is a binaryprediction problem.

The datasets are actually severely imbalanced. Considering two datasets that contain M and N items respectively, the total number of possible relations between them is M * N . Hence, there is up to 389, 130 relations between CAPEC and CWE, and up to 87, 726, 324 relations between CWE and CVE. By contrast there are only 700 CAPEC-CWE relations and 2, 190 CWE-CVE relations defined. Considering this issue, the regular accuracy metric should not be used [START_REF] Japkowicz | Evaluating learning algorithms: a classification perspective[END_REF] because it can be easily biased by majority classes, in that case the absence of a link.

Therefore, we used Area Under Curve (AUC) score to evaluate our prediction. AUC is suggested to be used in evaluating the prediction on imbalanced dataset [START_REF] Japkowicz | Evaluating learning algorithms: a classification perspective[END_REF] as [START_REF] Huang | Using AUC and accuracy in evaluating learning algorithms[END_REF] showed that is is a better metric in representing the performance of a learning algorithm compared to accuracy. We note that, given the output of our learning algorithm which can be considered as the probability of the existence of links, we prefer AUC to F -1 score. The reason is that AUC metric is defined on a list of probabilistic prediction while F -1 score is defined based on discrete predictions.

2) Link Recommendation: We select a set of SDN/NFV vulnerabilities from CVE dataset, apply our technique to recommend related attack patterns and security weaknesses and present them as a list of ranked related items ranking by the TF-IDF based similarity score. Then we assess manually the relatedness between the items.

In order to test the accuracy of the recommendation in a more automated manner, we also consider the metric recall. For instance, given a CVE item x. In the dataset, the human experts defined that x is related to CAPEC items y 1 and y 2 . In recommendation phase when we temporarily remove the existing links connected to x and present a list of recommended CAPEC items that our system determined as related to x, these recommended links should also include y 1 and y 2 .

C. Experimental Results

1) Link Prediction: In fact, the link prediction problem can be considered as the link-sign prediction problem in graphs [START_REF] Leskovec | Predicting positive and negative links in online social networks[END_REF] if we consider the existence of a link between two nodes as a link with value of "1" and the non-existence of a link is a link with value of "0".

We followed the evaluation configuration of [START_REF] Leskovec | Predicting positive and negative links in online social networks[END_REF]: to predict the link that already existed, we temporarily remove this link from the graph and use the remaining as the training data; and to predict the links that does not exist in the graph we simply use the entire original graph as the training data. The configuration is similar to "leave-one-out" cross validation [START_REF] Japkowicz | Evaluating learning algorithms: a classification perspective[END_REF]. We expect the predictive value in the former case will be close to 1 while the predictive value in the latter case will be close to 0. It is noteworthy that we expect that the values corresponding to the links that actually existed in the dataset should be close to "1". If there is a CAPEC item and a CVE item that seem to be related but the authors of the enumerations did not declare a relation between them, we expect that the value for this non-existed link should close to "0".

In fact, the graph features can be used directly as the unnormalized predictive values.

For instance, we can simply set a rule such as, if the graph feature is greater than 0.5 we will predict the missing link as an existed link and vice versa.

However, in order to increase the performance of the predictive algorithm we additionally performed logistic regression to train a classifier. We also applied sampling preprocessing techniques, i.e. over-sampling, under-sampling and SMOTE [START_REF] Branco | A survey of predictive modeling on imbalanced domains[END_REF] to further improve the predictive power of the classifier.

Over-sampling means that we create multiple copies of the minority class while with under-sampling we only take a subset of majority classes into training dataset. SMOTE [START_REF] Chawla | SMOTE: synthetic minority over-sampling technique[END_REF] stands for Synthetic Minority Over-sampling Technique, means that we create synthetic training data to increase the number of instances from minority classes. The SMOTE algorithm works as follows.

Consider a dataset with two classes A and B with B is the minority classes. We aim to increase the number of instances of B. Suppose that we want to create new T instances of B. We select randomly T instances from the class B. For each instances, we find k nearest neighbours (from B only) and select randomly 1 neighbour from these k instances. The idea is that, the new created synthetic data point will lie somewhere in between the original data point and the neighbour. The pseudo-code is displayed in Figure 4.

First of all, we evaluated the performance of models using different graph features. The prediction result when we use only graph similarity values as a feature is shown in Figure 5. We observed that, using preferential attachment value achieves much higher AUC value than other similarity values. In fact, the best learning model achieves a very high AUC value of 0.925. From now on we use the term graph feature to refer to preferential attachment value.

Secondly, we verified the performance of learning models using each individual feature. We display the result in Figure 6. We observed that the model using graph feature individually has significant improvement compared to other features.

The results shed the lights on the development process of the enumerations. Theoretically, we might expect an uniform distribution of relations, i.e. for instance in linking a new CAPEC item to existing CWE items we should expect that each CWE item has a similar probability to be considered. However the results suggested that the development process of the enumerations follow the Barabaśi-Albert graph model, i.e. when a new item is created, the authors might follow the existing links to determine related items rather than scanning through the entire enumerations. In order to validate the observation, we added text similarity feature into the learning model. We show the result in Figure 7. It turns out that adding text similarity feature decreases the performance of the learning model. In order to determine the links that will be actually added manually to the enumerations, graph features play a more important role than text similarity features.

Similarly, using recommendation-based approaches, i.e. matrix factorization, does not improve the performance of the learning model. It is mostly due to the cold-start problem where the matrix factorization techniques usually fail [START_REF] Tang | Social recommendation: a review[END_REF], i.e. it is because of the matrix is too sparse for the matrix factorization technique can be effectively performed.

2) Recommendation: We selected five CVE items that are related to SDN/NFV domains. The selected CVE items are CVE-2016-3708, CVE-2016-5363, CVE-2017-8189, CVE-2017-9265 and CVE-2017-1000411. To give the reader an idea of a CVE item, we present the content of existing CVE items as follows. We note that no relations from these CVE items to CAPEC or CWE were defined. Therefore, if any recommendation are proposed by our approach, it may bring new information to the enumerations.

CVE-2017-8189: FusionSphere OpenStack V100R006C00SPC102(NFV) has a path traversal vulnerability. Due to insufficient path validation, an attacker with high privilege may exploit this vulnerability to cover some files, causing services abnormal. CVE-2016-3708: Red Hat OpenShift Enterprise 3.2, when multi-tenant SDN is enabled and a build is run in a namespace that would normally be isolated from pods in other namespaces, allows remote authenticated users to access network resources on restricted pods via an s2i build with a builder image that (1) contains ONBUILD commands or (2) does not contain a tar binary. We calculated the similarity scores of these CVE items and recommend the top similar items from CAPEC and CWE enumerations.

For instance, regarding the item CVE-2017-8189, we recommend the following CAPEC and CWE items. Regarding the item CVE-2016-3708, we recommend the following CAPEC and CWE items.

• CAPEC-1: Accessing Functionality Not Properly Constrained by ACLs. • CAPEC-88: OS Command Injection.

• CAPEC-498: Probe iOS Screenshots.

• CWE-285: Improper Authorization.

• CWE-749: Exposed Dangerous Method or Function.

• CWE-862: Missing Authorization. Regarding the subjective evaluation, i.e. based on manual evaluation, while a few recommendations might not be strong related ones, we can see that most recommendations are meaningful (such as CAPEC-17, 23 and CWE-22, 99 for CVE-2017-8189). Therefore, these recommendations can provide valuable information for the authors of the enumerations in the linking process.

Regarding the objective evaluation, we calculate recall@10, i.e. we use the top-10 related items as our recommendation. By this configuration, we achieved the recall metric of 0.912. Hence we conclude that the recommendation are related to the given items.

V. CONCLUSIONS

In this paper, we research on the problem of automatic linking between different security enumerations and recommending attack patterns related to SDN/NFV vulnerabilities. The enumeration development follows the Barabasi and Albert model [START_REF] Barabási | Emergence of scaling in random networks[END_REF] and by using preferential attachment method we can effectively predict the links between enumeration items. On the other hand, by using TF-IDF method we can recommend missing links between enumerations. We demonstrated the usage of the method by recommending attack patterns and software weaknesses related to SDN/NFV vulnerabilities.

In the future, we might include more attack enumerations such as NVD 5 and FIRST 6 or real attack logs to analyze their relationship.

Fig. 1 .

 1 Fig. 1. Matrix factorization

Fig. 2 .

 2 Fig. 2. Relations between datasets

Fig. 4 .

 4 Fig. 4. SMOTE

Fig. 5 .Fig. 6 .

 56 Fig. 5. Performance in term of AUC values using different graph similarity functions.

Fig. 7 .

 7 Fig. 7. Performance in term of AUC values using preferential attachment and TF-IDF values.

• CAPEC- 23 :

 23 File Content Injection. • CAPEC-17: Accessing, Modifying or Executing Executable Files. • CAPEC-69: Target Programs with Elevated Privileges. • CWE-22: Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal'). • CWE-99: Improper Control of Resource Identifiers ('Resource Injection'). • CWE-23: Relative Path Traversal.

TABLE I Fig

 I

. 3. Area Under Curve

https://www.mitre.org

https://capec.mitre.org/data/

https://cwe.mitre.org/data/index.html

http://www.cve.mitre.org/data/downloads/index.html

https://nvd.nist.gov/

https://www.first.org/global/sigs/vrdx/vdb-catalog

ACKNOWLEDGMENT

This work was partially funded by HuMa, a project funded by Bpifrance and Region Grand Est under the FUI 19 framework, and by the NATO Science for Peace and Security Programme under grant G5319 Threat Predict: From Global Social and Technical Big Data to Cyber Threat Forecast. It is also supported by the High Security Lab hosted at Inria Nancy Grand Est.


Valentin Vielzeuf
email: valentin.vielzeuf@orange.com

Corentin Kervadec
email: corentin.kervadec@orange.com

Stéphane Pateux
email: stephane.pateux@orange.com

Alexis Lechervy
email: alexis.lechervy@unicaen.fr

Frédéric Jurie
email: frederic.jurie@unicaen.fr

An Occam's Razor View on Learning Audiovisual Emotion Recognition with Small Training Sets

Keywords: Computing methodologies → Computer vision tasks, Neural networks, Computer vision representations, Emotion Recognition, Deep Learning

INTRODUCTION

Emotion recognition is a current topic of interest, finding many applications such as health care, customer analysis or even face animation. With the advance of deep learning for face analysis, automatic emotion recognition might appear as an already solved problem. Indeed, large image datasets for facial expression recognition in uncontrolled conditions are emerging, allowing to learn accurate deep models on this kind of task. For instance, EmotioNet [START_REF] Benitez-Quiroz | EmotioNet Challenge: Recognition of facial expressions of emotion in the wild[END_REF] gathers one million faces annotated in Action Units [START_REF] Ekman | Facial action coding system[END_REF], AffectNet [START_REF] Mollahosseini | Affectnet: A database for facial expression, valence, and arousal computing in the wild[END_REF] proposes half a million usable faces annotated in both discrete emotion [START_REF] Plutchik | [END_REF] and arousal valence [START_REF] Feldman | The structure of current affect: Controversies and emerging consensus[END_REF], and Real-World Affective Faces [START_REF] Li | Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild[END_REF] is a dataset of around 30,000 faces with very reliable and accurate annotations of the discrete and compound emotions.

However, the algorithms proposed in the literature do not allow yet to reach a human-level understanding of the emotions. It is the reason why several multimodal and temporal datasets have been proposed recently. The AVEC challenge [START_REF] Ringeval | AVEC 2017: Real-life Depression, and Affect Recognition Workshop and Challenge[END_REF] contains a dataset of videos annotated with per-frame arousal valence and several features. The FERA challenge [START_REF] Michel F Valstar | Fera 2017-addressing head pose in the third facial expression recognition and analysis challenge[END_REF] presents a corpus of different head poses to allows action units detection in uncontrolled temporal conditions. Finally AFEW [START_REF] Dhall | Collecting large, richly annotated facial-expression databases from movies[END_REF][START_REF] Dhall | Video and image based emotion recognition challenges in the wild: Emotiw 2015[END_REF] is annotations in discrete emotions of 773 training audiovisual clips extracted from movies. These clips are therefore very noisy, due to the uncontrolled conditions.

Nevertheless these datasets contain a small amount of samples and therefore raise three issues for machine learning approaches: i) how to cope with the temporal aspect of emotions? ii) how to combine the modalities? iii) how to learn a meaningful representation from so few samples? We investigate these issues by focusing on the AFEW dataset annotated with discrete emotion. Several methods have been exploring these questions during the past years. The literature associated with the first challenge editions focuses on hand-crafted features [START_REF] Kächele | Revisiting the EmotiW challenge: how wild is it really?[END_REF][START_REF] Yao | Capturing au-aware facial features and their latent relations for emotion recognition in the wild[END_REF] (e.g. Local Binary Patterns, Gabor filters, Modulation Spectrum, Enhanced AutoCorrelation, Action Units), which are then fed to classifiers such as SVM or Random Forest. After 2015, visual learned features are becoming the dominating approach, with the use of large deep convolutional neural networks [START_REF] Fan | Video-based emotion recognition using cnn-rnn and c3d hybrid networks[END_REF]. To handle the problem of the small size of these datasets, recent approaches [START_REF] Fan | Video-based emotion recognition using cnn-rnn and c3d hybrid networks[END_REF][START_REF] Hu | Learning supervised scoring ensemble for emotion recognition in the wild[END_REF][START_REF] Knyazev | Leveraging large face recognition data for emotion classification[END_REF][START_REF] Vielzeuf | Temporal multimodal fusion for video emotion classification in the wild[END_REF] use transfer learning from models learned on larger image datasets. To handle the temporal nature of the signal, several authors use LSTM recurrent neural networks [START_REF] Fan | Video-based emotion recognition using cnn-rnn and c3d hybrid networks[END_REF][START_REF] Felix A Gers | Learning to forget: Continual prediction with LSTM[END_REF][START_REF] Hu | Learning supervised scoring ensemble for emotion recognition in the wild[END_REF][START_REF] Vielzeuf | Temporal multimodal fusion for video emotion classification in the wild[END_REF] even if no strong improvements have been obtained, compared to other simpler methods, as observed by Knyazev et al. [START_REF] Knyazev | Leveraging large face recognition data for emotion classification[END_REF]. Other authors propose to use 3d convolutions [START_REF] Fan | Video-based emotion recognition using cnn-rnn and c3d hybrid networks[END_REF] and, possibly combined with LSTM [START_REF] Vielzeuf | Temporal multimodal fusion for video emotion classification in the wild[END_REF]. The audio modality is often described by hand-crafted features on top of which a classifier is trained, even in recent approaches [START_REF] Hu | Learning supervised scoring ensemble for emotion recognition in the wild[END_REF][START_REF] Knyazev | Leveraging large face recognition data for emotion classification[END_REF][START_REF] Vielzeuf | Temporal multimodal fusion for video emotion classification in the wild[END_REF]. Pini et al. [26] also propose to use a Soundnet [START_REF] Aytar | Soundnet: Learning sound representations from unlabeled video[END_REF] as a features extractor. Finally, to combine the modalities, late fusion approaches are preferred and performs better on this dataset [START_REF] Vielzeuf | Temporal multimodal fusion for video emotion classification in the wild[END_REF].

We present in this paper a light-weight real-time neural network model based on an "Occam's razor" philosophy, consisting in always choosing the simplest method at equal performance. The methods section details the characteristics of this model, while results section reports its performance on several visual benchmarks and on the EmotiW audiovisual challenge [START_REF] Goecke | EmotiW 2018: Audio-Video, Student Engagement and Group-Level Affect Prediction[END_REF].

METHODS

This section presents the proposed framework for audiovisual emotion recognition, by discussing first the visual and audio modalities, and then by proposing a method for fusing them. We finally address the issues raised by the (small) size of the dataset.

Visual modality

The visual modality consists in sequences of images extracted from the AFEW videos. They are processed by: i) applying a face detector ii) aligning the landmarks with a landmark detector and an affine transform, resizing aligned faces to 224 × 224. The so-obtained faces are referred as the visual input in the rest of the paper.

Emotion classification with a ResNet-18 network. Our first objective is to learn a still image emotion classifier, based on the ResNet-18 [START_REF] He | Deep residual learning for image recognition[END_REF], which we considered to be well adapted to this task. However, the small amount of training clips (773) makes it hard to learn from scratch. Consequently, we first trained it on the larger AffectNet dataset [START_REF] Mollahosseini | Affectnet: A database for facial expression, valence, and arousal computing in the wild[END_REF], containing in the order of 300,000 usable faces, annotated with both emotion labels (8 labels, not the same as AFEW) and arousal valence values. Multi-task learning allows us to use these two types of annotations, by replacing the last dense layer of the ResNet-18 network by two dense layers: one arousal-valence linear regressor and one emotion classifier (softmax layer). Their two losses are optimized during training, leading to a more general 512-sized hidden representation. We also use standard regularization methods such as data augmentation (e.g. jittering, rotation) to be robust to small alignment errors, cutout [START_REF] Devries | Improved regularization of convolutional neural networks with cutout[END_REF] and dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF]. Applying directly this model to AFEW face images gives not better than chance results, due to the big differences in the annotations of the 2 datasets. To deal with this issue, we use two other emotion datasets: i) Real-World Affective Faces (RAF) [START_REF] Li | Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild[END_REF], containing 12,271 training images annotated with emotion labels, with a large number of annotators for each sample and hence a high confidence in the annotations. ii) SFEW [START_REF] Dhall | Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark[END_REF], containing fewer than 1,000 training images, but extracted from AFEW's frames and annotated with the same labels. The fine-tuning is done on these two datasets, in two times. In a first time, only the parameters of the last layer (regressor/classfier) are optimized. On a second time, all the parameters of the network are fine-tuned, but with a lower learning rate. The so-obtained classifier can be applied to the AFEW faces, giving, for each frame, the arousal-valence prediction and class label scores. We also output, for further use, the weights of the hidden layer (512 weights), used as face features.

From still image to video classification. Once processed by the above explained model, and assuming L denotes the number of frames of a video, we obtain a set of L 512-dimensional face descriptors with their associated scores (one score for each one of the seven categories) as well as arousal/valence predictions.

One very simple way to aggregate the temporal information, and produce classification scores at the level of the video, is to average the per-frame scores. As shown in the experiment sections, it already gives very good results (see Section 3.1). However, we propose to explore better ways in the following.

We select n faces (n = 16 in the rest of the paper) from the L original ones, combining down-sampling and max-pooling, as shown in Figure 2. We first divide the sequence into n chunks of equal length, and choose in each chunk the face with the highest score (across frames and categories), represented by 512d feature vectors. For the rest, the per-frame scores are not used anymore.

At the end of this stage, we have a nx512 tensor representing the faces of the video sequence. The final video clip classification is obtained by temporal pooling of the n face features. We consider in the experiment two alternatives: the first one consists in a simple average of the face features, the second one consists in a weighted average of these features. In both cases, once the features are averaged, it gives a 512-d vector on which a linear classifier (softmax with cross entropy) is applied. The n weights of the weighted average, one per selected frame, are regressed with a linear regressor (with sigmoid activation) applied to the arousal-valence representation of the frames. The regressor is trained jointly with the whole network. The third fusion method consists in training a LSTM network with 128 hidden units on the 512-d features of the n selected frames. The output of the LSTM is the scores of the 7 classes.

Please note that for the sake of simplicity and because we observed no improvement, we do not compute temporal features (such as C3D features), contrarily to several recent approaches [START_REF] Fan | Video-based emotion recognition using cnn-rnn and c3d hybrid networks[END_REF][START_REF] Vielzeuf | Temporal multimodal fusion for video emotion classification in the wild[END_REF].

Audio modality

Regarding the audio modality, we experimented two alternatives.

The first one consists in extracting 1582-d features designed for emotion recognition, using the OpenSmile toolbox [START_REF] Eyben | Opensmile: the munich versatile and fast open-source audio feature extractor[END_REF] trained on the IEMOCAP dataset [START_REF] Busso | IEMOCAP: Interactive emotional dyadic motion capture database[END_REF]. On top of these features, we apply a random forest classifier [START_REF] Breiman | Random forests[END_REF].

The second one consists in extracting the same OpenSmile features and train a fully connected classifier with 64 hidden units, batch-normalization, dropout, and reLu activation. The so-obtained model is then fine-tuned on the AFEW train set.

Fusion of the modalities

Image and audio give each one a set of scores (with one score per class). As shown by previous works [START_REF] Vielzeuf | Temporal multimodal fusion for video emotion classification in the wild[END_REF], sophisticated fusion methods often performs worse than simpler ones on AFEW, probably because of their large number of parameters and the small size of the dataset. We therefore experimented two simple approaches for computing the final scores: (a) the mean of the modality scores; (b) the weighted mean of the score, learned on the validation set, similar to the last year's winning approaches [START_REF] Fan | Video-based emotion recognition using cnn-rnn and c3d hybrid networks[END_REF][START_REF] Hu | Learning supervised scoring ensemble for emotion recognition in the wild[END_REF].

Ensemble learning

Ensemble learning is often used to boost the results, as observed in last competition works [START_REF] Antipov | Apparent age estimation from face images combining general and children-specialized deep learning models[END_REF][START_REF] Kim | Hierarchical committee of deep convolutional neural networks for robust facial expression recognition[END_REF][START_REF] Miech | Learnable pooling with Context Gating for video classification[END_REF][START_REF] Yu | Image based static facial expression recognition with multiple deep network learning[END_REF][START_REF] Zeng | Crafting gbd-net for object detection[END_REF]. We implement an ensemble of our temporal model, by learning several times the same model with different initializations, and averaging their predictions.

Dealing with the small size of AFEW

The proposed method has several hyper parameters or architectures details (size of layers, etc.) that have to be set. The most common way to set these parameters is to choose those giving the best performance on the validation set. However, due to the small size of the training and validation sets of AFEW, this is rather unreliable. We experimented with 3 alternatives : (i) training several times the model with different initialization and computing the mean performance and the standard deviation (std); (ii) merging the training and validation sets and applying cross-validation; (iii) using estimated per-class accuracy and weight classes according to the test set distribution. The third alternative, as previously done by [START_REF] Knyazev | Leveraging large face recognition data for emotion classification[END_REF][START_REF] Vielzeuf | Temporal multimodal fusion for video emotion classification in the wild[END_REF], uses the distributions given in Table 1. The accuracy on the weighted validation set is then computed as follows:

a pond = 7 i=1 a i n i
653 , where a i the estimated accuracy of the i t h class on the validation set, n i the number of elements of this class in the test set and 653 the number of samples of the test set. taken individually, and, combined in a second time for the audiovisual challenge. It is worth noting that our model uses a relatively small number of parameters and can work in a real-time setting (180M FLOPs). Our performance is measured by training several models (in the order of 50 models) with different initialization and measuring the mean accuracy and standard deviation (std).

RESULTS

This

Emotion Recognition in Images

We first compare our ResNet-18 model pre-trained on AffectNet, without temporal aggregation of the features, to several state-ofthe-art methods, on emotion classification in images. As shown in Table 2, despite its small number of parameters, our model gives very good results, outperforming its competitors on the AFEW validation set. In this case, temporal fusion is done by averaging per frame predictions. Note that we measure a std around 0.5%.

Evaluation of the Temporal Pooling

We provide here the evaluation of the 3 different temporal pooling methods we proposed in Section 2. Performance is measured as the accuracy on the validation set of AFEW (mean and std). We also indicate the weighted accuracy (see details in Section 2), which can be seen as a more accurate estimation of the actual performance on the test set. The standard deviation is, on average, around 0.6%.

Our main observations are: i) temporal features aggregation is useful, ii) average and weighted average have approximately the same performance and are better than our LSTM model. iii) combining weighted and non-weighted average pooling seems to help a little on the weighted validation set. iv) our best performance outperforms any results published yet on this dataset. However, with a std of 0.6%, we must be cautious about the conclusions.

We also note that the std on the weighted validation set is a bit lower (resp. 0.5% and 0.4% for av. pool. and weighted av. pooling). It can be explained by higher variations of performance inside the "difficult" classes (disgust, surprise), which are rare in the test set.

Model

Accuracy Weig. Acc.

Visual

Evaluation of the Audio Modality

As for the visual modality, we evaluate the 3 proposed methods. The std is of 1.5% for the Multi Layer Perceptron trained from scratch and of 0.8% for the Multi Layer Perceptron pre-trained on IEMOCAP. Even if the Random Forest yields good results, it overfits the training set with the accuracy of almost 100% with hence a high risk of bad generalizations. The pre-trained MLP achieves the best and most stable results. The audio modality is weaker than the visual one and the AFEW annotation of a video seems to rely more on the visual modality and on the context. Nevertheless, audio can bring a +3% accuracy gain when combined with visual modality, as reported by Fan et al. [START_REF] Fan | Video-based emotion recognition using cnn-rnn and c3d hybrid networks[END_REF], and is not an option for this challenge.

Audiovisual Challenge

This section describes our 7 submissions to the 2018 edition of the EmotioW challenge. For each submission, we explain the audio and the visual modalities in Table 4, as well as the performance.

First submission: most simple model with average pooling of visual features and MLP for audio. Video and audio scores are weighted resp. with 0.65 and 0.35 (weights learned on the validation set). Second submission: audio/video weights set to 0.5/0.5 and combination of RF and MLP. Third submission (our best one): ensemble of 6 models (see Table 4 for details). Fourth submission: same as the third submission, but replacing the Random Forest by a second MLP. Fifth and sixth submissions: larger visual ensembles. Seventh submission: same as third one using Train+Val for training (surprisingly 0.1% lower).

These results confirmed our three intuitions. First, performance on validation and test sets are very different, making it difficult to choose a model from the validation set accuracy. Second, adding the validation set to the train set for final training makes the performance worse, which is counter-intuitive. Last but not least, drastically limiting the number of trainable parameters on such a small dataset is one of the key ingredients to better generalizations.

These experiments also highlights the importance of having a weighting scheme to improve the selection of the model on the validation set, knowing that validation and test sets have different distributions. More generally, we also observed large standard deviations in our cross-validation experiments, explaining why it's difficult to compare different methods on the validation set.

On overall, the proposed light model is real time and achieved the accuracy of 60.64%, allowing it to ranked 4th at the 2018 edition of the EmotiW Challenge.

We do believe that the performance on this challenge starts saturating, which can be explained by the small size of the dataset and the subjective (and therefore noisy) nature of the annotations. We indeed noted that human performance reported on the validation set of AFEW by [START_REF] Vielzeuf | Temporal multimodal fusion for video emotion classification in the wild[END_REF] is comparable to the performance reached by our model. There is consequently a risk that improving the performance on this dataset will consist in exploiting its biases rather than actually learning a better representation of emotions.

CONCLUSIONS

This paper proposes a new audiovisual model for emotion classification in videos. This model is carefully designed following the "Occam's razor" principle, which can be summed up by "always choose the simplest approach". For both modalities we limited the number of trainable parameters to their minimum. Transfer learning is also used to include reliable a priori knowledge and solve the high-dimensional versus lack of data paradigm, especially for the visual modality. A basic but well-performing temporal pooling is also proposed, including a frame selection mechanism. Finally, a simple fusion method average of the score limits again the number of parameters of the model.

Figure 1 :

 1 Figure 1: Overview of our framework. Only the green part (on the right side) is trained on AFEW.

Figure 2 :

 2 Figure 2: n faces are selected from a sequence of L frames. s is the score of a given frame.

Table 1 :

 1 section experimentally validates the proposed model, by first presenting the results obtained by our audio and visual models An. Di. Fe. Ha. Sa. Ne. Su. All AFEW dataset: number of video sequences per class.

	Train 133 74 81 150 117 144 74 773
	Val	64	40 46 63	61	63	46 383
	Test	99	40 70 144 80	191 29 653
	Model		RAF SFEW AFEW FLOP Param
	CNN Ensemble [33] _	55.96	_		>2000 >500
	HoloNet [19]	_	_	46.5		75	_
	Cov. Pooling [2]	85.4 58.14 46.71	1600	7.5
	Transfer VGG [31]	_	45.2	41.4		1550	138
	Our (image)		80	55.8	49.4	180	2

Table 2 :

 2 Accuracy of different models for facial expression classification. Weights and FLOPs are in millions. Transfer VGG is a VGG-face model fine-tuned on FER-2013[START_REF] Goodfellow | Challenges in representation learning: A report on three machine learning contests[END_REF].

Table 3 :

 3 Accuracy for visual and audio modalities and for state-of-the-art visual models (italic). Indication '(x)' means: ensemble of x models with different initialization .

		no feat. aggregation	49.4	55.6
		av. pool. (1)	49.7	60.5
		av. pool. (4)	50.4	61.2
		av. pool. (50)	52.2	61.7
		weig. av. pool (1)	50.2	61.1
		weig. av. pool (4)	50.3	61.5
		av. pool (2) + weig. av. pool (2)	50.1	62.0
		LSTM 128 hidden units	49.5	58.2
		VGG-LSTM [31]	48.6	_
		FR-Net-B [22]	53.5	_
	Audio	MLP MLP (pre-trained) Random Forest	33.5 35.0 38.8	42.1 45.2 44.3

Table 4 :

 4 Our 7 seven submissions to the 2018 Emotion in the Wild challenge. Indication '(x)' means: ensemble of x models trained with different initialization. See text for details.

	# Visual	Audio	Weigh. Val Test
	1 av. pool. (1)	MLP (1)	62.1	57.2
	2 av. pool. (1)	RF (1) +MLP (1)	62.4	58.6
	3	av. pool. (2) + weig. av. pool.(2)	RF (1) +MLP (1)	62.7	60.6
	4 weig. av. pool. (2)	MLP (2)	63.5	59.4
	5	av. pool. (2) + weig. av. pool.(2)	MLP (2)	63.0	60.4
	6 av. pool. (50)	MLP (2)	63.6	59.4
	7 av. pool. (4)	MLP (1)	72.4	60.5


Corinne

Landure enseigne au département de Techniques de Commercialisation à l'IUT d'Épinal, composante de l'Université de Lorraine. Docteure en linguistique anglaise, spécialité didactique des langues, ses recherches portent sur l'exploitation directe de corpus générique et spécifique par des apprenants Lansad. corinne.landure@univ-lorraine.fr en Language corpora have been shown to be of substantial help in improving learner writing, both for advanced students majoring in the language as well as for students with lower levels of proficiency needing language for specific purposes. This paper outlines two very different ways an online corpus has been successfully used as a reference resource by our learners. The first is in a distance learning context: the teacher highlights issues in the student's academic writing and suggests corpus uses rather than simply giving the correction, the objective being to raise awareness of and sensitivity to the language problems identified. The second derives from a classroom situation where the learner himself chooses the items to check in a professional simulation writing task; the teacher guides the process, again with the intention of fostering the learner's autonomy.

Introduction

Corpora can be used to study language in all its forms and uses. In language teaching and learning, one of its most common functions has been to inform dictionaries, grammar books, usage manuals, textbooks, syllabuses, tests, and other resources. Most of these keep corpora in the hands of expert corpus linguists, but there is little reason why end-users cannot also make use of such tools and techniques. Such approaches go by different names, including corpus consultation and data-driven learning. While in reality both are used to refer to the same thing, there is potentially an enormous difference between consultation and learning; while one might assume that the act of looking up a word in a dictionary (consultation) might lead to long-term retention, there is no guarantee that this will be the case, even if it is the objective. Both the present authors have used corpora extensively for both purposes (e.g. Boulton 2016; Boulton & Tyne 2014; Landure et Boulton 2010), and propose some short examples here of some of the activities genuinely used with their students. The present focus is primarily on using corpora as a reference resource, in line with some of Tim Johns' original uses, often referred to as 'kibbitzing', where the process was written up and made available to outside learners. A sample corpus kibitzers are still available at http://lexically.net/TimJohns. In our set of two pedagogical notes, the focus is on French learners of English using similar techniques accessing the same corpus, but the first with a master's student in a distance English degree, the second with a lower-level IUT student in class.

The corpus used is the Corpus of Contemporary American English (COCA), a collection of 450 million words available on line (http://corpus.byu.edu/coca).1 While personal use requires free registration, any search conducted generates a distinct URL which can be passed on to others so that they can see the query performed and the results. The same website has a number of other corpora for different varieties of English, as well as corpora in Spanish and Portuguese. Corpora can be found for many other languages on the web, as well as for specific varieties of languages (e.g. law, medicine, etc.). What follows is merely an indication of the types of information that can easily be retrieved from a corpus and usefully passed on to students.

Case study 1 A Master's student writing a short research paper in English on 'erotic fiction' produced the following text as part of a draft abstract, submitted to the teacher for comment. The text is relatively accurate in most respects, with the exception of *storie. One can only assume that the student wasn't using a spellchecker, or hadn't switched it to English, otherwise it would be underlined in red on her screen. This is a simple example of the potential of basic technology to help students in their L2 writing -but is in fact underexploited.

The rest of this short text presents few obvious orthographic, lexical or grammatical errors, but will immediately appear to be odd to most expert speakers of English. The teacher's job here is not an easy one: s/he first has to identify a problem, then form an idea of what the learner wanted to say, and imagine how this could be better formulated. Then there is the question of what type of feedback to give, a subject which has generated enormous quantities of primary research as well as a number of meta-analyses (cf. Plonsky & Brown 2015). The main finding common to most is that feedback does help, but should not be too explicit. Systematically providing the (presumed) correct answers tends to lead to minimal cognitive processing and is quickly forgotten. Intriguingly, whether the teacher provides detailed indications (such as coding Prep for preposition, Tr for transitivity, etc.) or simply underlines the error does not seem to have much impact on the learners' ability to correct their errors. There is a lesson here that we should not over-correct in terms of quality, but the same applies to quantity: it seems to be more effective to focus on a small number of errors that are considered important so that the learner doesn't drown in too many errors of limited use for their present purposes. This is likely to be even more beneficial where the errors are related so that the learner has multiple opportunities to focus from different angles.

A final issue is the aim of the feedback itself. In the case of a CV, for example, the sole aim might be to help the learner improve her text as product. From a linguistic perspective, the emphasis might be on helping her to learn some new language for future use. In addition to these, we might even (or perhaps especially) want to help her improve the processes involved so that she can think about her future writing and consult resources successfully as the need arises.

In several places in this text, the teacher might not be able to find a dictionary or grammar reference that is explicit; such resources are wonderful generic tools, simplifying sometimes extremely complex points, but the goal being to deal with common issues they cannot, by definition, attempt to cater for every specific case. Where there is no obvious answer, the teacher might only be able to say 'because', a message that is difficult to transmit to the learner. This is where corpora can be helpful -not so much with errors as linguistic impossibilities, but with uses that are not 'usual'.

What follows is not a generalisable "fiche pédagogique" in the usual sense that it is ready for use with other students, but a genuine example of the type of feedback that can be useful for L2 learners. In the present case, it was fairly quick to produce, and made available electronically to the student concerned. An additional benefit, one might argue, is that it makes correction more interesting for the corrector. This is the original text of the Master' student mentioned above (reproduced with permission):

The following feedback was provided on the following elements only using MS Word comments. Other aspects may be debatable. "High criticism"

The student clicks on the URL to see immediately what the query was (on the left in Figure 1), i.e. what adjectives occur frequently before criticism.

Figure 1. Query for adj+criticism

There is no need to learn the codes for different parts of speech; these can be selected from the drop-down POS menu. While various functions are available, this is a simple query using entirely default settings for presentation (list) from the entire corpus (as opposed to defined sections such as academic). In case of doubt, the user can click on the question mark on the right for additional help. Figure 1 gives the first 20 results in rank order with the number of occurrences in the corpus on the right. Most of these have meanings and uses which are not relevant to this particular case, but the student should notice a number of possible options for her purposes: harsh criticism, sharp criticism, intense criticism, widespread criticism, heavy criticism, strong criticism, all of which occur over 50 times in this corpus. If hesitating between them, clicking on the term to see examples in context, along with information about the source text. The student could also formulate her own query for high criticism and receive the message: "Sorry, there are no matching records". In other words, for these two words which individually occur 209,144 and 16,848 times respectively, there is not one single attested use of them in this combination in a corpus equivalent to 7,000 books or 60,000 research articles. Such features highlight the importance of collocation (i.e. co-occurrence) in the 'naturalness' of language use.

"Put a finger on"

The URL provided this time shows the distribution of the phrase in the different sub-sections of COCA using the chart function at the top (Figure 2), a function which can be extremely useful when dealing with languages for specific purposes. Clearly there is nothing wrong with this phrase in itself, but it is very unlikely to occur in academic texts such as the student is attempting to produce here -on average only 0.03 occurrences per million words. The student can click on the column headings (e.g. fiction) to see distribution in sub-sections (general books, sci-fi/fantasy, etc.), the bars to see the occurrences in context. Awareness of genre and text type are crucial.

"Reason of"

Prepositions are a common source of difficulty, and here is perhaps the closest we get to a language 'error' as opposed to a usage issue. As figure 3 illustrates, the query uses an asterisk to stand for any word: the reason * "Gives the impression to be"

The URL shows there are 48 occurrences of gives the impression; clicking on this provides short contexts, the first 10 of which are given in Figure 4 below. Of these, 14 are followed by a that-clause; the remaining 6 are followed by of plus a noun (lines 9, 12, 13, 17) or, in two cases, Ving (lines 7, 19). This presentation encourages the student to look at usage in context and detect recurring patterns -an extremely useful skill if it can be drawn upon in other situations. The particular advantage here is the presence of multiple examples rather than occasional serendipitous encounters with isolated occurrences which are likely to go unnoticed. Possible supplementary searches show that there are only 12 occurrences of [give] the impression to (where the square brackets indicate any form of the word -give, gives, gave, given), and in 11 of those cases to is a preposition (e.g. I didn't want to give the impression to anybody that I was guilty of anything.).

"Heavy success"

Having seen various queries and the results they generate, the student might be in a position to use the corpus on her own. Heavy success gives no hits at all; using an almost identical query to #1, it should be easy enough to select adjective from the drop-down POS menu or input [j*] followed by success, as in Figure 5. As before, some of the results are clearly irrelevant, but there are various frequent possibilities to choose from, and which can be seen in context in case of doubt. Having the student look at or perform several queries of the same type is more likely to lead to uptake and continued, autonomous use in the future.

Case study 2

As in the previous case study, we look here at an actual case corpus consultation that was seen to be useful for an EFL learner, especially with regard to usual, normal, natural use of language in context. In this case, however, we are dealing with a student in the second year of a DUT Techniques de Commercialisation (a two-year post A-level degree in Business and Marketing), i.e. not majoring in English and with relatively low level of proficiency in the language -approximately A2. Of course, this experiment can apply to all fields of studies. The objective here was to use the same online corpus (COCA) to help improve the quality of a short professional text. While the query types are largely similar, the problem areas were not indicated by the teacher, and it was up to the students to choose the points to look up in the corpus. The teacher's role was largely reactive with respect to individual queries. This type of feedback was designed to help students improve their corpus consultation techniques, suggesting alternative search functions or ways of formulating their questions or more efficient queries including semantic searches, genresensitive queries, diachronic variations, etc.

The corpus was initially presented to the whole class, but students then used the corpus individually for their own questions. Such individualisation represents a substantial benefit of corpus consultation as each member of the class can work at their own pace on points that are relevant to them, a potential source of motivation. Students can also work in pairs or small groups, with the stronger and weaker students each benefiting from the others' input not just on language, but on the interface and the techniques involved, analysing and sorting the results, etc.

Where students are allowed the considerable freedom they are here, there may be initial questions focus on the content and the objective -what are we doing and why are we doing it like this? Such questions are an important step on the road to more autonomous language learning. In practice, the first step is to choose a word or phrase to check, the actual question being secondary; in other words, the nature of the problem and types of possible improvements emerge from the search itself. This often means a cycle of several searches, some of which may additionally involve dictionary use or other resources, until such time as the student is satisfied they have managed to find relevant data and interpret it appropriately for their specific contextual use. This case study examines the procedures used for the sentence below, produced by a student from this class, following the introduction to corpus use and a few tentative searches in previous lessons:

The student had a doubt about the use of information in this context. He has been warned about it before and knows it is 'uncountable', but such abstract knowledge is difficult to fully grasp and not always easy to translate into appropriate usage when the time comes. His first search is simply for information, but using the KWIC (keyword in context) presentation in COCA where the target item is centred and surrounding items colour-coded for part of speech (students should be aware though that the POS-tagging is automated and may contain some errors). Here is a sample: In other words, absence or rarity of a particular form in a large, wellconstructed corpus may be taken to imply that it is at best relatively unusual, where the learner is typically interested in normal usage.

The student followed up this search by clicking the drop-down POS List menu and selecting det.ALL for any determiner, followed by a space and then information -the first 12 are given in Figure 7. This lists the determiners occurring immediately to the left of information in descending order of frequency; since this information is in second place with 4339 occurrences while these information is not to be found anywhere near the top of the list, there is a clear reason to prefer the former, all things being equal.

A similar procedure allows a quick search for verbs following information (Figure 8). Here the student noted information are in 9 th position with 304 hits (i.e. a frequency of 0.68 occurrences per million words), and decided to go no further. Fortunately, the teacher was on hand and managed to encourage the student to pursue the query. Admittedly, the results suggest that information are is a possible string, but context is needed to see whether it a coherent syntagmatic unit (which it is not). Clicking on information are brings up concordance lines which clearly show that information on its own is not the subject of the verb (a random sample is shown in Figure 9).

Conclusion

The last example highlights that interpreting the results can be the trickiest phase in corpus consultation, requiring an ability to take a step back and think quite hard on occasion. On the one hand, one might think that this adds to the cognitive load; but on the other, it is crucial that it is relevant cognitive activity, and precisely the type of mental effort which is required for successful language learning. Similarly, such uses of corpus can be quite time-consuming -especially at first, though they speed up rapidly with regular use. But it is the process itself that is important, not the answer to a particular question. A learner who is capable of noticing reasoning language in this way, noticing usage in context and making sense of authentic data, is a more autonomous and sophisticated learner better equipped to deal with genuine future needs.

Figure 2 .

 2 Figure 2. Distribution of put a finger on

Figure 3 .

 3 Figure 3. Results for the reason * This is quicker than choosing the POS tag for adjective, although there is correspondingly more 'noise'. From the first 10 results above, the most frequent preposition by far is for with 2915 occurrences; the user can scroll down to find the reason of with only 68 occurrences (i.e. over 40 times less frequent), and click on it to see it in context -most involving different uses (e.g. the reason of course) or meanings (e.g. the reason of the law).

Figure 4 .

 4 Figure 4. Contexts for gives the impression

Figure 5 .

 5 Figure 5. Results for adj+criticism

Figure 6 .

 6 Figure 6. KWIC search for information Even in the random selection of 100 lines, there are no examples of these immediately to the left of information, nor are to the right. While corpus linguists are fond of pointing out that the absence of evidence is not evidence of absence, they are equally fond of stating that they are not interested in what's theoretically possible but in what is actually probable.In other words, absence or rarity of a particular form in a large, wellconstructed corpus may be taken to imply that it is at best relatively unusual, where the learner is typically interested in normal usage.The student followed up this search by clicking the drop-down POS List menu and selecting det.ALL for any determiner, followed by a space and then information -the first 12 are given in Figure7.

Figure 7 .

 7 Figure 7. POS List search for det+information

Figure 8 .

 8 Figure 8. POS List search for information+vb

Figure 9 .

 9 Figure 9. Concordance for information are

  Boulton, A. 2016. "Integrating corpus tools & techniques in ESP courses". ASp, 69 : 111-135. Boulton, A. & H. Tyne. 2014. "Des Documents Authentiques aux Corpus: Démarches pour l'Apprentissage des Langues". Paris : Didier. Landure, C. & A. Boulton. 2010. "Corpus et autocorrection pour l'apprentissage des langues". ASp, 57: 11-30. Plonsky, L., & Brown, D. 2015. "Domain definition and search techniques in meta-analyses of L2 research (Or why 18 meta-analyses of feedback have different results)". Second Language Research, 31 : 267-278.

  High criticism. This is where a corpus can be useful for collocation, as we don't often say 'high criticism'; try this and

	see	if	there's	anything	that	satisfies	you:

http://corpus.byu.edu/coca/?c=coca&q=43356224 Put a finger on. Similarly, look at the distribution of this expression: http://corpus.byu.edu/coca/?c=coca&q=43356250 Reason of. Check preposition: http://corpus.byu.edu/coca/?c=coca&q=43356267 Gives the impression to be. For how to use this, look at what comes after this phrase here: http://corpus.byu.edu/coca/?c=coca&q=43356308 Reason of. See above. Heavy success. Another collocation issue; can you formulate a query similar to that for 'high criticism" above?

corpus, ressource de référence, guide, éveil linguistique, accompagnateur, autonomie

COCA underwent a facelift in May

after the activities were conducted and the paper written up. The functions are the same and we have left the original screenshots for authenticity, but the new interface is cleaner, more intuitive, and works better on mobile devices.