Sophie Berjot
email: sophie.berjot@univ-reims.fr

Camille Amoura

Leïla Bensalah

Amandine Herbay

Stereotype threat among children attending adapted courses (7-10 years old): A study in a quasi-ordinary classroom

Keywords: Stereotype threat, bad reputation, performance, 7-10 years old students

The aim of the present studies was to explore if students attending AC (adapted classesremedial training, 3 times a week; 7-10 years) have a bad reputation (are considered as less able than RC students -Regular Classes) and if this bad reputation can make them vulnerable to stereotype threat. Study 1 assessed self-perceptions of students from adapted (AC) and regular classes (RC) as well as other-perceptions of RC students and meta-perceptions of AC students. Results showed that RC students perceived themselves more positively than they perceived AC students and that self-perceptions of AC students were less positive than those of RC students. They also showed that the meta-perceptions of AC students were similar to the other-perceptions of RC students. Study 2, ran 15 days after, showed that the performance of AC students was impaired when under stereotype threat. Study 3 replicated Study 2 and showed that the effect of stereotype threat on performance was still there even after controlling for initial self-perceptions. It also showed that self-perceptions of both groups lowered after the threat. Key words:

que la méta-perception des élèves en CA était similaire à l'hétéro-perception des élèves en CN. L'étude 2, réalisée 15 jours après, a montré que les performances des élèves en CA a été affectée par la menace de stéréotype. L'étude 3 a répliqué l'étude 2 et a montré que les effets de la menace de stéréotype sur la performance étaient toujours visibles, même après avoir contrôlé l'effet des auto-perceptions initiales. Ils ont également montré que les autoperceptions des deux groupes d'élèves diminuent après la menace.

Mots clefs: Menace de stéréotype, mauvaise réputation, performance, élèves de 7 à 10 ans.

Public schools offer in many countries basic and solid education that facilitates access to universities and higher education institutes. This system is generally quite efficient in France principally because it is based on shared and tested methodologies. But what works for the majority may not work for all, and this system turns out to be less efficient for some students because they learn faster and get bored during school, or because they learn slowly.

To compensate for this, schools set specific plans to allow slower learners to catch up. Those plans often consist of opening specific classes in which special teachers help these students to catch up with the program. The aim is to reintegrate children into the regular system. Many forms of such system exist in many occidental countries, which differ in the amount of specialized/specific courses children have to take. This can consist of a few hours' courses during the week after school time or of being inserted in a full-time specialized class. The present studies were run with remedial classes called CLAD (for 'CLasse d'ADaptation' or 'ADadped CLass'). This system aims at helping students from elementary schools (from 2 nd to 4 th grade) acquiring specific academic knowledge they did not learn during the year by giving them two or three times a week an hour of class with a specialized teacher. Students are still part of their normal class during regular school time but stay longer some days during the week to take this remedial training. While this system may be quite helpful for students, its positive impact on learning and performance may, we think, have side-effects that are not always foreseen. Indeed, if that system is implemented to help students, it is also intended to students who learn slowly. Given that in our occidental societies behaviors are often attributed to personal instead of situational factors [START_REF] Ross | The person and the situation: Perspectives of social psychology[END_REF], even by children [START_REF] Stipek | Children's judgments of their own and their peers' academic competence[END_REF], it is likely that those students are perceived as being slower and less smart, and as such are prone to be, even if temporally, stigmatized and victim of what is called stereotype threat. So the aims of those studies are to 1) show that students attending adapted classes (AC) have a bad reputation regarding their academic abilities and are conscious of this bad reputation, and 2) to explore if this bad reputation impacts their self-perception (Study 1).

It is also aimed at examining whether this bad reputation is powerful enough to impact their performance when a task is presented as a test. Study 2 and 3 explore if students attending AC are vulnerable to stereotype threat because of their (temporally) belonging to this specific adapted system (Study 2) and/or because of their initial low self-perception (Study 3).

Stereotype threat

Stereotype threat is "being at risk of confirming, as self-characteristic, a negative stereotype about one's group" (Steele & Aronson, 1995, p. 797). Far from being a kind of chronic vulnerability because of one's group membership, stereotype threat arises in specific situations where an individual is at risk of fulfilling the stereotype associated with his or her group membership. Many studies have shown that Black American students perform less well when a task is presented as diagnostic of their intellectual abilities [START_REF] Steele | Stereotype threat and the intellectual test performance of African Americans[END_REF] or when the lack of abilities of members of their group is made explicit [START_REF] Cadinu | Stereotype threat: the effect of expectancy on performance[END_REF]. The same was generally observed with other ethnic minorities such as Latinos in the States (for a review, see [START_REF] Guyll | The potential roles of self-fulfilling prophecies, stigma consciousness, and stereotype threat in linking Latino a ethnicity and educational outcomes[END_REF] or French-Arab students [START_REF] Chateignier | French-Arab students and verbal intellectual performance: Do they really suffer from a negative intellectual stereotype[END_REF]. Also, women, who are allegedly less able in Math, often perform less well on a Math test when it is presented as a test of quantitative capacities [START_REF] Schmader | Converging Evidence That Stereotype Threat Reduces Working Memory Capacity[END_REF] than when it is described as a test/task not relevant to Math. The same was also typically observed with many other social groups for which it exists a negative stereotype such as low Socio Economic Status students [START_REF] Croizet | Extending the concept of stereotype and threat to social class: The intellectual underperformance of students from low socioeconimic backgrounds[END_REF] or even students from psychology majors tested on their learning skills while explicitly compared with students from medical school majors [START_REF] Dutrevis | Reputation of Intellectual Infereiority Undermines Memory Efficiency Among College Students[END_REF].

Stereotype threat is foremost a situational threat [START_REF] Steele | Stereotype threat and the intellectual test performance of African Americans[END_REF]. This is the situation and its meaning for the individuals that make them particularly vulnerable. For example, [START_REF] Shih | Stereotype susceptibility: Identity salience and shifts in quantitative performance[END_REF] asked Asian American women to take a Math test under different implicit activations, either of their women identity or of their Asian identity. While women performed better on the Math test when their Asian identity was activated, they performed worse when their gender identity was activated. This was replicated more recently with a verbal test [START_REF] Shih | Domain-specific Effects of Stereotypes on Performance[END_REF].

However, if most studies were run on individuals with an enduring group membership, other studies have shown that a history of stigmatization was not always necessary for stereotype threat to appear [START_REF] Leyens | Stereotype Threat: Are Lower Status and History of Stigmatization Preconditions of Stereotype Threat?[END_REF]. For example, [START_REF] Koenig | Stereotype Threat in Men on a Test of Social Sensitivity[END_REF] showed that the performance of men on a social sensitivity test was lower than that of women when the test was presented as diagnostic of this ability and when men were told that women generally succeeded better than men. In a more extreme test of this idea, [START_REF] Martiny | When a new group identity does harm on the spot: Stereotype threat in newly created groups[END_REF] showed in a set of 3 studies, that the deleterious effects of stereotype threat can also appear with totally newly created groups.

Using a minimal group paradigm, the authors threatened their participants by telling them that members of their group (individuals who have a convex or a concave information processing style) usually perform less well on tests assessing this ability. Results showed that newly created group members underperformed when informed about the bad reputation of their group (presented as a fact or as a suspicion) compared with non-threatened group members. This was especially the case of highly identified group members. So, even if stereotype threat is a quite powerful phenomena that can impact performance of many members of stereotyped groups (e.g., women, Black and Hispanic Americans, low SES students) on many abilities [START_REF] Beilock | Stereotype threat and sport: Can athletic performance be threatened?[END_REF][START_REF] Chalabaev | Can stereotype threat affect motor performance in the absence of explicit monitoring processes?: Evidence using a strength task[END_REF][START_REF] Chateignier | Effet de la menace du stéréotype et de la colère sur la performances des femmes à une tâche liée à la conduite automobile[END_REF][START_REF] Hively | You throw like a girl:" the effect of stereotype threat on women's athletic performance and gender stereotypes[END_REF], those studies show that neither a history of stigmatization nor a well-grounded stereotype are necessary for stereotype effects to occur.

One of the reasons may be, as stated by value expectancy theory [START_REF] Wigfield | Expectancy-value theory of achievement motivation[END_REF], that the stereotype was partly endorsed and impacted the self-schemata and ability beliefs of the victims (Maass & Cadinu, 2003). If many stigmatized groups don't have a low self-esteem because of their ability to cope effectively with identity threats [START_REF] Crocker | Social stigma and self-esteem: The self-protective properties of stigma[END_REF], some group members can at times internalize the stereotype that is associated with their group or at least believe it is true for them. Several studies have shown that believing or endorsing a stereotype can be a handicap in coping with stereotype threat (Bonnot & Croizet, 2007a, 2007b;[START_REF] Chalabaev | Stereotype endorsement and perceived ability as mediators of the girls' gender orientation-soccer performance relationship[END_REF][START_REF] Schmader | The costs of accepting gender differences: The role of stereotype endorsement in women's experience in the Math domain[END_REF]). However, because no studies to the best of our knowledge directly tested the impact of initial self-perceptions on the performance of people under stereotype threat, an explanation in terms of expectancies effects cannot be totally ruled out.

Stereotype threat and children

Research on stereotype threat and its deleterious effects on children performance is scarcer but nevertheless tend to show that students also can be the victims of stereotype threat from the age of 7 years old [START_REF] Désert | So young and already victims of stereotype threat: Socio-economic status and performance of 6 to 9 years old children on Raven's progressive matrices[END_REF][START_REF] Muzzatti | Gender and mathematics: Attitudes and stereotype threat susceptibility in Italian children[END_REF][START_REF] Neuville | Can salience of gender identity impair math performance among 7-8 year old girls? The moderating role of task difficulty[END_REF]. Only two studies reported stereotype threat effects with children as young as 5 [START_REF] Ambady | Stereotype susceptibility in children: Effects of identity activation on quantitative performance[END_REF][START_REF] Tomasetto | Girls' math performance under stereotype threat: The moderating role of mothers' gender stereotypes[END_REF]. Among all those studies (for a review, see Ganley et al., 2013), most were run on girls and performance in Math [START_REF] Huguet | Stereotype threat among schoolgirls in quasi-ordinary classroom circumstances[END_REF][START_REF] Muzzatti | Gender and mathematics: Attitudes and stereotype threat susceptibility in Italian children[END_REF]. Only two studies concerned other abilities and groups. One concerned intellectual abilities of children from low SES backgrounds compared to high SES backgrounds [START_REF] Désert | So young and already victims of stereotype threat: Socio-economic status and performance of 6 to 9 years old children on Raven's progressive matrices[END_REF] and the other one concerned the lower performance at school of boys compared to girls [START_REF] Hartley | A stereotype threat account of boys' academic underachievement[END_REF]. None were run with another type of stereotyped group, which is our interest here.

What about a less enduring group membership such as belonging to AC? Can a negative stereotype be associated with this specific group, at least in the context of school? Also, can students, newly members of this group, be conscious of such a belief toward them, and does this consciousness impact their self-perception? Finally, can this belief impact the performance of those who hold it?

As already stated by [START_REF] Galdi | The roots of stereotype threat: When automatic associations disrupt girls' math performance[END_REF] as well as other researchers, specific prerequisites are necessary for stereotype threat to impact performance of children.

First of all, children must have developed an awareness of distinct categories, that is, they must be able to distinguish categories in their environment and to identify themselves as members of some of those categories. This is often the case by the age of 3 or 4 [START_REF] Martin | Patterns of gender development[END_REF]. Moreover, they must be conscious that categories are associated with specific attributes and valence. This is the case by the age of 4 for main categories such as gender.

However, refinements in the perception of groups such as the awareness of status differences are better understood by the age of 6 or 7 [START_REF] Martin | Patterns of gender development[END_REF]. Moreover, it is important to distinguish between stereotypic beliefs (what they really think about groups: other groupsother-stereotype, or their own -self-stereotype) and stereotypic consciousness which corresponds to their awareness of the stereotypes held by others (i.e., meta-stereotype if it concerns beliefs toward one's own group). Stereotypic consciousness appears a little bit latter than stereotypes, around 7, as it presupposes that the child is able to infer others' social beliefs [START_REF] Mckown | The development and consequences of stereotype consciousness in middle childhood[END_REF]. This is coherent with what was found in the literature on stereotype threat run with children. Children, as young as 6, know the stereotype that is associated with their gender group or with their socio-economic origin. For example, [START_REF] Désert | So young and already victims of stereotype threat: Socio-economic status and performance of 6 to 9 years old children on Raven's progressive matrices[END_REF] have shown that children as early as 6 years old believed that high SES children do better at school than low SES children. This was especially the case when the students were in an evaluative situation as this belief impacted performance. In a more recent study, [START_REF] Hartley | A stereotype threat account of boys' academic underachievement[END_REF] showed, using pictured scenarios, that children of both gender believed that girls were better at school than boys by the age of 7-8 years old. This belief increased with age. More interestingly, the authors showed that those children also developed by the age of 7 a metastereotype, reporting that adults believed that girls performed better at school than boys.

When told that girls were better on tests, boys underperformed compared to girls. In the nonthreatening condition, boys performed as well as girls. This study is interesting as it shows that by the age of 7, children can form stereotypes as well as meta-stereotypes and be impacted by them. It also showed, as with studies ran with adults, that a history of stigmatization is not necessary for stereotype threat effects to appear. However, as already mentioned, all those studies were run with enduring groups' memberships. Can children, as young as 7 years old, form specific beliefs about a newly formed group and can those beliefs be known and shared by the members of that group? If this is the case, can those beliefs be powerful enough to disrupt performance of children belonging to that group? These were the aims of the present three studies.

More specifically, we tried to assess the self and meta-perceptions of students aged from 7 to 10 years old attending an adaptive course and to assess the self and other-perception of students attending a regular course (Study 1). We also explored whether those students attending adaptive courses were susceptible to stereotype threat (Study 2), over and above lower initial self-perceptions (Study 3). As mentioned, the impact of stereotype threat on performance could be due totally or partially to initial lower self-perceptions of children.

Finally, Study 3 also aimed at testing the impact of stereotype threat and initial selfperceptions on post-task self-perceptions which, to our knowledge, has never been done before.

All studies were run in the children's school, thereby increasing ecological validity [START_REF] Désert | So young and already victims of stereotype threat: Socio-economic status and performance of 6 to 9 years old children on Raven's progressive matrices[END_REF][START_REF] Huguet | Stereotype threat among schoolgirls in quasi-ordinary classroom circumstances[END_REF]. Perceptions were assessed during school time by the teacher about 3 months after the beginning of the school year and the experiments (induction of stereotype threat) were run individually in an adjacent room during school time by a female experimenter.

Study 1

In this Study, we explored the self-perceptions of students from Regular Classes (RC) and from Adapted Classes (AC), as well as the meta-perception of AC children and the otherperception of RC children. If, as suggested, RC children perceive their AC counterparts as being less able, their self-perception should be higher than their perception of AC children (hypothesis 1). Additionally, if AC students are aware of their bad reputation, their metaperception (how they think they are perceived by RC children) should match how RC students perceive them. If not, AC children's meta-perception should be more positive than the perception RC students hold about them (hypothesis 2). Finally, if AC students agree with this bad reputation, their self-perception should be lower than that of RC children (hypothesis 3) and meet their meta-perception.

Method

Participants and Procedure

Participants were 104 elementary school students (53 girls, 51 boys, aged 7-101 ) from a school in a low socio-economic area of the town. Their mean age was 8.36 years (SD=1.07).

Among them, 61 were from a RC (31 girls, 30 boys; M age =8.28, SD=0.92) and 43 from an AC (20 girls, 23 boys, M age =8.49, SD=1.24). Among them, 37 were 2 nd grade (M age =7.41, SD=0.50), 42 from 3 td grade (M age =8.48, SD=0.67) and 25 from 4 th grade (M age =9.40,

SD=0.50).

Between two lessons, the teacher asked all her students to answer 7 questions taken from the Pictorial Scale of Perceived Competence and Social Acceptance for Young Children of [START_REF] Harter | The pictorial scale of Perceived Competence and Social Acceptance for Young Children[END_REF], using its regular assignment ("How do you perceive yourself ?").

RC students were then invited to answer a second time the 7 items but with an otherperception assignment ("How do you think students of adapted classes are?"). AC children were invited to answer these items with a meta-perception assignment ("How do you think students from normal classes perceive you?").

Measures

The Perceived Competence and Social Acceptance for Young Children of [START_REF] Harter | The pictorial scale of Perceived Competence and Social Acceptance for Young Children[END_REF], translated and validated in French by Gobancé, Rosnet and Gillet (in revision, see also [START_REF] Gobancé | Vers un modèle prenant en compte l'estime de soi dans le processus transactionnel de stress lors des premières confrontations à des situations d'évaluation sociale[END_REF] was used to assess perceptions. This scale assesses self-esteem in a way that is adapted for young children (6-8 years old) through the use of pictures depicting children (boys or girls) performing tasks or actions. First, children were asked to choose which of the children from the 2 pictures, both depicting the same action, was most like him or her. The first picture depicted an action that was performed poorly and the other one the same action performed well. Once the child had chosen a picture, she/he was asked to choose between two circles, a small one meaning "I'm quite like him/her" and a big one meaning "I'm really like him/her". Score were assessed using this 4-point rating scale.

Seven items were taken from this scale among the 22 possible items of the French version: 1 item concerned general knowledge, 2 concerned skills in Math, 2 skills in writing and spelling, 1 skills in sports, and 1 concerned social acceptance. The same scale with the same format was used for all children.

An exploratory factorial analysis run on the measure showed two dimensions, one with the social competence item, the other with all the other items. So, item 1 was deleted from subsequent analysis. After this deletion, alpha was equal to .63 for self-perception (all participants), .65 for other-perception (students from normal classes), and .71 for meta-perception (AC students2 ). In addition, and with the agreement of children's parents, the teacher reported the type of class, the age and sex of each student.

Results

To test for our hypothesis and in order to make all the comparisons we needed, we ran a mixed ANOVA with class as a between factor (Regular Class vs Adapted Class) and perceptions as a repeated measure (self-perception vs perceptions of AC3 ). Results showed that this interaction was significant (F[1, 102]=5.70, p<.02, η²=.05). We then ran a series of post-hoc analysis (Tukey HSD) to test for specific hypotheses.

To check whether RC students perceive their counterparts from AC as less able (hypothesis 1), we compared self-perceptions of RC students to the perception they had of AC students. As shown in Table 1, results indicated that the self-perception of RC students was higher (M=3.51, SD=.40) than their perception of AC students (M=2.14, SD=.56, p<.001).

Table 1

Mean and standard deviations for students' self-perceptions and perceptions of AC depending on their type of class (RC vs AC)

Self-perceptions

Perceptions of AC To examine whether AC students were aware of their bad reputation (hypothesis 2), we compared their perception of AC (meta-perception) to RC students' perceptions of AC's students (other-perception). Results showed that the meta-perception of AC students (M=2.16, SD=.70) did not differ from the other-perception of RC students (M=2.13, SD=.56; ns). This suggests that AC students were aware of their bad reputation. To check whether AC students endorsed the negative view of their group membership, we compared their selfperceptions with those of RC students. Results showed that self-perceptions of AC students were more negative (M=3.17, SD=.66) than those of RC students (M=3.51, SD=.40, p<.02).

Finally, we also compared the self-perception and meta-perception of AC children.

Results showed that they perceived themselves more positively (M=3.17, SD=0.66) than they thought they were perceived by RC's students (M=2.16, SD=0.70; p<.001). So, if AC students have a lower self-perception of themselves than have RC students, this self-perception does not meet their meta-perception.

Discussion

The aim of this study was to see if a bad reputation was associated with children attending AC. More specifically, a first hypothesis stated that the way RC children perceive themselves would be much more positive than the way they perceive AC students. This was what we found. However, this can be partly due to in-group favoritism, which has been found among children as soon as the age of 5 [START_REF] Dunham | Consequences of 'minimal' group affiliations in children[END_REF], or to selfenhancement motives that often drive self-perceptions of adults [START_REF] Dauenheimer | Selfenhancement, self-verification, or self-assessment? The intricate role of trait modifiability in the self-evaluation process[END_REF][START_REF] Sedikides | Self-enhancement and self-protection motives[END_REF][START_REF] Sedikides | Self-enhancement: Food for thought[END_REF] as well as children [START_REF] Trzesniewski | Self-enhancement and selfprotection in a developmental context[END_REF]. Indeed, children as young as 4 often over predict their performance, past and future, even in the face of failure [START_REF] Stipek | Developmental change in children's assessment of intellectual competence[END_REF]. This effect decreases with age (by 8-9 years old). They also overrate their own abilities compared to that of their peers, again, the difference between self and peer's rating decreasing with age (Deborah [START_REF] Stipek | Children's judgments of their own and their peers' academic competence[END_REF]. Note that our results showed that the difference of perception was quite large. Moreover, our results also showed that the way RC students perceived AC students (other-perception) was equivalent to the way AC students thought they were perceived (meta-perception of AC students). So, even if self-enhancement motives might explain this difference between the self-and other-perceptions of RC students, AC students were conscious of being perceived negatively and this perception seemed quite correct in regard to what RC students thought of them.

Finally, our goal was to see if this bad reputation impacted the self-perception of AC students. Our results showed that they did perceived themselves more negatively than did RC students. However, this self-perception did not meet their meta-perception. This means that they did not seem to have totally internalized their bad reputation yet, even if their initial selfperception was lower. However, this is difficult to tell if AC students' lower self-perceptions were due to their membership to this lower status group or if they were lower to begin with.

Research on stigmatization showed that not all lower status groups develop a lower selfesteem and many personal and situational factors can impact the self-esteem of children [START_REF] Crocker | Social stigma and self-esteem: The self-protective properties of stigma[END_REF][START_REF] Martinot | Relation entre estime de soi et identification ethnique dans des contextes scolaires menaçants pour l'identité ethnique des élèves[END_REF][START_REF] Redersdorff | Being outperformed in an intergroup context: The relationship between group status and self-protective strategies[END_REF]). More research is needed, using longitudinal design to test for the effect of the inclusion in those AC on the self-perceptions of children.

Study 2

The aim of study 2 was to explore if the bad reputation associated with attending to an AC can impact students' performance in a threatening situation, that is when a task is presented as a test (threat) instead of a game (no threat). Given that AC students seemed to be aware of their bad reputation and had a lower perception of themselves, they should perform less successfully when a task is presented as a test (i.e., as being relevant regarding their abilities) than when it is presented as a game (i.e., as irrelevant regarding their abilities).

Method

Participants and Procedure

Fifty seven elementary school students (33 girls, 29 boys, aged 7-104 ) taken from the sample of Study 1 volunteered to take part on this experiment which took place 15 days after The experimenter (a woman) took children from classrooms to a little adjacent room and proposed them to play a game (non-threatening condition) or to take a test of their general abilities (threatening condition). After agreement, she showed them a memory game and asked them to perform it. The Memory ® is well-known card game in which pairs of cards (depicting objects or animals) are presented face down on a table 6 . The task consists in randomly flipping two cards and see if they match. If they do match, the cards are kept aside.

If not, the cards are turned back over and two other cards must be flipped. All pairs have to be found as fast as possible. The game is over when all the pairs have been found. This game thus requires cognitive abilities such as concentration, observation and spatial memory. The more one is able to memorize cards, the faster he/she completes the game. After the experimental task, the experimenter praised and thanked children and sent them back to their classroom.

Measures

Performance. Performance was assessed via the time students took to completely perform the memory task (in seconds). It was assessed with a stop watch visible to the child.

Additionally, the experimenter wrote children's class (RC or AC), sex and age.

Results

To

Discussion

The aim of this study was to test for stereotype threat effects on the performance of children from AC. As shown in Study 1, those students are victims of a bad reputation concerning their academic abilities, even if this group membership is temporary. As a salient group associated with a bad reputation, we hypothesized that they should be vulnerable to stereotype threat. This is what we found. When asked to do a memory task in an evaluative setting (presented as a measure of their intellectual abilities), AC students took more time to resolve it than when the memory game was presented in a non-evaluative way, as a game.

This result is consistent with past literature on stereotype threat with children [START_REF] Désert | So young and already victims of stereotype threat: Socio-economic status and performance of 6 to 9 years old children on Raven's progressive matrices[END_REF][START_REF] Muzzatti | Gender and mathematics: Attitudes and stereotype threat susceptibility in Italian children[END_REF][START_REF] Neuville | Can salience of gender identity impair math performance among 7-8 year old girls? The moderating role of task difficulty[END_REF], and adds to this literature another proof of the impact of a bad reputation on performance. Another alternative explanation, however, might be that AC children performed poorly simply because, as low achievers, they expected to fail. Indeed, the evaluative setting might have triggered specific expectations about future performance, based on their lack of past success and on their lower self-perceptions [START_REF] Wigfield | Expectancy-value theory of achievement motivation[END_REF]. Because names were not reported between Study 1 and 2, we cannot test for this explanation. So, Study 3 aimed at replicating results of Study 2 and test for the impact of initial self-perceptions on performance of students under stereotype threat. It aimed also at testing if stereotype threat impacted post-task selfperceptions.

Study 3

To test for this alternative explanation, we run Study 3 in two distinct phases, each separated by 15 days with a first phase during which we assessed self-perceptions of RC students and AC (self-perceptions at Time 1) and a second phase during which we asked them to take a test versus play a game and then assessed again self-perceptions (Time 2). As in Study 1, we hypothesized that self-perceptions of AC students (at Time 1) would be lower than those of RC students (Hypothesis 1). We also predicted that AC students would need more time to complete the task when it is presented as a test (threatening condition) than when it is presented as a game (non-threatening condition), while performance of RC students would not differ depending on how the task is presented (Hypothesis 2). We also predicted that AC students would perceive themselves after the task less positively in the threatening condition than in the non-threatening one [START_REF] Steele | A threat in the air: How stereotypes shape intellectual identity and performance[END_REF]. Self-perceptions of RC students should not differ across conditions (Hypothesis 3). We also predicted that those effects of stereotype threat on performance and post self-perceptions should appear over and above the effect of initial self-perceptions.

Participants and procedure

Participants of Phase 1 were 105 elementary school students (53 girls, 52 boys, aged 7-108 ) from a school in a low socio-economic area of the town. after Phase 1, the experimenter came to the classes and, as in Study 2, took the children in an adjacent room to ask them to take a memory task presented either as a game (non-threatening condition) or as a test of abilities (threatening condition) 10 . After the task, the experimenter asked children to complete the same self-perception measures as in Phase 1.

Measures

Self-perceptions. Self-perceptions (at Times 1 and 2) were assessed in the same way as in Study 1 but only with the 5 items of the academic subscale of the Pictorial Scale of Perceived Competence and Social Acceptance for Young Children [START_REF] Harter | The pictorial scale of Perceived Competence and Social Acceptance for Young Children[END_REF] 1984) to capture perceptions linked to the academic domain. Alphas were .61 at Time 1 (collective) and .74 at Time 2 (individual).

Performance. Performance was assessed via the time students took to completely perform the memory task (in seconds). Additionally, the experimenter wrote children's name, class (RC or AC), sex and age.

Results

9 Names were collected in order to match students' self-perceptions with performance. They were deleted afterward from the database. 10 Only children who had their parental authorization filled and were present during Phase 1 participated to the experiment.

Discussion

The aim of this study was to replicate results of Study 2 and also to disentangle stereotype threat effects from an alternative explanation in terms of motivational expectancies. The effects of stereotype threat on performance were replicated, and were still visible after initial self-perceptions were kept constant, suggesting again the power of stereotype threat and ruling out an explanation in terms of expectancy effects. Note that if predicted by the literature, the effect of self-perceptions or self-esteem on performance when under stereotype threat had rarely been tested. The only study we are aware of did not show any impact of self-esteem on performance when under threat. However, as the authors assessed self-esteem just before the manipulation, self-esteem might have served as a buffer and reduced the impact of threat [START_REF] Rydell | Capitalizing on multiple social identities to prevent stereotype threat: The moderating role of self-esteem[END_REF]. This is not the case in our study that assessed initial self-perceptions 15 days before. No relation could have been made

between the two evaluations.

Finally, our results showed an impact of stereotype threat on post-task selfperceptions, but this effect, even if a little bit higher for AC students, also appeared for RC students suggesting an effect due to the immediate evaluative situation. Also, our results

showed that the effect disappeared when initial self-perceptions were kept constant. Initial self-perceptions explained post-task self-perceptions.

General Discussion

The aims of the present studies were to investigate if a bad reputation was associated to students attending AC, if this bad reputation was known by those students and if it was somewhat internalized (Study 1). It was also to investigate the impact of stereotype threat on performance (Studies 2 and 3) and post-task self-perceptions (Study 3) and to test which of the threat induced by the situation or the expectancy effects due to initial self-perceptions could explain the performance decrement. No studies to our knowledge explored the effect of stereotype threat on performance and post-task self-perceptions while assessing initial selfperceptions.

Our results were broadly consistent with past research. Even such small and circumscribed groups as are the AC can be rapidly associated with a bad reputation. This appeared as young as 7 years old, that is for students who have never entered those programs before and who have just entered a new school (pre-schools and elementary schools are often different schools or are at least physically separated in France). RC students identified AC students easily and already seemed to have a specific perception of them. Moreover, those beliefs seemed to be widely shared as victims of this bad reputation were aware of them.

Our results also showed that AC students had lower self-perceptions than those of RC students. Does this mean that AC students have already internalized the bad reputation of their group? Our data cannot definitely answer this question. Indeed, even if the AC program is probably new for students, they entered it because of their learning difficulties in some academic subjects. It is thus likely that these students already have a history of low achievement, even if this history is a short one (for a review, see [START_REF] Monteil | The social regulation of classroom performances: A theoretical outline[END_REF], which impacted their self-perceptions. More research is needed to disentangle the impact of academic history from the effects of membership to such a group on self-perceptions.

Until then, being part of an adapted class is not without consequences. Indeed, our results showed that belonging to those AC impacted students' performance on a task presented as assessing their intellectual abilities. When it was presented as a game, no performance differences emerged, what is classically found in the literature on stereotype threat. Therefore, the problem is not so much students' abilities level, but rather the way the task is presented. Given that evaluations are more and more explicit as students go from pre-school to elementary school, such a phenomenon is likely to be repeated over time. However and hopefully, the impact of stereotype threat was not explained by initial self-perceptions.

This means that if a past history of underachievement and/or membership to an adapted class can negatively impact self-perceptions, they do not seem to reinforce the effects of stereotype threat.

However, initial self-perceptions seemed to reinforce the impact of an evaluative situation on post-task self-perceptions as the self-perceptions of AC students lowered more than that of RC students. Evaluative situations, even if regularly used to assess progress, may be stressful for young children, especially if they occur outside their regular context. Further studies are needed to explore more fully the respective impact of initial self-conceptions of abilities and situational cues on subsequent self-perceptions.

The present studies contributed to the literature on stereotype threat, enlarging it to another group and to another stereotype, as suggested by the knowledge all children have of the bad reputation associated with being part of an adapted class. They also showed that even a short term group membership can have deleterious effects [START_REF] Martiny | When a new group identity does harm on the spot: Stereotype threat in newly created groups[END_REF] and that initial self-perceptions do not explain the impact of stereotype threat on performance.

However, those studies are not without limitations. The first one is the lack of measure of initial meta-perceptions before a stereotype threat induction (Study 3). Moreover, a better measure is needed to assess perceptions and stereotypes. Our choice was motivated by very practical reasons. The first was to use the same measures for all children. If a pictorial measure may be quite adapted for 7 or 8 years old children, it may appear quite childish for older children. The second was to avoid any link between measures captured at Phase 1 and the experiment conducted at Phase 2. We chose to ask students to complete self-perception measures during class and ask an experimenter to run the experiment. This had the disadvantage to imply a collective session not favourable to validity concerns.

In conclusion, our three studies showed again the power of stereotype threat. However it raised other questions, in particular about the impact of the AC on self-perceptions of students and also about the extent to which stereotype threat can impact subsequent selfperceptions. Longitudinal studies would be much more appropriate, with measures of selfperceptions taken at the beginning of the school year and a follow-up of children during the year.

  test for stereotype threat effects on children's performance, an Ancova was run with the type of class (Regular vs Adapted) and condition (Game vs Test) as independent variables, time as the dependent variable, and age as a covariate. Results showed a significant effect of condition (F[1, 52]=43.76, p<.001, η²=.49) . Students were faster when the task was presented as a game (M second =169.07; SD=18.92) than when it was presented as a test (M

		250
		200
	Time (in seconds)	100 150	Threatening Condition Non-Threatening Condition
		50
		0
		Adapted Class	Regular Class
		Type of class
	condition (F[1, 52]=20.41, p<.001), the reverse was true in the non-threatening condition in
	which AC students took a little bit less time to solve the task than RC students (F[1, 52]=4.83,
	p<.04).
	Figure 1 Study 2: Effect of condition and type of class on performance

second =211.07; SD=33.09). Moreover, there was an interaction between the type of class and condition (F[1, 52]=22.85, p<.001, η²=.30) as shown on Figure

1

. This effect was still significant with age as a covariate 7 (F[1, 52]=22.50, p<.001, η²=.30).

As shown in Figure

1

, AC students took more time to solve the task in the threatening condition (M second = 231.40; SD = 37.97) than in the non-threatening condition (M second =159.60; SD = 20.96; F[1, 52]=68.30, p<.001). RC students were a little slower in the threatening condition (M second =196.30; SD = 11.93) than in the non-threatening one (M second = 179.30; SD = 8.99), but this difference was not significant (F[1, 52]=2.63, p=.11). In addition, whereas AC students took more time to solve the task than RC students in the threatening

7

Performance, even in this type of task can be linked with age. To avoid its effect, age was entered as a covariate.

  Their mean age was 8.16 years (SD=0.92). Among them, 67 were from a RC (37 girls; M age =8.13, SD=0.95) and 38 from an AC (16 girls, M age =8.21, SD=0.87). Among them, 42 were 2 nd grade (M age =7.36, SD=0.53), 43 form 3 td grade (M age =8.44, SD=0.67) and 20 from 4 th grade (M age =9.25, SD=0.44). M age =8.15, SD=1) and 34 from an AC (15 girls; M age =8.21, SD=0.91). During Phase 1, as in Study 1, self-perceptions were assessed by the teacher during school time; names, age and class were reported 9 . During Phase 2 which took place 15 days

	Eighty eight students from Phase 1 (45 girls; aged 7-10; M age =8.17, SD=0.96)
	participated to the experiment 15 days after and were assigned either to the threatening
	condition (N=43) or to the non-threatening condition (N=45). Among them, 54 were from a
	RC (30 girls;

  To test for hypothesis 1, we compared RC students' self-perceptions with those of AC students (at Time 1), while controlling for students' age. Results showed that RC students reported higher self-perceptions students (M=3.58, SD=.38) than did AC students (M=2.88, Results also showed a significant effect of the type of class (F[1, 83]=38.98, p<.001, η²=.32). Finally, the interaction between Time, Type of Class and Condition was not significant suggesting that the decrease in students self-perceptions at Time 2 was mostly due to the evaluative component of the threatening condition and not to stereotype threat effects.

	250
	SD=.51, F[1, 102]=63.26, p<.001, η²=.38). 200
	To test for hypothesis 2, we run an Ancova with the type of class (Regular vs Adapted) and condition (Game vs Test) as independent variables, Time as a dependent variable and age as a covariate. Results showed a significant effect of condition (F[1, 100 150 Threatening Condition Time (in seconds) Non-Threatening Condition
	83]=69.61, p<.001, η²=.46). Students were faster when the task was presented as a game
	50
	(M second =130.1; SD=26.05) than when it was presented as a test (M second =177.7; SD=40.79.
	0
	Adapted Class	Regular Class
	RC students were faster M second =136.5; SD=27.10) than AC students (M second =180.1; Type of class
	SD=46.37). More importantly, there was a significant interaction effect between the type of
	To test for hypothesis 3, we ran again a mixed Anova with the type of class (Regular
	class and condition (F[1, 83]=28.45, p<.001, η²=.26). As shown in Figure 2, AC students took
	vs Adapted) and condition (Game vs Test) as independent variables, self-perceptions at Time
	more time to solve the task in the threatening condition (M second = 212.65; SD = 30.18) than in
	1 and 2 as the dependent variables. Results showed a significant effect of Time (F[1,
	the non-threatening condition (M second =133.50; SD = 12.62; F[1, 83]=79.44, p<.001). This
	84]=56.10, p<.001, η²=.40). Self-perceptions of students were lower at Time 2 (M =3.06;
	was also the case, although in a lesser extent, of RC students (respectively for the threatening
	SD=0.65) than at Time 1 (M=3.35; SD=.49). Results also showed a significant interaction
	and the non-threatening conditions, M second =147.22; SD = 17.54 and M second = 128.55; SD =
	between Time and the Type of Class (F[1, 84]=8.48, p<.01, η²=.09). If self-perceptions of AC
	30.30; F[1, 83]=5.90, p<.02). In addition, whereas AC students took more time to solve the
	students at Time 1 (M =2.95; SD=0.39) were lower than those of RC students (M= 3.60; SD =
	task than RC students in the threatening condition (F[1, 83]=69.69, p<.001), time was
	0.37; F[1, 84]=13.64, p<.001), this difference between AC (M =2.44; SD=0.46) and RC
	equivalent between AC and RC students in the non-threatening condition (F<1).
	students (M =3.44; SD=0.47) was larger at Time 2 (F[1, 84]=43.92, p<.001). Additionally,
	Finally, results showed that this interaction was still significant after initial self-
	results showed a significant interaction between Time and Condition (F[1, 84]=52.48, p<.001,
	perceptions were entered as a covariate (F[1, 82]=28.29, p<.001, η²=.26).
	η²=.39). While self-perceptions of students did not change according to Condition at Time 1
	(F<1), they were at Time 2 lower in the threatening condition (M =2.66; SD=0.55) than in the
	Figure 2 Study 3: Effects of condition and type of class on performance
	non-threatening one (M =3.44; SD=0.47).

Three children were excluded from that sample because of their age. One was 6 years old, one 11 and the other 12. The youngest might have some trouble with answering the scale, the older ones because they were at least

years late on their normal course of studies.

Note that students completed the scales during class with their regular teacher, which might explain the low alphas. However, this was important because we did not want the students to think about it as an experiment (which might impact results of Study 2).

Which of course correspond for AC students to their meta-perception and for RC students to their otherperception.

Five children were excluded for the same reasons as in Study 1. One was 12 years old, 4 were 11.

Only children who had a written consent from their parents were allowed to participate.

This game was chosen for several reasons. First, it is a well-known game with well-known rules that do not necessitate particular explanation or specific skills to be understood. Second, it doesn't depend on prior knowledge and as such is less prone to be affected by age or class. Finally, it can be presented as a game (which it is) or as a test of abilities (which is believable since it requires abilities in concentration, memory and observation).

As in Study 1, 4 children were excluded from that sample because of their age. One was 6 years old, 3 were 11.


Silvio Cordeiro

Carlos Ramisch
email: carlos.ramisch@lif.univ-mrs.fr

Marco Idiart
email: marco.idiart@gmail.comavillavicencio@inf.ufrgs.br

Aline Villavicencio

Predicting the Compositionality of Nominal Compounds: Giving Word Embeddings a Hard Time

Distributional semantic models (DSMs) are often evaluated on artificial similarity datasets containing single words or fully compositional phrases.

We present a large-scale multilingual evaluation of DSMs for predicting the degree of semantic compositionality of nominal compounds on 4 datasets for English and French. We build a total of 816 DSMs and perform 2,856 evaluations using word2vec, GloVe, and PPMI-based models. In addition to the DSMs, we compare the impact of different parameters, such as level of corpus preprocessing, context window size and number of dimensions. The results obtained have a high correlation with human judgments, being comparable to or outperforming the state of the art for some datasets (Spearman's ρ=.82 for the Reddy dataset).

Introduction

Distributional semantic models (DSMs) use context information to represent the meaning of lexical units as vectors. They normally focus on the accurate semantic representation of single words. It is based on single words that many optimizations for these models have been proposed [START_REF] Lin | Automatic identification of noncompositional phrases[END_REF][START_REF] Erk | Exemplar-based models for word meaning in context[END_REF][START_REF] Baroni | Distributional memory: A general framework for corpus-based semantics[END_REF]. This is particularly true for word embeddings, that is, a type of DSM where distributional vectors are obtained as a by-product of training a neural network to learn a function between words and their contexts (Mikolov et al., 2013a).

Simultaneously, there has been intensive research on models to compose individual word vectors in order to create representations for larger units such as phrases, sentences and even whole documents [START_REF] Mitchell | Composition in distributional models of semantics[END_REF]Mikolov et al., 2013a). Larger units can often be assumed to have their meanings derived from their parts according to the language's grammar, but this is not always the case [START_REF] Sag | Multiword expressions: A pain in the neck for NLP[END_REF]. Many multiword units are associated with idiomatic interpretations, unrelated to the meaning of the component words (e.g. silver bullet, eager beaver).

Precision-oriented NLP applications need to be able to identify partly-compositional and idiomatic cases and ensure meaning preservation during processing. Compositionality identification is a first step towards complete semantic interpretation in tasks such as machine translation (to translate non-compositional compounds as a unit), word sense disambiguation (to avoid assigning a sense to parts of non-compositional compounds), and semantic parsing (to identify complex predicates and their arguments).

Even when larger units are explicitly represented in DSMs [START_REF] Mccarthy | Detecting a continuum of compositionality in phrasal verbs[END_REF][START_REF] Reddy | An empirical study on compositionality in compound nouns[END_REF]Mikolov et al., 2013c;[START_REF] Ferret | Compounds and distributional thesauri[END_REF], it is not clear whether the quality of these representations is comparable to the representations of single words. In particular, when building vectors for larger units, their generally lower frequencies in corpora (Kim and [START_REF] Su | Interpreting semantic relations in noun compounds via verb semantics[END_REF]) may combine with morphosyntactic phenomena to increase sparsity even further, often requiring non-trivial preprocessing (lemmatization and word reordering) to conflate variants.

This paper presents a large-scale multilingual evaluation of DSMs and their parameters for the task of compositionality prediction of nominal compounds in French and English. We examine parameters like the level of corpus preprocessing, the size of the context window and the number of dimensions for context representation. Additionally, we compare standard DSMs based on positive pointwise mutual information (PPMI) against widely used word embedding tools such as word2vec, henceforth w2v (Mikolov et al., 2013c), and GloVe [START_REF] Pennington | Glove: Global vectors for word representation[END_REF]. We start with a discussion of related work ( §2) and the materials and methods used ( §3). We report on the evaluations performed ( §4) and finish with conclusions and future work ( §5).

Related Work

We define nominal compounds as conventional noun phrases composed by two or more words, such as science fiction [START_REF] Nakov | On the interpretation of noun compounds: Syntax, semantics, and entailment[END_REF]. In English, they are often expressed as noun compounds but their syntactic realization may vary for different languages. For instance, one of the equivalent forms in French involves a denominal adjective used as modifier (e.g. cell death and the corresponding mort cellulaire).1 In this paper, we focus on 2-word nominal compounds involving modifiers that are nouns (e.g. word embedding) or adjectives (e.g. hard time).

Semantically, nominal compounds may display a wide range of idiomaticity, from compositional cases like access road to idiomatic or noncompositional cases like gravy train, whose meaning is unrelated to its parts.2 Even when there is a level of compositionality in the compound, the contribution of each word may vary considerably, independently from its status as a syntactic head or modifier, as cash in cash cow versus tears in crocodile tears. Indeed, various annotation scales have been proposed as means to collect human judgments about compositionality. Particularly for nominal compounds, [START_REF] Reddy | An empirical study on compositionality in compound nouns[END_REF] used a 6-point scale to collect judgments on the literal or figurative use of nominal compounds and its components in English. Similar judgments have also been collected for 244 German compounds, for which an average of 30 judgments on a scale from 1 to 7 were gathered through crowdsourcing [START_REF] Roller | The (un)expected effects of applying standard cleansing models to human ratings on compositionality[END_REF]. An alternative to multi-point scales is the binary judgment adopted by [START_REF] Farahmand | A multiword expression data set: Annotating non-compositionality and conventionalization for english noun compounds[END_REF], for a dataset of English nominal compounds.

There has been much interest in creating semantic representations of larger units, such as phrases (Mikolov et al., 2013b), sentences and documents [START_REF] Quoc | Distributed representations of sentences and documents[END_REF], and in examining whether it is possible to accurately derive the semantics of a compound or multiword expression from its parts [START_REF] Mccarthy | Detecting a continuum of compositionality in phrasal verbs[END_REF][START_REF] Baldwin | An empirical model of multiword expression decomposability[END_REF][START_REF] Tratz | ISI: Automatic classification of relations between nominals using a maximum entropy classifier[END_REF][START_REF] Reddy | An empirical study on compositionality in compound nouns[END_REF]. For the latter, proposals include using additive and multiplicative functions to combine vector representations of component words [START_REF] Mitchell | Vector-based models of semantic composition[END_REF][START_REF] Reddy | An empirical study on compositionality in compound nouns[END_REF], calculating the overlap between the components and the expression [START_REF] Mccarthy | Detecting a continuum of compositionality in phrasal verbs[END_REF] and looking at the literality of translations into multiple languages [START_REF] Salehi | Using distributional similarity of multi-way translations to predict multiword expression compositionality[END_REF]. Other proposals to explicitly represent the semantics of nominal compounds include the use of paraphrases [START_REF] Lauer | How much is enough?: Data requirements for statistical NLP[END_REF][START_REF] Nakov | Paraphrasing verbs for noun compound interpretation[END_REF][START_REF] Hendrickx | Semeval-2013 task 4: Free paraphrases of noun compounds[END_REF], and inventories of semantic relations [START_REF] Girju | On the semantics of noun compounds[END_REF].

The ability of DSMs for accurately capturing semantic information may be affected by a number of factors involved in constructing the models, such as the source corpus, context representation, and parameters of the model. Relevant corpus parameters include size [START_REF] Ferret | Identifying bad semantic neighbors for improving distributional thesauri[END_REF]Mikolov et al., 2013c) and quality [START_REF] Lapesa | A large scale evaluation of distributional semantic models: Parameters, interactions and model selection[END_REF]. Factors related to context representation include the context window size and the number of context dimensions adopted for a model [START_REF] Lapesa | A large scale evaluation of distributional semantic models: Parameters, interactions and model selection[END_REF]; the choice of contexts to be used with targets (syntactic dependencies vs. bag-of-words) [START_REF] Agirre | A study on similarity and relatedness using distributional and wordnet-based approaches[END_REF]; the use of morphosyntactic information [START_REF] Padó | Constructing semantic space models from parsed corpora[END_REF][START_REF] Padó | Dependency-based construction of semantic space models[END_REF]; context filtering [START_REF] Riedl | Topictiling: A text segmentation algorithm based on LDA[END_REF]Padró et al., 2014a); and dimensionality reduction methods (van de [START_REF] Van De Cruys | Multi-way tensor factorization for unsupervised lexical acquisition[END_REF]. Important model parameters that have been studied include the choice of association and similarity measures [START_REF] Curran | Scaling context space[END_REF] and the use of subsampling and negative sampling techniques (Mikolov et al., 2013c). However, the particular effects may be heterogeneous and depend on the task and model [START_REF] Lapesa | A large scale evaluation of distributional semantic models: Parameters, interactions and model selection[END_REF]. In this paper, we examine the impact of both corpus and context parameters for a variety of models, for the task of nominal compound compositionality prediction in English and French.

For the choice of particular DSM, contradictory results have been published showing the superiority of neural models [START_REF] Baroni | Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors[END_REF] and of more traditional but carefully designed models [START_REF] Levy | Improving distributional similarity with lessons learned from word embeddings[END_REF]. The former were also reported as a better fit to behavioral data on semantic prim-ing tasks [START_REF] Mandera | Explaining human performance in psycholinguistic tasks with models of semantic similarity based on prediction and counting: A review and empirical validation[END_REF]. Moreover, these evaluations are often performed on single-word similarity tasks [START_REF] Freitag | New experiments in distributional representations of synonymy[END_REF][START_REF] Camacho-Collados | A framework for the construction of monolingual and cross-lingual word similarity datasets[END_REF] and little has been said about the use of word embeddings for the compositionality prediction of multiword expressions. Two notable exceptions are the recent works of [START_REF] Salehi | A word embedding approach to predicting the compositionality of multiword expressions[END_REF] and [START_REF] Yazdani | Learning semantic composition to detect non-compositionality of multiword expressions[END_REF]. [START_REF] Salehi | A word embedding approach to predicting the compositionality of multiword expressions[END_REF] show that word embeddings are more accurate in predicting compositionality than a simplistic count-based DSM. [START_REF] Yazdani | Learning semantic composition to detect non-compositionality of multiword expressions[END_REF] focus on the composition function, using a lightly supervised neural network to learn the best combination strategy for individual word vectors. In order to consolidate previous punctual results, we present a large-scale and systematic evaluation, comparing DSMs and their parameters, on several compositionality datasets.

Materials and Methods

We examine the impact of corpus parameters related to the target language and the degree of corpus preprocessing adopted. We also investigate context parameters related to the size of the context window and the number of dimensions used to represent context.

Corpora Preprocessing

We use the lemmatized and POS-tagged versions of the ukWaC for English (∼2 billion tokens) and frWaC (∼1.6 billion tokens) for French [START_REF] Baroni | The wacky wide web: a collection of very large linguistically processed web-crawled corpora[END_REF] to train the models and build vector representations of words and compounds. For each corpus, we re-tokenize all target compounds as a single word with a separator (e.g. monkey business → monkey business) and re-tag them using a single manually selected tag per compound to handle POS-tagging errors. 3 All forms are then lowercased (surface forms, lemmas and POS-tags); and noisy tokens, with special characters, numbers or punctuation, are removed. Additionally, ligatures are normalized for French (e.g. oe → oe) and a spellchecker4 is applied to normalize words across English spelling variants (e.g. color → colour).

To test the influence of preprocessing in model accuracy, for each corpus, we generate four variants with different degrees of abstraction:

1. surface + : the original corpus with no preprocessing, containing surface forms.

2. surface: stopword removal; generating a corpus of surface forms of content words. 3. lemma: stopword removal and lemmatization; generating a corpus of lemmas of content words. 4. lemmaPOS: stopword removal, lemmatization and POS-tagging; generating a corpus of content words, represented as lemma/tag.

The operation of stopword removal eliminates from the corpus all function words, leaving only nouns, adjectives, adverbs and verbs. In lemmatized corpora, the lemmas of proper names are replaced by placeholders.

Compositionality Datasets

For evaluation, we use nominal compound compositionality datasets for English (Reddy, Reddy++ and Farahmand) and for French (FR-comp). They provide annotations as to whether a given compound is more idiomatic or more compositional.

Reddy contains compositionality judgments for 90 compounds and their individual word components, in a scale of literality from 0 (idiomatic) to 5 (literal), collected with Mechanical Turk [START_REF] Reddy | An empirical study on compositionality in compound nouns[END_REF]. For each compound, compositionality scores are averaged over its annotators. Compounds included in the dataset were selected to balance frequency range and degree of compositionality (low, middle and high). We use only the global compositionality score, ignoring individual word judgments. With a few exceptions (e.g. sacred cow), most compounds are formed exclusively by nouns.

Reddy++ is a new resource created for this evaluation [START_REF] Ramisch | How naked is the naked truth? a multilingual lexicon of nominal compound compositionality[END_REF]. It extends the Reddy set with an additional 90 English nominal compounds, in a total of 180 entries. Scores also range from 0 to 5 and were collected through Mechanical Turk and averaged over the annotators. The extra 90 entries include some adjective-noun compounds and are balanced with respect to frequency and compositionality. We focus our evaluation on this combined dataset, since it includes Reddy. However, to allow comparison with state of the art, we also report results individually for Reddy.

Farahmand contains 1042 English compounds extracted from Wikipedia with binary noncompositionality judgments by four experts [START_REF] Farahmand | A multiword expression data set: Annotating non-compositionality and conventionalization for english noun compounds[END_REF]. We consider a compound as non-compositional if at least two judges agree that it is non-compositional, following Yaz-dani et al. (2015). In our evaluations, we use the sum of all judgments in order to have a single numeral compositionality score, ranging from 0 (compositional) to 4 (idiomatic).

FR-comp is also a new resource created for this evaluation [START_REF] Ramisch | How naked is the naked truth? a multilingual lexicon of nominal compound compositionality[END_REF]. It contains 180 adjective-noun and noun-adjective compounds in French, such as belle-mère (mother-in-law, lit. beautiful-mother) and carte bleue (credit card, lit. blue card). This dataset was constructed in the same manner as the extension to Reddy, that is, using crowdsourcing and average numerical scores. Special care was taken to guarantee that annotators were native speakers by asking them to provide paraphrases along with compositionality scores.

The new datasets Reddy++ and FR-comp are similar to Reddy. For instance, the average standard deviation of compound scores given by different annotators is σ = 1.17 for the new compounds in Reddy++, σ = 1.15 for FR-comp and σ = 0.99 for Reddy. Their detailed evaluation is presented by [START_REF] Ramisch | How naked is the naked truth? a multilingual lexicon of nominal compound compositionality[END_REF].

DSM Models

We build three types of DSMs: models based on sparse PPMI cooccurrence vectors, as well as those constructed with word2vec and GloVe.

PPMI For each target word or compound, we extract from the corpus its neighboring nouns and verbs in a symmetric sliding window of w words to the left/right5 , using a linear decay weighting scheme with respect to its distance d to the target [START_REF] Levy | Improving distributional similarity with lessons learned from word embeddings[END_REF]. In other words, each cooccurrence count of target-context pairs is incremented by w + 1 -d instead of 1. The representation of a target is a vector containing the positive pointwise mutual information (PPMI) association scores between the target and its contexts. 6In PPMI-thresh, we follow Padró et al. (2014b) to select the top k most relevant contexts (highest PPMI) for each target. No further dimensionality reduction is applied.

In PPMI-TopK, we use a fixed global list of 1000 contexts, built by looking at the most frequent words in the corpus: the top 50 are skipped, and the next 1000 are taken [START_REF] Salehi | A word embedding approach to predicting the compositionality of multiword expressions[END_REF]. No further dimensionality reduction is applied.

In PPMI-SVD, for each target, contexts that appear less than 1000 times are discarded. 7 We then use the Dissect toolkit8 [START_REF] Dinu | DISSECT -DIStributional SEmantics composition toolkit[END_REF] in order to build a PPMI matrix and reduce its dimensionality using singular value decomposition (SVD) to factorize the matrix.

w2v Uses the word2vec toolkit based on neural networks to predict target/context cooccurrence (Mikolov et al., 2013a). We build models from two variants of word2vec: CBOW (w2v-cbow) and skipgram (w2v-sg). In both cases, the configurations are the default ones, except for the following: no hierarchical softmax; negative sampling of 25; frequent-word downsampling weight of 10 -6 ; runs 15 training iterations. We use the default minimum word count threshold of 5.

glove We use the count-based DSM of Pennington et al. ( 2014), which implements a factorization of the co-occurrence count matrix. The configurations are the default ones, except for the following: internal cutoff parameter x max = 75; builds co-occurrence matrix in 15 iterations. Due to the large vocabulary size, we use a minimum word count threshold of 5 for lemma-based models, 15 for surface and 20 for surface + .

For each DSM, we evaluate the influence of a set of parameters. By varying the values of these parameters, we build a total of 408 models per language. The parameters are:

• WORDFORM: Refers to one of the four variants of each corpus: surface + , surface, lemma, and lemmaPOS. • WINDOWSIZE: Indicates within how many words to the left/right we are searching for target-context co-occurrence pairs. In this work we explore windows of sizes of 1, 4 and 8. • DIMENSION: Each model is constructed to have a maximum number of final dimensions for each vector. We generate models with 250, 500 and 750 dimensions.

Compositionality Prediction

To predict the compositionality of a nominal compound w 1 w 2 using the DSMs, we use as a measure the cosine similarity between the compound vector representation v(w 1 w 2 ) and the sum of the vector representations of the component words:

cos( v(w 1 w 2 ), v(w 1 + w 2 ) )
where for v(w 1 + w 2 ) we use the normalized sum

v(w 1 + w 2 ) = v(w 1 ) ||v(w 1 )|| + v(w 2 ) ||v(w 2 )|| .
In this framework, a compound is compositional if the compound representation is close to the sum of its components representations (cosine is close to 1), and it is idiomatic otherwise. One possible improvement of the predictive model would consist in using more sophisticated composition functions instead of sum, such as the multiplicative model of [START_REF] Mitchell | Vector-based models of semantic composition[END_REF]. However, we want to first assess the performance of a simple additive function. Other optimized functions like the ones proposed by [START_REF] Yazdani | Learning semantic composition to detect non-compositionality of multiword expressions[END_REF] could also be verified, but are out of the scope of this paper, since they are based on supervised learning.

Evaluation Setup

We evaluate the compositionality models and their parameters on the datasets described in Section 3.2. For Reddy, Reddy++ and FR-comp, we report Spearman's ρ correlation between the ranking provided by humans and those calculated from the models. We follow [START_REF] Yazdani | Learning semantic composition to detect non-compositionality of multiword expressions[END_REF] and report the best F1 score (BF1) obtained for the Farahmand dataset, by calculating the F1 score for the top k compounds classified as positive (noncompositional), for all possible values of k.

Given the high number of experiments we performed, we report the best performance of each model type. For instance, the performances reported for w2v-cbow using different values of WINDOWSIZE are the best configurations across all possible values of other parameters such as DIMENSION and WORDFORM. This avoids reporting local maxima that can arise if one fixes all other parameters when evaluating a given one [START_REF] Lapesa | A large scale evaluation of distributional semantic models: Parameters, interactions and model selection[END_REF].

For Reddy++ and Farahmand, we distinguish between strict evaluation, reported in the form of wider bars in the figures, and loose evaluation, shown as narrow blue bars in the figures. Strict evaluation corresponds to the performance of the model only on those compounds that have a vector representation in all underlying DSMs, 175 out of 180 for Reddy++ and 913 out of 1042 for Farahmand. Loose evaluation considers the full dataset, using a fallback strategy for the imputation of missing values, assigning the average compositionality score to absent compounds [START_REF] Salehi | A word embedding approach to predicting the compositionality of multiword expressions[END_REF]. This is particularly important for Farahmand, which contains more rare compounds such as universe human and mankind instruction so that 129 compounds are missing in the corpus. Only strict evaluation is reported for FR-comp, as all compounds are frequent enough in FRWaC.

The vectors generated by w2v and glove have some non-determinism due to random initialization. To assess its impact on results, we report the average of 3 runs using identical configurations and use error bars in the graphics.9

Results

We report results on each dataset separately and then discuss findings that hold for all datasets.

Reddy++ and Reddy Datasets

Figure 1 summarizes the results for Reddy++ dataset. 10 Overall, w2v-cbow (ρ = 0.73), w2v-sg (ρ = 0.73), PPMI-SVD (ρ = 0.72) and PPMIthresh (ρ = 0.71) obtain similar results. In spite of this, except for the two best w2v models, all differences were deemed statistically significant (Wilcoxon rank correlation test, p < 0.05).

Figure 1(b) shows the influence of the degree of corpus preprocessing (shown as WORDFORM in these figures). The results are heterogeneous, as the best w2v models seem to profit from the presence of stopwords, unlike the other models for which more preprocessing (lemma and lemma-POS) leads to better results. One exception is PPMI-SVD for which the use of lemmaPOS drastically reduces performance. 11For WINDOWSIZE, Figure 1(c), although increasing context size seems to help DSMs (at least up to 4), for the best w2v models, a better result is obtained with limited context of 1 word left/right. Probably the interaction between the subsampling strategy and randomized window size explains why increasing this value does not improve the The results for context vector dimensionality, Figure 1(d), show, as expected, that the best results are obtained with larger dimensions (DIMENSION=750) for all models, except for glove, which displays very similar results independently of the number of dimensions.

Examining the Reddy dataset alone, the same trends for all parameters were found, but with higher results. The overall best performances on Reddy were quite similar: w2v-cbow (ρ = 0.82), w2v-sg (ρ = 0.81), PPMI-SVD (ρ = 0.80) and PPMI-thresh (ρ = 0.79), and the differences are significant except for the two best w2v models. The 90 compounds added to Reddy++ seem to be more difficult to assess than the original ones, probably because they include many adjectives, which have been found harder to judge for compositionality than nouns [START_REF] Ramisch | How naked is the naked truth? a multilingual lexicon of nominal compound compositionality[END_REF].

Farahmand Dataset

Figure 2(a) shows the overall best model for the Farahmand dataset. PPMI-SVD reached a BF1 score of 0.52, with DIMENSION=750, WINDOW-SIZE=4, using lemma, and both w2v (BF1=0.51) obtain comparable results with similar configurations.

These results show a marked difference between the loose (the narrower bars in the figures) and the strict evaluation (wider bars). The former uses a fallback strategy for the imputation of missing values that does not accurately reflect how the compositionality scores vary. Indeed, we observed that compounds that do not appear very often in our corpora tend to be non-compositional, whereas most of the compound occurrences are compositional, increasing average compositionality. For instance, the 10 most compositional compounds in Reddy++ occur an average of 26551 times in the UKWaC vs 1096 times for the 10 least compositional ones. Spearman rank correlation between frequency and compositionality in Reddy++ is ρ = 0.43.12 In short, even if a fallback strategy is adopted as the means to obtain a lowerbound for performance, it may be unrelated to the real performance for the missing compounds.

For most models, corpus preprocessing resulted in better scores, with WORDFORM=lemma outperforming all other forms of preprocessing, especially for French. Concatenating lemmas and POS tags does not seem to help, probably due to decreasing word frequencies without substantial gain in informativeness (Figure 2(b)).

The impact of WINDOWSIZE has a similar trend to the one found for the Reddy++ and Reddy datasets (Figure 2(c)). That is, the larger window was preferred by most models, but the average difference between the best and the worst size for each DSM is only 0.01. For DIMENSION, a larger number resulted in better scores, as expected, with 750 being the best for all models in Figure 2(d). Nonetheless, here too the average difference in scores between DIMENSION=750 and 250 is 0.01.

FR-comp Dataset

Globally, for the FR-comp dataset, PPMI-thresh (ρ = 0.70) outperforms glove (ρ = 0.68) and w2v (ρ = 0.66), as can be seen in Figure 3(a). 13For morphologically rich languages like French, Figure 3(b) indicates that working on lemmatized data often yields better results than working on surface forms. Lemmas conflate the frequencies for all the many morphologically inflected variants which would otherwise be dispersed in different surface forms. Therefore, it is not surprising that the best results concerning WORDFORM are achieved by lemma. These results differ from English, where a corpus without any preprocess-

Discussion

When comparing DIMENSION across languages and datasets, larger values often bring better performance. Likewise, the lemma is usually the better WORDFORM. The recommended WINDOW-SIZE depends on the model and language, but for the best models in all datasets, a window of 1 outperforms the others. This may be a consequence of the linear decay context weighting process, which assigns higher weights to closer words as window size increases. As an overall conclusion, in combination with a large dimension and a small14 For w2v, the same parameters used for English were adopted also for French. As a sanity check, we tested a range of negative sampling values [5,15,25,35,50], as well as subsampling rates for powers of 10 in [10 -3 to 10 -7 ]. Variations in ρ are minor and do not show any clear trend. window size, investing in preprocessing provides a good balance of a small vocabulary (of lemmas) and good accuracy. This is especially clear for a morphologically richer language like French, where lemmatization is homogeneously better for all models, even in w2v, for which surface forms were better for English.

In terms of models, the w2v models performed better than PPMI for Reddy++, both were in a tie for Farahmand, and w2v was outperformed by PPMI-thresh for French. The performance of glove for English was underwhelming, probably because we did not perform parameter tuning. As shown by [START_REF] Salehi | A word embedding approach to predicting the compositionality of multiword expressions[END_REF], PPMI-TopK is not an appropriate DSM for this task, as it does not model relevant cooccurrence very well.

The average Spearman's ρ for Reddy over all tested parameter configurations was 0.71 for both w2v models and 0.67 for PPMI-SVD and PPMI-

Model & Parameters

Result [START_REF] Reddy | An empirical study on compositionality in compound nouns[END_REF] .71 [START_REF] Salehi | Using distributional similarity of multi-way translations to predict multiword expression compositionality[END_REF] .74 [START_REF] Salehi | A word embedding approach to predicting the compositionality of multiword expressions[END_REF] .80

Best w2v (sg, WF=surface, D=750, W=1) .82 (.80) Best PPMI (thresh, WF=surface, D=750, W=8) .80 (.80) Best glove (WF=lemmapos, D=250, W=8)

.76 (.76) thresh, and this was also observed for the other datasets. In short, both types of models can obtain good results. While PPMI-thresh is a simple, fast and inexpensive model to build, w2v has a free and push-button implementation, and requires less hyper-parameter tuning, as is it seems more robust to parameter variation. More generally, the best results obtained for Reddy and Farahmand are comparable and even outperform the state of the art, as shown in Tables 1 and2, when strict evaluation is adopted (that is, when not using a fallback strategy for missing compounds).

Conclusions

In this paper we presented a multilingual, largescale evaluation of DSMs for compound compositionality prediction. We have built 816 DSMs and performed 2,856 evaluations, examining the impact of corpus and context parameters, namely the level of corpus preprocessing, the context window size and the number of dimensions. Evaluation on 3 English datasets and a French one revealed that a large dimension is consistently better, and corpus preprocessing is usually beneficial. The choice of window size varies according to language and dataset, but a small window can often provide a good performance. The DSMs w2v and PPMI alternated in providing the best results. Moreover, the results obtained were comparable and even outperformed the state-of-the-art.

As future work, we plan to examine the use of a voting scheme for combining the output of complementary DSMs. Moreover, we also plan to combine additional sources of information for building the models, such as multilingual resources or translation data, to improve even further the compositionality prediction. We would also like to propose and evaluate more sophisticated compositionality functions that take into account the unbalanced contribution of individual words to the global meaning of a compound.

  (a) Overall best Spearman's ρ per DSM. (b) Best Spearman's ρ per DSM and WORDFORM. (c) Best Spearman's ρ per DSM and WINDOWSIZE. (d) Best Spearman's ρ per DSM and DIMENSION.

Figure 1 :

 1 Figure 1: Spearman's ρ for different DSM parameters on Reddy++ dataset.

  (a) Overall best BF1 score per DSM. (b) Best BF1 score per DSM and WORDFORM. (c) Best BF1 score per DSM and WINDOWSIZE. (d) Best BF1 score per DSM and DIMENSION.

Figure 2 :

 2 Figure 2: BF1 scores for different DSM parameters on Farahmand dataset.

  (a) Overall best Spearman's ρ per DSM. (b) Best Spearman's ρ per DSM and WORDFORM. (c) Best Spearman's ρ per DSM and WINDOWSIZE. (d) Best Spearman's ρ per DSM and DIMENSION.

Figure 3 :

 3 Figure 3: Spearman's ρ for different DSM parameters on FR-comp dataset.

Table 1 :

 1 Comparison of our best models with state-of-the-art ρ forReddy. 14

	Model & Parameters	Result
	Yazdani et al. (2015)	.49
	Best w2v (sg, WF=lemma, D=500, W=1) .51 (.47)
	Best PPMI (svd, WF=lemma, D=750, W=4) .52 (.45)
	Best glove (WF=lemma, D=500, W=8)	.40 (.36)

Table 2 :

 2 Comparison of our best models with state-of-the-art BF1 forFarahmand. 14

In French, one can also use a preposition and optional determiner, like cancer du poumon (lung cancer).

It refers to an initiative that provides money to many people without much effort.

We use a simplified tag set (e.g. v instead of vvz).

https://hunspell.github.io

Syntactic context definition is planned as future work.

PPMI vectors are built using minimantics https:// github.com/ceramisch/minimantics.

Aggressive filtering was required because SVD seems quite sensitive to low-frequency contexts.

http://clic.cimec.unitn.it/composes/toolkit/ index.html

Error bars are barely visible because results are stable.

In the remainder of this section, we will discuss strict evaluation results (outer bars).

Further investigation must be done to determine the cause of this reduction as an increase in vocabulary size alone is insufficient to explain the effect, given that both surface forms outperform it.

We report these figures for Reddy++ because Farahmand has many ties, given the binary nature of compositionality annotations.

As all compounds in the dataset occur in the corpus, only strict evaluation results are reported.

DSM parameters: WF: WORDFORM, D: DIMENSION, W: WINDOWSIZE. Results in parentheses for loose evaluation, using fallback.

Acknowledgments

This work has been partly funded by projects PARSEME (Cost Action IC1207), PARSEME-FR (ANR-14-CERA-0001), AIM-WEST (FAPERGS-INRIA 1706-2551/13-7), CNPq 482520/2012-4, 312114/2015-0, "Simplificac ¸ão Textual de Expressões Complexas", sponsored by Samsung Eletrônica da Amazônia Ltda. under the terms of Brazilian federal law No. 8.248/91.