Mariam Makramalla

Andreas J Stylianides

The contextual power dynamics in defining and utilising problem solving tasks: A case study at an Egyptian private school

Keywords: problem solving, socio-cultural power tensions, educational ideologies, teacher beliefs

This paper builds on Ernest's five ideologies for mathematics education in order to explore the socio-cultural buy-in of mathematical problem solving tasks within the context of private schools that are governed by the Egyptian national mathematics curriculum. The research contributes to the contextual understanding of teacher beliefs and school power dynamics that affect the adoption or rejection of a culturally foreign teaching ideology of problem solving. Findings from one teacher focus group revealed that the teachers had a good understanding of the features that align with a literature-based classification of problem solving tasks. Yet, a disparity was found between desired and existing best practices for classroom implementation of problem solving tasks. This disparity is argued to be a function of power tensions between inner and outer circles of contextual influence.

Introduction

Literature argues against school education being viewed as a neutral enterprise [START_REF] Apple | Ideology and curriculum[END_REF]. The skill and knowledge dissimilation process in schools is arguably better understood when viewed from the perspective of the schools' position in a dynamic and turbulent context that incorporates complex social and cultural power relations [START_REF] Bernstein | Class, codes and control: Towards a theory of educational transmittion[END_REF]. This study explores the specific context of mathematics instruction at Egyptian private schools that teach the national curriculum. As will be elaborated further in the upcoming sections, this context is particularly interesting as it embodiesat the private school levela teaching culture that appreciates an inquiry-based enactment [START_REF] Apple | Ideology and curriculum[END_REF] of the mathematics curriculum. This culture is however situated and controlled by an imposed national curriculum agenda that is rather stagnant and one-sided in nature [START_REF] Hargreaves | The diploma disease in Egypt: learning, teaching and the monster of the secondary leaving certificate[END_REF][START_REF] Naguib | The production and reproduction of culture in Egyptian schools[END_REF]. The study aims to unravel the mentioned inner and outer circles of influence and how these play a role in shaping mathematics teachers' beliefs and practices, particularly in relation to their teaching of mathematical problem solving.

Literature Review

Literature claims that social and cultural configurations govern the outcome of schooling in order to ensure the reproduction of an existing culture of stratified social class relations [START_REF] Apple | Ideology and curriculum[END_REF]. In line with that view, [START_REF] Alexander | Culture and pedagogy: International comparisons in primary education[END_REF] unpacks the term 'education' and focuses on three particular aspects, namely curriculum, assessment and inspection. He claims that the formation of those is bound by different forms of policy and power. Our study focuses on the first element and argues that, since education cannot be viewed as a neutral enterprise, similarly curriculum design and knowledge distribution schemes cannot be viewed as neutral. According to [START_REF] Apple | Ideology and curriculum[END_REF], curriculum design and enactment which govern the process of skills formation and cultural capital distribution [START_REF] Bourdieu | The forms of capital[END_REF] are controlled by power dynamics that relate to a given context's social class hierarchy.

Particularly in relation to science and mathematics curricula, literature suggests that the general tendency to move towards a curriculum design where students are taught to become independent inquirers is often quenched by a culturally selective curriculum enactment tradition where educators may choose to teach certain skills or forms of knowledge in order to re-enforce certain configurations of social power. This selective tradition, which in turn is governed by social and cultural power dynamics, can be better understood when considering the historical roots that led to its formation [START_REF] Apple | Ideology and curriculum[END_REF][START_REF] Bourdieu | The forms of capital[END_REF].

Scholars (e.g., [START_REF] Alexander | Culture and pedagogy: International comparisons in primary education[END_REF][START_REF] Apple | Ideology and curriculum[END_REF] have added to the discussion about forces affecting curriculum design and enactment the dimension of political control. In his comparative study on primary education in five different cultures, [START_REF] Alexander | Culture and pedagogy: International comparisons in primary education[END_REF] claims that the national constitutional identity plays a vital role in prescribing curriculum outlines and elevating certain forms of knowledge and skill transfer over others. According to scholarship, to better understand the question of political control, the question of centralizing (or de-centralizing) systems of curriculum design and enactment is crucial to consider [START_REF] Alexander | Culture and pedagogy: International comparisons in primary education[END_REF].

In light of that understanding, we intend to map the situation of primary stage mathematics curriculum enactment and design in Egypt. A wider study from which this work is derived has elaborately addressed, drawing on relevant literature [START_REF] Heyworth-Dunne | An introduction to the history of education in modern Egypt[END_REF][START_REF] Sika | Educational reform in Egyptian primary schools since the 1990s: A study of the political values and behavior of sixth grade students[END_REF], the historical evolution of education in modern Egypt. It sought to map out social and cultural configurations and power tensions against educational outcomes of skills and cultural capital distribution. While it is beyond the scope of this work to elaborately address the overall historical mapping of education in modern Egypt, we consider it as important to use sections of that mapping, particularly to better understand political and social control forces governing the purpose and identity of curriculum enactment and formation. In light of [START_REF] Alexander | Culture and pedagogy: International comparisons in primary education[END_REF] views about political control, it is worth mentioning that, since the republic of Egypt was formed in 1953, curriculum design and enactment agendas have been standardized. A centralized system that is governed and tightly controlled by the Ministry of Education (MOE) strategically selects the forms of knowledge acquisition and skill transfer that are to be enacted by the national curriculum to all schools of Egypt. Literature [START_REF] Naguib | The production and reproduction of culture in Egyptian schools[END_REF] shows that for the case of the mathematics curriculum the focus is on factual knowledge and the transmission of a one-sided set of facts that are not to be questioned or debated. Mathematics instruction is meant to take on the form of a procedural, teacher-led, examoriented structure [START_REF] Sika | Educational reform in Egyptian primary schools since the 1990s: A study of the political values and behavior of sixth grade students[END_REF]. The national curriculum in Egypt is taught in all public schools and also in the so-called 'national curriculum private schools'. Both forms of schooling stem from different social, cultural and historical roots [START_REF] Heyworth-Dunne | An introduction to the history of education in modern Egypt[END_REF]). The historical plotting, referred to earlier, indicates a so-called cultural migration [START_REF] Alexander | Culture and pedagogy: International comparisons in primary education[END_REF] of educational values which governed the original institution of foreign private schooling, even before the republic of Egypt was formed. Those values are centered around a more multi-façade, student-centered scheme for curriculum enactment. Nevertheless, being under the governance of the MOE, a centralized system ensures that the curriculum enactment follows the same format across all types of national schools in Egypt [START_REF] Hargreaves | The diploma disease in Egypt: learning, teaching and the monster of the secondary leaving certificate[END_REF][START_REF] Naguib | The production and reproduction of culture in Egyptian schools[END_REF][START_REF] Sika | Educational reform in Egyptian primary schools since the 1990s: A study of the political values and behavior of sixth grade students[END_REF]. [START_REF] Alexander | Culture and pedagogy: International comparisons in primary education[END_REF] claims that schools are micro-cultures mediating imposed curricula in such a way that reflects some of the school's own flavor. It is true that, despite the fact that the same mathematics curriculum is meant to be applied across the mentioned wide base of schooling, curriculum enactment remains quite a complex process [START_REF] Remillard | Conceptualizing the curriculum enactment process in mathematics education[END_REF]. This paper presents a case study that addresses the interplay between a centrally governed curriculum and its contextual enactment in the culture of a private school. More specifically, we will focus on how the factual nature of the national mathematics curriculum translates into the formation of teacher beliefs about problem solving and how in turn these beliefs affect, according to the teachers, their curriculum enactment decisions in the classroom. The research questions to be addressed are the following:

(1) Within the context of a private school in Egypt that is bound to follow the national curriculum, how do school mathematics teachers perceive features of problem solving tasks? (2) Within this same context, how do school mathematics teachers view the contextual suitability of adopting their perceived problem solving tasks into their local teaching setting?

Theoretical Framework

In his model of educational ideology for mathematics, [START_REF] Ernest | The philosophy of mathematics education[END_REF] situates the wider aims of mathematics education along with the specific function of school mathematics teaching, learning and assessment within a context that is bound by the theory of society, the theory of the child and the moral ideologies attached to a given society. [START_REF] Ernest | The philosophy of mathematics education[END_REF] differentiates between five educational ideologies in viewing teaching and learning mathematics and maps those out against cultural power configurations in society. He views these ideologies from various social and pedagogical perspectives. Three of those perspectives form the theoretical grounding of this paper (Figure 1). As illustrated in Figure 1, it is not possible to dissociate a teacher's perception of mathematics, which in turn translates into a teaching and learning theory, from the student's experience when learning mathematics and from the wider nature of viewing and experiencing mathematics both by the teacher and by the student. Figure 1 shows how five different mathematical ideologies manifest themselves in the experience of the teacher and the learner. These ideologies extend all the way from viewing mathematics as a set of fixed truths and teaching it in a rather authoritarian and transmission-based mechanism all the way to viewing mathematics as a subject of constructive nature and the teaching of it as being rather more inquiry-and discussion-based. In turn, the learning of mathematics is viewed as lying on a spectrum that extends from the mastery of formulas and procedures on the one end to the learning of inquiry skills and negotiation on the other end. [START_REF] Ernest | The philosophy of mathematics education[END_REF] discussion of the latter features align broadly with our understanding of problem solving skills as discussed in the literature [START_REF] Schoenfeld | Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics[END_REF][START_REF] Stein | Implementing standardsbased mathematics instruction: A casebook for professional development[END_REF].

Various scholars [START_REF] Polya | How to solve it[END_REF][START_REF] Schoenfeld | Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics[END_REF] have sought to conceptualize mathematical problem solving as a construct and apply that understanding in the formulation of mathematical tasks. As an extension of the scholarly view that perceives mathematics as ranging from the pure application of rules on the one side of the spectrum to problem solving on the other [START_REF] Schoenfeld | Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics[END_REF], [START_REF] Stein | Implementing standardsbased mathematics instruction: A casebook for professional development[END_REF] identified four types of mathematical tasks: memorisation tasks, procedures without connections tasks, procedures with connections tasks, and doing mathematics tasks. While we acknowledge that the term "problem solving" has been used in different ways in the literature [START_REF] Schoenfeld | Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics[END_REF][START_REF] Reiss | Problem solving in the mathematics classroom: The German perspective[END_REF], for the purpose of this work we have adopted [START_REF] Stein | Implementing standardsbased mathematics instruction: A casebook for professional development[END_REF] perspective on problem solving which, at the task level, has been conceptualized by the authors as "doing mathematics tasks" (p. 16). According to the authors, doing mathematics (or problem solving) tasks are unique in that they invoke student analysis based on conceptual understanding of the underlying mathematical principles, their solution process has an unpredictable nature and a high level of ambiguity, they place on the solvers high cognitive demands and they require solvers to self-regulate their thought process. The broad understanding of the ideologies that formulate a teacher's and a learner's view of mathematics along with the specific unfolding of that view in the form of mathematical tasks constitute the foundation to conceptualizing the upcoming analysis.

Methodology

As part of the larger project exploring contextual power dynamics in relation to framing teacher perceptions of mathematical problem solving, this research adopts the methodology of a multiple case study, conducting various focus groups across a wide range of private schools in Egypt. It is worth mentioning that, in Egypt, private schools annually serve a wide base of students (approximately two million) from various sociocultural family backgrounds (CAPMAS Yearly Report, 2017). The case studies have been carefully selected to represent the high end, the low end, and the middle of the socio-cultural map of private schooling. Each case study represents findings of two focus groups conducted at different schools that are both classified within the same sociocultural cluster of schooling. In the following, we present findings from one focus group that has been classified by local partners as lying on the high end of the private schooling spectrum. This classification has been also confirmed based on an assessment of the school location, the school fee and the selection criteria of both students and teachers. The sample comprised a group of six mathematics teachers.

Being a contextual study, it was important that the method selected was locally fitting. Researchers have investigated the suitability of using focus groups as a research method in the Arab context and have argued that focus groups are well received by participants [START_REF] Thomas | Focus groups in qualitative research: Culturally sensitive methodology for the Arabian Gulf?[END_REF]. Being a collective culture [START_REF] Thomas | Focus groups in qualitative research: Culturally sensitive methodology for the Arabian Gulf?[END_REF], teachers' choices and practices are governed and influenced by each other's perceptions making it unlikely for a teacher to adopt a practice in their classroom if it is not collectively perceived as best practice by most other teachers. Hence group power dynamics represented a cultural reality that needed to be taken into consideration.

The data collection protocol was framed around prompts of a Task Sorting Activity [START_REF] Friedrichsen | Using a card-sorting task to elicit and clarify scienceteaching orientations[END_REF] followed by a more holistic focus group discussion. For the Task Sorting Activity, eight tasks were put together ranging across [START_REF] Stein | Implementing standardsbased mathematics instruction: A casebook for professional development[END_REF] mathematical task spectrum. The tasks showed an interesting variation in terms of their presentation, including numerical tasks, word problems, and visual integration tasks. The participating teachers were unaware of the existing task pre-classification and were prompted to classify the eight tasks according to (a) which tasks they considered to be problem solving tasks (cf. RQ1) and (b) which tasks they would be more likely to use in their classrooms (cf. RQ2). Each of the sorting prompts was followed by a more elaborate focus group discussion about teachers' task sorting choices.

The data analysis followed an inductive data-driven approach. Incidents in the data demonstrating task features that were associated by teachers to problem solving tasks (cf. RQ1) were grouped together to form codes and clusters. Based on the constant comparative method [START_REF] Glaser | The constant comparative method of qualitative analysis[END_REF], the coding scheme was developed over three levels of analysis. First, incidents in the data were compared to each other. More specifically, three different focus group transcripts were chosen representing low, high and middle socio-cultural levels. Incidents in those three transcripts were cross-compared and clustered to create a preliminary coding scheme representative of all three sets of data. Secondly, the created code was compared to another set of incidents, represented by three other focus group transcripts also varying across socio-cultural standings. Resulting from that was a more elaborate coding scheme. Finally, created codes were compared against each other, yielding a saturated format of codes and clusters. This saturated code was then utilized for the frequency count analysis of all focus group transcripts. Occurrences in the transcripts that aligned with the codes in the saturated coding scheme were counted in order to trace the popularity of certain task features. Following the same method, a separate set of codes was created to represent task features that teachers perceived to be suitable for their local classroom practices (cf. RQ2) given the constraints of the national curriculum framework. Contrasting the findings obtained from these two sets of codes cast some light on the power tensions between teacher perceptions and the contextual reality.

Findings

Results of the analysis revealed the following task features to be most popularly associated by teachers with problem solving tasks:

Nature of the task: The task layout has to be relevant to students' everyday lives. Also, the task needs to be approachable by a range of different methods, thereby giving space for a wide audience of learners to address the task, each in their own way.

Teaching the task: Students need to independently tackle the task, thereby exploring new and often unfamiliar territories. However, the teacher needs to be in tight control of that exploration process.

Learning from the task: In order to solve the task, students would need to go through an inductive process, to develop a yet unknown formula or solution procedure, that can then be adopted to find the solution to the problem. This creative process of invention was said to be cognitively demanding and was argued to naturally trigger student enjoyment.

These results show a high degree of alignment of teacher perceptions of problem solving task features with [START_REF] Stein | Implementing standardsbased mathematics instruction: A casebook for professional development[END_REF] conceptualization of problem solving tasks (referred to as "doing mathematics tasks" in their classification). Indeed, teachers seem to agree with the literature about the knowledge creation process that ought to be triggered in students' minds as they engage in problem solving tasks.

When asked about the suitability of adopting such tasks in the classroom, teachers admitted that these tasks would not naturally align with the national curriculum agenda. Yet, they also made it clear that they would favor adopting this type of tasks over the tasks that fulfil the national curriculum textbook requirements. The teachers described the latter kind of tasks as follows:

Nature of the task: The task layout has to be tangible and relevant to students' everyday lives. Its formulation needs to be straightforward and needs to align with the formulation of expected examination questions.

Teaching the task: The teacher presents to the students the task solution process. This is often coupled with a more elaborate explanation of the concepts that relate to that solution process. Students are encouraged to follow teacher explanations and ask questions.

Learning from the task: The task needs to be quickly and easily solvable by an average student.

When contrasting the two descriptions of task features, we see a gap between (1) the kind of tasks that align with the curriculum requirements and that the teachers perceived to be suitable for local adoption and (2) the kind of tasks that the teachers perceived to be problem solving tasks and would rather enact in their classrooms if they were given the choice. As illustrated in Figure 2, there seems to be a variation between the ideology that the curriculum dictates and the mathematical ideology that would rather be implemented by the teachers and that in their view relates to problem solving. In their focus groups, teachers commented about them being governed by a one-size-fits-all curriculum model that does not align with their own views about mathematics. They also argued to be bound by a narrow time schedule that is dictated by the national curriculum requirements which results in them mostly having to enact a rather more procedure-based [START_REF] Stein | Implementing standardsbased mathematics instruction: A casebook for professional development[END_REF] mathematics view in their teaching.

Nevertheless, the discussion still revealed traces of teachers attempting to enact their own mathematical ideology in the classroom despite being bound by a different curriculum ideology. Teachers expressed their belief in the importance and relevance of integrating into their teaching the tasks that they defined to be problem solving tasks. Being foreign to the national curriculum, it would be difficult to integrate this type of tasks as a student-led activity. Nevertheless, teachers claimed that those tasks could still be useful as a pedagogical tool to introduce students to new mathematical concepts and to involve them in the development of formulae and procedures in relation to those. In that way, the teachers' definition of a problem solving task experience would be partially fulfilled. The task would trigger knowledge construction, yet it would only serve as a teacher explanation tool.

Discussion

In response to the research questions about teacher perception and enactment of problem solving within the context of the mathematics national curriculum, the following can be stated:

(1) Teacher perceptions about mathematical problem solving seem to align more with the ideology that is found in literature [START_REF] Stein | Implementing standardsbased mathematics instruction: A casebook for professional development[END_REF] rather than with the ideology that is dictated by the curriculum [START_REF] Naguib | The production and reproduction of culture in Egyptian schools[END_REF][START_REF] Sika | Educational reform in Egyptian primary schools since the 1990s: A study of the political values and behavior of sixth grade students[END_REF]. (2) In view of curriculum enactment, the ideology dictated by the national curriculum seems to have superiority on classroom enactment choices when compared to teachers' ideological preferences. Nevertheless, there is some evidence of a partial realization of problem solving task integration in the classroom, which aligns more with teachers' preferred ideology.

These results re-emphasize the indicated gap between the inner and outer circles of culture and contextual influence. In line with the plotting at the onset of this paper, the historical roots of cultural migration that led to the formation of a different cultural footprint of foreign private schooling in Egypt seem to still have an impact on teacher belief formation and desired curriculum enactment practices. As the results indicate, teachers seem to be more oriented towards inquiryorientated curriculum enactment practices. Despite the fact that the mathematics curriculum in Egypt is meant to be centralized around schemes dictated by the MOE and that its enactment is meant to be unified across all national Egyptian schooling types, the investigation of the actual enactment seems to offer more variety and complexity. Findings of this study indicate some traces that re-emphasize [START_REF] Alexander | Culture and pedagogy: International comparisons in primary education[END_REF] views of schools being microcultures transferring their own cultural flavor. As the findings indicate, teachers seem to find it important to integrate problem solving tasks into their daily instruction, even if those do not align with requirements of the national curriculum. Results also indicate an apparent power dynamic that gives superiority to the influence of the national curriculum ideology, resulting in teachers' mathematical ideologies being trapped and only being partially realized in terms of practical enactment in the classroom. This superiority seems to only influence the practical enactment, yet it seems not to affect the formation of teacher beliefs about mathematical problem solving. Findings reported in Figure 2 also indicate a disparity between the national curriculum's views of teaching and those of learning mathematics. While teaching mathematics seems to be associated with the teaching of structures, the learning of mathematics was associated by teachers as a routine process of recalling knowledge. According to [START_REF] Ernest | The philosophy of mathematics education[END_REF] framework there seems to be a mismatch between both views that is worth investigating in future research.

Figure 1 :

 1 Figure 1: Mathematical ideologies: Three perspectives (adapted from Ernest, 1995)

  Figure 2 maps our classification of the two aforementioned task types on to the framework developed in Figure 1. The symbols (x) and (*) have been used to mark the position of the first and second kind of tasks, respectively.

Figure 2 .

 2 Figure 2. Mapping of findings


Juuso Henrik Nieminen
email: juuso.h.nieminen@helsinki.fi

Discourse of otherness in a Universally Designed undergraduate mathematics course

Keywords: Discourse analysis, otherness, special needs, university mathematics, Universal Design

To meet the needs of student diversity, universities have often implemented the principles of Universal Design in their instruction. However, previous literature knows little about the effectivity of these practices. In this study, two students with special needs are given a voice to share their experiences about a Universally Designed undergraduate mathematics course. The study uses the view of a Foucauldian discourse analysis in order to investigate how these students construct discourses on their position in the context of the course. According to the results, the students positioned themselves as the Others, constructing a discourse of "different learners". The study implicates that there is a need to shift the existing discourses to open up alternative subjectivities.

Little is known about disabled students' perceptions of their studies in the field of postsecondary mathematics. Indeed, disabled students are often marginalized in mathematics educational research. These students are rarely seen as mathematical thinkers and doers rather than focusing on cognitive deficits (for a review see [START_REF] Tan | Calling for research collaborations and the use of dis/ability studies in mathematics education[END_REF]. What is known about disabled mathematics students is that they are not encouraged in their studies [START_REF] Feigenbaum | Algebra for students with learning disabilities[END_REF] and that university staff might also be reluctant to change their teaching methods in order to take these students into account [START_REF] Thurston | Postsecondary STEM education for students with disabilities: Lessons learned from a decade of NSF funding[END_REF]. It is also known that especially dyslexic students are shown to struggle in studying tertiary mathematics because of environmental barriers for learning [START_REF] Perkin | The dyslexic student and mathematics in higher education[END_REF]. In general, it is known that disabled students often struggle while participating in higher education. These students are less likely than others to persist until completing their educational program [START_REF] Mamiseishvili | Students with disabilities at 2-year institutions in the United States: Factors related to success[END_REF]. According to [START_REF] Weeber | Disability community leaders' disability identity development: A journey of integration and expansion[END_REF] students might see their disability status as an undesirable factor in how they perceive themselves as the campus environment does not provide enough support. In a Finnish report, it was found that accessibility was rarely thought about when learning environments were designed [START_REF] Laaksonen | Esteetön opiskelu yliopistoissa[END_REF]. However, in this study it was the staff who assessed the level of accessibility, and the disabled students themselves were never given a voice. I argue that there is a critical silence, in terms of [START_REF] Seale | E-learning and disability in higher education: accessibility research and practice[END_REF], when it comes to researching disabled students in the field of postsecondary mathematics.

Background of the study

Discourse and power as theoretical frameworks

This article draws on Foucauldian discourse analysis. Discourse analysis is not used simply as a research method; it serves as a broad theoretical framework for understanding discourse and its importance in social life. According to Foucault, discourses are "practices that systematically form the objects of which they speak … [they] are not about objects, they constitute them and in the practice of doing so conceal their own invention" (Foucault, 1977, p. 49). Discourses, therefore, produce normality by constructing what can be taken as granted. By studying discourse, we can gather information on social organizations and identities, since they are constructed by rules about what is normal. The way language is used shapes reality. Observing the ways in which social constructions form our world are the core of discourse analysis and form the theoretical base for this study.

Discourses produce meaning and knowledge, and these [START_REF] Foucault | Discipline and punish: The birth of the prison[END_REF] connected with power. [START_REF] Foucault | Discipline and punish: The birth of the prison[END_REF] proposed that in modern societies, we must move forward from analyzing dyadic relations of a ruler and a subject, and instead focus on finding these power relations in institutions that are so often labeled as 'humane'. He defined the concept of disciplinary power, a form of power that is understood to be the body of knowledge and those discourses takes as the 'truth'. According to [START_REF] Foucault | The subject and power[END_REF], personal identities are only produced through institutional or societal power and knowledge. In other words, subjectivity is constructed through the productive power of discursive practices. Various scholars have also emphasized the socially constructed nature of mathematical identityhow discourses construct mathematical identities and agency [START_REF] Alderton | Teaching mathematics to lower attainers: Dilemmas and discourses[END_REF]. Because discourses truly produce our social reality, they are all about power. Some voices are heard as meaningful and authoritative, while some others are not. When power and knowledge are combined, hegemonic discourses are constructed [START_REF] Dant | Knowledge, ideology and discourse[END_REF]. By this, Dant means that certain socially constructed discourses are considered as facts.

Universal Design

Inclusive pedagogy has often been introduced as a way to promote positive discourses of students with special needs by promoting a view of disability as a social construction. While the traditional, medical model views disability as an abnormality that has to be cured and assisted, the social model emphasizes the idea that disability is constructed when the environment cannot account to the needs of the disabled [START_REF] Seale | E-learning and disability in higher education: accessibility research and practice[END_REF]. The social model is often fostered in higher education by using Universal Design (UD), an environmental design that is accessible for everyone (Burgstahler, 2015). According to Burgstahler, the learning environments create barriers for learning, and UD aims to reduce these by taking into account the needs of the diversity of students. Often, UD is fostered by using the guidelines by the Center for Applied Special Technology: 1) engagement 2) representation 3) action & expression (CAST, 2018). These guidelines are tied to brain research on how people learn. Another common and practical way to enhance inclusion in higher education is to implement the Principles of UD of Instruction (UDI) into learning environments: 1) equitable use 2) flexibility in use 3) simple and intuitive 4) perceptible information 5) tolerance for error 6) low physical effort 7) size and space for approach and use (Burgstahler, 2015). Often, two additional principles are used: 8) community of learners and 9) instructional climate [START_REF] Shaw | Teaching college students with learning disabilities[END_REF].

Althought UD has been used as a base for legislation (e.g., the USA; Higher Education Opportunity Act 2008), we know surprisingly little about whether it works in higher education (for a review, see [START_REF] Roberts | Universal design for instruction in postsecondary education: A systematic review of empirically based articles[END_REF]. The warm-hearted idea of supporting inclusion through UD is not to be judged, but there is a need for more research-based evidence. Also, the voice of disabled students themselves is usually not heard in the research on UD in higher education [START_REF] Seale | E-learning and disability in higher education: accessibility research and practice[END_REF][START_REF] Griful-Freixenet | Higher education students with disabilities speaking out: Perceived barriers and opportunities of the universal design for learning framework[END_REF]. Griful-Freixenet and colleagues underline the fact that it is impossible to "tackle the need of all learners" and calls for a need of responsiveness. I argue that there is need for studies that would, at the same time, break the silence surrounding UD and also examine the power structures surrounding it. This study examines whether the framework of UD, aimed to promote the socially constructed view of disability, was able to promote discourses that would see students with special needs as an equal part of the learning community.

Aims of the study

In this study, the power relations in a Universally Designed university mathematics course are brought into light through students' own voices. The objective of this study is to examine the subjective positions that the students with special needs have constructed during the course.

Methodology Context of the study

The study was conducted on a proof-based undergraduate mathematics course with over 400 participants that lasted for 7 weeks. The course arrangements reflected the principles of UD: All of the three principles of CAST (2018) and the nine principles of Burgstahler (2015) and [START_REF] Shaw | Teaching college students with learning disabilities[END_REF] were implemented in the course examined in this study. Each week, students were given a set of problems to solve, both pen-and-paper and digital. After that, they received various feedback on the tasks (peer feedback, automatic feedback, feedback from the teacher and the teacher assistants) and participated in mathematical conversation during active lectures. There was no exam, and the final course grade was self-assessed. Table 1 shows an overview of the course components.

Engagement Representation Action and learning

Applied tasks relevant to different majors

Accepting and supportive classroom climate 8, 9 Active lectures based on discussion; minimizing the Course material been developed for years to be simple and intuitive , 2

Materials in Finnish and in Swedish 2

Mathematical discussion was one of the learning Various options for returning the tasks (LaTex, drawing...) 2

Digital WhatsApp groups for group discussion 8,9 Anonymous discussion forum 8, 9 Digital GeoGebra tasks social demands 7, 8 Learning objective matrix Formative assessment used to value process and effort 5

An open learning space with tutor students offering support when needed 2, 4, 6, 7 Peer-assessment 5

Formative and summative self-assessment 5

Flipped learning approach 5, 8 objectives 8, 9 Concept maps about the relationship of concepts A guide for producing clear mathematical text was provided 3

Dynamic geometry (GeoGebra) offering multiple representations

Progressive information releasing; first simple, then more advanced tasks Digital polls during lectures 8, Digital Moodle environment as a "base" for the course 1, 3 Social interaction in the open learning space; students are guided to tutor each others 5, 8, 9 Personal goal-setting through formative self-assessment

Various kinds of digital feedback; automatic, from tutors 5, 8, 9 Octave software for computing

Participants

All of the participants of the course were emailed an advertisement about the study. Six students, self-reporting as having some kinds of special needs during the course, were interviewed. Because of the page limit, only two of these six interviews were used in this study, the ones of Jesse and Nico; these two interviews were chosen as representative examples of the phenomenon. Jesse, aged 20, had been studying for two years and self-reported having difficulties with dyslexia. Nico, aged 23, was also studying in their second year and stated to have been diagnosed with ADHD. Both of the participants could explain as much about their diagnosis as they wanted to, and their choice was respected. Self-report about the diagnosis was needed; no diagnostic documents were collected for this study. Jesse and Nico were both majoring in mathematics or related science. Both of the names are gender-neutral pseudonymsgender of the participants is not reported in order to provide anonymization. Gender neutral pronouns they/them are used throughout the text. "Special needs" is the term used in this study to describe the participants. The purpose is not to label these students as one group and contribute in constructing a discourse of "students with special needs" as one subject.

Instead, this label is used to underline that the students, usually neglected in studies, are given a voice here [START_REF] Seale | E-learning and disability in higher education: accessibility research and practice[END_REF], even though mediated by the researcher.

Data collection and analytical process

Data was collected after the course using semi-structured interviews, with the questions concerning perceptions of the course components. The students were able to give feedback on the course arrangements and therefore participate in the development process of the course.

The first cycle of analysis consisted of careful familiarization with the data. The transcripts of the interviews were read through multiple times to gather a general understanding of the discourses of the students. The focus on the first cycle was on the position that the students created for themselves. Foucaldian concepts of discourse and power were used in order to unpack the ways students described themselves as participants of the course. This first part of the analysis was led by the question "Do the students position themselves as included or excluded in the learning community, and how is that inclusion or exclusion created?". The analysis was guided by identifying hegemonic discourses.

The second analysis changed the view towards examining how the students constructed the discourses on their subjective position and how they contributed in building hegemonic discourses.

The data was read through the lens on simplifications: When did the students choose to rely on simplifications over complexity and contradictions? How were these simplifications made seem convincing [START_REF] Dant | Knowledge, ideology and discourse[END_REF]? The concept of naturalization [START_REF] Fairclough | Language and power[END_REF] was used in the analysis as an ultimate form of simplification; this kind of a process happens when socially constructed discourses and practices are taken as natural and are even connected to the nature itself. The analysis of simplifications reconstituted as follows: when do the students see social orders arranged by nature and not by people, and how are these discourses justified? During the analysis it was noted that there is a need for an analytical tool that would frame the asymmetrical power constructed through naturalization. This is why the theory of otherness was implemented; as a part of the reliability of this study, I describe this process as it chronologically happened in my findings.

Findings

The first analysis cycle: The discourse of 'a different kind of a learner'

The first notice, after getting familiar with the data, was that both of the students described a very lonely learning process. This loneliness was connected with the difficulties the students faced in this course. An evident part of this loneliness was that throughout the data, the students spoke almost entirely in the first-person point of view as is evident through all the citations used in this article. In the entire data, there were no references to us.

Jesse:

When I am doing an exercise, the exercise number one. Everyone else has started that exercise at the same time as me but they are already doing the task number five when I'm moving forward to task number two. I just progress so much slower. --I realized a long time ago that for me it is better to work on my own.

The clear, choiceless segregations made between me and them were seen as building blocks of the discourse about difference. The discourse was named by the researcher as 'the discourse of a different kind of a learner' after Nico's feedback on the course:

Nico: I think it is good if there is a will to map out the needs of different kinds of learners.

One way of constructing difference between these students and the others was the discourse of different kinds of study methods. The students did not want to lean on to the ready-made methods, built for them by the teacher. On the contrary, they created their own study strategies by themselves, because they could not make use of the normal ways of studying. This smaller discourse was seen as a legitimator for the stronger discourse of difference. Even these kinds of helper discourses might have some serious real life consequences.

Jesse:

Well. I have been to small groups or so. Where you need to do group work with other people. These kinds of situations have always been very bad for me since. I mean, goodness. You have to write very fast, just whatever comes to your head. And the answers to the tasks, they are what they are.

Nico: I tried to study in a way everyone else is studying and. In the end of the year I realized that I wasn't even getting grades from my courses. Now then, I built this learning strategy. So, maybe I don't learn like other people, so I'll just have to learn completely in my own way. --I was using a timer with these exercises. For example, I decided that at 5 pm I'll start doing these exercises.

The discourse of difference was also identified when the students gave feedback on the course. This was most clear with Nico, who told that they almost did not use any of the social course components, yet still they praised the existence of these arrangements. They told that they did not attend the lectures nor the open learning space, and neither wanted to take part on the digital discussion groups of the course. According to Nico, there was nothing wrong with the course components, designed with the principles of UD; they were just meant for someone else.

Nico: I would say that compared to last year. Well, this is. A hundred times better arranged, this course. So, keep going like this and it'll be very good.

As shown, there was a hegemonic discourse of 'a different kind of a learner', a strong discourse that was obviously constructed way before attending this course. But how was this kind of a hegemony legitimised? For further analysis, a new tool was needed: the concept of otherness.

A plunge back to the theory: Otherness and power

Individual subjects construct their reality and identity under the supervisory of normalizing practices that impose homogeneity and build discourses on norms [START_REF] Foucault | Discipline and punish: The birth of the prison[END_REF]. This leads to monitoring one's behavior, a process that is constantly judged. [START_REF] Fenwick | The 'good' teacher in a neo-liberal risk society: A Foucaultian analysis of professional growth plans[END_REF] uses the concept of internalisation to describe the process where identities are produced through following norms, thus leading to being dependent on the disciplinary power the subjects have directed on themselves.

When norms produce inclusion and exclusion through disciplinary power, they produce otherness. [START_REF] Staszak | Other/otherness[END_REF] conceptualizes otherness as a discursive process which is produced by a dominant group by stigmatizing a difference. The creation of otherness thus constructs two different group: them and us. Asymmetry of power is the key feature in constructing otherness: Only the more powerful group of them and us is able to maintain hegemonies and construct their own identity, while the identity of the other is only constructed in a relation to these powerful discourses. Discipline power is found when the others define themselves in relation to the dominant group, not having equal power to define themselves by their own means. When otherness is seen in a negative way through dominant discourse, the results tend to be very real and reflect the power structures [START_REF] Okolie | Introduction to the special issue. Identity: Now you don't see it; now you do[END_REF].

The second analysis cycle: The discourse of otherness

Because of the asymmetrical power structure that was found at the first cycle of analysis, the 'discourse of a different kind of a learner' was renamed as the 'discourse of otherness'. It was kept up with simplifications; strong discourses that left no room for seeing things from different angles. Internalisation of norms created otherness when the students saw their differences as something that could not be taken account by the course or the university staff. The students positioned themselves as the others that cannot and therefore should not be helped by the course. This is a strong hegemonic discourse: This learning environment of university mathematics is not meant for them, even if it is constructed with the principles of UD. This is an obvious simplification, since the learning environments could, of course, be adjusted for these students too.

Jesse Naturalization was the key element in constructing otherness. This kind of a process was most evident in Nico's interview. Nico referred to themselves as an 'ADHD person' and constructed otherness through neurology. Also, Jesse made references of dyslexia as a biological state that prevented them of studying in a normal way. These kinds of discourses can be seen as ultimate ways of asymmetrical power between them and me.

Jesse:

Well. Always, it would be nice to get. Some kind of an assistance. When it comes to dyslexia. Which means more hours to a day, but that just isn't possible.

Discussion

This study examined how students with special needs position themselves in a Universally Designed undergraduate mathematics course. Even though the theories of UD and UDI were reflected in the course design, it was found that the two students interviewed constructed a discourse of otherness [START_REF] Staszak | Other/otherness[END_REF][START_REF] Okolie | Introduction to the special issue. Identity: Now you don't see it; now you do[END_REF]. Asymmetrical power structures were found in the way they distanced themselves from the other students.

Mathematical identities are known to be formed through institutional power. As educators and researchers, we need to see our responsibility in how we create discoursesor silence. There is a growing need to admit there are students with special needs in undergraduate mathematics, both in research and practice. What we can do is to transform the experiences and possibilities of students with special needs [START_REF] Alderton | Teaching mathematics to lower attainers: Dilemmas and discourses[END_REF]. This study showed how naturalization helped to build the discourse of otherness. If we normalize this kind of inequity, we are constructing a hegemonic discourse that legitimizes subordination through asymmetrical disciplinary power.

UD offers a way to change our view towards more equal mathematics teaching in higher education by challenging the medical model of disability. However, implementing the principles of UD into a course is not enough if discourses of otherness in mathematics are not shifted. The study suggests that that designing inclusive environments for disabled students is not enough, since, in the end, that would reflect the medical model of disability and not social. [START_REF] Seale | E-learning and disability in higher education: accessibility research and practice[END_REF] sees participation, both in research and practice, as a way to truly foster inclusion. Maybe this would be a way to shift the discourse of otherness and truly include disabled students in the mathematical learning environments. In university mathematics, the first thing to do would be to listen to these students.

Table 1 : The course components through the lense of the principles of UD (CAST, 2018), linked with the elements of UDI (with superscript; Burgstahler, 2015; Shaw, Scott, & McGuire, 2001)

 1

  :It would have been nice if I would have bought the physical copy. And not just read the material from the screen of my computer, since there is that backlight. I always become so confused about the line I'm reading. --It costs money. So no thank you.

	Nico:	I would say that if you are a different kind of a learner, then the most important
		thing you should do immediately after you start your university studies is to find
		out how you learn.


Arnaud Sallaberry
email: asallaberry@ucdavis.edu

Davis Kwan-Liu

Visualizing InfoVis Reaserchers with ContactTrees

Keywords: Computer Graphics [I, 3, 3]: Picture/Image Generation-

This poster presents an application of ContactTree, a new egocentered visualization design, to charactering the collaborative activities of selected InfoVis researchers in terms of their publications as listed in DBLP. The ContactTree visualization, based on a botanical tree metaphor, is designed for studying individuals and comparing their social behavior and relationships with others based on data with rich attributes. As shown, the resulting trees give each of the selected researchers a distinct look, and many interesting patterns reveal themselves. In social science research, the study of people's social contacts and activities is of high interest. ContactTree visualization well complements conventional network visualization, which is better for showing relations among people and activities in a global context.

INTRODUCTION

The analysis of social networks, which are composed of people (actors) and their relationships (ties), has been the subject of a very dynamic field of study for decades [START_REF] Scott | Social Network Analysis: A Handbook[END_REF]. Nowadays, the development of new information technologies helps to collect large data sets to discover many properties related to the formation of social web or the development of interpersonal relationships. Besides the scientific interest of such studies, the recent success of online social networks, personal such as Facebook as well as professionals such as LinkedIn, shows how the masses are concerned with creating and maintaining their own network influence. First reserved to sociologists, the study of social networks has quickly attracted other disciplines. The emergence of methods and tools devoted to social network analysis and visualization (e.g [1,[START_REF] Heer | Vizster: visualizing online social networks[END_REF][START_REF] Henry | Exploring Large Social Networks with Matrix-Based Representations[END_REF][START_REF] Perer | Integrating statistics and visualization for exploratory power: From long-term case studies to design guidelines[END_REF]) shows the popularity of InfoVis approaches. Most of these methods consider social networks as a whole relying on a global or complete network approach. As comprehensive as it can be, the global approach fails to capture some aspects about social relationships and the study and comparison of the ego-centered social circle representations allows one to discover trends that global representations fail to highlight [START_REF] Kadushin | Understanding Social Networks: Theories, Concepts, and Findings[END_REF].

ContactTree is a new ego-centered visualization design that helps to access data, compare persons' interpersonal relationships and make hypothesis about patterns or trends. The design is based on a botanical tree metaphor. The main idea is to use the features of a tree (the structure of its branches, leaves, fruits, colors, etc.) to map properties of one's relationships. This design fits well to visualize many aspects of one's social circle. We present here an example involving the collaboration network from 2000 to 2011 of some well-known researchers in information visualization. A technical report including a more complete description of the design and several case studies is also available [START_REF] Sallaberry | ContactTrees: Egocentered visualization of social relations[END_REF].

DESIGN DESCRIPTION

A tree can be seen as a structure composed by a trunk, which is divided into several parts to form main branches, themselves divided into smaller ones, etc. Starting from this observation, an intuitive approach consists in representing ties as the smallest branches. According to this, a small branch holds exclusive properties of a tie, while bigger branches represent common properties of the ties represented by the small branches growing up from them. Finally, the aspect of the trunk represents the whole relationships.

As the set of examples proposed in this poster is based on a coauthorship network, each tree focuses on an InfoVis researcher and the ties are his co-authors. The starting point of main branches growing up from the trunk can be characterized by two parameters: its position along the trunk and the side of the trunk it is on. Then, we can map on it two properties. The first one is on an ordinal scale (position along the trunk) and the second one is a boolean (right or left side of the trunk). In our examples, co-authors lying on the right side of the tree are persons presently (i.e. 2011) working in the same country as researchers while those lying on the left side work in another country (see Figure 1). Y-position of a main branch along the trunk represents the year of the first collaboration during the period, from 2000 (lowest branch) to 2011 (highest branch). These two properties characterize the set of ties represented by the main branches. In the same way, the position along the main branch and the side (above/bellow) where a smallest branch is starting from can be used to represent properties (one ordinal and one boolean). In our examples, positions are not used while small branches growing above (resp. below) main branches represent co-authors from industry (resp. public institutes).

Proceeding the botanical metaphor, leaves need to be added to these branches. They can represent two kinds of properties of ties. Each leaf also holds four features we can map a property on: color, size, position along the branch, side of the branch they are growing from. In our example, each leaf is an article published between 2000 and 2011 and written by the researcher and his co-author represented by the small branch it is growing from (see Figure 1). The size of a leaf is proportional to the number of pages of the article and its darkness represents the number of authors who have collaborated on it.

RESULTS

Figure 2 shows the ContactTrees of four U.S.A researchers in Information Visualization: Ben Shneiderman, Jeffrey Heer, John Stasko and Kwan-Liu Ma. These ContactTrees are very unbalanced: most co-authors lie on the right side of the trunk and on the lower side of the main branches. It means that most collaborators are academic researchers working in the same country as the researcher represented by the tree. This trend is not always true. For example, we see that Ben Schneiderman had many collaborators from industry during the year 2009. This "anomaly" is mainly due to 2 articles involving many industrial researchers. Another important collaborator who is pointed out by our visualization is Catherine Plaisant. She is represented by the long small branch growing from the lowest main branch. She has written more than 30 papers with Ben Shneiderman during the given period. One last interesting remark on the ContactTrees of Figure 2 is related to the distribution of new ties over the years. We see, for example, that in 2004 and 2005, Jeffrey Heer started to work with two new collaborators (Danah Boyd and Maneesh Agrawala, with whom he has published 12 papers). In contrast, the number of his co-authors sharply increased in 2011 with 32 new ties (with up to three papers for one of them). It would be interesting to see the evolution of these collaborations in the future. Maybe 2011 is an important step for Jeffrey Heer's career. Let's also have a look at two other well-known researchers from Europe: Jark van Wijk, from Netherlands, and Jean-Daniel Fekete, from France (Figure 3(a,b)). As for the previous examples, we see that most of their co-authors come from public institutions. Their ContactTrees are much more balanced regarding the country of origin of their ties. This is likely due to the smaller number of team devoted to Information Visualization in their respective countries. However, this observation must be tempered. We see that most of foreign co-authors, as for U.S.A researchers, have been involved in only one or two papers. So, these kind of collaborations are weaker.

CONCLUSION

Our ContactTrees allow people to map ties and contacts based on various priorities and preferences. Facing overly sophisticated network data, whether they are taken from contact diaries, citation records, survey archives, or online social media, researchers and users would benefit greatly by first looking at the key features of a tree. These features capture not only "ties", the underlying unit of analysis in most social network analyses, but also "contacts" among ties. The addition of this key property distinguishes our design from the majority of visualization tools for social networks. This work is performed by InfoVis researchers in collaboration with sociologists. We're currently working on a second dataset, provided by the sociologists, and including individual ties and their contacts for 2 waves of 3 months. While many properties are related to the individuals, their ties and their contacts, more features of the tree are used, including fruits and even birds. We're also exploring different mappings of properties with the same dataset. The technical report [START_REF] Sallaberry | ContactTrees: Egocentered visualization of social relations[END_REF] presents the current results.

Figure 1 :

 1 Figure 1: Features of the tree we map properties on.

Figure 2 :

 2 Figure 2: Four U.S.A researchers: (a) Ben Shneiderman, (b) Jeffrey Heer, (c) John Stasko and (d) Kwan-Liu Ma.

Figure 3 (

 3 c,d) shows two more recent researchers in Information Visualization: Nathalie Henry Riche and Tim Dwyer. They both work in the industry. Looking at the side of the main branches where the small branches start, we see that Nathalie Henry Riche (resp. Tim Dwyer) mostly work with industrial partners since 2008 (resp. 2010). This is because they have joined Microsoft Research in 2008 while they were previously in a public institution. Another interesting aspect of their career is shown by the side of the trunk their ties lie on. Nathalie Henry Riche started her career in France, Tim Dwyer in Australia. They both now work in U.S.A. Looking at Tim Dwyer's ContactTree, we clearly see that after 2010, most of his new collaborators are in U.S.A. This phenomenon is less salient for Nathalie Henry Riche. We see that she is still having a lot of new ties from another country.

Figure 3 :

 3 Figure 3: Two European researchers: (a) Jark van Wijk and (b) Jean-Daniel Fekete. Two more recent researchers: (c) Nathalie Henry Riche and (d) Tim Dwyer, both work in a U.S.A company and both come from another country


Charlotte Rochereau
email: charlotte.rochereau@gmail.com

Benoît Sagot
email: benoit.sagot@inria.fr

Emmanuel Dupoux
email: emmanuel.dupoux@gmail.com

Modeling German Verb Argument Structures: LSTMs vs. Humans

LSTMs have proven very successful at language modeling. However, it remains unclear to what extent they are able to capture complex morphosyntactic structures. In this paper, we examine whether LSTMs are sensitive to verb argument structures. We introduce a German grammaticality dataset in which ungrammatical sentences are constructed by manipulating case assignments (eg substituting nominative by accusative or dative). We find that LSTMs are better than chance in detecting incorrect argument structures and slightly worse than humans tested on the same dataset. Surprisingly, LSTMs are contaminated by heuristics not found in humans like a preference toward nominative noun phrases. In other respects they show human-similar results like biases for particular orders of case assignments.

Introduction

Among neural networks, LSTMs [START_REF] Hochreiter | Long short-term memory[END_REF] are commonly used for language modeling. Although new architectures [START_REF] Al-Rfou | Character-level language modeling with deeper self-attention[END_REF][START_REF] Dai | Transformer-xl: Attentive language models beyond a fixed-length context[END_REF] challenge this standard, LSTMs remain competitive for language modeling [START_REF] Melis | On the state of the art of evaluation in neural language models[END_REF]. However, despite the success of LM LSTMs, it is not clear what makes them so effective. In particular, are representations derived through language modeling able to effectively encode syntactic structures and relations? Do they encode them in a reliable and systematic way?

The typical metric used to compare LMs, perplexity, is not adapted to address these questions. Perplexity measures the probability assigned to held-out data from the corpus the LM is trained on. Because the held-out and training data are typically randomly extracted from an initial corpus, they have similar statistics, which is good from a machine learning viewpoint, but bad from the viewpoint of linguistic analysis: perplexity is mostly sensitive to the most common sentence types in the initial corpus and therefore will not reflect well the behavior of the LM in the tail of the distribution. In addition, the sentences extracted from a natural corpus confound several factors: syntax, semantics, pragmatics, etc. further complicating the interpretation of a good perplexity score.

To circumvent this limitation, recent work has focused on using probing techniques inspired by linguistic and psycholinguistics (for instance, grammaticality or acceptability judgments, or forced choice). In addition, instead of using sentences from the training corpus, studies rely more and more on automatically constructed test sentences, which enable for a removal of the bias in the original corpus and focus on particular linguistic phenomena. Here, we will use acceptability judgments operationalized by the log probability of sentences according to the LM and sets of synthetic sentences generated from template sentences to probe for a challenging linguistic structure: verb argument structure.

Verb argument structure provides languages with a way to link syntactic position in a sentence (subject, direct object, etc) with semantic roles (agent, patient, etc), in other words, to determine who is doing what. It is currently unknown whether neural LMs purely trained from surface statistics are able to capture this kind of structure, or whether additional information from another modality would be needed to provide some semantic grounding.

Verb argument structure is typically correlated to sentence position in many languages like English. But in other languages with relatively free word order, it is indicated by morphological markers. Here, we study German, where the arguments of a verb can occur in any position (when occurring within a relative clause), and is indicated by the case of the noun phrase (nominative, accusative, etc).

We setup a test of argument structure representation by presenting a trained LM with carefully constructed sets of sentences that either have the right set of arguments, or abnormal sentences where one case is missing or duplicated. We use word order permutations to control for unigram and positional statistics. If the LM is able to track argument structure irrespective of word order, it should assign lower grammaticality scores (log probabilities) to the incorrect sentences as compared to the correct ones.

Since at the level of the sentence, we study a global rather than local syntactic phenomenon, we depart from earlier work [START_REF] Linzen | Assessing the ability of LSTMs to learn syntax-sensitive dependencies[END_REF][START_REF] Gulordava | Colorless green recurrent networks dream hierarchically[END_REF][START_REF] Marvin | Targeted syntactic evaluation of language models[END_REF][START_REF] Tran | The importance of being recurrent for modeling hierarchical structure[END_REF] and do not compare pairs of sentences. Rather, we compare a set of valid grammatical variations of the template to a corresponding set of grammatical violations of the template. Thus, for each template, we measure the model's ability to discriminate grammatical sentences from ungrammatical ones using receiver operating characteristic curves, or ROC curves. We also compute the area under the ROC curve, or AUC. In our results, we often report the average AUC over templates as our metric.

We evaluate three LMs on our dataset, the two-layer LSTM of [START_REF] Hahn | Tabula nearly rasa: Probing the linguistic knowledge of character-level neural language models trained on unsegmented text[END_REF] trained on German Wikipedia text, as well as n-gram baselines using the same corpus. We ask proficient German speakers to annotate our sentences for grammaticality, providing a human comparison. Since some of these sentences are rather implausible because of the permutations, we also collect human meaningfulness scores. We find that our dataset is challenging for both LMs and humans and that LMs lag behind human performance.

Related work

Grammaticality judgments for recurrent networks have been investigated since [START_REF] Allen | The emer-gence of grammaticality in connectionist networks[END_REF], who use closely matched pairs of sentences to investigate grammatical correctness. This approach has been adopted recently to assess the abilities of RNNs, and LSTMs in particular, to capture syntactic structures. For instance, [START_REF] Linzen | Assessing the ability of LSTMs to learn syntax-sensitive dependencies[END_REF] and [START_REF] Gulordava | Colorless green recurrent networks dream hierarchically[END_REF] use word probes in minimally different pairs of English sentences to study number agreement. To discrimi-nate grammatical sentences from ungrammatical ones, they retrieve the probabilities of the possible morphological forms of a target word, given the probability of the previous words in the sentence. Practically, in the sentence "the boy is sleeping", the network has detected number agreement if P(w = is) > P(w = are). This methodology has also been adapted by [START_REF] Goldberg | Assessing bert's syntactic abilities[END_REF] to models trained with a masked language-modeling objective. Those works find that in the absence of many detractors or complex sentence features, recent language models perform well at the numberagreement problem in English.

More closely related to our work, [START_REF] Ravfogel | Can LSTM learn to capture agreement? the case of basque[END_REF] use word probes to examine whether LSTMs understand Basque agreement. Like German, Basque is a morpho-syntactically rich language with relatively free word order, thus providing a challenging setting for the LM. In contrast to our work, the LM's ability to understand verb argument structure is tested on number-agreement and on suffix recovery tasks, which involve localized changes rather than whole sentence perturbations and re-orderings.

In parallel to work focusing on word probe probabilities, another closely related line of inquiry has investigated surprisal, the inverse log probability assigned to a specific prediction by a model. For instance, [START_REF] Wilcox | What do RNN language models learn about filler-gap dependencies?[END_REF] and [START_REF] Futrell | Neural language models as psycholinguistic subjects: Representations of syntactic state[END_REF] examine many syntactic phenomena, including filler gap dependencies and garden path effects.

We depart from these approaches because our test set encompasses whole sentence variations, such as argument reordering. Word probes are therefore less apt to capture such changes. Instead, we choose to follow [START_REF] Marvin | Targeted syntactic evaluation of language models[END_REF] and [START_REF] Tran | The importance of being recurrent for modeling hierarchical structure[END_REF] in taking the more general approach of comparing whole sentence probabilities as our grammaticality probe. This method, which also corresponds to the sentencelevel LogProb acceptability measure of [START_REF] Jey | Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge[END_REF], evaluates whether the model assigns a higher log probability to sentences which are grammatical than to sentences which are not.

In contrast with approaches that seek to probe language models directly, other approaches involve fine-tuning representations to a specific syntactic task using a task-specific supervision signal. For instance, [START_REF] Warstadt | Neural network acceptability judgments[END_REF] introduce CoLA, a binary acceptability dataset whose example sentences are taken from linguistic publications. They train a classifier on top of frozen ELMo [START_REF] Peters | Deep contextualized word representations[END_REF] layers to assess performance at acceptability judgments. Later work [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Warstadt | Grammatical analysis of pretrained sentence encoders with acceptability judgments[END_REF] has focused on fine-tuning an entire pre-trained model to the acceptability task, such as is done for BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF]. Both of those paradigms do not directly evaluate syntactic ability but rather whether pre-trained representations can be effectively transferred to learn to solve specific syntax problems.

Verb Argument Structure Dataset Construction

Templates Our test sentences were automatically generated from fifty grammatical sentences which we call templates. These templates are all constructed the same way: the main clause "wir wissen, dass..." ("we know that"), followed by a subordinate clause with a subject (nominative case), a verb in the past tense form, a direct object (accusative case) and an indirect object (dative case). For simplicity purposes, we did not use any adjective. In the Template of Figure 1, "the minister" is the subject, "that bill" the direct object, and "the Senate" the indirect object of "announced".

We constructed a dataset designed to expose impossible verb argument structures by manipulating the arguments' case assignments. We introduced these changes within subordinate clauses rather than main clauses, because German subordinate clauses have a more flexible noun phrases order than main clauses. This specificity allows us to test whether models are able to capture syntactic dependencies when the arguments' positions vary.

In German, the syntactic role of noun phrases is indicated by the morphological form of its constituents: determiners and nouns take different suffixes, if not completely different forms, according to their case assignment. However, feminine, neutral and all plural noun phrases share common morphological forms. Thus, to avoid sentence duplicates within our dataset, all noun phrases are singular masculine.

Grammatical Sets

To control for all possible argument orders and words syntactic roles, for each template, we change (i) the positions of the three verb arguments in the subordinate clause and (ii) the case assignments of each noun group. There are three verb arguments, leading to six different position permutations. Similarly, they are three unique case assignments, leading to six possible case assignments. By generating all such permutations, we create 6 × 6 = 36 grammatical sentences for each template, yielding 1800 grammatical sentences in total. In Figure 1, we show an example where only the positions of the subject and the indirect object are switched, which does not alter the meaning. We also show an example where only the case assignments of the subject and the indirect object are switched: "The Senate" becomes the subject and "the minister" the indirect object. The case permutations were done by retrieving the desired case markings (nominative, accusative or dative) from a dictionary mapping the vocabulary's nouns to their morphological forms. Case permutations change sentence meaning. In practice, some of our sentences will be implausible yet grammatical, in contrast with [START_REF] Marvin | Targeted syntactic evaluation of language models[END_REF].

Case Violation Sets We constructed ungrammatical sentences using the same templates. Briefly, we substituted one of the case assignments by another one already present in the sentence, which creates a grammatical violation: sentences contain three noun phrases and only two case assignments, one being duplicated. In Figure 1, we show how we apply this to a template sentence to create grammatical violations.

For each case violation, we generated 36 sentences containing a case violation from every template. Thus, from each of our 50 templates, we generated 36 valid grammatical variations and 108 ungrammatical variations. Note also that throughout the number of words in our dataset stays constant (11 words per sentence), so that log probabilities are more comparable. Overall, our dataset comprises 7,200 sentences, of which 1,800 are grammatical and 5,400 are ungrammatical.

Methods

Human Evaluations

To generate human results for our dataset, we hire annotators proficient in German on Amazon Mechanical Turk. 1Sentence Grammaticality We asked Amazon Mechanical turkers to assess the sentence grammaticality on a scale from 1 to 10, where 1 means grammatically incorrect and 10 means grammatically correct. Before the task started, respondents were shown examples of grammatical sentences and ungrammatical sentences. Importantly, it was indicated that grammatical sentences were not necessarily meaningful. As an example, we translated to German Chomsky's famous quote: "Colorless green ideas sleep furiously" [START_REF] Chomsky | Syntactic structures[END_REF]. Each respondent graded 50 sentences, with the following constraints: (i) each sentence comes from a different template, to avoid that past sentences impact future ratings; (ii) twenty-five percent of the sentences shown are grammatical, mirroring the construction of the dataset; (iii) sentences selected are randomly chosen among the 144 possibilities for each template, so that each user is exposed to a wide variety of case assignments, argument orders and grammatical violations; (iv) no sentence is annotated twice.

Sentence Meaningfulness For grammatical sentences only, we also conduct meaningfulness evaluations. Similarly to our grammaticality experiment, users are asked to grade 50 sentences from 1 to 10, where 1 is meaningless and 10 is meaningful. They were also shown examples of meaningful and meaningless grammaticality correct German sentences before starting the evaluations. Constraints are the same as above, except that all sentences are grammatical and that there are thus only 36 possibilities per template.

Ensuring German Proficiency To ensure that all annotators are proficient in German, we took the following steps: (i) we only accepted annotators from German-speaking countries; (ii) instructions are given in German only; (iii) annotators took a short German grammar test on conjugation and declination knowledge; (iv) filler sentences (easy sentences for which answers are known and obvious to proficient German speakers) are inserted throughout the annotation process to ensure annotators stay focused; (v) we remove annotators who took less than a third of the average time to complete the assignment after checking that they also underperform on our test questions.

Pairwise Ranking and Individual Grading

As noted, we do not ask humans to compare minimally differing sentences, but rather to grade individual sentences. This setup differs from earlier work such as [START_REF] Marvin | Targeted syntactic evaluation of language models[END_REF] who show both sentences simultaneously and ask humans to pick the most grammatical one. This approach prevents humans from using the differences between the sentences to form a judgment on grammaticality; rather they must judge each sentence on its own. In doing so, the human setup is closer to that of language models: when we use log probability scores of LMs, we do not enable them to learn from the differences between the sentences to form a judgment.

In total, we collected 2,750 annotations from 55 annotators for sentence grammaticality (38% of the dataset) and 1,800 annotations from 36 annotators for sentence meaningfulness (100% of grammatical sentences). We do not have grammaticality annotations for all sentences due to a lack of proficient German annotators on Amazon Mechanical Turk. Our human results for grammaticality are computed on this subset of the dataset.

Language Models

We use the pre-trained word-level language model (German WordNLM) described and trained by [START_REF] Hahn | Tabula nearly rasa: Probing the linguistic knowledge of character-level neural language models trained on unsegmented text[END_REF]. The model is a two-layer LSTM without attention, a hidden dimension of 1,204, and word embeddings of dimension 200 for the 50,000 most frequent words. It was trained on a corpus from German Wikipedia, totalling 819 million words. The 50,000 most-frequent words in this corpus are used as the vocabulary and embedded in 200-dimensional vector space. The model reaches a perplexity of 37.96 on this dataset. We use unigram and bigram language models that use the same corpus with Laplace smoothing as baselines. The probability of test sentences according to the language models is computed using the chain rule:

P(w 1 . . . w n ) = log(P(w 1 ))+ n i=2 log P(w i |w 1:i-1 )
Each of these log probabilities can be read from the softmax outputs of the LSTM, or directly estimated in the case of the unigram and bigram models. We also tried normalizing for unigram frequency as proposed by [START_REF] Pauls | Large-scale syntactic language modeling with treelets[END_REF] but found like [START_REF] Marvin | Targeted syntactic evaluation of language models[END_REF] that it did not improve results for the LSTM.

Results

Main Classification Task

Figure 2 shows the distribution of the log probability scores predicted by the LSTM and the distribution of the grammaticality scores given by humans. Figure 3 presents the distributions and average of the AUC values computed per template (50 in total), both for the models' log probability scores and the human grammaticality scores. Performances are rather modest, with a mean AUC of 0.56 for the LTSM and of 0.58 for humans, compared to the chance score of 0.5 for the unigram and bigram models. As expected, the n-gram baselines perform exactly at chance, confirming that they do not represent verb argument structures and that LMs need a deeper encoding to be able capture syntax within sentences. We also notice that AUC varies relatively little across different templates for our models, indicating that the particular choice of template has little impact. For humans, the wider spread in results can be attributed partially to the fact that 55 random permutations out of the 144 permutations were annotated for each template. Therefore, it might have been easier to distinguish grammatical sentences from ungrammatical ones for some templates than others. Surprisingly, humans performed only slightly better than the LSTM. We believe that this is due two factors. First, we presented the sentences in a scrambled order and asked for an absolute grammaticality judgment. It may be more difficult to put a sentence on a 1 to 10 scale than making pairwise judgments. Second, our sentences may be particularly challenging. The grammatical sentences contained both unusual argument orders and semantically odd situations, thus inciting participants to rate them low. While these factors could be expected to impact the LSTM, it is more surprising that they impact humans, despite precise instructions to rate on grammaticality rather than meaning or frequency. In addition, as can be seen in Figure 2b, some ungrammatical sentences were rated as highly grammatical by humans. We suspect that these are cases of inattention, as in our test set the distinction between grammatical and ungrammatical rest on a single word, and even a single character (the distinction between 'der' and 'den', for instance).

Case Frequency Bias

In Table 1, we further investigate our grammaticality results by segregating them by case violati- on type (duplicate nominative, accusative or dative). While humans tend to give similar scores for each violation type, models tend to assign higher log probability scores to sentences with doubled nominatives than to grammatical sentences, leading to worse than chance performance on Nominative violations. Conversely, models tend to assign lower log probability scores to sentences with doubled datives, likely because these sentences lack either a nominative or an accusative, both of which are more frequent than dative. This leads to better than human performance on this case violation. Such behavior is probably due to the fact that German being a non pro-drop language, every verb must have a nominative case, making nominative more frequent than accusative, and that dative even rarer. This frequency bias is worse for models that are directly based on frequency, such as our unigram and bigram models. However, our LSTM is not exempt from it, confirming that RNNs rely in part on frequency cues. The grammatical vs non grammatical AUC scores based on log probability (models) and grammaticality scores (humans), for each type of case violation (e.g: Nominative compares grammatical vs double nominative sentences). Chance level corresponds to 0.5.

Argument Order Preferences

In Figure 4, we explore the effect of argument order. Despite the fact that all argument orderings should be equally valid from a grammatical perspective, we find that humans tend to favour more 'canonical' orders, with nominative-accusativedative being the preferred order. Models also assign higher log probability scores to the canonical order compared to others. It is likely that some orders occur more frequently than others in German, thus leading to a frequency bias for both models and humans. Although sentences with shuffled argument order have the same meaning as those without shuffled order, we find a similar bias for the meaningfulness scores. Interestingly, even though the case orders preferred by the LSTM correlate with those of humans, there are also subtle differences: we also find that models tend to prefer argument orders that start with dative to those that start with accusative, when the opposite is true for human grammaticality scores. The origin of such differences is unclear. Understanding it more fully would require to obtain distributional statistics on the order of such phrases in the original corpus.

Animacy Preferences

As mentioned in Section 3, some of our grammatical sentences are semantically implausible though syntactically valid. This is because we create highly unlikely associations of case assignments and thematic roles when we permute the case assignments from the original sentence template. For instance, one permutation has a bill announcing a minister to the senate. Such unlikely sentences may be rejected by participants as ungrammatical even though they were specifically requested to ignore semantic plausibility. Similarly, they may affect neural models through the distributional correlates of meaningfulness: in any language corpus, a bill being an inanimate object is more likely to be an object (accusative case) than a subject (nominative case).

To check for the existence of such effect, we categorized the nouns in all of our sentences as animate and inanimate, and computed the human and machine scores of our grammatical sentences as a function of the association between case and animacy. Table 2 shows that indeed, both humans and machines are biased by animacy-case associations: all share a preference for animate for nominative (subject) and dative (indirect object). By contrast, negative AUC values for accusative indicate that direct objects are preferred as inanimate.

Restricting the Analysis to Plausible Sentences

To see the impact of such biases, we re-analysed the human and machine scores by restricting the AUCs to the non-permuted sentences, i.e, the sentences whose case assignments correspond to that of the original templates. These templates were constructed to be plausible, and indeed the average human plausibility scores for these nonpermuted orders of 5.33 is higher than for the permuted ones 3.61. In this analysis, we therefore include the 6 valid grammatical argument order permutations and all 108 grammatical violations for each template sentence.

The results are shown in Table 3. As expected, the human AUC scores are higher in this restricted analysis than in the full dataset shown in Table 1. Note that the model scores are also higher, which suggests that the algorithms are also sensitive to meaningfulness, probably through its effects on the distribution of case for the different nouns in the training corpus.

Correlation between model and human ratings

In

Conclusions

We set up a well controlled grammaticality test for the processing of argument structure in neural language models and in humans. The results show that LSTMs are better than chance in detecting an abnormal argument structure, despite the fact that the arguments could occur in any position, due to the generally free word order of phrases in German relative clauses.

The average performance of models, though, is far from 100% correct and lower than humans, and the error patterns differ markedly. Contrary to humans, neural language models are overly sensitive to frequency distribution of phrase types. For instance, they assign a higher probability to sentences containing multiple nominative phrases than a correct sentence with only one nominative phrase. This frequency bias directly reflects the frequency of nominative, accusative and dative in the language, as the same bias is found in unigram and bigram models. Similar to the conclusion reached by [START_REF] Linzen | Distinct patterns of syntactic agreement errors in recurrent networks and humans[END_REF] in their investigation of the error patterns made by RNNs and humans on syntactic agreement, we find that the syntactic representations of humans and LSTMs differ in some respects. Despite this difference, neural models are able to mimic the pattern of human responses for grammatical sentences. As has been noted previously, humans are not uniformly considering all grammatical sentences as grammatical, i.e, grammaticality judgments are graded [START_REF] Phillips | Grammatical illusions and selective fallibility in real-time language comprehension[END_REF]. Humans tend to reject sentences with unusual word orders. For instance, they prefer the canonical Nominative-Accusative-Dative order over all of the others orders. A similar pattern is found in neural models, although the details differ somewhat.

Another point of convergence is found with regards to the association between case and semantic features: humans prefer that nominative phrases are animate, and accusative inanimate, a pattern also found in neural networks. This shows that humans have difficulties in judging grammaticality as separate from other factors like frequency and meaningfulness, especially when sentences are presented independently instead of in minimal pairs. In this respect, humans are quite comparable to neural models.

Overall, the difficulty of neural networks to detect incorrect argument structure as such (especially spectacular in the case of duplicate nominatives), provides us a clue that these models may not be fully able to represent such structures, above and beyond their probability distributions.

Figure 1 :

 1 Figure 1: Construction of grammatical examples by permuting the case assignment and the argument order in template sentences. For the construction of the ungrammatical examples, we doubled one of the cases, creating a declension error.

Figure 3 :

 3 Figure 3: Distributions of the AUC values per template and average over templates (in bold) for models and for humans.

Figure 2 :

 2 Figure 2: Distribution of log probability scores and grammaticality scores for grammatical sentences and ungrammatical sentences (a) for the LSTM and (b) for humans.

Figure 4 :

 4 Figure 4: Average log probability scores of LSTM against grammaticality or meaningfulness scores of humans for specific case orders on grammatical sentences.

  Wir wissen, dass der Minister dem Senat diesen Gesetzentwurf angekündigt hat. (We know that the minister announced that bill to the Senate.) 50 TEMPLATES Wir wissen, dass der Minister der Senat diesen Gesetzentwurf angekündigt hat. (We know that the minister the Senate announced that bill.) 36 PERMUTATIONS PER TEMPLATE Accusative case violation: Wir wissen, dass der Minister den Senat diesen Gesetzentwurf angekündigt hat. (We know that the minister announced that bill the Senate.) 36 PERMUTATIONS PER TEMPLATE Dative case violation: Wir wissen, dass der Minister dem Senat diesem Gesetzentwurf angekündigt hat. (We know that the minister announced to that bill to the Senate.) 36 PERMUTATIONS PER TEMPLATE

	Grammatical set permutations:
	Permuting argument order:
	Wir wissen, dass dem Senat der Minister diesen Gesetzentwurf
	angekündigt hat. (We know that the minister announced that bill to the
	Senate.)
	6 PERMUTATIONS PER TEMPLATE
	Permuting case assignment:
	Wir wissen, dass dem Minister der Senat diesen Gesetzentwurf
	angekündigt hat. (We know that the Senate announced that bill to the
	minister.)
	6 PERMUTATIONS PER TEMPLATE
	Permuting both
	Wir wissen, dass der Senat dem Minister diesen Gesetzentwurf
	angekündigt hat. (We know that the Senate announced that bill to the
	minister.)
	6 × 6 = 36 PERMUTATIONS PER TEMPLATE
	Ungrammatical set permutations:
	Nominative case violation:

Template:

Table 1 :

 1

	Model	Overall Nominative Accusative Dative
	Unigram	0.50	0.11	0.63	0.77
	Bigram	0.50	0.27	0.54	0.69
	LSTM	0.56	0.42	0.55	0.71
	Humans	0.58	0.56	0.60	0.57

Table 2 :

 2 Preference for animacy on grammatical sentences computed as the ROC-AUC for the scores as function of the association between case and the animate versus inanimate status of the noun. Less than .

		Nominative Accusative Dative
	LSTM log probability	0.51	0.43	0.59
	Human grammaticality	0.62	0.36	0.52
	Human meaningfulness	0.58	0.35	0.57

5: preference for inanimate. More than .5: preference for animate.

Table 4

 4

	, we show correlations between human
	judgments of grammaticality, meaningfulness
	and LSTM log probabilities. Unsurprisingly,
	all variables are positively correlated, which
	supports our earlier findings. More surprising
	is that the LSTM is more correlated with
	both grammaticality and meaningfulness than

Table 3

 3

	: The grammatical vs non grammatical
	AUC scores based on log probability (models) and
	gramaticality scores (humans), restricted to the original
	(plausible) template sentences plus their argument
	order permutations, for each type of case violation.
	meaningfulness is with grammaticality. Note that
	meaningfulness and grammaticality have been
	annotated by different annotators, which might
	help explain this finding.
		R
	grammaticality vs. meaningfulness (gr. only) 0.17
	LSTM vs. meaningfulness (gr. only)	0.23
	LSTM vs. grammaticality (gr. only)	0.27
	LSTM vs. grammaticality (all)	0.18

Table 4 :

 4

Spearman's Correlation Coefficient between log probabilities (models) and grammaticality and meaningfulness scores (humans). gr. only: restricted to the grammatical sentences; all: all sentences.

The average pay for our annotators was $20/hr.

Acknowledgments

The team's project is funded by the European Research Council (ERC-2011-AdG-295810 BOOT-PHON), the Agence Nationale pour la Recherche (ANR-10-LABX-0087 IEC, ANR-10-IDEX-0001-02 PSL* ), Almerys (industrial chair Data Science and Security), and grants from Facebook AI Research (Research Grant), Google (Faculty Research Award), Microsoft Research (Azure Credits and Grant), and Amazon Web Service (AWS Research Credits).