Rossella Garuti
email: rossella.garuti@libero.it

Francesca Martignone
email: francesca.martignone@uniupo.it

Assessment and argumentation: an analysis of mathematics standardized items

Keywords: Assessment, argumentation, standardized tests

In this paper, after a brief description of Italian National Guidelines and Evaluation System, we present and analyze two examples of standardized items focused on argumentation. The qualitative analysis is carried out by using theoretical lenses coming from studies about mathematical argumentation: in particular, we use Toulmin's model to identify common features in the structure of these standardized items. The first results of our analysis give us the elements to reflect on the multiple-choice items structures that could be used for identifying specific aspects about argumentation and for assessing students' skills.

Introduction

Since many years, mathematics education research has dealt with teaching and learning argumentation and proof [START_REF] Hanna | The ongoing value of proof[END_REF]. The importance of argumentation in mathematics education was perceived also at the institutional level and has led to important changes in the orientation of different countries curricula (from primary to secondary school) all over the world. Also, in the frameworks of the main international surveys, as IEA TIMSS and OECD PISA, argumentation is an important milestone. In the Mathematics Framework of IEA TIMSS 2019 (IEA, 2017) one of the three cognitive domains (reasoning) is described as:

Reasoning mathematically involves logical, systematic thinking. It includes intuitive and inductive reasoning based on patterns and regularities […]. Reasoning involves the ability to observe and make conjectures. It also involves making logical deductions based on specific assumptions and rules, and justifying results. (TIMSS 2019 Mathematics Framework, p. 24) As regards OECD PISA survey, in the recent version of the mathematical framework, we read: Mathematical literacy therefore comprises two related aspects: mathematical reasoning and problem solving. Mathematical literacy plays an important role in being able to use mathematics to solve real-world problems. However, mathematical reasoning also goes beyond solving problems in the traditional sense to include making informed judgements in general about that important family of societal issues which can be addressed mathematically. It also includes making judgements about the validity of information that bombards individuals by means of considering their quantitative and logical implications. (PISA 2021, Mathematics Framework, First draft, p. 10) In this paper we examine how Italian mathematics standardized tests try to assess specific aspects related to argumentation skills: we analyse two items selected from grade 5 (primary school) national standardized tests. We use theoretical lenses taken from research in mathematics education in order to highlight the argumentation features that these two standardized items focused on. Our reflections are grounded in the studies on argumentation processes carried out by some Italian researchers [START_REF] Boero | Approaching Theorems in grade VIII[END_REF][START_REF] Boero | Argumentation and proof: a contribution to theoretical perspectives and their classroom implementation[END_REF][START_REF] Arzarello | Semiotic and theoretic control in argumentation and proof activities[END_REF].

Italian National Guideline and standardized tests

In the last ten years, new Italian National Guideline (NG) for the first cycle of instruction (i.e. preprimary, primary and middle school) have been proposed by the Ministry of Education (Ministero dell'Istruzione, Università e Ricerca, MIUR). For primary and middle school (grades 1 to 8), the NG were first published in 2007 with the latest version arriving in 2012 (MIUR, 2012). Italian legislation does not lay down a strict curriculum, but it indicates the goals for competence development at the end of grades 3, 5 and 8. Some of these goals refer explicitly to the argumentative skills that students should acquire:

[the student] constructs reasoning by formulating hypotheses, by supporting his/her ideas and by dealing with others' points of view (grade 5, end of primary school, MIUR 2012, p. 50, translation by the authors) The Italian Ministry of Public Education has established the standardized assessment of the Italian educational system, and commissioned the INVALSI (www.invalsi.it) to carry out annual surveys nationwide to all students in the second and fifth grades of primary school, grade 8, and high school (grades 10 and 13). INVALSI is a research institute with the status of legal entity governed by public law. INVALSI carries out periodic and systematic checks on students knowledge and skills (about reading comprehension, grammatical knowledge and mathematical competency), and on the overall quality of the educational offers from schools and vocational training institutions; in particular, it runs the National Evaluation System (SNV). The INVALSI standardized tests were created for system evaluation and this is their primary purpose. The statistical representative sample comprises approximately 30,000 students (with tests administered under controlled conditions). Moreover, the tests are administered at census level and students results are provided to each school institution. The SNV Framework defines what type of mathematics is assessed by the SNV tests and how it is evaluated. It identifies two dimensions along which the items are built: the mathematical content, divided into four major areas (Numbers, Space and Figures, Relations and Functions, Data and Uncertainty), and the mathematical processes involved in solving the items (Knowing, Problem Solving, Arguing and Proving). These dimensions are closely and explicitly related to the goals for competence development of NG. The framework adopted by SNV assessment includes aspects of mathematical modelling as in PISA survey [START_REF] Niss | Mathematical Competencies and PISA[END_REF], and aspects of mathematics as a body of knowledge logically consistent and systematically structured, characterized by a strong cultural unity [START_REF] Arzarello | The impact of PISA Studies on the Italian National Assessment System[END_REF]. The INVALSI tests are designed by expert teachers, educational and disciplinary researchers, statisticians and experts of the school system. [START_REF] Garuti | The SNV (INVALSI) experience. Teaching and learning mathematics: resources and obstacles[END_REF][START_REF] Garuti | The Italian national education assessment system: building mathematics items[END_REF].

Argumentation in INVALSI tests

As stressed before, taking into account the NG suggestions, INVALSI tests aim to assess also argumentation skills. Therefore, our research questions are: which aspects related to argumentation skills can be assessed by a standardized test? And in which way? These are very general questions that we faced starting from a more specific investigation about which elements were taken into account in the construction of some INVALSI argumentative items.

In a INVALSI booklet there are approximately 40-50 items and about 10% of these are about argumentation. A standardized test cannot assess all the argumentative skills quoted in the NG (e.g. formulating hypotheses or exploring a problem situation in order to produce conjectures), but it can propose tasks that ask students to support his/her statements, to show examples and counterexamples, and to recognize the logical consequences of a correct argument. In the SNV framework the limits of standardized tests in the assessment of mathematical competencies, particularly with regard to the argumentative skills, are well explained, but it is also clear that some aspects of this capability can be assessed. For example, by means of items that ask to choose the correct answer and the right justification of it among the options proposed, or to produce and justify the answer. In INVALSI tests, two item-format types are used to assess the argumentative skills: open constructed-response items and selected-response (multiple-choice) items. The first may ask the student to explain how the answer was reached or to justify the answer of a given statement; the second requires to select one response among a number of options.

In this paper we analyze two multiple-choice items selected from grade 5 tests. We joined the groups of teachers and experts, who produced the INVALSI tests, therefore we can argue about the choices made during items productions. These choices are the results of discussion in which Mathematical Knowledge for Teaching [START_REF] Ball | Content knowledge for teaching: What makes it special[END_REF], the experiences developed in the classrooms, and knowledge about educational research studies merged.

Any discourse cannot be accepted as an argumentation, "[…] a reason or reasons offered for or against a proposition, opinion or measure" (Webster Dictionary), and may include verbal arguments, numerical data, drawings etc. Argumentation can indicate both the process which produces a logically connected discourse about a given subject and the text produced by the process [START_REF] Douek | Some remarks about argumentation and proof[END_REF].

In order to describe the structure of mathematics argumentations, as texts produced, different studies use Toulmin's model (1958). In the next paragraph we show some interpretative tools proposed by this model.

An interpretative tool for the argumentation structure

Toulmin's model (1958) has been used for the analysis of arguments in mathematics education [START_REF] Pedemonte | How can the relationship between argumentation and proof can be analyzed[END_REF] as a tool to analyses structural features of mathematics argumentation both with pre-service teachers [START_REF] Arzarello | Semiotic and theoretic control in argumentation and proof activities[END_REF]) and primary students [START_REF] Douek | About argumentation and conceptualization[END_REF]. Many studies report the limitations of this model in order to analyze mathematical argumentation [START_REF] Nielsen | Dialectical features of students' argumentation: A critical review of argumentation studies in science education[END_REF], in particular to study the dialogical and dialectical elements of verbal interaction that take place in the classroom. The "argumentation" is much more complex that the arguments that make it up, but Toulmin's model can be used to identify the argument structure, in particular to break down arguments into their constituent parts. In Toulmin's basic model an argument comprises three elements: the Claim (C), i.e. the statement; the Data (D), i.e. the data that justify the claim C; and the Warrant (W), i.e. the inference rule which allows data D to be connected to claim C.

We use this way of breaking down arguments into their constituent parts to identify and compare some elements in the structure of multiple-choice options of INVALSI argumentative items.

Research questions

In our study we focus on written texts, in particular INVALSI multiple-choice items in which students have to choose among different sentences. In this frame, we refine our initial research questions: can we identify common structures in the different answer options in INVALSI tests? How can we carry out a posteriori analysis of the argumentative items by using some interpretative tools coming from Toulmin's model?

Examples of INVALSI argumentative items

The first example (Figure 1) is a geometry item relating to equivalence between plane figures. The student must choose the correct answer: the options are built up by two "Yes" and two "No" followed by arguments that have to justify correctly the answer. The stem consists of two figures and a question that is common in activities about geometrical objects in primary school. This example shows a structure in the construction of the item that we identified as very common in the INVALSI argumentative items. In the different options, the student has to choose not only if the answer is Yes or No, but also the correct argumentation: i.e. the argumentation that is relevant and useful to answer correctly the question. For these reasons, all the sentences after Yes and No are true sentences, but in only one is there a correct argument. In particular, options A and C (together are chosen by about the 50% of students) are both referring to the fact that the dimensions of the rectangles are different: these may refer to a misconception like "if the sizes are different then the area must be different" quite common in primary school. Option D (chosen by 7.4% of the students) is weaker and refers to invariance of shapes (right triangles). Using the Toulmin's model to interpret the item structure, we can identify Data (D) (i.e. the representation of two equivalent rectangles made by the same triangles), Claim (C) (i.e. the recognition of the equivalence of the figures: "Yes, they have the same area" or "No, they don't have the same area") and Warrants (W) (i.e. the inference rule which allows Data (D) to be connected to Claim (C) about the identification of the same area of the rectangles). Therefore, focusing on the structure of the four options, the students have to choose the options where they recognize both the right answer to the question about the equivalence of the figures area and the right justification of it. The second example (Figure 2) deals with a situation framed in a field of experience external to mathematics. This item requires to understand that the difference in the amount of water contained in equal pitchers doesn't change if we add the same amount of water in each one of them. The students have to recognize the correct statement that justifies this invariance. The incorrect options highlight misconceptions related to the situation of invariance or difficulties in identifying a condition that is necessary, but not sufficient. In this example, the students can identify arguments (Warrants) linked also to everyday life experience, or can use forms of reasoning for general principles about adding quantities to different initial amounts. The statistical data show that this item has a higher percentage of correct answers: we can conjecture that it is because it refers to everyday life experiences. The option B focuses on the difference between relative and absolute increase and it was chosen by many students (29.1%). In fact, the other options show more general arguments about shape and content of the pitchers.

Using the Toulmin's model: data (D) is represented by the figure and the text of the question that describes the situation. Claim (C) is about the increase of water in the two pitchers after the addition of the same quantity of water. Warrant (W) is given by the arguments in support of responses (two for answer Yes and two for answer No). Although the mathematical content and context are very different from the previous example, we identify the same structure: all the arguments are true, but only one is adequate in order to support the correct answer.

Summarizing, the analysis of the structure of these two items show that all the Warrants are true and pertinent (i.e. relating directly) to the mathematical content of the item, but only one is adequate, that is useful to support the Claim.

Discussion

In this paper we discussed some features of INVALSI items focused on argumentation: multiplechoice items where the student has to identify the right arguments to support his/her answer concerning a given statement. Analyzing this tests, we demonstrate how certain aspects of argumentation can be assessed through the use of standardized items. In particular, the ability of the students to choose the correct argument to support a statement. We are aware that the analysis of other argumentative skills, such as the production of justifications to support a claim, requires analysis tools other than the Toulmin's model, which provides us with information only on the structural aspects of the argumentation. By means of two examples, we show some characteristics of argumentative items and we use Toulmin's model as a posteriori interpretative tool to describe and interpret the structure of these items. We chose Toulmin's model as an interpretative tool because it allows us to analyze retrospectively the structural aspects of argumentative items focusing on the characteristics of the justifications (Warrant), regardless of the mathematical content involved. The analysis carried out in this paper wants to show how, even if we chose two items that deal with different topics and contexts, we can identify in the structure of the items some similar characteristics: i.e. structural characteristics that the use of Toulmin's model can clearly highlight.

In the examples presented here all the justifications (Warrants) are true and pertinent, but in INVALSI tests we can also find items where all the justifications (Warrants) are true but only one is pertinent or items where the incorrect options are false but pertinent. In the latter case, as regards the first example (Figure 1), the three incorrect response options could have been as follows: No, because the two figures have different areas; No, because the two figures have a different number of triangles; Yes, because the two figures are congruent. The justifications are false, but directly relevant (i.e. pertinent) to the context of the question.

It is clear that, as underlined at the very beginning of the paper, standardized items can evaluate only some features of argumentation, as well as the Toulmin's model can identify only specific types of argumentation structures, but this doesn't mean that a study about these elements cannot give us some educational information about the assessment on argumentation and helpful tips for designing other kinds of standardized items about argumentation. As a matter of fact, in the further steps of our study, we are investigating if the Toulmin's model can become a useful a priori tool in order to construct items in which the argumentations have different structures: i.e. items with the same structure of those presented here; items where it is explicit that the answer is No (or Yes) and the students have to choose the only warrant that justifies it; items in which, starting from the data and by a given warrant, the students have to choose which statement, among those proposed, is justified by the argument (from Warrant to Claim).

Therefore, in our study the Toulmin's model is being transformed from a posteriori analysis tool that sheds light on the passage from Claim to Warrant (I have a Claim and I have to identify which Warrant is adequate to justify that Claim) to a tool for constructing argumentative items, that to say to a priori analysis tool.

[

  the student] produces arguments based on the theoretical knowledge acquired […] [the student] supports his/her beliefs by choosing examples and counterexamples and by using concatenations of claims; [the student] agrees to change his/her opinion recognizing the logical consequences of a correct argument" (grade 8, end of middle school, MIUR 2012, p. 51, translation by the authors).

Figure 1 :

 1 Figure 1: Item from SNV 2016, grade 5 (indicating the percentage for each option) (translation by the authors)

Figure 2 :

 2 Figure 2: Item from SNV 2016, grade 5 (indicating the percentage for each option) (translation by the authors)


Jokke Häsä
email: jokke.hasa@helsinki.fi

Johanna Rämö
email: 2johanna.ramo@helsinki.fi

Viivi Virtanen
email: 3viivi.virtanen@helsinki.fi

Evaluating students' self-assessment in large classes

Keywords: Self-assessment, assessment for learning, digital assessment, accuracy, large classes

This study is part of an ongoing larger project concerning student self-assessment skills in university courses. We have developed a method enabling large cohorts of students to assess their own learning outcomes and to give their own course grades with the help of an automatic verification system. This paper explores the question of accuracy, namely, whether the self-assessed grades correspond to the students' actual skills, and how well the automatic system can pick up issues in the self-assessment. Based on an expert's evaluation of the skills of two students, we conclude that although for large part the model works as intended, there are some cases where neither the self-assessment nor the computer verification seem to be accurate.

Introduction

The ability to judge the quality of one's own work is one of the core skills that should be developed during university studies. Self-assessment has been viewed as a valuable assessment process through which student can learn to understand the expectations, criteria and standards used in assessment, and further, to be able to regulate own learning and acquire skills for lifelong learning [START_REF] Falchikov | Student Self-Assessment in Higher Education: A Meta-Analysis[END_REF][START_REF] Kearney | Using self-and peer-assessment for summative purposes: analysing the relative validity of ASSL (Authentic Assessment for Sustainable Learning) model[END_REF]. However, academic community seem to be resistant to change the prevailing assessment practices focusing on testing and grading, and practices such as self-assessment are scarcely implemented at course level [START_REF] Boud | Reframing assessment research: through a practice perspective[END_REF][START_REF] Postareff | Academics' conceptions of assessment and their assessment practices[END_REF].

In this paper, we draw attention to assessment practices in university first-year mathematics by examining an implementation of student self-assessment processes into large class setting. During this process, students frequently evaluated the quality of their learning outcomes, received feedback on their performance, and finally decided their own grades according to particular criteria. The intended learning outcomes were made transparent through a rubric including both content knowledge and generic skills, such as writing mathematics. The digital environment gave opportunity for monitoring learning process and giving real-time formative feedback in line with previous research on online assessment [START_REF] Ćukušić | Online self-assessment and students' success in higher education institutions[END_REF]Gigandi, Morrow, & Davis, 2011;[START_REF] Ibabe | Online self-assessment with feedback and metacognitive knowledge[END_REF], and further, it formed a basis of assessment of the student's progress. The emphasis of self-assessment was in developing student capability in making evaluative judgements [START_REF] Ajjawi | Conceptualising evaluative judgement for sustainable assessment in higher education[END_REF] and building their metacognition skills [START_REF] Mok | Self-assessment in higher education: experience in using a metacognitive approach in five case studies[END_REF], so that the students' ability to self-regulate their learning for current and future learning would improve. We fill the gap in research by showing how, in the case of summative self-assessment, the problems aroused by large class setting were resolved by using digital and automatic verification and real-time feedback.

Self-assessment as a tool for learning

Self-assessment can be defined as a process during which student evaluate their own achievements and judge about their own performance [START_REF] Falchikov | Student Self-Assessment in Higher Education: A Meta-Analysis[END_REF]. The judgements students make are based on information and evidence about their own performance collected from various sources [START_REF] Yan | A cyclical self-assessment process: towards a model of how students engage in self-assessment[END_REF]. In this paper, we refer to self-assessment as a process in which the students evaluate their own progress and performance and give justifications for the result of their evaluation according to teacher-given criteria showing intended learning outcomes.

The use of self-assessment has been shown to improve student engagement and motivation (e.g. [START_REF] Andrade | Student responses to criteria-referenced self-assessment[END_REF][START_REF] Mok | Self-assessment in higher education: experience in using a metacognitive approach in five case studies[END_REF], self-efficacy [START_REF] Kissling | Increasing language awareness and self-efficacy of FL students using self-assessment and the ACTFL proficiency guidelines[END_REF] and academic performance [START_REF] Ibabe | Online self-assessment with feedback and metacognitive knowledge[END_REF], while the ability to self-assess is reportedly intermingled with ability to self-regulate own learning [START_REF] Panadero | The future of student self-assessment: A review of known unknowns and potential directions[END_REF] and with life-long learning skills [START_REF] Kearney | Using self-and peer-assessment for summative purposes: analysing the relative validity of ASSL (Authentic Assessment for Sustainable Learning) model[END_REF]. Consequently, the literature encourages the use of selfassessment for formative purposes. Research shows that in large class settings digital environments with effective formative online assessment can foster a learner-centred focus and engagement in learning (Gigandi et al. 2011;[START_REF] Ibabe | Online self-assessment with feedback and metacognitive knowledge[END_REF]. Recent results show that online selfassessment can also improve students' academic success (Ćukušić et al. 2014). However, the debate concerning students generating their own grades by self-assessing their own work is more complicated and constantly questioned [START_REF] Boud | Reframing assessment research: through a practice perspective[END_REF]Tejeiro et al., 2010). One of the main challenges regarding self-assessment for grading is the question of accuracy: How can we be sure that students' grades are valid and reliable?

The question of accuracy

Many studies have found high correlations between self-and teacher-ratings [START_REF] Falchikov | Student Self-Assessment in Higher Education: A Meta-Analysis[END_REF][START_REF] Kearney | Using self-and peer-assessment for summative purposes: analysing the relative validity of ASSL (Authentic Assessment for Sustainable Learning) model[END_REF]. The results indicate that students are able to make reasonable accurate judgements if they are properly provided with training and background information to the process. Also, students vary in their capability to evaluate their performance e.g., high achievers tend to underestimate their performance whereas low achievers tend to overestimate it [START_REF] Falchikov | Student Self-Assessment in Higher Education: A Meta-Analysis[END_REF][START_REF] Boud | Does student engagement in self-assessment calibrate their judgement over time?[END_REF][START_REF] Kearney | Using self-and peer-assessment for summative purposes: analysing the relative validity of ASSL (Authentic Assessment for Sustainable Learning) model[END_REF]. However, the accuracy of student self-assessments can be improved through using criteria and standards [START_REF] Andrade | Student responses to criteria-referenced self-assessment[END_REF], while students need to have multiple opportunities for practising self-assessment in relation to given criteria, with feedback to help calibrate the judgements [START_REF] Hosein | The relationship between students' prior mathematical attainment, knowledge and confidence on their self-assessment accuracy[END_REF][START_REF] Kearney | Using self-and peer-assessment for summative purposes: analysing the relative validity of ASSL (Authentic Assessment for Sustainable Learning) model[END_REF]. On the other hand, [START_REF] Boud | Does student engagement in self-assessment calibrate their judgement over time?[END_REF] argue that increase in accurate self-assessment is not immediately transferable, because standards and criteria are somewhat domain-specific. Hence, we suggest that in order to understand the expectations, criteria, and disciplinary standards of mathematics, and to develop capabilities to make accurate and realistic assessments on own learning processes and outcomes, it is required that self-assessment processes are implemented in first-year university mathematics. However, in large class setting, typical to that learning context, the challenge how to give evidence-based feedback for improving the accuracy needs to be resolved.

The DISA model

This study is part of a research project centred around an assessment model called DISA (Digital Self-assessment). In the model, students assess their own learning outcomes throughout the course by using a detailed rubric articulating the subgoals of the ultimate intended learning outcomes. Learning goals and criteria are clearly identified, and through self-assessment activities the students are actively engaged with them. Evidence of learning is elicited during the course, and students receive feedback for their self-assessment from an automatic digital system.

The feedback is generated in the following way. Every course task has been linked with the learning objectives it is supporting. This enables the automatic system to compute, based on the student's coursework, an index from 0 to 1 for each learning objective. This index estimates how well the student has acquired the learning objective. From these indices, the system then computes tentative grades in each course topic. These tentative grades are compared to the student's self-assessed grades, and the student is advised either to consider a higher or a lower grade for themselves.

In addition, self-assessment is used for summative purpose in the end of the course, as the students self-assess and justify how well they have achieved the intended learning outcomes, and proceed in deciding their own course grade based on the self-assessment. In order to prevent abuse of the selfassessment process, the system described above is used to verify the validity of the course grades. If the self-assessed topic grades differ too much from the computed ones, the student's final course grade is disputed. Earlier results imply that the model supports students in using deep learning approach, and encourages them to study for themselves, not for an exam [START_REF] Nieminen | Replacing Exam with Self-Assessment: Reflection-Centred Learning Environment as a Tool to Promote Deep Learning[END_REF].

Aim of the study

This study aims at gaining a better understanding of the use of self-assessment as an integral part of assessment in a large first-year university mathematics course. In the course context, selfassessment is used to give students an opportunity to think metacognitively about their learning. We hypothesise that student active engagement into self-assessment processes is enhanced if these processes are valued in grading, but then, the question of accuracy needs to be resolved. This question is two-fold: firstly, we are interested in the validity of the student grades, in other words, whether they reflect true learning, both in content knowledge and domain-specific generic skills such as writing mathematics. Secondly, we need to examine the reliability of the automatic verification system: can it spot the cases where self-assessment is inaccurate? The research questions in this study are:

1. How do the students' evaluations of their own skills compare with evaluations performed by the automatic verification system? 2. How does an expert judge the student's acquired skills in cases where the automatic verification disagrees with student's self-assessment?

Method

This study uses data collected from students taking a first year mathematics course at a major research-intensive university in Finland. The second author was the lecturer for the course. The course was a proof-based linear algebra course dealing with finite-dimensional vector spaces, and it lasted for seven weeks (half a term). During the course, students were given weekly problems to solve, part of which were assessed and given feedback on. Some of the tasks were assessed by the tutors, some by an automatic assessment system called Stack [START_REF] Sangwin | Computer aided assessment of mathematics[END_REF]. Some tasks were also peer-reviewed.

The course was not graded with a final exam, but grading was done by self-assessment using the DISA model. The self-assessment was based on a detailed learning objectives matrix prepared by the teacher. The learning objectives were divided into 10 topics: six content-specific and four generic skills topics, and the students were asked to give themselves a grade from 0-5 in each of these topics, 0 meaning fail. They were also asked to write down reasons for choosing that grade. In the end of the course, students chose their own final grade. They were left to decide by themselves how to combine the grades from the different topics. The DISA system was used to verify the final self-assessment.

It is worth noting that, in the Finnish context, although the teacher is responsible for the course grades, these can be awarded by any means the teacher chooses. There is little fear of distorting the grades, as the final grade of a first-year mathematics course carries very little weight in the final outcome of a student's study programme. Also, all courses and exams can be usually retaken as many times as the student wishes.

The participants of this study were 158 students who took the linear algebra course described above, gave themselves their own grades using the DISA model, and gave consent for using their data. Most of the students were majoring in either mathematics, mathematics education or some other field related to mathematics such as computer science, physics or chemistry. Most students were first year students, but the cohort included also older participants, up to post-doctoral level.

We narrow our study to two of the ten learning objective topics of the course: (1) "Matrices" (content-specific) and ( 2) "Reading and writing mathematics" (generic skill). These two topics were chosen since both are among the most central topics of the course and there were relatively many tasks linked to them. Also, we wanted to compare self-assessments on a content-specific topic with those on a generic skill. Henceforth, these topics are abbreviated as [M] and [RW]. Examples of learning objectives pertaining to these topics are given in Table 1. To answer Research question 1, we compared the grades students gave themselves on the two topics in the final self-assessment with the results of the automatic verification of that self-assessment. The computations were done with R version 3.5.0. For Research question 2, coursework and final selfassessment of two students whose self-assessment was poorly in line with the automatic verification were chosen for closer inspection. In this manuscript, we call them Student A and Student B. The two students' anonymised answers to all of the written tasks as well as their Stack exercise points were analysed by the second author. This author was also the teacher of the course and can be regarded as an expert in the subject. When the expert was grading the students, she did not know how the students had assessed themselves. The expert read every written solution the student had submitted, and evaluated which learning objects in topics [M] and [RW] the student had reached.

Topic

Every time the expert could see the student mastering a learning object, she made a note in the learning objectives matrix. After that, there were learning objectives for mastering of which the student had not provided any evidence in the written solutions. The expert then looked at the Stack exercises that were linked to these learning objectives to see how many points the student had received from them. She used the information in determining whether the student had reached the remaining learning objectives. When the expert had considered each learning objective, she awarded the student a grade in both topics by looking from the learning objectives matrix which grade the reached learning objectives corresponded to. In borderline cases, the expert used her expertise as a mathematician and teacher of the course. For the topic "Reading and writing mathematics", the expert could only evaluate the student's skills in writing as there were no tasks that were linked to reading skills.

Results

Research question 1: comparison of self-assessed grades with automatic verification

The distributions of the self-assessed grades in the two topics The computer verification system computed tentative grades for the two topics for each student.

The distribution of differences between the computed grade and student self-assessed grade are reported in Table 3. We see that the computer and student grades agree well. In [M], there are 86 matches, 53 cases in which the self-assessed grade was higher than the computed grade (negative difference), and only 19 cases in which the self-assessed grade was lower (positive difference). In [RW], there are 75 matches, 43 cases in which the self-assessed grade was higher, and 40 cases in which the selfassessed grade was lower. In both topics, between 81-83 % of self-assessed grades lie within 1 grade point from the computed grade.

Difference -5 -4 -3 -2 -1 0 1 2 [M]

Research question 2: Expert opinion in conflicted cases

Student A's self-assessed grades were lower than the computed ones. For both topics, the selfassessed grade was 4 and computed grade 5. The expert's evaluation agreed with the computed grades. The expert observed that Student A had done almost all tasks during the course. Even though not all the answers were correct, all the learning objectives in topic [M] were fulfilled.

Students were asked to make corrections to some tasks, and student A had always resubmitted solutions written in good mathematical style. The student's explanations were concise and readable, and the student was able to construct proofs concerning abstract mathematical objects. Based on this, the expert's grade for topic [RW] was 5.

Student B's self-assessed grades were greater than the computed ones. For topic [M], the selfassessed grade was 5 and the computed grade 3. The expert's evaluation yielded grade 4, that is, something in between. For [RW], the self-assessed grade was 3 and the computed grade 1. The expert's evaluation agreed with the self-assessed one. The expert observed that Student B had submitted only a fraction of the course tasks. However, the expert was able to evaluate from the solutions that Student B accomplished almost all learning objectives in [M]. Some of Student B's skills were shown in the intermediate steps of tasks that were not directly linked to topic [M]. For example, the student determined whether given vectors are linearly independent by forming a system of linear equations and calculating the determinant of the coefficient matrix. This showed that the student knew how invertibility of matrices is connected to the number of solutions of a system of linear equations even though the topic of the task was linear independence. Student B had not corrected any solutions when encouraged to. According to the expert, the student reached partially all the learning objectives in [RW], but did not fully master any of them, not even the ones corresponding to grade 1. For example, the student mixed up equivalence arrows with equality signs, wrote long, confused sentences and used "if-then" structures inside a proof in the place of assumptions and conclusions. However, the overall structures of the proofs were correct. Based on this, the expert's interpretation was that the student's grade for [RW] was 3.

Discussion

In this study, a new model of determining course grades via self-assessment was examined with a focus on the accuracy of the self-assessed grades. The students gave themselves grades in all course topics, and these grades were automatically verified by comparing them against the course work the students had done. We analysed the results of the verified self-assessment in two topics, one content-specific topic (matrices) and one subject-related generic skill (reading and writing mathematics).

The students' self-assessment agreed well with the automatic verification. Most discrepancies are within one grade point, which can be explained by the coarseness of the grading scale: the "real" skill level is often between two grade points and must be forced to one or the other direction. The high agreement is not surprising, as previous studies have shown that explicit criteria and standards support self-assessment, as does frequent practice and feedback [START_REF] Andrade | Student responses to criteria-referenced self-assessment[END_REF][START_REF] Kearney | Using self-and peer-assessment for summative purposes: analysing the relative validity of ASSL (Authentic Assessment for Sustainable Learning) model[END_REF]. It remains to be studied how great an effect the feedback that the students received for their self-assessment exercises had on their final self-assessment.

The students gave fairly good grades to themselves in both examined topics. For reading and writing mathematics, the grades were more concentrated around the second-best grade, whereas for matrices, the top grade was clearly the most common grade. Perhaps it was easier for the students to understand the learning objectives as well as recognise their achievements in the mathematical topic, and without clear evidence for mastery, they were hesitant to award themselves the best grade in a generic skill. Our results could be understood in the view of previous results [START_REF] Falchikov | Student Self-Assessment in Higher Education: A Meta-Analysis[END_REF] showing that in science courses, self-assessment was more accurate that in other fields.

We examined more closely two students whose self-examined and computed grades differed. In the first case, self-evaluated grades were below the computed ones. The expert's evaluation agreed with the computed grade. The student was a high achiever, and from previous studies we know that such students tend to underestimate their performance [START_REF] Boud | Does student engagement in self-assessment calibrate their judgement over time?[END_REF][START_REF] Kearney | Using self-and peer-assessment for summative purposes: analysing the relative validity of ASSL (Authentic Assessment for Sustainable Learning) model[END_REF]. In the second case, the self-evaluated grades were above the computed ones. The expert's evaluation was between the two for the mathematical topic and agreed with the self-assessed grade for the generic skill. In this case, the student had skipped many tasks which made it difficult for the automatic system to estimate the grade fairly. Also, the expert noted that the student seemed to have some skills from all grade categories in the learning objectives matrix, but not to have fully reached any. This kind of case would be very difficult for the automatic verification system to estimate correctly.

The study used a method in which an expert evaluated students' skills based on all the work they had done on the course, evaluating against the intended learning outcomes, not by grading individual tasks. The method suffered from some of the maladies related to teacher evaluation, such as time restriction and personal bias. The accuracy of teacher-grading is not an issue to be taken as obvious truth [START_REF] Brown | Assessing student learning in higher education[END_REF]. One should also note that neither the expert nor the automatic system were able to evaluate students' reading skills even though they were included in the self-assessed grades.

This study opened a new way to critically examine a self-assessment model as a viable option for grading students. We did not find any fundamental problem with reliability. However, at least in one of the studied cases, the verification system did not estimate the student's skills very well. A larger sample needs to be studied in order to find out whether such issues are common. Also, we need to study students' written justifications for their grades in order to better understand what is involved when the self-assessment process does not go as intended.

Table 1 : Part of the learning objectives matrix of the course. In total, there were 10 topics and 10-15 learning objectives in each topic

 1

		Skills corresponding	Skills corresponding to	Skills corresponding to
		to grades 1-2	grades 3-4	grade 5
	Matrices	I can perform basic	I can check, using the	I can apply matrix
	[M]	matrix operations and	definition of an inverse,	multiplication and
		know what zero and	whether two given matrices	properties of matrices in
		identity matrices are	are each other's inverses	modelling practical
				problems
	Reading	I use course's notation	I write complete, intelligible	I can write proofs for claims
	and writing	in my answers	sentences that are readable	that concern abstract or
	[RW]		to others	general objects

  [M] (Matrices) and [RW] (Reading and writing mathematics) are shown in Table2. We see that the students gave the grades 3 and 4 more often for [RW] than for [M], but the top grade 5 was more common in [M] than in[RW].

	Grade	1	2	3	4	5
	[M]	3	10	25	47	73
	[RW]	2	10	37	58	51

Table 2 : Frequencies of each grade in the two topics

 2

Table 3 : Frequencies of the differences: computed grade minus self-assessed grade in the two topics

 3


Teo Asplund

Andrés Serna

Beatriz Marcotegui

Robin Strand

Cris Luengo Hendriks

Cris L Luengo Hendriks

Mathematical Morphology on Irregularly Sampled Data Applied to Segmentation of 3D Point Clouds of Urban Scenes

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Asplund et al. introduced morphological operators able to deal with irregularly sampled data. These operators, initially defined for 1D-signals, better approximate the underlying continuous nature of the world [START_REF] Asplund | Mathematical morphology on irregularly sampled signals[END_REF][START_REF] Asplund | Mathematical morphology on irregularly sampled data in one dimension[END_REF] than regular, discrete morphology.

Large amounts of 3D data, acquired by Mobile Mapping System, are available nowadays (Oakland [START_REF] Munoz | Contextual classification with functional Max-Margin Markov Networks[END_REF], KITTI [START_REF] Geiger | Are we ready for autonomous driving? The KITTI vision benchmark suite[END_REF], Rue Madame [START_REF] Serna | Paris-rue-Madame database: a 3D mobile laser scanner dataset for benchmarking urban detection, segmentation and classification methods[END_REF] datasets). Such data is of interest for a wide range of applications, for example: urban planning, documentation of cultural heritage, city modeling, among others [START_REF] Vosselman | Airborne and terrestrial laser scanning[END_REF]. 3D point clouds represent discrete versions of real world scenes. The sampling is by construction irregular, as it depends on the distance from the surface to the device, the angle of the surface, occlusions, vehicle speeds, etc. Thus, such data are particularly suitable to illustrate the potential of irregular morphological operators (i.e. operators on irregularly sampled data).

Previous morphological approaches to the problem of segmenting urban scenes have made use of regular, discrete morphology, by projecting the point cloud onto a regular grid [START_REF] Serna | Detection, segmentation and classification of 3D urban objects using mathematical morphology and supervised learning[END_REF][START_REF] Serna | Segmentation of façades from urban 3D point clouds using geometrical and morphological attribute-based operators[END_REF]. A disadvantage of this approach is that the segmentation depends on the coarseness of this grid (since nearby points can no longer be differentiated). Moreover, the projection loses information where the scene contains one part overshadowing another, for example where ground is occluded by a roof. A 3D morphological approach is proposed by Calderon and Boubekeur [START_REF] Calderon | Point morphology[END_REF], that requires the estimation of the underlying surface.

In this paper we introduce a framework for applying 2D, real-valued morphology to irregularly sampled signals, extending previous work by Asplund et al. [START_REF] Asplund | Mathematical morphology on irregularly sampled signals[END_REF]. We show that our proposed approach can be applied to 3D point clouds with promising results for segmenting point clouds of urban scenes.

Morphology on Irregularly Sampled Data

Morphology on irregularly sampled data is related to morphology on graphs. Najman and Cousty provide a nice survey of a large number of papers regarding this issue [START_REF] Cousty | A graph-based mathematical morphology reader[END_REF]. Here we propose a new approach to morphology on point clouds.

Irregular morphology in the specific case of rectangular structuring elements (SE) for 2D real-valued signals was dealt with in a previous paper [START_REF] Asplund | Mathematical morphology on irregularly sampled signals[END_REF] focusing mostly on regularly sampled input. Moreover, the previously presented method has some limitations regarding the sampling of the transformed signal. The main issue is that edges end up being "fuzzy" because the number of samples that are generated near edges is too low. A proposed solution is to iterate operators. In this section we present a similar idea for irregular morphology, which however does not require such iterations.

Morphology on irregularly sampled signals has a number of advantages. For one, in the regular case, the SE depends on the sampling grid, i.e. a continuous shape must first be discretized, which creates artifacts. This is illustrated on a regularly sampled image in Fig. 1, where an input image is slightly blurred (Gaussian kernel with σ = 1.0 pixels) in order to remove high frequencies and then subsampled by taking every third pixel in every third row. The subsampled image is then dilated with a small disk (r = 2 pixels in the subsampled grid) using both regular, discrete morphology, as well as the proposed irregular approach detailed below. The figure also shows the result of applying a regular, discrete dilation to the original, blurred image (before subsampling), with a SE of corresponding radius. As can be seen, the small disk, after discretization, ends up looking more like a diamond. It is also worth noting that irregular morphology has a number of advantages over regular, discrete morphology when it comes to approximating continuous morphology on images in the discrete domain [START_REF] Asplund | Mathematical morphology on irregularly sampled signals[END_REF]. However, this is outside the scope of this paper.

We investigate the applicability of the proposed irregular morphological operators to point clouds. This seems a natural application, since they are naturally irregularly sampled.

Dilations of Irregularly Sampled Signals

The basic idea behind the proposed algorithm is to sample two disks at each input point. One with radius r, and one slightly larger with radius r + ε. Additionally, for each of these samples, examine whether a disk of radius r + ε placed at any other input sample at a larger z-value overlaps this sample, and if so, discard it. The remaining samples from the larger disk are then projected downward, until they hit a disk (of radius r) placed at a lower sample, or falls through to -∞. The remaining samples of the smaller disk are kept, defining a border between transitions in the value of the dilation. Here ε defines the width of this border.

We will use B x to denote the set B translated by x, i.e. B x = {b+x | b ∈ B}. To perform a dilation by a disk of radius r of an irregularly sampled signal, we propose the following algorithm:

Choose some small ε > 0. For a disk shaped structuring element of radius r

D(r) = {(x, y) | x 2 + y 2 ≤ r} (1)
let Ď denote a sampled version of D. In this paper, we will only consider the case where the disks are sampled uniformly along the border with 8 samples, and a ninth sample at the origin. This means that smaller circles are sampled more densely than larger ones. This keeps the number of samples down and larger SEs lead to larger plateaus, which probably do not need as many samples. However, less naive ways of sampling the disk are obviously of interest, but outside the scope of this paper.

To dilate some input I = {(x i , y i , z i )} i∈{1,2,...,N } containing N samples at positions (x i , y i , z i ) do the following:

1. Let I 0 δ = ∅. Here I 0 δ is empty, but will be filled with samples from the dilation as we process the samples in I. 2. In this step we iteratively select an unprocessed sample (x c , y c , z c ) ∈ I and mark it as processed. This is analogous to sliding the SE over the image. In Fig. 2 the blue X is the sample being processed. 3. For each sample (x, y) of the translated disk Ď(xc,yc) (r), and each sample The shaded area illustrates the part that is covered by a disk at a higher level. The samples that are in this shadow are discarded (blue and red stars as well as the blue X). The remaining samples are added to the dilation. This approach is similar to the paper by Asplund et al. [START_REF] Asplund | Mathematical morphology on irregularly sampled data in one dimension[END_REF].

(x i , y i , z i ) ∈ I \ {x c , y c , z c }, such that z i ≥ z c ,
(x, y) (see the red star in Fig. 2). For each remaining sample (x, y) and each (x i , y i , z i ) ∈ I, such that z i < z c , let H(x, y) be the set of values of samples for which the distance between (x, y) and (x i , y i ) is at most r (see the red circle and black solid disks in Fig. 2), i.e.

H(x, y) = {z i | (x i , y i , z i ) ∈ I ∧ z i < z c ∧ (x -x i ) 2 + (y -y i ) 2 ≤ r} (2)
Let z h (x, y) = H(x, y). Conceptually, z h is the value of a given sample (x, y) of the bigger disk, after projecting downward until hitting a disk at a lower value (or dropping out to -∞). See red circle in Fig. 2.

Finally, let

Dc (xc,yc) (r) = {(x, y, z c ) | (x, y) ∈ D(xc,yc) (r)}, (3)
Dh (xc,yc) (r + ε) = {(x, y, z h (x, y)) | (x, y) ∈ D(xc,yc) (r + ε)}, and (4)
I i+1 δ = I i δ ∪ Dc (xc,yc) (r) ∪ Dh (xc,yc) (r + ε) (5)
I.e. add the samples spawned by the processed sample to I i+1 δ , where i is the number of previously processed samples. These steps are then iterated until there are no unprocessed samples left.

Then I N δ is the dilated signal. That is, I N δ is the irregular dilation of I by a disk shaped structuring element of radius r. The parameter ε controls the distance between samples where translated disks overlap. This parameter should be small (as a rule, at least smaller than the distance between samples in the input). Figure 3 illustrates an example of applying this algorithm to an input containing five irregularly sampled points when seen from above. The samples that fall in the shadow of any SE placed at a higher sample are discarded. The remaining samples from the larger SE are projected down until they hit a SE at a lower input sample (or until they fall through to -∞). Here the SE samples that fell under a large disk at a higher location (i.e. one of the disks in the third figure) have been suppressed, and the samples of the larger disks have been projected down, helping define the border between the sampled disks (see for example the bottom right of the last figure).

Erosions, Openings, and Closings

To get the erosion ε(I) of an input signal I, one can simply compute the dilation δ(I ), where

I = {(x i , y i , -z i ) | (x i , y i , z i ) ∈ I}, the erosion is then δ(I ) .
Openings and closings can be formed by composition [START_REF] Serra | Image analysis and mathematical morphology[END_REF].

Irregular morphology for point clouds

Figure 4 shows some examples of different morphological operators applied to a simple point cloud depicting a signpost. The signpost is first eroded by a disk of radius 0.016m, followed by a dilation with the same SE (yielding an opening). The result is interpolated back onto the original input positions, and then subtracted from the input, resulting in the tophat, which has enhanced the thin pole holding the sign. The value of ε is set to 10 -6 m in all these cases.

We use nearest neighbor interpolation for a few reasons: Firstly, for the sake of simplicity. Secondly, empirically this seems to give good results. Finally the morphological operators will, in general, create cusps when applied to a signal, even if the input is smooth [START_REF] Asplund | Mathematical morphology on irregularly sampled signals[END_REF]. Therefore, a smooth interpolation of the opening is problematic. Some other possible candidates would be linear interpolation, or natural interpolation [START_REF] Sibson | A brief description of natural neighbour interpolation[END_REF], however in some preliminary testing, these seem to give worse results for our data.

An interesting property of the irregular operators is their tendency to homogenize the sampling density. The sampling of the ground in the input is less dense further from the camera (since these samples are further from the sensor). The eroded point cloud, on the other hand, has a similar sampling density close to, and further away from the camera. In essence, this behavior stems from not sampling inside the disks (except for one sample in the center) thereby reducing the sampling density where it is high (since disks are likely to overlap many nearby samples) accelerating further processing, while increasing the density where it is low (since one input sample gives rise to many output samples and there are fewer neighboring samples in the input that are likely to be suppressed). This homogenization may better define surfaces, enabling more accurately detecting object boundaries. Moreover, homogenizing the density introduces an implicit normalization of features and a more robust feature description, avoiding the excessive contribution of high density areas.

Application to Urban Point-Cloud Segmentation

As previously mentioned, the problem of segmenting point clouds of urban scenes has already been addressed using morphology on elevation images. However, using elevation images has some drawbacks. The images being projected onto a regular grid means that the segmentation will depend on the coarseness of said grid. Furthermore there is a question about how to deal with holes in the projected image. Thus, this problem seems like a natural candidate for applying irregular morphology.

We will consider a problem where the goal is to segment point clouds of urban scenes, to illustrate the applicability of our proposed method. In order to segment the point cloud, we first focus on separating out the facades and objects, followed by a refinement which yields the final segmentation. Figure 5 shows a diagram outlining the workflow.

We make use of point clouds from the Paris-rue-Madame dataset 5 [10], containing two point clouds from a 3D Mobile Laser Scanning (MLS) of a section of a street in Paris. Apart from the 3D-location, each point is also equipped with three additional pieces of data: reflectance, label, and class. However, we only use the position of the points for segmenting. We use the provided classes to test our segmentation. The reflectance and label information is ignored.

Tophat

An initial preprocessing step will be applied to the entire point cloud, the purpose being to flatten out the surface, removing large-scale variations in height (e.g. slopes and hills), while preserving local height differences (e.g. cars, pedestrians, facades). This preprocessing consists of a tophat applied to the point cloud. First we apply an opening to the set of points (by applying an erosion followed by a dilation), the result is then subtracted from the original signal (via interpolation as described in Sec. 3).

The size of the structuring element used by the opening controls the size of the support of height variations to be preserved. Based on the size of objects of interest one can make a decision on the radius of the SE. In this case, our objects of interest are on the scale of cars or smaller (we also capture facades, which are generally quite thin). Therefore, we set the radius of the disk to be 1.5m, since a disk of this size cannot fit inside a normal vehicle. Figure 6 shows an example of the result of taking the tophat of a point cloud from the dataset [START_REF] Serna | Paris-rue-Madame database: a 3D mobile laser scanner dataset for benchmarking urban detection, segmentation and classification methods[END_REF].

Segmenting Facades

Facades are initially segmented by thresholding the tophat at a large value h facade . We are assuming the highest objects in the scene are house facades. If this does not hold, one could use some more sophisticated feature, such as considering the elongation of objects [START_REF] Morard | Geodesic attributes thinnings and thickenings[END_REF][START_REF] Serna | Segmentation of façades from urban 3D point clouds using geometrical and morphological attribute-based operators[END_REF], or learned features, but for our purposes, this assumption is fine. The value of h f acade is not very sensitive. The important part is that the threshold contains the top part of facades. We simply fix this value to be 5 meters. This initial segmentation fails to include a large number of points that are actually parts of facades, however, it contains very few false positives (very tall objects might be included, e.g. lampposts, but they are easily eliminated). The purpose of this initial segmentation is to localize the facades. Following this localization, we can find candidate facade points by thresholding the result of the tophat at a lower value h f acade as described in Section 4.5. These candidates are then filtered based on their distance (in 2D) from the initially selected facade points.

Specifically, let This makes thresholding for segmentation feasible. The tophat contains some negative values (points below the plain), which stems from interpolation artifacts. These negative points happen mostly inside buildings, where the sensor has sampled parts of the insides of buildings so that it has some samples from rooms on top of each other.

F h facade = {(x, y, z) ∈ I | z > h facade }, F h facade = {(x, y, z) ∈ I | z > h facade } (6)
where I is the set of input samples and F h facade denotes the initial segmentation of the facades associated with h facade . The refined segmentation F is given by the following set of points:

F = {(x, y, z) ∈ F h facade | d((x, y, z), F h facade ) < ε f } (7)
here ε f > 0 is a parameter that controls the distance that the initial facade labels can be propagated. For our experiments we use ε f = 0.05m. The function d denotes the minimum 2D-distance between all points in a set and some sample:

d((x, y, z), F ) = min (x ,y ,z )∈F (x -x ) 2 + (y -y ) 2 (8)
By restricting the potential facade points to those that are near the initially selected set of points, we can rule out a lot of false positives, meaning that we can push the threshold value h facade close to 0, i.e. close to the height of the ground, even though the number of ground points that are above the threshold increases as h facade approaches 0. Thresholding at h f acades (for reasonable values) will clearly find points that are parts of objects too (e.g. pedestrians), since the lower parts of facades are at the same height as objects. However, such points will generally not be immediately next to facades (although it is possible) and are therefore discarded, since they are too far (≥ ε f ) from the initial facade points. The remaining points are labeled "facade". Figure 7 illustrates this process.

Segmenting Objects

We now focus on the objects, i.e. everything that is neither facade nor ground. At this point, we only need consider the non-facade points. Again, we follow the procedure outlined in the previous section. Objects are initially segmented by selecting a threshold h object . Again, this initial threshold is set at a height such that there are few non-object points above it, but many object points fall below it. To remedy this, a threshold of h object is selected (see Section 4.5) and applied to the remaining points. Finally, points that are too far from the initial set of object points (in 2D) are discarded (as in Eqn. 7 with appropriate substitutions). The remaining points are labeled "object".

Segmenting Ground and Refining Labeling

The points that are not labeled "object" or "facade" are labeled "ground". However, there may be some points that are close to objects and facades that are mislabeled as "ground," since the thresholds h facade and h object cannot be set too close to ground level without introducing noisy labels close to the borders between ground and not ground. An additional step to refine the segmentation is performed by finding the internal gradient G i [START_REF] Rivest | Morphological gradients[END_REF] of the set of points, G, labeled ground. I.e. G i = G -ε r (G), where ε r denotes the erosion with a disk shaped structuring element of radius r > 0. Here r is set to a small value which depends on the average minimum distance from any ground point p to the rest of the ground points G \ {p}, i.e.

r = C |G| p∈G d(p, G \ {p}) (9)
where C ≥ 1 is a constant increasing the radius to give more context (we fix C = 10). In other words, this radius depends on the point cloud density. We want it to be small, in order to get a good approximation of the internal gradient, but it cannot be too small, since this would essentially lead to each point being looked at in isolation, which is a problem, since the gradient obviously requires information about values at nearby points. The internal gradient G i enhances the edges, which are the ground points that are likely mislabeled. A final threshold h edge is chosen as described in Section 4.5. The set of points above this threshold E, are then used instead of F h facade and F h object (see Eqns. ( 6) -( 7)). As a result, more points close to the ground near edges can be correctly labeled, though this may lead to a small number of ground points being mislabeled.

Choosing the Threshold Value

For the upper thresholds (i.e. h f acade and h object ) we simply choose values by hand, based on the height of objects that we are interested in. The method is not particularly sensitive to the choice for these values. The main concern is to set values that capture the top part of facades and objects respectively. In this paper the values used are h f acade = 5m, and h object = 0.5m. I.e., we are assuming that facades are higher than 5 meters, and objects are at least half a meter tall. For the lower thresholds (i.e. h f acade h object and h edge ), ideally the value would be 0, but even after leveling the scene using the tophat, the ground is still not perfectly flat. For simplicity, we set the values of h f acade , h object , and h edge to the same value of 0.4m. One may select a threshold automatically by examining the histogram of heights of the tophat for values between 0 and h object . The histogram contains a large peak around the height of the ground, which can be used to set the other thresholds. It is possible to push the threshold value down, thus improving the labeling of the lower parts of objects and walls, at the cost of a somewhat noisier ground segmentation.

Results

The top of Figure 8 shows the segmentation result for GT Madame1 2 [START_REF] Serna | Detection, segmentation and classification of 3D urban objects using mathematical morphology and supervised learning[END_REF] for different sampling densities of the input (achieved by discarding every n:th sample). The result seems reasonably robust to changes in sampling density. To quantify the performance for different densities we consider the percentage of unchanged labels as more and more input samples are discarded (see bottom left of Fig. 8).

Additionally, we make use of the class labels provided in the dataset [START_REF] Serna | Detection, segmentation and classification of 3D urban objects using mathematical morphology and supervised learning[END_REF] to create a ground truth for our labeling by combining the class labels for facade, wall light, balcony plant, and wall sign into a single class label for the walls. Similarly, the ground label is comprised of the background class label and the ground class label, the remaining classes are considered objects. We then look at the percentage of correctly labeled samples for each of these three classes as the sampling density of the input is decreased (bottom right of Fig. 8).

Discussion and Conclusions

This is a first attempt at applying irregular morphology to 3D point cloud analysis. A previous morphological approach [START_REF] Serna | Detection, segmentation and classification of 3D urban objects using mathematical morphology and supervised learning[END_REF][START_REF] Serna | Segmentation of façades from urban 3D point clouds using geometrical and morphological attribute-based operators[END_REF] to the problem studied in this paper first projected the point cloud onto a regularly sampled range image, which causes issues where multiple points in the point cloud correspond to the same pixel in the range image. Working directly on the irregular domain means that there is no need to resample onto a regular grid, thus avoiding this problem. It also allows for morphological operators of any scale, unrestricted by any grid.

We argue that another possible benefit of this approach is the homogenizing tendency of the proposed operators on the point cloud density. This can speed up further processing, since oversampled parts of the signal become sparser, while better defining the surface where the density is low, which may allow for more accurate detection of object boundaries.

In experiments we show that the proposed method is relatively robust, yielding similar labels for the remaining samples as we decrease the input sampling, thus keeping the percentage of correctly labeled samples high even as the density drops to a hundredth. The most robust class is ground, which is to be expected, since large and smooth surfaces do not require a large amount of samples.

Worth noting is that the proposed approach is slower than regular, discrete morphology. Computation time depends on a number of factors, including point density, size of the SE, and size of the input. Processing inputs on the order of 10 6 samples, using a k-d tree to find neighbors, takes seconds for smaller SEs (on an Intel Xeon 3.40GHz), but may take several minutes if the SE is large (e.g. in the case of the large tophat used in our experiments), though further processing is normally faster, since the density is likely decreased, as previously described.

In the future, we would like to improve the sampling strategy. Instead of generating the same number of tentative output samples at each input sample, one could let this number depend on local properties of the signal (e.g. smooth-ness). Allowing for more general shapes of SEs is also of interest. This may also have an affect on the sampling strategy (e.g. pointy parts of the SE may need a denser sampling, while smoother parts can be sampled more sparsely). With regard to the segmentation approach, comparing our results should be the next step. It would be interesting to take into account some other features. Local thresholding could be useful. Finally, a postprocessing step could likely further improve the segmentation (e.g. via some regularization method).

Fig. 1 .

 1 Fig. 1. From left to right: input image, regular dilation on sampled input, irregular dilation on sampled input, and regular dilation on non-sampled input.

Fig. 3 .

 3 Fig. 3. A dilation of an irregularly sampled signal containing five samples (illustrated by crosses), by a disk shaped structuring element (solid black line). The first figure shows a SE placed on top of each input sample. The dashed circles illustrate the larger disk in step 3 (with radius r + ε). The second figure shows the sampled SEs (i.e. Ď(r)). The third figure shows the sampled larger disk (i.e. Ď(r + ε)). The last figure shows the final result (i.e. I N δ ).Here the SE samples that fell under a large disk at a higher location (i.e. one of the disks in the third figure) have been suppressed, and the samples of the larger disks have been projected down, helping define the border between the sampled disks (see for example the bottom right of the last figure).

Fig. 4 .

 4 Fig. 4. Examples of irregular morphology applied to a simple point cloud of a signpost.Here, the SE is a disk with radius 0.016m.

Fig. 5 .

 5 Fig. 5. Workflow diagram of the proposed algorithm. The output is a segmented point cloud with labels: ground, facade and object.

Fig. 6 .

 6 Fig.6. The first row shows the tophat (right) of the input (left). Also shown is a horizontal plane in purple that is approximately at ground level for the input figure, and at 0 for the figure of the tophat, to illustrate slope. Rows two and three show the same point clouds from the side. Notice that the slope of the input has been removed in the tophat (bottom), and large scale variations in height have been suppressed. This makes thresholding for segmentation feasible. The tophat contains some negative values (points below the plain), which stems from interpolation artifacts. These negative points happen mostly inside buildings, where the sensor has sampled parts of the insides of buildings so that it has some samples from rooms on top of each other.

Fig. 7 .

 7 Fig. 7. The left figure shows the initial facade segmentation in dark blue (other colors indicate height). On the right: points labeled facade after the process in Sec. 4.2.

Fig. 8 .

 8 Fig. 8. Top: Labeled scene for different point cloud densities. From left to right and top to bottom, we have: 10 7 , 10 6 , 10 5 , 10 4 samples. Red indicates object, blue indicates ground, and black indicates facade. Bottom left: The percentage of samples in the labeled point cloud, per class, that have changed when comparing less densely sampled inputs to the most densely sampled input. Bottom right:The percentage of correct labels for each class and different point cloud densities. The behavior of the curves for the ground label is explained by its simple nature. In general, the ground is relatively flat and does not need many samples to be well represented.

  Illustration of the algorithm from a side view. The disk shaped SE is located on top of each sample. The red circle is a sample (x, y) from D(r + ). The black dashed disks do not belong to H since they are too far from (x, y) (and one of them is above zc). Other two disks, placed below zc, belong to H. z h is found by projecting down until hitting the highest disk in H. In case no disk is hit, such a sample is discarded.

	}
	Fig. 2.
	check if the distance from
	(x, y) to (x i , y i ) is at most r + ε. If so, discard (x, y) (see the blue star in Fig. 2). Let D(xc,yc) (r) denote the remaining set of samples (blue circle). I.e.
	discard samples from the smaller disk that are overlapped.
	4. Similarly to the previous step, for each sample (x, y) of the translated disk
	with radius r + ε, and each sample in I \ {x c , y c , z c } no lower than z c ,
	check if the distance from (x, y) to (x i , y i ) is at most r + ε. If so, discard

c 2014 MINES ParisTech. MINES ParisTech created this special set of 3D MLS data for the purpose of detection-segmentation-classification research activities.

Acknowledgement. Teo Asplund was funded through grant 2014-5983 from the Swedish Research Council.


Mark Prendergast
email: mark.prendergast@tcd.ie

Páraic Treacy
email: p.treacy@brighton.ac.uk

Niamh O'meara
email: niamh.omeara@ul.ie

Teachers' Perceptions of Using Incentives in State Examinations to Increase the uptake of Higher Level Mathematics

Keywords: Education policy, state examinations, incentives, teacher perceptions

Background to the Study

In Ireland, all students sit a summative state examination known formally as the Leaving Certificate (LC) at the end of Senior Cycle (upper secondary level). The LC acts as a gatekeeper to tertiary level education with students awarded points based on their six best graded subjects. Different points are also awarded pending the level of which the subject was studied. For example, the majority of subjects have two levels. The most challenging level is Higher Level (HL -often referred to as Honours) and the next level down is Ordinary Level (OL -often referred to as Pass). Since 2012 mathematics has been assigned a special status within the Irish post-primary school curriculum with the introduction of a Bonus Points initiative (BPI). Students are now awarded an extra 25 points in their LC examination results if they achieve a passing grade at HL. The awarding of these extra points was introduced to encourage students to study mathematics at HL. Such incentive schemes for studying mathematics and improving student performance in the subject are not a new phenomenon. Similar schemes exist in Australia and Israel [START_REF] Treacy | Incentivizing advanced mathematics study at upper secondary level: the case of bonus points in Ireland[END_REF].

In Ireland while the numbers opting for HL mathematics have nearly doubled since the introduction of the BPI, anecdotally there have been many mixed reviews. Hence, while there have been calls in some quarters to extend this initiative to other subjects, it is important that the impact of the BPI on the study and assessment of mathematics is first appraised. This paper investigates the advantages and disadvantages associated with the BPI from the perspective of mathematics teachers (n=266) and aims to address the following research question: What are mathematics teachers' perceptions of the Bonus Points Initiative as an incentive in state examinations?

Methodology

The methodology of this study involved the distribution of a questionnaire to a representative sample of HL mathematics teachers in Irish post-primary schools (n = 723). The instrument was designed specifically for this study and was developed by the authors with the assistance of an advisory group involving five experienced post-primary mathematics teachers. It comprised of four main sections. Section 4, which is the focus of this submission, inquired of teachers' perceptions of the BPI through a series of multiple choice and open-ended questions. The finalized questionnaires were distributed (two to a stratified sample of 400 post-primary schools) in early April 2018. The response rate was 266 second level teachers (approx. 33%). The open-ended questionnaire data was analysed thematically in two stages. Firstly, the first coder coded the two sets of data (perceived advantages and disadvantages) himself two weeks apart, and the intra-coder agreement percentages were equally high (> 90%). The second coder subsequently coded around a third of each dataset and once again the inter-coder agreement percentages were > 90% for the perceived advantages and disadvantages, highlighting that the coding frameworks that we developed for this study's analysis were reliable.

Findings and Discussion

Findings from both the qualitative and quantitative data in this study indicate that teachers hold many mixed opinions regarding the BPI. While 46% of respondents indicated that they agree with the BPI, the rest were either not sure or did not agree. The main advantage of the initiative was that it increased uptake and perseverance of mathematics at HL. However, while the numbers taking HL may have increased, respondents noted some concerns about the standard of students now persisting at HL. A greater proportion of students that would have typically attempted the OL examination, were it not for the BPI, were now attempting the HL examination and thus there was an increase in those achieving low grades (SEC 2015). There were also fears that the LC HL mathematics examination would be 'dumbed down' to cater for the surge in students now taking it. It was feared that "Students of a stronger mathematics ability may not be extended to the same degree". However, while these concerns are real, there were also many advantages of the BPI which were noted by the participating teachers. The second most cited benefit was that it rewarded students for the time and effort required for the course. Concern has long been expressed at the workload and timeframe of the LC HL mathematics curriculum in Ireland (Cosgrove et al., 2013). In addition to the workload and timeframe, a study carried out by [START_REF] Smyth | From Leaving Certificate to leaving school: A longitudinal study of sixth year students[END_REF] found that mathematics is considered to be among the top five most difficult LC subjects. Thus the BPI rewards students for their exertions and supports the view that mathematics is not 'just another subject' [START_REF] Mcdonagh | Maths national competitiveness: a discussion document submitted to the national competitiveness council[END_REF].

However, the findings of this study indicate that while there are clear advantages to the special status that the BPI has given mathematics within the Irish post-primary school curriculum, there was also a number of interlinking disadvantages which must be considered if such incentive schemes are to be maintained, and indeed expanded.