text:
Ben Davies
email: b.m.j.davies@lboro.ac.uk

A case for a new approach to establishing the validity of comparative judgement as an assessment tool for mathematics

Keywords: Assessment, comparative judgement, conceptual understanding, reliability, validity

, may address the concerns raised regarding previous research on the validity of comparative judgement.

If presented with two large bags of gold and invited to keep one, most people would have a strong preference. While unlikely to know the exact quantity in either bag, it is usually not difficult to identify which is heavier by comparison. Comparative judgement is an assessment tool that exploits this innate capacity for comparison [START_REF] Thurstone | A law of Comparative Judgement[END_REF], "The Law of Comparative Judgment") rather than isolated evaluation against specified criteria. This well-established principle of psychology forms the basis for an innovative assessment that offers an alternative to the criterion-based status quo. This paper is a theoretical discussion of investigations into the reliability and validity of this approach to mathematics assessment. This is followed by a detailed theoretical examination of validity, resulting in the suggestion that further work and a new approach is necessary to establish validity in a more substantive manner. While making few definitive claims regarding the nature of the new approach, a case study in assessing conceptual understanding is used to illustrate the potential of two approaches based on what [START_REF] Trochim | Research Methods: The Essential Knowledge Base (2nd Edition)[END_REF] refers to as predictive validity.

What is comparative judgement?

Comparative judgement relies on expert judges making a series of comparisons between two responses to a given question. Without any specific criteria, judges are simply asked to 'choose the better response'. Using an automated system to generate pairs of students' responses for comparison, comparative judgement generates a ranked order from 'best' to 'worst'. This ranked order, produced after each response has been compared to others several times, is based on the Terry Bradley model [START_REF] Pollitt | Comparative judgement for assessment[END_REF] that generates a standardised parameter estimate (z-score) representative of the relative quality of each response. In the absence of specific marking criteria, the rankings produced are "instead grounded in the collective expertise of the judges" (Bisson et al., 2016, p. 143), avoiding the necessity to explicitly define the target of the assessment.

Thematic Working Group 21

Proceedings of CERME10

A summary of the literature on Comparative Judgement

Before examining the literature on reliability and validity, this section begins with a discussion of two key benefits claimed in the comparative judgement literature.

Benefits, motivations and intentions

Comparative judgement facilitates the targeting of specific aspects of mathematics without having to embed a precise definition into a marking schedule [START_REF] Pollitt | Comparative judgement for assessment[END_REF]. For example, [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF] used comparative judgement to investigate conceptual understanding in undergraduate mathematics. This "important but nebulous construct" (Bisson et al. 2016, p. 143) has been the topic of much debate. [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF] claim experts are capable of recognising examples of conceptual understanding, but find it difficult to generate comprehensive, reliable scoring rubrics. As suggested by [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF], by grounding the definition of an area of assessment solely in the collective understanding of judges, this problem is at least partially avoided. Assessment of conceptual understanding will be returned to in a later section of this work. [START_REF] Jones | Assessing Mathematical Problem Solving Using Comparative Judgement[END_REF] also argue for the need to "free assessment designers from restricting tasks to those that may be easily scored by conventional means" (Jones et al. p. 170). They claim this would allow for the introduction of components that more adequately reflect the assessments purported purpose. A study involving four experienced GCSE (General Certificate of Secondary Education for England, Wales and Northern Ireland) exam writers producing an exam explicitly for comparative judgement marking found that exam writers experienced greater freedom to write more open-ended questions providing students with the freedom to experience more genuine problem solving situations [START_REF] Jones | The problem of assessing problem solving: can comparative judgement help[END_REF]. Moreover, this freedom was successfully used, as evidenced by the survey responses of 23 teachers who evaluated the suitability of this exam script. Examinations of this type have been long sought-after but have seldom materialised due to the need for a demonstrably reliable and valid tool for assessment [START_REF] Jones | The problem of assessing problem solving: can comparative judgement help[END_REF].

These benefits, particularly that of eliminating assessment criteria, raise questions of reliability and validity. An assessment based on such apparently subjective foundations might initially attract substantial skepticism. These questions are the focus of the following two sections.

Reliability

In the context of comparative judgement, reliability has universally been used to refer to the notion of repeatability across different judges and judging communities. This is often referred to as interrater reliability.

There is a growing collection of case studies demonstrating high inter-rater reliability in the use of comparative judgement. [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF] demonstrated high inter-rater reliability in the contexts of conceptual understanding of entry-level undergraduate statistics (p-values), undergraduate pure mathematics (derivatives) and elementary school-level algebra (using letters in mathematics). Using a split-half technique in each of the three studies, judges were divided into two groups of equal size. As a result of the computerised method by which judges make comparisons, this split-half comparison was repeated 20 times in each study without requiring any additional comparisons from the judges. Pearson's correlation coefficients were computed for each iteration of each study, the results of which are summarised in Table 1. Jones, Swan and Pollitt ( 2015) produced results consistent with the above in a problem solving context. When applied to both traditional GCSE examination scripts and purpose-written open-ended examination scripts, comparative judgement performed well when implemented by a diverse group of 23 judges, consisting of ten GCSE examiners, one non-GCSE examiner, seven mathematics education lecturers, two researchers, one research student and two advisors. When separated into two groups (of 12 and 13), the Rasch sample separation reliabilities, often considered similar to Cronbach's  (e.g. [START_REF] Wright | Rating scale analysis: Rasch measurement[END_REF] were .80 and .93, indicating acceptably high internal inter-rater reliability for each group. Jones et al. (2015) also report a Pearson product-moment correlation coefficient of .87, providing further evidence for the inter-rater reliability of comparative judgement when assessing problem solving.

Participants

In every reviewed study on the reliability of comparative judgement to-date, the results have positively indicated that comparative judgement performed reliably across judging communities.

Validity

Consistent with the majority of studies in the comparative judgement literature this paper adopts the broad definition that a valid assessment measures that which it purports to measure [START_REF] Koretz | Measuring up: What educational testing really tells us[END_REF]. Most investigations of the validity of comparative judgement have focused on what [START_REF] Trochim | Research Methods: The Essential Knowledge Base (2nd Edition)[END_REF] refer to as convergent validity, based on comparison with traditional measures of mathematical achievement. [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF] has taken a different approach, by investigating content validity, based on comparisons with other psychometrically validated instruments.

A measure is said to have convergent validity if it produces results similar to others that should theoretically be similar [START_REF] Trochim | Research Methods: The Essential Knowledge Base (2nd Edition)[END_REF]. On investigating assessment of problem solving, [START_REF] Mcmahon | A comparative judgement approach to teacher assessment[END_REF] reported "comparative judgement outcomes correlated as expected both with test marks and with existing student achievement data, supporting the [convergent] validity of the [comparative judgement] approach" (p. 368). [START_REF] Jones | The problem of assessing problem solving: can comparative judgement help[END_REF] reported a correlation of .86 between students' GCSE marks and their parameter estimated z-score based on a comparative judgement assessment, providing further evidence in this direction.

Finally, [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF] investigated both convergent validity and content validity across all three studies on which they report; see Table 2. The former was evaluated with methods similar to those above, analysing correlations between traditional student achievement data and comparative judgement results. Content validity, a comparison of the measure against the relevant content domain [START_REF] Trochim | Research Methods: The Essential Knowledge Base (2nd Edition)[END_REF], was based on correlation analysis between comparative judgement results and other psychometrically validated measures of conceptual understanding; see column two of Table 2. It should be noted that the choice of topics (p-values, derivatives and algebra) in [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF] were "driven by the existence of validated instruments to measure conceptual understanding of those topics" (p. 144). As is discussed later, the absence of such instruments for most topics in mathematics appears to serve as a significant barrier for this line of research. For the list of topics available [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF] conclude that comparative judgement is a valid measure of conceptual understanding. Jones and Inglis (2015) also investigated content validity of exam scripts. They asked experienced experts to analyse the content of the exam scripts produced when writers were freed from the constraints of traditional assessment criteria. Their qualitative analysis resulted in the assertion, consistent with their hypothesis, that these exam scripts placed more emphasis on conceptual thinking than their GCSE counterparts. It should be noted that this aspect of the study was focused on the exam scripts themselves, not on students' responses.

The evidence in this section suggests that comparative judgement is likely to perform at least as well as established tools for mathematics assessment in many or even most contexts. Moreover, [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF] provide evidence of content validity in three specific mathematical domains. As discussed in the final section, the scope for expansion of the argument for content validity is limited, given the absence of other psychometrically validated measures. The following section focuses on the necessity for a complementary approach to establishing the validity of comparative judgement.

The necessity for a different approach to validity

This section has two aims. First, this section establishes the necessity for an approach other than the convergent validity-based studies discussed above. Second, this section discusses the limitations of the content validity approach proposed by Jones et al. (2015) as a suitable answer to the shortcomings of convergent validity discussed above.

Assessing the validity of comparative judgement only by comparison with traditional assessment appears to significantly limit the argument for comparative judgement. While arguments regarding efficiency [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF]) and the freedom to write examinations with a different mathematical focus [START_REF] Mcmahon | A comparative judgement approach to teacher assessment[END_REF] still stand, proponents of comparative judgement claim it to have the potential to assess something fundamentally different from that assessed by traditional approaches. [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF], for example, argue that comparative judgement can be used to assess 'conceptual understanding', an aspect of mathematics known to be difficult to assess using traditional assessment. However, in establishing the validity of comparative judgement in this area, they refer to comparisons with traditional assessment data. This approach to validity appears problematic from the outset as it limits a successful measure of validity to something not worse than traditional assessment; an approach claimed to be ineffective in assessing their domain of interest. Jones et al. (2015) note similar reservations pointing out that traditional student achievement data, often compared with comparative judgement, can be criticised for its overly-procedural focus. Jones et al. (2015) go on to assert the need for a different approach, pointing to comparisons with 'other psychometrically validated measures' as a satisfactory solution.

further analysis of the limitations of convergent validity

A key aspect of the argument for comparative judgement is that it facilitates reliable assessment of areas previously difficult to assess. Thus it seems logical to believe that comparative judgement, when explicitly focused on assessing those areas (e.g. conceptual understanding) should reward different abilities/strengths than those rewarded by traditional assessment.

Taking this reasoning to its logical conclusion results in the assumption that some students should perform better under comparative judgement than under traditional assessment. Thus it seems at best limited to evaluate a new assessment tool by its ability to reproduce the results of traditional assessment. Consider a hypothetical student with a particular aptitude for memorising procedures but difficulty with the conceptual understanding underlying and drawing together important ideas. Based on previous critiques of traditional criterion-based approaches, one might assume this student would perform well on such an assessment. However, one might hope that this student would score poorly on comparative judgement assessment given the claims of [START_REF] Bisson | Measuring Conceptual Understanding Using Comparative Judgement[END_REF]. Hence, arguing for the validity of comparative judgement by pointing to its comparability with traditional assessment seems inherently limited.

Establishing convergent validity has made a significant progress toward gaining legitimacy and the credibility to justify further, more detailed work. However, it seems that a complementary direction is necessary in establishing other aspects of validity, thereby further developing the argument for the adoption of comparative judgement.

The limitations of content validity

As noted by Jones et al. (2015), making comparisons with alternative psychometrically-validated instruments appears to be an obvious and fruitful way forward in establishing the validity of comparative judgement. However, the absence of such instruments in most domains serves as a substantial barrier to progress. Returning to conceptual understanding, the absence of such instruments in this realm is well-documented and is a significant reason Bisson et al. limited their study to just three content domains.

It is theoretically possible to develop more such instruments. However, the number of instruments will always be far fewer than the number of mathematical domains in need of assessment. At present, the content validity approach relies on inherently inductive reasoning. Evidence supporting comparative judgement in a small number of domains has little relevance for the new and previously untested. If comparative judgement is to be established as a viable assessment tool for a greater range of domains, I further claim that yet another approach is necessary.

The final section below seeks to identify a way forward that is cognizant of the issues raised above.

Predictive validity through the case of conceptual understanding

Having argued the necessity for something new, this final section offers a plausible approach attempting to avoid at least some of the above criticisms. This new approach, predictive validity, is illustrated here by further considering the case of conceptual understanding. It should be noted that this is just one such possible solution that, like previous sections, is intended to raise more questions than it answers. It too is not without its flaws.

As defined by [START_REF] Trochim | Research Methods: The Essential Knowledge Base (2nd Edition)[END_REF], predictive validity refers to a measure's ability to predict something it should theoretically be able to predict. If one is to accept conceptual understanding as fundamental to mathematical development (NCTM, 2000;Ofsted, 2008;[START_REF] Rittle-Johnson | Developing conceptual understanding and procedural skill in mathematics: an iterative process[END_REF], a valid measure of conceptual understanding should serve as a predictor of other mathematical success. In the absence of suitable comparative measures for evaluating content or convergent validity, predictive validity appears a logical alternative. This section addresses two possible such approaches that may aid in establishing the validity of comparative judgement.

First, comparative judgement could be evaluated as a predictor of performance on traditional assessment. Allowing for the passage of time, moving away from an immediate comparison with traditional assessment (see convergent validity) creates the opportunity for a student's general mathematical maturity and understanding to progress. This argument hinges on the assumption that conceptual understanding is so fundamental to mathematical development that it should act as a future predictor of mathematical performance in general, even on assessment for which conceptual understanding is not an explicit focus. If one is to accept this assumption regarding conceptual understanding then those that perform well on a comparative judgement assessment of conceptual understanding at time 1, should perform well on any assessment at time 2.

This approach may be criticised by those who argue that if an attempt to avoid immediate comparison with traditional assessment results in a delayed comparison with the same traditional assessment, then nothing has been gained by analysing predictive over convergent validity.

Another approach to predictive validity is to seek other measures of mathematics success not beholden to traditional, criterion-based assessment. Such measures of mathematical success may appear hard to come by. However, metrics such as success at postgraduate level, or success on relevant career pathways may provide an interesting solution. While arguably somewhat crude, these measures may provide useful sources of comparison. Using such measures, one could compare comparative judgement z-scores from prior years with success on these future measures to evaluate the predictive validity of comparative judgement here. For example, in an attempt to evaluate validity in the tertiary context, it would be possible to consider success in tertiary mathematics. By using comparative judgement to assess a series of tertiary exam scripts, it may be possible to evaluate predictive validity by comparison with future success in postgraduate studies or related mathematical careers. A significant correlation between comparative judgement rankings and future success would stand as strong evidence for the predictive validity of comparative judgement.

This second approach to predictive validity benefits from avoiding any comparison with traditional assessment. However, unsurprisingly raises many questions of its own, particularly regarding the quality and availability of data related to such future measures of success. Moreover, predictive validity still does not grapple with the ways that scores and interpretation of scores are used. Kane ( 2013) offers yet another approach, drawing on the general principles of construct validity (i.e., how well an assessment reflects the true theoretical measurement of a concept) to grapple with the assumptions underpinning the interpretation and use (IUA) of assessment scores, rather than the scores themselves. The implications of this 'argument-based approach' approach to validation of comparative judgment are unclear, but his detailed discussion of IUAs draws attention beyond the process of assessment in of itself to the use and interpretation of the assessment findings in the real world. For example, the evaluation of comparative judgment results could usefully require an evaluation of the consequences of the assessments' use, and negative consequences in some could render the use of the results unacceptable in some circumstances.

I do not claim to have covered the full range of criticisms that may be levied at the new approaches.

Nor do I claim predictive validity is the only response to the limitations of existing research discussed earlier. The intention of this section is to initiate a discussion regarding a new approach to validation of comparative judgement that might facilitate an extension of the argument for comparative judgement beyond the mathematical domains for which we have other tests.

Conclusions

This paper has been a discussion of an exciting and relatively new approach to assessment. In particular, the focus has been on the necessity for a new approach to evaluating the validity of comparative judgement. It should be noted this work does not advocate for the abandonment of one approach in favour of another. Rather, it advocates for a wider range of complementary approaches that together will form a comprehensive body of research.

This paper has argued that by moving beyond convergent validity, researchers will have the ability to investigate the validity of comparative judgement in a manner less limited by comparisons with the status quo. Moreover, it has been argued that the recent content validity solution, suggested by Jones et al. (2015) is insufficient in scope and is inevitably limited by the non-existence of psychometrically validated measures for the vast majority of mathematical content.

Finally, this paper proposed two new approaches, based on predictive validity, that avoid some of the discussed limitations. A research programme including investigation of predictive validity and the interpretation and use of assessments has the potential to extend the discussion about comparative judgement, and ensure that further uptake of this exciting new development in mathematics assessment is evidence-based and beneficial to the students for whom it is designed.

Range in r over 20 trials Mean r Median r Standard Deviation

	P-values	10 judges,	.664 -.855	.749	.762	.06
		20 students
	Derivatives	30 judges,	.826 -.907	.869	.872	.02
		42 students
	School Algebra 10 judges,	.678 -.837	.745	.742	.04
		46 students

Table 1 : Summary of comparative judgement reliability results from Bisson et al. (2016).

 1

Table 2 : Summary of comparative judgement validity results from Bisson et al. (2016). The 'traditional measure' line refers to the correlation between achievement data and comparative judgement z-scores. The 'instruments' column refers to the correlation between comparative judgement z-scores and scores on an existing psychometrically validated measure (RPASS-7, CCI and 'Concepts in Algebra', respectively).

 2

		Traditional measure	Instruments
	P-values	.555	.457
	Derivatives	.365	.093
	School algebra	.349	.428

Proceedings of CERME10


domain: math shs

text:
Bettina Dahl
email: bdahls@plan.aau.dk

Mathematics students' attitudes to group-based project exams compared to students in science and engineering

Keywords: attitudes, project exam, group exam

At Aalborg University, science, engineering, and mathematics students spent half the time each semester working in groups on projects within a problem based learning (PBL) curriculum. They are assessed through group exams. A survey showed that overall, the students are positive towards the group exam but there are significant differences between engineering, science, and mathematics students. Within this collectivistic student culture, some engineering students are very positive towards group exams, while mathematics, science, and other engineering students are less positive. In terms of the opportunity to obtain a fair grade in a group exam, the mathematics students are moderate positive, with different engineering students being more or less positive. All students agree that a group exam gives less differentiation of grades compared to an individual exam.

Introduction

At Aalborg University (AAU) in Denmark, science, engineering, and mathematics students work half the time each semester in groups of four to eight students on a project in a problem based learning (PBL) curriculum. PBL is student-centred and self-directed learning in teams with problem analysis and problem solving [START_REF] Barge | Principles of Problem and Project Based Learning: The Aalborg PBL Model[END_REF]. The project is documented as a joint written report with an oral group exam lasting around four hours. This exam has traditionally consisted of two phases: First the group presents the project, then the external examiner and the supervisor examine the group. Each student is awarded a grade that may not be the same as the others. From 2006-2012 the Danish government banned group exams, but during this time AAU students still worked in PBL groups but the exam became an individual oral exam of around half an hour per student. This situation led to research on assessment methods in PBL, and [START_REF] Kolmos | Alignment of PBL and assessment[END_REF] concluded that the students, the academic staff, and the external examiners preferred the group exam. One argument was that the students were not able to interact with each other during an individual exam, hence it was not possible to test PBL process competencies such as collaboration and teamwork. Since Danish law states that a grade solely depends on the student's performance at the exam (they cannot earn partial credits during the semester), this created a misalignment between PBL teaching and the assessment methods. In 2013 the group exam was reintroduced in Denmark and at AAU. The Faculty of Engineering and Science (FES) now added an individual phase into the previous group exam, where each student is questioned without interference from the group. [START_REF] Dahl | Students' attitudes towards group-based project exams in two engineering programmes[END_REF] found that the students overall were in favour of the reintroduction of the groupbased project exam, but that students from two different engineering programmes were not equally positive, partly owing to their previous experience with the individual project exam and partly owing to professional cultures of individualism or collaboration influencing their attitudes. [START_REF] Maull | A questionnaire to elicit concept images of engineering students[END_REF] and [START_REF] Bingolbali | Engineering students' conceptions of the derivative and some implications for their mathematical education[END_REF] showed that mathematics and engineering students are different in terms of their learning of mathematics, so one might ask whether a similar division between engineering and mathematics students is seen in relation to which type of exam is perceived appropriate. This paper therefore compares students from eight programmes in science, engineering, and mathematics in relation to how they perceive the new group exam, the new individual phase as well as the opportunity to obtain a fair grade.

Theoretical background

Alignment and exams [START_REF] Biggs | Teaching for Quality Learning at University[END_REF] argue that in order for students to learn the intended learning outcomes (ILOs), teaching should be constructively aligned with the ILOs and the exam. This theory fits other studies stating that an upcoming exam is a key factor for students' motivation and learning [START_REF] Boud | Aligning assessment with long-term learning[END_REF]; i.e. the 'backwash effect' of exams. Hence, one can argue that in a PBL curriculum, the exam method should be aligned with the team-based and collaborative teaching method and the ILOs on PBL process competencies. [START_REF] Romberg | Reform in School Mathematics and Authentic Assessment[END_REF] argues that a group exam is able to test "reflection on one's own thinking, reasoning and reflection, communication, production, cooperation, arguing, negotiating" (p. 165). One can thus argue that a group exam assesses PBL competencies of communication and cooperation. However, each programme also prepares the students for a professional life after the university so the problems that the students address vary and AAU's PBL model is developed "on the basis of both professional and educational argumentation" (Kolmos et al., 2004, p. 9). One might anticipate that professional culture influences the students' views of the group exam, particularly master students.

Cultural differences in engineering, science, and mathematics [START_REF] Murzi | Comparative dimensions of disciplinary culture[END_REF] studied how students perceived their discipline culture using Hofstede's dimensions. One dimension measures individualism versus collectivism. Overall, students had a high individualistic score. Mathematics, computer engineering, and electronic engineering students were among the less individualist students. This fits the study by [START_REF] Burton | Mathematicians as Enquirers[END_REF] where a majority of professional mathematicians worked co-operatively. [START_REF] Murzi | Comparative dimensions of disciplinary culture[END_REF] further argued that they had expected industrial design students to be more collectivistic as they rely on collective work in team projects but the results were opposite. Architect students' scores fell between mathematics and industrial design. [START_REF] Dahl | Students' attitudes towards group-based project exams in two engineering programmes[END_REF] also found significant differences between the engineering programmes Architecture and Design (AD) and Software Engineering (SE) at AAU. SE students were significantly more positive toward the group exam than those of AD. AD combines architecture with civil engineering and students here expect a more individual-oriented programme whereas SE is a system-oriented approach and a collaborative profession.

Research questions

How do the students from the eight programmes view the group-based project exam compared to the individual project exam and the individual phase of the new group exam? How do they experience the grading? What does this tell us about mathematics assessment in PBL?

Methodology

The questionnaire was piloted after the January 2013 exams and the revised questionnaire consisted of 20 questions of which most had several sub-questions. After the June 2013 exams, all 4,588 FES students received a link to this questionnaire and 1,136 responded. The response rate was relatively low (25%), which unfortunately is not uncommon for online surveys, but the level is still reasonable [START_REF] Nulty | The adequacy of response rates to online and paper surveys: What can be done?[END_REF]. The response rate for each study programme cannot be determined separately but the number of student responses were as follows: Computer Science (CS: 40), Energy (EnE: 50), Mechanics and Production (MP: 39), Physics and Nano science (PN: 27), Architecture and Design (AD: 79), Mathematics (M: 28), Software (SE: 51), and Electronic (ElE: 48). In this paper, all questions are translated from Danish by the author. The programmes compared all had a relative large number of students who responded. The engineering programmes are civil engineering.

Results

Views of individual versus group-based project exam

Of all FES students, 34% preferred the individual exam and 57% the group exam, but students in different programmes were not equally positive towards the group exam (see Figure 1). The survey asked a related question: "The time spent on the individual part is better spent on a longer joint part?" Of all FES students, 34% agreed, 50% disagreed, and 15% did not know. The students do not differ except MP, which is significantly different from AD (p = .005) and M (p = .030) (see Figure 2). Overall the students are positive towards the individual phase, particularly M. There is not significant difference (p = .114) between bachelor and master students for all FES.

When comparing students in each programme, MP bachelor and master students are significantly different (p = .024) but here the master students are the more positive toward the group exam. A question asked if the students were satisfied with their own grade. Overall, 83% agreed and there was not a significant difference between the eight programmes. Another question asked whether they found that all their group members had received a fair grade. Here only 66% agreed. In all groups, a majority of students had agreed but EnE (61%) and AD (59%) agreed significantly less than SE (76%) and PN (85%). EnE was different from SE (p = .046) and PN (p = .037) while AD was (p = .019) different from SE and PN. When comparing each programme, most had given significantly different answers to the two questions except ElE (p = .064), PN (p = .324), and SE (p = .760).

The survey also asked the students who had tried the individual exam whether the new group exam gave a better or a worse opportunity to obtain a fair grade. The views differ greatly (see Figure 3). When comparing the programmes, one sees that only M is not significantly different from any of the others while students from several of the engineering programmes answer significantly differently from each other (see Table 2). M therefore appears to be quite moderate in their views of whether or not the group exam gives a better or worse opportunity to obtain a fair grade. CS, ElE, SE, MP and to some extent PN are more positive towards the group exam as giving them a fair grade, while AD and EnE in general state that it provides them a worse opportunity to obtain a fair grade, compared with the individual exam. The students answering this question were all from the second year or older, as these were the only ones who had experienced both types of exams. The survey also asked the students whether there was a larger differentiation of grades in the group exam compared to the individual exam. The Danish grade scale has five passing grades (2, 4, 7, 10, 12) and two failing grades (-3, 0). This question relates to an internal discussion at both AAU and in Denmark debating if the group exam uses less of the grade scale as it is harder to give a precise individual grade and weaker students can hide and good students are not rewarded. Only a minority of the students confirmed that the group exam resulted in more differentiated grades. The lowest was ElE, where 7% said that the group exam resulted in more differentiated grades to some or to a higher extent, while the highest was M with 29%. This difference was not significant (p = .069).

CS

Conclusions Individual versus group exam and the individual part of a group exam

The students were in general very positive towards the group exam but there are significant differences. [START_REF] Murzi | Comparative dimensions of disciplinary culture[END_REF] found that mathematics, computer engineering, and electronic engineering were among the least individualistic within a very individualistic student culture. One might argue that in general the AAU students are used to working in PBL and thus have a more collectivistic culture since, overall, the majority of students were in favour of the group exam when asked to compare it to an individual exam. This study of AAU students shows that in agreement with [START_REF] Murzi | Comparative dimensions of disciplinary culture[END_REF], CS and ElE were among the most collectivistic students as they were among the most positive towards the group exams. However, this is stated within the frames of a collectivistic AAU culture. [START_REF] Murzi | Comparative dimensions of disciplinary culture[END_REF] found mathematics students to also be among the least individualist, however at AAU, M appeared to be among the more individualistic students, even though AD and EnE appeared to be even more individualistic. Architect students in [START_REF] Murzi | Comparative dimensions of disciplinary culture[END_REF] were not among the individualist groups. In relation to their attitudes to the time spent on the individual part of the group exam, M were the most positive toward the individual part. M was again close to AD but quite different from MP. PN students were closer to the M students.

Master students were generally more positive toward an individual exam than bachelor students, especially ElE. However MP master students appeared very positive toward having a longer joint part in the new group exam. One might argue that this is related to the fact that the master students have been used to the individual exam prior to 2013, or perhaps to how they perceive their future professional life (wrongly or rightly) might have an impact.

Fairness of grades

The students were overall satisfied with their own grade but relatively less satisfied with the grades given to their peers. Given the relatively low response rate, it could make sense that students answer this question significantly differently. The question in the questionnaire did not explicitly ask if their peers were over/under-graded, but it appears that seeing how their peers behave at the exam, perhaps with reference to their work during the semester and then experiencing what grade they received, often left the other students feeling some degree of unfairness. More research is needed here in order to determine why. The students also differed when they compared the group and the individual exam in relation to the opportunity to receive a fair grade. Here, M was more or less in the middle, not being significantly different from any of the other programmes. In general one sees that the same programmes as above show 'collectivist' preferences (CS, ElE, SE, MP) and 'individualistic' preferences (AD, EnE), which to some extent validate the results shown above and show that the students are consistent in their answers. However, one also needs to discuss to what extent their experience of receiving a fair grade is correct. Do students always know which grade they deserve? Furthermore, the perceptionrightly or wronglyof not being awarded a fair grade, might negatively influence their view of the exam. Students are occupied by fairness in grading and their perception of justice is significantly affected by the assessment method [START_REF] Burger | Student perceptions of the fairness of grading procedures: A multilevel investigation of the role of the academic environment[END_REF].

Students appear to obtain more similar grades when they are assessed as a group than if they are assessed individually. The question is thenwhich is the right grade? One might argue that in a group exam of up to eight students, it might be difficult to make a distinction between each group member, which to some extent might explain the different opinions about own grade and the grade of the group members, and the same question can only be asked once. On the other hand, one might also argue that since a group exam to some extent is able to test PBL process competencies, which an individual exam cannot, the grades given in a group exam are the more accurate.

Summing up and impact for mathematics assessment

It appears that mathematics students are not distinct from engineering or science students on the issue of preference for individual or group exams. The eight groups were mixed; ergo mathematics students were more similar to some engineering students but different from others. This is different from what is known from how the learning of mathematics takes place when comparing mathematics and engineering students. For instance [START_REF] Bingolbali | Engineering students' conceptions of the derivative and some implications for their mathematical education[END_REF] found that engineering students see mathematics as a tool and therefore wish to see the application side. In science, engineering, and mathematics PBL projects, mathematics is applied to solve problems, but in a mathematics project, the body of mathematical theorems used to solve a problem usually takes up a considerable part of the project work and the report. Thus, the role of mathematics is different in the PBL projects in each programme. Assuming that the group exam is the best fit to PBL, it is unexpected that the students are not more in agreement with each other about the group exam. An obvious answer is that the group exam does not fit each programme equally well. The more moderate views of the mathematics students could indicate that it is a reasonably good fit when there is an individual phase as this is a way to serve both the individual and collaborative aspects of mathematics.

The question of individualistic or collective attitudes may also depend on the overall student culture. [START_REF] Murzi | Comparative dimensions of disciplinary culture[END_REF] found that mathematics students were among the least individualistic within an individualistic culture, while this study found the mathematics students to be among the most individualist students within a collectivist culture. With caution, one might argue that the most individual student cultures were AD and EnE, with M and PN also being individualistic but not as much. The most collectivist student cultures were SE and CS, with ElE and MP also being collectivistic but not as much.

However, the above conclusions should be treated with caution as the students base their attitudes about the group exam on the 'learnt' curriculum [START_REF] Bauersfeld | Research related to the mathematical learning process[END_REF], which is not necessarily the same as the intended curriculum. The group exams were intended to be the same throughout the FES and prior to the reimplementation, workshops had prepared the supervisors for this type of exam. It is, however, unlikely that all group exams were identical as students, supervisors, and external examiners were different. One should therefore hesitate to draw too strong conclusions, particularly also taking into account the relatively small response rate. The results are indications of how students in different programmes at a Danish PBL university perceive a group exam and therefore which attitudes curriculum planners might expect from students if other universities wish to implement a type of project work or group exams. Curriculum planners need to consider what is the general 'culture' of collaboration both at the university and in the future profession, they need to consider how the exam has a backwash effect on how the students work, and that bachelor and master students might not perceive such an exam in the same way. The group exam might also be a better fit for some groups than others, but in terms of mathematics, it neither appears to be an obvious fit, since mathematics students might lean more towards the individualistic culture, nor does it appear to be a bad choice since the students in general are positive. This might reflect the mathematics culture as being both individual and collectivistic. In terms of grading, curriculum planners need to consider that group exams might result in a smaller distribution of grades.

Figure 1 :

 1 Figure 1: "I would prefer an individual project exam"EnE students were the least positive towards the group exam while SE students were the most positive. Table1shows the programmes that were significantly different.

Figure 2 :

 2 Figure 2: "The time spent on the individual part is better spent on a longer joint part" How did the students experience the opportunity to obtain a fair grade?

Figure 3 :

 3 Figure 3: "Does the group exam provide a better or worse opportunity to obtain a fair grade…?"

Table 1 : Significant differences in answers to the question if they preferred the individual exam

 1 Table1shows the programmes that were significantly different. SE and EnE are at opposite ends of the group-individual preference and they are each significantly different from many programmes. SE, CS and ElE are the most collective while EnE, AD, PN, M are the more individualistic, although M appears to be more moderately individual. MP are not significantly different from any. Master students are significantly more positive towards the individual exam than bachelor students (p = .001). Almost half the master students preferred the individual exam while only a third of the bachelor students did. Comparing bachelor and master students in each programme, there is only a significant difference for ElE (p = .002).

		EnE	AD	PN	M
	CS	.047	.037
	ElE	.043
	SE	.002	.004	.041	.031

Table 2 : Significant differences in answers to the question if the group exam provided a better or worse opportunity to obtain a fair grade

 2

			ElE	SE	MP	PN
	EnE	.001	.004	< .001	< .001	.015
	AD	.030	.039	< .001	.005

Proceedings of CERME10


text:
Johannes Beck
email: beck@dmuw.de

Written documentations in final exams with CAS

Keywords: Upper secondary education, exams, written solutions, Computer Algebra Systems, formative assessment

This article focuses on students' written solutions from CAS-allowed exams. Based on the analysis of students' solutions a descriptive model is set up. It can be used for assessing students' solution as well as creating exemplary documentations. The purpose of these documentations is to help teachers reflect about their practise of writing down solutions and the norms they set by this for exams. This paper also shows how formative assessment could be a means to help students develop their competencies in communicating mathematics.

Introduction: Challenges in exams with computer algebra systems

Digital technologies like graphical calculators and computer-algebra-systems (CAS) influence various aspects of mathematics education in general and the classroom-practise in particular (cf. Barzel ea. 2005). Most prominently, the way tasks can be solved changes drastically: where once it was necessary to differentiate a function with pen and paper, now every student can use a CAS and by pressing a short series of buttons the result appears on the screen. The new possibilities fuelled the hope that "[t]he new tool provided the chance to concentrate more on central competencies in mathematics education, concept formation, problem solving and modelling competencies, and to outsource algorithmic operations to the machine" [START_REF] Drijvers | The role of handheld technology in the mathematics classroom[END_REF]. While the shortcutting of the work of calculating with pen and paper enables classes to have more time for the central competencies, it also forces teachers and students to think about how to document the process of "working" on a task. This article focuses on the aspect of how communication in a CAS-environment could be shaped.

In this context, the study aims at identifying problems, difficulties and possibilities concerning students' solutions, categorize elements in those according to their function within the solving process and suggest a possible standard for written solutions with CAS. Against this background, the suggestions can also be interpreted as assessment criteria and, thus, a given students' solution can be assessed. Furthermore, it is a goal to provide a theoretical model that describes the development of students' documenting competency. The following research questions are 1. How do students write down their solutions in final exams? Which different forms of documentations do they use? What kinds of problems or difficulties (if any) are connected with these forms?

2. How can students' written solutions be described by means of a category system?

3. How could a developmental model look like that a. describes how students' competencies in documenting the solving process with CAS can be developed?

b. offers learning strategies and exemplary solutions for shape this development?

c. encompasses criteria for assessment in final exams?

The major motivation for this study is to create a sound set of suggestions for teachers, for application in the classroom and for preparing students for the final exams. Formative assessment plays a crucial role in developing an adequate documentation competence. This article gives an answer to the second question and presents an outline of ideas to research question (3).

Theoretical framework

The theoretical framework to tackle the questions above encompasses two important fields: written documentations as communicative texts, and formative assessment, which will be the basis for creating didactical material for teachers.

In exams the purpose of documentations is to enable others to understand how the solution has been gained and to evaluate to which degree the solution is correct or incorrect (cf. [START_REF] Ball | What Should Students Record When Solving Problems with CAS? -Reasons, Information, the Plan, and Some Answers[END_REF]. Thus, the communication of mathematical knowledge is the primary aspect. In terms of the communication model by [START_REF] Jakobson | Linguistics and Poetics[END_REF] the communication situation can be described as follows: the learner is the ADDRESSER, the teacher is the ADDRESSEE and the written-down solution is the CONTACT (or channel) for the MESSAGE (1960, p. 353). The CODE in this communication situation can be considered as coming from three different areas: (1) the natural language, (2) the mathematical language, encompassing the symbolic language as well as the mathematical register (cf. [START_REF] Pimm | Speaking mathematically: Communication in mathematics classrooms[END_REF], and (3) the computer world with CAS-commands and also its own register (cf. [START_REF] Siller | Mathematical Modelling in Class regarding to Technology[END_REF].

In exams the written documentation isaccording to the communication model abovethe only channel by which the message is sent from the addresser to the addressee. Naturally, in such a situation it is neither possible nor allowed for the corrector to inquire in case he or she does not understand a part of the solution. Busse speaks of all written communication as "reduced communication situation[s]" (2015, p 320, translation by the author), arguing that only the text itself and the recipient of the text are present in the situation. As a result, the understanding of texts can be reduced to the allocation of the recipient's knowledge to elements of the text.

The second part of the theoretical background is about formative assessment and how it might be used to develop the documentation competence of students over a longer period. "Assessment for learning", as formative assessment is sometimes called, can be outlined as "the process of seeking and interpreting evidence for use by learners and their teachers to decide where the learners are in their learning, where they need to go and how best to get there" ( ARG 2002, p. 2). [START_REF] Black | Developing the theory of formative assessment[END_REF] describe how five key strategies constitute formative assessment:

1. Clarifying and sharing learning intentions and criteria for success;

2. Engineering effective classroom discussions and other learning tasks that elicit evidence of student understanding;

3. Providing feedback that moves learners forward;

4. Activating students as instructional resources for one another; and 5. Activating students as the owners of their own learning [START_REF] Black | Developing the theory of formative assessment[END_REF].

The crucial and most difficult point here is to have criteria for good written solutions. As Weigand points out "there are no algorithmic rules or norms how to document a solution on paper" [START_REF] Weigand | Tests and examinations in a CAS-environment -The meaning of mental, digital and paper representations[END_REF](Weigand , p. 2772)). He reports from a long-term project in Bavaria (a part of Germany) that students have "difficulties in using SC [scientific calculator] and (problem-)adequate representations especially, as well as the documentation of the solution with paper and pencil" [START_REF] Weigand | Tests and examinations in a CAS-environment -The meaning of mental, digital and paper representations[END_REF](Weigand , p. 2763)). Therefore, teachers need to focus on the development of the competence to document adequately over a longer time. Students have to reflect about documentations and grow into the communication practices of the mathematics community. In Germany, the most important framework of mathematical competencies is the one by the KMK (cf. KMK 2012). The KMK distinguishes three requirementlevels to describe the requirements that can be addressed in tasks in relation to six central mathematical competencies and five central mathematical guiding ideas. This framework is not made for the development of the competence to document (which I see as only a part of the competence of communicating mathematically) but for describing and testing the competencies. Thus, a model was created that focuses more on the development and tries to reflect the difficulties of handling the CAS, too 3. Expert: Focus on modelling and problem-solving: Applying the mathematics to modelling problems with the help of the CAS.

Although high-school CAS-classes only use one device (like the TI-nspire or the CASIO Class Pad II) it is not the aim to restrict the mathematical competence to just this one device. Therefore, also the competence to document should not be chained to the tool that is used but be applicable to all kinds Novice Getting to know the device

Learning new mathematics

Experienced

Communication of mathematics in the foreground

Device is known

Expert

Math is used for modelling and problem-solving.

of tasks and problems. To achieve such a flexible competence teachers could use formative assessment as shown below.

It is the teacher's task to initiate a discussion about how the digital tool changes the nature of the solving process and, as a consequence, the written solution (strategy 2, see above). This is also a reflection of the benefits of a CAS. As already mentioned, it is not necessary anymore to write down every step in the solving process in order to perform it. A lot of algorithmic procedures can be outsourced to the CAS. This reflection might help students to progress further on their way from novices to experts.

Furthermore, as Black & Wiliam state "peerand self-assessment are activities that might be used to pursue the fourth and fifth [key strategy] respectively" (p. 8). As a useful activity students (and teachers as well) could check and discuss whether their own documentation and the documentation of classmates meet the criteria. Students can discuss problems regarding the understandability of solutions amongst themselves and with their teacher. From a theoretical perspective, it seems to be most promising to apply these activities with experienced students (Fig. 1), after the students are used to work with the CAS but before complex (modelling-)problems are treated. Yet, this assumption has not been tested empirically. Regarding the documentation of solutions one aim is a high level of language use (most prominently: mathematical terminology). This is part of the competences communicating mathematically and mathematical reasoning (cf. KMK-Bildungsstandards, K6 and K1; Blum 2010).

Methodology

As a first step, a descriptive model has been developed from students' authentic solutions from highstake final exams. These exams are the last time in the students' life and therefore these reflect (to a certain degree) the knowledge and the practise of the students. From a linguistic perspective, it is one aim to identify which elements students use in their documentations and which function is connected with each form. This is a typical pragmatic approach (cf. [START_REF] Meibauer | Pragmatik -Eine Einführung[END_REF]. The underlying question of this form-function-analysis can be formulated as follows: With which forms of representation do students document each step of the solving process? From a mathematical perspective, it is the aim to identify difficulties and problems in the students' solution. One problem is that traditional mathematical notation is mixed with computer language with the result that the created expressions do not fit the requirements of "the community of mathematicians".

Bavarian teachers of CAS-classes have been asked to send in nine written solutions each from the final exams. The students have three groups of tasks to solve (calculus, geometry, data & statistics) and have 180 minutes time. Three solutions came from students who have been average, three from students who have been above average, and three from students who have been below average in the preceding semester. Similar data has been collected every year (starting with 2014) for further evaluation and research. Four to five teachers answer this request every year.

The first research question is how students document their solving process in exams. So far, in Bavaria (Germany) only little official advice about documentations of solving processes is given. Normally, the Institute for School Quality and Educational Research (ISB) provides such material and official notes in addition to the curriculum. In order to develop such advice, it is a very valuable first step for researchers to analyse authentic documentations and to develop a descriptive model with which problems and difficulties can be identified and categorized.

The representational dimension describes with which forms of representation students document.

There might be expressions, which use some kind of formulaic symbols (traditional mathematical, computer-syntax, mixed-forms), verbalisations (both natural language and the special mathematical vernacular) and graphic representations. In the latter category, mixed forms (such as graphs, tables, sketches, etc.) are also counted. The second, activity dimension describes which purpose an element has, that is what actually is documented with it and which step, or activity, in the solving process it is related to. Central categories are:

• CAS-related notes make the use of CAS explicit, either by stating the CAS command (input), by writing down the output (e.g. "false", which is odd in a German text), or by unspecifically writing in short form -that the CAS was used (e.g. "CAS: …"). • According to [START_REF] Wagner | Erklären lernen -Mathematik verstehen -Ein Praxisbuch mit Lernangeboten[END_REF] explanations comprise three different facets: concepts and ideas (what-explanation), algorithms and procedures (how-explanation), argumentations and logical connections (why-explanation). They often focus on: o mathematizations, which show that information given in the task-description is translated into mathematical notation or terminology; o interpretations, which are translations of computer-output and the construction of meaning in relation to the task. The terms mathematizations and interpretations are related to the respective activities in the extended modelling cycle by [START_REF] Siller | Mathematical Modelling in Class regarding to Technology[END_REF].

• Furthermore, there are elements which refer to the underlying mathematical idea, e.g. in order to find the maximum of a function f it is necessary to solve the equation f'(x)=0. From this element, the mathematical idea can be reconstructed.

• Every mathematical activity leads to some result. This might be the answer to a posed question or a step that takes one closer to the final result. • Structuring elements are used to structure the text on the surface (the layout) and the way the information is presented. They can also be to set up links between pieces of information such as single steps in the solving process and the chronological order in which they were performed.

Results

The first result is that the category system above (Fig. 2) is suitable to describe students solutions. It can be observed that in regard to the documentation of CAS-commands the style was very homogenous throughout each class. In one of the classes, CAS-commands have been documented. In the second class, the CAS-use was indicated by writing "CAS" either over an equation or at the beginning of a line. In the third and fourth class there were no CAS-commands at all. This phenomenon can be explained by the normative standards that the respective teacher had set in the preceding year. Secondly, students who had a correct solution always showed the necessary mathematical ideas. Fig. 3 shows two different ways how mathematical ideas can be presented: either in verbalized form (line 1) or encapsulated in a formulaic expression (line 3).

A further result of the analysis of the students' documents is that written solutions without verbalised explanations were often harder to understand and that the solving process could not be reconstructed that easily.

Authentic and examplary solutions

As shown above, elements of written solutions can have different functions. Among them explanations can contribute a lot to make students' documents easily understandable. According to Jörissen and Schmidt-Thieme explanations can be characterised as "primarily verbal statements" with the goal that the reader can understand connections (2015, p. 401, translation by the author).

Furthermore, additional explanations extend the transmitted information with the possible consequence of redundancy. However, misunderstandings can possibly be prevented. As already mentioned Wagner & Wörn distinguish three different types of explanations: explain-what, explainhow and explain-why (2011). These sub-categories can be foundrudimentarily -in the students' solutions, too. It is most important to notice here that students often explained verbally although it was not explicitly asked to do so in the formulation of the task. The task of the example (Fig. 3) is to check whether there is a point at which the exit of a highwaymodelled by a polynomic function sruns parallel to another roadthe route B299.

In the example (Fig. 3) we see that the student explains the mathematical idea of his solution verbally at the beginning. It is a rudimentary how-explanation. The verbal inaccuracy at this point is not that important because the information given in the text is supported by the mathematical formulaic expression, which is the equation. The output ("{}") follows a CAS-use which is documented unspecifically (see abve). The student confuses proper mathematical syntax with device-specific CAS-output and mixes both into an incorrect expression. As a concluding answer to the task a verbal interpretation of this output is written down.

The categories from Fig. 1 can be used to describe and explain students' solutions. But they can also be used to help teachers to reflect written solutions and their own practise of writing mathematical texts. Furthmore, on the basis of the categories exemplary solutions can be created, as shown below (Fig. 4).

Category Exemplary solution

Explanation

The roads run parallel to each other when there is a point at which 𝑠 has the same gradient as 𝑠(𝑠) = -0.5𝑠. It cannot be expected of students' solution to show such a degree of verbalisation. It is not the purpose of exemplary solutions to set minimal standards for students but to show teachers how solutions can be documented. The categories help to structure the text and to make the function of single elements more apparent.

Discussion of results and conclusion

Teachers may apply the categories in two ways: firstly, the categories can provide guidelines for documentations in a constructive way. For example, every documentation should make clear what the mathematical idea was that lead to the solution. Furthermore, students and teacher could agree in classroom discussions that verbalized explanations help to make clear the connections of different (symbolic) elements and, thus, allow the reader(s) to follow the solving process more easily. The notation of CAS-commands (if and how) could also be agreed upon in the class or amongst teacher even on a school level. Secondly, they then might be used for assessing students' solutions for formative purposes as well as summative purposes. The categories might help teachers and researches alike to see documentations more clearly as a set of elements with different functions that make up the whole text. The exemplary solutions illustrate different possibilities of documentations. They can be used for reflection in pre-service professional development courses as well as in in-service professional development courses.

In conclusion, it is important that students develop the competence to assess by themselves if a solution is acceptable and to create good solutions on their own (formative assessment key strategies 4 and 5). The main determining factors are the purpose of the documentation and the intended addressee, both of which may make an additional verbal explanation necessary. To provide support for this development is a major challenge for modern mathematics education. The combination of "learning to document" with some elements of formative assessment is a promising way to meet this challenge.

  . The competence model by[START_REF] Dreyfus | Künstliche Intelligenz. Von den Grenzen der Denkmaschine und dem Wert der Intuition[END_REF] is insofar an important reference work that it describes how the development of competencies from a novice stage to an expert stage happens in five steps. According to Dreyfus & Dreyfus, novices act according to rules very explicitly while experts have internalized the rules so much that the behaviour has become part of them. The model for the development of the documentation competence distinguishes only three stages. For teachers it is thus easier to think of the stages as of the three consecutive years (10th to 12th grade) when CASclasses can be allowed permanently in special classes. The model tries to reflect some difficulties teachers reported in CAS-classes (cf.Beck 2015[START_REF] Weigand | Tests and examinations in a CAS-environment -The meaning of mental, digital and paper representations[END_REF]) by shifting the focus of each of the three stages:

Figure 1 -

 1 Figure 1 -Development model 1. Novice: Focus on technology-use: Learning to deal with already known and new mathematical content with the new tool, in order to get accustomed to it. 2. Experienced: Focus on communication: Reflecting about the use of the tool and the communication of mathematical content.

Figure 2 -

 2 Figure 2 -Category system for description of students' solutions

Figure 3 -

 3 Figure 3 -Student's solution: original and translation

  not run parallel to each other.

Figure 4 -

 4 Figure 4 -Exemplary solution for teachers

Proceedings of CERME10


text:
Catarina Andersson
email: catarina.andersson@umu.se

Formative assessment -and the component of adjusted teacher instruction

Keywords: Formative assessment, assessment for learning, mathematics, instruction

This theoretical paper is based on an empirical study where the framework of formative assessment by Wiliam and Thompson was used to analyze teachers' use of formative assessment in their mathematics classroom practice. The paper argues for treating a component named Adjusted Teacher Instruction (ATI) as a key strategy in complement to the five key strategies in the original framework. ATI is a significant component in formative assessment, but also particularly challenging for teachers to implement in their classroom practice. Treating ATI as a key strategy could facilitate the analysis of teachers' use of formative assessment activities and enhance the understandings about what kind of ATIs are most useful for whom under what conditions. Extended understandings about effective formative assessment activities are important in decisions about what formative assessment to include in teacher education and in-service training for teachers.

Introduction

The encouraging potential noticed in formative assessment has motivated scholars to further engage in both empirical studies and theoretical work in the research area of formative assessment. The theoretical understandings of formative assessment have evolved during a long time, often affected by empirical studies in which researchers has been responsive to teacher practice. This paper focuses on a component of formative assessment that is regarded particularly significant, but also difficult in carrying out formative assessment. In this paper this component is called Adjusted Teacher Instruction (ATI) and it is argued for treating ATI as a key strategy in parallel to teacher feedback. [START_REF] Black | Assessment and classroom learning[END_REF] in their research review demonstrated that large student achievement gains are possible when formative assessment is employed in classroom practice. This review received widespread attention and caused a discussion about the need for and role of an extended assessment culture and practice. Since then, implementation of formative assessment has been on school policy agendas in many countries [START_REF] Tierney | Changing Practices: Influences on Classroom Assessment[END_REF], but this implementation has often proven to be challenging [START_REF] Birenbaum | International trends in the implementation of assessment for learning: Implications for policy and practice[END_REF]. Several attempts have been unsuccessful in accomplishing a substantially developed formative assessment practice [START_REF] James | Teachers Learning How to Learn[END_REF][START_REF] Schneider | Research on characteristics of effective professional development programs for enhancing educators' skills in formative assessment[END_REF] and misunderstandings and distortions of essential features of formative assessment are detected in policy and practice [START_REF] Swaffield | Getting to the heart of authentic assessment for learning[END_REF]. Some factors facilitating and hindering the implementation regarding the teacher, student, assessment and context are identified [START_REF] Heitink | A systematic review of prerequisites for implementing assessment for learning in classroom practice[END_REF], but still a strong research base supporting how to effectively help regular teachers to implement a high quality formative assessment practice is lacking [START_REF] Schneider | Research on characteristics of effective professional development programs for enhancing educators' skills in formative assessment[END_REF][START_REF] Wiliam | An integrative summary of the research literature and implications for a new theory of formative assessment[END_REF]. Such a research base needs to include both how to design effective professional development programs for teachers and what kind of formative assessment to include in such programs. This paper is related to the latter issue. The discussion in the paper is theoretical, but originates from an intervention study in which a group of mathematics teachers learned about formative assessment (see [START_REF] Andersson | Professional development in formative assessment: Effects on teacher classroom practice and student achievement[END_REF]. In this study, formative assessment was conceptualized as one big idea and five key strategies (see [START_REF] Wiliam | Integrating assessment with learning: what will it take to make it work[END_REF] in a framework that was used in the professional development program and for structuring the data collection and data analysis.

Background

Black and Wiliam defined formative assessment as "encompassing all those activities undertaken by teachers, and/or by their students, which provide information to be used as feedback to modify the teaching and learning activities in which they are engaged" (Black & Wiliam, 1998, pp. 7-8); a definition that provide several possible focus in carrying out formative assessment. Consequently, Black and Wiliam's review included studies investigating different strategies for carrying out formative assessment, using the term formative assessment in different meanings or using alternative terms such as feedback, self-regulated learning or peer-assisted learning. As [START_REF] Bennett | Formative Assessment: A Critical Review[END_REF] points out, without a consensus about the term formative assessment, the effects will be unclear. A common and clear terminology and definition of formative assessment is also desired to eliminate misunderstandings and distortions in policy and practice.

To maximize instructional benefits, we need to know more about what constitutes effective formative assessment (Wiliam & Thomphson, 2008;[START_REF] Wiliam | Keeping learning on track: classroom assessment and the regulation of learning[END_REF]. To gain valuable insights about best practices it is important to be clear about the way formative assessment is conceptualized in for example studies of implementations of formative assessment that are empirically linked to student achievement. The formative assessment practice needs to be carefully analyzed and described to provide information about specifics of such practices as well as how these specific characteristics may have functioned as part of an enhanced learning process.

Black and Wiliam's review included studies showing the potential instructional benefits of different strategies for carrying out formative assessment. It can be expected that a classroom practice that integrate such key strategies to a unity would open up extended opportunities for learning and thus offer higher potential for improving student achievement. The empirical study motivating this paper is one of few studies investigating the impact of such an integrated practice on students' achievement. Following sections of the text outline Wiliam and colleagues' conceptualization of such a practice, followed by a description of the operationalization of that framework.

One big idea and five key strategies

A more recent definition of formative assessment by Black and Wiliam is more detailed:

Practice in a classroom is formative to the extent that evidence about student achievement is elicited, interpreted, and used by teachers, learners, or their peers, to make decisions about next steps in instruction that are likely to be better, or be better founded, than the decisions they would have taken in the absence of evidence that was elicited. (Black & Wiliam, 2009, p. 9) This definition clearly demands every formative strategy to fulfill the big idea of using evidence of student learning to adjust instruction to better meet students' learning needs. The conceptualization of formative assessment as one big idea and five key strategies [START_REF] Wiliam | Integrating assessment with learning: what will it take to make it work[END_REF][START_REF] Black | Developing the Theory of Formative Assessment[END_REF]) is visualized in Figure 1. The matrix visualizes how three processes (horizontally) and three agents in the classroom (vertically) construct five key strategies (KS) in formative assessment. The three processes constitute the defining characteristics of formative assessment inherent in the definition above and are central for the big idea of using evidence of student learning in decisions about how to proceed in the instruction. The three agents who are responsible for the learning in the classroom are defined as the teacher, the learner and the peers. The different key strategies in formative assessment are connected and sometimes dependent on each other's existence and performance. For example, clear learning intentions guide the teacher to chose questions/tasks that elicit relevant information about students' learning and help the teacher to provide goal directed feedback. In addition, learning intentions clear to the students enhance their opportunities to be engaged and involved in the learning process (their own or their peers').

Using the big idea and five key strategies to analyze classroom practice

The author of this paper participated in a research group responsible for a study about professional development in formative assessment for a group of randomly selected mathematics teachers. The framework above was used in the professional development program, in the data collection and in analysis of formative assessment used in the teachers' mathematics classroom practices.

Two rounds of data collection and analysis were made, before and one year after the professional development program. Both times, observations were completed in each teacher's mathematics classroom practice and all teachers were interviewed. The big idea and the five key strategies structured observation schemes and interview guides, with a focus on what formative assessment activities the teachers used in their mathematics classroom practice.

Each formative assessment activity was supposed to be classified in relation to one of the five key strategies or the big idea. From a teacher perspective the big idea pertains to Key strategies 1-3. Accordingly, activities aimed at clarifying where the learner is going could be classified as belonging to KS 1 and activities aimed at eliciting where the learner is right know as belonging to KS 2. Teacher feedback aiming at moving student learning forward was classified as belonging to KS 3. This process led to one remaining group of teacher activities concerning teachers' use of information about learning needs (the big idea) that did not fit to any of the other key strategies. Therefore, to clarify our data and for consistency reasons, we decided to include the new category Adjusted Teacher Instruction (ATI) as a new 'strategy' in parallel to feedback. Both strategies (ATI and feedback) aim at taking learning forward. Consequently, no activities were classified as belonging to the big idea.

The formative activities classified as ATI were the activities aiming at taking learning forward that did not concern teachers' oral or written feedback.

Before the professional development program (PDP) the most common ATI activity was to use results from a diagnosis in the textbook to choose a group of tasks (regular or advanced) for each student's individual work with the chapter in the textbook. Other ATI activities used by a smaller group of teachers were, for example: individualized tasks for a student; adapted materials for example work sheets, homework or tactile materials; extra or modified lecture for the class, a group of students or for individual students; and adaption of time set aside for a chapter in the textbook. After the PDP individual teachers extended their repertoire of ATI activities (from the same type of activities as identified before the PDP), for example lectures for group of students became more common. The teachers' use of ATI activities was also affected by new activities connected to Key strategy 2. Many teachers had started to make use of students' misunderstandings, which were often identified by using mini-whiteboards as an all-response system. In general, the teachers received information about student learning more often and in various ways and could consequently make adjustments of instruction more often and with more precision.

In our analysis of formative assessment activities in teacher's mathematics classroom we decided that teacher activities connected to the use of evidence of learning needs could either be classified as a feedback activity or as an Adjusted Teacher Instruction activity. Thus, feedback and Adjusted Teacher Instruction would have a shared position within the third teaching and learning process (How to get there) for the teacher's actions (see Figure 1). This proposed shared position will be discussed below.

Discussion

In this paper, Adjusted Teacher Instruction (ATI) is suggested as a component of particular significance in formative assessment to be treated as a key strategy, a proposal that could improve the use of the formative assessment framework by [START_REF] Wiliam | Integrating assessment with learning: what will it take to make it work[END_REF] in research, policy and practice. The advantage is twofold: (1) the ATI component will get a more prominent place and

(2) the framework will be more coherent. These advantages, but also some concerns, will be discussed below.

One advantage of treating ATI as a key strategy would be that teacher activities within this strategy could be defined and further studied in the same way as for activities belonging to other strategies. The few studies using frameworks unifying several formative assessment strategies in analysis of teacher classroom practices do not always provide specifics about teachers' adjustment of instruction (e.g. [START_REF] Wylie | The fidelity of formative assessment implementation: issues of breadth and quality[END_REF][START_REF] Randel | Classroom Assessment for Student Learning: Impact on Elementary School Mathematics in the Central Region[END_REF]. Such specifics are desirable because the instructional decisions and actions taken to better meet student needs are crucial for students' continued learning opportunities [START_REF] Wiliam | Keeping learning on track: classroom assessment and the regulation of learning[END_REF] and because using evidence of learning to inform next instructional steps has been experienced as a challenging aspect of formative assessment [START_REF] Cowie | A Model of Formative Assessment in Science Education, Assessment in Education: Principles[END_REF][START_REF] Heritage | From Evidence to Action: A Seamless Process in Formative Assessment?[END_REF][START_REF] Oláh | Learning to Learn From Benchmark Assessment Data: How Teachers Analyze Results[END_REF].

Research addressing this crucial and difficult component in formative assessment is desirable to enhance the understandings about what ATIs are effective under different circumstances, but this area of research needs improvement in terms of becoming more prominent, the definition of the area and the number of empirical studies conducted [START_REF] Bellert | Effective re-teaching[END_REF]. Research about feedback has resulted in guiding models for what type feedback is more or less effective, for different students and under different conditions (e.g. [START_REF] Hattie | The power of feedback[END_REF][START_REF] Shute | Focus on Formative Feedback[END_REF]. Similar knowledge about ATI is important because a main aspect of formative assessment is that planning of instruction is decision driven. To secure that the information from the assessment will be useful, a feasible way is to plan instruction backwards with a clear decision in mind and searching for relevant evidence to make decisions in a smarter way [START_REF] Wiliam | Keeping learning on track: classroom assessment and the regulation of learning[END_REF]. Skilled teachers can design teachable moments into their lesson because they have already thought of alternative instructional decisions before the information was collected (ibid., p. 1089).

If we know more about what instructional adjustments are likely to be most effective in different situations this would be helpful guidance for policy and practice, for example in teacher education and in-service training for teachers. The knowledge base can be extended from conducting studies that empirically link different types of ATI to student achievement and by careful analysis, descriptions and conclusions about ATI characteristics and their function as part of an enhanced learning process. One example of a study contributing to this knowledgebase is a study by Ruiz-Primo, [START_REF] Ruiz-Primo | Teachers' Judgments On-The-Fly: Teachers' Response Patterns in the Context of Informal Formative Assessment[END_REF]. This study of science and mathematics teachers was restricted to informal formative assessment, in which the interaction between teacher and students is central. Studied within this interaction, the characteristics of the teacher's response were classified according to type of oral feedback or type of instructional move. Additionally, the type of teachers' actions observed in more and less expert teachers was studied. The results were separated for individual student work and whole class work. Using a two step cluster analysis, there were five variables included for teachers' instructional moves: (1) re-teaching (e.g., going over content again in the same or similar way as before);

(2) solving problems with students (e.g., asking students for their input along the way while solving a problem); (3) solving problems without students (e.g., solving or modeling the solution to a problem without student input); (4) re-clarifying the task (e.g., reminding students of what they need to do); and (5) providing the correct answer (e.g., giving the answer without explanation) (Ruiz-Primo et al., 2015, p.17). This study does not only tell us that instructional adjustment where implemented, but also specifies these adjustments and compare the use of them in two kinds of work conditions and by two groups of teachers.

Another advantage of treating ATI as a key strategy in parallel with feedback is that the framework would be more coherent. In fact, also Wiliam includes instructional adjustment as a second aspect of feedback in the meaning that feedback is provided to the teacher so he or she can modify the instruction to be more effective (Wiliam, 2010, p. 33). However, the feedback to the teacher is comparable to elicited evidence of student learning in Key strategy 2, and this might generate confusion. The suggested shared position for feedback and Adjusted Teacher Instruction might clarify the framework to avoid misunderstandings and distortions in policy and practice and thus provide a better guidance. Such guidance might have affected the design of the professional development program in the study behind this paper (see [START_REF] Andersson | Professional development in formative assessment: Effects on teacher classroom practice and student achievement[END_REF]. In the program ATI was not treated as a key strategy and did not get the same focus and time set aside as the other key strategies did. This might have affected the limited extension of types of ATIs at group level in the results.

A concern about treating ATI as a key strategy regards the need and meaning of the big idea.

Elaborating the big idea from a teacher perspective, Key strategy 1 is not indispensable for teachers' eliciting and using information of students' learning needs, only critical for ending up with evidence useful for formative assessment. Wiliam distinguishes between diagnostic assessment and assessment that is instructionally tractable, where the latter form not only indicates what needs attention but also what needs to be done to address the issue [START_REF] Wiliam | Keeping learning on track: classroom assessment and the regulation of learning[END_REF](Wiliam, , p. 1063)). Wiliam points to the need of the teacher to have a range of instructional alternatives beyond just repetition:

For formative assessment to be instructionally tractable, the teacher must be clear about the range of alternative instructional moves that are possible, should then decide what kinds of evidence would be useful in choosing among the relevant alternatives, and only then elicit the evidence needed to make that decision. (Wiliam, 2010, p. 33) While Key strategy 1 is ultimate, Key strategy 2 is a prerequisite for the implementation of the big idea. Key strategy 3 concerns the very foundation of the big idea about formative assessment. The big idea is important in the evaluation of the function of the implementation because it reflects the whole assessment cycle, which Wiliam suggests should be performed backwards (see above).

Another concern regards the distinction between feedback and Adjusted Teacher Instruction. The distinction we made (see [START_REF] Andersson | Professional development in formative assessment: Effects on teacher classroom practice and student achievement[END_REF] do not match the distinction made by [START_REF] Ruiz-Primo | Teachers' Judgments On-The-Fly: Teachers' Response Patterns in the Context of Informal Formative Assessment[END_REF], probably caused by the different conceptualization of formative assessment. Even when using the same conceptualization of formative assessment, some activities will be a definite feedback or ATI activity, but other activities will have a more uncertain belonging. In our case, when teachers used the "thumb of role" giving feedback as two stars and a wish (showing the student two excellent aspect of their work and an idea for improvement), this is categorized as feedback. When a teacher decides on finishing work on algebra a week earlier than planned, this is ATI. A more uncertain activity would be when the teacher together with the student decides what tasks are most appropriate for the student to work with. We would classify this as an ATI activity, from the rationale of not being restricted to oral or written feedback from the teacher.

Conclusion

The advantage of treating ATI as a key strategy put forward in this paper is ultimately about improving the guidance and support of teachers' implementation of high quality formative assessment practice. ATI is experienced as difficult to implement by teachers. At the same time, the ATI component does not always receive much focus in analysis of teacher classroom practices. A more specified analysis and communication of research results about ATI could provide teachers with better guidance. In addition, a more coherent framework could be easier for teachers to understand. The big idea is an important guiding idea and the key strategies concretize this idea. The quality of any formative assessment activity is dependent on the extent the activity meet the aim of the key strategy as well as the big idea.

This paper argues for more studies conceptualizing formative assessment as a unity of different strategies of which ATI is one. Ultimately such studies examine the effect of implementation of formative assessment on both teacher classroom practice and student achievement. The formative assessment practice needs to be carefully analyzed and described to provide information about specific characteristics and their function as part of an enhanced learning process. Such research might end up in models similar to those of feedback [START_REF] Hattie | The power of feedback[END_REF][START_REF] Shute | Focus on Formative Feedback[END_REF], showing more and less effective ATI, for different students and under different conditions.

  Where the learner is going Where the learner is right now

				How to get there
	Teacher	KS 1 Clarifying learning	KS 2 Engineering	KS 3 Providing feedback
		intentions and criteria	effective classroom	that moves learners
		for success	discussions and other	forward
			learning tasks that
			elicit evidence of
			student understanding
	Peer	Understanding and	KS 4 Activating students as instructional resources for
		sharing learning	one another
		intentions and criteria
		for success
	Learner	Understanding learning
		intentions and criteria
		for success

KS 5 Activating students as the owners of their own learning Figure 1: The relationship between key strategies (KS), instructional processes and agents in the classroom (After a figure in Black & Wiliam, 2009, p. 8)

Proceedings of CERME10

Proceedings of CERME10


text:
Caitríona Wynne
email: caitrionawy@gmail.com

Lorraine Harbison
email: lorraine.harbison@dcu.ie

The application of the Universal Design for Learning framework to task design in order to support lower attaining children in the primary school mathematics classroom

Keywords: Universal task design, mathematics education, inclusion of lower attaining children

Background to the study

It is a requirement that all schools in Ireland administer standardized attainment tests in mathematics when children are in 2 nd Class (aged 7 -8). On the basis of these results, children who are deemed to have underperformed nationally (Standard Score <90), are usually offered additional support by means of withdrawal from their mainstream classroom for mathematics lessons to work on a complementary program with the learning support teacher. Although it is argued that such provision is offered to less able children as they tend to simply 'give up' in classrooms in which they find the mathematical tasks too challenging, the concern is that that these children will fall further and further behind. It is the contention of the authors, that it is not the tasks per se that are too challenging for the children but rather the nature of the tasks, which all too often tend to emphasize traditional practices of rote procedure and drill.

In this study, the researcher, who was working as the learning support teacher, and the class teacher planned together to integrate children back into their classroom by co-designing mathematically rich tasks. Over a four week period, three children who had formerly been withdrawn from the classroom were put to the fore in the planning of the tasks. Particular consideration was given to their learning styles. The mathematical tasks were inspired by the three key principles of the educational framework, Universal Design for Learning (UDL), which constitute Multiple Means of Representation; Multiple Means of Action and Expression; and Multiple Means of Engagement [START_REF] Rose | Universal design for learning[END_REF]. The term 'universal' is particularly pertinent in the design of the tasks as they were developed in line with specific mathematical learning goals for 'all' learners from the beginning rather than implementing a standard 'one size fits all' set of tasks and differentiating the tasks to cater for the marginalized, less able children, at a later stage.

Therefore, in order to create learning tasks that will engage all children by design, this research sought to ascertain:

1. How can the principles of UDL be used to design mathematically rich tasks? 2. Do UDL informed tasks engage and support children of low-ability in mathematics?

Methodology

It was a detailed body of work comprising collaborative universal lesson design on the topic of 'Measurement', implementation of lessons, critical analysis of tasks and peer review. Each lesson was assessed using an adapted scoring rubric developed by [START_REF] Spooner | Effects of training in universal design for learning on lesson plan development[END_REF]. Frequencies of different events were tabulated. Formal observational instruments were developed to recognize and discern certain types of behaviors such as children's degrees of engagement. Observations were supplemented by photographs. Teachers' daily reflections were analyzed using the analytic technique of pattern matching. A matrix of categories was developed and evidence placed within each classification. Information was put in chronological order. A follow up interview was held with the host teacher at the end of the intervention.

Results

The 14 lessons scored 82 points out of a maximum of 84 points on the adapted scoring rubric [START_REF] Spooner | Effects of training in universal design for learning on lesson plan development[END_REF] implying that a very high level of the UDL approach was used in the task design and implementation. Diversity was the starting point in planning the tasks, with lower ability children being accommodated within and enriching the regular class. The development of positive learning profiles for the three target children, such as, 'needs assigned role during group tasks', or, 'needs to have basic equipment available such as a pencil and a ruler prior to task allocation', helped to remove barriers to and enable participation in learning. UDL tasks offered the children various ways of acquiring information and knowledge; provided alternatives for demonstrating what they knew; tapped into children's interests, gave appropriate challenges, and increased motivation. Multiple means of representation, action and expression, and engagement were used in task design. Video clips were used on five occasions, concrete materials on eight occasions, ICT (PowerPoints, images, interactive stylus and interactive tools) were used during eight lessons, a parallel ICT mathematics program was set as homework for the children on each of the 14 days and the local environment was used on seven occasions. ICT was also found to be a key component that engaged children who were previously observed to be challenged by mathematics.

Discussion and conclusion

This research revealed that tasks which take into consideration UDL instructional goals, assessments, methods and materials are usable and accessible from the outset rather than having to retrofit the tasks to children's needs as an afterthought. Crucially, results from this study found that by intentionally creating flexible learning opportunities, less able children were engaged and understood difficult mathematical ideas when they were provided with UDL informed tasks.

Proceedings of CERME10