Solid findings in mathematics education:

A psychometric approach

Caterina Primi

The foundation of all rigorous research designs is the use of measurement tools that are psychometrically sound. The purpose of this paper is to present the scales' proprieties such as reliability, validity, and invariance that are fundamental prerequisite for assuring the integrity of study findings. Providing examples of how to assess the psychometric properties of tools used in mathematics research may be helpful for future researches in this topic.

In the document prepared by the Education Committee of the European Mathematics Society (2011), the description of "solid findings" includes an aspect of "robustness". That means that findings in the research on mathematics learning and teaching should be repeatedly observed or confirmed in many studies reporting the same or similar results leading to the same (general) conclusions. To achieve this goal rigorous research designs and measurement tools that are psychometrically sound are needed. Starting from these premises I will try to identify the contribution of psychometrics to solid findings in mathematics education.

Measurement

In many educational measurement situations, the variables of interest such as ability, beliefs, attitudes, and anxiety are not directly observable. As such, they are latent variables or traits. Indeed, they are easily described but they cannot be measured directly, as can height or weight for example, since these variables are concepts rather than physical dimensions.

To give an example of a measurement process, imagine that a researcher is interested in measuring mathematics anxiety (MA). Mathematical anxiety is commonly defined as an adverse emotional reaction to math or the prospect of doing math (Hembree, 1990). It is a state of nervousness and discomfort brought upon by the presentation of mathematical problems and may impede mathematics performance irrespective of true ability (Ashcraft, 2002;Ashcraft & Moore, 2009). The negative consequences of mathematical anxiety are well-documented (Morsanyi et al, 2017). Students with high levels of mathematical anxiety might underperform in important test situations, they tend to hold negative attitudes towards mathematics, and they are likely to opt out of elective mathematics courses, which also affects their career opportunities. Over the last decade there has been more interest in understanding how and when MA develops (Dowker, Sarkar, & Looiet, 2016;Harari, Vukovic, & Bailey, 2013;Jameson, 2013;Ramirez, Gunderson, Levine, & Beilock, 2013), investigating the incidence of MA, and its effects on primary school samples (e.g. Galla & Wood, 2012;Karasel, Ayda, & Tezer, 2010;Wu, Barth, Amin, Malcarne, & Menon, 2012), as well as its consequent influence on math achievement (Wu et al., 2012). Given the widespread prevalence of MA and its detrimental long-term impact on academic performance and professional development, it is important to measure this construct in a valid and reliable way.

From a measurement prospective it is not possible to directly observe MA. Following the latent trait theory (Lord & Novich, 1968), we can measure something that cannot be observed only by inference from what can be observed. Thus, while the trait itself is not observable, its interaction with the environment produces, at the surface level, observable indicators which can be used to infer the level or degree of the latent trait. Considering MA, although we cannot observe our latent variable, its existence may be inferred from behavioural manifestations or manifest variables (for example, as feeling tense, fearful and apprehensive about mathematics). These manifestations make it possible to measure MA asking, for example, a series of questions (the items of the instruments) that describe each manifestation (for example, "I feel nervous when I use numbers"). Indeed, a measurement instrument can be constructed using these items with the purpose of assessing the unobservable trait.

However, the primary goal of educational measurement is to determine the level of the latent trait that a person possesses. In general, scaling is the process of establishing the correspondence between the observations and the latent variable. Several mathematical approaches have been developed in order to define how to measure a latent trait through item responses, assuming that the latent trait is continuous. These approaches include Classic Test Theory (CTT) and the more recent Item Response Theory (IRT).

Traits, indicators, and their relationships can be represented graphically. Figure 1 represents the measurement structure of the Abbreviated Math Anxiety Scale (AMAS; Hopko, Mahadevan, Bare, & Hunt, 2003). This is a two-factor measure of MA that is considered a parsimonious, reliable, and valid scale. The two factors are Learning Math Anxiety, which relates to anxiety about the process of learning, and Math Evaluation Anxiety, which is more closely related to testing situations. The AMAS is one of the most commonly used measure of MA in college and high school students (for a review, see Eden et al., 2013). The scale has been translated into several languages, and it has been found to be a valid and reliable measure in Looking at the details of Figure 1, the ovals represent latent, unobserved variables, specifically, Learning Math Anxiety and Math Evaluation Anxiety. The squares represent the observed variables (items); five for Learning Anxiety (e.g., listening to a lecture in a math class), and four for Math Evaluation Anxiety (e.g., thinking about an upcoming math test one day before). The relations between items and the latent traits are represented with arrows that indicate that the traits cause the corresponding indicators. In Figure 1, the error components that we have to take into account in the measurement process are also represented. Any observed score has two parts: The true score part and the error part. Intuitively, we control for the error and we estimate the true score by taking several measures and averaging them. We assume that averaging several measures results in a better estimate of the true score. These ideas are formalized in the concept of reliability. We use multiple indicators or items to better measure the trait. This way, we can have more information and reduce the error components, that is, we can maximize the reliability or precision of the measurement tool.

Moreover, verifying the relationships among indicators and the corresponding traits, through a confirmative procedure, such as a confirmatory factor analysis (CFA), we can verify that the measurement tool truly captures the underlying trait, attesting the validity of the measurement tool (Zumbo, 2009). Indeed, obtaining evidence of validity is part of the measurement process.

Invariance

Measurement validity also implies that the meaning of the construct and its operationalization is the same in different social and cultural contexts. Testing the invariance of the test concerns the extent to which the psychometric properties of the test generalize across groups or conditions. Therefore, measurement invariance is a prerequisite of the evaluation of substantive hypotheses regarding differences between contexts and groups.

If the research question is, for example, about assessing gender differences in MA, and our test shows that female students have higher math anxiety scores than male students, we would be tempted to interpret test scores in terms of the trait that they are intended to reflect, i.e., that females have greater MA than males. However, it is possible that the test scores do not purely reflect the latent trait, i.e. MA in each group. That is, it is possible that the test is biased in some way.

Bias is used as a general term to represent the lack of correspondence between measures applied to different groups (Van de Vijver & Tanzer, 2004). There are different kinds of bias, for example construct bias, when the meaning of the studied trait varies among groups; item bias, when the meaning of the item content is different in certain groups, or method bias, when the characteristics of instruments induce measurement errors for particular groups of respondents.

These biases violate the assumption of measurement invariance, which holds that measurement properties should not be affected by group membership (Zumbo, 2009). In other words, the observed scores should depend only on the latent construct, and not on group membership. An observed score is said to invariantly measure the construct if it is affected by the true level of the trait in a specific person, rather than by group membership or context (Meredith, 1993). This means that people belonging to different groups, but with the same level of a trait, are usually expected to display similar response patterns on items that measure the same construct. Thus, when studying test invariance, we determine whether the tool functions equivalently in different groups, that is, we test the absence of biases in the measurement process.

A well-known method to assess invariance is multiple group confirmatory factor analysis (MGCFA) in which the theoretical model is compared to the observed structure in two samples.

Testing measurement invariance involves a step-by-step procedure in which nested models are organized in a hierarchal ordering. Specifically, the following invariance models are tested. The configural one, which refers to testing whether an instrument exhibits the same structure (Do the groups show the same general factor structure? Same number of factors? Same conceptual definition of latent constructs?). The next model, the metric one, tests whether the items function equally across groups. If this invariance is established, the groups can be said to have the same unit of measurement. The final model, the residual one, tests if measurement errors are the same across groups, which means that the scale is be equally reliable in both groups.

Applying this method, we tested the equivalence of the AMAS across male and female Italian students (Primi et al., 2015). With regard to the measurement issue, given that the assessment of MA relies on self-report measures, it is important to note that females are more willing to report their feelings of anxiety than males (e.g., Goetz, Bieg, Lüdtke, Pekrun, & Hall, 2013). This finding highlights the importance of employing measures of MA which are invariant across genders. That is, there is a need to test if the items measure the same construct when administered to male and female respondents, controlling for the differences in true group means. Indeed, to compare groups of individuals with regard to MA, one must be sure that the values that quantify the construct are on the same measurement scale.

The issue of measurement invariance has received considerable attention also in cross-cultural research because people from different cultures might have different understanding of the same questions included in an instrument (Milfont & Fischer, 2010). Indeed, testing invariance is of particular concern when using a translated version of a survey instrument, and it is a necessary prerequisite for the translated instrument to be used in cross-cultural research (e.g., Baumgartner & Steenkamp, 1998).

For this reason, we tested the invariance of the Italian version of the Statistical Anxiety Scale (SAS) developed by Vigil-Colet, Lorenzo-Seva, and Condon (2008). Learning statistics is often associated with statistics anxiety, defined as "extensive worry, intrusive thoughts, mental disorganization, tension, and physiological arousal [. . .] when exposed to statistics content, problems, instructional situations, or evaluative contexts" (Zeidner, 1991, p. 319). In the original validation study, Vigil-Colet et al. (2008) analyzed the internal structure of the SAS using exploratory factor analysis. The results attested a three-factor structure: Examination Anxiety (referring to the anxiety involved when taking a statistics class or test), Interpretation Anxiety (referring to the anxiety experienced when students are making a decision about or interpreting statistical data), and Fear for Asking for Help (referring to the anxiety experienced when asking a fellow student or a teacher for help in understanding specific contents). The primary aim of our work was to confirm this factorial structure of the Italian version using CFA. As confirmation of the same base factor model was not a sufficient condition to establish the equivalence of the Spanish and Italian versions of the SAS, we tested the invariance of the factor model's parameters between the Italian sample and a comparison Spanish sample. Since the results indicated a substantial equivalence of the Italian and Spanish versions of the SAS, we can use the translated instrument in cross-cultural research, we can make meaningful comparisons between Italian and Spanish students' statistics anxiety, and we can develop intervention strategies to enhance students' achievement across Spanish and Italian educational frameworks.

To sum up, if measurement tools are not "invariant", instruments do not measure the same trait across the different groups or contexts, results are not comparable, and inferences about differences are misleading. As a consequence, methods for investigating biases should be implemented when new measures are created, when existing measures are adapted to new contexts or for different populations, or when existing measures are translated.

Conclusion

The foundation of all rigorous research designs is the use of measurement tools that are psychometrically sound. Confirmation of the validity and reliability of tools is a prerequisite for assuring the integrity of study findings.

In empirical research, comparisons are often made between distinct population groups, including groups from different cultures, genders, or that speak different languages. These analyses implicitly assume that the measurement of these outcome variables is equivalent across groups, although this assumption often remains untested. Measurement invariance can be tested and it is important to make sure that the variables used in the analysis are indeed comparable across groups.

In conclusion, testing the psychometric properties of tools, such as measurement invariance might help in increasing the robustness of findings across various groups and contexts.

  a variety of populations (Polish version: Cipora, Szczygiel, Willmes, & Nuerk, 2015; Italian version: Primi, Busdraghi, Tomasetto, Morsanyi, & Chiesi, 2014; Persian version: Vahedi & Farrokhi, 2011). Recently, it has also been adapted for children between the ages of 8 to 11 (Italian version: Caviola et al. 2017), and 8 to 13 (English version: Carey et al. 2017).

Figure 1 :

 1 Figure 1: Model of the Abbreviated Math Anxiety Scale (AMAS)

Proceedings of CERME10


Gerry Shiel

Can the outcomes of PISA 2015 contribute to evidence-based decision making in mathematics education?

Drawing on data from the OECD's Programme for International Assessment (PISA), which assesses mathematical literacy and other domains among 15-year olds in over 70 countries every three years, this paper explores the extent to which PISA outcomes in 2015 can be described as 'solid' and hence contribute to evidence-based decision making. It identifies aspects of PISA that render its findings 'solid', but also points to pitfalls that arise in interpreting PISA outcomes related to achievement. The paper concludes by examining how PISA can contribute to thinking about the nature of evidence-based findings in mathematics education.

Introduction

A key feature of the educational landscape since 2000 has been the Programme for International Student Assessment (PISA), a study sponsored by the Paris-based Organisation for Economic Cooperation and Development (OECD) that assesses performance in mathematics, reading literacy and science among 15-year olds in over 70 countries every three years. In addition to administering tests to students, PISA administers questionnaires to students, their parents and their school principals. The student questionnaire asks about students' socioeconomic status, their attitudes towards mathematics and other subjects, and their instructional experiences. This paper looks at performance outcomes in the two most recent PISA cycles -2012, when mathematics was a major assessment domain, and 2015, when mathematics was a minor domain, and PISA moved from a paper-based to computer-based testing in most participating countries.

Interest in the extent to which PISA provides 'solid' or 'evidence-based' findings arises because of the strong impact that PISA has on policy making in many participating countries. In Ireland, for example, a significant drop in performance in mathematics and reading literacy in PISA 2009 led to the implementation of a National Strategy to Improve Literacy and Numeracy 2011-2020(DES, 2011). The strategy set out a series of measures designed to improve performance, including plans to enhance initial teacher education, curriculum and assessment. In parallel with the Strategy, revised curricula in mathematics at post-primary level have been rolled out in a phased basis since 2010 in an initiative known as 'Project Maths'. This involves a strong focus on developing students' conceptual understanding in mathematics, and on applying mathematical knowledge in solving problems in context using a range of methods. Ní Shuilleabháin (2013) described Project Maths as 'a philosophical shift in Irish post-primary classrooms from a highly didactic approach with relatively little emphasis on problem solving towards a dialogic, investigative problem-focused approach to teaching and learning mathematics ' (p. 23).

A key feature of the National Strategy is the inclusion of national targets for performance in PISA mathematics. In an interim review of the Strategy (DES, 2017), there are targets of 10.5% of students achieving below proficiency level 2 by 2020, and 12.0% achieving levels 5-6. The first of these is quite an ambitious relative to current performance (15% performed below Level 2 in 2015), while the second is more modest (11% performed at Levels 5-6 in 2015).

Efforts to ensure that PISA findings are solid

The procedures around the development of PISA survey instruments, including the mathematics test, are designed to ensure that findings can be relied on and used by participating countries to enhance teaching and learning, and raise performance standards. The development of the PISA mathematics test and scale encompasses the following:  An assessment framework is developed and published at the outset of each PISA cycle (e.g., OECD, 2013). The framework provides a definition of mathematical literacy in PISA, and outlines the content areas (mathematical content categories) and processes to be assessed, the contexts in which items are to be embedded and the item formats to be used. Items are then developed in a way that ensures that all elements of the framework are adequately addressed. The assessment framework is a key source of evidence to support the validity of the PISA tests.

 Items based on the framework are submitted by countries, or are developed by the consortium charged by the OECD with implementing PISA. Items are vetted by countries for cultural and linguistic appropriateness and suitable items are forwarded for field trialling.

 The PISA field trial is conducted on a sample of 15-year olds in each participating country, and the performance of items is assessed within and between countries. The outcomes of both classical item analysis and item response theory scaling are taken into account in determining the suitability of items. These items, along with any trend items not fieldtrialled, are then used to compile test forms for the main study.

 Considerable effort goes into ensuring that items are scored accurately, using scoring guides prepared by the PISA consortium. Many items are marked by two or four scorers, and realtime indices of inter-rater reliability are used to guide the quality of scoring.

 The PISA main study is implemented. Quality control is a key aspect of the Main Study, as countries are held accountable to quality standards (see below).

 Performance on PISA is scaled using Item Response Theory models and links with performance on earlier rounds are established.

A document, PISA Technical Standards (e.g., OECD, 2014), is issued in each cycle to guide countries in ensuring that their samples, response rates, security procedures, translation and coding practices are of a sufficiently high standard that their data warrants inclusion in international reports. For example, the 2015 Technical Standards indicate that response rates of 85% at school level and 80% at the student level are required. The achieved samples of countries failing to meet these criteria are examined in detail for potential bias. In some cases, countries have not been included in international reports because of low response rates (e.g., the Netherlands in 2000, and the UK in 2003).

At the end of each PISA cycle, a technical report is prepared by the PISA consortium and is issued by the OECD (e.g., OECD, 2017). It details the procedures used in each aspect of the implementation of PISA, including sample design, field operations, quality control, survey weighting, scaling, proficiency scale construction, and coding reliability.

The consortium charged with implementing PISA establishes expert groups for mathematics, science and reading literacy, and there is also a Technical Advisory Group, which advises the Consortium on its use of scaling and other procedures, and a Questionnaire Expert Group. These groups act as a further check on the quality of the PISA instruments and outcomes.

Hence, PISA has taken several precautions to ensure the quality and solidity of its findings.

Notwithstanding the fact that PISA assesses the mathematics that students require for life after they leave school (or mathematical literacy) and for future study, rather than mathematics based on school curricula, the steps taken to ensure that findings are solid are extensive.

The introduction of computer-based assessment as a threat to the solidity of PISA findings

Prior to 2015, PISA implemented computer-based testing in subsets of countries on an optional basis. In 2012, for example, mathematics was assessed on paper in all 65 participating countries, and on computer on an experimental basis in a subsample of 32 countries. In 2015, however, there was a shift to computer-based assessment in most participating countries, with 56 of 73 countries, including all 34 OECD member countries, administering PISA in this format. The remaining countries administered PISA on paper.

The transition to computer-based testing in PISA presented some significant challenges for the OECD. A key component of PISA is the availability of trend datathat is, performance from one PISA cycle to the next must be placed on the same underlying scale so that average performance and performance across proficiency levels in each country and on average across OECD countries can be tracked from cycle to cycle. The task facing the OECD and its contractors1 was to establish the feasibility of linking performance on the 2015 computer-based tests to scales based on performance on paper-based tests in earlier cycles. This was further complicated by a requirement to continue to provide trend data for countries that administered PISA in paper-based form in 2015.

There were several ways in which the transition to computer-based testing could have been managed, given the imperative to maintain trends. For example, all students (or equivalent samples of students) taking PISA 2015 could have been given paper-based and computer-based tests. Then trends could have been established with reference to performance on the paper-based measures and new computer-based scales could have been devised, based on the computer-based items, and used for trend analysis in the future. This would have eliminated any concerns about mode effects (an advantage or disadvantage arising from implementing PISA on computer).

The approach taken by the OECD and its contractors was to make adjustments in 2015 based on how the same items performed on paper and on computer in the PISA 2015 Field Trial, which took place in all participating countries in spring or autumn 2014. In the case of mathematics, which was a minor domain, items used in earlier PISA cycles (i.e., trend items) were transferred from paper to computer, and equivalent representative samples of students from each country took the paper-and computer-based tests. Hence, the purpose of the mode study was to ascertain whether tasks or items presented in one mode (i.e., paper) functioned differently when presented in another mode (i.e., computer) and vice versa. For the purpose of analysis, items were pooled across countries, as individual countries did not have sufficiently large samples of students to allow for reliable comparisons of individual items across modes, or for an analysis of item-by-country interactions.

Where item parameters were judged to be 'strongly invariant' (that is, similar on paper and computer), item parameters were constrained to be the same in the 2015 Main Study (OECD, 2017).

In the course of the Field Trial analysis, a subset of items showed mode effects. To account for these effects in the Main Survey, different item parameters were estimated for paired paper-and computer-based items. According to the OECD (2017, Chp. 7, p. 53), 'this established an invariance model that assumes scalar or strong invariance for the majority of items and metric invariance for a minority of items for which difficulty differences were detected'. A correlation of .95 was found between paper-based and computer-based item parameters for mathematics in the Field Trial, further supporting a link between performance on computer-based tests in 2015 and paper-based tests in earlier cycles, as well as between computer-and paper-based tests administered in 2015.

The PISA 2015 Field Trial yielded other interesting findings that applied to mathematics as well as other domains. For example, across countries, students taking the Field Trial tests on computer had significantly fewer omitted responses than students taking the paper versions. Furthermore, there were fewer effects of cluster positon on performance when tests were administered on computer (that is, items administered by computer were more likely to perform in the same way regardless of whether they appeared early or late in the test). However, as Jerrim et al. (in press) note, while the Field Trial did not yield large differences across modes for male and female students, no analyses were conducted to examine potential interactions with variables such as ethnicity or socioeconomic status. They also questioned the representativeness of the samples used in the Field Trial, which, in some countries, could be described as convenience samples. They viewed this as weakening the external validity of the results, given the implications for the adjustments made within Main Study scaling to enhance cross-mode comparability.

Overall performance on PISA 2015 mathematics

The PISA main study took place in all participating countries in 2015. The OECD issued two volumes of findings in December 2016 that included country mean scores in mathematics, and comparisons with performance in earlier cycles. The mean score of students in Ireland in 2015 was 503.7 (OECD, 2016). This was significantly above the average across OECD countries (490.2), and was about the same as in 2012 (501.5), 2006 (501.5) and 2003 (502.8). Indeed, the only year in which average performance moved outside the 501-504 range was in 2009 (487.1).

While the mean mathematics score of students in Ireland was stable in the transition to computerbased assessment, a number of countries saw large declines in performance between 2012 and 2015. These included Korea (down 29.7 score points, though still well above Ireland at 517.4), Chinese Taipei (17.5), Hong Kong (13.3), Poland (13 points), and the Netherlands (10.7 points). On the other hand, a small number of countries experienced increases in achievement, including Sweden (15.7 points), Norway (12.4), the Russian Federation (11.9), and Denmark (11.1).

It is noteworthy, however, that Norway, Denmark and the Russian Federation were among the countries with the highest use of computers by students in mathematics classes in PISA 2012 for purposes such as entering data on a spreadsheet, drawing a graph of a function, constructing geometric figures, re-writing algebraic expressions and solving equations (OECD, 2015). In contrast, Korea, Hong-Kong China and Ireland were among the countries with the lowest usage of ICTs by students in mathematics classes.

The fact that Ireland's overall performance on PISA 2015 is similar to 2012 can be interpreted in a number of ways:

 It suggests that students in Ireland are equally adept as solving mathematical problems in paper and computer-based formats; indeed, this would suggest that the mode of assessment does not matter, at least for students in Ireland.

 It suggests that students in Ireland improved in their mathematics between 2012 and 2015, but this improvement was largely hidden because of the transition to computer-based testing.

The second of these seems the most likely. PISA 2015 was the first PISA cycle in which all students in Ireland's sample had studied under the Project Maths syllabus. This interpretation is consistent with a finding that students in initial Project Maths schools (24 schools that had implemented Project Maths first) outperformed students in non-initial schools in PISA 2012 mathematics (see [START_REF] Perkins | Learning for life: The achievements of 15-year olds in Ireland on mathematics, reading literacy and science in PISA 2012[END_REF], though the difference was relatively small (4 score points) and not statistically significant.

A further relevant finding relates the optional computer-based assessment of mathematics administered as part of PISA 2012. In that assessment, students in Ireland had a mean score that was not significantly different from the corresponding OECD average score, despite achieving a mean score on paper-based mathematics that was significantly above the corresponding OECD average in the same year [START_REF] Perkins | Learning for life: The achievements of 15-year olds in Ireland on mathematics, reading literacy and science in PISA 2012[END_REF]. Hence, performance on PISA 2015 can be interpreted as being indicative of a possible improvement.

Interestingly, the OECD has continued to hold the positon that mode effects in PISA 2015 mathematics were small and did not impact on the performance of participating countries (OECD, 2016(OECD, , 2017)). Implicit in this is the view that performance on computer-based assessment in 2015 can be linked back to performance on paper-based assessment in earlier PISA cycles.

Other threats to the solidity of PISA 2015 findings

The transition to computer-based assessment in PISA is clearly one threat to the validity of scores reported by the OECD for PISA 2015 mathematics. However, there were several other changes to PISA 2015 which could also impact on the interpretation of outcomes, and hence the solidity of PISA findings. The changesseveral of which occurred because a new consortium was contracted by the OECD to gather and analyse PISA datainclude:

 Changes in the assessment designthe design of PISA 2015 was modified to reduce or eliminate differences in construct coverage for major and minor assessment domains for test takers. In practice, this meant that fewer students took mathematics in PISA 2015, compared with earlier cycles, but more mathematics items were included in the assessment, thereby allowing for broader construct coverage.

 Changes in the calibration sampleprior to 2015, item difficulty in PISA was estimated using the responses of students in the most recent cycle (e.g., in 2012, this comprised data from students who took PISA in 2009). Moreover, the calibration sample in earlier cycles comprised a random sample of 500 students per participating country. In 2015, item parameters were re-estimated using all students in all participating countries in the previous four PISA cycles. This change was implemented to reduce the uncertainty around estimates of the item parameters used in calibration.

 Changes to the scaling modelin earlier PISA cycles, a one-parameter Item Response Theory (IRT) model (with adjustment for partial credit) was used to scale performance. In 2015, item functions based on a two-parameter logistic IRT model for dichotomous data, and a generalized partial-credit model for polytomous data were used in scaling data in the case of new items, while functions based on a one-parameter model were used (as previously) with trend items. Unlike its predecessor, the new approach does not give equal weighting to all items when constructing a score, but assigns optimal weights to tasks based on their capacity to distinguish between high-and low-achieving students.

 Changes in the treatment of differential item functioning across countrieswhere items performed unexpectedly differently across countries, the calibration in 2015 allowed for a number of country-by-cycle-specific item parameters. In previous cycles, items that showed differential item functioning (e.g., because of differences across languages) were dropped from scaling. The change in 2015 was intended to reduce the dependency of country rankings on the selection of items included in the assessment (for a country) and hence improve fairness (OECD, 2016).

 Changes in the treatment of not-reached itemsin PISA 2015, not-reached items (unanswered items at the end of a section, such as at the end of the first and second hour of testing) were treated as not administered when estimating proficiency (i.e., scoring student responses), whereas in previous PISA cycles they were treated as incorrect. A reason for this change was to eliminate the opportunity for countries and test takers to randomly guess answers to multiple-choice questions at the end of a section of the test. As in previous cycles, not-reached items were treated as not administered when computing item parameters (i.e., during scaling).

The OECD (2016) acknowledges that improvements to the PISA test design and to scaling in PISA 2015 can be expected to result in reductions in link error (the error associated with particular sets of items used in a particular cycle) between 2015 and future cycles. However, it also acknowledges that the changes described above may result in increased link error between PISA 2015 and earlier cycles, as past cycles used a different design (paper-based assessment) and used different scaling procedures. Furthermore, the OECD (2016) acknowledges that the change in the treatment of notreached items could result in higher scores than would have been estimated in earlier PISA cycles for countries with many unanswered items.

Conclusion

The problem in terms of interpreting trend scores is that any of the changes implemented by the OECD and their contractors in relation to the design and scaling of PISA in 2015 could have impacted on the scale scores achieved by students. Interpretation becomes even more difficult when multiple changes are implemented, as these may interact with one another in complex ways. The OECD has sought to address this in a limited way by rescaling data from earlier PISA cycles using the methods implemented in 2015. Thus, in the case of Ireland, performance on PISA mathematics changed by +2 score points between 2012 and 2015 (see above), but, the change was 6.0 score points when newer scaling methods were applied to the 2012 mathematics data. On average across OECD countries, the impact of changes to scaling procedures was also reported to be small (a published drop of 3.7 score points between 2012 and 2015, and a drop of 2.5 score points following rescaling of the 2012 data) (OECD, 2016). For most countries, differences arising from re-scaling are within the error margins of the original difference scores reported by the OECD.

While the readjustment of scores from PISA 2012 using the new scaling procedures implemented in 2015 may go some way towards reassuring users that PISA outcomes are comparable over time, the sheer number of changes implemented in PISA 2015, including the change to computer-based testing, indicates that particular care should be exercised in interpreting PISA 2015 data.

Efforts to improve the design and scaling of PISA 2015 also contain some lessons for efforts to generate solid data in mathematics education. On the one hand, solid findings can be obtained by implementing the same testing procedures and methodologies on multiple occasions (e.g., pre-and post-intervention). In the words of Beaton (1990), 'when measuring change, do not change the measure' (p. 165). On the other hand, at least in the case of longitudinal, multi-year surveys such as PISA, there is an ongoing need to build innovation into all aspects of the project to maintain relevance and deliver more robust measures for the future. One clear danger is that, when mathematics becomes a major assessment domain in PISA 2012, the construct measured will also change, as new items specifically designed to take advantage of the affordances computers, will be introduced for the first time.

Proceedings of CERME10

The lead contractor in PISA

was the Educational Testing Service in the US. The lead contractor in all earlier cycles of PISA was the Australian Council for Educational Research.

Acknowledgements

Participants who sent us questions during and after the discussion debate are warmly thanked, with the hope that they will see their issues, concerns and interrogations included in the lines above.


Judith Huget
email: jhuget@math.uni-bielefeld.de

BiMathTutor -The program for further training of new tutors in University Mathematics

Keywords: 1, Teacher's and student's practices at university level, 7, Preparation

In the Faculty of Mathematics at Bielefeld University, new tutors have to attend the further training BiMathTutor, which took place for the first time in the winter term 2017. At this program, they learn about the essentials of teaching. This program is being researched to the effects on tutors and their students. The poster presents the theoretical framework, the concept of BiMathTutor, the research design and first results.

INTRODUCTION

In this paper, we focus on the development of a further training for tutors. The main aim of a further training for tutors is to improve teaching skills and students' learning. In most cases, the tutors are experienced students who are interested in passing down their knowledge and, of course, earning some money to support themselves. These teaching skills should be based on a qualification program to establish and maintain standards of teaching and learning.

In the Faculty of Mathematics at Bielefeld University, starting in October 2017, new tutors have to attend the program for further training called "BiMathTutor" (Bielefeld Mathematics Tutor Program). This program has been reformed over the past year and evaluation is happening under different aspects, namely, the change of attitude towards teaching, the expectations of the tutors of what is important for the tutorials and the effects of "BiMathTutor" on the exam results of the students attending the tutorials (see figure 1).

Figure 1: Time table of BiMathTutor

DATA COLLECTION

The flowsheet (see figure 1) shows the different elements of the intended surveys. Module 1-3 and the observation session plus feedback conversation is stated on the top. The semester week shows a typical semester at Bielefeld University. There are two general cohorts, one being the tutors and the other being students visiting the tutorials. We have several times of measurement indicated by black bordered boxes. The study uses a combination of quantitative and qualitative methods. The sample size of the tutor survey is N1=24. The sample size of the student survey is N2= 597. The test instruments are in form of questionnaires and exam results.

RESEARCH QUESTIONS

The object of research are the effects of is the tutor program. Hence, the research questions for the survey are the following:

1. How do new tutors assess the further training? 2. What expectations do students visiting the tutoring class have? 3. What is the grade of effectiveness of the further training on the students' exam performance? The first hypothesis is that new tutors will know the typical process and the organisational aspects of a tutoring class, will be able to write and apply a horizon of expectation, will execute different methods to engage students in a learning process, will establish characteristics of a good tutoring class, will be aware of students' difficulties dealing with university mathematics and will investigate their own strengths and weaknesses. On the second research question, one will expect that students have the goal to pass the exam. They will not be aware of the learning process involved and the possible simplification by a tutor, who attended BiMathTutor. The third hypothesis is that the further training has an effect on the students' exam result.

THEORETICAL FRAMEWORK

The theoretical framework is still in process of elaboration. The design of BiMathTutor was made by Dee Fink's theory "Designing courses for significant learning" (2003). The key components "Learning goals", "Teaching and Learning activities" "Feedback and Assessment" have to be in unison. Additionally, the situational factors must be considered.


Jean-Baptiste Lamy
email: jean-baptiste.lamy@univ-paris13.fr

A new diagram for amino acids: User study comparing rainbow boxes to Venn/Euler diagram

Keywords: Set visualization, User study, Euler diagram, Rainbow boxes, Amino acids, Bioinformatics, Education

Figure 1. Rainbow boxes diagram displaying 8 properties of the 20 amino acids.

I. INTRODUCTION

Set visualization [START_REF] Alsallakh | Visualizing sets and set-typed data: Stateof-the-art and future challenges[END_REF] considers several elements and several sets containing all or part of these elements. The sets are overlapping, i.e. an element can belong to more than one set. The visualization aims at showing clearly the elementset membership relations but also at providing answers to more complex questions, such as finding the intersection of two sets, and to elicit new insights, such as discovering similarities between elements or sets.

Euler and Venn diagrams are commonly used to visualize sets when the dataset is small, especially in biology. A widely known example is the Euler diagram of amino acid properties (known as "the Venn diagram of amino acid" despite the fact that it is actually an Euler diagram, Figure 2). In this example, each amino acid is an element and each property is a set (e.g. the set of small amino acids). There are 8 sets and 20 elements. This diagram is present in almost all textbooks and courses in molecular biology and bioinformatics (including the ones of the author), and a Google search yields more than 300,000 results ("Venn|Euler diagram amino acid", on 13/3/2018). However, Euler and Venn diagrams are not always easy to read. In our teaching experience, students often have difficulties to understand and use Figure 2. Other authors mentioned that, in general, Venn diagrams are "not effective for presentation of more than four categorical groups" [START_REF] Shade | Beyond the Venn diagram: the hunt for a core microbiome[END_REF].

Recently, we introduced a new technique for visualizing overlapping sets, rainbow boxes [START_REF] Lamy | Rainbow boxes: a technique for visualizing overlapping sets and an application to the comparison of drugs properties[END_REF], [START_REF] Lamy | Rainbow boxes: a new technique for overlapping set visualization and two applications in the biomedical domain[END_REF], which was initially applied to the visual comparison of drug properties [START_REF] Lamy | Using visual analytics for presenting comparative information on new drugs[END_REF]. In rainbow boxes, the elements are shown in columns, and the sets are represented by rectangular boxes placed below column headers (see example in Figure 1). Each box covers the columns corresponding to the elements belonging to a set. The column order is computed using a heuristic optimization algorithm. This algorithm tries to order the columns so as the elements belonging to each set are contiguous. When it is not possible to have them contiguous for a given set, "holes" are present in the set's box (in Figure 1, the polar box has 3 holes). Colors are added to columns and boxes, as follows: a specific color is associated with each column, ranging across the spectrum, and the color of a box is the mean of the colors (in Red-Green-Blue coordinates) of the columns it covers. Finally, boxes are stacked vertically, with the largest boxes at the bottom. Two boxes can be next to each other, as long as they do not occupy the same columns. However, the evaluation data of rainbow boxes are rather sparse.

The work presented here has two objectives: [START_REF] Alsallakh | Visualizing sets and set-typed data: Stateof-the-art and future challenges[END_REF] to propose an improved diagram for the presentation of the properties of amino acids, based on rainbow boxes, and (2) to evaluate this diagram and to compare it to Euler diagram. This user study involved 78 students in biology and followed a crossover protocol. We measured error rate, response time and user preference. We also discuss the limitations of the two types of diagrams, such as the ability to enrich the existing diagram with additional properties.

The rest of the paper is organized as follows. Section II presents related works. Section III explains how were produced the diagrams for the user study. Section IV describes the experimental design of the user study. Section V gives the results of the study. Section VI explores the possibilities to enrich the amino acid diagram with additional properties. Section VII discusses the results, proposes perspectives.

II. RELATED WORKS A. Set visualization

Alsakallah et al. [START_REF] Alsallakh | Visualizing sets and set-typed data: Stateof-the-art and future challenges[END_REF] reviewed the techniques for overlapping set visualization. Euler and Venn diagrams are one of the oldest approaches to set visualization [START_REF] Baron | A note on the historical development of logic diagrams: Leibniz, Euler and Venn[END_REF]. In Euler diagrams, each set is represented by a closed-area. The areas overlap in various regions that represent the intersections of the sets. Venn diagrams are a kind of Euler diagrams showing all the 2 n -1 possible combinations of overlaps (where n is the number of sets). A proportional Euler or Venn diagram is a diagram in which the size of the various areas and overlaps is proportional to the number of elements in each regions. The automatic drawing of Euler and Venn diagrams is still a challenge when the number of sets is above 6 [START_REF] Rodgers | A survey of Euler diagrams[END_REF].

Linear diagram is another diagram for set visualization [START_REF] Gottfried | Set space diagrams[END_REF]. In a linear diagram, the elements are displayed in columns and the set in rows. A piece of horizontal line is drawn in each cell at the intersection of an element that belongs to a set. Thus, a set is represented by one or more horizontal lines. Colors are usually added to identify the lines that correspond to a given set. Although not initially inspired by linear diagrams, rainbow boxes can be seen as an evolution of linear diagrams. The main difference between the two diagrams is that, in rainbow boxes, a set is represented by a box, which is a single visual object. In case of holes, the pieces of the box are connected by a small line. On the contrary, in linear diagram, a set is represented by several, independent, visual objects: a label and one or more lines. This has two important consequences. First, in rainbow boxes, several boxes can be placed next to each other, leading to a more compact diagram. Second, in rainbow boxes, there are three "free" visual variables that can be used to represent additional information: height, color and texture. In a previous work, we explored the use of the box height as a visual variable [START_REF] Lamy | Translating visually the reasoning of a perceptron: the weighted rainbow boxes technique and an application in antibiotherapy[END_REF]. Rainbow boxes have a default color scheme that facilitates the discovery of similarities between sets, however, the color can also be set manually. The use of texture (hatches) will be demonstrated in section VI. On the contrary, the choice of visual variables in linear diagrams is more limited: texture and height can hardly be applied to a line, and color is already used to identify the sets.

B. Amino acid properties

Amino acids [START_REF] Wu | Amino acids: metabolism, functions, and nutrition[END_REF] are the building blocks for peptides and proteins. There are 20 major amino acids, each of them shares a common part, containing an amine and a carboxyl functional groups, and involved in the peptide bond. The various amino acids differ in their side chain, which is specific to each amino acid. Each side chain exhibits different physical and chemical properties, such as size, electric charge or hydrophobicity. The properties of amino acids are usually represented using a Euler diagram (Figure 2), which is the results of the Taylor classification [START_REF] Taylor | The classification of amino acid conservation[END_REF]. The diagram was generated using Multi-Dimensional Scaling (MDS).

Few alternatives to Euler diagram have been proposed for comparing amino acids, and they are often not as informative or as easy to read. Nagano et al. [START_REF] Nagano | Strong hydrophobic nature of cysteine residues in proteins[END_REF] proposed a dendrogram, showing the similarity and the distance existing between each pairs of amino acids. It was computed from the amino acid substitution scores. The dendrogram displays the proximity between amino acids, but it does not show explicitly the properties of each amino acid. Kosiol et al.

[13] also proposed to classify amino acids according to replacement rate, using a Markov process.

III. THE TWO DIAGRAMS

In the Euler diagram (Figure 2), we used the usual position for each amino acid. We added colors on the diagram, in order to make it more appealing and to have a fair comparison with (colored) rainbow boxes.

We previously proposed a rainbow boxes diagram for amino acids [START_REF] Lamy | Rainbow boxes: a new technique for overlapping set visualization and two applications in the biomedical domain[END_REF]. The diagram we present here (Figure 1) has been improved on four points. First, we replaced the 3-letter codes for amino acids by the more common 1-letter codes. Second, the 8 properties commonly presented in the Euler diagram were considered. Third, colors and contrasts were improved. Fourth, we also improved the order of the amino acids. In order to produce a diagram with the lowest possible number of holes in the boxes, the optimal column order must be found. This is a combinatorial optimization problem. The complexity of the problem is O(n!), where n is the number of columns. For amino acids, there are 20 columns, thus there are 20! 2.43 × 10 18 possible orders. All possible orders cannot be tested, even on a fast computer. The previously published rainbow boxes diagram was produced using the heuristic algorithm we proposed [START_REF] Lamy | Rainbow boxes: a new technique for overlapping set visualization and two applications in the biomedical domain[END_REF]. It had three holes, one in the "small" box and two in the "polar" box.

For improving the column order, we first used a bruteforce algorithm on subsets of the amino acid dataset to verify that no order exists that leads to fewer than 3 holes. The 3 holes are caused by the interaction between the following sets of boxes : { small, tiny, polar, hydrophobic }, { small, polar, hydrophobic } and { positive, aromatic, polar, hydrophobic }. Second, we searched for a column order that would place the three holes in the same box. The presence of holes in a box makes this box more difficult to perceive, consequently, it is better to have all holes in the same box. Moreover, some set visualization tasks involve several boxes, for example for searching for the intersection of two sets. By having all holes in a single box, we avoid the most complex situation for this task, i.e. the situation where the two considered boxes have holes.

We used the Artificial Feeding Birds (AFB) metaheuristics [14] for optimizing the column order, and we minimized the triplet (b, h, l) where b is the number of boxes with at least one hole, h is the number of holes and l is the total length of holes. We considered a lexicographic order when comparing two triplets (b, h, l) and (b , h , l ), e.g. [START_REF] Shade | Beyond the Venn diagram: the hunt for a core microbiome[END_REF][START_REF] Lamy | Using visual analytics for presenting comparative information on new drugs[END_REF][START_REF] Taylor | The classification of amino acid conservation[END_REF] was considered as lower (i.e. better) than [START_REF] Shade | Beyond the Venn diagram: the hunt for a core microbiome[END_REF][START_REF] Rodgers | A survey of Euler diagrams[END_REF][START_REF] Lamy | Translating visually the reasoning of a perceptron: the weighted rainbow boxes technique and an application in antibiotherapy[END_REF]. We found a column order with three holes in a single box, the "polar" one (Figure 1, corresponding to b = 1, h = 3 and l = 8).

IV. EXPERIMENT DESIGN

A. Recruitment 78 biology students were recruited at University Paris 13, at two levels: third year of bachelor degree (B3, 56 students) and first year of master (M1, 22 students). Most of them were female; this is usual in biology courses in France. The study was performed during the course of bioinformatics. Students were not aware that the study will take place at this moment, and all present students were included in the study. Students passed the study during 5 sessions (3 for B3 + 2 for M1 students). Students were told that the objective of the study was to compare two diagrams, but not that the Euler diagram is a well-established diagram and that rainbow boxes are a new "challenger" diagram. B3 students were "naive" subjects, i.e. they were not exposed to Euler diagram yet. On the contrary, M1 students have more diverse origins, and some of them might have already seen the Euler diagram in previous years. All students were new to rainbow boxes.

B. Protocol

The study was anonymous and no personal information was recorded (such as age or sex). We used a crossover protocol in which each student tested both diagrams, and thus can be his own control. For each diagram, the student had to reply to several questions related to amino acids, using the diagram. We recorded the error rate and the response time.

Two sets of similar questions, A and B, were defined. Five categories of tasks were considered, corresponding to typical tasks in overlapping set visualization: membership (does q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q #1 Eu RB #2 Eu RB #3 Eu RB #4 Eu RB #5 Eu RB 0 20 40 60 80 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q B3-Eu B3-RB M1-Eu M1-RB element X belong to set Y?), count (how many elements are there in set Y?), intersection (what are the elements in both set Y and set Z?), disjoint (are set Y and set Z disjoint?) and inclusion (is set Y included in set Z?). For each category, we defined two questions, one in each set A and B. In addition, a "warm-up" question was added at the beginning of each set; the results of these questions were not taken into account during analysis. In each set, questions were ordered so as consecutive questions do not refer to the same properties.

Table I shows the 12 questions and their order. Students were randomly divided in two groups, with 39 students in each group (28 B3 + 11 M1). The first group had the Euler diagram for the questions in set A and the rainbow boxes for questions in set B, and the second group had the rainbow boxes for questions in set A and the Euler diagram for the questions in set B. Thus, both groups had the same questions in the same order, but not with the same diagram. Finally, there was a last question, asking the student about his preferred diagram, with three possible choices: Euler diagram, rainbow boxes, or no opinion.

C. Software

A dynamic website was developed for the study, implemented in Python 3. The website was responsible for the randomization in the two groups. The first page of the website presented briefly the objective of the study. The second page presented the first diagram (depending on the group in which the user was randomized). The 6 following pages corresponded to the 6 questions of set A, with the associated diagram. Then the second diagram was presented, followed by the 6 questions of set B. When the response to a question was an amino acid, students had to click on the right amino acid in the diagram, and a "none" button was also available. When the response was a number of a Boolean, student had to choose the response from a set of predefined values (with 2 possible values for Boolean and 7 for numbers). Finally, the last page asked the question related to user preference. The website was in charge of collecting error rates and response times.

D. Statistical analysis

Statistical analysis was performed using R software version 3.3.2 [START_REF]R team development core, R: A language and environment for statistical computing[END_REF]. The significance threshold was set at α = 0.05. Error rates were compared with Fisher's exact test and Global Linear Model (GLM), considering the following factors: diagram (Euler or rainbow boxes), question category (1-5, see table I) and user category (B3 or M1). Response times were log-transformed to normalize the distribution.

They were analyzed with Welch Two Sample T test and ANalysis Of VAriance (ANOVA), using the same three factors. User preference was analyzed Chi-squared test.

The raw data and the statistical analysis in R can be downloaded at the following address: http://www.lesfleursdunormal.fr/static/_downloads/amino_acid_user_study.zip

V. EXPERIMENT RESULTS

All students performed the entire study and responded to all questions. We collected 390 responses with each diagram. 38 errors were recorded with the Euler diagram (error rate 9.7%, 95% confidence interval: 6.8 -12.7) and 19 with rainbow boxes (error rate 4.9%, 2.8 -7.0). This difference is significant (p value = 0.013, Fisher exact test). The analysis with Global Linear Model (GLM) showed that there were significant relationships between error rate and two factors: the diagram used and the question category. On the contrary, no significant relationship was found with the user category factor (B3 or M1), i.e. B3 and M1 students did not make a significantly different number of errors.

The analysis of error rates per category of question and per diagram (figure 3, left) shows that rainbow boxes led to fewer errors for all categories but # 5 (inclusion). Category #5 was the most error-prone category of question, with both diagrams, and the number of errors in this category was the same for both diagrams. The analysis of error rates per category of user and per diagram (figure 3, right) shows that B3 and M1 students had a similar global error rate. Both categories of user made fewer errors with rainbow boxes, but the difference was more accentuated for M1 students.

The mean response time was 11.57 seconds (95% confidence interval: 11.02 -12.12) with Euler diagram vs 9.63 seconds (9.10 -10.16) with rainbow boxes. This difference is significant (p value < 10 -6 , Welch two sample T test performed on log(time)). The analysis of the variance of response times (ANOVA) showed that the three factors were highly significant (diagram, category of question, category of user), and that there was no interaction between them. The analysis of response time per category of question and per diagram (figure 4, left) shows that rainbow boxes led to faster response times for all categories of question. The analysis of response time per category of user and per diagram (figure 4, right) shows that both B3 and M1 students responded faster with rainbow boxes.

19 students preferred the Euler diagram, while 51 preferred rainbow boxes (the 8 others indicated no opinion). We noted several comments that the students spontaneously made orally after the evaluation session. Many students commented on their preference for rainbow boxes or (more rarely) for Euler diagram. One student stated that the Euler diagram was more complex at the beginning, but then "we deal with it". Another student mentioned the fact that rainbow boxes were easy to read because the boxes were rectangular while the closed areas in the Euler diagram were rounded.

VI. ENRICHING THE DIAGRAM

In this section, we investigate the possibility to enrich the rainbow boxes diagram with additional amino acid properties. Figure 5 shows an enriched rainbow boxes diagram, with 3 additional properties: C-β branching, essential and functional. C-β branched amino acids contain two nonhydrogen substituents attached to their C-β carbon. This property was proposed by Betts et al. [START_REF] Betts | Bioinformatics for geneticists[END_REF], but not visualized in a diagram.

Essential amino acids [START_REF] Aristoy | Handbook of analysis of active compounds in functional foods[END_REF] cannot be synthesized by Human cells. This is an important property in the nutrition field. Semi-essential amino acids (G, C and Y) can be synthesized, but not in sufficient quantity; they are displayed with hatches in the diagram. On figure 5, the similarity between the "hydrophobic" and the "essential" boxes is striking. The relation between amino acids hydrophobicity and essentialness has already been described [START_REF] Widyarani | Production of hydrophobic amino acids from biobased resources: wheat gluten and rubber seed proteins[END_REF]. But rainbow boxes made it clear that almost all hydrophobic amino acids are essential, excepted one, alanine (A). Notice how the box colors enhance this similarity: the two boxes have almost the same, light green, color.

Functional amino acids are those frequently involved in interaction with non-protein compound. The data presented here are based on the values computed by Holm et al. [START_REF] Holm | Decision support system for the evolutionary classification of protein structures[END_REF]. The authors computed a score for each amino acid, expressed as a real number. The score is positive if the amino acid tends to participate in non-protein interaction more than average, and negative otherwise. In the rainbow boxes, we used a plain box for amino acid whose score is superior or equal to 0.1, and a hatched box for amino acid whose score is superior or equal to 0.05 (but inferior to 0.1). Rainbow boxes clearly shows that all functional amino acids are polar. This is sounding, because many non-protein interactions involve ions or other polarized molecules.

VII. DISCUSSION AND CONCLUSION

In this paper, we presented an improved diagram for amino acid properties based on rainbow boxes, and we compared the ability of biology students to answer questions using the proposed diagram versus the traditionally used Euler diagram. The evaluation results show that the students made significantly fewer errors with the new diagram, they responded significantly faster, and they preferred it. Results were similar between bachelor and master students. We also proposed an enriched diagram with three additional properties. Consequently, the new rainbow boxes diagram of amino acid properties could be used in textbooks and courses, in complement (or instead) of the usual Euler diagram. High-definition images of the new amino acid diagram can be downloaded at the following address, in

A. User study

The user study showed that inclusion was the most error prone type of question, for both diagrams. This is quite surprising, because Euler diagrams are known to be efficient for visualizing inclusions. However, rainbow boxes performed equally well in terms of the number of errors (figure 3, left). The rectangular aspect of rainbow boxes was mentioned by a student as a reason that makes them easier to read than the Euler diagram. This argument is confirmed by a recent study that shows that horizontal and vertical lines are better perceived than diagonal ones [START_REF] Burch | A user study on judging the target node in partial link drawings[END_REF].

We used colors in rainbow boxes. Nevertheless, colors are not essential for the reading of the diagram, and therefore a colorless rainbow boxes diagram is still possible whenever colors are not available (e.g. for black-and-white printing or for color-blind persons).

B. Euler diagrams vs rainbow boxes

Compared to Euler diagrams, rainbow boxes are easier to generate automatically. In previous works, we proposed methods for producing diagrams up to 20-25 elements (using the heuristic algorithm) or 50-100 elements (using the AFB metaheuristic); the number of sets have little impact on the optimization problem and is rather limited by the height of the screen. On the contrary, the automatic drawing of Euler diagrams is a challenge above 6 sets; Simonetto et al. proposed a method that can produce an Euler diagram with 60 sets [START_REF] Simonetto | Fully automatic visualisation of overlapping sets[END_REF] but the resulting diagram is complex. Rainbow boxes also seem to be able to present larger dataset (as we seen in section VI when enriching the amino acid diagram).

C. Perspectives

Students were enthusiasts and they considered the evaluation like an exercise. They asked for consulting their personal results, and especially to be able to see in which questions they made errors. However, our evaluation website was not providing this feature. Students found interesting to learn about amino acids properties, but also to learn how to use Euler diagrams and rainbow boxes. A perspective of this work is to transform our evaluation software into a training software.

Venn diagrams are quite frequent in the biomedical field. In bioinformatics, Venn diagrams are used for comparing several gene lists. Two typical examples are the comparison of the genes common between several species, and the comparison of the genes found as differently expressed using several methods (e.g. microarrays vs next-generation sequencing (NGS), or microarray data analyzed with various software). Proportional Venn diagrams have also been used in medicine, for presenting proportional classifications of phenotypic subgroups for a given disease. For instance, several such diagrams have been proposed for obstructive lung diseases and Chronic Obstructive Pulmonary Disease (COPD) [START_REF] Marsh | Proportional classifications of COPD phenotypes[END_REF] and Behçet's disease-related manifestations [START_REF] Dinc | The proportional Venn diagram of Behçet's disease-related manifestations among young adult men in Turkey[END_REF]. All these applications are potential perspectives for rainbow boxes.

Figure 2 .

 2 Figure 2. Euler diagram displaying 8 properties of the 20 amino acids (known as "the Venn diagram of amino acid").

Figure 3 .

 3 Figure 3. Bar charts showing the error rates with each diagram (Eu : Euler diagram, RB : rainbow boxes) for each category of question (#1-5, left) and user (B3 or M1, right).

Figure 4 .

 4 Figure 4. Box plots showing the response time for each diagram (Eu : Euler diagram, RB : rainbow boxes) and each category of question (#1-5, left) and user (B3 or M1, right).

Figure 5 .

 5 Figure 5. Enriched rainbow boxes diagram displaying 11 properties of the 20 amino acids.

  [START_REF] Nagano | Strong hydrophobic nature of cysteine residues in proteins[END_REF] QUESTIONS WITH THE CORRECT ANSWERS. QUESTION #S INDICATE THE ORDER OF THE QUESTIONS IN THE SET.

		# Category	# Question	Right answer
		(Warm-up)	1 Click on the small and aliphatic amino acid	Valine (V)
	Set A	1 Membership 2 Is Valin (V) polar? 2 Count 5 How many tiny amino acids are there? 3 Intersection 6 Click on the aromatic and positive amino acid	No 4 Histidine (H)
		4 Disjoint	3 Click on the tiny and positive amino acid	None
		5 Inclusion	4 Are all small amino acids polar ?	No
		(Warm-up)	1 Is Cysteine (C) small?	Yes
	Set B	1 Membership 2 Is Tyrosine (Y) hydrophobic? 2 Count 5 How many positive amino acids are there? 3 Intersection 3 Click on the small and negative amino acid	Yes 3 Aspartate (D)
		4 Disjoint	4 Click on the aromatic and aliphatic amino acid None
		5 Inclusion	6 Are all aromatic amino acids hydrophobic ?	Yes
			Table I


Jean-Baptiste Lamy
email: jean-baptiste.lamy@univ-paris13.fr

Visualizing symmetric square matrices with rainbow boxes: methods and application to character co-occurrence matrices in literary texts

Keywords: Knowledge visualization, matrix visualization, overlapping set visualization, symmetric square matrix, distant reading, digital humanities

A symmetric square matrix is a type of dataset frequently found in various domains. An example is character co-occurrence matrices in digital humanities. However, the visualization of these matrices is difficult. In this paper, we propose a method for visualizing symmetric square matrices, by transforming a matrix into overlapping sets, and then by visualizing these overlapping sets using rainbow boxes, a recent set visualization technique. We apply this method to the visualization of character matrices in novels, using two datasets: a small one (21 characters) and a large one (80 characters). We show that this visualization allows the finding of several insights. Finally, we discuss the advantages and drawbacks of this method, and we compare it to other approaches in the literature.

I. INTRODUCTION

Symmetric square matrices and co-occurrence matrices are a kind of dataset that can be encountered in many domains [START_REF] Leydesdorff | Co-occurrence matrices and their applications in information science: Extending ACA to the Web environment[END_REF]. Any undirected graph can be represented by an adjacency matrix, leading to a symmetric square matrices. Such matrices are frequently encountered in visual text analysis [START_REF] Jänicke | Visual text analysis in digital humanities[END_REF], which applies information visualization methods to digital humanities. When the structure of the text is not preserved by the visualization, this is called distant reading (by opposition to close reading). In particular, character co-occurrence matrices can be extracted from novels and narrative texts, and visualized with various techniques. An example of a symmetric square matrix, representing the relations between the 21 main characters in a novel, is shown in Figure 2.

In this paper, we propose a method for visualizing symmetric square matrices, by transforming a matrix into overlapping sets, and then by visualizing these overlapping sets using rainbow boxes. Rainbow boxes is a recent set visualization technique [START_REF] Lamy | Rainbow boxes: a new technique for overlapping set visualization and two applications in the biomedical domain[END_REF], [START_REF] Lamy | Rainbow boxes: a technique for visualizing overlapping sets and an application to the comparison of drugs properties[END_REF]. In rainbow boxes (see example on Figure 1), elements are represented in columns and each set is represented by a rectangular box that covers the columns corresponding to the elements belonging to the set (the set label can be shown inside the box, however, in Figure 1 the sets are unlabeled). When the elements of a given set cannot be placed next to each other, holes are present in the box (e.g. the yellow box on the left of Figure 1 has one hole of size 2). The column order is optimized for minimizing the number of holes in the visualization. We initially developed rainbow boxes for the comparison of drug properties [START_REF] Lamy | Using visual analytics for presenting comparative information on new drugs[END_REF] and then we applied them to biomedical data and knowledge. Here, we present an application of rainbow boxes in a totally different domain.

The method we propose is particularly aimed toward the identification and the representation of subsets of interrelated elements, for example in a novel, groups of characters that know each other.

The rest of the paper is organized as follows. Section II describes the visualization method we propose. Section III presents the application of the method to the visualization of character matrices, with two datasets: one small (21 characters) and one large (80 characters) and show various insights that can be produced from these visualizations. Section IV discusses the method and the results, and compares our approach to literature, before concluding.

II. METHODS

A symmetric square matrix can be formalized as M = (M i,j ) 1≤i≤n,1≤j≤n ∈ R, with M i,j = M j,i . Our method for visualizing a symmetric square matrix follows two steps: first, producing a set of overlapping sets from the matrix, and second, visualizing these overlapping sets using rainbow boxes.

For the purpose of step 1, we consider each row/column of the matrix (i.e. each index i ∈ E = {1, 2, ..., n}) as an element, and then we define several overlapping sets made of these elements. Each set will include only elements related to each other.

For describing the method, and facilitating customization, we will consider 3 functions: the selection function, the aggregation function and the colorization function. The selection function select : R → {T rue, F alse} indicates, for a given value in the matrix, if the two elements are considered as related, i.e. i and j are considered as related if (and only if) select(M i,j ) = T rue. For example, in the matrix of Figure 2, two elements are considered as related if the value in the matrix is not zero.

Using the selection function, we can generate all element subsets s that include only elements related to each other (e.g. when considering a character matrix, all characters in a given subset s know all the other characters of the subset). We name S 0 the set of all subset s.

S 0 = s ⊆ E | |s| > 1 ∧ ∀(i, j) ∈ (s, s) with i = j, select(M i,j ) = T rue
Then we compute S, the set containing only the largest subsets s in S 0 (i.e. we only keep subsets s for which no superset exists in S 0 ). S = {s ∈ S 0 | ∃s ∈ S 0 , s ⊂ s }

The subsets s in S are the largest sets of interrelated elements. These subsets can be visualized using rainbow boxes: each element in E (corresponding to row/columns of the matrix) will be represented by a column in rainbow boxes, and each subset will be represented by a rectangular box. The box covers all columns corresponding to the elements in the subset.

In addition to set membership, the values present in the matrix can be represented in rainbow boxes using colors. For a given element i ∈ E in a subset s ∈ S, there can be several values in the matrix. For example, if we consider s = {i, j, k}, two values are present in the matrix for i and the other elements of s: M i,j and M i,k . In the general case, the number of values is s -1. For the sake of visualization, we choose to aggregate these values into a single value and then to represent it using colors. The aggregation is achieved by the function aggregate : R p → R (with p ≥ 1). It takes one or more values from the matrix, and returns a single aggregated value.

Finally, the colorization function colorize : R → color takes as parameter the aggregated value and return a color. The color will be applied to the rectangular box in the corresponding column (i.e. colors are defined on a persubset/box and per-element/column basis, thus a given box may have different colors).

Rainbow boxes require to optimize the column order, for minimizing the number of holes in the boxes. In previous works, we proposed a heuristic algorithm for solving this problem with up to 20-25 columns. However, one of the two datasets we will present in this paper includes 80 characters, leading to 80 columns. Therefore, we used the Artificial Feeding Birds (AFB) metaheuristic [START_REF] Lamy | Advances in nature-inspired computing and applications[END_REF] for optimizing column order. AFB is a nature-inspired metaheuristic [START_REF] Yang | Nature-inspired metaheuristic algorithms[END_REF], inspired by the behavior of pigeons, and is able to solve both non-linear and combinatorial optimization problems.

III. APPLICATION TO CHARACTER MATRICES

A. Small dataset (Sombre comme l'Aurore)

Figure 2 shows a character matrix for Sombre comme l'Aurore, a French yet-unpublished novel written by the author of the paper. This matrix includes the 21 main characters of the novel and was produced manually by the author. A rather conservative definition of "character relation" was considered: two characters are considered as related if it is plausible that they met together outside of the events directly reported in the novel. Since the novel covers a period of one year, not all events are reported and, in particular, characters that are friends to each other are likely to meet much more often that told in the novel. On the contrary, simple encounters that do not lead to regular relationship are not considered as "relation" in this matrix. Consequently, the matrix is not a simple co-occurrence matrix. When a relation holds between two characters, the matrix indicates by an integer number the part of the novel during which their relation starts: 1 means before the story begins, and 2-5 correspond to part #1-4 in the novel, respectively. Finally, 0 in the matrix indicates the absence of relation between two characters. We used the previously described method for transforming this matrix in overlapping sets, and for visualizing these sets using rainbow boxes. The three parameter functions were defined as follows. The selection function selects all pairs in the matrix were the value in the matrix is not zero. select : p → T rue if p = 0, F alse otherwise

The aggregation function returns the minimum value (consistently with the content of the matrix, which indicates the first part of the novel where two characters have a relation). aggregate : p 1 , p 2 , ... → min(p 1 , p 2 , ...)

The colorization function produce a bright color, whose hue indicates the first part of the novel in which a character joins a group, using the color key shown at the bottom of Figure 1. Cold/hot colors correspond to the beginning/end of the story, respectively.

colorize : a →                violet if a = 1 cyan if a = 2 green if a = 3 yellow if a = 4 red if a = 5
Figure 1 shows the rainbow boxes representing the matrix in Figure 2. The 21 characters are represented by columns. Each box represents a group of interrelated characters, i.e. any character in a given group is related to any other character of the group. For example, the top-most box indicates that Le Capitann, Raken, Tienn and Alyse are related to each other. In addition, the colors indicate when a character joined the group. The top-most box is entirely violet, indicating that the four characters were related to each other before the story begins. On the contrary, the box covering Alyse, Chéram, Gauve, Méric, Rashkang and Lescantelles is bicolor: Alyse, Chéram, Gauve and Lescantelles formed a group before the story begins (violet color) while Méric and Rashkang joined this group in part #1 (cyan color). Notice that, since we used the minimum function as aggregation function, the fact that Alyse is in violet in this box does not imply that she is related to all other characters in the group before the story begins: she may be related to only some characters in the group before the beginning, and then develop relations with the others later.

Several new insights can be obtained from Figure 1: (1) Many groups are already present before the story begins (the violet color is dominant). ( 2) Tienn is the character that belongs to the highest number of groups (five rectangular boxes in the Tienn column). Indeed, Tienn is the hero of the novel. (3) Tienn belongs to two groups before the story begins (the two violet boxes in his column). (4) In the first part of the novel, Tienn joins an already formed group of characters (La Rasinne, Werfam, Auguss and Le Lamineur). In fact, this corresponds to his colleagues in his first job. [START_REF] Lamy | Using visual analytics for presenting comparative information on new drugs[END_REF] This group will be joined by Nicol in part #4 (red segment in that box). He is a new colleague. [START_REF] Lamy | Advances in nature-inspired computing and applications[END_REF] In part #2, Tienn encounters Crépusculine (the green box at the bottom). In addition, we can see that Tienn and Crépusculine have no common friends at all (the green box includes no other characters), and that they are very distant in the visualization. Indeed, they come from two different worlds, and they fall in love. ( 7) Moreover, we can search the shortest path between Crépusculine and Tienn (ignoring their direct relation). This path is Crépusculine → King Saphir → Werfam → Tienn. [START_REF] Wu | Handbook of data visualization[END_REF] In part #3, Tienn joins a new group of three characters (Mabine, Malfred and Érard) that have no relations with any other characters. Indeed, those characters live isolated, in a kind of ghetto.

B. Large dataset (Les Misérables)

A well-known dataset from the Stanford GraphBase includes a character graph from Les Misérables, a famous novel written by Victor Hugo. The graph contains 80 characters and indicates their co-occurrences in the novel. Each co-occurrence is a separate relation in the graph, and is labeled with the part and chapter in which it occurs. We transformed this dataset into two symmetric squared matrices, with characters as rows and columns, one including the parts of the novel and the other the numbers of cooccurrence.

The first matrix, M , indicates for each character pair the part (1-5) of the novel during which the first co-occurrence occurs. This matrix contains values similar to the one shown in Figure 2. It was visualized using the same selection and aggregation functions as in section III-A. Since no character joined a new group in part #5, the colorization function was modified to produce only 4 different colors (violet, cyan, green, red).

The resulting rainbow boxes are shown in Figure 3. The following insights can be obtained from the figure : (1) Most of the new character-group relations occur at the first and third part of the novel (violet and green colors are dominant).

(2) Jean Valjean is the character that belongs to the highest number of groups. Indeed, he is the main character of the novel. (3) Cosette appears in part #2 and also joins new groups in part #3. (4) Gavroche appears in part #3, in a single group, but joins many other groups during part #4, two of them including Jean Valjean. ( 5) the characters seem quite well separated with regards to the parts of the novel: characters on the left appear mostly in part #3-4 while those in the right appear in part #1. The Thénardiers are the most noticeable exception: they appear in part #1, but join several groups later during parts #2 and #3.

The second matrix, M , indicates for each character pair the number of co-occurrences (between 0 and 31). We visualized it using the previously defined selection function, and an aggregation function that computes the sum of the co-occurrences. Thus, the aggregation function returns the total number of co-occurrence between the character and the rest of the group. The colorization function produce "hotspot" colors, mapping the lowest value (0) to violet and the highest value (77) to red, using a hue-based color gradient between the two extremes. Thus, cold/hot colors correspond to characters having few/many relations with the group.

In addition, the rainbow boxes column optimization was biased in order to favor holes in boxes associated with fewest co-occurrences (hence resulting in a different character order than in Figure 3). For each box, we computed m cooc the mean number of co-occurrence. Then, the cost for adding a hole in a given box was proportional to m cooc . This prevent splitting boxes corresponding to the most important (in terms of the number of co-occurrence) groups of characters.

The resulting rainbow boxes are shown in Figure 4. We can observe that: (1) Jean Valjean appear as the "central" character. (2) Myriel has several relations, but with very few co-occurrences (violet color). (3) There is a group of 8 interrelated characters (from Blacheville to Fantine, the cyan box on the left) that have very few relations with other characters (only Fantine and Félix are belonging to other groups). ( 4) In most groups he belongs to, Jean Valjean is the most interacting character (hotter color in his column); the group with the Gillenormands, Marius and Cosette is an exception, with Marius and Cosette being more active. [START_REF] Lamy | Using visual analytics for presenting comparative information on new drugs[END_REF] There are some highly interacting characters from Gavroche to Feuilly, spread over three groups, and Courteyrac and Enjolras are the most active in these groups.

In both figures, we can see that Javert is placed next to Jean Valjean (they have 11 groups in common). Indeed, Javert is a policeman tracking Jean during most of the novel. In addition, a characteristic pattern of triangular "Christmas tree" (fir) can be observed around Myriel and (to a lesser extent) Jean Valjean. This pattern indicates that the central character (forming the trunk of the tree) has many isolated relations with other characters (the leaves), i.e. the central character is related to many other characters without having common friends with them. Depending on the global shape of the rainbow boxes, this pattern can be bilateral (as in Figure 3) or unilateral (as in Figure 4 for Myriel).

IV. DISCUSSION AND CONCLUSION

In this paper, we proposed a method for visualizing symmetric square matrices using rainbow boxes, a set visualization technique not initially aimed at representing matrix datasets. We applied this method to the visualization of character matrices in novels. We showed that this method was efficient for identifying groups of interrelated elements and generating new insights. Thus, it may be interesting for visualizing highly connected networks. We also described a characteristic pattern of "Christmas tree" for elements related to many isolated others. The two datasets we presented differ in size, but also qualitatively. The first one was produced manually by the author, using a conservative definition of "relation" between two characters. On the contrary, the second one was built automatically by considering co-occurrence of characters in the novel. Figure 1 is visually much simpler and easier to read than Figure 3, this can be explained by the smaller size of the dataset, but possibly also by the difference in the nature of the matrices. It would be interesting to extract a co-occurrence matrix from Sombre comme l'Aurore, and to compare it with the manually produced matrix.

When building overlapping sets from matrices, we considered only pairwise relations. Thus, the character groups we extracted do not necessarily exhibit higher-level relations. For example, if A is related to B and C, and B is related to C, then A, B and C form a group. However, it does not necessarily imply that A, B and C meet all the three (i.e. ternary relation). Consequently, A might be unaware that B and C are related.

With regard to the matrix size, the proposed method is limited by the number of columns that can be shown on a screen, and by the optimization algorithm that reorders columns. On the 80-character dataset, the optimization took several minutes. As a consequence, the method is currently limited to matrices with fewer than 100 rows/columns. This limit might be overcome by performing clustering beforehand, in order to group similar rows/columns and reduce their number.

In the literature, various approaches have been proposed for the visualization of symmetric square matrices. First, these datasets are frequently visualized as colored matrices after reordering rows and columns [START_REF] Wu | Handbook of data visualization[END_REF]; for instance, an online matrix visualization of Les Misérables can be seen at https://bost.ocks.org/mike/miserables/. An example of matrix-based tools is MatLink [START_REF] Henry | MatLink: Enhanced matrix visualization for analyzing social networks[END_REF], which has been applied to the analysis of social networks.

Matrix reordering methods allow the identification of interrelated character groups: those groups form squares on the matrix diagonal (or triangle if only half of the matrix is shown). For example, in Figure 2, 5 such groups can be seen on the diagonal. However, with this approach, a character belongs only to a single group, or at most two if the character is placed in-between two squares on the diagonal. For instance, Tienn has been placed at the beginning of the matrix in Figure 2, but could also be placed at the end (due to the three "4" at the end of the Tienn's row). On the contrary, Tienn belong to not less than 5 groups in the rainbow boxes representation in Figure 1, and 9 interrelated groups have been identified (4 additional groups compared to the matrix reordering methods). Similarly, in Figure 4, we can see several overlapping groups at the top of the visualization, most of them including Jean Valjean. In addition, rainbow boxes allow a compact representation of the dataset: a group of n characters that know each other occupies n×(n-1) 2 cells in the matrix, but the entire group is represented as a single box in rainbow boxes, with a length of n columns.

Second, a character matrix can be treated as a similarity matrix. Therefore, dimension reduction techniques, such as Principal Component Analysis (PCA) or Multidimensional Scaling (MDS) [START_REF] Borg | Applied multidimensional scaling[END_REF], can be used in order to transform a matrix into a two-dimensional scatter plot, a topological landscape or a knowledge map. An example is the Text Variation Explorer [START_REF] Siirtola | Interactive Text Visualization with Text Variation Explorer[END_REF], which uses PCA to represent various sociolinguistic features in text fragments using a scatter plot. Another example is Memory Islands [START_REF] Yang | Creating knowledge maps using Memory Island[END_REF], which integrate hierarchical knowledge, e.g. from an ontology.

Third, a co-occurrence matrix can be represented as an undirected graph or a network. However, those graphs often become difficult to read when the number of nodes increases. For instance, word co-occurrences have been visualized using graphs for analyzing scientific literature on patient adherence [START_REF] Zhang | Mapping the Knowledge Structure of Research on Patient Adherence: Knowledge Domain Visualization Based Co-Word Analysis and Social Network Analysis[END_REF]. Graphs have also been used for visualizing plagiarism in pieces of music, i.e. co-occurrences of several identical notes in several pieces [START_REF] De Prisco | Music plagiarism at a glance: metrics of similarity and visualizations[END_REF].

Fourth, chord diagrams have been proposed for the visualization of character co-occurrences in novels [START_REF] Bilenko | The narrative explorer[END_REF]. The chord diagram displays all characters on a ring, and represent cooccurrences by "chords" linking two characters, and located in the middle of the ring. It allows selecting a character and observing its co-occurrences with other characters, but it does not help with the identification of groups of interrelated characters.

The perspectives of this work are (1) to extend the proposed method to the visualization of non-symmetric square matrices (including directed graphs), (2) to apply a similar method using overlapping set visualization techniques other than rainbow boxes, such as Euler diagrams, and (3) to apply the method to other domains, such as the visualization of FOAF (Friend Of A Friend) graphs in social media, proteinprotein interaction matrices in bioinformatics, drug-drug interaction matrices in pharmacology, or matrices generated with Linkography [START_REF] Hsieh | Whether the relationscape of Interaction design strategies during design process can be explained by Linkography[END_REF].

Figure 1 .

 1 Figure 1. Rainbow boxes displaying groups of interrelated characters in a novel. Characters are in columns, and each rectangular box corresponds to a group of characters. In a given group, each character is related to each other character of the same group. Box colors indicate the part of the novel during which a character joins the group.

Figure 2 .

 2 Figure 2. Example of a symmetric square matrix showing the relations between 21 characters in a novel. Numbers in the matrix indicates the parts of the novel in which the two characters meet for the first time (0: no relation, 1: before the novel starts, 2: during part #1, etc).

  aggregate : c 1 , c 2 , ..., c m → c 1 + c 2 + ... + c m

Figure 3 .

 3 Figure 3. Rainbow boxes displaying the symmetric square matrix of character co-occurrence in Les misérables. Hue (violet, cyan, green, red) indicates the parts (1-2-3-4) of the book, respectively. Characters mentioned in the text are highlighted with red dots.

Figure 4 .

 4 Figure 4. Rainbow boxes displaying the symmetric square matrix of character co-occurrence in Les misérables. Hue indicates the frequency of character interactions (hotter colors corresponding to characters that interact more frequently with the group).