Mathew Hillier
email: mathew.hillier@monash.edu

Nathaniel Lyon
email: nathaniel.lyon@monashcollege.edu.au

Keywords: e-Exams, writing strategies, student perceptions, affordances

This study examined students' expressed strategies, habits and preferences with respect to responding to supervised text-based assessments. Two trials of a computerised examination system took place in an Australian preuniversity college in 2016 and 2017. Students in several classes studying geography and globalisation completed a sequence of practice and assessed work. Data were collected using pre-and post-surveys about their preferred writing styles, habits and strategies in light of their choice to type or handwrite essay and short answer examinations. Comparisons were made between those that elected to handwrite and those who chose to type the examination, with several areas being significant. The performance (grades), production (word count) of the typists and hand-writers were also correlated and compared.

Introduction and background

This study is part of a nationally funded project [1 -3] looking at modernising supervised high stakes assessment within the Australian higher education context. This paper builds on previous e-examination (e-Exam or e-exam) trials held in Australian universities, investigating how students in a pre-tertiary pathway college context perceive the task of writing e-Exams.

We use the term e-Exam to specifically refer to a "timed, supervised, summative assessment conducted using each candidate's own computer running a standardised operating system" [START_REF] Fluck | eExams: Strength in Diversity[END_REF]. This differentiates our work from those employing 'online' testing tools that take the format of a data collection instrument (e.g. Moodle quiz, Blackboard test, TCExam, QuestionMark Perception, ExamSoft). In our case, we include the use of 'authentic' software applications fit for the purpose of the assessment task, e.g. a full office suite is provided to write reports or essays.

We will explore the literature related to the writing of high stakes assessment tasks and in particular computerised examinations before moving on to explain the process used in the study. This is followed by survey results and a discussion of the findings that draws out implications for future research and practice.

Literature

The idea of using computers for assessment has been around for 60 years [START_REF] Swets | Computer-Aided Instruction[END_REF], yet penon-paper still dominates most higher education and school examinations around the world. However, attention has recently shifted to modernising the examination room. The 'Dublin Declaration' [START_REF]Tomorrow's Learning: Involving Everyone. Learning with and about Technologies and Computing[END_REF] features the idea of 'authentic assessment' [START_REF] Crisp | Towards Authentic e-Assessment Tasks[END_REF] as one of its core recommendations when considering the use of computers for assessment. The Declaration specifically recommends that an e-Exam must be an:

"authentic assessment that matches modern workplace practices and many student learning experiences". (p. xviii)

If we consider the modern classroom, or the majority of work places, we find that computer and software technology is near ubiquitous. From accountancy to zoology, computers are now key tools of the trade. From keeping track of laboratory experiments to report writing and bookkeeping, we could be confident that a computer was involved. Similarly, in higher education, reports, essays and communications are conducted via the typed medium. The vast majority of students in the developed world today use software tools as part of their course work and in addressing unsupervised assessment tasks. University students could hardly remember or even know what it would be like to hand-write their class work, yet we ask them to write examinations that can take up to three or more hours. The keyboard is now so commonly used that the very skill of handwriting is in decline [START_REF] Sülzenbrück | The Death of Handwriting: Secondary Effects of Frequent Computer Use on Basic Motor Skills[END_REF], with a subsequent loss of the motor skills required to write proficiently. A previous study [START_REF] Hillier | To type or handwrite: student's experience across six e-Exam trials[END_REF] found that hand-writers in examinations beyond about 70 minutes felt physical discomfort while typists were unaffected. Fortunately, there are a number of examples where the transition to e-exams is well underway or at least beginning. Prior work [START_REF] Fluck | eExam symposium: design decisions and implementation experience[END_REF][START_REF] Fluck | eExams: Strength in Diversity[END_REF] described and reviewed several e-Exam projects, with a longer list available [START_REF] Hörnblad | Digital Exam. SUNET Incubator project[END_REF].

As part of the conversion effort, teachers and students may have an adjustment period when transitioning from hand-writing under examination conditions to typing. A survey [START_REF] Dermo | E-assessment and the student learning experience: A survey of student perceptions of e-assessment[END_REF] of students at the University of Bradford in the United Kingdom (UK) on their experience of using QuestionMark Perception covered a range of topics. Differences between handwriting and typing an essay-style examination were explored [START_REF] Mogey | Factors influencing student preference when comparing handwriting and typing for essay style examinations: Essay exams on computer[END_REF], and the authors looked at self-reported typing prowess, confidence, stress, use of time, writing strategies, pre-planning, structuring, editing and reviewing prior to submission. Similarly work by other authors [START_REF] Purcell | Exams: Comparing Handwritten Essays with those Composed on Keyboards (Final Report)[END_REF] looked at the preferences of students with regard to typing or handwriting essay examinations when they were given the choice. They noted a 10% uptake by students of the typing option. Another researcher [START_REF] Hillier | The Very Idea of e-Exams: Student (Pre)conceptions[END_REF] reported on students' preconceptions of what an e-exam may involve prior to the start of several e-Exam trials at an Australian university, with interest by students varying significantly between discipline areas. A follow up study of e-exam trials in six courses at the same institution [START_REF] Hillier | To type or handwrite: student's experience across six e-Exam trials[END_REF] showed the typing option was selected by 5% to 34% of students. The top three comments from typists were that they could type faster, it was neater than handwriting, and that they were easily able to edit their work leading to more polished responses.

Finally, we turn to the matter of performance. Stakeholders are concerned that computerising the examination room may have an uneven impact on student performance by advantaging some and disadvantaging others [START_REF] Mogey | Typing compared with handwriting for essay examinations at university: letting the students choose[END_REF]. Concerns over typing prowess, speed, and computer access have been raised. However, similar concerns could be raised with respect to the uneven abilities of students in terms of speed, neatness and physical strength for handwriting long examinations (ibid). Performance in an e-exam in a dentistry unit found that there was a moderate increase in marks for those that typed the examination compared to those handwriting. There could be no causal link established, due to the self-selecting sample groups [START_REF] Fluck | Innovative assessment with eExams[END_REF]. Handwriters that produced more words generally did better while more typed words did not see marks increase by as much. Another study [START_REF] Fluck | eExaminations Strategic Project Final Report for Academic Senate[END_REF] compared typists and hand-writers and found that the typists produced around 20% more words in an e-exam. Similarly, a further study [START_REF] Mogey | Typing compared with handwriting for essay examinations at university: letting the students choose[END_REF] found that typists produced more words than hand-writers but there was no statistically significant difference in grades. In terms of presentation effects, one researcher [START_REF] Rankin | A comparability study on differences between scores of handwritten and typed responses on a large-scale writing assessment[END_REF] found no significant difference in scores due to presentation mode (typed or transcribed handwritten) on a large-scale writing assessment. Prior studies have also looked at demographics such as gender [START_REF] Terzis | Computer based assessment: Gender differences in perceptions and acceptance[END_REF] and how this may impact a student's performance when faced with a typed examination.

The review of prior work has outlined several areas of focus that informed questions to be investigated (see Table 1), and provided an up-front frame for analysis.

Table 1. Areas for investigation

Area

Example research questions

Rationale of students

What proportion of candidates were in favour of typing their examination? What rationale was provided for their choice? Did the e-Exam environment support their writing? Writing strategies Were there differences in the writing preferences and strategies used by those that typed and those that handwrote? Student performance Were there differences in words produced and grades achieved by those that typed and those that handwrote?

Study context

In Australia, the first author is leading an Australian federal government funded grant project [START_REF] Teaa | Transforming exams across Australia[END_REF] investigating how authentic e-assessment can be introduced into the examination rooms of universities using bring-your-own-devices (BYOD). This study was undertaken within two units offered in the Foundation Year [START_REF]Foundation Year[END_REF] programme at Monash College, Australia. The programme is the equivalent of an Australian year 12 high school certificate or the International Baccalaureate. The second author is a unit coordinator and teacher of the units in which the e-exam trials were conducted. The trials were carried out using in-class supervised written assessments requiring short text and essay-style responses.

Method and approach

Students in several classes undertaking geography (Geo) in semester 1, 2016 and globalisation (Glo) in semester 2, 2017 at Monash College took part in a two phase trial of the e-Exam system. Ethics protocol approval was gained via Monash University prior to the trials. In this study a fully functional word processor was used that was part of the e-Exam platform as previously described [START_REF] Hillier | Transforming Exams -How IT works for BYOD e-Exams[END_REF]. Students used a custom Live Linux universal serial bus (USB) environment on their own laptops that provided a full office suite. They could use editing tools such as spelling, grammar and highlighting to help with their writing tasks. Being able to copy, delete or move text around with ease meant that there were functional differences to undertaking the same task using pen-on-paper.

In phase one, students participated in an in-class, ungraded preview session. They were provided with instructions on how to use the e-Exam system with laptops and opportunity to practice e-exam processes, use the software, and try the question response format. Students completed a pre-examination survey requesting technical information about their laptop and first impressions of the e-Exam process and software. Attitudes and responses to e-Exam software use are reported separately [START_REF] Hillier | Student experiences with bring your own laptop e-Exams in preuniversity college[END_REF].

Phase two occurred two weeks later in the form of an in-class, graded, supervised assessment task; students could choose to type or handwrite. Materials were provided on paper or as a word processor document and included photographs, diagrams, charts and data tables. In Geo, the assessment task was a single case study with an extended essay response. In Glo, two short answer sections and a mini-case essay response were required. Students then completed a post-examination survey before leaving the room. An extended account of the trial process is provided elsewhere [START_REF] Hillier | Student experiences with bring your own laptop e-Exams in preuniversity college[END_REF].

Qualitative survey data relating to students' opinions on writing in examinations was analysed using SPSS v24. Likert items were treated as non-parametric as per advice [START_REF] Jamieson | Likert scales: how to (ab)use them[END_REF]. This stance is supported by other authors [START_REF] Dermo | E-assessment and the student learning experience: A survey of student perceptions of e-assessment[END_REF] when analysing students' perceptions of e-Assessment. The Mann-Whitney U test [START_REF] Mann | On a Test of Whether One of Two Random Variables is Stochastically Larger than the Other[END_REF] was used to test the variance between groups (males versus females and typists versus hand-writers) of Likert responses. Chi-squared was also used to test if gender played a role in the decision to type the examination. A Fisher exact test [START_REF] Mann | On a Test of Whether One of Two Random Variables is Stochastically Larger than the Other[END_REF] was used when comparing categorical variables. When it came to performance data (grades) a T-test was used to compare between groups while a Spearman test [START_REF] Spearman | The proof and measurement of association between two things[END_REF] was used to test for correlation between word count and grades. We used an alpha level of .05 for all statistical tests, unless otherwise noted. However, the participants were not randomly assigned to the typing or hand-writing groups, which makes the results only descriptive of this study. In terms of items reflecting students' opinions, we used statistical tests as a tool to summarise rather than to be representative of an objective truth [see 12].

Findings

The trials involved 128 pre-tertiary students; 65% female and 35% male. In the examination, 52% of students elected to type. Table 2 shows participation at each stage. A Chi-squared test revealed a statistically significant difference in relation to gender and choice to type the examination (χ 2 (1)=5.299, p=0.021) with 68% of males choosing to type compared to 46% of females. However, the differences due to gender were not significant in the earlier stages of the trial. This may be due to the larger number of undecided ('missing') cases in the earlier stages.

Opinions

We examined the differences in opinions on five-point Likert items (where strongly agree=5, neutral=3, and strongly disagree=1) between those that typed the examination and those that handwrote it using a Mann-Whitney U test. Table 3 displays the extent of agreement, with means and standard deviations provided for clarity, along with the results of significance tests with the difference between the means shown. Those that chose to hand-write the examination were asked about the neatness of their writing and comfort levels experienced during the examination (see Table 4). The examination duration was 70 minutes in both cases.

Table 4. Post-examination survey responses on hand-writing

Question

n M SD I think my hand writing was neat and legible 53 3.6 0.9 I experienced discomfort (sore/tired/cramp) in my writing hand 53 2.9 1.2

Typists in the geography unit were also asked to reflect specifically on using a computer for the examination, given the nature of the task (see Table 5).

Table 5. Post-examination survey responses on using a computer for the assessment

Question

n M SD I was able to produce a better final version of this assessment 24 4.3 0.6 I was able to quickly complete the assessment 24 4.3 0.7 I was able to easily edit and make changes 24 4.6 0.6 I was able to easily refer to reference materials and resources 24 3.7 1.0 I was able to easily think and compose my answer using a computer 24 4.3 0.6 I would like to use a computer for similar assessments in the future 23 4.2 0.7

Students were asked about their preferences for production method ('computer' (C), 'same' (S) or 'pen and paper' (P)) for a range of writing activities, style and features. A Fisher exact test for categorical variables was used to see if their preferences may have influenced their choice to type or handwrite the examination (see Table 6). Counts and percentages in brackets are shown for each production method (C, S or P) for the two groups (typists and hand-writers). Significant differences were noted across many of the items with preferences in alignment with their actual choice of examination mode.

Table 6. Post-examination survey responses on examination writing strategies

Question

Typed Handwrote Fisher C=computer, S=same, P=pen C S P n C S P n p I write more words in an examination when... count certainly does not guarantee a pass. Similarly, a larger number of words above 600 did not see the highest grades; indeed, those with the highest grades wrote about the same as the group mean of 538 words. The scatter plot for Glo shows a more tightly monotonic relationship between words and grades. Again, correlation is certainly not causation in terms of word count -quality still matters!

Conclusion

Overall, we observed a significant alignment between preferred writing strategies and choice of text production method in the two trials. Further, the grades achieved between typists and hand-writers did not differ significantly. These two facts are not surprising when dealing with thinking, purposeful humans who aim to maximise outcomes. However, this does raise the concern that should a shift occur towards fully-typed assessment, then a period of transition with assistance for those who preferred handwriting should occur to ensure they are able to adapt successfully.

64 4 .

 4 2 0.9 53 3.0 1.3 1.2 721.5 >.001 My hand-writing is normally neat and legible 63 3.6 1.3 52 3.1 1.1 0.5 1183.5 0.008 I go back to re-read and revise my writing quite a lot 63 3.8 1.0 53 2.9 1.2 0.9 969.5 >.001 I prepare most of my assignments /reports using a computer 64 4.0 0.9 53 3.0 1.4 1.0 1012.5 >.001

Table 2 .

 2 Intention to type the examination at each stage of the study (counts)

	Participation	Initial interest	After practice	At the examination
	Type	73	72	64
	Handwrite	26	30	59
	Missing	29	26	5

Table 3 .

 3 Post-examination survey responses by text production mode

	Question	Typed Handwrote	MW
	Post-examination Survey	n M SD N M SD diff U	p
	I type faster than I handwrite	64 4.1 1.1 55 1.8 1.2 2.2 412 >.001
	I type accurately	63 4.1 0.8 52 2.5 1.1 1.7 379 >.001
	When I make errors, I am able to quickly	64 4.3 0.8 53 2.9 1.3 1.4 634 >.001
	correct them when typing
	I often rely on spell check to detect errors 62 3.8 1.3 54 2.9 1.3 0.9 1060.5 >.001
	I work more efficiently when I type on a
	familiar keyboard

 (56)

Acknowledgments. The authors would like to thank the Australian Government Department of Education and Training for financial support and the students at Monash College for being willing to 'give it a go'.

Performance

Performance was expressed as a percentage grade. There was a statistically significant performance difference between the two units (Geo, n=38, Mdn 73.75 and Glo, n=85, Mdn 57.14) using a Mann-Whitney U test (U=1100, p=.005).

When the grade data were grouped by gender (shown in Table 7) it was not found to be statistically significant, although the result was borderline at p=.050. In the geography unit, a significant difference in grades between genders was found (p=.006); however, this was not so in globalisation. Normal distributions were confirmed for each gender pair using standardised skewness and the Shapiro-Wilks test with Levene's test (shown in Table 7) demonstrating equivalence of variance. Means and standard deviations are also provided for clarity. When comparing the grades of typists and hand-writers (shown in Table 8) across both units, no statistically significant difference was found. However, within the globalisation unit there was a small statistically significant difference in grades between typists and hand-writers (p=.033). The distribution of each pair was found to be normal using standardised skewness and the Shapiro-Wilks tests, with Levene's test (shown in Table 8) establishing equivalence of variance. Means and standard deviations are provided for clarity. Within each of the geography group (n=38) and the globalisation group (n=32) positive, statistically significant correlations were found between the number of words written and the grade achieved from Spearman tests (Geo: r s =.628, p=>.001 and Glo: r s =.865, p=>.001). See Figure 1.

Geography Globalisation

• Typed • Handwrote

Figure 1. Correlation between number of words and grade

Mann-Whitney U tests found no statistically significant difference in the number of words produced by typists and hand-writers (see Table 9). It is worth noting that word count data were only available for half the globalisation group and that assessment tasks were different between Glo and Geo, so the two groups are not comparable.

Discussion

The choice to type or handwrite appears to be multifaceted, but with the strongest differences declared around being able to type more quickly than handwrite and typing accuracy. Overall, the results showed an alignment between writing strategies and choice of text production mode in the examination (see Tables 3 and6). The result of 52% of students electing to type is much higher than reported in previous studies involving free choice [START_REF] Hillier | To type or handwrite: student's experience across six e-Exam trials[END_REF][START_REF] Purcell | Exams: Comparing Handwritten Essays with those Composed on Keyboards (Final Report)[END_REF]. The use of in-class practice sessions compared to optional out-of-class practice sessions used [START_REF] Hillier | To type or handwrite: student's experience across six e-Exam trials[END_REF] may have contributed to the increased uptake of the typing option in our study.

In both 70-minute assessment tasks, only a minority of hand-writers experienced discomfort (i.e. agreement of 2.9 out of a maximum 5 with standard deviation 1.2 indicating that only some were impacted). This was consistent with previous work [START_REF] Hillier | To type or handwrite: student's experience across six e-Exam trials[END_REF] where a 70-minute duration was the cross-over point where discomfort started to become a problem.

Typists generally agreed that the computer allowed them to produce better responses on the assessment task (i.e. most Likert items in Table 5 were rated above 4 on the 5point scale), including that they would like to be able to use a computer for similar assessments in the future. The weakest agreement related to their ability to easily refer to reference materials. In this study, students had to scroll up and down between the materials and their written response. Providing reference materials in a separate file (that would allow side-by-side window arrangement or 'Alt-Tab' between windows), in a split screen format within the document [START_REF] Al Nadabi | Features of an online English language testing interface[END_REF] or on paper, may help in this regard.

In terms of performance, there were mixed results. The number of words produced by typists was greater in Glo, as shown elsewhere [START_REF] Mogey | Typing compared with handwriting for essay examinations at university: letting the students choose[END_REF][START_REF] Fluck | eExaminations Strategic Project Final Report for Academic Senate[END_REF], but less in Geo. The classes were relatively small and assessment tasks were different, so this may have played a role. In Geo, hand-writers achieved slightly better grades (but not significantly so) and in Glo, typists did better with a borderline statistically significant difference in grades. When comparing by gender, in Geo, females did significantly better. In Glo, they also did slightly better, but not significantly so. Overall, a weak link between gender and performance could be claimed within these groups. In both classes, we found a statically significant correlation between words produced and marks. The scatter plot in Figure 1 shows that the correlation occurred over a narrow band in Geo, with a minimum of 300 words required before a pass was possible. Yet an adequate word


Andrew E Fluck
email: andrew.fluck@utas.edu.au

C K H Chin

Dev Ranmuthugala

Transformative Computational Thinking in Mathematics A Comparison by Student Age

Keywords: Computational thinking, transformative uses, integral calculus, students

The Calculus for Kids project showed how Year 6 (aged 12 years) students could master integral calculus through the use of multi-media learning materials and specialist mathematics software. When solving real-world problems using integral calculus principles and the software to perform their calculations, they demonstrated ability commensurate with university engineering students. This transformative use of computational thinking showed age-extension, because the students were enabled to redefine the curriculum by accessing content normally taught to much older children. To verify this was not an accidental finding, further work was undertaken with a relatively smaller cohort of (n=44) Year 9 (aged 15 years) students. The results were similar to the earlier findings with an effect size of 24 (Cohen's d) recorded. The article explores the implications of these new findings, and the potential application to other subject areas and student age groups.

Introduction

There is a problem with computers in schools. Schools can rarely guarantee every student access to a computer, so the curriculum is largely designed without assuming this equipment will be available. Yet outside school, students, citizens and professionals use digital technologies intensively. This study aimed to reconcile this incongruity, and to counter diminishing learning achievements in Australian schools.

School curricula change very slowly. They are not controlled by content experts, but by communities, politicians, educators and others. Therefore, new knowledge can take time to be adopted. In 2005, we celebrated the centenary of Einstein's special theory of relativity [START_REF] Einstein | Zur Elektrodynamik bewegter Körper[END_REF], but this is rarely found in primary schools. Technology and science continue to advance rapidly [START_REF] Kurzweil | Essay: The Law of Accelerating Returns[END_REF], often outstripping community discussion and legal controls. Disparities between the school curriculum and the lived world of children embedded in artefacts created from recent knowledge can lead to tension for them and their teachers. One way to bring new knowledge into schools may be to use computers. However, even computers are not making the desired impact in education. An OECD report [START_REF]OECD: How computers are related to students' performance[END_REF] showed a weak inverse relationship between computer investments in schools and student achievement in the Programme for International Student Assessment (PISA). This is, therefore, a wicked problem in education. When students regularly use computers to create text using keyboards, they are unlikely to develop the handwriting skills demanded in the current curriculum. So, these findings are not entirely unexpected. But how can learning achievements be improved and new skills and ideas enter the curriculum? A transformative perspective is necessary, looking beyond the current curriculum to new skills and higher order thinking [START_REF] Downes | Making better connections[END_REF]. These authors initially saw transformation as "an integral component of broader curricular reforms that change not only how students learn but also what they learn" and went on to suggest it could become "an integral component of the reforms that alter the organisation and structure of schooling itself" (ibid, p.2).

It is suggested that new theoretical approaches are needed to compare transformative uses of computers in schools. Puentedura's Substitution, Augmentation, Modification, and Redefinition (SAMR) model [START_REF] Puentedura | Building transformation: An introduction to the SAMR model[END_REF] provides a framework for thinking about transforming school curricula with computers, but there is still neither a clear way of defining transformation, nor of measuring it. In this study, we tried using computers to change what students learnt in a very specific way. Material within the school mathematics subject is structured in a similar way across many jurisdictions. In particular, we could not find evidence of integral calculus being taught in primary school (up to age 12 years) in any widespread fashion. Therefore, the Calculus for Kids project used computers to teach this topic to both younger and older students. This article looks at the similarities and differences between the two groups.

Previous work

Before explaining how this project could solve these entwined wicked problems, it is important to have an accepted way to measure the impact of an educational innovation. Meta-studies such as that by Hattie and Yates [START_REF] Hattie | Visible learning and the science of how we learn[END_REF] have established 'effect size' (often using the statistical measure of Cohen's d) as a measure of innovation impact on learning achievement. A hinge point effect size of 0.4 has been identified as the minimum for a significant innovation [START_REF] Hattie | Visible Learning: A synthesis of over 800 meta-analyses relating to achievement[END_REF]. Studies have shown that "supplementing traditional teaching… computers were particularly effective when used to extend study time and practice, when used to allow students to assume control over the learning situation… and when used to support collaborative learning" [START_REF]OECD: How computers are related to students' performance[END_REF]. In these meta-studies, computer aided instruction was found to have an intervention effect size of 0.55 to 0.57.

When assessing the learning of any group of students, their achievements generally lie on a normal curve: a few do well; some under-perform; and most perform close to the average. After a group of students receives a successful teaching intervention, the group demonstrates higher achievement, so the distribution curve of achievement moves to the right. The quantum of the improvement is crucial to determining the effectiveness of the intervention. An effect size of 1.0 corresponds to an improvement of one standard deviation. Glass et al. [START_REF] Glass | Meta-Analysis in Social Research[END_REF] indicated that normal learning progress is about one standard deviation per year.

The Calculus for Kids intervention taught both Year 6 (aged 12 years) and Year 9 (aged 15 years) students to solve real-world problems using integral calculus with computer algebra software. This achieved a published intervention effect size between 1.85 and 25.53 [START_REF] Fluck | Large effect size studies of computers in schools: Calculus for Kids and Science-ercise[END_REF] for the Year 6 students. The effect size of this transformative intervention was over four times the size of Hattie's hinge point (0.4) for a significant intervention [START_REF] Hattie | Visible Learning: A synthesis of over 800 meta-analyses relating to achievement[END_REF]. The nearest other high effect size for new ways of using computers in class was reported by Puentedura [START_REF] Puentedura | A Brief Introduction[END_REF] with 'redefinition' types of computer uses providing effect sizes of 1.6.

David Perkins provided the concept of 'person-plus' to embody the shared cognition of a learner assisted by their notebook [START_REF] Perkins | Person-plus: A distributed view of thinking and learning[END_REF]. In our project, we acknowledge this notebook could be a notebook computer, or any kind of specialised electronic assistive device. This 'person-plus' view supports arguments for open book assessment, because it is closer to real-world problem solving. The assistive device improves information retrieval and accuracy, and Perkins argues this leads to higher order knowledge. By extension, a student using modern digital technology can solve problems that are more complex. This models human adoption of other mind tools such as language, numbering systems and so on. Moursund's extension [START_REF] Moursund | Introduction to Information and Communication Technology in Education[END_REF] divided problems into three groups: those which are more readily solved by humans alone; those best solved by computers alone; and those which are best solved by humans working in combination with computers. This project probed the transformative effects of students learning in partnership with their computers.

Computational thinking was popularised by Jeannette Wing [START_REF] Wing | Computational Thinking[END_REF] and encourages us to solve problems using method which can be automated. This approach promotes the application of computers beyond computer science. Wing sees computational thinking as recursive, parallel processing, choosing an appropriate representation for a problem, and modeling. Algorithms are explicitly labelled in computational thinking and involve decision steps. Many of these mental tools in computational thinking were used in Calculus for Kids, because the students had to decide how to represent problems in MAPLE and how best to model the real world.

This study grew from the Calculus for Kids work reported previously [START_REF] Fluck | Large effect size studies of computers in schools: Calculus for Kids and Science-ercise[END_REF] with Year 6 students. That project trained teachers to use supplied multi-media materials to teach the core concepts of integral calculus and how to solve real-world problems using them. Students were provided with a copy of the MAPLE specialist mathematics software for learning and assessment. The assessment test was based on a first year engineering degree integral calculus examination.

While the overall results with Year 6 students have already been reported, some schools in our project undertook work with Year 9 students as well. This paper compares the performance of these older students with the younger ones.

Method

Schools were invited to become part of the project, with a teacher attending our university campus for one day of training. During the training day, we introduced the teachers to the animated Microsoft (MS) PowerPoint files they could use to teach each concept; introduced them to the ideas of integral calculus and trained them to use the MAPLE computer algebra software. MAPLE was selected over Mathematica, Microsoft Mathematics and Maxima because it used conventional notation and because we were approached by a local re-seller of the product who supported our research.

We provided a MS PowerPoint slide deck for each lesson, together with a MAPLE worksheet and a Portable Document Format (PDF) page of example questions. Students were shown how to perform simple calculations using MAPLE, starting with those they already knew (addition, subtraction, area of a rectangle) and then moved immediately to the correct terminology for setting up algebraic expressions (see Figure 1). Integral calculus was introduced as a tool for calculating the area under a function curve. This area might represent the distance travelled for a speed-time function curve. Our basic approach was to use the strips method, whereby a curve is approximated by successively shorter straight lines. The animated MS PowerPoint slides provided powerful visual representations to illustrate the techniques of integration. While this introduction to calculus as the area under a curve was very conventional, we skipped the task of calculating examples by hand. Students were shown the integral of the sum of any number of functions is equal to the sum of the integrals of the several functions. Some 'principles' of integral calculus were not taught, such as manipulation of trigonometric functions, since the MAPLE software could perform this activity. Nor was differentiation discussed, since it was outside the scope of the project. Teachers were provided with a comprehensive document with solutions to all the activities. Figure 1 shows a screen from the MAPLE application. In the left-hand menu bar can be seen components for constructing mathematical expressions. The bottom-most lefthand icon is the template for a standard definite integral. When the user clicks on this icon, the template is transferred to the main page, whereupon the values of the lower bound (x 1 ) and upper bound (x 2 ) can be inserted. The mathematical expression for the target function 'f' can also be inserted to complete the definite integral. MAPLE recognises this notation and evaluates the numerical value of the integral when [CTRL]= is pressed. The MAPLE software is capable of symbolic manipulation in many other areas of mathematics. The advertising for the product claims "Maple has over 5000 functions covering virtually every area of mathematics, including calculus, algebra, differential equations, statistics, linear algebra, geometry, and much more" (www.maplesoft.com/products/Maple/features).

In the Calculus for Kids project, it is salient to recognise the software works with mathematically correct expressions and automates their calculation. From a student's point of view, a mathematically correct expression will generate a numerical result even if the expression does not fit the related real-world example. Similarly, a notationally deficient expression will not generate a numerical result, even if it has some semblance to the real-world situation. To obtain correct results, students need to use correct mathematical notation (on-screen) and frame their algebraic expressions to precisely match the real-world problem.

Question

Student response The stainless-steel Gateway Arch in St Louis, Missouri, has the shape of a catenary, which is a curve that approximates a parabola. It is 192 metres wide and 192 metres tall. Determine an equation that models the curve. Plot this curve on Maple Many of the activities contained real-world problems. Since our collaboration involved academics from the Australian Maritime College, such real-world problems often involved engineering or sailing boats. Other examples involved finding the area of a billboard to be covered with a poster, finding the amount of cloth to make a curtain for a stage, the amount of metal for a drain cover, and the quantity of grass seed for a curved garden path. Students calculated the area of a sail and the amount of fabric for the curved gable end of a tent. In some cases, students were expected to devise the parabolic equation representing a real-world situation. See Figure 2 for an example of this question type, with a student response.

To solve these real-world problems, students needed to use their computational thinking skills of modelling to formulate the curves as a mathematical expression for MAPLE. It was important to select data points which correctly matched the limits of the function to be integrated, and ensure the correct units were applied to the result. One topic relevant to this discussion was that relating to parabolic curves. We used this as an early introduction to non-linear functions, and taught how the parameters of the descriptive equation could be found from two known points on the curve. It is important to remember that students did not have to work out the value of mathematical expressions by hand. Instead, they needed their computational thinking skills to put the right parameters and equations into MAPLE, so the computer could undertake the calculation for them. This provides a new way of understanding the skill of integral calculus, and shows how computational thinking can render old methods of learning redundant.

Moving now to ways in which our results could be compared with other teaching interventions, it was previously noted that effect size is the commonly accepted measure of the impact of an innovation. It is dimensionless, and therefore allows findings from different-sized cohorts to be compared fairly. However, in the context of transformative uses of computers in schools, it has some disadvantages.

The calculation of an effect size uses the following equation [START_REF] Coe | It's the Effect Size, Stupid: What effect size is and why it is important[END_REF]:

When contemplating a transformative computer innovation, this equation can provoke speculation about the nature of the control group. Ordinarily, if students are randomly allocated to the experimental or control groups, the investigation would consider two different teaching methods. It goes without saying that the groups would be taught the same content in such a context, to show the difference in learning achievement between the two methods. However, in the current study, the definition of transformation that was adopted required new content to be taught; specifically, new content traditionally programmed for students at a much higher year level. Should the control group then comprise students in the higher year level? If so, there are two conflicting variables to consider: learning achievement and student age. If both groups are taught using the computer-based methods and tools, then it would not be easy to distinguish their impact on learning.

Another way to calculate an effect size without a control group is to compare learning achievement before and after the computer-based intervention. Where students are delivered content within the prescribed curriculum, standardised assessment tools can be used for this. However, age-accelerated transformative innovations cannot use such instruments. Therefore, Calculus for Kids used a method to calibrate each student to a notional academic year level scale. The initial measure was derived from national numeracy testing (www.myschool.edu.au), and effectively located each student as a bona-fide member of their chronological year group. Since the common post-test was derived from a first year university examination, it was possible to translate demonstrated learning achievement onto the same academic year level scale in the range Year 12-13 [START_REF] Fluck | Large effect size studies of computers in schools: Calculus for Kids and Science-ercise[END_REF]. This study followed the same protocol. The mean numeracy score was used in conjunction with the Australia-wide mean numeracy score and standard deviation to deduce a pre-activity numeracy Year level for our students. A similar process was used to map the post-activity test score percentages onto the notional national numeracy scale, on the assumption that a 50% pass mark would map onto the Year 12/Year 13 boundary between high school and university. The demographics for the Year 6 and Year 9 samples are given in Table 1. There were more males in each cohort, with 54% males in the younger group, and 64% males in the older group. There was a slight overlap between the cohorts in terms of age, but at least 2 years between their mean chronological ages. This provided the opportunity to see if student age had any effect on the learning outcomes.

Results

Teachers of Year 9 students used the same protocols as for Year 6 students. Therefore, the three teachers attended a training session in Launceston, and then taught the material over 13 lessons (including a post-test lesson). The post-tests were sent to the research team and after marking were returned to the teachers with a community report for inclusion in newsletters to parents.

The data from each class were added to the national database for the project, and effect sizes extracted for the Year 6 and Year 9 cohorts (see Table 2). Table 2 shows the numeracy aptitude for both cohorts was as expected for their respective ages. The older students demonstrated greater mastery of the high-end mathematics content than did the younger Year 6 students at the end of the intervention, but with greater variance among them.

The effect size has been calculated in three ways for each cohort. Firstly, the standard calculation using Cohen's d is given. Then, two annual maturation measures have been used. The first uses Glass et al.'s [START_REF] Glass | Meta-Analysis in Social Research[END_REF] supposition that one year of learning advancement corresponds to one standard deviation. Wiliam's refinement [START_REF] Wiliam | Standardized Testing and School Accountability[END_REF] showed annual growth in achievement between Grades 4 and 8 (as measured by standardised national testing) was 0.30 standard deviations. This has been used in the final part of the table.

Discussion and conclusion

These very high effect sizes are unusual in the study of teaching innovations. However, they are derived from standard practice for comparing the effectiveness of different approaches to learning. When proponents make claims to have enacted a transformative change through the use of computers, the SAMR model predicts dramatic results, especially at the redefinition end. These very large effect sizes of 25.53 and 13.79, using Cohen's d, justify such claims. They are well above the 'hinge point' of 0.4 for an effective intervention, and greater than the intervention effect size of 0.57 ascribed to computer aided instruction. It can be deduced that computer use is highly effective in schools when used to teach transformative content, and much more so than when used to teach traditional content.

The Year 9 student cohort performed in a very similar way to the Year 6 cohort. This is promising in several ways. First, it shows the original Calculus for Kids project did not accidently hit a mysterious 'sweet spot' which was only available for younger students for this topic of integral calculus. Second, the result validates the earlier work by showing a slightly different context does not change the outcome. Finally, it provides evidence this method of incorporating computational thinking [START_REF] Webb | Thematic Working Group 9: Curriculum -Advancing Understanding of the Roles of Computer Science/Informatics in the Curriculum in EDUsummIT[END_REF] into the curriculum can be done at any age between 12 and 14 years with the same beneficial effect. Others may wish to see if these chronological boundaries can be expanded further.

To progress this work, we propose to continue working with Year 6 students using short and effective computer-based interventions involving specialist application software. Schools will be invited to participate in guided workshops that will generate specific plans appropriate for their own students. Key elements in each of the interventions will be:

• Short in duration (typically 5 to 7 lessons).

•

Students to use a single computer software application for each intervention, generally free of cost to schools. • Learning outcome will be a significant departure from the existing curriculum (4-6 years in advance of chronological age). • Effect size will be calculated by comparing each student's National Assessment Program -Literacy and Numeracy (NAPLAN) ranking with a calibrated post-test of learning achievement accompanied by pre-and postactivity attitudinal surveys. We will modify the Attitudes toward Mathematics Inventory [START_REF] Tapia | An instrument to measure mathematics attitudes[END_REF] to make it subject-generic and link with our previous work; also, the Mathematics and Technology Attitude scale [START_REF] Barkatsas | A New Scale for Monitoring Students' Attitudes to Learning Mathematics with Technology (MTAS)[END_REF][START_REF] Pierce | A scale for monitoring students' attitudes to learning mathematics with technology[END_REF] will be similarly adapted. • Participating schools will be drawn from a wide range, going beyond the 26 schools from 5 the states of Calculus for Kids, and drawing on mixedability classes from a range of ICSEA [socio-economic and locational status] backgrounds. If we can demonstrate similar high effect size transformations by applying computational thinking in subjects across the breadth of the school curriculum, this would provide experimental evidence supporting curriculum change. We do not suggest every student should be taught in conjunction with a computer and powerful software in every lesson. However, there may be good reason to teach some topics, and perhaps introduce new knowledge into the curriculum where this is possible and desirable. Should we be able to demonstrate accelerated learning achievement of 4+ years with primary school students, similar processes can be put in place at other educational levels. For instance, first year undergraduates will be able to achieve learning outcomes at the Masters or even initial PhD candidate level. Year 10 students would be empowered to demonstrate understanding from first year degree courses. We seek to engender this quantum leap in learning in areas where humans working with computers can perform better than either alone. Our work with Year 6 students can be a lever for future work at all educational levels.

Figure 1 .

 1 Figure 1. Example of working in MAPLE with text and mathematical expressions

Figure 2 .

 2 Figure 2. A real-world problem to be solved in the Calculus for Kids project

Table 1 .

 1 Demographics for the two student groups

	Year 6	Year 9
	students	students

Table 2 .

 2 Effect size measures for younger and older students in the Calculus for Kids project

	Year 6	Year 9 students
	students

Acknowledgments. The authors would like to thank the Australian Research Council for supporting this study under LP130101088, the University of Tasmania and Australian Scientific & Engineering Solutions Pty Ltd for financial assistance to support this study.


Hana Voňková

Miroslava Černochová

Hasan Selcuk
email: hasan.selcuk@pedf.cuni.cz

Jan Hrabák

Kateřina Králová
email: katerina.kralova@pedf.cuni.cz

The Application of Anchoring Vignettes in the Analysis of Self-assessment of ICT Skills A Pilot Study among Czech Secondary School Students

Keywords: Information and communication technology, self-assessment, anchoring vignette, upper secondary schools

This paper presents pilot study findings of a research project about the application of anchoring vignettes in the analysis of Czech upper secondary school students' self-assessment of ICT skills. The pilot study was conducted in December 2017 with 166 respondents from four different types of upper secondary schools. Anchoring vignettes, which are brief texts describing hypothetical people who illustrate a certain level of the trait of interest (e.g. information and communication technology (ICT) skills), is a method implemented to identify response scale differences in survey questions and to adjust self-assessments caused by response scale differences. Methodologically, as there are only a few examples of the application of this method in the selfassessment of ICT skills and also in educational research, this pilot study has enabled the researchers to test how comprehensible a questionnaire with a set of vignettes was for the upper secondary school students. This enhanced research method based on anchoring vignettes will be used for the main study in spring 2018. The pilot study findings confirmed the high variability of the use of scale for respondents' self-assessments and vignettes.

Introduction

This paper presents pilot study findings of a research project about the application of anchoring vignettes in the analysis of Czech secondary school students' self-assessment of ICT skills. As explained by King et al. [START_REF] King | Enhancing the validity and crosscultural comparability of measurement in survey research[END_REF], anchoring vignettes is a method to identify response scale differences in survey questions and to adjust the self-assessments caused by response scale differences. For example, when students are asked to self-assess their ICT skills in a survey question on a scale of 1 to 7 (1 being the lowest and 7 the highest), students might either overvalue or undervalue their ability in the self-assessment. Hence, anchoring vignettes, which are brief texts describing hypothetical people who illustrate a certain level of the trait of interest (e.g. ICT skills), enable researchers to identify response scale differences in self-assessments and thus, to adjust them.

Since the early 2000s, the anchoring vignette method has been implemented in a variety of areas of research, such as political efficacy [START_REF] King | Enhancing the validity and crosscultural comparability of measurement in survey research[END_REF], work disability [START_REF] Kapteyn | Vignettes and self-reports of work disability in the US and the Netherlands[END_REF], job satisfaction [START_REF] Kristensen | New evidence on cross-country differences in job satisfaction using anchoring vignettes[END_REF], health ([4 -6]), health system performance [START_REF] Sirven | Comparability of health care responsiveness in Europe[END_REF], life satisfaction [START_REF] Angelini | Age, health and life satisfaction among older Europeans[END_REF], and satisfaction with social contacts [START_REF] Bonsang | Satisfaction with social contacts of older Europeans[END_REF]. However, there are only a few examples of the application of this method for educational research. Regarding one of these, Buckley and Scheider [START_REF] Buckley | Charter schools: Hope or hype?[END_REF] implemented anchoring vignettes when investigating charter schools in the United States of America (USA) and parents' satisfaction with different types of schools. Vonkova and Hrabak's study [START_REF] Vonkova | The (in) comparability of ICT knowledge and skill selfassessments among upper secondary school students: The use of the anchoring vignette method[END_REF] focused on Czech upper secondary school students' self-assessment of ICT knowledge and skills through the anchoring vignette method. Moreover, Vonkova et al. [START_REF] Vonkova | How Students Report Dishonest Behavior in School: Self-Assessment and Anchoring Vignettes[END_REF] investigated Czech lower-level secondary school students' self-assessment of dishonest behaviour in school by using the anchoring vignette method. Von Davier et al. [START_REF] Von Davier | The effects of vignette scoring on reliability and validity[END_REF] examined the effects of vignette scoring on reliability and validity of student self-assessment, according to the Programme for International Student Assessment (PISA) 2012 dataset. He et al. [START_REF] He | Effects of Anchoring Vignettes on Comparability and Predictive Validity of Student Self-Reports in 64 Cultures[END_REF] conducted a study on effects of this method on comparability and the predictive validity of student self-assessment in 64 countries based on data from PISA 2012. Vonkova et al. [15, p.3] looked into cross-country heterogeneity in students' self-assessment of their teacher's classroom management also based on the PISA 2012 dataset and they found that the anchoring vignette method was potentially a useful tool to enhance the comparability of the self-reported measures in education. To contribute to research in the field of education, we have conducted a study aimed at designing an enhanced data collection method based on anchoring vignettes to explore different realms in educational research, such as digital literacy.

Literature review: The application of anchoring vignettes in the self-assessment of ICT skills

This section presents a review of existing studies that are pertinent to the application of anchoring vignettes in the self-assessment of ICT skills in the context of education. Before moving to reviewing the existing studies, it is worthwhile mentioning the European Computer Driving Licence (ECDL) foundation study [START_REF]ECDL: Perception and Reality. Measuring Digital Skills in Europe[END_REF], which highlighted how people's self-assessment of their ICT skills can be different from their actual ability. The ECDL foundation's study was conducted in five European countries, namely Austria, Denmark, Finland, Germany and Switzerland, to identify people's ICT skills. Respondents (aged 15-64 years) were given a questionnaire in two parts, with the first focusing on self-assessment of ICT skills, and the second testing their real level. The findings of the study showed that in all the countries surveyed, respondents overestimated their ICT skills. Taking into account the study of the ECDL foundation [START_REF]ECDL: Perception and Reality. Measuring Digital Skills in Europe[END_REF], we suggest that anchoring vignettes could provide dependable findings from respondents' answers with respect to self-assessment in surveys.

Regarding studies that have applied anchoring vignettes in self-assessments, Vonkova and Hrabak [START_REF] Vonkova | The (in) comparability of ICT knowledge and skill selfassessments among upper secondary school students: The use of the anchoring vignette method[END_REF] compared the ICT knowledge and skills of two distinct groups of upper secondary school students by examining their self-assessed perspective on these, both before and after the anchoring vignette adjustment for the different usage of scale. The study findings revealed that the anchoring vignette method enabled the researchers to distinguish between the two groups of students' differences in terms of scale usage and showed how adjusted self-assessments corresponded to the assumed level of students' ICT knowledge and skills. Cerna [START_REF] Černá | Self-assessment of teacher students' knowledge in the field of information and communication technologies[END_REF] investigated the self-assessment of undergraduate university students' ICT knowledge by applying the anchoring vignette method. Respondents were students from the Faculty of Education at Charles University and specialised in three different study programmes (information technology (IT), social science, and mathematics). The findings of Cerna's study showed significant differences between respondents' self-assessment and their assumed actual ICT knowledge. Moreover, the author found that those who study IT as their specialisation in education have a greater tendency to exaggerate the level of their ICT knowledge compared to other respondents studying different subject areas.

Pilot study methodology 3.1 Aims of the pilot study

Generally, a pilot study refers to "a small-scale version or trial run, done in preparation for the major study" [18, p. 467]. As described by Vogt [START_REF] Vogt | Dictionary of statistics and methodology: A nontechnical guide for the social sciences[END_REF], it can be considered as a 'dress rehearsal' to identify any possible problems before undertaking the major study. The main aim of our pilot study was to inform and design the main study methodology, specifically to test the feasibility of the data collection method, which consists of a questionnaire with a set of vignettes focused on self-assessment of the ICT knowledge and skills of young people.

In the pilot study, we aimed to test how comprehensible our questionnaire was for upper secondary school students, who are at Year 1 (age 15 years) and 4 (age 19 years). The pilot was conducted in December 2017 among (N = 166) students from four upper secondary schools in the Czech Republic. Experiences gained from the pilot study were then used in the main study, conducted in spring 2018 with a representative sample of 2,600 students from 56 secondary schools in the Czech Republic.

Procedures of developing the data collection method

"The aim of the anchoring vignette method is to clear/correct the self-assessment of respondents so that they can be comparable" [20, p. 14]. Respondents in the area (in our case ICT skills and knowledge) use a self-assessment question, as well as evaluating hypothetical people described in a short story (anchor). The presented verbal characteristics of hypothetical people in vignettes in the case of categorical assessment scales can be evaluated by different respondents in different ways. Respondents' answers to self-assessment can be affected by the different use of scale categories. For example, respondents with a lot of ICT experience and a deep interest in ICT can use different scale categories for evaluating a given level of ICT knowledge than beginners and ordinary ICT users. Our main data collection method was a questionnaire, which consisted of a set of anchoring vignettes and fixed-choice questions to obtain background information about each respondent (age, sex, school, grade in ICT subjects, field of study, interest about ICT, participation in programming or informatics competitions, number of hours spent on a computer at school or at home, respondents' use of ICT within different types of activities, parents' education and parents' use of ICT in their job or free time). Briefly, in our pilot study, we used one self-assessment question (S) and three vignettes (V1, V2, V3), with a scale of 1 to 7 (1 -the lowest level and 7 -the highest level).

In the vignettes designed for our pilot study (see Table 1), we focused on five domains of computer literacy (information and data literacy, communication and collaboration, digital content creation, safety, and problem-solving) in accordance with the concept of digital literacy, as defined in DigComp (see [21, pp. 8-9]). In formulating the vignettes, we presented stories that would be comprehensible to students in the social sciences areas as well as the students of technical fields, including IT specialisation, in addition to being accessible to the two chosen age groups (Year 1 aged 15-16, and Year 4 aged 18-19 years). The vignettes should be understandable to all respondents also from a curricular point of view (note that in the Czech Republic, the current curriculum for ICT subjects does not cover all domains of DigComp).

The level of ICT skills and knowledge presented in vignettes is as follows: V1 describes a basic level (Filip's knowledge and skills in ICT correspond to the knowledge of a basic school graduate in the Czech Republic). V2 describes a more advanced level (Kristin uses ICT for creative activities and some problem-solving; she behaves safely on social networks); the majority of respondents from all secondary schools should achieve this level. V3 describes highly advanced ICT skills and knowledge beyond curriculum requirements (programming additional functions for more difficult tasks, programming a website, using multi-layered security, etc., which are not included in the curriculum for general education). This vignette should mainly distinguish students with ICT specialisation. By providing a score on the selfassessment question and for all three vignettes in Table 1, students provided a personal rating and calibration points for inter-participant comparison.

Procedures of data collection

We gathered data in December 2017 through an on-line questionnaire from 166 students of Year 1 and 4, attending four different public upper secondary schools (see Table 2): School 1 focuses on general education (a gymnasium) and is located in Prague. School 2 focuses on humanities and is located in Beroun, whilst School 3 is a technical school specialising in IT and located in Prague, and lastly, School 4 is a technical school specialising in transport and mechanisation in Mladá Boleslav. These schools were not selected at random, so respondents do not represent a representative sample of the secondary school population. However, these schools represent different specialisations among Czech upper secondary schools. Respondents filled out the online questionnaire at their schools with the participation of the researcher.

Procedures of data analysis

For data analysis, we used Microsoft Office Excel 2011 and the statistical software called GRETL [http://gretl.sourceforge.net]. Table 3 shows how all respondents (N=166) assessed the self-assessment question and three vignettes from across scale 1 to 7 (1' was the lowest and '7' was the highest). The variability of the use of scale for both self-assessments and vignettes is high; respondents use (almost) the whole range of the scale categories. Concerning vignettes, it shows a high heterogeneity in reporting behaviour of respondents -the same level of ICT skills described in the vignettes is evaluated differently by different respondents. We expected the respondents would assess vignettes in the natural order V1<V2<V3 (the order given by a researcher). It means the expected value for V1 would be lower than the value for V2 and the value for V2 would be lower than that for V3. However, 5.4% of respondents assessed the vignettes in another way; they typically tied their assessment to two subsequent vignettes (see Table 3).

Findings from the pilot study

For further data analysis, we analysed each school separately and tried to distinguish the differences in scale usage among different schools. Using the correction based on the non-parametric approach, which consists of how the self-assessment S of a respondent relates to his/her vignette evaluations V1, V2 and V3 "assuming that the vignettes are naturally ordered (for example, from the lowest skills level of a hypothetical vignette person to the highest)" (see [11, p. 192]), we corrected the values for each respondent for each school (see Table 4).

The findings in Table 4 show that the ranking of two schools has changed after the correction, with the most considerable change being in the ranking for School 4 -its position has decreased more, indicating its low standards for evaluating ICT skills. Table 5, and Figures 1 and2 show comparisons of self-assessments before and after correction for School 4 and the best performing School 3. We showed there were statistically significant differences between Schools 3 and 4, not only before the selfassessment correction, but also after the correction. To understand better how respondents evaluated their skills, we compared schools using the background variables of respondents. Analyses of questions focused on respondents' interests and ways of using ICT in their free time, which showed respondents do not differ too much. Generally speaking, ICT knowledge, skills and interests of respondents correspond to topics they have learned mainly at school. Regarding respondents' grades, 69.9% of all the respondents from Schools 1-4 have obtained an average grade between 1 to 2 in ICT school subjects (1 is the best grade and 5 is the worst grade in Czech schools); only respondents from School 4 got worse grades (the average is 2.45). This corresponds to our ranking of schools based on adjusted self-assessments. Nonetheless, 14.5% of respondents reported that they had never studied such a subject before.

As already mentioned, School 3 is a technical upper secondary school specialising in ICT located in Prague and School 4 is a secondary school without any specialisation in ICT located in an industrial city. In both schools, practically the same number of respondents (about 65%) agreed with a statement "I am doing my best to have good results in ICT because I am expected to do so". Respondents from School 3 differed in some characteristics from respondents in School 4. The ratio between universityeducated mothers and fathers of respondents from School 3 was 2.2 and for School 4 was 0.7. Respondents of School 3 spent 1.8-times more time on computers at weekends compared to working days, while at School 4 only 1.3-times more. On weekends, respondents in School 3 spent an average 4.2 hours/day on computers, while respondents in School 4 only 3.4 hours/day. Seventy-four per cent of respondents in Schools 3 and 4 spent practically every day surfing on the Internet for fun. Comparing respondents in School 4 with those in School 3, more enjoyed creating digital music, were active on social networks and enjoyed playing computer games. Respondents in School 3 dedicated more time to web design activities, work with graphics software, publishing on YouTube or creating digital animations than respondents in School 4. Respondents in School 3 were more interested (53%) in the latest ICT news, new technologies, computer graphics, etc., than respondents in School 4 (24%). Respondents in School 3 liked learning new things in ICT (87%) much more than respondents in School 4 (48%). Respondents in School 3 (68%) were fond of creative activities using ICT much more than respondents in School 4 (36%). Ninety-two per cent of the respondents in School 3 agreed with the statement: "To do my best in the ICT lessons pays off because it can help me to get a job I want to do in the future", while in School 4 this was only 60% of respondents. To summarise, the background characteristics of respondents are in line with our adjusted self-assessments using anchoring vignettes.

Conclusion

The main aim of our pilot study was to inform and design the main study methodology, specifically to test the feasibility of the data collection method, consisting of a questionnaire with a set of vignettes focused on self-assessment of ICT knowledge and skills of young people. From this pilot study, we identified some problems that shed light on the main research, conducted in spring 2018 in the Czech Republic.

The pilot study showed us that in terms of data interpretation, it was very important that researchers could visit all schools to instruct respondents what to do and how to fill out the on-line questionnaire. The researchers could understand better some contexts related to students' ICT knowledge, skills, motivation, and approaches to ICT. Some questions need to be adapted for the main research (questions about type of school, number of inhabitants living in a respondent's town/village, grades from ICT subjects, arrangement and ordering vignettes on a page, questions about the family). The pilot also highlighted questions about how to organise data collection through an on-line questionnaire in school computer laboratories, how to support teacher cooperation to motivate students to answer the questionnaire responsibly and maintain classroom discipline, etc. It is necessary in the classes to ensure peace and discipline in order for the respondents to read attentively all questions, especially vignettes.

Limitations of this study are as follows. Findings obtained in the pilot cannot be generalised; sampling does not allow this. However, findings are definitely of interest since they indicate huge differences in scale usage between different types of schools -students from some schools have high/low standards when evaluating their ICT skills. However, all three vignettes V1, V2 and V3 were presented to respondents on one page of the questionnaire -this might have affected respondents' assessment of vignettes.

Figure 1 .Figure 2 .

 12 Figure 1. Comparison of self-assessment of respondents from School 3 and School 4 before correction

Table 1 .

 1 Overview of the self-assessment question and three vignettes How do you evaluate your knowledge and skills in ICT? Note: the following was applied to each of the vignettes below:

	General
	self-assessment
	question (S)

Use a scale of 1 to 7 ('1' being the lowest and '7' the highest). Vignette 1 (V1) Filip can work with texts and charts with the use of basic functions available from the ribbon. He saves his files on a desktop, he doesn´t use sharing or cloud saving, but instead, sends the files via email. He uses the same password for the social network, email, etc. accounts. If he encounters any problem while working on a computer, he usually asks his friend for help. Vignette 2 (V2) Kristin can process texts and charts with the use of advanced functions (e.g. created personal styles, automatic table of content). She goes in for creating graphics (designing business cards) and short original footage, which she shares on YouTube. She doesn´t post any sensitive information on social networks. If she encounters any problem while solving a task, she searches for an instruction on the internet and determines the solution procedure with its help. Vignette 3 (V3) Adam can process texts and charts with the use of advanced functions. He is able to program his own functions for more difficult tasks. He goes in for computer graphics (designing posters), creating footage and programming websites. He manages two Facebook groups and verifies the credibility of the shared posts. He uses multi-layered security (SMS verification) for his accounts on the internet.

Table 2 .

 2 Characteristics of the pilot schools

		Number of	Male Female Male Female		Respondents
		respondents	(%)	(%)	Age	Age Year 1 (%) Year 4 (%)
	School 1	47	17.0	83.0	17.3	16.5	59.6	40.4
	School 2	56	12.5	87.5	17.0	17.6	53.6	46.4
	School 3	38	84.2	15.8	17.6	16.6	71.1	28.9
	School 4	25	84.0	16.0	17.5	16.7	60.0	40.0
	Total	166	41.0	59.0	17.4	16.9	60.2	39.8

Table 3 .

 3 Variability of the use of scale for self-assessment

	Scale	1	2	3	4	5	6	7
		(%)	(%)	(%)	(%)	(%)	(%)	(%)
	Self-assessment (S)	4,22	18,07	27,71	25,90	19,28	4,82	0,00
	Vignette 1 (V1)	7,23	26,51	35,54	26,51	3,61	0,60	0,00
	Vignette 2 (V2)	0,00	1,20	3,61	15,66	34,34	35,54	9,64
	Vignette 3 (V3)	0,60	0,60	1,20	5,42	10,24	31, 93	50,00

Table 4 .

 4 Average self-assessments of respondents in all schools before and after correction

			Average value of	Ranking of schools	Comment
			self-assessment S
		Number	Before	After	Before	After
		of respondents	correction	correction	correction	correction
	School 1	47	3.02	2.22	4	4	The position is
							the same.
	School 2	56	3.34	2.44	3	2	The position
							improved, high
							standards.
	School 3	38	4.29	3.27	1	1	The position is
							the same.
	School 4	25	3.72	2.43	2	3	The position
							decreased, low
							standards.

Table 5 .

 5 Comparison of self-assessment of respondents from School 3 and School 4 before and after correction

		Uncorrected self-assessment	Corrected self-assessment
			(before correction)			(after correction)
		School 3	School 4	School 3	School 4
	Scale	Absolute	Relative	Absolute	Relative	Absolute	Relative	Absolute	Relative
		value	value	value	value	value	value	value	value
			(%)		(%)		(%)		(%)
	1	0	0.0	0	0.0	5	13.51	4	19.0
	2	2	5.3	2	8.0	3	8.1	7	33.3
	3	5	13.2	10	40.0	14	37.8	7	33.3
	4	14	36.8	7	28.0	11	29.7	3	14.3
	5	14	36.8	5	20.0	1	2.0	0	0.0
	6	3	7.9	1	4.0	2	5.4	0	0.0
	7	0	0.0	0	0.0	1	2.7	0	0.0
	Number of
	respondents	38		25		37		21
	Average	4.29		3.72		3.27		2.43
	Standard	0.97		1.00		1.36		0.95
	deviation
	T-test: p		0.03414			0.01730
	value

Acknowledgments. This study is a result of the research funded by the Czech Science Foundation, project GA ČR 17-02993S "Factors influencing the ICT skill selfassessments of upper-secondary school students".