Yaacov J Katz
email: yaacov.katz@biu.ac.il

Social Networks as Learning Delivery Platforms

Keywords: WhatsApp, Facebook, Twitter, achievement, affective variables

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Social networks and learning

Mobile learning provides more flexibility, mobility, convenience and seamless integration of data access for students than other online distance learning environments [START_REF] Ducate | Going mobile: Language learning with an iPod touch in intermediate French and German classes[END_REF][START_REF] Premadasa | Mobile learning environment with short messaging service[END_REF]. Social networks may be defined as applications that utilise mobile learning technology to enable users to communicate with each other by posting information, comments, messages, images, etc. [START_REF] Wallace | A dictionary in education[END_REF]. Education and learning are perceived to be specific topics that can greatly benefit from social networks [START_REF] Kapuler | Top 20 social networks for education[END_REF]. Recent studies have indicated the increasing effectiveness of the contribution of social networks to the learning process. Gilroy [START_REF] Gilroy | Higher education migrates to YouTube and social networks[END_REF] intimated that social networks as educational tools in the academic landscape are catching on fast as universities, colleges and schools recognise the potential that social networks have for learning. Casey and Evans [START_REF] Casey | Designing for learning: Online social networks as a classroom environment[END_REF] reached the conclusion that learning via social networking is positively received by students and contributes to an enhancement of students' learning performance. According to Alvarez and Olivera-Smith [START_REF] Alvarez | Learning in social networks: Rationale and ideas for its implementation in higher education[END_REF] social networks offer ample and potentially effective opportunities to improve student learning at the university level. Sobaih et al. [START_REF] Sobaih | To use or not to use? Social media in higher education in developing countries[END_REF] confirmed that social networks used in the learning process increase motivation of students and stimulate study activities.

In summary, it may be said that research literature offers increasing evidence that supports the notion that social networks contribute to the fostering of student learning at the university, college and high school levels [START_REF] Derakhshan | Social networks for language learning[END_REF].

WhatsApp learning delivery platform

One popular application of mobile technology is WhatsApp "instant messaging". Aburezeq and Ishtaiwa [START_REF] Aburezeq | The impact of WhatsApp on interaction in an Arabic language teaching course[END_REF] found that the WhatsApp platform has the power to enhance students' learning and Shambare [START_REF] Shambare | The adoption of WhatsApp: Breaking the vicious cycle of technological poverty in South Africa[END_REF] confirmed that WhatsApp is userfriendly and contributes to the promotion of learning and student satisfaction with the learning process. Minimol and Angelina [START_REF] Minimol | Teenagers' perception of social network sites in relation to academic motivation and interpersonal relationships: A focus group approach[END_REF] intimated that the use of WhatsApp as a learning tool increases student curiosity and motivation in the learning process. Echenique et al. [START_REF] Echenique | Students in higher education: Social and academic uses of digital technology[END_REF] contended that all social network tools, but most especially WhatsApp, are advantageous and facilitate motivation and curiosity in the learning process.

Facebook learning delivery platform

Facebook, in addition to being the most popular social network for social groups, has also become a mobile learning-based learning resource. Isacsson and Gretzel [START_REF] Isacsson | Facebook as an edutainment medium to engage students in sustainability and tourism[END_REF] noted that students valued Facebook for providing an informal and motivating learning environment. Other research projects have indicated the positive potential of Facebook as a learning delivery platform at the university level [START_REF] Harris | The uses of Facebook technologies in hospitality curriculum on an experiential learning platform for a new generation of students[END_REF][START_REF] Harris | The uses of Facebook technologies in hospitality curriculum on an experiential learning platform for a new generation of students[END_REF]. De Villiers and Pretorius [START_REF] De Villiers | Evaluation of a collaborative learning environment on a Facebook forum[END_REF] found that when used as a learning delivery platform, Facebook enhances critical collaborative thinking and learning motivation. Facebook is also perceived to enhance student-centered as well as social learning [START_REF] Duncan | Facebook in the university classroom: Do students perceive that it enhances community of practice and sense of community[END_REF]. Mitchell [START_REF] Mitchell | A social tool: Why and how ESOL students use Facebook[END_REF] indicated that Facebookbased learning facilitates language learning of foreign students as well as their cultural concept learning. Kassem [START_REF] Kassem | Facebook as a nation-wide civic education classroom listening to the voices of Egyptian secondary school students[END_REF] found that the use of Facebook in the Egyptian secondary educational system led to the narrowing of social gaps between students studying in general (more elite) and technical (less elite) high schools.

Twitter learning delivery platform

Twitter is another social network application that has been used as a learning delivery platform. Junco et al. [START_REF] Junco | The effect of Twitter on college student engagement and grades[END_REF] postulated that the use of Twitter in university courses enhances students' learning potential. Other research results [START_REF] Prince | Learning technologies: Tweeting in a high school social studies class. I-Manager[END_REF] indicated that the use of Twitter at the university level enhances learning collaboration among students, increases learning motivation, encourages students feel that learning could be "trendy" and fun and focuses the attention of students on the topic under study. West et al. [START_REF] West | Beyond the tweet: Using Twitter to enhance engagement, learning, and success among first-year students[END_REF] confirmed that use of Twitter for learning delivery leads to a positive effect on students' achievement as well as enhancing students' willingness and desire to engage in learning tasks.

Academic achievement and social network learning

Several research studies have indicated that students' academic achievement is positively related to the use of technology-based learning delivery platforms. Ituma [START_REF] Ituma | An evaluation of students' perceptions and engagement with e-learning components in a campus based university[END_REF] confirmed that university students who were enrolled in courses where learning was delivered by digital technology had positive perceptions of the learning delivery platform and were in favour of participating in additional courses where learning was delivered by technology-based social networks. Harris [START_REF] Harris | The uses of Facebook technologies in hospitality curriculum on an experiential learning platform for a new generation of students[END_REF]) indicated that learning via digital social network delivery platforms contributes significantly to improved student achievement and Chandra and Watters [START_REF] Chandra | Re-thinking physics teaching with web-based learning[END_REF] confirmed that learning physics through the medium of technology-based social network learning delivery enhances students' learning outcomes. On the other hand, there are studies, such as research conducted by Gettman and Cortijo [START_REF] Gettman | Leave me and my Facebook alone! Understanding college students' relationship with Facebook and its use for academic purposes[END_REF] as well as Kon Shing and Paredes [START_REF] Kon Shing | Towards a social networks model for online learning & performance[END_REF] that cast doubt on the claim that technology-based social network learning delivery leads to significantly improved academic achievement. Thus, in the present study, the potential positive relationship between academic achievement and delivery of learning via WhatsApp, Facebook and Twitter applications will be examined.

Affective attitudinal variables and social network learning

Research studies quoted by Katz [START_REF] Katz | Mobile learning delivery via social networks: what platforms do first-year university students prefer[END_REF] indicate that affective (non-cognitive) variables such as attribution, autonomy, control of the learning process, creativity, curiosity, flexibility, locus of control, motivation, satisfaction, self-confidence, self-efficacy, selfimage, self-esteem and perception of user-friendliness are some of the major variables known to positively contribute to enhanced (improved and efficient) language and concept learning when delivered via social network-based delivery platforms. Furthermore, Katz [START_REF] Katz | Mobile learning delivery via social networks: what platforms do first-year university students prefer[END_REF] confirmed the positive relationship of some or all the above factors with effective social network-based delivery of language and concept learning. In this study, key affective (non-cognitive) variables, namely perception of userfriendliness, motivation and satisfaction of students will be examined vis-à-vis their relationship with different social network learning delivery platforms.

Perception of user-friendliness and social network delivery of learning

Chapman and Henderson [START_REF] Chapman | E-learning quality assurance: A perspective of business teacher educators and distance learning coordinators[END_REF] showed that perception of user-friendliness by students is a vital indicator that assures the quality of a digital learning delivery platform and Katz [START_REF] Katz | Mobile learning delivery via social networks: what platforms do first-year university students prefer[END_REF] confirmed that user-friendliness is a significant attitude that contributes to the enhancement of positive attitudes toward learning in a technology-based environment. Llorente-Cejudo [START_REF] Llorente-Cejudo | Assessing personal learning environments (PLEs): An expert evaluation[END_REF] presented expert evaluators' consensus that confirmed that perceived user-friendliness is a major variable vital for efficient use of different types of digital technology in the learning process. Thus, students' perceptions of user-friendliness of WhatsApp, Facebook and Twitter delivery of learning will be examined in this project.

Learner motivation and social network delivery of learning

Motivation regarding the learning process is another major variable that leads to efficient and effective learning. Moon [START_REF] Moon | Using Story: In Higher Education and Professional Development[END_REF] contended that information is retained longer if it is presented in an interesting way and evokes motivation and curiosity. Rosen and Beck-Hill [START_REF] Rosen | Intertwining digital content and a one-to-one laptop environment in teaching and learning: Lessons from the time to know program[END_REF] confirmed that learning delivery that arouses student motivation enhances the quality of learning and is necessary for learners to successfully utilise technology-based social network learning delivery platforms. Considering the above evidence, the contribution of learner motivation towards the use of WhatsApp, Facebook and Twitter learning delivery platforms will be investigated in this study.

Learner satisfaction and social network delivery of learning

With reference to learners' satisfaction, Katz [START_REF] Katz | Mobile learning delivery via social networks: what platforms do first-year university students prefer[END_REF] found in a comprehensive research study that one of the key constructs that positively affects student performance is the satisfaction derived from studying through social network delivery platforms. When students utilise social networks for their learning, the level of their satisfaction with the educational process increases as does their performance. Minimol and Angelina [START_REF] Minimol | Teenagers' perception of social network sites in relation to academic motivation and interpersonal relationships: A focus group approach[END_REF] confirmed how student satisfaction with the use of social networks, such as WhatsApp, Facebook and Twitter in the learning process, leads to higher levels of student engagement, learning, and success. The above evidence highlights the key relationship between satisfaction of students resulting from the use of social networks in the learning process. Thus, in the present study, the comparative learner satisfaction with the three social network delivery platforms will be examined.

Aims of the present study

In summary, the aims of the present study are twofold: a) to examine the contribution of WhatsApp, Facebook and Twitter learning delivery platforms to students' achievement; and b) to investigate students' affective perceptions of user-friendliness, learner motivation and learner satisfaction derived from studying via WhatsApp, Facebook and Twitter delivery platforms. These aims are based on significantly positive empirical evidence presented by Casey and Evans [START_REF] Casey | Designing for learning: Online social networks as a classroom environment[END_REF] who described the positive use of social networks such as WhatsApp, Facebook and Twitter for learning in communities, and by Shambare [START_REF] Shambare | The adoption of WhatsApp: Breaking the vicious cycle of technological poverty in South Africa[END_REF], who indicated the feasibility of using social networks such as WhatsApp, Facebook and Twitter as effective learning delivery platforms.

Method

Participants

The research sample consisted of 363 first year university social science students enrolled in a 14-week semester-long mandatory "Introduction to Ethics" foundation course offered at one of the seven chartered universities in Israel. The sample included students who came from similar socio-economic backgrounds, and studied in various departments attached to the Faculty of Social Sciences at the university. All students complied with university acceptance criteria (national psychometric university entrance examination and a school-leaving matriculation certificate). All participating students owned smartphones equipped with WhatsApp, Facebook and Twitter access. The students were randomly assigned to three comparison groups ensuring similarity of the groups. The first group of 131 students received their concept definitions via WhatsApp delivery; the second group of 107 students were sent concept definitions by way of Facebook delivery; the third group of 125 students obtained their definitions of concepts through Twitter delivery.

Research instruments

Two research instruments were administered to the students in this research study: a. The first instrument was a standardised ethical concept achievement test which was specially compiled to assess students' mastery of the 140 ethical concept definitions studied in the semester-long course (10 concepts sent to students weekly during the 14-week long course) The test scale ranged from 0-100, the higher grades indicating higher levels of achievement on the test. b. The second instrument administered to the participants was a 23-item Likert-type scale response questionnaire (students responded to a five-point scale with 1=totally disagree to 5=totally agree) designed to examine the attitudes of participants towards their particular learning delivery platform (WhatsApp, Facebook, Twitter) regarding their perceptions of user-friendliness (8 items; Cronbach α=.87); learner motivation (8 items; Cronbach α=.85); and learner satisfaction (7 items; Cronbach α=.89).

Procedure

Following the establishment of the three comparison groups to which the participants were randomly assigned, students in the first group received ethical concept definitions by way of WhatsApp delivery to their smartphones; students in the Facebook group received identical ethical concept definitions relayed to their smartphones; and students in the Twitter delivery group received their ethical concept definitions on their smartphones. The students in the three groups were sent concise definitions of ethical concepts studied in the course with each weekly list containing 10 identical definitions delivered via the three respective learning delivery platforms. Thus, each of the students received the same 140 academic ethical concept definitions during the 14-week long course. On completion of the course, the students in the three comparison groups were administered a standardised ethical concept achievement test to assess their level of knowledge and understanding of the 140 definitions sent to them during the course. In addition, they were administered the 23-item attitudinal questionnaire which examined their scores on the three attitudinal research factors, namely perception of userfriendliness, learner motivation and learner satisfaction regarding the learning delivery platform that they personally experienced during the course.

Twitter Facebook WhatsApp Figure 1. Example of Identical Philosophical Concept presented on the 3 Platforms

The research project adhered to the university research ethics criteria and was approved by the Ethics Committee of the university School of Education.

Results

The main aim of this study was to examine the contribution of WhatsApp, Facebook and Twitter learning delivery platforms to students' achievement as well as to investigate students' affective perceptions of user-friendliness, learner motivation and learner satisfaction derived from studying via WhatsApp, Facebook and Twitter delivery platforms. Descriptive statistics as well as results of analyses of variance were conducted to ascertain possible intergroup differences on the four research variables, as related to the three social network learning delivery platforms, are presented in Table 1. One-way analyses of variance (ANOVA) were conducted to investigate intergroup differences on the four research variables. This statistical procedure was chosen as best suited to assess the differences between the three research groups regarding the relationships between the different delivery platforms and the research variables. Results indicate significant differences on achievement scores between students in the WhatsApp, Facebook and Twitter delivery groups. Results of post-hoc Scheffe tests confirmed that students in both WhatsApp and Facebook delivery groups attained significantly higher grades than students in the Twitter group on the ethical concept definitions achievement test. There were no significant differences between grades of students in WhatsApp and Facebook learning delivery groups on the concept definitions test.

Similar significant results were evident from the one-way ANOVAs conducted to examine intergroup differences on the perception of user-friendliness, learner motivation and learner satisfaction variables. Post-hoc Scheffe tests indicated that students in WhatsApp and Facebook delivery groups were characterised by significantly higher levels of perception of user-friendliness and learner motivation than students in the Twitter group. No significant differences were indicated in the posthoc test between students in the WhatsApp and Facebook groups on these two variables. Regarding the learner satisfaction variable, post-hoc Scheffe tests confirmed that students in the WhatsApp group were typified by a significantly higher level of satisfaction than students in either Facebook or Twitter groups. Results of a post-hoc Scheffe test confirmed that students in the Facebook group were characterised by a higher level of learner satisfaction than students in the Twitter delivery group.

Discussion

The present study examined the comparative contributions of three social network learning delivery platforms (WhatsApp, Facebook and Twitter) to first-year university social science students' knowledge and understanding of definitions of ethical concepts studied in a semester-long "Introduction to Ethics" foundations course. In addition, the study examined the perceptions of the students regarding the user-friendliness of the three delivery platforms as well as the students' levels of learning motivation and learning satisfaction derived from the learning that they experienced in their studies via the different learning delivery platforms.

Results of statistical analyses of the data collected in this study regarding achievement indicate that students in the WhatsApp and Facebook delivery groups attained similarly high grades on the standardised ethical concept definitions knowledge and understanding assessment with students in the Twitter group achieving significantly lower grades on the same standardised measure. These results confirm results of previous research that indicated that both WhatsApp [START_REF] Aburezeq | The impact of WhatsApp on interaction in an Arabic language teaching course[END_REF] and Facebook [START_REF] De Villiers | Evaluation of a collaborative learning environment on a Facebook forum[END_REF] learning delivery contribute to enhanced student learning and achievement. The results are congruent with previous results [START_REF] Yakin | Using twitter as an instructional tool: A case study in higher education[END_REF] regarding the disadvantages of Twitter delivery vis-a-vis students' learning and achievement when compared to the contribution of other social networks, such as WhatsApp and Facebook, to students' performance.

Additional results of the statistical analyses regarding the affective attitudinal variables confirm that students who received their learning content through WhatsApp and Facebook delivery were characterised by similar levels of perceptions of userfriendliness and learner motivation that were significantly more positive than the levels of perception of user-friendliness and learner motivation of students who experienced their learning delivery via Twitter delivery. These results confirm research findings about perceptions of user-friendliness and learner motivation especially associated with WhatsApp [START_REF] Shambare | The adoption of WhatsApp: Breaking the vicious cycle of technological poverty in South Africa[END_REF] and Facebook [START_REF] Isacsson | Facebook as an edutainment medium to engage students in sustainability and tourism[END_REF] learning delivery as reported by students who experienced learning via social networks. A last finding of the present study showed that students in the WhatsApp delivery group were significantly more positive about learning satisfaction than students in both Facebook and Twitter delivery groups, with students in the Facebook group indicating a significantly higher level of learner satisfaction than students in the Twitter group. This result is congruent with research results reported by Echenique et al. [START_REF] Echenique | Students in higher education: Social and academic uses of digital technology[END_REF] regarding the relative advantage of WhatsApp learning delivery over delivery by either Facebook or Twitter.

Conclusions

It may be speculated that the use of WhatsApp and, to a lesser extent, of Facebook platforms, can enhance students' learning achievement significantly more than the use of the Twitter platform. Moreover, the use of these platforms seems to promote more positive feelings of user-friendliness, learner motivation and learner satisfaction. This result can be ascribed to the comprehensiveness of WhatsApp and Facebook styles of communication which are virtually unlimited when compared to a short communication style that characterises Twitter. In addition, WhatsApp, more than Facebook and Twitter, is more hermetically sealed against infiltration of unwanted members who could disrupt the learning process, with Twitter especially open to infiltration of members not interested in the learning process.

It also appears that, despite the vast popularity of Twitter as a means of communication within social communities, students do not rate Twitter as an efficient learning delivery platform that positively contributes to achievement or positive affective attitudes when compared to WhatsApp or, to a lesser extent, to Facebook as learning delivery platforms. Thus, it may be concluded from the results of the present study that students are positive about WhatsApp, and to a lesser extent, about Facebook delivery of learning regarding key variables such as student achievement, as well as learner perception of user-friendliness, learner motivation and learner satisfaction. The results also indicate that Twitter is not considered by students to be as effective as WhatsApp and Facebook as a medium of learning delivery. On the whole, results of the present study confirm and emphasise earlier research findings [START_REF] Ducate | Going mobile: Language learning with an iPod touch in intermediate French and German classes[END_REF][START_REF] Premadasa | Mobile learning environment with short messaging service[END_REF] that clearly indicated that social networks, and especially WhatsApp and Facebook applications, can serve as viable platforms for the delivery of learning content via mobile learning technology.

Table 1 .

 1 One-way ANOVA results for achievement, perception of user-friendliness, learner motivation and learner satisfaction in WhatsApp, Facebook and Twitter learning delivery groups

	Group	WhatsApp	Facebook	Twitter	F
	Variable				(2,360)


Yoshiaki Matsuzawa
email: matsuzawa@si.aoyama.ac.jp

Kazuyoshi Murata

Seiichi Tani

Multivocal Challenge toward Measuring Computational Thinking Bebras Challenge versus Computer Programming

Keywords: Programming, literacy, computational thinking, Bebras Challenge

Towards the establishment of an evaluation platform for computational thinking (CT), in this paper, we use the "Bebras Challenge" coined by Dr. Dagienė as a measurement tool of CT skills. This paper presents a "triangle examination" which includes three kinds of testing methods (programming testing, traditional paper testing, and Bebras Challenges). Approximately one hundred and fifty non-computer science (CS) undergraduate students participated in the examination as a part of an introductory programming course. The result indicated a weak but positive correlation (.38-.45) between the three methods. Additional qualitative analysis for each task in Bebras showed that requirements of algorithm creation and interpretation, and explicitness of the description, are two critical factors to determine a high correlation between other testing methods. We conclude our research by showing a clear correlation between the Bebras Challenge and actual programming.

Introduction

With the banner of "Computational Thinking (CT)" [START_REF] Wing | Computational Thinking[END_REF], a movement of promoting programming education as literacy for all citizens is increasingly growing all over the world. Following European and North American countries, the Japanese government issued a statement which includes compulsory programming education at elementary schools from 2020 [START_REF]Prime Minister of Japan and His Cabinet: 26th Council for Industrial Competitiveness[END_REF]. The purpose of the education is not merely to increase computer science engineers in the future, but to increase 'good users' of computers. While advanced technologies have made a fundamental change in science practices over the past 50 years, a renovation of learning environments for both teachers and learners is required to empower computing [START_REF] Weintrop | Defining Computational Thinking for Mathematics and Science Classrooms[END_REF]. Hence, the purpose of education is to develop citizens who can naturally employ new science practices and live in the knowledge society over the next 50 years.

However, the problem is how we can measure (or know) CT has been laid in front of teachers/researchers. Brennan and Resnick stated "there is little agreement about what CT encompasses, and even less agreement about strategies for assessing the development of computational thinking" on the basis of their long-term experiences in teaching children with Scratch [START_REF] Brennen | New Frameworks for Studying and Assessing the Development of Computational Thinking[END_REF]. Weintrop et al. [START_REF] Weintrop | Defining Computational Thinking for Mathematics and Science Classrooms[END_REF] asserted that "much of the difficulty stems from the fact that the practices collected under the umbrella term CT [START_REF] Wing | Computational Thinking[END_REF][START_REF] Papert | Mindstorms: Children, computers, and powerful ideas[END_REF]". Recently, several studies on CT have been published under the IFIP umbrella (e.g. [START_REF] Angeli | A K-6 Computational Thinking Curriculum Framework: Implications for Teacher Knowledge[END_REF][START_REF] Fluck | Arguing for Computer Science in the School Curriculum[END_REF]); however, in these papers, the discussion around assessment is limited. We think that a further clarification of CT needs a deeper discussion on assessment.

A number of research studies to evaluate student performance in undergraduate programming classes have been conducted using paper examinations. For example, Lister et al. [START_REF] Lister | A multi-national study of reading and tracing skills in novice programmers[END_REF] reported on reading and tracing, and Ford [START_REF] Ford | Assessing the success of an introductory programming course[END_REF] tried to assess the achievement of classes incorporating tests used in cognitive studies in programming. However, there have been very few research studies that discuss methods of evaluating CT as formative problem-solving skills, instead of skills to tackle programming language elements.

Our purpose in this paper is the establishment of an evaluation platform for CT based on the "Bebras Challenge", a measurement tool of CT skills. A "triangle examination" is conducted, which includes three types of testing methods (programming testing, traditional paper testing, and the Bebras Challenge), and the correlation between them is analysed.

Theoretical framework

Computational thinking

There is some consensus between researchers that the movement of computing education is a revival of 1980s programing education with Logo [START_REF] Tedre | The long quest for computational thinking[END_REF]. The origin of this movement originated from Papert, who coined the term computational thinking [START_REF] Papert | Mindstorms: Children, computers, and powerful ideas[END_REF], but was not primarily intended to develop programming skills but to open a new method of learning mathematics through programming. By working in situated environments, children could construct their ideas by directly operating these ideas in a situated world [START_REF] Papert | Perestroika and Epistemological Politics[END_REF]. Papert criticised technocentrism of programming education, and he expressed the purpose as "to give children a greater sense of empowerment, of being able to do more than they could do before" [START_REF] Papert | A Critique of Technocentrism in Thinking about the School of the Future. Talk presented at Children in an Information Age: Opportunities for Creativity, Innovation, and New Activities[END_REF].

More than 20 years after the first generation of programming education, as discussed above, Wing started to use the term CT [START_REF] Wing | Computational Thinking[END_REF] independently. Wing's CT has similarities with the term introduced by Papert, since both of them focus on the necessity of developing those general skills which are needed by all citizens in the knowledge society. Wing's CT was initially discussed in the computer science (CS) community, whereas Papert's term was considered from Piaget's constructivism perspective and has been discussed in the cognitive and learning sciences. This difference brought about a difference in focus: Wing's definition sounds like CS, technology-centric concepts; whereas Papert's use of the term is aimed to foster a greater sense of empowerment in solving problems through computing.

Weintrop et al. [START_REF] Weintrop | Defining Computational Thinking for Mathematics and Science Classrooms[END_REF] offer a recent attempt to define CT by comprehending the literature over 30 years including the two generations of CT discussion, as mentioned above. They emphasised the change in science practices while advancing technology, and how to educate to develop a sensibility in order to literally survive in the world where changed practice is common sense. The higher level problem-solving skills are summarised as: "Data practices"; "Modeling and Simulation practices"; "Computational Problem Solving practices"; and "Systems Thinking practices". The practice to formulate a problem into computational models is defined as "Designing Computational Models", which is included in "Modeling and Simulation practices". "Computer Programming" or "Creating Computational Abstractions" is merely a part of "Computational Problem Solving". Accordingly, the large problem-solving cycle from the formulation of a problem to an evaluation of solutions was defined as "Computing" and then the competencies to conduct the process were defined as CT.

Although the paper claimed it was originally designed for application of CT in science and mathematics, the definition is applicable to other disciplines.

The Bebras Challenge

The Bebras Challenge is an educational practice where students are challenged to do several small tasks, using CS/CT concepts to complete those tasks [START_REF] Dagiene | Bebras a Sustainable Community Building Model for the Concept Based Learning of Informatics and Computational Thinking[END_REF][START_REF] Dagiene | Information technology contests -introduction to computer science in an attractive way[END_REF]. It is also an international informatics contest where a large number of students participate from over 40 countries. Although the activity looks like quizzes that can be used in a classical classroom for grading students, we can see the difference in task design, being described as "wrap[ping] up serious scientific problems of informatics and the basic concepts into playful tasks, inventive questions thus attracting students' attention" [START_REF] Dagiene | Bebras a Sustainable Community Building Model for the Concept Based Learning of Informatics and Computational Thinking[END_REF]. Accordingly, the thoughtfully-designed tasks are playful and appreciated by students.

Bebras includes a "contest" where students can compete with each other through their scores, but that is not the main goal. The goal is "to motivate pupils to be interested in informatics topics and to promote thinking which is algorithmic, logical, operational, and based on informatics fundamentals" [START_REF] Dagiene | Bebras a Sustainable Community Building Model for the Concept Based Learning of Informatics and Computational Thinking[END_REF]. In other words, it can be explained as promoting the enjoyment of thinking, the learning of activities embedded in the procedure, and consequences performed as a formative evaluation of the learning process.

The Bebras international contest was created in 2004 by Dr. Dagienė, with a first report published in 2006 [START_REF] Dagiene | Information technology contests -introduction to computer science in an attractive way[END_REF]. From the practice and research conducted over a decade, a review paper was published in 2016 [START_REF] Dagiene | Bebras a Sustainable Community Building Model for the Concept Based Learning of Informatics and Computational Thinking[END_REF], which included approximately 50 papers published during the previous ten-year period.

The Bebras Challenge as assessment tools

Although the Bebras challenge is not designed to assess students' knowledge or skills, Dr. Dagienė refers to the capability of Bebras as an assessment tool over a long time period [START_REF] Dagiene | Bebras a Sustainable Community Building Model for the Concept Based Learning of Informatics and Computational Thinking[END_REF]. One paper [START_REF] Dagiene | Bebras a Sustainable Community Building Model for the Concept Based Learning of Informatics and Computational Thinking[END_REF] describes how one of the most important and required cognitive operations in CSTA (Computer Science Teachers Association) K-12 Standards -using visual representations of problem states, structures, and datainvestigates the "task-based assessment" approach to assess CT. Hubwieser et al. validated the use of item response theory, focusing on whether Bebras tasks could assess CT in CSTA [START_REF] Hubwieser | Playing PISA with Bebras[END_REF][START_REF] Hubwieser | Investigating the Psychometric Structure of Bebras Contest: Towards Measuring Computational Thinking Skills[END_REF]. Dolgopolovas et al. [START_REF] Dolgopolovas | Exploration of Computational Thinking of Software Engineering Novice Students Based on Solving Computer Science Tasks[END_REF] expanded on Hubwieser's work and tried a validation of Bebras as assessment tools of CT with first-year software engineering university students. This study included the following two research questions (RQs): RQ1: How can the CT skills of novice software engineering students be evaluated independently of programming language? RQ2: What is the relation between novice software engineering students' computational thinking skills and programming course results?

For RQ1, a study was attempted by Bebras with 65 university students. They succeeded in validating the test using item response theory, as described in a previous study [START_REF] Dolgopolovas | Exploration of Computational Thinking of Software Engineering Novice Students Based on Solving Computer Science Tasks[END_REF]. However, for RQ2, they failed to find a correlation between Bebras and examination scores in the programming course. The paper [START_REF] Dolgopolovas | Exploration of Computational Thinking of Software Engineering Novice Students Based on Solving Computer Science Tasks[END_REF] discussed how the failure resulted from the quality of questions in the examination, which asked for practical knowledge in programming (e.g. grammar and knowledge of a particular library) instead of algorithms, formalisation, or abstraction that are required in CT.

Djambong et al. [START_REF] Djambong | Task-Based Assessment of Students' Computational Thinking Skills Developed through Visual Programming or Tangible Coding Environments[END_REF] offers another influential study, which attempts a summative assessment of programming education where both grade 6 and grade 9 students engage in a robot programming activity using LEGO Mindstorms. Bebras was examined using a pre-and post-test after five hours (during a five-week period) in a classroom, but significant differences were not identified.

The study detailed in this paper can be seen as a revised version of this work. We conducted an experimental study using the same research question (RQ2), but we asked for actual programming construction in the examination in order to reveal any correlation between computer programming and Bebras activities with non-CS students.

Method

Research question

Our research question is as follows:

RQ: Could we use Bebras as an assessment tool for computational thinking, and if so, to what extent? (What is the correlation between Bebras, the actual programming test, and the paper test?)

The aim of the RQ is an evaluation of Bebras as an assessment tool of CT by examining the correlation between Bebras, the actual programming test, and the paper test. Although the result of this question in the previous research [START_REF] Dolgopolovas | Exploration of Computational Thinking of Software Engineering Novice Students Based on Solving Computer Science Tasks[END_REF] was negative, we hypothesised that a positive result could be achieved by improving the paper and programming tests by asking about CT instead of practical programming knowledge.

Experimental study environment

An experimental study was conducted in an introductory programming course designed for social informatics (non-CS) major students. The class was designed to develop language independent CT skills through the practical programming experiences with Visual Basic, and HTML (HyperText Markup Language) for all students in the major class (compulsory).

The time schedule of the class and the examination is shown in Figure 1. The class was scheduled over 15 weeks (three hours per week), including an examination in the weeks, and then engaged in web page authoring in HTML during the last four weeks.

Figure 1. Time schedule

The first trial of Bebras (Bebras1) was carried out in week 2 as a pre-test, and the second trial was carried out in week 15 as a post-test. The programming test was conducted in week 9 and the paper test was conducted in week 16. As there was a winter break in week 13, there was an approximate 10-week interval between the programming and paper tests. Despite this gap, we assumed the skill levels of the students were essentially the same, as the content of the class during the term was on web page authoring in HTML. As the Bebras trial was addressed as a part of the lectures, students were given as an incentive a maximum score of 10% of their grade.

Design of the Bebras task

The Bebras task used in this study was from the senior level Japanese version of Bebras tasks. We considered that the difficulty of the senior tasks fitted well for first-year university students. Ten tasks were selected from Bebras 2015 and 2016 Challenges for the first trial (Bebras1). As a pilot trial had been carried out with a different student cohort one year earlier, the tasks which had a statistically high correlation were selected. Twelve tasks in all were selected from the Bebras 2017 Challenge for the second trial (Bebras2). All of the selected tasks are listed in the results section.

The time limit for the challenge was set to 40 minutes in both trials. The time was a little longer than that in the international challenge in order to encourage students to think deeply while doing each task. The web-based system used in the international challenge was used for the experiment. Students were given 10 minutes to practice the use of the system; consequently, there were no students who failed to answer the tasks.

Design of the programming test

The programming test examined during the 9 th week of class was an examination to develop a program to meet the requirements presented to students using a specification sheet. All of the students were assigned a single computer, and then were asked to submit a workable code in Visual Basic. The students were allowed to access all course materials during the examination, although access to the Internet was prohibited.

Students were required to create four tasks, given on the specification sheets, within 85 minutes. The easiest task did not require using any loops, but to use branching. The most difficult task required using a collection (the course teaches Listbox for abstract data collection) and developing an algorithm to process inside the collection.

During the test, the students could use a development environment (Visual Studio). Therefore, the activity measured comprehensive skills of program construction: designing data structure and algorithms; correcting compilation errors; or debugging. Additionally, the time constraint required a certain level of fluency in their programming processes; we observed that some students gained full marks close to the time limit.

Design of the paper test

The paper test administered during the 16 th week of the class was the traditional examination carried out using paper and pencil. Students could access all course materials during the examination as well as the programming test, while the use of all electronic devices was prohibited. The test was in six sections, lasting 85 minutes.

There were three kinds of tasks (code tracing, code complementing, and code ordering) among the sections analysed. Code tracing is a type of task where students read the code written in Visual Basic and then answer the outputs, values of variables at a particular state, or figures expected to be drawn by the code. Code complementing is a type of task where students complete a part of the code to complement the blanked, uncompleted code. Code ordering is a type of task where students initially order code fragments in random order to complete a workable code. In this manner, the tasks of the paper test were designed to assess algorithmic thinking, which is located within basic grammar knowledge.

Results

Descriptive statistics

The study was conducted in our introductory programming class during the 2017 academic year. Approximately 200 students in the school are required to register for this course, with the students being randomly assigned to one of four separate classes in order to reduce the number of students in each class. Each of the four classes was managed by different teaching staff, but the course materials and examinations were the same. Three of the four classes were randomly selected for the analysis. The descriptive statistics of the four examinations are shown in Table 1. In Table 1, "n" refers to the number of students, and the other number indicates the average score of the four tests: Programming; Paper; Bebras1; and Bebras2, respectively. A total of 137 students' data were used in the analysis. No significant differences in scores were found between the classes according to these descriptive statistics.

Correlation analysis between the four examinations

Figure 2 is a correlation table, showing the correlation and scatter plot between the four examinations. The correlation between Programming and Bebras was approximately 0.4. The correlation between Paper and Bebras was approximately 0.45. Although a strong correlation was not indicated, this is clearly a different result from the previous research [START_REF] Dolgopolovas | Exploration of Computational Thinking of Software Engineering Novice Students Based on Solving Computer Science Tasks[END_REF], as no relationship was reported between the examinations in that study.

Figure 2. Correlation table of the four examinations

Additionally, the correlation value can be considered as reaching a significant level when considering it in relation to the other correlations. The paired correlations between Programming and Paper, and Bebras1 and 2, respectively, are approximately .6. We assume the two pairs of tests were quite close in terms of methods, respectively. Therefore, these correlation values were high; however, they could be considered as not being higher than what we had expected. Notably, the correlation between Bebras1 and Bebras2 could be expected to be much stronger (.8-.9) if the validity of the tasks was improved. Accordingly, the correlation value between Bebras and other tests (.4-.45) can be considered sufficient to show a significant relationship between the actual programming and Bebras.

Detailed analysis for the tasks

A detailed analysis of the correlation between the programming test and each task in Bebras was performed. Table 2 shows the percentage of correct answers, and the significant differences between the paper test, programming test, and each task, respectively. The significant differences are indicated by the average scores between the group that answered correctly and the group that failed in answering for each task. The average score of the group that answered correctly was higher than the group that failed their answers in all the tasks. The result of a qualitative analysis for each task was conducted by the first author, although he was not a specialist in Bebras tasks, or in the interpretation of the results from the viewpoint of the correlation between Bebras, Programming, and CT.

Overall, the results of the programming and paper test indicate a similar tendency in the correlations. For example, significant differences were consistently found in AQ1, AQ6, AQ7 and BQ5, BQ7, BQ9, whereas no significant differences were consistently observed in AQ3, AQ9, AQ10, and BQ1, BQ2, BQ3, BQ11, or BQ12.

The tasks showing significant difference involved algorithm comprehension, creation, and abstraction. A typical example is AQ1 "Pencils alignment" where the task asks about the result of sorting algorithms written in natural language. There was a direct relationship to the topic of sorting algorithms that was taught in the lecture. Tasks which included an algorithm in a geometric field such as AQ5 "Finding the final state" or BQ6 "Exit the maze", were expected to show strong correlation. For the two tasks, only the paper test indicated significant differences, so further consideration is needed for this in terms of interpretation.

The tasks indicating no significant differences included relatively little algorithm comprehension, but also included other CT concepts, for example, data structure expressions, such as AQ4 "Mobiles." Although the lecture included basic data structure expressions with linear collection, further data structure expression, used to model the actual world, seemed to be difficult in introductory programming for non-CS students.

For other examples, AQ2 "Ice Cream Shop" requires the concept of database structure, and BQ1 "Grandmother's jam" requires the concept of task scheduling.

Another possible deterministic factor to be considered was whether an expression of a task was explicit or not. For example, in BQ7 "Robot", the rule to be applied is simple and also clearly given by illustrations, whereas in BQ3 "Files" students have to construct a procedural algorithm by declarative rules. Another example, BQ12 "Library" was seen to create difficulty in description, although this may have been caused by local translation.

Discussion

The RQ of the study was "Could we use Bebras as an assessment tool for computational thinking? And if so, to what degree?" As shown in Section 4.2, the results indicate a positive correlation (.38-.45) between the three methods. As we discussed, this was an opposite result from previous research [START_REF] Dolgopolovas | Exploration of Computational Thinking of Software Engineering Novice Students Based on Solving Computer Science Tasks[END_REF]. One notable difference of the two separate research studies is the design of the test; the test in this research required algorithm creation with actual programming, whereas the previous research required declarative information for a programming language. The results of the qualitative analysis for each task in Bebras support this consideration. The analysis revealed two critical factors: a requirement of algorithm creation and interpretation; and the explicitness of the description. Hence, we conclude here that the research showed clearly the difference between the Bebras Challenge and actual programming.

As was expressed earlier, a significant assumption in this research was that the actual programming requires a certain level of assessment ability in CT in higher-level problem-solving skills [START_REF] Weintrop | Defining Computational Thinking for Mathematics and Science Classrooms[END_REF]. One significant criticism is that Bebras can be more accurate in measuring language-independent CT than actual programming. However, even if Bebras tasks enable users to operate CT concepts in a language-independent way, the concepts should be finally applied in practical situations: computer programming. Bebras Challenge trials lead to basic understanding of CT concepts, while subsequent programming experiences make the understandings deeper, and consequently they appear as a Bebras score. This cycle is expected in the design of Bebras. Accordingly, we believe our results will encourage all programming researchers/practitioners who are engaged in supporting language-independent and creative programming practices instead of conveying only syntax knowledge.

Table 1 .

 1 Descriptive statistics

		n	Programming	Paper	Bebras1	Bebras2
	Class A	43	49.2	50.5	64.9	55.4
	Class B	45	53.3	50.4	71.6	57.4
	Class C	49	44.7	44.0	62.3	55.3
		137	48.9	48.2	66.2	56.0

Table 2 .

 2 Percentage correct answers and average score significant differences by task

	Q ID	Task ID	Name	Correct Program Paper
	AQ1	2015-LT-04	Pencils alignment	0.78 **	**
	AQ2	2015-JP-05	Ice Cream Shop	0.74 ns(p<.1) *
	AQ3	2015-CA-02	Fireworks	0.64 ns	ns
	AQ4	2015-DE-05	Mobiles	0.39 ns	*
	AQ5	2016-CZ-06	Finding the final state	0.76 ns(p<.1) **
	AQ6	2016-MY-02	Scanner code	0.85 *	**
	AQ7	2016-NL-04	KIX Code	0.58 **	**
	AQ8	2016-AT-06	Recursive painting	0.49 ns	*
	AQ9	2016-JP-02	Paint it black	0.70 ns	ns(p<.1)
	AQ10 2016-IT-02b	Red and blue marbles	0.69 ns(p<.1) ns(p<.1)
	BQ1	2017-RU-04	Grandmother's jam	0.42 ns	ns(p<.1)
	BQ2	2017-JP-04	Colorful Building	0.64 ns	ns(p<.1)
	BQ3	2017-AT-03	Files	0.67 ns	ns
	BQ4	2017-MY-05	Moving Die	0.60 **	ns
	BQ5	2017-IR-06	Bebragram	0.80 **	**
	BQ6	2017-CH-01a	Exit the maze	0.73 ns(p<.1) **
	BQ7	2017-SK-12b	Robot	0.54 **	**
	BQ8	2017-KR-07	Icon Image Compression	0.48 ns(p<.1) **
	BQ9	2017-KR-02	A Stray Baby Beaver	0.69 *	*
	BQ10 2017-DE-09	BikeFun	0.72 ns	**
	BQ11 2017-RU-02	Digit recognition	0.09 ns	ns
	BQ12 2017-IT-10	Library	0.37 ns(p<.1) ns
					*p<.05, **p<.01

Acknowledgments. This work was supported by JSPS KAKENHI Grant Numbers 16K00488, and 17H06107.


Halvdan Haugsbakken
email: halvdan.haugsbakken@ntnu.no

Inger Langseth
email: inger.langseth@ntnu.no

Designing an Educational Action Task Force for MOOCs and Online Course Production

Keywords: Online courses, MOOC, organisational design, network, coaching

Although the research literature on online courses, such as massive open online courses (MOOCs), has proliferated, surprisingly few studies have explored the organisational approach to a generic institutional strategy for supporting educators when developing online courses in higher education (HE). The goal of this paper is therefore to describe and conceptualise the outline of an infrastructure for organising the production of online courses in continuous and further education. Central to the infrastructure is the Educational Action Task Force (EATF), a network consisting of employees with complementary competences (c.f. technical, pedagogical and multimedia) that can coach, mentor and support educators through the entire online course production process in designated teams. In this article, we outline the design of the online course production process in the EATF teams. The design is stepwise and collaborative, and aims to contribute to a seamless and quality-assured strategy that caters for the various goals that content creators may have within the scope of the strategic goals in the organisation.

Introduction

The rationale for designing an Educational Action Task Force "EATF", a flexible, collaborative and networked support unit that will support faculty to make online courses, is based on research and experiences acquired at a large Norwegian university. This research shows the increased need for putting focus on how online courses are made [START_REF] Langseth | Introducing Blended Learning MOOC -A Study of One bMOOC in Norwegian Teacher Education[END_REF][START_REF] Jacobsen | Dropping Out or Dropping In? A Connectivist Approach to Understanding Participants' Strategies in an e-Learning MOOC Pilot[END_REF]. There is a demand for a technological infrastructure, and pedagogical support for faculties that wish to make high-quality online courses. At our university, MOOC initiatives are short-lived and significantly rely on project funding and enthusiasts to survive. Even so, there has been no centralised strategy for producing certified online courses and MOOCs for continuous education. Such factors inhibit scalability and flexibility in online course production. Consequently, the need for an institutional support unit that can assist educators and assure the quality in (massive, open) online course production, other than ordinary courses delivered on the learning management system (LMS), has emerged.

Inspiration for establishing an EATF

In research on MOOCs, we find few studies focusing on the essential requirements and conditions for supporting faculty in their production of a quality-assured MOOC for the first time. For example, little research addresses the implications related to the collaborative production process, the choice of platform, the planning of the course design and the video production time involved, which are essential. In fact, a MOOC production is a collaborative activity that can last for months. Moreover, we argue that ideas about making high-quality MOOCs are often scrapped, partly due to the commitment that educators must make and partly due to the lack of technical and pedagogical support in higher education (HE). Instead, a prolific MOOC research literature has emerged, focusing on the analysis of the activities and outputs in MOOCs, like decomposition of user groups [START_REF] Kizilcec | Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses[END_REF], video engagement among learners [START_REF] Guo | How video production affects student engagement: an empirical study of MOOC videos[END_REF], drop-out rates [START_REF] Clow | MOOCs and the funnel of participation[END_REF] etc., overlooking the efforts to make one. In a recent research review, Sanchez-Gordon and Luján-Mora [START_REF] Sanchez-Gordon | Research challenges in accessible MOOCs: a systematic literature review 2008-2016[END_REF] suggest that researchers need to redirect their focus towards developing clearer strategies and standards for MOOC course design, which supports our claim that the strategic and organisational aspects of faculty's MOOC production processes must be highlighted. Their argument is supported in a recent study on the design of online learning opportunities associated with MOOCs where they found that the quality of instructional design across 76 MOOCs was limited [START_REF] Margaryan | Instructional quality of Massive Open Online Courses (MOOCs)[END_REF].

The proposed outline for the EATF for MOOC and online course production is a lasting, non-project based organisational construct that can support faculty, who will share their subject-specific expertise in a MOOC or an online course. The EATF is organised around three core principles: collaboration; distributed network online; and a stepwise production process design.

First, the adoption of cloud-based services for teaching and learning facilitates scalable and flexible courses. The EATF is a collaborative endeavour, based on a formalised partnership between the educator(s) and designated EATF-team members during the production period. Together, they form a network of some 3 to 7 participants, meeting at regular intervals to discuss, learn and inform stakeholders at different stages in the MOOC production process. Each meeting has a fixed agenda to ensure progress. Second, the EATF performs all internal and external activities and assignments in a distributed network online. This implies that the EATF is mainly cloud-based and that meetings and collaboration are largely online. This allows for a more flexible and transparent workflow. An overall goal is to coach and mentor faculty in digital collaborative spaces (Office 365, Skype, etc.) instead of having many face-to-face meetings on campus. Consequently, the EATF networked design is closely related to what Groth calls "ad-hoc organizations" [START_REF] Groth | Future organizational design: the scope for the IT-based enterprise[END_REF]. Third, the EATF aims to establish and maintain a coherent and stepwise course production process design, a "virtual assembly line" that contributes to transparency and course content quality assurance. Transparency is also a cornerstone for continuity and quality performance in the EATF over time.

In the EATF, we aim at using coaching techniques to uncover faculties' initial motivation and intentions in order to support their aims and objectives. Coaching is an emerging research field that has been successfully used in health care, sports and private business to make performers stay on target, enhancing performance, self-esteem and intrinsic motivation [START_REF] Horn | Coaching effectiveness in the sport domain[END_REF]. Very little research has been conducted on the gains from coaching in formal educational [START_REF]Erasmus+: COACH (Coaching SchOols to fAce Change aHead)[END_REF][START_REF] Van Nieuwerburgh | [END_REF]. We also aim at using mentoring to inform faculty of online course design, video production and platform technicalities. The details about how we envision our approach is outlined in the next section of the paper.

The EATF support process

Our experience, from MOOC production since 2013, is that online course production easily drags out in time terms. Educators tend to underestimate the workload and focus too quickly on video production. They base their idea of the online course on previous experience from teaching on campus and have little understanding of the pedagogical limitations of the platforms and of the difference between online and on-campus teaching. The latter is also reflected in studies that we have previously referred to in [START_REF] Margaryan | Instructional quality of Massive Open Online Courses (MOOCs)[END_REF].

To enhance the quality of the instructional design and make the online course production process more efficient, the EATF works in a streamlined production process consisting of seven steps. The main goal is to form a growth spiral where the educator and the EATF team members collaborate in a network to complete the course production. The network typically consists of 3 to 7 members with complimentary competences in: 1) online course design and pedagogy; 2) multimedia production; 3) frontend representation and platform functionalities; and 4) expertise in the course content area. In each step, the network collaborates to understand and carry out tasks that have to be completed, before the team moves on to the next step in the production process. Information about the tasks in the various steps is also available online to support the educators between the meetings in the network, with a form that shows an overview of the activities and progress made.

The EATF makes use of coaching and mentoring as well as reflection-on-action [START_REF] Schön | The reflective practitioner: how professionals think in action[END_REF] to motivate educators and help them understand the actions required to make an online course. Coaching is an emerging research field and can be described as questioning and listening techniques that may help educators see more clearly where they are, where they want to be when the online course is finished, and how to get there. Coaching is a method that has the capacity to motivate educators to stay on track to reach their goals. Research from the Erasmus+ COACH project [START_REF]Erasmus+: COACH (Coaching SchOols to fAce Change aHead)[END_REF] shows that coaching has positive benefits in educational environments. An important strength is what may emerge in terms of increased reflectivity, stronger cultures of collaboration, sharing of knowledge and greater engagement with professional development [START_REF] Horn | Coaching effectiveness in the sport domain[END_REF]. The GROW model, which is an acronym for Goal, Reality, Options and Will, has been used with great success in sports and corporate business. It is an approach that can be used to aid educators to positively reach their final goal through a series of supported steps. Mentoring is rooted in Vygotsky's theories of the "zone of proximal development" and can be understood as adult learners engaged in new learning and relearning in changing educational contexts that demand a new view on education [START_REF] Fletcher | A: Sage handbook of mentoring and coaching in education[END_REF].

Following, we outline some initial ideas of the potential content and composition in the EATF support process.

Step 1, Initial clarification of goals and motivation: The first step is to invite the educators to an initial meeting. The meeting intends to raise their awareness of the educational context and help them clarify the goal and the steps that will have to be taken to reach their goal by means of coaching techniques. Also, the objective is to form an EATF team that can support the educators in the best way. The conversation initiates a formalised, collaborative networked process to support the educators so that they can reach their goal.

Step 2, Selection of platform and work flow: In this step, the educators make various decisions that are supported by mentoring. First, the educators are introduced to the EATF-team members and the result from the first meeting is discussed to contribute to further specify the conditions for the course production and inform all stakeholders. Second, the educators get a better understanding of the potential and limitations of the various course platforms and the nature of the support the other team members can offer. Third, the educators select an appropriate platform for the course. An important activity is to introduce the educators to a course on the platform, which serves as an example of what the course might look like. Fourth, the educators decide upon a work flow to commit to a timeline for the course production and to decide upon online collaboration software and when to meet next and whether to meet face-to-face.

Step 3, Course design: In this step, the educators are introduced to online instructional design. They are invited to reflect on learning objectives, the number of modules, types of activities and forms of assessment in the context of their chosen course platform. They explore the selected platform and discuss possibilities and limitations with more experienced platform users. An important activity in this process is to discuss what type of content is better suited as text, picture or video. At the end of the meeting, the educators have a clearer idea of what the MOOC will look like and is able to go to the next step with a draft or overview of how many pictures, animations and videos they will create and content that can be reused in the course.

Step 4, Multimedia workshop: In this step, the educators are invited to a multimedia workshop. The focus is on preparing the educators for the video production and support the multimedia course content. In the workshop, they are first introduced to video-production methodology and how to visualise knowledge in a video for an online audience. Educators come to the workshop with a written text, a treatment or a draft that will be completed with multimedia content in collaboration with the team. They will typically discuss different concepts like a talking head, two-dimensional (2D) animation, voiceover, slide layout, illustrations, etc., to support the written content that will be read on a teleprompter. In the workshop, the team finally outlines a detailed video production schedule for the next step.

Step 5, Video production: In this step, the focus is on the actual video production process. The project team strives to create an atmosphere of trust, where uncomfortable and inexperienced educators are supported to make the best possible product.

Step 6, Uploading content to platform: In this step, the course content production is completed, and the educators are ready to start uploading the course to the platform. The EATF team informs the educators about technical platform support depending on the educators' technical competence. The team also discusses how to proceed with user testing, piloting, and feedback from peers on content quality when the course is online.

Step 7, Evaluation: In the last step, the team meets to finalise the online course and support the educators for launching and possible marketing. The educators are also invited to evaluate the EATF and pertaining pedagogy, methods and technologies to contribute to quality assurance. Feedback is essential, to improve the supportive network.

Conclusion

This paper intends to address a missing link in the MOOC research literature -how to organise support for educators who want to make MOOCs. Researchers have examined student user patterns and outputs but seldom cast lights on the efforts involved in making them. This paper attempted to outline some initial ideas on how HE institutions can organise support for educators, who want to produce MOOCs, in a conceptual framework for MOOC and online course production. The seven steps outlined contribute to a scaffolded technical, pedagogical and quality-assured process, which contributes to efficiency in online course production and quality enhancement for online learners. The introduction of an organisational approach to structuring support for online course production contributes to closing the gap in the MOOC research literature.


Bastian Küppers
email: kueppers@itc.rwth-aachen.de

Ulrik Schroeder
email: schroeder@cs.rwth-aachen.de

Students' Perceptions of e-Assessment A Case Study from Germany

Keywords: Computer-based Assessment, e-Assessment, BYOD, Cheating

In order to verify common findings in the literature regarding the conception of e-assessment among students, we carried out a survey based on common findings. Our survey, which has been carried out over several higher education institutes, enhances the already existing findings by adding new facets. The achieved results are promising in that students seem to be open-minded regarding e-assessment, which is in line with the findings in the already existing literature. However, there are some open points that have to be resolved in a reliable way in order to completely convince the students of the opportunities offered by e-assessment.

Introduction

If e-assessment is to be introduced into the examination system of an institute of higher education (IHE), it is not only the staff of the latter that have to accept this type of assessment, but also the students [START_REF] Terzis | The acceptance and use of computer based assessment[END_REF], especially when it comes to e-assessment on students' devices (BYOD) [START_REF] Hillier | e-Exams with student owned devices: Student voices[END_REF]. Therefore, it is of interest for IHEs that are willing to introduce e-assessment, to be aware of the possible limiting factors from the students' points of view, to tailor the e-assessment system and the process of integration to the students' needs.

To verify that the findings regarding the students' points of view in the literature are valid for our institution, we carried out our own survey about e-assessment, BYOD scenarios and cheating in examinations.

The paper is structured as follows: in the second section, we give a brief overview of the findings already presented in the literature. In the third section, we discuss the setup of our survey, followed by a discussion of the achieved results in the fourth section. The paper closes with a summary and an outlook.

There is a lot of literature about students' perceptions of e-assessment, which has been written over the last years. Most of these papers focus on a particular IHE, e.g. Saudi Electronic University, Saudi Arabia [START_REF] Alsadoon | Students' Perceptions of E-Assessment at Saudi Electronic University[END_REF] and Dow University of Health, Pakistan [START_REF] Jawaid | Computer Based Assessment (CBA): Perception of residents at Dow University of Health Sciences[END_REF]. Some papers focus even on a single study course, e.g. Polytechnic Institute of Porto, Portugal, Marketing Degree [START_REF] Babo | Students' Perceptions about Assessment Using an e-Learning Platform[END_REF], University College London, UK, Chemical Engineering [START_REF] Sorensen | Implementation and student perceptions of e-assessment in a Chemical Engineering module[END_REF], Hong Kong Polytechnic University, Hong Kong, Rehabilitation Sciences) [START_REF] Hodgson | Effective Formative E-Assessment of Student Learning: A Study on a Statistics Course[END_REF] and Kocaeli University, Turkey, Desktop Publishing [START_REF] Özden | Students' Perceptions of Online Assessment: A Case Study[END_REF]. The findings reported in these papers testified generally positive students' attitudes regarding eassessment.

For the course of this paper, the most important publication is "e-Exams with student owned devices: Student voices" by Matthew Hillier [START_REF] Hillier | e-Exams with student owned devices: Student voices[END_REF], since his paper focuses on a BYOD scenario. There are many interesting findings about students' perceptions not only regarding e-assessment, but especially about their perceptions regarding eassessment on their own devices. However, even Hillier's research was conducted only in one IHE.

Design of the survey

We constructed our survey based on the findings in a previous paper [START_REF] Hillier | e-Exams with student owned devices: Student voices[END_REF], to answer our research question: Which factors influence students' perceptions of e-assessment?

We anticipated that the perception of e-assessment is influenced by: • gender • age • the study programme (science, technology, engineering and mathematics (STEM) versus humanities, for example) • technology affinity • the stage of study (Bachelor versus Master level) Since we expected the results to be additionally influenced by the general technology affinity of the students, we incorporated another questionnaire as part of our survey to be able to distinguish technology accepting students and technology reluctant students. This questionnaire is the TA-EG questionnaire by Karrer et al. [START_REF] Karrer | Technikaffinität erfassen -der Fragebogen TA-EG. Der Mensch im Mittelpunkt technischer Systeme[END_REF], which is designed to measure technology affinity. The items of the TA-EG questionnaire have been reordered to eliminate effects that could originate from the clustered answers of the original questionnaire. Additionally, unlike existing surveys, we wanted to carry out the survey at multiple IHEs and for different study courses.

Altogether, this resulted in the, originally German, survey as shown in Table 1. The survey was carried out mainly with students of RWTH Aachen University and FH Aachen University of Applied Sciences, but also students at Maastricht University, Alpen-Adria-University Klagenfurt, TU Berlin, FOM Hochschule für Oekonomie und Management (Study Centre Aachen) and Albstadt-Sigmaringen University were invited to participate. The study programmes mentioned explicitly in the survey are the main study programmes, which are related computer science courses at those universities. Five-level Likert Scale (5LLS)

Electronic devices cause illness.

I like to go to stores for electronic devices.

I (would) have problems understanding electronic and computer magazines.

Electronic devices provide a high standard of living.

Electronic devices lead to intellectual impoverishment.

Electronic devices make many things more complicated.

I inform myself about electronic devices, even if I have no intention to buy them.

Electronic devices make you independent.

I enjoy trying out electronic devices.

Electronic devices make everyday life easier for me.

Electronic devices increase security.

Electronic devices reduce personal contact between people.

I know most of the functions of the electronic devices I own.

I am thrilled when a new electronic device comes onto the market.

Electronic devices cause stress.

I know about electronic devices.

It is easy for me to learn how to operate an electronic device.

Electronic devices help to obtain information.

e-Assessment

E1: I think it is very good to have electronic examinations in my studies. 5LLS E2: I think that electronic examinations are a good complement to paper-based examinations. E3: I think that electronic examinations are a good substitute to paper-based examinations.

E4: I see advantages of electronic examinations

Analysis of the results

In total, 408 students responded to the survey with demographics as shown in Table 2. About three quarters of the participating students were male and one quarter were female. A similar distribution can be seen for the age, where about three quarters were aged between 18 and 25 years and nearly a fifth of the students were aged above 25 years.

The students came from a variety of study programmes, as can be seen from Figure 1. Other programmes of study included artificial intelligence, engineering and physics. So, despite individual students studying in programmes like economics and literature, the vast majority of the study programmes were related to a STEM topic. Therefore, it is not surprising that the results of the TA-EG questionnaire did not allow for identifying subgroups with different affinities regarding technology.

The plots in Figure 2 refer to the original grouping of the TA-EG questionnaire, which has four groups: Enthusiasm, Competency, Positive Attitude, and Negative Attitude. The five subplots in each of these plots refers to a question in the corresponding group of questions of the TA-EG questionnaire. Please note that every item in the TA-EG questionnaire in our survey used a five-level Likert scale ranging from 1 ("Strongly agree") to 5 ("Strongly disagree"). The overall variance of these items was 0.76, which accounts for the indistinguishability of different subgroups. Since there were too few students enrolled in a study course that were not from the STEM field, the collected data were not suitable to answer whether the study course influenced the students' perceptions of e-assessment. The absence of those students may be caused by the decision to carry out the survey via an online portal, which may have biased the results so that only students participated who had an affinity for technology. However, that cannot be concluded from the data.

Influences of gender and age

To examine the influence of gender, age, and study level (bachelor or master) the data set was split into subsets accordingly. These subsets were then tested for significant differences with a Fisher test [START_REF] Upton | Fisher's Exact Test[END_REF]. The results for the Likert-scaled questions can be found in Table 3. Given these p-values, conclusions about the influence of gender, age and study level are possible to a certain extent. Regarding question E1, it seems that women are more hesitant to accept e-assessment as part of the examination system. In addition, students between 18 and 25 years seem to be more positive about e-assessment than students of other ages. For question E2, the age again makes a difference, as students older than 25 years seem to be less convinced that e-assessment is a good complement to paper-based examinations compared to younger students. The same tendency is revealed when considering the question about whether the study level influences the perception of eassessment. Students that are enrolled in a master's programme seem to be more reluctant regarding e-assessment than students in a bachelor's programme. If this tendency is caused by the progress in the studies or by age, again, is a crucial factor that cannot be concluded from the data, which is shown in

Discussion

The achieved results from the survey show a rather clear picture. The students would like to have electronic examinations in their studies, but not necessarily as a replacement for paper-based examination, but rather as a complementary approach, as the answers of questions E2 and E3 (see Figure 4) suggest. This perception of eassessment is understood through advantages, which cover topics like faster correction (75.98%, E4), more realistic assignments (62.74%, E4), more diverse examination tasks (45.34%, E4), and readability (which was stated in free text comments). However, students are also concerned about disadvantages, like security (41.67%, E5), usability (42.64%, E5), and fairness (34.56%, E5). Additionally, technical difficulties and the subsequent loss of already solved assignments are mentioned very often in the comments. Overall, less than half of the students sees disadvantages in e-assessment; however, especially when it comes to a BYOD approach, the students are afraid that technical difficulties may lead to a handicap for them or that they have to have a capable device on their own. Still, the tendency seems to be positive regarding a BYOD approach (see B1 in Figure 4), as students see the advantage of a familiar device (89.7%, B2). However, due to the reported concerns, it is very important to have a reasonable backup strategy for these situations. As we have discussed elsewhere [START_REF] Küppers | Reliable e-Assessment with git -Practical Considerations and Implementation[END_REF], it is important to regularly have backups during an e-assessment, so that a student can simply switch to an emergency device provided by the IHE in case her own device breaks down. These emergency devices could also be used for students that cannot afford a device on their own in order to enable these students to participate in electronic examinations. Additionally, the topic of fairness is important to the students, as they state differences between the students' devices as the main concern when utilising BYOD (82.84%, B3). Furthermore, topics like security or cheating are of importance for the students. The students are rather split about the risk of cheating in paper-based examinations; however, there is a tendency that students think that it is easier to cheat in electronic examinations (see C1 and C2 in Figure 4). Therefore, new ways of reducing the risk of cheating in electronic examinations have to be found, as we have discussed elsewhere [START_REF] Küppers | Beyond Lockdown: Towards Reliable e-Assessment[END_REF] and presented an approach to security [START_REF] Küppers | Practical Security for Electronic Examinations on Students' Devices[END_REF].

Age is seemingly a factor; it does influence the perception of e-assessment in line with the concept of Digital Natives introduced by Marc Prensky [START_REF] Prensky | Digital Natives, Digital Immigrants[END_REF]. He claims that "[t]oday's students have not just changed incrementally from those of the past", but underwent a drastic change of attitude, because "the arrival and rapid dissemination of digital technology in the last decades of the 20th century [was] an event which changes things so fundamentally that there is absolutely no going back". The evidence gained from the survey suggests a similar conclusion, because there is a statistically significant difference between students over the age of 25 years in comparison to younger students. Shelley White states in her article "The Generation Z effect" [START_REF] White | The Generation Z effect[END_REF] that "Gen Z is loosely accepted as people born in the mid-to late-1990s and later. (According to the Pew Research Center in the United States, the last Gen Y was born in 1997, while Statistics Canada says Gen Z starts with people born in 1993)". The timespan mentioned in her article is exactly in line with our findings of the age that has an influence on the perception of e-assessment.

Gender having an influence on the perception of e-assessment is actually not surprising, as many studies show that women seem to have a lower confidence in using technology in general than men, for example Kadijevich [START_REF] Kadijevich | Gender Differences in Computer Attitude among Ninth-Grade Students[END_REF], Kahveci [START_REF] Kahveci | Students' perceptions to use technology for learning: Measurement integrity of the modified Fennema-Sherman attitudes scales[END_REF], and Yau and Cheng [START_REF] Yau | Gender Difference of Confidence in Using Technology for Learning[END_REF], whether this is justified or not. Therefore, it is reasonable to assume that the same tendency can be observed when examining the perception of eassessment.

Summary and outlook

In order to identify factors that influence students' perceptions of e-assessment, we carried out our own survey based on the findings in a previous paper [START_REF] Hillier | e-Exams with student owned devices: Student voices[END_REF]. However, we extended our survey over multiple IHEs to gain a broader view. The results are promising, in that students seem to be open-minded regarding e-assessment, which is in line with the findings in the already existing literature. However, there are open points that have to be reliably resolved in order to convince the students completely of e-assessment. Therefore, more research is needed to uncover all the open questions that exist among the students as well as to find solutions to these open questions. Further research could also tackle the question as to whether affinity to technology and the field of study have a direct influence on the perception of e-assessment. In addition, it could be further investigated if the influence of the study level is indeed significant, due either to further progress in studying or if there is a hidden correlation between age and level of study.

Figure 1 .Figure 2 .

 12 Figure 1. Distribution of study programmes.

Figure 3 . 1 Figure 3 .

 313 Figure 3. Violin plots for questions E1 and E2.

Figure 4 .

 4 Figure 4. Violin plots of questions E3, B1, C1, and C2.

Table 1 .

 1 The Survey (translated to English).

	Part	Item

Table 2 .

 2 Demographics of the participating students.

		Male	Female	Not Answered	∑
	< 18	1.2 %	0.25 %	0 %	1.5 %
	18 -25	60.3 %	16.67 %	0.5 %	77.5 %
	> 25	14 %	6.4 %	0 %	20.3 %
	Not Answered	0.46 %	0	0.25 %	0.7 %
	∑	75.96 %	23.3 %	0.74 %	100 %

Table 3 .

 3 p-values for the Fisher test.

			p-Value
		Gender	Age	Study Level
	E1	0.04536 < 0.05	0.0951 < 0.1	0.1661
	E2	0.4115	0.002503 < 0.01	0.0534 < 0.1
	E3	0.6161	0.14	0.7155
	B1	0.2079	0.211	0.1844
	C1	0.5356	0.6287	0.3458
	C2	0.5694	0.5445	0.185