Mathew Hillier
email: mathew.hillier@monash.edu

Nathaniel Lyon
email: nathaniel.lyon@monashcollege.edu.au

Student Experiences with a Bring Your Own Laptop e-Exam System in Pre-university College

Keywords: e-Exam system, assessment, student perceptions, acceptance

This study investigated students' perceptions of a bring-your-own (BYO) laptop based e-Examination system used in trials conducted at an Australian pre-university college in 2016 and 2017. The trials were conducted in two different subjects, in geography and globalisation. Data were gathered using pre-post surveys (n = 128) that comprised qualitative comments and Likert items. Students' perceptions were gathered relating to the ease of use of the e-Examination system, technical reliability, suitability of the assessment task to computerisation and the logistical aspects of the examination process. Many of the typists were taking a computerised supervised test for the first time. A divergence of opinions between those that typed and those that hand-wrote regarding students' future use intentions became more prominent following the examination event.

Introduction and background

In this study, we characterise an e-Examination (e-Exam or e-exam) as a "timed, supervised, summative assessment conducted using each candidate's own computer running a standardised operating system" [START_REF] Fluck | eExams: Strength in Diversity[END_REF]. We would add that the use of authentic software applications as part of the examination environment is an important element of our approach. As such, we distinguish our approach to computerised examinations from 'online assessments' that are limited to the test or quiz functionality of a learning content management system (Moodle, Blackboard) or specialised testing software (TCExam, QuestionMark Perception, ExamSoft) that may or may not be directly supervised by human invigilators.

This study is part of a wider project [2 -4] investigating authentic approaches to supervised high stakes assessment typically carried out in examination halls and classrooms suited to the Australian higher education context. In this respect, our paper takes a departure from other trials we have conducted in that we focus here on e-exam use in a pre-tertiary pathway college context.

In this paper, we explore the literature related to e-exams, including matters relating to student choice and acceptance of the e-exam approach. Data were collected on the students' impressions of the process as expressed through written comments and selected response items in a pre-and post-assessment surveys.

Computerised examinations have been increasingly gaining attention in the last decade. Whilst one of the first reported uses of computers for assessment was in 1965 [START_REF] Swets | Computer-Aided Instruction[END_REF], little movement away from pen-on-paper examinations has occurred in the higher education or school systems around the world. Only recently has attention shifted to modernising the examination room. Examples of efforts underway in higher education and other sectors were reviewed [START_REF] Fluck | eExam symposium: design decisions and implementation experience[END_REF][START_REF] Fluck | eExams: Strength in Diversity[END_REF]. The 'Dublin Declaration' developed by the International Federation of Information Processing Technical Committee Three for Education conference sets a future direction for computers in assessment [START_REF]Tomorrow's Learning: Involving Everyone. Learning with and about Technologies and Computing[END_REF] (pp. xvii -xviii):

"To see computers used effectively in education, it is necessary to develop fair, reliable and resilient computer-based assessment methods. Assessment methods must go far beyond imitating paper-based assessment, and prioritise the pedagogical affordances of computers over administrative convenience. The use of computers in timed, supervised assessments offers the chance to transform curricula in the light of computational thinking".

In particular, the Declaration recommends that e-Exams must be: "authentic assessment that matches modern workplace practices and many student learning experiences". ( [START_REF]Tomorrow's Learning: Involving Everyone. Learning with and about Technologies and Computing[END_REF], p. xviii)

We particularly find resonance with the idea of promoting authentic assessment [START_REF] Crisp | Towards Authentic e-Assessment Tasks[END_REF] in high stakes examinations. One way to enable such assessment is to provide a rich array of software tools of the trade to candidates in the examination room. Doing so opens up the possibility of designing complex constructed assessment tasks to be done under supervised conditions. This enables assessment designers to push into the Modification and Redefinition stages of the Substitution, Augmentation, Modification and Redefinition (SAMR) model [START_REF] Puentedura | A Matrix Model for Designing and Assessing Network-Enhanced Courses[END_REF][START_REF] Puentedura | Transformation, Technology, and Education[END_REF] or to target 'higher order thinking' of Bloom's taxonomy [START_REF] Krathwohl | A Revision of Bloom's Taxonomy: An Overview[END_REF] with respect to pedagogical efficacy. Systems such as 'Secure Exam Environment' and the work described in [START_REF] Hillier | Transforming Exams -How IT works for BYOD e-Exams[END_REF] place authentic assessment at the heart of the project. The e-Exam platform [START_REF] Hillier | Transforming Exams -How IT works for BYOD e-Exams[END_REF] used for this study uses a bring-your-own laptop approach and provides the same full operating system and application suite that includes an office suite, multimedia tools and optional discipline-specific applications (e.g. mathematics, computer aided design (CAD), chemistry, accounting). In this instance, we used the fully functional word processor as the question presentation and response environment, thus providing an authentic tool typically used to produce essays and reports.

The perceptions and attitudes of users with respect to ease of use and usefulness (being fit for purpose) have been found to be important factors in people accepting new computer technology [START_REF] Davis | Perceived usefulness, perceived ease of use, and user acceptance of information technology[END_REF][START_REF] Farzin | Attitude of Students Towards E-Examination System: an Application of E-Learning[END_REF], but not the only factors at play [START_REF] Lunceford | Reconsidering Technology Adoption and Resistance: Observations of a Semi-Luddite[END_REF]. This particularly applies to the students as the people most directly impacted by e-exam systems, although they often do not have a strong voice when it comes to selecting software deployed in education institutions. Therefore, it is important to ensure their views are heard if we desire smooth acceptance and operation of a high-stakes e-exam system. A survey of students [START_REF] Dermo | E-assessment and the student learning experience: A survey of student perceptions of e-assessment[END_REF] at the University of Bradford in the United Kingdom (UK), following their use of QuestionMark Perception, included a range of topics. Other studies include the use of Examsoft in pharmacy courses in Canada [START_REF] Bussières | Use of Moodle, ExamSoft, and Twitter in a First-Year Pharmacy Course[END_REF] and an institution-wide survey [START_REF] Hillier | The Very Idea of e-Exams: Student (Pre)conceptions[END_REF] capturing students' hopes and fears prior to the trialling of e-Exams at The University of Queensland in Australia. e-Exam trials followed the latter study, exploring the students' experiences of the process [START_REF] Hillier | e-Exams with student owned devices: Student voices[END_REF]. These studies on student perceptions of the e-exam process informed construction of survey tools in this study.

Another characteristic is how each system architect treats the idea of technology reliability. Where an e-Exam solution has a heavy reliance on a network during the examination, the risk of a 'single-point of failure' impacting on a whole cohort of students is increased and so the need to ensure extra redundancy measures is heightened. One study [START_REF] Hillier | The Very Idea of e-Exams: Student (Pre)conceptions[END_REF] reported that the fear of technology failure is a major barrier to the adoption and intention to use e-Exams by students. A recently publicised case of failure during a national high stakes Medical board e-exam event in Australia [START_REF] Aubusson | Burnt-out doctors deeply distressed by botched high-stakes exam[END_REF][START_REF] Aubusson | Company behind botched medical exam has track record of failure[END_REF] highlights the critical need to ensure a robust system and to avoid 'single point of failure' designs. Earlier online assessment systems tended to stop working the moment the network dropped out. Advances in web technologies mean that some systems may handle or mitigate network outages of a short duration (e.g. auto-save for Moodle quiz) but extended outages will result in an unscheduled end to the session. Only a small number of e-exam systems are able to continue to operate and successfully complete the e-exam session without a network connection. This includes the commercial product 'Examsoft ' [21] and the e-Exam platform [START_REF] Teaa | Transforming exams across Australia[END_REF][START_REF] Hillier | Transforming Exams -How IT works for BYOD e-Exams[END_REF] used in this study. Avoiding system-wide failures means that any technical issues that do occur are likely to be isolated to a single student. Therefore, an issue can be managed according to existing examination protocols with respect to individual interruptions, breaks and extra time.

This review of prior work has outlined several areas of concern that will serve as a focus for our evaluation in this study. These are summarised in Table 1.

Table 1: Areas for investigation

Area

Example research questions

Intention and attitude

Were candidates in favour of the e-Exam system? Would they recommend it to others or use it again? Did they have any concerns about undertaking an e-exam? If they typed their examination, was their attitude changed or any concerns lessened following the event? Ease of use Were the students able to use the system with relative ease? Did they have any issues related to the e-exam processes? Technical reliability Did any technical issues or interruptions arise? If so, did such issues interfere with the examination or result in lost work?

Study context

In Australia, the lead author is conducting a nationwide project investigating the scalable provision of authentic assessment in the examination room using BYO laptops [START_REF] Teaa | Transforming exams across Australia[END_REF][START_REF] Hillier | Transforming Exams -How IT works for BYOD e-Exams[END_REF]. The study reported in this paper investigates if prior work carried out in the higher education sector would work successfully in the pre-university context. The study was undertaken at Monash College, Australia within the 'Foundation Year' [22] programme. This programme is at the equivalent level as an Australian year 12 high school leaving certificate or the International Baccalaureate. The study was run in conjunction with the second author who is a unit coordinator and teacher in the two units in which trials were conducted. The trials were carried out using in-class supervised written assessments. These took the form of a couple of mini-cases that included photographs, charts and data tables, each with one or several questions requiring a short text or essay-style response.

Method and approach

This study examined two live trials of the e-Exam system and approach in two separate units at the College, involving 128 students. The units selected were geography (Geo) in semester 1, 2016 and globalisation (Glo) in semester 2, 2017. The process used to run each trial is represented in Table 2. Preview the examination process and practice following provided instructions for starting up their laptop from an e-Exam universal serial bus (USB) stick. Students complete the practice questions. Data collected about hardware compatibility via hardware logging, observation of use and student impressions via pre-examination survey. Following the session, data analysis of the surveys was carried out to detect any concerns.

Real examination for both typists and handwriters

Individual desks were set up with a paper copy of the 'e-Exam quick start guide' and post-examination survey. A power socket was provided for each typist. Hand-writers were given a paper copy of the examination questions and response booklets. These were available to typists upon request. Both typists and hand-writers sat in the same room. 1. Students entered the room and were seated at a suitable desk. 2. Typists were given an e-Exam USB stick containing the questions. 3. Students started their computer with a USB stick progressing to the e-Exam desktop. A desktop background image provided a visual check that all had booted from the correct USB. 4. Invigilator announced the start of the examination. Students entered their student ID and name into the starter screen. The system then opened the examination document. Auto-save occurred every two minutes. 5. Examination ended: saved work one last time and shut down the computer. 6. Students returned the USB sticks containing their responses. 7. Students completed the post-examination survey before leaving the room. 4. Grading In the following week, the teacher did the grading. Students were given grades and feedback comments. Surveys results were analysed.

The formative ungraded practice session was run in-class time with all students participating. Students were free to choose typing or handwriting for the real examination.

Selected response survey questions as shown in Tables 3 to 5 were analysed using SPSS v24 using an alpha level of .05. Likert data pertaining to students' opinions were treated as non-parametric [START_REF] Jamieson | Likert scales: how to (ab)use them[END_REF]. Another study [START_REF] Dermo | E-assessment and the student learning experience: A survey of student perceptions of e-assessment[END_REF] did the same when analysing students' perceptions of their experience with an e-Assessment system. Mann and Whitney's U test [START_REF] Mann | On a Test of Whether One of Two Random Variables is Stochastically Larger than the Other[END_REF] was used to test the variance between groups (typists versus handwriters) on Likert items. When comparing paired pre-post Likert items, a Wilcoxon Signed Ranks Test [START_REF] Wilcoxon | Individual comparisons by ranking methods[END_REF] was used with the requirement of a normal distribution of differences met. Chi-squared was also used to test if experiencing a technical issue impacted in the decision to type the examination.

It is important to note that participants were not randomly assigned to the typing or hand-writing groups so results are only descriptive of this group. As per [START_REF] Dermo | E-assessment and the student learning experience: A survey of student perceptions of e-assessment[END_REF], we take the stance that statistical tests serve as a tool to summarise the body of students' opinions rather than to be representative of an objective truth.

Findings

The trials involved 128 pre-tertiary students; 65% were female and 35% were male. We examined the students' opinions regarding their first encounter with the e-Exam system in terms of differences between those that went on to type the examination and those that handwrote the examination using a Mann-Whitney U test. Table 3 displays the results from Likert items (strongly agree=5, neutral=3 and strongly disagree=1) collected on the pre-examination survey (done at the practice session). The strongest difference was for "I would like to use a computer for exams in the future" (U=842.5, p=<.001). Means and standard deviations are provided in the tables for clarity. Following the examination, typists (52%) were asked to reflect on the e-Exam system itself with regard to suitability, usability and reliability (see Table 4 and Figure 1). The majority of items received positive agreement, most with mean agreement ratings of 4 or above out of 5. The sentiment within the group was relatively uniform, as evidenced by the small standard deviations (Table 4) and boxplots in Figure 1. Typists were asked "Did you experience any technical difficulties during this exam?" Responses yes (n=17, 24%) and no (n=53, 76%) were gathered via a comment box and a list covering usability, technology and logistics. It should be noted that all those who typed successfully completed and submitted their work. A comparison of problems encountered in the pre-and post-sessions is shown in Figure 2. A Chi-squared test indicated that there was no statistically significant relationship between encountering a problem in the practice session and electing to type or handwrite the examination (χ 2 (1) = 0.003, p = 0.956). This could indicate that the practice session did its job in preventing serious problems from reaching the examination room, or that problems were considered to be minor by those that encountered them. A comparison between typists' and hand-writers' intentions to use a computer for future examinations following the examination event (see Table 5) showed a significant Mann-Whitney U test result (U=160.5, p=<.001). Finally, we examined if students' declared future use intentions may have changed between pre-and post-examination surveys for typists and hand-writers using the question "I would like to use a computer for examinations in the future". Those that typed the examination were in slightly stronger agreement following the examination (n=61, M=4.2 SD=0.7) than prior (n=55, M=4.0, SD=1.0). However, the difference was not significant when tested with a Wilcoxon Signed-Ranks test, Z=-1.763, p=0.078. Both pre-and post-median agreement was 4. Those that handwrote the examination became more negative following the examination to a significant extent (Z=-3.757, p=>.001), with the median agreement 3 prior and 2 following the examination. For clarity, mean agreement for hand-writers pre-examination was M=3.0 (SD=1.2, n=56) and post-examination M=2.1 (SD=1.0, n=53).

Discussion

In the practice session, most students were able to successfully undertake the steps required for doing the e-exam using their laptop, although some did require assistance. This included starting up their laptop from the USB stick and using the software (see Table 3). The Chi-squared result shows that encountering a problem in the practice session did not impact the decision to type or handwrite. The practice session appeared to resolve most serious problems (see Figure 2) before they reached the examination itself as seen by the reduced number of issues reported between the pre-and postsessions. Most problems that remained related to user familiarity with the software or process (i.e. forgetting the boot key, not realising that short-cut keys behaved like 'Windows' rather than Apple OSX) or minor hardware incompatibility (i.e. their laptop touchpad being too sensitive; although a wired mouse would have solved the issue). However, the persistence of these issues indicates that further opportunities for practice and increased awareness of the option to bring a wired mouse were needed.

Those that went on to type the examination, not surprisingly, expressed stronger agreement in being able to undertake the practical steps of the e-exam process but the differences in opinions with hand-writers were not statistically significant. However, the items reflective of confidence "I feel confident I will be able to do these steps in a real examination", "I now feel relaxed about using the e-Exam system for my examination" and future intentions "I would like to use a computer for examinations in the future" did show a significant difference between the groups. This gap between their perceptions of process and their expressed levels of confidence or intentions could be indicative that other matters beyond those surveyed played a role in students' decision making. Additional findings related to students' preferences with respect to writing styles, behaviours and proficiency, where a link to their selected text production mode was found to be stronger, are reported separately [START_REF] Hillier | Writing e-Exams in pre-university college[END_REF].

Following the examination event, the results in Table 4 and Figure 1 showed that a large majority of students who typed the examination were satisfied that the assessment task was suited to computerisation, they appreciated being able to use their own computer, and that the system was easy to use, reliable and secure against cheating. Most also agreed that they would recommend the e-exam system to others. This was consistent with prior work in the university sector [START_REF] Hillier | e-Exams with student owned devices: Student voices[END_REF].

A moderate, but not statistically significant, divergence of opinion between handwriters and typists emerged across most items in the pre-examination survey. Overall, it would appear that people tended to reaffirm their choice to type or handwrite in terms of their future intentions following the examination. This divergence can be seen when looking at future intentions stated prior to the examination with difference in agreement of 1 widening to 2.1 in the post-examination survey. It would appear that students' opinions 'hardened' once the real examination was over, in that typists became more positive about their future intentions to type an examination and hand-writers more negative.

Finally, the decision to allow students to self-select typing or handwriting served to lessen the stress for students, but it also limited the degree of task sophistication that was possible (i.e. keeping to the lower levels of SAMR). However, this can only ever be a temporary state of affairs if we want to progress up the SAMR ladder to include re-designed, higher order assessment tasks that assume sophisticated tools will be available. To take advantage of the affordances of modern software, means all students must ultimately use a computer in the examination. Our work on e-Exams is also about providing a strategy [START_REF] Hillier | Transforming Exams -How IT works for BYOD e-Exams[END_REF] for moving from paper-equivalent e-exams to sophisticated post-paper e-exams where all must type. This phased strategy, along with associated support, will be important in helping staff and students to make this transition.

Conclusion

We have successfully completed two trials of e-exams centred on the use of a fully featured word processor in two different units within a pre-university context. From this point of view, we broadly achieved what we set out to do in that the e-Exam technology and BYO laptop centric processes were shown to have worked in this context. We have also seen that most students were satisfied with the approach to doing e-Exams within a classroom setting. The strength of opinions regarding the process and technology between those that typed the examination and those that elected to handwrite were not significantly different, although there was a general trend towards typists holding more positive opinions. Their levels of confidence did differ significantly and this likely played a role in their choices. The results are not at all surprising given the self-selecting nature of the groups who went on to type the examination. However, it does reinforce the need to ensure adequate support is available to students who are not all equally prepared for the computerisation of high-stakes examinations. Future work will involve comparisons with similarly run examinations in the university system and within different discipline contexts. The next phase will be to trial e-Exams using post-paper, higher-order tasks where all members of the class will type. Further technical work on the e-Exam system is progressing that will see integration with the Moodle quiz tool alongside the ability to use authentic software tools in a manner that is robust against network outages [START_REF] Hillier | Robust networked e-Exams with Moodle[END_REF].

Table 3 :

 3 Pre-examination survey responses by text production mode Question Type Hand write MW Pre survey (Practice run) n M SD n M SD diff U p Written instructions were easy to follow 55 4.1 0.7 55 4.0 1.0 0.2 1466 0.759 It was easy to learn the necessary technical steps 55 4.2 0.7 56 3.9 1.1 0.3 1307 0.128 It was easy to start my computer using the e-Exam USB stick 55 4.0 1.0 56 3.7 1.2 0.3 1333 0.194 I feel confident I will be able to do these steps in a real examination 55 3.9 1.0 56 3.3 1.2 0.6 1093 0.006 The software within the e-Exam System was easy to use 54 4.0 0.8 56 3.8 1.0 0.2 1439.5 0.642 I now feel relaxed about using the e-Exam system for my examination 54 3.8 1.0 56 3.3 1.1 0.5 1171.5 0.034 I would like to use a computer for examinations in the future 55 4.0 1.0 56 3.0 1.2 1.0 842.5 <.001

Table 4 : 9 Figure 1 :

 491 Figure 1: Opinions of the e-exam system

Figure 2

 2 Figure 2: Reported issues

Table 2 :

 2 e-Exam trial process

	Stage	Activities
	1. Call for	Students indicate interest in either typing or handwriting their
	interest	examination and complete research study consent forms.
	2. Practice
	session
	done two
	weeks prior
	to
	examination

Table 5 :

 5 Post-examination survey future intention to use

	Question	Type Hand write	MW
	Post survey	n M SD n M SD diff U	p
	I would like to use a computer for	61 4.2 0.7 53 2.1 1.0 2.1 160.5 <.001
	examinations in the future

Acknowledgments. The authors would like to thank the Australian Government Department of Education and Training for financial support and the students at Monash College for being willing to 'give it a go'.


Andreas Bollin

Peter Micheuz

Computational Thinking on the Way to a Cultural Technique A Debate on Lords and Servants

Keywords: Computational thinking, computer science, life-long-learning, engineering, curriculum development

Based on a thorough literature review and on personal expertise in different areas of computer science (education) fields, we reflect and debate on computational thinking from different perspectives. One is that of an Austrian teacher who is confronted with a curriculum for a new subject called 'Basic Digital Education', with computational thinking as an explicit part of it. The other view is that from a reflective software engineer with a holistic perspective on computational thinking and concrete ideas about its limitations. The debate concludes with an agreement on computational thinking as a cultural technique and a mutual approach to a refined working definition.

Introduction

In an interview with the German "Süddeutsche Zeitung" at the end of January 2018, Armin Grunwald, head of the Office of Technology Assessment at the German Bundestag, said that if "we just have to work to run after the technology, then something is wrong. Hegel has already put this in a nutshell, with the relationship of master and servant… The more the Lord relies on his servant, the more dependent he becomes on him" [START_REF] Bauchmüller | SZ Online[END_REF]. Indeed, for decades, educators (and politicians) have had to deal with the question of what to teach and how to be able to produce (or perhaps guarantee) politically mature and technologically up-to-date people. But, what does this mean in the context of current developments: robots, drones, artificial intelligence, smart devices and, not to forget, the Internet of things and smart homes? Is digital education following a scattergun approach?

With the publication of a CACM viewpoint article about computational thinking (CT) in 2006, Jeanette Wing popularised the idea of a new fundamental skill used by everyone in the world by the middle of the 21 st century [START_REF] Wing | Computational Thinking[END_REF]. She defined computational thinking as "the thought processes involved in formulating a problem and expressing its solution(s) in such a way that a computer -human or machine -can effectively carry out". In the Gödel Lecture at Vienna University of Technology on June 9 2016, she also shared many examples of where to find computational thinking aspects in different disciplines, be it economics, law, healthcare or geosciences. So, together with introducing computer science to our classes, do we have a silver-bullet for dealing with today's challenges?

In her 2006 seminal paper, Wing did not primarily think of computational thinking in primary and secondary education [START_REF] Wing | Computational Thinking[END_REF]. It was not foreseeable which worldwide avalanche has been set off by her, especially among educationalists and teachers. Maybe it goes too far to refer to it as a hype. But if not, it could be a worthwhile endeavour to show that Gartner's Hype Cycle can even be applied to this phenomenon, with the peak of inflated expectations being apparently behind us. Currently, we find ourselves at the slope of enlightenment, in the form of reasonable and viable definitions of that all-inall still fuzzy term. Reviews on existing literature strengthen some core concepts of CT: logical and algorithmic thinking, decomposition, generalisation and pattern recognition, modelling and abstraction [START_REF] Selby | Computational Thinking: the developing definition[END_REF].

CT is seen as a fundamental set of mental skills used by everybody, as fundamental as reading, writing and arithmetic [START_REF] Wing | Computational Thinking benefits Society[END_REF]. Martin describes CT simply in a few words: "It is about connecting computing to the world" [START_REF] Martin | Rethinking Computational Thinking[END_REF]. Moreover, it seems to be widely accepted that coding is an indispensable part of CT [START_REF] Prottsman | Computational Thinking and Coding for Every Student: The Teacher's Getting-Started Guide[END_REF]. It is seen as an aid to learning software development [START_REF] Beecher | Computational Thinking. A beginner's guide to problem-solving and programming[END_REF], and is thus coupled with software engineering [START_REF] Bollin | Teaching Software Engineering in Schools -On the right time to introduce Software Engineering Concepts[END_REF][START_REF] Bollin | Software Engineering in Primary and Secondary Schools -Informatics Education is more than Programming[END_REF].

The list of publications in the context of CT is amazing. But, even more surprising is the fact that the debate about "How much of computer science and computational thinking should be taught?" is camouflaged by the support from technology enthusiasts, industry, and politicians. So, though there are numerous different curricula and definitions of computational thinking around [3 -7], there still is no common understanding about how far we need to go. This paper, therefore, tries to answer the question of what computational thinking (also at primary and secondary schools) is, and elaborating on it closer, what it is not. We try to approach the border between computational thinking (as a set of skills that is needed due to contemporary demands, addressing the characteristics of a new cultural technique) and the skills of an engineering education that are needed by professionals.

The idea behind this paper was born during a trip of the two authors from Klagenfurt to Vienna, where we were trying to define computational thinking in the Austrian context. Both authors have years of teaching experience in computer science, but the first author has a strong engineering background, whereas the second author is involved in political discussions, adapting the school system in Austria for many years. It seems natural to us to approach this topic in the form of a debate, where statements are presented and redefined along the discussion, finally coming up with a reliable definition of the border between CT and not-CT.

A debate on CT in the (Austrian) educational system

"Basic Digital Education" is the name for a new subject which will be introduced in all Austrian lower secondary schools beginning in the school year 2018-2019. There is one curriculum covering four years of lower secondary education (age groups from 10-14 years) and encompassing eight main topics:

• Social aspects of media change and digitisation Obviously, these topics stand for a broad curriculum, which encompasses digital competence, media competence, and political competence as well. Digital competence in particular is expected to empower pupils, based on a comprehensive overview of digital tools (hardware and software), for coping with certain scenarios in educational, vocational and private contexts in a reflective manner.

At first sight, informatics (computer science) does not play a prevalent and visible role. Media pedagogy and digital literacy are apparently better represented than core informatics. At a second glance, computer science is represented explicitly as CT. This term has not been translated into the German synonym "Informatisches Denken" and appears in the curriculum as the (global) driving force for implementing elements of core informatics into a seemingly overcrowded curriculum.

All topics of the curriculum are divided into further subtopics and detailed competence descriptions and learning goals. The main topic CT is split into a basic and advanced level (see Table 1 for more details). For the first time in the history of computing education in Austria, all lower secondary schools -"some" teachers and all pupils -are exposed to a binding curriculum wherein computing, algorithms and programming play a more or less clearly defined and specified role. Digital education in general and CT in particular are no longer optional for a special cohort of pupils, but obligatory for all. It would be too much to go into the details of the challenging organisational issues required to implement the curriculum. All there is to say about it is that within the framework of school autonomy, schools can change the number of hours within a certain range. They have to decide autonomously to introduce the curriculum for 'Basic Education' as an independent subject (2 to 4 hours per week, which means an assumed 64 to 128 hours of lessons) within 4 years, in a completely integrative way in other subjects (64 to 128 hours) or all hybrid forms between these extremes. But independent from the justified question of how to cover all topics, with, all-in-all, over one hundred learning goals in a very limited time, the main questions are: to what extent is the Austrian definition of CT in line with international views among experts including scientists and teachers, and does it already answer the question of what CT is definitely not? The explicit topic CT in the Austrian curriculum could lead to the assumption that it has nothing to do with the other topics, like standard applications, technical problem-solving or security or even media design.

Proper or -mproper

The opening statement of A. Bollin: The article of Bocconi et al. [START_REF] Bocconi | Developing Computational Thinking: Approaches and Orientations in K-12 Education[END_REF] about computational thinking approaches and orientations in K-12 education brings us exactly to the point: there are still different views on what computational thinking is. Some of them include programming; some of them do not. For motivational reasons, curricula sooner or later will introduce coding or programming in their lecture units. According to the new Austrian curriculum, pupils should be able to produce simple programs or webapplications and should know different programming languages. This seems to be in- [START_REF] Missiroli | Cooperative Thinking, or: Computational Thinking meets Agile[END_REF] and following Computing at School [START_REF] Beck | Extreme programming explained[END_REF] and Beck [START_REF] Csizmadia | Computational thinking: A guide for teachers[END_REF]) line with the current trend, but when it is not introduced for good reasons and correctly, this is putting the cart before the horse.

In the vision of Wing, everyone should be able to use well-established techniques that have been applied by engineers for a long time already. It is about formulating problems in such a way that, maybe with the help of others (or even machines), problems can be solved easier. Thus, it is not surprising that the problem-solving activity includes logical reasoning, algorithmic thinking, abstraction, decomposition, generalisation, pattern detection and languages (notations) for communication and representing information. But, this is quite individualistic, and nowadays by far not enough to solve larger (and more complex) problems. Missiroli et al. [START_REF] Missiroli | Cooperative Thinking, or: Computational Thinking meets Agile[END_REF] thus suggest combining computational thinking skills and team-based skills, as needed by software developers, when developing software in an agile manner. Figure 1 summarises their concept, combining problem solving and social skills to a new literacy they call "cooperative thinking".

So, when introducing programming, it should not be done to just to illustrate how algorithms can be used and executed. We should not be dumb Lords. One should be honest and state that nowadays it is about solving real-world problems. But, this then includes more than just being able to write some lines of (computer-readable) text. It is also about including some software engineering skills.

Step-by-step

The follow-up statement of P. Micheuz: "The secret of getting ahead is getting started. The secret of getting started is breaking your complex overwhelming tasks into small manageable tasks and starting on the first one." This little-known quote from poet Mark Twain on problem-solving can be seen as a remarkable historical precursor, long before CT began to rack the brains of thousands of educational experts in the field. Every software engineer working on solving so-called real-world problems in teams, and with the aid of digital tools, has learned "simply to go" by practicing and internalising basic concepts and, to a reasonable extent, also some (especially among many educationalists), disreputable rote learning. There is another quote, "He who wants to build high towers must dwell with the fundament for a long time" from the Austrian composer Anton Bruckner. It supports the truism that CT as a cultural technique needs an early beginning, a coherent and sustainable construction of skills and competences in the form of a spiral curriculum.

Looking at the ambitious and overloaded curriculum, with CT as a comparatively small part, and even under the assumption that motivated and CT-proven teachers follow the intended curriculum, it is rather unlikely that all pupils can meet all goals of the whole curriculum in 64 to 128 hours of lessons within four years.

It is self-evident that CT for primary, lower/upper secondary and tertiary level (must) have different characteristic forms. I can live with the fact that advanced aspects of software engineering and bigger projects should play a role, at the earliest, at upper and tertiary level, but for primary and lower secondary level that would go too far. Nonetheless, cooperative thinking, or better, cooperative acting, can be harnessed as a valuable general teaching method, especially in programming and CT-related lessons. I doubt that software engineering in its full definition is an adequate term for lower age groups. That would overstrain teachers who are already struggling with CT, but willing to undergo professional development in that field. But, often things are not as bad as they seem. From this perspective, the fact that CT lacks a precise definition must be considered predominantly an advantage, as it is scalable and adaptable for various age groups and even interdisciplinary implementations.

The explicit learning goal of producing (simple) programs confronts Austrian teachers and pupils with a fait accompli. CT in the new curriculum covers more than mere algorithmic thinking, but less than dealing with (complex) real-world problems.

Past and future

The follow-up statement of A. Bollin: Interestingly, history is repeating. The introduction of spreadsheets decades ago led a situation where every user was using sheets without noticing that he or she was (and is) in fact programming [START_REF] Mittermeir | Protecting Spreadsheets against Fraud. Proceedings of the European Spreadsheet Risks International Group[END_REF] (and not following quality standards a software engineer would naturally follow). Debugging aids for spreadsheets are getting better nowadays, but still a lot of erroneous spreadsheets are around, forming the basis for (private as well as industrial) disastrous decisions. We should not educate pupils in a way where oversimplifications potentially lead to a misuse or misunderstanding of reality.

The "proper or improper" claim does not mean that even more skills are to be packed into the tight schedule in schools under the umbrella of computational thinking, but it is a hint toward a problem that we are running into. In one work, Hermans and Aivaloglou [START_REF] Hermans | Do code smells hamper novice programming: A controlled experiment on Scratch Programs[END_REF] show that using block-oriented languages (which are quite often chosen for novice programmers) dramatically hampers learning programming later at universities. Without taking care of software engineering practices from the beginning (as examined in their paper), a lot of effort, time and resources are needed to produce the engineers that the industry is longing for. Now, not everybody will (and should) strive for a career as a (software) engineer. But, this is not a reason for showing programming in a way nobody ever would and should program. The problem focus and the context are missing, and in our classrooms, we continue having a lot of bored pupils.

This viewpoint is partially also supported by neuro-didactic findings: people have enormous difficulty learning when either parts or wholes are neglected. Hence, the learning brain needs the whole and the details; it requires both a big picture and paying attention to the individual parts [START_REF] Caine | Understanding a brain-based approach to learning and teaching[END_REF]. In our case, the real world is needed, and the individual parts could be programming tasks or the meaningful use of technology.

To me, programming already is part of the engineering discipline, and CT covers parts of the skills an engineer needs to successfully solve problems and to create something new. Sure, for didactic reasons, one might start to introduce small programs to show the application of CT techniques in a bigger context. It is also clear that one has to start step-by-step. But, as another comparison with a cultural technique shows, when we start learning to read, we do not stop after recognising the letters. We continue with combining the letters together and with learning to recognise syllables, words, etc., until we reach some level of proficiency.

To summarise, when defining programming as not being part of CT, then we definitely should add computer science (and software techniques) to our curricula in order to show the whole picture. When we say that programming is part of CT, then we should not do it in an inappropriate (context-neglecting) way and need to add more hours to our syllabi. It also means investing more resources and training our teachers (including all the lecturers at universities) accordingly. This is the only way that pupils (and parents and teachers) will responsibly know why to decide either for or against a technical study or technique-related working place later on. It is also the only way to keep technological change being the servant and not allow it to take over.

In the right place at the right time with a sense of proportion -P. Micheuz

The final statement of P. Micheuz: Digital education in its full complexity will remain a big challenge in traditional formal educational settings. So will the recent decision from an expert group to embed CT explicitly into the curriculum. It does not solve the problem of age-appropriateness and does not even guarantee dedicated lessons for computing. It transfers the responsibility for its implementation -to what extent and at which age level -to schools and teachers. In contrast to traditional subjects such as language education and mathematics, the drivers of the main cultural techniques of reading, writing and arithmetic, currently CT cannot rely on sequenced and coherent age appropriate lessons. Accordingly, there is legitimate concern that within 4 years of lower secondary education, CT will not be taught properly. It may be assumed that in the initial phase of executing the curriculum of 'Basic Digital Education' in some schools, CT will play little role.

Regarding the introduction of programming with a block-based approach (Scratch or similar development environments), I am quite optimistic. The question remains when to switch to the first steps of textual coding. As for today, the Austrian curriculum for 'Basic Digital Education' has no answer for that.

It cannot be expected that (m)any teachers at this school level see the whole picture of software development. Basic CT education with first steps in problem-solving, algorithmic thinking and first programming experiences on a small scale should be also feasible without having deep experiences in software engineering.

But even these first steps cannot be taken for granted. CT education for pupils needs CT-educated teachers and professional development in that field on a large scale. In the next years, nationwide measures in the form of pre-service and in-service training in various formats will need to be taken.

A working definition of computational thinking

In the previous section, we tried to take two positions, one from the viewpoint of a software engineer, and one from the viewpoint of teachers who need to make the next generations fit for exciting technological changes and current threats. Now, we try to converge and to find a working definition in the context of the Austrian situation and with regard to CT as a cultural technique.

About cultural techniques

When reflecting on cultural techniques, we think primarily of the cognitively most fundamental cultural techniques of reading, writing and arithmetic, and of its lengthy and laborious acquisition in dedicated school subjects. But, in our increasingly digitally penetrated culture, there are demands for extended skills and competences as widely elaborated in the Digital Competence Framework for Citizens [START_REF]Digital Competence Framework for Citizens[END_REF].

If we accept CT as a new cultural technique, it might be helpful to look closer at what a cultural technique really is. Cultural techniques are a set of skills, concepts and competences that help human beings "function properly" in a given culture. They help in dealing with tasks and solving problems in different situations of life, like making a fire, using a calendar, or being able to communicate in social networks.

As cultural techniques are solution concepts for tasks and problems of human beings, we have to be clear about current human needs (cf. Maslow's hierarchy). In the context of new technological demands and secondary education, this list of needs encompasses being able to:

• Communicate with others using state-of-the-art communication technology.

• Search for, assess and work with available information.

• Solve tasks in a sustainable manner with the help of new state-of-the-art technology. • Protect him-or herself against fraud. Apart from these needs, which are well covered also in the new Austrian curriculum for 'Basic Digital Education' (see Section 2), one also needs to know about the limits of the solution concepts and being able to protect oneself from a misuse, so the list has to be extended by:

• Knowing what computer science and software engineering is about.

• Being aware of potential limitations and side-effects. Last, but not least, computational thinking definitely does not include the cultural technique of typewriting or information technology (IT)-literacy such as using office software and digital devices at a cursory level. And, it is rather unthinkable that there are computational thinkers who are not fluent in harnessing computers, but it is quite possible that fluent computer users are not yet educated computational thinkers.

With all these reflections in mind, one can give a quite crisp definition of what computational thinking in secondary education is and what it is not.

A working definition

Computational thinking is a cultural technique consisting of a set of skills needed to complete a task in a responsible, sustainable manner including problem-solving, evolutionary and reflection steps. These steps encompass logical reasoning, algorithmic thinking, abstraction, generalisation, decomposition, design/solution patterns, evaluation techniques, and as computers might be involved in the solution process, different representation forms. It also includes knowing about related disciplines like computer science and software engineering. As such, it should be thought about to its fullest extent, but in an age-appropriate manner, at the secondary level in Austria.

Computational thinking is not about being able to work like a software engineer or computer scientist. It is not necessarily about finalising (software) products in a correct, efficient and cost-oriented manner. It is also not about proving and searching for algorithmic properties or creating new physical devices. But, it is knowing about the limits of one's own solution ideas. Now, as our debate was also about programming and coding, it is up to the educator to what extent to include programming languages (in graphical or textual form) to motivate for a technique or skill. However, it is then his or her responsibility to make the difference to software engineering clear. The implications for a teacher (and for teacher education) are obvious: he or she needs to know more about computer science and software engineering, as at least a portion of teaching CT is about teaching the differences/boundaries to neighbouring disciplines.

Summary and outlook

In this paper, we tried to further the approach to defining computational thinking, reflecting on discussion among scientific and educational experts in the educational field of computing. It stresses the fact that CT is at the border of engineering disciplines, and, when coding is involved, it is also close to the border of software engineering.

More than a decade after the seminal work of Jeanette Wing [START_REF] Wing | Computational Thinking[END_REF], the wave of its public perception reached Austria. Since computational thinking is an explicit part of the new curriculum 'Basic Digital Education', it will be a starting point of many discussions and debates.

The debate in this paper results in a working definition from the perspective of software engineering and CT as a cultural technique, having in mind the limits and challenges of school education in general and the introduction of a new subject in particular.

We agree that CT in the way we see it must be considered an important cultural technique in the 21 st century. But, we are realistic enough to know that CT as imagined in the heads of many educational experts, including our abstract working definition above, still has a long and difficult way to go from the conception stage to its implementation.

There is hope that its future will not be that of the term and subject of informatics in Austrian lower secondary schools where, according to a very broad interpretation of its definition, the subject informatics in Austrian schools created its own reality. Maybe a reality with some CT-related parts were included, but from a vast majority of teachers they were not realised as CT defined above. Only when carefully knowing the borders, were we able to deal with current and future human needs. And thus, it is more likely to keep the role of a Lord not being dependent on his or her (technical) servants.

Figure 1 .

 1 Figure 1. Cooperative Thinking as a combination of CT and Agile Development breakdown (according to Missiroli et al.[START_REF] Missiroli | Cooperative Thinking, or: Computational Thinking meets Agile[END_REF] and following Computing at School[START_REF] Beck | Extreme programming explained[END_REF] and Beck[START_REF] Csizmadia | Computational thinking: A guide for teachers[END_REF])

Table 1 .

 1 CT in the Austrian Curriculum for Basic Digital Education[START_REF]Digital Basic Education[END_REF]

	Computational	Basic Level	Advanced Level 1	Advanced Level 2
	Thinking	(2 hours per week)	(+ 1 hour)	(+ 1 hour)
		Pupils	Pupils	Pupils
		-name and describe eve-	-discover similarities	can evaluate intuitive
		ryday processes	and rules (patterns)	user interfaces and its
		-use, build and reflect	within instructions (al-	underlying processes
		codes (e.g. secret writ-	gorithms)
	Working with	ing, QR-Code)	-discover the im-
	Algorithms	-reproduce distinct in-	portance of algorithms
		structions (algorithms)	in automatic digital
		and carry them out	processes (e.g. auto-
		-formulate distinct in-	mated proposal of po-
		structions verbally and in	tentially interesting in-
		written form	formation)
		Pupils	Pupils	Pupils
		-produce simple pro-	-master basic pro-	reflect the boundaries
		grams or web applica-	gramming structures	and options of simula-
	Creative Use of	tions with appropriate	(decision, loops, pro-	tions
	Programming	tools to solve a problem	cedures)
	Languages	or to complete tasks
		-know different program-
		ming languages and
		production processes


Elisa Reçi
email: elisa.reci@aau.at

Andreas Bollin
email: andreas.bollin@aau.at

A Teaching Process Oriented Model for Quality Assurance in Education -Usability and Acceptability

Keywords: CMMI, teaching quality, maturity model, higher education 1 Introduction

The lack of standards to objectively assess the quality of teaching opened a new path of research. Teaching involves a lot of different tasks and activities that should be explored, so, consequently, when talking about quality of teaching, it makes sense to look at teaching as a process and to assess its maturity. This contribution briefly looks at existing approaches, and introduces the idea of a teaching maturity model (TeaM) for school and university teachers. Such a framework, even though it proves helpful from a measurement perspective, might not be acceptable by teachers, so this paper presents the results of a study for testing the TeaM model in respect to its usability and acceptability with informatics lecturers at the Alpen-Adria-Universität Klagenfurt. The results show the interest of our teachers in the model, but also some of the impediments that have to be dealt with when applying the model on a larger scale.

On the other hand, the introduction of the model in the educational domain raises the question of how to integrate it in one's daily (teaching) life. Another issue is how to integrate the TeaM in educational institutions so that teachers can use it for assessing the quality of teaching in their lectures. This requires testing of the model and improving it based on feedback that we get, and, with it, also to look at its usability and acceptability. The objective of this paper is to describe our results in checking for the applicability of the TeaM.

Within the scope of the paper, we collected opinions of informatics lecturers at the Alpen-Adria-Universität Klagenfurt when using the model, and the paper reports on the most important findings.

The rest of the paper is organised as follows: section 2 describes related work by giving an overview of how models like CMMI-Services and others are related to the educational system in respect to their usability and acceptability. A detailed description of how the TeaM is tested and the feedback from the lecturers is presented in section 3. In section 4, the results of the study in respect to our model are discussed. Future work and the findings are described in section 5.

Background

This section gives a short description of related models. It introduces briefly how CMMI is structured and discusses, for related models, their applicability and usability in practice.

Related work

Traditional forms for addressing quality of teaching, such as student evaluations, feedback, peer evaluation and inspections are seen as quite subjective. This opened up a path for research for assessment models that rely on standards. In this, a lot of authors address the quality of teaching by mainly focusing either on teachers (preparation, communication, engagement), or pupils/students, or course content or the environment. Taking a closer look at existing work, these models can be divided into several groups.

There are models that, to address the quality of teaching, focus only on teachers. The AQRT model addresses quality of teaching by assessing teacher teaching practices [START_REF] Chen | Assessing the Quality of Teachers' Teaching Practices[END_REF]. In this case, Chen at al. applied the model in thirty physical education lessons with nine elementary physical teachers. The results emphasised the applicability of the model. The competence-based model is another model that assesses teaching quality through teacher-licensure tests [START_REF] Mehrens | Assessing the Quality of Teacher Assessment Test. Assessment of Teaching: Purposes, Practices and Implications for the Profession. The Buros-Nebraska Series on Measurement and Testing at DigitalCommons at University of Nebraska[END_REF]. Mehrens's study is more an investigation and analysis of licensure and teachers' competency tests. A similar model is the competence-based model for teachers on how to teach [START_REF]Kultusminister der Länder in der Bundesrepublik Deutschland: Standards für die Lehrerbildung: Bildungswissenschaften. Beschluss der Kultusministerkonferenz. Ständige Konferenz der Kultusminister der Länder in der Bundesrepublik Deutschland[END_REF], and based on this, it assesses quality.

There are other approaches that consider pupils/students and the teachers' interactions for addressing quality. The CEM model is one of them, which assesses teacher quality based on students' outcomes [START_REF] Azam | Assessing the Teaching Quality in India[END_REF]. Azam and Kingdon applied their model to compare students' results of examinations from the tenth-grade to the twelfthgrade. Based on the results (improved or not) the teacher contribution was estimated.

The National Education Association uses a standards-based learning and assessment system to show how student learning standards can be connected with teacher education and assessment [START_REF]Framework for Transforming Education Systems to Support Effective Teaching and Improve Student Learning[END_REF]. Although there is no concrete implementation in practice, this is how they suggest measuring quality of teaching. The assessment of teacher competences and students' learning and feelings is integrated into another model presented by Snook et al. [START_REF] Snook | The Assessment of Teacher Quality. An investigation into current issues in the evaluating and rewarding teachers[END_REF], where they run an investigation in the New Zealand school system. The Angebots-Nutzungs Model is another model used to address quality based on teacher-student interaction (results, feelings, and environment) [START_REF] Helmke | Studienbrief Unterrichtsdiagnostik. Projekt EMU (Evidenzbasierte Methoden der Unterrichtsdiagnostik) der Kultusministerkonferenz[END_REF], while TEQAS is a model where quality is addressed by assessing teaching education [START_REF] Dilshad | Assessing Quality of Teacher Education: A Student Perspective[END_REF]. Dilshad showed the applicability of the latter model by covering five quality variables through interviews (questionnaire) with 350 students on MEd programmes.

Furthermore, there is the TALIS model, which assesses quality based on the working conditions of teachers and the learning environment [START_REF]OECD: TALIS Technical Reports. Teaching and Learning International Survey[END_REF]. This OECD article was a technical report where they applied this model in a pilot test (which was successful) with five volunteered countries: Brazil, Malaysia, Norway, Portugal and Slovenia.

Beyond traditional forms and assessment methods mentioned above, some maturity models based on the CMMI's principles have been created. Researchers in the field of computer science education adapted and created maturity models to assess and to improve the curricula or the institution itself [13 -15]. In these cases, validation of models is referred to at a later stage, but so far, no results have been published. Ling et al. applied their model through a case study in a private institution of higher learning (IHL) in Malaysia and mentioned that a larger participation of IHLs will be used in the future for a better validation of the model [START_REF] Ling | A Review Study: Applying Capability Maturity Model in Curriculum Design Process for Higher Education[END_REF].

The adaption of CMMI in the educational domain is seen also in course design, either in a classroom environment [START_REF] Petrie | A Model for Assessment and Incremental Improvement of Engineering and Technology Education in the Americas[END_REF] or online [START_REF] Marshall | Applying SPICE to e-learning: an e-learning maturity model[END_REF][START_REF] Neuhauser | A maturity model: Does it provide a path for online course design[END_REF]. The model of Petri is not validated yet [START_REF] Petrie | A Model for Assessment and Incremental Improvement of Engineering and Technology Education in the Americas[END_REF], but Neuhauser did validate the model in relation to usability, and the answers from the questionnaires revealed that 88% of the respondents found themselves in a cell within each process area [START_REF] Neuhauser | A maturity model: Does it provide a path for online course design[END_REF]. Similarly, Marshall and Mitchell validated the processes and the model in the analysis of an e-learning module at New Zealand University [START_REF] Marshall | Applying SPICE to e-learning: an e-learning maturity model[END_REF].

Likewise, in primary and secondary schools, some CMMI-like implementation models focus on the institutional level or on the syllabus [19 -21]. Montgomery applied her model in six schools for defining the level of using computers and technologies in schools. The model provided goals and practices for making improvements [START_REF] Montgomery | Developing a Technology Integration Capability Maturity Model for K-12 Schools[END_REF]. Solar et al. conducted a pilot study to test the validity of their model and its associated websupport tool [START_REF] Solar | A maturity model for assessing the use of ICT in school education[END_REF]. They tested the applicability of the model in different schools and obtained positive feedback from them.

Only Chen et al. established a maturity model for observing the teaching process. The model is limited to a subset of possible process areas and focusses on tertiary teachers [START_REF] Chen | Teaching Quality in Higher Education: An Introductory Review on a Process-Oriented Teaching-Quality Model[END_REF] only. In their paper, Chen et al. address the implementation of a model for primary and secondary schools, but to the best of our knowledge, such a model has not been implemented and/or yet published.

We believe that the quality of teaching is more than just focusing on the teacher or on the students, and it is more than just looking at the institution or the course content. It is rather a process that includes all the above and more. So, unlike the aforementioned models, like Chen et al., we address the quality of teaching by looking at the teaching process as a whole. However, in contrast to Chen et al., our model considers not only tertiary teachers but primary and secondary teachers as well. A more elaborated (tabular) overview about the differences between CMMI, TeaM and the T-CMM models can be found in the work of Reçi and Bollin [3, p. 7] where the authors compare the different process areas tackled and include these in the respective model.

Maturity model in practice

The application of maturity models is straightforward for engineers, but for teachers, such an assessment might be new. This section describes the application of a maturity model in practice, and briefly discusses usability and acceptability concerns.

The Capability Maturity Model Integration (CMMI) stemmed from the need to assess and improve the quality of products. After many years of research, the SEI collected and grouped together some relevant tasks and activities (by naming them Process Areas (PAs)). These tasks were further split into basic ones (named Specific Goals (SGs)) and their related activities (named Specific Practices (SPs)). The specific tasks and activities are unique to a PA. When talking about the generalisation and standardisation of processes, then some general tasks (named Generic Goals (GGs)) and related general activities (named Generic Practices (GPs)) were also defined. The latter tasks and activities are common for all PAs [START_REF] Forrester | CMMI for Services. Guidelines for Process Integration and Product Improvement[END_REF]. The assessment of the process for producing a product (software/service, etc.) with this model has a twofold meaning. It can focus on different PAs and defines at which level (Capability Level (CL)) the correlated specific tasks and activities are fulfilled, or it controls the fulfillment of tasks and activities on a predefined group of PAs that correspond to a Maturity Level (ML). Such outputs reveal at which maturity level a process for producing a product is. Further improvement for the process means fulfillment of the group of PAs corresponding to a higher ML [START_REF] Forrester | CMMI for Services. Guidelines for Process Integration and Product Improvement[END_REF].

Naturally, the question of how an assessment with maturity models looks might arise. For conducting the assessment, CMMI has specific models, which consist of steps of implementations. The assessment is conducted by a CMMI institute certified assessor. The steps of the assessment start with the analysis of the requirements, which determine what processes (sectors) a company wants to assess. This is followed by an appraisal plan development and a selection and preparation of a team for doing the assessment. The PAs are selected and a catalogue with questions is prepared. CMMI-Services contains a total of 24 PAs and each of them has corresponding goals and practices. This means that a catalogue with several questions needs to be answered by the interviewees. For this, considerable time is required, and the quality and quantity of questions is important as it might influence the results for ranking the company at the appropriate maturity level. In the last steps of the implementation, artifacts are obtained and the appraisal is conducted [START_REF] Forrester | CMMI for Services. Guidelines for Process Integration and Product Improvement[END_REF].

One major problem when addressing the quality of maturity models is related to time consumption for planning, answering and conducting the appraisal. It is also related to the quality and quantity of the questions, and consequently that a rating to the maturity model might influence the company (in terms of money, success, etc.). However, the published "appraisal results directory" from the CMMI institute manifests the usability and applicability of the CMMI model [22].

The Software Engineering Institution puts much effort in coming up with a consistent version of CMMI, involving a long process of studies and improvements within the last 30 years. Nowadays, although the model is applied in practice, there are still parts of it being improved. It is a continuous process of improvements. The same problem holds for the TeaM model. Several studies are required to produce a better version of the model.

Validating the TeaM

The TeaM is built up from the necessity for some standards to address the quality of teaching. The particularity of the model is on addressing quality by considering the teaching process as a whole with regard to teachers at university, primary and secondary schools. Making use of the model then either helps the educational institution in evaluating and improving its quality of teaching (by, when required, producing a ranking), or it helps teachers to evaluate and improve their teaching process on their own. Within the TeaM, the teaching process is composed of four phases:

-Initialisation -where administrative issues are managed; -Preparation -where the course is planned and prepared by teachers; -Enactment -where the implementation of the teaching unit takes place; -Quality and Incident Control -where possible incidents and the teaching process itself are observed, analysed and refined.

For each of these phases, factors related to the quality of teaching are determined, and in the TeaM terminology they are called Process Areas (PAs). Each PA contains a collection of goals and activities (practices). The implementation of these goals and practices indicates which PA is satisfied. In TeaM this is called "reaching a Capability Level". When a predefined group of PAs is satisfied, until the maximum Capability Level is reached, then also a Maturity Level is reached. The latter expresses how mature the teaching process is. Achieving a higher Maturity Level (so improving the teaching process) means satisfying all the PAs associated with that Maturity Level.

A detailed description of the TeaM and its related PAs can be found in the paper of Reçi and Bollin [START_REF] Reçi | Managing the Quality of Teaching in Computer Science Education[END_REF], where also a first assessment related to its consistency is presented. For the validation as presented in this study, another survey (including two questionnaires and one interview) were conducted. For this, the practices of the TeaM were mapped to 76 questions in the first questionnaire (comparable to CMMI appraisals), helping us to assess the quality of the model. The second questionnaire (containing 7 questions) then focused on applicability considerations.

Study objectives

Having the size of the TeaM and the time-requirements (when applying it in practice) in mind, the objective of the study was to test the TeaM in terms of usability and acceptability with teachers at the University of Klagenfurt. In the context of this paper, we tried to answer the following question: how is the applicability of the TeaM perceived by lecturers at the Alpen-Adria-Universität Klagenfurt?

To deal with the objective and for answering the question, a structured interview accompanied by a questionnaire were performed.

Research settings

A survey (including questionnaire and interviews) was used as a research instrument to assess the applicability of the TeaM in practice. The assessment was planned in a similar way to CMMI appraisals, and at first, we identified potential lectures and lecturers at our university. At random, 30 informatics courses from our bachelor and master programmes at Alpen-Adria-Universität were selected. The experimental subjects were the lecturers of these courses who were then interviewed. From 30 informatics courses that were selected for the study, only 13 lecturers participated and answered the questionnaire. The lecturers varied in their experience in teaching, from 3 years to 25 years. Only one lecturer was female, but all of them were specialised in the field of informatics and are teaching in the bachelor and the master programmes.

In comparison to CMMI, the TeaM has a total of 12 PAs with related goals (31) and practices (76). The practices of each PA were taken and a catalogue with questions was provided. The catalogue contained 76 "yes/no" questions representing the 76 practices of the TeaM. For instance, the practice "SP1.2.1.2 Arrange the Classroom Atmosphere" is mapped to the questionnaire as the question (translated to English): "7. Do you attempt to provide an adequate atmosphere in the classroom?" The same strategy is applied to all the other practices. For supporting the appraisal process, the 76 questions were provided in an electronic format using Google forms. This makes the questions public and accessible by those who are interested to use such a model. Moreover, the participation remains anonymous as no personal data are collected. The link to the questionnaire is maintained on the website of our Department (in the project section with the name "TeaM model"). On the project website, you find both the link for the questionnaire and the file containing the detailed description of the TeaM Version 1.6 (including the 76 practices) [START_REF]IID: Research projects[END_REF]. Teachers and educators are invited to join the project and to report on their personal experience with it.

For performing the appraisal, two non-expert assessors (members from the Informatics Didactic Department of the Alpen-Adria-Universität Klagenfurt) were involved in the interviews. During the interviews, the teachers were given two questionnaires. The first questionnaire contained 76 questions related to 76 practices of the TeaM. This was necessary in order to introduce the model to the teachers (by applying it in practice). The second questionnaire (with 7 questions) then focused on the two dimensions, usability and acceptability, and it was given to teachers after applying the TeaM. The questions focused on:

• The results are presented in detail in section 3.3, while the presentation of the results from the first questionnaire (the TeaM assessment) is not in the scope of this paper. In a next step, however, the results from applying the TeaM in practice will be analysed to see if there is a correlation between the generated TeaM's maturity levels for each course with the feedback provided in the ZEUS system at the University of Klagenfurt.

Study results

The 13 lecturers participating in the questionnaire worked through all the "yes/no" questions about the practices of the TeaM, and at the end they provided their opinion about missing/relevant practices of the model. Additionally, a questionnaire with 7 questions was given to them to better understand their perceptions about usability and acceptability of the model.

(Q1) The first question was related to the time required to fill out the questionnaire. The average time was 30 minutes to answer the 76 questions. Only one interview lasted longer (56 minutes) because the assessor read the questions and the interviewee read the questions himself one more time.

(Q2) The second question dealt with the understandability of the questions from the first questionnaire. We were looking for any ambiguities. Five questions needed explanation from the assessor, because their structure was misleading for the interviewees. Basically, these questions were connected with "and/or" conjunctions and they confused the interviewees. Examples of such questions were: "Do you consider other requirements that might come from students/pupils (like explanation of a new term, repetition of an exercise, etc.), OR administration (like substituting a colleague in one teaching hour because she/he is sick?)"; "Do you consider AND document problems during units' delivery?" Another problem was a set of questions related to existing curricula. As there are courses which are not based on only one curriculum, a correct answer was impeded as well.

(Q3) The third question produced a ranking from unpleasant (1) to wonderful (10) of the process for filling out the questionnaire. The interviewees rated it with 6. This was related to the unclear structure of the sentences and due to the fact that they had to think about their teaching process for the first time. This created a little tension for them and they were trying to explain the reason why their answers were "no" or why "bad" things happened in their course. The assessors think that the TeaM questionnaire might work better without the presence of an assessor. However, the interviewees expressed their deep interest in the model.

(Q4) The benefit in using this model for the future was the fourth question. The interviewees liked the idea of thinking about the questions that helped to improve their teaching, so they thought that it was an advantage to use the model. The only problem identified was related to documentation practices that were required by the model.

(Q5) The fifth question revealed if the TeaM is relevant or appropriate to be used in order to assess the quality of teaching. None of interviewees raised a concern that any of the questions was not related to the quality of teaching. They saw it as a good collection of standards to follow for addressing the quality.

(Q6) The sixth question looked closer at the fear of the interviewees if such an assessment could criticise their way of teaching. In a way, the answers were "yes".

They expressed this response even in question 3. There, they expressed worries about some questions that they could only answer with "no", and this was in a major way related to the documentation practices.

(Q7) Last but not least, they were asked about other observations or ideas to share. They thought that providing more information on the questions in such a way that no assessor had to participate during the assessment would make them answer with less tension. Most questions were well understandable and also interesting to think about. Already the process of trying to answer the questions and thinking of their own process was felt to be worthwhile.

Discussion

By analysing the collected feedback, it is noticeable that the model somehow surprised the interviewees. It made them think (maybe for the first time) about teaching as a process. If we go back to the questionnaire, it is obvious that, in comparison to the CMMI questions catalogue, answering the questions concerning the TeaM takes not so much time (referring to Q1). This is worthwhile when thinking about the model as a part of assessing and improving your work.

Based on the results (Q2), we see that the TeaM needs to be improved regarding the structure of its "and/or" sentences, even though splitting them will yield a slightly larger number of questions and consequently lead to a higher time consumption.

When answering the main question related to the objective of this paper, the TeaM is perceived as interesting from the general point of view of the lecturers at Alpen-Adria-Universität. Providing an improved version of the model (with clearer questions and with no assessor) will further motivate the teachers to use it in practice. Clearly, at least within the scope of the study, the model is applicable by the teachers at Alpen-Adria-Universität Klagenfurt. Another benefit to be considered is: by just introducing the TeaM, the idea of seeing their own teaching process in more detail was planted into the heads of the participants. When perceiving TeaM more as a self-assessment framework rather than as a raking generator, then its integration in practice in the educational domain could be greater.

Summary and future work

The TeaM is an ongoing project running at the Alpen-Adria-Universität Klagenfurt. At first, it can be seen as a model for ranking. This might create doubt with teachers as to whether to use it or not. However, the main aim of the TeaM is not to create a ranking between teachers or educational institutions (even though one might do so). TeaM aims at providing a framework that helps teachers to assess the quality of teaching and to tell them how to improve.

After a lot of theoretical research, the TeaM is now consistent, and its applicability in practice was tested for the first time. Based on the results presented in this paper, it seems that it can be used by teachers to assess their teaching process.

As to future work, we plan to test the model in other courses at the University and schools and to produce stable maturity levels based on the results. Further future work will be the extension of the TeaM by an advisory framework. The practices of the models will then be presented in a form of a checklist, clearly defined and annotated, and future users will not need the presence of an assessor to apply the appraisal.

  Q1) Time to fill out the TeaM questionnaire • (Q2) The understandability of the questions • (Q3) How much they liked filling out the questionnaire • (Q4) The assumed benefit of the model in the future • (Q5) The relevance of the model for assessing the quality of teaching • (Q6) Whether the model would criticise the teachers' way of teaching • (Q7) Other observations or ideas to share