Athanasis Karoulis
email: karoulis@csd.auth.gr

Andreas Pombortsis

The Cognitive Transfer and the Tutor's Role in a CBL Environment

Keywords: computer based learning, tutor, instructional methodology

In this paper we report on a longitudinal study of a Leonardo da Vinci program regarding the application of a Computer Based Learning environment in three EU countries, Greece, Germany, and Holland. The "Orestis" CD-ROM was a multimedia CBL environment aimed at teaching young offenders the basic and advanced skills for the use and maintenance of a photocopy shop. The complete project consisted of three phases: construction of the CD-ROM, application of the instructional methodology in the participating countries and evaluation of the outcome. In this paper we firstly clarify the notion of CBL environments, in comparison to multimedia and distance learning ones, and then we provide some theoretical background on these environments. The main research question we tried to answer was whether the application of the environment under real circumstances could perform adequately without the presence of a domain expert as a "teacher", but rely on a "tutor" who had five main tasks to perform during the instructional phase, as described in the paper. In order to answer this question, we designed and constructed the aforementioned integrated CBL environment and, in addition to this, an instructional methodology in order to apply it effectivelly in the participating countries. The application of the environment by the partners showed our hypothesis about the tutor to be correct, moreover we elicited results about the different user groups that worked with the environment and their performance under different circumstances. We studied also the combination of CBL training and real practice in real environments, among some other issues concerning the application of CBL environments in general, and we provide some statistical facts as observed and reported during the two pilot studies and the five applications of the environment in the three participating European countries. We conclude by arguing that the tutor, as described in this paper, performs more than satisfactorily and demands far fewer resources than a real domain expert. However, it must be emphasized once more at this point, that this conclusion implies the application of a predefined instructional methodology that aids the tutor in his role, since the teaching procedure will be a task for the hypermedia enriched CBL environment, designed ad hoc for the intended user group and according to the appropriate educational theories.

Introduction

Computer Based Learning (CBL) environments have been used since the early days of the utilisation of computer technology in education. [START_REF] Cullen | Review of Current Pedagogic Research and Practice in the Fields of Post-Compulsory Education and Lifelong Learning[END_REF] provide a wide definition of a CBL environment to include any learning involving the application of telem-atics, Information and Communications Technology (ICT). [START_REF] Maier | Integrating Technology in Learning and Teaching[END_REF] provide overviews of learning and teaching for the range of different types of learning technologies. However, [START_REF] Sieber | Learning technologies and learning theories[END_REF] pinpoint that there is a considerable variation between individuals in the way they learn, and an individual may well learn in different ways at different times. In this context, a CBL environment can be defined as a piece of software that cognitively covers a particular domain and provides the individual student with all the means to gain knowledge on the domain. This definition implies two assumptions: the cognitive coverage and the presentation of the domain is sound (theory, exercises, simulations) and the learner can interact with the piece (interactivity with the software, multimedia elements); in other words, there is a communication channel in order for the student to acquire the offered knowledge.

The cognitive transfer, namely the ability of every educational environment to facilitate the acquisition of knowledge, is an issue of paramount importance. In CBL environments the communication and computation technologies play a major role in the storage, process and presentation of educational related information. Moreover, these technologies establish an interaction mechanism in order for the user to communicate with the software. This definition may be adequate to define CBL environments in a technical manner, it does not however answer the question of the cognitive transfer, since the technology by itself can not guarantee it. Furthermore, even motivated students cannot be relied on to take full advantage of exploratory opportunities of the CBL environment. Studies (e.g., [START_REF] Pane | Assessing dynamics in computer-based instruction[END_REF] showed that substantially more guidance must be built into such experimentation environments. Other studies argue that computer-based learning may be a cost effective way to deliver improved tutoring, but only if the computer-based systems can perform as true "intelligent" tutoring systems (ITS) and accomplish explanation of the material, not just drill-and-practice or on-line answers [START_REF] Warren | Investigation of tutorials in a core computing subject[END_REF]. On the other hand, some researchers (e.g., [START_REF] Corbett | Instructional interventions in computer-based tutoring: differential impact on learning time and accuracy[END_REF] argue that we can reliably build "second generation" intelligent computer tutors that are approximately half as effective as human tutors. They are often reffered to as "cognitive tutors", however, an intelligent and personalized assistance that a teacher or a peer student can provide in a normal classroom situation is not easy to get [START_REF] Brusilovsky | Adaptive and intelligent technologies for web-based education[END_REF]. So, it is obvious that the optimal equilibrium between the computer scaffolding factor and the human presence must be in more detail investigated, however, the corresponding educational theories must firstly be considered.

Educational Background

As theories of learning have developed and educationalists have gained more experience in using computer-based technology, there has been a shift of emphasis from the behaviourist paradigm, through the weak artificial intelligence approach, as described by [START_REF] Atkins | Theories of learning and multimedia applications: an overview[END_REF], to a constructivist view. For most educationalists, constructivism offers far more scope for realising possible learning benefits of using information and communication technology. In fact [START_REF] Reeves | Evaluating what really matters in computer-based education[END_REF] refers to the claim by Gagné and Glaser (1987) that virtually all self-respecting instructional design theorists now claim to be cognitivists [START_REF] Squires | Predicting quality in educational software: evaluating for learning, usability, and the synergy between them[END_REF].

Many writers have expressed their hope that constructivism will lead to better educational software and better learning (e.g. [START_REF] Brown | Situated cognition and the culture of learning[END_REF][START_REF] Papert | The Children's Machine: Rethinking School in the Age of the Computer[END_REF][START_REF] Jonassen | Thinking technology: Toward a constructivist design model[END_REF]. They stress the need for open-ended exploratory authentic learning environments in which learners can develop personally meaningful and transferable knowledge and understanding. The lead provided by these writers has resulted in the proposition of guidelines and criteria for the development of constructivist software and the identification of new pedagogies. A recurrent theme of these guidelines, software developments and suggestions for use is that learning should be authentic, on a cognitive and contextual level. A tenet of constructivism is that learning is a personal, idiosyncratic process, characterised by individuals developing knowledge and understanding by forming and refining concepts [START_REF] Piaget | The Origins of Intelligence in Children[END_REF], which finally leads to the five main socio-constructive learning criteria [START_REF] Squires | Predicting quality in educational software: evaluating for learning, usability, and the synergy between them[END_REF] that must be met in order to characterize an educational piece as socio-constructive: credibility, complexity, ownership, collaboration and curriculum.

As regards the use of technologies, such as hypermedia and the web in CBL environments, [START_REF] Marchionini | Evaluating hypermedia-based learning[END_REF] argues that the use of them allows the learners access to vast quantities of information of different types, control over the learning process and interaction with the computer and other learners. A pilot study performed at Cornell University [START_REF] Fitzelle | Survey Evaluation of Web Site Instructional Technology: does it Increase Student Learning?[END_REF] had as its primary research question whether the web site enhanced student perceptions of learning. The research findings showed that students thought that the web site significantly enhanced their learning of course content. Student perceptions of performance in the course were also predicted by variables of enjoyment and control of the learning pace.

Every web-based instructional program is a collaborative environment on its own, allowing users to communicate and interact with all participating entities, therefore, in common with current thinking, cognition is "distributed" between users, the environment and learning artefacts, including computers, when learning takes place [START_REF] Brown | Situated cognition and the culture of learning[END_REF]Salomon, 1996). The distribution of cognition leads to learners constructing their own concepts which they use to learn. [START_REF] Ester | CAI, lecture, and student learning style: the differential effects of instructional method[END_REF], in reviewing literature on Computer Assisted Instruction (CAI) and learning style, found that CAI can significantly improve student achievement and attitudes while decreasing necessary instructional times. A meta-analysis of the effects of CAI on student academic achievement and performance by [START_REF] Khalili | The effectiveness of computer applications: a meta-analysis[END_REF] found that in 151 published comparative studies, the use of CAI raised student performance on exams by an average of 0.38 standard deviation. [START_REF] Kulik | Effectiveness of computer based instruction: An updated analysis[END_REF] in examining a large number of studies found that computer based tutorials produce improvements in learning outcomes on an average of 20 percent greater than average. Simulation, interactive video instruction, hypertext programs, bulletin boards and networks have also all been found to be effective in enhancing learning [START_REF] Cronin | A critical analysis of the theoretical foundations of interactive video instruction[END_REF][START_REF] Khalili | The effectiveness of computer applications: a meta-analysis[END_REF][START_REF] Kulik | Effectiveness of computer-based instruction in adult education[END_REF][START_REF] Schlecter | Computer-based simulation systems and role playing: an effective combination for fostering conditional growth[END_REF]. Finally, [START_REF] Bagui | Reasons for increased learning using multimedia[END_REF] refers to several studies showing that computer-based multimedia can help people learn more information and learn it more quickly compared to the traditional classroom lectures. So to summarize, there is good research for demonstrating that instructional technology often optimizes learning. However, nagging questions remain, such as what features work best, differentiating effects between subgroups of learners, determining how the content of the information makes a difference and specifying how outcomes may be more systematically evaluated, as well as the question of how one should evaluate the learnability of an on-line learning course.

Educational psychology provides many theoretical principles to be applied in the development and evaluation of on-line instructional technology. Milheim and Martin (1991) in studying learner control motivation, attribution and informational processing theory, identify learner control as an important variable in developing the pedagogy of web sites. It is beneficial to generally maximize learner control as it increases the relevance of learning, expectations for success and general satisfaction contributing to heightened motivation (Keller and Knopp, 1987). This research looked specifically at the control of pace by the student as a factor in building on existing theory. A tenet of constructivism is that learners direct their own learning either individually or through collaborative experiences. This implies that learners need to find their own pathways through learning; a philosophy that under-pins hypertext and many web-based instructional systems (Murray, 1997). E-mail, listservers and web browsers also support this approach by enabling students to search for information and discuss issues with others around the world. So, one can infer that the collaborative and interactive nature of the web supports learning mainly by means of augmented motivation of the student. It is common that students make wrong assumptions or possess a false mental model of the domain they study.

Another study, by Sloan (1997), states that the WWW represents a step forward towards the use of resources that are often difficult or impossible to obtain from traditional information sources, and gives seven key areas to consider before using the medium to assist learning, simultaneously or supporting the traditional paradigm:

• information availability, that is to ensure that there is an adequate range of material available to provide a varied selection for the intended group of students; • ease of access to that information; • control of access, which is a problem because the Internet has a lot of inappropriate material which is easily accessible; • group dynamics, as web-assisted instruction is in most cases collaborative; • specification, to determine the correct information from the great deal of data one can gather from the Internet; • appropriateness, the material must be of a type that is useful to both, students and teacher; • teacher input, the additional input required to complete the instructional material.

The Research Question

CBL environments may become very complex, in order to completely cover the cognitive domain they deal with. On the other hand, the needs of the potential users must be taken into consideration as well. Some studies, like [START_REF] Rappin | Balancing usability and learning in an interface[END_REF] have showed that students needed a system that would encourage them to examine their assumptions as they worked through problems in their domain. Students also need motivation in cases of educational software, such as simulation environments, which provide students with realistic experience, even in domains where realistic activities are too complex to be performed by novices, too expensive or too dangerous to allow students to make mistakes. This makes the presence of a teacher inavoidable.

However, CBL environments are built for repetitive use, for use over a long period of time, or, as already stated, as knowledge repositories, where students should be able to work on their own and "on-demand", namely to study only the needed chunk of the offered knowledge. Under these circunstances, the presence of a teacher could be too expensive, or simply impossible.

On the other hand, CBL environments contain all the required information to be applied successfully. This implies a certain overlapping in roles between the teacher and the environment, since both can offer knowledge. From a motivational perspective, most CBL programs are designed to serve as a self-study environments with frequent feedback. This tends to be a highly effective and efficient method for delivering some instruction (Keller, 1983, p.412). So it is a pity to abandon their use because of the lack of an appropriate teacher, or because a teacher over a long period of time would be too expensive. In practice, this is what really happens: distance learning environments are designed and applied (whatever it means), because they don't preassume a teacher, while CBL environments are rarely constructed, although they are much cheaper and easier to produce, because organisations cannot afford a teacher to utilize them. So, in this work we stated two null hypotheses: H 1 : in a CBL environment, no teacher is needed, and H 2 : in a CBL environment a domain expert is needed as a teacher.

We wanted to study following alternative hypothesis H a : Instead of employing a teacher, we use a "tutor", who has only basic knowledge on the cognitive domain under consideration, yet he is an expert in the use of the environment. By utilizing a previously designed educational path through the software, is he/she able to "lead" the students through the environment?

To prove this hypothesis we worked on a Leonardo da Vinci EU-Program, described in the next chapter, which we designed as a self-study complete environment. In addition to this, we designed a detailed presentation methodology for the tutor, which was more a "time-scheduled navigation" through the software. The coverage of the cognitive part of our experiment was completely left to the CBL environment.

Description of the Software

The "Orestis" CBL environment is part of the Leonardo da Vinci project "Abilities and skills enhancement of minors offenders and minors at risk", and was a collaboration between many organizations in four European Countries, Greece, Germany, Holland and Hungary.

The main CBL software was designed and constructed in the Multimedia Laboratory of the Aristotle University of Thessaloniki in Greece. Four identical versions in four languages, Greek, German, Dutch and Hungarian were produced.

The "Orestis" CD-ROM was a multimedia CBL environment aimed at teaching young offenders the basic and advanced skills for the use and maintenance of a photo-copy shop. The complete project consisted of three phases: construction of the CD-ROM, application of the instructional methodology in the participating countries and evaluation of the outcome.

Part 1. The "Orestis" Environment

The "Orestis" CD-ROM simulates a photo-copy shop with the basic machines (black and white photocopier, phototyper, spiral-binder, thermo-binder and laminator) and consuming materials (papers, transparencies, toners, graphite etc). The knowledge offered is structured in theory, exercises and practice. The theory is offered with text and multimedia elements, such as audio, video and images. Through a series of "chapters" and "paragraphs" the student is provided with the complete theory on the particular domain. There are three kinds of exercises:

1. Simple exercises of type "multiple choice" or "true -false". They aim as self control exercises, since there is scorekeeping, in order for the student to assess his/her own progress. From a cognitive perspective, these exercises aim to assess the aquisition of the knowledge offered by the student.

2. Costructive exercises where the user is asked to perform a task by "drag and drop" on the screen or by manipulating some variables. From a cognitive perspective, these exercises aim to assess the synthesis of knowledge by the student. That is, the tasks that the students are asked to perform are similar, but not identical, to the material already presented in the theory.

3. The third kind of exercises is also the third structuring modul of the environment: the practice. In a separate simulation environment the student has to control a photo-copy shop on his/her own. Customers give orders, choosen among the most common real orders in real photo-copy shops. The student now has to fulfil the customers request by applying all he/she has learnt. From a cognitive perspective, this environment aims to assess the transfer of knowledge, namely the ability to rely on previous acquired knowledge and apply it to an environment outside of the instuctional one, constructing thus his/her own experience.

In addition to this, there are two levels of assisting facilities. One, which is screen sensitive and is available throughout the whole program, and a second which applies only to the exercises and depends on them, so that the student is always supported to complete them.

Part 2. The Istructional Methodology

As already stated, this was the main research point of the environment. The core point of the work with CBL environments is the non-linear structure of the course. The knowledge offered may be organzed in logical chapters, however the instructional methodology may have a very different logic. In our example the knowledge was organized into four main chapters: the machines, the materials, problems and technology. Some secondary utilities were also presented, such as guidance for newcomers, a lexicon, help and historical facts, but they played a less significant role as the instruction proceeded.

The instructional methodology, on the contrary, follows the logic of completing a "cognitive chapter", for example the black and white photocopier. So the students were asked to navigate through the machines-chapter, but only the paragraphs that affected the black and white photocopier, then through the materials-chapter but only through the paragraphs that affected the materials for the black and white photocopier, continuing, the corresponding paragraphs in the technology-chapter and so on. Obviously, the same methodology was applied to the exercises of every chapter.

An important person in the application of the software was the tutor -we no longer call him/her "teacher" because of the alternate role he had to play during the instructional period. The tutor was given a booklet with detailed instructions and a detailed time schedule with the chapters and the exercises he should guide the students through, for every day of the instruction. It was made completely clear to the tutor, that he/she should NOT give any cognitive help to the students, a task that was left to the software alone. The tutor could only give advice, on how the student could find the solution to his/her problem. Example: "The copy is too light. What went wrong?" Answer: "Why don't you look at the chapter about the adjustments on the control panel?".

A second task the tutor had to perform was the observation of the students' progress and the adaptation of the instructional pace for every student. As it may be obvious up to now, the targeted users were a special user group, namely young offenders. They provide in general some special characteristics, common for this user group that should have been taken into consideration through the design phase of the software as well as through the application phase. It is outside of the scope of this paper to present the complete design methodology. We only mention here that the main characteristics we took into consideration were impatience, disorganization, distraction and fragmentation of the thought and actions of this user population, and secondly the fact of their low cultural status and the possibility of illiteracy or different mother language. The influence of these characteristics in the application of the software was that the tutor had to face a variety of performance paces in the "class". So, he/she had to assess up to a certain level the success of the exercises for every student and, if nessecary to provide him with easier tasks, or, for "good" students, to guide them to more detailed information or to additional exercises to lead them further.

To conclude this point, the tutor had four main tasks:

1. Guidance. He/she had to guide the students through the software.

2. Scheduling. He/she had to keep the correct pace in the class, so that the time schedule was respected.

3. Support. He/she had to help the students in user interface issues, such as the manipulation of the mouse or its "click and drag" function.

4. Observational/assessing role. This role was indispensable, because of the evaluation of the program that performed in parallel.

The Application in Practice

We applied a longitudinal study in three countries, Greece, Germany and Holland. Unfortunately, there has no been any report from Hungary. The reporting here is based on one report from Germany (Report 1, 2000) concerning the application in one prison, one report from Holland (Report 2, 2000) concerning the application in two prisons and two applications in Greece.

We firstly applied a pilot study in Germany. Four subjects were asked to work with the program, without a teacher or a tutor and without any further support. In this phase we wanted to exclude our first null hypothesis H 1 , as we believed that no user belonging to this special user group could work with a CBL environment without any support. Indeed, the first null hypothesis was quickly excluded by the observations in the usability laboratory of the Technologiezentrum at the University of Bremen. A second pilot study to validate these results was then applied in Greece. Four subjects were asked to work with the environment without any further help. Here we observed different results with the same conclusion. This is how: in two days the subjects declared to have completely studied the environment. In a test that followed, with simple questions that were handled in the software, the failure was almost complete: of the 10 stated questions only one subject answered one and another three, that is a percentage of 90% failure. So, although it seemed here that the use of the software could take place without any support, the cognitive transfer was negligible. This again leads to the conclusion that the use of a CBL environment, by novice users in the domain, cannot take palce without a tutor.

After this pilot phase, the main application begun. The partners in Germany applied the environment in a prison, under realistic circumstances for twelve days with the participation of five subjects. Four of them were using drugs before their arrest. One tutor served during the whole period.

In Holland the partners applied the environment in two prisons. In the first prison, seven subjects between 15 and 18 years participated for one week intensive course and in the second four subjects between 19 and 35 years for a three weeks period. In both prisons there was a tutor plus the working personal. Most of the subjects had history with drug consumption.

In Greece, we applied the environment with 5 subjects, 15-19 years old, which were in the category "young at risk"; they were not in prison, yet most of them had history of arrests and drug consumption. The course lasted for two weeks and two tutors were present. A second application of the environment took place with a very different user group. For a one week course twelve unemployed women between 20 and 40 years worked with the environment, as part of a vocational training program and finally evaluated it. One tutor was present during this phase and only two real machines could be used, a black and white photo copier and a spiral binding machine.

Results

Since this study mainly concerns a qualitative investigation of the acceptance of the turor in his/her new role, a detailed statistical elaboration of the collected data has been omitted. So, the opinions of the test subjects have mainly been grouped into major categories, expressing their percieved performance during their work in the integrated environment of the CBL system and the tutors.

The application in all countries showed that there were no problems with the use of the software, nor with the first two categories of the exercises. However, the tutor in Germany had often to intervene during the work with the exercises of the third kind; the subjects obviously had problems in transfering the knowledge, which is partially explainable by their drug history and the special characteristics of the particular user group. However, in complete contrast to the above, these users characterized the exercises of the first kind as extremely simple, as "for children" (their words) (Report 1, 2000).

There are some interesting observations concerning the notion of CBL environments, as perceived by the users. The German users stated their work to be "very serious", while the tutor stated that it seemed to be "the ideal media, in order to wake up the particular user group to some activity in the domains of problem-statement and solution-finding" (Report 1, 2000). In addition to this, they seemed to work without any anxiety on the computer and they established a kind of co-operative work, as they helped each other in cases of uprising problems.

In Holland, 6 of the 7 subjects questioned on whether they should be able to handle the program without the aid of a tutor, answered "yes". This is contrary to the observations of the personel, who pinpointed certain difficulties in the use of the software or in the navigation. However, this statement implies the subjective satisfaction of the users, which is stated also in the question on the overall satisfaction with the program. On the other hand, some users complained that "they could not remain on the same subject", which implies a navigational disorientation. The conclusion of all this is that many times the users' opinions can be influenced by the special characteristics of the particular user group or by other random factors. However, in some issues there was an impressive coincidence. In the question whenether the exercises enhanced their knowledge on the domain, all the answers were 8 and 9 in a 1 to 9 scale [START_REF]Bericht über die Praxisevaluation in den Jugendvollzugsanstalt Bremen Blockland[END_REF]. Following this statement, the users declared the whole environment as positive for instructional purposes, aiding thus the use of CBL environments under those particular circumstances, such as special user groups, short-time training or difficulty in providing "normal" training etc. However, the tutors and the aiding personel had a different opinion. In Holland it was not possible to complete the course with real photo-copy machines for practical exercise. So, the personel believes the CBL course to be just a part from the whole procedure. On the contrary, in Germany the subjects worked on real machines and the personel there observed a good integration of the aquired knowledge in real work circumstances. These observations were enhanced in Greece, where during the practical phase with real machines, the subjects faced new problems. So, some quick references to the software were made with the help of a laptop and many problems were solved on site with the help of the tutor always acting as navigator, e.g., "Why don't you look for the solution in chapter X?". So, it was apparent that the combination of the software with real practice provided the best results.

During the applications in Greece we studied the issue of motivation as well. We used some questions in the questionnaires to clarify the four main factors of motivation, according to [START_REF] Keller | Motivational Design of Instruction[END_REF]: curiosity, expectancy, relevance, and satisfaction. The main outcome was that the motivation was high, which is in accordance with the observed results in the other countries, yet the coresponding acquisition of knowledge was not as high. The subjects thought they had learned things that they really didn't. This is due to the particularities of the user groups, since this particular result was not observed during the second application with the unemployed women. Here, the motivation was also high, but in any case more in accordance to the cognitive transfer. Here no subject "thought" that she had learned something she really hadn't, just because it was taught by a computer program.

During this second application in Greece, we also collected some interesting statistical results concerning the use of the multimedia elements during the instruction. So, for "normal" users, everybody reads the text, however only 60% do so carefully, therefore not needing to re-read it; only 40% listen to the audio, and only 40% look at all images (if they are hyperlinked, as it was the case in "Orestis"), while 80% watch all the videos. In the case where there is text and video, they prefer firstly to watch the video and after to carefully read the text, in order to clarify the procedure they just watched.

Observations about the process speed of the users showed that 60% of the users proceeded more or less at a logical pace, while 20% were too fast and 20% too slow for the rest of the class. However, all of them showed the same success rate in the exercises.

Conclusions and Discussion

Let us at this point answer the three hypotheses we made.

H 1 : In a CBL environment, no teacher is needed. This is our first null hypothesis and must be rejected, as the results in Germany showed, where we stated total failure to manipulate the environment and the results in Greece, where there was neglectic cognitive transfer. However, in the case of repetitive use, such as a knowledge database, it is practically impossible to employ a teacher or a tutor, yet in instructional ones it is inavoidable. H 2 : In a CBL environment a domain expert is needed as a teacher. Our research showed that this hypothesis is also not true. Rarely was more knowledge demanded than offered by the environment. A domain expert would eventually be able to add more knowledge or to offer more exercises and questions to the students, however this issue was not within the scope of this research. Our scope is stated through the alternative hypothesis we made:

H a : In a CBL environment we can use a "tutor", who has only basic knowledge on the cognitive domain under consideration, yet he is an expert in the use of the environment. In addition, he adheres to a former declared instructional approach. This hypothesis has been proved as true in all applications, since none of the tutors in Germany, Holland and Greece was a domain expert. Yet, as reported by all partners, the cognitive transfer was significant, even in cases where no real machines were available for practice, as was the case in Holland.

So, the suggested tutor no longer needs to be a domain expert. It is sufficient for him/her to be a power-user of the environment, he/she could even be a former user of it, or a person who is especially trained in it. The main advantage of this approach is that it demands fewer less resources (time, cost, availability etc.) than required when recruiting a real domain expert. In the survey domain, there are also a number of studies, like [START_REF] Bagui | Reasons for increased learning using multimedia[END_REF] and [START_REF] Roselli | Artificial intelligence can improve hypermedia instructional materials[END_REF], arguing that hypermedia environments, equipped with a tutor capable to appropriately drive the learner, proved to be efficient learning systems [START_REF] Costabile | Experimental evaluation of an intelligent tutoring system[END_REF].

The presence of the tutor depends also on the user group. A clear conclusion from all studies was that CBL environments work excellently with "normal" user groups, namely with relative homogen classes. If it is not the case, then a tutor with basic understanding of the domain is inavoidable. This is because an extremely differentiated user group in general provides a great variety of reactions during the instruction, so it is indispencible to have a tutor to handle the issues that arise. However, it is not necessary to be a "teacher", in our studies the personel of the prisons performed without problems. This position is also strengthened by the interaction of the tutor with the subjects during the application in Germany, because of the many questions they asked. On the other hand, for "normal" user groups, as it was the case during the second application in Greece, there is no such need at all.

Another interesting result concerns the long standing discussion about CD-ROM vs. Book. 44% of the subjects stated, reading from a monitor to be better than from a book, "because of the multimedia elements and absence of boredom", 28% thought it is the same, "reading is reading" and 28% thought the book is more "personal". In addition to this, they answered the question if it is better to learn from a book or with the aid of a computer. 72% of the subjects stated the computer to be more stimulating, while the rest preferred the book. The foundation of these statements relies mostly on the augmented attraction and motivation of the medium computer, yet parts of the medium are confused with the whole (Report 2, 2000), e.g., "reading from screen is better, because of the multimedia elements". So, these statements, although interesting, can not contribute a lot in the direction CD vs. Book, they just indicate a user preference to CBL.

As regards the special user group for this particular environment, there are some additional advantages of the CBL model. According to [START_REF] Friedrich | Telelrnen im strafvollzug[END_REF]:

• The prison sentences for young offenders has decreased during recent years and are now in average between some months and a year. This makes a long instructional period practically unapplicable.

• There is a great cultural and intellectual differentiation between the young offenders in prisons which prohibits the structuring of homogen classes. So approaches of modularization of the instruction and of individualisation of the learning environment are demanded.

It seems that for practical skills, the combination of the CBL environment with real practice is the correct solution. Even more, if the software can be used during the practice, as it was the case in the first application in Greece, the results are even better.

The problem of the adequate cognitive coverage of the domain under consideration remains open. In the evaluation of the program, many deficiencies in the cognitive domain were detected, as for example with the colour photocopier. In Holland [START_REF]Bericht über die Praxisevaluation in den Jugendvollzugsanstalt Bremen Blockland[END_REF] the tutors were often asked for cognitive issues the software didn't cover, or for more advanced issues the software couldn't cover. On the usability field, although the program provided few usability problems, there were sometimes problems with the navigation in it. The current work can not address this issue of the cognitive coverage, but it seems that the application of the environment in practice and the revisions of the software according to the notifications of the users is a step in the right direction, an approach that is strongly recommended by almost all authors.

Another observation in Holland was that the subjects preferred to work on their own. This is in contrary to the collaborative model CBL environments are supposed to support. An obvious explanation lies here with the different learning pace of the students and the diversity of the classes. The same phenomenon was observed more or less in all applications of the program, even in the more homogen unemployed women class in Greece. So, this observation brings into question the long standing opinion that, in general, computer-based instructional environments facilitate CSCW (Computer Supported Co-operative Work). Our opinion is that it may be suitable for distance learning environments, where asynchronous modes of communication, such as mail, discussion forums etc. play a vital role. In CBL environments there is a clear domination of the model "working on the PC on my own". Another observation supports this statement as well. In Holland a great variety in the working mode of the subjects has been observed. The same observation we also noted in Greece, where some subjects, in both applications, relied solely on the audio facilities of the program, barely reading the text, while others relied strongly on the video to understand the procedure under consideration. This diversity in the working mode breaks the coherence of the class and would even cause problems for a teacher in the core meaning of the term.

An interesting issue arose from the questionnaires in Greece. All of the subjects stated in the corresponding questions that the CD provided them with knowledge they couldn't have acquired in practice. This is based on the fact that in the CD many theoretical issues (e.g., how the machines operate) or technical ones (e.g., operation temperatures or consistence of the materials) or historical ones were exploited. This leads to the known result that any instruction must have a well balanced theoretical background and practical session.

Concerning the research future on the domain, it is interesting to investigate the possibility for an interface agent to replace the tutor. As we described the role of the tutor in this work, it could be an adaptive interface agent as well. The only nagging question is whether it could act in cases not foreseen by the designers, in case of collaboration or an external problem. In addition to this, it is under question whether the contemporary adaptive technology can really produce a capable interface agent to replace the tutor; however we hope that this limitation is only temporary.

To conclude, we strongly suggest the design of CBL systems with a tutor, who has to perform the four tasks already mentioned: guidance, scheduling, support and observation, but not the teaching of the material. It must be emphasized once more at this point, that this conclusion implies the application of a predefined instructional methodology, ad-hoc prepared for the particular environment, that aids the tutor in his role, since the teaching procedure will be a task for the hypermedia enriched CBL environment, designed for the intended user group and according to the appropriate educational theories. We recognize that this may be a hard task, yet there are no known simple tasks in the fusion of education with the IT technologies.

A. Karoulis has a BSc in mathematics from the Aristotle University of Thessaloniki, a degree in educational technologies from the University of Macedonia (Greece), a degree in open and distance learning from the Greek Open University, a MSc in information systems from the University of Macedonia (Greece), and a PhD in informatics, in the domain of human-computer interaction from the Aristotle University of Thessaloniki, Greece. He is currently active as an instructor in educational techologies for secondary education, as a multimedia and web project manager and as a researcher in the domains of HCI and of distance learning, regarding the application of new technologies at the Department of Informatics at Aristotle University. He is author of two books and co-author of another five which are published in Greece and he has managed more than six multimedia projects. His scientific interests concern human-computer interaction, multimedia and web design, educational technologies and distance learning.

A. Pombortsis received a BSc degree in physics and an MSc degree in electronics and communications (both from the University of Thessaloniki), and a diploma degree in electrical engineering from the Technical University of Thessaloniki. In 1987 he received a PhD degree in computer science from the University of Thessaloniki. Currently, he is professor in the Department of Informatics, Aristotle University Of Thessaloniki, Greece. His research interests include computer architecture, parallel and distributed computer systems, and multimedia systems.

Žini u perdavimas ir instruktoriaus vaidmuo dirbant su kompiuterine mokymosi aplinka

Athanasis KAROULIS, Andreas POMBORTSIS Šiame straipsnyje supažindinama su Leonardo da Vinči programos projektu kompiuteriu pagr istam mokymui skatinti. Šis projektas vykdytas trijose ES šalyse: Graikijoje, Vokietijoje ir Olandijoje. CBL (Computer Based Learning -kompiuterinė mokymosi aplinka) aplinkos "Orestis" kompaktinė plokštelė buvo skirta jauno amžiaus nusikaltėliams išmokyti pagrindini u bei sudėtingesni u igūdži u dirbant su grafikos, nuotrauk u apdorojimo programine iranga. Eksperimentas vykdytas trimis etapais: 1) parengta kompaktinė plokštelė, 2) projekte dalyvavusiose šalyse pagal ši a plokštel e buvo pritaikyta mokymo metodika bei 3) ivertinti rezultatai. Šiame straipsnyje pirmiausia išryškinama pati CBL aplinkos s avoka, lyginant j a su multimedija bei nuotoliniu mokymu bei pateiktas, o taip pat pateiktas šios aplinkos teorinis pagrindimas. Pagrindinis nagrinėtas klausimas buvo tai, ar šios aplinkos pritaikymas yra tiek pat veiksmingas, kai yra pateikiamas ne dalyko žinovo ir "mokytojo", bet "instruktoriaus", turinčio aiškius, straipsnyje aprašytus, tikslus. Norėdami sužinoti atsakym a i š i klausim a mes parengėme ir sukūrėme minėt aj a kompleksin e CBL aplink a bei, tam kad ji būt u efektyviau taikoma, parengėme speciali a mokymo metodik a. Šios aplinkos taikymas patvirtino mūs u hipotez e apie "instruktoriaus" pranašum a, be to ištyrėme skirting u vartotoj u grupi u, dirbusi u su šia aplinka, pasiektus rezultatus bei ivairiomis s alygomis priimtus sprendimus. Taip pat mes nagrinėjome CBL mokymo ir realaus veikimo realioje aplinkoje kombinacij a, CBL aplinkos taikym a apskritai bei pateikėme statistinius duomenis apie du bandomuosius tyrimus bei penkis šios aplinkos pritaikymus trijose Europos šalyse. Baigėme straipsn i tvirtindami, kad šioje situacijoje instruktorius būtinas, o ir ištekli u jis reikalauja kur kas mažiau nei "tikras" srities žinovas. Vis dėlto, reikia dar kart a pabrėžti, kad čia kalbama konkrečiai apie mokomosios medžiagos, padedančios instruktoriui, taikym a; o pats mokymo procesas pagr istas hipermedija papildyta CBL aplinka, kuri, remiantis atitinkamomis edukacinėmis teorijomis, sukurta tam tikrai vartotoj u grupei.

Informatics inEducation, 2003, Vol. 2, No. 2, 241-256

Acknowledgements

The authors want to thank all the contributor organizations to the success of this project: the Law Department of the Aristotle University of Thessaloniki -Greece; the Multimedia Laboratory of the Department of Informatics of the Aristotle University of Thessaloniki -Greece; the Technologiezentrum Informatik of the University of Bremen -Germany; the Institut für Kriminalpolitikforschung of the University of Bremen -Germany; the Juristic Faculty of the University of Leiden -Holland; the Ministry of Justice of Hungary.

Many people have worked for these organizations. The listing of their names would take up an enormous amount of space, so here we want to thank all of them cordially for their co-operation.


Ana Iglesias
email: aiglesia@inf.uc3m.es

Paloma Martínez

Fernando Fernández

An Experience Applying Reinforcement Learning in a Web-Based Adaptive and Intelligent Educational System

Keywords: web-based adaptive and intelligent educational systems, intelligent tutoring system, reinforcement learning, curriculum sequencing

The definition of effective pedagogical strategies for coaching and tutoring students according to their needs is one of the most important issues in Adaptive and Intelligent Educational Systems (AIES). The use of a Reinforcement Learning (RL) model allows the system to learn automatically how to teach to each student individually, only based on the acquired experience with other learners with similar characteristics, like a human tutor does. The application of this artificial intelligence technique, RL, avoids to define the teaching strategies by learning action policies that define what, when and how to teach. In this paper we study the performance of the RL model in a DataBase Design (DBD) AIES, where this performance is measured on number of students required to acquire efficient teaching strategies.

Introduction

Web-based education (WBE) is currently a hot research and development area. They have two useful benefits: classroom independence and platform independence. Traditional web-based courses usually are static hypertext pages without student adaptability. However, since last ninetieths, several research teams are implementing different kinds of adaptive and intelligent systems for WBE [START_REF] Brusilovsky | Adaptive and intelligent technologies for web-based education[END_REF].

Web-based Adaptive and Intelligent Educational Systems provide intelligence and student adaptability, inheriting properties from Intelligent Tutoring Systems (ITS) and Adaptive Hypermedia Systems (AHS). "Intelligent Tutoring Systems (ITSs) are computer-aided instructional systems with models of instructional content that specify what to teach, and teaching strategies that specify how to teach" [START_REF] Wenger | Artificial Intelligence and Tutoring Systems[END_REF]. Adaptive Hypermedia Systems adapt the content of a hypermedia page to the user's goals, knowledge, preferences and other user's information for each individual user interacting with the system [START_REF] Brusilovsky | Adaptive and intelligent technologies for web-based education[END_REF].

Both, ITSs and AHS, share one of the oldest problems at educational systems: the ITS Curriculum Sequencing, which goal is to provide the student with most suitable individually planned sequence of knowledge units to learn and sequence learning tasks (examples, questions, problems, etc.) to work with the AHS Adaptive Navigation Support (ANS) technology, which goal is to support the student in hyperspace orientation and navigation by changing the appearance of the pages and their visible links.

In Curriculum Sequencing, pedagogical strategies must specify how the content is sequenced, what kind of feedback is provided, when and how to coach, explain, summarise, give an exercise, problem, question or analogy, etc. The choice of the best strategy has been widely studied, but there exist some drawbacks in their definition [START_REF] Beck | Learning to teach with reinforcement learning agent[END_REF]. On the one hand, it is expensive to encode all of the pedagogical rules that the system requires to teach effectively. On the other hand, it is difficult to incorporate all the knowledge of the human tutor and to decide how much strategies are necessary, the differences among them, the moment to apply them, why they fail, how to solve them, etc. To overcome the difficulty derived from the definition and election of pedagogical strategies, other artificial intelligence techniques have been incorporated, like semantic nets, neural nets, and others [START_REF] Murray | Authoring intelligent tutoring systems: an analysis of the state of the art[END_REF].

In this paper, we eliminate the pedagogical strategy concept using a knowledge representation based on a Reinforcement Learning (RL) model [START_REF] Kaelbling | Reinforcement learning: a survey[END_REF] that allows AIESs to adapt tutoring to student's needs, sequencing the content in an optimal way based on the student's performance, lesson objectives and the relationships between course modules, avoiding to define all static and predefined pedagogical strategies for each student [START_REF] Iglesias | Learning to teach database design by trial and error[END_REF]. The RL is an artificial intelligent technique that is able to find the optimal behaviour policy for a system, based only in previous experience. In this work, an example of an DataBase Design (DBD) AIES is presented and some experiments on how the tutor learns to teach by trial and error at the same time that simulated students learn AIES's contents are explained, where the main goal is to determine how many students are necessary for the AIES convergence to the optimal policy on teaching the system's contents. This will allow to conclude if it is viable to apply this method in real tutoring systems applied on real students.

The paper is organised as follow: first, the architecture of the DBD AIES used in this work and the reinforcement learning model are described in Sections 2 and 3 respectively. In Section 4, the definition of the DBD AIES system as a reinforcement learning problem is defined. Then, different experiments using different learning parameters of the system are presented in Section 5, and finally, the main conclusions and further research of this work are given.

Architecture of an Adaptive Intelligent Educational System

A typical structure of an ITS, and hence, of an AIES, is composed of four well differentiated modules [START_REF] Burns | Foundations of intelligent tutoring systems: an introduction[END_REF]. The student module contains all important information about the student in the learning process: student knowledge, personal characteristics, historical behaviour, etc. The interface module facilitates the communication between the AIES and the student. The domain module contains all characteristics of the knowledge to teach. Finally, the pedagogical module decides what, how and when to teach the domain module contents, taking the better pedagogical decisions according to the user needs.

Student Model

It is necessary, for the effectiveness of this proposal, to construct a good student model and to classify learners according to their critical characteristics in learning the AIES knowledge. A great variety of student models and techniques have been studied [START_REF] Sison | Student modeling and machine learning[END_REF], and any classification technique, like C5.0 [START_REF] Quinlan | C4.5 Programs for Machine Learning[END_REF] or Reinforcement Learning techniques [START_REF] Beck | Advisor: a machine learning architecture for intelligent tutor construction[END_REF], could be used to assort students according to their learning characteristics. That is to say, the learning process of all the students of a certain class will be similar.

For instance, three classes of students can be defined, depending only on two characteristics (in order to simplify the example): what kind of tasks must be used in order they learn (definition, introduction and problem) and what format the material must be shown (video, text, etc.). Table 1 shows the three classes of students, where the 100% of the class1 students require the definition task and the video format to learn. Class2 students require, like class1 students, only the definition task, but a 95% of them learn only with the video format and at the other 5% learn only with text format. Finally, class3 students require definitions and introductions, both with a percentage of 50%, and the video and text format with a percentage of 75% and 25% respectively.

Domain Model

AIES knowledge is stored in the domain module. The traditional knowledge structure based on topics, sub-topics, tasks, etc. could be an advantage in the pedagogical strategy in AIES. That is why we propose to use this hierarchy to define the tutor system's objectives. Fig. 1 shows a proposal of a hierarchical knowledge structure, where each topic has been divided into sub-topics, and these in others sub-topics, and so on. At the same time, each node of the tree contains sets of definitions, examples, problems, exercises, etc. in several formats (image, text, video, etc.). Database Design (DBD) domain have been proposed as example of an AIES domain. This domain has been included too in PANDORA project1 [START_REF] Castro | Integrating intelligent methodological and tutoring assistance in a CASE platform: the PANDORA experience[END_REF], whose main goal is to define methods and techniques for database development implemented in a CASE tool, useful for students and practitioners, where one of the modules of this project is concerned with tutoring student learning via web.

The knowledge model used for this system is based on the methodology proposed by Teorey [START_REF] Teorey | A logical design methodology for relation databases using the extended entity-relationship model[END_REF] in which three main phases in the database design are defined: conceptual, logical and physical phases. In order to simplify the example, we focus at the Entity Relationship (E/R) model [START_REF] Chen | The entity-relationship model -toward a unified view of data[END_REF], and, exactly, on the Conceptual Design topic. In Fig. 2, the AIES DataBase Design knowledge of the example is presented.

In order to experiment with the convergence of the Q-learning algorithm in this system, we have worked with two reduced knowledge model. The A-model of knowledge (see Fig. 3) presents only three topics, but each one of them has ten tasks (definitions, introductions and problems) that can be shown to the student in order to teach him/her this knowledge material. In the other hand, the B-model of knowledge presented in the Fig. 4 has eleven topics, but each topic only has two tasks for their execution, to be exact only one definition, but in two different formats: text and video.

Interface Model

In the interface model, the AIES adapts the content of a hypermedia page to the user's goals, knowledge, preferences and other user's information for each individual user interacting with the system, changing the appearance of the pages and their visible links.

In the experiments presented at this paper, we have not used the interface model, because the system have interacted with simulated students. Now, we are building an interface model in order the real students can learn the system's contents.

Pedagogical Model

In the pedagogical model, the AIES finds the best way to teach the knowledge items, corresponding with the internal nodes of the tree (topics), to the current student. The definition of this problem as a Reinforcement Learning problem is fulfilled in the Section 4, where a guide to apply Reinforcement Learning model in Intelligent Tutoring Systems is given, defining the RL components introduced in next section.

Reinforcement Learning

Reinforcement Learning (RL) is defined as follows [START_REF] Kaelbling | Reinforcement learning: a survey[END_REF]: "an agent is connected to its environment via perception and action. On each step of interaction the agent receives as input, i, some indication of the current state, s, of the environment; the

Q(s, a) = (1 -α)Q(s, a) + α {r + γ max a Q(s , a )} - Sets to s
agent then chooses an action, a, to generate as output. The action changes the state of the environment, and the value of this state transition is communicated to the agent through a scalar reinforcement signal, r. The agent's behaviour, B, should choose actions that tend to increase the long-run sum of values of the reinforcement signal. It can learn to do this over time by systematic trial and error, guided by a wide variety of algorithms."

Q-learning Algorithm

Updating the action policy can be performed by different algorithms, for instance, the Q-learning algorithm [START_REF] Watkins | Learning from Delayed Rewards[END_REF]. This algorithm allows to learn directly from the experience (systematic trial and error) with others students with similar learning characteristics. It is based on the value-action function, Q(s, a), to define its action policy. One of the main characteristics of this algorithm is that it does not require that system executes optimal sequences to converge [START_REF] Mitchell | Machine Learning[END_REF], being able to learn the Q function and, therefore, the optimal policy. This is very important for ITS, given that learning is based on the interaction with users, and this implies a high cost.

The Q-learning algorithm is described in Table 2. It requires the definition of the possible states, S, the actions that the agent can perform in the environment, A, and the rewards that it receives at any moment for the states it arrives to after applying each action, r. It dynamically generates the action-value table, Q(s, a), that allows to follow a potentially optimal policy. The γ parameter controls the relative importance of future actions rewards with respect to new ones, and α parameter is the learning rate, that indicates how quickly the system learns.

Application of Reinforcement Learning in Adaptive and Intelligent Educational Systems

In this section, we define how the Reinforcement Learning model is applied in Adaptive Intelligent Educational Systems and how the Q-learning algorithm is adapted to the tutor domain.

Reinforcement Learning Components

In this section, the components of a reinforcement learning problem in the database design adaptive intelligent educational system environment are defined.

Set of states (S).

The state is defined as the description of the student knowledge. It is represented by a vector which stores as many representative values of learner knowledge items (internal node of the knowledge tree) as is wished the student learns. In order to simplify the example, it is supposed that these values are defined in the set {0,1}. The items of the knowledge tree are enumerated in an pre-order way. The zero value indicates that the student does not know the item, and the one value indicates that the item has been correctly learned.

Set of actions (A).

The actions that the tutor can execute are those that teach the ITS subject, i.e., to show the knowledge items defined in the leaves of the knowledge tree. It is allowed to show macro-actions, that define a set of leaves of the tree that will be shown at the same time. It is necessary to know if a leaf of the tree has been shown to a student, because it affects on the state of the system. For instance, a student could learn a topic just after teaching him/her a leaf, but he/she could understand it thanks to the fact that previously, the system has shown him/her another topic. Thus, let us suppose a system with the macro-actions allowed to teach the topic 1 (E/R model) of the reduced A-model of knowledge model (see Fig. 3) defined in Table 3. There are actions composed by only one leaf and actions composed by several leaves. Let us suppose, that the student is in a state s and the system executes the action A1, i.e., the system shows to the student the definition one (def1). If the student would not learnt the item, the system have to execute another action that will content the previous one, for instance, action A7.

Perception of the environment (I: S → S).

This function indicates how the AIES perceives the state the student is into. In order to perceive the consequences of the execution of to evaluate the student knowledge, a given action is required. The only way an AIES could perceive the knowledge state of the student is through evaluating his/her 2). Each knowledge item in the ITS has associated a set of tests, that will allow to evaluate how much the student knows about the item. These tests will be centred in the last action executed by the AIES, assuming that other student capabilities evaluated before do not vary from the last evaluation. However, this could be false, varying them. This problem implies that an AIES could receive from the environment states with different levels of information, that could make the system not to differentiate some states from others, or perceiving two different states as equal (POMDPs: Partially Observable Markov Decision Problems), as it happens in teaching with human tutors. To avoid this problem, there will be necessary to accomplish continuous re-evaluations.

4. Reinforcement (R: SxA → R). This function defines the reinforcement signals (rewards) provided by the environment. This reinforcement function supplies a maximum value upon arriving to the goals of the tutor; in conclusion, the aim of the AIES will be to maximise its long term reward. For example, following knowledge tree defined in Fig. 3 a goal for the tutor would be that the user learns the "Entities" item, receiving only a positive reinforcement signal when the component of the state vector in this item is one. It is only in this moment when the teaching process is considered finished. The application of the reinforcement signal is a key issue in the learning technique, because to decide when and how to apply the reward is crucial for the learning system. The AIES learning is a delayed positive reward problem, given that it is impossible to know the kindness of the actual state perceived immediately after the performance of an action. So, the system does not immediately know whether it has achieved its goal after executing an action, since it might be only achieved (or not) after the execution of several ones. Furthermore, a higher reinforcement signal will be applied when the student may have learnt in less time and better (positive impact on achieving the goal).

Value-action function (Q: SxAxΠ → R).

This function estimates the usefulness of showing leaves of the tree to a student when he is in certain knowledge state. This function provides an evaluation method for the tutor action policies. Therefore, the goal of the learning process is to find the policy such that it maximises this function, i.e., to obtain the optimal value-action function.

In its learning, the AIES uses the state description (s) as input of the function approximator (Q), using this function to select a teaching action (a). After executing actions, the student changes its state to the new one perceived by the AIES (using tests). Depending on the kindness of the state reached in order to attain the goals, the system will receive a positive or null reward.

In [START_REF] Iglesias | Learning to teach database design by trial and error[END_REF] appears an example of the learning to teach process of an Database Design ITS with the Q-learning algorithm.

Adaptation of the Q-learning Algorithm

In this section, how the Q-learning algorithm is adapted to the tutor domain is explained. In Table 4 a comparison of the theoretical Q-learning algorithm and the adapted Q-learning algorithm to de AIES domain is presented. as follows:

Q(s, a) = (1 -α)Q(s, a) +α {r + γ max a Q(s , a )} → -Update the table entry for Q(s, a),
that estimates the usefulness of executing the a action when the student is in a particular knowledge state:

Q(s, a) = (1-α)Q(s, a)+ α {r + γ max a Q(s , a )} -Set s to s → -Let us s the current student knowl- edge state, s .

Experiments

In this section we present the experiments carried out on the AIES learning process. First, the learning variables are explained and finally, the experimental results are discussed setting our attention specially in how these variables affect the system learning convergence.

Learning Variables

It is very important to define the learning variables of our experiments and how they can vary. We have chosen four learning parameters to study:

• Domain size: In the experiments, we are going to use the A-and B-models of knowledge described in Fig. 3 and Fig. 4, respectively, and explained at Section 2.2.

The A-domain model has only three topics, but thirty actions to execute (tasks) and the B-domain model has eleven topics, but only twenty-two tasks.

• Determinism of students learning capability: When using the Q-learning algorithm in a given domain, it is very important to define if the execution of an action when the agent is in a given state, does or not always arrive to the same state. When it does not achieve always the same state is because of noise associated to perception and actions. This implies that the reward of executing an action in a given state can be different depending on whether the next observed state is a goal state or not. In Adaptive and Intelligent Educational System, like at Intelligent Tutoring Systems, there exists indeterminism in actions, because their execution could arrive to different final states due, for instance, to the different students learning characteristics (the student can learn or not the item, with different results on the tests). In the experiments we have used the three student classes defined in Section 2.1, each of them with different determinism level.

• Exploration versus exploitation: Exploration implies to try new alternatives, while exploitation implies to use pre-acquired knowledge. It is important to define the exploration/exploitation system strategy because it influences the learning rate and its quality. In our experiments, we are going to use de e-greedy exploration policy, where the e parameter indicates the probability of choosing the action with the best value of Q(s, a) being at the s state at this moment. We are going to vary the e parameter from the value 1 (when the system choose the next action to execute randomly) to the value 0.1 (when the system choose the next action trusting in the Q values obtained by the previous experience).

• The learning rate (α): This parameter indicates how quickly the system learns.

We are going to vary the α parameter of the

Q update function (Q(s, a) = (1 - α)Q(s, a) + α {r + γ max a Q(s , a
)}) at Q-learning algorithm from the value 0.9 (when the system quickly learns) to the value 0.1 (when the system learns to slowly teach).

Experimental Results

In this section, the results of the experimentation varying the learning parameters of the system described in previous section are commented. All of these experiments are going to show how these parameters affect the convergence of the AIES learning, dealing with the goal of the AIES: to minimise the time spent in teaching process, using less actions as possible. Therefore, the goal of these experiments is to analyse the number of students needed to use the system until the AIES learn to teach optimally. We divide the experiments in two groups: the experiments with the A-model of knowledge, and the experiments with the B domain model:

A-model of Knowledge

In the first set of experiments (see Figs. 5, 6 and7), we have used the A-model of knowledge shown in Fig. 3 and we have fixed the e parameter of the e-greedy exploration policy to 1 (the system randomly choose the next action to execute). The goal is to obtain the learning curve of the AIES in order to study how many students must work with the AIES in order the AIES learns an optimal pedagogical strategy on teaching policy for each class of students.

• Fig. 5 shows learning evolution of the system interactions with class1 students (see Table 2). We can see different learning evolution depending on learning rate α. We can observe that when de value of α is high, the curve decreases faster than when de value of α is low, because the system learns slower. Furthermore, the figure shows that the average number of actions needed to arrive to the AIES goal is 3, and that the system learns to teach optimally all topics when only six students have interacted with de tutor if α > 0.5. When α = 0.1, the system needs 15 class1 students to effectively teach.

• In Fig. 6 the AIES learning evolution for interactions with class2 students (see Table 2) is represented. From this figure, we can affirm that the best policy for this kind of student needs to show an average of four tasks, and when the learning rate (α) is equal to 0.1, the system learns slowly. When the system has interacted with 63 students, it has not learn the best policy.

• In Fig. 7, it is represented the system learning evolution when the tutor is interacting with students of class3 (see Table 2). It is shown that the actions for these students have a high level of indeterminism, and this is why the curve has a lot of "pikes", that represent changes at the action policy of the system. Moreover, we can observe how the system learns more constantly when α = 0.1, obtaining the optimal policy when the system needs six actions to arrive the goal with 36 students.

Next, we have modified the e parameter of the e-greedy exploration policy, and we have used the reduced A-model of knowledge in Fig. 3.

• In Fig. 8 we present the learning evolution of the system, where class3 students interact with the tutor. We have fixed the learning rate parameter (α) to 0.9 (the system learns quickly). In this figure we can observe that when e = 0.5, the tutor keeps in a local minimum (of eight actions), because it does not explore enough, and that is why is does not reach the optimal policy (six actions) when 72 students have interacted with the system.

• In Fig. 9, a comparison of the learning curve when the three classes of students interact is presented. We have fixed in this experiment the parameters e = 1(randomly) and α = 0.9 (learns quickly). Seeing the graphic, we can conclude that the tutor learns slower when indeterminism in actions exists.

B-model of Knowledge

In the second set of experiments, we have used the reduced B-model of knowledge shown in Fig. 4 and we have fixed the e parameter of the e-greedy exploration policy to 1, as well as the previous experiment. Due to students usually implies non-determinism in actions, in the following experiments, we have used only class2 and class3 students.

• In Fig. 10, class2 students have interacted with our system. We can compare the learning evolution of Fig. 10 with the Fig. 5, because they are similar: when the value of α is high, the curve decreases faster than when de value of α is low, because the system learns slower. Furthermore, the average number of actions needed to arrive to the AIES goal is around 30, and the system learns to teach optimally all topics when only 18 students have interacted with de tutor if α > 0.5. When the system have interacted with 18 students and the learning rate (α) is 0.1, the systems needs to teach all the topics executing approximately 60 tasks, that is not a very good teaching policy.

• In Fig. 11, class3 students have interacted with the AIES, and, again, we have modified the α parameter for the learning evolution study. In this figure we can clearly see how the learning rate affects to the system learning: we observe that the tutor learns more quickly when α is close to 1 and the tutor learns slower when the parameter is close to zero. In this way, the learning curve when α = 0.9 converges to more or less 30 actions when 12 students have interacted with the system; when α = 0.5 the curve converges to the same value when 45 students have interacted; and when α = 0.1 the evolution curve have not yet converge when 66 students have interacted with the AIES.

Then, we have modified the e parameter of the e-greedy exploration policy, where we have used the B-model of knowledge in Fig. 4. In Fig. 12 we present the learning evolution of the system, where class3 students interact with the tutor and we have fixed the learning rate parameter (α) to 0.9. We can see at this figure that the tutor keeps in local minimums when the exploration is not complete (when the e parameter is different that 1 (completely random)): when e = 0.5, the tutor keeps on the average of 70 actions; when e = 0.5, the tutor keeps on 100 actions, but at the 69 students, the system changes its action policy and decrease to 78 actions; finally, when the tutor randomly chooses the action, the system achieves the optimal policy of around 30 actions.

To conclude, we have done experiments comparing the learning curve when class2 and class3 students interact with the system. For this experiment, we have fixed the parameters e = 1(randomly) and α = 0.9 (learns quickly). In Fig. 13 we can see how the two curves converge at the same time.

Conclusions and Further Research

In this paper we propose the use of a Reinforcement Learning model that avoids the problems derived from construction of different pedagogical strategies for each student in adaptive intelligent educational systems. This model consists on defining the pedagogical module of any AIES (domain independent) as an action policy learned by trial and error (as human tutors do), applying the reinforcement learning model to the pedagogical module of the Adaptive Intelligent and Educational System. The use of reinforcement learning techniques provides many advantages on the techniques of traditional dynamic planning. First, it avoids the cost associated to rules construction and heuristic creation for each teaching strategy adapting to the needs of each student. Second, it allows the system to individually adapt to the student in real time, based only on previous information of interactions with other students with similar characteristics. Third, the system is able to adapt to the user not only deciding what to show, or when to show a given knowl-edge, but also how to do it. Fourth, the system updates tutoring models that serves to generalise teaching interaction over several students of a particular classification. Finally, it is considered a general technique, that can be applied in any ITS independently of the domain. This paper exposes an example of an AIES (a database design adaptive and intelligent educational system) and system executions with simulated students. We have done experiments with some parameters of the learning algorithm in order to study their effects at the convergence of the system and, at the same time, we have experimentally shown that AIES can learn an optimal policy to teach students interacting with reasonably few students. This makes possible its implementation in AIES with human students.

Actually, we are involved in the evaluation of the system with real students instead of simulated students and the study of hierarchical or compositional reinforcement learning algorithms.

Fig. 1 .

 1 Fig. 1. Hierarchical structure of knowledge.

Fig. 3 .

 3 Fig. 3. A-model of knowledge.

Fig. 4 .

 4 Fig. 4. B-model of knowledge.

A1

  = to show the Definition in a Text format A2 = to show the Definition in a Video format A3 = to show the Introduction in a Text format A4 = to show the Introduction in a Video format A5 = to show the Problem in a Text format A6 = to show the Problem in a Video format A7 = to show the A1+A3 at the same time A8 = to show the A2+A4 at the same time A9 = to show the A1+A5 at the same time A10 = to show the A2+A6 at the same time knowledge by tests (presented by shadow shapes in the Fig. 1 and Fig.

Fig. 5 .

 5 Fig. 5. AIES learning curve for A-model of knowledge students and 1-greedy exploration policy interacting class1 students.

Fig. 7 .

 7 Fig. 7. AIES learning curve for A-model of knowledge students and 1-greedy exploration policy interacting class3 students.

Fig. 8 .

 8 Fig. 8. AIES learning curve for A-model of knowledge students and α = 0.9. Curve varying the e parameter of the 1-greedy exploration policy.

Fig. 10 .

 10 Fig. 10. AIES learning curve for B-model of knowledge students and 1-greedy exploration policy interacting class2 students.

Fig. 11 .

 11 Fig. 11. AIES learning curve for B-model of knowledge students and 1-greedy exploration policy interacting class3 students.

Fig. 12 .

 12 Fig. 12. AIES learning curve for B-model of knowledge class3 students and α = 0.9. Curve varying the e parameter of the 1-greedy exploration policy.

Fig. 13 .

 13 Fig. 13. AIES learning curve for B-model of knowledge students α = 0.9 and e = 1. Comparison of the indeterminism in actions for different students.

Table 1

 1 Student classes and their characteristics

	Student Classes	Tasks	Formats
	Class 1	Definitions	Video
	Class 2	Definitions	Video (95%) & Text (5%)
	Class 3	Definitions (50%) & Introductions (50%)	Video (75%) & Text (25%)

Table 2 Q

 2

		-learning algorithm
		Q-learning Algorithm
	Do forever
	-	Select an action, a, and execute it
	-	Receive the immediate reward, r
	-	Observe the new state, s

•

For each pair (s ∈ S, a ∈ A), initialise the table entry Q(s, a)

Observe the current state, s -Update the table entry for Q(s, a) as follows:

Table 3

 3 Macro-actions for topic1 of the A knowledge model

Table 4

 4 Adaptation of the Q-learning algorithm to AIES domain

		Q-learning Algorithm		Q-learning adapted to AIES domain
	•	For each pair (s ∈ S, a ∈ A), initialise	→ •	For each pair (s ∈ S, a ∈ A), initialise
		the table entry Q(s, a)		the table entry Q(s, a)
		Observe the current state, s	→	Test the current student knowledge, s .
		Do forever	→	Do forever
		-Select an action, a, following an	→	-Select a set of knowledge tree
		explotation strategy, and execute it		leaves, a, to show to the student.
		-Receive the immediate reward, r	→	-Receive the immediate reward, r.
				A positive reward is received when
				the AIES goal is achieved. A null
				reward is obtained in any other
				case.
		-Observe the new state, s	→	-Test the current student knowl-
				edge, s .
		-Update the table entry for Q(s, a)

CASE Platform for Database development and learning via Internet. Spanish CICYT project (TIC99-0215).

Fig. 2. AIES DataBase Design (DBD) knowledge.

Acknowledgements. We thanks Juan Carlos Martínez for his support on the implementation of the Q-learning algorithm.

A. Iglesias Maqueda got a degree in computer science from Universidad Carlos III of Madrid in 1999. Since 2000, she works as assistant lecturer at the Advanced Databases Group in the Computer Science Department at Universidad Carlos III of Madrid. She is currently teaching database design and advanced databases. She has been working in several national research projects about advanced database technologies and CASE environments. Her research interests include adaptive intelligent educational systems, machine learning and database design.

P. Martínez

Sustiprinto mokymo modelio taikymas žiniatinkliu pagr istose adaptyvinėse ir intelektinėse mokymosi sistemose

Ana IGLESIAS, Paloma MARTÍNEZ, Fernando FERNÁNDEZ Vienas svarbiausi u uždavini u taikant adaptyvines ir intelektines mokymo sistemas (AIES) yra pasirinkti tinkam a pedagogin e student u mokymo strategij a. Taikant sustiprinto mokymo model i sistema automatiškai išaiškina, kaip mokyti kiekvien a student a individualiai, remiantis jau igyta patirtimi mokant kitus panaši u sugebėjim u studentus, -lygiai kaip elgt usi ir mokytojas. Sustiprinto mokymo modelio taikymas nereikalauja išlaid u susijusi u su taisykli u kūrimu ar kiekvienos mokymo strategijos euristine analize, kuria būt u siekiama išsiaiškinti ko, kada ir kaip reikėt u mokyti kiekvien a student a. Be to, sustiprinto mokymo modelis leidžia sistemai taikytis prie studento individualiai -išanalizuojamas prieš tai vyk es panašaus pajėgum u mokini u mokymas. Šiame straipsnyje aptarta adaptyvini u ir intelektini u mokymo sistem u struktūra bei išnagrinėta, kaip sustiprinto mokymo modelis yra taikomas bet kuriam adaptyvini u sistem u pedagoginiam moduliui (šios sistemos technika gali būti pritaikoma mokant daugelio disciplin u), pasitelkiant Q-learning algoritm a. Pateiktame pavyzdyje paaiškintas sustiprinto mokymo modelio vaidmuo, kai sistema taiko bausmes nepažangiems studentais. Su kai kuriais algoritm u mokymosi parametrais buvo atlikti eksperimentai, siekiant ištirti j u darom a itak a sistem u konvergencijai, išsiaiškinti sistemos efektyvumo kriterijai bei patikslinta, kiek sistema ivykdė veiksm u mokydama studentus atitinkam u dalyk u. Šiame straipsnyje eksperimentiškai parodyta, kad sustiprinto mokymo modelio taikymas adatyvinėse mokymo sistemose leidžia šiai sistemai parinkti optimali a student u mokymo strategij a, ypač kai su šia sistema dirba nuovokūs studentai. Aprašyt aj i model i galima taikyti internete.