Gilbert Paquette
email: gpaquett@licef.teluq.uquebec.cairosca@licef.teluq.uquebec.ca

Ioan Rosca

An Ontology-based Referencing of Actors, Operations and Resources in eLearning Systems

We explore some of the multiple relationships between two very active research fields in eLearning and Knowledge Management research: educational modeling languages and ontologies. Our previous research projects in the last 10 years have shown the central importance of the association between the learning activities and the knowledge and skills that they target. Studies on this relationship have led to the concept, we will present here, of a semantically referenced educational function grouping actors, operations and resources or learning objects. The referencing method proposed here is both qualitative (structured by an ontology) and quantitative, using a bi-dimensional skill/performance metric to situate the mastery level of knowledge associated to an actor, an operation or a resource in a multi-actor learning scenario.

Introduction.

Functions are editable and executable graphic models representing multi-actor processes where activities are executed, using and producing resources. This editing process is similar to methods of task representation, flowchart or workflow modeling, and UML use cases and activities diagramming. When executed at run time, Functions facilitate the coordination of the participants who achie ve the represented activities, using and producing resources (so-called "learning objects"). The representation of the cognitive processes involved while executing a Function Map demands the semantic referencing of its components: the operations (virtual or actual activities), the actors (abstracted or concretized by participants), the instruments (virtual or concretized by different kinds of resources).

In this communication a summary of our previous work will be outlined. Then we will present the concept of a Function Model, central to the LORNET project, as a generalisation of IMS-LD, a specification for the educational modeling of multi-actor activities. The third part will focus on ontology referencing of components in a n eLearning Function Model and thus enabling eLearning systems to help users find persons, activities and resources that can enhance their knowledge and competencies. Then we will discuss the notion of competency equations to help improve the design, the support and the evaluation of learning activities.

Modeling Knowledge and Competency for eLearning

We will first briefly summarize our previous research on Instructional Engineering methodology, Educational Modeling Languages and Web-based eLearning delivery systems.

1.1 Central Instructional Engineering Operations

The fast evolution of learning technologies has multiplied the number of decisions one must make to create an eLearning system. While it is true that a majority of the first Web-based applications have been mostly ways to distribute information, more and more educators have become aware of the need to go beyond these simple uses of information and communication technologies. One of our contribution to this field has been to propose a new approach to Instructional Design [START_REF] Merrill | Principles of Instructional Design[END_REF], founded on cognitive science, labeled as "Instructional Engineering"; it is defined as: "a methodology that supports the analysis, the design and the delivery planning of a learning system, integrating the concepts, the processes and the principles of instructional design, software engineering and cognitive engineering". [START_REF] Paquette | [END_REF] We have used a knowledge modeling approach to define such an instructional engineering method. This effort, started in 1992, and has led to the actual MISA 4.0 method [START_REF] Paquette | Designing Virtual Learning Centers[END_REF] and to ADISA, a Web-based support system for designers using MISA. [START_REF] Paquette | Web-based Support for the Instructional Engineering of E-learning Systems[END_REF]. In one of the main tasks, the instructional designer constructs the structure of the activities, a network of Learning Events. A second step in the elaboration of the instructional model is to build a learning scenario for each Learning Unit (task 320). A learning scenario is a multi-actor process model where the actors are involved into learning or support activities using and producing resources for themselves or other actors.

Prior or in parallel to that, the instructional designer will à built a Knowledge Model that is a graphic structured representation of the concept, procedures, principles and facts in the domain to be learnt. For this, a graphic modeling editor, MOT, is used, as a standalone or as a module of the ADISA Web-based design workbench. A companion task consists in defining entry and target skills and competencies for different types of learners, associated to some of the knowledge units in the domain model. A competency is a statement that a learner can apply a cognitive, socioaffective or psycho-motor skill to unit of knowledge at a certain degree of performance When the learning units have been defined, a sub-model of the domain model, with its linked competencies, is associated to each learning unit providing a knowledge representation of its content. Later on, when the learning scenarios are defined, other knowledge sub-models can be assigned to the learning resources, in each scenario, providing a knowledge representation of their content.

Adding Knowledge Representation to IMS-LD

The work on Educational Modeling Languages [START_REF] Koper | Modeling units of study from a pedagogical perspective -The pedagogical metamodel behind EML Retrieved march[END_REF][START_REF] Rawlings | Survey of Educational Modelling Languages (EMLs), version 1[END_REF] , and the subsequent integration of a subset in the IMS Learning Design Specification [START_REF] Ims-Ld | IMS Learning Design. Information Model, Best Practice and Implementation Guide, Binding document, Schemas[END_REF], is the most important initiative to date, to integrate Instructional Design preoccupations in international standards [START_REF] Wiley | Connecting learning objects to Instructional design theory: a definition, a methaphor, and a taxonomy[END_REF]. It describes a formal way, and an XML binding that can be read by any compliant eLearning delivery system, to represent the structure of a Unit of Learning and the concept of a pedagogical Method specifying roles and activities that learners and support persons can play using learning objects.

The learning Design is composed of a method, learning objectives and prerequisites, and metadata referencing the unit-of-learning as in the IMS content packaging specification. The Method element and its sub-elements are central as they control the behavior of the unit-oflearning as a whole, coordinating the activities of the actors in the various roles and their use of resources. Like MISA's instructional model, it is a multilevel structure where altenative plays are decomposed in a series of one or more acts, each act being composed of one or more role -parts associating an actor to a single activity or an activity structure that can be decomposed further.

This IMS-LD Function on figure 1 is composed of two alternative plays, Module A and B, intended for different users. Only the first play is developed here, it contains two acts and 5 learning activities, ruled by individual learners or group of learners, and support activities by a teacher. Each couple Role/Activities correspond to IMS-LD role parts. Resources or learning objects link to activities (by IP links) are used or produced in the corresponding activities. While IMS-LD is a great progress in eLearning specifications, norms and standards, we believe it has to be expanded, first to a more general Function model that we will present in section 2, and also to associate knowledge and competencies to its components as we do in MISA. Actually, the only way to describe the knowledge in the activities or in the resources is to assign optional educational objectives and prerequisites, to the Unit of learning as a whole and/or to all or some of the learning and support activit ies. Objectives and prerequisites, although they correspond to entry and target competencies, are essentially unstructured pieces of text composed according to the IMS RDCEO specification [START_REF] Ims | IMS Reusable Definition of Competency or Educational Objective -XML Binding[END_REF].

Figure 1 -An IMS-LD Method

Unstructured texts are difficult to compare: consistency checking cannot be supported by a system between different levels of the LD structure, and even, at the same level, between the content of learning activities and resources, and the actors' competency. In fact, the content or learning resources is not described at all, and the actor's competencies are only indirectly defined by their participation in learning units or activities where educational objectives are associated. What we need is both a qualitative structural representation of knowledge in activities and resources, but also a quantitative one that can be provided by adding a metric to knowledge elements.

1.3 Adding Knowledge Representation Capabilities to a Delivery System

Without any representation of the knowledge to be processed in a eLearning environment, a delivery system will be unable to help its users according to their present and expected state of knowledge and competency acquisition. To expand the capacity of such systems, the Explor@-2 delivery system [START_REF] Paquette | Designing Virtual Learning Centers[END_REF]) has been based from its inception in on two structures, the instructional structure (corresponding to the learning design) and the knowledge/competency structure (corresponding to a domain ontology).

Figure 2 shows screens of the knowledge/competency editor provided to Explor@-2 designers. On the left, we see a hierarchy of concepts in the domain of ecological agriculture where the terminal nodes are skills associated to their parent nodes. If we select one of the skills (on the figure, "Analyze-6" is selected), plus some performance criteria (selected in the little window on the left), we can associate to its parent knowledge, here " Agriculture Processes", a target competency and an entry competency.

Figure 2 -A Knowledge and Competency Editor

Competency statements are texts that have a precise interpretation as knowledge/skill+performance couples. For example, the competency statement "classify the agriculture processes according to their greenhouse effect, autonomously in easy and difficult situations", is interpreted as analyze (skill level 6), autonomously in all situations (performance level B) to be applied to agriculture practices (knowledge). The skill and the performance levels form two ordered sets of value that enable comparison between competency statements for a certain knowledge. Performance levels A, B, C and D are derived from a combination of increasingly demanding performance criteria and can be transformed in numbers (A = 2, B = 4, C = 6 and D = 8) so as to obtain a metric enabling to represent the distance between entry and target competency for that knowledge (agriculture practice). This means that learners will stay at the "Analyze-6" level, with an expected increase of performance from A to B, that is from 6.2 to 6.4.

On figure 4, the lower right corner of the Knowledge/Skill editor is where the designer associates the selected knowledge/skill+performance)/ couples (the entry and target competency statements) to the components of the activity structure that has been built using the Explor@-2 activity editor. In this way, all the activities and the resources can be described by knowledge and competencies.

Function Models

There are limitations to the semantic referencing method we have just presented. The most important is that Explor@-2 do not support multi-actor processes and no competencies are assigned to actors except indirectly by competencies assigned to activity structure components. It is now our goal to both generalize the activity structure using a multi-actor workflows called Function Models, and also to reference later on, the actors, operations and resources in these Functions Maps, with Ontology components (for the knowledge part) and a competency metric.

Function Models as Multi-actor Workflows

A Function Model (or simply Function) within a eLearning system is composed of actors playing a role in activities (or operations) where they use and produce resources for themselves or other actors [START_REF] Paquette | Organic Aggregation of Knowledge Objects in Educational Systems[END_REF]. From a computer science view, a Function Model corresponds to a use case (Bosch et al. 2001) of that system. From a conceptual point of view, in a biological or ecological sense, a function is a particular physiology, an interesting subsystem of operations within of the learning system organism. Particular cases are instructional scenarios, delivery models and IMS-LD methods.

For example in a delivery model , learners are work colleagues can use an electronic performance support system (EPSS) offering integrated training and work activities. They would equipped with the same databases, documents, and tools as those used at work. These organizational resources are provided and supported by the workpla ce technicians. The learners acquire knowledge and competencies by solving problems similar to those experienced in the workplace. Learners use hyperguides that provide activity assignments to be completed with the training material published on the Internet. The learning material is created and maintained by the training organization designers. The target competencies are validated though various exercises and tests. A trainer-manager supervises the learners' work and training by providing advice and assessing their work and knowledge, skills and competency progress. All these operations can be represented as a multi-actor workflow where the different actors rule operations where the use resources and produce resources for other actors or with other actors.

Embedding Knowledge and Competency in a Function Model

We obviously need a unique semantic referential (for knowledge and competencies) for all the components of a Function that must reach competence equilibrium in a given domain. For example, if a le arning activity, to be achieved, requires a level of mastery of certain knowledge, then the resources provided to the learner (persons, documents and tools) must enable the learner to progress from a lower entry mastery level to the one required by the activity.

But if we use only a knowledge referential without précising the mastering level, we obtain a coarse granulation of sense and, as a consequence, weak semantic management services. We need measures of a knowledge mastering, a weighted ability defined on that knowledge. We can use different mastering scales: simple quantitative 1-10, Bloom taxonomy, combinations between skills taxonomies and performance level as in MISA, etc. It would be preferable that the level scale that describes the mastering of any knowledge be reasonably simplified, to be manageable. Still, the levels must correspond to clearly identify cognitive processes such as memorizing, applying, analysis, synthesizing or evaluating knowledge [START_REF] Paquette | Meta-knowledge Representation for Learning Scenarios Engineering[END_REF]. The evolution of a learner on a competence scale materializes a learning process: therefore, it should be managed explicitly and expressively.

Figure 3 -A Skill/Performance Scale

Figure 3 presents such a competency scale that is used in the Explor@ knowledge/activity editor presented above. It has also proved a solid tool for instructional engineering projects using the MISA method. Here the mastery of a knowledge term, Multimedia Production Method, is evaluated on bi-dimensional scale. The skills scale , from 0 to 10, is complemented by a performance scale, where values are decimals from 0 to 0.9, corresponding to qualitative terms like "Aware", "Familiarized", "Productive" or "Expert". On the figure, we see that PeterM's knowledge is evaluated at 8.3: he can synthesize MM production methods, at a performance level showing he his familiarized with synthesizing such procedures. On the other hand, Book X is evaluated near 9.7; that might be too much for the actual competency of PeterM, unless we aim a target competency at that level or higher. On the other hand, a lecture on VideoY is evaluated near 6.9 so it is below PeterM's actual competency and might not prove very useful, except maybe as a review.

The knowledge space of a domain can also be structured in many ways: dictionaries, thesaurus, book summary, library catalog, indexes and metadata, knowledge graphs, ontologies, etc. The tree organization of the knowledge referential seems an important aspect because it allows the competence inheritance of a parent node to his children, if these have no other explicit specifications. This can reduce significantly the mechanisms of competence analysis and management. But the tree organization is two restrictive for describing the rich network of relations that ties the concept structures. A relational (predicate) logic must then complete the concept tree and sustain more refined mechanism of conceptual matching.

These requirements suggest that a good candidate for the semantic indexing of the function components will be a combination between domain ontologies [START_REF] Davies | Towards the Semantic Web, Ontology-Driven Knowledge Management[END_REF][START_REF] Breuker | Ontological modelling for designing educational systems[END_REF] and a simple and operational competency scale.

Ontology Referencing of a Function Model

In the TELOS architecture framework we are developing in the LORNET 1 project, the construction of a domain ontology and its use for referencing actors, operations and resources represented in a Function, is central. It is in continuity with the work presented in section 1. We present here a tool and a process to integrate qualitative and quantitative ontology referencing in a Function Model.

Construction of a Domain Ontology

Some technologies and methods to develop the Semantic Web are now becoming mature. On February 10, 2004, the World Wide Web Committee has issued a recommendation for the definition of a standard Ontology Web Language: "OWL is a revision of the DAML+OIL web ontology language incorporating lessons learned from the design and application of DAML+OIL." (W3C, 2004) Many ontology engineering methods exists such as the one presented in [START_REF] Sure | A Methodology for Ontology-based Knowledge Management[END_REF]. We present here, as an example, an ontology aiming to identify Agriculture practices that influence the greenhouse effect. A function model (such as the IMS-LD method presented on figure 1) describes the use of the ontology to find out agriculture alternative practices, in at least five agriculture domains, and to build a transition plan towards the replacement of the old practices by the more ecological ones. The ontology would serve in a browsing mode to access related resources and to launch search agents to find persons, information resources and learning activities useful to solve the problems. Figure 4 presents a simple example to illustrate the use of the MOT+ knowledge editor, used in MISA, as a graphic tool for Ontology Engineering2 . The upper part of the graph presents the top levels of three hierarchies of concepts (linked by sub-class links "S"): agricultural practices, fertilizers and gases. Some properties of these concepts are shown on the graph. An Agriculture practice, such as Rice Production Processes, has inputs including fertilizers and outputs that can be gases. Fertilizers can also produce gases, some of which are greenhouse gases. Figure 4 also shows a few of the instances that will constitute the knowledge base. Here, we see an agriculture practice, Traditional Rice Production, having among its outputs methane gas. It also has Nitric Oxide amongst its inputs, a chemical fertilizer that produces Carbon Dioxide. Both of these gases are example of greenhouses gases harmful to the environment.

We can now proceed to reference the components of the Function Model presented on figure 5. For this, sub-graphs of figure 4 are associated to actors, acts and activities (operations) and resources used and produced by the activities.

Competency equations

We will now use the skill/performance scale presented above to show how we can add a metric to represent the mastery of knowledge terms that have been associated to operations, actors and resources of the IMS-LD scenario (or function) presented on figure 1. On this figure, we see that for a certain knowledge term in the ontology, for example "rice production processes", the target competency (TC) of activity 5.3 is evaluated at a 7.4 level (skill: "Repair", performance: B) on the skills/performance scale . Then the produced resource (an exam, an essay, a classification table,….) should show a TC level equal or higher than 7.4.

Since the learner has an entry competency (EC) of 5.2 (skill: "Apply", performance A), he needs help. Here we have a trainer with EC = 6.4 so he alone cannot bring the learner all the way up, but he can certainly help him fill part of the gap. Also, the learner can use two input resources. Resource A is at TC=5.2 so it can only serve to test the entry competencies of the learner, to make sure he has the prerequisites. Fortunately, Resource B has a TC = 7.4, the right target, but lets hope it not a lecture that starts at 7.4, but maybe an aggregate of learning objects that can help him progress with the help of the trainer. By the way the trainer will also learn a little bit in the process, so at the end the activity, we could consider raising his EC for the next run of the activity.

There are many other situations to investigate where competency equations such as these will prove useful, but this example shows that this kind of analysis, by humans, by machines or both, can bring more intelligence in learning environments before learning takes places (at design time), during learning (to help learners use available resources adequately) and after learning (to evaluate and improve designs).

Conclusion

By definition, domain ontologies are continuously evolving in the semantic Web. Furthermore, in communities of practice and project-based learning, the learners themselves can be the ones who will make the ontology evolve. This is why it is important to lower the barrier to ontology construction and referencing, taking in account that ontologies are a moving target. To achieve this, more user friendly graphic tools will have to be designed.

On a more theoretical level, the maintenance of the coherence of the Ontology through versioning by different actors is a huge problem researchers have just started to address. Our approach here is to classify the possible changes from a version to a subsequent one and to provide a tool that will assist the Ontology designer [START_REF] Rogozan | Methodology for the Ontology Evolution within a Dynamic and Distributed Environment[END_REF] in the different cases that can occur.

Finally, the knowledge referencing of Functions or multi-actor workflows using an ontology needs to be complemented by a knowledge mastery metric such as the one proposed here. This important aspect will be further investigated to better understand the multiple possible configurations.

The Semantic Web and its associated technologies and methods are recent developments that have barely begun to be used in applications. Extensive solution-oriented applied research in knowledge engineering and distributed computing is needed to adapt, integrate and evolve these technologies, and to produce a framework for building and using learning object (or resource) repositories on the Web, for knowledge management and learning.

Figure

  Figure 4 -A Simple Ontology

FigureFigure 5

 5 Figure 5 -Associating knowledge mastery values to scenario components

C C Persons, resources for the Classification of Agriculture Domains and Methods Module A (For beginners) IP Activity 2 Make a synthsis Activity 4 Present a Systhesis on Gaz Properties R Activity 5 Classify Gases by their presence in Agriculture Domain and methods IP Role 2: Group of Learners R Role 2: Group of Learners Role 1: Individual Learners R C C Role 3: Teacher Act 2 Analyze Gas Properties Activity 1 Explore the Agriculture Domains and Methods IP IP IP C Synthesis C Classification Table IP Lecture Notes and Web Sites Course on Ecological Agriculture Act 1 Synthetize domains and methods R P Activity 3 Evaluate and discuss synthesis Validated Synthesis Loosely Aggregated Information C Module B (For agriculture expert)

		R
		C
		C
	IP	IP
		IP
		IP

We are actually working towards having MOT+ produce automatically a standard OWL description of an application domain instead of its actual XML schema


Dietrich Albert
email: dietrich.albert@uni-graz.at

Christina M Steiner
email: chr.steiner@uni-graz.at

Representing Domain Knowledge by Concept Maps: How to Validate Them?

Keywords: concept map, semantic net, ontology, domain knowledge, validation, content validity, application validity

For one and the same domain several alternative concept maps may exist, originating from different world views or purposes. Some of these concept maps may be valid, however not all of them. Thus, strategies for empirically and objectively validating concept maps in the respective context are necessary. We outline two methodological approaches for empirically validating concept maps, one for giving evidence of content validity and one for application validity. One procedure is to validate a given concept map with concept maps systematically generated by others. As a second method we suggest to observe the behaviour and the performance in a relevant situational context as a validation criterion. In this scope, a method for predicting persons' problem solving behaviour by using a given concept map is outlined. In general, the purpose and ultimate use of a given concept map has to be taken into consideration for choosing a validation procedure and interpreting its results.

Introduction

Concept maps (semantic nets) provide a valuable tool for organising and presenting knowledge within an ontological framework. They serve a variety of purposes, especially in educational contexts (for an overview see e.g. [START_REF] Coffey | A Summary of Literature Pertaining to the Use of Concept Mapping Techniques and Technologies for Education and Performance Support[END_REF][START_REF] Novak | The Theory Underlying Concept Maps and How To Construct Them[END_REF], but e.g. also in the context of hypertexts (as tools for hypermedia design and navigation). Most commonly, a concept map is depicted by a labelled, directed graph with the vertices (nodes) representing concepts of a domain and the directed and labelled edges (arcs) representing the relationships that exist between those concepts. Figure 1 shows an example of a concept map, representing what a concept map is. The combination of two concepts and the link relating them forms a meaningful statement and therefore constitutes a proposition. Hence, another way of representing a concept map is a list of propositions. Furthermore, the information contained in a concept map may also be presented in form of a matrix, with the set of concepts labelling the columns and rows and the relations specified in the cells of the matrix.

Mathematically, a concept map can be defined as follows. A concept map is a directed graph consisting of a finite, non-empty set C = {c 1 , ..., c n } of nodes, representing concepts (or concept labels) and a finite, non-empty set A of arcs, representing the relations between concepts. Every arc in A is an ordered pair of concepts from the set C. The relations in a concept map are labelled; each relation label i defines a binary relation R i on the set of concepts. A concept c p is in relation R i to another concept c q , i.e. R i (c p ,c q ), if and only if there exists an arc in the concept map with the label i and with an arrowhead pointing to the second node of the ordered pair (c p ,c q ). Each arc is an element of one relation R i , the arc set A is the union of all relations R i .

Representations of semantic knowledge as described by superordination (e.g. [START_REF] Heller | Semantic Structures[END_REF][START_REF] Heller | Representation and Assessment of Individual Semantic Knowledge[END_REF], mind maps [START_REF] Buzan | The Mind Map Book: How to Use Radiant Thinking to Maximize Your Brain's Untapped Potential[END_REF], cognitive maps (e.g. [START_REF] Ackermann | Cognitive Mapping: Getting Started with Cognitive Mapping[END_REF] constitute special cases of a concept map, each featuring only one relation (is a, related to, leads to). Please note, that semantic knowledge representations known as e.g. semantic nets (e.g. [START_REF] Fisher | Mapping biology knowledge[END_REF] and knowledge maps (e.g. O 'Donnell, Dansereau, & Hall, 2002) are also in line with the mathematical definition of concept maps given above. Thus, although in the sequel we concentrate on concept maps, the given considerations and explanations actually also hold for those methods of knowledge representation.

For generating concept maps, aside from the traditional methods using paper and pencil or other devices, a variety of software exists, e.g. Inspiration (www.inspiration.com), Hozo (http://www.hozo.jp), CmapTools (http://cmap.ihmc.us), or Protégé-2000 (http://protege.stanford.edu). Utilising such software provides significant support in creating, manipulating, storing, and reusing concept maps. A concept map may represent a knowledge domain, e.g. for presenting learning material in an e-learning environment. A concept map is also able to represent personal knowledge, e.g. when a student is asked to generate a concept map specifying his/her personal understanding of a specific domain. Here, we focus on concept maps as domain knowledge representations.

Basically, two approaches can be distinguished in building a concept map for a particular domain, a normative and a descriptive one. When holding the view of a normative approach, it is assumed that there exists only one concept map of complete consensus for the domain. Contrary, according to the descriptive approach, it is assumed that there may exist alternative concept maps for a specific domain. In other words, for a given domain there may be different, but not arbitrary concept maps. Generally, the descriptive approach seems to be more reasonable instead of demanding 'the one correct' concept map. This is, because a concept map necessarily entails some sort of world view or opinion regarding a particular domain. Hence, concept maps representing the same domain may differ from each other. Moreover, differences may occur by reason of the intended purpose and ultimate use of concept maps.

One general aim in building concept maps is to obtain at least one of possibly several valid concept maps of a domain. A crucial question in this context therefore refers to the validity of a given concept map. As a result, a framework for evaluating the adequacy of a concept map or of different proposals for a concept map of a specific domain should be provided. This field of research, however, still lacks efficient strategies. In the present paper we elaborate on the question of validating concept maps representing domain knowledge. Before outlining two types of validity that we distinguish and two approaches for examining them, the topic of reliability of concept maps is shortly discussed.

Reliability of Concept Maps

Before addressing the validity of a given concept map, the issue of its reliability should be regarded. Test-retest reliability can easily be determined by having a concept map for the same domain generated again. Let us assume an expert of a specific knowledge domain who constructed a concept map for this domain. By asking the respective person in a different point of time to generate a concept map for the same domain again, the reliability of the concept map can be examined. For this, it is assumed, that there is no indication of a change in knowledge or understanding of the person. If both concept maps correspond to each other, they reliably represent (the domain or at least) the understanding of the respective person regarding the domain. If the expert generates a different concept map each time, though, it is obvious that no indication exists that one of them will be a reliable model of the knowledge in question. A given concept map can be described by attributing specific characteristics or features to the respective map. This is the case when scoring a concept map according to a particular scoring system (e.g. [START_REF] Novak | Learning how to learn[END_REF][START_REF] Ruiz-Primo | On the Use Of Concept Maps As An Assessment Tool in Science: What We Have Learned so Far[END_REF] that considers different map characteristics. Some of these features can be objectively and clearly determined, e.g. number of concepts, number of propositions. Contrary, there are also map characteristics that have to be determined by subjective ratings, e.g. proposition accuracy. The reliability of such ratings can be examined by using again the test-retest method. Another procedure refers to the interrater-reliability, i.e. the consistency of ratings assigned to a concept map (e.g. [START_REF] Ruiz-Primo | On the Use Of Concept Maps As An Assessment Tool in Science: What We Have Learned so Far[END_REF]. This aspect of reliability can be examined when having two or more judges that independently from each other score a given concept map.

Assuming again an expert that generated a concept map representing a particular domain, parallel-forms reliability can be examined by asking to construct a concept map in alternative forms of representation (e.g. as a directed graph and as a list of propositions). A reliable concept map should represent the same model of knowledge (for the domain or at least) for the respective person, regardless in which representation format it is generated.

Validity of Concept Maps

Having considered the reliability of a concept map, the aspect of validity can be addressed. A concept map representing domain knowledge constitutes a model of a part of the current knowledge about the world for a given domain in a given context. Such a model may serve e.g. for presenting learning material, for predicting problem solving performance etc. Thus, it is important to ensure, that a concept map is well founded and valid.

Subjective evidence for the correctness of a concept map is not enough for making a statement regarding validity. Even the principle of consensus with respect to the correctness of a concept map does not suffice. In fact, objective and empirical criteria are needed for giving evidence of the validity of a given concept map.

The validity of a concept map can be understood from two perspectives. On the one hand, it may be examined whether a concept map serves the purpose for that it has been designed. This aspect of validity could be denoted as 'application validity' of a concept map. It refers to the practical usability and usefulness of a concept map. Therefore, different kinds of intended application will require different means of validating a concept map.

Before examining the quality of a concept map by applying it for the purpose it has been generated, though, evidence needs to be given regarding its validity of the knowledge in question. This means, it has to be determined whether the concept map constitutes a valid model of a part of the current knowledge about the world. This aspect of validity we call 'content validity'. Content validity is an important issue, as the evaluation of the content of concept maps is critical for using them. Of course, the evaluation whether a concept map adequately reflects the respective knowledge will also need to take into account its intended purpose and ultimate use.

In the following sections we concentrate on both, content and application validity.

Content Validity

As a valuable approach for giving evidence of the content validity of a given concept map, we suggest to take empirically collected concept maps representing personal knowledge (in the sequel denoted as 'criterion maps') as a criterion. Concept mapping tasks are an appropriate way for eliciting persons' understanding of a domain. Thus, for gaining information regarding the content validity of a given concept map representing a particular domain, it may be compared with empirically gathered concept maps of individuals of different knowledge level, including experts. For this, the similarity between the given concept map and the criterion maps is examined.

This can be done by investigating the conformance of propositions. To this end, for instance 2x2 tables can be established for comparing the given concept map with a criterion map, detailing how many propositions are contained either in both or in only one of them. Based on this, in correspondence to [START_REF] Ruiz-Primo | On the Use Of Concept Maps As An Assessment Tool in Science: What We Have Learned so Far[END_REF] for example a so-called 'convergence score' (Con) can be calculated. Utilising this score for validating a given concept map representing domain knowledge, it would express the proportion of propositions in the given concept map out of the total number of propositions in one criterion map. In this case, the respective score is derived by Con = x / y k , whereas x denotes the number of propositions in the given concept map that are also contained in the criterion map k and y k denotes the total number of propositions in the criterion map k. Other statistical methods for determining the similarity between given concept map and criterion concept maps can be found e.g. in [START_REF] Goodman | Measures of association for cross classifications[END_REF] or [START_REF] Tversky | Features of similarity[END_REF].

Of course, such similarity measures can not only be determined for propositions, but also only for relations or concepts, respectively, if only those special parts of a given concept map are to be validated.

If in sum a high similarity between the given concept map and the criterion maps can be determined, this indicates, by definition, the content validity of the given concept map.

A given concept map may not necessarily have to be validated as a whole. Possibly only a part of a concept map has to be validated, e.g. the contained concepts or relations, a substructure of the map etc. In this case either only parts of the criterion maps are used for validation or only partial criterion maps are collected. Thus, depending on the validation objectives, a set of different procedures for posing a concept mapping task is available. Applying the 'construct-a-map' or 'map creation' method (e.g. [START_REF] Ruiz-Primo | On the Use Of Concept Maps As An Assessment Tool in Science: What We Have Learned so Far[END_REF], individuals are asked to generate a concept map concerning a specific knowledge domain from scratch -either by providing concepts and/or relations or not. This may be done by drawing the concept map by hand, by arranging note-cards, or of course by using suitable software as mentioned in the introductory section. The 'fill-in-themap' or 'map completion' technique, (e.g. [START_REF] Schau | Select-and-Fill-In Concept Map Scores as a Measure of Students' Connected Understanding of Science[END_REF] is characterised by providing a concept map of a particular domain to individuals, where all or some of the concepts and/or relations have been left out. The blanks then have to be filled in. The software CMap Pro1 (http://www.unisaarland.de/~su11pshb/forsch/cmap.html), for example, features the possibility of creating fill-in maps that can afterwards be presented and filled in directly on the computer display. One further alternative of a concept mapping task makes use of relatedness ratings between pairs of concepts (e.g. [START_REF] Schau | Select-and-Fill-In Concept Map Scores as a Measure of Students' Connected Understanding of Science[END_REF]. For this, an individual is asked to rate the degree of relatedness between pairs of previously defined concepts on a numerical scale. Through applying a mathematical algorithm, e.g. by using the Pathfinder software (http://interlinkinc.net/Pathfinder.html), the relatedness ratings of an individual can then be visually represented in form of a graph. Based on the resulting graph an individual may additionally be asked to label the arcs between concept nodes (Shavelson, Ruiz-Primo, & Wiley, in press) in order to get a concept map. Another kind of concept mapping task consists in presenting the propositions of a concept map as a correct-incorrect discrimination task, which should additionally include distractor items and confidence ratings [START_REF] Steiner | Beziehung zwischen Concept Maps und Kompetenzstrukturen[END_REF]. A version of this method -however without the possibility of including distractors or confidence ratings -has also been implemented in the software CMap Pro, mentioned before, namely for assessing pre-knowledge.

The different techniques of posing a concept mapping task vary considerably in the extend of constraints imposed and information provided to individuals. All of them are conceivable for empirically collecting criterion maps, when the content validity of a given concept map, or parts of it, is to be examined. Of course, each method has special characteristics with respect to validation. Which method is most suitable will depend very much on the particular validation issue and its requirements, e.g. whether a whole given concept map or only specific parts of it are to be validated. Furthermore e.g. the cognitive and the time demands will influence the method for obtaining criterion maps. In any case the purpose and context has to be taken into account when collecting criterion maps for validation.

Application Validity

For examining the application validity of a given concept map, we propose utilising situational performance as a criterion of validation. With situational performance we mean behaviour in real-world situations which does not consist in performing a concept mapping task. This could for example be problem solving, answering questions, or even social behaviour in given situations. It seems natural and obvious, that an individual's personal understanding of a domain is reflected in his/her behaviour and performance in given situations, at least as declarative knowledge is involved. Situational performance is therefore useable as a criterion for examining the validity of a concept map. For validating a given concept map, a kind of situational performance has to be chosen, that is related to the purpose and intended application of the given concept map. Due to different purposes (e.g. presenting learning material, describing social skills) that are intended for a given concept map, different kinds of situational performance (e.g. problem solving, behaviour in social situations) will be appropriate for validation. Thus, specific situational performance scores or profiles of individuals of different knowledge level, including experts, are collected and constitute the validation criterion.

Performance scores such as number of correct answers in e.g. multiple choice tests (e.g. [START_REF] Rice | Using Concept Maps to Assess Student Learning in the Science Classroom: Must Different Methods Compete[END_REF][START_REF] Ruiz-Primo | On the Use Of Concept Maps As An Assessment Tool in Science: What We Have Learned so Far[END_REF][START_REF] Schau | Select-and-Fill-In Concept Map Scores as a Measure of Students' Connected Understanding of Science[END_REF], standardised tests (e.g. [START_REF] Rice | Using Concept Maps to Assess Student Learning in the Science Classroom: Must Different Methods Compete[END_REF], or problem solving tasks (e.g. [START_REF] Steiner | Beziehung zwischen Concept Maps und Kompetenzstrukturen[END_REF]) have already been applied as criteria for validating personal concept maps. In sum, it could be shown that measures of situational performance are suitable for validating concept maps representing personal knowledge or at least, that individual concept maps and performance are somehow positively related, indicating application validity in this special case. Of course these performance measures can be used for validating any given concept map. However, measures like test scores constitute only summary validation criteria.

Instead of or in addition to these criteria, it would be desirable to apply more sophisticated performance profiles as criteria for validation. This is why we suggest to take Knowledge Space Theory [START_REF] Albert | Knowledge Spaces: Theories, Empirical Research, and Applications[END_REF][START_REF] Doignon | Knowledge Spaces[END_REF][START_REF] Falmagne | Introduction to Knowledge Spaces: How to Build, Test, and Search Them[END_REF]) as a suitable framework for validation. We propose a validation approach that utilises Knowledge Space Theory for predicting problem solving behaviour, i.e. for predicting answer patterns, based on a given concept map. The given concept map then can be validated by comparing the predicted answer patterns with empirically obtained answer patterns. In the following section a short introduction into the basic notions of Knowledge Space Theory is given. Subsequently, the validation approach mentioned above will be outlined in more detail.

Basic Notions of Knowledge Space Theory

Knowledge Space Theory provides a formal model for structuring a domain of knowledge and for representing the knowledge of individuals based on prerequisite relationships. A domain of knowledge is characterised by a finite, non-empty set Q of problems. The knowledge state K of a learner is represented by the subset of problems that he/she is capable of solving.

Due to mutual dependencies among the problems of a domain, from the correct solution of certain problems the mastery of other problems can be surmised. Such relationships between problems are captured by the socalled surmise relation. , {a}, {c}, {a, c}, {a, b}, {a, b, c}, {a, b, d}, {a, b, c, e}, {a, b, c, d}, Q}.

K = {Ø

Validation Approach

Let us assume a concept map representing the declarative knowledge of a particular domain, for which application validity is to be examined. As an appropriate measure of situational performance to be applied as validation criterion collecting problem solving patterns has been chosen. To this end, a set of typical and representative problems of the domain is selected. Each problem is identified with a subset of propositions of the given concept map, representing those elements of declarative knowledge that are required for solving the respective problem. In other words, each problem is mapped on the given concept map, by assigning the subset of propositions required for mastering the respective problem. Each proposition can be considered as an atomic skill or competency in the sense of the approaches of Doignon, Düntsch and Gediga, and Korossy (for references see [START_REF] Albert | Knowledge Spaces: Theories, Empirical Research, and Applications[END_REF][START_REF] Doignon | Knowledge Spaces[END_REF]. Most likely, the subsets of propositions assigned to the problems will overlap.

Based on the representation of the problems by subsets of the given concept map, dependencies between problems in terms of a surmise relation can be derived. This could be done e.g. by set inclusion, i.e. if the representation of a problem a in the concept map is a subset of that of problem b, then problem a is a From the dependencies between the problems derived in this way, possible knowledge states can be identified and a knowledge structure can be established. This means that specific answer patterns, that are expected to be observable, out of all possible subsets of items (2 |Q| ) can be predicted. In the following, the described procedure is illustrated by an example. Assuming a given concept map representing the domain in question, containing a set P of propositions (whereas a proposition (c p ,c q )∈R is abbreviated by p j ), that is given by P = {p 1 , p 2 ,…, p j ,…, p 7 , p 8 }. Let furthermore the selected set Q of problems representing the same domain be Q = {a, b, c, d}. Assume, the mapping m of the problems on the concept map is specified as follows:

m(a) = {p 1 , p 3 } m(b) = {p 2 , p 3 , p 4 } m(c) = {p 1 , p 2 , p 3 , p 4 , p 5 } m(d) = {{p 1 , p 2 , p 3 , p 4 , p 5 , p 6 , p 7 , p 8 } From the assignment of propositions to the problems given above it can be seen e.g., that problem a can be identified with the propositions p 1 and p 3 , i.e. p 1 and p 3 represent the declarative knowledge required for solving problem a. By applying the principle of set-inclusion, dependencies between the problems can be derived:

S d c d m c m S d b d m b m S c b c m b m S d a d m a m S c a c m a m ∈ ⇒ ⊆ ∈ ⇒ ⊆ ∈ ⇒ ⊆ ∈ ⇒ ⊆ ∈ ⇒ ⊆ ) , ( ) ( ) ( ) , ( ) ( ) ( ) , ( ) ( ) ( ) , ( ) ( ) ( ) , ( ) ( ) (
From the derived dependencies a surmise relation can be deduced, which is illustrated in Figure 3. The knowledge structure that is induced by the surmise relation shown in Figure 3 is given by , {a}, {b}, {a, b}, {a, b, c}, {a, b, c, d}}. In this way, based on the mapping of the problems on the concept map, six possible knowledge states have been identified. These knowledge states constitute answer patterns that are expected to be observed, provided that the given concept map adequately represents the domain. The next step in the validation approach is therefore to collect empirical answer patterns, by posing the problems to individuals of different knowledge level, including experts. In this way, it can be investigated empirically whether the observed answer patterns correspond to the identified and predicted knowledge states. This would be done e.g. by using a discrepancy index describing the similarity between the knowledge structure and the set of answer patterns (e.g. Doignon & Falmagne, 1999, Chapter 12). As the knowledge structure has been established based on mapping the problems on the given concept map, the empirically obtained answer patterns serve as a criterion for validity.

K = {Ø
If the empirical answer patterns correspond well to the predicted knowledge states, the given concept map can be considered to be valid -provided that both, the chosen set of problems as well as the sample of persons are adequate and representative.

Conclusions

In general, it is not sufficient to subjectively evaluate a concept map, judging it as valid by subjective evidence or uncontrolled consensus. Actually, there is an urgent need of objective, empirical measures and criteria for giving evidence of the validity of a concept map (semantic net). Efforts for empirically and objectively validating concept maps have proven reasonable and promising.

We distinguish two types of validity, content validity and application validity. The former refers to the adequate representation of the domain modelled, the latter refers to the practical usefulness of a concept map. We suggested and outlined two approaches for empirically validating a given concept map, one for giving evidence of content validity and one for application validity. Both approaches constitute useful and valuable procedures. For building up a coherent picture about the validity of a given concept map, both approaches can complement each other, providing a comprehensive view from different perspectives. Of course, the outlined approaches are not the only possibilities for validating concept maps. There are also other ways that constitute suitable validation approaches, as e.g. consulting the published literature of the given knowledge domain for examining content validity.

One special feature of the suggested validation approaches is that not only experts in a given field, for which a concept map is to be validated, are queried or consulted. In fact, the proposed procedures rather involve individuals of different knowledge level, who will possibly afterwards perform occupational tasks based on validated concept maps, e.g. developing a curriculum.

In general, when validating a given concept map, the purpose and ultimate use (e.g. predicting problem solving behaviour) needs to be taken into account. This issue is very important, as the content and structure of a concept map may highly depend on its intended purpose. Furthermore, validation efforts of concept maps also need to consider which aspects are intended to be validated (e.g. the whole concept map, the concepts, the relations, substructures of the given concept map). Another critical point is to choose appropriate statistical measures, depending on the particular criterion for validity and its constraints.

It should be mentioned that to date there is also little attention paid to the reliability of concept maps. Actually, before focusing on the validity of a given concept map, some considerations should be dedicated to reliability. Does the given or generated model of knowledge reliably represent the knowledge in question? The question of reliability or at least the circumstances under which the concept map originated, should be taken into account when the validity of a concept map is addressed.

For a given knowledge domain there might be several alternative concept maps that validly represent the respective domain. Most likely, such alternative concept maps will match in parts, i.e. with respect to concepts and relations, or even in whole substructures. An interesting research topic in this context is that of merging concept maps. Merging means a process that tries to integrate the information from two or more concept maps (or other forms of semantic knowledge representation) into a single one. How this can be done best is a subject of graph theory and ongoing research (e.g. [START_REF] Noy | PROMPT: Algorithm and Tool for Automated Ontology Merging and Alignment[END_REF]Stefanutti, Albert, & Hockemeyer, 2000).

Summarising, the issue of validating concept maps and other forms of semantic knowledge representation, such as semantic nets, knowledge maps etc., is an emerging field of research. Having available wellfounded and valid concept maps is critical for their effective use, be it in the classroom, in the context of hypertexts etc. Publishing a concept map, reusing an existing map for building a new one, or implementing an application that relies on concept maps written by others or even yourself without first evaluating it would be very unwise [START_REF] Gómez-Pérez | Evaluation of Ontologies[END_REF]. The methodological considerations presented in this paper provide interesting starting points for practical validation efforts.

Figure 1 :

 1 Figure 1: Concept map describing what a concept map is (adapted from Ruiz-Primo, 2000)

  The surmise relation S is a binary relation on the set Q of problems, that is reflexive and transitive. Two problems a and b are in a surmise relation, i.e. (a,b)∈S, whenever from a correct solution to problem b the mastery of problem a can be surmised. In other words, problem a is a prerequisite problem for problem b.

Figure 2 :

 2 Figure 2: Example of a Hasse diagram illustrating a surmise relation on a knowledge domain Q = {a, b, c, d, e} (adapted from Falmagne et al., 1990) A surmise relation can be depicted by a so-called Hasse diagram (see Figure 2 for an example). In such a diagram descending sequences of line segments indicate a surmise relationship. According to the surmise relation illustrated in Figure 2, from a correct solution to problem b the correct solution to problem a can be surmised, while the mastery of problem e implies correct answers to problems a, b, and c. The surmise relation forms a quasi-order on the set Q of problems and thus restricts the number of possible knowledge states (i.e. subsets of problems) that are expected to be observable. The collection of all possible knowledge states, including the empty state Ø and the whole set Q, constitutes the so-called knowledge structure K. The knowledge structure K corresponding to the surmise relation shown in Figure 2 is given by

Figure 3 :

 3 Figure 3: Surmise relation based on the mapping of the set Q of problems on the given concept map

The development of this software unfortunately has stopped.

Acknowledgements

Part of the work reported in this paper was financially supported by the European Commission through the iClass project (Contract Number 507922) within the FP6-IST programme.

We wish to thank Jürgen Heller for his helpful comments on an earlier version of this paper.


Dietrich Albert Cord Hockemeyer
email: fdietrich.albertjcord.hockemeyerg@kfunigraz.ac.at

Adaptive and Dynamic Hypertext Tutoring Systems Based on Knowledge Space Theory

Knowledge space theory provides a formal model for representing students' knowledge and describing the structure of a domain of knowledge. A similar formal structure can be used to described the structure of hypertexts. The combination of knowledge space theory and the formal hypertext model leads to a framework for intelligent tutoring systems which provides individualized learning paths to a student. Using powerful procedures from relational database systems and from knowledge space theory, w e get e. g. an ecient selection of appropriate teaching documents.

Introduction and previous results

Doignon and Falmagne [START_REF] Doignon | Spaces for the assessment of knowledge[END_REF] introduced the theory of knowledge spaces which provides a mean to formally describe the structure of a given domain of knowledge. We i n troduce the theoretical concepts in Section 1.1 below. The basic idea is the description of a student's knowledge by the set of problems (items) he or she is able to solve. The set of possible knowledge states is restricted by prerequisite relationships between the items.

A similar structure as knowledge spaces can be used to describe a hypertext: prerequisite relationships between dierent hypertext components can be specied by prerequisite links [START_REF] Albert | A relational model for describing hypertext structures[END_REF]. These prerequisite links build a structure which is comparable to a knowledge space. In section 1.2, we describe a formal model for hypertext structures.

In Section 2, we describe how the combination of the formal hypertext model and the theory of knowledge spaces leads to an individualized and ecient h ypertext tutoring system. 1.1 Formal concepts of knowledge space theory The formal concepts in knowledge space theory have been introduced in [START_REF] Doignon | Spaces for the assessment of knowledge[END_REF]. Some more recent research developments can be found e. g. in [START_REF] Albert | Knowledge Structures[END_REF].

In knowledge space theory, a eld of knowledge is specied by a nite set of items, i. e. problems or tasks a student m a y o r m a y not be able to solve. Each student can be described by his/her knowledge state, i. e. the subset of items which this student masters. However, since there exist prerequisite relationships between the items, not all possible subsets of items are knowledge states. The set of all possible knowledge states is called a knowledge space. Such a knowledge space contains the empty set ; and the complete item set Q as elements, and, for any t w o knowledge states K; K 0 2 K , their union K [ K 0 is also a memberofK.

We h a v e already mentioned that the set of possible knowledge states is restricted by prerequisite relationships between the items. In knowledge space theory, these relationships are formalized by surmise relations, i . e . quasiorders on a set Q of items. Such a surmise relation v may beinterpreted as q v q 0 if and only if from a correct response to problem q 0 we can surmise a correct response to problem q. Instead, we could also say that mastering item q is a prerequisite for mastering item q 0 . 1.2 A mathematical model for the structure of hypertext Albert, Held, and Hockemeyer [START_REF] Albert | A relational model for describing hypertext structures[END_REF] have suggested a mathematical model for the structure of a hypertext using the terminology of the Dexter hypertext reference model [START_REF] Halasz | The Dexter hypertext reference model[END_REF]. Within the Dexter model, a hypertext consists of a set of components and a set of links between these components. A component consists of a base component (unit of information) and some source and destination anchors which are located on the base component. Links are specied by source anchors and destination anchors and the components on which the anchors are located. In the following, we present the core of [START_REF] Albert | A relational model for describing hypertext structures[END_REF].

We assume the existance of three sets B , S , and D of base components, source anchors, and destination anchors, respectively. 2 Combining hypertext structures and knowledge space theory Knowledge space theory and the mathematical hypertext model can becombined to build a framework for intelligent hypertext tutoring systems. As a central point in the combination of knowledge spaces and hypertexts we use the concept of prerequisite relationships. These prerequisite relationships are specied using prerequisite links which is a special example of a link type. 2.1 Prerequisite links in dynamic hypertexts. Albert, Held and Hockemeyer [START_REF] Albert | A relational model for describing hypertext structures[END_REF] introduce the concept of link types. Assume a hypertext H = (C;L) as dened in Section 1.2 above. For an arbitrary subset P S , we call a link l = ( ( c; s) ; (c 0 ; d ) ) a link of type P if and only if s 2 P . We denote by L P = fl = ( ( c; s) ; (c 0 ; d ) ) 2 L j s 2 P g the set of all links of type P . We dene a binary relation `P ( C C ) . F or any components c; c 0 2 C , w e obtain c `P c 0 if and only if there exists a link l = ( ( c; s) ; (c 0 ; d ) ) 2 L P . This relation `P is called P relation.

In this paper, we make only use of the prerequisite links. A prerequisite link from a component c to a component c 0 (i. e. c `P c 0 ) means that c 0 is a prerequisite for c, i. e. for understanding the contents (or solving the problem) of component c, the content of component c 0 must be known. Semantically, this prerequisite relation has a similar meaning to that of the surmise relation with the components corresponding to the items. While we use the surmise relation to select new problems during the student's knowledge assessment, we can use the prerequisite relation to select appropriate components to be oered to the student. 2.2 Individualized learning path. The transfer of the concept of surmise relations from knowledge spaces to hypertexts enables us to use procedures developed within the frame of knowledge space theory. One of these concepts is the learning path [START_REF] Falmagne | Stochastic learning paths in a knowledge structure[END_REF].

Let H = ( C; L) beahypertext and let `P beaprerequisite relation on C as specied in Section 2 above. A sequence X 0 ; X 1 ; : : : ; X n of subsets of components is called a learning path from X 0 to X n if and only if the following two conditions hold. (i) For any i = 0; 1; : : : ; n , for any c 2 X i , and for any c 0 2 C , if c `P c 0 then c 0 2 X i and (ii) for any i = 1; 2; : : : ; n , there exists a c 2 C n X i 1 such that X i = X i 1 [ f c g . A learning path from ; to C is simply called a learning path. Condition (i) ensures that the steps of the learning paths are states, i. e. subsets of components which conform to the prerequisite relation. Condition (ii) provides that the learning path describes a stepbystep learning.

Individualized learning paths are constrained by the educational objective and the student's current knowledge state. The former may be specied by the student o r b y some teacher while the latter can be determined with assessment procedures known from knowledge space theory [START_REF] Dowling | Adaptive assessment and training using the neighbourhood of knowledge states[END_REF][START_REF] Albert | Knowledge Structures[END_REF]. In general, there will exist several learning paths from the starting to the destination state [START_REF] Falmagne | Stochastic learning paths in a knowledge structure[END_REF].

During the tutoring process, the student's knowledge state can be updated at any step corresponding to the lessons learned. At a n y time, the student has access to those components which are the next step on some learning path from the current state to the destination state.

Discussion

In this paper, we suggest the combination of knowledge space theory and a mathematical hypertext model to build ecient and adaptive intelligent tutoring systems. The eciency of such a system has two sources: First, knowledge space theory provides us with ecient procedures based on dierent representations (surmise relations, knowledge spaces and their bases, prerequisite relations) for the structure of a domain of knowledge. We can, for example, implement storage and timeecient procedures for the assessment of knowledge using a theorem of Birkho. Second, the formal description of hypertext structures using mathematical relations enables us to apply methods from relational database theory to these structures quite directly. This concerns not only individual access to document appropriate for the student's actual knowledge but also the construction of sub hypertexts due to educational objectives or to the student's prior knowledge.

We also want to mention some extended concepts from knowledge space theory. One idea is the generalization of the surmise relation to a surmise system, which reects the fact that there may exist several ways to solve a problem. Another, recent extension is the distinction between observable behaviour (items or performance) on the one hand and underlying skills (or competencies) on the other hands. Such extensions have been suggested by Doignon, Düntsch and Gediga, and Korossy.

These extensions of the knowledge space theory can lead to an extension of the framework for intelligent hypertext tutoring systems.

  A component c = (b; S c ; D c ) 2 ( B 2 S 2 D ) is a triple constituted by a base component b 2 B, a subset S c S of source anchors, and a subset D c D of destination anchors. A set L (C S)(CD) where each l = ( ( c; s) ; (c 0 ; d ) )2L fullls the condition s 2 S c and d 2 D c 0 is called a set of links. A pair H = ( C; L) is called a hypertext. A link l = ( ( c; s c ) ; (c 0 ; d c 0 ) ) connects a source anchor s c located on a component c with a destination anchor d c 0 on a component c 0 . Based on this formalization of links, we introduce a binary relation `which describes the linkage between components. For a Hypertext H = (C;L), we dene the link relation C C on the set C of components. For any components c; c 0 2 C , w e obtain c `c0 if and only if there exists a link l = ( ( c; s) ; (c 0 ; d ) ) 2 L .

Acknowledgments

This work was supported by the grant ERBCHBICT941599 in the program on human capital and mobility (HCM) of the Commission of the European Communities. The authors are grateful to Theo Held for his valuable suggestions.


Dietrich Albert
email: dietrich.albert@uni-graz.at

Luca Stefanutti
email: luca.stefanutti@uni-graz.at

Knowledge Structures and Didactic Model Selection in Learning Object Navigation

In this paper two levels of content aggregation are considered: (i) a 'conceptual level' of the content of the learning objects, and (2) a 'didactic preference level' of the specific preferences expressed by a taecher or educational institution. We then present a knowledge structure model in which the learning objects are organized according to constraints imposed at these two distinct levels. The structure on the learning objects is then used to restrict the number of learning paths available to a learner during her/his navigation of the learning environment. In particular, only paths that are consistent with both the content constraints and the didactic constraints are available.

Introduction

According to the SCORM (Shareable Content Object Reference Model) specification, an e-learning system is broken into two main and independent components of learning content interoperability: the so called Learning Mangement Systems (LMS), and the Shareable Content Objects (SCOs). In this model the learning objects (the SCOs) are assumed to be standardized and reusable units. The process of finding and organizing the learning objects into a specific structure is called content aggregation. This process is responsible, for instance, of deciding which learning object comes next in the learner's interaction with the LMS. In doing so, the system may take into account a number of different aspects related to the specific content of the learning objects, the didactic preferences of the teacher or educational institution which applies the system, and the learner's personal preferences.

Elsewere some methods and models were presented for structuring reusable distributed learning objects according to their content (Stefanutti, Albert & Hockemeyer, in press). These models are based on a mathematical framework called knowledge space theory (Albert & Lukas, 1998;[START_REF] Doignon | Knowledge spaces[END_REF], whose key concepts and applications are summarized in the next section. In this paper the attention is focused on some didactic aspects that may be considered in structuring a set of learning objects, as well as on some ways of integrating the didactic preferences, expressed e.g. by a teacher, with the content structure on the learning objects.

We make a distinction between two levels of content aggregation. At the first level the learning objects are organized and related one another on the basis of their content in terms of the concepts and notions they contain. The structure obtained in this way puts constraints on the possible sequences of learning objects presented to the learner. At this level the teacher's didactic preferences have no influence on the structure. However, a second level of content aggregation is provided in which (at least some of the) didactic preferences of the teacher are transformed into rules that put further constraints on the structure of the learning objects. Combining together these two levels results in a learning environment in which the possible learning paths (sequences of learning objects) are consistent with both the content of the learning objects, and the preferences of the teacher.

Knowledge structures

Knowledge space theory [START_REF] Doignon | Spaces for the assessment of knowledge[END_REF][START_REF] Albert | Knowledge spaces: theories, empirical research, applications[END_REF][START_REF] Doignon | Knowledge spaces[END_REF] offers a rigorous and efficient formal framework for the construction, validation, and application of e-assessment and e-learning adaptive systems. It is a psychometric theory for the assessment and acquisition of knowledge; however, as far as adaptivity and optimal learning paths are concerned, its methods and models can be fruitfully exploited for e-learning purposes as well.

According to this theory, a domain of knowledge is a collection Q of problems in a given field of knowledge (e.g., mathematics, physics, chemistry, biology, etc.). Then, the knowledge state of a student is the set K of all problems in Q that this student actually masters, and a knowledge structure for Q is a pair (Q, K) in which K is the collection of all knowledge states that can be observed in a certain population of students. If K is closed under union (i.e., K ∪ K ∈ K whenever K, K ∈ K) then it is called a knowledge space. Sometimes also closure under intersection holds (K ∩ K ∈ K whenever K, K ∈ K), in which case, K is called a quasi-ordinal knowledge space.

The above-mentioned theory is at the basis of some existing e-learning and assessment adaptive systems in the U.S. and in Europe. Two of them are the ALEKS (Adaptive LEarning with Knowledge Spaces) system developed by the research group of Irvine, CA supervised by Jean-Claude Falmagne (http://www.aleks.com), and the RATH prototype (Relational Adaptive Tutoring Hypertext) system of the research group of Graz, Austria (http://wundt.uni-graz.at/rath, [START_REF] Hockemeyer | RATH -a relational adaptive tutoring hypertext WWW-environment based on knowledge space theory[END_REF] supervised by Dietrich Albert.

Learning paths and navigation history

We denote with L a set of learning objects. Each learning object in L can be a problem, an explanation, an example, and so on. All objects in L are supposed to be on the same domain of knowledge. In particular, each of them contains some specific notion or concept in this domain.

Navigating through the learning objects the learner specifies implicitly some order on the set L. The sequence in which the objects are visited is a result of the preferences and decisions of both the learner and the author of the learning management system.

In this development, with the term learning path we mean any sequence (linear order) of learning objects in L which is consistent with the criteria, preferences and decisions of both the learner and the author. The way in which these two actors specify their criteria and preferences, however, differs in the sense that the criteria and preferences of the author are specified in advance through some set of rules that, usually, reduce the number of possible learning paths available to the learner.

If the rules specified by the author are represented through a knowledge structure on the learning objects, i.e., by any collection K of subsets of L such that both ∅ and L are in K, then, a learning path in K is any maximal chain C in the partially ordered set (K, ⊆)1 . As an example consider the structure K on a set L = {1, 2, 3, 4} of learning objects:

K := {∅, {1}, {2}, {1, 2}, {1, 3}, {2, 3}, {1, 2, 3}, {1, 2, 4}, {1, 2, 3, 4}}.
Then, for instance, the collection C := {∅, {1}, {1, 2}, {1, 2, 3}, {1, 2, 3, 4}} is a maximal chain in (K, ⊆) and thus it is a learning path in K. Note that this path corresponds to the sequence (1, 2, 3, 4) of learning objects. On the other hand, consider the collection

C := {∅, {1}, {1, 4}, {1, 2, 4}, {1, 2, 3, 4}}.
This is not a learning path in K, since {1, 4} / ∈ K. Thus the sequence (1, 4, 2, 3) is not consistent with the constraints imposed by the structure K. This means that K prevents the learner to navigate the learning objects in this particular sequence.

If the the learning objects visited by the learner are recorded as a set H ⊆ L of learning objects, namely, the set of all learning objects already visited by her/him, then H is consistent with the constraints imposed by K if and only if H ∈ K. In this sense, K contains all consistent histories.

One question that arises at this point is how an author may construct the structure K. One very direct way would consist in specifying which of the 2 n , for n = |L|, subsets of L are consistent histories. This way, however, seems unfeasible for different reasons (see, in this connection, Falmagne, Koppen, Villano, Doignon, Johanessen, 1990). First, the number of such subsets grows exponentially in n and with a realistic number of learning objects, the job of separating the consistent subsets from the inconsistent ones would result in an enormous task for any human. The second reason is that there are indirect ways of establishing such collection by imposing explicit rules on the set L of the learning objects which are meaningful, for instance, from a didactic and/or a cognitive point of view. In the next section we present one possibility along this direction.

Concept assignments and didactic preferences

In this section we present a model that makes a distinction between two kinds of constraints on the learning history. The first kind of constraints is directly related to the content of the learning objects in terms of the concepts and the notions they contain. The second kind of constraints pertains to the different types of learning objects one can have in a learning management system. In this respect, not all learning objects are designed for the same purpose. Some of them may provide general or specific information (like definitions or explanations) on some content while others are aimed at the assessment of the knowledge of the learner (like problems or questions), still others are provided in order to give the learner concrete instances of abstract concepts and notions (like examples or exercises). The author of the learning management system might want to keep information about the content of a learning object separate from the information on its type. Keeping the content information separate from the type information the author of a learning management system might embed in the system the didactic constrainst on the content, leaving however to the final user the opportunity to choose the constraints on the types according to her/his didactic preferences.

Let C be a set of concepts (or notions) in a given field of knowledge. For example, in the field of linear algebra, a set of notions might contain the following concepts: (c 1 ) vector space, (c 2 ) spanning set of vectors, (c 3 ) basis of a vector space, (c 4 ) dimension of a vector space, (c 5 ) rank of a matrix (c 6 ) linear independence. On the other hand, we consider a set T of learning object (LO) types (or classes). Examples of learning object types are (t 1 ) the LO is a problem or question, (t 2 ) the LO is an explanation or instruction (t 3 ) the LO is an example, (t 4 ) the LO is an exercise. The pair (T, T ), where ' T ' is a partial order on the types in T is called a didactic preference. Thus, a didactic preference is expressed by (partially) ordering the different types of learning objects so that, for instance, in a 'task oriented didactic' one might require that an explanation page on a certain topic should be presented to the learner only if some problem on the same topic was already presented. In this case, if we denote with p the class of learning objects that are problems, and with e the class of learning objects that are explanations (p, e ∈ T ), then we will have p T e. In general, for t , t ∈ T , the interpretation of t T t is that a learning object belonging to class t should be presented before (or immediately before -see Section 6 in this connection) a leaning object of class t . We thus call the partial order T a predecessor relation for the didactic preference (T, T ).

It remains to establish how concepts and types are assigned to learning objects. To this aim we introduce two distinct mappings. A concept function for L and C is a mapping f : L → 2 C \ {∅} assigning a given subset of concepts to each learning object x ∈ L, so that f (x) ⊆ C (note that f (x) = ∅ for all x ∈ L, that is, each learning object must have at least one concept assigned to it). On the other hand, we call type function a mapping t : L → T assigning a given type in T to each of the learning objects. Thus, for x ∈ L, t(x) ∈ T is the type assigned to x.

It should be noted that the notion of a concept function is germane to that of a skill function (or skill map) introduced and discussed elsewere (see e.g., [START_REF] Doignon | Knowledge spaces and skill assignments[END_REF][START_REF] Korossy | Extending the theory of knowledge spaces: A competence performance approach[END_REF][START_REF] Doignon | Knowledge spaces[END_REF][START_REF] Düntsch | Skills and knowledge structures[END_REF]. In this context, however, we prefer to use the term 'concept' rather than 'skill' since this last is closely related to cognitive aspects that we do not consider explicitly here. Instead, our focus is on the content of a learning object in a didactic sense, that is, in terms of the 'concepts' and 'notions' that can be learned through the learning object itself. Nonetheless, the mathematical concept behind the two terms 'skill function' and 'concept function' is exactly the same.

Considering any learning object in L, a distinction should be made between the concepts that a learner needs to know in order to master the learning object itself (i.e., to solve it if it is a problem or an exercise, or to understand it if it is some explanation or instruction) and those that are new (just presented in this learning object for the first time). This distinction can be found, e.g. in [START_REF] Hockemeyer | Extending the Competence-Performance-Approach for Building Adaptive and Dynamic Tutoring Systems[END_REF]. Here we focus on the first kind of concepts and we assume that, for x ∈ L, f (x) is the set of concepts required to master x. It seems thus reasonable to assume that a learning object x is a 'prerequisite' for some other learning object y whenever the set of concepts required by x is a subset of that required by y. Formally, we call a prerequisite relation for the learning objects in L a partial order L such that, for x, y ∈ L,

x L y ⇐⇒ f (x) ⊆ f (y).
A partial order on the learning objects which is consistent with both the prerequisite relation L and the didactic preference T is thus defined by

x y ⇐⇒ f (x) ⊆ f (y) and t(x) T t(y).

(4.1)

Since each learning object x ∈ L is represented by the corresponding pair (f (x), t(x)), the order ' ' follows from a coordinatewise ordering of such pairs. In particular, the partial order (L, ) is embedded into the Cartesian product (L, ⊆) × (T, T ), where

L := {f (x) : x ∈ L}.
This fact is established by the following Proposition 1. Let (P, P ) be the Cartesian product of (L, ⊆) and (T, T ) (in which ' P ' is the coordinatewise ordering). Then there exists a total function h : L → P such that, for any x, y ∈ L, x y ⇐⇒ h(x) P h(y).

Proof. For any x ∈ L, let h(x) = (f (x), t(x)). Clearly, h maps L into P i.e., h(L) ⊆ P . Given any x, y ∈ L, from (4.1) we have x y iff f (x) ⊆ f (y) and t(x) T t(y). Moreover, the order ' P ' is such that, for (A, t ), (B, t ) ∈ P , (A, t ) P (B, t ) iff A ⊆ B and t T t . Thus, x y iff h(x) P h(y) for all x, y ∈ L.

As an example, consider the set C = {c 1 , c 2 , c 3 , c 4 } of the first four concepts on linear algebra mentioned at the beginning of this section, and the set T = {t 1 , t 2 } of the first two types. An assignment of the concepts in C and types in T to a set L of ten different learning objects (numbered from 1 to 10) is displayed in Table 1. Table 1: An example of concept and type assignment to a set of 10 learning objects (see text).

LO

c 1 c 2 c 3 c 4 Type 1 × t 1 2 × × t 1 3 × × t 1 4 × × × t 1 5 × × × × t 1 6 × t 2 7 × × t 2 8 × × t 2 9 × × × t 2 10 × × × × t 2
A cross (×) in the cell corresponding to object i and concept j indicates that concept j is assigned to learning object i. The last column of the table (Type) shows how the two 4.1) to the 10 learning objects of the example in Table 1.

types t 1 and t 2 are assigned to the learning objects. According to Table 1, for example, we have that f (1) = {a} and f (2) = {a, b}, thus 1 L 2. If now a didactic preference is expressed such that type t 1 is a predecessor of type t 2 (t 1 T t 2 ), the partial order resulting from an application of (4.1) is that depicted in Figure 1. It can be observed that, if the collection L := {f (x) : x ∈ L} is considered: embedding is, in fact, an isomorphism.

L := {{a},
The question now arises of how to construct a knowledge structure K containing all histories H ⊆ L that are consistent with the order relation ' ' on L established by (4.1). The answer comes from a classical result in lattice theory due to [START_REF] Birkhoff | Lattice theory[END_REF]: given a finite set X, the family of all partial orders on X is in one-to-one correspondence with the family of all collections of subsets of X that are both closed under union and under intersection2 . Thus, the partially ordered set (L, ) corresponds to a unique collection K of subsets of L, and this correspondence is established by the relation

K = {X↓: X ⊆ L},
where, for X ⊆ L, X↓ is the down-set generated by X in (L, ):

X↓:= {y ∈ L : y x for some x ∈ X}.
Given a history H ∈ K, its consistency with ' ' is guaranteed by the fact that if x ∈ H and y

x, then also y ∈ H holds. Thus, for instance, the set = {1, 2, 6, 8} is in the knowledge structure corresponding to the partially ordered set of Figure 1. The set {1, 2, 4, 6} however, is not. In fact, 4 requires 3, which is not in this set.

Constrained navigation

When constraints are imposed on the learning objects in L in the form of the partial order ' ', not all learning objects are available to a learner with history H ⊆ L. More precisely, which learning objects are available to her/him depends on the history H of the learner, and the partial orders L and T . Thus, going back to the partially ordered set of learning objects (L, ) displayed in Figure 1, if for instance the history of a learner is H = {1, 2, 6, 8}, the learning object 5 should not be available to this learner simply because, moving to this object her/his new history would become H = {1, 2, 5, 6, 8}, which is not an element of the structure K corresponding to (L, ) (the reader can easily verify this).

In order to establish which learning objects are accessible to a learner with history H, we need some further notation. First, given two objects x, y ∈ L, we say that x is equivalent to y (x ≡ y) whenever both x y and y x hold true, and we denote with [x] := {y ∈ L : x ≡ y} the equivalence class of x. From a didactic point of view, this is interpreted as a 'content equivalence' i.e., the two learning objects have equivalent content. Then, for H ∈ K, the collection

H o := {y ∈ L \ H : H ∪ [y] ∈ K}
is called the outer fringe of H [START_REF] Doignon | Knowledge spaces[END_REF]. With this new definition we require that a learning object y not in H is accessible from H whenever y ∈ H o . In other words, if the learner's last visited learning object is x ∈ H, then a link between x and y is dynamically created for all y ∈ H o . Thus, for H 1 := {1, 2, 6, 8}, the last visited learning object of the learner will be linked to all objects in the set H o 1 := {3}, in this case there is only one object, whose type is problem 3 . Note that, objects 6 and 7 are both of type explanation and 7 would be immediately accessible from 6 if there was no order specification on the object types. The constraint imposed on the types requires that problem 3 be visited before to access the explanations contained in 7. This just reflects the didactic preference that 'whenever compatible with the content constraints, problems should be presented before explanations'. Once problem 3 is visited, the new history will be H 2 := {1, 2, 3, 6, 8}, and the corresponding outer fringe H o 2 := {4, 7}. Table 2 shows, as an example, the learning path of a learner in the knowledge structure K and the corresponding set of accessible learning objects.

Conclusions

Adaptivity occurs at different levels. In the content aggregation process of a LMS at least two levels should be considered: (i) the level of the content of the learning objects, and (ii) that of the didactic preferences of the educational institution that applies the e-learning system. Different teachers may have different didactic preferences and the system should be able to re-organize its structure on-the-fly according to such preferences.

In this paper we presented a simple model for content aggregation based on these two levels. At the 'content' level the structure on the learning objects corresponds to partially ordering the objects according to the concepts and notions they contain. At the 'didactic preference' level learning objects are classified according to their 'purpose'. Not only learning objects differ in their content, they also have different purposes: a quiz is presented to the learner to test her/his knowledge, skills, degree of competency; an instruction is provided whenever the learner needs more information on a certain topic; an example is given to introduce to the learner some new concept or to make concrete an abstract concept, and so on. The priority of the different purposes may change in different didactic situations and educational contexts, and the LMS should be able to adapt the structure on the learning objects according to these changes.

Learning objects are thus classified according to their types, where different types reflect different purposes. When a partial order is specified on the learning object types, it can be combined with the order on the learning object content to produce a structure that is consistent with both the conceptual content of the objects, and the didactic preferences of the teacher. This structure can then be used to decide which learning object comes next according to content constraints, teacher's preferences, and learner's preferences.

The simple model presented in this paper can be extended in different ways. One possibility, for instance, is represented by a learning object belonging to more different types. Some learning objects can be used for different purposes, depending on the context. For instance, a problem can be used as a step-by-step exercise, or in some cases instructions may be appropriate as examples. Another possibility pertains the distinction between 'before' and 'immediately before' made in section 4. Sometimes, a learning object is required to be presented immediately after some other, like a quiz that tests the learner after some explanation. In other cases this requirement is less strict: after failing a problem, the learner might be free to choose between reading some explanation on that problem, or trying to solve some less demanding problem.

Figure 1 :

 1 Figure1: Partial order resulting from the application of rule (4.1) to the 10 learning objects of the example in Table1.

Figure 2 :

 2 Figure 2: Cartesian product of the two partially ordered sets (L, ⊆) and (T, T ).

Table 2 :

 2 Example of a learning path, the corresponding collection of outer fringes and the type of the current learning objects at each step of the learning path.

	Step	History	Outer Fringe Current Object
	0	∅	{1}	-
	1	{1}	{2, 3, 6}	problem
	2	{1, 2}	{3, 6}	problem
	3	{1, 2, 6}	{3, 8}	explanation
	4	{1, 2, 6, 8}	{3}	explanation
	5	{1, 2, 3, 6, 8}	{4, 7}	problem
	6	{1, 2, 3, 4, 6, 8}	{5, 7}	problem
	7	{1, 2, 3, 4, 5, 6, 8}	{7}	problem
	8	{1, 2, 3, 4, 5, 6, 7, 8}	{9}	explanation
	9	{1, 2, 3, 4, 5, 6, 7, 8, 9}	{10}	explanation
	10	L	∅	explanation

A chain in (K, ⊆) is any subset C ⊆ K such that, for X, Y ∈ C, either X ⊆ Y or Y ⊆ X hold. A maximal chain is a chain M ⊆ K for which there is no other chain C ⊆ K such that M ⊂ C .

In the terminology adopted in knowledge space theory, these structures are often called quasi-ordinal knowledge spaces[START_REF] Doignon | Knowledge spaces[END_REF].

A link to all objects already visited (i.e., the elements of H), however, may always be present.