PhD Thanasis Daradoumis
email: adaradoumis@uoc.edu

PhD Alejandra Martínez-Monés

PhD ; Fatos Xhafa

Fatos Xhafa
email: fxhafa@uoc.edu

A Layered Framework for Evaluating Online Collaborative Learning Interactions

Keywords:

Evaluating online collaborative learning interactions is a complex task due to the variety of elements and factors that take place and intervene in the way a group of students comes together to collaborate in order to achieve a learning goal. The aim of this paper is to provide a better understanding of group interaction and determine how to best support the collaborative learning process. To that end, we propose a principled framework for the study and analysis of group interaction and group scaffolding which is built by combining different aspects and issues of collaboration, learning and evaluation. In particular, we define learning activity indicators at several levels of description which prompt to the application of a mixed interaction analysis scheme and the use of different data types and specific tools. At an initial layer, the basis of the approach is set by applying a qualitative process for evaluating the individual and group task performance as well as the group functioning and scaffolding. The interaction analysis process is completed by defining and applying two more layers: a social network analysis of the group activity and participation behaviour, and a quantitative analysis of group effectiveness as regards task achievement and active interaction involvement. Our work defines a grounded and holistic conceptual model that describes online collaborative learning interactions sufficiently and applies it in a real, webbased, complex and long-term collaborative learning situation. An in-depth empirical evaluation of the conceptual model is fully discussed, which demonstrates the usefulness and value of the approach.

Introduction

Several issues and questions have risen up in online collaborative learning and need to be responded in the most effective way. These questions are closely related to the extensive research on interaction analysis in the CSCW and CSCL fields that has been oriented to identifying and exploring the factors that affect the effectiveness and success of online group work and learning [START_REF]Collaborative Learning. Cognitive and Computational Approaches[END_REF]. However, this line of research has proposed rather limited approaches, focusing on a single collaboration channel, such as dialogue [START_REF] Barros | Analysing student interaction processes in order to improve collaboration. The DEGREE approach[END_REF] or action [START_REF] Mühlenbrock | Action-based collaboration analysis for group learning[END_REF]. Some researchers have recently proposed an integration of different sources of data in the analysis [START_REF] Collazos | Evaluating Collaborative Learning Processes[END_REF][START_REF] Sfard | On two metaphors for learning and the dangers of choosing just one[END_REF][START_REF] Wortham | Nodal and Matrix Analyses of Communication Patterns in Small Groups[END_REF]. In spite of this fact, existing approaches have not yet managed to meet these needs satisfactorily, since most of them focus on experimental situations, which do not exactly reflect the issues and problems of a real situation.

From the evaluator's point of view, a critical issue in the groups' lifecycle is how to analyse collaborative learning interactions in order to be able to assess and support self, peer and group performance through efficient and functional assessment and scaffolding techniques [START_REF] Barros | Analysing student interaction processes in order to improve collaboration. The DEGREE approach[END_REF][START_REF] Martínez | Combining qualitative and social network analysis for the study of social aspects of collaborative learning[END_REF][START_REF] Reiser | Why Scaffolding Should Sometimes Make Tasks More Difficult for Learners[END_REF][START_REF] Soller | Supporting Social Interaction in an Intelligent Collaborative Learning System[END_REF][START_REF] Zumbach | Supporting Distributed Problem-Based Learning: The Use of Feedback in Online Learning[END_REF]. Most of the existing learning systems still have limitations when used by students in real settings. Some of the limitations are attributed to the fact that students have difficulties to develop meta-cognition on their own actions and processes, or to self-estimate the appropriateness of their participation in a collaborative group or a wider learning community. Students seem to need information (in a literal or visual form) on their own actions, that could support awareness, meta-cognition and thereby self-regulation of their learning activity [START_REF] Dimitracopoulou | Designing advanced collaborative learning environments: Current trends and future research agenda[END_REF].

In addition, CSCL applications for online collaborative learning are characterized by a high degree of user-user and user-system interaction and hence generate a huge amount of information usually maintained in the form of events aggregated in log files [START_REF] Zumbach | Multidimensional Tracking in Virtual Learning Teams[END_REF]. This information is an important data source for understanding, explaining and predicting patterns of group behaviour, detecting collaboration breakdowns and therefore supporting group activity with adequate feedback. Thus it needs to be appropriately structured and represented at several levels of description so that it can be automatically processed in order to extract essential knowledge related to the quality of the collaborative learning product (task performance) and the quality of the collaboration itself (group functioning, help and social support). Some approaches were based on pre-structured dialogue to represent interaction.

Katz [START_REF] Katz | The Cognitive Skill of Coaching Collaboration[END_REF] developed two rule learning systems, String Rule Learner and Grammar Learner. These systems learn patterns of conversation acts from dialog segments that target particular pedagogical goals. Inaba & Okamoto [START_REF] Inaba | The Intelligent Discussion Coordinating System for Effective Collaborative Learning[END_REF] describe a model that draws upon the ideas of finite state machines and utility functions. They used a finite state machine to control the flow of conversation and to identify proposals, while applying utility functions to measure participants' beliefs with regard to the group conversation. However, both approaches restrict the way interaction is described.

Since interaction analysis is a core function for supporting both students' selfregulation and evaluation in CSCL environments, it is crucial to develop a principled framework for evaluating collaborative interactions that allows studying and analysing the collaborative behaviour and success of online learning teams at several levels. The evaluation of collaborative learning has to be performed at least at two levels, separating the process (or group functioning) from the product (or task performance) of collaboration [START_REF] Collazos | Evaluating Collaborative Learning Processes[END_REF][START_REF] Daradoumis | Evaluating Collaborative Learning Practices in a Virtual Groupware Environment[END_REF][START_REF] Macdonald | Assessing online collaborative learning: process and product[END_REF]. According to [START_REF] Sfard | On two metaphors for learning and the dangers of choosing just one[END_REF], participation is a further important aspect since, together with acquisition, constitutes one of the two main metaphors of learning.

As a consequence, our framework is conceptualised in a layered manner that results from the definition of generic group activity indicators, which represent high-level collaborative learning processes and which are further decomposed into more specific ones. Then, in order to address and evaluate them, we propose a mixed evaluation scheme [START_REF]User-Friendly Handbook for Mixed Evaluations[END_REF] that, on the one hand, consists of qualitative, quantitative and social network analysis techniques and, on the other hand, integrates different sources of data (i.e. types of interaction, products, student's opinions, etc.) and supporting tools.

To our knowledge, no other approach so far has defined or developed any principled, effective and holistic framework for analysing the interaction in-depth and assessing the performance of online learning groups. Our work provides a new step towards the fulfilment of the demand for an in-depth formative analysis and evaluation of online collaborative learning interactions in real CSCL settings and is based on previous work done in [START_REF] Daradoumis | Exploring Interaction Behaviour and Performance of Online Collaborative Learning Teams[END_REF][START_REF] Daradoumis | An Integrated Approach for Analysing and Assessing the Performance of Virtual Learning Groups[END_REF].

The rest of the paper is structured as follows. First, we present the case study on which our empirical analysis was carried out. Section 3 describes how high-level indicators are set and measured by a qualitative evaluation process. This leads to the definition of the top layer of our conceptual model for analysing and evaluating collaborative learning interactions. In Section 4, the framework is then complemented by defining further specific mid-and low-level indicators which build more refined layers of the framework and are evaluated and interpreted by further analysis techniques: a Social Network Analysis and a quantitative statistical analysis. Both techniques support and fill any remaining gaps and thus complete the analysis process. In addition, the data sources and supporting tools associated with the above techniques are also described. Finally, we present the future lines of our research that aim at extending the use of the framework from an assessment tool employed so far to group awareness and scaffolding.

Case Study Description

The context of our academic work and research is mostly situated in the area of distance education which is completely supported by a virtual learning environment.

For this reason, we have defined a learning situation that consisted of setting online learning groups to work on a real, long-term, complex, collaborative problem-solving practice that was carried out in the scope of distance learning undergraduate courses, such as "Case studies in Information Systems Management", or "Software Development Techniques".

Both experiences usually run over a period of 14 weeks and involve at least 10 tutors and more than 500 students distributed into more than 90 online groups of 5 to 6 members. In the first experience, students have to collaborate and work out a case study that simulates a real project in a business or organisation. The second experience is based on the Project-Based Collaborative Learning paradigm. The findings of this work are based on the first experience, so it is important to present first a sufficient description of the workings of the case study that was used for the purpose of our analysis.

The case resolution consists of a set of target goals (phases) that are realised collaboratively (except the first one which aims at studying and understanding the case presented). The instructional design of each target goal includes several learning tasks, adequately linked to each other, that students should carry out individually (such as readings) or collaboratively (such as activities and exercises) in order to achieve the goal. In addition, the design of some target goals also dictates the realisation of specific asynchronous debates at group and class level aiming at decision making on specific questions set.

The whole project was carried out mostly asynchronously; synchronous interaction occurred in few specific cases of the decision-making process. All asynchronous collaborative interactions took place on the Basic Support for Cooperative Work (BSCW) system, a groupware tool that enables asynchronous and synchronous collaboration over the web [START_REF] Bentley | Basic Support for Cooperative Work on the World Wide Web[END_REF]. BSCW offers shared workspaces that groups can use to store, manage, jointly edit and share documents, realise threaded discussions, etc.

To structure the whole collaborative learning process, we set two particularised shared workspaces in the BSCW system. The first one is a general workspace, which can be accessed by all students of the virtual class. The main purpose of this workspace is to let the students interact with each other in order to form the virtual learning groups. In addition, it is used to effectuate specific debates, which form part of the project requirements and involve all students, as well as to share important information about the project among tutors and students. The other workspace type is a private space designated to house each virtual group, that is to record and structure the interaction of its members that aims to achieve the project target goals through the resolution of the specific tasks and problems the project consists of.

Our analysis was carried out at both the general and the private group spaces, using specific evaluation criteria as parameters to measure the groups' real effectiveness regarding learning outcome and collaborative skills, as explained in the following sections.

An aspect, which is of particular interest of our analysis, is the coordinator role that a member assumes during the realisation of a target goal. To that end, we assess the degree of success of this role by examining whether the following tasks, which a coordinator should carry out, are accomplished:

-Planning, assignment and management of the target goal activities -Setting and monitoring of virtual meetings -Workspace organisation and maintenance -Monitor the completion of task planning, by notifying group members for any eventual delay or failure to fulfil a given task -Mediate and provide support to the group members when needed -Prepare the definite version of the product that has to be delivered at the end of the phase, watching for its robustness and coherence -Deliver the collaborative product and the group functioning report; the latter is elaborated by all members and describes how they collaborated as a team to develop the product, including all significant matters or problems that took place during the collaborative learning process.

The assessment of this collaborative learning practice is a crucial aspect, so the aim of this work is to provide the means to carry out an efficient and in-depth evaluation of each student's work and learning. To that end, the tutor effectuates a qualitative evaluation of the collaborative learning process which is explained in detail in section 3. This evaluation is further supported and complemented by two more analysis techniques, a Quantitative and Social Network Analysis, which form the layers of a complete framework for evaluating collaborative learning interactions. We now turn to describe our approach.

Setting the Basis of a Layered Conceptual Model for

Evaluating Collaborative Learning Interactions

Collaborative distance learning involves a variety of elements and factors that have to be considered and measured in order to analyse and assess group and individual performance more effectively and objectively [START_REF] Collazos | Collaborative Scenarios to Promote Positive Interdependence among Group Members[END_REF][START_REF] Katz | The Cognitive Skill of Coaching Collaboration[END_REF]. Our approach integrates different data sources, tools and techniques which allow the tutor to supervise, guide and evaluate the collaborative learning process.

First, analysis data may come from different sources such as the group activity log files maintained in the collaborative learning system. Log files capture the actions performed by the group participants on the shared workspaces. Other important sources are the content of these actions and of the collaborative activity products, group and individual self-assessment reports, reports and/or logs of virtual meetings (synchronous communication and interaction), as well as a final self-evaluation questionnaire. Both reports and questionnaire aim at extracting specific information from the participants related to task achievement (the activity product), the learning processes and the quality of the collaboration itself.

Second, specific (custom-designed) software tools can be also used as supporting means for the filtering and processing of the above data as well as for the visualisation of the information and knowledge derived from the interaction analysis. Finally, analysis or evaluation techniques may involve several qualitative, quantitative or other approaches that should be combined appropriately to produce an effective study and assessment of the collaborative learning interaction. Previous work explained in [START_REF] Daradoumis | Exploring Interaction Behaviour and Performance of Online Collaborative Learning Teams[END_REF][START_REF] Daradoumis | An Integrated Approach for Analysing and Assessing the Performance of Virtual Learning Groups[END_REF] focused on the development of such an initial integrated approach for analysing and assessing group and individual performance. However, a more grounded and holistic layered framework is needed to account for a more complete and effective evaluation of online collaborative learning interactions. This is the focus of the current paper.

Design of the basic layer of our framework: A qualitative analysis process

The need for a layered framework for evaluating collaborative interactions comes forth from the need of defining a top-down hierarchy of important indicators in order to be able to describe different aspects of group activity at different levels of detail.

To that end, based on the theoretical principles and indicators of effective collaboration of [START_REF] Macdonald | Assessing online collaborative learning: process and product[END_REF][START_REF] Mcgrath | Time, Interaction and Performance (TIP). A Theory of Groups[END_REF][START_REF] Sfard | On two metaphors for learning and the dangers of choosing just one[END_REF][START_REF] Soller | Supporting Social Interaction in an Intelligent Collaborative Learning System[END_REF][START_REF] Webb | Testing a theoretical model of student interaction and learning in small groups[END_REF] we specified four high-level indicators, namely task performance (or learning outcome), group functioning (or participation/ interaction behaviour), social support, and help services (or task/process scaffolding), which represent high-level collaborative learning processes [START_REF] Daradoumis | An Integrated Approach for Analysing and Assessing the Performance of Virtual Learning Groups[END_REF]. For sake of convenience, we reproduce these indicators (and their associated weights) in Table I below.

The association of weights to each indicator is an important feature of our approach since it determines not only the importance of each evaluation means but also the way these means can be combined to carry out the analysis and evaluation process. The assignment of the weights was performed according to the needs of the particular case study, so the weights are not fixed but they are worked out and settled by the evaluator for each case. In general, this depends on premises, such as the evaluation goals, the context or situation surrounding the collaborative learning experience and its specific tasks, as well as the available evaluation techniques and data sources.

Consequently, Table I shows a way to set weights for the analysis and assessment of our particular case study. We are currently exploring a more principled mechanism (such as a regression statistic model) to derive relative weights for each indicator.

At this level, the indicators are more generic (composite), so they form the top layer of our framework and they can be best measured and evaluated by a qualitative analysis method. A qualitative analysis of these indicators enables the tutor to infer some specific conclusions about individual and group performance or competence by distinguishing some particular cases of inadequate contribution or interaction behaviour as well as insufficient supply of social support or help. Our case study offers the tutor the context to perform a continuous qualitative evaluation of the students' work and collaborative activity. Thus, a formative qualitative evaluation takes up an important value and constitutes the basis of our framework. In fact, all the four analysis aspects are measured and assessed qualitatively by the tutor at the end of each project phase as follows.

In each phase of the case resolution, every group delivers the tutor its learning products (the outcome of the collaborative work toward the achievement of the target goal). The tutor corrects it thoroughly, which allows him/her to assess the group problem-solving capabilities (indicator variable TP1), as well as to send his/her feedback to the group. In addition, during each phase the tutor performs the following analysis tasks:

• A selective qualitative examination of the students' most significant actions/ contributions to the task. The tutor is able to determine these types of actions in two ways: First, the BSCW system itself allows the creator of a contribution (e.g., a document) to rate it by assigning it a value that ranges from poor, passable, fair, good to excellent. Second, the study derived form the Social Network Analysis helps the tutor identify which actions of an actor were most accessed (read or modified) by the other members. Then, on the one hand, the evaluation of the content of these actions enables the tutor to reason out the production function, problem-solving capabilities and the active learning skills exhibited by each group member (indicator TP2), as well as to get an initial evidence of the type of social support and help provided by each member, especially the coordinator of the phase. On the other hand, the identification of the type of actions (create, read, change or move) performed by each member allows the tutor to draw some initial conclusions about the active participation behaviour and social grounding of the group (indicators GF1, GF2).

• A qualitative analysis and assessment of a group self-evaluation report elaborated by all members of the group and delivered by the coordinator of the phase. This report presents an evaluation of group work and is guided by specific questions that aim at knowing the students' personal opinion, perception and impression about individual contribution and overall group performance regarding the task (indicator TP3), as well as the process and the quality of group functioning, social support and help services. As regards the last three aspects, there is at least one question addressing and measuring every particular indicator variable of each aspect.

• A group report that results from the interactions that take place at the virtual meetings of the group. This report provides further information about each generic aspect and its indicator variables.

When a group delivers its learning outcomes and group functioning report, the tutor communicates back to the group an assessment mark 1 as well as valuable remarks about the workings of the group (that includes the problems identified and suggestions for overcoming them). Consequently, each member prepares a specific report in which, on the one hand, he/she proposes the tutor a mark the member believes he/she deserves for him/her self; on the other hand, the member assigns a mark to each of his/her peers. The goal of this self-assessment is to make the students reflect upon the self, peer and group work, learning and performance.

Indeed, each member has to describe the specific tasks of the target goal in which he/she led their resolution. In addition, the member has to outline his/her collaborative contributions to the rest of the tasks. All this guides the tutor to decide better about the final mark of each individual member. Given the different synergies that take place in a group, the individual mark may be different for each group member.

Finally, at the end of the case resolution, each student has to work out a detailed self-evaluation questionnaire. Being at the end of the collaborative learning experience, the questionnaire now gives students the possibility to carry out a critical reflection on the degree of achievement of all the four main aspects of the collaborative learning process, that is task performance, group functioning, social support and help services. In doing so, they are able to perform an ultimate self, peer and group assessment of the whole work carried out during the experience, and reason out the degree of achievement of their learning objectives and group performance.

We have seen that the qualitative evaluation process is spread along the four toplevel analysis axes. As such, we consider it to be the basic layer of our framework upon which further analysis layers (more specific indicators, techniques, data and tools) can be incorporated and applied. These layers work in parallel with the basic qualitative evaluation layer and are used by the tutor to fill whatever gaps left as well as to evaluate all those issues that could not be covered sufficiently by the qualitative evaluation method alone. We now turn to discuss the subsequent layers of our framework.

Completing the Layered Model of Interaction Analysis

Composite or high-level indicators, as the ones defined in Table I, provide a basis for evaluating collaborative learning interactions as well as group and individual performance. However, our work in [START_REF] Daradoumis | An Integrated Approach for Analysing and Assessing the Performance of Virtual Learning Groups[END_REF] showed that these indicators alone, as measured and interpreted by the qualitative evaluation process above, do not suffice to provide an in-depth and effective evaluation of collaborative learning interactions.

To that end, we need to determine more specific indicators at a mid and low level, as well as to develop and apply new analysis techniques associated to further data sources and supporting tools. In doing so, our framework is endowed with further specific layers of interaction analysis.

At a conceptual level, we proceed to decompose the high-level indicators defined in Table I to more refined ones. So far this decomposition has been effectuated for the task performance and group functioning aspects. Due to their intrinsic difficulties, social support and help services aspects are currently under a more thorough study.

Table II shows the mid-and low-level indicators in the form of the skills and subskills that students should have in order to achieve effective group and individual performance regarding the task and thus obtain a successful learning outcome. To measure each indicator (or skill), we associate it with the actions that students perform and which represent each indicator in the best possible manner. Though we have established this correspondence between skills and BSCW actions, our approach can be easily adapted and tailored to the targeted groupware platform which is supposed to be used.

Table III shows the mid-and low-level indicators in the form of the skills and subskills that students should exhibit in order to enhance participation, promote better communication and coordination, and thus achieve the effective interaction and functioning of the group in a Web-based collaborative situation. Again, to measure each indicator (or skill), we associate it with specific student actions which best describe each skill to be accomplished. To describe and measure these specific indicators, we developed two more analysis techniques which, in conjunction with the qualitative evaluation process, provide a more complete framework of interaction analysis. Next, we describe the analysis layer that is built by the Social Network Analysis technique which focuses on analysing two very important aspects of group functioning: active participation behaviour and social grounding.

A Social Network Analysis of group activity and participation behaviour

At first, Social Network Analysis (SNA) [START_REF] Wasserman | Social Network Analysis: Methods and Applications[END_REF] has proved to be an adequate and sufficient technique to analyse the structure of the social interactions that take place in the virtual workspaces (first two indicators of Table III). This structure allows for the study of individual properties (prominence of the actors), small groups and the whole network. Indeed, SNA seeks to describe patterns of relationships among actors, to analyse the structure of these patterns and discover their effects on people and organizations. Social networks can be visualized as graphs called sociograms, which

Meeting accommodation

ChangeDesc/ChangeDate /ChangeLocation Appointment represent the actors as nodes of the graphs and the links among them as lines in the graph. Several studies have demonstrated its value within the CSCL field for the study of structural properties of individuals learning in groups [START_REF] Cho | Social information sharing in a CSCL community[END_REF][START_REF] Wortham | Nodal and Matrix Analyses of Communication Patterns in Small Groups[END_REF].

The basic source that provides data for this type of analysis is the BSCW daily log files. Every log file records all the interaction data (events) occurred in all active BSCW workspaces. As seen in Tables II and III, BSCW distinguishes and generates four generic action types related to an object: Create, Change, Read and Move events.

To support SNA automatic processing, we use a tool called SAMSA (System for Adjacency Matrix and Sociogram-based Analysis). This tool contains several input modules, one of which takes data from BSCW event logs and transforms them into an XML file representing the interactions. Then, SAMSA allows us to select and configure the network we want to study (selecting dates, actors, and relationship type). The tool builds the matrix that represents the network, known as sociomatrix, and computes the indexes chosen for the analysis. It also shows the sociogram representing the network, and allows for visualising the actors' attributes.

The SNA technique can be used in a top-down analysis approach so that the evaluator can start from a very general perspective of the classroom interactions and detect which are the groups or actors that need further analysis. In order to perform Social Network Analysis, it is necessary to define the networks and the set of indexes that will be used for the study.

Networks are relationships established among a set of actors. In this study we considered the relationships composed by the indirect links between an actor that creates an object in the BSCW workspace and those that access this object in order to read it. This is by far the most frequent type of interaction in the context where BSCW is used as shown by the daily report and the log files maintained by the system. The set of actors included both the students and the teachers.

We have identified a set of SNA indicators for the study of participatory aspects of learning, which were the ones used in this study, namely: Network density (D), actor's degree centrality (C D (n i )), and network degree centralization (C D ) [START_REF] Martínez | Combining qualitative and social network analysis for the study of social aspects of collaborative learning[END_REF]. D measures how knitted a network is, with values ranging from 0 (most sparse) to 1 (most dense).

Degree centrality is an index of the actor's prestige. Given an actor n i , C D (n i ) is the proportion of actors that are adjacent to n i . It reflects the activity of the actors. In the case of directed relationships that consider the direction of the link, two degree indexes are defined: indegree, or the number of links terminating at the node; and outdegree, or the number of links originating at the node. Finally, network degree centralization (C D ), is a group-level measure based on actor's degree centrality. It gives an idea about the dependency of the network on the activity of a small group of actors. Its values range from 0 (even distribution of activity) to 1 (most centralized network). Directed networks define the corresponding indexes of indegree centralization (C ID ) and outdegree centralization (C OD ). All of these indexes and ranges apply to dichotomous relationships that can have only one out of two possible values: 0 when there is no link and 1 when there is a link between two actors. It is also possible to consider valued relationships that include a number showing their strength. The indexes computed on these relationships are more difficult to generalize than those computed from the dichotomous relationships, but sometimes are important to provide additional information. All of these indexes provide basic information about the activity of the actors in the network and about the global structure of the network according to different relationships. Moreover, they are simple to understand and to interpret, which are important features for facilitating their use by evaluators, who are not expected to be experts in SNA methods. These indexes were applied in order to study and compare interactions at the general and at the private workspaces, as well as to identify who were the more and the less active students at both levels. The following two subsections develop these issues.

Student Participation in the General Workspace

The first aspect we wanted to analyse was the general structure of the relationships in the classrooms, which was studied by the indirect relationships network at the general workspace of the virtual classroom. A first analysis can be performed by studying the network of the aggregated relationships during the last four phases of the course, once the groups had been created and the students were focused on their project-oriented tasks. The indexes of this network (D=14,24%, C ID =42,22%, C OD= 63,33%) show that the indirect links considered in this network were quite frequent (if we take into account the size of the network) but too much centralised as regards both reading (C ID ) and specially writing (C OD ), which means that the activity was concentrated on a very reduced set of actors.

The examination of the sociogram of this network (Fig. 1) allows us to go deeper in this issue in a very intuitive manner that can be easily built by the teacher or evaluator by means of SAMSA. The actors are represented by different shapes according to the group they belong to, and the links between them as directed lines that go from the actor that creates a document to the one(s) that read it. While the high concentration of the lines and arrows makes difficult a detailed analysis of the specific links, it is still possible to draw some initial conclusions that complement the values mentioned above. Firstly, it can be observed that some actors appear as isolated nodes at the left, which means that they did not intervene in the shared workspace at all. It is also possible to see that the teacher (ifa) has a central position in the network, shared with an important number of students. It is possible to identify at a glance who were the most active students (at the centre) and the less active ones (at the periphery). For example, actors like xva, cao, dac, and fca played a prominent role, while others, like jga, Ton, ngu, etc. played a peripheral role, as their only connection to it consists of a single link to another actor.

The exploration of the actors' centrality values, which is also calculated by SAMSA, complements the analysis of the sociogram, as they allow identifying the most active students, with the added value that these indexes inform also about the reading and writing activities. It is especially interesting to detect who were the students with a higher value of out-degree centralization, which means that they wrote documents that were read by more students. According to [START_REF] Cho | Social information sharing in a CSCL community[END_REF], this index can be considered as a measure of actors' prestige, and can help a teacher to detect who are the students which act as leaders in the sense measured by the network. In our study the students with a higher out-degree centrality value were san, fer, jur, fgu, and car.

The analysis performed so far shows a static view of the aggregated relationships during the course. This view was complemented with an analysis of the evolution of the networks, which is performed by comparing the indexes and sociograms obtained for each one of the phases in which the course was divided. It is not possible to show the details of this evolution for lack of space, but we comment here the main results: density remained stable through the course (with values around 5-6% in each one of the phases), with a slight drop at the last phase; and the prominent actors we have identified above showed a quite regular participation throughout the course.

Interestingly the teachers, in spite of their high level of activity in the general workspace, are never on the top position, which is considered a positive indicator in the sense that the students (or at least some of them) indeed got involved in the classroom activities. As it has been explained above, the activity at the private workspaces has a different nature from the one performed at the general workspace. While the analysis of the general workspace shows the structure that yields from the interactions that take place in the context of discussions that are indirectly related to the fulfilment of the assignments, collaboration at the private group level is mainly focused on the development of a common product (or a set of common products), namely: the set of assignments required by the teacher for the fulfilment of the project tasks.

In order to analyse this workspace, we performed a similar study to the one previously presented. Fig. 2 shows the indirect relationships network at the group private workspaces during the same period of time as the one previously considered.

As expected, it displays a rather different structure, due to the restricted access to each group workspace, which results in a set of independent sub-networks that represent the relationships in each group. By simple observation of the sociogram it is easy to detect which were the most and the least active and homogeneous (balanced) groups.

These groups are respectively labelled X and Y. The evolution of these groups is shown by the indexes in Table IV. We can observe that, while group X has a 100% of density and a 0% of centralization during all the phases of the course, group Y never achieves these desired values, with the lowest value at the last period of the course. This difference is even more outstanding if we compare the valued densities obtained by both groups throughout the course. On the one hand, according to [START_REF] Wortham | Nodal and Matrix Analyses of Communication Patterns in Small Groups[END_REF], group X shows an ideal pattern of interaction, with all the members interacting with each other, and none of them taking a too central position. Moreover, if we go back now to the indirect relationships at the general workspace network analysed in the previous subsection, we can observe that most of the members of group X are at the centre of the network and belong to the list of prominent actors (actually, only car does not occupy a central position at the sociogram of Fig. 1). This can mean that the members of group X obtained a good result regarding the first two indicators of Table III. On the other hand, looking at the workings of the ineffective group Y, it seems that there are two members, jdo and por, that show to be the most prominent actors in the group, thus leading the group as regards active participation behaviour. The contributions of the rest of the members seem quite poor, which yields in a quite unbalanced group as regards both active or reaction attitudes and role playing.

As seen from the above, the SNA technique presents the evaluator a useful but general perspective of group interactions, especially since it detects which groups or individuals need further analysis. For this reason, it constitutes an important layer of our framework through which the evaluator is able to have a global view of the participation and activity of all the groups and their members and to detect problematic groups or individuals. As a consequence, further analysis of the behaviour and performance of specific groups and members can be carried out by both qualitative and quantitative techniques.

Next, we focus on the description of a quantitative statistical analysis that explores the interactions (action types) that capture and describe the members' skills which are related to task achievement (Table II) and group functioning (Table III).

A Quantitative Analysis of Group Performance and Functioning

The quantitative analysis performed here is a descriptive statistical analysis that aims to provide a complementary and more focused analysis of the task performance and group functioning aspects of the groups X and Y, more importantly to explore and understand the real performance and achievement of each member. This analysis proves to be a necessary aid to the SNA carried out before, on the one hand, for identifying patterns of effective or ineffective collaboration at group and individual level, on the other hand, for determining particular details on the attitudes of group members not been able to be tracked through SNA.

As in Social Network Analysis, the basic source that provides data for the statistical analysis is the BSCW daily log files. However, the way the content of these data is presented by these logs makes it difficult to structure and analyse the information. For this reason, to facilitate the quantitative analysis, we initially use a specific software tool that extracts and filters the data contained in the event logs according to desired parameters defined by our analysis needs (for instance, events can be classified by user and action type, or can be distributed in specific periods of time).

Based on the indicators that model task performance and group functioning of Tables II and III, we provide a comparative analysis of groups X and Y as regards the interactions that took place in their BSCW workspaces. SNA showed that group X has been one of the most active and well-balanced groups of the experience whereas its members were also considered as prominent actors (to a greater or lesser degree) in the general classroom workspaces, whereas group Y proved to be quite the opposite.

As a consequence, the aim of this study is twofold: First, it determines patterns of collaboration that show how the four basic actions (create, change, move and read) are globally distributed in a group and how individual contributions are distributed among members in the group. This helps the tutor to identify at first sight whether a group or member is performing an effective or ineffective collaboration, that is, it constitutes an important initial level of awareness which complements the results obtained from the Social Network Analysis.

Second, it can provide a more detailed analysis that covers all the specific indicators of Tables II andIII. This analysis can reveal which particular indicators a member fails to achieve, allowing the tutor to identify insufficient contribution on specific aspects of task performance or group functioning, and even detect whether there is a member who is "over-leading" a particular aspect. This forms a distinct level of awareness and contributes to a more effective and objective assessment of the group and its members. It also enables the tutor to intervene to point out and correct a non-desirable participation or contribution of a member.

The results of the analysis (the values of the indicators) are displayed to the tutors (and accordingly to the groups) via specific forms: numerical and/or diagrammatic.

Both visualised forms currently concern the variation of an indicator variable (e.g., create) in relation to the values of another indicator variable (e.g., read). Their purpose is to describe and compare the state, process and quality of group or individual activity and performance.

In particular, Fig. 3 and 4 present a comparative study carried out between the effective group X (the learning outcomes delivered to the tutor were of a very good quality) and the ineffective group Y (with rather poor learning outcomes).

Fig. 3 shows a quite different distribution of the action types performed in the two groups. In fact, as concerns group Y, the percentage of the read actions which occurred is out of proportion (too low) with respect to the percentage of create actions (very high). Fig. 4 shows that only one member (jdo) was very active in reading. The rest of the members (especially members jta and xga) showed very little interest in reading. This means two things: First, most of group members were not aware of the creating or changing actions of their peers. Second, though the creative activity seems high, it seems not significant enough to pay attention to by the members. So the quality of the creative contributions proved to be quite low. Moreover, Fig. 4 shows that most of the create actions was carried out by two members (jdo and por).

Going back to the SNA analysis (Fig. 2), we can see that members jdo and por are at the leading position of the group, since they do not only create documents but their documents are read by the rest of the members. This is not the case for the rest of the members; especially member xga is found at a complete isolated position.

Moreover, the high amount of move actions (in Fig. 3) sharpens more the disproportion shown in group activity. This action type refers to workspace organisation and maintenance, a function attributed to the coordinator role of the group, and it should not be so high. From Fig. 4, it seems that only members jdo and por satisfied this coordinator function sufficiently.

As seen from the above, the analysis done at this initial level provides a general but sufficient perspective of a group's state of collaboration and its members' degree of participation and contribution. At the end of the experience, it allowed us to draw the conclusion that an ineffective group, like Y, may show the pattern of group interaction shown in Fig. 3 and the one of participant interaction shown in Fig. 4.

Using this pattern at an early stage of the collaborative learning process, the tutor is able to predict problematic behaviours and take appropriate measures to correct them.

In contrast, the distribution of the four action types of the effective group X, shown in Fig. 3, can be considered as an acceptable pattern of group activity. A first observation is that the read action constitutes the 80% of the whole activity that took place in the group. This fact manifests that all members were fairly aware of the documents/messages created in the group's workspaces. In fig. 4 there is a further evidence of that; most importantly, the read action is one of the most balanced activities performed by the group members.

Having been an effective group at the end of the experience, the proportion of the four actions shown in Fig. 3 can be considered to provide a pattern of effective collaboration, at least as regards the quality of the collaborative learning outcomes produced. As for the individual contributions shown in Fig. 4, the two most important actions (create and read) show to be within acceptable limits that configure a pattern of effective interaction, the realisation of the change action presents some deviation as concerns members cao and car (with less participation at this aspect), whereby the move action was basically centred on two members (fer and san). As such, group X does not exhibit an ideal pattern of effective collaboration and interaction, though it came quite close to that. In general, our approach does not seek to identify the ideal group patterns, since the definition of good (or bad) groups is quite subjective. The real aim of our approach is to help the teacher evaluate a specific situation and let him/her extract as much information as possible about group and individual performance. At this point, it is interesting to perform a more refined analysis of the workings of group X so that to see in detail which are the weak points of its collaborative activity, that is, which members achieve insufficient values for which indicators of Tables II and III. This level of analysis was carried out in [START_REF] Daradoumis | An Integrated Approach for Analysing and Assessing the Performance of Virtual Learning Groups[END_REF] and showed that members fer and fgu had a distinguishing contribution to task achievement, that is, they had high values as regards the indicators of Table II, in comparison to their peers (jur and san had a rather middle-rate contributing activity, whereby cao and car achieved the lowest rate in the group). As concerns the indicators related to group functioning (Table III), member fer showed a distinguishing performance (he achieved high values at all indicators), whereby the rest of the members achieved a good value only at very specific indicators, which means that they played a rather supporting (though important) role at this aspect of collaborative process. More details of this analysis level can be seen in [START_REF] Daradoumis | An Integrated Approach for Analysing and Assessing the Performance of Virtual Learning Groups[END_REF].

As shown above, the comparison of basic activity of an effective and ineffective group indicated different patterns of group interaction that correspond to each case and allows the tutor to have, at first sight, a quite clear picture of the workings of the group and to intervene in order to advise and guide at both group and individual level.

We have also shown that, if necessary, further detailed statistical analysis can be performed to measure and evaluate specific indicators related to task performance and group functioning in order to identify insufficient values, which allows the tutor to do specific corrective interventions to particular members.

Finally, the layered definition of a variety of indicator variables provides the tutors, who are in charge of several groups of students, layered visualisation forms of the analysis results, with several levels of detail. The same information can be also provided to the students in different formats and granularity.

Reflections on the Approach Followed

Our study showed that the evaluation of real collaborative learning interactions is a very complex task, since one has to consider a variety of aspects. To accomplish this task we proposed an approach that builds on a layered framework which consists of a hierarchy of high-mid-and low-levels indicators that describe collaborative learning interactions at several levels of description and as sufficiently as possible, whereas it integrates several analysis techniques, data sources and supporting tools. The analysis techniques included a formative qualitative evaluation, a social network analysis, and a quantitative statistical analysis. The use of all these different techniques proved to be complementary and was guided by the indicators that each technique best accomplished. The classification of each indicator into a specific category (aspect) or skill of the collaborative learning process dictates the way each technique is used and influences the evaluation process and how it is positioned and related to the others.

The evaluation and comparison of a specific effective group X and an ineffective one Y, on the one hand, indicated specific patterns of effective or ineffective collaboration; on the other hand, it showed that the application of different analysis levels is essential to unfold the group's internal workings and achieve a more objective interpretation of each member attitude and competence.

The ultimate aim behind the layered approach used for evaluating collaborative learning interactions is to provide a more complete vision to interaction analysis and scaffolding, by tailoring collaboration analysis indicators, methods and presentation of the analysis results to different types and profiles of users as well as to different cognitive systems involved in collaborative learning settings.

On the one hand, the definition of different layers of analysis provides a generic framework that can be adapted to different cognitive systems (individuals, peers, group, classroom, and teachers) and give support to their specific needs; that is, it can be used to allow students' self-regulation, facilitate peer and group evaluation, enable classroom monitoring by pedagogic coordinators and help the teachers themselves to control and assess the collaborative activity and learning more effectively. This paper presents a way that the approach can be applied to a particular case, where a cognitive system (e.g., an evaluator) adapted the framework to a concrete scenario by defining specific interaction analysis indicators and providing different levels of analysis.

All in all, the framework provides each cognitive system with the means to define adequate indicators and analysis techniques so that the form and details of information derived from the analysis to conform and account for the needs of each cognitive system accordingly. For instance, every teacher could use this framework as a guide to define his/her own particular viewpoint of product and collaboration quality, that is, to determine what aspects are important to evaluate, what indicators best describe these aspects, and what weights to attribute to each of them. In doing so, our framework can, in principle, be tailored to every specific situation and context according to the main criteria used to define product and collaboration quality. As such, the analysis can produce a variety of outcomes, ranging from identifying general patterns of effective or ineffective collaboration (as shown in this work) to determining particular participatory attitudes when carrying out different tasks.

On the other hand, this approach could account for and measure the different roles and functions assumed by the participants in a collaborative learning experience;

indications of how the latter can be explored and achieved have been presented in this work.

Another problem that our approach wants to address effectively is the identification of the needs of each learner and other potential actors (including the teacher) in every moment and be able to decide what information is required to provide, in which granularity and how to present it. Moreover, users may come from different backgrounds (interdisciplinary groups) and thus have different cognitive and metacognitive skills, needs, interests, motivation, time availability, contexts or conditions of learning. This is a very complex issue to address sufficiently, but we believe our approach can constitute a starting point to explore it.

Finally, another issue addressed in this work is related to the ethical and information privacy problems that result from the observation of students' contributions and actions that take place on the shared workspaces. Students were certainly informed and aware of the fact that their actions were registered by the system. They were told that their products, reports and the actions recorded in log files would go through a detailed analysis for the purpose of our research and this would have a positive impact to their work and learning as well. For this reason, they had no objection to that; instead, they showed very eager to participate and they did it in a very natural manner.

In addition, students showed a supportive attitude if any group members faced a problem at a certain moment of the case/project development and they were not able to contribute to the task and group functioning as expected. However, they were usually not willing to accept a lengthy unjustified inactivity or a deficient contribution or participation of a member and thus they expected the teacher to intervene and evaluate accordingly.

Conclusions and Future Work

In It is important to remind here that the whole evaluation process took place in real settings, not experimental ones, which gives an added value to the approach.

Moreover, the approach is general enough so that it can be easily tailored to allow the monitoring of the interaction by the teacher as well as to facilitate students' selfregulation of their learning activity.

As regards the first, the teacher can supervise the individual and group learning process by applying the mixed evaluation scheme during and at the end of each phase of the case study. For instance, the SNA method will enable the teacher to identify at a glance who are the most or less active students as well as which are the problematic groups, then suggest him/her to perform further analysis of specific actors or groups by means of the quantitative and/or qualitative techniques and finally allow him/her to intervene accordingly. Thus, our framework gives the teacher the possibility to observe and collect data on each group as it works in a systematic way. Then the teacher can intervene, when he/she considers it necessary, to provide task assistance or to enhance group functioning (i.e., students' interpersonal and group skills) [START_REF] Johnson | Cooperation in the Classroom[END_REF].

As for the second, the knowledge extracted from the analysis can be fed back to the students in appropriate formats and detail levels as awareness information about their activity, behaviour and performance regarding all the four basic collaborative learning processes, (i.e., the task, group functioning, social support, and help). Since the knowledge provided to the students can be structured and classified according to high-, mid-and low-level indicators, students not only become aware of their strong and weak points during collaboration, but also they receive adequate feedback and indications that can help them self-regulate their activity and attitude for those aspects needed.

In general, there are many variables that could influence task performance, group functioning and the success of a collaborative activity. Certainly, this work does not aim at covering all of them. For instance, some more variables, not taken into account, that could be also included to assess task performance could be: cognitive empowerment (self-esteem, self-knowledge, self-efficacy in the domain of interest, in this case, learning), locus of control, self-knowledge, ambition, general efficacy, motivation to action and community orientation, capacity for life-long learning, attitudes to information technology, and attitudes to collaborative work.

Kozma [START_REF] Kozma | Students collaborating with computer models and physical experiments[END_REF] has found, in an analysis of student interaction, that the amount and nature of collaboration between partners had less to do with the availability of computer software and more to do with the way the instructor designed and structured the task. The instructional design of our case study surely played an important role in the success of the collaborative activity. For this reason, at the end of the experience, the outcomes of the evaluation provided us several insights about the effectiveness of the particular design of the case study and prompted to specific improvements.

In any case, the proposed model tries to include as many interaction aspects as possible that may occur in a collaborative activity. As such, it constitutes a generic framework that does not currently offer specific guidelines to help teachers in order to make decisions about the presented analysis; instead, it provides resources that any teacher can use in every case in the best possible way. Experience with the framework may end up providing guidelines that correspond to and describe repeated situations.

Further work will show whether this can be possible.

Finally, further work focuses, on the one hand, on developing a more comprehensive multi-level statistical model of interactions; on the other hand, we are working to develop automated awareness, assessment and scaffolding tools that provide appropriate, structured information of the state of the interaction in different formats, for different users, specific needs and interests, contexts or conditions of learning.

Fig. 1 .

 1 Fig. 1. Indirect relationships network at the general workspace during the project working period

Fig. 2 .

 2 Fig. 2. Sociogram representing the indirect relationships network at the group workspace. The line thickness represents the intensity of the links between actors.

Fig. 3 .

 3 Fig. 3. A global comparison of the activity of an effective and ineffective group

  this work we have presented a layered framework for evaluating collaborative learning interactions. The layers of the framework were initially specified by defining high-, mid-and low-level indicators that explicitly (and implicitly) model different aspects of group interaction. Through these indicators the model intends to provide a conceptual understanding of the collaborative learning process that takes place in a group. To that end, we associated a qualitative evaluation method to measure the high-level indicators whereby mid-and low-level indicators were interpreted through a SNA and quantitative analysis techniques. Finally, specification of the source data and specific supporting tools associated with each analysis techniques complete the components of the framework. The overall interpretation of group interaction and performance is achieved by relating all the available descriptive indicator values.

Table I .

 I Description of the high-level interaction analysis indicators

	GF4	Group processing (examine whether each member learnt how to	20%
		interact and collaborate more effectively with his/her teammates)
	Social support	15%
	SS1	Members' commitment toward collaboration, joint learning and	30%
		accomplishment of the common group goal
	SS2	Level of peer involvement and their influential contribution to the	30%
		involvement of the others
	SS3	Members' contribution to the achievement of mutual trust	10%
	SS4	Members' motivational and emotional support to their peers	20%
	SS5	Participation and contribution to conflict resolution	10%
	Help services	15%
	HS1	Help is timely	25%
	HS2	Help is relevant to the student's needs	10%
	HS3	Help is qualitative	30%
	HS4	Help is understood by the student	25%
	HS5	Help can be readily applied by the student	10%
	Indicators	Weight
	Task performance	50%
	TP1	The students' individual and group problem-solving capabilities and	40%
		learning outcomes (acquisition metaphor)
	TP2	The students' contributing behaviour during task realisation	40%
		(production function and use of active learning skills)
	TP3	The students' individual and group ongoing (and final) performance	20%
		in terms of self-evaluation
	Group functioning	20%
	GF1	Active participation behaviour	30%.
	GF2	Social grounding (well-balanced contributions and role playing)	20%
	GF3	Active interaction or processing skills that monitor and facilitate the	30%
		group's well-being function

Table IV .

 IV Evolution of the values of density and centralisation for groups X and Y

			Group X			Group Y
	Phase	D	D v	C iD C oD	D	D v	C iD C oD
	Ph2	100 % 2833 % 0 % 0 % 63,33 % 186,67 % 25 % 25 %
	Ph3	100 % 1493 % 0 % 0 % 46,67 % 76,67 % 50 % 50 %
	Ph4	100 % 1696 % 0 % 0 % 6,67 % 13,33 % 50 % 20 %
	Ph5	100 % 980 % 0 % 0 % 23,33 % 43,33 % 55 % 25 %

We use a 5-point scale mark A (excellent), B (fairly good), C+ (good or passable), C-(not passable) and D (fail)

Acknowledgments

We thank our colleagues at UPC, UOC and UVA for their human and scientific support as well as all the students who eagerly participated in this experience. This work has been partially supported by Spanish MCYT project TIC2002-04258-C03-03.