Jose Sanchez

Carlos M Mateo

Juan Antonio Corrales

Belhassen-Chedli Bouzgarrou

Youcef Mezouar

Online Shape Estimation based on Tactile Sensing and Deformation Modeling for Robot Manipulation

Precise robot manipulation of deformable objects requires an accurate and fast estimation of their shape as they deform. So far, visual sensing has been mostly used to solve this issue, but vision sensors are sensitive to occlusions, which might be inevitable when manipulating an object with robot. To address this issue, we present a modular pipeline to track the shape of a soft object in an online manner by coupling tactile sensing with a deformation model. Using a model of a tactile sensor, we compute the magnitude and location of a contact force and apply it as an external force to the deformation model. The deformation model then updates the nodal positions of a mesh that describes the shape of the deformable object. The proposed sensor model and pipeline, are evaluated using a Shadow Dexterous Hand equipped with BioTac sensors on its fingertips and an RGB-D sensor.

I. INTRODUCTION

As interest in automating applications that require manipulation of soft objects keeps increasing, promising results have recently been achieved by applying robotic manipulation in a diversity of industries where deformable objects are commonplace. For instance, in the food industry, robots have been used to debone chickens [START_REF] Hu | Intelligent automation of bird deboning[END_REF] and for meat cutting [START_REF] Long | Force/vision control for robotic cutting of soft materials[END_REF]. The aerospace and automotive industry could reduce manufacturing costs as robots become able to manipulate cables for assembly purposes [START_REF] Shah | Towards manipulation planning for multiple interlinked deformable linear objects[END_REF]. Automating surgical procedures would benefit the health industry, as in the case of robotic suturing [START_REF] Jackson | Needletissue interaction force state estimation for robotic surgical suturing[END_REF]. Robot manipulation of soft objects has also domestic applications such as automated folding of clothes [START_REF] Maitin-Shepard | Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding[END_REF] and assisting impaired people dress themselves [START_REF] Yamazaki | Bottom dressing by a life-sized humanoid robot provided failure detection and recovery functions[END_REF]. A survey of further robot-based applications for manipulating deformable objects can be found in [START_REF] Sanchez | Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey[END_REF].

Despite these advances, many challenges still remain unsolved in automating tasks involving deformable objects. One such challenge, is that of manipulating the shape of a deformable object, most recently referred to as shape servoing [START_REF] Navarro-Alarcon | Automatic 3-D Manipulation of Soft Objects by Robotic Arms With an Adaptive Deformation Model[END_REF]. To perform shape servoing on a deformable object, the robot must be able to accurately track the object's shape in order to monitor its deformation until reaching the desired shape. In this paper, we propose an approach to continuously estimate the shape of a deformable object by coupling tactile sensing with a deformable object simulator. Here, the contact information, provided by the tactile sensors, is used by a deformation model in order to estimate the shape of the object while it deforms. We identify our contributions as follows:

All authors are with Université Clermont Auvergne, SIGMA Clermont, Institut Pascal BP 10448, F-63000 Clermont-Ferrand, France Corresponding author jose.sanchez.loza@gmail.com

• A sensor model to estimate a contact's magnitude and location. • A modular pipeline, combining tactile sensing with a deformation model, that estimates the shape of an object as it is deformed by a robotic hand 1 . • Quantitative evaluation of the sensor model and the deformation sensing pipeline. Following this introduction, we review related works on deformation tracking in Section II. In Section III we describe our approach and its evaluation is presented in Section IV. The results of our evaluation are discussed in Section V and finally, our conclusions and future work are outlined in Section VI.

Z X Y

World frame

Fig. 1: Real deformation, shown on the left side, compared to the estimation of the proposed approach, visualized in RViz2 , shown on the right side.

II. RELATED WORK

Although some approaches have considered controlling the shape of a deformable object via a robot manipulator by either assuming that the object's shape is available [START_REF] Berenson | Manipulation of deformable objects without modeling and simulating deformation[END_REF] or by using fiducial markers to extract a set of points that describe the object's shape [START_REF] Navarro-Alarcon | Automatic 3-D Manipulation of Soft Objects by Robotic Arms With an Adaptive Deformation Model[END_REF]; some real-life applications might not be feasible without directly sensing the shape of the deformable object. Thus, the ability to track the shape of an object while being deformed would greatly benefit robotic manipulation tasks.

In order to track the changing shape of a soft object, various approaches have been proposed using different sensing modalities such as vision and force, and by relying on mesh models (e.g. based on the Finite Element Method (FEM) or on a Mass-Spring formulation) or on mesh-free models such as Meshless Shape Matching (MSM). For instance, Cretu et al. applied a neural gas network to the output of a vision system, to track the contour of an object while a robotic hand was deforming the object [START_REF] Cretu | Soft object deformation monitoring and learning for model-based robotic hand manipulation[END_REF]. To improve the tracking accuracy, they also trained a neural network to map position and force data (obtained by joint encoders and strain gauges at the fingertips of the robotic hand) to the deformed contour as computed by the neural gas network at every time step. Although this approach does not rely on a priori information of the object (e.g. mesh model, Young Modulus and Poisson ratio), it is limited since it only provides the contour on one side of the object that is visible to the vision system.

In contrast, an approach that required the elasticity parameters (i.e Young modulus and Poisson ratio) to be known a priori, was proposed by Tian and Jia [START_REF] Tian | Modeling Deformations of General Parametric Shells Grasped by a Robot Hand[END_REF]. This approach used an FEM simulation based on shell theory to estimate the deformation caused by a robotic hand on thin shell objects such as a tennis ball, where the inputs to the model were the contact forces of the hand while grasping an object. Similarly, but relying on visual sensing rather than force, Petit et al. recently proposed an approach that was able to perform at 35 Hz with accurate results [START_REF] Petit | Real-time tracking of 3D elastic objects with an RGB-D sensor[END_REF]. An approach that combined both, force and visual data, with an FEM model was described in [START_REF] Frank | Learning the elasticity parameters of deformable objects with a manipulation robot[END_REF]. Moreover, unlike the previous two approaches, the Young modulus and the Poisson ratio were not provided in advance and were instead estimated by minimizing the error between the observed and the simulated deformation.

Other alternatives to FEM modeling have been examined. For instance, Fugl et al. used an Euler-Bernoulli beam model to estimate the Young's modulus of a flexible barlike object to tracks its deformation [START_REF] Fugl | Simultaneous Estimation of Material Properties and Pose for Deformable Objects from Depth and Color Images[END_REF]. The approach required the object to be divided into sections having specific curvatures caused by the deformation. These curvatures and the object's pose, obtained via an RGB-D sensor, represented the deformation state of the beam. By having both, simulated and sensed, representations of the object's deformation, they could minimize the error between them as a function of the elasticity parameters. Mass-Spring models have also being used in combination with visual data as in the approaches proposed in [START_REF] Schulman | Tracking deformable objects with point clouds[END_REF], [START_REF] Leizea | Real-time deformation, registration and tracking of solids based on physical simulation[END_REF]. Although approaches based on Mass-Spring models are very fast, their drawback lies on their inaccuracy when large deformations occur. A recent approach that did not require a mesh was described in [START_REF] Güler | Estimating the deformability of elastic materials using optical flow and position-based dynamics[END_REF], where they relied on a position-based physics simulation known as Meshless Shape Matching (MSM). As the name implies, only position information is required, which was obtained through an optical flow algorithm applied to a sequence of images. However, as the images are taken from a static camera position, the deformation can only be estimated on one side of the object.

III. DEFORMATION SENSING PIPELINE

We propose a modular pipeline able to estimate the shape of a soft object being deformed by a manipulator. The pipeline is composed of a sensor model, a force transformation component, and a deformation model, as shown in Figure 2. The approach assumes that the contact forces, along with their locations, can be extracted via a sensor model. The deformation model takes a tetrahedral mesh (i.e. nodes connected by tetrahedral elements) describing the object's geometry and a force vector representing the forces applied to each node on the mesh. As output it produces the new positions of the mesh's nodes.

For this work, we use the BioTac ® sensor, a biologically inspired sensor with a rigid core covered with 19 impedance electrodes. The core is wrapped by a flexible skin and the space between them is filled with a conductive liquid. As contacts deform the skin, the liquid changes its distribution, thus modifying the values of the impedance electrodes. Furthermore, the sensor is also equipped with a thermistor and pressure transducer, to produce low and high frequency sampled values of temperature and pressure [START_REF] Wettels | Multimodal Tactile Sensor[END_REF].

A. Sensor Model

We model the BioTac sensor [START_REF] Wettels | Multimodal Tactile Sensor[END_REF] to compute a threedimensional force caused by contacting an object as well as the location of the contact.

1) Contact force magnitude estimation: Due to the complex fabrication of the BioTac tactile sensor, researchers have relied on machine learning algorithms to develop models that map the tactile signals into a three-dimensional force. These machine-learning based models have outperformed previously analytic formulations (see for instance [START_REF] Su | Force estimation and slip detection/classification for grip control using a biomimetic tactile sensor[END_REF]). We use Recurrent Neural Networks (RNNs) to exploit the sequential structure of the tactile signals. Specifically, we use an RNN architecture called Long Short-term Memory (LSTM) that has been successfully applied on speech recognition problems [START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF].

2) Contact localization: As the tactile sensor has 19 impedance electrodes distributed underneath its surface, they can be used to retrieve spatial information and thus to find the location of a contact. To localize a contact we first filter the active electrodes, e.g. the ones close enough to the contact point such that their values exceed their resting values3 (Figure 3b). We then compute the geometric centroid of the active electrodes as the mean of the positions of the m active electrodes.

x = 1 m m i=1 p e i (1)
where p ei represents the position of the i-th electrode. Once the centroid is known, it is used to find the contact point by computing the direction vectors d i between the centroid and the active electrodes. In order to locate the contact point the direction vectors are multiplied by the normalized intensity of the electrodes, thus assuring that the contact point is closer to the electrodes with the highest intensities.

d i = (p ei -x) I ei I e (2)
where I ei is the intensity value of the i-th electrode and I e represents the sum of all active electrodes. By summing these distances we can compute the contact location c (as shown in Figure 3c),

c = m i=1 d i m + x (3)
Finally, we project the contact point onto the finger's surface by using a geometric model of the finger (Figure 3a). We model the finger's surface as a sphere:

c sensor = o + r(c -o) c -o (4)
where r represents the radius of the sphere (we set r = 7 mm) and o is the origin, except when the contact c is negative on the X axis i.e. it is in the cylindrical part of the sensor. In that case, we set o = (x, 0, 0) to avoid distortions caused by using a spherical projection on a cylinder, where x is c x . An example of a projection when a contact lies on the spherical section is shown in Figure 3d.

B. Force Transformation

In order to apply a force to the deformation model, the forces computed on each sensor frame must be first transformed into a common frame. The object frame is used as a reference to transform the forces

f o n = T o n f n , (5)
where T o n is a transformation matrix relating the nth sensor frame to the object frame. Furthermore, as the deformation model requires forces to be applied directly on the mesh nodes, each force needs to be distributed among the surface nodes. Since the mesh elements are tetrahedra, they can be treated as triangles on the surface and this allows for the use of a linear shape function H to distribute the force onto the nodes in an inversely proportional manner based on the distance from the force's location to the three nodes.

H =   a1 a 0 0 a2 a 0 0 a3 a 0 0 0 a1 a 0 0 a2 a 0 0 a3 a 0 0 0 a1 a 0 0 a2 a 0 0 a3 a   (6)   f 1 f 2 f 3   = H T   f o x f o y f o z   (7)
where f i is a force vector applied to node i on the X, Y and Z axes. The total area of the triangle where the force is applied is denoted by a, and a i stands for the sub-triangle area formed between the opposite nodes of the i-node and the contact point. An example of a force being distributed on the three outer nodes of a tetrahedral element is shown in Figure 4.

a 3 a 1 a 2 f 1 f 2 f 3 f o
Fig. 4: Nodal distribution on a tetrahedral element.

To select the three nodes forming the triangle where the contact point is located, we apply the k-nearest neighbors algorithm with k set equal to 3. In order to guarantee the forces are caused by contacting the object we verify that the tactile sensor frame is within a threshold distance of the closest node, in our case we used a 1 centimeter threshold.

C. Deformation Model

To compute the deformation of the object caused by external forces, internal forces must be propagated throughout the mesh to obtain the new positions of the mesh's nodes. This can be achieved by solving the following differential equation:

f ext = Mq + D q + f int (q) ( 8
)
where f ext is the external force caused by gravity and contacts. Where the gravity vector, expressed in the world frame, is set to g = (0, 0, -9.80567 m s 2 ) as depicted in Figure 1. The position, velocity and acceleration of each node n is represented by q, q and q, respectively, with q ∈ R 3n . The mass matrix is contained in the M ∈ R 3n×3n and D is the damping matrix. The internal forces are described by

f int (q) ∈ R 3n .
As the deformation model, we use a Co-rotational Linear FEM model [START_REF] Müller | Interactive Virtual Materials[END_REF], since it provides a balanced trade-off between accuracy and speed as it can handle large deformations and it does not require nonlinear computations. The model requires a volumetric mesh and the elasticity parameters, i.e. Young modulus and Poisson Ratio. The deformation model computes Equation 8 with initial and boundary conditions. The initial conditions are the positions and velocities of the nodes when the object is at rest, e.g. undeformed; and the boundary conditions are constrained nodes, e.g. nodes that do not change their position at any time.

IV. EXPERIMENTAL EVALUATION

This sections describes the methods used to evaluate the accuracy of our sensor model to estimate force magnitude

A. Contact force magnitude estimation

To evaluate the ability of the sensor model to estimate a three-dimensional force, we collected a dataset of the BioTac signals and the output of a force/torque sensor ATI Gamma 4 . To generate contacts on the tactile sensor with different areas shapes and sizes, we fixed probes with different tips on the force/torque sensor. We then move each finger separately, except the thumb, downwards ten times for ten seconds at nine locations on each probe (see Figure 5). This produced a total of 228 recordings, each having close to 12,000 time steps, containing 21 tactile signals (19 impedance and two pressure signals) and a three-dimensional force. Given that the sensors operate at different rates, the force/torque sensor has a rate of 1 KHz and the tactile sensor runs at 100 Hz, we recorded the data at the lowest rate. The force values ranged between 0.1 to 1.0 N 5 . The data was then divided by using 80% as the training dataset and 20% as the test 4 http://www.ati-ia.com/products/ft/ft_models. aspx?id=Gamma 5 As we are concerned mainly with manipulating soft objects these are usual force values that occur when making contact with such objects. The structure of the RNN consisted of two hidden layers, each composed of 20 LSTM units, and a fully connected output layer as shown in Figure 6. A hyperbolic tangent sigmoid function was used for the hidden layers and a linear activation function was used for the output layer. The first hidden layer returns the full sequence (50 time steps), while the second hidden layer returns only the last time step. Thus, while the input is a sequence of tactile signals, the output is the estimated force vector at a single time step. As the optimizer, we applied Stochastic Gradient Descent with a learning rate of 0.01 in the regression layer. The RNN was implemented in Python using TFLearn [START_REF] Aymeric | TFLearn[END_REF].

In order to compare the performance of our approach we implemented a feedforward deep neural network (DNN) as described by Su et al. in [START_REF] Su | Force estimation and slip detection/classification for grip control using a biomimetic tactile sensor[END_REF], where the 19 impedance electrode values were used as input. Furthermore, we also evaluated networks with two additional vectors to investigate the effects of the pressure values. Thus, we compared four architectures, namely:

1) dnn

Hidden layers

Output layer

y 1 Input layer X 1 X 2 X n y 2 y m
Fig. 6: RNN architecture used to estimate a threedimensional contact force from tactile signals.

The four architectures are evaluated using the Root Mean Square Error (RMSE) and the Standardized Mean Square Error (SMSE) for the three components of the force. A comparison between the measured and estimated force magnitude is shown in Figure 7 and the evaluation results are shown in Table I. For a single sensor, the sensor model runs at an average speed of 185 Hz.

B. Contact localization

The contact localization algorithm was evaluated by contacting a probe, as the one shown in Figure 5, five times in 12 locations distributed along the tactile sensor's fingertip. The error was measured as the difference between the contact location, computed by the proposed algorithm, and the distance between the probe's tip and the sensor's frame, obtained by the robot's kinematics. The results are summarized in Figure 8.

C. Deformation sensing

To evaluate our proposed pipeline, we used the implemenof the Co-rotational Linear FEM offered by the Vega FEM library [START_REF] Sin | Vega: Non-linear FEM deformable object simulator[END_REF] and, as this is a self-contained library, we integrated it with ROS [START_REF] Quigley | ROS: an open-source Robot Operating System[END_REF] to couple it with the output of our developed sensor model. The volumetric mesh of the test objects were generated using the commercial software ANSYS and their elasticity parameters were obtained experimentally.

We used nine test objects 6 with three shapes (see Fig. 9 and Table II) and three different material properties (see Table III). To deform the objects we used the Shadow Dexterous Hand 7 . The cube objects were grasped using two fingers, while the rest of the objects were pushed by a finger of the robot hand as they were fixed, using double sided tape, on their sides with their longest axis being parallel to the Y axis, as shown in Figure 1. The sponge and bar objects start from an undeformed state and end in a deformed state. Figure 10 shows the states used for the cube objects, namely, when the object is fully visible, unoccluded; once contact has been made but without deformation, occluded; and finally, the deformed state. Since there is no direct way to measure the entire shape of an object while deforming, we evaluated our approach on a single object face. To this end, we compared the similarity between two point clouds: measured and simulated. The measured point cloud was obtained by a Microsoft Kinect RGB-D sensor from the real object, as shown in Fig. 11a; while the simulated one was rendered by placing a virtual Kinect at the same position and where the real object was replaced by the mesh output of the deformation sensing pipeline. By applying ray tracing and adding Gaussian noise, the virtual Kinect generated the simulated point cloud from the mesh as it can be seen in Fig. 11b. To track the real object, we first segmented the point cloud using the colorbased segmentation proposed in [START_REF] Zhan | Color-Based Segmentation of Point Clouds[END_REF], which uses similarity in color and spatial proximity to create clusters, where the cluster representing the object to be tracked is selected manually by the user. An example of this segmentation can be seen in Figure 12. To measure the similarity of these point clouds we generated an octree based on the measured point cloud with a minimum leaf size of 1 cm 8 , where the accuracy is given by the ratio of points from the simulated point cloud that are inside the leafs from the octree (Fig. 11c). The results are summarized in Figure 13. The deformation sensing component, implemented in ROS, runs at an average of 50 Hz. While the complete pipeline runs at around 20 Hz.

V. DISCUSSION

The force estimation results, shown in Figure 7, demonstrate the ability of the RNNs to accurately predict threedimensional forces. However, the estimation of the tangential forces (f x and f y ) is poorer than that of the normal force (f z ). This is a consequence of how the data was collected, namely, by only moving the fingers downwards tangential forces were not generated as much as the normal forces. Table I shows that the RNNs clearly outperformed the feedforward DNNs, but the addition of pressure information does not lead to a significant improvement. The contact localization algorithm was able to estimate the position of a contact with an error below five millimeters on all axes as it can be seen in Figure 8. Although, due to the location of the electrodes, the algorithm fails to localize contacts when they occur on the extreme sides.

The results of the deformation estimation are close to those obtained while no deformation was occurring (i.e. during the occluded and undeformed states), demonstrating the accuracy of our method, except for the case of the sponge, where the low accuracy was due to its small size. Although the deformation sensing pipeline appears to have a high variance in the accuracy results, as it can be seen in Figure 13, it is within the range of variance shown when no deformation was occurring. Other factors also contribute to errors in accuracy. For instance, our method is highly dependent on the location of the contacts. Thus, errors in the robot model in simulation and between the object's real pose and its pose in simulation have a direct impact on the performance of the pipeline. Another source of error, resulting in the oscillation of the mesh, is due to intermittent contacts caused by the softness of the objects as the tactile sensors fail to detect these contacts. Other errors stemming from the tactile sensors are that they cover a small surface of the fingers is relatively small. This results on the finger contacting the object with parts of the finger's surface that are not covered by the tactile sensor. Thus, deforming the object but not having any sensor feedback.

VI. CONCLUSION AND FUTURE WORK

We have presented a deformation sensing pipeline to estimate the shape of a deformable object while it is manipulated by a robotic hand by combining tactile sensing with a deformation model. Although tactile sensors perceive only local information we showed that, by combining them with a deformation model, it is possible to continuously estimate the global shape of an object while it is being deformed. The proposed pipeline also allows the use of multiple and different sensors by defining an interface to interact with a deformation model, namely, a sensor model. In our case, we developed and evaluated a sensor model for the BioTac sensor able to compute a three dimensional contact force along with its location. Although the sensor model proved accurate in estimating the force magnitude as well as the contact location, other factors not considered by the sensor model influenced the performance of the overall approach. For instance, the current sensor model assumes contacts to be points rather than areas, which causes the load distribution on the nodes to be less realistic. Also, setting contacts as boundary conditions on the mesh when the fingers touch the object is necessary to mitigate the oscillatory behavior of the simulated mesh. Nonetheless, the deformation sensing pipeline shows promising results and tactile sensing could be a good complement to existent approaches relying on vision sensors. In our future work, besides addressing the issues mentioned above, we will investigate further sensor models based on force/torque information as well as control schemes that can be coupled with our shape estimation approach.

Fig. 2 :

 2 Fig. 2: Components of the deformation sensing pipeline.

  Projection of the contact.

Fig. 3 :

 3 Fig. 3: Finger model, shown in (a), and the steps of the contact localization algorithm: (b) thresholding of the active electrodes where the electrodes' size is shown proportional to their intensity values, (c) contact localization based on the active electrodes and their geometric centroid, (d) projection of the contact to the sensor's surface.

Fig. 5 :

 5 Fig. 5: Data collection for the force magnitude estimation.

Fig. 7 :

 7 Fig. 7: Force magnitude estimation.

Fig. 8 :

 8 Fig. 8: Errors in the X, Y and Z axes for the contact localization algorithm.

Fig. 9 :

 9 Fig. 9: Test objects: cube (hard), sponge (mid) and bar (soft).

  (a) Unoccluded state. (b) Occluded state. (c) Deformed state. (d) Unoccluded state. (e) Occluded state. (f) Deformed state.

Fig. 10 :

 10 Fig. 10: A cube-like object tested in the three states. Front view is shown on the top row and a side view is shown on the bottom row.

  (a) Measured point cloud. (b) Simulated point cloud.(c) The generated octree, where the minimum leaf size is 1 cm.

Fig. 11 :

 11 Fig. 11: Similarity evaluation of a bar-like object using RGB-D data: (a) point cloud as measured by the Kinect, (b) point cloud generated by a virtual Kinect based on the output mesh of the proposed approach, (c) octree (white) generated from the measured point cloud to measure the similarity with the simulated point cloud (green).

  (a) Raw point cloud. Kinect frame (b) Segmented point cloud.

Fig. 12 :

 12 Fig. 12: Point cloud segmentation for a sponge-like object.

Fig. 13 :

 13 Fig. 13: Evaluation results of the deformation sensing.

  19 : DNN with impedance values. 2) dnn 21 : DNN with impedance and pressure values. 3) rnn 19 : RNN with impedance values in 50 time steps. 4) rnn 21 : RNN with impedance and pressure values in 50 time steps.

TABLE I :

 I Evaluation results of the force estimation.

			RMSE (in mN)		SMSE
		fx	fy	fz	fx	fy	fz
	dnn 19	41.74 94.38 344.74	1.6127 1.713	2.5225
	dnn 21	41.95 94.59 344.71	1.6294 1.7207 2.5222
	rnn 19	18.64 35.91 53.11	0.3213 0.2477 0.0599
	rnn 21	18.07 31.07 51.71	0.3018 0.1854 0.0569

TABLE II :

 II Geometric information of the test objects.

		Dimensions (cm)		Mesh
		Length Width Height Nodes Elements
	Cube	6	6	6	153	486
	Sponge	8	5	2	118	304
	Bar	20	4	4	152	385

TABLE III :

 III Material properties of the test objects.

			Elasticity parameters
		Material	Mass density	Young modulus	Poisson
		name	(kg/m 3 )	(P a)	ratio
	Hard	HR 45	45	3800	0.15
	Medium Bultex 30	30	3200	0.15
	Soft	Bultex 26	26	3000	0.15

An implementation of the code is available at https://github. com/jsanch2s/uca_deformation_sensing

http://wiki.ros.org/rviz

The BioTac sensor does not always initialize with the same impedance values, thus these values at rest are subtracted at each time step.

The objects were bought from the following vendor: http://www. moussesurmesure.com/

https://www.shadowrobot.com/products/ dexterous-hand/

We chose this size due to the accuracy of the Kinect, which is 0.5 cm.

ACKNOWLEDGMENT This work has been sponsored by the French government research program Investissements d'Avenir through the RobotEx Equipment of Excellence (ANR-10-EQPX-44) and the IMobS3 Laboratory of Excellence (ANR-10-LABX-16-01), by the European Union through the program Regional competitiveness and employment 2007-2013 (ERDF -Auvergne region) and by the Auvergne region.


Andrea Bagnasco
email: bagnasco@dibe.unige.it

Arianna Poggi
email: apoggi@dibe.unige.it

Anna Marina

Scapolla Dibe
email: scapolla@dibe.unige.it

Computational GRIDs and Online Laboratories

Keywords: Grid, online laboratory, remote experiments, grid services, measurement instruments, e-learning

Many projects addressing the creation of online laboratories have been developed recently. They have a common goal: to enable students and professional learners to get hands-on experience without moving from the places where they are. Nevertheless, the found solutions are often very different among them, and interoperability between different online laboratories systems does not exist. In this paper we present our work around the exploitation of Grid technologies for sharing instrumentation and linking together different online laboratories.

INTRODUCTION

The remote control of measurement instruments and devices enables students and professional learners to get hands-on experience without the need to move from their homes or their workplaces to a traditional laboratory. Compared with traditional laboratory practice, remote labs offer flexible learning in time and place, access to a wide number of distributed experiments and cost-cutting strategies. During the last years, many pilot projects addressed the development of online laboratories; nevertheless, it is still difficult to share instrumentation and experiments among laboratories [1]. Each one has its own security policy and adopts a proper technology in accessing and controlling real devices. A common integrated framework, offering indexing facilities, unique logins, file sharing and the seamless access and run of experiments, is the main challenge in order to create a network of online laboratories. In a previous work [START_REF] Bagnasco | A Grid of Remote Laboratory for Teaching Electronics[END_REF] we showed how the requirements and the functionalities of a remote laboratory system, which we developed for education in electronics using proprietary protocols and APIs, fit in the Foster's Grid model [START_REF] Foster | The Anatomy of the Grid: Enabling Scalable Virtual Organization[END_REF].

As OGSA [START_REF] Foster | The Physiology of the Grid: An Open Grid Service Architecture for Distributed System Integration[END_REF] evolved into OGSI specification and, the Globus Toolkit (GT) [5] became a standard solution for implementing Grids, we started to work around its exploitation for sharing online instrumentation and linking remote laboratories. This paper presents the path we are following. It is organized as follows: next section gives an overview of the state of the art in the usage of grid technologies and instrumentation; Section three illustrates how to create a network of online laboratories on the base of the Globus system architecture; Section four presents technical details for implementing instrument Grid Services.

INSTRUMENTATION ON GRID: STATE-OF-THE-ART

The Grid architecture seems very promising to meet the requirements of a network of remote laboratories, bearing well-defined tools for location, security, and integration of resources. The service-oriented view allows advertising features and services offered by instrumentation and measurement set-ups on the network. In this way, users and applications can easily find and make use of these services. Services can be expressed in a standard form, so that any implementation of a service can be invoked in the same way, hiding the variety of real hardware devices and ensuring location transparency. In order to create a grid of remote laboratories we need to solve the sub-task related to the exposure of instruments as grid services. Instruments have a strategic role in many scientific and industrial processes in the phases of data acquisition and data analysis. We can distinguish large and rare instrumentation, used mostly to acquire data, from instruments characterized by high interactivity with end users and real-time control requirements. The first ones are already part of relevant Grid projects. The two giant spectrometers, ATLAS [6] and CMS [7], that will detect the products of collisions at the LHC, will be the source of very large collections of measured data. Grid projects, such as GriPhyN [8] or DataGrid [9], are deploying computational environments that meet the resulting data-intensive computational needs. The focus of these projects is on the extraction of complex scientific information from massive data, which must be distributed for storage and computation to a community of scientists spread across the globe. Most of them do not have a direct access to the measurement plant. The benefits of Grid technologies concern analysis, sharing and distributed storage of information. These tasks can be performed offline with respect to the instrumentation, in charge of capturing the data, and do not require a real interaction with the instrument itself.

2

Concerning the second type of instruments, Grid technologies have not yet found a full exploitation. This is the target of ongoing research projects. The GRIDCC project [10], a three years project just started within the VI Framework Program, is the main European effort in this direction. It aims to integrate the real-time management of remote instrumentation (from temperature sensors to an array of telescopes), storage and computational resources, and real-time data analysis and visualization services. An important part of the project development plan deals with the design and the implementation of Virtual Instrument Grid Services (VIGS). A further interesting project is CIMA [START_REF] Bramley | Instruments and Sensors as Network Services: Making Instruments First Class Members of the Grid[END_REF]. It aims at developing a common middleware to facilitate the use of a wide range of instruments into a Grid computing environment, exposing them as Grid services. The project goals are the description of the characteristics of instruments by metadata and the development of a standard methodology for grid-enabling instruments.

A GRID OF ONLINE LABORATORIES

In this section we describe how Grid technologies, and in particular the Globus Toolkit, can be used to set up a network of laboratories for education. We start from our specific expertise in the development of remote laboratories in electronics [START_REF] Bagnasco | XML Technologies to Design Didactical distributed Measurement Laboratories[END_REF] and examine the evolution of the current client/server architecture to a grid based architecture.

Requirements

Basically, a remote laboratory used for educational purposes, has two categories of users, teachers and students: teachers plan and publish lab experiments, while students execute them. The release of new experiments requires preparing the hardware setup and developing or assembling the experiment software components. This software provides the control of the instruments, the scheduling mechanism, the communication layer services, the web interface to the experiment, the instrument virtual panels. As teachers are not usually expert programmers, these tasks are not so trivial, if not adequately supported. Moreover, it would be desirable to share hardware and software components of experiments among many distributed laboratories and to let teachers set up experiments assembling simple "bricks" corresponding to the resources they need. From the student's point of view, the final target is the availability of a wide number of remote workbenches, that can be accessed from a unique learning space and without taking care of the real location of physical resources [START_REF] Bagnasco | Authoring and Automatic Generation of Virtual Panel For Controlling Instrumentation in a Remote Laboratory Context[END_REF]. Thus, we must manage both software applications (i.e. virtual instrumentation panels) and physical devices (the measurement instruments) and we would like to treat them as distributed resources that can be combined in different ways, to set up diverse experiments according to our educational targets. Grid Services, the core of the OGSA architecture, offer an answer to this challenge. They are based on Web Services, which are software components characterized by high interoperability and accessibility, and extend them to the management of physical resources. In this way, we can use the same approach towards all the elements of the laboratory and can think to the real laboratories as nodes of a network of Grid service providers. Grid Services, such as Web Services, use Internet protocols and XML technology to ensure the interoperability. The services can be implemented in different programming languages and use SOAP message-based communication to talk to each other. The interfaces of the components are clearly defined by GWSDL (Grid WSDL), together with state information and service metadata (the so-called Service Data). A specific and already available service, the Index Service, can be used for service discovery and integration. In our opinion, Grid Services have characteristics very promising to connect remote laboratories and compose experiments by the integration and composition of services.

I N D E X S E R V I C E L A B S E R V I C E S Client i Authoring Service Device Service i Measure Service i Measure Service i Device Service i G R I D P O R T A L Client i FIGURE 1: Architecture

Design Choices

In this paragraph we present the architecture of a distributed laboratory system, based on Grid technology. In Figure 1 the Grid Portal is the front-end for clients. The back end consists of the Index Service, which provides the support for service delivering, and the laboratory services. Three classes of laboratory services are defined: the "Device Services" (DS), the "Measure Services" (MS), and the "Authoring Service" (AS). DSs wrap the hardware resources in order to expose the control of their functionalities in the form of methods. As in the real world there are different classes of instruments, we have different classes of DSs, i.e. one for oscilloscopes, one for waveform generators, and so on. Each DS exposes methods that are typical of its class. It is possible to make the service independent from the device details, such as model, supplier, or communication interface. The WSDL of a DS describes the operations and the arguments that control them. The term measure is used to define an ordered sequence of calls to the methods exposed by the DSs. The MS invokes the DS methods in the right order and with the proper parameters. As the MS is the sequential invocation of a number of independent DS, when a measurement occurs it is important to assign exclusively the hardware devices to one MS until this finishes, avoiding that one instrument changes its status before the whole measure has been processed. A device lock/unlock mechanism is required to guarantee the coherence of data during the measurement execution. The AS is a component that supports the process of creating new MSs. It works on the basis of the DSs descriptions and allows assembling DSs and defining specific measures. As both authoring and measure services expose their methods in a standard way, it is possible to call them from any application. Thus it is possible to integrate their control panels in whichever computer-based learning environment. Figure 2 gives a simple view of the interactions between the users and the laboratory grid services. To execute a measure, the student runs the MS and interacts with a graphical interface where all the device interfaces involved in the measure are present. At each interaction, the MS executes its embedded measurement program, invokes the proper DSs, collects the results and returns them to the student. Students, who work as clients of the system, can consume only MSs. They do not access DSs directly. The direct access to DSs is reserved for testing the devices and for the definition of new measures. To set up a new measure sequence, an experiment, the teacher configures and connects the physical workbench, the instruments and the circuit under test, and assembles the software through the AS. This service uses the WSDL descriptions of the DSs to automate the composition of the measurement program and the creation of the MS.

DESIGN AND IMPLEMENTATION OF A DEVICE GRID SERVICE

To prove the feasibility of the system, we started the development of a DS. The instrument chosen for exposing its functionalities was an oscilloscope. The complete software stack wrapping the instrument is visible in Figure 3. At the top, we have the Grid Service. We choose to write it in Java, because at this time, C/C++ services do not seem well supported in the GT framework. As instrument drivers are typically available in C language, we had to use Java Native Interface (JNI) to interface it. The oscilloscope service is an in-out type service; it receives the input arguments embedded in a request SOAP message and returns the results in a response SOAP message. It is a stateful service, so it is able to maintain the state between invocations. The service exposes methods for the main settings of the oscilloscope, they are part of those ones defined by the IVI specification [14]. Some of these methods are listed in Table 1. The WSDL file describes how these methods must be invoked. A chunk is in Figure 4.

<!--define the response message --> <wsdl:message name="readWaveformResponse"> <wsdl:part element="impl:readWaveformResponse" name="parameters"/> </wsdl:message> <!--define the request message --> <wsdl:message name="readWaveformRequest"> <wsdl:part element="impl:readWaveform" name="parameters"/> </wsdl:message> <!--define the operation --> <wsdl:operation name="readWaveform" parameterOrder=""> <wsdl:input message="impl:readWaveformRequest" name="readWaveformRequest"/> <wsdl:output message="impl:readWaveformResponse" name="readWaveformResponse"/> </wsdl:operation> FIGURE 4: WSDL for the oscilloscope grid service For demo and evaluation purposes we have developed a Java application, a virtual panel that implements graphical metaphors representing the instrument's controls, and invokes the service.

CONCLUSIONS

The grid-based laboratory system takes advantage of the GT3 OGSA for publishing services and accessing measurements and devices. Although we are still at the beginning, we developed a demo that shows the feasibility of the approach. We know that crucial aspects in the implementation of a network of distributed laboratories are the quality of the service (QoS), the system performance and response time. Surely, grid protocols add latencies and worsen the QoS. This is an open problem, but it is not so critical when the measurement loop is closed by a human being, as in the didactical experiments we consider.

Figure 2 :

 2 Figure 2: User scenario

FIGURE 3 :

 3 FIGURE 3: Device service stack

TABLE 1 :

 1 The oscilloscope grid service methods

	Method

Acknowledgment.

The Authors would like to thank Alberto Boccardo and Paolo Buschiazzo for their precious contribution in the services development.


Barry Tao

Hugh Davis

David Millard

Arouna Woukeu

The Semantic Aspects of e-Learning: Using the Knowledge Life Cycle to Manage Semantics for Grid and Service Oriented Systems

The aim of bringing semantics to learning content and services is to enable large scale collaboration of elearning activities over the Grid infrastructure. As machines and software applications will be ubiquitously involved in enabling and facilitating this collaboration, it requires a common understanding of the domain, in particular at the conceptual level, so that both computers and human participants of the elearning activities understand and are able to communicate among themselves though a common conceptualisation of the learning domain. In this paper we propose that Ontology is required in this context which is shared as a language to enrich resources with semantics. In line with the pedagogical view of the semantics obtained from experts in the learning domain, we describe a semantics driven knowledge life cycle to address the key phases in managing semantics in e-learning. Service oriented architecture is adopted for the vision of Grid infrastructure, reusable components and easy integration at a later phase.

Introduction

Learning related content and services are two main resources that are manipulated and interacted with in the e-learning activities. The concept of semantics is brought in for resource enrichment so that they can be better understood and processed by third-parties. In this aspect, we describe a technical view and a pedagogical view of the semantics. From the technical points of view, Ontology is a specification of a conceptualization. As an improvement over metadata, it is the backbone to better realize the Semantic Web/Grid vision -the idea of having resources on the Grid semantically enriched and linked for more effective discovery, automation, integration, and reuse across various applications.

To start, we give a simple example to illustrate the potential benefit of bringing semantics into the learning domain. The ontology defines the concept of student, teacher and course. The relationships are student assignedWith course, teacher deliver course. Apart from the "assignedWith" and "deliver" properties that are associated with their corresponding concepts, each concept also has its own properties like "name", "course ID", etc. Real people (students and lectures) and courses are the raw resources that need to be semantically enriched with the ontology. Ontology driven templates can be automatically generated so that any student/lectures can fill them out and generate corresponding semantic instances that represent a semantic enriched instance of the raw resource of themselves. As the system has a shared ontology, it knows how to process these requests and handle the semantically enriched resources. For example, when a teacher wants to register a new course, the course generation service knows from the ontology what properties of the course it should ask the teacher to fillout in order to generate a semantic instance of the "course concept". Furthermore we envisage that by adding "sequence" concepts the system could automatically generate the "workflow" that will result from the introduction of this course. On the other side, when a student wants to search for a course, the course query service knows from the shared ontology what the search criteria are. By this means, the two services deployed at different locations on the Grid (maybe by different software developers and running in different operation systems) can understand each other by following the shared ontology. Furthermore, the Declarative Logic (DL) based inference allows students to classify their courses and select potentially beneficial ones automatically based on the inferred facts.

The Grid is now a well established infrastructure that underpins the above purposes very well. Firstly it allows resources (content and services) to be located in a geographically distributed environment yet the semantic aspects can still guarantee the seamless integration and process automation. Secondly, it provides a potentially large pool of CPU and memory to handle the computationally intensive DL reasoning for inferring new facts automatically from existing facts stated in the semantics.

The vision is that a portal can be architected to embrace a distributed, service orientated architecture, where different parties can log in, discover, share, integrate and collaborate on learning resources. These activities are driven by a set of predefined ontologies and their upper level semantics bond with the resources.

Purpose and scope

This section aims to help us agree a common understanding of what it means to add semantics to the grid, and to the learning grid in particular.

Figure 1 Research aspects of semantics in e-learning

As illustrated in Figure 1, the semantics can be brought in to describe resources, which can be categorised into two major types -the content and the services in Grid-based elearning.

On the other hand, the semantics can also be examined from a pedagogical view and from a technical view. The technical view on Semantics focuses on generic specifications and technologies on Ontology, Semantic Web, and Semantic Grid, etc. The pedagogical view on Semantics places more emphasis on concrete impacts that semantics bring to distance education or e-learning activities.

Pedagogical view of semantics

The purpose of this section is to examine the affordances of semantics from a pedagogical point of view, with the intention of motivating the semantic layer of the ELeGI architecture. We are not interested here in how the semantics may be implemented but in the uses to which semantic mark-up can be put in order to benefit and improve learning and the formal and informal processes surrounding learning. The implementation of metadata and semantics for learning is covered in [2], particularly sections 5,6,7 and 8, and in [3].

The constituent elements of learning

From a learner's point of view there are a number of interactions which may take place:

• It is not our intention here to get into profound arguments about the meanings of these terms. We use the term teacher to mean a person who has a formal role in guiding student's learning, an expert as a person who may help a learner but without the formal role, and the learner learner interaction acknowledges that learning will often happen in discussion with other learners; the distinction between these three roles is not fixed. Similarly we characterize the difference between a resource (something which is essentially static and should not change between accesses) and a service (a dynamic process producing results varying according to context). Again, there is no hard line between these elements; we are using the terms in order to aid communication and understanding in the later discussion rather than as a formal taxonomy of learning.

We should also consider the role of those whose job is to support learning. Often this role is taken by teachers, but may also include librarians, learning technologists, web authors, and technicians. ELearning practitioners often comment that they believe they spend as much time organising the materials, setting access permissions, dividing students into working groups,

Technical view of Semantics

Pedagogical view of Semantics

Semantics and Content

Semantics and Services setting access times etc. as they spend on teaching and the production of teaching materials.

We suppose that in the future systems are going to deliver greater intelligence in order to improve learning and increase the efficiency of the management of learning. In order to achieve these goals, clearly we it will be necessary for systems to have an understanding of the learners, the teachers, the domain experts, the resources and the services.

What are the services?

The E-Learning Framework [4] provides a useful mapping of the functionality of E-Learning Systems onto services. It distinguishes between three layers of services:

Common Services: These provide the base level system functionality on which other services rely and which are not really specific to the learning domain. It would be fair to say that the functionality of most of these services already exists in the current monolithic Managed Learning Environments such as Blackboard and WebCT. There are many advantages to adopting a service oriented architecture to supply such functionality (support for pedagogic diversity, composable, flexible functionality, easier collaboration between institutions etc. [START_REF][END_REF]) but the issue we are concerned with here is how adding semantics to the constituent elements of learning (learners, teachers, domain experts, resources and services) can improve the learning environment and the learning itself.

The most obvious improvements of having well defined metadata are the opportunities for reuse of learning materials, the ease of interoperability and the general improvements in the ease of accurately searching for information. However, in the following sections we will attempt to move beyond the obvious to describe examples where intelligent reasoning applied to the semantics adds novel value to the pedagogic endeavor.

How semantic enrichment can improve learning

In this subsection we briefly describe a number of the ways in which intelligent reasoning about semantics can improve the opportunities for a student to learn.

Most, but not all, of these scenarios are specifically concerned with formal learning; they address the whole educational cycle from studying, researching, assessment and feedback.

• Connecting communities: Services can put people in contact with other people who are experts or learners with similar interests. This is a particularly important tool in informal learning scenarios.

• Personalized content: Intelligent tutoring systems have for some time being delivering content that was personalized for the user, based on an understanding of their goals and previous knowledge. • Analytic Tools: The e-Science community is leading the way in production of tools that harvest, store and analyse data from a range of sources.

How Semantic Enrichment can improve the Management of learning

• Production of materials: Production of teaching materials is a notoriously time consuming task, and the ability to locate and to re-use existing materials is a primary motivation for providing metadata for learning resources. The next stage is to provide services to assist in the location of suitable materials from heterogeneous sources. • Student Management: An understanding of the roles of the actors (teachers, students, experts, assessors etc.) makes the production of services for assigning students to the correct classes, discussion groups, experimental teams etc. possible. • Timetable management: An important task for teachers of on-line tasks is the timing of events, such as the release of some new materials, the closing date of some assessment, the exact time of a synchronous group chat session etc. These events can be made to happen automatically when a course is described in some language such as IMS Learning Design.

• Record Keeping: Record keeping and quality assurance can be the bane of a teacher's life, requiring them to spend much time ensuring that all the results are kept in the correct places such as institutional enterprise systems, student portfolios as well as made available for QA purposes by whatever external authorities might be involved. All of this work is an obvious target for automation by services that understand the goals. • Quality Assurance: Quality assurance often involves the maintenance of sample work and feedback/reflections, as well as ensuring that new programmes, courses and assessments have been through appropriate validation. Again, this is a task that could be assisted by intelligent services which could guide such tasks through the set of other services involved.

Much of what has been described in this section is in effect suggesting the use of some kind of agent to orchestrate services to achieve some goal. For example, an assessment system might call a service to handle some marks. This service might then ask an enterprise system service to store the marks in a database, it might call a service to annotate the student records with the new information, then might call an email service to inform the students of the need to update their personal development plans accordingly. This sort of orchestration of services looks very much like "workflow". It is our belief that appropriate semantic enrichment of the elements in the learning domain should make possible the automatic creation of such workflows by the composition of appropriate services.

Knowledge life cycle of semantics

In order to achieve these visions, we proposed a Semantic Web based knowledge management approach. We illustrate in Figure 2 the key steps in the knowledge life cycle as well as tools/APIs that we intend to use in each step.

Knowledge acquisition is carried out to acquire key information in the learning domain, the result of which is then fed to ontology modeling where we use the protégé ontology tool with an OWL plug-in to build a formal ontology. Ontology forms the conceptual structure of the knowledge base and the semantic annotation populates the knowledge base by generating instances. This is done by binding content with the relevant part of the ontology. The ontology and the semantics (instances) are stored in OWL format for easy machine processing at the programming level.

Knowledge reuse is then done through consuming these instances to generate knowledgeable decisions. Two types of reuse are envisaged:

1. Ontology driven query of the learning resources: dynamic forms can be automatically generated according to the pre-defined ontology so that end users know what the query criteria are.

2. Semantic matching based knowledge advisor on course recommendation: recommend and rank courses according to course ontology and learners' profile data. E.g., semantics on course pre-requisitions, learner's level, learning curve and subject, etc.

Note that the ontology is the share consensus and has been stored in the OWL format, Third-party programs must be able to access it and process the instances in the knowledge base for different knowledge reuse purposes. In order to allow the knowledge to be used potentially outside the awareness of its providers, a generic OWL API must be used. We use the Jena semantic toolkit [6] to process the OWL format ontologies and semantic information.

Technical view of semantics

Content enrichment is an important step in the semantics life cycle from knowledge acquisition to reuse. It binds together relevant parts between content and ontology so that raw content is enriched with more formal "meanings" pre-defined in a shared ontology.

Enriching content is also termed as knowledge binding [11], which depends upon human effort to tag thus making the knowledge accessible.

Key mark-up languages, such as XML, RDF and OWL1 are often chosen to represent the semantics via ontologies and make them machine accessible and interpretable.

This section will be informed by sections on Learning Object standards from ELeGI deliverable D6 (Projects, technologies and Standards evolution monitoring report) and the SEES where appropriate.

We address here how e-learning activities benefit from semantically enriched content.

Ontology Engineering

Prior to the enriching phase, the ontology must be made available as a formal, explicit and shared consensus in the domain. We intend to build the ontology by working together with domain experts in e-learning and utilize existing key specifications of content representation as described in the previous section.

Ontology is a specification of conceptualization [START_REF] Gruber | A translation approach to portable ontologies[END_REF]. It explicitly defines the domain concepts and their relationships. It is similar to a dictionary or glossary, but with richer structure, relationship and axioms that describe a domain of interest more precisely.

As can be seen in Figure 4.1, we build an initial ontology in protégé with an OWL plug-in. "Person", "Topic", "Learning_Event", etc. are key concepts under which the taxonomy is further expanded to express hierarchical relationships (parent/children) among concepts. Each concept also has its properties defined to express the subject/predicate relationship (who uses who). The ontological information is saved in OWL format for content enrichment through instance generation.

Enriching resources

While ontology is important in specifying the conceptual structure and a constrained vocabulary set, instances are treated as the concrete content in a semantic knowledge base. Generating the instances involves annotating the raw data source using predefined ontologies. In this paper, two methods are used to generate instances. Based on their operational mechanism, they are called "Ontology Instantiation" and "Resource Annotation" respectively.

1) Ontology instantiation

Protégé 2000 [9] is an ontology building and knowledge acquisition tool that has been frequently used for knowledge modelling purposes. It allows knowledge engineers to focus on modelling without worrying about the underlying language and syntax.

The modelling work can be saved in various formats including RDF and OWL (given OWL plug-in has been used).

Generating semantic instances in protégé

As illustrated in Figure 4.2-a, to create semantic instances of courses, relevant information in the course content is used to instantiate its corresponding ontology classes, such as "Assessment", "Course code" and "Subject", as defined in the ontology in Figure 4.1. Each instance (in the middle column) in Figure 4.2-a represents a course instance. Its properties ("Authorship", "Prerequisition" as defined in the ontology) are also filled with object instances, the class of which is constrained by class properties defined in the ontology. The object instances can be created on the fly or selected from previously generated instances.

In order to reach the maximum capacity of expressiveness, many languages have been designed to express the ontology and semantic information. Among them, the most recent is the Web Ontology Language (OWL), which has evolved from RDF and DAML+OIL to provide more expressive power. RDF stands for Resource Description Framework, a graph model (triple statement) which is designed for describing and searching resources on the web. The DAML+OIL is a schema language that adds constraints on properties to assist machine reasoning. For example when "daml:TransitiveProperty" is added as a constraint on the property "P1:older_than" of a RDF model, if we have A1:P1:A2 and A2:P1:A3, then A1:P1:A3 can be inferred. This is very useful for reasoning and inferring new knowledge that has not been directly stated. DAML+OIL also uses subProperty to describe relationships at different granularities. OWL is close to DAML+OIL with additional constraints such as "sameAs", "cardinality", etc. for more expressive power.

Instances generated in this way can be exported from protégé (with OWL plug-in) as can be illustrated in Figure 4.3, where the instances are represented using RDF as well as OWL enhancement for extra semantics.

The RDF can be also interpreted as N-Triples for efficient machine processing. While in Protégé, knowledge engineers acquire information about resource to instantiate an ontology, this is often too complicated for resource providers. On the other hand, there are many occasions when the resources are actually generated by the learners. In this case, a annotaion tool will be preferable to allow the end user to semantically annotate thier own resources for reuse.

Learning services are another important resource that need to be annotated so that we can be easily located for service orchestration. OWL-S (evolved from DAML-S [14]) is a standard language initially designed to semantically enrich services. Further development can be found in [START_REF] Gruber | A translation approach to portable ontologies[END_REF].

Advanced semantics reuse

Once the resources are enriched with semantics, more advanced reusing can be carried out in the form of discovery, automation and integration. At the moment, these are only our visions. More work need to be done to demonstrate these visions 1.

Resource discovery: This is in line with the Semantic Web and Semantic Grid, which intend to enrich data/services on the Web/Grid so that they can be easily identified and located for reuse purpose.

2.

Process automation: As services /content have their interface/structure semantically described, automation is possible by using agent technologies.

3.

Service integration: This is about exploiting semantics to assist the service oriented architecture where simpler services can be orchestrated together to realize more complex customized functionalities.

Conclusion

The semantic aspects are viewed from two perspectives: the technological view and the pedagogical view. We argue that the semantics can be brought in to enrich learning resources -both content and services in the learning Grid. Key technologies such as ontologies, the Semantic Web, the Semantic Grid and web services are intended to contribute to a new paradigm where the learning environment becomes service oriented, context aware and more user friendly.

Experience from Semantic Grid research in Geodise [START_REF] Geodise | Grid Enabled Optimisation and Design Search for Engineering[END_REF] illustrated that although ontology building is the kickoff activity for semantics based knowledge management, ontologies manipulation and maintenance activities happen all the way toward the end of the knowledge management life cycle.

In particular, the design of the semantics should be motivated by smart knowledge reuse scenarios that intend to facilitate e-learning related resource sharing and collaboration.

Figure

  Figure 2 Semantics aspect of the knowledge life cycle

  Figure 4.2 Generating semantic instances in protégé

  Figure 4.3 Function semantic instances

  When information has been semantically enriched then search engines can be much more powerful, having a much better ability to isolate relevant parts of documents and the ability to identify a much greater range of media types (videos, on-line experiments, simulations, data feeds, etc). Where services are semantically enriched, then search engines can choose suitable services to manage the query.

	attempt to provide feedback and links to suitable
	material to assist the learner.
	• Recommender Agents: Intelligent Agents that
	observe the information searching and studying
	patterns of users may recommend alternative
	materials. In a formal learning setting, the agent
	could query the syllabus and the course timetable
	to recommend a plan of study.
	• Annotation Tools: Allowing users to annotate
	information is an excellent source of semantic
	information providing useful information for
	others and allowing both readers and other
	services the opportunity to process the information
	in alternative ways (e.g. a search engine might be
	able to find all commentaries by a critic of a given
	author's work).
	• Search Engines:
	• Personalized sequencing: Adaptive Hypertext
	Systems attempt to provide pathways through
	materials by matching domain ontologies with
	dynamically evolving user models.
	• Adaptive Assessment: Systems may choose
	questions for the learner that attempt to test at just
	below the boundary of their knowledge and
	understanding, thus improving the efficiency of
	assessment and providing feedback that provides
	detail in the area that the student can most use it.
	Such assessment can be in the form of "self test"
	questions (formative assessment) or more formal
	assessments used for the purpose of grading
	(summative assessment).
	• Feedback Agents: Intelligent agents that observe
	student behaviour (e.g. assessment results,
	interactions with a virtual experiment etc.) can

XML, RDF and OWL are W3C standards, which can be found in http://www.w3c.org


Nicola Capuano

Pierre Carrolaggi

Jerome Combaz

Fabio Crestani

Matteo Gaeta

Erich Herber

Enver Sangineto

Krassen Stefanov

Mikel Vergara

A Virtual Organisation for e-Learning

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

You cannot mandate productivity, you must provide the tools to let people become their best.

-

Steve Jobs, CEO of Apple Computer

It is a well-known fact that many corporations would like to increase the global productivity of their production processes using e-Learning techniques as an alternative way to improve the effectiveness of their human resources. This general comment is also applicable to the SME's (Small-Medium-Enterprise) in the ITC (International Technology and Communication) market sector, in general lacking adequate personnel training.

Diogene is an European funded project under the 5th Framework Programme whose mission is to innovate the vocational and post-degree training in the ICT field making it more flexible and effective through the advanced use of the Information Technology. Particular interest is placed in the application of methodologies for distance learning and in the evaluation of their impact. The activity of production, experimentation and research of Diogene focus in conjugating demands and didactic objectives of free-lance teachers, learning material providers and learners through well defined training strategies analysing, defining, planning and appraising the most suitable methodologies to support them. Diogene objective is to design, implement and evaluate with real users an innovative training Web brokering environment for ICT individual training (but based upon a domain-independent platform) able to support learners during the whole cycle of the training, from the definition of Learning Objectives to the assessment of results through the construction of custom self-adaptive courses.

The system uses several state-of-the-art technologies such as: metadata and ontologies for knowledge manipulation, fuzzy learner modelling, intelligent course tailoring, co-operative and online training support. Besides, it includes a set of innovative features such as: dynamic learning strategies, Semantic Web openness, Web services for Learning Object handling and property rights management, Curriculum Vitae generation and searching facilities, free-lance teachers support, assisted Learning Objectives definition.

The system is accessible through the Web exploiting an ASP (Application Service Provisioning) methodology. Once a learner accesses the Diogene e-learning system, he/she can select a particular set of topics from an ontology of arguments and let the system to arrange a personalised self-adaptive course about chosen topics (personalisation is based on Learning Preferences). on Distributed e-Learning Environments Workshop

The Diogene e-learning platform includes eight courses covering different topics in three languages (English, Spanish and French). The following paragraphs describe briefly its innovative features with respect to commercial e-learning platforms presently available on the scene.

Metadata and Ontologies for Knowledge Management

All Diogene learning material is organised in Learning Objects indexed through IMS compliant metadata in order to let the system know what each one of them is about and how it can be used during the learning process.

To provide, also, information about Learning Objects relations and interdependency, Diogene applies Ontologies allowing to design abstract, simplified views of training domains. Within Diogene, ontologies are used to define and relate concepts of a training domain with three kinds of relations: (is_part_of, requires and suggested order) and, also, to link concepts to Learning Objects.

Semantic Web Openness

Another characteristic of Diogene e-learning system is its openness to the new Semantic Web paradigm. A key idea behind the project is, in fact, to not only rely upon ad-hoc generated learning material (i.e. high quality Learning Objects) but also to exploit the power given by the Web of the next generation to find useful and freeware learning material.

Web Services for Learning Object Handling

High quality learning material to be used in Diogene e-learning system is organised by content providers in several Learning Objects and maintained on their own Web servers. This means that such material is never imported on the main Diogene site avoiding, in this way, any property right related problem and giving content providers, in the same time, the possibility to manage contents (re-organise, update, delete) locally and not through complex HTTP or FTP based remote interfaces.

Intelligent Course Tailoring

A Diogene course is composed by an user selected set of learning goals (key concepts that the learner wants to learn) and by a learning path (a sequence of Ontology's concepts that has to be used to provide, to a specific learner, all necessary knowledge to fully understand chosen goals). Different learners can require different paths to learn the same goals depending on their Student Models. For this reason, Diogene provides an automatic learning path generation procedure: the learner can choose what to learn (goals) and let the system organise a personalised learning path for her/him. Such path can change dynamically during the learning process adapting to learner needs in relation to learner performed activities.

Curriculum Vitae Search Engine

Information about Diogene learner models is exportable in a standard format (PAI, LIP, etc.) in order to allow the possibility to create a standard Curriculum Vitae (CV) of the learner to be published, with respect to privacy requirements. Such curriculum includes information about: identification of biographic data, qualifications, learning related activities, competencies, language capabilities and preferences.

Free-Lance Teachers Support

Not only learning contents are indexed through Metadata but also other learning resources such as teachers are defined and described in that way. Free-lance teachers have, in fact, the possibility to subscribe Diogene and to describe (in a formal way) their professional abilities.

The Diogene network

Virtual Organizations (VO) are coordinated groups of individuals and institutions who collaborate towards a common interest and share, on the basis of some policies, a set of resources. Members of a VO can be geographically distributed and can access to the resources any time they are allowed. The element that characterizes the participants of a VO is the common purpose or business goal.

The Diogene project realised a Web Service based VO for e-learning. Diogene uses state-of-the-art technologies like metadata and ontologies for knowledge representation and management, fuzzy learner modelling, intelligent course tailoring and includes a set of innovative features such as 1 st International Kaleidoscope Learning GRID on Distributed e-Learning Environments Workshop dynamic learning strategies, Semantic Web openness, Web services for Learning Objects handling and IPR management, Curriculum Vitae maintenance and searching facilities, free-lance teachers support and assisted Learning Objectives definition.

The Diogene VO is populated by entities offering and consuming services. Such entities are grouped into two main categories: users (human beings that provide and consume services) and organizations (software components that provide and consume services).

Diogene Users

Diogene Users can be grouped into the following main categories.

 Learner: is a student registered in the Diogene VO. He/she acquires knowledge through learning experiences provided by Diogene specific learning services.

 Expert Learner: is a Learner skilled about some topics. He/she can offer mentoring support to other learners about such topics, usually for free.

 Freelance Tutor: is a professional tutor offering his/her specialized mentoring support to learners about specific topics usually under the payment of some price.

 Skill Searcher is an enterprise manager interested in hiring certified staff. He/she performs queries on skill repositories.

Further users like Content Providers, Knowledge Managers and Administrators are not discussed in this paper.

Diogene Organisations

Organizations of different type offer services with respect to the specifications of the Diogene VO and collaborate for the realisation of their own services. The kind of organisations supported by Diogene are summarised in the following list.

 Publishing Houses (PH). They store training content and provide remote access to it. They provide search and retrieval functions on the local repository via metadata-based queries.

 Web Catcher Agencies (WA). They are able to extract training content directly from Web and Semantic Web. Through a keyword-based text categorisation algorithm they can, where absent, semi-automatically extract metadata from textual learning objects ad link them to structures maintained by the Knowledge Agency.

 Tutor Agencies (TG). They work as entry point for freelance tutors, they manage the freelance tutor archive (containing tutor models) and provide searching facilities on such archive.

 Brokerage Offices (BO). They research, prepare and provide training offers for learners on demand, based on customisation information (learner model, learning strategies, price, learning goals). They collaborate with PH and WA in order to retrieve didactical material that best fit the customisation requests. They collaborate with TG to retrieve and provide freelance tutors that best fit learner needs.

 Training Agencies (TA). They provide the basic environment for learning experiences and tutoring activities. They are responsible for the delivery of courses and for the provision of course management and execution functions. They maintain learner models and are able to provide (on demand) course offers by exploiting services offered by BO.

 Knowledge Agencies (KA). They maintain and manage knowledge structures (concept dictionaries and ontologies) for the whole Diogene Network.

 Café (CA). They support social interactions, mentoring and information exchange by providing users a set of collaborative synchronous and asynchronous facilities. They automatically arrange groups among users of registered TA by individuating and grouping learners with similar needs and/or profiles.

 Skill Agencies (SA). They provide search engine capabilities on Learner Models Databases of registered TA in order to let third parties interested to hire certified staff to find qualified professional (with respect to privacy requirements). They maintain, moreover, requests statistics in order to rank the required competencies.

1 st International Kaleidoscope Learning GRID on Distributed e-Learning Environments Workshop  Bank (BA). It's a singleton organization that executes and logs transactions information occurred inside the Diogene VO. It interacts with an e-commerce engine to execute transactions involving real payment processes.

More instances of the same Organisations live inside the VO. The various instances stand on different hosts and are managed (administered) by different users. Figure 1 depicts some interactions between Diogene organisations.

The next chapters will focus on each of the Diogene Agencies.

The Main Services

In this section we present in more details some of the most important organizations explaining the way in which they work.

Training Agency, Brokerage Office and Knowledge Agency

These three agencies are the core of the whole net. A typical learning session works as follows.

A learner submits to the TA a query specifying the set of arguments she/he is interested to learn.

The set of topics is chosen among the set of all the Domain Concepts of the Ontology, provided by the KA. Then, the TA sends to the KA the student's training query (called Target Concepts). Now each BO has a Learning Path (a list of necessary arguments), different sets of Learning Objects (coming from the contacted PHs and the WAs) and the Student Model (provided by the KA through the TA). Thus the BO is able to select the best training offer for the student. This is done by comparing the available Learning Objects with both the Student Model (in order to select the ones nearest to the student's learning preferences) and the student's price request. The output of each BO is a Presentation, which is an ordered set of Learning Objects explaining the whole Learning on Distributed e-Learning Environments Workshop

Path. The Presentation is finally sent to the TA which presents it to the student. The student will select the best Presentation among all the Presentations provided by the different BOs contacted by her/his TA.

Web Catcher Agency

The purpose of the WebCatcher Agency (WA) is to discover free content on the Web, and to provide this to learners through the Diogene system. This free content may be of lower quality than the training materials provided by registered content providers, but can be used as additional material to aid learners.

In order to discover this material, the WA performs searches on the Web and Semantic Web using a mixed approach based on keywords and ontologies, whose details are reported in [Villa et al. 04a]. This approach enables to bypass compatibility problems between different ontological representations of the same domain that can be found on the Semantic Web and, at the same time, offering a search service, matching learning objects that can be found on the Web to users' requests.

The WA also uses a keyword-based text categorisation algorithm to enable, where absent, to automatically extract metadata from textual learning objects and to link them to Diogene's ontology concepts [Villa et al. 04b].

The WA is directly linked to Diogene's Brokerage Office. The Brokerage Office collaborates with the Publishing House and the WA in order to retrieve didactical material that best fits the customisation request.

Publishing Houses

The Publishing house is a Diogene Organization focused on cataloguing and managing high quality didactical material. There could be more than one Publishing House inside DIOGENE Network. The didactical material is composed by Learning Objects (LO), specified in term of content (i.e. HTML, PDF) and metadata (IMS-LOM). For each learning object the Publishing house stores content in LO Content Repository, stores metadata in LO Metadata Repository and updates indexes of LO Metadata. Other Diogene organizations can demand learning objects from Publishing house and can get them in the form of a content package. For that purpose they have to use the services of the Brokerage office.

Publishing house provides the following web services:

• Store package of learning objects and update the metadata indexes -unpacks learning objects pack into several learning objects composed by content and metadata. For each learning object stores content in LO Content Repository, stores metadata in LO Metadata Repository and updates indexes into LO Metadata Repository.

• Retrieve personalized learning object's list -searches the LO Metadata that best matches the input LO Metadata Query. All LO Metadata founded will be returned in a LO Metadata List. The LO Metadata Query is represent as XML structure. Every Query is composed by one or more query nodes. The node's value is matched against every LO Metadata. If match succeed then the Metadata is selected. The process returns a union of all selected Metadata.

Deliver learning objects -This process creates the Learning Object Pack, that contains the Content and Metadata of the LO required, and returns it.

Tutor Agency

The Tutor Agency is a Diogene Organization that enables the storage and retrieval of information about tutors, their models, and their schedules.

Companies registered into the Diogene Network can perform queries on a Tutor Agency in order to retrieve tutors with specific skills such as language, education level and areas of experience.

Tutor Agency has several important features:

• Support the management of tutor profiles (each tutor can add, view, update and delete private information such as contact information). • Support the management of tutor schedules (each tutor can add, view, and update his/her schedule). • Support the management of tutor model -that is the language, level of education, and skills in specific subjects.

1 st International Kaleidoscope Learning GRID on Distributed e-Learning Environments Workshop

• Support the tutor searching activity that allows students to searcher and find a tutor that best fits the criteria such as language, level of education and skills in a specific area of interest. In order to use the Tutor Agency, the user have to be registered inside the Skill Agency from the Diogene Network Reception.

Skill Agency

The Skill Agency can be queried by third parties interested to hire specialized staff. The Skill Agency (SA) receives from the Training Agency (TA) a data base of Student Models. In this data base the SA will search for people to propose as answer to the query. Searching in the data base works as follows. The input query specifies a set of Domain Concepts (see Section 3.2), i.e. a set of topics/skills in which the external company is interested as the wished application curriculum. Since the Student Model of each learner contains, for each Domain Concept, the representation of the knowledge degree shown by the learner on that subject, the SA can rank the Students' Models provided by the TA by selecting in each Student Model the Domain Concepts specified in the query and then maximizing the sum of all the knowledge degree corresponding to each student. Finally, the SA will propose to the skill searcher the ranked list of learners with the possibility to contact them.

Examples of complete courses provided by Diogene

Eight courses have been adapted from existing linear courses. Courses have been split in many LOs to make it Diogene compliant. It has been necessary to rewrite some parts of the content, to ensure that each LO is atomic (no reference is made to another LO or to another concept). A Main Ontology covering the ICT domain was created to index these LOs, based on the ACM-CCS classification (Computing Classification System). ACM was chosen for different reasons : it has a widespread ICT taxonomy, it is easily expressed using the Diogene relations, and it is directly linked with the ACM & IEEE Computing curricula guidelines, describing how courses in the field of computing have to be arranged in the suitable training programmes at various levels. The actual Ontology contains and extends the CCS, using the Requires/Suggested Order/Has part relations to organise 1652 concepts. The eight genuine courses cover a wide range of ICT concepts among the Ontology : how to secure data access, processing and transmission for SMEs dealing with information systems; the theory of digital images and how ICT workers can exploit them; how to set up a basic development environment to develop dynamic web pages with the PHP language; Business Process Management is a set of methods to help better decision making; some other courses cover software engineering, like 'COTS' approach for large-scale software development, Balanced IT Scorecard for software process improvement, object oriented analysis and design which covers all the UML notations, and a set of LOs about the fundamentals aspects of XML.

The content producers used the KMS tool to make interactive exercises used by the system to test the learners on each LO. Test and LO have been meta-tagged with the same tool regarding their content: language, interactivity level, typical learning time, learning styles, etc. All LOs and test are available in English and optionally in the content producer's languages (Spanish and French for the experimentation). Diogene deals the content in the language of the learner regarding its profile whether localized LOs are available. It is easily expendable to other languages. Actually the experimentation includes a total of 716 LOs.

Evaluation results

Experimentation and evaluation of the DIOGENE system was conducted between April and October 2004 and built up on several use scenarios that realised a real life learning environment in a VO. The results of this process proved that DIOGENE has had a significant impact on the brokerage of ICT courses and the support of the learning life-cycle by use of the underlying organisations. One of the major benefits that DIOGENE enables, as many evaluators pointed out, is providing the available courses on a resource list directly to concepts of a structured knowledge domain, and supporting fundamental processes of the whole learning life-cycle. Learners proved highly satisfied with the allocation of courses they achieved by use of their profile definitions (72 %) and the adequacy of the courses that were suggested by the system to represent their lack of knowledge (78 %). Tutors evaluated DIOGENE as a useful platform for the support of their tutorial services. It 1 st International Kaleidoscope Learning GRID on Distributed e-Learning Environments Workshop was appreciated from tutors that DIOGENE allows a mechanism that supports collaboration with learners and that enables tutors to support the learning process. Skill searchers confirmed the value of the services related to skills requests in the process of higher qualification and/or recruitment. According to the innovative aspects that can be associated with DIOGENE, evaluators assessed DIOGENE to be a highly innovative (63 % of evaluators) and highly useful (94 % of evaluators) system that provides easy to use and good quality (91 % of evaluators) services of which they would want to make use again once DIOGENE will be available as a commercial service (87 % of evaluators). It was also clearly found out that where constant communication and interaction between the stakeholders of the VO was held, learning outcomes and progress could be improved and processes made easier as well as more transparent to stakeholders. Figure 2 shows some evaluation criteria on which DIOGENE had been evaluated. The evaluation results demonstrate that DIOGENE as a VO has a good level of utility for providing learning resources of various aspects and domains and supporting the learning life-cycle in a distributed environment, but complexity of knowledge domains is an issue that requires good methods and concepts in order to be translated to users. The major impact of the work of the DIOGENE project and evaluation outcomes will be on taking forward the results and lessons learnt from the project and improving further such Virtual Organisations to support the processes of learning. Applied Methodology: Evaluation involved 170 users of various roles and types (unskilled / skilled / expert users with diverse language backgrounds) from 6 European countries and was based on a range of well-known evaluation methods: use scenarios, performance recordings, online questionnaires, interviews, and document analysis.

Conclusions

We have presented a distributed e-learning system created during the European funded project Diogene (5th Framework Programme, Information Society Technologies, contact number IST-2001-33358, see [Diogene] for more details). Diogene is based on a network of independent organizations each of which realized as a Web Service. These agencies provide several e-learning and related facilities, such as course delivering, course customization, free document retrieval and categotization on the Web and on the Semantic Web, learning objects' repositories based on a metadata indexing, human tutor support and external skill search. During the project we also have built a complete Ontology of the ICT topics, [Stefanov & Todorova 03], used by the system as domain knowledge representation, which completes the ACM taxonomy of the Computer Science. Moreover Eight courses have been adapted from existing "traditional" courses splitting them in different Learning Objects and indexed using Semantic Web standards. Finally, the project has been evaluated using 170 users of various roles (learners, tutors, ...) with very encouraging results.

Figure 1 .

 1 Figure 1. Interactions among different Diogene Organisations.The KA, starting from the Target Concepts and using the pedagogical information on the domain of interest represented by its Ontology[Capuano et al. 02, Capuano et al. 03], generates a Learning Path, i.e. the set of all the Domain Concepts necessary to learn the input Target Concepts set. In the Learning Path there is no concept already known by the student (this can be verified using the Student Model), moreover the KA includes in the Learning Path all the necessary pre-requisite concepts and finally the Learning Path is ordered following the pedagogical instructions explicitly declared in the Ontology. Once the Learning Path has been built, it is sent by the KA to the TA, which, in its turn, sends the Learning Path to the BOs. Each BO sends the Learning Path to all the PHs and WAs it is in contact with. Each PH, received the Learning Path (i.e. an ordered set of Domain Concepts), sends back to the BO the set of Learning Objects Metadata related to at least one concept of the Learning Path. Analogously, each WA sends back to the BO all those documents (previously retrieved on the Web and off-line categorized) which are related to the concepts of the Learning Path.

Figure 2 .

 2 Figure 2. The added value of DIOGENE

[

  Capuano et al. 02] Capuano N., Gaeta M., Micarelli A. and Sangineto E. An integrated Architecture for Automatic Course Generation. Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT 02), Kazan, Russia, 2002. [Capuano et al. 03] Capuano N., Gaeta M., Micarelli A. and Sangineto E., An Intelligent Web Teacher System for Learning Personalization and Semantic Web Compatibility, Proceedings of the Eleventh International PEG Conference, St. Petersburg, Russia, 28 June-1 July 2003.