Pierre Nagorny
email: pierre.nagorny@univ-smb.fr

Thomas Lacombe
email: thomas.lacombe1@univ-smb.fr

Hugues Favreliere
email: hugues.favreliere@univ-smb.fr

Maurice Pillet

Eric Pairel

Ronan Le Goff
email: ronan.legoff@ct-ipc.com

Marlene Wali

Jerôme Loureaux

Patrice Kiener
email: patrice.kiener@inmodelia.com

Hugues Favrelière

Marlène Wali

Jérôme Loureaux

Generative Adversarial Networks for geometric surfaces prediction in injection molding

Keywords: Injection molding, Quality prediction, Thermography, Generative Adversarial Networks, Discrete Modal Decomposition

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I. INTRODUCTION

Thermoplastics injection molding can produce complex parts in large quantities. Quality requirements are increasing to guaranty the final product functionality. The final part quality depends on multiple settings and external non-controllable factors, from raw material hygrometry to in-mold pressure. Thus, closed-loop process settings adjustments are needed to achieve optimal quality [START_REF] Johnston | On-line multivariate optimization of injection molding[END_REF]. Moreover, the final part quality can only be measured on stabilized parts, days after the production. Molded parts need to cool down with internal mechanical constraints relaxation. Thus, it is not possible to measure part quality just after production. To achieve next part process settings adjustment in closed loop, hot parts must be measured, and final part quality must be inferred. It is interesting to measure the hot part and infer the final part quality. Measurements and settings adjustments computation must be done in the industrial cycle time (less than thirty seconds). Thermographic imagery is a fast and easy to set-up measurement, which can be used on the production line. Inference of specific final parts quality based on hot parts measurements can be achieve using regressive models or neural networks. A previously learned model is fast to execute for inference on a recent computer. Convolution Neural Networks [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF] have been used to predict a continuous geometry measurement of final part from thermographic images of hot parts [START_REF] Nagorny | Quality prediction in injection molding[END_REF]. However, a unique geometrical value is not meaningful for a human operator, and even more information could be extract from thermographic images. Recent research about Generative Adversarial Networks [START_REF] Goodfellow | Generative Adversarial Networks[END_REF] used as images translators are applicable on industrial problems. In this paper, we propose the use of a GAN to translate thermographic images into final surface geometry (Fig. 1). We measure the prediction performance with different images similarity comparison methods. Furthermore, we introduce the Discrete Modal Decomposition analysis [START_REF] Favreliere | Modal Tolerancing : From metrology to specifications[END_REF] as an innovative tool to characterize the GAN generation performance to retrieve geometric parameterization. With some enhancement, this method could be proposed as a real-time tool to inspect the future part geometry, even in an augmented reality real-time overlay for human operators.

II. RELATED WORK

Regulation of injection molding is a productive literature subject, using multiple advanced methods. Many works have been done using in-mold measurements with good success [START_REF] Michaeli | Online control of the injection molding process based on process variables[END_REF], but far lesser proposed to measure hot parts just after production. The serious disadvantage of in-mold measurements lays on its need of expensive sensors placed inside the mold with a custom mold design for wire routing. Besides, robotic is now heavily deployed in the industry. Thus, hot parts measurements can be done in-situ, in the production line and cycle time. If the measurement process is fast enough, every part can be measured. A robotic arm can move parts on different measurements stands. Thermography imagery, 3D scanning, weighing or photography are easy to set up. Then, mathematical models must be construct between hot parts measurements and final parts quality. Neural networks models can benefit from multi modal sensors fusion [START_REF] Ngiam | Multimodal deep learning[END_REF]. In this study, we work specifically on thermographic images as model input. Generative Adversarial Networks were introduced in 2014 and have since shown success in very specific image processing applications. Recent research used GAN to translate images, after the learning of a transformation model on pair images (pix2pix), or even with no pair prior at all (Cycle GAN) [START_REF] Zhu | Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks[END_REF].

III. METHODS

A. Experimental dataset creation

Under industrial setting, it is usually difficult to create a big dataset. This paper aims to use a GAN network architecture to solve this limit. Generative networks are known to require a small dataset. We actually have a dataset with ten times fewer elements than classical dataset used for GAN. Our study is exploratory and aims to study the performance of GAN in industrial constraints. The dataset creation was done using a Taguchi orthogonal L12 Design of Experiments [START_REF] Taguchi | Introduction to Quality Engineering: Designing Quality into Products and Processes[END_REF], with twelve different trials. Furthermore, we choose to put two parts of each trial in the training dataset. An exception was done for on trial which parts are not at all in the training dataset but only in the validation dataset. These special not learned parts will be used for further validation of the network performance. We use a robotic arm to grab the part right after molding and to show them in front of a thermographic camera. A still frame was acquired for each part 10 seconds after the ejection of the part Then, we have been waiting a month before geometrical measurements to be sure that parts were geometrically stable. Surfaces were scanned on a confocal optical profilometer (Altimet AltiSurf 520). The scanning takes one hour; this duration is the actual limit to the size of our dataset.

From the nearly two hundred parts we have produced and measured on-line, we have actually only scanned 37 parts: 23 parts are used as a training dataset and 14 parts are used for validation.

B. Dataset preprocessing

The robotic arm and the vacuum prehension device were not enough repeatable to guarantee pixel to pixel position similarity between measurements. Thus, thermographic images were stabilized using the Lucas-Kanade optical flow tracking algorithm [START_REF] Lucas | An Iterative Image Registration Technique with an Application to Stereo Vision[END_REF]. Images were normalized on the minimum and the maximum values of all images. Images were then cropped and scaled to 71 by 71 pixels in grayscale, without multiple color channels. Finally, images are upsampled (bicubic interpolation) to 128x128 to be used as networks input. Upsampling the small definition acts as a 1.8x1.8 pixels Gaussian filter. We voluntary choose small size images to simulate industrial constraint where multiple parts must be imaged with a unique camera, leading to a small image definition peer part. After normalization and reduction to 8 bits images the thermographic image has a resolution of 0.902 degrees Celsius. After normalization and reduction to 8 bits images, the height map images have a resolution of 1.57 µm. This is quite low, but it is acceptable to characterize low frequency geometric form defects. In our case, better results in term of geometry precision could be obtained with floating points input instead of 8 bits images, but the network size and training time should be far bigger.

C. Pix2pix Generative Adversarial Networks architecture

A "Generative Adversarial Network" is composed by a generator and a discriminator which are both trained sequentially one after the other. The generator is trained to produce realistic images from a random normal distribution. Then, the discriminator must distinguish the "fake" generated images from the real images. At each training loop, the generator gets the difference of the output between the discriminator and the "fake" generated sample. The generator is train to increase the error of the discriminator by producing perfect counterfeit image. The discriminator is train on real images. The convergence of the network is achieved when the generator and the discriminator reach an equilibrium point between counterfeit performances and detection. The pix2pix framework [START_REF] Zhu | Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks[END_REF] use a deep convolutional GANs (DCGANs) architecture proposed by Radford, Metz and Chintala [START_REF] Radford | Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[END_REF]. Pix2pix also proposes the use of conditional GAN [START_REF] Pathak | Context Encoders: Feature Learning by Inpainting[END_REF], which learn their loss function from the observation of the inputted images. This framework simplifies many previous works and propose to use a "U-Net" architecture [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF]. The U-Net architecture has skip connection between each center symmetric layers. Each skip connection concatenates channels from layer i with channels from layer n -i, with n the total number of layers in the network. Pix2pix also use a convolutional classifier, which work on small patches of the input image (PatchGAN). Thus, the convolutional layer will be only receptive to pattern at the patch size scale. The patch size choice must be investigated. For our problem, the patch size will certainly change the prediction performance of various geometrical scale. The generator and the discriminator have the same layer architecture: each layer has a convolution layer and LeakyReLu [START_REF] Maas | Rectifier Nonlinearities Improve Neural Network Acoustic Models[END_REF] as an activation function. The augmentation part of the generator use Rectified Linear Unit [START_REF] Glorot | Deep Sparse Rectifier Neural Networks[END_REF]. In our study, we do not use any batch training nor normalization because we have a small dataset. Finally, pix2pix introduces a L1 distance loss between the generated images and the real ground truth images. Thus, the generator must fool the discriminator and minimize the L1 Manhattan distance with the real image. The complete network architecture is show in Fig. 2. We have only trained two hundred epochs for now. We use Xavier layers initialization [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF] based on success in the literature. We envisage hyper parameters tuning as a direct pursuit of this research. Other architectures must also be evaluated, particularly the Perceptual Adversarial Network [START_REF] Wang | Perceptual Adversarial Networks for Image-to-Image Transformation[END_REF], which shows better performance in specific cases.

D. Pix2pix GAN training

The network was trained using the Adam stochastic gradient descent solver [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF]. We kept the image augmentation techniques used in the pix2pix [START_REF] Zhu | Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks[END_REF] proposition: random jitter and mirroring. The network training takes 3 hours on a Core i7 CPU. We will investigate multiple hyper parameters tuning as soon as we have new GPU computing resources. We observe the importance of the Conditional L1 loss in the training process (Fig. 3, Fig. 4). The training completion was effective after 180 epochs. The training is relatively fast. It can be done over a night for industrial applications.

E. Discrete Modal Decomposition

In order to describe geometrically the surfaces and check if the GAN generation enables to retrieve geometrical features from the true images, we introduce here some shape parameterization methods, especially the Discrete Modal Decomposition (DMD). Global shape parameterization techniques lay on the identification of parameters which characterize the shape geometrical elements as well as possible. In practice, we find descriptors by decomposing the surface into a descriptors space, specific to each method. The DMD uses a decomposition space based on a vibration mechanics problem [START_REF] Favreliere | Modal Tolerancing : From metrology to specifications[END_REF], [START_REF] Goïc | Multiscale roughness analysis of engineering surfaces: A comparison of methods for the investigation of functional correlations[END_REF]. Among the other widespread methods used for parameterization in the literature:

i. The Discrete Cosine Transform (DST) decomposition [START_REF] Rao | Discrete Cosine Transform: Algorithms, Advantages, Applications[END_REF] uses a cosine harmonics space to describe the surface. ii.

The Fourier series decomposition is based on the Discrete Fourier Transform (DFT). This method enables to decompose a surface into a sine and cosine harmonics space, more general than the DST one. iii.

The Spherical harmonics decomposition describes complex shapes into a spherical harmonics space. This technique is especially used to characterize 3D shapes. If we work with a plane geometry, the DMD descriptors are the natural vibration eigenmodes of the reference plane geometry (Fig 5). Le Goïc [START_REF] Goic | Multi scale modal decomposition of primary form, waviness and roughness of surfaces[END_REF] has shown that this method can be used to characterize global as well as local surfaces geometries. We propose to use this method to parameterize the geometry of our samples, and to see if this parameterization can be retrieved through the GAN image generation.

IV. PREDICTION PERFORMANCE ANALYSIS

We use multiple methods to determine the predictive performance of the network. We firstly compare real measured height map images with images generated by the GAN (Fig. 6). The validation is done on a dedicated dataset of 14 parts (Fig 7). Simple statistical features and histogram comparisons with different metrics are used. Then, we introduce Discrete Modal parameters as an innovative prediction performance tool on specific geometrical features.

A. Statistical and Haralick features comparisons

Generated images visually show good similarities with real images. In order to validate the similarity, we compute different statistical features on each couple of real and generated images. We compute the mean, median, kurtosis, skewness and multiple quantile of the images. We also compute the Haralick features [START_REF] Haralick | Textural Features for Image Classification[END_REF] as they are good textural descriptors. We study the difference between those values. We compute the p-value to validate the result on this really small population (14 elements). The standard deviation on all the tested results is high and the p-value was superior to 5% on most of the metrics. Thus, we cannot reject the null hypothesis [START_REF] Colquhoun | An investigation of the false discovery rate and the misinterpretation of p-values[END_REF]. We then must increase the test dataset size to be able to get significant results with these textural parameters. The only features with a p-value superior to 5% are Haralick's difference variance, energy, entropy and homogeneity. The energy and the homogeneity metrics indicate the uniformity of the image. The entropy measured the presence of random patterns. Thus, our real and generated images are similar in their variance distribution.

B. Real vs generated image similarity

We avoided pixel to pixel comparison because our generated images can be slightly translated due to the dataset augmentation methods used. Histogram comparisons are robust to scaling and movements. We used various distance metrics on histograms (Fig. 8):

Bhattacharya [START_REF] Bhattacharyya | On a Measure of Divergence between Two Multinomial Populations[END_REF], Khisquared, correlation [START_REF] Székely | Measuring and testing dependence by correlation of distances[END_REF], cosine (similar to Euclidian L2), Hellinger, Kullback-Leibler [START_REF] Kullback | On Information and Sufficiency[END_REF], Manhattan L1 and Minkowski distances. We used the p-value test to test for the null hypothesis and only the cosine and correlation distances were kept. We also compute the Structural Similarity [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF] and the Peak Signal to Noise Ratio (PSNR), two widespread measures for image comparison. Results on the test dataset are shown in Table I. test06 and test06-bis are images obtained for the same settings in the design of experiment: no part with similar settings were present in the train dataset. Results on the test06 and test06-bis are used to study the generalization capability of this predictive method. Although test06 results are good, test06-bis is not on the same level of prediction quality based on cosine and correlation distances on histograms. The median value on the 14 parts shows the overall good predicting performance of the methods. Further analysis is done with the Discrete Modal Decompositions.

C. Discrete Modal Decomposition analysis

We introduce here a new performance indicator for the GAN predictions: the DMD analysis. The objective is to study if the GAN is able to retrieve the surfaces in the DMD geometrical parameterization. DMD analysis is especially interesting in the case of plastic parts deformation. Our objective is to anticipate deformation on the real part. The firsts DMD coefficients give the global part geometry. The last DMD coefficients concern the roughness and very local deformation.

In plastic molding, we particularly have to anticipate the global part geometry, so, the first modal coefficients are especially appropriate. The real pictures modal spectrums are calculated and compared with the generated images ones (Fig. 9). Similarity between the 2 spectrums vectors for the 14 pairs of images have been calculated thanks to correlation coefficient and cosine similarity (Table II). The results of the spectrums comparison study show a very good similarity between the spectrums with a median R² coefficient value of 0.98 and a median cosine distance value of 0.99 (very close to 1). Moreover, standard deviations are very low for both measurements, which shows a strong consistency for the 14 tested parts results. These measures bring to light the good performances of the GAN to retrieve the geometric parameterization of the parts' surfaces. It also shows the opportunity of using DMD analysis as an innovative performance indicator for neural network image generation. This paper illustrates a preliminary study for the use of GAN to predict geometry parameters through DMD. For further developments, we also propose to study these results from "the modes viewpoint". Instead of analyzing the correlation between images global modal spectrums, we want to investigate each mode independently to see which ones are best predicted by the network. Through this work, we hope to show what types of shapes and deformations the network is able to generate the best. As explained in III.E., DMD is not the only method to geometrically characterize surfaces. It could be also interesting to apply other type of decompositions and compare with the DMD analysis. In this study, we have shown the possibility of using Generative Adversarial Networks pix2pix on a very small dataset, to comply with industrial constraints. Based on the measured similarity between final image and generate image, we show that the prediction is acceptable in the context of our very limited learning study. Quantities of information contained in the thermal image seem sufficient to predict our geometry in our specific application case. We introduce Discrete Modal Decomposition (DMD) to analyze the prediction performance from a geometrical parameterization side. The DMD presents a clear meaning to analyze geometrical properties but it could also further be used to analyze any generative network performance. In thermoplastic injection molding, we don't to predict the rugosity (high modes in DMD) but only geometric form defects (low modes in DMD). Our problematic is to anticipate the global cold part geometry, so, the first modal coefficients are appropriate. Performance evaluation and hyperparameters tuning must be done on a larger dataset. Further studies can also be designed to analyze more precisely the GAN performances to retrieve independent modes to show what shapes can be best predicted.

Finally, further research needs to be done on the complete 3D model generation based on thermographic prior, as literature shows encouraging result in 2D images to 3D shapes transformation [START_REF] Wu | Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling[END_REF].

Fig. 1 .

 1 Fig. 1. Process ajustement based on thermography and GAN.

Fig. 2 .

 2 Fig. 2. Pix2pix UNet_128 GAN network architecture.

Fig. 3 .

 3 Fig. 3. Training loss over epoch.

Fig. 5 .

 5 Fig. 5. Vector basis of modals descriptors (DMD) for a plane geometry [18].

Fig. 6 .

 6 Fig. 6. Right: real geometry image, center: generated geometry image, left: inputed thermographic image (images are normalized between 0-255).

Fig. 7 .Fig. 8 .

 78 Fig. 7. Complete test dataset for the trained network after 200 epoch

Fig. 9 .

 9 Fig. 9. DMD modal spectrum real/generated images comparison

TABLE I . IMAGES SIMILARITIES Part name Similarity between real and generated images Cosine distance

 I

			Correlation distance	PSNR	SSIM
	test01	0.95	0.92	14.24	0.79
	test06	0.90	0.83	17.91	0.84
	test06-bis	0.60	0.40	13.59	0.81
	MEDIAN 14 parts	0.90	0.84	18.35	0.86
	STD 14 parts	0.10	0.15	3.95	0.07

TABLE II . SPECTRUMS SIMILARITY Part name Similarity between real and generated images' spectrums vectors

 II

		Cosine distance	Correlation coefficient R²
	test01	0.94	0.88
	test06	0.98	0.96
	test06-bis	0.99	0.99
	MEDIAN 14 parts	0.99	0.98
	STD 14 parts	0.08	0.04


J Y Didier
email: didier@iup.univ-evry.fr.

Abstract² Augmented reality has been shown to be useful in many application areas such as maintenance skills, urban planning, interior design and entertainment, etc.. The development of the mobile augmented reality in these last years is due to the evolution of the technology at various levels: from sensors (GPS, inertial sensor, etc.) to mobile devices (Tablet-PC, PDA, etc.). In this paper, we present a mobile augmented reality system dedicated to outdoor applications. It encompasses a localization process based on an assistance scheme and combining several data acquired from a hybrid sensor (camera, GPS and inertial sensor). The visualization and interaction are performed using a tablet-PC. This system has been tested in an application intended to geologists in order to monitor and supervise the restoration of a castle on a long term. This paper details various components of the system, some results that were obtained and how the whole application was evaluated through end-user.

I. INTRODUCTION

ugmented Reality (AR) enhances our perception of the real world with virtual objects. The considered applications in this area are numerous, going from assistant tools for the experts (like in medicine, maintenance) to entertainment purposes for the general public. However, these applications are generally restricted to indoor and small-scale environments. Nevertheless, few research works ZHUH SXEOLVKHG LQ WKH ¶V RQ RXWGRRU DXJPHQWHG UHDOLW\ such as the MARS project [START_REF] Hollerer | ³Exploring mars: Developing indoor and outdoor user interfaces to a mobile augmented reality system´[END_REF]. Since few years, we notice the increase of this kind of application. This is motivated by the progress made in terms of technology on handheld devices (Tablet PCs, PDAs and cell phones, etc.) and the development of new generation of localization sensors that are more accurate and smaller. Indeed, some of these sensors (like GPS receiver and accelerometers) are now integrated into the mobile devices like smartphones and allow achieving interesting outdoor AR applications. In addition, the advances made in the fields of visualization, interaction and localization have contributed in the development of the outdoor AR projects where the user has a small control on its working environment. The RAXENV (outdoor augmented reality applied to environmental science) project that we present here offers an assistance solution based on AR for the geosciences. This project aims to develop a system combining complementary solutions of localization, visualization and interaction on a mobile device. The localization is used to estimate the point of view of the handheld camera and then to register the real and virtual worlds. The user can interact in real time with the rendered scene which is displayed on a wearable tablet-PC.

In this paper, we describe the system proposed in the RAXENV project. We present each component including: localization, visualization and interaction. The paper is organized as follows. In the next section, we briefly present existing outdoor augmented reality systems. Section 3 overviews the RAXENV project and its objectives. In section 4 describe the hardware and software architecture of the designed system. The details functioning of the whole system are provided in section 5. Experiments conducted in real conditions and obtained results are presented in section 6. Finally, we conclude in section 7 and suggest some future works.

II. RELATED WORKS

During the last years, outdoor AR projects have proliferated over the world. These works focus on different areas such as military, civil engineering or cultural heritage. The existing applications in this field can be generally subdivided into two categories:

ƒ Navigation applications which provide to the user relevant information in order to help him to evolve in his environment; ƒ Assistance applications which assist the users in his task (for example inspection task, etc.).

The first developed applications in this area had for main objective to demonstrate the feasibility of an AR system in outdoor environments. For example MARS [START_REF] Hollerer | ³Exploring mars: Developing indoor and outdoor user interfaces to a mobile augmented reality system´[END_REF] and Tinmith [START_REF] Piekarski | ³Tinmith-evo5architecture for supporting mobile augmented reality environments´[END_REF] projects have proposed original hardware and/or software architectures in adequacy with outdoor AR needs. The MARS project prototyping equipment has inspired several other projects such BARS project [START_REF] Livingston | ³Military applications of augmented reality´[END_REF] which was developed for the military navigation. In the Tinmith project, the researchers have proposed hardware/software architectures managed by the Unix operating system. All theses systems were equipped by large backpacks holding the processing unit (laptops) which is connected to various types of sensors, such as GPS receiver and inertial sensor. Also, the processing unit can in some case communicate with remote database servers using WiFi as in the ARCHEOGUIDE project [4]. The display devices were either see-through display or terminals. However, in recent years, mobile handheld device provides a significant gain in computational

Outdoor Augmented Reality System For Geological Applications F. Ababsa, Member, IEEE, I. Zendjebil, J.Y. Didier, J. Pouderoux, and J. Vairon A power, autonomy and miniaturization. It allows replacing the processing unit and displaying device by a single processing unit, which is also used for visualization. They are more and more used for outdoor AR applications. This is the case of the Vidente [START_REF] Schall | Handheld augmented reality for underground infrastructure visualization[END_REF] and IpCity projects [START_REF] Markus | Urban sketcher: Mixed reality on site for urban planning and architecture[END_REF] where a UMPC (Ultra-Mobile PC) is used. Other applications have used PDAs or mobile phones as it is the case in the TimeWarp project [START_REF] Herbst | ³Timewarp: Interactive time travel with a mobile mixed reality game´[END_REF]. Actually, with the emergence of new generation of mobile devices integrating the navigation sensors (accelerometer, GPS, etc.), new AR applications are proposed for mainstream user to help him in his common life, like for example to guide him to the closest metro stations or to find an interesting restaurant. The progress made in this filed allows the development of new AR systems where the visual augmentations are not only restricted to simple annotations or avatars. Thus, the recent mobile outdoor AR systems provide the access to remote databases such as Geographic Information Systems (GIS) and represent knowledge in visual form. For example in RAXENV system, the geologist can view, on site, the geological sections superimposed on the volcano view. In addition, RAXENV uses new interaction methods which facilitate the data manipulation by the end-users and thereby enriches the data bases (GIS, CAD models, etc.) in real time.

Various challenging issues were addressed in the RAXENV project; they are detailed in the next sections.

III. RAXENV PROJECT

RAXENV is an exploratory project whose main objective is to demonstrate the feasibility of AR paradigm applied to the environmental science and technology such as geology. In this context, several challenges have been identified in order to realize an efficient system able to: ƒ Localize itself using heterogeneous sensors (such as GPS, inertial and camera) and a priori knowledge on the environment (geo-referenced 3D models, Geographic Information Systems (GIS), digital terrain models (DTM)). ƒ Offer features to the user in order to visualize and query the local or remote geological data, using appropriate interaction methods with mobile devices. The RAXENV system was designed for different scenarios involving different needs and skills in the field of environment sciences:

ƒ An instrumented scenario (the castle of Saumur) to manage the monitoring of rebuilding a rampart. ƒ An urban scenario to manage sewerage network based on geo referencing data provided by a GIS. ƒ A panoramic scenario for geology applications to see geological survey map, or communication sites.

IV. SYSTEM OVERVIEW

A. Hardware platform The RAXENV main hardware component is a tablet-PC (CORE 2 DUO U7700) that is simultaneously the processing unit and the display terminal. The Tablet-PC is connected to a localization system relying on a hybrid sensor (figure 1). The hybrid system contains a GPS receiver, A Trimble GPS Pathfinder ProXT receiver, worn by the user. The ProXT receiver integrates a multipath rejection technology providing sub meter accuracy. Its rate update is about 1 Hz. The ProXT receiver uses a Bluetooth wireless connection to communicate with the computer. Add to the GPS receiver, the hybrid sensor is composed of a camera coupled with an inertial sensor which is rigidly attached. The inertial sensor, a Xsens MTi, contains gyroscopes, accelerometers and magnetometers. The advantage of the MTi is that it incorporates an internal digital signal processor which runs a real-time sensor fusion algorithm providing a reliable 3D orientation estimate. Data from MTi are synchronously measured at 100Hz. For the vision, we opted for a USB uEye UI-2220RE industrial camera with 8mm focal length which is extremely compact, low-cost and well adapted for outdoor environments. Colour images with resolution of 768u576 pixels at a frame rate of 25 Hz are streamed to the tablet-PC using a USB 2.0 connection. For interaction, 4 Phidgets [START_REF] Greenberg | ³Phidgets: easy development of physical interfaces through physical widgets´[END_REF] components are connected to the platform. Phidgets are a set of small analogical building blocks which can be plugged on a PC through USB and easily monitored and controlled using an easy to use API. We found that Phidgets give a very efficient way to enrich the plateform prototype by offering the user (in mobility with his only fingers to interact) haptic devices to accomplish some interaction tasks with the system.

B. Software platform

There are three different software parts integrated inside RAXENV system. A database representing the 3D model of the real environment is used to extract a priori knowledge. The sensors and the database are connected to libraries that are developed with our ARCS framework (Augmented Reality System Component) which is using a componentbased approach. So, components are modeled as objects with specific inputs and outputs called respectively slots and signals. An AR application is modeled as a set of nominal states (e.g. initialization, sensor calibration, tracking, reinitialization, etc) that share the same components. Depending on the current state of the application, components may be configured differently and connected to each other in another scheme. Thus, it allows the data-path between components to be configured differently, each one of these configuration (also called sheets) corresponding to a state. In order to control the application we used a finite state machine which also can communicate with the sheet components. In technical terms, components are written in C++ and stored inside dynamic libraries loadable at runtime by the ARCS engine. In order to simplify application descriptions, composite components, represented in an XML format, are allowed to be composed using native components or composite components. The ARCS engine has several functionalities: it parses XML descriptions of AR applications, loads dynamic libraries, instantiates components and finally controls the application behaviour through a finite state machine. The ARCS framework also proposes a graphical editor in order to visually program and configure an application. The whole framework heavily relies on Qt3, which has XML capabilities and possesses an implementation of the signal/slot mechanism. ARCS contain about 120 components implementing data acquisition, image processing, graphical engine as well as some logical glue capabilities. In addition, the database is used by another framework called Elkano developed by Iparla research group. Elkano is a 3D engine of an object-oriented scene graph compatible with OpenGL and OpenGL ES and allows to quickly setting up visualization and interaction techniques for heterogeneous devices (PC, mobile phones, etc.) Written in C++, Elkano can be easily extended through a plugging mechanism and easily integrated in any application to manage 3D contents. Indeed, the pluggin mechanism is intended to provide access to many kinds of "external" processing through a consistent / generic interface.

The GUI of the RAXENV application was designed in C++/CLI using the WinForms.NET API (Figure 2). There is only one main application for all the scenarios but small changes on the GUI are done according the specificity of each scenario needs. ƒ A visualization method for representing subsurface objects (surveys, pipes, cavities, etc.) in AR context; ƒ An interaction approach suitable for mobile and handheld AR system. The developed libraries concern the localization process and the sensors interfacing. The localization library contains the pose estimation approaches using data from the sensors and the database. Other libraries interface the sensors and forward the acquired data to the localization library. The estimated camera poses and the sensors data are then provided to the Elkano platform which performs the rendering of the augmented scene. This is visualized on the screen of the Tablet PC. Moreover, several phidgets are connected the Tablet-PC and allow the interaction with the visualized 3D data.

A. 3D Localization

The localization process is crucial for AR applications. Most AR systems, mainly those based on video see-through, use vision-based approaches to estimate the camera pose [8][9] [START_REF] Didier | ³+\EULG &DPHUD 3RVH (VWLPDWLRQ Combining Square Fiducials Localisation Technique and Orthogonal[END_REF]. However, in outdoor environments, these approaches are sensitive to work conditions such as brightness changes, occlusions and sudden motions. For this reason, these applications converge to use hybrid sensor systems to overcome the drawbacks of using a single type of sensor (i.e. camera) and increase the robustness and the accuracy. The localization module used in the RAXENV platform is combined with data provided by the camera, the GPS receiver and the inertial sensor. it operates using an assistance scheme. The used approach is described in detail in [START_REF] Zendjebil | ³Large scale localization for mobile outdoor augmented reality applications´ International Conference On Computer Vision Theory and Applications[END_REF]. Briefly, the assistance scheme subdivides the localization system into two parts: a main subsystem and an assistance subsystem. The main subsystem is vision-based approach for markerless tracking. 3D salient points extracted from the 3D model are matched with the first frame using a semi-automatic approach based on the SURF descriptor [START_REF] Bay | ³Surf: Speeded up robust features´[END_REF]. Such matched points are then tracked through the video stream using the KLT algorithm [START_REF] Lucas | ³An iterative image registration technique with an application to stereo vision´[END_REF]. When the vision-based tracking fails because of occlusion or blur in images, the system switch automatically to the assistance subsystem, called Aid-Localization (AL) [START_REF] Zendjebil | ³On the hybrid aid-localization for outdoor augmented reality applications[END_REF], including the GPS receiver and inertial measurement. In this case the AL replaces the camera to provide an estimation of the localization until the vision process is able to regain control. For that, the data from GPS and inertial sensor are transformed in the world reference frame defined during the calibration phase [START_REF] Zendjebil | ³A gpsimucamera modelization and calibration for 3d localization dedicated to outdoor mobile applications´[END_REF]. Since our system follows an assistance scheme, we will decompose it into four states:

ƒ Init state: the system initializes itself using a semiautomatic approach; ƒ Vision predominance state: the system uses a visionbased approach to localize itself. ƒ AL predominance state: in this state, the system uses AL subsystem to estimate the localization; ƒ Reinit state: through this state, the system tries to reinitialize the vision after its failure. The system switches from one state to another according to different criteria using the finite state machine (see figure 4) formalism which is mainly used in the theory of computability and formal languages.

B. Visualisation and 3D Interaction

RAXENV project involves a huge quantity of 2D and 3D georeferenced data which must be graphically represented on the screen. We achieved this thanks to the Elkano framework. The different data sources are managed by layers which can be shown or hidden like in a classical GIS. We mainly distinguished 3 main data sources: 1. 3D models and scenes which can be used to represent purely visual elements manually modelled from real world objects (buildings, walls, vegetation, road, etc.); 2. GIS data which can describe, using 2D vectors and some metadata, ground and underground elements. The 3D representation of such objects is achieved in a procedural way by extensions of Elkano; 3. The geological data, one of the most important components of the project. One of these data is the geological model which is represented by a simplified IRUP FDOOHG ³PXOWL-OD\HU´ D VWDFN RI 2.5D grids where each grid represents the elevation of the highest of lowest limit of a given geological layer). To represent those layers, we developed an algorithm which virtually represents a volumetric model from the set of grids. The considered resolution of the grids is about 1 m. Indeed, borders of the geological model are reconstructed on the fly by generating a vertical mesh connecting the layers two by two. This technique also allows us to offer different configurations of cutting planes (like excavation or drill holes) which are very useful tools to analyze geological models for geologists as illustrated by figure 5. In the context of an augmented reality application, the visualization of the captured video frame in real time is of course one of the main need. We offer the user to show this image in foreground or in background of the 3D model with an adjustable transparency level. During the calibration step, the wireframe model and the points are shown similarly. We should also mention that 3D objects are mutable: the user can select each of them and edit their rendering properties (colour, transparency, etc.) or, for GIS objects, their metadata which can also have a direct effect on their graphical representation (for instance the radius or material of a pipe). The augmented view (mixing the camera image and the 3D view of the georeferenced objects) can also be deactivated to offer the user a virtual navigation in the 3D scene around him. The user can thus step back and put into situation elements of interest. To navigate the virtual scene, the prototype implements different interaction techniques specifically designed for a multitouch screen. Among them we can cite Navidget [START_REF] Herbst | ³Timewarp: Interactive time travel with a mobile mixed reality game´[END_REF], ScrutiCam [START_REF]HFOH 6 .QRGHO DQG 3 *XLWWRQ ³1DYLGJHW IRU HDV\ G FDPHUD SRVLWLRQLQJ IURP G LQSXWV´ ,Q 3URFHHGLQJV RI[END_REF], a continuous go to technique and a binoculars-like mode which allows to quickly observe a particular point from the current point of view. As mentioned earlier, some interaction tasks can be done using one of the 4 Phidgets components chosen for the prototype. For instance, a touch button can be used to switch between the augmented view and the free virtual navigation, a slider can be used to go forward/backward in virtual navigation or to move a cut plane in a given direction, a button allows changing the transparency of a selected 3D element in the scene, etc.

VI. RESULTS

Numerous experiments are done in order to evaluate the performances of the RAXENV platform, in particular the software frameworks assessment, the localization accuracy and the end-user reviews.

A. Software framework assessment RAXENV enabled to assess the ARCS framework in the context of mobile augmented reality. Amongst the interesting points that we have observed, we can cite:

ƒ the use of a component based architecture makes the composition of applications quite easy;

ƒ decomposing the application in several sheets and using a state machine in order to control the whole system is very powerful to manage hybrid sensor tracking.

B. 3D Localization results

The localization approach used in RAXENV project was tested in different situations. The results of these experiments are described in details in [START_REF] Zendjebil | ³Large scale localization for mobile outdoor augmented reality applications´ International Conference On Computer Vision Theory and Applications[END_REF]. We will focus mainly on the results obtained on tests conducted in the Saumur castle using the whole system. The tests were performed in the courtyard outside the castle, where we have a priori knowledge of the partial 3D model. The hybrid sensor was calibrated in advance using a set of data (GPS positions, RULHQWDWLRQV RI WKH LQHUWLDO VHQVRU DQG FDPHUD ¶V SRVHV collected on the site. In the figure 6, we can observe the registration of the 3D model represented the castle on the actual view. This registration is obtained from the poses estimated by the localization module and returned to the visualisation module which performs a rendering of the augmented scene. We obtain a mean reprojection error around 1.67 pixels with a standard deviation about 2.37. The mean error of the orientation obtained from inertial sensor is about (x = 0; 27; y = 0:41; z = 0:24) with standard deviations of (0:28;0:43;0:25). In generally the system behaviour is correct and obtained results are satisfactory especially when the position errors are corrected using the error prediction module. Concerning the position obtained from the GPS receiver, the mean error is around (1:84m;1:48m) with standard deviation equal to (1:11m;0:96m) respectively on xaxis and y-axis. The GPS receiver is used only for 2D positioning. In fact, the elevation returned by the GPS is less accurate. To supplement the GPS data, the position on the zaxis is set at the average size of the user with the elevation of the ground, this correspond to the position of the sensors handhold by the user.

C. AR Contributions: end-users reviews

The RAXENV prototype has been tested on the site of the castle of Saumur. No in-depth user study has been done yet but the first reviews and experiments on the site were very encouraging. The end-users (Geologists and Geotechnical experts) have tested the Raxenv tool during the design process and also in real working conditions. They feel very interesting to interact with the 3D geological model when it is mixed with the camera video. They found the AR view very useful to have a better understanding of the underground structures. The use of Phidgets was reported to be a very good idea because it avoids the imprecise contact with the touch screen.

VII. CONCLUSION

In this paper, we have presented the RAXENV platform, an augmented reality system which aims to provide an assistance tool for geoscientists to help them in their outdoor tasks. A great care has been take in the design of the final application to offer the mobile user the possibility to interact with the model using haptic or touch screen interfaces. The project demonstrated the feasibility and highlights the interest of augmented reality in the context of trades applications or general public from geosciences. The demonstrators and prototypes offer to the end-user innovative ways to navigate, interact and interview data implemented. However, the work is underway to be completed mainly concerning the 3D localization. Indeed, the used vision-based approach is dependent on the accuracy of the 3D model. The suggested idea is to move towards SLAM approaches to break away from the use of the 3D model mostly in urban environments where we have only the data extracted from GIS and sensors.

1 .

 1 Hardware platform of Raxenv Project. (a) Tablet PC, camera and inertial sensor (b) GPS receiver.

Fig. 2 .

 2 Fig. 2. Raxenv User Interface V. FUNCTIONAL REQUIREMENTS Figure 3 shows the data flow in the RAXENV platform. The project has focused on different aspects namely: real/virtual registration, access to remote data in a mobility context, visualization and interaction. The developed techniques include principally: ƒ A localization system based on multi-sensors technology

Fig. 3 .

 3 Fig. 3. Data flow in the RAXENV system

Fig. 4 .

 4 Fig. 4. The proposed state machine for 3D localization

Fig. 5 .

 5 Fig. 5. Different cutting planes configuration on the geological model.

Fig. 6 .

 6 Fig. 6. Example of registration on the castle site

ACKNOWLEDGEMENTS

This work is supported by the RAXENV project funded by WKH )UHQFK 1DWLRQDO 5HVHDUFK $JHQF\ ´$15´