De Luca

Laying the foundations for an information system dedicated to heritage building degradation monitoring based on the 2D/3D semantic annotation of photographs

Keywords: Descriptors (according to ACM CCS): Computer Graphics [I, 3, 3]: Picture/Image Generation-Digitizing and scanning Computer Graphics [I, 3, 6]: Methodology and Techniques -Graphics data structures and data types

In the last decade many 3D digitization techniques have emerged allowing the generation of dense and precise digital representations of historical building. However, regardless their level of geometric accuracy or visual realism, 3D models are not yet fully adapted to the conservation analysis purposes. In fact, even if a 3D model can be considered an efficient way to accurately record the state of a building, its potentiality in terms of semantic annotation and spatial distribution of heterogeneous data still remain almost unexplored today. Since several years, photographs appear to be a flexible and well diffused portable support for the heritage documentation. They are a natural medium to annotate and compare temporal states. Thanks to the recent advances in photogrammetry, computer vision and augmented reality, photographs can be also considered as an excellent support for accurate spatial localization. This article presents the first principles for the development of an information system to monitor the historic building degradation based on three main components: a high dynamic range (HDR) imagebased automatic pipeline, an hybrid (2D/3D) semantic annotation method and a domain ontology describing knowledge related to degradation phenomena. The innovative integration of these main components allows us to introduce the notion of "informative continuum" as a key for interconnecting spatialized and semanticallyenriched photographs to populate a knowledge base on the building degradation. The first steps of this on-going project are illustrated by an experimentation carried out on the Caromb church in the south of France.

Introduction

In the field of built heritage, various data describe the states of the monument: scientific imagery, damage mapping, historical archives, etc. Given the difficulty to collect, compare, analyze and validate data prior to restoration, this project aims to mobilize various disciplines (architecture, conservation, mechanics, computer science) to define a novel information processing chain including: spatial metrics, analysis of surface, geometric models of structures, heterogeneous documentary sources, spatiotemporal data, etc. This objective requires the definition of a common and continuous process that establish a technological and conceptual interconnection between the stages of 3D digitization, semantic annotation and structuring of the model (including multi-layers analysis of surfaces), characterization of the state of the building and management of restoration actions. This project, funded by ANR, integrates works of MAP and ENSG concerning the image-based-modeling of architectural heritage and works of CICRP, CRMD focusing on the development of a 3D information system for the management of conservation data. Finally it includes the LMGC expertise physical and structural analysis (FEM-DEM). This article presents the first principles (see section III) for the development of an information system for monitoring the historic building degradation based on three main components: a HDR image-based 3D digitization automatic pipeline, an hybrid (2D/3D) semantic annotation method and a domain ontology describing knowledge related to degradation phenomena. The innovative integration of these main compo-nents today related to the photogrammetry, 3D modelling, knowledge engineering and augmented reality spheres allows us to introduce the notion of "informative continuu" as a key for interconnecting a huge amount of spatialized and semantically-enriched photographs able to populate a knowledge base on the building degradation. The first steps of this on-going project are illustrated by an experimentation carried out on the Caromb church in the south of France.

Related Works

2D and 3D Annotations

The cultural heritage documentation is performed by semantically annotating different representations and supports. Increasing information on a building goes through the annotation of iconographic sources and more specifically photographs (2D annotations) and 3D representations (3D annotations). Annotations on 2D sources can be set up by means of three main methods: manual, automatic and semiautomatic annotation methods. With manual methods, images are annotated one by one in their entirety or partially. Keywords [PAS * 06] or ontologies [HWJA * 05] can be associated to the annotated parts. Automatic methods automatically assign description to images by means of an analysis of image content [START_REF] Shotton | TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout and Context[END_REF]. Finally semi-automatic methods use automatic methods' processes but ask also for manual intervention [START_REF] Barrat | Classification et extension automatique d'annotations d'images en utilisant un réseau Bayesien[END_REF]. Annotations on 3D representations consist in attaching annotations to parts of the 3D representations i.e. to points , to segments, to surfaces [START_REF] Attene | Part-based Annotation of Virtual 3D Shapes[END_REF] or objects [HSB * 08] in the scene. In past years the use of 3D information for images' annotation has known an increased interest. In Phototourism N. Snavely [SSS06] use images orientation around SIFT points so as to annotate images. The work of C. Busayarat [START_REF] Busayarat | A Database of Spatialized and Semantically-Enriched Iconographic Sources for the Documentation of Cultural Heritage Buildings[END_REF] includes the use of a segmented digital mock-up in order to project semantic annotation on images. All these works show that there is an important interest in connecting images with 3D representations for the process of images annotation.

Domain Ontology

An ontology is used to share and reuse knowledge between software and humans among them. Several ontologies emerge in particular knowledge fields (medicine, mathematic, law etc...) in order to analyze and formalize terms [Gru93]. In our case our research is oriented toward heritage building degradation monitoring. The cultural heritage field is really a challenge about its description and analysis. Firstly masonry structure description is realized by lots of heterogeneous information: survey data, scientific imagery, photographs and texts. These data provide constructive, historical and archeological information. The best way to link all these data is creating a semantic model. Furthermore an heritage building is managed by different actors from various scientist domains. In this field, several research group have created their own semantic web ontology in cultural heritage domain. Two main of them emerge: the ontology dedicated to huge data integration through information portals for cultural heritage [START_REF] Hyvönen | Publishing and Using Cultural Heritage Linked Data on the SemanticWeb[END_REF] and MONDIS (MONument Damage Information System) [CVK * 13] dedicated to alteration phenomenon monitoring. Our project aims to combine their specificities into one ontology integrated in an information system.

Images Orientation

To link an image with the real world its localization needs to be known. Two main methods exist: either using physical markers [KB99] or using the combination of internal and external sensors such as GPS and gyroscopes coupled to the camera. Our goal is to mark up fine details on large scale building through images annotations. As sensors do not yet provide an accurate enough localization (precision close to the meter scale) and it is inconceivable to cover an entire historical masonry structure with markers, a new method was needed.

Main approach structure

Our approach aimed at establishing a "digital continuum" from the acquisition to the heritage building information. First an image base covering the building is processed through an automated image orientation method. This method based on a dense image correlation creates an inherent 2D/3D. Images can then be annotated in order to highlight building alterations and an automated propagation of annotations in the spatially oriented image base is set up. The data structuring method allows to manage several analysis levels by overlapping semantic annotations and related information.

Automatic Image-Based Modeling

The core of our approach is based on an ongoing automated image-based 3D reconstruction [START_REF] Pierrot-Desseiligny M | Automated image-based procedures for accurate artifacts 3d modeling and orthoimage generation[END_REF]. The process consist of an automated calibration and relative orientation of photographs then an absolute orientation using 3D scan survey and finally a dense multi-stereo correlation leading to an inherent relation between each pixel and its real 3D coordinates. The photographic acquisition aims to cover most points of view as possible around the building while following a protocol for multi-stereo correlation. For each point of view three bracketed photographs are taken in order to combine them in a HDR image. HDR images allow to get the stone patterns and details independently of the shadows and other diachronic issues for further comparisons. The HDR images are then computed with the SIFT [Low99] algorithm adapted by A. Vedaldi [START_REF] Vedaldi | An open implementation of the sift detector and descriptor[END_REF] in order to extract common features. The SIFT detectors are then used to find a global geometric model of each lens. Thank to the lens' calibration and the SIFT points, a relative orientation is automatically computed through an iterative process and a bundle adjustment to refine each orientation. This first orientation is only relative with no metric measure. A dozen of characteristic points are extracted on a 3D scan point cloud to finely adjust the images orientation in an absolute coordinate reference. This whole process takes about six hours for 1200 photographs on a recent double quad core computer. Then a dense matching surface based reconstruction method [START_REF] Pierrot-Desseiligny M | Automated image-based procedures for accurate artifacts 3d modeling and orthoimage generation[END_REF] generates on each relevant image a depth map. As the images' orientation is precisely known the depth map can be projected to link for each pixel its corresponding 3D coordinates.

Semantic annotation

Images contain a high level of information about shape and colors and are an effective support for documentation. Thus architectural and state conservation analysis can be performed directly on images. The process presented in 3.1 generates 3D information inherent to images. This means that a 2D/3D bijective link is introduced between pixels and their 3D coordinates. This relation allows us to transfer annotations between images. An image of the set is annotated and the 2D-to-3D relation extracts the corresponding 3D point cloud of the drawn area. The corresponding areas on other images of the set is then retrieved by using the 3D-to-2D relation (Figure 1). Starting from a selection on one image the 2D-to-3D relation is used to create a list of 3D coordinates of the annotation (Figure 2).The 3D-to-2D relation conversely permits to retrieve positions in other images where 3D coordinates of the annotation are detected and highlight the corresponding areas [MSDLV13]. This process can be easily transposed on multiple images to define the area to annotate before transferring the annotation on all images. As a consequence, this method provides a simple and fast way to annotate images all at once instead of one by one.

Ontology design

The previous annotation process aims to a building morphological identification. Our ontology will provide a solution to characterize those annotations. The Lassila's method [Her06] appears to be the best to fit our needs: the domain's actors validate a controlled vocabulary which terms set will remain unchangeable. Those terms definitions are also immutable and compose a glossary. Then a thesaurus aims to precisely define the complex and structured relationships between the terms. Finally a hierarchy is set between the previous relations. Two glossary types are exploited in our project : Illustrated glossary on stone deterioration patterns [START_REF] Bromblet | Illustrated glossary on stone deterioration patterns[END_REF] to describe degradation and alterations phenomenon and architecture vocabulary [START_REF] De | Architecture : methode et vocabulaire, inventaire general des monuments et des richesses artistiques de la France[END_REF] to identify masonry structure architectural components. The figure 3 presents a typical encountered alteration example allowing us to explain the usefulness of a domain ontology and to understand the complexity of its conception. The concerned masonry structure is composed of architectural entities built with porous stone containing soluble salts. Due to the effect of the rain or underground circulation, water (active agent) seeps in stone by an alteration mechanism named rising damp. As a consequence, a chemical reaction between water and soluble salts in infiltrated water volume creates tensions because of salt crystallization. This internal pressure is cause of a typical feature by inducing material loss and by producing an alteration named alveolization. If one of these factors is missing, this decay would not appear. Thus five strongly-related components have been identified in order to fully characterize an alteration: masonry structure, architectural entities, associated material, active agent and alteration mechanisms.

Conclusions

This paper has described the bases of an information system dedicated to heritage building degradation monitoring. Our goal is to provide an innovative way to analyze the morphologic and visual state and to describe the architectural point of view of a structured masonry thank to 2D/3D annotations based on spatially oriented photographs. This IS is based on an ontology capitalizing on MONDIS, ICOMOS, and description of architectural components. As this project is at its first steps, only one historical site was covered and the ontology is still to be completed. Caromb church experimentation shows encouraging results on annotations transfer with HDR images: they provide a large range of visual information allow a higher precision than usual photographs in shad- ows and highlights. But even HDR images can only allow a human eye range of information. Other scientific imagery methods such as infra-red thermography, X-ray and LIDAR could bring even more. Our next implementation will then be to integrate them into a multi-layer system capable to combine all those different visual range of data. Also, an imagebased solution for the accurate on-site retrieval of semantic annotation is being developed. Some future informatics implementations are still required.

Figure 1 :

 1 Figure 1: Transfer of annotation between images of the set by using the 2D/3D relation.

Figure 2 :

 2 Figure 2: Research of 3D coordinates of the annotation.

Figure 3 :

 3 Figure 3: An ontology sample to describe architecture components, alteration phenomena, degradations and their relationships.