Nous proposons une solution générique qui peut être appliquée dans différents domaines d'applications métiers. Cependant, dans cette thèse, les expérimentations ont été appliquées au domaine du Bâtiment et Travaux Publics (BTP), en s'appuyant sur des projets de construction.

x

ful to Dr. Christian Sallaberry for his close supervision, constant presence, generous guidance, patience, comprehension and trust during the thesis. I am profoundly thankful to Dr. Sébastien Laborie for his continuous support, motivation, practical suggestions and constructive comments during the completion of my Doctorate. I would also like to express my greatest gratitude to Prof. Richard Chbeir for his warm encouragement, profound belief in my abilities, helpful contributions, valuable advice, exigence and immense knowledge, which inspired me to widen my research from various perspectives. It is a pleasure to have known them and a privilege to have worked with them.

My sincere thanks also go to Prof. Sylvie Calabretto and Prof. Max Chevalier for the time they dedicated to read my dissertation along with their helpful comments.

I am also grateful to the rest of my thesis committee Dr. Pieter Pauwels, Dr. Pierre Bourreau, Mr. Frederic Betbeder and Prof. Kokou YETONGON for their valuable questions, insightful comments, and encouragement. To my closest friend Lara, thank you for listening, offering me advice, and supporting me through this entire process. I owe my deepest gratitude to my love Wissam Bejjani whom without his love, listening, support, inspiration and advice I would not have lasted.

Lastly and most importantly, I am extremely grateful to Jesus Christ who filled me with strength, patience and perseverance to overcome obstacles and finish this work.

vii

Abstract

The recent advances of Information and Communication Technology (ICT) have resulted in the development of several industries. Adopting semantic technologies has proven several benefits for enabling a better representation of the data and empowering reasoning capabilities over it, especially within an Information Retrieval (IR) application. This has, however, few applications in the industries as there are still unresolved issues, such as the shift from heterogeneous interdependent documents to semantic data models and the representation of the search results while considering relevant contextual information.

In this thesis, we address two main challenges. The first one focuses on the representation of the collective knowledge embedded in a heterogeneous document corpus covering both the domain-specific content of the documents, and other structural aspects such as their metadata, their dependencies (e.g., references), etc. The second one focuses on providing users with innovative search results, from the heterogeneous document corpus, helping them in interpreting the information that is relevant to their inquiries and tracking cross document dependencies.

To cope with these challenges, we first propose a semantic representation of a heterogeneous document corpus that generates a semantic graph covering both the structural and the domain-specific dimensions of the corpus. Then, we introduce a novel data structure for query answers, extracted from this graph, which embeds core information together with structural-based and domain-specific context.

In order to provide such query answers, we propose an innovative query processing pipeline, which involves query interpretation, search, ranking, and presentation modules, with a focus on the search and ranking modules.

Our proposal is generic as it can be applicable in different domains. However, in this thesis, it has been experimented in the Architecture, Engineering and Construction (AEC) industry using real-world construction projects.

Résumé

Les avancées récentes des Technologies de l'Information et de la Communication (TIC) ont entraîné des transformations radicales de plusieurs secteurs de l'industrie.

L'adoption des technologies du Web Sémantique a démontré plusieurs avantages, surtout dans une application de Recherche d'Information (RI) : une meilleure représentation des données et des capacités de raisonnement sur celles-ci. Cependant, il existe encore peu d'applications industrielles car il reste encore des problèmes non résolus, tels que la représentation de documents hétérogènes interdépendants à travers des modèles de données sémantiques et la représentation des résultats de recherche accompagnés d'informations contextuelles.

Dans cette thèse, nous abordons deux défis principaux. Le premier défi porte sur la représentation de la connaissance relative à un corpus de documents hétérogènes couvrant à la fois le contenu des documents fortement lié à un domaine métier ainsi que d'autres aspects liés à la structure de ces documents tels que leurs métadonnées, les relations inter et intra-documentaires (p. ex., les références entre documents ou parties de documents), etc. Le deuxième défi porte sur la construction des résultats de RI, à partir de ce corpus de documents hétérogènes, aidant les utilisateurs à mieux interpréter les informations pertinentes de leur recherche surtout quand il s'agit d'exploiter les relations inter/intra-documentaires.

Pour faire face à ces défis, nous proposions tout d'abord une représentation sémantique du corpus de documents hétérogènes à travers un modèle de graphe sémantique couvrant à la fois les dimensions structurelle et métier du corpus. Ensuite, nous définissons une nouvelle structure de données pour les résultats de recherche, extraite à partir de ce graphe, qui incorpore les informations pertinentes directes ainsi qu'un contexte structurel et métier. Afin d'exploiter cette nouvelle structure dans un modèle de RI novateur, nous proposons une chaine de traitement automatique de la requête de l'utilisateur, allant du module d'interprétation de requête, aux modules de recherche, de classement et de présentation des résultats. Bien que nous proposons une chaine de traitement complète, nos contributions se focalisent sur les modules de recherche et de classement.

Introduction

De nos jours, les Systèmes d'Informations (SI) mettent à notre disposition des données non ou semi-structurées. Ces données proviennent de documents hétérogènes c.à.d. ayant des contenus multimédia (p. ex., image, texte, audio, vidéo), de formats différents (p. ex., pdf, txt, mp3, png, etc.) et couvrants des sujets divers. Il arrive souvent que ces documents soient liés les uns aux autres par des liens explicites (p. ex., des références à tout ou partie de documents introduites par des auteurs différents) ou implicites (p. ex., selon les thèmes abordés dans les documents). Dans ce contexte, plusieurs exemples d'application peuvent être identifiés : les SIs exploitant les données sur le web, les données personnelles disponible sur nos ordinateurs (emails, notes, photos, etc.), les données échangées dans des projets industriels (p. ex., dans le domaine de l'industrie manufacturière, du bâtiment, de la médecine, de l'agriculture, etc.).

Dans cette thèse, nous nous focalisons sur les projets industriels multidisciplinaires, avec une application particulière sur les projets de construction qui impliquent un échange de documents hétérogènes interdépendants (p. ex., les Cahiers des Clauses Techniques Particulières ou CCTP, les rapports thermiques, les plans 2D, les photos de chantiers, etc.) encapsulant des données non structurées entre plusieurs acteurs ayant des domaines d'expertises et des intérêts différents (p. ex., architectes, ingénieurs, bureaux d'étude technique, etc.) tout au long du cycle de vie d'un bâtiment.

Le domaine du BTP constitue un example pertinent car, malgré la transformation radicale liée à l'explosion des technologies de la TIC (p. ex., la maquette numérique du bâtiment appelé Building Information Modeling ou BIM) et du Web Sémantique (p.ex, les ontologies modélisant les données du batiment telles que l'ontologie ifcOWL), ce domaine subit une transition numérique relativement lente par rapport aux autres industries. Cela est dû à plusieurs obstacles, parmi lesquels le manque d'un système de RI qui, à partir d'un corpus de documents hétérogènes liés à un projet, retourne des informations pertinentes spécifiques au domaine métier (p. ex., acoustique, thermiques, etc.) ainsi que des informations liées à la structure des documents (p. ex., niveaux de granularités divers tels que les parties pertinentes des documents, dépendances pertinentes entre documents). Cela concerne plusieurs défis scientifiques : (i) le stockage des données, (ii) la représentation des données encapsulées dans les documents hétérogènes, (iii) l'écriture de la requête pour les utilisateurs non experts, (iv) la représentation des résultats de RI avec des informations contextuelles pertinentes, (v) l'optimisation de la RI, (vi) la sécurité des données. Dans cette thèse, nous abordons deux défis principaux dans le cadre d'une démarche de RI novatrice : la représentation des données (Défi 1) et la représentation des résultats de recherche (Défi 2).

Pour ce faire, nous proposons une nouvelle plateforme générique, applicable xi dans plusieurs domaines, intitulée FEED2SEARCH (FramEwork for hybrid molEcule-baseD SEmantic SEARCH) qui a pour but de faciliter la RI sémantique pour les utilisateurs non experts à partir d'un corpus de documents hétérogènes. Plus précisément, les contributions scientifiques de la thèse se présentent dans le cadre de cette plateforme et se déclinent principalement en quatre propositions :

• L'ontologie LinkedMDR : une nouvelle ontologie multi-couches pour repésenter la sémantique d'un corpus de documents hétérogènes tout en considérant les deux dimensions du corpus (structurelle et métier) ;

• Le graphe sémantique Tightly Coupled Semantic Graph : défini formellement pour représenter la connaissance collective d'un corpus de documents multimédias en se basant sur la sémantique de l'ontologie LinkedMDR ;

• Les molécules hybrides Hybrid Molecules : définies formellement en tant que sous-graphes du graphe sémantique pour représenter une nouvelle structure pour les résultats de la RI tenant compte des informations pertinentes accompagnées d'informations contextuelles couvrant les dimensions structurelles et métiers du corpus ;

• Un nouveau modèle de recherche et de classement à partir de graphe, fondé sur la proposition d'un algorithme appelé HM_CSA et des fonctions de classement adaptées, dont le but est de générer les moléules hybrides à partir du graphe sémantique et de les classer convenablement.

Chapitre 2 Représentation Sémantique d'un Corpus Documentaire Hétérogène

Le chapitre 2 présente les premières contributions de la thèse en se focalisant sur le problème de représentation de la sémantique d'un corpus documentaire hétérogène.

Une étude de l'état de l'art autour des standards et des modèles de données existants, qu'ils soient généraux (p. ex., EXIF, TEI, DC, MPEG-7, COMM, M3O, Me-diaOnt, Mpeg-7 Rhizomik, OntoText Data model, XCDF Data Model et LINDO Data Model) ou bien dédiés au domaine du BTP (p. ex., IFC, COBie, gbXML, ifcOWL, BOT, PRODUCT, OPM, les modèles de données Newforma et Kroqi), est décrite à la lumière des cinq défis suivants : (i) la représentation de liens inter et intradocuments, (ii) la représentation de métadonnées génériques, structurelles et relatives au contenu des documents, (iii) la représentation de la sémantique des informations documentaires, (iv) la représentation de la multi-modalité des documents et (v) la prise en considération de l'évolutivité des informations et des documents.

A la base de la synthèse de l'état de l'art qui montre une limitation des solutions existantes vis à vis des défis dégagés, nous proposons une approche sémantique pour représenter les dimensions structurelles et métier d'un corpus documentaire hétérogène à travers un graphe sémantique appelé Tightly Coupled Semantic Graph xii tout en répondant aux cinq défis dégagés. Tout d'abord, nous introduisons l'ontologie LinkedMDR qui comprend trois couches : (i) la couche noyau (Core) qui joue le rôle de médiateur entre les différentes couches et qui introduit de nouveaux et riches concepts et relations (non définis dans les standards existants), (ii) la couche intégratrice de méta-données standards (Standardized Metadata) liant des descripteurs de standards existants (comme DC, TEI ou MPEG7) et (iii) la couche spécifique au domaine (Pluggable Domain-Specific) qui s'adapte à tout domaine d'application. LinkedMDR fournit l'infrastucture nécessaire pour produire le graphe sémantique.

Ce denier est ensuite défini formellement. Enfin, nous présentons un algorithme, appelé Tight Coupling, qui décrit le processus de génération de ce graphe.

Chapitre 3 Recherche d'Information dans un Corpus Documentaire Hétérogène

Le chapitre 3 aborde le problème de la RI sur un corpus documentaire hétérogène. Il montre l'intéret d'exploiter l'ontologie LinkedMDR et le graphe sémantique Tightly Coupled Semantic Graph dans un SI tout en fournissant aux utilisateurs des résultats de recherche couvrant des informations contextuelles à la fois structurelles et spécifiques au domaine métier.

Une étude de l'état de l'art est faite sur les modèles de RI classique (p. ex., Booléens, Vectoriels, Probabilistes) et la RI sémantique (p. ex., les modèles de recherche conceptuelle) et les approches qui les implémentent tout en insistant sur les aspects sémantiques (i) Fully-Fledged, (ii) orientées SPARQL, (iii) orientées graphes, (iv) orientées molécules RDF. Une étude comparative est faite à la lumière des trois défis suivants concernant les résultats de recherche : (i) fournir des niveaux de granularités pertinents, (ii) prendre en considération les dépendences inter et intradocumentaires, et (iii) représenter les résultats dans une structure pertinente avec de l'information contextuelle comprenant à la fois les dimensions structurelles et spécifiques au domaine métier.

A la base de la synthèse de l'état de l'art qui montre une limitation des solutions existantes vis à vis des défis dégagés, nous proposons une nouvelle structure de données, appelée Hybrid Molecules, fondée sur la notion de sous-graphes bien formés et issues du graphe sémantique Tightly Coupled Semantic Graph. Les molécules hybrides apportent une information essentielle ainsi que des informations contextuelles utiles sur les documents, y compris les dimensions structurelles et spécifiques au domaine.

Nous proposons ainsi un nouveau modèle de RI basé sur la notion de molécule hybride. Bien que nous détaillons la chaine complète de traitement de requêtes avec des algorithmes dédiés à l'interprétation de requêtes, la recherche à base de graphe, le classement et la présentation des résultats aux utilisateurs finaux, les contributions de la thèse se situent au niveau des modules de recherche et de classement.

De ce fait, nous proposons HM_CSA, un nouvel algorithme de recherche basé sur xiii des graphes qui génère les molécules hybrides à partir du graphe sémantique, ainsi que des fonctions de pondération qui classent les résultats sous forme de molécules hybrides.

Chapitre 4 Evaluation Expérimentale

Le chapitre 4, présente une évaluation expérimentale des contributions de cette thèse.

Nous avons tout d'abord présenté le prototype qui a implémenté, en Java, les différentes couches de la plateforme FEED2SEARCH dans le contexte du domaine du BTP. Ce prototype est formé de deux modules principaux. Le premier, intitulé LMDR Annotator, annote automatiquement un corpus documentaire hétérogène donné et génère le graphe sémantique Tightly Coupled Semantic Graph. Le second, intitulé HM Query Processor, utilise le graphe généré pour fournir une liste classée de molécules hybrides en réponse à une requête écrite en langage naturel par l'utilisateur.

Nous avons ensuite présenté un ensemble d'expérimentations menées sur des projets de construction, fournies par le partenaire Nobatek/INEF4 et évalués par des utilisateurs du domaine. D'une part, nous avons comparé la concision de la représentation d'un corpus documentaire hétérogène basé sur l'ontologie LinkedMDR avec des modèles de données alternatifs pour la représentation de documents. Nous avons ensuite évalué l'efficacité du module LMDR Annotator dans la génération des annotations automatiques. D'autre part, nous avons évalué l'efficacité du module HM Query Processor dans la génération des molécules hybrides résultant de l'algorithm HM_CSA et des fonctions de pondérations associées. Les expériences ont montré des résultats prometteurs qui nous motivent à poursuivre la mise en oeuvre des prototypes et à l'étude de leur efficacité pour une adoption dans des projets réels.

Chapitre 5 Conclusion

Le chapitre 5 conclut cette étude et présente plusieurs axes de recherche futurs que nous prévoyons d'explorer à court et à long terme.

A court terme, nous travaillerons sur l'interface graphique du prototype pour améliorer surtout la présentation des résultats de recherche. Par exemple, dans l'état actuel du prototype, les molécules hybrides sont visualisées à l'aide d'un outil de visualization de graphe (p. ex., NavigOwl) intégré dans le module HM Query processor. Nous pensons qu'il faudrait améliorer l'affichage afin d'aider les utilisateurs non experts en informatique à mieux interpréter les résultats de leur recherche. Cela pourrait être fait en explorant l'état de l'art sur les métodologies de SERP (Search Engine Result Page) adoptées par les moteurs de recherche actuels et leur impact sur les utilisateurs finaux. xiv A long terme, les contributions de la thèse seront intégrées dans deux projets européens qui ont déjà démarrés en 2018 : BIM4REN avec le partenaire Nobatek et E2S avec le laboratoire LIUPPA. BIM4REN1 vise à fournir un écosystème numérique facilitant l'intégration d'outils numériques innovants, compatibles avec le modèle BIM, dans le processus de rénovation énergétique des bâtiments. Les contributions de la thèse seront intégrées dans le projet pour aider à la RI dans des corpus de documents textuels sur la CVC (Chauffage, Ventilation et Climatisation). Ceci a pour but de fournir aux maquettes BIM vides, générées à partir d'un outil d'analyse 3D lors de la phase de rénovation du bâtiment, les informations manquantes. Quant au projet E2S2 , il vise à fournir un SI générique pouvant être configuré, à la demande, par n'importe quelle organisation afin d'intégrer des services multimédias dédiés à l'indexation, au stockage, à l'enrichissement et au traitement de la sécurité des données de différents domaines, telles que les données relatives à l'énergie et l'environnement. Les contributions de la thèse seront intégrées dans le projet et davantage approfondies dans le contexte du Big Data et de la sécurité des données.

Conclusion

5

Introduction

"We are drowning in information but starved for knowledge."

-John Naisbitt

An Insight into Industry 4.0

Over the past two decades, the Digital Revolution has begun to spread its benefits over the developing world [START_REF] Brynjolfsson | Race against the machine: How the digital revolution is accelerating innovation, driving productivity, and irreversibly transforming employment and the economy[END_REF]. The emergence of Information and Communication Technology (ICT) has spurred opportunities for increased productivity and efficiency in several industries, such as the medical, automotive, aerospace and construction industries. The use of digital technology has been shown to facilitate collaborative approaches to drive innovation and reduce waste [START_REF] Tidd | Managing innovation integrating technological, market and organizational change[END_REF].

The fourth industrial revolution, commonly known as Industry 4.0, is designed to prepare the industries to adapt to the era of the so called "Smart Factory" which involves Internet of Things (IOT) 1 , Cyber-Physical Systems (CPS) 2 , Cloud Computing 3 and Cognitive Computing4 [START_REF] Lu | Industry 4.0: A survey on technologies, applications and open research issues[END_REF]. Along with the surge in these trends comes a large availability of data. This information revolution created a new stage of the Internet development, the Web 3.0, often referred to as, the Semantic Web (SW) or the Web of Data [START_REF] Berners-Lee | The semantic web[END_REF].

"The Semantic Web is an extension of the current Web in which information is given well-defined meaning, better enabling computers and people to work in cooperation", says Tim Berners-Lee.

The initiatives towards these major transformations of the World Wide Web (WWW)

deeply rooted in day-to-day actions of the industries [START_REF] Pawełoszek | Web 3.0 applications in enterprise strategy[END_REF]. The proper handling of the semantic information has become a must for enabling the full potential of Industry 4.0 and its subsidiary fields. However, moving from raw and unstructured data to intelligent data models that can be semantically reasoned upon remains a major mile stone that is yet to be crossed.

The Architecture, Engineering and Construction (AEC)

Industry

AEC is the term used to represent three different but related entities in one common industry where architects, engineers and contractors work together for the achievement of a common goal: the efficient accomplishment of a construction project. Although the adoption of the advances of ICT has been slow within the AEC industry in comparison to other industries [START_REF] Jacobsson | The role of industry: an analytical framework to understand ICT transformation within the AEC industry[END_REF], many government agencies and businesses from the private sector are nowadays imposing strategies to accelerate the digital transition within this industry. Below are some examples:

• The French Minister of Territorial Equality and Housing announced, in December 2014, the launch of the so called "Plan Transition Numérique dans le Bâtiment" (PTNB) as part of the construction stimulus plan for the purpose of reaching savings of nearly e1 trillion in 2025. The dedicated funds are up to e70 million [START_REF]Plan Transition Numérique dans le Bâtiment (PTNB). Feuille De Route Opérationnelle[END_REF];

• The UK Government Construction Strategy (GCS) 2016-2020 [START_REF]Government Construction Strategy[END_REF] sets out the Government's plan to develop its capability as a construction client and act as an exemplary client across the industry. One of its principle objectives is to embed and increase the use of digital technology. The resulting projects are worth £163 billion;

• Some of the largest construction companies around Europe have joint forces to put together the European Construction Industry Manifesto for Digitalisation 5 which calls on the European Union (EU) to take strong political action for setting the appropriate regulatory framework on data policy and budgetary focus on digital skills, and Research and Development (R&D).

Building Information Modeling (BIM)

BIM is an intelligent 3D model-based process that gives AEC actors (e.g., architects, engineers, consultants, etc.) the insight and tools to more efficiently plan, design, construct, and manage buildings and infrastructure 6 throughout their whole life-cycle (See Fig. 1.1). The use of BIM empowers the collective approach helping actors of the same construction project in exchanging information [START_REF] Jiang | Research on BIM-based Construction Domain Text Information Management[END_REF].

BIM covers more than only building geometry, but also various relationships between objects and information beyond geographic data, such as light analysis, quantities and properties of building components. Thus, it allows the actors to perform calculations, analysis and simulations on 3D object properties, which helps them in the process of decision-making [START_REF] Huovila | Linking IFCs and BIM to sustainability assessment of buildings[END_REF]. Recent studies invoke seven dimensions of BIM extending its 3D representation to cover (i) the temporal dimension (4D), (ii) the financial dimension (5D), (iii) the energy dimension (6D), and (iv) the operational dimension (7D) [START_REF] Mcarthur | A building information management (BIM) framework and supporting case study for existing building operations, maintenance and sustainability[END_REF].

Obstacles to Construction 4.0

While BIM presents a major breakthrough in the AEC industry, there are still barriers putting off companies, especially Small-to-Medium sized Enterprises (SMEs), from fully adopting it and keeping up with the recent ICT advances in order to accomplish the shifting to Industry 4.0, also known in the construction industry as Construction 4.0:

• On one hand, the construction industry is a large and old well established sector of the economy making it more resistant to changes relative to other industries. Adopting new technologies, even when the long term advantages are clear, attributes a large upfront cost. Companies are required to invest in acquiring new software tools and professionals. Also, the existing labor force has to be trained to use and rely on new software tools. This problem is aggravated by the lack of enough trained personnel who are qualified to support this transition.

• On the other hand, the uncanny gap between the new technologies and the current status of the industry also plays a major role in hindering the transition to Construction 4.0. For instance, many construction projects rely heavily on the use of documents as they contain interdependent information required to manage the work progress. The BIM model, overlooking the notion of documents, still cannot replace all this information. Further, the users still have to manually intervene in order to enrich the model as there are no existing methods or tools to assist them. As a consequence, the adoption of BIM has somehow been restricted to the early conception phase.

Project Information Management (PIM) solutions [START_REF]Newforma Project Center Twelfth Edition[END_REF][START_REF]Kroqi Platform[END_REF] are recently being adopted by SMEs as intermediary digitization tools and platforms compatible with the BIM model (by means of plugins, external links to BIM Software). They mainly offer project management services and traditional operations (such as classification, filtering, comparison, and archiving) on documents of several formats. However, they do not offer any solution that uses the information embedded in the documents to serve the BIM model.

Thesis Context

Nobatek/INEF4

Nobatek/INEF4, the co-financier of the thesis, is a French Institute for the Energy Transition of the Building 11 . It provides innovative solutions to support the entire construction sector (architects, contractors, consultants) for the energy and environmental transition.

Since 2014, NOBATEK/INEF4 has been renowned as the French leader for investing in European H2020 projects in the construction sector involving several R&D areas (e.g., Sustainable Districts, Digital Ecosystems, Innovative & accessible BIM tools for all the building renovation value chain). Nobatek/INEF4 sets out its strategic digital road map for 2024 that emphasizes the development of digital tools and platforms supporting BIM, and collaborations with active entities in this sector such as laboratories (e.g., Le2i laboratory 12 ), enterprises (e.g., EnerBIM 13 ) and communities (e.g., Linked Building Data (LBD) Community Group14 of the World Wide Web Consortium (W3C)). Although the motivating scenario of Fig. 1.3 is applicable to several industries (such as the medical, manufacturing industries), we particularly apply it to the AEC industry (See Fig. 1.4) and give examples on its key elements:

Motivating Scenario

• The multi-disciplinary project comes down to a given construction project;

• The large amount of unstructured information comes down to text and multimedia content describing several topics such as the building data including all the underlying systems (e.g., Heating, Ventilation, and Air Conditioning (HVAC), Electricity, Plumbing, etc.), project costs, etc.;

• The heterogeneous documents come down to contracts, administrative forms, technical specification documents, 2D drawings, BIM files, on-site photos, etc.

These documents involve the diverse information described above. They do not follow a common structure and are serialized in different formats (e.g., pdf, docx, xlsx, png, ifc, dwg, etc.);

• The interplay between documents comes down to several implicit and explicit dependencies between them. For instance, a general technical specification document describing general characteristics of the exterior facades of the buildings may refer to a more detailed document describing their thermal properties (e.g., light transmission, solar factor, etc.)

• The actors exchanging these documents come down to owners, architects, contractors, consultants, etc. having different expertise (Architecture, HVAC, etc.)

• The project life-cycle comes down to the building life-cycle starting from the early design stages such as the feasibility (mainly for cost and schedule estimations) and authorization of the project, the analysis and execution studies to the construction, site management and maintenance stages.

In this context, at any point in time of the project, an actor may need to search for a particular information. For instance, consider an engineer who works in a specialized consultancy office supporting the owners of the project in quality performance studies. His main role is to ensure the compliance of the building's exterior facades with environmental standards. For his task to be done, he needs to search for all the information regarding the exterior facades from the whole collection of documents.

This means exploring the variety of diverse information related to the exterior facades (such as thermal studies, acoustic studies, etc.) in the form of EXCEL sheets, WORD documents, PDF, technical 2D drawings, etc., finding relevant parts from a bunch of other non relevant information within a single document, and tracking document dependencies to search for complementary or related information.

Main problems

Currently, actors of multi-disciplinary projects have difficulties while searching for a specific information from the collection of heterogeneous documents within these projects. This is due to:

• Problem 1 -The lack of an adapted digital tool (e.g., enterprise search engine tool) that is capable of providing the users with the required domain-specific information (e.g., acoustic and thermal studies conducted on the exterior facades of the building) from the heterogeneous document corpus representing the project while considering further structural characteristics of the documents such as the relevant granularity levels (e.g., the section of the document that details on the exterior facades) and the documents' dependencies (e.g., references between the technical specification documents).

For instance, as mentioned in Sect. 1.2.2, PIM solutions offer traditional operations on documents, such as a keyword-based search over the metadata of documents, which neglects the semantic handling of information contained in these documents and their dependencies;

• Problem 2 -The lack of an underlying robust data model which is capable of representing the knowledge embedded in these documents including the domain-specific information and the structural characteristics of the documents.

For instance, as mentioned in Sect. 1.2.1, although the BIM model is an innovative solution in the AEC industry, it still cannot replace the construction related documents, as it neglects crucial information embedded in these documents (e.g., the dependencies between them).

Consequently, actors of multi-disciplinary projects, especially in SMEs, often manually search for information, which is a very tedious and time-consuming job.

Main Challenges

In this thesis, we mainly address the challenge of searching, in a novel way, for relevant information from the heterogeneous document corpus representing the multidisciplinary project. By novel we mean (i) considering information beyond what is explicitly shown in the raw documents (e.g., their dependencies), (ii) responding to users inquiries by relevant answers helping them to reduce their workload, time and efforts. More concretely, this can result in two main challenges, among several other challenges that remain out of this thesis scope 15 :

• Challenge 1 -Representing the collective knowledge embedded in a heterogeneous document corpus through a robust data model;

• Challenge 2 -Providing novel search results based on the proposed data model.

Tackling these challenges is crucial and a step forward for reducing the gap between new technologies imposed in the era of Industry 4.0, such as Construction 4.0, and the current status of the market. For instance, in a particular application to the AEC industry, this innovative search could serve the BIM users in enriching the model with the required missing information.

Proposal

"Semantic Technology makes everything connected to anything and helps to build Linked Data through intricate models for representing information", Source: Ontotext 16 , 2018

In this thesis, we first propose to inject semantics on a heterogeneous document corpus and represent it through a semantic network (i.e., knowledge graph using the advances of the SW) while considering two different dimensions:

• A structural dimension which represents structural aspects of the documents such as their metadata information, and possible dependencies between them;

• A domain-specific dimension which represents the knowledge describing the content of the documents related to a specific application domain, such as the AEC industry.

We then propose to rely on the resulting knowledge graph, which we call a tightly coupled semantic graph for an enriched Information Retrieval (IR) over the heterogeneous document corpus.

FEED2SEARCH Framework

Based on our proposal, we introduce a novel framework, entitled FEED2SEARCH, which stands for FramEwork for hybrid molEcule-baseD SEmantic SEARCH over a 15 Such as handling Data Storage, Big Data, Data Security, and Query Writing for non expert users. 16 https://ontotext.com/ heterogeneous document corpus. It is a generic framework since it is applicable to a heterogeneous document corpus in many domains, such as the AEC industry. The main purpose of FEED2SEARCH is to facilitate the query processing over a heterogeneous document corpus for non computer expert users by providing them with relevant answers in response to their natural language queries. As shown in • The Knowledge Representation Layer -It introduces a novel multi-layered ontology, entitled LinkedMDR [START_REF] Charbel | LinkedMDR: A Collective Knowledge Representation of a Heterogeneous Document Corpus[END_REF] which provides the infrastructure for the generation of the tightly coupled semantic graph. It is based on external knowledge bases (such as metadata standards and domain-specific ontologies) and handled by an ontology engineer;

• The Indexing Layer -It provides services, technologies and Application Programming Interfaces (APIs) (based on well-known techniques such as automatic metadata extraction [START_REF] Greenberg | Metadata extraction and harvesting: A comparison of two automatic metadata generation applications[END_REF], semantic annotation [START_REF] Cunningham | A framework and graphical development environment for robust NLP tools and applications[END_REF], Natural Language processing (NLP) and Text Engineering [START_REF] Maynard | Natural language processing for the semantic web[END_REF]) in order to index the given external heterogeneous document corpus and generate the tightly coupled semantic graph representing it based on the backbone LinkedMDR ontology. This layer is handled by a corpus expert which provides the document corpus as well as some parameters required to build the graph (e.g., the granularity levels required to describe the documents);

Main Contributions

The key contributions, previously identified in the different layers of FEED2SEARCH framework, are described as follows.

LinkedMDR ontology

The ontologies provide high semantic expressiveness power for knowledge representation [START_REF] Staab | Handbook on ontologies[END_REF]. Thus, LinkedMDR ontology is one of the major contributions of this study. The proposed ontology is made of three main layers: the Core layer serving as a backbone and a mediator among the different layers, the Standardized Metadata layer relying on external metadata standards (e.g., [START_REF]Dublin Core Metadata Initiative[END_REF][START_REF]The Moving Picture Experts Group[END_REF][START_REF]The Text Encoding Initiative Consortium[END_REF]), the Pluggable Domain-Specific layer which can adapt to any domain application through a flexible and easy mechanism to align with external domain-specific ontologies, such as well-known ontologies in the AEC industry (e.g. [START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF]).

Tightly Coupled Semantic Graph

The tightly coupled semantic graph is one of the major contributions of this study as it represents the collective knowledge embedded in a heterogeneous document corpus based on the infrastructure provided by the proposed LinkedMDR ontology. We formally define this graph using two coupled external resources: a heterogeneous document corpus and a domain-specific ontology (for the pluggable domain-specific layer of LinkedMDR). Thus, it embeds instances representing the domain-specific components of the corpus, others representing the structural components, and hybrid links representing relations between the two different components. A dedicated tight coupling algorithm describes the mechanism of the generation of the graph.

Hybrid Molecules

The definition of a novel structure extracted from a tightly coupled semantic graph, which we call the hybrid Molecules, is one of the major contributions of this study as it is a means to provide innovative search results covering both the domain-specific and the structural-based dimensions of the documents. The Hybrid Molecules consist of well-defined sub-graphs that we formally define in view of the characteristics of a tightly coupled semantic graph and the definition of a molecule concept in the literature [START_REF] Valle | A first step towards stream reasoning[END_REF][START_REF] Ding | Tracking rdf graph provenance using rdf molecules[END_REF][START_REF] Endris | MULDER: Querying the Linked Data Web by Bridging RDF Molecule Templates[END_REF][START_REF] Galkin | SJoin: A Semantic Join Operator to Integrate Heterogeneous RDF Graphs[END_REF][START_REF] Newman | A scale-out RDF molecule store for improved co-identification, querying and inferencing[END_REF]. They are hybrid as they encapsulate domain-specific information coupled with related structural-based information of the documents.

The hybrid molecules bring in a core information together with helpful contextual information of the documents.

Hybrid Molecule-based Search and Ranking

On the basis of the proposed Hybrid Molecules, one of the major contributions of this study is to provide the underlying IR model that exploits this novel structure.

Although we present an overall pipeline (Hybrid Molecule-based Query Processing)

with a bunch of algorithms for each stage of the query processing, we focus in this thesis on the IR model behind the Hybrid Molecule-based Search and Ranking modules.

The former presents a novel graph-based search algorithm, entitled HM_CSA which generates relevant hybrid molecules from a tightly coupled semantic graph w.r.t. a user's natural language query. The latter relies on a series of weight mapping functions which rank the molecules conveniently.

Publications

The contributions of this thesis are published and submitted to the following conferences:

1. Nathalie Charbel, Christian Sallaberry, Sébastien Laborie, Gilbert

Report Organization

The rest of this report is organized as follows.

Chapter 2 tackles the problem of representing the collective knowledge embedded in a heterogeneous document corpus. We first review existing standards and data models for document representation regardless of the application domain, then considering the AEC industry. We then describe our proposal within a semantic tight coupling approach which comprises (i) a formal definition of a tightly coupled semantic graph, (ii) an introduction to LinkedMDR ontology and a description of its layers, and (iii) a tight coupling algorithm which provides the required pseudo code in order to generate the graph using the services of the Indexing Layer together with the knowledge provided by LinkedMDR in the Knowledge Representation Layer of FEED2SEARCH.

Chapter 3 tackles the problem of IR over a heterogeneous document corpus while providing the users with innovative search results. In other words, it addresses the challenge of searching for relevant information from the proposed tightly coupled semantic graph representing the corpus and providing query answers with meaningful context including both structural and domain-specific dimensions of a heterogeneous document corpus. We first review existing IR models including traditional and semantic models, and the variety of approaches and systems that implement them. We then propose a novel data structure for query answers which we call Hybrid Molecules, extracted from a tightly coupled semantic graph. We also detail on the novel Hybrid Molecule-based Query Processing Layer of FEED2SAERCH, with a focus on the Search and Ranking modules.

Chapter 4 presents the experimental evaluation study for the purpose of evaluating the contributions of Chapter 2 and Chapter 3 in the context of a domain-specific application, particularly the AEC industry. We first describe the implemented Javabased prototype with its two dedicated modules. The former, entitled LMDR Annotator, implements the Indexing Layer of FEED2SEARCH. It automatically annotates a heterogeneous document corpus and generates the tightly coupled semantic graph.

The latter, entitled HM Query Processor, implements the Query Processing Layer of FEED2SEARCH. It exploits the generated semantic graph and provides a ranked list of Hybrid Molecule-based answers in response to a user's natural language query. We then describe a set of experiments conducted on construction projects, provided by Nobatek/INEF4. These experiments show promising results for an adoption in real-world applications.

Chapter 5 concludes this study and presents several future research directions that we are planning to explore afterwards. The ever increasing need for extracting knowledge from heterogeneous data has become a major concern. One of the utmost challenges remains in representing this data, especially when it is dispersed among various types of interdependent documents generated from different sources. This chapter tackles the problem of representing the collective knowledge embedded in a heterogeneous document corpus for it to be exploited by a Semantic Information Retrieval (SIR) system. We propose a modular and generic semantic approach relying on a tightly coupled semantic graph where we associate semantics on two different dimensions: the content of the documents which depends on a domain-specific knowledge, and the structural and metadata information related to them. One of the major characteristics of this graph stands in the coupling information between hybrid components i.e., components of the two different dimensions. We generate such a graph based on a tight coupling algorithm and a backbone ontology which we call LinkedMDR. We introduce Linked-MDR as a novel multi-layered ontology. It offers core components modeling the documents, their relations and their metadata information at different granularity levels together with a pluggability feature that makes it adaptable to any domainspecific knowledge.

Introduction

Over the past two decades, modeling heterogeneous data has posed significant research [START_REF] Fernández | Semantically enhanced information retrieval: An ontology-based approach[END_REF][START_REF] Kiryakov | Semantic annotation, indexing, and retrieval[END_REF][START_REF]Ontotext Platform[END_REF][START_REF] Suárez-Figueroa | The landscape of multimedia ontologies in the last decade[END_REF]. This has increasingly migrated into industrial applications aiming at providing the different actors intervening in a modern project with a powerful and intelligent search engine reducing their workloads, cognitive efforts and errors over heterogeneous document corpora. As mentioned in Chapter 1, this is particularly observed in the AEC industry. For instance, Fig. In order to provide the basis for IR and search applications, several challenges arise for modeling such documents. In our context, we are particularly interested in the following challenges2 :

• Challenge 1. • Challenge 1.5: Ensuring extensibility -The information may evolve over time.

This may involve new document link types, new vocabulary, new document formats, new media types, etc. For instance, an extension of the scenario illustrated in Fig. 2.1 would integrate an audio file that analyzes the noise impact before and after the cladding of the facades.

In the literature, several works have been undertaken to define standards [START_REF]Dublin Core Metadata Initiative[END_REF][START_REF]Exchangeable Image File Format for digital still cameras[END_REF][START_REF]The Moving Picture Experts Group[END_REF][START_REF]The Text Encoding Initiative Consortium[END_REF] and data models [4,[START_REF] Bloechle | XCDF: a canonical and structured document format[END_REF][START_REF] Brut | Integrating Heterogeneous Metadata into a Distributed Multimedia Information System[END_REF][START_REF] Garcia | Semantic integration and retrieval of multimedia metadata[END_REF][START_REF]Ontotext Platform[END_REF][START_REF] Saathoff | Unlocking the semantics of multimedia presentations in the web with the multimedia metadata ontology[END_REF] for document representation. On the other side, other domain-oriented works, such as building-oriented standards [START_REF]Construction Operations Building Information Exchange (COBIE)[END_REF][START_REF]buildingSMART. IFC-Industry Foundation Classes, IFC Releases[END_REF][START_REF]Green Building XML[END_REF] and data models [START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group. Building Product Ontology[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group[END_REF]] have taken up the challenge of modeling building data and managing related documents [START_REF]Newforma Project Center Twelfth Edition[END_REF][START_REF]Kroqi Platform[END_REF]. To the best of our knowledge, none of the existing works have tackled the above challenges combined.

In this context, our aim is to provide a clear and a complete picture of the collection of heterogeneous documents w.r.t. the identified issues, so it could be exploited by an IR system. Thus, in this chapter, we present a modular and generic tight coupling approach where, given a heterogeneous document corpus and an external domain-specific knowledge, it generates a tightly coupled semantic graph representing the collective knowledge embedded in that corpus based on two different dimensions: the domain-specific information that reflects the content of the documents, and the structural and metadata information describing them. We thus introduce LinkedMDR [START_REF] Charbel | LinkedMDR: A Collective Knowledge Representation of a Heterogeneous Document Corpus[END_REF], a multi-layered ontology, which adapts to any domain application and provides the required infrastructure to generate this graph, including the capability of: (i) representing various inter and intra-document relations, (ii) representing metadata information at different levels of precision while relying on the existing standards, (iii) handling advanced information semantics on the two different dimensions, (iv) handling multimodal documents, and (v) ensuring extensibility. We also present a tight coupling algorithm which details the procedure of generating such a graph.

The contributions of this chapter are summarized by:

• The Tight Coupling Approach, which provides a general overview of our proposal;

• The Tightly Coupled semantic graph, which is formalized on the basis of the two identified dimensions of a heterogeneous document corpus;

• LinkedMDR [START_REF] Charbel | LinkedMDR: A Collective Knowledge Representation of a Heterogeneous Document Corpus[END_REF], which is the backbone ontology to generate this graph;

• The Tightly Coupled Algorithm, which provides a means to implement our proposal.

The remainder of this chapter is organized as follows. Sect. 2.2 reviews existing standards and data models for document's structure and content description as well as for the building data in the AEC industry. Sect. 2.3 describes the proposed tight coupling approach which comprises the tightly coupled semantic graph, LinkedMDR ontology and the underlying tight coupling algorithm. Sect. 2.4 concludes the chapter.

Related Work

In this section, we review existing standards and models addressing (i) metadata for document representation in general, and (ii) building data in particular. This literature review is based on academic researches and industrial solutions as well.

Metadata Standards and Data Models for Document Representation

The metadata standards and data models for document representation are domain independent. They can be divided into four different categories: (1) single mediabased standards which handle one type of media content, (2) multimedia standards which handle multiple types of media content, (3) ontology and knowledge-based models which build on existing multimedia standards to provide more semantics, and (4) other models which build on traditional ways of representing the data neglecting semantics of the information.

Single Media-based Standards

We consider single media-based standards those basically describing only text or image contents. Renowned standards dedicated for audio and video are, in general, multimedia-based as they involve at least two media content types in their descriptions 3 . Although there exist many other standards in this category, we present an example on the most commonly used standard for image description and another one for text description:

EXIF -The Exchangeable Image File (EXIF) format is a widely used standard for describing digital images [START_REF]Exchangeable Image File Format for digital still cameras[END_REF]. It mainly supports a set of tags related to image data structure (e.g., width, height, pixel composition), version, data characteristics (e.g., color space information), configuration (e.g., image compression mode), user information (e.g., user comments), date and time, recording offset (e.g., bytes of JPEG data), picture-taking conditions (e.g., exposure time, brightness), GPS (e.g., latitude, altitude) and many other tags (e.g., image title, software used, creator, copyright).

TEI -

The Text Encoding Initiative (TEI) [START_REF]The Text Encoding Initiative Consortium[END_REF] is a commonly adopted text-driven descriptive standard. It is based on the eXtensible Markup Language (XML) format and provides a way to describe information and meta-information within a textual document. TEI offers a representational form of a text as well as a set of semantically rich elements that could improve IR. There is no formal grouping of TEI elements.

However, it is possible to classify them along the following categories: the structural elements (e.g., chapters, sections, paragraphs, lists, items, page break, table, cross reference links), the highlighting elements (e.g., highlighted text, italic, bold), the logical and semantic elements (e.g., title, name, measure, date, address, abbreviation, emphasis), the analytical elements (i.e., additional information, e.g., notes, corrections, indexes), and the figures and graphics4 .

Multimedia-based Standards

Although there exist many other standards in this category, we describe the most adopted ones in the literature:

DC -The Dublin Core (DC) Metadata Initiative5 is a metadata standard for describing a wide range of online multimedia resources. The DC Metadata Element Set consists of 15 Elements describing the content of a document (e.g., title, description, etc.), the intellectual property (e.g., creator, rights, etc.), and its instantiation (e.g., date, format, etc.) [START_REF] Weibel | Dublin Core Metadata for Resource Discovery[END_REF]. This standard also offers a set of qualifiers which aim to modify the properties of the DC statements [START_REF]Dublin Core Metadata Initiative[END_REF].

MPEG-7 -

The Multimedia Content Description Interface (MPEG-7) is an ISO/IEC standard developed by MPEG (Moving Picture Experts Group) [START_REF]The Moving Picture Experts Group[END_REF]. Its aim is to provide a rich set of complex standardized tools describing low and high level features as the introduction of new components for specific domains [START_REF] Hunter | An overview of the MPEG-7 Description Definition Language (DDL)[END_REF].

Ontology and Knowledge-based Models

There is often the need to combine several standards in order to meet the requirements of complex multimedia applications [START_REF] Scherp | A method for integrating multimedia metadata standards and metadata formats with the multimedia metadata ontology[END_REF]. Many initiatives have been taken

for the purpose of building multimedia ontologies and knowledge bases, such as [4,[START_REF]Ontotext Platform[END_REF][START_REF] Saathoff | Unlocking the semantics of multimedia presentations in the web with the multimedia metadata ontology[END_REF]102], or transforming existing formats into ontologies, such as [START_REF] Garcia | Semantic integration and retrieval of multimedia metadata[END_REF]. The aim of these studies is to bridge the gap between low level features with automatically extractable information by machines and high level human interpretable features of the same information [START_REF] Suárez-Figueroa | The landscape of multimedia ontologies in the last decade[END_REF]. subject to multiple alignments with several existing multimedia metadata, such as MPEG-7, DC and EXIF. The main purpose of the mappings is to ensure interoperability among the most commonly adopted metadata formats on the web describing media resources [START_REF] Suárez-Figueroa | The landscape of multimedia ontologies in the last decade[END_REF].

Mepg-7 Rhizomik -The Mpeg-7 Rhizomik ontology is the first complete MPEG-7 ontology designed based on one-to-one automatic mapping of the entire standard to Web Ontology Language (OWL) [START_REF] Garcia | Semantic integration and retrieval of multimedia metadata[END_REF] using XSD2OWL 6 . The XML schema to OWL mapping is complemented with a mapping of XML metadata instances to Resource Description Framework (RDF) in order to generate semantic-enabled representations of the input metadata instances and thus facilitate data integration when several sources are available. The Mpeg-7 Rhizomik ontology have been used as an upper-level ontology for other domain ontologies, such as music ontology.

OntoText Data Model -The OntoText company 7 provides proprietary solutions for data analytics for global businesses. These solutions are based on semantic data models in order to (i) represent large amount of information coming from diverse data sources, (ii) create meaningful connections between structured and unstructured data, and (iii) obtain valuable insights by reasoning over the semantic graphs.

The basic approach behind the construction of such graphs is not to rely on a common data model but to automatically extract Named Entities and rules that create relations between them [START_REF]Ontotext Platform[END_REF]. These entities basically cover Person, Organization, Location from renowned knowledge graphs such as DBpedia8 , Wikidata 9 and Schema.org10 .

Other Models

Although there exist many other non ontology or knowledge-based models, we present the following two models as examples on a renowned data model for a specific document format (PDF) and a renowned generic data model for multimedia content and various document formats:

XCDF Data Model -Problems such as over-segmentation, additional noisy information, and lack of structure preservation produced by PDF generators are often associated with the PDF format. Great effort has been put into overcoming these issues [START_REF] Bloechle | XCDF: a canonical and structured document format[END_REF]: XCDF is a canonical format which purpose is to represent the results of the physical structure extraction of the PDF documents in a single and structured format. It is based on the XML format and its DTD provides basic elements for general document representation (page, fonts, image, graphic, textblock, textline and token).

LINDO Meta Model -In the context of distributed multimedia information systems, such as the video surveillance applications that use different indexing engines, interoperability problems arise when metadata of different formats are combined together. The LINDO project (Large scale distributed INDexation of multimedia Objects) 11 takes up the challenge of handling different metadata standards within its distributed information system, such as DC, EXIF and MPEG-7. Thus, the authors in [START_REF] Brut | Integrating Heterogeneous Metadata into a Distributed Multimedia Information System[END_REF] define a unified XML-based metadata model that encapsulates these standards based on two levels: general metadata information describing the entire document and metadata related to multimedia contents (image, text, video and audio).

Building-Oriented Standards and Data Models

The building-oriented models are limited to applications in the AEC industry. They can be divided into three different categories: (1) standards which model the building data, (2) ontology-based models which build on existing standards to provide more semantics, and (3) other models which build on traditional ways of representing the building data neglecting semantics of the information.

Standards

Although there exist many other standards in this category, we present the widely known standards:

IFC -The IFC (Industry Foundation Classes) [START_REF]buildingSMART. IFC-Industry Foundation Classes, IFC Releases[END_REF] is one of the most recent and widely used information exchange standards in the building and construction industry. It is developed by buildingSMART alliance 12 in the context of the open-BIM approach for collaborative design, realization and operation of buildings. The main purpose is to define an open and common data schema that is independent of the different software applications. This standard is object-oriented [START_REF] Belsky | A framework for semantic enrichment of IFC building models[END_REF]. It involves building and construction objects (including physical components, spaces, systems, processes and actors) and relationships between them [START_REF] Huovila | Linking IFCs and BIM to sustainability assessment of buildings[END_REF]. IFC specifications are written using the EXPRESS data definition language 13 , but they are also published in XSD (XML Schema Definition).

COBie -The Construction Operations Building information exchange (COBie) [START_REF]Construction Operations Building Information Exchange (COBIE)[END_REF] is a non proprietary international standard for the exchange of non geometric building information. It was initially led by the Engineer Research and Development

Center, Construction Engineering Research Laboratory of the U.S. Army, Corps of Engineers. It is also implemented by the buildingSMART international as a subset of the IFC [START_REF]buildingSMART. IFC-Industry Foundation Classes, IFC Releases[END_REF], specifically as a Model View Definition (MVD) [START_REF]buildingSMART. Model View Definitions[END_REF] of this standard with simplified information needed by facility managers. It focuses on the operations, maintenance and asset management information related to equipment and spaces. The COBie data is serialized in EXCEL spreadsheets or IFC file formats such as XML and STEP files. For the purpose of providing a specification of the way that information can be transferred between the spreadsheets and IFC versions, the COBie standard is integrated as part of the US National BIM Standard (NBIMS) 14version 2.

gbXML -The Green Building XML standard (gbXML) [START_REF]Green Building XML[END_REF] is an open XML schema, originally submitted by Green Building Studio Company 15 , in order to ease the transfer of BIM data between disparate building design software tools for energy performance analysis. The information mostly covered by the gbXML standard is related to spaces and surfaces [START_REF] Zhang | Evaluations on XML standards for actual applications[END_REF] along with other elements leveraging the overall energy consumption of the building (e.g., schedule of use, thermostat temperature set-points, internal loads, materials and constructions). The core idea behind it is to describe multiple buildings that are somehow related and located in the same climate region.

Ontology-based Models

Although there exist many other ontologies in this category, we present the most referenced by the literature:

ifcOWL -The ifcOWL [START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF] is the representation of the IFC standard [START_REF]buildingSMART. IFC-Industry Foundation Classes, IFC Releases[END_REF] into ontology by converting the EXPRESS schema of the standard into OWL. The major goal is to migrate into SW technologies to support data interoperability, flexibility and extensibility within information system applications in the construction industry. In this direction, there have been several approaches for the EXPRESS-to-OWL conversion, such as [2,[START_REF] Barbau | OntoSTEP: Enriching product model data using ontologies[END_REF][START_REF] Schevers | Converting the industry foundation classes to the web ontology language[END_REF][START_REF] Terkaj | Virtual factory data model[END_REF]. Among all approaches, Pauwels and Terkaj [START_REF] Pauwels | EXPRESS to OWL for construction industry: Towards a recommendable and usable ifcOWL ontology[END_REF] propose a recommendable and usable ifcOWL ontology.

BOT -

The Building Topology Ontology (BOT) [START_REF]Linked Building Data (LBD) W3C Community Group[END_REF] is a modular ontology covering core concepts of the building and three methods for extending it with domain specific ontologies (such as ontologies modeling information related to geographical location, sensor data, domotics, and construction data). The main purpose is to reduce redundancies caused by overlapping data found in these ontologies, which is currently violating the W3C best practice rules. Further, this approach helps the development of distributed ontologies, which have several advantages over a onesize fits all, such as interoperability and simple reuse. The BOT ontology comprises 7 key Classes (Building, Element, Interface, Site, Space, Storey, and Zone), 14 Object properties linking them and 1 Data Property linking a Zone or an Element to an IRI that identifies its 3D Model. As for the linking methods, it comes down to (i) extending the core classes with subclasses of more specialized ontologies, (ii) defining equivalent classes, and (iii) establishing typed links between instances [START_REF] Rasmussen | Proposing a central aec ontology that allows for domain specific extensions[END_REF].

PRODUCT -

The Building Product Ontology (PRODUCT) [START_REF]Linked Building Data (LBD) W3C Community Group. Building Product Ontology[END_REF] is another modular ontology developed along the same lines as BOT [START_REF]Linked Building Data (LBD) W3C Community Group[END_REF], however for describing a product i.e., an article or substance that is manufactured or refined for sale. It encapsulates one main Class (Product) and one main Object Property (Aggregates). The latter represents a simple decomposition tree of products. A custom product of more specific ontologies can be specified using this class and object property (such as the Element Class of the BOT ontology being a subClass of Product).

OPM -

The Ontology for Property Management (OPM) [START_REF]Linked Building Data (LBD) W3C Community Group[END_REF] is an ontology for describing temporal properties that are subject to changes as the building design evolves. It is made of 8 Classes, 5 Object Properties, and 2 Data Properties modeling the building properties and their states along the project life-cycle.

Other Models

Although there exist many other non ontology-based models, we present two wellknown of them here: the first one in the international market and the second one in France:

Newforma Data Model -The Newforma company provides proprietary solutions for PIM [START_REF]Newforma Project Center Twelfth Edition[END_REF] mainly dedicated for architects, engineers, contractors and owners. The main purpose of these solutions is to index all information in a given project including unstructured data (such as emails, construction related files of different formats). This helps the different users in organizing and managing project data, and collaborating with the different team members. The underlying data model represents general metadata information, syntactic words extracted from the content of the documents, and keywords manually added by the users.

Kroqi Data Model -Kroqi [START_REF]Kroqi Platform[END_REF] is a free collaborative platform developed by the Scientific and Technical Center Building 16 (CSTB) as a tool for the digital transition in the AEC industry to reduce the gap of the application of the BIM in large enterprises and small to medium-sized enterprises. Kroqi comprises several modules including the document management module. The latter provides several features, yet we focus on the document representation feature. The underlying data model represents general metadata describing the documents and tags manually added by the users.

Discussion

We evaluated the existing standards and models based on the challenges previously mentioned in Sect. 2.1. The results are depicted in Table 2.1 where we used the following symbols to evaluate a given standard or data model: " " to express an exhaustive coverage, "" to express a lack of coverage, and "Partial" to express a partial coverage.

• Inter/Intra-document Relations (Challenge 1.1): In general, there is no existing standard or data model capable of fully representing inter and intra-document relations. For instance, TEI [START_REF]The Text Encoding Initiative Consortium[END_REF] provides re f element which is only limited to cross references. DC [START_REF]Dublin Core Metadata Initiative[END_REF] comprises a set of relation qualifiers (e.g., References, isVersionOf ) between different resources but excludes intra-document relations (e.g., reference between parts of the same resource) and more complex inter-document relations (e.g., a spatial relation where a specific part of a resource is contained in another resource). Mpeg-7 [START_REF]The Moving Picture Experts Group[END_REF] and multimedia ontologies (e.g., [4,[START_REF] Garcia | Semantic integration and retrieval of multimedia metadata[END_REF][START_REF] Saathoff | Unlocking the semantics of multimedia presentations in the web with the multimedia metadata ontology[END_REF]102]) provide capabilities of linking only audiovisual descriptors (e.g., a semantic relation between two objects of a video segment).

The LINDO data model [START_REF] Brut | Integrating Heterogeneous Metadata into a Distributed Multimedia Information System[END_REF] covers only spatio-temporal relations between objects describing the content of multimedia elements (e.g., Localization relation). The Ontotext knowledge-based graph [START_REF]Ontotext Platform[END_REF] is capable of describing relations between entities representing the content of the same or different web pages (e.g., a semantic relation between an entity representing an organization and another entity representing its CEO). Yet, this is still limited as it does not cover other relations (e.g., references between structural metadata). This is also the case with building-oriented standards (e.g., [START_REF]buildingSMART. IFC-Industry Foundation Classes, IFC Releases[END_REF][START_REF]Construction Operations Building Information Exchange (COBIE)[END_REF][START_REF]Green Building XML[END_REF]) and ontologies (e.g., [START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group. Building Product Ontology[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group[END_REF]) which only describe relations between entities representing the building data. The other building-oriented data models (e.g., [START_REF]Newforma Project Center Twelfth Edition[END_REF][START_REF]Kroqi Platform[END_REF]) provide inter-document relations limited to document versions since they are conceptually designed towards document management solutions. there is no existing standard or data model providing a full coverage of the three aspects combined. Although, the majority fully cover generic metadata, they do partially cover content description and often lack structural metadata description. For instance, EXIF [START_REF]Exchangeable Image File Format for digital still cameras[END_REF] and DC [START_REF]Dublin Core Metadata Initiative[END_REF] are limited to metadata information describing a document as a whole entity. The XCDF data model [START_REF] Bloechle | XCDF: a canonical and structured document format[END_REF] only focuses on structural metadata describing the decomposition of PDF documents. TEI [START_REF]The Text Encoding Initiative Consortium[END_REF] provides a rich set of content description and structural metadata focusing however on the text. On the other hand, the multimedia-based standards (e.g., [START_REF]The Moving Picture Experts Group[END_REF]), ontology and knowledge-based models (e.g., [4,[START_REF] Garcia | Semantic integration and retrieval of multimedia metadata[END_REF][START_REF]Ontotext Platform[END_REF][START_REF] Saathoff | Unlocking the semantics of multimedia presentations in the web with the multimedia metadata ontology[END_REF]102]), and other data models (e.g., [START_REF] Brut | Integrating Heterogeneous Metadata into a Distributed Multimedia Information System[END_REF]) provide a wider coverage for the description of multimedia content while failing in the description of structural metadata, especially those related to the text (e.g., the page encapsulating a given textual content). The building-oriented standards (e.g., [START_REF]buildingSMART. IFC-Industry Foundation Classes, IFC Releases[END_REF][START_REF]Construction Operations Building Information Exchange (COBIE)[END_REF][START_REF]Green Building XML[END_REF]) and ontologies (e.g., [START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group. Building Product Ontology[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group[END_REF]) are not conceived to describe metadata information. The data models behind building-oriented document management solutions (e.g., [START_REF]Newforma Project Center Twelfth Edition[END_REF][START_REF]Kroqi Platform[END_REF]) are however capable of representing general metadata on the documents. Although they do not provide metadata on the content of the documents, they allow the users to add tags that might describe the content of the documents.

• Advanced Semantics (Challenge 1.3): As for the advanced semantics capabilities, none of the existing standards or data models contains the required knowledge for their associated systems to reason over the documents together with their content and their structure. However, the ontology and knowledge based models, whether representing multimedia document's content (e.g., [4,[START_REF] Garcia | Semantic integration and retrieval of multimedia metadata[END_REF][START_REF]Ontotext Platform[END_REF][START_REF] Saathoff | Unlocking the semantics of multimedia presentations in the web with the multimedia metadata ontology[END_REF]102]) or the building data (e.g., [START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group. Building Product Ontology[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group[END_REF][START_REF]Linked Building Data (LBD) W3C Community Group[END_REF]), partially integrate some advanced semantic aspects which are limited to reasoning capabilities over the content of the documents.

• Multimodality (Challenge 1.4): The multimodality aspect is covered by few of the existing standards and data models. For instance, DC [START_REF]Dublin Core Metadata Initiative[END_REF] is capable of describing any resource regardless of its type, its media content types, and its serialization technology. The LINDO data model represents multimedia contents that could be encoded in different formats. The building-oriented data models behind the industrial solutions for document management (e.g., [START_REF]Newforma Project Center Twelfth Edition[END_REF][START_REF]Kroqi Platform[END_REF]) are basically designed to support multimodality (e.g., emails, textual documents of different formats, images of different formats, 3D files). The multimedia standards, ontologies and knowledge-based models [4,[START_REF] Garcia | Semantic integration and retrieval of multimedia metadata[END_REF][START_REF]Ontotext Platform[END_REF][START_REF] Saathoff | Unlocking the semantics of multimedia presentations in the web with the multimedia metadata ontology[END_REF][START_REF]The Moving Picture Experts Group[END_REF], 102] naturally handle multimedia contents, yet they partially cover the multimodality aspect since they are not further capable of representing textual documents in their different forms of serialization.

• Extensibility (challenge 1.5): We consider all the reviewed standards and data Document Corpus models as partially extensible i.e., the evolution of the model itself is possible, yet following narrow directions. For instance, the nature of XML-based standards and data models makes them partially extensible as they could support new elements, however describing their particular coverage. For instance, defining new DSs in Mpeg-7 [START_REF]The Moving Picture Experts Group[END_REF] is possible, which makes it somehow extensible. Yet, defining more complex elements such as inter and intra-document links is still out of its scope.

To sum up, to our knowledge, there is currently no available standard or data model that addresses all the challenges and answers all the requirements we described in Sect. 2.1. A naive solution could be the simple combination of the most convenient standards or data models for document representation, and the building data. However, this yields interoperability issues since they rely on different semantics, structures and formats. We believe that ontologies provide a reliable and efficient means to resolve these problems and support SIR from heterogeneous data [START_REF] Guo | SOR: An optimized semantic ontology retrieval algorithm for heterogeneous multimedia big data[END_REF]. Thus, a more complex solution could be also the alignment of the best suited ontologies. Nonetheless, this would partially solve their current limitations as shown in Table 2.1. Furthermore, we aim to provide a modular and generic solution that is able to adapt to particularities of a given domain (e.g., the AEC industry) but also interpolated in any other domain.

Tight Coupling Proposal

Our aim is to provide powerful knowledge representation capabilities over the various data embedded in a heterogeneous document corpus for it to be exploited by any domain-oriented IR application. In this context, we need to leverage semantics on two different dimensions of a document corpus: the content of the documents which depends on the knowledge of a given application domain, and the structural and metadata aspect describing the documents regardless of that domain. Once semantics are handled, we propose to couple domain-specific information with its related structural and metadata information.

Overview

We provide a modular and generic semantic approach capable of generating a semantic network called tightly coupled semantic graph describing the collective knowledge of any heterogeneous document corpus in view of the limitations of current standards and data models (See Sect. 2.2). Our proposal is depicted in Fig. 2.2, which corresponds to the Indexing and Knowledge Representation layers of FEED2SEARCH (See Sect. 1.4.1).

In our approach, we rely on ontologies since they are proven powerful in: (1) conceptualizing heterogeneous data, (2) handling interoperability between vocabularies We propose LinkedMDR, a novel multi-layered ontology, which provides the infrastructure for the generation of a tightly coupled semantic graph. A middleware is required in order to benefit from the different existing services, technologies and APIs, and translate the generated information into LinkedMDR instances, thus progressively building the semantic network. Given a heterogeneous document corpus, one or multiple external domain-specific ontologies describing the knowledge embedded in the corpus, LinkedMDR ontology and its dedicated middleware:

• Structural and metadata instances describing the heterogeneous document corpus are generated using concepts, relations, and semantic rules of the core layer and the standardized metadata layer of LinkedMDR,

• Domain-specific instances describing the content of the heterogeneous document corpus are generated using concepts, relations, and semantic rules provided by the core layer of LinkedMDR and its pluggable domain-specific layer which adapts to the external domain-specific ontology(ies),

• Relations between instances describing the structural and metadata information and those describing the domain-specific information are generated using the semantic knowledge provided by the core layer of LinkedMDR and a tight coupling algorithm.

In the following, Sect.

LinkedMDR Ontology

We introduce a novel multi-layered ontology, entitled LinkedMDR17 [START_REF] Charbel | LinkedMDR: A Collective Knowledge Representation of a Heterogeneous Document Corpus[END_REF], which stands for Linked Multimedia Document Representation. The main purpose of this ontology is to provide the infrastructure to model the knowledge embedded in a heterogeneous document corpus of any domain-specific application. LinkedMDR is made of three main layers:

• The Core layer serving as a backbone and a mediator among the different layers,

• The Standardized Metadata layer linking descriptors of existing standards in metadata representation,

• The Pluggable Domain-Specific layer that can adapt to any domain-specific ontology.

The multi-layering of LinkedMDR ensures its genericity and extensibility. Fig. 2.3

shows an example of LinkedMDR application to the AEC industry using DC [START_REF]Dublin Core Metadata Initiative[END_REF],

TEI [START_REF]The Text Encoding Initiative Consortium[END_REF] and MPEG-7 [START_REF]The Moving Picture Experts Group[END_REF] standards for the standardized metadata layer and the ifcOWL [START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF] ontology for the pluggable domain-specific layer.

Core Layer

This layer introduces new concepts and relations that are either not covered or partially covered by existing standards (See Sect. 2.2), mainly:

• Concepts that model the global composition of a given document and the metadata properties associated to it (i.e., Document, Media, MediaComponent and DocumentProperty). This allows the description of various characteristics on the different granularity levels of the document.

• Object concept which abstracts Document, Media and MediaComponent so as to define on the Object common characteristics that are then inherited by these concepts.

• A rich set of relations associated to the Object concept (i.e., Part-Whole, Semantic, Temporal and Spatial relations) to define possible relations between any two document components (i.e., Document, Media and MediaComponent) or associated to a specific document component (e.g., Order and Syntactic relations associated to the Media concept) to define particular relations between two entities of the same type.

• Concepts that are equivalent to other concepts of adjacent layers (e.g., Text and Image describe respectively text and image concepts of the standardized metadata layer). This ensures easy and flexible alignments with similar concepts of adjacent layers that are defined by external standards or data models. is available online at: http://spider.sigappfr.org/linkedmdr/documentation/.

Standardized Metadata Layer

This layer builds upon metadata information defined by existing standards. Its main purpose is to reuse the most convenient descriptors representing general metadata of the document, textual and multimedia components while adapting them to the previously mentioned requirements (See Sect. 2.1). As an example, we select DC [START_REF]Dublin Core Metadata Initiative[END_REF],

TEI [START_REF]The Text Encoding Initiative Consortium[END_REF] and MPEG-7 [START_REF]The Moving Picture Experts Group[END_REF] as they present a rich set of descriptors satisfying the three above categories of descriptors respectively (See Sect. 2.2). Although our selection is rational, we still leave flexibility for the use of other standards in the future for later upcoming versions of LinkedMDR.

Pluggable Domain-specific Layer

The previous layers model a heterogeneous document corpus independently of the content of the documents. We introduce the domain-specific layer as a pluggable layer to make the same version of LinkedMDR ontology easily adaptable to any domain-specific knowledge.

In the core layer, the concept Domain is linked to the concept Object through the relation isRelatedTo (See Fig. A detailed example on LinkedMDR instances considering its different layers is explained in Sect. 2.3.3.2 and shown in Fig. 2.11.

Tightly Coupled Semantic Graph

In the literature, a tightly coupled semantic graph is known as a semantic graph where data objects (e.g., documents, web pages, tuples, etc.) and their metadata are individuals coupled with those of a lexical knowledge base or domain-specific ontology. Existing works [START_REF] Chbeir | SemIndex: Semantic-aware inverted index[END_REF][START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF] have adopted tightly coupled semantic graphs in their approaches, yet on different data types. For instance, Rocha et al. [START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF] coupled web pages and a domain-specific ontology.

Underlying External Resources

We define a heterogeneous document corpus and a domain-specific ontology as follows: • C is the set of domain-specific concepts.

For instance, considering a simplified version 19 of the ifcOWL [START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF] from the AEC industry. The ontology concepts include, but are not limited to, entities describing the building (IfcBuilding), the building elements (e.g., IfcWindow, IfcDoor, IfcCurtainWall) and the building systems (e.g., IfcSystem describing Plumbing, Cooling, Fenestration, etc.).

• R ⊆ C × C is the set of relations between domain-specific concepts in C.

For instance, r 1 =(IfcBuilding, IfcCurtainWall) is a spatial containment relation between IfcBuilding and IfcCurtainWall, where r 1 ∈ R.

• Lit = {Integer, Decimal, String, ...} is the set of literal types.

• A ⊆ C × Lit is the set of attributes describing domain-specific concepts i.e., relations between domain-specific concepts in C and literals in Lit.

For instance, WindowHeight is an attribute property linking IfcWindow to Decimal.

• L is the set of relation labels.

• f L : R → L is the association function that assigns a label l ∈ L to a domainspecific relation r ∈ R, hence f L (r) = l.

For instance, f L (r 1 ) = "contains".

For ease of presentation, O D (C, R, Lit, A, L, f L ) will be referred to as O D in the remainder of the thesis report.

Tightly Coupled Semantic Graph Model

We formally define a tightly coupled semantic graph as follows (Examples are based on Fig. 2.11 which resumes our motivating scenario depicted in Fig. 2.1): Definition 3 (Tightly Coupled Semantic Graph). Given a heterogeneous document corpus δ and a domain-specific ontology O D , we define a tightly coupled semantic graph G δ (V, E, Val, f Val , Lab, f Lab , W, f W v , f W e ) as the instances graph describing the structural and domain-specific knowledge in δ following the infrastructure provided by LinkedMDR, where:

• V is the set of nodes representing instances of O D and δ:

-V = V d ∪ V s . -V d ⊂ V is the subset of domain-specific nodes where a node v d ∈ V d represents an instance of c ∈ C in O D .
For instance, curtainwall4 is an instance of the concept IfcCurtainWall.

-V s ⊂ V is the subset of structural-based nodes where a node v s ∈ V s represents any granularity element in δ i.e., a document d i ∈ δ (e.g., d4), a metadata meta k ∈ d i (e.g., d4.title), a media med l ∈ d i (e.g., d4.imagegraphic1 or more precisely a media component medComp r ∈ med l (e.g., d4.stillregion1).

-

V d ∩ V s = ∅ 20 .
• E is the set of directed edges representing relations between nodes: For instance, e d 1 = (bldg1,curtainwall4) is the spatial containment edge linking bldg1 to curtainwall4.

-E = E d ∪ E s ∪ E h . -E d ⊆ V d × V d is
-E s ⊆ V s × V s is the subset of structural-based edges where an edge e s ∈ E s represents a relation between structural-based nodes in V s (such as partwhole, reference, etc.), thus augmenting the representation of δ.

For instance, e s 1 = (d1, d1.div7) is the edge linking d1 to d1.div7.

-E h ⊆ V d × V s is the subset of hybrid edges where an edge e h ∈ E h represents a tight coupling, i.e. a relation between a node v d ∈ V d and a node v s ∈ V s .

For instance, e h 1 = (curtainwall4, d1.div7) is the edge linking curtainwall4 to d1.div7.

-E d ∩ E s = ∅, E s ∩ E h = ∅, E d ∩ E h = ∅, and E d ∩ E s ∩ E h = ∅ 21 .
• Val is the set of node literal values.

The value of a node v s ∈ V s is made of its content.

For instance, "Exterior Facades" is the string value associated to d1.div7.

The value of a node v d ∈ V d consists of its concatenated attributes A v d ⊆ A in O D together with their associated values.

For instance, "label:facteur solaire; hasValue:0.64" is the value associated to solar-factor4 obtained from the concatenation of its attributes label, hasValue and their values "solar factor" and 0.64 respectively.

For the sake of simplicity, we omit these values from the graph depicted in Fig. 2.11.

Chapter 2. Towards a Collective Knowledge Representation of a Heterogeneous Document Corpus

• f Val : V → Val is the association function that assigns to a node v ∈ V a literal value val ∈ Val, hence f Val (v) = val.
For instance, f Val (d1.div7) = "Exterior Facades".

• Lab is the set of edge labels.

• f Lab : E → Lab is the association function that assigns a label lab ∈ Lab to an edge e ∈ E, hence f Lab (e) = lab.

For instance, f Lab (e s 1 ) = "hasPart".

• W is the set of both nodes and edges' weights.

• f W v : V → W is the node weight mapping that assigns a weight w v ∈ W to a node v ∈ V.

• f W e : E → W is the edge weight mapping that assigns a weight w e ∈ W to an edge e ∈ E. f W e consists of a set of functions which adapt to E s , E d , and E h depending on different heuristics.

The weight mapping functions f W v and f W e are used in the search process of query answers. They are detailed in Chapter 3, Sect. 3.4.4.

For ease of presentation, G δ (V, E, Val, f Val , Lab, f Lab , W, f W v , f W e ) will be referred to as G δ in the remainder of the thesis report. • d 4 contains several technical drawings, each related to a specific building floor and described on different pages of the document. As mentioned in Sect. 2.2, existing multimedia standard such as MPGE-7 [START_REF]The Moving Picture Experts Group[END_REF] help in describing the different regions of the drawings but without any information on the corresponding pages. Existing text encoding standards such as TEI [START_REF]The Text Encoding Initiative Consortium[END_REF] can represent the pages of each drawing but without structural description of their content.

Existing general metadata standards such as DC [START_REF]Dublin Core Metadata Initiative[END_REF] can provide us with descriptors on d 4 as a whole entity without details on pages and content of each 22 Details will be provided in Chapter 4. drawing. Through LinkedMDR core concepts and relations, it is possible to take advantage of the specialized features of existing standards while associating them to provide extended capabilities. For instance, this is represented by the structural-based node d4.imagegraphic1 and its associated structural-based edges: e s 4 = (d4.imagegraphic1,d4.stillregion1) with f Lab (e s 4 ) = "hasPart", and e s 5 = (d4.imagegraphic1,d4.page1) with f Lab (e s 5 ) = "isOn". This particularly addresses Challenge 1.2.

• • All the nodes in the graph (See Fig. 2.11) are instances of LinkedMDR ontology: d4 is an instance of the core concept Document of LinkedMDR; d4.stillregion1 is an instance of the standardized metadata concept mpeg7:StillRegion of Linked-MDR which is provided by the MPEG-7 standard [START_REF]The Moving Picture Experts Group[END_REF]; curtainwall4 is an instance of the domain-specific concept ifc:IfcCurtainWall of LinkedMDR which is provided by the ifcOWL [START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF]. The nature of the ontology makes the graph easily extensible at different levels (new concepts, new relations, new instances).

As an example, at the instances level, consider an external knowledge-base or ontology describing an audio file d 7 which represents the noise before and after the cladding of a given facade: ext:facade. If the latter finally corresponds to curtainwall4, then the two resources are linked through owl:sameAs relation.

This particularly addresses Challenge 1.5.

Tight Coupling Algorithm

In this section, we present the pseudo code of our proposed tight coupling algorithm report"). This outputs standard metadata output (Output s ) mostly from existing standards for document's structure and content description (See Sect. 2.2), thus describing the structural part of the corpus δ.

• Step 2 (lines 3-4): It converts the generated descriptors into structural-based instances of LinkedMDR by using well-defined converters offered at the middleware indexing level. These converters are based on the semantics provided by the core layer and the standardized metadata layer of LinkedMDR i.e., their concepts, relations and semantic rules. This generates nodes V s and E s which are then added to the empty graph G δ (e.g., structural-based nodes d1.div7.p1 and d2.div2, and edge e s 6 = (d1.div7.p1, d2.div2) with f Lab (e s 6 ) = "references", See Fig. 2.12).

• Step 3 (lines 5-7): It looks over the direct textual content of each generated structural-based node v s ∈ V s ( f Val (v s )), as long as it is not empty, to identify one or multiple domain-specific instances v d ∈ V d based on the knowledge embedded in a domain-specific ontology O D . This is done using existing automatic techniques for semantic annotation [START_REF] Cunningham | A framework and graphical development environment for robust NLP tools and applications[END_REF] offered at the middleware indexing level. The rationale behind is to benefit from the automatic extraction of domain-specific information from textual content at different levels of where it was identified in the previous step. The rationale behind is that if a node v d where identified from the direct textual content of a node v s then it is directly related to it as it provides domain-specific information describing its content.

Summary

This chapter tackles the problem of representing a heterogeneous document corpus for it to be exploited in a SIR system, which has been partially solved by existing standards and data models for document representation. We introduce a novel semantic-based approach, which we call Tight Coupling approach, aiming to represent the collective knowledge embedded in a given heterogeneous document corpus through a tightly coupled semantic graph. The contributions of this chapter come down to: (i) defining a backbone multi-layered ontology, entitled LinkedMDR [START_REF] Charbel | LinkedMDR: A Collective Knowledge Representation of a Heterogeneous Document Corpus[END_REF],

which provides the required infrastructure to build this graph through core components easily connected to existing standardized metadata standards and easily pluggable to domain-specific ontologies, (ii) defining the tightly coupled semantic graph that embeds instances of LinkedMDR where semantics are associated to structural components of the documents as well as their domain-specific information, and (iii) defining the tight coupling algorithm which generates this graph. To our knowledge, this supports the first graph capable of: (1) representing the various inter and intradocument links within the corpus, (ii) describing general metadata information of the document, its content, and its structural text and multimedia components all combined, (iii) associating semantics at content and structural levels of the document, (iv) handling document multimodality, and (v) ensuring extensibility. Also, the proposed approach is modular and generic, so it does cope with any domainspecific application. Several experiments were conducted to validate our proposal within real-world applications. These experiments are dedicated to Chapter 4.

Introduction

Web and Information Systems are increasingly adopting semantic knowledge-based models to represent the data encapsulated in heterogeneous resources [START_REF] Beetz | Open-ease[END_REF][START_REF] Huang | Sharing knowledge in a supply chain using the semantic web[END_REF]. This has several proven benefits in improving users' experience in search applications [START_REF] Mangold | A survey and classification of semantic search approaches[END_REF].

As mentioned in Chapter 1 and Chapter 2, in several industries involving multidisciplinary projects, users looking for a particular information have to search through interdependent heterogeneous documents provided by different sources. Consider that the user is searching for "the solar factor of windows". Several challenges arise in order to provide him with relevant query answers, that do not only include documents, but also more refined and enriched information (See Fig. 3.2):

• Challenge 2.1: Providing relevant granularity levels of the documents -Relevant granularity levels of the documents associated to domain-specific information searched by the user (e.g., section 2 of document d2 entitled "2) Characteristics of Window Frames" describing the value of the solar factor i.e., "SW = 0.45") help the user in locating desired information from possibly large documents involving other irrelevant sections.

• Challenge 2.2: Providing relevant inter and intra-document dependencies -Relevant dependencies between documents and parts of documents (e.g., sub-section 2 of document d 1 entitled "Thermal Properties" that involves additional information regarding the solar factor coefficients described in section 2 of document d 2 )

enrich the search results as they provide further related and useful information across the documents.

• Challenge 2. Traditional IR approaches mainly rely on syntactic keyword-based search [START_REF] Manning | Introduction to information retrieval[END_REF].

To overcome their limitations, there has been significant interest in taking semantics into account leading to the emergence of SIR systems. Although suitable for several applications [START_REF] Chechev | The patents retrieval prototype in the MOLTO project[END_REF][START_REF] Fernández | Semantically enhanced information retrieval: An ontology-based approach[END_REF][START_REF] Giunchiglia | Concept search[END_REF][START_REF] Kiryakov | Semantic annotation, indexing, and retrieval[END_REF][START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF][START_REF] Tekli | Full-fledged semantic indexing and querying model designed for seamless integration in legacy RDBMS[END_REF][START_REF] Zhong | Conceptual graph matching for semantic search[END_REF], SIR systems provide documents as query answers without considering in their search results (i) detailed information regarding relevant granularity levels of the documents, (ii) various inter and intra-document dependencies, and (iii) relevant contextual information. To the best of our knowledge, none of the existing approaches have tackled the above challenges combined.

In this chapter, we provide a solution to the aforementioned limitations of current SIR systems. As we already discussed, in Chapter 2, the importance of adopting The contributions of this chapter can be summarized by:

• Hybrid molecule, a novel data structure for query answers based on tightly coupled semantic graphs;

• HM Query processing, a comprehensive query processing pipeline over a heterogeneous document corpus, where the hybrid molecule structure intervenes in each of its stage;

• HM_CSA, an algorithm that constructs relevant hybrid molecules from a tightly coupled semantic graph;

• Weight mapping, a series of weighting functions that rank the hybrid-molecule query answers.

The remainder of this chapter is structured as follows. Sect.

Background and Related Work

In this section we first present some basic background concepts in IR and the classical pipeline behind (Sect. 3.2.1). Then, we provide a literature overview on the different existing IR models and the existing approaches which implement them.

We mainly distinguish between 2 categories of IR models: traditional IR models (Sect. 3.2.2) and SIR models (Sect. 3.2.3). We then focus on the approaches based on SIR models since we rely on semantic graphs and ontologies as mentioned in Chapter 2.

Information Retrieval (IR)

IR is a wide area in Information Science and has been subject to many research works for decades. The main purpose of an IR system is to search and retrieve relevant resources from a collection of resources in order to satisfy user needs expressed as queries. IR systems have significantly evolved especially with the emergence of web search engines [START_REF] Aggarwal | Information Retrieval and Search Engines[END_REF]. However, IR systems mainly rely on the following classical pipeline2 for query processing (See Fig. 3.3):

• Query Interpretation: the system understands the user's query expressed in terms of keywords, natural language text or visual components and translates it into its internal knowledge structure. This sometimes includes preprocessing techniques (such as NLP for natural language text) and query modification (such as query re-writing) to increase both precision and recall [START_REF] Mangold | A survey and classification of semantic search approaches[END_REF].

• Search: the system searches for relevant information that matches the user's query from the descriptors (e.g., inverted lists, graphs, etc.) representing the collection of resources. The system retrieves partially or exactly matched resources depending on the underlying adopted IR model.

• Ranking: the retrieved resources are given scores according to their degree of relevance w.r.t. the user's query. The system ranks the results based on these scores.

• Presentation: the system presents the results to the user in response to their original query, within a Graphical User Interface (GUI), in an understandable format (such as SERP).

Traditional IR models

The classical IR approaches are based on Boolean, Vector Space and Probabilistic models [START_REF] Belkin | Information filtering and information retrieval: Two sides of the same coin?[END_REF][START_REF] Manning | Introduction to information retrieval[END_REF]. They mainly focus on the bag of words theory to represent the documents without considering the context of these words nor the different meanings that might be associated to them [START_REF] Kara | An ontology-based retrieval system using semantic indexing[END_REF]. In the following, we first describe these models, then we provide examples of existing works and systems built on them.

The Boolean retrieval is based on the exact match between the set of terms representing the documents and the Boolean expressions3 representing the query [START_REF] Heaps | Information retrieval, computational and theoretical aspects[END_REF].

In this model, there is no distinction between the retrieved documents i.e., there is no form of relevance ranking. The Vector Space and Probabilistic retrieval are both based on the best match theory. The Vector Space Model (VSM) [START_REF] Salton | Introduction to modern information retrieval[END_REF] is the most widely used retrieval model. It consists of representing the documents and the query by means of vectors of weighted terms, where weights are calculated using statistical distributions such as the term frequency-inverse document frequency (tf-idf) function. Document vectors are then compared to the query vector and assigned scores representing the relevance of documents to the query. The Probabilistic retrieval model [START_REF] Fuhr | Probabilistic models in information retrieval[END_REF] is based on the Probability Ranking Principle (PRP). The documents are ranked based on the probability of relevance of their texts to the query. This probability is often calculated using the Bayes Theorem 4 .

Despite the drawbacks of the Boolean retrieval model, the latter was the main adopted retrieval model for decades until the arrival of the WWW. At this point, extended models have started to emerge. For instance, Salton et al. [START_REF] Salton | Extended Boolean information retrieval[END_REF] propose an extended Boolean model to overcome the drawbacks of the standard Boolean model and improve the effectiveness of the search results. They combine the characteristics of VSM with the properties of Boolean algebra and calculate the similarity between queries and documents, so as to cover partial matching and ranking. Turtle and Croft [START_REF] Turtle | Inference networks for document retrieval[END_REF] are the first to introduce the use of Bayesian networks in IR to represent probabilistic dependencies between a document and a user query by means of a directed and sophisticated graph. They combine Boolean, statistical and probabilistic models. Haines and Croft [START_REF] Haines | Relevance feedback and inference networks[END_REF] extend this work by including relevance feedback techniques. This aims to improve retrieval performance for a particular query by modifying the query based on the user's reaction to the initial retrieved documents.

In other words, the user judges the relevance or non-relevance of some of the documents retrieved and this feedback is later used to add new terms to the query and to re-weight query terms.

On the other side, there exist many open-source systems, toolkits and platforms on the basis of the aforementioned IR models. Lucene 5 is an IR system which uses a combination of the Boolean model and VSM. It essentially remains a VSM-based system, but uses the Boolean model to first narrow down the documents that need to be scored based on the use of Boolean logic in the Query specification [START_REF] Gospodnetic | [END_REF]. It also adds some capabilities and refinements onto this model to support Boolean and fuzzy searching. Lemur6 is an IR toolkit that is based on VSM and Probabilistic IR models. It offers several retrieval algorithms including simple tf-idf, Okapi (BM25)

weighting scheme, and Kullback-Leibler (KL) divergence measure [START_REF] Manning | Introduction to information retrieval[END_REF]. Terrier7 is a modular platform for the rapid development of large-scale IR applications. It supports several IR approaches such as Divergence From Randomness, BM25F, and dependence proximity. It also offers supervised ranking models via Learning to Rank techniques [START_REF] Ounis | Terrier information retrieval platform[END_REF].

SIR models

Many research works have attested the benefits of incorporating knowledge bases and domain specific ontologies in their index, query, and search process to overcome the semantic gap between keywords found in the documents and those in the user's query [START_REF] Giunchiglia | Concept search[END_REF]. The main advantage of such methods resides in maximizing the precision and recall w.r.t. to traditional IR [START_REF] Manning | Introduction to information retrieval[END_REF] where the search is limited to syntactic keyword matching. In the literature, there have been many attempts to provide classification categories for SIR models [START_REF] Mangold | A survey and classification of semantic search approaches[END_REF][START_REF] Ramkumar | Ontology Based Semantic Search: An Introduction and a Survey of Current Approaches[END_REF]. However, this depends on various and numerous criteria such as the annotation models behind, the semantic similarity measures used to compare concepts representing the queries with those representing the documents (e.g., Resnik [START_REF] Resnik | Using information content to evaluate semantic similarity in a taxonomy[END_REF], Wu and Palmer [START_REF] Wu | Verbs semantics and lexical selection[END_REF]). In general, independently of the classification criteria used, SIR models rely on the conceptual search which replaces the bag of words paradigm with the bag of concepts outstripping traditional IR models [START_REF] Buscaldi | Using the semantics of texts for information retrieval: a concept-and domain relationbased approach[END_REF].

In this study, we focus on existing works which are based on SIR models. We distinguish between fully-fledged SIR approaches which consider semantics in their in-

Fully-fledged approaches

The first attempt towards injecting semantics in IR was the adoption of Word Sense Disambiguation (WSD) techniques [START_REF] Navigli | Word sense disambiguation: A survey[END_REF] where terms are associated to concepts of a thesaurus or knowledge base such as WordNet8 . Mihalcea and Moldovan [START_REF] Mihalcea | Semantic indexing using WordNet senses[END_REF] extend the Boolean IR model by adding word semantics to free-text index. Their approach expands both the queries and the indices with WordNet synsets provided by a semi-complete yet highly precise WSD algorithm. The disambiguation process is based on pairs formed by the word to disambiguate and neighboring words in its context and their co-occurrence in SemCor, a semantic tagged corpus. Tekli et al. [START_REF] Tekli | Full-fledged semantic indexing and querying model designed for seamless integration in legacy RDBMS[END_REF] build upon the idea of semantic aware search to target textual databases. They propose a general framework for modeling and processing semantic-aware querying.

They first generate a semantic-aware inverted index structure, called SemIndex. It consists of a tightly coupled inverted index graph that combines a semantic network (such as WordNet) and a standard inverted index on a collection of textual data.

They also provide a general keyword query model and query processing algorithms to retrieve semantic-aware results.

Kiryakov et al. [START_REF] Kiryakov | Semantic annotation, indexing, and retrieval[END_REF] introduce a semantic platform, called the Knowledge and Information Management (KIM), for information extraction and retrieval over large document collections. Their semantic annotation is based on the Named Entity (NE) recognition i.e., the process that assigns to the entities encountered in the text (web pages, non-web documents, text fields in databases, etc.) links to their semantic descriptions. This is done based on an underlying ontology, called KIM Ontology (KIMO). Documents are then retrieved on the basis of relevance to NEs instead of words. Chechev et al. [START_REF] Chechev | The patents retrieval prototype in the MOLTO project[END_REF] present a prototype for patent retrieval within the Molto project. It addresses high quality domain-specific machine translation on web documents covering up several languages. It is based on a semantic tagger that annotates the patent content using a patent-specific ontology. In the search results, the tool provides a list of classes from the ontology that match the query and a list of matching documents. It also offers links to access the semantically annotated documents and the original patents. In the text of annotated documents, there are highlights on the words that are related to any semantic item. Fernandez et al. [START_REF] Fernández | Semantically enhanced information retrieval: An ontology-based approach[END_REF] present an ontology-based IR model that takes advantage of domain knowledge bases to support semantic search capabilities over large document repositories such as the web environment. The proposed system takes a formal SPARQL query as input, then executes the query against a knowledge base. This returns a list of matching semantic entities that satisfy the query based on the exact match strategy. Finally, the documents that are annotated with the previously returned instances are retrieved, ranked, and presented to the user based on an approximate match strategy.

SPARQL-based approaches

RDF is the widely used data model for representing semantic graphs [START_REF]World Wide Web Consortium (W3C). Resource Description Framework[END_REF]. Simple Protocol And Rdf Query Language (SPARQL) [START_REF]SPARQL Query Language for RDF[END_REF] is the predominant language used to query and manipulate RDF graph content on the Web or in an RDF store.

Writing SPARQL queries is a tedious task and requires a technical knowledge in RDF and SPARQL syntax. It also requires full knowledge of the complex and heterogeneous structure of the linked data and handling its evolution over time. In order to enable users with limited technological skills to express their information retrieval requirements and query linked data, several works have been defined to support visual querying, query rewriting and refinement.

Haag et al. [START_REF] Haag | Visual SPARQL querying based on extended filter/flow graphs[END_REF] propose a visual querying framework for the formulation of SE-LECT and ASK SPARQL queries without any text input. In another work [START_REF] Haag | QueryVOWL: Visual composition of SPARQL queries[END_REF], they develop QueryVOWL, a visual query language which defines SPARQL mappings for graphical elements of the ontology visualization VOWL. Benedetti et al. [START_REF] Benedetti | Lodex: A tool for visual querying linked open data[END_REF] present LODeX, an interactive tool that provides the users with a summary view of the dataset structure and supports them in creating a visual query and refining it.

Soylu et al. [START_REF] Soylu | Ontology-based visual query formulation: An industry experience[END_REF] present an ontology-based visual query system, called OptiqueVQS, that helps domain experts users in formulating visual queries for industrial applications.

Frosini et al. [START_REF] Frosini | Flexible query processing for SPARQL[END_REF] propose a flexible query processing approach. It is based on query rewriting using query approximation and relaxation operators. These operators are defined within SPARQL AR , an extension of a fragment of SPARQL 1.1 9 .

Query Approximation consists of applying edit operators (such as deletion, insertion and substitution) to transform a well-defined regular expression into a new one.

Query Relaxation relies on a fragment of RDF Schema (RDFS) entailment rules. In general, Query Relaxation is one of the most used cooperative techniques that provide users with alternative answers instead of an empty result. Fokou et al. [START_REF] Fokou | Cooperative techniques for SPARQL query relaxation in RDF databases[END_REF] approach this problem by finding subqueries responsible of the failure, called Minimal Failing Subqueries (MFSs). These subqueries explain the empty returned result and guide the user to perform the relaxation process. They also compute a particular type of RDF queries, called Maximal Succeeding Subqueries (XSSs), which are subqueries with maximal number of triple patterns of the initial query. The rationale behind is that the set of triple patterns that are not in an XSS could be removed or made optional in the relaxed query. Angles et al. [3] propose an extension of SPARQL, called SPARQL Ext which preserves the original semantics while incorporating all known types of subqueries in a modular fashion: subqueries in FROM clauses, subqueries as graph patterns, and subqueries in filter constraints (Set membership, quantified and existential conditions).

Graph-based approaches

As previously mentioned, SIR systems rely on semantic graphs representing the data at hand. Thus, the retrieval process mainly consists of finding the relevant information from the graph substituting the data. Existing works have approached this problem differently. Some of them adopted subgraph matching approaches which consist of representing the data and the query through graphs G and Q respectively and then retrieve all sub-graphs of G that are similar to Q. Other approaches adopted graph exploration strategies where query terms are used to find relevant nodes in the graph. From these nodes, an algorithm traverses the graph to determine semantically related resources.

Zoo et al. [START_REF] Zou | gStore: a graph-based SPARQL query engine[END_REF] Among the many heuristic graph-based search methods [START_REF] Russell | Artificial intelligence: a modern approach[END_REF], Constrained Spread Activation (CSA) has been widely adopted in many IR applications where it proved its effectiveness [START_REF] Mangold | A survey and classification of semantic search approaches[END_REF][START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF]. It is based on a Breadth-First Search (BFS) graph traversal strategy. It works by spreading out the activation from a set of start nodes to adjacent nodes progressively in the graph until predefined constraints are met. Cohen and Kjeldsen [START_REF] Cohen | Information retrieval by constrained spreading activation in semantic networks[END_REF] use CSA algorithm to realize intelligent matches between user requirements and relevant agents in a Q&A application. Crestani et al. [START_REF] Crestani | Searching the web by constrained spreading activation[END_REF] were the first to apply CSA technique to the World Wide Web to retrieve information using an ostensive approach to querying similar to query-by-example. Griffith et al. [START_REF] Griffith | A constrained spreading activation approach to collaborative filtering[END_REF] propose to enhance users recommendations using a collaborative filtering approach. They rely on graph-based representation of the problem domain and a CSA approach.

Sun et al. [START_REF] Sun | A spreading activation algorithm of spatial big data retrieval based on the spatial ontology model[END_REF] combine CSA algorithm with a spatial ontology to improve results in associative retrieval of spatial big data. Gouws et al. [START_REF] Gouws | Measuring conceptual similarity by spreading activation over Wikipedia's hyperlink structure[END_REF] compute semantic relatedness by applying spread activation over the hyperlink structure of Wikipedia.

One of the most prominent uses of CSA is the one proposed by Rocha et al. [START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF]. It consists of the combination of CSA techniques with classical search techniques for searching in the semantic graph of a given domain. The semantic graph consists of a tight coupling between web pages and concepts of a domain-specific ontology. Their query execution follows a hybrid approach: given a user query, a classical search engine identifies a set of matched nodes; these nodes are then used as the start nodes of a spreading activation algorithm which subsequently activates possibly relevant neighbors in the graph. At the end of the algorithm, nodes (i.e., documents) with highest activation values are ranked the highest in the result set. 10 Flexible criteria added to the query.

Molecule-based approaches

Molecules, mainly RDF molecules, have received a wide attention over the past two decades within different applications.

It is first introduced by Ding et al. [START_REF] Ding | Tracking rdf graph provenance using rdf molecules[END_REF] for the purpose of tracking RDF prove-

Discussion

Both existing traditional IR and SIR approaches use the same classical pipeline for query processing, yet they rely on different models and techniques as shown in Tables 3.1 and 3.2. This proves that the underlying 4 stages of this pipeline are the basis of any IR system.

Ranking

No ranking (for Boolean Retrieval), Scores of vectors/terms (e.g., tf-idf, BM25)

Presentation

Resources matching the query (for Boolean Retrieval), Ranked list of resources relevant to the query

SIR Models & Techniques

Query Interpretation

Keywords, Concepts, Graphs Visual SPARQL, NLP, Semantic Annotation techniques, Query Modification (Disambiguation, Relaxation, etc.)

Search

Best Match (VSM, Probabilistic, Combined Retrieval) Graph traversal techniques (e.g., CSA), Sub-graph matching, etc.

Ranking

Scores of vectors/terms/concepts/nodes/graphs (e.g., using semantic similarity measures) Presentation

Ranked list of resources relevant to the query To recap, we summarize the adopted models and techniques w.r.t. each stage as follows:

• Query Interpretation: Traditional IR approaches basically use keywords, Boolean expressions and vector of terms to express user's queries [START_REF] Manning | Introduction to information retrieval[END_REF]. A major breakthrough has been achieved with SIR approaches using knowledge-based concepts and semantic graphs to represent these queries [START_REF] Mihalcea | Semantic indexing using WordNet senses[END_REF][START_REF] Zou | gStore: a graph-based SPARQL query engine[END_REF]. Further advanced techniques to query modification has been also considered by these approaches, such as query disambiguation [START_REF] Navigli | Word sense disambiguation: A survey[END_REF] and relaxation [START_REF] Fokou | Cooperative techniques for SPARQL query relaxation in RDF databases[END_REF][START_REF] Frosini | Flexible query processing for SPARQL[END_REF], which augment the query with further semantics. Traditional IR approaches [START_REF] Haines | Relevance feedback and inference networks[END_REF] adopt query modification techniques yet adding/removing/modifying syntactic terms following classical Relevance Feedback. It is to note that, the same models and techniques used for query interpretation are usually used for the document representation in the annotation phase.

• Search: Traditional IR approaches are based on either the exact match or the best match theory [START_REF] Belkin | Information filtering and information retrieval: Two sides of the same coin?[END_REF]. SIR approaches focus on the best match model to search for resources that are exactly or partially relevant to the user's queries. They also differ from traditional approaches in the way they search for these resources. As they are based on semantic graphs, new search techniques have been introduced such as the graph traversal [START_REF] Russell | Artificial intelligence: a modern approach[END_REF][START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF][START_REF] Sun | A spreading activation algorithm of spatial big data retrieval based on the spatial ontology model[END_REF] and the sub-graph matching [START_REF] Galkin | SJoin: A Semantic Join Operator to Integrate Heterogeneous RDF Graphs[END_REF][START_REF] Zhong | Conceptual graph matching for semantic search[END_REF].

• Ranking: Despite Boolean Retrieval models which do not consider scores for matched resources w.r.t. user's queries, traditional IR approaches mainly compute scores for terms and vectors of terms (e.g., using tf-idf, BM25, etc.). All these models are still used by SIR approaches, however the latter often apply them on concepts, nodes and graphs. SIR approaches also rely on semantic measures [START_REF] Resnik | Using information content to evaluate semantic similarity in a taxonomy[END_REF][START_REF] Wu | Verbs semantics and lexical selection[END_REF] to compute these scores. It is to note that Search and Ranking phases are often merged in one single phase.

• Presentation: IR approaches based on the Boolean Retrieval model do not provide a ranking for the set of retrieved resources. Other traditional IR approaches [START_REF] Haines | Relevance feedback and inference networks[END_REF][START_REF] Salton | Extended Boolean information retrieval[END_REF][START_REF] Turtle | Inference networks for document retrieval[END_REF] and SIR approaches [START_REF] Chechev | The patents retrieval prototype in the MOLTO project[END_REF][START_REF] Fernández | Semantically enhanced information retrieval: An ontology-based approach[END_REF][START_REF] Kiryakov | Semantic annotation, indexing, and retrieval[END_REF][START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF][START_REF] Tekli | Full-fledged semantic indexing and querying model designed for seamless integration in legacy RDBMS[END_REF] output, at the final stage of query processing, a ranked list of relevant resources. We evaluated the existing IR models based on the challenges previously mentioned in Sect. 3.1. The results are depicted in Table 3.3 where we used the same symbols as those for Sect. 2.2.3. In general, none of the existing IR models is able to answer the three challenges combined. In the following, we summarize their limitations w.r.t. each challenge:

• Relevant Granularity Levels (Challenge 2.1): Although current IR models (including traditional and SIR) search for relevant information contained in different granularity levels of the document structure, they do not provide in their results detailed information regarding the relevant parts of the documents. Instead, they provide the user with the whole documents that might contain many other irrelevant parts.

• Relevant Inter/Intra-Document Dependencies (Challenge 2.2): Traditional IR models do not consider the various dependencies between documents or parts of the documents. This is because their underlying models focus on the presentation of documents as bag of words neglecting the relations between them. SIR models only focus on the lexical dependencies or domain-oriented links between the documents, thus partially considering these dependencies in their results.

• Contextualized Query Answers (Challenge 2.3): Existing IR models do not provide both structural and domain-specific context for query answers. Although partial contextual information could be extracted from the content of the provided resources (e.g., snippet of the relevant content) or from knowledge-graph relations between them (e.g., resources involving related semantic concepts), they still neglect structural information (e.g., document cross references, granularity levels). Thus, their retrieval results do not help the user in easily locating straightforward information and interpretation the results, especially in tracking cross document dependencies.

To sum up, existing SIR works [START_REF] Fernández | Semantically enhanced information retrieval: An ontology-based approach[END_REF][START_REF] Frosini | Flexible query processing for SPARQL[END_REF][START_REF] Kiryakov | Semantic annotation, indexing, and retrieval[END_REF][START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF][START_REF] Sun | A spreading activation algorithm of spatial big data retrieval based on the spatial ontology model[END_REF][START_REF] Tekli | Full-fledged semantic indexing and querying model designed for seamless integration in legacy RDBMS[END_REF][START_REF] Zou | gStore: a graph-based SPARQL query engine[END_REF] are suitable for several applications, however there are still some limitations w.r.t. the challenges we aim to cope with. This does also concerns the data models behind which completely neglect the structural information of the documents (e.g., parts of the documents) and the various dependencies between these structural parts (e.g., references). We discussed in Chapter 2, the advantages of adopting a tightly coupled semantic graph to represent the collective knowledge of a heterogeneous document corpus within a SIR system supporting the structural and the domain-specific dimensions, and the coupling between them. Thus, we consider adopting such a model as a prerequisite to deal with the identified challenges. A naive solution would be to apply one of the existing approaches to SIR on our tightly coupled semantic graph considering the four stages of the classical pipeline for query processing. In this thesis, we will not focus on query interpretation since we consider that relying on existing techniques would be sufficient to tackle the main problem. For instance, natural language queries, NLP and semantic annotations techniques [START_REF] Cunningham | A framework and graphical development environment for robust NLP tools and applications[END_REF] are convenient in our context for non computer expert users (such as actors of the AEC industry).

Although visual SPARQL-based approaches [START_REF] Benedetti | Lodex: A tool for visual querying linked open data[END_REF][START_REF] Haag | QueryVOWL: Visual composition of SPARQL queries[END_REF][START_REF] Haag | Visual SPARQL querying based on extended filter/flow graphs[END_REF][START_REF] Soylu | Ontology-based visual query formulation: An industry experience[END_REF] and query re-writing or refinement techniques [3,[START_REF] Fokou | Cooperative techniques for SPARQL query relaxation in RDF databases[END_REF][START_REF] Frosini | Flexible query processing for SPARQL[END_REF] aim at improving queries for users with limited technological skills, however they still require a minimum level of knowledge on the background data model's structure or the SPARQL language respectively.

The main focus of this Chapter is to consider the identified challenges while searching, ranking and presenting relevant parts of our tightly coupled semantic graph w.r.t. a given user's query. Existing graph-based SIR approaches [START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF][START_REF] Zhong | Conceptual graph matching for semantic search[END_REF][START_REF] Sun | A spreading activation algorithm of spatial big data retrieval based on the spatial ontology model[END_REF] do not provide an exhaustive solution since their search and ranking strategy do not consider the different dimensions of our graph. Most importantly, the presentation of their results relies on node-based query answers representing the whole documents or web pages. Molecule-based SIR approaches [START_REF] Valle | A first step towards stream reasoning[END_REF][START_REF] Ding | Tracking rdf graph provenance using rdf molecules[END_REF][START_REF] Endris | MULDER: Querying the Linked Data Web by Bridging RDF Molecule Templates[END_REF][START_REF] Galkin | SJoin: A Semantic Join Operator to Integrate Heterogeneous RDF Graphs[END_REF][START_REF] Newman | A scale-out RDF molecule store for improved co-identification, querying and inferencing[END_REF] take the advantage of relying on well-defined sub-graphs, called RDF molecules, for different purposes (e.g., tracking RDF provenance, stream reasoning, federated SPARQL queries). Along the same lines, we consider that (i) defining a novel molecule data structure that copes with the characteristics of our tightly coupled semantic graph, and (ii) adapting current search, ranking and presentation strategies based on the proposed data structure would respond to the three identified challenges. In other words, this would generate meaningful augmented query answers that further consider the documents' structure and their inter and intra-document dependencies, and provide users with helpful information.

Hybrid Molecules for Tightly Coupled Semantic Graphs

In this section, we first give a brief overview on the motivations behind introducing a novel data structure, called hybrid molecule, on the basis of a tightly coupled semantic graph that we introduced in Chapter 2 (See Sect. 2.3.3). We then provide a formal definition for this structure. We also use Fig. 3.4, which depicts our motivating scenario (See Fig. 3.1) in a tightly coupled semantic graph, to provide illustrative examples.

Overview

As we explained in Chapter 2, given a heterogeneous document corpus, one important feature of its tightly coupled semantic graph stands in the coupling information between hybrid components i.e., the hybrid edges between components of the structural-based dimension and those of the domain-specific dimension. We consider that defining a novel data structure that emphasizes the hybrid information in a meaningful (well-defined) sub-graph is crucial for the use of this graph in several applications (such as clustering, query processing, etc.) and for the exploitation of interesting information.

In this study, we will focus on the use of this sub-graph structure for query processing. For instance, Fig.

Hybrid Molecules

In this section, we introduce hybrid molecules which we build upon the definitions of molecules in the literature [START_REF] Valle | A first step towards stream reasoning[END_REF][START_REF] Ding | Tracking rdf graph provenance using rdf molecules[END_REF][START_REF] Endris | MULDER: Querying the Linked Data Web by Bridging RDF Molecule Templates[END_REF][START_REF] Galkin | SJoin: A Semantic Join Operator to Integrate Heterogeneous RDF Graphs[END_REF][START_REF] Newman | A scale-out RDF molecule store for improved co-identification, querying and inferencing[END_REF], yet regardless of the serialization technology. Molecules are sub-graphs of connected nodes. They are extracted from an initial graph using a decomposition function. We propose adjusting the decomposition to better cope with our tightly coupled semantic graph (See Sect. 2.3.3) and provide meaningful sub-graphs i.e., sub-graphs encapsulating a core information and a meaningful context that covers both structural and domain-specific features of a given heterogeneous document corpus. We formally define a hybrid molecule as follows:

Definition 4 (Hybrid Molecule). Given the instances graph G δ describing a heterogeneous document corpus δ, we define a hybrid molecule m(e h , V m , E m , ∆ m max , w m , f W m ), also denoted by m ∈ M, as a sub-graph decomposition result from the initial graph G δ based on a coupling between a domain-specific node and its related structuralbased node, where:

• M = d E h (G δ ) is
• V m ⊆ (V s ∪ V d )
is the subset made of domain-specific and structural-based nodes forming m.

For instance, solarfactor4 and bldg1 are examples of domain-specific nodes v d ∈ V m 1 . d1.div7.div1 and d1.div7.p1 are examples of structural-based nodes v s ∈ V m 1 (See Fig. 3.6).

• E m ⊆ ({e h } ∪ E d ∪ E s ) is
∆ m max = max v∈V m (min(shortestPath(v, e h .v s ), shortestPath(v, e h .v d ))), (3.1)
Where, shortestPath refers to the minimum number of edges between two nodes regardless of the weights of these edges.

For instance, ∆ m 1max = 5, which comes down to shortestPath(d1.div7, d2.div2.tab1. cell1) and shortestPath(d1.div7, d2.div2.tab1.cell2) given that both d2.div2.tab1.cell1 and d2.div2.tab1.cell2 are the farthest nodes to the core e h 1 , more precisely to its structural-based node e h 1 .v s i.e., d1.div7 (See Fig. 3.6).

• w m is the overall weight of the molecule m such that w m ∈ W m , where W m is the set of molecules' weights.

• f W m : M → W m is

Query Processing over a Heterogeneous Document Corpus

This section provides an overview on a query processing pipeline over a heterogeneous document corpus. Although we provide a general algorithm including a bunch of other algorithms for each module of the whole pipeline, we particularly focus on the search and ranking modules.

Overview

Our aim is to retrieve relevant information from a given heterogeneous document corpus w.r.t. to non computer expert users' queries. To do this, we propose a query processing strategy that relies on:

• A tightly coupled semantic graph representing the heterogeneous document corpus, so as to take advantage from the richness of this model and the collective knowledge embedded in it,

• Hybrid molecules defined in Sect. 3.3, which provide a novel data structure for query answers made of core information (i.e., hybrid couples) and additional contextual information from both structural and domain-specific dimensions,

• Adapted algorithms that are able to identify components of the proposed hybrid molecules and construct them progressively, starting from the query interpretation stage until the graph-based search, ranking and presentation strategies handling the characteristics of these molecules.

Our proposal aligns with the previously mentioned challenges (See Sect. 3.1) since the hybrid molecules query answers provide users with (i) structural granularity parts of the documents associated with relevant domain-specific information (from the core), (ii) relevant information on the various dependencies between the documents and parts of the document (from structural and domain-specific contextual information associated to the core), and (iii) a helpful frame to interpret the results based on a meaningful sub-graph structure. Fig. 3.12 illustrates our proposal and recalls the classical pipeline for query processing in an IR system (See Fig. 3.3), yet integrating the notion of Hybrid Molecules (HM) in all the stages and relying on a tightly coupled semantic graph for representation of the heterogeneous document corpus, and on LinkedMDR and a domainspecific ontology (for its Pluggable Layer) as the underlying knowledge structure. This corresponds to the upper layer of FEED2SEARCH (See Sect. 1.4.1). The input of our proposed strategy to query processing is a natural language query (expressed as plain text). The final output comes down to relevant hybrid molecules query answers representing parts of documents or documents with additional contextual information. Since the end users are non computer experts, these molecules are presented in a SERP-like results.

Overall Algorithm

In this section, we present the overall algorithm of our proposed HM-based query processing (See Algorithm 2) over a heterogeneous document corpus.

query answers after applying a ranking strategy using rank_params. The latter provide the configurations required for assigning weights or scores to the hybrid molecules.

•

Hybrid Molecule-based Query Interpretation

This section provides details on the hybrid molecule-based query interpretation phase (See Algorithm 3). More specifically, it describes HM_QueryInterpretation function used by Algorithm 2.

HM_QueryInterpretation converts the user's natural language query q into an understandable format that is compatible with the system's internal knowledge structure (i.e., the tightly coupled semantic graph). Since the user's background corresponds to knowledge related to a domain-specific application, we consider that elements describing the domain-specific dimension of the graph are the best to interpret the user's needs. Thus, HM_QueryInterpretation extracts, from q, domain-specific instances I d_out that could later match the graph G δ , specifically domain-specific nodes of its hybrid edges. This is done using the same techniques offered at the middleware indexing layer and used during the generation of the domain-specific parts of the graph (See Sect. 2.3.3.3):

• Step 1 (line 1): It generates semantic annotations (Output d ) describing domainspecific information identified from the textual content of the query q using the semantics of the external domain-specific ontology O D .

• Step 2 (line 2): It converts the generated descriptors of the previous step into domain-specific instances (I d_out ) of LinkedMDR using the semantics of Linked-MDR's domain-specific layer.

Hybrid Molecule-based Search

This section provides details on the hybrid molecule-based search phase (See Algorithm 4). More specifically, it describes the HM_Search function used by Algorithm 2.

HM_Ranking assigns scores or weights to the retrieved hybrid molecules from the search stage. It ranks them according to the computed weights. This is done using two major steps:

•

3). The main purpose

of HM_CSA is to cope with the characteristics of a tightly coupled semantic graph G δ representing a heterogeneous document corpus δ, and retrieve relevant hybrid molecules from G δ . We first provide some background on the CSA theory on which our proposed algorithm is build and the motivations behind our choice of this theory as a basis for our proposed algorithm, recall limitations of existing CSA-based algorithms, and present the main novelty of our proposal. We then describe HM_CSA in details with examples based on Fig. 3.4.

Constrained Spread Activation (CSA)

From the various graph-based search approaches (See Sect. 3.2.3.3), our proposed algorithm relies on a Breadth-First Search (BFS) graph traversal strategy 11 , mainly the CSA algorithm [START_REF] Cohen | Information retrieval by constrained spreading activation in semantic networks[END_REF]. CSA works by spreading out the activation from a set of start nodes to adjacent nodes progressively until predefined constraints are met. We consider CSA as a suitable search strategy for our proposed tightly coupled semantic graph as (i) it handles the heterogeneity of the graph since it can explore possibly relevant structural-based and domain-specific nodes located anywhere in the graph, (ii) it constructs progressively multiple target nodes from activated nodes, and (iii) it supports the incorporation of useful constraints, either at the beginning to select start nodes or at termination point to stop the spreading in the graph.

Although, in its standard form, CSA stands out as a simple yet effective solution in many IR applications [START_REF] Cohen | Information retrieval by constrained spreading activation in semantic networks[END_REF][START_REF] Crestani | Searching the web by constrained spreading activation[END_REF][START_REF] Griffith | A constrained spreading activation approach to collaborative filtering[END_REF][START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF][START_REF] Sun | A spreading activation algorithm of spatial big data retrieval based on the spatial ontology model[END_REF], however, CSA-based algorithms still have some limitations w.r.t. the identified challenges (See Sect. 3.1):

• They output a ranked list of single nodes as query answers, which does not align with our main objective (i.e., to output hybrid-molecules as query answers),

• They weigh nodes and edges based on variant weight mapping strategies (e.g., cluster similarity measure 12 presented in [START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF]), however only adapted to domainspecific semantic graphs where nodes represent documents or web pages. This partially copes with the characteristics of our tightly coupled semantic graph as the latter involves heterogeneous nodes including structural-based ones belonging to different granularity levels of the documents and more sophisticated edges including hybrid edges, which may wrongly guide the activation through the graph towards undesired nodes.

Thus, we propose extending CSA to HM_CSA (See Algorithm 7) with the following advances w.r.t. existing CSA-based algorithms:

• It generates a list of hybrid molecules as query answers,

• It adapts the edge weights differently in order to handle the characteristics of desired hybrid molecules (i.e., hybrid core, structural-based nodes and edges, and domain-specific nodes and edges) and prepare them to be ranked conveniently for the ranking phase of the query processing.

HM_CSA Algorithm

The pseudo code of HM_CSA algorithm is presented in Algorithm 7. The input consists of (i) a set of domain-specific nodes V d_in generated based on the output of the query interpretation module as they matched the user query (See Algorithm 4), (ii) a tightly coupled semantic graph G δ representing a heterogeneous document corpus δ, and (iii) a set of constraints parameters, search_params. The latter is made of pre-adjustment parameters (search_params.preadjustment) to be checked before each spread iteration occurs (e.g., a firing threshold F, and a maximum spread distance D from start nodes), post-adjustment parameters (search_params.postadjustment) to be checked by the end of each spread iteration (e.g., a maximum number of iterations I, and a maximum processing time T), and spread configurations (e.g., an activation percent decrease γ which imposes a decay on the propagation of the activation through the graph). The choice of these parameters is application-dependent. The output of the proposed algorithm consists of a list of relevant hybrid molecules M out , where composing nodes are weighted based on their final activation values.

others and will go in the final output. At start time, V in contains previously selected start nodes V d_in where the activation value Activation(v i ) of each node v i ∈ V in is set to 1 (max value), and V out is initially empty. The algorithm works by checking constraint parameters to decide whether to trigger or not a spread iteration where a firing node v i ∈ V in propagates its activation to its neighbors. The same process repeats until V in has no further nodes to process or post-adjustment constraints are met (line 4). This is done in six major steps described as follows:

• Step 1 (lines 5-6): It consists of removing a firing node v i from V in i.e., the node with the highest activation value. Note that, when many nodes have an equal highest activation value, HM_CSA selects the first node.

• Step 2 (line 7): It checks search_params.preadjustment i.e., pre-adjustment parameters to decide whether current firing node v i is allowed to spread its activation or not to its neighbors. These constraints are met if v i has a distance to start nodes less than or equal to the maximum spread distance D and an activation value higher than or equal to the firing threshold F.

• Step 3 (line 8): It explores direct neighbors of the firing node v i through outgoing edges E ij from v i where each edge e ij ∈ E ij connects v i to its direct neighbor v j .

• Step 4 (lines 9-13, 30): Firing node v i spreads out its activation to each neighbor v j . In other words, each explored node v i contributes to the input of its

neighbor v j by ∆ input(v j ) ← Output(v i ) × f W (e ij ) × (1 -γ) (line 11
), where Output(v i ) is its current activation value (i.e., Activation(v i )), f W (e ij ) computes the weight w e ij of the link e ij connecting v i to v j (See Sect. 3.4.4.1) and

(1γ) is the decay factor. Each ∆ input(v j ) is then added to the input of v j i.e., Input(v j ) 13 (line 12). The actual activation value Activation(v j ) = Output(v j ) of a node v j is obtained by normalizing the sum of all the contributions it receives (line 13). HM_CSA uses a simple feature scaling function 14 to rescale values between 0 and 1. After having spread its activation, current node v i is added to the set of spread nodes V out (line 30) so it could be visited for molecules processing in future iterations.

• Step 5 (lines 14-15): It adds activated neighbor v j to the set of activated nodes V in if it is visited for the first time.

• Step 6 (line 32): It checks search_params.postadjustment i.e., post-adjustment parameters to decide whether to stop the spread iterations afterwards. These constraints are met (i.e., stopSpread is f alse) if current spread iteration is less 13 At start time, the input value Input(v j ) is set to the initial activation value of v j i.e., 1 for start nodes and 0 for others. 14 Output(v j ) =

Input(v j )-Input min Input max -Input min where Input min , Input max are respectively the minimum and the maximum values of all nodes' inputs in the graph.

than the maximum number of allowed iterations I and the current processing time of the algorithm is less than the maximum processing time T.

Molecules Construction and Processing -One major added value of HM_CSA remains in the construction and processing of the hybrid molecules, as described in the red-highlighted section of Algorithm 7 (lines [START_REF]buildingSMART. Model View Definitions[END_REF][START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF][START_REF] Buscaldi | Using the semantics of texts for information retrieval: a concept-and domain relationbased approach[END_REF][START_REF] Charbel | LinkedMDR: A Collective Knowledge Representation of a Heterogeneous Document Corpus[END_REF][START_REF] Charbel | Resolving XML Semantic Ambiguity[END_REF][START_REF] Chbeir | SemIndex: Semantic-aware inverted index[END_REF][START_REF] Chechev | The patents retrieval prototype in the MOLTO project[END_REF][START_REF] Cohen | Information retrieval by constrained spreading activation in semantic networks[END_REF][START_REF] Crestani | Searching the web by constrained spreading activation[END_REF][START_REF] Cunningham | A framework and graphical development environment for robust NLP tools and applications[END_REF]. We consider that all nodes that have spread out their activation i.e., contained in V out , should take part in the final results in the form of hybrid molecules. One naive way to construct the hybrid molecules from the standard implementation of CSA and according to Definition 4 (See Sect. 3.3) is to post-process the nodes in V out . However, this would require reexploring connected nodes in V out in order to construct the sub-graphs and would enormously increase the processing time of the algorithm. Instead, HM_CSA integrates the hybrid molecules construction process within the graph traversal while fetching and activating neighboring nodes.

Two cases arise when processing a neighbor v j ∈ V out connected to v i through an edge e ij :

1. Neighboring edge e ij is hybrid -i.e., e ij = e h (lines [START_REF] Buscaldi | Using the semantics of texts for information retrieval: a concept-and domain relationbased approach[END_REF][START_REF] Charbel | LinkedMDR: A Collective Knowledge Representation of a Heterogeneous Document Corpus[END_REF][START_REF] Charbel | Resolving XML Semantic Ambiguity[END_REF][START_REF] Chbeir | SemIndex: Semantic-aware inverted index[END_REF]: a new molecule m i is constructed (according to Definition 4) with e ij being the core of this molecule (line 20). Since a molecule should be also made of contextual information, HM_CSA appends connected nodes (except those connected through hybrid edges 15 ) from existing molecules in M out , created in previous iterations, whenever one of the core's nodes (i.e., e h .v s or e h .v d ) is found in these molecules For instance, Fig. 3.13a shows an example of a new molecule m new with e ij = e h = (thermal1, d1.div7.div2) being its core, and an existing molecule m k ∈ M, where e h .vs = d1.div7.div2 i.e., the structural-based core's node of m new is found. Fig. 3.13b shows connected structural-based nodes, that are selected from m k using the operator σ s , which are then appended to m new in Fig. 3.13c.

The resulting hybrid molecules are depicted in Fig. 3.13d. 15 The only hybrid edge of the molecule corresponds to its core.

Algorithme 9 : appendToMolecules

Inputs : Neighboring edge e ij connecting node v i to node v j ; Set of of existing hybrid molecules M. Output : Updated set of hybrid molecules M with possibly new elements appended to its hybrid molecules. if ({v j } ∈ M AND {v i } / ∈ M) then // Case 1: only neighboring node v j exists in M M j ← getMolecules(v j ) ; // set of molecules containing v j , M j ⊆ M foreach m j ∈ M j do m j ← m j ∪ {e ij } ; // appends e ij to matched molecule m j end else if ({v j } ∈ M AND {v i } ∈ M) then // Case 2: both neighboring node v j and firing node v i exist in

M M i ← getMolecules(v i ) ; // set of molecules containing v i , M i ⊆ M M j ← getMolecules(v j ) ; // set of molecules containing v j , M j ⊆ M if (getType(e ij ) == "Structural") then m temp ← σ s (M i ∪ M j ) ∪ {e ij } ;
// merges structural-based edges and nodes of hybrid molecules in M i and M j (by using σ s as structural-based selection operator) and appends e ij . else The first case remains simple as e ij is appended to each hybrid molecule m j ∈ M j , where M j is the set of hybrid molecules that contain v j (lines 2-5).

m temp ← σ d (M i ∪ M j ) ∪ {e ij } ; //
For instance, Fig. 3.14a shows an explored neighboring edge e ij = (d2.div2, d1.div7.p1) where v j = d1.div7.p1 is found in m 3 , m 4 ∈ M j . Thus, edge e ij is simply appended to m 3 Fig. 3.14b and m 4 Fig. 3.14c.

The second case (lines 8-20) is more complicated as e ij should be further appended to each hybrid molecule m i ∈ M i , where M i is the set of molecules involving v i . By appending edge e ij to hybrid molecules of both sets M i and M j , the algorithm bridges over other connected nodes among the two sets of hybrid molecules. In other words, it merges either structural-based or domainspecific components of the two sets, depending on the type of edge e ij (lines 10-13). To simplify this operation, a temporary hybrid molecule m temp is created involving either all structural-based or all domain-specific components of the two sets. This is done by using a structural-based or domain-specific selection operator σ s or σ d respectively. The edge e ij is also appended to m temp . After merging the appropriate parts of the two sets M i , M j and the edge e ij in m temp , the hybrid molecules of the two sets are updated with the new content of m temp (lines [START_REF] Brynjolfsson | Race against the machine: How the digital revolution is accelerating innovation, driving productivity, and irreversibly transforming employment and the economy[END_REF][START_REF]buildingSMART. IFC-Industry Foundation Classes, IFC Releases[END_REF][START_REF]buildingSMART. Model View Definitions[END_REF][START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF][START_REF] Buscaldi | Using the semantics of texts for information retrieval: a concept-and domain relationbased approach[END_REF][START_REF] Charbel | LinkedMDR: A Collective Knowledge Representation of a Heterogeneous Document Corpus[END_REF]. The final output is M, the updated set of hybrid molecules. For instance, Fig. 3.15a shows the same example of Fig. 3.14a, however with additional hybrid molecule m 5 ∈ M i where v i = d2.div2 is also found.

CSA vs HM_CSA

In this section, we discuss the importance of applying HM_CSA over a standard implementation of CSA using the graph G δ 1 depicted in Fig. 3.4. We consider the user's needs detailed in Sect. 3.1 and expressed by means of the following natural language query q = "Solar factors of windows". Providing that instances of concepts SolarFactor and IfcCurtainWall are identified from q in the query interpretation module (See Sect. 3.4.2.1), the search module (See Sect. 3.4.2.2) matches nodes solarfactor1 and solarfactor4, and curtainwall1 and curtainwall4 respectively from the graph G δ 1 , and set them as start nodes for the application of the CSA theory. For the sake of simplicity, we neglect post-adjustment parameters and we choose D = 2 (the maximum spread distance) and F = 0 (the firing threshold) as pre-adjustment constraint parameters. Based on the selected start nodes, the tightly coupled semantic graph G δ 1 , and the constraint parameters, we simulate the implementation of a standard CSA (i.e., Algorithm 7 excluding the red-highlighted section, with V out as final output) vs HM_CSA (i.e., Algorithm 7):

Standard CSA-based algorithm -The output of a standard CSA-based algorithm is a list of spread nodes V out i.e., all nodes of the graph G δ 1 except acoustic1 and d2.div2.tab1.cell2 (since their distance to start nodes is greater than D = 2), ranked by their activation values (See Fig. 3.16a). In some applications, the output is augmented with sub-graphs consisting of the shortest paths connecting start nodes to output nodes [START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF]. At termination point, one of the resulting spread nodes is d1.div7.p1.

The latter is related to d1 as it is part of it and to d2 as it references one of its sections.

Thus, neither presenting d1.div7.p1 as a single node result, nor within its augmented sub-graph connecting it to the start nodes (e.g., d1.div7.p1 → d2.div2 → curtainwall1, See Fig. 3.16b) would help the user in getting this information. The latter still need to search for relevant structural and domain-specific context to understand the result, which is time and effort consuming especially in large graphs. HM_CSA algorithm -The output of HM_CSA consists of five hybrid molecules constructed from spread nodes and their connecting edges (See Fig. 3.17): m 1 , m 3 , m 4 , m 5 , and m 6 16 having cores e h1 = (curtainwall4, d1.div7), e h3 = (solarfactor4, d1.div7.div2), e h4 = (thermal1, d1.div7.div2), e h5 = (curtainwall1, d2.div2), and e h6 = (solarfactor1, d2.div2.tab1.cell1) respectively. The core of each molecule holds the central information from which other relevant contextual information is provided by either its connected structural nodes or its domain-specific ones. For instance, d1.div7.p1 is now part of the five hybrid molecules. However, d1.div7.p1 plays different roles within these molecules. In m1, m3, and m 4 d1.div7.p1 adds containment information to e h1 , e h3 , and e h4 through the isPartOf relation. In m 5 and m 6 , d1.div7.p1 adds an interdocument link information to e h5 and e h6 through the references relation. This allows the user to better interpret the results, especially when he tracks fine granularity levels of the documents and cross-document dependencies between them.

Weight Mapping

In this section, we present the weighting functions used to score edges, nodes, and hybrid molecules in a tightly coupled semantic graph G δ . The edges and nodes' weights are calculated during the search module, more precisely in HM_CSA. The hybrid molecules' weights are calculated during the Hybrid Molecule-based Ranking module (See Sect. 3.4.2.3) and correspond to the final weights of query answers.

Edge Weight Mapping

In HM_CSA, the edge weight mapping function f W e , used in the blue-highlighted section of Algorithm 7, is the weighting function that affects the most the output of the search module (See Sect. 3.4.2.2). This is because it directly controls the contributions of neighboring nodes on the activation value of a given node (lines 11-13), which in turn, affects the final weight of the hybrid molecule query answers encapsulating it. In the literature, however, there is no proof on the best edge weighting function. The choice remains application dependent [START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF]. Thus, we rely on commonly used strategies and adapt them to best suit each edge type in G δ providing rationales behind each choice of adoption or adaptation.

The weight w e ij of an edge e ij ∈ E, connecting node v i to node v j is calculated by f W e (e ij ) following the below strategies:

Structural-based Edge Weight -Considering that the edge e ij is a structural-based edge i.e., e ij = e s , where e ij connects two structural-based nodes v i and v j (v i , v j ∈ V s ), the corresponding edge weight w e ij comes down to w e s = f W e (e s ) ∈ [0, 1] and is manually acquired from the corpus expert.

The rationale is that a structural-based edge weight is a data design issue. It is set by the corpus expert to best suit the application.

For instance, consider the structural-based edge e s 6 = (d1.div7.p1, d2.div2) with f Lab (e s 6 ) = "references", and e s 7 = (d1.div7.div2, d1.div7.p1) with f Lab (e s 7 ) = "hasPart" (See Fig. 3.4). The corpus expert may decide that e s 6 is more important than e s 7 as it provides crucial information on an inter-document link. Thus, w e s 6 may be set to 0.7 while w e s 7 may be lower and equal to 0.5 Domain-specific Edge Weight -Considering that the edge e ij is a domain-specific edge i.e., e ij = e d , where e ij connects two domain-specific nodes v i and v j (v i , v j ∈ V d ), the corresponding edge weight w e ij comes down to w e d = f W e (e d ), such that:

f W e (e d ) = 1 f an -in lab (v j ) ∈ ]0, 1] (3.2)
Where,

• 1 f an-in lab (v j )
is the specificity measure of the edge e d incoming towards v j

• f anin lab (v j ) is the number of incoming nodes towards v j having the same label lab of e d i.e., lab = f Lab (e d )

The rationale is that the specificity measure reflects the importance of a domainspecific edge w.r.t. a target node. The less incoming edges with the same label, the more important the edges become for a node. This measure is commonly used in the literature of semantic graphs (e.g., [START_REF] Chbeir | SemIndex: Semantic-aware inverted index[END_REF][START_REF] Rocha | A hybrid approach for searching in the semantic web[END_REF]).

For instance, consider the domain-specific edge e d 2 = (solarfactor4, curtainwall4) with lab 2 = f Lab (e d 2 ) = "isIfcPropertyOf" (See Fig. 3.4). f anin lab 2 (e d 2 ) = 1 as it is the only domain-specific edge with label lab 2 incoming towards curtainwall4, thus w e d 2 = 1 1 = 1.

Hybrid Edge Weight -Considering that the edge e ij is a hybrid edge i.e., e ij = e h , where e ij connects a domain-specific node v i ∈ V d to a structural-based node v j ∈ V s or vice versa, the corresponding edge weight w e ij comes down to w e h = f W e (e h ), such that:

f W e (e h ) = TF(s i , v j ) × IDF(s i , V s ) ∈ [0, 1[ (3.3)
Where,

• TF(s i , v j ) × IDF(s i , V s ) is inspired by the notion of TF-IDF (Term Frequency -Inverse Document Frequency) score;

• TF(s i , v j ) is the frequency of the value s i of the domain-specific node v i ∈ V d occurring in a structural-based node v j ∈ V s (e.g., a document or a part of a document), where

s i = f Val (v i ).
It is calculated as follows:

TF(s i , v j ) = NbOcc(s i , v j ) Max(NbOcc(s k , v j )) ∈ ]0, 1] (3.4)
Such that, -NbOcc(s i , v j ) is the number of occurrences of the string value of v i in the content of its related structural-based node v j .

Note that, the minimum value of NbOcc(s i , v j ) is equal to 1 since the presence of the hybrid relation means that, during the annotation phase, an occurrence of s i was identified in v j and lead to the creation of the hybrid edge (as discussed in Chapter 2, in Sect. 2.3.3.3).

-Max(NbOcc(s k , v j )) is the maximum number of occurrences of the string value s k of any domain-specific node v k ∈ V d connected to the same structural-based node v j through another hybrid edge e kj . Thus, TF(s i , v j ) is never equal to 0.

• IDF(s i , V s ) is the inverse frequency of the value s i of the domain-specific node v i ∈ V d occurring in the set of all structural-based nodes V s . It is calculated as follows:

IDF(s i , V s ) = 1 - DF(s i , V s ) |V s | ∈ [0, 1[ (3.5)
Such that, -DF(s i , V s ) is the number of structural-based nodes v s involving at least one occurrence of the string value s i of v i .

Note that the minimum value of DF(s i , V s ) is equal to 1 since s i has at least occurred in one structural-based node v s ∈ V s , specifically in v j ∈ V s . Thus, IDF(s i , V s ) is never equal to 1.

-|V s | is the total number of structural-based nodes in the graph G δ .

The rationale is that, in G δ , a domain-specific node v i describes the content of a structural-based node v s , thus the value of a v i can be perceived as a term and v s as a document in a TF-IDF like notion (in a similar vein with other IR applications, such as [START_REF] Tekli | Full-fledged semantic indexing and querying model designed for seamless integration in legacy RDBMS[END_REF]). Consequently, a hybrid edge weight w e h is directly proportional to the number of occurrences of the domain-specific information in a structural-based element and inversely proportional to the number of occurrences of the domainspecific information contained in other structural-based elements.

For instance, consider the hybrid edge e h 5 = (curtainwall1, d2.div2) (See Fig. 3.4) with f Lab (e h 5 ) = "isRelatedTo". Given v i = curtainwall1 and has a string value s i = f Val (v i ) = 'window frames", and v j = d2.div2 and has a string content s j = f Val (v j ) = "Characteristics of Window Frames" ⇒ TF(s i , v j ) = 1 1 = 1 since NbOcc(s i , v j ) = 1 and there is no other value of a domain-specific node v k connected to v j that occurs several times in its content s j . Also, considering that s i did not occur in any other structural-based node v s ∈ V s and |V s | = 10, DF(s i , V s ) = 1 ⇒ IDF(s i , V s ) = 1 -1 10 0.9. Thus, w e h 4 = 1 × 0.9 = 0.9

Node Weight Mapping

In any CSA-based application, the node weights come down to their final activation values [START_REF] Cohen | Information retrieval by constrained spreading activation in semantic networks[END_REF]. Naturally, the activation function reflects the contribution of neighboring nodes considering the strength of linking edges (as described in Sect. 3.4.3.2).

Along the same lines, we consider that the node weights w v s of a structural-based node v s ∈ V and w v d of a domain-specific node v d ∈ V are calculated by f W v (v) based on the final activation value of a node v ∈ V regardless of its type i.e,:

f W v (v) = Activation(v) ∈ [0, 1] , ∀v ∈ V. (3.6)
The rationale is that given the different strategies that are applied to calculate the edge weights (See Sect. 3.4.4.1) based on their types, the activation function implicitly weighs the nodes differently considering their nature i.e., whether they are structural-based nodes or domain-specific nodes.

For instance, consider the structural-based node v j = (d2.div2) in Fig. 3.18. The activation value Activation(v j ) is the result of the contributions of firing nodes V i = (curtainwall1, d2.div2.tab1, d1.div7.p1, d2) at different points of time t k of Algorithm 7.

At t 0 , consider that the domain-specific node v i = (curtainwall1), where v i ∈ V i , spreads out its activation to its neighboring node v j . Providing that the initial activation values of nodes v i and v j are set to 1 and 0 respectively i.e., Activation(v i ) = 1 and Activation(v j ) = 0, and γ = 0.2 i.e., the decay factor (1γ) is equal to 0.8: ∆ input (v j ) = 1 × 0.9 × 0.8 = 0.72 ⇒ Input(v j ) = 0 + 0.72 = 0.72 ⇒ Output(v j ) = Activation(v j ) = 0.72. Similarly, at t k , another node v i ∈ V i spreads out its activation to v j , until the termination point t END , where the final activation value of v j is set to the contributions of all nodes v i ∈ V i . Consequently, if at t END of Algorithm 7 Activation(v j ) = 1 ⇒ the node weight w v j of the structural-based node d2.div2 comes down to w v j = 1. This means that, over spread iterations, firing nodes V i strengthened the importance of the neighboring node v j = (d2.div2) as it is possibly relevant w.r.t. initial start nodes.

f W m (m) = α × ∑ w v s + β × ∑ w v d |V m | × w e h ∈ [0, 1] (3.7)
Where,

• w v s and w v d are the weights computed by f W v (v) and assigned to a structuralbased node v s ∈ V m and a domain-specific node v d ∈ V m respectively.

• |V m | is the total number of nodes in m.

• α and β are the weight parameters that balance the contribution of the structural and domain-specific parts of m, such that α and β ∈ [0, 1].

• w e h is the hybrid edge weight of the core e h of m Note that, α and β come down to rank_params in Algorithm 5.

The rationale is that, as the hybrid molecules are constructed progressively, they are changing from one spread iteration to the other. Hence, their weights are the best to be calculated based on the contributions of their components at termination point of HM_CSA. This is done using the final activation values of the nodes composing them which implicitly reflect the contributions of the edge components as well. For the sake of simplicity, this is expressed through an equation of weighted average values of structural-based and domain-specific contributions within a hybrid molecule, which is then multiplied by the weight of the hybrid edge identifier of the molecule.

The latter is done for two main reasons: (1) the hybrid edge is the unique identifier of a molecule as it holds core information (as discussed in Sect. 3.3.2), (2) some molecules may involve the same structural-based and domain-specific components yet different core information, thus the hybrid edges should leverage their weights so they could be scored differently and conveniently.

For instance, Fig. Although m 1 and m 3 have exactly the same structural-based and domain-specific components, their hybrid edges e h 1 and e h 3 respectively are different and have different weights. w e h 1 = 0.9, being higher than w e h 3 = 0.8, the hybrid molecule weight w m 1 of m 1 is higher than the weight w m 3 of m 3 . Considering α = 1, β = 0.5 to emphasize the contribution of structural-based nodes on the weights of m 1 and m 4 :

w m 1 = 1 × (0.
w m 1 = 1 × (0.

Summary

This chapter tackles the problem of handling IR over a tightly coupled semantic graph representing a heterogeneous document corpus. The main purpose is to provide the users with augmented search results i.e., query answers including both the structural and the domain-specific dimensions of the document corpus. This is not covered by current SIR systems which mainly neglect, in their search results, relevant granularity levels of the documents and dependencies between them. The contributions of this chapter come down to:

Introduction

This chapter describes the experimental study we conducted in order to validate the contributions of Chapter 2 and Chapter 3 in the context of the AEC industry, using real-world construction projects from Nobatek/INEF4. We set two main global objectives: (i) evaluate the knowledge representation of a heterogeneous document corpus on the basis of the contributions of Chapter 2 (Objective 1), and (ii) evaluate the quality of the IR model on the basis of contributions of Chapter 3. To so do, we first implemented FEED2SEARCH framework layers (See Chapter 1) within a Java-based prototype as follows:

• We used LinkedMDR OWL file (lmdr.owl) 1

for the Knowledge Representation

Layer. This version is built on OWL 2 in the Protégé environment and serialized in RDF/XML. It relies on DC [START_REF]Dublin Core Metadata Initiative[END_REF], TEI [START_REF]The Text Encoding Initiative Consortium[END_REF] and MPEG-7 [START_REF]The Moving Picture Experts Group[END_REF] standards for the Standardized Metadata Layer and on an adapted ifcOWL [START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF]-based ontology as an external domain-specific ontology for the Pluggable Domain-Specific Layer;

• We developed a module for the Indexing Layer, entitled LMDR Annotator, in order to generate a tightly coupled semantic graph representing a heterogeneous document corpus. Although LMDR Annotator provides automatic pipelines from the stage of upload of the document corpus to the stage of the generation of the corresponding semantic graph, the users still have the possibility to intervene in all the intermediate stages in order to collect the generated results for experimental purposes;

• We developed a module for the Query processing Layer, entitled HM Query Processor, in order to generate a ranked list of Hybrid Molecule-based query answers w.r.t. a user's natural language query. Although the main purpose of the Query Processing layer of FEED2SEARCH is to output the hybrid molecules in SERP-like results, we do not so far focus on GUI issues while implementing HM Query Processor as it is not the main contribution of the thesis. Instead, we give an intermediary visualization layout of the hybrid molecules, which, with our assistance, is presented to users in the AEC industry for experimental evaluations.

The remainder of this chapter is organized as follows. Sect.

Overall Architecture

Automatic Metadata Extraction

The main purpose of this sub-module is to automatically generate descriptors of DC [START_REF]Dublin Core Metadata Initiative[END_REF], TEI [START_REF]The Text Encoding Initiative Consortium[END_REF] and MPEG-7 [START_REF]The Moving Picture Experts Group[END_REF] that are reused in LinkedMDR ontology based on existing tools, APIs or web services:

Automatic Generation of DC Metadata

Among the various DC-based tools2 , LMDR Annotator uses the Apache Tika 1.4

Toolkit3 as it provides a fully automatic technique to generate, from a wide variety of document formats(JPG, TXT, PDF, etc.), a list of DC descriptors covering all the elements reused in LinkedMDR.

The Apache Tika toolkit automatically detects and extracts metadata and text using metadata harvesting techniques [START_REF] Greenberg | Metadata extraction and harvesting: A comparison of two automatic metadata generation applications[END_REF]

Automatic Generation of TEI Metadata

Among few available TEI-based tools 5 , LMDR Annotator uses Oxgarage6 web RESTful service as it provides a fully automatic technique to generate, from a wide variety of document formats (e.g., TXT, DOCX, XLSX, etc.), a reliable list of TEI [START_REF]The Text Encoding Initiative Consortium[END_REF] descriptors covering all the elements reused in LinkedMDR. It is also recommended by the TEI consortium 7 .

Oxgarage web service is based on the Enrich Garage Engine8 which provides services for data conversion, transformation and validation. Among the underlying converters, the TEI converter is based on eXstensible Stylesheets Language (XSL) to convert between different forms of XML documents. The output of the web service calling the TEI converter is an XML format following specifications of TEI standard.

Automatic Generation of MPEG-7 Metadata

In LinkedMDR ontology, we so far reused MPEG-7 descriptors related to the image metadata with different levels of precision (e.g., mpeg7 : Image , mpeg7 : StillRegion ), its related visual descriptors, and semantic descriptors. For practical reasons, in the implementation of LMDR Annotator, we only focus on the automatic generation of MPEG-7 visual descriptors for two main reasons: (i) the automatic French construction related documents. In order to generate the required annotation sets, LMDR Annotator executes a pre-configured GATE pipeline application and provides it with the required knowledge resources. The GATE pipeline is made of the following main processing resources:

• Document Reset, Tokeniser, Sentence Splitter, POS Tagger, and Morphological Analyser for linguistic pre-processing of the textual content;

• ANNIE Gazetteer which consists of a pre-defined list of expressions that yield inter and intra-document references;

• OntoGazetter which creates mappings between lists containing documents' information (e.g., lists of document names dynamically created at execution time of LMDR Annotator) and corresponding LinkedMDR instances 12 (e.g., a mapping between a document name and its corresponding LinkedMDR instance URI);

• JAPE Transducer13 which allows to recognize regular expressions and to subsequently create inter and intra-document references.

For instance, Fig. 4.6 shows an example of a pre-defined rule in the JAPE Transducer, which detects an inter-document reference InterDocLink whenever it identifies a Link entity (i.e., an entry from the pre-defined list of expressions in the ANNIE Gazetteer), followed by 0 to 3 Token entities (i.e., entries from the pre-processed list of

Semantic Annotation for Automatic Generation of Domain-Specific Annotation Sets

We focus on ontology-based annotation techniques which link pre-processed forms of textual content to a specific concept or relation of a domain-specific ontology (which is integrated in the domain-specific layer of LinkedMDR). This is to automatically generate domain-specific annotation sets that will be later used to create LinkedMDR domain-specific instances.

To do this, we also use the GATE API as it provides robust techniques for automatic semantic annotation and has been frequently used in the literature for this purpose [START_REF] Chechev | The patents retrieval prototype in the MOLTO project[END_REF][START_REF] Kiryakov | Semantic annotation, indexing, and retrieval[END_REF]. As it is the case with the use of GATE in the automatic generation of inter and intra-document references, LMDR Annotator executes a main preconfigured GATE pipeline application. For this sub-module, the GATE pipeline is made of the main following processing resources:

• Document Reset, Tokeniser, Sentence Splitter, POS Tagger, and Morphological Analyser for linguistic pre-processing of the textual content;

• FlexibleGazetteer where it is possible to configure (i) feature names of annotation sets that will replace the original text, and (ii) a gazetteer instance which generates ontology-based annotations with the configured features. The latter comes down to an OntoRootGazetteer which is fed with the chosen domainspecific ontology in order to pre-process its classes, properties, labels, etc. (using the same above linguistic processing resources) and generate their humanunderstandable forms.

LMDR Annotator outputs the generated annotation sets in XML format. An extract of the generated output following the execution of the main GATE pipeline, described above, is illustrated in Fig. 4.8. For instance, the concept SolarFactor is identified from the occurrence of "facteur solaire" in the French textual context of a construction related document.

LinkedMDR Converters

LinkedMDR Converter is the core sub-module of LMDR Annotator as it is made of a list of converters that bridge the gap between annotations of existing standards and tools, and those of LinkedMDR over heterogeneous documents. The main purpose of this sub-module is to automatically generate instances of LinkedMDR based on the outputs of all previously described sub-modules.

Each sub-module in LMDR Annotator has a dedicated LinkedMDR converter (See In order to do the required conversions, LMDR Annotator relies on eXtensible Stylesheet Language Transformations 14 (XSLT) as it is the widely used language to transform XML documents into other formats. Consequently, each converter comes down to a tailored XSLT processor: An example of a LinkedMDR converter, DC to LinkedMDR, is provided in Appendix A together with the RDF output generated when executing it on an XML file.

The chosen XML file is the same one presented in Fig. 4.3b, which is considered a previously generated output from the Automatic Metadata Extraction sub-module of LMDR Annotator. Note that, all the DC descriptors that are contained in the XML file were transformed into LinkedMDR instances of the Standardized Metadata Layer (e.g., instance of the concept dc:title) in the generated RDF file. Also note that, the converter involves a set of rules that infer LinkedMDR instances of the Core Layer (e.g., instance of the concept Document) that did not exist in the XML file.

A full list of LinkedMDR converters is available online at spider.sigappfr.org/ linkedmdr/lmdr-annotator/.

Tightly Coupled Semantic Graph Builder

This sub-module is responsible for the aggregation of all the previously generated RDF annotations of LinkedMDR converters' sub-module and the generation of a tightly coupled semantic graph G δ m representing a heterogeneous document corpus δ m . We use the Apache Jena 3.2.0 API 15 as it is the widely used API to extract data from and write to RDF graphs. On the basis of Jena features, LMDR Annotator is capable of:

• Merging RDF files into one file describing the graph G δ m , while easily handling RDF instances contained in different RDF files and referring to the same resource, based on the latter's Universal Resource Identifier (URI);

• Creating hybrid edges which link LinkedMDR domain-specific instances to core instances or standardized metadata-based instances where they were identified during the semantic annotation process;

• Dynamically generating inferred instances in G δ m based on the semantics of LinkedMDR ontology.

HM Query Processor

We For the implementation of these algorithms, HM Query Processor uses:

• Two of LMDR Annotator's sub-modules for the query interpretation: the semantic annotation sub-module relying on Gate Developer 8.1 API (See Sect. 4.2.4)

and LinkedMDR Converters sub-module (See Sect. 4.2.5) in order to use the same semantic annotation technique as for the corpus annotation phase and then transform the annotations into LinkedMDR instances;

• Jena 3.2.0 API, which provides a means to navigate within the semantic graph, extract data and reason over it in the search and ranking phases;

• NavigOWL16 java-based visualization tool which is called by HM Query Processor in the presentation phase, specifically for the visualization of the domainspecific and structural-based context of each hybrid-molecule query answer in an appealing graph layout [START_REF] Hussain | Scalable visualization of semantic nets using power-law graphs[END_REF]. The tool supports both RDF and OWL files.

Note that, the implementation of HM Query Processor is still on-going, particularly for the presentation module. Our current effort focuses on the GUI in order to improve the presentation of the hybrid molecules and make it more adapted to non computer expert users. The user has the choice to explore the structural-based and the domain-specific con-

text of this answer by checking one or both checkboxes below the answer. This displays the required contextual information, in a graph layout, in the NavigOWL tool. The user can manage the graph in a more clear and consistent view. When the mouse is over a particular node, he can see its value and relations to other nodes within the context (e.g, a reference from the relevant paragraph of CCTP.pdf describing the solar factor to another document (rapport thermique.pdf ) that was not listed in the result list.

Evaluation of the Annotation of a Heterogeneous Document Corpus based on LinkedMDR

We conducted several experiments in order to assess the annotation of a heterogeneous document corpus based on the proposed infrastructure of LinkedMDR ontology (See Chapter 2). This is done regardless of the pluggable Domain-Specific Layer as we focus on the invariant knowledge part of LinkedMDR across domains. To do so, we targeted two main objectives:

• Objective 1.1: Compare LinkedMDR's comprehensive annotations with alternatives ones (i.e., annotations based on existing standards e.g., DC [START_REF]Dublin Core Metadata Initiative[END_REF], TEI [START_REF]The Text Encoding Initiative Consortium[END_REF], MPEG-7 [START_REF]The Moving Picture Experts Group[END_REF], and the naive combination of their annotations 17 ) regardless of the annotation tools;

• Objective 1.2: Evaluate the automatically generated LinkedMDR annotations using LMDR Annotator with a focus on those generated following the Core Layer and the Standardized Metadata Layer.

Experimental Context

In this section, we present the test corpora used throughout this experimental study together with their corresponding annotations considering different scenarios for Objective 1.1 and Objective 1.2.

Test Corpora

We hand-picked 5 heterogeneous document corpora (δ m with m ∈ {1, . . . , 5}) of realworld construction projects from Nobatek/INEF4. Table 4.1 shows the document composition of each test corpus. Each corpus is made of different documents, where we varied their number, their formats and their size.

Test Annotations

We prepared two categories of annotations (i) comprehensive annotations representing corpus δ 1 (for Objective 1.1), and (ii) automatically generated LinkedMDR annotations for the remainder corpora i.e., corpora δ 2 , δ 3 , δ 4 and δ 5 (for Objective 1.2).

As for the comprehensive annotations, we first used annotations generated by LMDR Annotator sub-modules on corpus δ 1 (i.e., δ 1

[DC].xml, δ 1 [TEI].xml, δ 1 [MPEG7]
.xml, and G δ 1 .rd f ), and then manually adapted them. This ensured the best possible repre- sentation of δ 1 following DC [START_REF]Dublin Core Metadata Initiative[END_REF], TEI [START_REF]The Text Encoding Initiative Consortium[END_REF], MPEG-7 [START_REF]The Moving Picture Experts Group[END_REF] and LinkedMDR (excluding Domain-Specific Layer's annotations). We further combined the generated XML annotations of the three standards in one XML file (i.e., δ 1 [combined].xml) to create comprehensive annotations of δ 1 considering their naive combination.

As for the automatically generated LinkedMDR annotations, we ran LMDR Annotator on the four document corpora: δ 2 , δ 3 , δ 4 and δ 5 . For each document corpus δ m , we collected the generated RDF Annotations of the different sub-modules separately (i.e., δ m [LinkedMDR_xLinks].rd f ) representing LinkedMDR instances related to inter and intra-document references, and δ m [LinkedMDR_DC].rd f , δ m [LinkedMDR_TEI].rd f and δ m [LinkedMDR_MPEG7].rd f representing LinkedMDR instances related to the standardized metadata layer. We further collect the generated tightly coupled semantic graph (i.e., G δ m .rd f ) representing all the generated LinkedMDR instances.

Evaluation Criteria and Metrics

In this section, we present the evaluation criteria and the underlying metrics. For Objective 1.1, we choose the conciseness criterion that we explain in Sect. 4.4.2.1 which is a well-known criterion among the ontology evaluation methods [START_REF] Raad | A survey on ontology evaluation methods[END_REF]. We adapt it to our context and define the required metrics in order to take into consideration the evaluation of different data models (ontology-based and non ontology-based) which have different formats and semantics. For Objective 1.2, we choose the effectiveness that we explain in Sect. 4.4.2.2 as it is commonly used to evaluate the quality of systems and tools' results in the context of IR [START_REF] Manning | Introduction to information retrieval[END_REF].

Conciseness

We evaluate the conciseness of each adopted annotation model i.e., DC [START_REF]Dublin Core Metadata Initiative[END_REF], TEI [START_REF]The Text Encoding Initiative Consortium[END_REF], MPEG-7 [START_REF]The Moving Picture Experts Group[END_REF], the three standards combined, and LinkedMDR ontology in the representation of corpus δ 1 . More particularly, for each annotation model we look at:

• The number of annotated documents;

• The number of resulting annotation files;

• The cumulative number of annotation elements (i.e., the number of XML tags in the XML annotation files and the number of RDF triples in the RDF annotation file) within the annotation files;

• The number of redundancies i.e., the overlapping annotation elements;

• The percentage of covering a pre-defined list of relevant criteria. This list comes down to categories of annotation elements aligned with the requirements we set in Chapter 2 such as inter and intra-document links, general metadata of the documents, structural metadata for the text, structural metadata for the image, etc. (See Sect. 2.1).

We consider an annotation model to be the most concise if it is capable of annotating all the documents in the corpus δ 1 with a minimum number of annotation elements, a minimum number of annotation files and a minimum number of redundancies, while covering a maximum number of relevant criteria from the pre-defined list.

Effectiveness

We evaluate the effectiveness of LMDR Annotator in automatically annotating corpora δ 2 , δ 3 , δ 4 , and δ 5 in terms of:

• Precision (P) to identify the number of relevant annotations among the automatically generated ones;

• Recall (R) to identify the number of relevant automatically generated annotations among the total number of expected relevant annotations;

• F 2 -score to evaluate a weighted average of P and R. Note that we choose F 2score since it emphasizes the recall measure. This indirectly highlights missed expected relevant annotations, which is an important factor to consider in the evaluation of the quality of the annotation phase.

These measures are calculated as follows:

P = a a + b (4.1) R = a a + c (4.2)
F 2 -Score = 5 × P × R 4 × P + R (4.3)
Where,

• a is the number of automatically generated relevant annotations (true positives);

• b is the number of automatically generated annotations that are not relevant (false positives).

• c is the number of relevant annotations that are not generated by LMDR Annotator (false negatives).

Experimental Results

In this section, we present the results of two conducted experiments considering the context defined in Sect. 4.4.1, and the evaluation criteria and metrics defined in Sect. 4.4.2.

Evaluating the conciseness of LinkedMDR and its alternatives

We evaluate the conciseness of LinkedMDR, DC [START_REF]Dublin Core Metadata Initiative[END_REF], TEI [START_REF]The Text Encoding Initiative Consortium[END_REF], MPEG-7 [START_REF]The Moving Picture Experts Group[END_REF], and the three standards combined in annotating δ 1 as described in Sect. 4.4.2.1.

Table 4.2 shows that when using DC, the three standards combined, and Linked-MDR as annotation models, all the documents involved in corpus δ 1 were annotated. In contrast, when using TEI and MPEG-7 as annotation models, the annotations were not exhaustive. This is due to the incapacity of annotating images and technical drawings in TEI and textual documents in MPEG-7. The annotation files generated based on DC standard contain a very small number of annotation elements while covering few relevant criteria from the pre-defined list (21%). This is because DC covers generic metadata and neglects structure and content representation of the documents. The naive combination of the three standards produces a significant number of annotation elements since TEI and MPEG-7 standards are very verbose while covering 77% of relevant pre-defined criteria. Examples on the important relevant criteria that the annotations of the three standards combined still lack are inter and intra-document dependencies. Also, the naive combination of the three standards involves many redundancies caused by mutual annotation elements between DC, TEI and MPEG-7 standards, mainly general metadata information of the documents (e.g., document title, author name, etc.).

The current experiment shows that LinkedMDR is the most concise annotation model representing corpus δ 1 since it has the highest coverage of relevant pre-defined criteria (100%) with a relatively small number of annotation elements, all in a single annotation file, and without any redundancies. requires sophisticated computer vision and machine learning algorithms (such as [START_REF] Opencv | [END_REF]) which, so far, are not adopted in our prototype. The sub-module generating δ m [LinkedMDR_TEI].rd f provides relatively good results in automatically generating structural metadata related to the text using Apache Oxgarage web service (See Sect. 4.2.2.2). However, there are still some limitations due to originally poor structured textual documents (especially with some PDF files) resulting in missing annotations regarding headings numbering and information on their related pages. As for the sub-module generating δ m [LinkedMDR_xLinks].rd f , one can see that LMDR Annotator is reliable in automatically extracting explicit inter and intra-document references from pre-defined regular expressions encountered in the textual documents using the Java GATE API (See Sect. 4.2.3). However, in some cases, such as in corpus δ 4 , the sub-module obtains relatively lower score. This is due to some expressions representing ambiguous references that could not be handled automatically without the use of advanced semantic disambiguation techniques [START_REF] Charbel | Resolving XML Semantic Ambiguity[END_REF][START_REF] Tekli | Building semantic trees from XML documents[END_REF]. by LinkedMDR ontology. The average recall value is equal to 0.43 over the four corpora. Intuitively, the recall scores decrease when the number of documents increases (e.g., from R = 0.5 for corpus δ 2 to R = 0.34 for corpus δ 4 ) since more complex inter and intra-document links are involved, which cannot yet be generated by LMDR Annotator (e.g., spatial inter-document dependencies: overlaps of images representing extract of technical drawings across documents). This emphasizes that LMDR Annotator, at its current state, offers relatively low recall scores when compared to the annotation potential proposed by LinkedMDR ontology.

Discussion

The first experiment demonstrates that, considering optimal annotations, the annotation capability provided by LinkedMDR ontology is better than the one provided by commonly used existing metadata standards for document representation (Objective 1.1).

In a real-world application, manually annotating a document corpus is a tedious job that requires much effort and technical knowledge. The second experiment shows promising results for an automatic annotation of a heterogeneous document corpus using LMDR Annotator, which automatically generates LinkedMDR instances representing the corpus (Objective 1.2). Although the average obtained effectiveness value is still low (average F 2 -score= 0.6), it reflects the quality of the so far implemented sub-modules of LMDR Annotator mostly relying on existing APIs, tools and services embedded in an automatic pipeline. However, these results are still encouraging since the pipeline provides more automatic annotation capabilities over a given corpus than each existing API, tool or web service alone. Our current effort focuses on furthering the annotation capabilities of LMDR Annotator in order to obtain higher effectiveness results enabling its adoption in real-world projects.

the two groups. Figure 4.14 shows the average values of P, R, and F 1 -score per run configuration per query group.

We select the optimal values of constraint parameters based on the results of the F 1 -score.

Evaluating the effectiveness of HM_Ranking

We also evaluate the ranking of the hybrid molecule-based query answers considering the optimal constraint parameters of HM_CSA (F = 0.3 and D = 4 from the previous experiment) over the same two groups of queries. We vary α and β parameters (See Section 3.4.4.3) and study their impact on the MAP values of HM_Ranking.

We choose 3 configurations: (i) α = 0 and β = 1, (ii) α = 1 and β = 0, and (iii) α = 1 and β = 1 to emphasize respectively domain-specific nodes' contribution, structural-based nodes' contribution and the equal contribution of both in the overall weight of a hybrid molecule. These contributions also reflect the impact of the different strategies adopted for the weight mapping (See Section 3.4.4).

Discussion

In the context of the given heterogeneous document corpus, the two conducted experiments demonstrate that, using optimal constraint and weight parameters, HM_CSA reaches an overall F 1 -score of 0.75, and HM_Ranking reaches overall MAP values > 0.6. The obtained results are considered as promising results in IR [START_REF] Manning | Introduction to information retrieval[END_REF]. They demonstrate that our proposed Hybrid Molecule-based Search and Ranking modules are capable of providing relevant query answers in the form of hybrid molecules i.e., the augmented contextualized results satisfy the user's needs.

Summary

This chapter presents an experimental evaluation of the contributions of Chapter 2 and Chapter 3 in regard to the semantic representation of a heterogeneous document corpus based on LinkedMDR ontology, and the adoption of such representation in an innovative IR model respectively. In this study, we used real-world data from projects in the AEC industry.

On the one hand, we assessed the annotations of a heterogeneous document corpus considering two different scenarios (i) regardless of the annotation tools i.e, based on comprehensive (ideal) annotations, and (ii) using LMDR Annotator module of FEED2SEARCH Java-based prototype that we implemented to automatically generate the annotations. The corresponding experimental results show that:

• Considering optimal annotation capabilities (ideal annotations), the annotations based on the infrastructure provided by LinkedMDR ontology are more concise than those provided by commonly used existing metadata standards (e.g., DC [START_REF]Dublin Core Metadata Initiative[END_REF], TEI [START_REF]The Text Encoding Initiative Consortium[END_REF], MPEG-7 [START_REF]The Moving Picture Experts Group[END_REF], and the three standards combined) for document representation;

• Considering automatically generated annotations using LMDR Annotator, we obtained promising effectiveness results (i.e., average F 2 -score= 0.6) that still can be improved. However, LMDR Annotator provides more automatic annotation capabilities over a given corpus than each existing API, tool or web service alone which are dedicated to a specific document representation based on a specific standard or data model.

On the other hand, we assessed the quality of the retrieved hybrid molecule-based query answers using HM Query Processor module of FEED2SEARCH Java-based prototype that we implemented to validate that, given a user's natural language query and optimal annotations of a heterogeneous document corpus: (i) the proposed HM_CSA algorithm provides relevant hybrid molecules query answers, and

(ii) HM_Ranking module ranks the generated hybrid molecules conveniently. The corresponding experimental results show that:

• Considering optimal constraint parameters and diverse queries, the effectiveness reached with HM_CSA is relatively good (average F 1 -score= 0.75), which demonstrates that the retrieved hybrid molecules correspond to users expectations. The Hybrid Molecules being a novel data structure, we are currently conducting a qualitative study to compare the richness of such data structure w.r.t. the State-of-the-art (e.g., a standard CSA algorithm that provides single nodes results).

• Considering a balanced contribution of the structural and domain-specific parts of the hybrid molecules in the calculation of their weights, HM_Ranking attains promising results of effectiveness (average MAP values > 0.6). Higher effectiveness values can be achieved in the future by investigating further alternative weight mapping functions.

Chapter 5

Conclusion

"The important thing is not to stop questioning. Curiosity has its own reason for existing."

-Albert Einstein

Recap

In this thesis, we proposed a semantic representation of a heterogeneous document corpus enabling an innovative IR over it. Our proposal enhances the current status of industries in inferring relevant information for the progress of their projects. Our evaluation was based on examples from construction projects in the AEC industry.

In Chapter 1, we highlighted the remaining disregarded issue of substituting raw and unstructured information with intelligent data models empowering knowledge reasoning capabilities and enabling the full potential of Industry 4.0. We then gave a particular attention to the Construction 4.0 in the AEC industry which struggled to keep up with the advances of ICT, such as the BIM multi-dimensional model-based process.

We presented a motivating scenario from the context of a multi-disciplinary project (e.g., the construction project), through which the thesis addressed two main challenges: (i) the knowledge representation of a heterogeneous document corpus, and (ii) the innovative information search enabling the users to extract relevant information from raw documents required for their tasks in the project.

The contributions of the thesis were presented within a generic FEED2SEARCH framework and detailed in the remaining chapters.

In Chapter 2, we focused on the main challenge of representing the collective knowledge embedded in a heterogeneous document corpus while considering two dimensions: the structural dimension and the domain-specific dimension of the corpus.

We identified the following sub-challenges: (i) representing the various inter and intra-document links within the corpus, (ii) describing general metadata information of the document, its content, and its structural text and multimedia components all combined, (iii) associating semantics at content and structural levels of the document, (iv) handling document multimodality, and (v) ensuring extensibility. According to the literature that we reviewed, there is no existing standard or data model for document representation that addressed these challenges combined.

We introduced a novel semantic-based approach, entitled Tight Coupling approach, which aims to represent the collective knowledge embedded in a heterogeneous document corpus through a tightly coupled semantic graph. We then proposed a backbone multi-layered ontology, entitled LinkedMDR [START_REF] Charbel | LinkedMDR: A Collective Knowledge Representation of a Heterogeneous Document Corpus[END_REF], which provides the required infrastructure to build this graph through core components easily connected to existing standardized metadata standards and easily pluggable to domain-specific ontologies. We then presented a tight coupling algorithm which generates the required graph.

In Chapter 3, we focused on the main challenge of handling IR over the proposed tightly coupled semantic graph while providing the users with innovative search results i.e., augmented search results with meaningful context including both the structural and the domain-specific dimensions of a heterogeneous document corpus.

We identified the following sub-challenges: (i) providing relevant granularity levels of the documents, (ii) providing relevant inter and intra-document dependencies, and (iii) providing a meaningful context for query answers. We reviewed existing IR models including traditional IR models and SIR models and the variety of approaches and systems that implement them. To our knowledge, none of the existing models addressed these challenges combined.

We first proposed a novel data structure for query answers based on well-defined sub-graphs, which we call Hybrid Molecules, extracted from a tightly coupled semantic graph. The Hybrid Molecules bring in a core information together with helpful contextual information of the documents including both the structural and the domainspecific dimensions. We then provide a comprehensive query processing pipeline, entitled HM Query Processing, on the basis of the proposed hybrid molecules. Its main purpose is to generate a ranked list of relevant hybrid molecule-based query answers w.r.t. the user's natural language query. We mainly focused on the Search and Ranking modules which consist of: HM_CSA, a novel graph-based search algorithm that generates the hybrid molecules from the tightly coupled semantic graph, and Weight Mapping functions which score the hybrid molecules conveniently.

In Chapter 4, we focused on the experimental evaluation study that assesses the main contributions of Chapter 2 and Chapter 3 in the context of the AEC industry.

We first introduced the implemented Java-based prototype with its two dedicated modules. The first one, entitled LMDR Annotator, automatically annotates a given heterogeneous document corpus and generates the tightly coupled semantic graph. The second one, entitled HM Query Processor, uses the generated graph to provide a ranked list of Hybrid Molecule-based answers in response to a user's natural language query. We then presented a set of experiments conducted on construction projects, provided by Nobatek/INEF4. On the one hand, we compared the representation of a heterogeneous document corpus based on LinkedMDR ontology with alternative data models for document representation. We then tested the automatic annotation capabilities of LMDR Annotator. On the other hand, we evaluated the search results provided by HM_SIRP w.r.t. ground truth provided by users in the AEC industry.

The experiments showed promising results that motivate us in further extending the implementation of the prototype and investigating its efficiency for an adoption in real world projects.

Future Works

There are several future works we can adopt to further improve (i) the contributions of this thesis, and (ii) their validation process. Some of these works are already in progress (mainly works mentioned in Sect 5.2.5, and Sect 5.2.6).

Extending LinkedMDR

LinkedMDR ontology was introduced in Chapter 2 as one of the main contributions.

Future improvements of LinkedMDR include, but are not limited to:

• Integration of descriptors for audio and video (e.g., from MPEG-7 [START_REF]The Moving Picture Experts Group[END_REF], ID31 , etc.) since the current version (lmdr-v1.owl) mainly focuses on the selection of metadata descriptors for the image and the text;

• Definition of semantic rules (using SWRL2 ) to empower its reasoning capabilities since the current version involves only semantic characteristics for data properties (e.g., Transitive, Symmetric, Inverse properties, etc.) generating inferred relations when the reasoner is started.

Defining Properties and Operations on the Hybrid Molecules

The Hybrid Molecules were introduced in Chapter 3 as one of the main contributions. A formal definition was provided and exploited by HM_CSA, the graph-based search algorithm, in order to construct hybrid molecule-based query answers.

However, the obtained hybrid molecule-based query answers still have some shortcomings such as a huge number of composing nodes, and a great number of overlapping components between them, which can add noises on the presentation of the results). We believe that, defining properties on the hybrid molecules (e.g., connectivity to other molecules) and operations (e.g., merge overlapping molecules' components and keep the core of the molecule that has the higher connectivity) would improve their quality and provide a better structure for query answers.

Improving the Query Processing

HM Query Processing was introduced in Chapter 3 as a novel pipeline for IR relying on the Hybrid Molecules. We focused on HM_CSA algorithm and the Weight Mapping functions as the main contributions for the search and ranking modules respectively. Future improvements of HM Query Processing include, but are not limited to:

• Adopting advanced disambiguation techniques [START_REF] Tekli | Building semantic trees from XML documents[END_REF] in the query interpretation module which can leverage initial start nodes for HM_CSA algorithm;

• Optimizing the search module by either adopting search parallelism strategies to construct the Hybrid Molecules from the graph, or preparing clusters of hybrid molecules before the search is started. The latter can be easily applied since the Hybrid Molecules are defined regardless of the context of the search process;

• Including users relevance feedback techniques [START_REF] Haines | Relevance feedback and inference networks[END_REF] to improve the search results. In other words, users' feedbacks launch another iteration of HM_CSA algorithm with modified initial weights for start nodes.

Improving the Prototypes

LMDR Annotator and HM_SIRP were introduced in Chapter 4 for testing the automatic annotation process of a heterogeneous document corpus based on LinkedMDR and IR based on the proposed HM Query Processing pipeline respectively. Due to development requirements in terms of time and resources, we focused on specific modules when implementing these prototypes. Future improvements of the prototypes include, but are not limited to:

• Automatically generating more LinkedMDR instances. For instance, so far, the Automatic Semantic Annotation module of LMDR Annotator generates, from textual context, inter and intra-document references. However, some references could be ambiguous and could implicitly refer to documents or parts of documents. We believe that using advanced disambiguation techniques [START_REF] Tekli | Building semantic trees from XML documents[END_REF]) could help resolving the ambiguity of such references;

• Improving the presentation module for a more user friendly display of the hybrid molecule-based query answers. For instance, the structural-based and domain-specific contextual information of a query answer are still visualized using NavigOwl java-based tool integrated in HM_SIRP prototype. We believe that more display strategies should be adopted in order to improve the presentation of the results and help the non computer expert users to better interpret them. This could be done by exploring state of the art regarding the display of SERP-like results adopted by current search engines and their impact on end users [START_REF] Bajpai | An Estimation of User Preferences for Search Engine Results and its Usage Patterns[END_REF].

Extending the Experiments

Chapter 4 focuses on experimental evaluation showing the power of the annotation of a document corpus based on LinkedMDR, and the effectiveness of LMDR Annotator and HM_SIRP prototypes. In order to further validate our contributions in the context of real-world applications, it is crucial to:

• Evaluate the efficiency of LMDR Annotator in terms of time and memory resources;

• Evaluate the efficiency of HM_SIRP prototype in terms of time and memory resources;

• Compare the richness of the hybrid molecule-based query answers w.r.t. the state-of-art in IR in order to quantify the impact of the contextual information of the results on the user's experience, especially regarding the time needed by the users to interpret and track the results;

• Apply the conducted experiments in the context of other domains (e.g., the medical domain) using another external domain-specific ontology or data model (e.g., MeSH3 ) for LinkedMDR's Pluggable Domain-specific Layer.

On-going Projects

The contributions of this thesis will be integrated and further explored in the following two projects. HM_SIRP prototype would be further enhanced and integrated in the project to help in searching for HVAC data from textual related documents. This is to provide

BIM4REN European Project

I

  am immensely grateful to Nobatek/INEF4 which co-funded my thesis. Without its aid and support, it would not be possible to conduct this research. Thank you Mr. Frederic Betbeder, Dr. Pascale Brassier, Dr. Christophe Cantau and Dr. Pierre Bourreau for helping me promote my research in useful industrial applications. I would like to thank the departments of innovative services and technologies for their help in the data collection and the experimental evaluation processes. Special thanks to Mr. Fabian Bertocchi. I am particularly grateful for the close assistance given by Dr. Gilbert Tekli during the first year of the thesis. It is with pleasure that I acknowledge his efforts and contributions. Discussions with Dr. Joe Tekli have been illuminating. I have greatly benefited from his knowledge and feedback. I thank the IUT Bayonne for permission to use the materials and facilities. Special thanks to Prof. Philippe Aniorte and Dr. Philippe Lopisteguy for their kindness and support. I appreciate the help of Benoit Coumy-Casteret in the implementation of the prototype. I would like to acknowledge the moral and emotional support of all my colleagues in Nobatek and LIUPPA laboratory, especially Lara Kallab, Elio Mansour, Karam Bou Chaaya, Fawzi Khattar and Joelle Céméli.

  vi I am thankful to my friends in Lebanon, France and abroad for their limitless love, support and availability in the hard moments. Thank you Clea Abi Khalil, Khouloud Salameh and Eliana Raad.

I

  am deeply indebted to my beloved family who has never left me alone despite the distance. Thank you Mom and Dad for supporting me and giving me wings to fly and follow my dreams. Without your drive and support, I might not be the person I am today. Thank you Sister and Brother for being by my side whenever I need.

Fig. 3 .

 3 1. . . . . . . . . . . . . . . . . . . . . 3.5 Extract of a sub-graph of G δ 1 involving hybrid information with its contextual domain-specific information and structural-based one. . . . 3.6 Example of a Hybrid molecule m 1 extracted from G δ 1 depicted in Fig. 3.4. 3.7 Example of a Hybrid molecule m 2 extracted from G δ 1 depicted in Fig. 3.4. 3.8 Example of a Hybrid molecule m 3 extracted from G δ 1 depicted in Fig. 3.4. 3.9 Example of a Hybrid molecule m 4 extracted from G δ 1 depicted in Fig. 3.4. 3.10 Example of a Hybrid molecule m 5 extracted from G δ 1 depicted in Fig. 3.4. 3.11 Example of a Hybrid molecule m 6 extracted from G δ 1 depicted in Fig. 3.4. 3.12 A Hybrid Molecule (HM)-based pipeline for query processing in IR. .3.13 Example on Algorithm 8: Appending components from each existing hybrid molecule m k ∈ M to a newly created molecule m new . . . . . . . 3.14 Example on Algorithm 9 -Case 1: Appending neighboring edge e ij to each existing hybrid molecule m j ∈ M j . . . . . . . . . . . . . . . . . . . 3.15 Example on Algorithm 9 -Case 2: Appending neighboring edge e ij to each existing hybrid molecule m i ∈ M i and m j ∈ M j . . . . . . . . . . . . 3.16 Example of an output provided by CSA on the graph G δ 1 of Fig. 3.4. . . 3.17 Example of hybrid molecule-based output provided by HM_CSA on the graph G δ 1 of Fig. 3.4. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.18 Contributions of firing nodes in the calculation of the weight of a neighboring node over spread iterations at different points of time. . . 3.19 Hybrid molecules m 1 , m 3 and m 4 together with their weighted components at termination point of Algorithm 7. . . . . . . . . . . . . . . . 4.1 Overview of the different sub-modules of LMDR Annotator. . . . . . . 4.2 Screen-shot of the current version of LMDR Annotator. . . . . . . . . . 4.3 Example of automatic generation of DC metadata on a PDF document describing general technical specifications. . . . . . . . . . . . . . . . . 4.4 Extract of the XML output of Oxgarage web service for the automatic generation of TEI metadata on a WORD document describing technical specifications regarding the Exterior Carpentry. . . . . . . . . . . . . 4.5 Extract of the XML output of MPEG-7 Visual Descriptors library for the automatic generation of MPEG-7 metadata on a JPG photo capturing on-site construction works. . . . . . . . . . . . . . . . . . . . . . . . 4.6 Example of a pre-define rule in a JAPE Transducer. . . . . . . . . . . . . 4.7 Extract of the generated annotation sets describing inter document references using the GATE API. . . . . . . . . . . . . . . . . . . . . . . .xxi 4.8 Extract of the generated domain-specific annotation sets using GATE for automatic semantic annotation. . . . . . . . . . . . . . . . . . . . . . 4.9 Extract of RDF file describing G δ m related to a heterogeneous document corpus δ m involving 8 construction related documents. . . . . . . 4.10 Extract of RDF file illustrated in Fig. 4.9 with a zoom in describing LinkedMDR document instance. . . . . . . . . . . . . . . . . . . . . . . . 4.11 Example of the desired HM Query Processor's GUI. . . . . . . . . . . . .

  The first commercial software to implement BIM was ArchiCAD of Graphisoft 7 company. Although many Computer-Aided Design (CAD) vendors implement BIM in their proprietary software solutions, there are initiatives towards a universal approach, such as the openBIM8 launched by the buildingSMART alliance9 . The latter aims at ensuring interoperability for the use of BIM across several leading software vendors. Fig.1.2 shows statistics 10 regarding the rates of BIM adoption in construction projects in some European countries, where France stands in the 3rd position.

FIGURE 1 . 2 -

 12 FIGURE 1.2 -Rates of BIM adoption in some European countries.

7

  http://www.graphisoft.com/ 8 https://openbim.fr/openbim/ 9 https://www.buildingsmart.org/ 10 Source: BIM In Motion, April 2018, http://biminmotion.fr/ ladoption-du-bim-en-europe-la-france-en-troisieme-position/.

  On the basis of what we presented in previous sections regarding the unresolved semantic information handling issues to enable the full potential of Industry 4.0 in general (See Sect 1.1) and more specifically Construction 4.0 (See Sect. 1.2), we present the following motivating scenario.

FIGURE 1 . 3 -

 13 FIGURE 1.3 -Motivating Scenario illustrating a large-scale multidisciplinary project in the era of Industry 4.0.

Fig. 1 .

 1 Fig. 1.3 illustrates a large scale and multi-disciplinary project involving a large amount of unstructured information dispersed across various heterogeneous documents i.e., documents having different formats, different content, and structure.These documents, although generated from different sources, are interconnected as they have strongly related information (whether related topics, complementary information) that is all together required for the progress of the project. The entities generating these documents are actors of the projects, who have different expertise and intervene in any stage of the project life-cycle.

FIGURE 1 . 4 -

 14 FIGURE 1.4 -Application of the motivating scenario of Fig. 1.3 to the AEC industry in the era of Construction 4.0.

FIGURE 1 . 5 -

 15 FIGURE 1.5 -An overview of the proposed framework:FEED2SEARCH.

Figure 1 . 5 ,

 15 Figure 1.5, FEED2SEARCH is made of three interconnected layers where the upper layers use data and services provided by the lower layers:

  .when you connect data together, you get power." -Tim Berners-Lee

  2.1 illustrates some of the various heterogeneous documents 1 that are involved in a given construction project: d 1 and d 5 describe technical specifications of the building; d 2 describes thermal properties; d 3 describes acoustic properties; d 4 is an excerpt of a technical drawing related to the ground floor; and d 6 is an image depicting a material pattern used along the construction process.

FIGURE 2 . 1 -

 21 FIGURE 2.1 -Example of heterogeneous documents exchanged within a construction project.

Chapter 2 .

 2 Towards a Collective Knowledge Representation of a Heterogeneous Document Corpus

  different granularities of data (e.g., collection, image, video, audio, segment) and different areas (content description, management, organization and navigation). The three main standardized components of MPEG-7 are: Descriptors (Ds), Description Schemes (DSs) and Description Definition Languages (DDL). While the MPEG-7 Ds are representations of features, the MPEG-7 DSs support complex descriptions and specify the structure and the semantics of the relationships among its constituents: Ds and DSs [84]. The MPEG-7 DDL is a standardized language based on XML schema. It allows the extension of existing Ds and DSs as well

  The following are examples on well-known ontologies and knowledge bases: COMM -The Core Ontology for MultiMedia (COMM)[4] is an MPEG-7 compliant ontology, designed to facilitate multimedia annotations. Although it is not aligned with the XML Schema of the MPEG-7 standard, COMM covers its most important parts, specifically the structure of the media and the content of multimedia documents, while providing more formal semantics. It is designed based on the Descriptive Ontology for Linguistic and Cognitive Engineering (DOLCE) as a foundational ontology, and two Ontology Design Patterns (ODPs): the Descriptions & Situations (D&S) and the Ontology of Information Objects (OIO) formalizing contextual knowledge and information objects respectively. A Java API is provided in order to translate the objects of the MPEG-7 classes into instances of the COMM concepts. M3O -The Multimedia Metadata Ontology (M3O) [83] aims to provide abstract pattern-based models for multimedia metadata representation. It is centered on the formal upper-level ontology DOLCE+DnS Ultralight (DUL) of which it provides three specialized patterns (Description and Situation (D&S), Information and Realization, and Data Value) together with two other patterns (Annotation and Decomposition). M3O is aligned with several ontologies such as COMM, Media Resource Ontology and the EXIF standard. MediaOnt -The Media Resource Ontology (MediaOnt) is developed by the W3C Media Annotation Working Group [102] for describing media resources, specifically images, video and audio fragments. It is made of a core vocabulary comprising a set of descriptive properties covering identification, creation, content description, rights, relational, distribution, fragments and technical properties. MediaOnt is

FIGURE 2 . 2 -

 22 FIGURE 2.2 -Overview of the our Tight Coupling Approach

FIGURE 2 . 3 -

 23 FIGURE 2.3 -Overall schema architecture of LinkedMDR ontology.

FIGURE 2 . 4 -

 24 FIGURE 2.4 -Overview of LinkedMDR Core layer.

Fig. 2 .

 2 Fig. 2.4 shows the overall schema of the core layer and its connections to elements of other layers. A detailed documentation on LinkedMDR core concepts and relations

FIGURE 2 . 5 -

 25 FIGURE 2.5 -Example of LinkedMDR Standardized Metadata sublayer dedicated to DC standard.

FIGURE 2 . 6 -

 26 FIGURE 2.6 -Example of LinkedMDR Standardized Metadata sublayer dedicated to TEI standard.

FIGURE 2 . 7 -

 27 FIGURE 2.7 -Extract of relations between concepts from TEI standard in the corresponding LinkedMDR Standardized Metadata sub-layer.

FIGURE 2 . 8 -

 28 FIGURE 2.8 -Example of LinkedMDR Standardized Metadata sublayer dedicated to MPEG-7 standard.

Chapter 2 .

 2 Towards a Collective Knowledge Representation of a Heterogeneous Document Corpus

  2.9). This way, only two main steps are required at this stage in order to adjust LinkedMDR to a specific domain: (1) creating a sub-concept of Domain describing the application at hand (e.g., Construction for the AEC industry, Medicine for the medical domain), and (2) creating a sub-concept of the previously created concept (e.g., IFC as a sub-concept of Construction) that generalizes concepts of one or multiple external domain-specific ontologies. By means of inference rules, these concepts will be related to sub-concepts of Object (i.e., Document, Media and MediaComponent).

FIGURE 2 . 9 -

 29 FIGURE 2.9 -Pluggability of a domain-specific layer in LinkedMDR.

Fig. 2 .

 2 Fig. 2.3 shows an example of the pluggable domain-specific layer where we integrated a simplified version of the ifcOWL [18] ontology. Fig. 2.10 shows another example of a domain-specific layer where we migrated into the medical domain and plugged in a portion of the Medical Subject Headings (MeSH) RDF schema 18 .

FIGURE 2 . 10 -

 210 FIGURE 2.10 -Example of LinkedMDR application in the medical domain.

  the subset of domain-specific edges where an edge e d ∈ E d represents an instance of a relation r ∈ R in O D .

Figure 2 . 1 (

 21 11 shows an extract of a tightly coupled semantic graph representing the heterogeneous document corpus depicted in the motivating scenario of Sect. 2.See Fig. 2.1), where each node and relation correspond respectively to instances of LinkedMDR concepts and relations of the three different layers. In the following, we demonstrate how this example of graph satisfies the challenges we mentioned in Sect. 2.1, regardless of the techniques used to generate it 22 : • d 5 contains a figure which itself is an excerpt of the technical drawing d 4 . This is represented by the following structural-based edges: e s 2 = (d5, d5.div3.fig-ure15) with f Lab (e s 2 ) = "hasPart", and e s 3 = (d4,d5.div3.figure15) with f Lab (e s 3 ) = "includes". It represents an inter-document spatial relation between the two documents d 4 and d 5 . This particularly addresses Challenge 1.1.

FIGURE 2 . 11 -

 211 FIGURE 2.11 -Extract of a tightly coupled semantic graph representing the collection of heterogeneous documents in Fig. 2.1.

FIGURE 2 . 12 -

 212 FIGURE 2.12 -Example of generated structural-based nodes an edges of the tightly coupled semantic graph depicted in Fig. 2.11 following Steps 1 and 2 of Algorithm 1.

FIGURE 2 . 13 -

 213 FIGURE 2.13 -Example of generated domain-specific nodes and edges of the tightly coupled semantic graph depicted in Fig. 2.11 following Steps 3 and 4 of Algorithm 1.

FIGURE 2 . 14 -

 214 FIGURE 2.14 -Example of a generated hybrid edge of the tightly coupled semantic graph depicted in Fig. 2.11 following Step 5 of Algorithm 1.

FIGURE 2 . 15 -

 215 FIGURE 2.15 -Example of generated inferred edges of the tightly coupled semantic graph depicted in Fig. 2.11 following Step 6 of Algorithm 1.

"

  Real search is about providing valuable information when it's really needed to those who are actually looking for it."-David AmerlandThe increased availability of interdependent heterogeneous data generated from different sources is fostering the incorporation of semantic knowledge-based graphs in information management and search applications. One of the utmost challenge remains in searching for relevant information among heterogeneous interdependent documents. One of the major limitations of the existing SIR systems is that they mainly rely on semantic graphs representing lexical and domain-specific information contained in heterogeneous data, without considering the structural-based components of the documents and dependencies between them. We consider that relying on tightly coupled semantic graphs is one step forward towards overcoming this major problem as they handle the representing of such information from the design phase. This Chapter tackles the problem of searching for relevant information from tightly coupled semantic graphs while augmenting the search results with meaningful context including both structural and domain-specific dimensions of a heterogeneous document corpus. We propose a novel data structure for query answers based on well-defined sub-graphs, which we call Hybrid Molecules, extracted from a tightly coupled semantic graph. We also provide a comprehensive query processing pipeline, entitled HM Query Processing, based on the defined hybrid molecules. Its main purpose is to generate a ranked list of the desired hybrid molecule-based query answers based on the user's query. Our main contributions, within the hybrid molecule-based query processing, are located in the Search and Ranking modules and they consist of: HM_CSA, a novel graph-based search algorithm over a tightly coupled semantic graph, and Weight Mapping functions which score the components of the hybrid molecules and rank the query answers conveniently.

Fig. 3 . 1 illustrates

 31 a sample of two interdependent documents from the AEC industry recalling an extract of the motivating scenario presented in Sect. 2.1 of Chapter 2: d 1 (a technical specification document) and d 2 (a material thermal report) related to the same project. Parts of document d 1 describe acoustic and thermal properties (subsection 1 and sub-section 2 of section 7 respectively). Document d2 describes a thermal study which includes details on the solar factors of windows (table 1 of section 2), implicitly described by "SW coefficient 1 ". The reference relation between the two documents shows that section 2 of document d 2 contains information complementary to sub-section 2 of document d 1 .

FIGURE 3 . 1 -

 31 FIGURE 3.1 -Extract of Fig. 2.1: sample documents from the AEC industry.

3 :

 3 Presenting contextualized query answers -Contextual information related to the relevant information covers both dimensions of the documents i.e., the structural-based dimension (e.g., "Page 3" of document d 2 ) and the domain-specific dimension (e.g., the heat transfer coefficient) explained in Chapter 2. The query answer and its contextual information should be presented in a meaningful structure to help the user in interpreting the results and tracking cross-document dependencies, which reduces their efforts, wasted time and errors.

FIGURE 3 . 2 -

 32 FIGURE 3.2 -Example of a contextualized query answer regardless of the display methods.

a

  tightly coupled semantic graph to represent the collective knowledge embedded in a heterogeneous document corpus, we take advantage of such a model to provide a novel data structure for query answers, which we call Hybrid Molecules. The latter consist of hybrid sub-graphs encapsulating domain-specific information coupled with related structural-based information of the documents. The hybrid moleculebased query answers bring in helpful contextual information of the documents improving the search results and reducing users' efforts in tracking and interpreting them. We formally define the hybrid molecule's structure in view of the characteristics of a tightly coupled semantic graph and the definition of a molecule concept in the literature[START_REF] Valle | A first step towards stream reasoning[END_REF][START_REF] Ding | Tracking rdf graph provenance using rdf molecules[END_REF][START_REF] Endris | MULDER: Querying the Linked Data Web by Bridging RDF Molecule Templates[END_REF][START_REF] Galkin | SJoin: A Semantic Join Operator to Integrate Heterogeneous RDF Graphs[END_REF][START_REF] Newman | A scale-out RDF molecule store for improved co-identification, querying and inferencing[END_REF]. We then integrate the notion of hybrid molecules in a query processing pipeline, where users submit their natural language (e.g., plain English text) queries over a heterogeneous document corpus and obtain relevant answers in the form of hybrid molecules. Although we present a bunch of hybrid molecule-based algorithms for each stage of the query processing, we focus on the graph-based search algorithm which generates a list of hybrid molecules, and the weight mapping which introduces weighting functions to rank the molecules conveniently.

  3.2 provides fundamental notions of IR and reviews the related work regarding traditional IR and SIR systems. In Sect. 3.3, we introduce the Hybrid Molecules and formally define them based on a tightly coupled semantic graph's definition. Sect. 3.4 details on the hybrid molecule-based query processing pipeline over a heterogeneous document corpus with a particular focus on the underlying HM_CSA algorithm and Weight Mapping for search and ranking modules respectively. Sect. 3.5 concludes the chapter.

FIGURE 3 . 3 -

 33 FIGURE 3.3 -Classical pipeline for query processing in IR.

  nance and evaluating trustworthiness against RDF data in SW applications. They define it as the finest and lossless connected subgraph decomposition of the original RDF graph based on Functional Properties (FP) and Inverse Functional Properties (IFP), which are specified in a background ontology. Della et al.[START_REF] Valle | A first step towards stream reasoning[END_REF] expand on the same concept of RDF molecules yet for Stream Reasoning applications. The main purpose is to reason, in real time, over ontological knowledge by combining data streams and reasoning techniques. They introduce RDF Molecule Streams to abstract an aggregation of sampled input data streams. An RDF Molecule Stream has a timestamp that denotes the end of the aggregation interval and the logical arrival time of the molecule on the outgoing stream for reasoning. Newman et al.[START_REF] Newman | A scale-out RDF molecule store for improved co-identification, querying and inferencing[END_REF] also use an extended version of the original definition of RDF molecules[START_REF] Ding | Tracking rdf graph provenance using rdf molecules[END_REF] for distributed querying using MapReduce. They propose a MapReduce-based RDF molecule store that decomposes the graph into molecules, uses SPARQL to query the molecules, and merges them to generate search results. In their definition of RDF molecules, they take into account hierarchy and ordering to overcome existing limitations in the decomposition and merging. For instance, the hierarchy feature helps in distinguishing single level triples with two blank nodes. The ordering feature allows for the rapid retrieval of triples.Endris et al.[START_REF] Endris | MULDER: Querying the Linked Data Web by Bridging RDF Molecule Templates[END_REF] introduce RDF Molecule Templates (RDF-MTs) in the context of federated SPARQL query processing over RDF datasets. RDF-MTs are used to describe the structure of RDF datasets. They are then used to bridge between parts of a query to be executed in a federated manner, thus guiding the source selection, query decomposition and optimization. The authors define an RDF-MT as an abstract description of entities belonging to the same RDF Class together with related properties and object properties. Consequently, internal and external links are created between RDF-MTs representing object properties from internal and external RDF datasets respectively. The proposed RDF-MTs improve the performance of the federation process. Galkin et al.[START_REF] Galkin | SJoin: A Semantic Join Operator to Integrate Heterogeneous RDF Graphs[END_REF] rely on the same definition of RDF molecules, however, the intention is to identify semantically equivalent RDF subgraphs. They present SJoin, a semantic similarity join operator to solve the problem of matching semantically equivalent RDF molecules from RDF graphs.

3 . 5

 35 shows an example of an extract of the tightly coupled semantic graph presented in Fig.3.4 with a focus on the hybrid information involved between a domain-specific instance of SolarFactor (i.e., solarfactor1 which label is SW) and a structural-based instance containing the value ("SW = 0.45") of solar factor (i.e., a table cell in the section entitled "2) Characteristics of Window Frames" of d 2 ), together with their domain-specific context (i.e., information related to SolarFactor concept), and their structural-based one (i.e., granularity levels of the documents, references, etc.). Components of this sub-graph throws back to elements of the desired contextualized query answer depicted in Fig 3.2 (See Sect. 3.1).

FIGURE 3 . 4 -FIGURE 3 . 5 -

 3435 FIGURE 3.4 -Extract of a tightly coupled semantic graph G δ 1 repre- senting the two heterogeneous documents in Fig. 3.1.

  the molecule weight mapping that assigns the weight w m ∈ W m to the molecule m ∈ M. The molecule weight mapping f W m is used in the ranking process of query answers. It is detailed in Sect. 3.4.4.3.To sum up, the core of a hybrid molecule holds the molecule's central information as it is where the domain specific knowledge is anchored to a structural component in the document corpus. The rest of the molecule's nodes and edges augments the core with additional relevant information. For instance, in Fig 3.6, e h 1 = (curtainwall4, d1.div7) is the core of the molecule m 1 . d1.div7 contains relevant information on the curtain walls (exterior facades). Other structural-based components (e.g., e s 1 = (d1, sect1)) and domain-specific ones (e.g., e d 1 = (curtainwall4, bldg1)) in V m 1 and E m 1 provide m 1 with further useful information (i.e., d1.div7 is part of document d 1 and bldg1 contains curtainwall4).

FIGURE 3 . 6 -FIGURE 3 . 7 -

 3637 FIGURE 3.6 -Example of a Hybrid molecule m 1 extracted from G δ 1 depicted in Fig. 3.4.

FIGURE 3 . 8 -

 38 FIGURE 3.8 -Example of a Hybrid molecule m 3 extracted from G δ 1 depicted in Fig. 3.4.

FIGURE 3 . 9 -

 39 FIGURE 3.9 -Example of a Hybrid molecule m 4 extracted from G δ 1 depicted in Fig. 3.4.

FIGURE 3 . 10 -

 310 FIGURE 3.10 -Example of a Hybrid molecule m 5 extracted from G δ 1 depicted in Fig. 3.4.

FIGURE 3 . 11 -

 311 FIGURE 3.11 -Example of a Hybrid molecule m 6 extracted from G δ 1 depicted in Fig. 3.4.

FIGURE 3 .

 3 FIGURE 3.12 -A Hybrid Molecule (HM)-based pipeline for query processing in IR.

Step 4 (

 4 line 4): It calls HM_Presentation algorithm (See Algorithm 6) described in Sect. 3.4.2.4. The hybrid molecules M out , which were ranked from the previous step, are now used as input for this algorithm. The main purpose is to transform and present these molecules in SERP-like results for them to be more understandable by the user. The notion of hybrid molecules intervenes in the four steps. Starting from the query interpretation, domain-specific elements are identified from the query, which will then match domain-specific nodes in the graph. More specifically, the matching parts of the graph come down to domain-specific node parts of the cores of possibly relevant hybrid molecules. The search, ranking and presentation phases rely on our proposed definition of hybrid molecules to construct them, rank them and present them conveniently.

Step 1 (• 3 . 4 . 2 . 4 Algorithme 6 :

 134246 lines 1-3): It assigns a score or weight w m to each hybrid molecule m ∈ M in retrieved from the previous stage using rank_params. The HM_Score function uses the weight mapping f W m associated to the hybrid molecule's definition (See Sect. 3.3.2) and is detailed in Sect. 3.4.4. Step 2 (line 4): It ranks the weighted hybrid molecules in descending order following their individual weights and generates a list of ranked hybrid molecules M out . Algorithme 5 : HM_Ranking Inputs : Set of Hybrid Molecules M in ; Set of ranking parameters rank_params. Output : List of ranked hybrid molecules M out . // **STEP 1** scoring the hybrid molecules 1 foreach m ∈ M in do 2 m ← HM_Score(m, rank_params) ; // assigns a score (weight) w m to each molecule m ∈ M in using rank_params 3 end // **STEP 2** ranking the hybrid molecules 4 M out ← rankMolecules(M out ) ; // rank the hybrid molecules in descending order following their individual scores 5 return M out ; Hybrid Molecule-based Presentation This section provides details on the hybrid molecule-based presentation phase (See Algorithm 6). More specifically, it describes the HM_Presentation procedure used by Algorithm 2. HM_Presentation aims at presenting the ranked list of hybrid molecules generated by the previous ranking stage in SERP-like results. This is to translate the hybrid molecule query answers into a more understandable output for non computer expert users (such as the output presented in Fig. 3.2, in Sect. 3.1). Regardless of the GUI behind, Algorithm 6 provides the following strategy in order to present a hybrid molecule m ∈ M in : • Step 1 (line 2): It displays core information i.e., values of the core's structuralbased node e h .v s and the core's domain-specific node e h .v d . Additional information on the structural-based node e h .v s is also provided such as the corresponding document name and a link to it. • Step 2 (line 3): It displays domain-specific contextual information i.e., the context of the core's domain-specific node e h .v d made of connected domain-specific nodes and edges. • Step 3 (line 4): It displays structural-based contextual information i.e., the context of the core's structural-based node e h .v s made of connected structuralbased nodes and edges. HM_Presentation Input : Set of Ranked Hybrid Molecules M in ; Output : Printed hybrid molecules in SERP-like results. 1 foreach m ∈ M in do // **STEP 1** printing core information 2 print(e h ) ; // prints values of core's domain-specific and structural-based nodes // **STEP 2** printing domain-specific contextual information 3 print_ContextD(e h .v d ) ; // prints values and labels of connected domain-specific nodes and edges respectively in m // **STEP 3** printing structural-based contextual information 4 print_ContextS(e h .v s ) ; // prints values and labels of connected structural-based nodes and edges respectively in m 5 end 6 return ; 3.4.3 Hybrid Molecule-based Search by Constrained Spread Activation (CSA) This section presents a novel graph-based search algorithm, HM_CSA, used during the Hybrid Molecule-based Search module (See Sect. 3.4.2.

(line 21 )

 21 . Algorithm 8 provides details on this operation. Given a newly created hybrid molecule m new with e h being its core and a set of previously created hybrid molecules M, the algorithm fetches each hybrid molecule m k ∈ M (line 1). Whenever the structural-based core's node e h .v s is found in the hybrid molecule m k , the algorithm uses σ s , a structural-based selection operator, to only select the structural-based nodes and edges of m k and adds these components to m new (line 2-4). Likewise, whenever the domain-specific core's node e h .v d is found in a hybrid molecule m k , the algorithm uses σ d , a domainspecific selection operator, to only select the domain-specific nodes and edges of m k and adds these components to m new (line 5-7). This results in an updated hybrid molecule m new with related contextual information from possibly appended structural-based and domain-specific elements.

(

  A) Structural-based node v j found in existing hybrid molecules m 3 and m 4 . (B) Updated hybrid molecule m 3 at termination point of Algorithm 9. (C) Updated hybrid molecule m 4 at termination point of Algorithm 9.

FIGURE 3 . 14 -

 314 FIGURE 3.14 -Example on Algorithm 9 -Case 1: Appending neighboring edge e ij to each existing hybrid molecule m j ∈ M j .

Fig. 3 .

 3 15b shows the constructed temporary molecule m temp involving only structuralbased components from m 3 ∪ m 4 ∪ m 5 since e ij = (d2.div2, d1.div7.p1) is a structural-based edge. Fig. 3.15c, Fig. 3.15d, Fig. 3.15e show the updated molecules m 3 , m 4 and m 5 respectively, after m temp has been appended to them.

(

  A) Structural-based nodes v i and v j found in existing hybrid molecules m 5 , and m 3 and m 4 respectively. (B) Temporary hybrid molecule m temp constructed after applying structuralbased selection on m 3 ∪ m 4 ∪ m 5 . (C) Updated hybrid molecule m 3 at termination point of Algorithm 9. (D) Updated hybrid molecule m 4 at termination point of Algorithm 9. (E) Updated hybrid molecule m 5 at termination point of Algorithm 9.

FIGURE 3 . 15 -

 315 FIGURE 3.15 -Example on Algorithm 9 -Case 2: Appending neighboring edge e ij to each existing hybrid molecule m i ∈ M i and m j ∈ M j .

(FIGURE 3 . 16 -

 316 FIGURE 3.16 -Example of an output provided by CSA on the graph G δ 1 of Fig.3.4.

(FIGURE 3 . 17 -

 317 FIGURE 3.17 -Example of hybrid molecule-based output provided by HM_CSA on the graph G δ 1 of Fig. 3.4.

FIGURE 3 . 18 - 3 . 4 . 4 . 3

 3183443 FIGURE 3.18 -Contributions of firing nodes in the calculation of the weight of a neighboring node over spread iterations at different points of time.

  3.19 shows three hybrid molecules m 1 , m 3 and m 4 together with their component nodes' weights from the output of HM_CSA applied on the example described in Sect. 3.4.3.3. Considering α = β = 1:

(

  FIGURE 3.19 -Hybrid molecules m 1 , m 3 and m 4 together with their weighted components at termination point of Algorithm 7.

  (i) formally defining Hybrid Molecules, a novel data structure for query answers based on a tightly coupled semantic graph's definition and the definitions of molecules in the literature, (ii) constructing the Hybrid Molecules progressively throughout a Hybrid Molecule-based Query Processing which involves Query Interpretation, Hybrid Molecule-based Search, Hybrid-Moleculebased Ranking, and Hybrid Molecule-based Presentation, (iii) providing a novel graphbased search algorithm HM_CSA inspired by the CSA theory, which details on the search module, and (iv) Weight Mapping functions that provide edge, node and hybrid molecule weight strategies handling the characteristics of a tightly coupled semantic graph and used in the ranking module. To our knowledge, the Hybrid Molecules are the first query answers providing the user with (i) relevant granularity levels of the documents, (ii) relevant inter and intra-document dependencies, and (iii) contextual information helping the users in interpreting the search results and tracking cross document dependencies. Several experiments were conducted to validate our proposal within real-world applications. These experiments are dedicated to Chapter 4.

  not think; I experimented." -Wilhelm Conrad Röntgen In Chapter 2, we propose a collective knowledge representation of a heterogeneous document corpus, which comes down to a tightly coupled semantic graph generated based on the infrastructure provided by a novel multi-layered ontology, entitled LinkedMDR. In Chapter 3, we propose an innovative IR model which exploits such a graph in a query processing pipeline relying on a novel data structure for query answers, the Hybrid Molecules, and a set of dedicated algorithms. This chapter addresses the validation of these proposals in the context of a domain-specific application, particularly the AEC industry. We implemented a Java-based prototype made of two main modules: an LMDR Annotator and an HM Query Processor.The former automatically annotates a heterogeneous document corpus and generates the tightly coupled graph. The latter exploits the generated semantic graph and provides a ranked list of Hybrid Molecule-based answers in response to a user's natural language query. The two modules provide a means to evaluate annotations of the heterogeneous document corpus and query results separately. The experiments were conducted on construction projects provided by Nobatek/INEF4 in order to demonstrate that the proposed contributions are applicable in real-world applications.

Fig. 4 .

 4 Fig. 4.1 illustrates an overview of the different sub-modules of LMDR annotator, which is based on LinkedMDR ontology implemented in the context of the AEC industry. The user's input is the heterogeneous document corpus δ and the system's final output is the graph G δ . We choose RDF for the serialization of G δ (G δ .rd f ) as it is a widely used and reliable data model for representing a semantic and extensible graph [104].

FIGURE 4 . 1 -

 41 FIGURE 4.1 -Overview of the different sub-modules of LMDR Annotator.

FIGURE 4 . 2 -

 42 FIGURE 4.2 -Screen-shot of the current version of LMDR Annotator.

Fig. 4 .

 4 Fig.4.2 shows a screen-shot of the implemented prototype. For instance, it provides an example of the Automatic Metadata Extraction module where the user uploads a set of heterogeneous documents simultaneously (e.g., an XLSX technical specification document, an on-site JPG image, a PDF document describing the exterior carpentry, etc.). The system automatically checks, for each document, the submodules capable of generating descriptors based on existing standards. However, the user still have the possibility to uncheck them. When he clicks on the Download button, the system automatically downloads XML files describing these documents based on the chosen standards. One document could have more than one XML file describing it. For instance, the first XLSX will have two XML documents, one involving DC descriptors and another one involving TEI descriptors.

  and basic textual content extraction respectively. Metadata harvesting techniques detect META tags either produced by humans or supported by the Software tool at creation or update time of the document [73]. Although these META tags follow several descriptors, we only extract DC compatible metadata [29] (e.g., dc:creator, dc:format, dc:terms:created). As shown in Fig. 4.3a, the output generated by this API is in the form of plain text or XHTML. Thus, LMDR Annotator transforms the selected DC metadata descriptors into an XML format following specifications of DC standard i.e., in DC XML encoding format 4 (See Fig. 4.3b). (A) Extract of Tika API output in form of plain text. (B) Transformed Tika output in DC XML.

FIGURE 4 . 3 -

 43 FIGURE 4.3 -Example of automatic generation of DC metadata on a PDF document describing general technical specifications.

Fig. 4 .

 4 Fig. 4.4 shows an extract of the XML output of Oxgarage web service which generates TEI structural metadata describing the structural decomposition of a word document regardless its language (e.g., French) into a section containing a list and a paragraph.

FIGURE 4 . 4 -

 44 FIGURE 4.4 -Extract of the XML output of Oxgarage web service for the automatic generation of TEI metadata on a WORD document describing technical specifications regarding the Exterior Carpentry.

  tokens), followed by a Document entity (i.e., an entry from the generated list of document information in the OntoGazetteer). LMDR Annotator outputs the generated annotation sets in XML format. An extract of the generated output following the execution of the above JAPE rule is illustrated in Fig. 4.7 where InterDocLink represents an annotation set identified from the following French textual content: "Pour plus de détails, cf. notice thermique".

FIGURE 4 . 6 -

 46 FIGURE 4.6 -Example of a pre-define rule in a JAPE Transducer.

FIGURE 4 . 7 -

 47 FIGURE 4.7 -Extract of the generated annotation sets describing inter document references using the GATE API.

FIGURE 4 . 8 -

 48 FIGURE 4.8 -Extract of the generated domain-specific annotation sets using GATE for automatic semantic annotation.

Fig. 4 .

 4 Fig. 4.1). The latter is responsible for the conversion of an XML file into an RDF file describing instances of LinkedMDR ontology. The advantage of LinkedMDR converters is that, even if we modify the underlying APIs and web services of adjacent sub-modules generating the XML files, the converters remain the same. The conversions do not consist of simple transformations of XML descriptors into RDF instances. They further create the required LinkedMDR core instances that are not generated by any of the existing APIs or web services. These core instances ensure the connection among other instances generated based on the existing descriptors.

FIGURE 4 . 9 -

 49 FIGURE 4.9 -Extract of RDF file describing G δ m related to a heterogeneous document corpus δ m involving 8 construction related documents.

FIGURE 4 . 10 -

 410 FIGURE 4.10 -Extract of RDF file illustrated in Fig. 4.9 with a zoom in describing LinkedMDR document instance.

Fig. 4 .

 4 Fig. 4.9 shows an extract of an RDF file describing a given heterogeneous corpus δ m containing 8 different construction related documents. Fig. 4.10 shows an extract of a zoom in describing a PDF construction related document. One can see that, for the same document, there are LinkedMDR instances generated based on descriptors of the different previous LMDR Annotator sub-modules (e.g., Automatic MetadataGeneration of DC and TEI Metadata). For instance, the LinkedMDR instance describing the PDF document has several document properties: LinkedMDR instances describing general metadata of the document (e.g., dc:title describing the title of the document), and LinkedMDR instances describing text metadata (e.g., tei:div describing a section of the same document).

FIGURE 4 . 11 -

 411 FIGURE 4.11 -Example of the desired HM Query Processor's GUI.

Fig. 4 .

 4 Fig. 4.11 shows an example of the GUI that we are currently working on. The query results are hybrid molecules retrieved following the user's natural language query, in French plain text, regarding the solar factors of windows. For each answer, the user first gets information from the core of the hybrid molecule: the relevant document (e.g., CCTP.pdf ) and the value of the core's structural-based node of the molecule containing information regarding the solar factor (e.g., "Pour éviter les risques de surchauffes dus à la verrière, celle-ci doit avoir un facteur solaire faible (20%)").

FIGURE 4 . 12 -

 412 FIGURE 4.12 -F 2 -scores measuring the effectiveness of LMDR Annotator in annotating corpora δ 2 , δ 3 , δ 4 , and δ 5 .

Fig. 4 .

 4 Fig. 4.13 illustrates the recall (R) values w.r.t. to the total expected LinkedMDR instances per corpus. This is to compare current state of LMDR Annotator in automatically generating LinkedMDR instances with the annotation potential proposed

FIGURE 4 .

 4 FIGURE 4.13 -Recall (R) scores based on the total expected Linked-MDR instances per corpus δ m .

Figure 4 .

 4 [START_REF] Brut | Integrating Heterogeneous Metadata into a Distributed Multimedia Information System[END_REF] shows that the highest average values of F 1 -score for both Query Group 1 (See Figure4.14a) and Query Group 2 (See Figure4.14b) are attained with F = 0.3 and D = 4 (F 1 -score 0.75). The optimal values of the constraint parameters portray a trade off between P and R. For instance, high precision is achieved with higher F values as only the most relevant nodes (with very high activation values) are selected. However, a high F value restricts the spread of the activation in the graph resulting in lower recall values. We also notice that, with the optimal constraint parameters (F = 0.3 and D = 4), HM_CSA attains slightly higher average precision and lower average recall with Query Group 1 when compared to Query Group 2. This is because increasing the concept types of the start nodes ensures that larger portion of the graph is searched but at the cost of increased false positive results. (A) Query Group 1. (B) Query Group 2.

FIGURE 4 . 14 -

 414 FIGURE 4.14 -Average Precision (P), Recall (R), and F 1 -score of HM_CSA considering different values of threshold F and maximum spread distance D for (a) Query Group 1 and (b) Query Group 2.

Figure 4 . 15 shows

 415 average values of MAP per configuration per group.The results show that considering only the contribution of domain-specific nodes (i.e., α = 0, β = 1), Query Group 2 attains a higher average MAP value in comparison to the result obtained when considering only the contribution of structural-based nodes (i.e., α = 1, β = 0). Query Group 1 shows an opposite behavior as it has less concept types for starting nodes, thus it is less influenced by the domain-specific contribution. The highest values of MAP (MAP= 0.62 for Query Group 1, MAP= 0.74 for Query Group 2) are reached when taking into account both the structural-based and domain-specific aspects in the weight calculation of the molecules (i.e., α = 1, β = 1). This further highlights the importance of the hybrid aspect of the molecule.

FIGURE 4 . 15 -

 415 FIGURE 4.15 -Average MAP values of HM_Ranking per α and β configuration per Query Group.

BIM4REN 4

 4 is a European project which aims at providing a digital ecosystem to facilitate the integration of innovative digital tools, compatible with the BIM model, in the energy renovation process of the buildings. It has just been started in October 2018 and is expected to finish in 2022. It involves 23 partners (SMEs, Universities, etc.), including Nobatek/INEF4, from 10 different countries in Europe.

of LinkedMDR Converter: DC to LinkedMDR Bibliography xix List of Figures 1

  .1 Recap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 Extending LinkedMDR . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Defining Properties and Operations on the Hybrid Molecules . . 5.2.3 Improving the Query Processing . . . . . . . . . . . . . . . . . . 5.2.4 Improving the Prototypes . . . . . . . . . . . . . . . . . . . . . . 5.2.5 Extending the Experiments . . . . . . . . . . . . . . . . . . . . . 5.2.6 On-going Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.6.1 BIM4REN European Project . . . . . . . . . . . . . . . 5.2.6.2 E2S Project . . . . . . . . . . . . . . . . . . . . . . . . . .1 BIM throughout the project life-cycle (Source: The BIM Hub, 2018). . . . 1.2 Rates of BIM adoption in some European countries. . . . . . . . . . . . 1.3 Motivating Scenario illustrating a large-scale multi-disciplinary project in the era of Industry 4.0. . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Application of the motivating scenario of Fig. 1.3 to the AEC industry in the era of Construction 4.0. . . . . . . . . . . . . . . . . . . . . . . . . Extract of Fig. 2.1: sample documents from the AEC industry. . . . . . 3.2 Example of a contextualized query answer regardless of the display methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Classical pipeline for query processing in IR. . . . . . . . . . . . . . . . 3.4 Extract of a tightly coupled semantic graph G δ 1 representing the two heterogeneous documents in

	2.13 Example of generated domain-specific nodes and edges of the tightly
	coupled semantic graph depicted in Fig. 2.11 following Steps 3 and 4

A Example 1.5 An overview of the proposed framework: FEED2SEARCH. . . . . . . . 2.1 Example of heterogeneous documents exchanged within a construction project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Overview of the our Tight Coupling Approach . . . . . . . . . . . . . . 2.3 Overall schema architecture of LinkedMDR ontology. . . . . . . . . . . . 2.4 Overview of LinkedMDR Core layer. . . . . . . . . . . . . . . . . . . . . 2.5 Example of LinkedMDR Standardized Metadata sub-layer dedicated to DC standard. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Example of LinkedMDR Standardized Metadata sub-layer dedicated to TEI standard. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Extract of relations between concepts from TEI standard in the corresponding LinkedMDR Standardized Metadata sub-layer. . . . . . . . . 2.8 Example of LinkedMDR Standardized Metadata sub-layer dedicated to MPEG-7 standard. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9 Pluggability of a domain-specific layer in LinkedMDR. . . . . . . . . . . 2.10 Example of LinkedMDR application in the medical domain. . . . . . . . 2.11 Extract of a tightly coupled semantic graph representing the collection of heterogeneous documents in Fig. 2.1. . . . . . . . . . . . . . . . . . . 2.12 Example of generated structural-based nodes an edges of the tightly coupled semantic graph depicted in Fig. 2.11 following Steps 1 and 2 of Algorithm 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . of Algorithm 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.14 Example of a generated hybrid edge of the tightly coupled semantic graph depicted in Fig. 2.11 following Step 5 of Algorithm 1. . . . . . . 2.15 Example of generated inferred edges of the tightly coupled semantic graph depicted in Fig. 2.11 following Step 6 of Algorithm 1. . . . . . . xx 3.1

  4.12 F 2 -scores measuring the effectiveness of LMDR Annotator in annotating corpora δ 2 , δ 3 , δ 4 , and δ 5 . . . . . . . . . . . . . . . . . . . . . . . . . . 4.13 Recall (R) scores based on the total expected LinkedMDR instances per corpus δ m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Evaluation of the existing standards and data models w.r.t. the identified challenges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.1 Models and techniques adopted by existing traditional IR approaches w.r.t. the classical pipeline in query processing. . . . . . . . . . . . . . . 58

4.14 Average Precision (P), Recall (R), and F 1 -score of HM_CSA considering different values of threshold F and maximum spread distance D for (a) Query Group 1 and (b) Query Group 2. . . . . . . . . . . . . . . . . 4.15 Average MAP values of HM_Ranking per α and β configuration per Query Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii List of Tables 2.1 3.2 Models and techniques adopted by SIR approaches w.r.t. the classical pipeline in query processing. . . . . . . . . . . . . . . . . . . . . . . . . 58 3.3 Evaluation of the existing IR and SIR models (considering different approaches) w.r.t. the identified challenges. . . . . . . . . . . . . . . . . 59 3.4 Core information of molecules extracted from G δ 1 of Fig 3.4 together with examples of their structural-based and domain-specific contexts. 68 4.1 Document composition of the 5 test corpora. . . . . . . . . . . . . . . . 107 4.2 Evaluating the conciseness of the existing standards and LinkedMDR in annotating corpus δ 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 4.3 Heterogeneous document corpus δ 6 . . . . . . . . . . . . . . . . . . . . . 114 Chapter 1

• The Hybrid Molecule-based Query Processing Layer -It provides a compre- hensive query processing pipeline, entitled Hybrid Molecule-based Query Pro- cessing, based

  on a novel data structure for query answers, which we call Hy-

	are constructed progressively from the tightly coupled semantic graph, to pro-
	vide users with relevant information w.r.t. their queries. Consequently given
	a user's natural language query (expressed as plain text), this layer provides
	relevant answers in Search Engine Results Page (SERP)-like results on the basis
	of the information provided by the hybrid molecules.
	brid Molecules. The latter intervene in each stage of the proposed pipeline and

  For instance, d 1 , d 2 and d 3 are Word documents, d 4 is a CAD in a PDF format, d 5 is a PDF document and d 6 is a PNG image.

	its section 9.3. d 5 has an external reference to the image d 6 and the technical
	drawing d 4 , which itself has several versions.
	• Challenge 1.2: Representing metadata on the documents, their content, and their
	structure -There are metadata descriptors providing useful generic informa-
	tion on the document as a whole entity (e.g., a given version of the technical
	drawing d 4 ), on its content (e.g., a description on the content of the image d 6 ),
	and on its structure on different depth levels (e.g., section 7 of d 1 , figure 15 of
	d 5 ).
	• Challenge 1.3: Handling advanced information semantics -Semantics should be as-
	sociated with the content of the documents, their relations, and their metadata
	information on the different granularity levels. This helps to smartly reason
	over the collection of documents. For instance, one needs to locate, on the
	finest granularity levels, relevant information regarding the exterior facades
	which is contained in several documents: section 7 of d 1 , section 2.2.11 of d 2
	referenced by d 1 , the related technical drawing d 4 , etc.
	• Challenge 1.4: Handling documents multimodality -Coping with multimodality
	means handling multimedia content but also the different formats the docu-
	ments could have.
	1: Representing various inter and intra-document links -A document d i
	has various relations among its components as well as with other documents.
	For instance, sections of d 1 and d 3 have related topics since they both describe
	the building's exterior facades. Section 2 of d 3 has an internal reference to

TABLE 2 .

 2

1 -Evaluation of the existing standards and data models w.r.t. the identified challenges.

  (Domain-specific Ontology). Given an application domain D, a domainspecific ontology designated as O D (C, R, Lit, A, L, f L ) is the semantic knowledge describing information in D, where:

	Chapter 2. Towards a Collective Knowledge Representation of a Heterogeneous
	Document Corpus
	Definition 2

Definition 1 (Heterogeneous Document Corpus). A heterogeneous document corpus δ is defined as a set of n documents originated from different sources. The structure of the documents does not necessarily follow a common standard. Formally, δ = {d 1 , . . . , d n }. A document d i ∈ δ is characterized by a set of p metadata (e.g., author, format, creation date, etc.) and a set of q media (e.g., text, image, video, or audio), such that d i = {meta 1 , . . . , meta p , med 1 , . . . , med q }. Further, a media med l ∈ d i is made of s media components (e.g., section, paragraph, etc. for the text; still region, object, etc. for the image), such that med l = {medComp 1 , . . . , medComp s }.

  Section 6 of the document d 1 , which describes the sanitary plumbing of the building, is represented by the structural-based node d1.div6. Section 7 of the document d 1 , which describes the exterior facades, is represented by the structural-based node d1.div7. It is possible to associate these sections with their related domain-specific information. This is done through hybrid edges e h 1 = (curtainwall4, d1.div7) with f Lab (e h 1 ) = "isRelatedTo" and e h 7 = (system1, d1.div6 with f Lab (e h 7 ) = "isRelatedTo" linking d1.div7 and d1.div6 to domain-Lab (e h 8 ) = "isRelatedTo" and space1 instance of IfcSpace. At this stage, the semantics associated to the different structural-based and domain-specific nodes, at different granularity levels, allows further inferred relations enriching the linked data graph. For instance, since the document d 4 contains the figure 15 of d 1 , which is related to space1, then d 4 is also related to space1. Hence, a new hybrid edge e h 4 = (space1, d4) is created with f Lab (e h 4 ) = "isRelatedTo". This particularly addresses Challenge 1.3. • d1, d4, d5 are instances of Document representing the technical specification Word document d 1 , the technical CAD PDF document d 4 and the technical specification PDF document d 5 respectively. They have different media content types (text and image) and formats. This particularly addresses Challenge 1.4.

specific nodes curtainwall4 and system1 representing respectively instances of concepts IfcCurtainWall and IfcSystem from the ifcOWL ontology

[START_REF]BuildingSMART Linked Data Working Group (LDWG). ifcOWL Ontology[END_REF]

. Likewise in the document d 5 , d5.div3.figure15 represents a drawing related to the office space of a building floor. This is represented by e h 8 = (space1, d5.div3.figure15) Document Corpus with f

  LinkedMDR ontology. Its final output is the tight coupled semantic graph G δ .The overall process consists of six major steps: Heterogeneous document corpus δ; Domain-specific ontology O D ; LinkedMDR ontology. Output : Tightly coupled semantic graph G δ . 1 G δ ← ∅ ; // initializes graph // **STEP 1** using techniques for automatic metadata extraction and text engineering 2 Output s ← middleware(δ) ; // generates structural-based descriptors of δ as output // **STEP 2** using our tailored converters 3 V s ∪ E s ← middleware(Output s , LinkedMDR) ; // converts standard output into LinkedMDR structural-based instances V s and E s relying on the semantics of the core and standardized metadata layers of LinkedMDR 4G δ ← G δ ∪ (V s ∪ E s ) ;// updates the graph with structural-based nodes and edges // **STEP 3** using techniques for automatic semantic annotations5 foreach v s ∈ V s do 6 if ( f Val (v s )! = null) then Output d ← middleware( f Val (v s ), O D ) ; //generates semantic annotations as output from the non null direct textual content of each generated v s // **STEP 4** using our tailored converters 8 V d ∪ E d ← middleware(Output d , LinkedMDR) ; // converts standard output into LinkedMDR domain-specific instances V d and E d relying on the semantics of the core and domain-specific layers of LinkedMDR 9 G δ ← G δ ∪ (V d ∪ E d ) ; // updates the graph with domain-specific nodes and edges

	• Step 1 (line 2): It uses existing automatic metadata extraction [42] and text en-gineering techniques [64] offered at the middleware indexing level. The former automatically generates metadata information of the document corpus δ (e.g., end end title1 of d 13 14 15 end // **STEP 6** running the reasoner

(See Algorithm 1). Although we give some examples on the techniques used by the middleware, we leave technical details regarding the underlying technologies, APIs and tools for Chapter 4. The algorithm takes as input: (i) a heterogeneous document corpus δ, (ii) a domain-specific ontology O D , and (iii) 4 , div7 and div6 of d 1 ). The latter automatically generates some dependencies encountered in the text based on pre-defined regular expressions (e.g., the cross-reference dependency between p1 of div7 of d 1 and div2 of d 2 encountered in the text of p1: "performances based on section 2.2.11 in the thermal Algorithme 1 : Tight Coupling Inputs : 7 10 foreach v d ∈ V d do // **STEP 5** coupling domain-specific node to its related structural-based node 11 e h ← hybridCoupling(v d , v s , LinkedMDR) ; // creates hybrid edge between v d and v s using the semantics of the core layer of LinkedMDR 12 G δ ← G δ ∪ {e h } ; // updates the graph with previously created hybrid edge 16 G δ ← runReasoner(G δ , LinkedMDR) ; // dynamically creates inferred relations enriching G δ based on semantic rules in LinkedMDR 17 return G δ ;

  The linking process is based on the semantics provided by the core layer of LinkedMDR precisely by the relation isRelatedTo between a concept Domain and Object (See Sect. 2.3.2.1). The generated hybrid edge e h is finally added to the graph G δ (e.g., e h 1 = (d1.div7, curtainwall4) with f Lab (e h 1 ) = "isRe-latedTo", See Fig.2.14).• Step 6 (line 16): It runs the semantic reasoner so that further inferred relations are dynamically added to the graph G δ based on the semantic rules provided by LinkedMDR. For instance, the relation hasPart is transitive, thus given the two edges e s 2 = (d5, d5.div3) with f Lab (e s 2 ) ="hasPart" and e s 2 = (d5.div3, d5.div3.fig15) with f Lab (e s 2 ) ="hasPart", the edge e s 2 = (d5, d5.div3.fig15) with f Lab (e s 2 ) ="hasPart" is dynamically inferred after the reasoner is started (See Fig.2.15). Furthermore, given the two edges e h 9 = (curtainwall2, d5.div3.fig15)

  represent SPARQL queries as query graphs. They answer the query following a subgraph matching approach. The matching is based on find-

	ing, in an RDF graph, sets of connected vertices that are similar to the one rep-
	resenting the query. Their proposal consists of a graph-based SPARQL query en-
	gine, called gStore. Its main purpose is to handle, in a uniform and scalable man-
	ner, SPARQL queries with wildcards 10 and aggregate operators over dynamic RDF
	datasets. Zhong et al. [108] propose an approach for semantic search by matching
	pairs of Conceptual Graphs (CGs): the first one representing the query graph and
	the second one representing the candidate resource graph from web pages describ-
	ing garment shops. A query CG has an entry representing a central word pointed by

the user from the query. Candidate resource CGs are those which entries (concepts representing garments) are mapped to the query CG entry. Hierarchical relations in WordNet are considered for this mapping. The matching algorithm is based on both semantic similarities between central entries of two CGs and semantic similarities between sub-graph pairs of CGs representing relations affiliated to these entries.

TABLE 3 .

 3 1 -Models and techniques adopted by existing traditional IR approaches w.r.t. the classical pipeline in query processing.

	IR	Models & Techniques
	Query	Keywords, Boolean expressions, Vectors of terms
	Interpretation	NLP techniques, Query Modification
	Search	Exact Match (Boolean Retrieval), Best Match (VSM, Probabilistic, Combined Retrieval)

TABLE 3 .

 3 2 -Models and techniques adopted by SIR approaches w.r.t. the classical pipeline in query processing.

TABLE 3 .

 3 3 -Evaluation of the existing IR and SIR models (considering different approaches) w.r.t. the identified challenges.

		Challenge 2.1	Challenge 2.2	Challenge 2.3
		Relevant Granularity Levels	Relevant Inter/Intra-Document Dependencies	Contextualized Query Answers
	Traditional IR Models	Partial
	SIR Models

  the set of hybrid molecules representing sub-graphs obtained from the decomposition function d E h . This function splits the initial graph G δ into molecules whenever a hybrid edgee h ∈ E h is identified, such that each molecule m ∈ M has a unique e h ∈ E h . For instance, d E h (G δ 1 ) = {m 1 , m 2 , m 3 , m 4 , m 5 , m 6 } decomposes the graph G δ 1 , depicted inFig. 3.4, into six different molecules since there are six different hybrid edges identified in G δ 1 (See Fig. 3.6 → Fig. 3.11).• e h ∈ E h is the hybrid edge identifying the molecule m. Also, we refer to e h as the core of the molecule m. We denote by e h .v d ∈ V d the domain-specific node of the molecule's core and e h .v s ∈ V s the structural-based node of the molecule's core.For instance, e h 1 = (curtainwall4, d1.div7) is the core of the molecule m 1 (See. Fig. 3.6) where e h 1 .v d refers to the domain-specific node curtainwall4 and e h 1 .v s refers to the structural-based node d1.div7.

  the subset made of the core edge, domain-specific and structural-based edges forming m and linking nodes v ∈ V m . ∆ m max is the maximum distance in a molecule m that corresponds to the largest of a set of shortest paths from a node v ∈ V m to its closest molecule's core vertex (i.e., e h .v d or e h .v s ), such that:

For instance, e h 1 = (curtainwall4, d1.div7), e d 1 = (curtainwall4, bldg1), and e s 1 = (d1, d1.div7) are examples of hybrid, domain-specific, and structuralbased edges respectively where e h 1 , e d 1 , e s 1 ∈ E m 1 (See Fig.

3

.6).

•

Table 3

 3

.

4

gives further examples for the 6 molecules.

TABLE 3 .

 3 4 -Core information of molecules extracted from G δ 1 of Fig 3.4 together with examples of their structural-based and domainspecific contexts.

	Hybrid Molecules	Core Information	E.g. of Additional Structural-based Information	E.g. of Additional Domain-specific Information
	m 1	e h 1 Information on the curtain wall curtainwall4 in d1.div7	d1.div7 is part of document d 1	curtainwall4 is contained in bldg1
		e h 2
	m 2	Information on acoustic properties	d1.div7.div1 is part of document d 1	(not depicted in Fig. 3.7)
		acoustic1 in d1.div7.div1
	m 3	e h 3 Information on the solar factor solarfactor4 in d1.div7.div2	d1.div7.div2 is part of document d 1	solarfactor4 is property of curtainwall4
		e h 4
	m 4	Information on thermal properties	d1.div7.div2 is part of document d 1	(not depicted in Fig. 3.9)
		thermal1 in d1.div7.div2
	m 5	e h 5 Information on the curtain wall curtainwall1 in d2.div2	d2.div2 is part of document d 2	curtainwall1 has property solarfactor1
		e h 6
	m 6	Information on the solar factor solarfactor1 in	d2.div2.tab1.cell1 is part of document d 2	solarfactor1 is property of curtainwall1
		d2.div2.tab1.cell1

  merges domain-specific edges and nodes of hybrid molecules in M i and M j (by using σ d as dual domain-specific selection operator) and appends e ij .

end foreach m i ∈ M i do m i ← m i ∪ m temp ; //

updates m i by appending new content from m temp end foreach m j ∈ M j do m j ← m j ∪ m temp ; // updates m j by appending new content from m temp end end end return M;

  33 + 0.33 + 0.83 + 1 + 0.29 + 1 + 0.66 + 0.78 + 0.51) + 1 × (1 + 1 + 0.64) 12 × 0.9 = 0.63 w m 3 = 1 × (0.33 + 0.33 + 0.83 + 1 + 0.29 + 1 + 0.66 + 0.78 + 0.51) + 1 × (1 + 1 + 0.64) 12 × 0.8 = 0.56 w m 4 = 1 × (0.33 + 0.33 + 0.83 + 1 + 0.29 + 1 + 0.66 + 0.78 + 0.51) + 1 × (0.46) 10 × 0.9 = 0.56

  33 + 0.33 + 0.83 + 1 + 0.29 + 1 + 0.66 + 0.78 + 0.51) + 0.5 × (1 + 1 + 0.64) 12 × 0.9 = 0.52 w m 4 = 1 × (0.33 + 0.33 + 0.83 + 1 + 0.29 + 1 + 0.66 + 0.78 + 0.51) + 0.5 × (0.46) 10 × 0.8 = 0.54 Since m 4 involves relatively less domain-specific nodes compared to m 1 , it has been less affected when reducing the contribution of domain-specific nodes on the weights of the hybrid molecules.

  We developed LMDR Annotator, which stands for Linked MDR Annotator, within FEED2SEARCH Java-based prototype. Its main purpose is to automatically annotate a given document corpus δ. It generates a tightly coupled semantic graph G δ representing δ. LMDR Annotator has several sub-modules involving services of the

	4.2 LMDR Annotator
	Indexing layer of FEED2SEARCH (See Fig. 1.5 in Chapter 1). It is, in fact, an imple-
	mentation of the tight coupling algorithm that we presented in Sect. 2.3.3.3 of Chap-
	ter 2 (See Algorithm 1). In this section, we first present an overview of the different
	sub-modules of LMDR Annotator (See Sect. 4.2.1), then we provide implementation
	details regarding the underlying technologies, APIs, and web services adopted for
	each of its sub-modules (See Sect. 4.2.2, 4.2.3 4.2.4, 4.2.5, 4.2.6).
	4.2 and Sect. 4.3
	describe technical details regarding the so far implemented sub-modules of LMDR
	Annotator and HM Query Processor respectively. Sect. 4.4 and Sect. 4.5 present ex-
	perimental protocols and results of two different sets of experiments in the context
	of knowledge representation (for Objective 1) and IR (for Objective 2) respectively.
	Sect. 4.6 concludes the chapter.

•

  DC to LinkedMDR is an XSLT processor that transforms DC descriptors into LinkedMDR instances based on the semantics provided by the Core Layer (See TEI to LinkedMDR is an XSLT processor that transforms TEI descriptors into LinkedMDR instances based on the semantics provided by the Core Layer and the Standardized Metadata Layer, more precisely the sub-layer dedicated to

	TEI standard;
	• MPEG-7 to LinkedMDR is an XSLT processor that transforms DC descriptors
	into LinkedMDR instances based on the semantics provided by the Core Layer
	and the Standardized Metadata Layer, more precisely the sub-layer dedicated
	to MPEG-7 standard;
	• xLinks to LinkedMDR is an XSLT processor that transforms GATE annotation
	sets describing inter and inter-document references into LinkedMDR instances
	based on the semantics provided by the Core Layer;
	• Domain to LinkedMDR is an XSLT processor that transforms GATE annotation
	sets describing domain-specific information into LinkedMDR instances based
	on the semantics provided by the Core Layer and the pluggable Domain-Specific
	Layer (See Sect. 2.3.2.3).

Sect. 2.3.2.1) and the Standardized Metadata Layer, more precisely the sublayer dedicated to DC standard (See Sect. 2.3.2.2); 14 A W3C Recommendation, https://www.w3schools.com/xml/xsl_intro.asp •

  developed HM Query Processor, which stands for Hybrid Molecule-based Query Processor, within FEED2SEARCH Java-based prototype. Its main purpose is to retrieve a ranked list of relevant hybrid molecule-based query answers w.r.t. a given user's natural language query. Its architecture conforms to the Hybrid Molecule-based Query Processing Layer of FEED2SEARCH (See Fig 1.5 in Chapter 1). HM Query Processor is the implementation of HM Query Processing algorithms detailed in Chap-

	ter 3:
	• HM_QueryInterpretation for hybrid molecule-based query interpretation mod-
	ule (Sect. 3.4.2.1), which extracts domain-specific concept occurrences from the
	user's natural language query;
	• HM_Search for hybrid molecule-based search module (Sect. 3.4.2.2), which tra-
	verses the RDF graph starting from nodes matching the previously extracted
	domain-specific occurrences and searches for relevant hybrid molecules;
	• HM_Ranking for hybrid molecule-based ranking module (Sect. 3.4.2.3), which
	assigns scores to the extracted hybrid molecules and ranks them in descending
	order;

• HM_Presentation for hybrid molecule-based presentation module (Sect. 3.4.2.4), which displays the ranked list of hybrid molecules and shows, for each hybrid molecule query answer, its core information, its domain-specific and structuralbased context.

TABLE 4 .

 4 1 -Document composition of the 5 test corpora.

	Corpus (δ m )	No. of Documents	Document Formats	Corpus Size (MB)
	m = 1	10	3 docx, 4 pdf, 3 png	20
	m = 2	10	7 pdf, 2 png, 1 jpeg	27.4
	m = 3	17	5 docx, 10 pdf, 2 png	54.8
	m = 4	15	5 xslx, 1 docx, 7 pdf, 2 png	112.3
	m = 5	12	1 xslx, 10 pdf, 1 png	38.2

TABLE 4 .

 4 2 -Evaluating the conciseness of the existing standards and LinkedMDR in annotating corpus δ 1 .

	Annotation Model	No. of Annotated Documents	No. of Annotation Files	Cumulative No. of Annotation Elements	No. of Redundancies	Coverage of Relevant Criteria
	DC [29]	10	10	131	0	21%
	TEI [98]	5	5	807	0	49 %
	MPEG-7 [97]	5	5	495	0	28%
	Baseline	10	20	1433	128	77%
	LinkedMDR	10	1	604	0	100%
	4

.4.3.2 Evaluating the Effectiveness of LMDR Annotator

  We evaluate the effectiveness of LMDR Annotator in automatically annotating corpora δ 2 , δ 3 , δ 4 , and δ 5 as described in Sect. 4.4.2.2. Fig. 4.12 shows F 2 -scores evaluating the outputs of each LMDR sub-module separately (i.e., δ m [LinkedMDR_DC].rd f , δ m [LinkedMDR_TEI].rd f , δ m [LinkedMDR_MPEG7].rd f , and δ m [LinkedMDR_xLinks].rd f ) then their union (i.e., G δ m .rd f excluding the domainspecific part i.e., δ m [LinkedMDR_Domain].rd f ). In general, F 2 -scores for the overall LMDR Annotator's sub-modules range from 0.48 (for G δ 4 .rd f ) to 0.63 (for G δ 2 .rd f ).Looking over the individual annotation sub-modules, F 2 -scores slightly change

from one corpus to another. LMDR Annotator's sub-module generating δ m [LinkedMDR_DC].rd f is, in general, the most effective one since it involves LinkedMDR instances generated from documents' meta-tags (e.g., title, creator, date, format, etc.) which are easy to extract automatically using the Apache Tika API (See Sect. 4.2.2.1). On the other

https://www.ef-l.eu/our-projects/bim4ren/

https://e2s-uppa.eu/en/index.html

The network of connected physical devices which allows the collection and exchange of data.

A mechanism which provides combination and coordination between physical and computational elements.

The concept of using a network of remote servers hosted on the Internet to store, manage, and process data.

Technology platforms that are based on artificial intelligence and signal processing.

https://www.nobatek.inef4.com/

http://le2i.cnrs.fr/

http://main.enerbim.com/en/partners/

https://www.w3.org/community/lbd/

For the sake of simplicity, we only present 6 documents. However, other documents could be also involved such as videos, audios and 3D drawings.

Although other challenges exist (e.g., data confidentiality and security), they remain out of scope.

For instance, ID3 standard describes digital audio files which can contain, in addition to the audio track, related text and/or graphical information.

http://teibyexample.org/modules/TBED01v00.htm

http://dublincore.org/specifications/

http://rhizomik.net/html/redefer/#XSD2OWL

https://ontotext.com/

https://wiki.dbpedia.org/

https://www.wikidata.org/wiki/Wikidata:Main_Page

https://schema.org/

https://itea3.org/project/lindo.html

https://www.buildingsmart.org/

Defined as ISO10303-11 by the ISO TC184/SC4 committee

https://www.nationalbimstandard.org/

http://www.gbxml.org/

http://www.cstb.fr/

Available at http://spider.sigappfr.org/linkedmdr/

MeSH provides a hierarchically-organized terminology for indexing and cataloging of biomedical information such as MEDLINE/PUBmed databases. It is available online at ftp://nlmpubs.nlm.nih. gov/online/mesh/rdf/

We distinguish nodes of the subset V s from those of the subset V d to explicitly differentiate between structural characteristics and domain-specific ones.

We also distinguish between edges of different subsets such as the case for nodes.

SW stands for Short Wavelength shading coefficient and represents the capacity of the glazing to transmit the solar heat inside the building.

Inspired by the general model of IR presented by Belkin and Croft[START_REF] Belkin | Information filtering and information retrieval: Two sides of the same coin?[END_REF].

Words or phrases combined using the standards operators of Boolean logic.

The Bayes Theorem describes the probability of a term appearing in a document, based on prior knowledge of conditions that might be related to the document such as its relevance.

http://lucene.apache.org/

https://www.lemurproject.org/

http://terrier.org/

An electronic lexical database where words are grouped into sets of synonyms (synsets), each expressing a distinct concept. Synsets are interlinked by means of conceptual-semantic and lexical relations.

The latest version of SPARQL, available at https://www.w3.org/TR/sparql11-overview/

BFS searches the graph data structure starting at chosen nodes in the graph of the neighbor nodes at the current depth prior to moving on to the nodes at the next depth level.

Measures the similarity between two concepts based on the number of common relations with other concepts.

For ease of presentation, the indices of the hybrid molecules follow those of their hybrid edges and are conform to the ones used in Fig.3.6 → Fig.3.11.

Available at http://spider.sigappfr.org/linkedmdr/

E.g., the Dublin Core Advanced Generator available at https://nsteffel.github.io/dublin_ core_generator/generator.html, the Editor-Converter Dublin Core available at http://library. kr.ua/dc/dcedituni

Available at https://tika.apache.org/

Guidelines available at http://dublincore.org/documents/dc-xml-guidelines/

E.g., GeneRation Of BIbliographic Data (GROBID) available at https://grobid.readthedocs.io/ en/latest/

Related API available at https://github.com/sebastianrahtz/oxgarage

Recommendations available at http://www.tei-c.org/tools/

Developed by Poznan Super computing and Networking Center, and Oxford University Computing Services for the EU-funded ENRICH project, https://sourceforge.net/projects/enrich-ege/

E.g., Caliph & Emir Java Open source Library available at http://www.semanticmetadata. net/features/, MPEG-7 Feature Extraction Library available at http://www.cs.bilkent.edu.tr/ ~bilmdg/bilvideo-7/Software.html

Available at https://chatzichristofis.info/?page_id=19

Gate Developer 8.1 available at https://gate.ac.uk/download/

Instances of documents are created by any of the LinkedMDR converters' sub-module (e.g., DC to LinkedMDR).

JAPE is a Java Annotation Patterns Engine providing finite state transduction over annotations based on regular expressions.

Available at https://jena.apache.org/

Available at http://home.deib.polimi.it/hussain/navigowl/.

A simple aggregation of the annotations.

For the sake of simplicity, they only pointed missing hybrid molecules' cores.

Standard for MP3 files, http://id3.org/

Semantic Web Rule Language, https://www.w3.org/Submission/SWRL/

An example of plugging in the Medical Subject Headings (MeSH) schema in LinkedMDR was presented in Chapter 2. Theoretical efforts have been done, yet awaiting real-world projects in the medical domain for experimental tests.

 4 https://www.ef-l.eu/our-projects/bim4ren/

https://e2s-uppa.eu/en/index.html

Acknowledgements

My research and dissertation would have been impossible without the presence and support of many people and institutions.

Foremost, I would like to express my sincere gratitude to my advisors Dr. Christian Sallaberry, Dr. Sébastien Laborie and Prof. Richard Chbeir. I am deeply thank-

List of Acronyms

AEC

Architecture, Engineering, and Construction Algorithme 2 : HM Query Processing

Inputs : User's natural language query q; External domain-specific ontology O D ; Underlying background ontology LinkedMDR; Tightly coupled semantic graph G δ describing a given heterogeneous document corpus δ; Set of search parameters search_params; Set of ranking parameters rank_params. Output : List of hybrid molecule-based answers M out printed in a SERP-like format. // **STEP 1** HM-based Query Interpretation 1 I d_out ← HM_QueryInterpretation(q, O D , LinkedMDR) ;

// extracts LinkedMDR domain-specific instances I d_out from the query using O D and LinkedMDR 's domain-specific layer // **STEP 2** HM-based Search 2 M out ← HM_Search(I d_out , G δ , search_params) ; // applies a graph-based search strategy on G δ using search_params and generates a set of possibly relevant Hybrid Molecules w.r.t. the user's query q // **STEP 3** HM-based Ranking 3 M out ← HM_Ranking(M out , rank_params) ; // applies a ranking strategy on the Hybrid Molecule results using rank_params and generates a ranked list of Hybrid Molecules // **STEP 4** HM-based Presentation 4 HM_Presentation(M out ) ; // transforms and prints the Hybrid Molecule-based answers in a SERP-like format 5 return ;

The algorithm takes as input: (i) a natural language query q expressed by a non computer expert user (such as an actor in the AEC industry) in plain text, (ii) the external domain-specific ontology O D (See Sect 2.3.3.1) that was plugged into LinkedMDR's domain-specific layer (See Sect. 2.3.2.3), (iii) LinkedMDR ontology (See Sect. 2.3.2), (iv) a tightly coupled semantic graph G δ generated based on a heterogeneous document corpus δ, an external domain-specific ontology O D and the infrastructure of LinkedMDR (See Sect. 2.3.3), (v) a set of search parameters search_params used to configure the graph-based search strategy, and (vi) a set of ranking parameters rank_params used to configure the ranking strategy of the query answers. The final output is a list of ranked molecule-based query answers presented in a SERP-like format. This is done following four major steps. They are summarized as follows:

• Step 1 (line 1): It calls HM_QueryInterpretation algorithm (See Algorithm 3) described in Sect. 3.4.2.1. The main purpose is to extract domain specific instances I d_out from the user's natural language query q. This is done by using the semantics of O D and the infrastructure provided by LinkedMDR's domainspecific layer. Inputs : User's natural language query q; External domain-specific ontology O D ; Underlying background ontology LinkedMDR. Output : List of LinkedMDR domain-specific instances I d_out . // **STEP 1** using techniques for automatic semantic annotations 1 Output d ← middleware(q, O D ) ; // generates semantic annotations as output from the textual content of the natural language query q // **STEP 2** using our tailored converters 2 I d_out ← middleware(Output d , LinkedMDR) ; // converts the query output into LinkedMDR domain-specific instances I d_out relying on the semantics of LinkedMDR 's domain-specific layer 3 return I d_out ;

HM_Search applies a graph-based search strategy on G δ in order to retrieve relevant hybrid molecules. This is done following two major steps:

// **STEP 2** searching for possibly relevant hybrid molecules in G δ 6 M out ← HM_CSA(V d_in , G δ , search_params) ; // triggers a graph-based traversal algorithm on the graph G δ starting from nodes in V d_in and using search_params to generate a set of possibly relevant Hybrid Molecules 7 return M out ;

Hybrid Molecule-based Ranking

This section provides details on the hybrid molecule-based ranking phase (See Algorithm 5). More specifically, it describes the HM_Ranking function used by Algorithm 2.

Output(v j ) ← normalize(Input(v j )) ; // output of v j i.e., its activation value after normalization function // **STEP 5** Processing Neighbor In general, HM_CSA can be divided into two main parts which are detailed in the remainder of this section: (1) the implementation of the CSA theory (i.e., Algorithm 7 excluding red-highlighted section) with adapted weight mapping (i.e., the blue-highlighted section of Algorithm 7), and (2) the construction and processing of hybrid molecules (i.e., the red-highlighted section of Algorithm 7).

Implementation of the CSA Theory -We use two sets of nodes V in and V out . The former is fed with activated nodes as the spread activation processes through the graph while the second consists of spread nodes i.e., nodes which have activated HM_CSA appends it to existing molecules in M out where neighboring node v j is found (line 24). Algorithm 9 details on this operation. Given the neighboring edge e ij connecting current firing node v i to neighboring node v j , and a set of existing hybrid molecules M, the algorithm checks whether only v j or both v i and v j exist in M (lines 1 and 7, respectively).

generation of the different granularity levels of the image metadata requires computer vision and machine learning algorithms [START_REF] Opencv | [END_REF], which is not the core purpose of the thesis, and (ii) the automatic generation of semantic descriptors still require human intervention.

Among several MPEG-7-based tools 9 , LMDR Annotator uses MPEG-7 Visual Descriptors 10 as it provides a fully automatic technique to generate, from image content files (e.g., JPG, PNG, etc.), MPEG-7 visual descriptors [START_REF]The Moving Picture Experts Group[END_REF] reused in LinkedMDR.

The library is the implementation of the Open Source Java Content Based Image Retrieval Library (Lire) [START_REF] Lux | Lire: lucene image retrieval: an extensible java cbir library[END_REF]

NLP and Text Engineering for Automatic Generation of Inter/Intra-Document References

We focus on the automatic generation of specific types of inter and intra-document dependencies which are cross-references between documents or parts of documents identified from their textual content. These relations are agreed to be frequently encountered by actors of the construction projects and very important in tracking complementary information among several related documents.

We use the General Architecture for Text Engineering (GATE) API 11 as it provides robust techniques for NLP and Text Engineering [START_REF] Cunningham | A framework and graphical development environment for robust NLP tools and applications[END_REF] to generate, from textual content of various document formats (e.g., TXT, PDF, DOCX, etc.), annotation sets based on predefined rules and knowledge resources. Consequently, we consider cross-references as annotation sets identified from the occurrences of pre-defined expressions in the heterogeneous document corpus. For instance, "cf.", "voir", "performances selon", "se référer" are examples of frequently encountered expressions in Ranking 113

Evaluation of the Quality of the Proposed Hybrid Moleculebased Search and Ranking

We conducted several experiments in order to assess the quality of the retrieved hybrid molecule-based query answers using the proposed HM_CSA algorithm and weight mapping functions. This is done using HM Query Processor described in Sect 4.3. To do so, we targeted two main objectives:

Experimental Context

In this section, we describe the users' queries together with the test data used throughout this experimental study for Objectives 2.1 and 2.2.

Queries

We collected 25 queries from Nobatek/INEF4 based on frequently required information searched by actors with different expertise (architects, technicians and engineers) throughout the different stages of real-world construction projects. We divided the queries into two groups:

• Query Group 1: q 1 → q 12 for simple queries i.e., queries firing start nodes of 1 concept type. For instance, q 1 = "Charateristiques des chassis vitrés" fires 1 domain-specific concept type which is IfcCurtainWall;

• Query Group 2: q 13 → q 25 for more diverse queries i.e., queries firing start nodes of 2 or more concept types. For instance, q 13 = "Facteur solaire des baies" fires 2 domain-specific concept types which are SolarFactor and IfcWindow.

Test Data

We generated a tightly coupled semantic graph δ 6 of 30 000 RDF triples over a heterogeneous document corpus δ 6 of 15 documents hand-picked from a real-world construction project in Nobatek/INEF4 (See Table 4.3). The graph was first generated using the implemented part of LMDR Annotator (See Sect. 4.2).

In order to asses the quality of the search and ranking modules regardless of the annotation tool that generates the graph, we manually completed the annotations in order to ensure the best representation of the document corpus.

Evaluation Criterion and Metrics

We choose the effectiveness of the retrieval as evaluation criterion since it is the most renowned in the evaluation of the retrieval results of an IR system [START_REF] Manning | Introduction to information retrieval[END_REF]. In this section, we present the different metrics we use in order to evaluate the effectiveness in two different scenarios: (i) regardless of the order of the hybrid molecule-based query answers (for Objective 2.1), and (ii) considering their order (for Objective 2.2).

As for the first scenario, we evaluate the effectiveness of HM_CSA algorithm in terms of:

• Precision (P) to identify the number of relevant hybrid molecules among the retrieved results;

• Recall (R) to identify the number of relevant hybrid molecules that are retrieved among the total number of expected relevant results;

• F 1 -score to evaluate the harmonic mean of P and R, which is the most used in the evaluation of IR results [START_REF] Manning | Introduction to information retrieval[END_REF]. the number of wrongly retrieved hybrid molecules (false positives), and c the number of hybrid molecules that are not retrieved although they are relevant (false negatives). F 1 -score is calculated as follows:

Where,

• q is a user query;

• Q is the total number of users' queries;

• AveP(q) is the average precision of query q, such that:

with P(k) being the precision at rank k, rel(k) equal to 1 if the kth hybrid molecule is relevant and 0 otherwise.

Note that, for the assessment of the relevance of a hybrid molecule used in the calculation of the above metrics, we relied on users' judgments. For each query, we asked 12 users (who did not take part in the queries formulation), highly involved in the construction project from which the documents of the corpus δ 6 were taken, to provide a score for each answer (1 for relevant and 0 for not relevant) independently from each other. Afterwards, the users were asked to validate the judgments collectively. These users also provided the false negative hybrid molecules 18 for each query.

Experimental Scenarios and Results

In this section, we present the results of two conducted experiments considering the context defined in Sect. 4.5.1, and the evaluation criterion and metrics defined in Sect. 4.5.2.

Evaluating the effectiveness of HM_CSA

We evaluate the effectiveness of HM_CSA for the 25 given queries in terms of P, R

and F 1 -score (Objective 2.1). We study the impact of constraint parameters, mainly the firing threshold F and the maximum spread distance D, on the query answers.

To do so, we consider 3 different values for F (0.1, 0.3, and 0.5) and 4 different values for D (2,4,6, and 8). This resulted in 12 run configurations for each query execution. We further examine the influence of the diversity of the queries considering empty BIM models (generated from a 3D scan tool during the renovation phase of the building) with the missing information.

E2S Project

The Energy Environment Solutions (E2S) project 5 aims at providing a generic information system that can be configured, on demand, by any organization in or-