Karolina Bataityte
email: k.bataityte@londonmet.ac.uk

Vassil Vassilev
email: v.vassilev@londonmet.ac.uk

Olivia Jo Gill
email: o.gill@londonmet.ac.uk

Ontological Foundations of Modelling Security Policies for Logical Analytics

Keywords: Security Policies, Modelling, Ontologies, Knowledge Representation, Situations and Actions, Frame Problem

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

In recent years there has been an increase in the interest of analysing the logical vulnerability and the security policies of cyber systems. The security policies cover a wide range of situations: how to prevent unauthorized access to the information, secure the operations, control the transactions, neutralize malicious activities, etc. Any gaps or inconsistencies in the security policies can open the door for logical vulnerabilities and leave the system exposed [START_REF] Jizba | Web logic vulnerability[END_REF]. Logical analysis of the vulnerability requires modelling of the online operations with sufficient background information to cover the security -user credentials and profiles, needed for identification, authentication and authorisation, communication channels, physical connections and logical sessions for operations and transaction control, threat intelligence for security protection, etc. Our approach for addressing it is to represent the domain knowledge in the ontological model and to formulate the security policies as a system of rules, so that we can analyse them formally. For this purpose, we developed a theory of Situations and Actions in Description Logic (DL) and modelled the Security Policies in Clausal Logic (CL), which can be implemented using the standard languages of Semantic Web -Ontology Web Language (OWL) [START_REF] Hitzler | Foundations of Semantic Web Technologies[END_REF] and Semantic Web Rule Language (SWRL) [START_REF] Lawan | The semantic web rule language expressiveness extensions-a survey[END_REF]. We can model dynamic changes and synchronous actions with different security events asynchronously.

The paper is organized as follows. In Section 2 we will present the overall methodology which we follow. In Section 3 we will present logical foundations. In Section 4 we will introduce the ontological level. Section 5 will consider the security policies as rules on logical level. Section 6 we will conclude the paper and comment on the security policy analysis on Analytical Level.

Methodology

There are a number research projects being conducted that are developing ontological models for the different security purposes. They each use their own vocabulary, however, they use the same semantic web technologies (e.g., [START_REF] Pereira | An ontology based approach to information security[END_REF], [START_REF] Granadillo | An ontology-driven approach to model siem information and operations using the swrl formalism[END_REF]). We separate the model of the world (ontological level ) from the model of the policies which govern the changes in the world (logical level ) and the model of the dynamic changes as a result of decisions (analytical level ) (see Fig. 1). For each of the three levels we will use different formal systems, suitable for modelling of an aspect of the problem in a manner, similar to the infamous "layered cake" of the Semantic Web [START_REF] Passin | The Explorer's Guide to the Semantic Web[END_REF].

Ontological Level

Logical Level

Analytical Level

the world as such governing policies decision making

Fig. 1. Multi-level Model for Analysis

The Ontological Level models the world using the vocabulary presented in Section 4.1. The conceptualization is similar to the famous situation calculus (SitCalc) [START_REF] Mccarthy | Some philisophical problems from the standpoint of artificial intelligence[END_REF], but formulating it using the language of DL makes it more "objectoriented" and allows for a new solution of the frame problem [START_REF] Reiter | Knowledge in Action: Logical Foundations for Specifying and Implementing Dynamical Systems[END_REF]. Using DL on this level allows us to implement the model entirely using OWL.

The Logical Level models the policies, captures constraints and completeness. It reflects the expert knowledge in the domain, which can be formulated as logical rules in CL and can be represented in computer format using SWRL.

The Analytical Level will deal with the analysis of the policies on a directed graph, considering the situations as nodes and the actions as edges however it is beyond the scope of this paper and is left for the next publication.

Logical Foundations

For developing of the theory of situations and actions we consider DL called ALC [START_REF] Tsarkov | Efficient reasoning with range and domain constraints[END_REF] which is not the most expressive but is expressive enough to support our needs without being too complicated beyond the necessity. More constructors can be added to extend ALC if the modelling requires it. As we choose DL for comfortable implementation in OWL, similarly we choose CL as we can implement rules in SWRL. The following two logics can be glued together for modelling the domain ontology and the policies within that domain.

Description Logic ALC as a Modelling Language

The syntax and the semantic interpretation is shown in Table 1. The interpretation I is a pair I = ( I , • I ), where I is a non-empty set (domain) and • I is a mapping function [START_REF] Szeredi | The Semantic Web Explained: The Technology and Mathematics Behind Web 3.0[END_REF].

R R I ⊆ I × I ⊥ ∅ Domain(R, C) < a, b >∈ R I → a ∈ C I A A I ⊆ I Range(R, C) < a, b >∈ R I → b ∈ C I ¬C I \C I C D C I ∩ D I C D C I ∪ D I ∀R.C {a ∈ I |∀b.(< a, b >∈ R I → b ∈ C I )} ∃R.C {a ∈ I |∃b.(< a, b >∈ R I ∧ b ∈ C I )}
where C, D are concepts, A is an atomic concept, R is a role.

Given interpretation I in model M with axiom α, we say that M is a model of α under I if M satisfies α, written I |= α. We will be expressing the domain restrictions as ∃R.

C and the range restrictions as ∀R.C [START_REF] Tsarkov | Efficient reasoning with range and domain constraints[END_REF]. By adding domain and range axioms we are able to have a fixed structure of the real world we are modelling without the necessity to use more expressive language or nonstandard semantics.

Clausal Logic and SWRL

In most logical languages it is possible to formulate rules, which are necessary for modelling structural constraints and dynamic changes. We have chosen a version of the first order clausal logic similar to the horn-clause predicate logic because its serialized version SWRL refers directly to the terms of OWL.

SWRL Knowledge Base (K) is defined as follows: K = (Σ, R) where Σ is KB of ALC and R is set of rules. The rule is composed of body and head which is represented as following: body → head. It consists of a conjunctions of atoms which are classes C(i) (concepts in ALC) and object properties R(i, j) (roles in ALC) [START_REF] Lawan | The semantic web rule language expressiveness extensions-a survey[END_REF].

Ontological Level: The Domain Model

The term ontology in a narrow logical sense provides the terminology, which can be used for building the domain model, together with its interpretation in the semantic domain [START_REF] Sãnchez | The Road Toward Ontologies[END_REF]. The cyber security operations require accounting of both static and dynamic semantic considerations, in order to have an adequate and semantically rich ontology for the analysis.

Terminological Vocabulary

In our ontology the semantic domain, , is a non-empty set, split into three disjoint subdomains: Entities, Events and Situations (plural) as Entities ,

Events and Situations respectively. In our theory we will use three terms with predefined meaning: Entity, Event and Situation (singular), which will be three separate taxonomies representing the static model of the world. The interpretation of ALC concepts in the domain are as follows: Entity I ⊆ I Entities , Event I ⊆ I Events and Situation I ⊆ I Situations . Our terminology (Table 2) will also include some predefined roles, one of them is Action (Action I ⊆ I Situations × I Situations ), which can be used as a top of the hierarchy of actions. The ontology can have as many specific named concepts and named roles as needed, (noted as Entityx, Situationy, Evente, Actionz), with the intended meaning and interpretations in the semantic subdomains introduced above in accordance with the syntax and semantics of ALC as presented in Section 3.1. Concepts from three subdomains must be disjoint as follows:

Situation Event ⊥, Situation Entity ⊥, Entity Event ⊥.

(1) On the ontological level we are using the ALC TBox for formulating the terminological axioms and the RBox for the relational axioms, while the ABox will incorporate the assertions later on.

Static Modelling of the World

Here we are defining a fix static structure of the modelling world using terms above. A Situation is a concept, which represents a partial description of the world in a specific moment of time. Two Situation concepts can be connected via Action roles to model the potential change:

∃Action. Situation, ∀Action.Situation (2)
The events are asynchronous activities which are modelled using Event concepts, linked through the predefined role chain in a causal chain (axiom 3). The intended meaning of Event is to represent a real-world events which can occur in the situations through the predefined role occur-in with domain Event and range Situation (axiom 4). This way we can formulate security policies with regard to planned and unexpected activities (events), which may or may not happen in the situations.

∃chain.

Event, ∀chain.Event

(3) ∃occur-in. Event, ∀occur-in.Situation (4)
The Entity concepts are used to describe situations and events using the predifined roles from the vocabulary: part-of with domain

It is important to note that the events do not change the situations in our theory, they can only occur in them; the changes can be caused only by actions. So that events are described as asynchronous activities while actions are purely synchronous activities

World Dynamics

In state-based dynamic theories which uses DL, the actions are represented as precondition, occlusion, post-condition triplets [START_REF] Baader | Integrating description logics and action formalisms: First results[END_REF], [START_REF] Chang | A dynamic description logic for representation and reasoning about actions[END_REF]. Unfortunately, there is no easy implementation of such a formalism since it has additional syntactic structure.

We have adopted the view that the dynamic changes are possible only through actions, similar to the original SitCalc from the early days of AI [START_REF] Reiter | Knowledge in Action: Logical Foundations for Specifying and Implementing Dynamical Systems[END_REF]. This logic formalism encounters the infamous frame problem, caused by the propositional treatment of the situations which require them to incorporate their parameters as arguments.

However, in our approach the definition of the actions (as relations between the situations) looks almost identical to SitCalc approach. The partitioning of our ontology has interesting and unexpected characteristics with practical importance for applications. We define the parameters of the actions contextually. In our approach the actions can change the situations only through their parameters, which are entities, but the action parameters are no longer attributed to the actions -they are attributed to the situations which the actions relate instead. This completely eliminates the need for heavy "frame axioms" because the complete absence of any "side effect" of the actions.

If we have TBox T with situations and entities as follows:

T := {Entityx Entity, Situationy Situation} (8)
and Entityx describe Situationy, T is extended as follows:

T := T ∪ {Entityx ∃present-at.Situationy}. (9)
Example 1. Let's consider the situation LoggedIn and the entity U ser. For this scenario the TBox T is as follows:

T := {U ser Entity, LoggedIn Situation, U ser ∃present-at.LoggedIn} Each situation can be described by a number of entities. Since the actions change the situations, they will affect these entities but not directly. So, we can consider the entities which describe all situations in which a given action applies as its input parameters and similarly, entities which describe the situations to which the action leads as its output parameters. NB: not all entities are input and/or output parameters, some of them just describe the situation without being needed for an action. To specify the parameters of all actions, we can create a GBox G as follows:

Definition 1. A GBox G = { Entityx, Actionz , Actionz,
Entityy } is a set of pairs of actions and entities, representing the action parameters where pair Entityx, Actionz is for input parameters and pair Actionz, Entityy is for output parameters.

The action parameters will be important on the Analytical Level since the input parameters are binding the actions, making them executable, while the output parameters are producing the effect, determining the changes in the situations.

In order for an entity to be an input parameter, it must meet the following conditions:

1. ∃Actionz.
Situationx, 2. Entitye ∃present-at.Situationx.

If both conditions hold, we can say GBox G = { Entitye, Actionz }. It can be formalized as the following axiom:

Entitye ∃present-at.(Situationx ∃Actionz. ) [START_REF] Reiter | Knowledge in Action: Logical Foundations for Specifying and Implementing Dynamical Systems[END_REF] which says that Entitye is connected to a Situationx via present-at and there is an Actionz starting at Situationx and leading to another unknown Situation. This gives us the first criteria for analysing the descriptive completeness of the security policies with respect to the possibility of binding the input parameters of the applicable actions to the descriptions of the situations in which they apply.

In order for an entity to be an output parameter, it must meet the following conditions:

1.

∀Actionz.Situationy, 2. Entitye ∃present-at.Situationy.

If both conditions hold, we can say GBox G = { Actionz, Entitye }. It can be formalized as follows:

Entitye ∃present-at.∃Actionz.Situationy [START_REF] Sãnchez | The Road Toward Ontologies[END_REF] which says that Entitye describes Situationy via present-at and Actionz leads to Situationy after it executes.

Example 2. In Fig. 2 we have a scenario which starts in situation Situation1 and finishes in Situation3 after executing Action1 and Action2 . The two actions have parameters amongst the entities which are present in the corresponding situations. In this case G = { Entity2, Action1 , Action1, Entity3 , Action1, Entity4 , Entity4, Action2 }. Amongst the parameters Entity4 is both input and output parameter of Action2. Entity1 and Entity5 simply describe the situations without being needed for actions.

Situation1 Situation2 Situation3 Entity1 Entity2 Entity3 Entity4 Entity5 Entity6 Event1 Action1 Action2 present-at present-at present-at present-at present-at occur-in part-of

Fig. 2. A graphical representation of two-step journey

The ontological considerations we have presented so far can be constructed in any variation of DL. Since such a theory can be serialized directly in OWL, the process of developing the ontology can be done entirely interactively using any standard ontology editor, such as Protégé.

Logical Level: Constraints, Dependencies and Domain Policies

In order to describe the logical characteristics of the model, as well as to represent adequately the domain policies controlling the execution of the actions, we can use axioms, rules of inference and heuristic rules. Although DL and CL, as theoretical base of our framework, have well-defined inference mechanisms for practical purposes, it is more convenient to work with derived inference rules rather than the rules of inference within the underlying logic. In this section we will discuss some derived rules of our framework which allow us to automate this process.

Parameter Binding and Entity Completion

To make sure that our KB is descriptively complete, we need to guarantee that it contains all needed information in the TBox (the ontology model) to match the SWRL rules (the policies) so that the policy rules which prescribe actions actually lead to executable actions. In practice this means that all parameters of the actions in the head of the rules must be bound to the situations in which the rules apply. This can be implemented using an algorithm which uses the ontology in the TBox to check if the parameters of the actions prescribed by the rules are defined.

The following derived rule captures the parameters of various events in the situations to prevent the loss of bindings. It is used to implement a "reasoner" which performs a secondary logical inference according to the following schema:

Entity ∃part-of.Event Event ∃occur-in.Situation ∴ Entity ∃present-at.Situation Derived Inference Rule 1 (Entity Triangulation). Let the following TBox T be given:

T := {Entity ∃part-of.Event, (12a)
Event ∃occur-in.Situation} (12b)

Then the following holds:

T := T ∪ {Entity ∃present-at.Situation}. ( 13
)
Proof. The TBox T holds since it states the domain and range of part-of (12a) and occur-in (12b) roles which satisfy the axioms 5 and 4 respectively. The same concept Event is used as range of part-of (12a) and as a domain of occur-in (12b). Therefore, we can substitute Event in 12a by the right-hand side of 12b to derive Entity ∃part-of.∃occur-in.Situation. As we can see, Entity is connected to Situation via two roles. We know from Section 4.2, this can be done via present-at (axiom 6), therefore, it can be expressed as Entity ∃present-at.Situation (13).

Transitivity of the Roles and Entity Propagation

The next derived rule reflects the abstract "transitivity" of the logical descriptions within one and the same situation. It can be accounted by another "reasoner" which performs secondary inference according to the following schemas against concept Situation or Event:

Entityy ∃describe.Entityx Entityy ∃describe.Entityx Entityx ∃present-at.Situationx

Entityx ∃present-at.Eventz ∴ Entityy ∃present-at.Situationx ∴ Entityy ∃present-at.Eventz Derived Inference Rule 2 (Entity Transitivity). Let the following TBox T be given:

T := {Entityy ∃describe.Entityx, (14a)
Entityx ∃present-at.Situationx} (14b)

Then the following holds:

T := T ∪ {Entityy ∃present-at.Situationx}. ( 15
)
Proof. The TBox T holds since it states the domain and range of describe (14a) and present-at (14b) roles which satisfy the axioms 7 and 6 respectively. The same concept Entityx is used as range of describe (14a) and domain of present-at (14b). Therefore, we can substitute Entityx in 14a by the right-hand side of 14b to derive Entityy ∃describe.∃present-at.Situationx. As we can see, Entityy is connected to Situationx via two roles. Therefore Entityy is connected to Situationx and we can simply rewrite it as Entityy ∃present-at.Situationx (15).

Conceptual Taxonomies and Entity Inheritance

Although the DL allows to automate the subsumption of concepts, we can extend our framework with additional inheritance mechanisms to allow full "parameter inheritance" in the style of object-oriented programming. This is possible because the entities, which are connected to situations or to events, are like the class attributes in object-oriented parlance. It is relatively straightforward to construct algorithmic reasoners which tackle more complex inheritance of entities, along the taxonomic hierarchies of situations and events.

Situationy ∃Situationx

Eventy ∃Eventx Entityx ∃present-at.Situationx

Entityx ∃present-at.Eventx ∴ Entityx ∃present-at.Situationy ∴ Entityx ∃present-at.Situationy Derived Inference Rule 3 (Entity Inheritance). Let the following TBox T be given:

T := {Situationy Situationx, (16a)
Entityx ∃present-at.Situationx}

Then the following holds:

T := T ∪ {Entityx ∃present-at.Situationy}. (17)
Proof. The TBox T holds since it states that Situationy is a sub-concept of Situationx (16a) and Entityx is related to Situationx via present-at (16b). Therefore, Entityx is also related to sub-concept of Situationx, which is Situationy (17).

Frame Problem

In our dynamical model the situations change as a result of the actions. The only way the change from one situation to another situation can affect the descriptions (entities) of the latter situation, is through the output parameters of the actions causing the transition. The specific changes caused by the actions must be specified by the corresponding rules of the security policy. The following two principles allow us to avoid the frame problem by formulating rules according to those principles. They will also guide the changes on the Logical Level. Principle of preservation: Any description of the situations within the domain of the action in terms of input parameters remains unchanged.

Principle of propagation: Any description of the situations within the range of the action in terms of output parameters may change as a result of the action.

Policy Rules

The policies on the Logical Level are rules which link the concepts and roles from the Ontological Level. Such rules have clausal form and can be represented as SWRL expressions (Section 3.2). This makes possible the use of the ontological editors like Protégé for modelling of the policies as well.

The policy rules can be modelled as SWRL rules using different templates which combine Situation, Event, Entity and Action atoms in the body and the head of the rule to serve different purposes -for analysis of the situations, making decisions for continuation of the journey, or responding to events. Two such templates are shown below, which can be finely tuned to the particular need of the analysis.

1. situationi(?sa) ∧ entity k (?ia) ∧ present-at(?ia, ?sa) ∧ ... ∧ actionn(?sa, ?sb) → situationj(?sb) ∧ entity l (?ib) ∧ present-at(?ib, ?sb) ∧ ... 2. situationi(?sa) ∧ entity k (?ia) ∧ present-at(?ia, ?sa) ∧ event l (?ea) ∧ occur-in(?ea, ?sa)∧...∧entity k (?ib)∧part-of (?ib, ?ea)∧...∧actionn(?sa, ?sb) → situationj(?sb)∧ entity k (?ic) ∧ present-at(?ic, ?sb)... In the templates above situationi(?s), entity k (?en), event l (?ev) are SWRL classes (which correspond to ALC concepts), actionn(?sa, ?sb) is an SWRL object property (which corresponds to ALC role) and they have to be adopted to the specific scenario.

Other classes/concepts and object properties/roles do not have to be adopted to the scenario and can be used as it is (present-at(?ib, ?sb), occur-in(?ea, ?sa), etc.)

Detailed Example

In this section we will present a more detailed example of the use of our framework for the analysis of a typical online banking transaction. The fragment was built in Protégé 5.1.0 with FaCT++ 1.6.5 reasoner. WebVOWL 1.1.7 was used for visualization of Fig. 3 and Fig. 4 was created using a drawing tool since software to generate these graphs is still in the development stage. Some specifications such as TBox, RBox and some of the named concepts are omitted for the sake of clarity and brevity. The purpose of this example is to illustrate our framework as well as show the interpretation and understanding of it. Lets consider the case when transaction is requested: we start in the initial situation S T ransactionRequested; then there are three possible events which may or may not happen: E AccountIn Overdraf t, E M axOverdraf tReached, E AccountOverloaded. Reaching the final situation will depend on the policy rules expressed in SWRL. Fig. 3 shows the interpretation fragment of ontological vocabulary from Ontological Level where yellow arrows represents some of the derived inference rules from Logical Level. Fig. 4 visualises SWRL rules on the Logical Level which will be used on the Analytical Level for the analysis. Some of the SWRL rules are as follows: T ransactionAmount(?it) ∧ present-at(?it, ?sa) ∧ E AccountInOverdraf t(?ea) ∧ occur -in(?ea, ?sa) ∧ Overdraf tAmount(?io) ∧ part-of (?io, ?ea) ∧ present-at(?io, ?sa) ∧ A T ransactionApproved(?sa, ?sb) → Balance(?iy) ∧ S T ransactionExecuted(?sb) ∧ present-at(?iy, ?sb) -S T ransactionRequested(?sa) ∧ Balance(?ix) ∧ T ransactionAmount(?it) ∧ present-at(?ix, ?sa) ∧ present-at(?it, ?sa) ∧ E M axOverdraf tReached(?eb) ∧ occur-in(?eb, ?sa) ∧ A T ransactionDeclined(?sa, ?sb) → S T ransactionRejected(?sb) -S T ransactionRequested(?sa) ∧ Balance(?ix) ∧ present-at(?ib, ?sa) ∧

T ransactionAmount(?it) ∧ present-at(?it, ?sa) ∧ E AccountOverloaded(?eb) ∧ occur-in(?eb, ?sa) ∧ A AccountSuspended(?sa, ?sb) → S AccountBlocked(?sb)

Although some of the rules may look too complex, most of the literals in it are type checking conditions which can be eliminated from the formulation by adopting a separate type checking algorithm.

Conclusion and Further Work

In this paper we presented ontological and logical considerations of knowledge representation for security analysis of the cyber systems operating in a workflow manner. As well as the processing of transactions in dynamic systems, which involve synchronous and asynchronous activities such as events and actions have been described. We outlined a multi-level framework for representing the ontology and modelling the security policies which enables analysing of some logical problems such as vulnerability analysis and risk assessment. It is entirely based on the use of standard modelling languages of the Semantic Web, which greatly simplifies the implementation, makes it transparent and efficient. Our framework provides a theoretical basis for solving some of the hard problems in modelling dynamic behaviour such as the infamous frame problem. We utilize the concept of state, to provide a proper distinction between the static characteristics of the situations and the possible side effect of the actions on them. We have a pilot implementation, of the framework, written in Java, which makes use of the APIs for OWL and SWRL available in Jena for processing the ontological representation and the security policies in symbolic form [START_REF] Vassilev | Intelligence graphs for threat intelligence and security policy validation of cyber systems[END_REF]. It allows us to perform various logical analytics related to logical vulnerability, risk assessment and policy validation. We are currently use this framework to cross-channel transaction processing, in digital banking, for preventing social engineering fraud. The semantic and logical considerations discussed above provide the formal ground for formalizing the concepts of accessibility, logical vulnerability and risks. Within our framework this can be done by simulating different scenarios for execution of the actions, under the conditions imposed on the situations and with possibility for events happening in them. Although such an analysis is beyond the scope of this paper, the experiments we conducted using our prototype implementation, have demonstrated that this approach is both transparent and convenient to be used for practical purposes [START_REF] Vassilev | Intelligence graphs for threat intelligence and security policy validation of cyber systems[END_REF].

Currently, we are working on an extension of the framework (to equip it) with risk analysis capabilities, based on the naive Bayesian theory. We are also exploring the potential use of the same framework in other areas, related to workflow control such as production line fault recovery and safety management, like evacuation in the event of fire or other disastrous situations.

Fig. 3 .Fig. 4 .

 34 Fig. 3. Example Visualization of Ontological and Logical Levels

Table 1 .

 1 Syntax and Semantics

		Concepts		Roles
	Syntax	Semantics	Syntax	Semantics
		I

Table 2 .

 2 Vocabulary of the Domain Ontology

	Term	DL Category	Use in modelling	Condition
	Situation	concept	partial static description of the world	axiom 1
	Event	concept	asynchronous activity	axiom 1
	Entity	concept	qualitative descriptor	axiom 1
	Action	role	synchronous activity	axiom 2
	occur-in	role	event occurrence	axiom 4
	present-at	role	situation description	axiom 6
	part-of	role	event description	axiom 5
	describe	role	describing entities and specifying dependencies axiom 7
	chain	role	connecting events causally	axiom 3


Chryssanthi Iakovidou
email: c.iakovidou@iti.gr

Symeon Papadopoulos

Yiannis Kompatsiaris

Knowledge-Based Fusion for

Keywords: Image Tampering Localization Chryssanthi Iakovidou, Symeon Papadopoulos, Ioannis Image tampering localization, late-decision fusion, passive image forensics

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Image forensics techniques have an important role in determining the authenticity of digital images. This is evident by the plethora of scientific approaches available in the literature that have carefully designed mechanisms to reveal different types of digital manipulations and traces that are expected to be generated during a given tampering process [START_REF] Korus | Digital image integrity-a survey of protection and verification techniques[END_REF]. Producing robust tools for detecting and localizing a specific type of forgery has proven to be challenging, even when testing their effectiveness on benchmark datasets and controlled scenarios [START_REF] Zampoglou | A large-scale evaluation of splicing localization algorithms for web images[END_REF] and becomes even greater when dealing with real-world scenarios; images are being edited and manipulated in a variety of ways during a single forgery session in order to produce a convincing outcome, and are then forwarded and shared over the Internet, further undergoing transformations (e.g. cropping, resizing, re-compression). These uncontrolled factors inevitably force many of the standalone methods to suffer in terms of detection accuracy and localization robustness, presenting noisy outcomes and higher false positive rates when applied to new datasets [START_REF] Kaur | A fusion framework based on fuzzy integrals for passive-blind image tamper detection[END_REF][START_REF] Zampoglou | A large-scale evaluation of splicing localization algorithms for web images[END_REF][START_REF] Barni | A fuzzy approach to deal with uncertainty in image forensics[END_REF]. Thus, researchers have come to realise that there is a true benefit in acquiring different reports from independent tools and evaluating the multiple clues in conjunction in the context of blind/passive image forensics (i.e. forensic analysis where prior knowledge regarding the original capturing circumstances, the manipulations or other post processing transformations is unknown) [START_REF] Kaur | A fusion framework based on fuzzy integrals for passive-blind image tamper detection[END_REF][START_REF] Li | Image forgery localization via integrating tampering possibility maps[END_REF][START_REF] Fontani | A framework for decision fusion in image forensics based on dempster-shafer theory of evidence[END_REF].

Several fusion approaches have been proposed in the literature, aiming at a synergistic analysis that improves the overall robustness and reliability of the forensic report. The different strategies can be roughly categorized based on the level at which fusion is carried out and the traces that are considered. Frameworks proposed for feature-level fusion often suffer from drawbacks related with selecting and handling a large number of features and scalability when adding new tools [START_REF] Barni | A fuzzy approach to deal with uncertainty in image forensics[END_REF][START_REF] Chetty | Nonintrusive image tamper detection based on fuzzy fusion[END_REF], while approaches based on measurement level fusion are best suited for the tampering detection problem as they provide a more high-level response in terms of confidence for particular traces being present or not [START_REF] Kaur | A fusion framework based on fuzzy integrals for passive-blind image tamper detection[END_REF][START_REF] Fontani | A framework for decision fusion in image forensics based on dempster-shafer theory of evidence[END_REF]. On the other hand, pixel-level fusion is more effective for tampering localization and techniques proposed in this direction usually involve utilizing probability output maps and a fusion model to refine the final output and improve the localization of the tampered region [START_REF] Li | Image forgery localization via integrating tampering possibility maps[END_REF][START_REF] Korus | Multi-scale fusion for improved localization of malicious tampering in digital images[END_REF]. However, several important issues have not been comprehensively studied, for example, how to select the appropriate "base" forensics approaches, how to fuse their detection results, and how to refine the fused localization map.

In this paper, we introduce an extensible fusion framework for tampering localization and output refinement. The design strategy focuses on analyzing tampering localization approaches from the literature that are selected and categorized based on a multi-criterion ranking process integrating also expert background knowledge regarding their domain of application (types of images and encoding, supported traces, known limitations, etc.). Next we employ a fusion mechanism based on local and cross-tool statistics to produce a single, refined fused heat-map output for tampering localization. We primarily focus on splicing localization which is a very common and effective type of tampering that occurs when parts of the original image are replaced by alien content, while also prioritizing including methods that base their detection mechanisms on JPEG-related traces as it remains the dominant image codec for digital images in devices and on the Internet.

The main objectives of the delivered framework are: i) to exploit tools that are complementary to each other, such that the robustness and reliability of the overall localization system can be improved, and ii) communicate the tampering detection and localization results to end users in a manner that is easier to interpret compared to existing forensics approaches.

The proposed tampering localization fusion framework

Selecting tampering localization base methods

In our effort to integrate different forensic approaches into a single framework, we begin by investigating the properties of state-of-the-art to assess what background information can benefit the fusion scheme. We first start by grouping candidate methods based on: (i) their known domain of application (type of tampering), (ii) their detection mechanisms (types of trace) and (iii) their reported performance (reliability of localization and readability of outputs). In order to limit the possible choices between methods, we primarily focus on splicing localization, a very common and effective type of tampering, and we prioritize passive methods that base their detection on analysis of the JPEG compression given its dominance on the Internet. Figure 1 depicts the general diagram of the selection process for including tampering localization methods as modules in our fusion framework. The various methods are organized in groups depending on the traces they detect, so that, when becoming part of the fusion framework, grouped methods can reinforce each other's results, while results deriving from different groups can be evaluated in a complementary fashion. In parallel, the candidate methods are undergoing a set of evaluation experiments on diverse datasets so as to assess their effectiveness in terms of tampering detection, localization and output readability. For that matter, we utilized the large volume of experiments we conducted in [START_REF] Zampoglou | A large-scale evaluation of splicing localization algorithms for web images[END_REF] and in [START_REF] Iakovidou | Content-aware detection of jpeg grid inconsistencies for intuitive image forensics[END_REF] as a guide for the selection of the most effective methods. In [START_REF] Zampoglou | A large-scale evaluation of splicing localization algorithms for web images[END_REF] we evaluated 14 established state-of-the-art methods for image splicing localization that cover the full spectrum of known tampering traces. In [START_REF] Iakovidou | Content-aware detection of jpeg grid inconsistencies for intuitive image forensics[END_REF] we extended the evaluations of seven techniques from [START_REF] Zampoglou | A large-scale evaluation of splicing localization algorithms for web images[END_REF] (Table 1, rows 1-7) and added a novel algorithm recently developed by us [START_REF] Iakovidou | Content-aware detection of jpeg grid inconsistencies for intuitive image forensics[END_REF] (Table 1, row 8).

Candidate Methods

Apply on diverse

The evaluations concern i) the ability of a method to retrieve true positives of tampered images at a low level of false positives (KS@0.05); ii) the ability to achieve good localization of the tampered region within the image (F1); and iii) the readability of the produced heat map, i.e., a high distinction of assigned values for pixels belonging to tampered versus untampered regions, expressed as the range of different binarization thresholds that result in high F1 scores. The Table 1. Image forensics methods considered as tampering detection modules.

Acronym Description DCT [START_REF] Ye | Detecting digital image forgeries by measuring inconsistencies of blocking artifact[END_REF] A simple fast detection method for inconsistencies in JPEG DCT coefficient histograms. ADQ1 [START_REF] Lin | Fast, automatic and fine-grained tampered jpeg image detection via dct coefficient analysis[END_REF] Tampering localization by exploiting the characteristics of double DCT quantization. BLK [START_REF] Li | Passive detection of doctored jpeg image via block artifact grid extraction[END_REF] Detection of disturbances of the JPEG 88 block grid in the spatial domain. NOI1 [START_REF] Mahdian | Using noise inconsistencies for blind image forensics[END_REF] Modeling of the local image noise variance using wavelet filtering. NOI2 [START_REF] Lyu | Exposing region splicing forgeries with blind local noise estimation[END_REF] Modeling of the local image noise variance utilizing the properties of the Kurtosis of frequency sub-band coefficients in natural images. NOI3 [START_REF] Cozzolino | Splicebuster: a new blind image splicing detector[END_REF] Computes a local co-occurrence map of the quantized high-frequency component of the image and locates inconsistencies in local statistical properties. CFA1 [START_REF] Ferrara | Image forgery localization via fine-grained analysis of cfa artifacts[END_REF] Models the Color Filter Array interpolation patterns as a mixture of Gaussian distributions and locates tampering based on detected disturbances. CAGI [START_REF] Iakovidou | Content-aware detection of jpeg grid inconsistencies for intuitive image forensics[END_REF] Detects JPEG grid abnormalities in the spatial domain, taking into account the contents of the image. Multiple grid positions are evaluated with respect to a fitting function, and areas of lower (for CAGI) or higher (for inv-CAGI) contribution are identified as tampered.

experiments were performed on three publicly available datasets1 , including both synthetic and real-world tampering cases while their performance robustness when input images are subjected to common post-processing operations was also investigated. Through these evaluations, summarised in Figure 2, we were able to assess, rank and correlate their classification ability and overall performance over a wide spectrum of cases and conditions. Based on the evaluation results and taking also into account the selection principles described above, a set of five methods were selected as the "base" building blocks of the framework; these include: i) ADQ1 and DCT that both base their detection on analysis of the JPEG compression, in the transform domain; ii) BLK and CAGI that base their detection on analysis of the JPEG compression in the spatial domain; and iii) NOI3 that is a noise-based detector selected as a complementary tool mainly due to its high reported performance and the good interpretability of its produced outputs. Any new candidate method that will be considered for inclusion in the framework, will go through the same evaluation and grouping steps, being additionally ranked against the base methods on these multiple criteria, so as to decide whether it is expected to contribute to the fusion (include or not), how (in which group/what trace), and by how much (confidence weights based on ranks).

Fusion and refinement of tampering localization masks

The objective of designing a fusion framework is to improve the system's robustness and reliability. If one detector produces noisy or erroneous scores, having other detectors at hand makes it possible to complement, correct and refine the final localization. Figure 3 depicts the block diagram of the proposed fusion framework. For each input image I, we calculate a set of different tampering maps obtained according to the selected subset of detection methods M k . Based on those, we formulate the fusion task as a labeling problem and we work towards denoting forged pixels with label "0" and authentic pixels with label "1". Normalization and Binarization Units: First, output maps are normalized in the [0, 1] range at image level. Next, we are able to cost-effectively automate the binarization of the maps by choosing the appropriate binarization threshold as a value belonging to the respective safe ranges per method as these are determined through the analysis that was performed during the selection process (Figure 2). The binarized maps allow easy analysis of their respective spatial and visual properties. We model as valuable and favor outputs that are easy to interpret visually; we are expecting useful maps to have well defined tampered pixel areas that are spatially concentrated and form "blob-like" structures of significant size.

Connected Components Unit: For each binarized map M b k , we calculate the center of mass (i.e., centroid) for every 8-connected region that is marked as tampered:

R c = 1/N N i=1 R i and C c = 1/N N i=1 C i (1)
where (R C , C C ) are the row and column coordinates of the centre of mass of the region under test, R i , C i are the i-th pixel coordinates of the region, i.e., matrix elements with zero value, and, N is the total number of pixels in the region. Next we build a feature vector describing the number of the detected connected regions, the location of their centroids (R C , C C ), the spatial standard deviation of the pixels belonging to a region from their respective centroid, and the image area of each connected region expressed as the smallest possible rectangle (bounding box) containing the pattern of interest. Additionally, for each method, we produce maps of the connected components maps M cc k , where pixels belonging to each region (hereinafter referred to as blobs) are marked with unique labels. Filtering Unit: The normalized maps, M n k , are forwarded to the Filtering Unit together with the outputs of the Connected Component Unit in order to filter the binary maps, M b k . Two types of filtering take place. First, we filter based on findings of each method independently from one another:

-Blobs that present bounding boxes of dimensions bigger than 50% or less that 5% of the image's largest dimension are automatically discarded. This contributes towards fast filtering of spurious, noisy and overall falsely detected results i.e. big blobs that are the result of densely, over-activated maps or isolated small groups of pixels. -Blobs whose bounding boxes overlap by more than 90% are merged.

-If after the two above steps, the number of blobs is more than five, we calculate the Center of Mass for each M n k (as in Eq. 1 but now all pixels are considered and weighted by their actual value in the map) and rank the blobs based on i) their centroids distance from the overall Map centroid (the smaller the distance the better the score), ii) the density of the pixels in the blob (the denser the better the score), and iii) their size (the bigger the size the bigger the score). We then keep the top five based on their mean score in all three criteria. Second, we perform a content-aware filtering step that depends on the particular methods. Utilizing the content annotation process implemented in CAGI [START_REF] Iakovidou | Content-aware detection of jpeg grid inconsistencies for intuitive image forensics[END_REF] that provides information about areas that are expected to present no noise traces at all (i.e., over and under exposed areas) and the fact that DCT also outputs zero pixel map scores for image blocks of 8-by-8 pixels that share the same intensity value, we are able to filter blobs that may occur as false localizations in BLK and NOI3 outputs; BLK areas that lack any kind of grid pattern are considered tampered; for NOI3 complete lack of noise activates false alarms as they are recognized as inconsistencies in noise distributions. ADQ1 is not triggered by content and thus not affected by this filtering step. Statistics Extraction: Finally, we extract statistics to automate the evaluation of the outputs' usefulness. These constitute an additional layer of confidence in selecting from the various intermediate maps the ones that are appropriate for use in the fusion step. We mainly rely on multilevel measurements of the entropy of the data. Image entropy is a quantity used to express the randomness of an image, computed by:

E = - i p i log 2 p i (2)
where p i is the probability that the difference between two adjacent pixels is equal to i. Measuring the entropy of the visual output maps can give us an immediate rough measure of the interpretability of the result. Low entropy corresponds to clear distinction between foreground to background, while noisy outputs with values ranging over many areas will have high entropy. We calculate the following levels of entropy: i) the overall entropy of the normalized map (M n ), per method; ii) the entropy of its binarized counterpart (M b ), and iii) the entropy of each blob region against the entropy of the remaining image. Additionally, we calculate the Kolmogorov-Smirnov (KS) statistic to compare the value distribution for the different regions of the maps (tampered and untampered) as follows:

KS = max u |C 1 (u) -C 2 (u)| (3)
where C 1 (u) and Fusion Unit: Figure 4 showcases the intermediate steps of the fusion process. This leverages the intermediate calculations to produce a single fused output. To this end, a set of fusion quality properties have been defined:

-Interpretability of the methods' localization maps: Maps are ranked and assigned a confidence score, C i , based on the difference of the entropy before and after the map's binarization. -Compatibility between the traces detected by the different methods: Confidence of a method is reinforced if other methods detecting similar traces also achieve high confidence. Thus, if the C i is high for more methods from the grouped set of tools (e.g., BLK/CAGI/NOI3 or ADQ1/DCT) the confidence score is boosted. -Reliability of the method as measured and assigned after performing extensive evaluations: The reliability of the tools is also a factor for ranking. All methods are ranked to contribute based on their historical performance (Tables (b) and (c) in Figure as long as their outcome interpretability score surpasses a given threshold. -Confidence in the presence or absence of identified tampered regions: For labeling regions as tampered or not, we also consider the original values of pixels region in the normalized tampering map. The KS statistic is calculated for regions belonging to blobs and background per method. The blobs with highest KS score of the best ranking method serves as our baseline detected tampered region. The refinement of the localization of the blobs is based on comparing it with the blob masks of the other methods in a ranked weighted order.

Experimental Evaluation

We tested our proposed fusion framework on two publicly available datasets. The First IFS-TC Image Forensics Challenge training set [1], contains 450 usersubmitted forgeries and was designed to serve as a realistic benchmark. Focusing on splicing tampering localization, we excluded cases that were produced by copy-move operations resulting in a set of 306 forgery cases produced through spicing operations only. Tampered images in this dataset are accompanied by Ground Truth (GT) maps. The second dataset is the CASIA V2.0 dataset [START_REF]CASIA TIDE v2[END_REF] that contains 5,123 realistically tampered color images of varying sizes. During the tampering process post-processing of spliced boundary regions is also considered. This dataset does not come with GT maps. In order for us to be able to perform localization tests, we manually produced 2, 195 reliable GT maps through semi-automated procedures involving image differencing, thresholding and morphological operations. In experiments that follow, when we refer to the CASIA2 dataset we only account for the 2,195 images for which we produced GT binary maps The overall localization quality and output readability is based on the pixelwise agreement between the reference mask (GT) and the produced tampering localization heat map and it is measured in terms of the achieved F-score (F1). This evaluation methodology requires the output maps to be thresholded prior to any evaluation. To this end, we first normalize all maps in the [0, 1] range and proceed by successively shifting the binarization threshold by 0.05 increments, calculating the achieved F1 score for every step.

Figure 5 presents the mean F1 scores curves per binarizarion step over the Challenge and CASIA2 collection for the outputs of each individual method along with the fused output. The achieved localization is evaluated by the maximum mean F1 score for each method, at its respective best performing binarization threshold (Table 3). As an indicator of a method's output interpretability we consider the range of the binarization threshold values, where the achieved F1 remains high (> 0.7 of the best reported score). A wide range suggests that the tampered and untampered image regions are characterized by significantly different values in the output maps making the respective heat map easy to interpret. Table 3 also reports the best localized detections achieved per method. The detection threshold was set to 0.7 and the search was performed for the best binarization step for each method. Unique Localizations corresponds to the number of detections exclusively achieved by that method.

From the experimental results in both datasets we can see that the fused output heat maps achieve high F1 scores over a wide range of thresholds. This verifies that the method produces outputs that exhibit increased localization ability and interpretability. In the Challenge dataset, the next best method (NOI3) achieves similar localization scores but is somewhat worse in terms of interpretability, while all other methods achieve significantly lower F1 scores. In CASIA2, the fusion method is the second best performing method in terms of F1 scores, while it still presents the best interpretability with F1 scores remaining high for a wider range of binarization steps. DCT, which is the leading method in this dataset, is significantly outperforming the rest of the individual methods, which is probably due to the tampering process followed in this specific dataset. The fusion framework manages to produce outputs that generally localize tampering better than most of the individual methods (Figure 5(b)) but, in its current state, does not take full advantage of the very good DCT localizations in building its final output. Instead, while trying to construct hybrid outputs with low risk by collectively examining the various outputs and not heavily relying on only one method, the good DCT localizations were undermined by the many unsuccessful localizations of other methods. Motivated by these findings, assigning better weighting factors and ranking criteria will be at the heart of our next efforts.

Finally, in both datasets the fused method reports a high number of absolute localizations, which is indicating that the fusion criteria set in this framework manage to take advantage of the correctly localized outputs of the individual methods, and more importantly the framework contributes additional unique localizations through fusion and refinement, especially so in the CASIA2 database. Various localization outcomes are depicted in the Figure 6.

Overall, this first set of experimental evaluations verifies the importance of exploiting the available state-of-the-art methods in a manner that improves the robustness and reliability of the system. In our next steps, we will continue to further test and refine the framework, while we also plan to introduce more localization methods in the system.

Conclusions

In this paper, we addressed the splicing tampering localization problem focusing on traces and methods that apply to JPEG images. To this end, we proposed an extensible tampering localization fusion and map refinement framework that combines multiple state-of-art techniques by exploiting their complementarities. We performed and took advantage of extensive evaluation experiments with the goal of selecting the most appropriate "base" methods to be fused so as to produce a single refined localization map outcome. Our experimental findings indicate that the fused output achieves high performance and interpretability by managing to exploit the correctly localized outputs of the individual meth- ods while contributing with unique accurate tampering localizations. While we consider the results of our fusion approach promising, we also recognize the fact that the fusion is based on hard-coded expert knowledge that is directly implemented in the fusion criteria and rules. To this end, we plan to also investigate the potential of fusion approaches based supervised learning.

Fig. 1 .

 1 Fig. 1. Diagram of the selection process for localization methods as modules in the image forensics fusion framework.

Fig. 2 .

 2 Fig. 2. Summary of results: a) performance: KS score, max F1 score; threshold binarization range, b) average performance of methods based on normalized KS, F1 and threshold range per dataset; c) rank aggregation results based on Borda count, Copeland and Kenemy-Young voting.

Fig. 3 .

 3 Fig. 3. Block diagram of the proposed fusion framework.

  C 2 (u) are the cumulative probability distributions inside and outside the mask, respectively. High KS values indicate that what we have marked as a blob of tampered pixels presents a very different distribution of values compared to the rest of the image.

Fig. 4 .

 4 Fig. 4. Intermediate outcomes and final fused output of an example input image (taken from the Challenge dataset [1]).

Fig. 5 .

 5 Fig. 5. F1 score curves on the (a) Challenge and (b) CASIA2 datasets for FUSED, CAGI, BLK, NOI3, DCT, ADQ1 of the tampering fusion framework

Fig. 6 .

 6 Fig. 6. Various examples of tampering localization outputs from the fusion framework and the individual methods; for the first four rows, images are taken from the Challenge dataset, for the last four rows images are taken from the CASIA2 dataset.

Rank and evaluate the candidate method against the base methods Draw implementaƟon decision and assign confidence scores

	Datasets	EvaluaƟon Metrics
	Categorize according to the type of traces it bases its detecƟon	Grouped Methods 1 Grouped Methods 2 … Grouped Methods n

Method 2 Method 1 Method 3 Method 4 Method 5 StaƟsƟc ExtracƟon FUSION Fused Output Ground Truth Input Image

	NormalizaƟon
	Unit
	BinarizaƟon
	Unit
	Connected
	Component
	Filtering
	Unit

  2 .

		0.2			Challenge
							FUSED
		0.18					CAGI
							BLK
		0.16					NOI3
							DCT
		0.14					ADQ1
		0.12
	F1	0.1
		0.08
		0.06
		0.04
		0.02
		0
		0	0.2	0.4	0.6	0.8	1	1.2	0	0.2	0.4	0.6	0.8	1	1.2
				Binarization Thressholds					Binarization Thressholds

Table 2 .

 2 Best mean F1 score and binarization range that allows F1 to remain high (> 70% of respective maximum F1 score) and reported detections for F1 score >= 0.7 at each method's best binarization threshold for Challenge and CASIA2 datasets.

			Challenge			CASIA2
	Method F1 score Binarization Localizations	Unique	F1 score Binarization Localizations	Unique
			Range		Localizations		Range		Localizations
	FUSED 0.191	0.0-0.4	37	7	0.238	0.0-0.6	490	88
	CAGI	0.159	0.3-0.8	16	3	0.073	0.0-0.5	29	4
	BLK	0.103	0.0-0.35	8	1	0.085	0.0-0.4	48	3
	NOI3	0.183	0.05-0.3	38	15	0.075	0.0-0.05	66	29
	ADQ1	0.109	0-0.5	4	1	0.241	0.05-0.9	371	22
	DCT	0.105	0-0.65	5	0	0.234	0.55-0.65	488	30

The First IFS-TC Image Forensics Challenge training set [1]; the synthetic Fontani et al. dataset[START_REF] Fontani | A framework for decision fusion in image forensics based on dempster-shafer theory of evidence[END_REF]; and the real cases of the Wild Web dataset[START_REF] Zampoglou | Detecting image splicing in the wild (web)[END_REF].

Acknowledgements

This work was partially funded by the European Commission under contract numbers H2020-825297 WeVerify and H2020-700024 TENSOR.


Evangelos A Stathopoulos

Alexandros Kokkalas
email: akokkalas@iti.gr

Eirini E Mitsopoulou

Athanasios T Patenidis
email: apatenidis@iti.gr

Georgios Meditskos
email: gmeditsk@iti.gr

Sotiris Diplaris
email: diplaris@iti.gr

Ioannis Paliokas
email: ipaliokas@iti.gr

Stefanos Vrochidis
email: stefanos@iti.gr

Konstantinos Votis
email: kvotis@iti.gr

Dimitrios Tzovaras
email: dimitrios.tzovaras@iti.gr

Ioannis Kompatsiaris

Knowledge-Based Management and Reasoning on Cultural and Natural Touristic Routes

Keywords: Ontologies, Reasoning, Semantically enriched geodata, Data homogenization, Route recommendation subsystem

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Vast amounts of data are effusive throughout every ecosystem. Gradually, the ability to effectively capture data for knowledge extraction has increased. Digital agents tend to pseudomimic mental processes, such as deductive reasoning, intricate decision making, inferring general assumptions and so on.The chasm between data and knowledge is bridged by semantics, inserting and fusing contexts into otherwise meaningless data. Ultimately, the interest is not on the value itself but on its representation and meaning inside a system and its exploitation. Today, there are many methodologies followed towards knowledge design and manipulation; in that aspect, one can amalgamate disparate conceptions into a unified model. To achieve the wished level of homogeneity, the affinities of entities must become firmly established, the capitalization of which is the rapidity and deftness of knowledge elaboration to infer with logic as a basis.

In this paper we describe an ontology-based framework for capturing and interlinking assets of cultural and natural substance facilitating the formation of routes via the utilization of spatio-temporal rule-based reasoning. The ontological model encompasses and depicts every unique data genre present in the workflow of the platform: genres related to weather, topological formats expressing geometry and geospatiality (such as points in 3D space and routes), time formats expressing temporality, user profiling, Augmented Reality (AR) and hierarchical content categorization of places along with other information. That way, places might be discarded or included in a final route recommendation for the end-user. To complete this task, a systematic evaluation was performed to assess the abundance of our approach.

Our methodology starts by identifying each relevant entity tagged as coherent with the e-Tracer objectives, as described above. Thereinafter, an extensive research on pre-existing ontologies encapsulating relative knowledge concepts was conducted. Upon the dilemma emerged concerning either the entire imports of concepts (where the majority of structures were needed) or manufacturing our own, we concluding in building custom concepts targeting precisely our objectives and linking afterwards where applicable. At the same time, we paid effort to keep a minimalistic approach in the design of the overall system. Based on the ontology already created, the aggregation of web content or content from official databases was populated in the knowledge base. Finally, rules that should diminish significantly the offered number of selections based on property constraints were implemented and incorporated into web Application Programming Interfaces (APIs).

The rest of the paper is structured as follows: Section 2 presents work which at some extent seems coherent to this paper. Sections 3 provides an overview of the framework and bestows the overall vision and motivation. In Section 4 elaboration on the inference, validation and consistency capabilities are exhibited, while in Section 5 fundamental reasoning functionalities relying on time and geo-spatial properties are showcased. In Section 6 quantitative and qualitative evaluation is displayed and, finally, Section 7 concludes our work.

Related Work

Data about Cultural and Natural places are bound with their location information and time parameters. With the increasing amount of geospatial data being published online and the geographic information taking a crucial part in several central hubs on the Linked Data Web; geospatial semantics, geo-ontologies and semantic interoperability can have a key role in supporting publishing, retrieving, and reusing data while reducing the risks of misinterpretations [START_REF] Janowicz | Geospatial semantics and linked spatiotemporal data-Past, present, and future[END_REF], [START_REF] Homburg | Interpreting heterogeneous geospatial data using semantic web technologies[END_REF]. Moreover, various semantic web technologies have been adapted for geospatial data, with progress in the effectiveness of the methods used in [START_REF] Zhao | Ontology-based geospatial data query and integration[END_REF] with space and time having a key role for definition, organization and mutual interaction between concepts for knowledge engineering [START_REF] Janowicz | The role of space and time for knowledge organization on the semantic web[END_REF]. In similar approach, RDF models regarding spacetime events have been designed, that integrate spatial, temporal and semantic relations for capturing factors behind certain geographic changes [START_REF] Fan | Modeling and Reasoning about Geospatial Event Dynamics Using Semantic Web Technologies[END_REF] Aggregating geospatial datasets into a single one is a challenging task. To tackle this problem semantic technologies were deployed so as to automate the geospatial data conflation process. By using ontology, RDF data conversion and a set of SWRL rules, one can produce a Points of Interest (POIs) dataset with reduced duplicates and improved accuracy [START_REF] Yu | Semantic web technologies automate geospatial data conflation: conflating points of interest data for emergency response services[END_REF]. Furthermore, the semantic ontological network graph (SONET) [START_REF] Palumbo | SONET: a semantic ontological network graph for managing points of interest data heterogeneity[END_REF] is an ontological network to match categories across multiple heterogeneous sources of POIs of Volunteered Geographic Information (VGI) data. This ontological network advances the study of VGI data by enabling cross-platform analysis while it supports the use of POI data in land use mapping and population modeling applications. Deepening in the issue of heterogeneity since particularly cultural heritage consists of multiple resources which might include entities such as places, events, availability, and others that have special characteristics and might be connected with each other. Cultural-ON is a suite of ontology modules, to model the principal elements identified in the cultural heritage data type classification. The result is a knowledge base consisting of semantic interconnections with also other data available in the Web to be exploited according to different tasks and users preferences [START_REF] Lodi | Semantic web for cultural heritage valorisation[END_REF].

Taking advantage of cultural assents and towards touristic recommendations, there are several ontology-based systems available. They provide personalized suggestions to users, based on user profiles and information concerning the suggested locations. The results are ranked based on profile assignment, content filtering and user feedback [START_REF] Alonso | Ontology-based tourism for all recommender and information retrieval system for interactive community displays[END_REF] or by ontology-based content analyzer, ontologybased profile learner, and ontology-based filtering component [START_REF] Bahramian | AN ONTOLOGY-BASED TOURISM RECOMMENDER SYSTEM BASED ON SPREADING ACTIVATION MODEL[END_REF] or in the case of STAAR (Semantic Tourist information Access and Recommending) where algorithms take into consideration itinerary length and user interests [START_REF] Cao | An ontology based approach to data representation and information search in smart tourist guide system[END_REF].

e-Tracer Framework

In a world with liberty and convenience of locomotion, data can be aggregated in abundance, enabling the sector of personalized tourism to attribute with further enhancements to provide a more pleasant experience to the end-user, an effort also facilitated by e-Tracer [START_REF] Stathopoulos | Smart Discovery of Cultural and Natural Tourist Routes[END_REF], a national funded project.

Key Concepts and Vision

The conceptual architecture of the platform is depicted in Fig. 1. Briefly, the Web Content involves data derived from official websites and open governmen- tal databases. For the former the easIE framework [START_REF] Gkatziaki | easIE: Easy-to-use information extraction for constructing CSR databases from the web[END_REF] was used to scrap content, while for the later massive file exports sufficed. Furthermore, the Route Formation Pipeline expresses a complex algorithm to create routes, containing not only reasoning functionalities but also personalization based on similarity measures and graph algorithms such as the traveling salesman approaches, Dijkstra's shortest path and so on. Moreover, the Augmented Reality Pipeline encompasses search, identification, retrieval and display of objects on a Smart Device. Concluding, for aggregating information and presenting it to the User the Frontend Platform Services are responsible for.

Our work is focused on the semantics and aims to enrich e-Tracer with such capabilities. It acts as a semantic middleware, capturing, interlinking and serving results. This is materialized in the "Analysis & Unification with Ontologies" component where dissimilar content is homogenized and stored locally in compliance with Resource Description Framework (RDF) triplets inside a Knowledge Base (KB). Additionally, it provides the semantic infrastructure to retrieve stored assets in an asynchronous on-demand manner to fulfil dynamic querying requirements. Furthermore, the reasoning occurs either automatically on each update of the KB or by calling it as a service where data are reciprocated amphidromously. Concluding, it is obvious from Fig. 1 that the use of semantics stands in the core of the platform adding extra value and coordinating a majority of processes. temporal entity, route, user profile, user feedback and weather. Each distinct connotation is potentially intertwined with others in manifold degrees, both at root and lower modeled levels, in a way that there should not be perceived in the overview graph such a phenomenon as an orphaned node when data increment efficiently and satisfactorily. Intuitively, entities will be linked with pre-existing online entities as soon as the core model reaches near its terminal configuration after numerous successive iterations. In the succeeding subsections we circumscribe extensively each key concept along with its semantic correlation with the rest.

e-Tracer Ontological Core Model

Re-used Concepts. The concept of the Event is described both accurately and sufficiently by the ontology for linking open descriptions of events, LODE [START_REF] Shaw | Lode: Linking open descriptions of events[END_REF]. In its simplicity, it is explained more thoroughly in the next subsection along with how supplementary content was formed on top of this work to extend it.

The GeoSPARQL ontology [START_REF] Battle | Geosparql: enabling a geospatial semantic web[END_REF], abiding by strictly to the Open Geospatial Consortium's standards, was shaped to represent and provide functionalities for objects possessing physical extent. In that aspect, as a "Feature" can be characterized anything from points in space to more intricate corporeal spatioformations such as polylines and acanonic, convex or non-convex, polygons. Noticeably, this supplementary property can be exploited to typify e-Tracer's concepts such as routes, POIs, events and furthermore the weather.

In conjunction with efforts towards the aforementioned localization, OWL-Time [START_REF] Hobbs | Time ontology in OWL[END_REF] instils the nature of temporality into the e-Tracer ontology. Periodic and sporadic chronic intervals can be analytically specified and represented, spanning from time-limited events to recurring schedules of points of particular interest, as well as unfolding iterative events such as weather phenomena. Consequently, the fusion of space and time capacitates a thorough designation for when an entity is shifting through those dimensions, facilitating perpetual and unremitting knowledge monitoring of its evolutionary existence.

Finally, we took into consideration previous work regarding the User Profile concept and collected very few content from [START_REF] Maria | Creating an ontology for the user profile: Method and applications[END_REF] as described further below and extended it according to our needs with additional properties and relations with other entities.

Points Of Interest, Events & Routes. The necessity for differentiation between the POI and the Event relies on the logical assumption which dictates that an Event might occur at a POI, coinciding spatially at the exact same longitude and latitude, thus being associated with it or at a place which is not regarded as a POI at all, also containing coordinates unclaimed by any POI. On the contrary, a POI might or might not ever hold an event, without being self-defined by it in any case. In terms of OWL 2 semantics, this is defined as:

Route ≡ F eature ∀includes.(P OI ∀isAssociatedW ith.Event)

Regarding temporal discrepancies, we take into consideration that a POI is permanently established, mayhaps withholding a somewhat fixed schedule, whereas an Event is strictly time-limited and even if periodicity is witnessed, each resumption will be considered a distinct instance. Apart from the dimensional perspective, a qualitatively extensive context-based research, based on local governmental resolutions1,2,3,4,5 , has been conducted so as to conclude to a class-based hierarchical categorization of eventual supported types of POIs and Events of interest.

Ultimately, the existence of an instance of the class Route solely depends on it incorporating either an instance of a POI or that of an Event at least, in a ranked manner which mandates the order of visit each Route suggests. Routes are being generated dynamically in a more compound integrated algorithm, which is out of the scope of this paper, due to the humongous population of potential options, thus not pre-processed or stored locally where the later occurs only after an end-user has completed successfully a proposed detour, on several occasions accompanied by an evaluation of his overall experience in the form of User Feedback.

User Profile, User Feedback & History Of Usage. The User Profile construction encapsulates basic demographic information plus personal interests complemented by the end-user himself or potential impairments. Each instance is bound to possess a History of Routes and a History of POIs archiving each Route, and within, each POI the user has indeed paid a visit to, which is cross validated via the end-user's smart device geolocalization. Furthermore, the subclasses of User Feedback are related at a lower level with the historical assets of the POI and the Route of the ontology, rendering infeasible to submit a personal standardized evaluation about a spatial entity without a priori ratified attendance by the appropriate digital agent of the platform.

U serP rof ile ≡ ∀hasF eedback.U serF eedback ∀hasRouteHistory .(HistoryOf Routes ∃hasRoute.Route)

Augmented Reality & Weather Features. The Augmented Reality Object intends to enclose a profusion of essential properties and describe thoroughly in a semantic way 2D, 2.5D and 3D objects. Unfortunately, there is a scarcity in modelled digital assets to the applied interests of e-Tracer, nevertheless, due to the purposeful adaptability of the system beyond fixed use cases, it was deemed vital to patronize such features. Pragmatically, such an object might only be related to spatial entities, potentially deriving implicit inferences from them according to each scenario. Moreover, the Weather in its simplicity is considered to hold, apart from self-explanatory data properties, both temporal and spatial attributes, achieving to monitor evolutionary weather phenomena across regions pertaining POIs or Events, eventually served to the end-user as plain information or taken into consideration in rule-based reasoning upon nominating routes.

Inference and Validation

Implicit Relationships

Extra logical assumptions are the outcome of blending native OWL 2 RL reasoning and manually constructed custom rules, where the prior relies on the OWL 2 RL profile semantics [START_REF] Motik | OWL 2 web ontology language profiles[END_REF]. Sadly, OWL 2 is limited as it serves modeling only for instances related in a tree-like approach [START_REF] Motik | Structured objects in OWL: Representation and reasoning[END_REF]. Our framework implements domain rules on top of the standard graphs in order to enunciate richer relations by the utilization of CONSTRUCT graph motifs, thus enabling the identification of valid inferences. For example, an Augmented Reality Object instance never contains information about its geolocalization but in our ontology is always attached to a spatial entity, which in turns contains coordinates that can be bequeathed to it via the suitable SPARQL CONSTRUCT query shown below:

CONSTRUCT { ?arobject geo:asWKT ?coordinates. } WHERE { ?arobject etr:relatedTo ?a2. ?a2 geo:hasGeometry ?a3. ?a3 geo:asWKT ?coordinates. }

Consistency & Validation Check

The validation procedure guarantees the consistency of the framework along with the quality, both morphologic and syntactic. This scope is fulfilled through the usage of both custom SHACL [START_REF] Knublauch | Shapes Constraint Language (SHACL), W3C Recommendation[END_REF] validation rules and native ontology consistency checking, always adhering to the closed-world paradigm. The latter manages validation by considering the semantics at TBOX, such as class disjointness, whereas the first discerns constraint contraventions like imperfect information or cardinality contradictions. For example, a SHACL shape representing a constraint which forces all POIs to contain exactly a single ID as a data property of type string is shown below: etr:POIIDShape a sh:NodeShape; sh:targetClass etr:POI; sh:property [ sh:path etr:hasID; sh:datatype xsd:string; sh:minCount 1; sh:maxCount 1; ].

Spatio-temporal Rule-based Reasoning

Entire concepts were imported from well-known ontologies and were combined so as to administer especial properties to specific instances, thus conferring additional capabilities on ruled-based reasoning in order to succeed in a significant diminish in the pool of recommended selections which flow to posterior in-chain services with ultimate objective to deliver delightful route suggestions to the end-user.

The very essence of reasoning in e-Tracer relies on the meaningful restriction of proffered choices. It has already been stated explicitly that each place of interest withholds formal standardized coordinates nearby a major traffic network. On top of those coordinates, functions and APIs based on [START_REF] Battle | Geosparql: enabling a geospatial semantic web[END_REF] were developed and utilized so as to estimate euclidean distances between interchanges of the initial route and the places of interest, realistically serving as a lower distance bound where at best case the euclidean distance equals the actual driving distance.In addition, the fixed traffic network speed limits facilitated the development of an algorithm about approximate calculation of the time needed to arrive from one place to another, also serving as a lower estimation bound.

The algorithms described above were fused into dynamic hybrid rules expressed in complex SPARQL queries, combining time and space dimensions and an additional boolean variable of accessibility impairment to showcase the true potential of complex rule-based reasoning. Furthermore, it is not obligatory to set all limitations at once for the API to function, e.g. sometimes we only mind for distance and not at all for time or accessibility. Consequently, by applying limitations when retrieving places of interest correlated to each interchange within the main route of the end-user, the options stand fewer than before based on logic and necessarily satisfy either default constraints or constraints set by the end-user himself. A sample SPARQL pseudocode applying time and space constraints is provided in Algorithm 1 where if input variables are set to zero the algorithm does not consider that variable for filtering at all:

Evaluation

Currently, a user-centered evaluation stands infeasible as the pilots are due to commence in subsequent months, ipso facto we focalized in system-wise benchmarking. In that aspect, we demonstrate the population of the stored entities, shown in detail in Table 1. The triple store at our disposal is a GraphDB 9.1.1 Free Edition with currently stored 15286 triples which was deployed at a server with Ubuntu 18.04.4 LTS (Bionic Beaver) 64-bit operating system, an Intel Xeon(R) Silver 4108 CPU @ 1.80GHz x 32, 62.5GB of RAM and a Hard Disk Drive of 3.6TB capacity. Unfortunately, it is only anticipated to incline the evaluation towards the engineering response times in a manner where the bias is eliminated. All but one methods were manufactured as dynamic RESTful, thereby we ensured upon summoning that the variables on call conform to a uniformly distributed pseudogenerator with their range values spanning with equal probability of selection to all meaningful and valid content. All of them gratify the competency questions which were documented formerly of the creation of the e-Tracer ontology, a subset of which is showcased in Table 2, along with mean response times and standard deviations, elicited from 1000 executions for each. A simulation example of our approach is displayed in Fig 3, while moving from point A to point B, where all POIs retrieved from the interchange without any reasoning occurring stand 43. On the contrary, it is conceived that the number of 5 POIs is noticeably less when the constraint of time is set to 30 minutes and that of the distance to 20 kilometers.

Conclusion And Future Work

In this paper we presented an ontology-based framework for encapsulating and interlinking resources of cultural and natural nature towards the construction of suggestive enhanced routes. On top of the structured knowledge we practised rule-based reasoning based on spatial and temporal properties of the assets.

At the moment, the work featured is part of a synthesis of services, where dynamic routes are formed based only on the distinct unary level. Consequently, looking to the future, the next step is to exploit knowledge at a more aggregated level, such as applying reasoning at route level. Apart from reasoning, at final stages the resources of e-Tracer ought to be openly linked to other efforts, following the principles of Open Data & Linked Data. The evaluation plan at a cultural level will be orchestrated by Piraeus Bank Group Cultural Foundation whereas at environmental level Axios -Loudias -Aliakmonas Delta, Koronia-Volvi and Pamvotis lakes protected area management bodies are responsible for. The platform encapsulates the Egnatia Motorway axis for pilots and content provided by the prior organizations. Finally, the assessment of the prototypes will be conducted during the pilot tests applied to 2 collaborating museums (the silversmithing museum and the silk museum) and at least 3 areas of environmental interest.

Fig. 1 .

 1 Fig. 1. The conceptual framework of e-Tracer

Figure 2

 2 Figure 2 illustrates the upper-level concepts of the e-Tracer hierarchical model where each differentiated arrow line is depicting a distinct type of connection and dotted lines implying customness. The conceptual model revolves around the notions of point of interest (POI), event, augmented reality object, spatial object,

Fig. 2 .

 2 Fig. 2. The upper-level core concepts of the e-Tracer model and the semantic conjugation

Algorithm 1 :

 1 Spatio-temporal Reasoning SPARQL Pseudocode Data: Interchange, POI Coordinates, Interchange Coordinates Input : DistanceOfTravel, TimeOfTravel ∈ N Output: A list of POIs initialization; foreach Interchange do GET each POI coordinates and the Interchange coordinates; foreach POI Coordinates do if DistanceOf T ravel = 0 then X=euclidean distance( POI Coordinates, Interchange Coordinates); end FILTER (X ≤ DistanceOf T ravel) ; if T imeOf T ravel = 0 and DistanceOf T ravel = 0 then Y = (X × 60) ÷ 90000; end FILTER (Y ≤ T imeOf T ravel) ; if T imeOf T ravel = 0 and DistanceOf T ravel = 0 then Z=euclidean distance(POI Coordinates, Interchange Coordinates)×60 ÷ 90000; end FILTER (Z ≤ T imeOf T ravel); end end

Fig. 3 .

 3 Fig. 3. Geographic map of simulation

Table 1 .

 1 The number of POIs and Events with (average) sum of properties for each,

	inside the knowledge base
	#POIs & Events #Properties Avg. Properties per POI /Event
	257	11627	≈45

Table 2 .

 2 Exemplary competency questions

	# Question

http://odysseus.culture.gr

http://listedmonuments.culture.gr

http://estia.minenv.gr/

http://www.minagric.gr

http://www.opengov.gr/

Acknowledgements

This work is co-financed by the European Union and Greek national funds via the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH-CREATE-INNOVATE (project code: T1E∆K-00410).