Roland Brémond
email: roland.bremond@ifsttar.fr

Jean-Michel Auberlet

Viola Cavallo

Lara Désiré

Vérane Faure

Sophie Lemonnier

Régis Lobjois

Jean-Philippe Tarel

Where we look when we drive: A multidisciplinary approach

Keywords: visual attention, saliency, eye movements, blinks attention visuelle, saillance, mouvements oculaires, clignements

This paper reviews a cluster of recent and current researches in the field of visual attention conducted at the LEPSIS Lab (IFSTTAR). Taking advantage of the multidisciplinary structure of the research team, contributions have been proposed in the fields of image processing, artificial intelligence, virtual reality, cognitive psychology and ethology. In image processing, we have contributed to predictive models of the gaze fixations in a driving task, namely, looking for road signs, proposing a computational model of top-down visual attention. In artificial intelligence, a simple computational model of visual attention was proposed and implemented in an agent-based microscopic traffic simulation model. In virtual reality, the blink frequency was proposed in order to assess the relevance of driving simulators in terms of the driver's mental workload. In cognitive psychology, the visual conspicuity of powered two wheels was assessed on a driving simulator, leading to recommendations about their lighting system design; another contribution gave evidence that oculomotor patterns can reveal crossing decisions at crossroads. In cognitive ethology, a joint work is in progress with the CEREMA in order to study oculomotor patterns at crossroads in a field test, thanks to a dedicated vehicle.

Résumé

Cet article présente un ensemble de recherches récentes et en cours dans le domaine de l'attention visuelle, menées au laboratoire LEPSIS de l'IFSTTAR. Grâce à la structure pluridisciplinaire du laboratoire, des contributions ont été proposées dans les domaines du traitement d'images, de l'intelligence artificielle, de la réalité virtuelle, de la psychologie cognitive et de l'éthologie. En traitement d'images, nous avons proposé un modèle de calcul de la saillance visuelle des panneaux routiers, en référence à une tâche de conduite. En intelligence artificielle, nous avons implémenté un modèle d'attention visuelle pour améliorer la simulation de trafic microscopique. En réalité virtuelle, nous avons proposé d'utiliser les clignements pour mesurer la charge mentale sur simulateur de conduite. En psychologie cognitive, la saillance visuelle des deux roues motorisés a été étudiée, ce qui a permis d'aboutir à des recommandations sur le design des phares ; nous avons également montré que les patterns oculomoteurs peuvent nous informer sur la décision de traverser en carrefour. En ethologie, des travaux conjoints avec le CEREMA sont en cours pour étudier le comportement oculomoteur en carrefour, grâce à un véhicule instrumenté.

Introduction

In humans, vision is a sense that allows resolving fine detail in a scene, at close and far distance and in a fast sequence [START_REF] Snowden | Basic vision: An introduction to visual perception[END_REF]. All these properties make it a valuable sense to gather information from such a rapid self-motion activity as driving. As such, there is no doubt that driving is a visually guided self-motion [START_REF] Sivak | The information that drivers use: Is it indeed 90% visual?[END_REF], and twenty years ago, in an influential paper, [START_REF] Land | Where do we look when we steer[END_REF] addressed the question "where we look when we steer". The issues of visual attention and gaze tracking have become, since, a major trend in transportation research, taking advantage of the fast development of eye tracker devices, together with the development of computational models of visual attention.

This paper reviews a cluster of recent and current researches in this area, conducted at the LEPSIS Lab (IFSTTAR). Taking advantage of the multidisciplinary structure of the research team, contributions have been proposed in the fields of image processing, virtual reality, artificial intelligence, cognitive psychology and ethology.

In the field of image processing, we proposed a predictive model of gaze fixations in a driving task, namely, looking for road signs, proposing a computational model of top-down visual attention [START_REF] Simon | Alerting the drivers about road signs with poor visual saliency[END_REF]. We have also proposed a bottom-up computational model relevant to compute the saliency in High Dynamic Range images [START_REF] Petit | Une carte de saillance adaptée aux images HDR[END_REF]. In the field of artificial intelligence, a simple computational model of visual attention was proposed and implemented in an agent-based microscopic traffic simulation model of crossroads [START_REF] Ketenci | Improved road crossing behavior with active perception approach[END_REF]. In the field of virtual reality, various oculometric indexes have been investigated in order to assess the relevance of driving simulators for specific driving tasks. For instance, it was proposed to use the blink frequency as a marker of the driver's mental workload [START_REF] Faure | Effects of driving environment complexity and dual task on eye blink rate[END_REF], allowing to compare virtual and actual driving. In the field of cognitive psychology, the visual conspicuity of powered two wheels was assessed on a driving simulator, leading to recommendations about their lighting system design [START_REF] Cavallo | Are car daytime running lights detrimental to motorcycle conspicuity?[END_REF]; another contribution gave evidence that oculomotor patterns can reveal crossing decisions at crossroads (Lemonnier, Brémond and Baccino, 2013). In the field of cognitive ethology, a joint work is in progress with the CEREMA in order to study oculomotor patterns at crossroads in a field test, thanks to a dedicated vehicle [START_REF] Lemonnier | Validation of a visual attention model in a driving field test: difficulties and benefits[END_REF].

Visual saliency on the road

Visual saliency refers to the objects attributes which attract one's attention. These attributes are labeled as bottom-up or top-down: bottom-up saliency is data-driven, as when a salient event pops out in the environment (for instance, a red ball among green ones). Top-down saliency is task-driven, and refers to the object's characteristics which are relevant with respect to the ongoing task (for instance, traffic lights convey relevant information in urban traffic). On the road, these two levels of visual saliency apply, and can be considered with various approaches. We give in the following three examples of the relevance of visual saliency in road applications, in the field of Computer Vision and Experimental Psychology.

Computer Vision approach

A number of computational models for bottom-up (pop-out) saliency have been proposed in the last fifteen years. However, these algorithms are rarely applied on High Dynamic Range (HDR) images. We thus derived a variation of Itti's classical algorithm [START_REF] Itti | A model of saliency based visual attention for rapid scene analysis[END_REF] which extracts salient areas in both HDR and Low Dynamic Range (LDR) images [START_REF] Petit | Une carte de saillance adaptée aux images HDR[END_REF]2010). From an eye tracking experiment with a HDR scene, we show that the proposed algorithm leads to a better fit between the saliency map and the ocular fixation map than using Itti's algorithm. Such a bottom-up algorithm is useful in order to compute the saliency in High Dynamic Range road situations, such as driving at night, in tunnels, or even when car lights or traffic lights are present in the scene.

Another topic related to both image processing and visual attention is the Automatic Diagnostic of Road Signs Saliency. Road signs, the main communication media towards the drivers, play a significant role in road safety and traffic control trough drivers' guidance, warning and information. Nevertheless, not all traffic signs are seen by all drivers, leading to accident risks. In order to manage safer roads, the estimation of road infrastructure's saliency is thus of importance for road engineers and authorities who aim at making and keeping traffic signs salient enough to attract attention regardless of driver's workload. What attracts a driver's attention is related both to psychological factors (motivations, driving task, etc.) and to the photometrical and geometrical characteristics of the road scene (colors, background, etc.). The saliency (or conspicuity) of an object is the degree to which this object attracts the driver's visual attention, on a given background. We have seen that road signs perception depends on the two main components of visual attention: objects pop-out and visual search. The first one, also mentioned as bottom-up saliency, is not very relevant when the task is to search for a particular object, whereas one important driving task is to look for road signs. As current computational models of visual search saliency are limited to laboratory-situations, we have proposed a new model allowing computing visual search saliency in natural scenes [START_REF] Simon | Alerting the drivers about road signs with poor visual saliency[END_REF]2010). Relying on statistical learning algorithms, the proposed algorithm captures the priors a driver learns on object appearance for any given class of road signs. Fig. 1 shows salient areas computed with a classical bottomup algorithm [START_REF] Itti | A model of saliency based visual attention for rapid scene analysis[END_REF], our algorithm, and the scanpath of a subject during a road sign search task. The proposed model has been evaluated with success on a panel of subjects; see [START_REF] Simon | A Model for Automatic Diagnostic of Road Signs Saliency[END_REF] for more details.

The core of our system is an automatic estimation of road sign saliency along a road network, from images taken with a digital camera onboard a vehicle. This system can be used for accident analysis and prevention. It enables a fine diagnostic of the road signs saliency allowing to decide accurately on which signs the road manager must act, replacing it, or modifying its peripheral environment. This leads to improved asset management, road infrastructure maintenance and road safety.

Experimental psychology approach

The question of visual conspicuity has long been considered decisive for improving motorcyclist safety. The main safety measure in the past has been the use of Daylight Running Lamps (DRL) by motorcycles, which have become compulsory in many countries since the seventies. The efficiency of motorcycle DRLs, evidenced by many studies, is due to a double effect: they increased the motorcycle's contrast with the environment and thus improve its detectability; they also provide a consistent feature, a visual signature that makes them easy to identify. This conspicuity advantage of motorcycles as the only vehicles with DRLs is now questioned by the growing practice among car drivers of also using DRLs.

We have studied the effect of several visual factors that influence motorcycle conspicuity, linked to the characteristics of the road environment (presence or absence of car DRLs) and of the motorcycle (headlight configurations). In these experiments, participants were briefly presented with photographs of real-world traffic scenes showing complex urban environments and containing motorcyclists, cyclists and pedestrians as visual targets. The participants' task was to say whether they had seen one of these vulnerable road users. The experimental conditions (short presentation time, complex visual environments, several kinds of targets presented at different distances and eccentricities) were chosen to favor attention conspicuity and to limit the participants' use of expectation-driven search strategies. The findings of the first study [START_REF] Cavallo | Are car daytime running lights detrimental to motorcycle conspicuity?[END_REF] indicate that car DRLs create a visual noise that decreases the detection and identification of motorcycles, but also has an adverse effect on the conspicuity of cyclists and pedestrians (Fig. 2, left). The detrimental effect of car DRLs in the vicinity of motorcycles, acting as visual distractors, has been clearly demonstrated.

The second study explored ways of defining a new, unique visual signature for motorcycles that could make them easier to detect and identify in a car DRL environment. Three innovative headlight configurations were compared to the standard headlight, in conditions similar to those of the first study. Color coding (yellow motorcycle headlight) proved most effective, and an additional light on the motorcyclist's helmet also improved the motorcycle conspicuity, whereas a triangle headlight proved ineffective (Fig. 2, right). The findings indicate several ways of improving motorcycle safety through innovative headlight ergonomics (Pinto, Cavallo and Saint Pierre, in press).

Eye movements in driving simulation

Driving simulators are developed by research laboratory in order to study driver's behavior and their interaction with transport systems (road signs, infrastructure and other users). They are based on virtual reality techniques, which allow the driver to interact with an artificial environment, through a variety of intuitive controls.

Driving simulators assessment

The development of driving simulators allows studying the driver's behavior in various driving context, the same for all participants, and safely; hopefully the conclusions of these study would apply to the driving behavior on the road (see for instance, at TRA 2014, Vienne et al.; Shahar and Brémond). Virtual reality is a fairly new research area. Standards are not ready for its development and its use. For a specific virtual reality system, such as a driving simulator, one may consider that its design should take into account the user's needs. Conversely, each simulator has its own technical characteristics, and may be ready to use for various applications, and at various costs. Recent technological advances allow increasing the simulators' fidelity, with respect to the "real world", that is, driving on the road. Technical developments are intended to reduce the gap between performance in the real and virtual worlds, to assist immersion, and to improve the usability of simulators. They may also reduce simulator sickness. However, it must be pointed that progress is still needed in these fields.

Yet, the general view seems to be that technical improvements can also increase the simulator's validity, which relies on the correspondence between the simulated and the real world in the way the human operator behaves. Malaterre and Fréchaux (2001), described two levels of validity: the ethological validity, which corresponds to the resemblance to the observed behavior, and the psychological validity, which is defined as the sameness of the psychological processes underlying the (driving) activity. In the past few years, a "user based approach" emerges to assess the validity of driving simulators. This approach involves three procedures:

(1) to take account of subjective data, such as questionnaires or interviews;

(2) to collect behavioral data;

(3) to measure physiological data, such as eye movement.

The subjective approach provides information about presence, and about the difficulties that a person can feel in a virtual environment. The behavioral approach gives insights about whether a simulator induces a natural driving behavior, and about the participant's ability to complete a specific task in a virtual environment. But these observations do not provide information about the cognitive processes involved in the task. For instance, a driver may allocate more attention to a simulated driving task, compared to real driving. Therefore, in order to assess the validity of a driving simulator, we must reach beyond the subjective and behavioral data. Our approach, based on physiological data, allows assessing the ongoing cognitive processes of the drivers. Eye movement data collection is increasingly popular in behavioral research. For instance, they are used to indicate the cognitive strategies developed to perform a specific task (see also sections 3.2 and 4). Several characteristics of eye movements, such as eye blinks, pupil dilation and saccadic peak velocity reflect the variations of attention, emotion and mental workload over time. These information seem relevant for the assessment of a driving simulator, especially in terms of psychological validity.

An observation of the eye blink frequency could also help to discriminate the source of mental workload [START_REF] Faure | Effects of driving environment complexity and dual task on eye blink rate[END_REF]. In a simulated driving task, we have observed a decrease in the eye blink frequency, when the level of task difficulty increases. On the other hand, we have observed an increase in the eye blink frequency in a dual task condition compared to a single task condition, while the mental workload increases because of attention switching. As a result, eye blinks seem to be a relevant indicator of the mental workload caused by the complexity of the driving task or of the driving environment, in single task condition. But an increase of the eye blink frequency may also reflect an increase of mental workload due to a dual task, such as driving while using a phone, or interacting with an In-Vehicle Information System (IVIS).

Driving behavior studies

Behavioral studies on driving simulators may also use eye movements in order to get a better understanding of the driving task. [START_REF] Yarbus | Eye movements and vision[END_REF] gave evidence for a strong link between oculomotor patterns and the ongoing task. Following this idea, we have explored such a link for cognitive processes active during a driving task. We considered a decision-making task in a driving situation, using Svenson's model of decision making [START_REF] Svenson | Differentiation and consolidation theory of human decision making: A frame of reference for the study of pre-and post-decision processes[END_REF], the Differentiation and Consolidation Theory (Diff/Cons). Differentiation consists in an evaluation and produces a diagnostic: the prior decision. Then, Consolidation leads to the final decision.

In our study, the decision was to stop or not at a crossroad (NoGo/Go decision). This decision was modulated by the road sign (Stop, Priority or Give-Way) and the traffic density (no, few or many vehicles). For each condition, predictions were expressed in terms of cognitive processes, based on the Diff/Cons theory. In the Priority condition, a "Cons" process was expected. In the Stop condition, another "Cons" process was expected, but the NoGo decision needs no interaction with others vehicles, so the risk level is lower and the process was expected to be closer to the Baseline (no decision-making task). Then, when approaching the crossroads, the driver anticipates the next decision (when to restart), with a "Diff" process. In the Give-Way (GW) condition, a "Diff" process was expected, followed by a "Cons" process with a Go decision (as the Priority condition), and a "Diff" process in the NoGo decision (as the Stop condition). In addition, the visual activity is more important in Diff than in Cons, and more important in Cons than in Baseline; moreover, when the visual activity increases, the saccade amplitudes (SA) are expected to increase and the fixation durations (FD) to decrease [START_REF] Chapman | Visual search of driving situations: Danger and experience[END_REF]. In consequence, predictions were possible in terms of SA and FD, that is, in terms of eye movements.

Thirty-four participants took part in a driving simulation experiment; they were equipped with SMI eye tracker glasses. The experiment lasted approximately one hour [START_REF] Lemonnier | To what extent visual search depends on the task? The role of attention[END_REF][START_REF] Lenné | The contribution of on-road studies of road user behaviour to improving road safety[END_REF]. The independent variables were the Position ("Before" and "After" the prior decision in the GW condition) and the Situation (stop, priority, GW Go and NoGo). In the "Before" section, the visual activity was highest for SA and FD in the GW condition. This is congruent with the presence of a "Diff" process before the prior decision. A significant difference was found between the Stop and Priority conditions for the SA, which is congruent with the fact that the "Cons" process is closer to the Baseline in the Stop condition. In the "After" section, significant differences were found between the Stop and Priority conditions, with a higher visual activity in the Stop condition. Comparing the GW-Go and GW-NoGo conditions, no significant effect was found for the mean values. However, significant differences were found for the mean slopes, showing increasing visual activity in the GW-NoGo condition.

In short, in the "After" section, similar patterns were found in the GW-Go and in the Stop conditions, corresponding to a Diff process, which anticipates the next decision at the crossroads (when to restart). Also, similar patterns were found in the GW-Go and Priority conditions, corresponding to a "Cons" process. These results suggest that it is possible to discriminate cognitive processes from eye movement patterns in a driving situation.

Naturalistic studies

Snowden et al. (2006) emphasize that "Our visual system is not there to faithfully record the image outside, it is there to give us the necessary information for us to behave appropriately". This sentence illustrates the move from a traditional approach that used to study human vision in highly controlled laboratory experiments to its study in the natural world [START_REF] Hayhoe | Vision in the natural world[END_REF]. This paradigm shift is requested in the study of human cognition and behaviour in natural settings within the framework of Cognitive Ethology [START_REF] Kingstone | Cognitive Ethology: A new approach for studying human cognition[END_REF].

In the context of driving, this approach is not so new but in recent years, progress in data acquisition and recording made the conduct of on-road studies easier and at lower cost [START_REF] Lenné | The contribution of on-road studies of road user behaviour to improving road safety[END_REF]. Instrumented vehicles allow observing the driving activity in natural traffic conditions. If we want to get insights about the information used to drive these behaviors, the measurement of eye movements in natural driving conditions is appealing. Even if objects that are glanced are not always processed ("looked but did not see") and that objects that are not fixated may have been noticed through peripheral perception, the direction of gaze is nonetheless considered to be an overt manifestation of the momentary deployment of attention in a scene [START_REF] Hayhoe | Vision in the natural world[END_REF]. In recent years, the eye tracker technology improved a lot and different manufacturers developed systems aiming at fulfilling the specific needs of human factors researchers to use eye-trackers in the field. Thus, if in the past researchers conducted researches with home-made eye-tracking system (see [START_REF] Land | Predictable eye-head coordination during driving[END_REF]), more and more studies use eye-tracker systems manufactured by eye-tracking specialized companies [START_REF] Taylor | The view from the road: The contribution of on-road glance-monitoring technologies to understanding driver behavior[END_REF].

Many eye-movement measures are available [START_REF] Holmqvist | Eye Tracking: A comprehensive guide to methods and measures[END_REF] and some are easier than other to calculate.

The most common eye movement events are fixation and saccade. A fixation corresponds to the period of time where the eye is nearly still and a saccade is the rapid motion of the eye from one fixation to another. Because it is considered valid that we are blind during a saccade, fixation events are especially interesting in driving studies. When we are interested in the information collected by the driver outside the vehicle (e.g. from traffic signs, signals, pavement markings), the prime measures are those that determine whether the driver is looking at a particular location or signal. Those eye-movement measures which are referred to as Area Of Interest (AOI) measures are particularly time consuming to obtain in the context of driving. Because they change with time, their determination has to be made mostly manually. Another difficulty is that various information may be present in the same AOI at the same time. For instance, the use of AOI to understand which information is used in curve driving (tangent point versus point on the future path) may not be relevant, because the AOIs of the tangent point and future path may frequently overlap [START_REF] Lappi | Pursuit eye-movements in curve driving differentiate between future path and tangent point models[END_REF]. However, because visual attention is highly task-dependant, and should be studied in relation with the driver behaviors, the CEREMA (Saint Brieuc Lab) instrumented a vehicle (see Fig. 3) which can be described as a High Instrumentation Vehicle [START_REF] Valero-Mora | Is naturalistic driving research possible with highly instrumented cars? Lessons learnt in three research centers[END_REF]. The Data Acquisition Systems combines three systems:  A data-logger (DIRCO), developed at IFSTTAR, which synchronizes the data acquired from a GPS receiver, the controller area network (CAN) of the vehicle, and a gyro;  A video acquisition system (The Observer, Noldus Information Technology) which captures videos from three interior cameras (face of the driver, rear view, driver from the back) on a single frame using a video quad splitter;  An eye-tracking equipment (SmartEye Pro version 5.9): the glance monitoring system corresponds to a vehicle-mounted combination of four eye trackers and three scene cameras. The direction of the gaze is overlaid on a panoramic view of the road. An experiment was conducted with this vehicle, to study the driver's behavior (glance behavior and deceleration behaviors) during their approach to intersections (crossroads and T-junctions) with different traffic control signs (stop versus yield signs), see [START_REF] Lemonnier | Validation of a visual attention model in a driving field test: difficulties and benefits[END_REF]. Another configuration of the vehicle is available with a head-mounted eye-tracker (Pertech). An experiment studying the driver's behavior in transitions between rural and urban areas is in progress.

Application to traffic simulation

The above examples show that the study of visual attention on the road may use theoretical frameworks from various academic fields. In this last example of the LEPSIS contributions, we show that common knowledge on visual attention may also contribute to traffic simulation, via an agent-based model.

The main goal of traffic modelling (and of the associated simulation tools) is to predict the flow and interactions of road traffic. In urban environments, previous researches manly focused on traffic flow prediction before and after crossroads. The main reason for this situation may be that in urban areas, crossroads are signalized, and thus they are considered as regulation points that need not to be simulated. Indeed, from the road engineer point of view, the main parameters at crossroads are the flow and the waiting time in the queue.

But another point may be considered. The crossroads is a complex area, where interactions are different from what they are in car following or overtaking. Today, a number of emerging issues (traffic jam, pollution, pedestrian issues) point out the need for a new definition of the public space, both in space and time. In order to assess the crossroads functions, simulation tools are useful, and their relevance depends on how they take into account the interactions, between drivers and between drivers and pedestrians. The following contribution is a tentative in this direction.

Traffic flow regulations are expected to follow some formal rules, such as the highway code, but this is not always the case, and some informal rules may also apply [START_REF] Björklund | Driver behavior in intersections: Formal and informal traffic rules[END_REF]. Among these rules, an important one says that that you should not enter a crossroads, if there is no way out. This rule is very popular in driving behavior and traffic studies (gap acceptance), while one can easily see that every driver has his own understanding of this rule. True drivers actively use the physical space at the center of the crossroads, which may lead to both congestions and accidents.

We have tried to improve microscopic traffic simulations, not only when approaching the crossroads, but also inside the crossroads, including near-accidents. An agent-based model of microscopic traffic simulation was proposed, based on common knowledge in cognitive science about the limited amount of percepts that one can process at the same time [START_REF] Marois | Capacity limits of information processing in the brain[END_REF]. The driver selects the more salient and relevant items in his environment, with respect to the ongoing task, and only a limited subset of percepts are processed. Based on the game theory, a simulated driver can cross a crossroads (Mandiau et al., 2008), while processing only a limited amount of information [START_REF] Ketenci | Improved road crossing behavior with active perception approach[END_REF]Ketenci, et al., in press).

Conclusion

It appears from these examples that our contributions address three main issues. The first one is to improve the state of the art about "where we look when we drive", which is known to be very taskand condition-dependent. These contributions follow an important trend in the field of visual attention, addressing an ecological situation, driving, with a high social impact: thus, we participate to the growing interest in studying visual attention outside of the lab. The social impact of this field of studies is related to the varieties of potential applications. New knowledge is likely to contribute to a better design and diagnosis of road design (including road signs, road studs, etc.) and in-vehicle systems (ADAS, automotive lighting).

In addition to direct contribution to visual attention on the road, the second issue that we address is the development of computational models using the current state of the art in terms of visual attention, with applications, so far, in traffic simulation and road sign saliency diagnosis. These contributions in image processing and multi-agent systems are bio-inspired, in the sense that they implement some key components of the driver's visual attention into quantitative models of visual perception.

Finally, based on the same state of the art, the gaze behavior is considered as a component of the driving behavior, allowing a better comparison between real and virtual driving, and thus a better estimation of driving simulators validity, possibly giving insights for new methodological approaches in virtual reality evaluation issues. The strong links between these three approaches of visual attention in driving encourage us in our multidisciplinary approach.

Fig. 1 .

 1 Fig. 1. Left: salient areas in 3 sample images, as computed by a bottom-up saliency algorithm. Right: salient areas computed by our top-down algorithm. Middle: scan-paths of observers asked to search for road signs.

Fig. 2 :

 2 Fig.2: Detection rates. Left: for motorcyclists, cyclists and pedestrians, according to whether the car's DRLs were off or on.Right: for motorcycles equipped with standard, yellow, helmet or triangle configurations.

Fig. 3 :

 3 Fig. 3: Views of the CEREMA's instrumented vehicle (and instrumented driver).


Yongxin Liao
email: ayongxin.liao@pucpr.br

Eduardo Rocha Loures
email: beduardo.loures@pucpr.br

Osiris Canciglieri
email: cosiris.canciglieri@pucpr.br

Hervé Panetto
email: dherve.panetto@univ-lorraine.fr

A Novel Approach for Ontological Representation of Analytic Hierarchy Process

Keywords: Decision Making, Analytic Hierarchy Process, Ontology, Reasoning Rule

The Analytic Hierarchy Process (AHP) is one of the widely accepted and frequently used priority assessment methods. A number of studies, which employed the AHP method, have already been carried out in different domains to support their multi-criteria decision-making definitions and applications. However, few researchers have paid attentions to extract and represent the important concepts and their relationships from the AHP method itself. The aim of this study is firstly to propose a way for representing the AHP method as an AHP Ontology together with a set of reasoning rules. Then, a prototype assessment tool is developed to show the possibility of obtaining more flexibility and reusability of this ontological representation.

Introduction

The Analytic Hierarchy Process, which is proposed by Saaty [START_REF] Saaty | A scaling method for priorities in hierarchical structures[END_REF] since the 1970s, is one of the widely accepted and frequently used mathematic analysis method that supports multi-criteria decision-making. A significant number of research projects employed the AHP method to support decision-making in their works and have already been benefited from it [START_REF] Vaidya | Analytic hierarchy process: An overview of applications[END_REF]. Meanwhile, keeping pace with these increasing needs, the development of decision-making tools based on the AHP method has attracted many attentions of some software companies, such as, Decision Lens1 , Expert Choice2 , and Logical Decision 3 . However, among these research projects and implemented toolkits, little work has been done in specifying and formalizing the important concepts together with their relationships from the AHP method itself.

Recently, this existing issue is noticed by some researchers, who began to apply a kind of incomplete ontological representation of the AHP method in their decision-making approaches. For example, Kornyshova and Deneckère [START_REF] Kornyshova | Decision-Making Ontology for Information System Engineering[END_REF] proposed a Decision Making Ontology, which aims to formalize the concepts and their relationships inside the knowledge that related to decision-making procedures and the AHP method, for supporting the decision-making during the information systems engineering. Tran, et al. [START_REF] Tran | A new QoS ontology and its QoS-based ranking algorithm for Web services[END_REF] developed a QoS ontology to describe the information about Web Services in detail and a QoS-based algorithm that adopted the AHP method as the evaluation mechanism to assist Web Services ranking. Wasielewska, et al. [START_REF] Wasielewska | Applying Saaty's Multicriterial Decision Making Approach in Grid Resource Management[END_REF] presented a domain ontology, which captures the concepts that are needed by an AHP algorithm, for assisting an user in specifying required resources to execute a job and then to evaluate the contract proposals based on the those resources and their comparisons. Although efforts have been made by these researches, several existing drawbacks still can be found as follows: (i) Some researches only propose a class hierarchy in an ontology to capture the concepts from the AHP method, but without specifying the relationships between these concepts; (ii) There exist some researches, which formalized both concepts and their relationships from the AHP method. However, the evaluation mechanisms in the AHP method are always designed as part of fixed algorithms that are embedded inside priority assessment tools; (iii) The ontological representation of the AHP method is usually mixed with the information that are needed to be assessed, which lacks of flexibility and reusability. Therefore, there remain needs for an approach to address these limitations.

The purpose of this paper is to propose a way to ontologically represent the AHP method, which not only specifies the concepts and their relationships that are necessary for applying the AHP method, but also realizes the priority assessment and consistency evaluation mechanisms in the AHP method through corresponding reasoning rules. In the remains of this paper, a brief introduction of the AHP method is firstly given to describe the background of our proposition. Then, the proposed ontological representation is presented to highlight our contribution. After that, a priority assessment prototype, based on the ontological representation, is used to show the possibility of obtaining more flexibility and reusability. At the end, we conclude this paper and discuss on future works.

The Analytic Hierarchy Process (AHP)

The priority assessment in the AHP method is based on the mathematical analysis of a set of pairwise comparisons, in which, the priority scales between each two related terms are relying on the professional judgments of domain experts [START_REF] Saaty | How to make a decision: The analytic hierarchy process[END_REF]. Of course, these human judgments may not be always consistent. The AHP method also provides a theory of consistency evaluation to assist experts to improve their judgments and to obtain better consistency [START_REF] Saaty | Decision making with the analytic hierarchy process[END_REF]. In general, the process of applying the AHP method can be divided into the following five steps:

(1) Define a problem area and determine a goal (the top level) of the decision-making.

(2) Select a set of evaluation criteria (the intermediate level) and a set of alternatives (the lowest level) to structure a decision hierarchy. An example of this hierarchy can be seen in Fig. 1. , then

         n z yz y q n p 1 1 .
d. The global priority of a criterion is obtained from its temporary priority in the matrix multiplied by the global priority of the term in its upper level that the matrix respects. e. The final priority of an alternative is the sum of all its temporary priorities in the related matrices multiplied by the global priority of the corresponding criterion. The Ontological Representation of the AHP Method Based on the above-introduced AHP method, an ontological representation is proposed to deal with the three existing drawbacks that we mentioned in the first section. In order to use an existing reasoning engine to assist the creation of the decision hierarchy and the calculation process, the ontological representation is in line with the OWL2 4 standard and results in an ontology, which is named AHP Ontology. An overview of the main structure of this ontology is presented in Fig. 3. To be more specific, following the ontology creation guideline that is presented in [START_REF] Noy | Ontology development 101: A guide to creating your first ontology[END_REF], the AHP Ontology is introduced with more details in the remains of this section. Domain Determination. The scope of the AHP Ontology is defined as the decision-making domain. For the step (1), (2), and (3) of the AHP method, which we mentioned in previous section, this ontology is supposed to use appropriate Classes, Properties, and Individuals to represent the main concepts and their relationships. Meanwhile, for the step ( 4) and ( 5), this ontology should also contain a number of reasoning rules that can perform priority assessment and consistency evaluation.

Define Classes and Class Hierarchy. As can be seen from Table 2, nine main classes together with their descriptions are listed.

Classes Descriptions Decision

The judgment of selecting an alternative for an objective.

Goal

The objective that a decision making intends to achieve.

Alternative

The available candidate that can be chosen during the decision-making.

Criterion

The rule or principle for comparing something. Priority GlobalPriority The indicator that shows the importance of a Criterion over others.

FinalPriority

The indicator that shows the importance of an Alternative over others.

ComparisonPair

The pair that compares two terms with respect to a Criterion or a Goal.

ComparisonMatrix

The matrix that is composed of a number of Comparison Pairs.

Define Properties and Property Restrictions.

As can be seen from Table 3 and Table 4, ten object properties and five data properties together with their restrictions are listed. Create Reasoning Rules for the Calculation. In order to perform the priority assessment and consistency evaluation through a reasoning engine, a number of Properties are added to support the creation of reasoning rules. Limited by the space, in this paper, only a small part of the reasoning rules that are used to calculate a 3×3 matrix is presented as an example. Table 5 and Table 6 list one additional object property and ten additional data properties together with their descriptions and restrictions for the example.

Fig. 1 . 3 )Fig. 2 .

 132 Fig. 1. An Example of the Decision Hierarchy (3) Generate a set of pairwise comparison matrices, in which, all compared terms in a matrix are in the same level of the decision hierarchy and all comparisons in a matrix are with respect to a related term from upper level. Let 1 term , … ,

( 5 )

 5 Evaluate the consistency of the priority assessment. This checking process can be decomposed into three sub-steps: a. Calculate the Principal Eigenvalue (PE) of a matrix. Let max  be the PE of the matrix, the Consistency Index (CI) of a matrix. Let CI be the CI of the matrix, the Consistency Ratio (CR) of a matrix based on the Random Consistency Index (RI, as shown in Table1

Fig. 3 .

 3 Fig. 3. An Overview of the Main Structure of the AHP Ontology

Table 3 .

 3 Object Properties and Their Restrictions Object Properties Descriptions (D) and Property Restrictions (R) hasGoal D: Each Decision has one or more Goal as its objectives. R: Decision hasGoal min 1 Goal hasAlternative D: Each Decision Making has at least two Alternatives to be selected. R: Decision hasAlternative min 2 Alternative hasCriterion D: Each Goal has one or more Criterion for the priority assessment. R: Goal hasCriterion min 1 Criterion hasSubCriterion D: Each Criterion can be decomposed into zero or more sub Criteria. R: Criterion hasSubCriterion some Criterion hasFinalPriority D: Each Alternative has exactly 1 Final Priority as the result of a decision-making R: Alternative hasFinalPriority exactly 1 FinalPriority hasGlobalPriority D: Each Criterion has exactly 1 Global Priority with respect to Goal R: Criterion hasGlobalPriority exactly 1 GlobalPriority hasComparisonPair D: Each Comparison Matrix has one or more Comparison Pairs R: ComparisonMatrix hasComparisonPair min 1 ComparisonPair withRespectTo D: Each Comparison Matrix is with respect to exactly one Criterion or Goal R: ComparisonMatrix withRespectTo exactly 1 (Criterion or Goal) hasFirstTerm D: Each Pair of Comparison contains two terms that compare to each other. It has exactly one Alternative or Criterion as the first term. R: ComparisonPair hasFirstTerm exactly 1 (Alternative or Criterion) hasSecondTerm D: Each Pair of Comparison contains two terms that compare to each other. It has exactly one Alternative or Criterion as the second term. R: ComparisonPair hasSecondTerm exactly 1 (Alternative or Criterion) Table 4. The Data Properties and Their Restrictions Data Properties Descriptions (D) and Property Restrictions (R) hasValue D: Each Priority has exactly 1 Value as the indictor of importance. R: Priority hasValue exactly 1 double hasNumberOfCompared -Terms D: Each Matrix has exactly 1 Number to show the size of that matrix, namely, how many terms are compared to each other in that matrix. R: ComparisonMatrix HasNumberOfComparedTerms exactly 1 integer hasI D: Each Comparison Pair has exactly 1 Number to show which column it is located in a matrix R: ComparisonPair hasI exactly 1 integer hasJ D: Each Comparison Pair has exactly 1 Number to show which row it is located in a matrix R: ComparisonPair hasJ exactly 1 integer hasWeight D: Each Comparison Pair has exactly 1 Weight, which is the comparison of importance from its Frist Term to its Second Term R: ComparisonPair hasWeight exactly 1 double

  Sum up the weights in each column in a matrix. Let i c be the sum value of the column Divide the weight of each comparison pair by the sum value of its column. Let ji q be the quotient at the raw j and the column i of that matrix, where Calculate the average values in each row to obtain the priority of the term in that row.

	i , where	i ,..., 1 	n	. Then	i c		 x 1  n	w	xi	.
	b. n i ,..., 1 
	and	j ,..., 1 	n	. Then	q	ji		w	ji	/	i c
	c. Let y p be the priority of	y term , where	y ,..., 1 	n
												Goal	The Top Level
	Criterion 1							Criterion 2	...	Criterion x
	Criterion 1.1	...	Criterion 1.y					...	The Intermediate ...
												Level
	Alternative 1			Alternative 2	...	Alternative z	The Lowest
												Level

In this paper, we only introduce one way to calculate the priority of each term in a matrix, which can be decomposed into five sub-steps: a.

Table 1 .

 1 ). If the CR< 10%, this measurement is consistent and acceptable. Let CR be the CR of the matrix, then Part of The Random Consistency Indexes[START_REF] Saaty | Decision making for leaders: the analytic hierarchy process for decisions in a complex world[END_REF]

	CR		CI	/	RI

Table 2 .

 2 The Classes and Their Descriptions

Table 5 .

 5 The Additional Object Property for a 3×3 matrix

	Object Property	Description (D) and Property Restriction (R)
	hasSuggestedAlternative	D: Each Decision has one or more alternatives as suggested selections
		R: Decision hasSuggestedAlternative min 1 Alternative

Table 6 .

 6 The Additional Data Properties for a 3×3 matrix ComparisonMatrix hasAggregateOfThirdColumn exactly 1 double hasFirstRowPriority D: Each Matrix has exactly 1 priority (a temporary priority inside a matrix) for the first term of those comparison pairs in its first row. R: ComparisonMatrix hasFirstRowPriority exactly 1 double hasSecondRowPriority D: Each Matrix has exactly 1 priority (a temporary priority inside a matrix) for the first term of those comparison pairs in its second row. R: ComparisonMatrix hasSecondRowPriority exactly 1 double hasThirdRowPriority D: Each Matrix has exactly 1 priority (a temporary priority inside a matrix) for the first term of those comparison pairs in its third row. R: ComparisonMatrix hasThirdRowPriority exactly 1 double hasPrincipalEigenvalue D: Each Matrix has exactly 1 Principal Eigenvalue

	Data Properties	Descriptions (D) and Property Restrictions (R)
	hasAggregateOfFirstColumn	D: Each Matrix has exactly 1 aggregate of weights in its first column
		R: ComparisonMatrix hasAggregateOfFirstColumn exactly 1 double
	hasAggregateOfSecondColumn D: Each Matrix has exactly 1 aggregate of weights in its second column
		R: ComparisonMatrix hasAggregateOfSecondColumn exactly 1 double
	hasAggregateOfThirdColumn	D: Each Matrix has exactly 1 aggregate of weights in its third column
		R:

R: ComparisonMatrix hasPrincipalEigenvalue exactly 1 double hasConsistencyIndex D: Each Matrix has exactly 1 Consistency Index R: ComparisonMatrix hasConsistencyIndex exactly 1 double hasConsistencyRatio D Each Matrix has exactly 1 Consistency Ratio R: ComparisonMatrix hasConsistencyRatio exactly 1 double isConsistent D: Each Matrix has a boolean value to tell whether it is consistent or not R: ComparisonMatrix isConsistent exactly 1 boolean

Decision Lens: http://www.decisionlens.com/

Expert Choice: http://expertchoice.com/

Logical Decision: http://www.logicaldecisions.com/

Acknowledgements

The authors would like to thank the financial support provided by the Coordination of Improvement of Higher Education Personal (CAPES) and Pontifícia Universidade Católica do Paraná (PUCPR) in Brazil.

Taking into account the concepts and their relationships in the AHP Ontology and following the rule syntax that are proposed in Jena 5 , some rules that correspond to each sub-step of the step (4) and [START_REF] Wasielewska | Applying Saaty's Multicriterial Decision Making Approach in Grid Resource Management[END_REF] in the AHP method are created. An example rule is presented in Fig. 4, which is related to the sub-step "a" and to the sub-step "e" in the step [START_REF] Tran | A new QoS ontology and its QoS-based ranking algorithm for Web services[END_REF]. The first part of the example rule sums up of all the weights in the first column of a 3×3 matrix, and the second part shows how to use the priority assessment results to suggest an alternative that has the top priority.

The Prototype Tool based on the AHP Ontology

In order to show how the AHP Ontology with its rules can be applied in reality, we prototyped a priority assessment tool, which is partially based on the framework that we proposed in our previous research [START_REF] Liao | Formal Semantic Annotations for Models Interoperability in a PLM environment[END_REF]. As shown on the left side of Fig. 6, the assessment framework contains four main modules. The Protégé 6 environment is employed as an ontology editor and browser, which is used to 5

For each found decision individual, get all its alternatives and put them in three sets

Make these sets different from each other

Get the final priority for each alternative

Get the Value for each final priority

Get the alternative that has top final priority Get all decision individuals.

Suggest the top priority alternative to its corresponding decision

We omit the rest of the calculations in this example represent the AHP method into the AHP Ontology with its rules. The AHP Ontology with its rules are saved in the Knowledge Cloud repository. The Jena Reasoner is employed to perform priority assessment and consistency evaluation. The priority assessment tool acts as an intermediate to manage the communications between the users and the other three modules. As can be seen on the right side of Fig. 6, the priority assessment workflow describes how the AHP Ontology with its rules are used in this prototype. The priority assessment example, which is presented in the thesis [START_REF] Song | Contribution of ontology alignment to enterprise interoperability[END_REF], is employed as an example to verify this prototype. As it can be seen from Fig. 7, the results that are produced by this prototype shows the AHP Ontology with its rules are capable of performing the priority assessment and consistency evaluation appropriately. In summary, this paper reviews some researches that used ontological representations, which related to the AHP Method, to support their decision-making approaches. Based on the investigation, it identifies three existing drawbacks in current research works. The main contribution of this paper is presenting a novel approach that ontologically represents the AHP Method. It not only offers a way to clearly specify and formalize the concepts, relationships and rules in the AHP Method, but also shows the possibility of obtaining more flexibility and reusability through the ontological representation. Furthermore, with some adjustments of the priority assessment and consistency evaluation rules, we are able to extend the AHP method to adopt different needs. Of course, it should be more interesting, therefore, to ontological represent other kinds of decision-making methods into the knowledge cloud. Our results are encouraging and should be used in a real assessment case. In the next stage, one of our perspective is to apply in the research, which relates to assessment of public administration interoperability [START_REF] Cestari | A Research Strategy for Public Administration Interoperability Assessment[END_REF], to validate the applicability of this study.