C W Moritz

Arjan J H Wolf

Silvia V Frijns

Ryan Nedea

Enright

Arjan Wolf

Arjan J H Frijns

Silvia V Nedea

Ryan Enright

COMPARATIVE STUDY OF THE EVAPORATION COEFFICIENT PREDICTING METHODS USING MOLECULAR DYNAMICS SIMULATIONS

Keywords: Molecular Dynamics, Evaporation/Condensation coefficient, Liquid-Vapor interphase, Kinetic boundary conditions (KBC), S-model kinetic equation

HAL is

Introduction

Evaporation and condensation is a non-equilibrium process which has many applications in different fields including biology, astronomy, physics, chemistry and engineering [START_REF] Persad | Expressions for the evaporation and condensation coefficients in the hertz-knudsen relation[END_REF]. Accurate models to describe the transfer of mass and energy across the liquid-vapor interface have become increasingly important in recent years [START_REF] Tsuruta | Condensation/evaporation coefficient and velocity distributions at liquid-vapor interface[END_REF] and are seen as a key to design and development of future high-performance

Proceedings of the International Symposium on

Thermal Effects in Gas flows In Microscale October 24-25, 2019 -Ettlingen, Germany two-phase microscale cooling systems [START_REF] Hanks | Evaporation from Nanoporous Membrane for High Heat Flux Thermal Management[END_REF]. In principle, molecular dynamics (MD) can describe this process precisely. However, due to the large computational cost of MD, practical modelling typically consists of a multiscale approach in which continuum, e.g. Navier-Stokes and gas kinetics, e.g. BGK, S-model [START_REF] Graur | Comparison of different kinetic models for the heat transfer problem[END_REF], DSMC models are used to capture heat/mass transfer in the bulk. MD can be used to study the liquid-vapor interface on the molecular scale to generate the appropriate molecular parameters to the kinetic boundary condition (KBC). In particular, accurate definition of coefficients appearing in the KBC as a function of system operating conditions are of particular interest in order to refine predictive models. Here we highlight different techniques that have been demonstrated in the literature to define these coefficients from MD simulation results and present our own analysis which includes the results of our new alternative method. The results of our analysis are used to define rational confidence bounds on the evaporation and condensation coefficient for Argon in equilibrium for temperatures between the boiling (T b = 87.3K) and critical (T C = 150.7K) temperature. In the next section, the definition of the evaporation/condensation coefficient is given in terms of mass fluxes.

The evaporation extraction methods of Meland [START_REF] Meland | Nonequilibrium molecular-dynamics simulation of net evaporation and net condensation, and evaluation of the gas-kinetic boundary condition at the interphase[END_REF], Gu [START_REF] Gu | Molecular dynamics simulation of the inverted temperature gradient phenomenon[END_REF] and Ishiyama [START_REF] Ishiyama | Molecular dynamics study of kinetic boundary condition at an interface between argon vapor and its condensed phase[END_REF] are briefly explained and our alternative method is introduced and in detail presented. Thereafter, the properties of the equilibrium molecular dynamics simulations are given. In section results, we show the average evaporation coefficient α e and its ±1σ confidence bounds as function of temperature. These results are compared with the coefficients obtained using the half-range Maxwellian distribution for the outgoing mass flux.

The final section contains our conclusions of the present study.

Liquid-Vapor boundary

The kinetic theory is capable of describing the vapor flow in combination with suitable KBC to describe the liquid-vapor interface. As mentioned, they play an important role, because they describe the interaction of the vapor with its condensed phase in the interphase region. In kinetic theory, it is assumed that the interphase region (Knudsen layer) has zero thickness [START_REF] Meland | Nonequilibrium molecular-dynamics simulation of net evaporation and net condensation, and evaluation of the gas-kinetic boundary condition at the interphase[END_REF].

The fraction of molecules which condense to the total colliding molecules on the condensed phase is termed the condensation coefficient (α c ). The fraction of evaporated molecules to the total outgoing molecules from the condensed phase is defined as the evaporation coefficient (α e ). The part (1-α c ) vs (1 -α e ) are those molecules which reflect back into the vapor vs condensed phase respectively.

At the molecular level, these coefficients are defined as,

α e = J evap J evap + J ref = J evap J out , α c = J cond J cond + J ref = J cond J coll (1)
where J ... is the time-averaged mass fluxes of the evaporating, condensing, outgoing, colliding and reflecting molecules. To properly account for these fluxes in MD simulations, it becomes necessary to define liquid and vapor boundaries in the interphase region, see Fig. 1.

In the case of an equilibrium state (α e = α c ), it is assumed that the molecules colliding with the liquid-vapor interface will reflect diffusively and that distribution function of the colliding molecules is given by the half-range Maxwellian f e (ξ) with ξ • n < 0. Hence, the mass flux from the liquid to the vapor phase is defined as [START_REF] Ishiyama | Molecular dynamics study of kinetic boundary condition at an interface between argon vapor and its condensed phase[END_REF],

Proceedings of the International Symposium on Thermal Effects in Gas flows In Microscale October 24-25, 2019 -Ettlingen, Germany

J out e = ξz>0 ξ z f (x, ξ)dξ = ξz>0 ξ z f e (ξ)dξ = ρ v RT L 2π (2)
where we have assumed a planar interface such that ξ • n = ξ z the velocity component perpendicular to the interface. Eq. ( 2) is the outgoing mass flux at the equilibrium state.

In Fig. 1, the liquid boundary can be considered as the "end" of the liquid phase and the vapor boundary as the "beginning" of the vapor phase. In between an interphase layer is defined.

In our previous study [START_REF] Wolf | Comparison of numerical results of molecular dynamics simulations and s-model kinetic equations for evaporation and condensation of argon[END_REF][START_REF] Wolf | The position of the liquid and vapor boundaries and its influence on the evaporation/condensation coefficients[END_REF] we observed a large discrepancy in the heat flux and energy flux between the numerical results of the MD simulations and the steady-state S-model equation. This was traced back to the precise definition of the liquid and vapor boundary positions in the MD simulations used to extract coefficients for the KBC. The implication of this boundary position in different methods used to derive the evaporation/condensation coefficients is investigated. [START_REF] Gu | Molecular dynamics simulation of the inverted temperature gradient phenomenon[END_REF] mentioned that this way of defining both boundaries is impractical for an unsteady non-equilibrium state where a liquid layer keeps growing or decreasing due to the evaporation and condensation. The statistical noise of the MD results will make it difficult to use both methods to locate the liquid and vapor boundary without having enough quasi-steady samples.

Gu et al. [START_REF] Gu | Molecular dynamics simulation of the inverted temperature gradient phenomenon[END_REF] focussed on a new procedure to address the problems of Meland [START_REF] Meland | Nonequilibrium molecular-dynamics simulation of net evaporation and net condensation, and evaluation of the gas-kinetic boundary condition at the interphase[END_REF]. They used a microscopic approach in which the location of the liquid and vapor boundary are based on the average number of interaction partners N k (t) per molecule per bin k. To determine the position of

Proceedings of the International Symposium on

Thermal Effects in Gas flows In Microscale October 24-25, 2019 -Ettlingen, Germany the boundaries, two criteria values, C v and C l are set as the limit on N k (t) to consider the fluid to be in the vapor or liquid phase. When moving from the liquid to the vapor phase, the first bin for which N k (t) < C l is recorded as the bin in which the liquid boundary is located. Similarly, the vapor boundary is locate in the first bin for which N k (t) > C v when moving from the vapor to the liquid phase. The precise location of the liquid/vapor boundary within the bin is determined by the position of the molecule which has the largest/smallest number of interaction partners respectively. The limit values, C l , C v , are obtained from separate MD simulations of pure liquid and pure vapor. Hence, for an equilibrium simulation at temperature T it requires two additional MD simulations. Whereas for a non-equilibrium simulation with temperature T 1 and T 2 , 4 additional MD simulations are needed. Furthermore, in case of non-equilibrium it becomes unclear at which temperature the additional MD simulations should be performed. Due to evaporation and condensation, a temperature gradient exists in the liquid and vapor phase.

Ishiyama et al. [START_REF] Ishiyama | Molecular dynamics study of kinetic boundary condition at an interface between argon vapor and its condensed phase[END_REF] calculated the evaporation coefficients in equilibrium state by introducing the concept of the spontaneously evaporating mass flux J sp evap . The coefficient, according to [START_REF] Ishiyama | Molecular dynamics study of kinetic boundary condition at an interface between argon vapor and its condensed phase[END_REF], is defined as,

α e = J sp evap J out e = α c (3)
where J out e is defined in Eq. ( 2). The spontaneously evaporating mass flux is obtained from simulations of evaporation into vacuum. A vacuum boundary is placed within the vapor phase close to the interphase. Molecules that cross this boundary will be removed, hence no molecules can come back from the vacuum side. The vacuum boundary is located far enough from the liquid-vapor interphase such that at positions z * < z vacuum-boundary the net mass flux of molecules is equal to the outgoing mass flux of molecules i.e. ( J + -J -)| z * = J + | z * . The net mass flux of molecules is defined as J + -J -with J + the outgoing mass flux and, J -is the colliding mass flux. The spontaneously evaporating mass flux is defined as [START_REF] Ishiyama | Molecular dynamics study of kinetic boundary condition at an interface between argon vapor and its condensed phase[END_REF],

J sp evap = J + -J -= J + | vacuum-boundary (4)
Using this method, defining the liquid and vapor boundary location becomes redundant. However, Kobayashi [START_REF] Kobayashi | Molecular dynamics study on evaporation and reflection of monatomic molecules to construct kinetic boundary condition in vapor-liquid equilibria[END_REF] used this concept to improve the previous methods [START_REF] Gu | Molecular dynamics simulation of the inverted temperature gradient phenomenon[END_REF][START_REF] Meland | Nonequilibrium molecular-dynamics simulation of net evaporation and net condensation, and evaluation of the gas-kinetic boundary condition at the interphase[END_REF] of determining the liquid and vapor boundary positions. An average position z can be calculated from the positions z * < z vacuum-boundary at which the net mass flux is equal to outgoing mass flux. The vapor boundary position was defined at the average position z. The liquid boundary location was set at the position where the spontaneously evaporating mass flux J sp evap from the vacuum simulations is equal to the evaporating mass flux J evap from equilibrium simulations. According to [START_REF] Kobayashi | Molecular dynamics study on evaporation and reflection of monatomic molecules to construct kinetic boundary condition in vapor-liquid equilibria[END_REF], this method is appropriate to determine the liquid and vapor boundary in equilibrium and non-equilibrium simulations as well as multi-component systems. Similar to the previous method, additional MD simulations (into vacuum) have to be performed to calculate the mass flux.

We developed an alternative method (Wolf ) which rely on shifting the vapor boundary position. This method explores the behaviour of the evaporation coefficient (Eq. ( 1)) in the interphase region. Applying one of the existing methods [START_REF] Gu | Molecular dynamics simulation of the inverted temperature gradient phenomenon[END_REF][START_REF] Ishiyama | Molecular dynamics study of kinetic boundary condition at an interface between argon vapor and its condensed phase[END_REF][START_REF] Meland | Nonequilibrium molecular-dynamics simulation of net evaporation and net condensation, and evaluation of the gas-kinetic boundary condition at the interphase[END_REF] to define the liquid boundary, the vapor boundary is shifted from the liquid to the vapor phase and thereby extracting the coefficient. Fig. 2

Equilibrium Molecular Dynamics simulation

The system to extract the evaporation and condensation coefficients consists of Argon vapor confined between its condensed phase as show in Fig. 3. Software package LAMMPS [START_REF] Plimpton | Fast parallel algorithms for short-range molecular dynamics[END_REF] has been used to execute the equilibrium MD simulations. The system contains between N = 8000 -10000 molecules depending on the temperature which ranges from T = 90-150K (T /T C = 0.6-1). This corresponds to a liquid-vapor equilibrium above its boiling point T b = 87.3K and below its critical point T C = 150.7K. A Nosé-Hoover thermostat is applied within the grey regions denoted by T in Fig. 3 to control the temperature. The dimensions of the simulation domain are L x = L y = 5.7nm and L z = 75.0nm. The intermolecular forces between the Argon atoms are calculated using the 12-6 Lennard-Jones potential,

U (r) = 4 σ D r 12 - σ D r 6 (5)
where the molecule diameter σ D = 0.3405nm and energy /k b = 119.8K (k b is the Boltzmann constant). The cut-off distance is set to 1.5nm [START_REF] Ishiyama | Molecular dynamics study of kinetic boundary condition at an interface between argon vapor and its condensed phase[END_REF]. Periodic boundary conditions are imposed in all three directions. To obtain steady-state, the center of mass position of the system has been fixed to avoid drifting of the atoms in any direction during the simulation. Newton's equation of motion are solved with time step of 4fs. After steady-state is obtained, the simulation continued for 20000000 time step (80ns) and the configuration of the molecules has been extracted every 500 steps. Hence, N s = 2 × 40000 = 80000 data sets (two liquid-vapor interfaces) are used to extract the evaporation/condensation coefficient. The macroscopic properties are obtained by dividing the z-direction of

Results

As shown in Fig. 2 Because of this sensitivity and the spread of the coefficients found in the literature [START_REF] Gu | Molecular dynamics simulation of the inverted temperature gradient phenomenon[END_REF][START_REF] Kobayashi | Molecular dynamics study on evaporation and reflection of monatomic molecules to construct kinetic boundary condition in vapor-liquid equilibria[END_REF][START_REF] Meland | Nonequilibrium molecular-dynamics simulation of net evaporation and net condensation, and evaluation of the gas-kinetic boundary condition at the interphase[END_REF], which were performed under the same thermodynamic conditions (temperature, pressure, density), an unique evaporation coefficient for each temperature cannot be defined. Since it is unclear where the exact location of the liquid and vapor boundary is and thus the value of the coefficient, we investigated the behaviour of coefficients for each temperature when different methods (Meland, Gu, Wolf) are combined. This implies that for the liquid boundary position, the method of Meland [START_REF] Meland | Nonequilibrium molecular-dynamics simulation of net evaporation and net condensation, and evaluation of the gas-kinetic boundary condition at the interphase[END_REF] or Gu [START_REF] Gu | Molecular dynamics simulation of the inverted temperature gradient phenomenon[END_REF] can be used. For the vapor boundary position, either the method of Meland, Gu or Wolf (alternative method) can be applied. Hence, 6 different combinations can be formed to determine the evaporation coefficient.

Our aim is to determine the range these coefficients can vary for a certain temperature T when different assumptions on the boundary positions are considered.

The results are depicted in Fig. 4 for different reduced temperatures. L B : X means that method X (Meland or Gu) is used to defined the liquid boundary position and V b : X means that method X (Meland, Gu or Wolf) is used for the vapor boundary position. This results in 6 different coefficients for each temperature for which an average value α e is calculated. These average values are used to fit a polynomial function (solid line), the standard deviation (±1σ) are shown by the error overlines. The relative standard deviation (RSD), defined as RSD = (σ/α e )×100, varies between 7%-10%.

As mentioned before, in case of an equilibrium state, the outgoing molecular velocity distribution

Proceedings of the International Symposium on Thermal Effects in Gas flows In Microscale October 24-25, 2019 -Ettlingen, Germany is assumed to be described by a half-range Maxwellian. Hence, the outgoing mass flux can be defined as J out e , see Eq. ( 2). The influence of using Eq. ( 2) for the outgoing mass flux instead of the mass flux calculated from MD simulations, J out M D , is depicted in Fig. 5. This has been done for all the 6 combinations of methods. A polynomial fit to the average values and ±1σ bounds are also included. It can be observed that at low reduced temperatures (T /T C ), some of these methods predict the evaporation coefficient larger than unity. This is not possible according Eq. ( 1) because J ref ≥ 0 which implies α e ≤ 1. When the vapor boundary location is determined with the method of Gu, the coefficient is much larger than unity for T /T C = 0.6 (α e ≈ 3) and T /T C = 0.66 (α e ≈ 1.5), both values are not shown in the Fig. 5. These values are related to the position of the vapor boundary which for low reduced temperatures T /T C ≤ 0.73 are predicted to be closer to the liquid phase for the method of Gu compared to Meland and Wolf. Therefore, it is reasonable to believe that the position of the vapor boundary is closer to the liquid phase compared to the position at which J out e = J out M D . Hence, J evap M D will not only capture evaporating molecules but also molecules reflecting back into the liquid phase so that J evap M D > J out e .

Proceedings of the International Symposium on

Thermal Effects in Gas flows In Microscale October 24-25, 2019 -Ettlingen, Germany

In Fig. 6 the previous obtained polynomial fits of the averaged evaporation coefficients (Fig. 4, Fig. 5) are plotted together with values from the literature. The results of Nagayama [START_REF] Nagayama | A general expression for the condensation coefficient based on transition state theory and molecular dynamics simulation[END_REF] are based on the transient state theory and are close to the average which also holds for the coefficients given by Ishiyama [START_REF] Ishiyama | Molecular dynamics study of kinetic boundary condition at an interface between argon vapor and its condensed phase[END_REF]. The values of Gu [START_REF] Gu | Molecular dynamics simulation of the inverted temperature gradient phenomenon[END_REF] are below the average and outside the ±1σ bounds for all temperatures. This can be again related to the vapor boundary position. Especially for the low temperatures, Z V b (Gu) < Z V b (M eland) < Z V b (W olf ) holds, with Z V b the position of the vapor boundary. According to Fig. 2, this implies that the coefficients of Gu will provide the lowest values in most cases.

Conclusions

• By shifting the vapor boundary from the liquid boundary towards the vapor phase, the behavior of the evaporation coefficient is explored in the interphase region. This lead to the development of an alternative method (Wolf) to determine the evaporation coefficient. Its local maximum attained in the interphase region, Fig. 2, is defined as the evaporation coefficient.

• Performing additional MD simulations like Gu and Ishiyama are unnecessary in the present simulation approach (Wolf). However, depending on which method is used for the liquid boundary, extra simulations may be required.

• Due to the influence of both boundaries on the evaporation coefficient and the spreading shown in the literature, a unique coefficient cannot be defined. We attributed to this by exploring the behavior of the evaporation coefficient when combining different methods to determine the liquid and vapor boundary positions. This lead to the introduction of an average evaporation coefficient 0.9 > α e > 0.4 as function of temperature with the relative standard deviation RSD = σ/α e × 100 varies between 7%-10%.

• Depending on which definition is used for the outgoing mass flux, J out M D or J out e (Eq. ( 2)), different average evaporation coefficients are obtained, Fig. 4 and Fig. 5. Coefficients larger than unity, shown in Fig. 5, are not possible. They are related to the position of the vapor boundary which will be located closer to the liquid phase compared to the position at which J out e = J out M D . Hence, J evap M D becomes larger than J out e .

Proceedings of the International Symposium on Thermal Effects in Gas flows In Microscale October 24-25, 2019 -Ettlingen, Germany

• The results of Ishiyama [START_REF] Ishiyama | Molecular dynamics study of kinetic boundary condition at an interface between argon vapor and its condensed phase[END_REF] and Nagayama [START_REF] Nagayama | A general expression for the condensation coefficient based on transition state theory and molecular dynamics simulation[END_REF] are close to the average value and within the ±1σ bound, see Fig. 6. The results of Gu [START_REF] Gu | Molecular dynamics simulation of the inverted temperature gradient phenomenon[END_REF] are below the average and outside the ±1σ bound.

Figure 1 :

 1 Figure1: Schematic of the liquid-vapor interphase and the corresponding mass fluxes[START_REF] Kobayashi | Molecular dynamics study on evaporation and reflection of monatomic molecules to construct kinetic boundary condition in vapor-liquid equilibria[END_REF]

  depicts the behaviour for different liquid boundary positions Z L b (randomly chosen). The alternative method defines Proceedings of the International Symposium on Thermal Effects in Gas flows In Microscale October 24-25, 2019 -Ettlingen, Germany the maximum value attained in the interphase/vapor region as the evaporation coefficient, denoted by the black dot in the figure. The position of this maxima could be regarded as the location of the vapor boundary. Hence, no additional simulations are needed to define the vapor boundary position or evaporation coefficient. Depending on which method is used for the liquid boundary, additional simulations may be required.

Figure 2 :

 2 Figure 2: The behavior of the evaporation coefficient αe for different liquid boundary positions ZL b = 0.0 -3.0nm. The results corresponds to an equilibrium MD simulation in which αe = αc. The solid line represents the normalized density profile ρ * = ρ/ρL. Black dots defines the evaporation coefficient for different liquid boundary positions (randomly chosen) according to the alternative method.

  Proceedings of the International Symposium onThermal Effects in Gas flows In Microscale October 24-25, 2019 -Ettlingen, Germany the simulation domain into bins and time-averaging over the extracted data. The liquid and vapor properties (pressure, density, temperature) are within 5% of the NIST-REFPROP[START_REF] Lemmon | Reference fluid thermodynamic and transport properties-refprop[END_REF].

Figure 3 :

 3 Figure 3: Schematic of the MD simulation domain. Grey area denotes the region of the Nosé-Hoover thermostat.

  , the position of the liquid boundary Z L b influence the behaviour of the evaporation coefficient. When the liquid and vapor boundaries are assessed at the same location (Z L b = Z V b ), the value of α e collapses to unity. As the vapor boundary is shifted right away from the defined liquid boundary location, it attains a minimum approximately half-way the decreasing density profile and increases until the end of the interphase region where a local maximum is attained. Shifting the liquid boundary to the right increases the local maxima and its location moves slightly to the left. Moving the liquid boundary towards the vapor phase implies that more molecules are considered to be in the liquid phase (z < Z L b ). Hence, the mass flux of evaporating molecules increases which reflects back into an increase of the calculated evaporation coefficient.

Figure 4 :

 4 Figure 4: Evaporation/condensation coefficient as function of reduced temperature (TC = 150.7K) for equilibrium MD simulations of Argon. Jout defined in Eq. (2) is calculated from MD simulation data.

Figure 5 :

 5 Figure 5: Evaporation/condensation coefficient as function of reduced temperature (TC = 150.7K) for equilibrium MD simulations of Argon. Jout M D defined in Eq. (1) is replaced by the equilibrium outgoing flux Jout e defined in Eq. (2).

Figure 6 :

 6 Figure 6: Evaporation/condensation coefficient as function of reduced temperature (TC = 150.7K) for equilibrium MD simulations of Argon. The polynomial fits of the average coefficient αe and its ±1σ shown in Fig.4and Fig.5are plotted with data from Gu[START_REF] Gu | Molecular dynamics simulation of the inverted temperature gradient phenomenon[END_REF], Ishiyama[START_REF] Ishiyama | Molecular dynamics study of kinetic boundary condition at an interface between argon vapor and its condensed phase[END_REF] and Nagayama[START_REF] Nagayama | A general expression for the condensation coefficient based on transition state theory and molecular dynamics simulation[END_REF].

Acknowledgements

This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sk lodowska-Curie grant agreement No. 643095.


Xavier Tannier Limsi
email: xavier.tannier@limsi.fr

NLP-driven Data Journalism: Time-Aware Mining and Visualization of International Alliances

We take inspiration of computational and data journalism, and propose to combine techniques from information extraction, information aggregation and visualization to build a tool identifying the evolution of alliance and opposition relations between countries, on specific topics. These relations are aggregated into numerical data that are visualized by time-series plots or dynamic graphs.

Introduction

Information and communication technologies have provided tools and methods to make the production of information more democratic. As a result, a vast amount of content is available, which arguably creates more noise than knowledge at the end of the day. Without hierarchical organization and contextualization, users may lack perspective to understand and assimilate the multiplicity of events that they come accross every day, and to link them to related events in the past.

In this context, journalists and technologists developed the notion of "data journalism", which takes advantage of the growing popularity of Open Data, the development of structured knowledge bases such as DBPedia [START_REF] Lehmann | [END_REF], YAGO [START_REF] Hoffart | [END_REF] or OpenCalais and many others, as well as recent work in data visualization, to facilitate information analysis and access a variety of points of view. However, knowledge is still far from being entirely represented in structured databases, and much information remains available in text format. For this reason, natural language processing has a lot to offer to modern journalism.

In this paper, we present a tool that automatically identifies alliance and opposition relations between countries, on a specific subject defined by a user query (e.g. situation in Syria, nuclear proliferation, North Pole ownership). The evolution of the relations over time are then illustrated by a time-series plot (see Figure 3) or dynamic graphs and maps (Figure 4).

Related Work

The idea of automatically finding topically related material in streams of newswire data goes back to Topic Detection and Tracking evaluation campaigns [Allan, 2002].

Our work contributes to this research effort and explores the quantitative aspects of knowledge that can be extracted from textual documents. With that respect, as well as on the topic of alliance and opposition relations, this can be connected to opinion mining. [START_REF] Chambers | [END_REF]] also consider these relations, based on Twitter data.

We also use well-known techniques of feature-based, supervised relation extraction. The aim is to identify and classify relations between two entities in the same sentence, by learning these relations on a training, manually annotated dataset. Examples of such works are [Miller et al., 2000], [Kambhatla, 2004], [START_REF] Boschee | [END_REF], [START_REF] Zhou | [END_REF], among many others. Our approach differs from most works in the fact that each relation is associated to a date, which makes the classification time-aware.

Also, we bias our classifier towards precision. This approach relies on linguistic variation and redundancy in a large amount of documents to ensure a good coverage. It is related to works in question-answering [START_REF] Dumais | [END_REF], temporal information aggregation [START_REF] Kessler | [END_REF] or opinion mining [Turney, 2002].

System Overview

We apply information extraction techniques to a large amount of newswire textual documents, in order to acquire enough data to make significant statistics on them. These data can then be accessed by a query-based visualization tool.

Relations that we extract are opposition (NEG) or alliance (POS) relations between two countries, explicitly expressed in a same sentence, such as:

(1) Indonesia voiced support for East Timor's bid to join the ASEAN. → POS(Indonesia, East Timor)

(2) London's recent condemnations of Libyan leader Moamer Kadhafi's bloody crackdown [...].

→ NEG(U.K., Libya)

(3) Chavez has stoop up for his longtime ally Kadhafi,

[...]. → POS(Venezuela, Libya)
The alliance or opposition can be made explicit mainly by an action verb (protested), an event noun (condemnations) or a state noun (ally). Countries (relation arguments) can be designated by their actual names, the name of their capital or of a person representing this country. Figure 1 shows the general architecture of the system. At indexing time, the corpus is processed (step x and Figure 2) with chunking, coreference resolution, named entity tagging, temporal information normalization and association between some people names and their countries. A classifier extracts POS and NEG relations in each sentence.

At query time, all sentences that are relevant to the query are retrieved (z). For each day, POS and NEG relations are aggregated and a tendency of this day is obtained, as well as a trend over time ({). A graphical visualization of this trend is then proposed to the user (| and Figures 3,4).

Alliance and Opposition Classification

Resources

We used a corpus of English newswire texts provided by the AFP French news agency. The English AFP corpus is composed of 1.79 million texts that span the 2004-2013 period (635 documents/day in average and 600 millions words). Entities. Country, nationality and capital name lists were extracted from the linked data repository of the CIA World Factbook, and linguistic variations of these names were collected from DBPedia. All these elements were considered as entities potentially parts of an alliance/opposition relation. Alliance/Opposition Gazetteers. We constituted manually a restrained lexicon of "relation triggers" containing 110 words used in the corpus to express alliance and opposition (and not a more general "polarity"). These words are verbs (agree, support, accuse, condemn, slam...), event nouns (congratulation, accusation, sanction...) and other nouns (ally...).

Date Normalization. Our aim is to associate alliance and opposition relations to dates, in order to observe the evolution of these relations in time. We used Heideltime [START_REF] Strötgen | [END_REF] to normalize dates inside the documents. Normalization is the operation of turning a temporal expression (e.g., July 26th, yesterday, Last Tuesday) into a formated, fully specified representation (2013-07-26). Chunking.

Parse trees are obtained from the Stanford parser [START_REF] Klein | [END_REF]], and we extract minimal chunks (NPs, VPs) from these trees.

Classification

It is important to note that we do not claim to build a generic classifier of such relations. Our aim is to extract enough relations to be able to produce realistic, substantial and significant data about international relations between countries. At classifier level, a good precision will be favored, because we rely on the redundancy of information in the collection to achieve an appropriate coverage (i.e., recall) of relations. Therefore, we will learn a classifier to identify relations that are expressed explicitely, in the same sentence. Training Set. We randomly selected sentences containing at least two entities separated by less than 15 chunks, and at least one relation trigger between the two entities.

Each pair of entities satisfying these constraints are an instance of the classifier, which means that one sentence can lead to several instances. e.g., in: We annotated 2105 such instances with relation NIL (no relation, 1463 instances), NEG (opposition, 349 instances) and POS (alliance, 293 instances). Classifier. Sentences do not differ in their structure whether they express alliance or opposition. Entities have the same kinds of interactions with each other. Only the polarity of trigger words matters. Therefore, we implement a two-step SVM classification:

A. Filtering out pairs that have no relation at all, i.e. classifying between NIL and non-NIL relations; B. Among non-NIL relations, classifying between POS and NEG relations;

These classifiers were evaluated by a 10-fold crossvalidation. Results are presented in Table 2. The final model used for next steps is trained with all annotated instances.

Time-Aware Aggregation and Visualization

The alliance classifier described in previous section extracts a total of 330,222 instances of relations from the corpus. If a date has been identified and normalized in a sentence, then this date stamps the sentence. If Heideltime was unable to normalize the date, then the sentence is skipped. Otherwise (no date found), the document creation time stamps the sentence. A typical query is then composed of a few keywords representing the topic, a temporal interval (minimum of maximum dates) and zero or more country names on which the user wants to restrict relation extraction.

For all pairs of considered countries, inside the same day d, the weight for the pair and the day d is:

w(d) = log( 1 + P (d) 1 + N (d) )
where P (d) and N (d) are the number of POS and NEG relations between the two countries. w(d) is a number between -∞ and +∞, where w = 0 is neutral, w < 0 is an opposition and w > 0 is an alliance. The noise is then reduced by a weighted mean smoothing over a temporal window of 5 days.

Visualization

For bilateral relations (field countries containing two items), we provide the user with a time-series plot representing sw(d), and show them on demand the sentences which led to this value. Figure 3 shows an example of results concerning the relations between United States and Russia ("countries:United States AND Russia") concerning the situation in Syria ("keywords:syria").

When zero, one or more than two countries are specified by the user, we generate a graph of countries, where the distance between vertices reflects the opposition between the countries in a given time span (Figure 4). We use the Barnes-Hut force-directed layout algorithm, where a value between two vertices is considered as a repulsive force. The color of the nodes reflects the their proximity with each other (using a first-neighbor shortest path algorithm).

For this algorithm, we need to transform our weights sw(d) into positive numbers (repulsive forces). We also need to damp the noise and the variations of the weights that are introduced by the volume of data. For example, two positive values of 1 and 3.5 (diff. = 2.5) should be considered as close to each other, while a positive value of 1 should be far from a negative value of -1 (diff. = 2). This kind of effects can be corrected by a "S-shaped", logistic function, that models a level of saturation after an approximately exponential growth (or, in our case, decrease): (d) (5)

sw (d) = 1 - 1 1 + e -sw
This function levels off high weights (both negative and positive), increases differences between positive and negative values and thus helps reducing noise without having to discretize values arbitrarily. Resulting numbers are all positive, between 0 and 1, where 1 is a strong opposition (then, repulsion in the graph) and 0 is a strong alliance (attraction in the graph), while 0.5 is neutral.

Evaluation

Evaluating the relevance of the produced trends is very subjective and would require a high level of expertise in every tested domain. Even if this work is carried out in collaboration with journalists, we cannot afford such an effort. That is why we opted for a protocol that is at the same time more objective and easier to conduct:

1. We chose 14 queries with the following information:

names of two countries (or unions of countries) involved in the relation, and an optional keyword-based thematic restriction. We selected queries having potentially a high density of extracted relations (e.g. North Korea vs. South Korea, or Russia vs. United Nations on "Syria"), as well as sparser topics (France vs. Germany on "austerity", China vs. Japan on maritime affairs). 2. On the resulting plot, we selected up to 5 strong peaks -abs(sw(d)) > 1 -and 5 weak peaksabs(sw(d)) ≤ 1 (total of 97 relations).

3. For each of these peaks, we estimated whether the polarity of the peak was relevant or not. For that purpose, we seeked news articles from a time-stamped web collection that was not part of the tested collection, in order to validate whether the two countries rather agreed or opposed at the date indicated by the peak. This is still a heavy task, which explains the low number of tested instances.

The accuracy of strong peaks is 0.90, making them highly reliable. Accuracy of weak peaks is 0.702. Note that very 6 Discussion and Perspectives

The system described in this paper shows that it is possible to use NLP techniques to aggregate information and provide reliable numerical data that could hardly be obtained in another way. This kind of NLP-driven data journalism can bring significant added value to journalists, as well as end users, in many fields. This opens the way for many applications of the same kind, for studying relations between people, countries, organisations in any domain.

The main guidelines for building such a system are:

• Think about precision first. Do not neglect recall, since it is all about getting data, but the vast amount of information can make up the lack of coverage.

• High variation in the corpus is better than low variation, to have more chance to get data from your accurate classifier, and to give less importance to misclassifications. • Smoothing the resulting data within temporal windows and correcting them with logistic functions is essential for hiding errors and producing reliable information. • Make it time-aware. Interests are two-fold: some data are valid only in a limited time range, and the evolution of data can be a main interest of the study. A wide new range of knowledge can thus become available to data journalists, who generally make use of factual data from structured bases. Such applications would bring high added value to the final users, in terms of aggregation, contextualization and hierarchical organization of information. We need however to reduce drastically the amount of needed supervision in order to make NLP enter the journalism world.

Figure 2 :

 2 Figure 1: System overview.

( 4 )

 4 In Jerusalem, Prime Minister Silvio Berlusconi pledged Italy's firm support for Israel and urged effective sanctions against Tehran. Entities are in bold and relation triggers in italics. Relations to classify are then {Italy (S. Berlusconi), Israel}, {Italy (S. Berlusconi), Iran (Tehran)}, {Italy, Israel}, {Italy, Iran (Tehran)} and {Israel, Iran (Tehran)}. The resulting relations would be NEG(Italy, Iran) and POS(Italy, Iran).

Figure 3 :

 3 Figure 3: Example of plot produced by the system for bilateral relations between United States and Russia on the query "Syria". The bottom left frame shows sentences corresponding to the user-selected date (Sep. 17, 2013). Circled numbers have been manually added to the screenshot. They correspond to: x Mutual accusations of supplying arms to Syrian authorities or opposition (bad relation, sw(d) << 0); y Planning of a meeting to discuss the problem (better relation, sw(d) > 0); z Vetos of China and Russia for United Nations resolutions; { Announcement of a peace conference; | Agreement at this conference.

Figure 4 :

 4 Figure 4: Example of graph produced by the system for relations between different states on the query "syria", for the year 2012. The graph is based on information collected in 18,582 sentences. Edge colors indicate the kind of relation (from dark red for strong alliance to dark blue for strong opposition), and vertice colors reflects proximity of countries with each other.

Table 1 :

 1 Features used for the classifiers A and B.

		STEP A
	Entities
	-Distance between entities, sentence length, positions of entities
	-Type of E1 and E2 (capital, person name, country name)
	-Number of entities around and inside entities
	Lexical features
	-Presence, number and type (verb, noun) of triggers around or
	between entities.
	-Negation between E1 and E2
	-Among 20 most frequent prepositions in the corpus: those oc-
	curring just before and just after E1 or E2
	Syntactic features
	-Whether E1 (resp. E2) is the head of its chunk, size of chunks
	-Number of verbs, nouns around entities
		STEP B
	Distinction between NEG and POS triggers, negation
	-Number of positive (resp. negative) triggers in the sentence
	-Number of positive (resp. negative) triggers around entities
	-Negation marks
		Relation	Precision Recall	F1
	Step A non-NIL	0.82	0.76	0.79
		NIL	0.90	0.93	0.91
		average	0.87	0.88	0.87
	Step B POS	0.95	0.99	0.97
		NEG	0.99	0.95	0.97
		average	0.97	0.97	0.97
	A + B	POS & NEG	0.80	0.73	0.76

Table 2 :

 2

10-fold cross validation results for the alliance/opposition classifier.


Xavier Tannier Limsi

Frédéric Vernier
email: frederic.vernier@limsi.fr

Creation, Visualization and Edition of Timelines for Journalistic Use

We describe in this article a system for building and visualizing thematic timelines automatically. The input of the system is a set of keywords, together with temporal user-specified boundaries. The output is a timeline graph showing at the same time the chronology and the importance of the events concerning the query. This requires natural language processing and information retrieval techniques, allied to a very specific temporal smoothing and visualization approach. The result can be edited so that the journalist always has the final say on what is finally displayed to the reader.

Introduction

Timelines are a natural way to describe series of related events in a compact manner, and journalists use them a lot. However, writing and maintaining such timelines, as well as building a comprehensive visualization, requires a considerable amount of human effort.

For this reason, automatic timeline summarization (TS) has known a wide interest in the last past years. TS is generally seen as a special case of multi-document summarization. For that matter, multi-document summarization systems have been used to generate timelines, and focus on the selection of the most representative sentences in an already time-stamped corpus [START_REF][END_REF]Chieu and Lee, 2004;Tran et al., 2015b]. Some previous work have focused on extracting salient dates before selecting the description of events corresponding to these dates [START_REF][END_REF]2015a;[START_REF] Kessler | [END_REF][START_REF] Nguyen | [END_REF].

The final output of these systems is generally made of the k top ranked events, presented in chronological order. The visualization can then be obtained with traditional librairies such as TimelineJS, SIMILE, TimeGlider, vis.js (see Figure 1). The information concerning the rank and the weight of the events is only used for selecting the top k, and is then lost.

We argue that readable timelines (or chronologies) should present first an overview with the most important events, but also let the reader discover intermediate events at will. A timeline must certainly be followed along a temporal axis, but a feedback of the importance of the events should also be displayed.

In this paper, we follow [START_REF] Nguyen | [END_REF]] and describe a system taking a set of keywords as an input, producing an output that is not a constrained summary or list of events, but a weighted list of dates, together with a description of the event that occurred at each date. We first focus on how the articles are processed in order to rank the dates, and especially on how the events are time-stamped. We show that considering only the article publication date does lead to shifted peaks, and then to irrelevant timelines. For this reason, we use a temporal normalization of texts to adjust the peaks. Then, we choose the best article headline related to the date and topic, as an event description.

Finally, we present a visualization tool specially dedicated to this system, where all the extracted events in the considered time span can be shown, and where the importance of the events is symbolized by a time-series graph filtered through event-specific smoothing functionalities.

The system is demonstrated on French data1 .

System

The first step of our approach can be seen as a task of "date extraction". Our system extracts a maximum of temporal information and uses only this information to detect salient dates for the construction of event timelines. Then, textual content is used for selecting the description of each event. Finally, an original data visualization is proposed.

Event Extraction

Figure 2 shows the general architecture of the system:

x The system Heideltime [START_REF] Strötgen | Jannik Strötgen and Michael Gertz. Multilingual and Cross-domain Temporal Tagging[END_REF][START_REF] Moriceau | Véronique Moriceau and Xavier Tannier. French Resources for Extraction and Normalization of Temporal Expressions with HeidelTime[END_REF] is used to normalize temporal expressions in the texts. Absolute (e.g. "January 6, 2016") and relative dates (e.g. "on Friday") are turned into a YYYY-MM-DD common format (see examples in Figure 3). This allows us to link event to specific dates, instead of relying on the document creation time. y The corpus is indexed by the Solr search engine2 , where one document per sentence is created, considering only sentences containing a normalized date. Each sentence is indexed together with the dates and the title of its article, using stemming. z At query time, documents are retrieved from the index, without any number limit. { Dates are extracted from the documents and weighted according to the number of occurrences of the date in the retrieved documents. Thus we obtain a plot where each peak corresponds to an "important" date. This is why considering dates inside the text instead of the document creation time is important: using document creation time gives us a measure of the mediatic response to events, making the peaks match with the days after the events. Retiming the events w.r.t. the dates specified in the text allows to reposition the peaks in front of the actual date of the events (see an illustration at Figure 4). | We then need to associate a textual description to each event. This is done by collecting the more important words for each date with a classical tf.idf:

tf.idf (w, d) = tf (w, d) log N df (w)
where tf (w, d) is the frequency of word w in all sentences containing the date d, df (w) is the frequency of

Visualization Tool

Figure 5 shows an example of graph produced by the system. On top, the most relevant events are presented (headline and picture from the selected article for the specific day). At bottom, the graph is displayed along the same temporal axis. The graph represents the weights of each day as calculated at step {. However, like all measures representing a human activity, these weights lead to a very noisy graph. We provide then a smoothing function to make the graph more readable to the user. A traditional Gaussian blur can be added and controled to obtain a smoother curve, but it also shifts the maxima to the right. Therefore, it would lose the temporal signature of the burst and decoy model (Descending Triangle Reversal [START_REF] Hochheiser | [END_REF]). In consequence, we added another smoothing functionality based on Bilateral Filtering [START_REF] Paris | [END_REF] to preserve the discontinuity at event burst. However it does not match our burst and decoy model since days just before a burst are raised by the following days if decoy happens into the kernel size. We refined then the Bilateral Filtering by accounting only past events in the smoothing function. This function is then: Even if a smoothing is necessary, we still aim at obtaining strong and sharp peaks when important events occur. Instead of using a pure burst model as in Kleinberg [2002] or [START_REF] Zhu | [END_REF], which have already been applied to media content [START_REF] Xie | [END_REF]Takahashi et al., 2012], we prefer using our refined Bilateral Filtering with a decreasing threshold detection. We use the double gaussian Kernels of the Bilateral Filtering as the aggregate function F of the Shasha Model [START_REF] Zhu | [END_REF]. The article at highest burst is selected then removed from the time serie. This process is repeated until the timeline is filled with the targeted number of selected events.

Selected articles are displayed with both an image and the title of the article underneath. The system crops the image to a flag shape to highlight the temporal nature of events. The flag shape is attached to the graph with a line from the triangle. When two events happen very close to each other the flags can float on different sides of their pole (first event is dis-played on the left to avoid overlapping). When more than two events occur very close we chose to display them arbitrarily on different tracks. As we process selected events from highest to lowest ranked, most important events are displayed on the top track and least important ones appear underneath. The smaller, not selected peaks display vertically the titles of the selected articles for these days.

The users of the system (i.e. journalists) can zoom on both axes by using the range sliders at bottom and at left of the combined graph. A temporal legend always display time ticks at bottom of the graph. The users can interactively rearrange events since the layout mechanism can fail to optimize screen real estate. Users can flip the flags on both side, change track (up or down) of an event and switch between two sizes (big or small). Furthermore, users can downgrade an event (and make it a smaller peak) or upgrade a smaller peak by double clicking on an vertical title above the graph. This makes the result fully editable, so that the journalist has the final say on what is displayed.

Discussion

Limits

The first important limit of the system is that its event granularity is fixed. This leads to two main issues: 1/ The tool is not able to detect more than one event per day. 2/ A macroevent that would last more than one day (e.g. a conference) could not be extracted nor vizualized.

Workarounds have been considered [START_REF] Nguyen | [END_REF]] but tend to reduce the overall accuracy of the system. Our further work will focus on this issue.

The definition of an event "importance" is also open to question. In this paper we considered to the importance depends only on repetition. Other factors have been studied and applied with learning-to-rank approaches [START_REF] Kessler | [END_REF], and should be integrated into this system.

Finally, the process requires a large number of search engine queries, which makes it time-consuming. A first query returns a potentially high number of documents; then, one query per day in the time span is run to select the best article. Even if they can easily be parallelized, all these queries make the entire process quite heavy3 .

Adaptation to Other Languages

This study has been achieved on a French dataset. Only two steps are language-dependent and need little adaption to another language:

• Tokenization and stemming (widely available in many languages) • Temporal normalization. Heideltime is available in 13 languages at the time of writing, and other tools may be existing for other languages.

The next step that is now being conducted consists in an evaluation with our journalist partners, and considers both the accuracy of the timeline and the ergonomics.

Figure 1 :

 1 Figure 1: An existing, manually produced timeline on the French presidential race (TimelineJS).

Figure 2 :

 2 Figure 2: System overview.

  Figure 3: Examples of sentences refering to November 13 events with absolute or relative dates. The normalized is "2015-11-13" for all the sentences.

Figure 5 :

 5 Figure 5: Visualization for the query "attentats" ("attacks") in 2015.

Figure 4 :

 4 Figure 4: Raw graphs on query "attacks" in November and Decembre 2015, with time weights given by the document creation time (left) or by the temporal normalization (right). November 13 is represented by the red vertical line.

This work has been partly founded by a Google award for computational journalism.

2 http://lucene.apache.org/solr/

Within a single thread on a simple server, about one minute for a one-year query on a popular subject as "attacks".


Maria Pontiki

Dimitrios Galanis

Haris Papageorgiou

Ion Androutsopoulos

Suresh Manandhar

Mohammad Al-Smadi

Mahmoud Al-Ayyoub

Yanyan Zhao

Bing Qin

Orphée De Clercq

Véronique Hoste

Marianna Apidianaki

Xavier Tannier

Natalia Loukachevitch

Evgeny Kotelnikov

Nuria Bel

Salud María Jiménez-Zafra

Gülşen Eryiğit

SemEval-2016 Task 5: Aspect Based Sentiment Analysis

This paper describes the SemEval 2016 shared task on Aspect Based Sentiment Analysis (ABSA), a continuation of the respective tasks of 2014 and 2015. In its third year, the task provided 19 training and 20 testing datasets for 8 languages and 7 domains, as well as a common evaluation procedure. From these datasets, 25 were for sentence-level and 14 for text-level ABSA; the latter was introduced for the first time as a subtask in SemEval. The task attracted 245 submissions from 29 teams.

Introduction

Many consumers use the Web to share their experiences about products, services or travel destinations [START_REF] Hyan | What Motivates Consumers to Write Online Travel Reviews?[END_REF]. Online opinionated texts (e.g. reviews, tweets) are important for consumer decision making [START_REF] Chevalier | The effect of word of mouth on sales: Online book reviews[END_REF] and constitute a source of valuable customer feedback that can help companies to measure satisfaction and improve their products or services. In this setting, Aspect Based Sentiment Analysis (ABSA) -i.e., mining opinions from text about specific entities and their aspects [START_REF] Liu | Sentiment Analysis and Opinion Mining[END_REF] -can provide valuable insights to both consumers and businesses. * *Corresponding author: mpontiki@ilsp.gr. An ABSA method can analyze large amounts of unstructured texts and extract (coarse-or fine-grained) information not included in the user ratings that are available in some review sites (e.g. Fig. 1).

Sentiment Analysis (SA) touches every aspect (e.g. entity recognition, coreference resolution, negation handling) of Natural Language Processing [START_REF] Liu | Sentiment Analysis and Opinion Mining[END_REF] and as [START_REF] Cambria | New Avenues in Opinion Mining and Sentiment Analysis[END_REF] mention "it requires a deep understanding of the explicit and implicit, regular and irregular, and syntactic and semantic language rules". Within the last few years several SA-related shared tasks have been organized in the context of workshops and conferences focus-ing on somewhat different research problems [START_REF] Seki | Overview of Opinion Analysis Pilot Task at NTCIR-6[END_REF][START_REF] Seki | Overview of Multilingual Opinion Analysis Task at NTCIR-7[END_REF][START_REF] Seki | Overview of Multilingual Opinion Analysis Task at NTCIR-8: A Step Toward Cross Lingual Opinion Analysis[END_REF][START_REF] Mitchell | Overview of the TAC2013 Knowledge Base Population Evaluation English Sentiment Slot Filling[END_REF][START_REF] Nakov | SemEval-2013 Task 2: Sentiment Analysis in Twitter[END_REF][START_REF] Rosenthal | SemEval-2014 Task 4: Sentiment Analysis in Twitter[END_REF][START_REF] Pontiki | SemEval-2014 Task 4: Aspect Based Sentiment Analysis[END_REF][START_REF] Rosenthal | SemEval-2015 Task 10: Sentiment Analysis in Twitter[END_REF][START_REF] Ghosh | SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter[END_REF][START_REF] Pontiki | SemEval-2015 Task 12: Aspect Based Sentiment Analysis[END_REF][START_REF] Saif | SemEval-2016 Task 6: Detecting Stance in Tweets[END_REF][START_REF] Reforgiato | Eswc'14 challenge on concept-level sentiment analysis[END_REF][START_REF] Ruppenhofer | IG-GSA Shared Tasks on German Sentiment Analysis (GESTALT)[END_REF][START_REF] Loukachevitch | SentiRuEval: Testing Objectoriented Sentiment Analysis Systems in Russian[END_REF]. Such competitions provide training datasets and the opportunity for direct comparison of different approaches on common test sets.

Currently, most of the available SA-related datasets, whether released in the context of shared tasks or not [START_REF] Socher | Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank[END_REF][START_REF] Ganu | Beyond the Stars: Improving Rating Predictions using Review Text Content[END_REF], are monolingual and usually focus on English texts. Multilingual datasets [START_REF] Klinger | The USAGE Review Corpus for Fine Grained Multi Lingual Opinion Analysis[END_REF][START_REF] Salud | A Multilingual Annotated Dataset for Aspect-Oriented Opinion Mining[END_REF] provide additional benefits enabling the development and testing of crosslingual methods [START_REF] Lambert | Aspect-Level Cross-lingual Sentiment Classification with Constrained SMT[END_REF]. In this direction, this year the SemEval ABSA task provided datasets in a variety of languages.

ABSA was introduced as a shared task for the first time in the context of SemEval in 2014; SemEval-2014 Task 4 1 (SE-ABSA14) provided datasets of English reviews annotated at the sentence level with aspect terms (e.g., "mouse", "pizza") and their polarity for the laptop and restaurant domains, as well as coarser aspect categories (e.g., food) and their polarity only for restaurants [START_REF] Pontiki | SemEval-2014 Task 4: Aspect Based Sentiment Analysis[END_REF]. SemEval-2015 Task 12 2 (SE-ABSA15) built upon SE-ABSA14 and consolidated its subtasks into a unified framework in which all the identified constituents of the expressed opinions (i.e., aspects, opinion target expressions and sentiment polarities) meet a set of guidelines and are linked to each other within sentence-level tuples [START_REF] Pontiki | SemEval-2015 Task 12: Aspect Based Sentiment Analysis[END_REF]. These tuples are important since they indicate in which part of the text a specific opinion is expressed. However, a user is also interested in the overall rating of the text towards a particular aspect. Such ratings can be used to estimate the mean sentiment per aspect from multiple reviews [START_REF] Mcauley | Learning Attitudes and Attributes from Multiaspect Reviews[END_REF]. In addition to sentence-level annotations, SE-ABSA16 3 accommodated also text-level ABSA annotations and provided the respective training and testing data. Furthermore, the SE-ABSA15 1 http://alt.qcri.org/semeval2014/task4/ 2 http://alt.qcri.org/semeval2015/task12/ 3 http://alt.qcri.org/semeval2016/task5/ annotation framework was extended to new domains and applied to languages other than English (Arabic, Chinese, Dutch, French, Russian, Spanish, and Turkish).

The remainder of this paper is organized as follows: the task set-up is described in Section 2. Section 3 provides information about the datasets and the annotation process, while Section 4 presents the evaluation measures and the baselines. General information about participation in the task is provided in Section 5. The evaluation scores of the participating systems are presented and discussed in Section 6. The paper concludes with an overall assessment of the task.

Task Description

The SE-ABSA16 task consisted of the following subtasks and slots. Participants were free to choose the subtasks, slots, domains and languages they wished to participate in.

Subtask 1 (SB1): Sentence-level ABSA. Given an opinionated text about a target entity, identify all the opinion tuples with the following types (tuple slots) of information:

• Slot1: Aspect Category. Identification of the entity E and attribute A pairs towards which an opinion is expressed in a given sentence. E and A should be chosen from predefined inventories4 of entity types (e.g., restaurant, food) and attribute labels (e.g., price, quality).

• Slot2: Opinion Target Expression (OTE).

Extraction of the linguistic expression used in the given text to refer to the reviewed entity E of each E#A pair. The OTE is defined by its starting and ending offsets. When there is no explicit mention of the entity, the slot takes the value null. The identification of Slot2 values was required only in the restaurants, hotels, museums and telecommunications domains.

• Slot3: Sentiment Polarity. Each identified E#A pair has to be assigned one of the following polarity labels: positive, negative, neutral (mildly positive or mildly negative sentiment). An example of opinion tuples with Slot 1-3 values from the restaurants domain is shown below: "Their sake list was extensive, but we were looking for Purple Haze, which wasn't listed but made for us upon request!" → {cat: "drinks#style_options", trg: "sake list", fr: "6", to: "15", pol: "positive"}, {cat: ``service#general", trg: ``null", fr: "0", to: "0", pol: "positive"}. The variable cat indicates the aspect category (Slot1), pol the polarity (Slot3), and trg the ote (Slot2); f r, to are the starting/ending offsets of ote.

Subtask 2 (SB2): Text-level ABSA. Given a customer review about a target entity, the goal was to identify a set of {cat, pol} tuples that summarize the opinions expressed in the review. cat can be assigned the same values as in SB1 (E#A tuple), while pol can be set to "positive", "negative", "neutral", or "conflict". For example, for the review text "The So called laptop Runs to Slow and I hate it! Do not buy it! It is the worst laptop ever ", a system should return the following opinion tuples: {cat: "laptop#general", pol: "negative"}, {cat: "laptop#operation_performance", pol: "negative"} .

Subtask 3 (SB3): Out-of-domain ABSA. In SB3 participants had the opportunity to test their systems in domains for which no training data was made available; the domains remained unknown until the start of the evaluation period. Test data for SB3 were provided only for the museums domain in French.

Datasets

Data Collection and Annotation

A total of 39 datasets were provided in the context of the SE-ABSA16 task; 19 for training and 20 for testing. The texts were from 7 domains and 8 languages; English (en), Arabic (ar), Chinese (ch), Dutch (du), French (fr), Russian (ru), Spanish (es) and Turkish (tu). The datasets for the domains of restaurants (rest), laptops (lapt), mobile phones (phns), digital cameras (came), hotels (hote) and museums (muse) consist of customer reviews, whilst the telecommunication domain (telc) data consists of tweets. A total of 70790 manually annotated ABSA tuples were provided for training and testing; 47654 sentencelevel annotations (SB1) in 8 languages for 7 domains, and 23136 text-level annotations (SB2) in 6 languages for 3 domains. Table 1 provides more information on the distribution of texts, sentences and tuples per dataset.

The rest, hote, and lapt datasets were annotated at the sentence-level (SB1) following the respective annotation schemas of SE-ABSA15 [START_REF] Pontiki | SemEval-2015 Task 12: Aspect Based Sentiment Analysis[END_REF]. Below are examples 5 of annotated sentences for the aspect category service#general in en (1), du (2), fr (3), ru (4), es (5), and tu (6) for the rest domain and in ar (7) for the hote domain:

1. Service was slow, but the people were friendly. → {trg: "Service", pol: "negative"}, {trg: "people", pol: "positive"} 2. Snelle bediening en vriendelijke personeel moet ook gemeld worden!! → {trg: "bediening", pol: "positive"}, {trg: "personeel", pol: "positive"} 3. Le service est impeccable, personnel agréable. → {trg: "service" , pol: "positive"}, {trg: "personnel", pol: "positive"} 4. Про сервис ничего негативного не скажешьбыстро подходят, все улябаются, подходят спрашивают, всё ли нравится. → { "сервис", pol: "neutral" } 5. También la rapidez en el servicio. → {trg: "servicio", pol: "positive" } 6. Servisi hızlı valesi var. → {trg: "Servisi", pol: "positive"} 7. .. ﺳﺮﯾﻌﺔ و ﺟﺪا ﺟﯿﺪة اﻟﺨﺪﻣﺔ → {trg: "اﻟﺨﺪﻣﺔ" , pol: "positive"}

The lapt annotation schema was extended to two other domains of consumer electronics, came and phns. Examples of annotated sentences in the lapt (en), phns (du and ch) and came (ch) domains are shown below:

1. It is extremely portable and easily connects to WIFI at the library and elsewhere. → {cat: "laptop#portability", pol: "positive"} , {cat: "laptop#connectivity", pol: "positive"} 2. Apps starten snel op en werken vlot, internet gaat prima.

→ {cat: "software#operation_performance", pol: "positive"}, {cat: "phone#connectivity", pol: "positive"} 5 The offsets of the opinion target expressions are ommited.

3. 当然屏幕这么好 →{cat: "display#quality", pol: "positive"} 4. 更 轻 便 的 机 身 也 便 于 携 带。→ {cat: "camera# portability", pol: "positive"}

In addition, the SE-ABSA15 framework was extended to two new domains for which annotation guidelines were compiled: telc for tu and muse for fr. Below are two examples:

1. #Internet kopuyor sürekli :( @turkcell → {cat:

"internet#coverage", trg: "Internet", pol: "positive"} 2. 5€ pour les étudiants, ça vaut le coup. → {cat: "museum#prices", "null", "positive"}

The text-level (SB2) annotation task was based on the sentence-level annotations; given a customer review about a target entity (e.g. a restaurant) that included sentence-level annotations of ABSA tuples, the goal was to identify a set of {cat, pol} tuples that summarize the opinions expressed in it. This was not a simple summation/aggregation of the sentencelevel annotations since an aspect may be discussed with different sentiment in different parts of the review. In such cases the dominant sentiment had to be identified. In the case of conflicting opinions where the dominant sentiment was not clear, the "conflict" label was assigned. In addition, each review was assigned an overall sentiment label about the target entity (e.g., restaurant#general, laptop#general), even if it was not included in the sentence-level annotations.

Annotation Process

All datasets for each language were prepared by one or more research groups as shown in Table 2. The en, du, fr, ru and es datasets were annotated using brat [START_REF] Stenetorp | BRAT: A Web-based Tool for NLP-Assisted Text Annotation[END_REF], a web-based annotation tool, which was configured appropriately for the needs of the task. The tu datasets were annotated using a customized version of turksent [START_REF] Eryigit | TURKSENT: A Sentiment Annotation Tool for Social Media[END_REF], a sentiment annotation tool for social media. For the ar and the ch data in-house tools6 were used. Below are some further details about the annotation process for each language.

English. The SE-ABSA15 [START_REF] Pontiki | SemEval-2015 Task 12: Aspect Based Sentiment Analysis[END_REF] training and test datasets (with some minor corrections) were merged and provided for training (rest and lapt domains). New data was collected and annotated from scratch for testing. In a first phase, the rest test data was annotated by an experienced 7 linguist (annotator A), and the lapt data by 5 undergraduate computer science students. The resulting annotations for both domains were then inspected and corrected (if needed) by a second expert linguist, one of the task organizers (annotator B). Borderline cases were resolved collaboratively by annotators A and B.

Arabic. The hote dataset was annotated in repeated cycles. In a first phase, the data was annotated by three native Arabic speakers, all with a computer science background; then the output was validated by a senior researcher, one of the task organizers. If needed (e.g. when inconsistencies were found) they were given back to the annotators.

Chinese. The datasets presented by [START_REF] Zhao | Creating a Fine-Grained Corpus for Chinese Sentiment Analysis[END_REF] were re-annotated by three native Chinese speakers according to the SE-ABSA16 annotation schema and were provided for training and testing (phns and came domains).

Dutch. The rest and phns datasets presented in (Clercq and Hoste, 2016) were initially annotated by a trained linguist, native speaker of Dutch. Then, the output was verified by another Dutch linguist and disagreements were resolved between them. Fi-7 Also annotator for SE-ABSA14 and 15.

nally, the task organizers inspected collaboratively all the annotated data and corrections were made when needed.

French. The train (rest) and test (rest, muse) datasets were annotated from scratch by a linguist, native speaker of French. When the annotator was not confident, a decision was made collaboratively with the organizers. In a second phase, the task organizers checked all the annotations for mistakes and inconsistencies and corrected them, when necessary. For more information on the French datasets consult [START_REF] Apidianaki | A Dataset for Aspect-Based Sentiment Analysis in French[END_REF].

Russian. The rest datasets of the SentiRuEval-2015 task [START_REF] Loukachevitch | SentiRuEval: Testing Objectoriented Sentiment Analysis Systems in Russian[END_REF] were automatically converted to the SE-ABSA16 annotation schema; then a linguist, native speaker of Russian, checked them and added missing information. Finally, the datasets were inspected by a second linguist annotator (also native speaker of Russian) for mistakes and inconsistencies, which were resolved along with one of the task organizers.

Spanish. Initially, 50 texts (134 sentences) from the whole available data were annotated by 4 annotators. The inter-anotator agreement (IAA) in terms of F-1 was 91% for the identification of OTE, 88% for the aspect category detection (E#A pair), and 80% for opinion tuples extraction (E#A, OTE, polarity). Provided that the IAA was substantially high for all slots, the rest of the data was divided into 4 parts and each one was annotated by a different native Spanish speakers (2 linguists and 2 software engineers). Subsequently, the resulting annotations were validated and corrected (if needed) by the task organizers.

Turkish. The telc dataset was based on the data used in [START_REF] Yıldırım | The impact of nlp on turkish sentiment analysis[END_REF], while the rest dataset was created from scratch. Both datasets were annotated simultaneously by two linguists. Then, one of the organizers validated/inspected the resulting annotations and corrected them when needed.

Datasets Format and Availability

Similarly to SE-ABSA14 and SE-ABSA15, the datasets 8 of SE-ABSA16 were provided under specific license terms through META-SHARE 9 , a repository devoted to the sharing and dissemination of language resources [START_REF] Piperidis | The META-SHARE Language Resources Sharing Infrastructure: Principles, Challenges, Solutions[END_REF].

Evaluation Measures and Baselines

The evaluation ran in two phases. In the first phase (Phase A), the participants were asked to return separately the aspect categories (Slot1), the OTEs (Slot2), and the {Slot1, Slot2} tuples for SB1. For SB2 the respective text-level categories had to be identified. In the second phase (B), the gold annotations for the test sets of Phase A were provided and participants had to return the respective sentiment polarity values (Slot3). Similarly to SE-ABSA15, F-1 scores were calculated for Slot1, Slot2 and {Slot1, Slot2} tuples, by comparing the annotations that a system returned to the gold annotations (using micro-averaging). For Slot1 evaluation, duplicate occurrences of categories were ignored in both SB1 and SB2. For Slot2, the calculation for each sentence considered only distinct targets and discarded null targets, since they do not correspond to explicit mentions. To evaluate sentiment polarity classification (Slot3) in Phase B, we calculated the accuracy of each system, defined as the number of correctly predicted polarity labels of the (gold) aspect categories, divided by the total number of the gold aspect categories. Furthermore, we implemented and provided baselines for all slots of SB1 and SB2. In particular, the SE-ABSA15 baselines that were implemented for the English language [START_REF] Pontiki | SemEval-2015 Task 12: Aspect Based Sentiment Analysis[END_REF], were adapted for the other languages by using 8 The data are available at: http://metashare.ilsp. gr:8080/repository/search/?q=semeval+2016 9 META-SHARE (http://www.metashare.org/) was implemented in the framework of the META-NET Network of Excellence (http://www.meta-net.eu/). the appropriate stopword lists and tokenization functions. The baselines are briefly discussed below: SB1-Slot1: For category (E#A) extraction, a Support Vector Machine (SVM) with a linear kernel is trained. In particular, n unigram features are extracted from the respective sentence of each tuple that is encountered in the training data. The category value (e.g., service#general) of the tuple is used as the correct label of the feature vector. Similarly, for each test sentence s, a feature vector is built and the trained SVM is used to predict the probabilities of assigning each possible category to s (e.g., {service#general, 0.2}, {restaurant#general, 0.4}. Then, a threshold10 t is used to decide which of the categories will be assigned11 to s. As features, we use the 1,000 most frequent unigrams of the training data excluding stop-words.

SB1-Slot2: The baseline uses the training reviews to create for each category c (e.g., service#general) a list of OTEs (e.g., service#general → {"staff", "waiter"}). These are extracted from the (training) opinion tuples whose category value is c . Then, given a test sentence s and an assigned category c, the baseline finds in s the first occurrence of each OTE of c's list. The OTE slot is filled with the first of the target occurrences found in s. If no target occurrences are found, the slot is assigned the value "null".

SB1-Slot3: For polarity prediction we trained a SVM classifier with a linear kernel. Again, as in Slot1, n unigram features are extracted from the respective sentence of each tuple of the training data. In addition, an integer-valued feature12 that indicates the category of the tuple is used. The correct label for the extracted training feature vector is the corresponding polarity value (e.g., positive). Then, for each tuple {category, OTE} of a test sentence s, a feature vector is built and classified using the trained SVM.

SB2-Slot1: The sentence-level tuples returned by the SB1 baseline are copied to the text level and duplicates are removed.

SB2-Slot3:

For each text-level aspect category c the baseline traverses the predicted sentence-level tuples of the same category returned by the respec- tive SB1 baseline and counts the polarity labels (positive, negative, neutral). Finally, the polarity label with the highest frequency is assigned to the textlevel category c. If there are no sentence-level tuples for the same c, the polarity label is determined based on all tuples regardless of c.

The baseline systems and evaluation scripts are implemented in Java and are available for download from the SE-ABSA16 website 13 . The LibSVM package 14 [START_REF] Chang | LIBSVM: A library for support vector machines[END_REF] is used for SVM training and prediction. The scores of the baselines in the test datasets are presented in Section 6 along with the system scores.

13 http://alt.qcri.org/semeval2016/task5/index. php?id=data-and-tools 14 http://www.csie.ntu.edu.tw/~cjlin/libsvm/

Participation

The task attracted in total 245 submissions from 29 teams. The majority of the submissions (216 runs) were for SB1. The newly introduced SB2 attracted 29 submissions from 5 teams in 2 languages (en and sp). Most of the submissions (168) were runs for the rest domain. That was expected, mainly for two reasons; first, the rest classification schema is less fine-grained (complex) compared to the other domains (e.g., lapt). Secondly, this domain was supported for 6 languages enabling also multilingual or language-agnostic approaches. The remaining submissions were distributed as follows: 54 in lapt, 12 in phns, 7 in came and 4 in hote.

An interesting observation is that, unlike SE-ABSA15, Slot1 (aspect category detection) attracted significantly more submissions than Slot2 (OTE extraction); this may indicate a shift towards conceptlevel approaches. Regarding participation per language, the majority of the submissions (156/245) were for en; see more information in Table 5. Most teams (20) submitted results only for one language (18 for en and 2 for ru). From the remaining teams, 3 submitted results for 2 languages, 5 teams submitted results for 3-7 languages, while only one team participated in all languages.

Evaluation Results

The evaluation results are presented in Tables 3 (sb1rest-en), 4 (sb1-rest-es,fr,ru,du,tu & sb1-hotear), 6 (sb1-lapt,came,phns), and 7 (sb2) 15 . Each participating team was allowed to submit up to two runs per slot and domain in each phase; one constrained (C), where only the provided training data could be used, and one unconstrained (U), where other resources (e.g., publicly available lexica) and additional data of any kind could be used for training. In the latter case, the teams had to report the resources used. Delayed submissions (i.e., runs submitted after the deadline and the release of the gold annotations) are marked with "*".

As revealed by the results, in both SB1 and SB2 the majority of the systems surpassed the baseline by a small or large margin and, as expected, the unconstrained systems achieved better results than the constrained ones. In SB1, the teams with the highest scores for Slot1 and Slot2 achieved similar F-1 scores (see Table 3) in most cases (e.g. en/rest, es/rest, du/rest, fr/rest), which shows that the two slots have a similar level of difficulty. However, as expected, the {Slot1, Slot2} scores were significantly lower since the linking of the target expressions to the corresponding aspects is also required. The highest scores in SB1 for all slots (Slot1, Slot2, {Slot1, Slot2}, Slot3) were achieved in the en/rest; this is probably due to the high participation and to the lower complexity of the rest annotation schema compared to the other domains. If we compare the results for SB1 and SB2, we notice that the SB2 scores for Slot1 are significantly higher (e.g. en/lapt, en/rest, es/rest) even though the respective annotations are for the same (or almost the same) set of texts. This is due to the fact that it is easier to identify whether a whole text discusses an aspect c than finding all the sentences in the text discussing c . On the other hand, for Slot3, the SB2 scores are lower (e.g. en/rest, es/rest, ru/rest, en/lapt) than the respective SB1 scores. This is mainly because an aspect may be discussed at different points in a text and often with different sentiment. In such cases a system has to identify the dominant sentiment, which usually is not trivial.

Conclusions

In its third year, the SemEval ABSA task provided 19 training and 20 testing datasets, from 7 domains and 8 languages, attracting 245 submissions from 29 teams. The use of the same annotation guidelines for domains addressed in different languages gives the opportunity to experiment also with crosslingual or language-agnostic approaches. In addition, SE-ABSA16 included for the first time a text- level subtask. Future work will address the creation of datasets in more languages and domains and the enrichment of the annotation schemata with other types of SA-related information like topics, events and figures of speech (e.g. irony, metaphor).

Figure 1 :

 1 Figure 1:Table summarizing the average sentiment for each aspect of an entity.

Table 1 :

 1 Datasets provided for ABSA.

	Lang. Domain Subtask	Train #Texts #Sent. #Tuples #Texts #Sent. #Tuples Test
	EN	REST	SB1	350	2000	2507	90	676	859
	EN	REST	SB2	335	1950	1435	90	676	404
	EN	LAPT	SB1	450	2500	2909	80	808	801
	EN	LAPT	SB2	395	2375	2082	80	808	545
	AR	HOTE	SB1	1839	4802	10509	452	1227	2604
	AR	HOTE	SB2	1839	4802	8757	452	1227	2158
	CH	PHNS	SB1	140	6330	1333	60	3191	529
	CH	CAME	SB1	140	5784	1259	60	2256	481
	DU	REST	SB1	300	1711	1860	100	575	613
	DU	REST	SB2	300	1711	1247	100	575	381
	DU	PHNS	SB1	200	1389	1393	70	308	396
	FR	REST	SB1	335	1733	2530	120	696	954
	FR	MUSE	SB3	-	-	-	162	686	891
	RU	REST	SB1	302	3490	4022	103	1209	1300
	RU	REST	SB2	302	3490	1545	103	1209	500
	ES	REST	SB1	627	2070	2720	286	881	1072
	ES	REST	SB2	627	2070	2121	286	881	881
	TU	REST	SB1	300	1104	1535	39	144	159
	TU	REST	SB2	300	1104	972	39	144	108
	TU	TELC	SB1	-	3000	4082	-	310	336

  EnglishInstitute for Language and Speech Processing, Athena R.C., Athens, Greece Dept. of Informatics, Athens University of Economics and Business, Greece Arabic Computer Science Dept., Jordan University of Science and Technology Irbid, Jordan Chinese Harbin Institute of Technology, Harbin, Heilongjiang, P.R.

	Lang.	Research team(s)
		China
	Dutch	Ghent University, Ghent, Belgium
	French	LIMSI, CNRS, Univ. Paris-Sud, Université Paris-Saclay, Orsay, France
	Russian	Lomonosov Moscow State University, Moscow, Russian Federation Vyatka State University, Kirov, Russian Federation
	Spanish	Universitat Pompeu Fabra, Barcelona, Spain SINAI, Universidad de Jaén, Spain
	Turkish Dept. of Computer Engineering, Istanbul Technical University, Turkey
		Turkcell Global Bilgi, Turkey

Table 2 :

 2 Research teams that contributed to the creation of the datasets for each language.

Table 3 :

 3 English REST results for SB1.

Table 4 :

 4 REST and HOTE results for SB1.

Table 5 :

 5 Number of participating teams and submitted runs per language.

Table 6 :

 6 LAPT, CAME, and PHNS results for SB1.

Table 7 :

 7 Results for SB2.

	LANG./ Slot1	Slot3
	DOM	F-1	Acc.
	EN/	GTI/U/83.995	UWB/U/81.931
	REST	UWB/C/80.965	ECNU/U/81.436
		UWB/U/80.163 UWB/C/80.941
		bunji/U/79.777	ECNU/C/78.713
		basel./C/78.711	basel./C/74.257
		SYSU/U/68.841 bunji/U/70.545
		SYSU/U/68.841 bunji/C/66.584
			GTI/U/64.109
	ES/	GTI/C/84.192	UWB/C/77.185
	REST	GTI/U/84.044	basel./C/74.548
		basel./C/74.548
		UWB/C/73.657
	RU/	basel./C/84.792 basel./C/70.6
	REST
	RU/	basel./C/84.792 basel./C/70.6
	REST
	DU/	basel./C/70.323 basel./C/73.228
	REST
	TU/	basel./C/72.642 basel./C/57.407
	REST
	AR/	basel./C/42.757 basel./C/73.216
	HOTE
	EN/	UWB/C/60.45	ECNU/U/75.046
	LAPT	UWB/U/59.721 UWB/U/75.046
		bunji/U/54.723	UWB/C/74.495
		basel./C/52.685 basel./C/73.028
		SYSU/U/48.889 ECNU/C/67.523
		SYSU/U/48.889 bunji/C/62.202
			bunji/U/60
			GTI/U/58.349

The full inventories of the aspect category labels for each domain are provided in Appendix A.

The ar annotation tool was developed by the technical team of the Advanced Arabic Text Mining group at Jordan University of Science and Technology. The ch tool was developed by the Research Center for Social Computing and Information Retrieval at Harbin Institute of Technology.

The threshold t was set to 0.2 for all datasets.

We use the -b 1 option of LibSVM to obtain probabilities.

Each E#A pair has been assigned a distinct integer value.

No submissions were made for sb3-muse-fr & sb1-telctu.

Acknowledgments

The authors are grateful to all the annotators and contributors for their valuable support to the task: Konstantina Papanikolaou, Juli Bakagianni, Omar Qwasmeh, Nesreen Alqasem, Areen Magableh, Saja Alzoubi, Bashar Talafha, Zekui Li, Binbin Li, Shengqiu Li, Aaron Gevaert, Els Lefever, Cécile Richart, Pavel Blinov, Maria Shatalova, M. Teresa Martín-Valdivia, Pilar Santolaria, Fatih Samet Çetin, Ezgi Yıldırım, Can Özbey, Leonidas Valavanis, Stavros Giorgis, Dionysios Xenos, Panos Theodor-akakos, and Apostolos Rousas. The work described in this paper is partially funded by the projects EOX GR07/3712 and "Research Programs for Excellence 2014-2016 / CitySense-ATHENA R.I.C.". The Arabic track was partially supported by the Jordan University of Science and Technology, Research Grant Number: 20150164. The French track was partially supported by the French National Research Agency under project ANR-12-CORD-0015/TransRead. The Russian track was partially supported by the Russian Foundation for Basic Research (RFBR) according to the research projects No. 14-07-00682a, 16-07-00342a, and No. 16-37-00311mol_a. The Spanish track has been partially supported by a grant from the Ministerio de Educación, Cultura y Deporte (MECD -scholarship FPU014/00983) and REDES project (TIN2015-65136-C2-1-R) from the Ministerio de Economía y Competitividad. The Turkish track was partially supported by TUBITAK-TEYDEB (The Scientific and Technological Research Council of Turkey -Technology and Innovation Funding Programs Directorate) project (grant number: 3140671).

Appendix A. Aspect inventories for all domains

Entity Labels laptop, display, keyboard, mouse, motherboard, cpu, fans_cooling, ports, memory, power_supply optical_drives, battery, graphics, hard_disk, multimedia_devices, hardware, software, os, warranty, shipping, support, company Attribute Labels general, price, quality, design_features, operation_performance, usability, portability, connectivity, miscellaneous Entity Labels phone, display, keyboard, cpu, ports, memory, power_supply, hard_disk, multimedia_devices, battery, hardware, software, os, warranty, shipping, support, company Attribute Labels Same as in Laptops (Table 8) with the exeption of portability that is included in the design_features label and does not apply as a separate attribute type. Entity Labels camera, display, keyboard, cpu, ports, memory, power_supply, battery, multimedia_devices, hardware, software, os, warranty, shipping, support, company, lens, photo, focus Attribute Labels Same as in Laptops (Table 8).