T Kosub

C Schubert

H Schletter

M Daniel

M Hietschold

V Neu

M Maret

D Makarov
email: denys.makarov@physik.tu-chemnitz.de

M Albrecht

Coercivity enhancement in exchange biased CoO/Co 3 Pt bilayers

A study on the exchange bias effect in an hcp (0002)-textured Co 3 Pt alloy film biased by a thin CoO film is presented. The Co 3 Pt film reveals an out-of-plane easy axis of magnetization although it was grown on an amorphous silica substrate at room temperature. Similar exchange bias fields measured in in-plane and out-of-plane directions of the CoO/Co 3 Pt bilayer system were observed for maximum cooling field. Whereas a pronounced cooling field dependence of the exchange bias field is observed in-plane, it remains constant in out-of-plane direction. The temperature evolution of the coercivity and the shift of the hysteresis loop caused by the exchange bias effect are discussed in terms of thermal activation accounting for the temperature dependent competition between the interfacial exchange energy and the anisotropy energy of antiferromagnetic grains.

Introduction

Discovered in 1956 as a unidirectional anisotropy [1], the exchange bias (EB) effect attracted much attention from a fundamental point of view and for applications in the field of sensorics. EB is a proximity effect appearing when a stack consisting of a ferromagnetic (F) and an antiferromagnetic (AF) layer is cooled through the Néel temperature of the AF layer in an applied magnetic field [2][3][4]. Usually EB is expressed in terms of the shift of the magnetic hysteresis loop and a modification of the coercive field of the F layer. The physical origin of EB is generally accepted to be the exchange coupling between the F and uncompensated AF spins at the F/AF interface. The microscopic way, in which this coupling translates into EB is more controversial and many models have been proposed [5][6][7][8][9][10][11][12]. So far most of the studies of the EB effect were focused on F films with in-plane (IP) easy axis of magnetization [4,13,14]. However, AF/F composites exhibiting an out-of-plane (OOP) exchange bias effect are rather attractive for applications in magnetic sensor devices operating in a current-perpendicular-to-plane geometry as they reveal superior signal-to-noise ratio compared to the current-in-plane sensors [15][16][17][18][19]. In this respect, only a few reports on OOP exchange bias systems are available [20][21][22][23][24][25]. Maat et al. [20] reported on the EB effect in Co/Pt multilayer stacks with strong perpendicular magnetic anisotropy (PMA) biased by a thin CoO AF layer. It was shown that CoO grown on (111)-textured [Co/Pt] multilayers results in an anisotropic exchange bias effect with substantially larger shift of the magnetic hysteresis loop in IP direction compared to the OOP one. In addition, the scaling dependence of the OOP exchange bias effect in nanostructures consisting of CoO or IrMn coupled to Co/Pd(Pt) deposited onto arrays of spherical nonmagnetic particles was recently investigated [21][22][23].

Here, we present a study of the exchange bias effect in a bilayer consisting of a 5-nm-thick Co 3 Pt alloy film with an out-of-plane easy axis of magnetization and a naturally oxidized 1-nm-thick Co layer. The cooling field dependence as well as the temperature dependence of the exchange bias field and the coercivity of the CoO/Co 3 Pt bilayer system will be discussed.

Sample preparation and experimental techniques

5-nm-thick Co 3 Pt alloy films were prepared at room temperature on thermally oxidized Si(100) substrates with a 100-nm-thick amorphous SiO 2 layer. The deposition was carried out by dc-magnetron cosputtering of Co and Pt in a high vacuum chamber with a base pressure of 1 × 10 -6 mbar. Argon was used as a sputter gas at a pressure of 3.5 × 10 -3 mbar. The composition of the alloy film was controlled by adjusting the deposition rates of Co (0.22 Å/s) and Pt (0.1 Å/s). A composition of Co x Pt 100-x with x = (73 ± 1) was determined by Rutherford backscattering spectroscopy. These films serve as reference samples for further investigations. In addition, a series of CoO/Co 3 Pt film samples was prepared by introducing the initial film samples into the sputter chamber and depositing an additional 1-nm-thick Co layer followed by the oxidation of the Co layer at ambient conditions. Note that the Co 3 Pt samples were exposed to air before the Co layer deposition, which led to an oxidation of the top surface.

The structural characterization was carried out by x-ray diffraction (XRD) using Cu-K radiation ( = 1.5406 Å), by atomic force microscopy (AFM), and by transmission electron microscopy (TEM) in both conventional and high resolution (HRTEM) modes employing a 200 kV Philips CM 20 FEG TEM.

Temperature dependent magnetic measurements were performed in IP and OOP geometry using a Quantum Design SQUID-VSM magnetometer. The measurement routine included a warming process to 320 K, which is above the Néel temperature, T N = 290 K, of bulk CoO. A maximum field of +70 kOe was applied to achieve complete saturation of the ferromagnetic Co 3 Pt layer. Then, the cooling field, H cool , was set at 320 K. With the cooling field applied, the samples were cooled down to the desired measurement temperature, T meas , at which the hysteresis loop was measured. T meas and H cool were varied over a wide range to investigate the dependence of coercivity, H C , and exchange bias field, H EB , on both parameters.

Structural properties

The x-ray diffraction pattern of the Co 3 Pt film measured in -2 reflection geometry shows a single peak at = 21.285°(figure 1(a)). The position of the peak corresponds to an interplanar distance of 2.122 Å, which is close to the value of the fcc (111) Co 3 Pt bulk alloy [26] and the one found in epitaxial hcp (0002) Co 3 Pt films [27][28][29]. The full width at half maximum of the peak leads to a perpendicular coherence length of By analysis of the cross-section TEM images (figure 2), the thickness of the Co 3 Pt layer was estimated to be (4.8 ± 0.3) nm whereas the lateral size of the Co 3 Pt grains is in the order of 20 nm. However, within one grain, there might be a number of small-angle boundaries. Besides this, areas with smaller Co 3 Pt grains of about 2 nm were found (not shown). Furthermore, a tilt of the c-axis of the grains of up to 10°from the film normal was observed (figure 2(b,c)), which is in agreement with the broad rocking curve (figure 1(a)). Please note that the observed canting of the c-axis has important consequences on the magnetic properties of the Co 3 Pt alloy film as discussed later.

In the CoO layer, small grains with a typical size below 3 nm were found. However, also a few larger grains with sizes of more than 5 nm were observed. The thickness of the CoO layer could not be determined, since there is no clear separation between CoO and the epoxy adhesive used for TEM sample preparation, but an estimation with the densities and molar masses or Co and CoO leads to a CoO thickness of at least 1.7 nm from the deposition of the additional Co layer.

An AFM study was carried out to access the overall surface morphology. The study revealed a root mean square (rms) roughness of the single Co 3 Pt layer of about 0.3 nm. With the addition of the CoO layer an increase of the rms roughness to about 0.5 nm was observed.

Magnetic characterization

The saturation magnetization, M S , of the Co 3 Pt film was estimated from SQUID measurements to be (900 ± 140) emu/cm 3 which is slightly lower compared to the bulk value of 1100 emu/cm 3 [29,30]. The OOP loop (figure 3) shows a sharp switching behaviour at a field of H C = 525 Oe and a remanent magnetization of about 0.85•M S ; whereas the IP loop displays characteristic hard axis behaviour. The presence of a perpendicular easy axis of magnetization is in agreement with earlier works reported by Yamada et al. [31,32]. Based on our structural study, the PMA is caused by the growth of hcp Co 3 Pt grains with their c-axes pointing preferentially in OOP direction. Please note that a high magnetic field of about 40 kOe is necessary to completely saturate the layer in OOP geometry (figure 3) which is likely caused by the canting of the caxes of the Co 3 Pt grains (figure 2(c)). Thus, the reversal in the high field region reflects a rotation away from the canted c-axis towards the field direction.

(A) Exchange bias field in OOP and IP direction

The magnetic hysteresis of a Co 3 Pt layer with an oxidized 1-nm-thick Co film is seen in figure 4. The Above 150 K the exchange bias field becomes zero, suggesting an upper limit for the blocking temperature T B of the same order. With decreasing measurement temperature, the EB field measured on CoO/Co 3 Pt gradually increases down to 10 K indicating that some AF grains are still thermally instable at this temperature. CoO is a rather strong AF with an anisotropy constant, K AF , of 29 × 10 7 ergs/cm 3 [33]. However, the mean size of the CoO grains of about 3 nm leads to a reduced thermal stability of the AF grains [12,34].

In this respect, only AF grains with high magnetic anisotropy energy, which are stable at T meas , can contribute to EB. Following the model by O'Grady et al. [12] which accounts for a size distribution of the AF grains, the temperature dependence of H EB in OOP geometry was measured (figure 6(a)) and its derivative was fitted with a log-normal peak function (figure 6(b)). The area under the curve includes all AF grains that are stable and thus allows to extract the maximum exchange bias field, H EB,max . With this method we found H EB,max = 433 Oe. Using this value, the exchange energy at the AF/F interface was calculated to be exchange bias [20]. The latter might be attributed to the different crystalline quality of F grains in the two systems. Indeed, in contrast to the room temperature growth of the Co 3 Pt alloy, the deposition of Co/Pt multilayers was carried out at an elevated temperature of 420 K [20], which might result in larger CoO grains with improved texture. This assumption is supported by the observed difference in blocking temperature (220 K for CoO/[Co/Pt] and 100 K for CoO/Co 3 Pt), which is related to the anisotropy energy of the AF layer and thus to the AF grain size.

J EB,OOP = M S •

(B) Coercive field dependence

The temperature dependence of the coercivity caused by the EB effect is one of the least understood phenomena in EB. A recent review of the existing theoretical models is given by O'Grady et al. [12]. With decreasing measurement temperature T meas , the difference in the coercive field of the Co 3 Pt film with and without CoO layer is found to become more pronounced (figure 6

(c)). According to Stiles and

McMichael [5,6], there are two possible mechanisms for increased coercivity in EB bilayer systems assuming a polycrystalline microstructure of the AF: (i) inhomogeneous reversal in the ferromagnet and (ii) irreversible switching of the antiferromagnetic order in AF grains as they follow the F magnetization due to interfacial coupling. The first mechanism is caused by spatially inhomogeneous coupling of the F to the uncompensated frozen AF spins which contributes at all temperatures below T B : With the decrease of T meas the contribution of this mechanism becomes more pronounced as more grains get frozen during the cooling procedure and the interfacial exchange area increases. Therefore, the observed gradual increase in H C (T meas ) (figure 6(c), dashed line) can be attributed to the increasing number of frozen uncompensated spins. As these are also responsible for the shift of the hysteresis loop (EB field), the background adopts the same functional dependency on T meas as the exchange bias field, which was approximated by an exponential decay.

In contrast, the contribution from irreversible switching of AF spins is expected to be weak at low temperatures having the strongest impact in the vicinity of the blocking temperature of the AF, where the anisotropy of the AF grains becomes largest before ceasing to be rotatable by the reversal of the F layer [5,6]. Thus, the non-monotonic dependence of H C (T meas ) is a result of AF spins, which are stable in zero field at T meas , but can switch irreversibly following the orientation of the magnetic moment of the F layer, when driving the field from positive to negative saturation. These AF spins are called rotatable spins which give rise to an additional energy barrier that needs to be overcome for the magnetic reversal process and, thus, lead to an increase of the coercive field of the coupled AF/F layers. Such non-monotonic H C (T meas ) behaviour was observed in soft magnetic layers exchange biased by CoO [14,36] and in CoO/[Co/Pd] films with OOP easy axis of magnetization [21]. By subtracting the background due to the inhomogeneous F reversal (figure 6(c), dashed line), we are left with the contribution of irreversible switching of AF spins to the coercive field of the F layer. Please note that the resulting H C (T meas ) dependence should resemble the AF grain size distribution (fraction of AF grains becoming thermally stable at T meas ) and was thus fitted using a log-normal peak function (figure 6(c)). Indeed, for a given temperature T meas , only spins in AF grains, which have their thermal destabilization temperature just above T meas , are subject to such switching and exhibit rotatable anisotropy. In contrast, spins in larger (more stable) AF grains will not switch and thus can contribute to unidirectional anisotropy resulting in a shift of a hysteresis loop. The limit at which the anisotropy changes from rotatable to unidirectional is defined by the coupling strength. AF spins coupled to the F layer with the exchange energy J EB can be reversed if their anisotropy energy, K AF •t AF (t AF is the thickness of the AF layer), can be overcome by the coupling energy. Usually the anisotropy energy is much larger than the exchange coupling energy, but the energy barrier is reduced by the aid of the thermal energy. If the energy barrier can be easily overcome at high temperatures, AF grains are not stable and behave paramagnetically contributing to neither H EB nor H C . Due to the different stability requirements for unidirectional and rotatable anisotropy, the peak in the derivative of H EB occurs at a lower temperature than the one in the H C (T meas ) dependence (figure 6) as the frozen state is reached later than the rotatable state during cooling.

Please note that in IP geometry the investigation of the EB related coercivity enhancement was not carried out due to the presence of an in-plane anisotropy caused by the CoO layer, which is the main source for the increased coercivity.

(C) Cooling field study

A strong cooling field dependence of both the exchange bias field and coercivity is typically observed for exchange bias systems [4,12,21,37].

As can be seen from figure 7(a), no difference in the OOP hysteresis loops was observed for the CoO/Co 3 Pt film even when a reverse cooling field (smaller than coercive field at room temperature) was applied to the initially saturated sample. However, the sign of H EB was changed when the reverse cooling field exceeded the coercive field at room temperature (not shown). This observation suggests that the stray field provided by the Co 3 Pt film at remanence (about 10 kOe) is already sufficient to align the CoO spins when cooled through T N .

In this respect, it is interesting to note that due to the relatively small IP remanent magnetization, the setting of the AF layer at remanence in IP geometry is not efficient compared to the OOP case. Therefore, a pronounced cooling field dependence of the exchange bias field was observed (figure 7(b)). In this case H EB,IP (H cool ) is correlated with the strength of the IP magnetization at room temperature thus following the room temperature IP hysteresis loop (figure 4(c)).

It is important to note that the coercive field, H C , reveals no cooling field dependence in both OOP and IP geometries (not shown) because the F layer in the studied system provides a sufficient magnetic field while cooling through the Néel temperature to align the spins in the thin AF layer with respect to the local orientation of the F spins. As the F spins reverse during the measurement of a hysteresis loop, the torque provided by the magnetization of the F layer will switch the rotatable AF spins without regards to their initial state after cooling. This means that the exhibited rotatable anisotropy is not bound to a specific cooling

4. 7 5 ,

 75 nm, which is slightly smaller than the nominal Co 3 Pt film thickness. The width of the corresponding rocking curve (RC) is equal to 12°(figure1(a)). Such broadening of the rocking curve can be attributed to a small lateral size of the Co 3 Pt grains and/or high mosaicity. Neither reflections coming from oblique planes characteristic of the fcc or hcp stacking, nor a superstructure peak indicative of long-range chemical ordering as measured for epitaxial hcp Co 3 Pt films[27,29] were observed.The structural properties of the CoO/Co 3 Pt sample were further investigated by TEM. A selected area electron diffraction (SAED) pattern of a plan-view sample is shown in figure1(b). In the lower right part of the electron diffraction pattern, simulated rings corresponding to a (0002)-textured hcp phase are superimposed on the experimental pattern (using values a = 2.61 Å and c = 4.24 Å deduced from the SAED pattern and from XRD, respectively). In the same way, in the upper right part of figure 1(b), simulated rings for a (111)-textured fcc phase (bulk value a = 3.675 Å) are shown. These simulations indicate that the diffraction rings 2, and 6a can be indexed both as(11)(12)(13)(14)(15)(16)(17)(18)(19)(20), (3-300) and (22-40) reflections of the (0002)-textured hcp phase or as the (220), (224) and (440) reflections of the (111)-textured fcc phase. In contrast, the positions of the (1a: 1-100), (3a: 2-200), (4a: 12-30), and (6b: 31-40) rings are exclusively characteristic of the (0002)textured hcp phase. Other weak rings (1b and 3b) refer either to the (111) and (113) fcc reflections, or to the (0002) and(11)(12)(13)(14)(15)(16)(17)(18)(19)(20)(21)(22) reflections of the hcp phase but not to CoO. Please note, that these rings are neither expected for the (0002)-textured hcp phase nor for the (111)-textured fcc phase, indicating that a small amount of crystallites has different orientations. From the SAED pattern, no conclusive picture about the occurrence of an additional fcc phase can be drawn. However, high resolution images of the Co 3 Pt layer indicate the existence of fcc grains similar to earlier observations[28]. Nevertheless, the high intensity of the (1-100) ring suggests a strong (0002)-textured hcp phase in the Co 3 Pt film. Moreover, it has been checked, that the weak reflections do not originate from CoO.

additional

  CoO layer leads to a reduction of the OOP remanent magnetization and to an increase of the remanence in IP direction when compared to the Co 3 Pt single layer (figure4(a, c)). The reduction of the OOP remanence might be related to the modification of the surface anisotropy of the Co 3 Pt film triggered by the deposition of the crystalline Co capping layer, as it was recently reported by Shipton et al.[25] for exchange biased CoO/[Co/Pt] films. Furthermore, the addition of the AF layer leads to a substantial enhancement of the coercivity in both OOP and IP directions.The hysteresis loop measured at T meas = 10 K on the CoO/Co 3 Pt film shows an exchange bias field of about 340 Oe after field cooling in 70 kOe in both OOP and IP geometries (figure4(b, d)). Note that also for the single Co 3 Pt layer a comparatively small exchange bias effect (H EB < 50 Oe) with a blocking temperature, T B , of about 30 K was observed. It can be concluded that the natural oxidation of the single Co 3 Pt layer forms only a marginal amount of AF CoO likely due to the surface segregation of Pt during co-deposition resulting in a partial passivation of the surface. Figure5summarizes the EB and coercive fields as a function of T meas in both measurement geometries (OOP and IP) for CoO/Co 3 Pt in comparison with single Co 3 Pt layers.

Figure 6 (

 6 Figure 6(c) displays the difference, ΔH C , between the coercivities of the Co 3 Pt films with and withoutCoO in OOP geometry. The temperature dependence of H C was fitted with a log-normal peak function subtracting an exponential background (figure 6(c), dashed line). Please note that even at temperatures higher than T B , the difference in H C does not vanish and remains at about 75 Oe. This might be related to the modification of the interface properties between the Co 3 Pt and CoO layers which will affect the distribution of nucleation/pinning sites. However, an influence of the paramagnetic CoO due to fluctuations of magnetic moments on the magnetization reversal behaviour of the F layer might also contribute to the coercive field of the CoO/Co 3 Pt bilayer. A similar idea was applied by Cai et al.[35] to explain the influence of an AF layer with T N > T C on a ferromagnetic film above its Curie temperature.

Fig. 1 :

 1 Fig. 1: (a) XRD -2 pattern and rocking curve (RC) of the 5-nm-thick Co 3 Pt film. (b) SAED pattern of a planview sample. The calculated rings corresponding to the (111) fcc and (0002) hcp textured phases are drawn in the upper and lower right parts, respectively. The bright spots on the left and right sides come from the Si(100) substrate.

Fig. 2 :

 2 Fig. 2: (a) Cross-sectional TEM image of the CoO/Co 3 Pt sample. The lower panels show high resolution images taken of different Co 3 Pt grains: (b) lattice planes are almost parallel to the substrate plane (out-ofplane c-axis); (c) lattice planes are approximately 10°tilted with respect to the substrate plane (canted caxis). Lattice contrast in panels (b) and (c) has been enhanced by Fourier-filtering and standard image processing techniques which led to delocalized lattice fringes in the amorphous adhesive used for sample preparation in the top left corners.

Fig. 3 :

 3 Fig. 3: Out-of-plane and in-plane SQUID hysteresis loops of the single Co 3 Pt film measured at 300 K. The inset shows a zoomed region of the loops around the zero field axis.

Fig. 4 :Fig. 5 :Fig. 6 :

 456 Fig. 4: SQUID hysteresis loops of the Co 3 Pt samples without (open symbols) and with (solid symbols) CoO layer. Panels (a) and (c) show loops measured at T meas = 300 K; panels (b) and (d) show loops measured at T meas = 10 K after field cooling in H cool = 70 kOe. Please note the difference in the shown field range of this figure compared to Fig. 3, hiding the further increase of the OOP magnetization with increasing magnetic field.

Fig. 7 :

 7 Fig. 7: (a) Out-of-plane SQUID hysteresis loops of the CoO/Co 3 Pt stack measured at 10 K after field cooling in various H cool . The image shows 5 hysteresis loops for cooling fields of H cool = +70, +5, +0.4, 0, -0.4 kOe (H C of Co 3 Pt at 300 K is -0.5 kOe). (b) Cooling field dependence of the EB fields in IP and OOP direction.The data is extracted from the hysteresis loops taken at T meas = 10 K in IP and OOP geometry after cooling in a field H cool .

Figure

  Figure 1a (revision/figure1a.eps)

  Pt][20]. Furthermore, in the CoO/Co 3 Pt system the interfacial exchange coupling energy in OOP and IP direction is found to be the same within the margin of error (J EB,IP /J EB,OOP 1) in agreement with the expected isotropy of the orientation of the easy axes in the grains of the polycrystalline AF film. In contrast to this finding, a value of J EB,IP /J EB,OOP 2 was obtained for CoO/[Co/Pt][20]. The smaller interfacial exchange energy measured in IP direction for CoO/Co 3 Pt compared to CoO/[Co/Pt] indicates that a smaller amount of AF grains have their easy axes close to the IP cooling direction, as required for IP

		H EB,max •t FM = 0.19 ergs/cm	2 . The value of J EB,OOP is lower than the interfacial exchange energy of
	0.25 ergs/cm	2 reported for CoO/[Co/

Acknowledgement

We would like to thank Dr. F. Radu (HZB, Berlin) and C. Brombacher for fruitful discussions, Prof. R. Magerle for the opportunity to perform AFM studies, P. Matthes for assistance in AFM measurements, and G. Baumann for TEM sample preparation. Part of this research was carried out at the light source HASYLAB (beamline G3) at DESY (Hamburg) and the financial support (project No. I-20090119) is gratefully acknowledged. In addition, H.S. acknowledges financial support through the Landesinnovationspromotion (financed by Free State of Saxony and European Social Fund).

configuration in the AF which is in agreement with the theory by Stiles and McMichael [4]. Therefore, even if the mean unidirectional anisotropy induced by the cooling procedure varies (figure 7(b), IP curve), the rotatable anisotropy does not change, resulting in constant H C of the obtained loops.

Summary

We presented a study on the exchange bias effect in hcp (0002)-textured Co 3 Pt alloy films coupled to a thin antiferromagnetic CoO layer. The Co 3 Pt layer shows an out-of-plane easy axis of magnetization. The model of grainy polycrystalline AF was applied to interpret the obtained experimental data on the magnetic properties of the samples. It was shown that in contrast to previous observations on perpendicular CoO/[Co/Pt] systems, CoO/Co 3 Pt reveals similar exchange bias fields in OOP and IP geometry for maximum cooling fields, i.e. when the F layer is fully saturated. This is attributed to the different quality of the AF/F interface.

Furthermore, the dependence of the coercivity enhancement due to the EB effect was investigated by comparing identical F films with and without CoO. It was found that there are two contributions to the enhancement of the coercive field in perpendicular CoO/Co 3 Pt systems, which can be identified following the theory developed by Stiles and McMichael as inhomogeneous reversal in the F film and irreversible switching of AF spins. While the first mechanism has the same origin as the loop shift (uncompensated frozen spins), the other one is related to the rotatable anisotropy characteristic of partially stable AF spins which are just below their destabilization temperature. Furthermore, the difference in the temperature evolution of the coercive field given by rotatable spins and the exchange bias field given by frozen spins indicates that AF grains undergo two transitions during the cooling process: First, from the paramagnetic state to the state exhibiting rotatable anisotropy and second, to the frozen state. The transition behaviour can be accessed by measuring the temperature evolutions of H C and dH EB /dT meas .

Interestingly, the CoO/Co 3 Pt samples did not reveal a cooling field dependence of the exchange bias field when initially saturated in OOP direction. It is assumed that the high remanence magnetization of the Co 3 Pt film results in a strong stray field at the AF/F interface leading to a setting of the AF layer even without an applied magnetic field. In contrast, in IP direction the exchange bias field is correlated with the strength of the IP magnetization at room temperature thus following the room temperature hysteresis loop of the Co 3 Pt film. The coercive field was not dependent on the cooling configuration for both OOP and IP geometries.


L Giancardo

❄② F Meriaudeau

❄ T P Karnowski

② Y Li

③ K W Tobin

E Chaum

AUTOMATIC RETINA EXUDATES SEGMENTATION WITHOUT A MANUALLY LABELLED TRAINING SET

Keywords: One, two, three, four, five

Diabetic macular edema (DME) is a common vision threatening complication of diabetic retinopathy. In a large scale screening environment DME can be assessed by detecting exudates (a type of bright lesions) in fundus images. In this work, two new methods for the detection of exudates are presented. The methods do not require a lesion training set so the need to ground-truth data is avoided with significant time savings and independence from human error. We evaluate our algorithm with a new publicly available dataset from various ethnic groups and levels of DME. Also, we compare our results with two recent exudate segmentation algorithms on the same dataset. In all of our tests, our algorithms are either outperforming or in line with existing methods. Additionally, the computational time is one order of magnitude less than similar methods.

INTRODUCTION

Diabetic retinopathy (DR) is a progressive eye disease that currently affects 250 million of people worldwide. Diabetic macular edema (DME) is a complication of DR and it is a common cause of vision loss and blindness. DME occurs from swelling of the retina in diabetic patients due to leaking of fluid from microaneurysms within the macula. Ophthalmologists can infer the presence of the fluid that causes the retina thickening in diabetic patients by the presence of accompanying lipid deposits called exudates. They appear as bright structures with well defined edges and variable shapes.

The approaches to exudate segmentation presented in the literature can be roughly divided into four different categories. Thresholding methods base the exudate identification on a global or adaptive grey level analysis. A first attempt was presented in [START_REF] Phillips | Automated detection and quantification of retinal exudates[END_REF] and recently a more sophisticated method based on image normalisation and distribution analysis was presented in [START_REF] Sanchez | Retinal image analysis based on mixture models to detect hard exudates[END_REF]. Region growing methods segment the images using the spacial contiguity of grey levels; a standard region growing approach is used in [START_REF] Sinthanayothin | Automated detection of diabetic retinopathy on digital fundus images[END_REF], which is very computationally expensive by being employed to the whole image.

In [START_REF] Li | Automated feature extraction in color retinal images by a model based approach[END_REF] the computational issues are limited by employing edge detection to limit the size of regions. Morphology methods employ greyscale morphological operators to identify all structures with predictable shapes (such as vessels). These structures are removed from the image so that exudates can be identified [START_REF] Walter | A contribution of image processing to the diagnosis of diabetic retinopathy-detection of exudates in color fundus images of the human retina[END_REF][START_REF] Sopharak | Automatic detection of diabetic retinopathy exudates from non-dilated retinal images using mathematical morphology methods[END_REF]. Classification methods build a feature vector for each pixel or pixel cluster, which are then classified by employing a machine learning approach into exudates or not exudates [START_REF] Gardner | Automatic detection of diabetic retinopathy using an artificial neural network: a screening tool[END_REF][START_REF] Osareh | Automated identification of diabetic retinal exudates in digital colour images[END_REF][START_REF] Garcia | Neural network based detection of hard exudates in retinal images[END_REF] or additional types of bright lesions [START_REF] Niemeijer | Automated detection and differentiation of drusen, exudates, and cottonwool spots in digital color fundus photographs for diabetic retinopathy diagnosis[END_REF][START_REF] Fleming | The role of haemorrhage and exudate detection in automated grading of diabetic retinopathy[END_REF] (such as drusen and cotton wool spots).

In this paper we present two variations of a new exudate segmentation method that falls into the category of thresholding methods. Hence, the methods do not require any manual segmented lesion map for training to achieve good performance. In order to reduce the problem of noise we have developed a new way to normalise the fundus images. We directly compare our method with an implementation of a morphology based technique [START_REF] Sopharak | Automatic detection of diabetic retinopathy exudates from non-dilated retinal images using mathematical morphology methods[END_REF] and another thresholding based technique [START_REF] Sanchez | Retinal image analysis based on mixture models to detect hard exudates[END_REF]. We selected the techniques that were tested with the largest datasets in the respective class.

In Section 2 we introduce the new public dataset used for testing the algorithm; Section 3 presents the details of the two automatic exudate segmentation methods; Section 4 presents the results by comparing them to other two published techniques; finally, Section 5 concludes the discussion.

MATERIALS

For the past 5 years, our group has been designing a semiautomated, HIPAA-compliant, teleophthalmology network for the screening of diabetic retinopathy and related conditions by employing a single macula centred fundus image acquired with a Zeiss Visucam PRO fundus camera. We have randomly extracted a set of 169 images representative of various degree of DME and patient ethnicity (the appearance of the retinal fundus greatly varies between ethnic groups). Each image of the dataset was manually segmented by one of the co-authors (E. Chaum, a practising Retina specialist). The manual segmentation allowed us to test the algorithm and it was not employed in the training phase. We have decided to release the dataset to the research community on the website: http://risa.uthsc.edu/dmed.

METHOD

Preprocessing

Background Estimation

In our approach we employ noth the green channel ■ ❣ of the RGB colour space and the ■ ✐ channel of the HSI colour space. We start the analysis by estimating the background by the means of a large median filter, whose size is ✶ ✸✵ the height of the fundus image on ■ ✐ . This approach has been used by other authors (such as [START_REF] Niemeijer | Automatic detection of red lesions in digital color fundus photographs[END_REF][START_REF] Fleming | Automated microaneurysm detection using local contrast normalization and local vessel detection[END_REF]) and it has great computation performance advantages over the method from Foracchia et al., mainly because it does not use bilinear interpolation or multiple passes. In other median filtering normalisation techniques, the background is subtracted from the original image in order to obtain a normalised version. In our approach, we enhance the normalisation with the addition of a morphological reconstruction step [START_REF] Vincent | Morphological grayscale reconstruction in image analysis: applications and efficient algorithms[END_REF]. This seems to improve the removal of nerve fibre layer and other structures at the edges of the ON, at little or no expense for the exudates. The following pseudocode illustrates this step in more details. for ① ❂ ✵ to WIDTH(■❣ ) ✶ do

6:

if ■❣✭①❀ ②✮ ❁ ❜❣❊st✭①❀ ②✮ then 7:

❜❣▼❛s❦✭①❀ ②✮ ✥ ❜❣❊st✭①❀ ②✮

Image Normalisation

Once the background ❜❣❊ st ✷ is estimated, it is subtracted from the original image. The subtraction operation is performed with 16 bit signed precision in order to maintain the negative pixels. The image obtained shows a peculiar greylevel distribution: the highest peak of the histogram is always centred on zero regardless of the ethnicity of the patient or disease stratification. The histogram provides a clear distinction of two classes of structures: dark structures and bright structures. Dark structures are located in the left side of the histogram and represent the vasculature, the macula, dark lesions (such as microaneurysms and haemorrhages) and other structures due to the retinal pigment epithelium layer.

Their distribution seem to vary depending on the ethnicity; the reason is related to the pigmentation of the patient, the less the pigmentation the more the choroidal vasculature is visible, hence it is visible as a bump in the negative side of the histogram. On the other hand bright structures are found in the positive side of the histogram which contains the optic nerve, bright lesions (such exudates and cotton wool spots), nerve fibre layer reflectance residuals (from the background removal), artefacts due to dirty lenses and other structures related to very bright retinal pigment epithelium layer. In this case, the distribution is fairly constant across different ethnicities.

Because of the alignment of the histogram after normalisation we can select all the exudate candidates ■ ❝❛♥❞ with a hard threshold t❤ ❝❛♥❞ . This has obvious computational advantages in comparison with any model fitting operations, that not only do not seem to provide any quantifiable advantage for the selection of a threshold, but are also more sensitive to suboptimal background estimation. We have selected t❤ ❝❛♥❞ ❂ ✸ given that the original image ■ ✐ has a grey level range between 0 and 255. This parameter has not been selected by trial and error on the whole dataset, but by empirically choosing a value slightly above 0 in order to accommodate small background estimation errors.

Exudate detection

The exudate detection is performed by assigning a score for each exudate candidate. The exudate candidates are selected by running a 8-neighbour connected component analysis on ■ ❝❛♥❞ . We have implemented two ways to assign this score, one based on Kirsch's Edges [START_REF] Kirsch | Computer determination of the constituent structure of biological images[END_REF] and the other based on Stationary Wavelets. Both try to capture the higher edge values of the exudates in comparison the one of other candidates.

Kirsch's Edges

Kirsch's edges try to capture the external edges of the lesion candidate. This edge detector is based on the kernel ❦ evaluated at 8 different directions on ■ ❣ . The kernel outputs are combined together by selecting the maximum value found on each pixel output. The result is stored in the final ■ ❦✐rs❝❤ image.

❦ ❂ ✷ ✹ ✺ ✶✺ ✸ ✶✺ ✸ ✶✺ ✺ ✶✺ ✵ ✸ ✶✺ ✺ ✶✺ ✸ ✶✺ ✸ ✶✺ ✸ ✺ (1)
As proposed by Sanchez et al., the average edge outputs of ■ ❦✐rs❝❤ under each lesion cluster are calculated and assigned to the lesion in its entirety. The thresholds used to evaluate the final output are t❤ ❢✐♥ ✷ ❢✵ ✿ ✵✿✺ ✿ ✸✵❣. (c) the exudate segmentation performances of our method using Kirsch's edges in the macula area; each line corresponds to a growing circular area centred on the fovea.

Stationary Wavelets

Quellec et al. [START_REF] Quellec | Optimal wavelet transform for the detection of microaneurysms in retina photographs[END_REF] proposed a method for the detection of retina microaneurysms in the wavelet space. We try to capture the strong peak at the centre of exudates by developing a method that employs a similar wavelet as Quellec et al., but that evaluates the results in image space. A stationary Haar wavelet analysis is performed up to the second level on ■ ❣ . The process is inverted maintaining the last vertical, diagonal and horizontal details only, as these are the wavelet coefficients that seem to contain most of the foreground structures. By transforming back to the image space we obtain a background-less image ■ ✇❛✈ with a strong response at the centre of exudates. It is interesting to notice that the distribution of ■ ✇❛✈ has similar properties as the image obtained during the image normalisation phase (i.e. centred at 0 and with the exudates located on the positive side).

Similarly to the previous approach, we estimate the chance of being an exudate by evaluating the ■ ✇❛✈ area which corresponds to each lesion cluster of ■ ❝❛♥❞ . However, instead of calculating the average coefficient, we attempt to detect spikes in greyvalues as follows.

✇❛✈❙ ❝♦r❡ ❂ ♠❛①✭♣① ✇❛✈ ✮ ♠✐♥✭♣① ✇❛✈ ✮ ♠❛①✭♣① ✇❛✈ ✮ (2)
where ♣① ✇❛✈ are the pixels of ■ ✇❛✈ corresponding to a lesion candidate cluster. The thresholds used to evaluate the final output are t❤ ❢✐♥ ✷ ❢✵ ✿ ✵✿✵✺ ✿ ✶❣.

RESULTS

In our results, we compare our two techniques described in this paper with our implementation of [START_REF] Sopharak | Automatic detection of diabetic retinopathy exudates from non-dilated retinal images using mathematical morphology methods[END_REF] and [START_REF] Sanchez | Retinal image analysis based on mixture models to detect hard exudates[END_REF]. In all instances, we evaluate the performance of the lesion segmentation algorithms on a lesion by lesion basis for each image. The analysis starts from the labelled image in the dataset and each lesion is compared to the automatic segmentation one by one. A lesion is considered a true positive (TP) if it overlaps at least in part with the ground-truth; a false negative (FN) if no corresponding lesion is found in the automatic segmentation; a false positive (FP) if an exudate is found in the automatic segmentation but no corresponding lesion has been manually segmented. In the evaluation of the segmentation, we did not employ any true negatives (TN). As such, we avoid any evaluation of specificity, which is inherently high in images where the lesions represent a very small percentage of the total image area. In order to compare the methods fairly, all the images in the dataset have been resized to the resolution used in the original papers (but maintaining the original width/height ratio). It should be noticed that in the test results of this section all the FP corresponding to bright lesions other than exudates have not been taken into account.

Fig. 1(a) shows the free-response receiver operating characteristic (FROC) analysis [START_REF] Bunch | A free-response approach to the measurement and characterization of radiographic-observer performance[END_REF] of the implemented algorithms on the entire dataset. The global Sensitivity is evaluated as ❚ P ❚ P ✰❋◆ on each image and then averaged together.

The global Positive Predictive Value (PPV) is computed in a similar way but employing ❚ P ❚ P ✰❋P . It can be noticed how our two methods and the technique developed by Sanchez et al. performed comparably, with our technique (with Kirsch's edges) performing somewhat better.

Similar results are shown in Fig. 1(b). In this case only the 54 image with DME are used in order to have a better evaluation of the segmentation performance. In this case both of our methods seem to have a higher sensitivity than the other two methods implemented.

When exudation is close to the fovea it is more likely to cause vision loss. Therefore, we tested the ability of identifying lesions close to the macula of our method with Kirsch's edges, which seems to be the best performing algorithm of the previous tests. We have evaluated the lesion segmentation on the most dangerous area which might be affected by exudationi.e. A growing area around the fovea. All the lesions outside these areas have been discarded. Fig. 1(c) shows how the lesion segmentation improves when the lesion is closer to the fovea. This means that our algorithm is more effective on more sight threatening lesions. This is because the macular pigment in the area around fovea tends to be darker than the rest of the image, therefore exudates have more contrast. Also, when the area close to the vessels is discarded there are less chances of false positives due to the reflections of nerve fibre layer or to the muscular tissues of large vessels.

When developing an exudate segmentation method it must be taken into account that, in itself, the localisation of the lesions is not the final aim of an automatic diagnosis tool, but rather only a step towards the final diagnosis. As such, the detection of every single exudate in an image with many of them is not as important as finding a single lesion on a image with a single one. Therefore, we have evaluated the algorithms based on their ability to discern patients with or without DME by employing the hard threshold of a single lesion. If one or more exudates are found, the image is diagnosed with DME, otherwise the patient is classified as being negative. Fig. 2 shows the results of this experiment by the means of a standard ROC analysis on our dataset. Again, our methods seem to perform better or comparably to the other algorithms. Although the area under the ROC curve (AUC) is highest for our method with Kirsch's edges, the technique with wavelet shows a higher sensitivity at a higher specificity, a useful aspect for the development of a automatic DME screening tool.

The computational performance are evaluated on a Dual Core 2.6 GHz machine with 4 GB of RAM. All the algorithms are implemented in unoptimised Matlab code. Table 1 shows the average time to segment an image.

✘39

Sopharak et al.

✘36

Our method (with Wavelet)

✘2.4

Our method (with Kirsch)

✘1.9

The reason for such performance discrepancy is likely to be due to the bilinear interpolation and expectation maximization for Sanchez et al. The calculation of the local standard deviation for each pixel is the cause for the computational performance degradation with the method by Sopharak et al.

DISCUSSION AND CONCLUSION

We have evaluated the global segmentation performance with the real distribution of patients in a screening setting and on only the patients showing signs of exudation. The results are particularly encouraging especially because of the comparison with the other techniques by Sopharak et al. and Sanchez et al. The method by Sanchez et al. is somewhat close to ours tests, however, the image normalisation procedure gives a substantial advantage to our method. The median filter with morphological reconstruction approach maintains a good contrast of the foreground structures by limiting the effects of the noise due to nerve fibre layer reflections and other small artefacts.

Also, we have evaluated our algorithms ability in identifying patients with DME based on the segmentation of one or more lesion in a fundus image. This is a naive method for DME diagnosis because neither it takes into account the many other features available in an image nor it employs any state of the art classification techniques. However, this test provides a baseline of the possible screening performances that can be achieved employing the output of the segmentation as a classification feature.

Many other exudate segmentation methods have been published throughout the years, however all of them seem to employ different datasets and different evaluation methods. This makes a direct comparison almost impossible, as is underlined by many of the authors themselves. We contribute to a solution to this problem by making available our heterogeneous, ground-truthed dataset to the research community. The need of having a common dataset to compare different techniques has been confirmed in this paper, where the two algorithms implemented did not perform as well as in the respective datasets employed by the original authors.

  for ② ❂ ✵ to HEIGHT(■❣ ) ✶ do 5:

Fig. 1 .

 1 Fig. 1. FROC curves. (a) the exudate segmentation performances on all images; (b) the performance on the images diagnosed with DME only; (c) the exudate segmentation performances of our method using Kirsch's edges in the macula area; each line corresponds to a growing circular area centred on the fovea.

Fig. 2 .

 2 Fig.2. ROC curves of the diagnosis of DME by employing only the lesions segmented, i.e. an image is considered positive if it shows at least a lesion. All the segmented lesions overlapping with the areas marked as "Other Bright Lesion" in the ground-truth are not taken in consideration.

Table 1 .

 1 Computational Performance

	Method	Seconds per image
	Sanchez et al.

ACKNOWLEDGEMENTS

These studies were supported in part by grants from Oak Ridge National Laboratory, the National Eye Institute, (EY017065), by an unrestricted UTHSC Departmental grant from Research to Prevent Blindness (RPB), New York, NY, Fight for Sight, New York, NY, by The Plough Foundation, Memphis, TN and by the Regional Burgundy Council, France. Dr. Chaum is an RPB Senior Scientist.


Estelle Delpech

Evaluation of terminologies acquired from comparable corpora : an application perspective

This paper describes a protocol for the evaluation of bilingual terminologies acquired from comparable corpora. The aim of the protocol is to assess the terminologies'added-value in a task of specialized translation. The protocol consists in having specialized texts translated in various situations: without any specialized resource, with an domain-related bilingual terminology or using Internet. By comparing the quality of the segments translated using these various resources, we are able to assess the impact of our bilingual terminologies on the quality of the translation.

Introduction

Evaluation plays an important role in NLP developments: it assesses the quality of tools, brings out the progress made between two developments, spots the limitations and highlights possible lines of research. Regarding the evaluation of terminologies, [START_REF] Rayson | Évaluation des outils terminologiques : enjeux, difficultés et propositions[END_REF] show that terminologies are complex objects and that their evaluation can be quite arduous. These authors distinguish between three evaluation modes: evaluation through reference : the terminology is compared to a standard reference, the evaluation metric indicates the adequacy between the assessed terminology and the reference terminology.

evaluation through interaction : the evaluation aims at measuring the cost of the transformation of the raw terminology as outputted by the system into the final, validated, ready-touse terminology.

evaluation through application : the evaluation's purpose is to compare the performance of a given application with and without the terminology, the metric indicates the addedvalue of the terminology and depends on the application.

( [START_REF] Rayson | Évaluation des outils terminologiques : enjeux, difficultés et propositions[END_REF] propose protocols and metrics for the first two evaluation modes and focus on monolingual terminologies only. The aim of this paper is to propose a protocol for the evaluation through application of bilingual terminologies acquired from comparable corpora. The considered application is human specialized translation.

Terminologies acquired from comparable corpora are usually assessed using an evaluation through reference protocol [START_REF] Fung | Finding terminology translations from non-parallel corpora[END_REF][START_REF] Sadat | Learning bilingual translations from comparable corpora to Cross-Language information retrieval: Hybrid statistics-based and linguistics-based approach[END_REF][START_REF] Koehn | Learning a translation lexicon from monolingual corpora[END_REF]. Algorithms which extract bilingual terminologies from comparable corpora output a list of 1-to-n alignments: each source term is aligned with the n best candidate translations, most of the time the Top20 candidate translations. The output of the algorithm is compared to a reference lexicon and the evaluation metric is a precision score computed on the Top1, Top10 or Top20 candidates. For example, a 50% precision on the Top20 candidates indicates that the correct translation is found among the first top 20 candidates for 50% of the source terms.

Although evaluation through reference is useful to monitor the effect of changes in the alignment algorithm and to compare the alignment techniques, we believe it is important to demonstrate the impact and the usefulness of terminologies and lexicons acquired from comparable corpora in real-life applications. [START_REF] Renders | Assessing automatically extracted bilingual lexicons for CLIR in vertical domains: XRCE participation in the GIRT track of CLEF[END_REF] showed the influence of such lexicons on crosslingual information retrieval. We would like to determine the added-value of these bilingual terminologies when they are used in a task of human specialized translation.

Section 2 explains how terms are aligned in comparable corpora and examines the issue of translation quality assessment. The evaluation protocol is defined in section 3. The experimentation and results are described in section 4. Perspectives and future work are discussed in section 5.

Background

In this section, we describe the algorithm used for term alignment (section 2.1) and give a brief stateof-the-art survey in translation quality assessment (section 2.2).

Term alignment from comparable corpora

Comparable corpora are sets of texts written in two languages which are not translations of each other but which share a substantial part of their vocabulary, mainly because they are topic-related. The major advantage of comparable corpora is that it is much more available than parallel corpora and enables the processing of unprecedented language pairs. It is also often argued that the target language texts found in comparable corpora contain more spontaneous / natural terms and expressions than in parallel corpora because the target texts are not translations and they have not been influenced by the language of the source text. Term alignment from comparable corpora was initiated by the work of [START_REF] Fung | Finding terminology translations from non-parallel corpora[END_REF] and [START_REF] Rapp | Identifying word translations in Non-Parallel texts[END_REF]. The alignment algorithm is based on distributional linguistics and considers that two terms are probable translations if they occurr in similar contexts. The context of a term T is represented by a vector indicating the number of times T cooccurs with each word within a given contextual window (for instance: three words on the left of T and three words on the right of T). The cooccurrence frequencies are normalized using the loglikelihood ratio [START_REF] Dunning | Accurate methods for the statistics of surprise and coincidence[END_REF]. Words in the source context vectors are translated into the target language using a bilingual seed lexicon. Then, the source and target vectors are compared using a similiarity measure such as the Cosine similarity measure. The most similar the vectors, the most likely the target and source terms are translations of each other. [START_REF] Morin | Compositionality and lexical alignment of multi-word terms[END_REF] report that the correct translation is to be found among the top 20 best candidates for 42% to 80% of the source terms depending on corpus size, on the complexity of the terms and whether the alignment is made using specialized corpora or general language corpora. As a consequence, the output lexicon is ambiguous and sometimes, the correct translation does not appear among the candidates.

Translation quality assessement (TQA)

Because we want to compare the quality of translations made by humans with and without bilingual terminologies, we need to find a way to assess human translation quality. If Machine Translation (MT) enjoys well-defined and rather consensual metrics to evaluate its quality, evaluation of human translation poses a real challenge. These two domains use different protocols for the assessment of translations. On the one hand, MT evaluation focuses on comparing the output of different MT systems. This evaluation is done in reference to one or several human translations. On the other hand, translation studies seek to assess the quality of a human translation on its own, without any reference to a standard translation. In fact, the only reference is the judge himself/herself.

TQA and machine translation

There are two ways of assessing machine translations. One is called objective or automatic evaluation. The other is called subjective or human evaluation.

In objective evaluation, translations are evaluated through a measure which is automatically computable and which has the advantadge of being reproducible. Examples of such measure are: BLEU [START_REF] Papineni | BLEU: a method for automatic evaluation of machine translation[END_REF] which is based on the count of common n-grams between the assessed translation and reference translation(s) and METEOR [START_REF] Banerjee | METEOR: an automatic metric for MT evaluation with improved correlation with human judgments[END_REF] which is similar to BLEU but leaves room for variation by including morphological variants and synonyms in the n-gram comparison. These metrics are in turn meta-evaluated by computing their correlation with human jugements. Though handy for the evaluation of MT systems on a daily basis, these metrics were not used in the shared translation tasks of the ACL Workshop on Statistical Machine Translation where they are perceived as imperfect substitutes of human assessment, see [START_REF] Callison-Burch | Findings of the 2009 workshop on statistical machine translation[END_REF][START_REF] Callison-Burch | Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation[END_REF] for example.

MT evaluation campaigns led to the development of a series of protocols for what is called subjective or human evaluation of MT. Two evaluation protocols stand out : judgement task The judge grades each translation independently. The grading scale can be quite complex, like 5 points scales over two criteria (fluency, adequacy) or simple binary judgements (correct/incorrect).

ranking task The judge ranks several translation of the same source segment from worse to best, each translation being produced by a different system.

These protocols are meta-evaluated using interand intra-annotator agreement measures which gives some indication on the coherence of the judgements. Experiments by [START_REF] Callison-Burch | Meta-) evaluation of machine[END_REF] show that annotation tasks which involve complex sets of categories (e.g. 5 points scales over adequacy and fluency vs. binary judgements) and larger segments (sentences vs. phrases) tend to be more time-consuming and to result in lower agreement. The ranking task is considered easier and less time-consuming than the judgement task. It also yields a higher annotator agreement.

TQA and translation studies

In translation studies, TQA is mainly used by the translation industry as a way to monitor the quality of its products. [START_REF] Secarȃ | Translation evaluation -a state of the art survey[END_REF] gives an overview of various translation grids. Although there is no consensus, all grids follow more or less the same methodology. Translation errors are categorized (e.g. spelling, grammar, terminology) and each error type is assigned a certain cost which is proportional to its gravity. A passage of a given length is randomly selected from the translation under assessment. Errors are marked and the cost points add up. If the sum of the points exceeds some threshold, the translation is deemed unacceptable. These grids are criticised by theoretical approaches to TQA -see [START_REF] Williams | The application of argumentation theory to translation quality assessment[END_REF] for instance -because they stick to the lexical and syntactic levels and do not take into account higher linguistic levels like discursive or argumental structures. They are also monolithic and supposed applicable to any kind of text whereas authors like [START_REF] Reiss | Translation criticism, the potentials and limitations : categories and criteria for translation quality assessment[END_REF] have argued that the evaluation criteria and their weight should be adapted to the text's function.

Protocol

The evaluation protocol is based on the ranking and judgement tasks used in MT subjective evaluation. These tasks were chosen because of their relative simplicity (compared to traditional 5-points scales) which also results in more reliable judgements as shown by [START_REF] Callison-Burch | Meta-) evaluation of machine[END_REF]. Automatic evaluation metrics were discarded because their only advantage -reproducibility -is of no use in this kind of evaluation: the protocol includes a subtask which is not reproducible (the translation) which makes the overall evaluation non-reproducible anyway. Evaluation grids were also discarded because they are too complex to put in practice, difficultly available and scarcely documented.

The evaluation's protocol is as follows:

1. Translators translate specialized texts in three different situations which we call "situations of translation". These situations of translation share a common base of identical generic resources (two monolingual and one bilingual dictionnaries). Translation are made from second-language to native language: situation 0 : translate with generic resources only situation 1 : translate with generic resources + a bilingual terminology extracted from comparable specialized corpora. situation 2 : translate with generic resources + full access to Internet where the translator can find all sorts of translation aids Situation 0 acts as a baseline where the translator has no specialized resource. The terminology used in situation 1 is the terminology under assessment. In situation 2, the Web is considered as some sort of "super-" or "meta-" specialized resource, because the translator will have access to all the specialized lexicons and termbases that are available online.

2. Once the translations are done, translators note down the time they spent in translation as well as the terms or expressions that they found problematic to translate and which drove them to use a linguistic resource. They also note down which resources they used to make the translation.

3. For each problematic term, judges rank the translations produced in the different situations 1 (ranking task). They also judge each translation separately using three categories: exact, acceptable or wrong (judgement task).

4. The added-valued of the bilingual terminology (situation 1) is measured by the comparing the quality of the translations produced in situation 1 with the quality of the translations produced in situations 0 and 2.

We decided to restrict the evaluation to the problematic terms rather than evaluating the quality of the whole translation because it appears from works in translation studies [START_REF] Williams | The application of argumentation theory to translation quality assessment[END_REF][START_REF] Reiss | Translation criticism, the potentials and limitations : categories and criteria for translation quality assessment[END_REF] that the overall quality of the translation of a text emerges from the complex interaction of various parameters (register, syntax, argumental structure, spelling, etc.) most of which terminologies have no influence upon. By focusing on the problematic terms and expressions, we isolate the part of the translation that terminologies are meant to improve. As a side effect, evaluating small segments also saves time and yields more reliable judgements as demonstrated by [START_REF] Callison-Burch | Meta-) evaluation of machine[END_REF].

The judgement task is based on three categories presented in table 1. These categories were chosen in accordance with [START_REF] Reiss | Translation criticism, the potentials and limitations : categories and criteria for translation quality assessment[END_REF] who states that the translation of "content-focused texts" (e.g. scientific and technical texts, manual for use...) should favor the transfer of the source text's meaning over the transfer of the source text's form. An acceptable translation is a translation which conveys the meaning of the source term. An exact translation is a translation which makes use of the expected, standard target term. In a way, the "meaning transfer" and "accurate form" criteria parallel the more classical "adequacy" and "fluency" criteria found in MT campaigns. meaning transfer accurate form exact ✔ ✔ acceptable ✔ wrong

Table 1: Translation quality criteria

In order to leverage differences in the quality of the translations which would arise from the translator's expertise rather than from the quality of the 1 Ties are allowed. language resource, each situation of translation is evaluated on the basis of texts translated by several translators. In turn, one has to be cautious that translators do not translate texts from the same domain in different situations of translation. Indeed, if a translator translates a text from domain A in situation 1, he/she must not translate a text from domain A in situation 2: there is a risk that the translator re-uses some terms'translations he/she has learnt in the previous situation.

A critical point when judging the translation of technical texts is that the judges often lack domain expertise and that domain experts are rarely available. One can get round this trouble by choosing specialized texts which already have an existing translation, like research paper abstracts for example. Judges can also get help from general terminological databases such as Termium2 .

The consistency of the judgements can be improved by first running a blank evaluation on a small set of data and then discussing the disagreements with the judges [START_REF] Blanchon | Pour l'évaluation externe des systèmes de TA par des méthodes fondées sur la tâche[END_REF]. In any case, it is necessary to provide the judges with clear instructions and examples of annotations on debatable cases.

Experimentation

This section describes the experimental framework (section 4.1) and the result of the evaluation (section 4.2).

Experimental framework

Data

Two bilingual terminologies were built for the evaluation. One was acquired from comparable corpora on BREAST CANCER and the other from comparable corpora on WATER SCIENCE . The WA-TER SCIENCE corpus is quite large (2M words per language) and its topic is coarse-grained. Texts are research papers from the journals Sciences de l'eau 3 and Water Science and Technology 4 . Conversely, the BREAST CANCER corpus is small (400k words per language) with a fine-grained topic. Texts come from various research papers of the publications portal Elsevier 5 . The texts to be translated belong to the same domains. They are divided into scientific texts and popular science texts. The scientic texts are 2 × 3 research papers abstracts taken from Elsevier and the water science journals. The popular science texts are 2 × 1 webpages taken from bilingual websites on breast cancer6 and water treatment7 ).

BREAST CANCER WATER SCIENCE scientific 508 499 pop. science 613 425

Table 2: Size of texts to be translated (number of words)

Data processing

The algorithm described in [START_REF] Fung | Finding terminology translations from non-parallel corpora[END_REF] was applied to the terms and to every open-class word occurring more than 5 times in the corpus. Extra knowledge was automatically added to the terms and open-class words in order to help the translators: part-of-speech, frequency, collocations8 , variants9 , related terms10 , definitions11 , concordances. Translators could browse the terminology via a dedicated interface designed for terminologies acquired from comparable corpora [START_REF] Delpech | Dealing with lexicon acquired from comparable corpora : validation and exchange[END_REF].

People involved

Due to the lack of human resources to perform the evaluation, there was some collisions in the roles of translator/judge and translator/organizer. Three persons were involved in the test of the protocol. The author of the paper, who is not a trained translator, translated the texts in the baseline situtation (general resources only) and organized the evaluation. Two trained translators translated the texts in situation 1 (terminology) and 2 (Internet) and also judged the translations. The translations were anonymized and randomly shuffled so that the judges would not know the origin of the translations.

Texts, domains and situations were distributed as follows :

Translators' feedback

It was difficult for translators to adapt to the ambiguity of the alignments. Although the aim and the context of the evaluation had been explained to them, they still expected the correct translation to appear "on click", just like it happens with the traditional languages resources they are accustomed to. Another obstacle was the coverage of the terminology, especially for the WATER SCIENCE domain whose topic was not refined enough. Table 3 shows the percentage of words of the texts to be translated which also appear in the terminology. Clearly, fined-grained corpora should be favored over large corpora.

BREAST CANCER WATER SCIENCE EN texts 94% 14% FR texts 67% 78%

Table 3: Terminology coverage of the vocabulary of the texts to be translated (EN) and their reference translation (FR)

Problematic terms

Problematic terms are terms or expressions that a translator found difficult to translate. Problematic terms retained for the evaluation are terms which were tagged problematic by at least 2 translators. We collected 148 problematic terms (26 tagged by 2 translators and 122 tagged by 3 translators). Table 4 shows the repartition of problematic terms among domains and types of corpora.

Time

The texts to be translated amounted to 2,147 words. Translators were quicker in situation 0 which is normal because they had less resources to browse (7.15 words/sec. on average). There is no significant time difference between situation 1 and situation 2 (11.18 and 11.6 words/sec. respectively).

Agreement between judges

Agreement was computed using the Kappa coefficient [START_REF] Carletta | Assessing agreement on classification tasks: The kappa statistic[END_REF] which takes into account the observed agreement P (A) and the agreement which would have occurred by chance P (E).

Kappa = P (A) -P (E) 1 -P (E)
Agreement was better for the ranking task: 0.65 (substantial) than for the judgement task: 0.36 (fair) which is consistent with the findings of [START_REF] Callison-Burch | Meta-) evaluation of machine[END_REF]. Agreement was better for the popular science texts: 0.57 (moderate) than for the scientific texts: 0.48 (moderate).

Judgement task

Table 5 gives quality judgements for the translations of the BREAST CANCER texts. The proportion of translations judged wrong is almost equivalent for all situations. Translations produced in situation 1 (with the terminology) are more often judged exact than the translations produced in situation 0 (only generic resources). 6 gives quality judgements for the translations of the WATER SCIENCE texts. One can see that translations produced in situation 1 are of lesser quality than those produced in situation 0. This is unexpected because situation 1 and situation 2 share a common base of generic resources. Translations produced in situation 1 should be at least as good as translations produced in situation 0.

The fact is that the translators used the languages resources in different manners depending on the situation in which they performed the sit.0 sit.1 sit.2 exact 59% 56% 77% acceptable 23% 23% 16% wrong 18% 21% 7%

Ranking task

The ranking task results are similar to those of the judgement task. When different translations of the same terms are compared, those produced in situation 2 are always better, whatever the domain.

Those produced in situation 1 are better than the ones produced in situation 0 only for the BREAST CANCER domain, probably because of divergences in the exploitation of the language resources as explained above.

5 Discussion and future work

We have described a protocol which assesses the added-value of terminologies acquired from comparable corpora when used for specialized human translation. This protocol consists in comparing sit.0 sit.2 sit.1 better than 28% 26% sit.1 as good as 47% 42% sit.1 worse than 26% 32% 9: Translations' ranking -WATER SCIENCE domain several situations of translation in which the translators have access to diverse language resources: only generic resources, generic resources and the evaluated terminology, generic resources and full access to Internet. The added-value of the terminology is supposed to be evidenced by the difference in the quality of the translations produced in the three situations. We have described in section 4 a first trial of the protocol. This first trial showed that some hitches in our procedure prevent us from clearly demonstrating the added-value of terminologies acquired from comparable corpora : we had contradictory results for the BREAST CAN-CER and WATER SCIENCE domains. Nonetheless, this first experimentation, although carried out with a small set of data and participants, allowed us to test the feasibility of the protocol and pinpointed problems which must be solved before launching a more thorough evaluation :

• The observed added-value of the terminology highly depends on its coverage of the texts used to evaluate it. Any measure of this added-value should also mention the adequacy between the assessed terminology and the texts to be translated, otherwise it is not interpretable. We determined this adequacy in a simple manner, by computing the proportion of words in the texts to be translated that also occurr in the terminology. This leaves some room for improvement. The comparability of the corpora used form terminology extraction and alignment must also be taken into account. For this, we are planning to use the comparability measure developped by [START_REF] Bo | Improving corpus comparability for bilingual lexicon extraction from comparable corpora[END_REF].

• The joint use of several language resources seems to bias the results as the translators' behaviour changes in function of the resources he/she has as his/her disposal. It is better to have only one resource per situation of translation, for instance:

situation 0: no resources, situation 1: assessed terminology only, situation 2: Internet only.

• Translators should be prepared to translate in a situation which is unusual to them. Ideally, one shoud run at first a blank translation task so as to discuss it with the translators and help them apprehend these new situations and resources.

The next step is to scale-up the protocol. We will renew the experiment on a much larger scale (a whole class of students translators) and include all the improvements listed above.

Finally, even if it was not the goal of this work, this first evaluation gives rise to some lines of research to improve the usefulness of terminologies acquired from comparable corpora. First, we have seen that the acquisition corpus should be collected in function of the texts that are to be translated and that the topic should be fine-grained. Second, it is clear that the Internet is a huge repository of linguistic resources and translations. A nice development would be to add a new functionality to the terminology software which, when the queried term is not present in the database, would either automatically generate a translation and filter it on the Internet or search it in pre-selected online resources. However, the worth of Internet as a linguistic resource should not be overestimated. In most professional translations, translators have to translate texts whose vocabulary can not be found on the Internet. It is especially the case with corporate translations : companies use their own terminologies, which can only be found in the texts produced by the company itself. Thus, we can not expect to rely on Internet as a unique source of translations and still need to improve the term alignment program. For this, we are planning to use translation techniques relying on the compositionality of terms [START_REF] Morin | Compositionality and lexical alignment of multi-word terms[END_REF] in addition to the distribution-based approaches [START_REF] Fung | Finding terminology translations from non-parallel corpora[END_REF] presented in section 2.1 and which we used for this evaluation.

Table 4 :

 4 Problematic terms used for evaluation

		BREAST CANCER WATER SCIENCE
	pop. science	34	10
	specialized	43	51
	total	87	61

Table 5 :

 5 Translations' quality -BREAST CANCER domain

Table

Table 6

 6

	: Translations' quality -WATER SCIENCE do-
	main
	translations. Thanks to the data collected dur-
	ing the translation phase, we are able to tell, for
	each term translation if it was produced using the
	generic resource or the specialized resource (ter-
	minology/Internet) or relying on intuition (not ex-
	clusive). Table 7 shows that the translators who
	had access to a specialized resource scarcely used
	the generic resource. It might be because they felt
	the generic resource was useless to get the trans-
	lation of technical terms and they prefered to use
	directly the specialised resource. But as the WATER
	SCIENCE terminology covers only a small part of
	vocabulary of the texts to translate, it was barely
	advantageous. A systematic exploitation of the
	generic resource in situation 1 would have led to
	translations at least as good as those produced in
	situation 0.
		sit.0 sit.1 sit.2
	gen. ress. 43% 14% 3%
	spec. ress.	-	25% 56%
	intutition	79% 77% 44%

Table 7 :

 7 Exploitation of the language resources depending on the situation of translation

Table 8 :

 8 Translations' ranking -BREAST CANCER domain

	sit.0 sit.2
	sit.1 better than 18% 16%
	sit.1 as good as 49% 41%
	sit.1 worse than 33% 43%
	Table

http://www.termiumplus.gc.ca/

http://www.rse.inrs.ca/

http://www.iwaponline.com/wst/

http://www.elsevier.com/

http://www.cbcf.org/

http://www.lenntech.com/

most remarkable cooccurrents, the association measure is the log-likelihood ratio[START_REF] Dunning | Accurate methods for the statistics of surprise and coincidence[END_REF]

phrases which have not been identified as terms by the term extractor but have words in common with the entry term

terms which have words in common with the entry term

either the Wikipedia or Wiktionary article if avalaible or a sentence extracted from the corpus and containg a very simple pattern like "A $TERM is a..."

Acknowledgments

This work was funded by the company Lingua et Machina and the French National Research Agency (funding no. ANR-08-CORD-009). I would like to thank Clémence de Baudus and Mathieu Delage for their participation in the evaluation.