Janine Emile

Olivier Emile
email: olivier.emile@univ-rennes1.fr

Pierre Gaudriault

Resonant vibration of a thin polymer film under optical excitation

We report on the mechanical excitation of a 220 µm thick thermoplastic film in its amorphous state by the radiation pressure of light. By modulating a low power visible laser (from 100 to 600 mW) at low frequencies (below 100 Hz), we observe a deformation of the film interfaces. The phenomenon, that is independent of the laser wavelength, is amplified at a resonant frequency and reaches 0.68 µm. The deformation is reversible and varies linearly with the optical power. Using the damped oscillator model, we show that the resonant frequency depends on the surface tension of the film. The associated free energy is then compared with the energy lost, taking into account the contribution of the damping corresponding to the imaginary part of the Young's modulus.

INTRODUCTION

Resonant phenomena are universal in nature. They occur when the frequency of a periodically applied force is in harmonic proportion to a natural frequency of the system on which it acts. They have been extensively considered and exploited in many scientific areas [START_REF] Ogata | System dynamics[END_REF][START_REF] Halliday | Fundamentals of physics[END_REF]. Pushing a person on a swing is probably the most common example of it. Among all these resonant phenomena, the vibration of thin films such as membranes bears a special status because of its close links with acoustical vibration [START_REF] Jenkins | [END_REF] and hearing [START_REF] Fay | Proc. Natl. Acad. Sci[END_REF]. It is thus compulsory to fully characterize the mechanical properties of the membranes [START_REF] Nan | [END_REF]. This also includes a freely suspended soap film that was early recognized as an archetype of a vibrating membrane [6].

In this case, the vibrating mode is an asymmetric mode, where both interfaces are curved in an asymmetric way.

Recently, the vibration of a soap film in a symmetric mode, where the interfaces of the soap film undulate in anti-phase, has been reported [7,8] at low frequencies. This has also been evidenced in an amorphous material deposited on a glass substrate under optical radiation pressure excitation [9]. The phenomenon has been attributed to the contribution of the surface tensions of the different interfaces probed. Nevertheless, no link has been established with the mechanical properties of the material since: (i) the soap film is very fragile with a limited lifetime due to environmental conditions; (ii) the deposited films under resonant vibrations delaminate from the substrate or deteriorate over time. For these two examples, the duration of experiments with reproducibility was rather short to extract reliable information. One may then wonder whether such resonances could be encountered in polymer films with well-known elastic properties. The aim of this article is thus to investigate the dynamical response of a thin thermoplastic sample excited by modulated light, coupling optical radiation pressure to mechanical resonance phenomena.

2

A c c e p t e d m a n u s c r i p t

EXPERIMENTAL SET-UP

We have studied the vibration of a rectangular 60 × 42 mm 2 (thickness e = 220 µm, mass m = 565 mg) commercially available Topas R 8007 film. Topas (abbreviation of Thermoplastic Olefin Polymer of Amorphous Structure) is a Cyclic Olefin Copolymer (COC) [10] that is used for packing, optics and fiber spinning. It is also widely used in the healthcare area because of its biocompatible properties and also in microfluidics [11]. Thin films made of COC can be obtained, with high optical surface quality and with less aging problems [12]. The Topas glass temperature transition is T G = 78 • C, with a Young's modulus E = 2.4 GPa and Poisson's ratio σ = 0.41. Its optical index in the visible spectrum is n = 1.53 [13,14]. In contrast to elastomers where capillary forces induce significant interfacial deformations [15], the surface stress in thermoplastic films is negligible.

Our experimental set up is displayed in figure 1a. A laser beam combiner (L4Cc Oxxius, wavelength λ = 532 nm, output power P = 250 mW at the end of the output coupling fiber) is directed on the film nearly at normal incidence. The output power can be increased up to P = 600 mW by using other wavelengths (λ = 404 nm and λ = 488 nm). Because of the optical index difference between the air and the film, there is discontinuity of the linear momentum at the interface [16,17] that induces a force at the interface between the two media. It goes from the higher optical index media towards the lower optical index media. Since the Topas index is n = 1.53 and the index of the air is around 1, the force is directed from the Topas to the air, whatever the interface, in particular, for a fixed direction it changes sign from one interface to the other (see figure 1b). More precisely, the force of the transmitted light on the first air/film interface is

-→ F 1 t = (1 -n) 4n (n + 1) 2 N - → k = (1 -n) 4n (n + 1) 2 P c - → u (1)
where N = P/hν l is the number of photons impinging on the interface per second, -→ k is the linear momentum per photon, = h/2π is the reduced Planck constant, ν l is the optical frequency, and -→ k is the wavevector (k = 2πν l /c, c is the velocity of light in vacuum). -→ u is a unitary vector collinear to -→ k . The term (1 -n) refers to the optical index discontinuity, 4n/(n + 1) 2 accounts for the light intensity transmission. Since the force is associated with a change of linear momentum, it can thus be called an optical radiation pressure force [17].

This force has to be corrected for the force due to reflected light acting on this first interface -→ F 1 r (see figure 1b). Since the momentum change upon reflexion air/film interface is F A/F = F t 1 -F r 1 = 7.0 × 10 -10 N for a 500 mW laser. For the second film/air interface, one has to consider the transmitted light from the air/film interface. The force due to the optical index variation is -→

F 2 t = - -→ F 1 t × 4n/(n + 1) 2
and the force due to the reflected light is

-→ F 2 r = -→ F 1 r
× 4n 2 /(n + 1) 2 acts on the same direction (see figure 1b) leading to a total force equal to

F F/A = F t 2 + F r 2 = 1.02 × 10 -9 N.
On each interface, the force is thus of the order of F = 1 × 10 -9 N and is always directed from the film to the air. The waist of the laser is w w = 0.6 mm, the value of the optical radiation pressure is then p = 1.4 × 10 -3 Pa. When applied in a continuous way, this value is too low to induce any noticeable thickness variation due to stretching or surface bending linked to the stiffness properties of the sample.

For example, considering the Young's modulus E, for a e = 220 µm thickness, the thickness variation is ∆e = p/E × e = 1.5 × 10 -16 m, which is hardly noticeable.

The difficulty to deform an interface with the light radiation pressure is not limited to solids. This is also true for liquids. However, in this later case, in order to bend the interface, the light radiation pressure has to compensate for the surface tension of the liquid [18]. Unlike acoustic radiation pressure, the optical force is applied on a much smaller area of the sample allowing its interfacial properties to be probed locally.

RESULTS

To increase the surface deformation, as for the bending of air/liquid interfaces [7], we use the resonance of the sample under modulated optical radiation pressure force. We chop the laser beam at a given controlled frequency ν. We measure the mean variation thickness of the film using the interference fringes from two He-Ne collimated laser probe beams: a red one (λ R = 633 nm, beam waist w r = 400 µm) and a green one (λ G = 543 nm, beam waist w g = 380 µm) [19] (see figure 1a). Both beams are partly reflected on each side of the film, leading to constructive or destructive interferences depending on the film thickness. As the thickness varies, the transmission varies accordingly, leading to fringes. A variation of one fringe then corresponds to a thickness variation of λ/2n.

An example of the transmission of the two laser beams versus time (sampling time: 0.01 s) is displayed in figure 2a, for a 500 mW laser beam at a modulation frequency of 65.7 Hz. The two probe lasers are on. We switch on the modulated laser at time t = 0 and we register the transmission of the probe beams versus time.

The frequency has been chosen to optimize the number of fringes. A variation of a single fringe in the laser transmission in figure 2a corresponds to a thickness variation of half a wavelength (λ/(2n)). There is thus a direct correspondence between the variation of the transmission and the variation of the thickness. We can follow the thickness variation of the film, which is then confirmed by the transmission of the second laser (see Fig. 2b). Note that the variation has to be higher than λ/2n in order to register at least one fringe and thus to calibrate our The film experiences a succession of swelling and deflation leading, as for the liquid film [7], to a mean thinning. Usually, optical radiation pressure induces bulges on the film interfaces but we here measured dimples (see Fig. 2b). Even at resonance, and whatever the direction of the applied force, one should observe a mean response equal to zero, i.e. no variation in thickness. One has to dissociate (i) the vibrations of the system with forced oscillations, (ii) the maximum amplitude of the oscillations, and (iii) the mean thinning of the system induced by the vibrations. This thinning phenomenon may be attributed to the effects of negative compressibility of the material [20] that also appears in polymers [21].

This negative compressibility could be associated to rearrangements of the polymer chains due to resonant forced oscillations of the system. Then, within the stressed film, thinning is more effective than swelling, giving an asymmetric mechanical response.

The deformation equals ∆e = -680 nm. It is worth noting that it corresponds to a static deformation of the film due to the modulated radiation pressure. The time constant associated with this deformation is τ = 9.1 s (exponential adjustment). The amplitude of the oscillations was measured thanks to a lock-in amplifier. It corresponds to approximately 2 nm for a 500 mW excitation at the resonant frequency. The amplitude of the oscillation follows the same resonance curve as the mean deformation. For a power of 600 mW, the thinning saturates to this same value. We have also performed experiments on a 20 µm PVC film but didn't succeed in measuring thickness variations. Actually, assuming that the physical properties of PVC and Topas materials are similar and that the thinning of the sample is correlated with the sample thickness, a 60 nm thickness variation would be expected for the PVC. This value is well below λ/2n, and is then very difficult to measure.

We repeat the measurement for different frequencies and different optical pow-

7

A c c e p t e d m a n u s c r i p t ers, and register the transmission of the green and red lasers. We apply the modulated laser for 2 minutes in order to reach a steady state, and switch off the laser for another 2 minutes to let the film relax before changing the frequency. The relaxation time constant is also equal to 9.1 s (see Fig. 2b). It is also the same for each laser power. Such transmissions versus the modulation frequency appear in figure 3a for a laser power of 300 mW. Note that the number of fringes is higher for 500 mW, but the results are less demonstrative. For the sake of simplicity, we present them for a 300 mW laser. From the transmission curves, we deduce the mean thickness variation versus the modulation frequency (see figure 3b). One clearly notices a resonant behavior at a frequency of ν = 65.7 Hz. The quality factor equals Q = 4, assuming that the amplitude of the vibration is proportional to the mean thickness variation. Around resonance, the time constant τ associated to the thinning is unchanged. Far for resonance, it is more difficult to evaluate it. It is worth noting that the deformation is reversible since the polymer film recovers its initial shape after switching off the laser. Actually, the transmission of the probe lasers recovers its initial value.

This thickness variation must be linked to the resonance response of the sample under light radiation pressure. It cannot be attributed to a change of conformation due to the light irradiance [START_REF] Aguilar | Smart Polymers and Their Applications[END_REF] nor to a local heating because the light flux is the constant whatever the modulation frequency. Besides, Topas is transparent at these wavelengths and absorption is negligible. Since light is nothing but electromagnetic fields, one may wonder whether electrostriction could be responsible for this effect. However, electroactive polymer have been widely studied [START_REF] Bar-Cohen | [END_REF] and resonant responses under modulated excitation have never been reported.

We have varied the optical power of the modulated light (see figure 4a) and found the same resonant behavior. From a theoretical adjustment of the experimental data, we find the same resonant frequency and the same Q factor for all the optical powers. One also notices that the mean thinning varies linearly with From the fringes, one can deduce the absolute value of the thickness variation versus modulation frequency. The solid line corresponds to a fit of the experimental curve using equation 3. Error bars: same as Fig. 2.

the optical power as expected (see figure 4b) up to a power a 500 mW. We were not able to make a complete cartography of the surface of the film because the amplitudes, out of the irradiated surface must be too low.

DISCUSSION

Optically induced mechanical oscillations

What could be the origin of such a resonance that resembles the mechanical oscillator behavior? Let us consider the equation of the weakly damped mechanical oscillator excited at a pulsation ω applied to our experiment

mẍ + α ẋ + k s x = F A/F cos(ωt) (2)
where we consider a single interface. Actually, since we excite the interface in a symmetric way (see section ), we thus expect the deformation to be symmetric,

9

A c c e p t e d m a n u s c r i p t corresponding to a breathing mode where both interfaces oscillate in a symmetric way. This type of vibration is opposed to the usual bending mode where both interfaces oscillate in an antisymmetric way. We can then only consider half of the film.

x corresponds to the amplitude of the forced oscillation around a zero position.

It is in the direction of the force. m is the mass of the film, α is the viscous damping coefficient, k s is a spring constant, and ω is the driving pulsation. Note that the laser is modulated with a chopper leading to a square modulation of the light intensity. However, we don't observe experimentally any response at twice, and three times the ω pulsation, either on the mean thinning or on the amplitude of the oscillation. We could then consider a cosine modulation of the laser only. Note also that Eq. 2 corresponds to a forcing with positive and negative values whereas the optical radiation pressure force here always acts outward. Nevertheless, we can add a constant force in Eq. 2 that equals half of the value of the optical radiation pressure force term. Thus, the remaining part of the force would be a forcing term with positive and negative values. This constant term could then be moved to the other side of the equation and would behave as an offset (or a change of origin in the vibration amplitude x). Let us introduce the natural resonant pulsation

ω 0 = k s /m. The well-known solution of equation 2 is x(t) = x A cos(ωt + φ), φ
being the phase between the excitation and the response, and

x A = F A/F m 1 (ω 2 -ω 2 0 ) 2 + α 2 ω 2 /m 2 (3)
The resonant pulsation ω R in position equals

ω R = ω 0 (1 - 1 2Q 2 ) 1/2 (4)
Since the experimental resonant pulsation is 2π × 65.7 rad×s -1 , the natural resonant pulsation is equal to ω 0 = 2π × 66.7 rad×s -1 , according to equation 4.

It is worth noting that in the case of capillary waves in ultra thin liquid films (thickness of the order of 100 nm), the resonant frequency depends on the film thickness [24]. However, since the thickness in our experiment is 220 µm, we can neglect this dependency and can consider the resonant frequency as a constant.

Let us comment briefly on some of the terms that appear in the simple model of Eq. 2. Whereas the force is applied on a very specific zone of the surface of the sample, its response in our model includes its whole mass and not only the mass underneath the laser. In order to check this hypothesis, we have cut our sample in two pieces with the same thickness 60 × 21 mm 2 , total mass m = 282 mg), and we have looked for a resonance. The new natural resonant pulsation should be ω 0 = 2π × 93.6 rad×s -1 , and the quality factor Q = 2 according to Eq. 4. The new resonant pulsation should be ω R = 2π × 87.5 rad×s -1 .

The resonance curve versus frequency for m appears in Fig. 5, together with the resonance curve for m taken from Fig. 4, for a laser power of 500 mW.

Experimentally we find ω R = 2π × 84. from Eqs. 3 and 4. The fitting parameters are also in agreement with Eq. 3.

By analogy with our experiments on air/liquid interfaces [7], we attribute this resonance phenomenon to the surface tension contribution of the film. The sample is too thick to take into account elastic bending [25] and compressibility effects.

By analogy with the tensiometer [START_REF] Butt | Physics and chemistry of interfaces[END_REF], the spring constant should be proportional to the surface tension γ times the contour on which the tension is applied, divided to the maximum elongation which is ∆e/2 for one interface. It can be written as k s = 2πw w γ/(∆e/2). Let us recall that w w is the beam waist. Then the natural resonant pulsation is

ω 0 = 4πw w γ m|∆e| (5)
For m = 5.65 × 10 -4 kg, ∆e = -680 nm, and for an optical power of 500 mW and beyond, this leads to γ = 1.0×10 -2 N×m -1 which is of the same order of the value of γ = 3.2 × 10 -2 N×m -1 measured indirectly using contact-angle techniques [START_REF] Jena | [END_REF],

although the measurement was performed with a different Topas grade.

One can argue that this resonance may be due to the bulk elastic proper-

12

A c c e p t e d m a n u s c r i p t ties of the polymer film. Then the natural resonant pulsation equals ω 0 = A Ee 2 /(ρ(2πw w ) 4 (1 -σ 2 )) where the calculated constant A depends on the film geometry [START_REF] Harris | Harris' shock and vibration handbook[END_REF] (here A = 6.09 rad×s -1 ). For our experimental parameters, the ω 0 value would be ω 0 = 2π × 150 × 10 3 rad×s -1 , i.e. between 100 kHz and 1

MHz. This is very different from the experimental values we found. Besides, as already mentioned, the stress imposed by the laser light would correspond to a pressure below the mPa range and a deformation of the sample that is very low (in the 10 -16 m range) if it was only due to the elastic properties of the film. Even with a sharp and strong resonance, this elastic effect should not be noticeable. We can thus neglect the mechanical properties of the film in volume and concentrate on the deformation due to the interfacial properties only.

One may nevertheless express some reservations about the one-dimensional model (Eq.3), since the Gaussian laser beam acts on a surface perpendicular to the thickness. At least a two-dimensional description could be needed, including the radial intensity distribution of the laser beam. Although this description is far beyond the scope of the article, one may note that our one-dimensional model gives the correct orders of magnitude. Indeed, the capillary length, λ c , is used to estimate the size of the deformed surface. Knowing that λ c = (γ/(ρg)) 1/2 , where g is the gravitational acceleration, one finds that λ c is approximately equivalent to the laser waist (about 1 mm). We could thus in first approximation forget the radial distribution of the laser intensity and also of the deformation of the film surface.

Energy storage

From equation 3, one can deduce the Q factor of the resonance Q = mω 0 /α, and thus, from the value derived from figure 3, we estimate the damping coefficient α = 6.23×10 -2 kg×s -1 . It has also to be noted that the time constant deduced from this One hypothesis is that the relaxation time of the polymer chains, especially at the interface, is much longer than the elastic relaxation time of the system. Its characteristic time could be much higher than the one-second range [START_REF] Paeng | [END_REF]. It is difficult to compare the relaxation time they found with what we measured since they used a photobleaching technics that depends on the fluorescence probes.

Besides, since this coefficient α corresponds to an energy damping, it should give access to the elastic properties of the sample. In particular, the imaginary part of the Young's modulus E is related to the energy dissipation. One can define α = E (2πw w e/V ), where 2πw w e is the surface of the cylinder underneath the region enlightened by the laser, and V = E/(2ρ(1 + σ)) = 913 m×s -1 is the sound velocity in the medium. This then leads to E = 74 MPa and to a loss factor tan δ = E /E 0.03 that is compatible with the values deduced from Dynamical Mechanical Analysis (DMA) [START_REF] Menard | Encyclopedia of Analytical Chemistry: Applications, Theory and Instrumentation[END_REF]. The damping coefficient could then be associated with the imaginary part of the elastic mechanical coefficient of the sample. However, whereas for a usual damped mechanical oscillator, the energy is dissipated into heat, here, part of the energy is stored within the material and then restored to the system that recovers its initial shape.

This stored energy can be estimated in two ways. First, the power lost by a mechanical damped harmonic oscillator is (1/2)α ẋ2 A , where ẋA is the oscillation velocity amplitude. At resonance, on the first interface ẋA = F A/F /α, according to equation 3. During a time τ , the energy dissipation is then τ F 2 A/F /2α = 3.6×10 -17 J, for the first interface and 7.6 × 10 -17 J for the second interface. There is also a mean variation ∆e of the thickness during this time. The corresponding velocity 14 A c c e p t e d m a n u s c r i p t is (∆e)/τ , associated with a damping force α(∆e)/τ on a distance (∆e), leading to a lost (α(∆e) 2 )/τ = 3.1 × 10 -15 J. The total energy lost is of the order of ∆E = 3.3 × 10 -15 J. The mean thinning when light is on, as the one shown in figure 3b, corresponds to an energy storage in the film.

The stored energy can also be estimated using a thermodynamic approach and considering the free energy of the system. The temperature is constant (see experimental conditions described in figure 1a). The surface deformation on each interface equals ∆S = π/2(∆e/2) 2 , assuming a deformation following the Gaussian laser intensity distribution. The same results hold for a conical deformation, and the order of magnitude is the same whatever the deformation shape. Then, for the two interfaces, the free energy variation is ∆F = (π/4)∆e 2 γ. Considering the value of γ = 1.0 × 10 -2 N×m -1 that has been already estimated, the variation of the free energy is 3.6 × 10 -15 J. It is of the same order of magnitude as the energy lost during the time τ . This reinforces the hypothesis that the deformation is due to the interfacial properties of the polymer film.

CONCLUSION

To conclude, we have experimentally shown a resonant breathing mode of a 220 µm thick thermoplastic film. The excitation is performed by the radiation pressure of a visible laser that leads to an antisymmetric vibration on both interfaces of the film. The thinning of the film is a fraction of a micrometer, i.e. in the 0.1 percent range of the initial thickness. However, it could be detrimental when the thermoplastic is used as an optical material. For example, in some optogenetic interrogation experiment, light is guided by polymer fibers [START_REF] Park | [END_REF]. Modulating the light intensity may lead to the deformation of the fiber surface and consequently to the change of the light injection, that thus may change the diagnosis or the analysis.

Alternatively, such resonant phenomena could initiate the bending or the twisting 15 A c c e p t e d m a n u s c r i p t in multilayers thermoplastic polymers [32] or favor the return to equilibrium of shape memory polymers [33]. This thickness variation is reversible. It disappears as the laser excitation is switched off. Besides, the effect is a local effect. The response of the film is governed by its surface tension, deduced from the experimental resonant frequency.

Using a spring model, the damping of the oscillations is linked to the imaginary part of the Young's modulus. These experimental results show that the surface tension of a solid material can be measured directly, without using contact angle methods with a series of liquids. Besides, the technique developed here is not limited to thermoplastic polymers and may be generalized to measure the surface tension of solids in general whose thermodynamic definition is confusing compared to liquids [34,35], and where reliable experimental data are scarce [36,37].

3 AFIG. 1

 31 FIG. 1. a) Experimental set up: a 532 nm intensity modulated laser excites a polymer film. Its thickness variation is measured with the interference fringes of two He-Ne lasers (λ R = 633 nm and λ G = 543 nm) detected on photodiodes PD R and PD G . Reference signals correct variations of the laser intensity. F: color filters. BS: beam splitter. The temperature is controlled to be 20 ± 0.2 • C. b) Higher magnification of the film showing the optical radiation pressure forces on the two interfaces (1-air/film interface, 2-film/air interface). The resultant forces are directed from the higher to the lower index medium. c) Picture of the film support that shows how the film is suspended.

4 A c c e p t e d m a n u s c r i p t

 4 force is F r 1 = ((n -1)/(n + 1)) 2 2P/c. Finally, the force on the

5

 5

FIG. 2 .

 2 FIG. 2. a) T: intensity transmission recorded on the two photodiodes (PD R and PD G versus time, at a frequency of 65.7 Hz and for a laser power of 500 mW), when the laser is switched on; frequency sampling: 0.2 Hz. b) Circles and solid line: variation of the film thickness versus time deduced from the interference fringes from a). Diamonds and doted line: variation of the film thickness when the laser is switched off. The error bars, obtained from at least 5 measurements at the same frequency, appear on the graph.

6 A

 6 c c e p t e d m a n u s c r i p t observations.

8 AFIG. 3 .

 83 FIG. 3. a) Interference fringes of the transmissions T of the red and green laser versus modulation frequency, for a P=300 mW laser power, frequency sampling: 0.2 Hz. b)

10

 10

A c c e p t e d m a n u s c r i p t 20 FIG. 4 .

 204 FIG. 4. Absolute value of the thinning versus the modulation frequency for various laser powers (the solid lines correspond to theoretical fits), together with the maximum thickness variation versus power at the resonant frequency ν = 65.7 Hz. Straight line: theoretical adjustment according to Eq. 3. Error bars: same as Fig. 2.

6 rad×s - 1 11 AFIG. 5 .

 1115 FIG. 5. Absolute value of the thinning for two different masses (m = 565 mg, m = 285 mg) of the sample versus the modulation frequency. Straight line: theoretical adjustment. Error bars: same as Fig. 2. Resonant frequency ν R = 65.7 Hz for m, and ν R = 84.6 Hz for m .

13 A c c e p t e d m a n u

  s c r i p t equation m/α = Q/ω 0 = 9.07 × 10 -3 s is very different from the value of τ = 9.1 s deduced from figure 2. Actually, they do not correspond to the same phenomenon. m/α is the time constant associated with a single excitation of the sample, whereas τ is the time associated with the mean thinning of the sample while periodically excited. It is worth noting that the value of τ = 9.1 s is a rather long time.

  s c r i p t


Nicolas Zampieri

Irina Illina

Dominique Fohr

Multiword Expression Features for Automatic Hate Speech Detection

Keywords: Social media, Hate speech detection, Deep learning

published or not. The documents may come   L'archive ouverte pluridisciplinaire

Introduction

Hate speech detection (HSD) is a difficult task both for humans and machines because hateful content is more than just keyword detection. Hatred may be implied, the sentence may be grammatically incorrect and the abbreviations and slangs may be numerous [START_REF] Nobata | Abusive language detection in online user content[END_REF]. Recently, the use of machine learning methods for HSD has gained attention, as evidenced by these systems: [START_REF] Pamungkas | Automatic identification of misogyny in english and italian tweets at evalita 2018 with a multilingual hate lexicon[END_REF][START_REF] Indurthi | FERMI at SemEval-2019 task 5: Using sentence embeddings to identify hate speech against immigrants and women in Twitter[END_REF]. [START_REF] Lee | Comparative studies of detecting abusive language on twitter[END_REF] performed a comparative study between machine learning models and concluded that the deep learning models are more accurate. Current HSD systems are based on natural language processing (NLP) advances and rely on deep neural networks (DNN) [START_REF] Mozafari | A bert-based transfer learning approach for hate speech detection in online social media[END_REF].

Finding the features that best represent the underlying hate speech phenomenon is challenging. Early works on automatic HSD used different word representations, such as a bag of words, surface forms, and character n-grams with machine learning classifiers [START_REF] Waseem | Hateful symbols or hateful people? predictive features for hate speech detection on twitter[END_REF]. The combination of features, such as n-grams, linguistic and syntactic turns out to be interesting as shown by [START_REF] Nobata | Abusive language detection in online user content[END_REF].

In this paper, we focus our research on the automatic HSD in tweets using DNN. Our baseline system relies on Universal Sentence Embeddings (USE). We propose to enrich the baseline system using word-level features, called multiword expressions (MWEs) [START_REF] Sag | Multiword expressions: A pain in the neck for nlp[END_REF]. MWEs are a class of linguistic forms spanning conventional word boundaries that are both idiosyncratic and pervasive across different languages. [START_REF] Constant | Multiword expression processing: A Survey[END_REF] We believe that MWE modelling could help to reduce the ambiguity of tweets and lead to better detection of HS [START_REF] Stanković | Multi-word expressions for abusive speech detection in Serbian[END_REF]. To the best of our knowledge, MWE features have never been used in the framework of DNN-based automatic HSD. Our contribution is as follows. First, we extract different MWE categories and study their distribution in our tweet corpora. Secondly, we design a three-branch deep neural network to integrate MWE features. Finally, we experimentally demonstrate the ability of the proposed MWE-based HSD system to better detect hate speech: a statistically significant improvement is obtained compared to the baseline system. We experimented on two tweet corpora to show that our approach is domain-independent.

Proposed methodology

In this section, we describe the proposed HSD system based on MWE features. This system is composed of a three-branch DNN network and combines global feature computed at the sentence level (USE embeddings) and word-level features: MWE categories and word embeddings representing the words belonging to MWEs.

Universal sentence encoder provides sentence level embeddings. The USE model is trained on a variety of data sources and demonstrated strong transfer performance on a number of NLP tasks [START_REF] Cer | Universal sentence encoder for English[END_REF]. The HSD system based on USE obtained the best results at the SemEval2019 campaign (shared task 5) [START_REF] Indurthi | FERMI at SemEval-2019 task 5: Using sentence embeddings to identify hate speech against immigrants and women in Twitter[END_REF]. This power of USE motivated us to use it to design our system. MWE features. A multiword expression is a group of words that are treated as a unit [START_REF] Sag | Multiword expressions: A pain in the neck for nlp[END_REF]. For example, the two MWEs stand for and get out have a meaning as a group, but have another meaning if the words are taken separately. MWEs include idioms, light verb constructions, verb-particle constructions, and many compounds. We think that adding information about MWE categories and semantic information from MWEs might help for the HSD task.

In our work, we focus on social media data. These textual data are very particular, may be grammatically incorrect and may contain abbreviations or spelling mistakes. For this type of data, there are no state-of-the-art approaches for MWE identification. A specific MWE identification system is required to parse MWEs in social media corpora. As the adaptation of an MWE identification system for a tweet corpus is a complex task and as it is not the goal of our paper, we decided to adopt a lexicon-based approach to annotate our corpora in terms of MWEs. We extract MWEs from the STREUSLE web corpus (English online reviews corpus), annotated in MWEs [START_REF] Schneider | A corpus and model integrating multiword expressions and supersenses[END_REF]. From this corpus, we create an MWE lexicon composed of 1855 MWEs which are classified into 20 lexical categories. Table 1 presents these categories with examples. Each tweet of our tweet corpora is lemmatized and parsed with the MWE lexicon. Our parser tags MWEs and takes into account the possible discontinuity of MWEs: we allow that one word, not belonging to the MWE, can be present between the words of the MWE. If, in a sentence, a word belongs to two MWEs, we tag this word with the longest MWE. We do not take into account spelling or grammatical mistakes. We add a special category for words not belonging to any MWE.

HSD system proposal. In this part, we describe our hate speech detection system using USE embeddings and MWE features. As USE is a feature at the sentence level and MWE features are at the word level, the architecture of our system is composed of a neural network with three branches: two branches are dedicated to the MWE features, the last one deals with USE features. Figure 1 shows the architecture of our system.

In the first branch, we associate to each word of the tweet the number of the MWE category (one-hot encoding). This branch is composed of 3 consecutive blocks of CNN (Conv1D) and MaxPooling layers. Previous experiments with different DNN structures and the fast learning of CNN allow us to focus on this architecture. The second branch takes into account the semantic context of words composing MWE. If a given tweet has one or several MWEs, we associate a word embedding to each word composing these MWEs. We believe that the semantic meaning of MWEs is important to better understand and model them. This branch uses one LSTM layer. We propose to use two types of word embeddings: static where a given word has a single embedding in any context, or dynamic, where a given word can have different embeddings according to his long-term context. We experiment with word2vec and BERT embeddings [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]. BERT uses tokens instead of words. Therefore, we use the embedding of each token composing the words of the MWEs. We think that using two branches to model MWEs allows us to take into account complementary information and provides an efficient way of combining different features for a more robust HSD system.

The last branch, USE embedding, supplies relevant semantic information at the sentence level. The three branches are concatenated and went through two dense layers to obtain the output. The output layer has as many neurons as the number of classes.

3 Experimental setup

Corpora

The different time frames of collection, the various sampling strategies, and the targets of abuse induce a significant shift in the data distribution and can give a performance variation on different datasets. We use two tweets corpora to show that our approach is domain-independent: the English corpus of SemEval2019 task 5 subTask A (called HatEval in the following) [START_REF] Basile | SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter[END_REF] and Founta corpora [START_REF] Founta | Large scale crowdsourcing and characterization of twitter abusive behavior[END_REF]. We study the influence of MWE features on the HatEval corpus, and we use the Founta corpus to confirm our results. Note that these corpora contain different numbers of classes and different percentages of hateful speech. We evaluate our models using the official evaluation script of SemEval shared task 51 in terms of macro-F1. It is the average of the F1 scores of all classes.

HatEval corpus. In the HatEval corpus, the annotation of a tweet is a binary value indicating if HS is occurring against women or immigrants. The corpus contains 13k tweets. We We apply the following pre-processing for each tweet: we remove mentions, hashtags, and URLs. We keep the case unchanged. We use this pre-processing because the systems using this pre-processing obtained the best results at the SemEval2019 task 5.

For train and development sets, we keep only tweets that contain at least two words. Thus, we obtain 8967 tweets for the training set and 998 tweets for the development set. We split the training part into two subsets, the first one (8003 tweets) to train the models, and the second one (965 tweets) for model validation. In the test set, we keep all tweets after pre-processing, even empty tweets. We tag empty tweets as non-hateful.

Founta corpus contains 100k tweets annotated with normal, abusive, hateful, and spam labels. Our experiments focus on HSD, so we decided to remove spams and we keep around 86k tweets. The vocabulary size of the corpus is 132k words. We apply the same pre-processing as for the HatEval corpus. We divide the Founta corpus into 3 sets: train, development, and test with 60%, 20%, and 20% respectively. As for the HatEval corpus, we use a small part of training as the validation part. Each set contains about 62%, 31%, and 6% of normal, abusive, and hateful tweets.

System parameters

Our baseline system utilizes only USE features and corresponds to figure 1 without MWE branches. The system proposed in this article uses USE and the MWE features as presented in figure 1 2 . For the USE embedding, we use the pre-trained model provided by google 3 (space dimension is 512) without fine-tuning.

We tag the MWE of each tweet using the lexicon, presented in the section 2. If an MWE is found, we put the corresponding MWE category for all words of the MWE. To perform finegrained analysis, we decided to select MWE categories that have more than 50 occurrences (arbitrary value) and occurrences appear less than 97% in hate and non-hate tweets at the same time. We obtain 10 MWE categories: called MWE5 and VMWE5 which are respectively the first and second part of Table Concerning the MWE one-hot branch of the proposed system, we set the number of filters to 32, 16, and 8 for the 3 Conv1D layers. The kernel size of each CNN is set to 3. For the MWE word embedding branch, we set the LSTM layer to 192 neurons. For BERT embedding, we use pre-trained uncased BERT model from [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] (embedding dimension is 768). The BERT embeddings are extracted from the last layer of this model. For word2vec embedding, we use the pre-trained embedding of [START_REF] Godin | Improving and Interpreting Neural Networks for Word-Level Prediction Tasks in Natural Language Processing[END_REF]. This model is trained on a large tweet corpus (embedding dimension is 400). In our systems, each dense layer contains 256 neurons.

For each system configuration, we train 9 models with different random initialization. We select the model that obtains the best result on the development set to make predictions on the test set.

MWE statistics

We first analyze the distribution of the MWEs in our corpora. We observe that about 25% of the HatEval training tweets contain at least one MWE and so the presence of MWE can influence the HSD performance.

As a further investigation, we analyze MWEs appearing per MWE category and for hate/nonhate classes. In the training set of the HatEval corpus our parser, described in section 2, annotated 4257 MWEs. Table 1 shows MWEs that appear only in hateful or non-hateful tweets or both in HatEval training part. We observe that some MWE categories, as symbol and interjection, do not appear in HatEval training set. We decided to not use these two categories in our experiments. Most of the categories appear in hateful and non-hateful tweets. For the majority of MWE categories, there are MWEs that occur only in hateful speech and MWEs that occur only in non-hateful tweets.

Finally, we analyze the statistics of each MWE category for hate and non-hate classes. As in HatEval the classes are almost balanced, there is no bias due to imbalanced classes.

We observe that there are no MWE categories used only in the hateful speech or only in the non-hateful speech excepted for the cause light verb construction category, but this category is underrepresented. We note that there is a difference between the use of MWEs in the hateful and the non-hateful tweets: MWEs are used more often in non-hateful speech. These observations reinforce our idea that MWE features can be useful for hate speech detection.

Experimental results

The goal of our experiments is to study the impact of MWEs on automatic hate speech detection for two different corpora: HatEval and Founta. We carried out experiments with the different groups of MWE categories: MWEall, including all MWE categories, and the combination of VMWE5 and MWE5.

Table 2 displays the macro-F1 on HatEval and Founta test sets. Our baseline system without MWE features, called USE in Table 2, achieves a 65.7% macro-F1 score on HatEval test set. Using MWE features with word2vec or BERT embeddings, the system proposed in this paper performs better than the baseline. For instance, on HatEval, MWEall with BERT embedding configuration achieves the best result with 66.8% of macro-F1. Regarding Founta corpus, we observe a similar result improvement: the baseline system achieves 72.2% and systems with MWE features obtain scores ranging from 72.4% to 73.0% of macro-F1. It is important to note that according to a matched pair test in terms of accuracy with 5% risk [START_REF] Gillick | Some statistical issues in the comparison of speech recognition algorithms[END_REF], the systems using MWE features and word2vec or BERT embeddings significantly outperform the baseline system on the two corpora. Finally, the proposed system with MWEall and BERT embedding for HatEval outperforms the state-of-the-art system FERMI submitted at HatEval competition (SemEval task 5): 66.8% for our system versus 65% for FERMI of macro-F1 [START_REF] Indurthi | FERMI at SemEval-2019 task 5: Using sentence embeddings to identify hate speech against immigrants and women in Twitter[END_REF].

To analyze further MWE features, we experiment with different groups of MWE categories: VMWE5, MWE5, and MWEall. Preliminary experiments with the two-branch system with USE and word embeddings branches only gave a marginal improvement compared to the baseline system. Using the three-branch neural network with only VMWE5 or MWE5 instead of MWEall seems to be interesting only for word2vec embedding. With BERT embedding it is better to use MWEall categories. Finally, the use of all MWEs could be helpful rather than the use of a subgroup of MWE categories. Comparing word2vec and BERT embeddings, dynamic word embedding performs slightly better than the static one, however, the difference is not significant.

We compare the confusion matrices of two systems: the baseline system and the proposed one with MWEall and BERT embeddings. On the HatEval, the proposed system classifies better non-hateful tweets than the baseline system. In contrast, on Founta our system is more accurate to classify hateful tweets. We think that the balance between the classes plays an important role: in the case of HatEval corpus, the classes are balanced, in the case of Founta, the classes are unbalanced.

To perform a deeper analysis, we focus our observations on only the tweets from the test sets containing at least one MWE: 758 tweets from the HatEval test set and 3508 tweets from the Founta test set. Indeed, according to section 4, there is about 25% of tweets containing MWEs. The second part of table 2 shows that the results are consistent with those observed previously in this section, and the obtained improvement is more important.

Conclusion

In this work, we explored a new way to design a HSD system for short texts, like tweets. We proposed to add new features to our DNN-based detection system: mutliword expression features. We integrated MWE features in a USE-based neural network thanks to a neural network of three branches. The results were validated on two tweet corpora: HatEval and Founta. The models we proposed yielded significant improvements in macro-F1 over the baseline system (USE system). Furthermore, on HatEval corpus, the proposed system with MWEall categories and BERT embedding significantly outperformed the state-of-the-art system FERMI ranked first at the SemEval2019 shared task 5. These results showed that MWE features allow to enrich our baseline system. The proposed approach can be adapted to other NLP tasks, like sentiment analysis or automatic translation.

Fig. 1 .

 1 Fig. 1. Proposed hate speech detection system using USE and MWE features.

Table 1 .

 1 MWE categories with examples from STREUSLE corpus[START_REF] Schneider | A corpus and model integrating multiword expressions and supersenses[END_REF] and the number of occurrences of MWEs. The train set of HatEval. The column Hateful (Non-hateful) represents MWE occurences that appear only in hateful (non-hateful) tweets. The column Both represents MWE occurrences that appear in hateful and non-hateful tweets.

		MWE categories	Examples Hateful Non-hateful Both
		Adjective	dead on	9	8	255
	MWE5	Adverb Discourse Nominal	once again thank you tax payer	1 12 25	5 15 36	194 401 189
		Adposition phrase (idiomatic) on the phone	9	36	134
	VMWE5	Inherently adpositional verb Full light verb construction Verbal idioms Full verb-particle construction	stand for have option Give a crap take off	11 9 14 11	21 10 24 20	447 36 384 387
		Semi verb-particle construction walk out	6	18	153
		Auxiliary	be suppose to	4	0	475
		Coordinating conjunction	and yet	1	0	8
		Determiner	a lot	1	2	242
		Infinitive marker	to eat	0	0	12
		Adposition	apart from	3	13	573
		Non-possessive pronoun	my self	0	3	11
		Subordinating conjunction	even if	0	0	28
		Cause light verb construction	give liberty	1	0	0
		Symbol	A+	0	0	0
		Interjection	lo and behold	0	0	0
	use standard corpus partition in training, development, and test set with 9k, 1k, and 3k tweets
	respectively. Each set contains around 42% of hateful tweets. The vocabulary size of the corpus
	is 66k words.

  1. VMWE5 is composed of Verbal MWE categories and MWE5 with the rest of the categories. The training part of the HatEval corpus contains 1551 occurrences of VMWE5 and 1329 occurrences of MWE5. During our experiments, we experiment with all MWE categories presented in Table 1 (containing 19 categories: 18 categories, and a special category for words not belonging to any MWE) and with the combination of VMWE5 and MWE5 (10 MWE categories and a special category).

Table 2 .

 2 The first part represents F1 and macro-F1 scores (%) on HatEval and Founta test sets. The second part represents F1 and macro-F1 scores (%) on tweets containing at least one MWE in HatEval and Founta test sets.

			HatEval	Founta
	Features	F1 Hateful Non-hate	Macro-F1	F1 Norm Abus Hate	Macro-F1
			All test set
	USE	64.9	66.4	65.7	94.2 87.8 34.6	72.2
	USE, MWEall, word2vec	64.5	68.2	66.3	93.8 86.9 36.5	72.4
	USE, VMWE5, MWE5, word2vec 66.1	67.0	66.5	93.9 87.1 37.2	72.7
	USE, MWEall, BERT	64.2	69.4	66.8	94.0 87.1 37.5	72.9
	USE, VMWE5, MWE5, BERT	64.8	68.2	66.5	93.8 86.9 38.2	73.0
	Tweets containing at least one MWE
	USE	67.8	62.3	65.0	91.1 94.1 41.6	75.6
	USE, MWEall, word2vec	71.7	61.4	66.6	91.4 86.9 44.6	76.5
	USE, MWEall, BERT	73.9	61.3	67.6	90.9 94 43.3	76.1

https://github.com/msang/hateval/tree/master/SemEval2019-Task5/evaluation

https://github.com/zamp13/MWE-HSD

https://tfhub.dev/google/universal-sentence-encoder-large/3


Aladine Chetouani

Maurice Quach

Giuseppe Valenzise

Frédéric Dufaux

COMBINATION OF DEEP LEARNING-BASED AND HANDCRAFTED FEATURES FOR BLIND IMAGE QUALITY ASSESSMENT

Keywords: Image Quality, Deep learning-based Features, Handcrafted Features, Bilinear Pooling

This last decade, a plethora of handcrafted-based image quality metrics has been proposed in the literature. Some of them are based on structural analysis, while some others exploit mutual information or perceptual characteristics. Nowadays, deep learning-based methods are widely used in several domains due to its ability to well fit the target directly from the image. In this paper, we study the impact on the performance of combining handcrafted and Deep Learning-based (DL) features, since each of them extracts specific information. Indeed, DL-based image quality assessment methods often extract local information by extracting small patches, while the handcrafted ones provide global information through a global analysis. We analyzed the performance before and after combining the two using bilinear pooling strategy. Experimental results on commonly used datasets show the relevance of combining both approaches.

INTRODUCTION

Quality of multimedia contents is a sensitive notion in several applications such as computer vision where the performance depends on the quality of the data, medical imaging where the region-of-interest must be not affected by any distortion, biometrics where the data collected must allow to well identify the person, etc. To answer this request, numerous studies were carried out in order to propose efficient metrics that ensure a high quality of experience. Existing metrics can estimate the quality with reference (FR), without reference (NR) or by extracting only some information from the reference image (RR). In this article, we focus on the NR approach since the latter is more applicable in most real applications.

Traditional NR metrics employ handcrafted features to characterize the distortion. For instance, the perceptual quality of blurred images is often estimated by analyzing edges [START_REF] Marziliano | A no-reference perceptual blur metric[END_REF], while the perceptual impact of block-based processing such as JPEG compression is often measured by analyzing Funded by the ANR ReVeRy national fund (REVERY ANR-17-CE23-0020).

the neighborhood of blocks [START_REF] Wang | No-reference perceptual quality assessment of jpeg compressed images[END_REF]. This kind of metrics is dedicated to specific distortions that limits their use. To overcome this limitation, multi-degradation metrics have been then proposed. The latter are used to estimate the quality of any distorted image (blocking, blurring, ringing, noise, combined degradations, etc.). These metrics are generally based on two steps: feature extraction and feature combination through a machine learning algorithm such as Artificial Neural Networks (ANN) [START_REF] Chetouani | A free reference image quality measure using neural networks[END_REF] or Support Vector Machine [START_REF] Moorthy | A modular framework for constructing blind universal quality indices[END_REF].

Nowadays, a large part of the published work focuses on Deep Learning-based methods. This is mainly due to their performance and their ability to automatically learn features from the raw data. In [START_REF] Kang | Convolutional neural networks for no-reference image quality assessment[END_REF], the authors used their own CNN model to estimate the quality of 2D images. The idea is quite simple and provides promising results. An RGB color image is first decomposed into patches of size 32 × 32 × 3. After a local normalization, these patches are used as inputs to a CNN model and the overall quality score is computed by averaging the predicted patch scores. In [START_REF] Kang | Simultaneous estimation of image quality and distortion via multi-task convolutional neural networks[END_REF], the method was extended to predict the quality and degradation type simultaneously. The two previous methods use the subjective score of the whole image, often called Mean Opinion Score (MOS), as the target (i.e. desired output) of each patch without considering additional information from the whole image. The authors assumed that the patches extracted from a given image have the same perceptual impact which is not totally in accordance with our Human Visual System (HVS). In order to overcome this assumption, some studies proposed to consider perceptual characteristics. In [START_REF] Jia | Saliency-based deep convolutional neural network for no-reference image quality assessment[END_REF], the authors considered rather saliency information as patch selector. In [START_REF] Chetouani | A blind image quality metric using a selection of relevant patches based on convolutional neural network[END_REF][START_REF] Chetouani | On the use of a scanpath predictor and convolutional neural network for blind image quality assessment[END_REF], the author considered rather scanpath. In [START_REF] Abouelaziz | Convolutional neural network for blind mesh visual quality assessment using 3d visual saliency[END_REF][START_REF] Abouelaziz | Noreference mesh visual quality assessment via ensemble of convolutional neural networks and compact multi-linear pooling[END_REF][START_REF] Abouelaziz | 3D visual saliency and convolutional neural network for blind mesh quality assessment[END_REF], a similar approach was applied for quality assessment of 3D meshes.

In this work, we study the impact on the performance of combining both approaches (i.e. handcrafted and Deep Learning-based (DL) features), since each of them extracts specific information. Indeed, DL-based image quality assessment methods often extract local information, while the handcrafted features provide global information. The idea is to compensate the lack of these patch-based methods by integrating global information from the whole image. To this end, we first extract DL-based feature vectors for each patch of the image as well as a handcrafted feature vector from the whole image. Then, we combine those vectors through the Bilinear Pooling (BP) strategy. The resulting vectors are finally fed to a CNN model to predict the quality. Our method was evaluated on well known databases and the results showed the improvement given by the proposed strategy. It is worth noting that in [START_REF] Chetouani | Image quality assessment without reference by mixing deep learning-based features[END_REF], the author studied the impact of combining DL-based features.

Our paper is organized as follows: In Section 2, we present the proposed method by describing the CNN model and the handcrafted features used. In Section 3, we show the results obtained in terms of correlation with the subjective judgments and we discuss the impact of the combination step. The last section is dedicated to the conclusion.

PROPOSED METHOD

The pipeline of the proposed method is summarized in Fig. 1. For a given image I, we first extract handcrafted features V H (I) from the whole image and DL-based feature vectors V DL (I, p) for each patch p of the image I (i.e. one V DL (I, p) per patch). The extracted vectors are then pooled using BP strategy and the resulting vectors are fed to Fully Connected (FC) layers. The overall quality index is finally given by averaging the predicted patch scores.

Handcrafted features

A lot of handcrafted features has been used to estimate the quality. Some of them are based on structural information [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF], while some others exploit degradation types [START_REF] Moorthy | A modular framework for constructing blind universal quality indices[END_REF] or even statistical information [START_REF] Mittal | No-reference image quality assessment in the spatial domain[END_REF]. In this study, we focus more on statistical features, since damage caused by distortions imply an alteration of the statistics of the image. Here, we employ statistical features based on Gradient Magnitude (GM) and Laplacian of Gaussian (LOG) that have been widely used in computer vision for image matching and were efficiently used for image quality in [START_REF] Xue | Blind image quality assessment using joint statistics of gradient magnitude and laplacian features[END_REF]. More precisely, we employed the marginal probability distributions of GM and LOG as well as their independency measures. The former allow to get information about the statistical distribution of the image, while the latter give information about the statistical interaction between GM and LOG. All these features are computed after applying a joint adaptive normalization that aims to remove content-dependent characteristics. For more details, the reader is referred to [START_REF] Xue | Blind image quality assessment using joint statistics of gradient magnitude and laplacian features[END_REF] where the authors well demonstrated the impact on the performance of each set of features. At the end, 10 features are extracted for each statistical set. So, our handcrafted feature vector V H is constituted of 40 features, extracted from the whole image.

Deep Learning-based features

Several CNN models have been employed in the literature with different strategies (i.e. from scratch, transfer learning or fine-tuning). In this work, DL-based features are extracted from a modified version of VGG that was fine-tuned [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. This model, initially pre-trained on ImageNet [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF], was proposed in 2014 by the Oxford Visual Geometry Group. Its initial input is an image of size 224x224x3, while its output is composed of 1000 classes. For a given image, P feature vectors of size 128 are thus extracted from the image I (i.e [V DL (I, 1), V DL (I, 2), ..., V DL (I, P )]).

Bilinear Pooling

To combine the handcrafted and DL-based features, Bilinear Pooling (BP) is applied [START_REF] Lin | Bilinear cnn models for fine-grained visual recognition[END_REF]. It allows a multiplicative interaction between all features of the two vectors and thus improves the interaction between the local (i.e. DL-based) and global (i.e. handcrafted) features. It consists of applying the outer product of two vectors as follows:

V BP I,p (V DL , V H ) = V DL (I, p) ⊗ V H (I) (1)
where V DL (I, p) is the DL-based feature vector of the patch p of the image I. V H (I) is the handcrafted feature vector extracted from the image I. ⊗ represents the outer product.

Before combining the vectors, we projected the DL-based features into a lower dimensional space. This makes it possible to reduce the number of learnable parameters and above all, to have a balanced representation of the features (i.e. handcrafted and DL-based). For that, the well-known Count Sketch (CS) projection function is used to obtain a vector V P DL of size D from a vector V of size N with D < N [START_REF] Pham | Fast and scalable polynomial kernels via explicit feature maps[END_REF]. Fig. 2 summarizes this procedure. The resulting matrix is of size D × 40 that is then vectorized and fed to three FC layers. The first FC layer is followed by a ReLu layer and a dropout layer, while the second FC layer is followed by only a ReLu layer. The third FC layer is a logistic regression layer with one output (i.e. predicted MOS). The model was trained through the Stochastic Gradient Descent (SGD) optimization method using the MSE as loss function. The learning rate and momentum were set to 0.1 and 0.9, respectively. The batch size was fixed to 32 and the number of epochs was equal to 20. The model was saved after each epoch and the training set was shuffled. The model that provided the best result was retained. The overall quality index was finally obtained by averaging all the predicted patch scores.

Datasets

Our experiments were carried-out on four well-known datasets:

• LIVE -Phase 2 (LIVE2-P2) [START_REF] Sheikh | Live image quality assessment database release 2[END_REF]: LIVE-P2 dataset is one of the first publicly available datasets and is still commonly used to evaluate existing It contains 29 pristine images from which 982 degraded images were derived. Five different degradation types were considered: JPEG2000, JPEG, White Noise, Gaussian Blur and Fast Fading. The DMOS (Differential Mean Opinion Score) was used to annotate the degraded images (0 indicates the best quality, while 100 indicates the worst quality).

• TID 2008 (TID08) [START_REF] Ponomarenko | Tid2008-a database for evaluation of full-reference visual quality assessment metrics[END_REF]: Constituted of 17 degradation types, TID08 dataset consists of 1700 degraded images obtained from 25 pristine images (i.e. 100 degraded images per degradation) and their corresponding MOS (Mean Opinion Score). The latter varies between 0 and 9 where 0 denotes the worst quality and 9 the best quality.

• TID 2013 (TID13) [START_REF] Ponomarenko | Image database tid2013: Peculiarities, results and perspectives[END_REF]: TID13 dataset is an extended version of the previous one. More degradation types were considered (24 instead of 17) with more degraded images per degradation type (125 instead of 100). A total of 3000 degraded images and their corresponding MOS are provided.

• CSIQ [START_REF] Larson | Most apparent distortion: full-reference image quality assessment and the role of strategy[END_REF]: This dataset is composed of 866 degraded images achieved from 30 pristine images and 6 degradation types (JPEG2000, JPEG, White Noise, Contrast, Gaussian Blur and Pink Gaussian Noise). The normalized DMOS are given for each degraded image. This value varies between 0 and 1, where 0 corresponds to the best quality and 1 the worst quality.

EXPERIMENTAL RESULTS

Evaluation Criteria

The performance evaluation is carried out by computing the Pearson (PCC) and Spearman (SROCC) correlation coefficients. These values are calculated between subjective scores and the corresponding predicted ones. A correlation of 1 means a perfect prediction, while a correlation of 0 indicates no correlation. The predicted scores are mapped to the subjective ratings using the following non-linear logistic function:

Q = β 1 ( 1 2 - 1 e -β2(Qp-β3) ) + β 4 * Q p + β 5 (2)
where Q p and Q are the predicted and the mapped scores, respectively.

Individual Evaluation

Our method is first evaluated by randomly splitting the LIVE-P2 database into 60%, 20% and 20% for the training, validation and test sets, respectively. This procedure is repeated 10 times to ensure the generalization ability of our method and the mean correlation across the 10 iterations is computed and used as performance indicator. We compare the performance of each extracted features individually. Table 1 presents the PCC and SROCC obtained. As expected, DL-based features achieve better results (PCC=0.967) than the handcrafted ones (PCC=0.955). The combination of the two allows an improvement of 1.14% and 2.41% compared to the DL-based and handcrafted features, respectively. Therefore, the integration of global information through the handcrafted features has a benefit impact on the performance. Fig. 3 shows the PCC of the CNN model (red line) for each iteration as well as the PCC obtained after combination of the two features (blue line) across the 10 iterations. As can be seen, the combination allows an improvement for each iteration. The mean, the maximum and the minimum PCC improvements are equal to 0.010, 0.013 and 0.008 respectively.

Fig. 3. PCC of the DL-based features (red line) and the combined ones (blue line).

A one-way ANalysis Of VAriance (ANOVA) test is carried out to show the statistical significance of improvement gain. Fig. 4 presents the boxplot of the predicted and subjective scores. This plot depicts the distribution of our data around their median values. It can be seen that the predicted scores of the combined features and subjective scores have close distributions with similar median and variance values. The predicted scores of the DL-based features has a higher variance. We compute the p-value between the predicted scores using the combined features and the DL-based features. The value is less than the significance level 0.05, which means that the improvement gain is statistically significant. We compare bilinear pooling strategy to some common strategies: Concatenation, Summation and Multiplication. Table 2 shows the correlations obtained for each of them. As can be seen, bilinear pooling outperforms all the compared [START_REF] Bosse | Deep neural networks for no-reference and full-reference image quality assessment[END_REF] 0.980 0.970 --NR-IQA DIIVINE [START_REF] Moorthy | Blind image quality assessment: From natural scene statistics to perceptual quality[END_REF] 0.917 0.916 0.900 0.880 BLIINDS-II [START_REF] Saad | Dct statistics model-based blind image quality assessment[END_REF] 0.930 0.931 0.930 0.910 BRISQUE [START_REF] Mittal | No-reference image quality assessment in the spatial domain[END_REF] 0.942 0.940 0.797 0.756 CORNIA [START_REF] Ye | Unsupervised Feature Learning Framework for No-reference Image Quality Assessment[END_REF] 0.935 0.942 0.781 0.714 IQA-CNN [START_REF] Kang | Convolutional neural networks for no-reference image quality assessment[END_REF] 0.953 0.956 0.954 0.948 IQA-CNN++ [START_REF] Kang | Simultaneous estimation of image quality and distortion via multi-task convolutional neural networks[END_REF] 0.950 0.950 --SOM [START_REF] Zhang | Som: Semantic obviousness metric for image quality assessment[END_REF] 0.962 0.964 --CNN-Prewitt [START_REF] Li | No-reference image quality assessment using prewitt magnitude based on convolutional neural networks[END_REF] 0.966 0.958 --Image-wise CNN [START_REF] Kim | Deep convolutional neural models for picturequality prediction: Challenges and solutions to data-driven image quality assessment[END_REF] 0.963 0.964 0.791 0.812 BIECON [START_REF] Kim | Fully deep blind image quality predictor[END_REF] 0.960 0.958 0.823 0.815 DIQA [START_REF] Kim | Deep cnn-based blind image quality predictor[END_REF] 0.972 0.970 0.880 0.844 DIQa-NR [START_REF] Bosse | Deep neural networks for no-reference and full-reference image quality assessment[END_REF] 0.972 0.960 --Our method 0.978 0.977 0.970 0.949 methods. Concatenation, Summation and Multiplication achieve close results. These results are certainly due to the fact that bilinear pooling allows a multiplicative interaction between all features and thus combines the advantages of the Concatenation and Multiplication strategies. Indeed, Concatenation permits an interaction between all features, while Multiplication permits only a multiplicative interaction element by element.

Comparison with the state-of-the-art

Our method is compared to the state-of-the-art including handcrafted-based FR and NR metrics (PSNR, SSIM [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF], FSIM [START_REF] Zhang | Fsim: A feature similarity index for image quality assessment[END_REF], DIIVINE [START_REF] Moorthy | Blind image quality assessment: From natural scene statistics to perceptual quality[END_REF], BLIINDS-2 [START_REF] Saad | Dct statistics model-based blind image quality assessment[END_REF], BRISQUE [START_REF] Mittal | No-reference image quality assessment in the spatial domain[END_REF] and CORNIA [START_REF] Ye | Unsupervised Feature Learning Framework for No-reference Image Quality Assessment[END_REF]) as well as CNN-based FR and NR metrics (DeepIQA [START_REF] Kim | Deep learning of human visual sensitivity in image quality assessment framework[END_REF], IQA-CNN [START_REF] Kang | Convolutional neural networks for no-reference image quality assessment[END_REF], IQA-CNN+/IQA-CNN++ [START_REF] Kang | Simultaneous estimation of image quality and distortion via multi-task convolutional neural networks[END_REF], SOM [START_REF] Zhang | Som: Semantic obviousness metric for image quality assessment[END_REF], CNN-Prewitt [START_REF] Li | No-reference image quality assessment using prewitt magnitude based on convolutional neural networks[END_REF], BIECON [START_REF] Kim | Fully deep blind image quality predictor[END_REF], DIQA [START_REF] Kim | Deep cnn-based blind image quality predictor[END_REF], DIQa-FR & DIQa-NR [START_REF] Bosse | Deep neural networks for no-reference and full-reference image quality assessment[END_REF] and Image-wise CNN [START_REF] Kim | Deep convolutional neural models for picturequality prediction: Challenges and solutions to data-driven image quality assessment[END_REF]). Two well-known databases are used (LIVE-P2 and CSIQ). The protocol described in Section 3.2 is applied on both databases. Table 3 shows the results obtained. The first three best methods are marked in bold and CNN-based methods are in italic. The overall best result is represented with a grey background. On LIVE-P2 database, our method achieves the third best result outperforming all the compared NR methods and being competitive to CNN-based FR methods. DeepIQA and DIQa-FR obtain the best results on this database with a PCC equal to 0.981 and 0.980, respectively. However, the latter necessitate the pristine image (FR-IQA).

In addition, CNN-based methods outperform all the handcrafted metrics for both approaches (FR-IQA and NR-IQA), which confirms the contribution of such methods. On CSIQ database, our method achieves the best result with high correlation (PCC=0.970). The second best result is obtained by a CNN-based method (IQA-CNN). The latter use only patches of the whole image to predict the quality and thus not integrate information from the whole image.

Cross-database Evaluation

Finally, we evaluate the generalization ability of our method to predict the quality whatever the dataset. For that, we apply a cross-database evaluation by training our method on LIVE-P2 dataset and testing it on CSIQ, TID08 and TID13 datasets. Table 4 shows the correlations for the three datasets. As can be seen, our method predicts well the quality for all datasets. The best result is obtained for CSIQ (0.896), but still close to those obtained for TID13 (0.894).

CONCLUSION

In this paper, we studied the impact of combining handcrafted and DL-based features. For that, bilinear pooling was employed after decreasing the size of the DL-based features through a Count Sketch (CS) projection. We compared the performance of both kinds of features individually and after their combination. We also evaluated the impact of the CS projection on the performance as well as the statistical differences between the predicted scores of the DL-based features and those obtained after combination. The best configuration was retained and compared to the state-of-the-art. Experimental results showed the contribution of handcrafted features.

As future work, we plan to study other handcrafted features (i.e. quality indexes) as well as other pooling strategies using a feature selection method. We will also try to improve the generalization ability of our method, since the performances obtained are not high as expected.

Fig. 1 .

 1 Fig. 1. Flowchart of the proposed method.

Fig. 2 .

 2 Fig. 2. Bilinear Pooling applied between the handcrafted and DL-based features.

Fig. 4 .

 4 Fig. 4. Boxplot for Anova test of the subjective scores and the predicted scores obtained by the CNN and those obtained by the combination of both kinds of features.

Table 1 .

 1 Mean PCC and SROCC of the handcrafted and DLbased features on the LIVE-P2. The best results are represented in bold.

	Features	PCC SROCC
	Handcrafted	0.955	0.951
	DL	0.967	0.971
	BP(DL,Handcrafted) 0.978	0.977

Table 2 .

 2 Comparison of some pooling strategies using the LIVE-P2 dataset.

	Pooling strategy PCC SROCC
	Concatenation	0.970	0.969
	Summation	0.967	0.971
	Multiplication	0.970	0.971
	Bilinear Pooling 0.978	0.977

Table 3 .

 3 Performance comparison on LIVE-P2 and CSIQ datasets. Mean PCC and SROCC of each dataset across 10 random splitting. The first three best methods are marked in bold and DL-based methods are in italic. The overall best result is with a grey background.

	LIVE-P2	CSIQ
	PCC SROCC PCC SROCC

Table 4 .

 4 Cross-dataset evaluation. The model was trained on LIVE-P2 dataset and tested on CSIQ, TID08 and TID13 datasets.

		PCC SROCC
	CSIQ	0.896	0.878
	TID08 0.881	0.863
	TID13 0.894	0.873