Madona Khoury
email: madonakhoury@hotmail.com

Marc El Beyrouthy

Naïm Ouaini

Véronique Eparvier

Didier Stien
email: didier.stien@cnrs.fr.

Hirtellina lobelii DC. essential oil, its constituents, its combination with antimicrobial drugs and its mode of action

Keywords: essential oil, antimicrobial activity, synergy, scanning electronic microscopy, Hirtellina lobelii, α-bisabolol

With the goal of unravelling antimicrobial agents and mixtures inspired by plant defences, we investigated the antibacterial and antifungal efficacy of Hirtellina lobelii DC. essential oil (EO), both alone and in combination with antimicrobial drugs.

Hirtellina lobelii DC. EO was analysed by GC, GC-MS and partial fractionation/NMR. It was essentially composed of oxygenated sesquiterpenes (75.2%), with α-bisabolol (34.5%), fokienol (12.0%) and T-muurolol (6.8%) serving as the main components. Microbial susceptibility was determined by the broth microdilution method and was expressed as minimum inhibitory concentration (MIC) and minimum bactericidal or fungicidal concentration (MBC or MFC). This EO was found to possess remarkable bactericidal (MBC/MIC = 2) and fungicidal (MFC/MIC = 1 -4) potential, particularly against the Gram (+) bacteria Staphylococcus aureus, including its methicillin-resistant forms, the yeast Cryptococcus neoformans and dermatophytes from the genus Trichophyton (MICs 8 -8 μg/ml). The examination of the combined effects of the EO with antimicrobial drugs revealed synergisms of the EO with vancomycin against S. aureus and of the EO with fluconazole and griseofulvin against dermatophytic fungi (FICI 0.2 -0.5). The effect of H. lobelii EO on the morphologies of fungal hyphae and bacteria, as determined by scanning electronic microscopy (SEM), showed fungal hyphae swelling and bulging.

These results suggest that H. lobelii EO and its major constituent, α-bisabolol, have remarkable antimicrobial potential. Combination therapies of this EO with antifungal drugs could offer a promising alternative for treatment of human mycoses caused by filamentous dermatophytic fungi.

Introduction

Antimicrobial resistance has become a major therapeutic challenge, as a variety of multiresistant pathogenic microbes have emerged that defy commonly available treatments.

The decline in effectiveness of existing drugs is partly due to natural selection and partly to their intensive use and misuse. As a matter of concern, the number of immunocompromised patients, who frequently develop opportunistic systemic infections, has dramatically increased since the 1990s [START_REF] Linden | Approach to the immunocompromised host with infection in the intensive care unit[END_REF]. Thus, infectious diseases are considered a real global threat, representing 26% of overall mortality in 2001 [START_REF] Becker | Infectious diseases -a global challenge[END_REF] and almost 30% in 2011 [3].

The development of microbial resistance to conventional treatments along with drug-related toxicities and costs have generated a clear need for new therapeutic strategies, new antimicrobial compounds and, particularly, new combination treatments and active mixtures.

Combinatorial therapies can be less vulnerable to the development of drug resistance and can increase therapeutic efficacy [START_REF] Groll | Antifungal chemotherapy: Advances and perspectives[END_REF][START_REF] Zimmermann | Multi-target therapeutics: when the whole is greater than the sum of the parts[END_REF]. Their potential synergistic effects provide broader pharmacological windows and lower toxicities [START_REF] Fitzgerald | Systems biology and combination therapy in the quest for clinical efficacy[END_REF]. Thus, efforts toward discovering multi-agent therapies that can overcome the limitations of monotherapies are highly encouraged [START_REF] Zimmermann | Multi-target therapeutics: when the whole is greater than the sum of the parts[END_REF][START_REF] Fitzgerald | Systems biology and combination therapy in the quest for clinical efficacy[END_REF][START_REF] Borisy | Systematic discovery of multicomponent therapeutics[END_REF][START_REF] Onyewu | Unique applications of novel antifungal drug combinations[END_REF].

In the quest for new antimicrobial drugs and bio-inspired mixtures, medicinal plants must not be overlooked. The antimicrobial potential of aromatic plants has been recognized since antiquity and is mainly attributed to their volatile oils, which contain wide chemical diversity [START_REF] Bakkali | Biological effects of essential oils -a review[END_REF][START_REF] Edris | Pharmaceutical and therapeutic potentials of essential oils and their individual volatile constituents: a review[END_REF]. Essential oils (EOs) are known for their antiseptic, i.e., bactericidal, fungicidal and virucidal, properties, and some of them have been claimed to cure microbial infections and have been proposed for use in complementary medicine [START_REF] Stea | Essential oils for complementary treatment of surgical patients: state of the art, Evidence-Based Complement[END_REF]. Thus, EOs could transition from being used solely in traditional medicine to also being used in modern medicine.

Although many studies have focused on showing the antimicrobial activity of EOs, few have investigated the origin of their bioactivities and their exact mechanisms of action by examining their effects on the morphologies and ultrastructures of pathogenic strains.

Hirtellina lobelii DC., formerly known as Staehelina lobelii DC., is an herbaceous plant belonging to the Asteraceae (or Compositae) family. The genus Staehelina (tribe Cardueae) is an extremely small genus that consists of only a few species worldwide and has rarely been documented. The ascription of this genus is highly problematic, and the subtribal placement of Staehelina remains unresolved. Based on Dittrich [START_REF] Dittrich | Die bedeutung morphologi -f systematik der tribus Echinopeae Cass. und Carlineae Cass[END_REF], the two species of Staehelina with hirsute pericarps (S. fruticosa L. and S. lobelii DC.) should be classified in a distinct genus, Hirtellina. We followed the classification proposed by Dittrich, which is also accepted by most databases such as Euro+Med PlantBase (http://ww2.bgbm.org/EuroPlusMed) and the plant list (http://www.theplantlist.org).

Hirtellina lobelii grows on rock crevices in large clumps and is a locally important element of Mediterranean chasmophytic vegetation. Its distribution is restricted to Lebanon, Cyprus, Syria and Asiatic Turkey [13, Euro+Med PlantBase (http://ww2.bgbm.org/EuroPlusMed)].

We evaluated the chemical composition of the EO of H. lobelii, its antimicrobial potential alone and in combination with antibiotics and antifungal drugs as well as its mechanism of action by observing the morphological alterations to pathogen structures that it caused by scanning electron microscopy (SEM). To the best of our knowledge, this is the first description of the chemical composition and the antimicrobial potential of Hirtellina lobelii DC. EO. In addition, no volatile organic compound has ever been described from species of the Hirtellina or Staehelina genera.

Experimental part

Plant material and essential oil extraction

The aerial part of the plant (fresh leaves and stems) was collected in June 2012 from Qartaba, Mount Lebanon (34°05'58.70" N 35°48'46.39" E) at an altitude of 1250 m. A voucher specimen was deposited at the Herbarium of the Department of Botany and Medicinal plants, Holy Spirit University, Faculty of Agricultural and Food Sciences (USEK-Lebanon) under the registry number MNV446a. Hydrodistillation of the plant was performed for 3 h using a Clevenger-type apparatus according to the European Pharmacopoeia, 1997. The EO was obtained with a yield of 0.1%.

Essential oils analyses

GC analyses

Analytical gas chromatography was performed using a Thermo Electron Corporation gas chromatograph fitted with a DB-5 MS capillary column (30 m × 0.25 mm, 0.1 µm film thickness) or a fused silica HP Innowax polyethylene glycol capillary column (50 m × 0.20 mm, 0.20 µm film thickness). Helium was the carrier gas (0.7 ml/min). The column temperature was initially set to 6 35 °C and was gradually increased to 85 °C at 5 °C/min. It was held at 85 °C for 20 min and then raised to 300 °C at 10 °C/min. Finally, it was held at 300 °C for 5 min. Diluted 1 µl samples (1/100, vol/vol) were manually injected at 250 °C in the splitless mode. Flame ionisation detection (FID) was performed at 310 °C.

GC/MS analyses

The GC/MS analyses were performed using an Agilent gas chromatograph 6890 coupled with a Mass Detector 5975. The 7683 B autosampler injected 1 µL of each oil sample. A fused silica capillary column DB-5 MS (30 m × 0.25 mm internal diameter, 0.1 µm film thickness) or a fused silica HP Innowax polyethylene glycol capillary column (50 m × 0.20 mm, 0.20 µm film thickness) was used. Helium was the carrier gas (0.7 ml/min). The oven temperature program was identical to that described in 2.2.1. The mass spectra were recorded at 70 eV with an ion source temperature of 310 °C and a transfer line heated to 320 °C. The acquisition was recorded in full scan mode (50 -400 amu).

Identifications and quantifications

Most constituents were identified by GC and GC/MS by comparing their retention indices (RI) with those from the literature [START_REF] Jennings | Qualitative analysis of flavor and fragrance volatiles by glass capillary gas chromatography[END_REF][START_REF] Davies | Gas chromatographic retention indices of monoterpenes and sesquiterpenes on methyl silicon and Carbowax 20M phases[END_REF] or those of authentic compounds obtained from Sigma-Aldrich (Lebanon). The retention indices were determined relative to a homologous series of nalkanes (C8 to C24) that had been analysed under the same operating conditions. Their mass spectra using both columns were compared with those provided in the NIST and Wiley 275 libraries, our home-made library constructed with pure compounds and EOs of known compositions or mass spectra from the literature [START_REF] Jennings | Qualitative analysis of flavor and fragrance volatiles by glass capillary gas chromatography[END_REF][START_REF] Adams | Identification of essential components by gas chromatography/mass spectroscopy[END_REF]. The relative concentrations of the components were calculated based on the GC peak areas without correction and are reported in Table 1.

Essential oil fractionation by HPLC

Analytical and preparative HPLCs were conducted using a Gilson system equipped with a 322 pumping device, GX-271 fraction collector, 171 diode array detector, and ELSII preparative electrospray nebulizer detector. Phenomenex Luna C18 columns of two sizes were used for these experiments: a 4.6 × 250 mm column with μ film thickness for analytical HPLC and a 21.2 × 250 mm column with μ film thickness for preparative HPLC. The flow rates were set to 1 and 21 ml/min for analytical and preparative HPLC, respectively, using a linear gradient of water mixed with an increasing proportion of acetonitrile (30/70 to 0/100 over 35 min and then 100% CH 3 CN for 24 min). The EO was diluted in acetonitrile at 10 mg/ml for analytical HPLC. The EO was also diluted in acetonitrile for prep-HPLC (60 mg EO in 300 µL CH 3 CN), and 250 µL of the diluted solution was injected.

Forty-nine 25 ml fractions were collected between 2 and 59 min and were combined into 22 fractions according to their HPLC profiles. These 22 fractions were analysed by GC/MS. Fraction 23/24 contained 4 compounds, among which the unidentified component was the major one and accounted for 47% of the mixture according to GC/MS integration. The volatile organic compounds were collected from the fraction as follows: the fraction was separated between ether (250 ml) and water (250 ml). The organic layer was washed with water (3 × 50 ml), dried with Na 2 SO 4 , and evaporated without heating.

NMR spectroscopy

The nuclear magnetic resonance (NMR) spectra ( 1 H-NMR, 13 C-NMR, 1 H-1 H COSY, HSQC and HMBC) of fraction 23/24 allowed us to confirm the identification of one of the main EO constituents (fokienol, 2, Fig. 1) by comparison with NMR data reported in the literature [START_REF] Pascual-T | Nerolidol-5,8-oxides from the essential oil of santolina oblongifolia[END_REF][START_REF] Stoessl | The structures of some stress metabolites from Solanum melongena[END_REF]

Antimicrobial activity

Microorganisms

The antimicrobial activity of the EOs against the following microbial strains was investigated: fumigatus SNB-AF1).

Microdilution method

The broth microdilution method was used to determine the minimal inhibitory concentration (MIC) of the EOs according to the Clinical and Laboratory Standards Institute guidelines [START_REF] Clsi | Methods for dilution antimicrobial susceptibility tests for bacteria that grow aerobically[END_REF][START_REF] Clsi | Reference method for broth dilution antifungal susceptibility testing of yeasts; third informational supplement[END_REF][START_REF] Clsi | Reference method for broth dilution antifungal susceptibility testing of filamentous fungi; approved standard-second edition[END_REF][START_REF] Stien | Search for antifungal compounds from the wood of durable tropical trees[END_REF].

The essential oil and its major components diluted in DMSO were tested at concentrations ranging from 512 to 1 μg/ml. Oxacillin, vancomycin and gentamicin (16 -0.03 μg/ml) were used as reference antibiotics, while itraconazole (16 -0.03 μg/ml) and fluconazole (64 -0.125 μg/ml) were used as positive controls for antifungal activity. These antimicrobial standard drugs were purchased from Molekula, and pure terpene was purchased from Sigma-Aldrich. The microplates were incubated at 37 °C for 24 h for bacteria, 48 h for yeasts and A. fumigatus, and five days for the other filamentous fungi. The MIC values corresponding to the lowest concentration that prevented visible microbial growth are reported in Table 2.

Minimum bactericidal and fungicidal concentrations (MBCs and MFCs)

After MIC determination, the bactericidal and fungicidal activities of the EO were determined as previously described [START_REF] Clsi | Methods for Determining Bactericidal Activity of Antimicrobial Agents; Approved Guideline Volume 19 Number 18 Methods for Determining Bactericidal Activity of Antimicrobial Agents; Approved Guideline[END_REF][START_REF] Espinel-Ingroff | Testing conditions for determination of minimum fungicidal concentrations of new and established antifungal agents for Aspergillus spp.: NCCLS collaborative study[END_REF][START_REF] Prado | Subculture on potato dextrose agar as a complement to the broth microdilution assay for Malassezia pachydermatis[END_REF]. From each well with no detected microbial growth, 20 μl of the culture medium of bacterial and fungal cultures were subcultured on Mueller-Hinton and Sabouraud dextrose agar plates, respectively; the contents of the wells were not agitated prior to removal of the specified volume. The growth control wells of the microdilution plate were used as the growth control, and oxacillin, itraconazole and fluconazole were used as positive controls. The plates were incubated at 35 °C until growth was observed in the growth control subculture (24 h for bacteria, 48 h for yeasts and 5 days for dermatophytic fungi). The MBC and MFC was defined as the lowest concentration that resulted in no visible bacterial or fungal growth on agar plates, respectively, and those values are reported in Table 2.

Synergy test by Microdilution Checkerboard assay

This test is based on the microdilution method, and the protocol was described by Shin and Lim [START_REF] Shin | Antifungal effects of herbal essential oils alone and in combination with ketoconazole against Trichophyton spp[END_REF] and Houël et al. [START_REF] Houël | In vitro antidermatophytic activity of Otacanthus azureus (Linden) Ronse essential oil alone and in combination with azoles[END_REF]. EO activity was tested in combination with the following drugs against the following microorganisms: vancomycin, S. aureus; fluconazole, C. neoformans; and fluconazole and griseofulvin, dermatophytic fungi Trichophyton sp. The drug solutions were diluted along the horizontal orientation so that the plates contained final concentrations of vancomycin and griseofulvin ranging from 16 to 0.03 μg/ml and final concentrations of fluconazole ranging from 64 to 0.125 μg/ml. The EO solution was diluted along the vertical orientation so that the final concentrations ranged from 64 to 1 μg/ml for bacteria and dermatophytes and from 256 to 4 μg/ml for yeast.

Fractional inhibitory concentrations (FIC), which represent the gain in activity of individual components of the mixture, were determined for each point by dividing the MIC of the combinations of the two products by the MIC of the essential oil or the drug alone. The FIC index (FICI), which is a measure of synergy, was obtained by adding both FICs. The FICI was interpreted as follows: ≤ 0.5 indicated a synergistic effect, values > . ≤ 2.0 indicated an indifferent effect, and values > 2.0 indicated an antagonistic effect [START_REF] Shin | Antifungal effects of herbal essential oils alone and in combination with ketoconazole against Trichophyton spp[END_REF][START_REF] Pyun | Antifungal effects of the volatile oils from Allium plants against Trichophyton species and synergism of the oils with ketoconazole[END_REF].

The results were also analysed by the isobologram method (see supplementary material, S1).

Cytotoxicity Assays

Cytotoxicity assays were conducted with MRC5 (human foetal lung fibroblast) and MDA435

(melanoma) cell lines according to the procedure described by Rochais et al. [START_REF] Rochais | MR22388, a novel anti-cancer agent with a strong FLT-3 ITD kinase affinity[END_REF].

Scanning electron microscopy observations

To examine the effect of H. lobelii EO on the surface topography of S. aureus, T. rubrum and T.

soudanense, samples for SEM observation were prepared as follows. For bacteria, a conventional broth microdilution assay was performed in a 96-well microtitre plate; then, 40 µL of the suspension was sampled and deposited on sterilized glass squares distributed in a 24well polystyrene plate. Sedimentation lasted three hours. Fungi were directly grown on the same system.

Samples treated with a sub-inhibitory concentration were fixed with 2.5% glutaraldehyde in 0.1 M sodium cacodylate buffer at pH 7.4 for 1 hour at RT. The untreated control microorganisms were processed in parallel for the SEM analyses. After three washes of 10 min with the same buffer, the samples were post-fixed with 1% osmium tetroxide in 0.1 M sodium cacodylate buffer at pH 7.4 for 1 hour at RT and then washed in sterile water three times for 10 min. The cells were dehydrated with increasing concentrations of ethanol (50%, 70%, 90%, 3x100%) at RT for 10 min for each bath. The samples were critical-point dried at 75 bar and 37 °C with liquid CO 2 as the transition fluid and were then depressurized slowly (400 cm 3 /min) in a Quorum Technologies K850 device (Elexience, France). Then, the samples were sputtercoated A p w ≈ in a Polaron SC7640 device (Elexience, France) at 10 mA and 0.8 kV for 200 s. Observations were performed using a FE-SEM Hitachi S4500 (Hitachi, Japan) in a high vacuum with a sample holder tilted at 45° and a low SE detector at 2 kV and 21 mm WD. The experiments were performed using the MIMA2 microscopy platform (http://www6.jouy.inra.fr/mima2).

Results and Discussion

Essential oils analyses

The essential oil of the fresh aerial parts of Hirtellina lobelii was obtained by hydrodistillation with a yield of 0.1% (vol/wt, relative to dry weight material). The chemical composition of the EO and the relative proportions of the components are reported in Table 1.

GC, GC-MS, and fractionation/NMR led to the identification of [START_REF] Prado | Subculture on potato dextrose agar as a complement to the broth microdilution assay for Malassezia pachydermatis[END_REF]

Antimicrobial activity

The minimum inhibitory concentrations (MICs) of H. lobelii EO and its main compound, αbisabolol, as well as the minimum bactericidal and fungicidal concentrations (MBCs and MFCs)

of the oil are reported in Table 2. The oil was considered active if the minimal inhibitory concentration was 128 µg/ml or below [START_REF] Cos | Anti-infective potential of natural products: how p 'p f-of-p[END_REF]. Most of the tested pathogens were sensitive to H. lobelii EO. The only resistant strains were the bacterium E. coli, the Candida yeasts, and the clinical isolate A. fumigatus. The EO was very active against the five dermatophytic Trichophyton species, T. rubrum, T. mentagrophyte, T. violaceum, T. soudanense and T.

tonsurans (MIC values ranging from 8 to 64 μg/ml) and against the Gram (+) bacterium S.

aureus with a MIC value of 32 μg/ml. It was also moderately active against the encapsulated yeast C. neoformans with a MIC value of 128 μg/ml. Interestingly, the growth of methicillinresistant S. aureus (MRSA) was also inhibited by the EO, with a MIC value of 128 μg/ml.

Although this value is higher than that for the non-resistant S. aureus, this result supports the use of combination therapies for MRSA infections that include H. lobelii EO.

Table 2. Antimicrobial activity (MIC, MBC or MFC in µg/ml) of Hirtellina lobelii essential oil and its major compound.

Compounds

In the context of this study, it was interesting to discern whether the EO possesses bactericidal and fungicidal properties capable of destroying bacterial and fungal cells or simple growth inhibition effects (bacteriostatic and fungistatic activities). The bactericidal and fungicidal activities of the oil were evaluated on the most sensitive pathogens (MIC < 128 µg/ml) and were compared to those of the positive control drugs, oxacillin, intraconazole and fluconazole (Table 2). The MBC/MIC and MFC/MIC ratios were calculated for each microorganism; compounds are considered bactericidal or fungicidal when the MBC/MIC or MFC/MIC ratio ≤ 4 [START_REF] Meletiadis | Differential fungicidal activities of amphotericin B and voriconazole against Aspergillus species determined by microbroth methodology[END_REF]. H. lobelii EO was found to be bactericidal against S. aureus (MBC/MIC = 2) and fungicidal against all tested Trichophyton spp. (MFC/MIC = 1 to 4). Additionally, the EO was more effective than the positive control antifungal drugs (MFC/MIC = 4 to > 128). Even itraconazole, which inhibited the growth of the dermatophytic fungi at much lower concentrations (MIC 0.25 to <

. μg/ml), was essentially fungistatic. The development of fungicidal therapies is crucial because the prophylactic use of fungistatic drugs has been shown to be associated with an increased frequency of acquired drug resistance in clinical isolates [START_REF] Monk | Outwitting multidrug resistance to antifungals[END_REF]; thus, these results add more value to the H. lobelii EO.

Combined effects of H. lobelii oil and antimicrobial drugs

Based on the above results, it was relevant to evaluate the combined antimicrobial effects of H.

lobelii EO with various antimicrobial drugs. Synergies between EOs and antimicrobial drugs can shorten the duration of therapies, decelerate the emergence of drug resistance and reduce the possible side effects of current therapies by decreasing the necessary doses of the current drugs and EO. The results of the checkerboard assay (FIC and FICI) are reported in Table 3, and the isobolograms are presented in Fig. S1 (Supplementary Material). Most of the tested combinations of the oil and the drugs showed synergistic activity. We observed a synergistic interaction between vancomycin and H. lobelii EO against S. aureus (FICI = 0.5). For the yeast C. neoformans, an additive effect was only detected for the combination of EO and azole drugs (FICI = 1.5). However, the most interesting results of the combination treatments were observed for the dermatophytic fungi. H. lobelii EO was synergistic with fluconazole and griseofulvin against all tested Trichophyton spp. (FICI values from 0.2 to 0.5), with one exception of an indifferent interaction between the EO and fluconazole against T. tonsurans (FICI = 0.6). The greatest synergism was observed for the combination treatment of EO with fluconazole against T. violaceum, with a FICI value of 0.2.

Overall, the combinations tested in this study would allow for a notable decrease in the necessary concentrations of commercial drugs due to a 2-to 8-fold decrease in MIC values.

Likewise, the oil concentration required to inhibit the growth of the pathogens was reduced to very small amounts (MIC C 1 -μg/ml), thus, limiting the potential cytotoxic effect of the EO (Table 3, see also Fig. S1). Indeed, the selectivity indexes (SI) of the synergetic mixtures at the FICI values showed that the antimicrobial activity did not exceed the cytotoxicity (with the exception of T. violaceum). For S. aureus, T. rubrum, T. mentagrophytes and T. soudanense, the selectivity indexes of the combination treatments ranged from 5.3 to 7.5, indicating relatively low cytotoxicities. The selectivity indexes were calculated based on the cytotoxicity measurements of the drugs in the skin cancer cell line MDA435, which is a relevant cell line in the context of the topical application of antimicrobial agents.

The mechanism by which some EOs have synergistic interactions with antimicrobial drugs is not yet clear. It has been postulated that some terpenes may act as solvents for the antimicrobial drug, facilitating its passage across cell membranes [START_REF] Gershenzon | The function of terpene natural products in the natural world[END_REF]. However, H. lobelii EO is strongly antimicrobial on its own. This may indicate a specific mode of action and could explain why combination treatments with this EO are so effective.

Scanning electron microscopy analysis of in vitro effects of H. lobelii EO

Changes in morphology of bacteria and filamentous fungi caused by antimicrobial compounds have often provided insight into their mechanism of action [START_REF] Gunji | Screening of antifungal antibiotics according to activities inducing morphological abnormalities[END_REF]. To investigate the effect of H.

lobelii EO on the morphological characteristics of S. aureus, T. rubrum and T. soudanenese, bacterial and fungal samples treated with sub-inhibitory concentrations (sub-MICs) of the oil were observed by SEM. At sub-MICs, microbial growth is not severely affected while morphological alterations can sometimes be detected [START_REF] Lorian | Antibiotiques à concentrations sudinhibitrices. Effet sur la morphologie et la croissance[END_REF][START_REF] Frost | A whole-cell Candida albicans assay for the detection of inhibitors towards fungal cell wall synthesis and assembly[END_REF].

The morphological characteristics of S. aureus bacteria were not affected by EO treatment. No morphological changes were detectable by SEM (data not shown).

Untreated T. rubrum and T. soudanenese showed typical structures of healthy hyphal elements, including rod-shaped filaments of uniform width with lines of separation (septa) and a smooth surface. Hyphae showed regular branching (fig. 2a,e). In contrast, an unusual pattern of hyphal growth, including alterations in cell shape and size were evident in both T. rubrum and T.

soudanenese hyphae in response to the EO (fig. 2b,c,d,f). The treated hyphae showed aberrant morphologies, such as a loss of linearity with bulging, swollen cells and anomalous branching. A portion of the T. rubrum mycelia seemed to be particularly inflated with a rough surface, probably because they were covered by extruded cell material (fig. 2c). Interestingly, fig. 2d shows a flattened and shrivelled hyphal element with partial distortion and a few small vesicles on the surface. At this time point, hyphal or cytoplasmic debris were dispersed on the 20 surface of the hyphae. These morphological alterations have been previously observed in dermatophyte hyphae treated with terbinafine [START_REF] Ghannoum | Evaluation of the morphological effects of TDT 067 (terbinafine in Transfersome) and conventional terbinafine on dermatophyte hyphae in vitro and in vivo[END_REF]. These different morphological characteristics could give us a glance on the sequence of events during the exposure of fungi to the EO, starting with cell swelling and cellular leakage, leading to the breakage and collapse of hyphal cells. In previous studies, similar cell damaging effects were also observed in hyphae exposed to terpenes or plant extracts [START_REF] Zacchino | Evidence for the mechanism of action of the antifungal phytolaccoside B isolated from Phytolacca tetramera Hauman[END_REF][START_REF] Romagnoli | Chemical characterization and antifungal activity of essential oil of capitula from wild Indian Tagetes patula L[END_REF]. It has been postulated that terpenoids interfere with the phospholipid bilayers of membranes and increase their permeability [START_REF] Vila | Antifungal compounds from plants[END_REF]. Our SEM observations validate the disruption of the fungal membrane in response to H. lobelii EO.

Interestingly, our results with H. lobelii EO do not necessarily correlates with those obtained with closely related antifungal sesquiterpenes. For example, nerolidol did not induced membrane swelling in Trichophyton mentagrophytes [START_REF] Park | Effect of citral, eugenol, α-terpineol on the ultrastructural changes of Trichophyton mentagrophytes[END_REF]. At high concentration (0.11 mg/mL and above), internal vesicles appeared in the cells, ultimately leading to membrane breakdown and abnormal mitochondria structure.

Our results are in agreement with the observed synergetic combinations because alteration in membrane permeability by the EO would facilitate the infiltration of the cell cytoplasm by more hydrophilic molecules, including a wide range of antibiotics and antifungal drugs. However, in this case, H. lobelii EO is active at low concentration, lower than that of many terpenes described in the literature. This might indicate an additional specific interaction of EO constituents against cell membrane components p α-bisabolol which was active on its own.

Conclusions

Our data demonstrated that Hirtellina lobelii EO and its major component, α-bisabolol, have potent antimicrobial activities, as manifested by drastic morphological changes, particularly in dermatophytic fungal cells. Although the mechanism of action of this EO and α-bisabolol against S. aureus remains unresolved, SEM observation on EO-treated fungi revealed alteration of cell membrane permeability, presumably associated with a specific antifungal potential p f α-bisabolol. Altogether, these results demonstrate that both H. lobelii EO and α-bisabolol could be used in antifungal mixtures of either the EO or the terpene in combination with antifungal drugs, such as fluconazole and griseofulvin. These combinations should improve the efficiency of the drugs in local applications, in particular, when the drugs alone are essentially fungistatic. Combination treatments will also reduce the amount of drug used and restrict the use of newly developed drugs to serious clinical cases, therefore, slowing the development of chemoresistant strains. Additional combinations of this EO with other antifungal drugs are worth testing.

  13 C-NMR (125 MHz, CDCl 3 ) δ 16.0 (C-14), 18.5 (C-12), 22.6 (C-5), 27.7 (C-15), 41.8 (C-4), 42.8 (C-8), 73.2 (C-3), 111.6 (C-1), 114.4 (C-13), 125.0 (C-6), 128.5 (C-9), 133.9 (C-10), 134.1(C-7), 141.9 (C-11), 145.0 (C-2).

  Gram (-) bacterial strain, Escherichia coli ATCC 25922; Gram (+) bacterial strains, Staphylococcus aureus ATCC 29213 and methicillin-resistant Staphylococcus aureus (also called oxacillin-resistant Staphylococcus aureus) ATCC 33591; yeasts, Candida albicans ATCC 10231, Candida parapsilosis ATCC 22019 and Cryptococcus neoformans SNB-CN1; and filamentous fungi, Trichophyton rubrum SNB-TR1, Trichophyton violaceum SNB-TV1, Trichophyton soudanense SNB-TS1, Trichophyton tonsurans SNB-TT1, Trichophyton mentagrophytes SNB-TM1 and Aspergillus fumigatus SNB-AF1. The ATCC strains were purchased, while the other strains by Prof. Philippe Loiseau and Christian Bories; the molecular analyses were conducted by BACTUP. The ITS sequences were deposited in the NCBI GenBank database under the following registry numbers: KF360235 (C. neoformans SNB-CN1), KC692746 (T. rubrum SNB-TR1), KF360236 (T. violaceum SNB-TV1), KF360237 (T. soudanense SNB-TS1), KF360238 (T. tonsurans SNB-TT1), KF360239 (T. mentagrophytes SNB-TM1) and KC692747 (A.

8 a

 8 Bacteria: S.a. = Staphylococcus aureus (ATCC 29213), b Yeast: C.n. = Cryptococcus neoformans (SNB-CN1), c Filamentous fungi: T.r. = Trichophyton rubrum (SNB-TR1), T.m. = Trichophyton mentagrophytes (SNB-TM1), T.v. = Trichophyton violaceum (SNB-TV1), T.s. = Trichophyton soudanense (SNB-TS1), T.t. = Trichophyton tonsurans (SNB-TT1). MIC a : MIC of the product alone (in µg/ml); MIC c : MIC of the drug or the essential oil representing the highest synergy or antagonism (in µg/ml); MIC M : MIC of the mixture (MICc EO + MICc drug); FIC: fractional inhibitory concentration; FICI: FIC index; SI: selectivity index (SI = IC 50 /MIC). *H. lobelii EO: itraconazole combinations could not be tested on dermatophytic fungi because itraconazole has very low MIC values. Instead, H. lobelii EO: griseofulvin combinations were tested, given that griseofulvin is frequently prescribed for dermatophytosis.

Fig. 2 .

 2 Fig. 2. Scanning electron micrographs of dermatophytic hyphae (5000x magnification)

  . The NMR spectra were recorded using a Bruker 500 MHz spectrometer equipped with s, d, t, and br standing for singlet, doublet, triplet and broad, respectively. The NMR signals of fokienol as extracted from the NMR analysis of fraction 23/24 are as follows:

	p	.	f δ	p	pp	the TMS
	signal, with

Fokienol (2):

1

H NMR (500 MHz, CDCl 3 ) δ 1.29 (3H, s, H-15), 1.61 (3H, s, H-14), 1.56-1.62 (2H, m, H-4), 1.84 (3H, s, H-12), 2.02-2.08 (2H, m, H-5), 2.76 (2H, br d, J = 7.2 Hz, H-8), 4.89 (2H, br s, H-13), 5.19 (1H, m, H-6), 5.07 (1H, dd, J = 11.0; 1.2 Hz, H-1a), 5.19 (1H, m, H-6), 5.23 (1H, dd, J = 17.1; 1.2 Hz, H-1b), 5.62 (1H, dt, J = 15.9; 7.2 Hz, H-9), 5.93 (1H, dd, J = 17.1; 11.0 Hz, H-2), 6.14 (1H, br d, J = 15.9 Hz, H-10).

Table 1 .

 1 Composition of the essential oil of Hirtellina lobeli Retention index on a HP-5MS column; b Retention index on an MS: identification based on comparison of mass spectra. Co-GC: retention time identical to authentic compounds; NMR: comparison of NMR spectra with those reported in the literature.

	components representing

Fig. 1. Major compounds identified in the H. lobelii EO 248 a

Table 3 .

 3 Combined effects of Hirtellina lobelii essential oil and antimicrobial drugs

									EO			Drug			MRC5	MDA435
	Path	Combination	MIC a	MIC c	FIC	MIC a	MIC c	FIC	FICI MIC M IC 50	SI	IC 50	SI
	S. a	a EO:Vancomycin (1:0.5)	32	1	0.03	1	0.5	0.5	0.5	1.5	8	5.3	9	6
	C. n	b EO:Fluconazole (1:0.125) 128	64	0.5	8	8	1	1.5
							EO:Itraconazole(1:0.016) 128	64	0.5	1	1	1	1.5
	T. r	c EO:Fluconazole (1:0.5)	64	1	0.01	2	0.5	0.25 0.3
							EO:Griseofulvin* (1:0.25)	64	1	0.01	1	0.25 0.25 0.3	1.25	7	5.6 8.5 6.8
	T. m	c EO:Fluconazole (1:8)	32	2	0.06	64	16	0.25 0.3
							EO:Griseofulvin* (1:0.25)	32	1	0.03	0.5	0.25	0.5	0.5	1.25	7	5.6 8.5 6.8
	T. v	c EO:Fluconazole (1:0.5)	16	1	0.06	4	0.5	0.13 0.2
							EO:Griseofulvin* (1:0.03)

Innovax column; c R i Retention index identical to bibliography.

Acknowledgments

This work was supported by " 'A " the Agence Nationale de la Recherche (CEBA, ref. ANR-10-LABX-25-01). The authors gratefully acknowledge Pr. Soizic Pardo (MNHN) for fruitful discussions and Alexis Acanette (MIMA2 microscopy platform) for his technical support and expertise in electron microscopy.

Next, we investigated the source of the antimicrobial potential of this EO. The major constituent of H. lobelii oil, α-bisabolol, represented 34.5% of the total oil, and thus, it was tested on the microorganisms that showed the greatest susceptibility to the EO. The activity of α-bisabolol was roughly equivalent to that of the crude oil, with MIC values ranging from 8 to 64 μg/ml (Table 2). This result suggests a crucial role of α-bisabolol in the observed antimicrobial activity of this EO.


Ali El-Amine

Hussein Al

Haj Hassan
email: hhajhassan@aust.edu.lb

Mauricio Iturralde

Loutfi Nuaymi

Location-Aware Sleep Strategy for Energy-Delay Tradeoffs in 5G with Reinforcement Learning

Keywords: Energy consumption, service delay, 5G, sleep modes, LTE positioning, Q-learning

In this paper, we propose a sleep strategy for energyefficient 5G Base Stations (BSs) with multiple Sleep Mode (SM) levels to bring down energy consumption. Such management of energy savings is coupled with managing the Quality of Service (QoS) resulting from waking up sleeping BSs. As a result, a tradeoff exists between energy savings and delay. Unlike prior work that studies this problem for binary state BS (ON and OFF), this work focuses on multi-level SM environment, where the BS can switch to several SM levels. We propose a Q-Learning algorithm that controls the state of the BS depending on the geographical location and moving velocity of neighboring users in order to learn the best policy that maximizes the tradeoff between energy savings and delay. We evaluate the performance of our proposed algorithm with an online suboptimal algorithm that we introduce as well. Results show that the Q-Learning algorithm performs better with energy savings up to 92% as well as better delay performance than the heuristic scheme.

I. INTRODUCTION

In order to support future traffic requirements (high-speed mobile data services and better coverage), the next generation cellular networks are expected to deploy denser mobile access networks with different Base Stations (BSs) sizes (large, small and pico). Although this improves the capacity and coverage of the network, it also brings new challenges such as pushing the limits of energy consumption and CO 2 emissions. In order to meet these challenges, green cellular networks with improved energy efficiency have become a key factor in future 5G cellular networks. Since more than 50% of the network power is consumed by the Radio Access Network (RAN) and in particular the BS [START_REF] Suarez | An overview and classification of research approaches in green wireless networks[END_REF], it has become a high-priority objective for network management to reduce the energy consumption of this part of the network. Under these circumstances, BS sleep strategies are drawing increasing attentions recently [START_REF] Han | Survey of strategies for switching off base stations in heterogeneous networks for greener 5G systems[END_REF], [START_REF] Feng | Base station ON-OFF switching in 5G wireless networks: Approaches and challenges[END_REF].

The introduction of BS Sleep Mode (SM) is consistent with the objective of reducing the energy consumption of the network. For instance in [START_REF] Peng | Stochastic analysis of optimal base station energy saving in cellular networks with sleep mode[END_REF], the authors studied the energy saving problem by switching-off some macro BSs under coverage constraints. Using tools from stochastic geometry to model the location of the BSs, the authors achieved an energy efficiency gain up to 60% compared to when all the BSs are active. BS sleep strategies also play an important role in renewable energy equipped BSs to better manage the energy harvested and the energy stored in the battery [START_REF] El-Amine | Analysis of energy and cost savings in hybrid base stations power configurations[END_REF], [START_REF] El-Amine | Services kpi-based energy management strategies for green wireless networks[END_REF], [START_REF] El-Amine | Battery aging-aware green cellular networks with hybrid energy supplies[END_REF]. However, such management of energy saving has to be carefully coupled with managing the Quality of Service (QoS) to ensure the user satisfaction since it can bring additional delay. In [START_REF] Guo | Delay-constrained energyoptimal base station sleeping control[END_REF], the authors studied the tradeoff between energy savings and delay for different wake-up schemes. Similarly, in [START_REF] Chamola | Delay aware resource management for grid energy savings in green cellular base stations with hybrid power supplies[END_REF] this tradeoff is studied in a renewable energy environment where the BSs switch to sleep mode in a cooperative manner to further reduce the grid energy consumption.

In contrast to the above binary sleep scheme models (ON and OFF), [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF] proposed a multi-level sleep model for future 5G BSs. The model categorizes the power consumption for different types of BSs (e.g., Large, small, signal, data). This model provides different stages of SM levels. Each stage is characterized by an activation/deactivation period and a power consumption. In 4G networks (LTE), implementing these SM levels is challenging due to backwards compatibility and reference signaling requirements. However, in 5G networks the Cell Reference Signals (CRS) are removed [START_REF]Radio Resource Control (RRC)[END_REF] making it possible for a BS to explore these different SM levels paving the road to the ultimate goal of achieving almost zero power consumption at zero load. The work in [START_REF] Ezzahra | Reinforcement learning approach for advanced sleep modes management in 5G networks[END_REF], [START_REF] El-Amine | A distributed Q-Learning approach for adaptive sleep modes in 5G networks[END_REF] explored the potential of these SM levels. Using Reinforcement Learning (RL), the authors achieved an energy saving gain up to 90% in a low load traffic.

On the other hand, Location Based Services (LBS) are increasing in both commercial applications and emergency services. LBS involves the process of determining the geographical position of a device, such as a mobile phone. Due to the increase demands on positioning, the Federal Communications Commission (FCC) set several location accuracy and reliability requirements, especially for emergency services [START_REF] Baker | Wireless enhanced 911 working group: report of proceedings[END_REF]. Unlike previous radio-access standards, LTE incorporates positioning capabilities to support higher level of application-adaptive requirements in order to meet the requested positioning QoS. LTE adopts three independent positioning techniques: Assisted Global Navigation Satellite Systems (A-GNSS), Enhanced Cell ID (E-CID) and Observed Time Difference of Arrival (OT-DOA) [START_REF] Del Peral-Rosado | Achievable localization accuracy of the positioning reference signal of 3gpp lte[END_REF].

In this paper, we study the joint problem of energy savings and delay in a multi-level SM environment, where the BS can switch to several SM levels to save energy while maintaining the QoS of the users. We propose a methodology that allows the operator to freely manage the tradeoff between energy consumption and service delay according to the different requirements of the 5G use cases, such as Ultra-Reliable Low Latency Communications (URLLC). Different from the above mentioned work, in this paper we propose a Q-Learning-based algorithm that controls the state of the BS depending on the geographical location of the user, and his/her moving velocity in order to learn the best policy that maximizes the tradeoff between energy savings and delay. In order to evaluate the performance of the Q-Learning algorithm, we propose a heuristic policy that maximizes the energy savings while minimizing the delay of the network. We show that Q-Learning outperforms the heuristic scheme. To the best of our knowledge, the joint study of energy-delay and device positioning has not been well investigated in the literature. The closest work to ours can be found in [START_REF] Liu | Small cell base station sleep strategies for energy efficiency[END_REF] where the authors studied the state of small cells depending on the position of the users. However, the work focuses on maximizing the energy efficiency and is based on an offline algorithm using tools from stochastic geometry that cannot be implemented in an online manner.

This paper is organized as follows. In Section II, we detail the system model along with the 5G sleep mode model. In Section III, we start by presenting an overview of the positioning techniques for LTE before proposing our Q-Learning and heuristic algorithms. Finally, we present the simulation results in Section IV before concluding in Section V.

II. SYSTEM MODEL

We consider a 5G network composed of M gNBs serving k users. A gNB is a part of the Next Generation Radion Access Network (NG-RAN) that is part of the 3GPP 5G NextGen System [START_REF]System architecture for the 5G system (Rel. 15)[END_REF]. Each user requests a real-time service, e.g., VoIP call. We further assume the BS has active mode and different SM levels with different activation times. When a user requests a service from a gNB in SM, it triggers the activation mode and the user is buffered until the BS wakes up. This wake-up delay could have an impact on the latency added to the system. The deeper the SM is, the more time the user will have to wait until the gNB reactivates.

A. Sleep modes in 5G networks

In [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF], GreenTouch Project identified four distinct SM levels by grouping sub-components with similar transition latency when being activated or deactivated. The presented model enables to quantify the power consumption of the BS in each of the four SMs. These are:

• SM 1: It considers the shortest time unit of one OFDM symbol (i.e. 71µs) comprising both deactivation and reactivation times. In this mode only the power amplifier and some processing components are deactivated. where the BS is out of operation but retains wake-up functionality. Higher energy savings can be achieved when switching BSs to a deeper SM, since more components will be deactivated. However, this will be associated with longer transition latency which may impact the QoS for the users. In Table I, we present the SM levels characteristics.

Along with SM and users' dynamics, the BS has to wake up periodically to send signaling bursts of Synchronization Signaling (SS) and Physical Broadcast CHannel (PBCH). It has been agreed in Third Generation Partnership Project (3GPP) [START_REF]Radio Resource Control (RRC)[END_REF] that the transmission periodicity of the SS/PBCH block can be set to any value among [5, 10, 20, 40, 80, 160 ms]. With these values, SM 4 cannot be used. Hence, we limit our work to the first three SM levels.

B. Dynamic user model

We consider two cases for users service requests from a BS: 1) Camping in a gNB: in this case, the user is in any of the three 5G Radio Resource Control (RRC) modes: connected, idle or inactive [START_REF]System architecture for the 5G system (Rel. 15)[END_REF]. 2) Handover arrival: in this case, a user already performing a service transmission/reception from a BS is moving towards the coverage of a neighboring BS. The user is handed over the other BS for service continuation. In the first case, when a service is being called for the first time, the delay impact on the user is tolerable since the transmission has not yet started. However, in the case of a handover (case 2), the service might be interrupted or degraded if the BS hosting the user is inactive (i.e., in sleep mode). This might have a severe impact on the QoS of the transmission. It is important to note that if the requested service is of type non real-time (e.g., web browsing), then this delay will have a less impact on the QoS since non real-time services are delay tolerant.

In this work, we consider the case of real-time services, and in particular the handovers of users between BSs. The objective is to jointly minimize the delay associated with handovers, and the energy consumption of the network. As shown in Fig. 1, we consider a user being served by BS A moving towards BS B with a velocity v (Km/h). Since BS B is not serving any users, it decides to switch to a specific SM level. This decision depends on the position of the user from BS B, i.e., the closer the user moves towards BS B, the lighter its SM level becomes. Since positioning with LTE is not always accurate and is prone to measurement errors, we divide the geographic region of interest (the region of neighboring sites) into several zones (z 1 , z 2 , . . . , z n ). Each zone corresponds to a region that can be detected by the BS. For example, if a user is in zone j (z j ), the BS senses the presence of the user in z j , but it does not know its exact location. The size of these regions depends on the accuracy of the positioning method used, i.e., the smaller the zone size, the more accurate the user position is measured.

III. LOCATION-AWARE SLEEP MODE STRATEGY A. Positioning methods with LTE

In wireless networks, positioning is very challenging due to the user mobility and the dynamic nature of both the environment and radio signals. On the other hand, the FCC set some stringent requirements on the accuracy of positioning especially for emergency services [START_REF] Baker | Wireless enhanced 911 working group: report of proceedings[END_REF]. As a result, 3GPP described an architecture, standards and methods for LTE to support different positioning techniques that reach a high level of accuracy as requested by the regulations.

LTE supports three positioning techniques: A-GNSS, E-CID, and OTDOA.

1) A-GNSS: To overcome the line of sight and low signal level drawbacks problems of GNSS, the cellular network assists the GNSS receiver to increase its positioning operation. A-GNSS works best in outside conditions with clear view of the sky (line of sight). 2) E-CID: This method is based on Cell of Origin (COO) that estimates the position of the device in the geographical area of its serving BS. Since this method is not accurate enough (linked to the cell size), E-CID performs measurements on radio signals such as Reference Signal Received Power (RSRP) and Time Different Of Arrival (TDOA) to measure the Round Trip Time (RTT) and Angle-of-Arrival (AoA) for more accuracy. With E-CID, it is also possible to have information on the direction of the device. 3) OTDOA: According to the LTE specification [START_REF]Stage 2 functional specification of User Equipment (UE) positioning in E-UTRAN (Rel. 9)[END_REF], this method relies on the downlink radio signals from multiple BSs to compute the user position. In [START_REF]Stage 2 functional specification of User Equipment (UE) positioning in E-UTRAN (Rel. 9)[END_REF], LTE standard specifies Positioning Reference Signal (PRS) that is dedicated for positioning purposes. By properly configuring the PRS and with interference management techniques, the error in position can be significantly improved to 1 meter, however, at the expense of decreasing the spectral efficiency [START_REF] Del Peral-Rosado | Achievable localization accuracy of the positioning reference signal of 3gpp lte[END_REF].

Each of the above proposed technique has its advantages and limitations. To improve positioning in challenging radio environment, hybrid positioning using the different above methods is also supported in Release 10 [START_REF]LTE; Evolved Universal Terrestrial Radio Access Network (E-UTRAN); Stage 2 functional specification of User Equipment (UE) positioning in E-UTRAN (Release 10)[END_REF]. In Table II, we summarize the differences between these methods. For more details on LTE positioning, the reader is referred to [START_REF]Positioning with LTE[END_REF]. Distributed Q-Learning is an online optimization technique that aims at finding the best policy of an agent (e.g., BS) via real-time interactions with the environment. Q-Learning obtains the optimal policy by maximizing the expected value of the total reward (Q-value) over all successive episodes.

In Q-learning, an agent m takes an action a t m from an action set A, then moves to a new state s t+1 m while receiving a reward r t m . The Q-value is then updated locally indicating the level of convenience of selecting action a t m when in state s t m . The Qvalue is updated as follows:

Q(s t m , a t m ) ← Q(s t m , a t m )+ α[r t m + γ max a∈A Q(s t+1 m , a t+1 m ) -Q(s t m , a t m )] ( 1
)
where α is the learning rate that represents the speed of convergence, γ ∈ [0, 1] is the discount factor that determines the current value of the future state costs, and t is the time at which the action has been taken. During the learning phase, the agent selects the corresponding action based on the -greedy policy, i.e., it selects with probability 1the action associated with the maximum Qvalue, and with probability selects a random action:

a t m =    argmax a∈A Q(s t m , a t m ), if y > rand(A), otherwise m = 1, . . . , M. (2)
By implementing the -greedy policy, the agent would have explored all possible actions and avoided local minima. For more details on RL and Q-learning the reader is referred to [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF].

In this work, we define the set of possible actions A the state of the BS, i.e., active or in sleep mode (SM 1 , SM 2 or SM 3 ). The BS chooses the appropriate action based on the geographical zone the user lies in. Thus, we define the state space and action space as follows:

S = {z 1 , z 2 , . . . , z n } A = {Active, SM 1 , SM 2 , SM 3 }
The number of zones depends on the accuracy of the positioning method used, and it spans the area of the closest neighboring BSs.

An episode starts when a user asks for a service, moves towards a neighboring cell, and finishes when the handover is complete. During each episode, the neighboring BS that the user is approaching to, changes its state by taking an action following Eq. ( 2). Then, it stores a quality-value linking the states s ∈ S to the chosen action a m ∈ A following Eq. ( 1). The optimal policy consists of choosing the best sleep mode level having the highest Q-value.

The goal is to find the best policy for each state (zone) along which the user is moving in order to maximize the reward r t m . We define the reward as the weighted-sum of the energy gain G and the added delay D, both resulting from the sleep mode level chosen during an episode.

r = (1 -η)G -ηD (3)
where η ∈ [0, 1] is a parameter which controls the trade-off between energy gain and the delay performance. We note that in the delay-tolerant services case (e.g., web browsing), η → 0, thus emphasizing on saving energy. In the other case where the service is delay-sensitive (e.g., VoIP), η → 1. We note that the weight parameter (η) is freely chosen by the operator depending on the 5G use cases.

Algorithm 1 : Q-Learning Algorithm 1: Initialize q(s, a) = 0, ∀s ∈ S and ∀a ∈ A.

2: Set the weight η, and the average user velocity v. Visit state s.

6:

Select an action a using -greedy rule in (2).

7:

Receive a reward r.

8:

Observe next state s .

9:

Update the Q-value q(s, a) from (1). 10: end while 11: end procedure 1: procedure Online 2: From Q(s, a), store best action in Q-table ∀s ∈ S and ∀a ∈ A. 3: Run Q-Learning. 4: end procedure In details, the training phase consists of running the BS (agent) with different instances of user arrivals in an offline fashion. After this step is completed, the system goes online. During this phase, a Q-table stores the best policy using the trained Q-values of the previous offline step. The algorithm is described in Algorithm 1.

C. Delay-Sensitive Sleep Mode (DS-SM) algorithm: Heuristic approach

In contrast to the Q-Learning-based algorithm that includes an offline training phase with a high degree of system infor- mation knowledge (e.g., exact user position, d, and velocity, v), we propose a Delay-Sensitive Sleep Mode (DS-SM) online algorithm that takes decisions on which sleep mode level to switch to based on the estimated measured position ( d) and velocity (ṽ). We use this algorithm as a benchmark to study the performance of the Q-Learning algorithm described in the previous section. When a user reaches a geographical zone (z j ), the estimated distance from the neighboring cell ( d) is calculated from the center of the zone it is located, as shown in Fig. 2. The estimated velocity on the other hand is related to the actual velocity by the following expression: ṽ = v +α, where α is the error in the measurement. The BS takes the decision to switch to SM i in a geographic zone (z j ) if after the sleep duration T SMi has elapsed, the user did not enter the neighboring cell, thus minimizing the delay. If the user enters the neighboring cell while it is inactive, his/her QoS will degrade due to the added delay resulting from the waking up time from SM i level. In order to maximize the energy savings, the algorithm starts with the deepest sleep mode level allowed (i.e., SM 3 ). The algorithm is summarized in Algorithm 2.

Algorithm 2 : Delay-Sensitive Sleep Mode (DS-SM) Algorithm 1: Measure user location d and velocity ṽ. if T SMi × ṽ < d then end if 13: end for 14: Output: Energy consumption and network added delay.

IV. SIMULATION RESULTS

A. Simulation parameters

In this section, we demonstrate the performance of the proposed approach for deciding SM levels for neighboring BSs. We consider a low traffic period where a BS serves a user with a continuous real-time service. We focus on the delay as the performance metric for QoS. The reason is twofold. First, we consider a low traffic period. Thus, bandwidth is not a problem. Second, we focus on real-time services, such as VoIP call, that do not require high throughput, but are delay sensitive. The user is randomly generated in a geographical zone and moving towards a neighboring BS with a constant velocity. We further consider that the inter-site distance is 800m.

In order to point the impact of the different SM levels on the energy savings, we consider a low load traffic. From [START_REF]IMEC power model tool[END_REF], we found the power figures for the different states of the BS. Then we define the sleep duration times for each SM level as: T SM1 = 0.5s, T SM2 = 10s and T SM3 = 30s. The reason behind these values is to minimize the measurements done by the BS for user positioning. Table III

B. Convergence analysis

First, we analyze the convergence of the proposed Q-Learning algorithm for η = 0.5. In Fig. 3, we evaluate the impact of the system positioning accuracy on the performance of the Q-Learning training phase. It is clear to observe that as the system accuracy decreases (e.g., from 10m to 80m), the learning delay decreases as well. For instance, after 50 episodes the Q-Learning algorithm converges for a positioning accuracy of 80m, whereas, 400 episodes are required for a sharper accuracy of 10m. This is related to the number of states that increases with higher accuracy. Thus, requiring more time to converge. Once the algorithm converges, we stop the training phase and exploit the obtained policies. Then, we store the best policy in a look-up table that will be used during the exploitation phase.

C. Energy consumption and service delay trade-off

In Fig. 4, we present the tradeoff between the average network energy consumption and added delay for different values of η and with a positioning accuracy of 10m. In other words, the difference between two geographical zones is 10m. We normalize the energy consumption with the case when the BS is always in idle mode. First, we observe that significant energy savings up to 92% (corresponding to the normalized energy consumption of 0.098) can be achieved using the multilevel sleep modes. The proposed Q-Learning algorithm allows a wide range of control over the energy consumption. For instance, with η = 0, the energy consumption is the lowest. However, this is at the expense of added service delay that is at its highest (5ms). This is because for this case, the SM policy is chosen solely considering energy savings. The resulting delay corresponds to the waking up time when the BS is in the deepest sleep state (SM 3 ). For η = 1, we observe that the energy consumption is higher, but the delay is zero. So when the user arrives to the neighboring cell, the BS is already active. It is then important to carefully choose η in order to satisfy the requirements of the different 5G use cases.

D. Performance evaluation

As a benchmark for comparison, we consider the DS-SM online scheme presented in Section III-C. This algorithm does not depend on η. However, its goal is to minimize the service delay while reducing the energy consumption. Hence, the corresponding η in the Q-Learning algorithm can be found for the case where the delay is minimized. Since DS-SM also takes into account the energy consumption, we look for the smallest value of η that minimizes the delay. For example in Fig. 4, this corresponds to η = 0.9. We note that the value of η for the designated condition changes for different user velocity and position accuracy. In the following, we compare the reward expression in Eq. ( 3) that takes into account both energy consumption and service delay. For a fair comparison, this weighted parameter will be also used when computing the reward resulting from the DS-SM algorithm the same way it is computed for the Q-Learning algorithm. We note that this reward can be regarded as a utility function of η. From Fig. 5, we show that the Q-Learning algorithm leads to a higher reward than DS-SM and in particular for low positioning accuracy. The poor performance in DS-SM results from choosing the policy based on the inaccurate positioning (always in the center of the zone), since it does not include a training phase to tune these decisions to optimize the reward. For low position accuracy (e.g., 90m), the gap between the user's real and measured positions is increased. This results in a performance degradation for both algorithms. However, DS-SM adds high delay when R(η) < 0.

V. CONCLUSION

In this work, we investigated the tradeoff between energy consumption and service delay associated with sleep strategies in 5G networks. While BS sleeping significantly reduce the energy consumption of the BS (up to 92%), it is coupled with QoS degradation by bringing additional delay to the users. We proposed a methodology for reducing the energy consumption of the BS while ensuring a good QoS. This methodology also permits the operator to freely manage the tradeoff between energy consumption and service delay. This objective was achieved by switching the neighboring BS to different SM levels depending on the location of the user in the network, and his/her moving velocity towards another cell. In order to choose the best SM policy, we proposed a Q-Learning algorithm. Compared with the heuristic DS-SM algorithm, the proposed Q-Learning algorithm outperforms the benchmark scheme in terms of bringing more energy savings to the network while maintaining a good QoS, and it allows the operator to manage the tradeoff factor (η) according to the 5G use cases.

• SM 2 :

 2 It corresponds to the case of sub-frame or Transmission Time Interval (TTI) (i.e. 1 ms). In this SM, more components enter the sleep state. • SM 3: It corresponds to the frame unit of 10 ms. Most of the components are deactivated in this mode. • SM 4: This is the deepest sleep level. Its unit corresponds to the whole radio frame of 1s. It is the standby mode

Fig. 1 :

 1 Fig. 1: The position of the user moving towards a neighboring gNB can be measured by the LTE positioning system.

  40 m but can go down to 1 m at the expense of SE) Requires synchronized network and operator dependency B. Sleep mode policy: Q-Learning approach

3 :

 3 procedure Training (Q(s, a)) 4: while Learning do 5:

Fig. 2 :

 2 Fig. 2: User positioning measurement following DS-SM algorithm.

2 :

 2 Repeat until handover is complete. 3: for i = 3..1 do 4:

5 : 6 :

 56 Switch the BS to SM i . Update d after T SMi is elapsed.

Fig. 3 :Fig. 4 :

 34 Fig. 3: Convergence of the Q-Learning algorithm for η = 0.5.

Fig. 5 :

 5 Fig. 5: Performance comparison between Q-Learning and DS-SM algorithms with different user velocity measurement accuracies.

TABLE I :

 I BS Sleep Modes Characteristics[START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF].

	Sleep	Deactivation	Activation	Power
	level	duration	duration	consumption
	SM 1	35.5 µs	35.5 µs	48%
	SM 2	0.5 ms	0.5 ms	13%
	SM 3	5 ms	5 ms	9%
	SM 4	0.5 s	0.5 s	7.5%

TABLE II :

 II Comparison of LTE positioning techniques.

  summarizes these values.

TABLE III :

 III Power Consumption of a 2 × 2 MIMO BS.

	State	SM 1	SM 2	SM 3	Idle
	Power consumption (W)	52.3	14.3	9.51	109
	Sleep duration (T SM i ) (s)	0.5	10	30	-