Laura Rowenczyk

Céline Picard
email: celine.picard@univ-lehavre.fr

Cécile Duclairoir-Poc

Nicolas Hucher

Nicole Orange

Marc Feuilloley

Michel Grisel

Development of preservative-free nanoparticlesbased emulsions: effects of NP surface properties and sterilization process

Keywords: Emulsion formulation, Sterilization, TiO 2 NP, Particle size measurement, Rheology, Texturometry, Microbiological assays

Model emulsions were developed with or without commercial titanium dioxide nanoparticles (NP) carrying various surface treatments in order to get close physicochemical properties whatever the NP surface polarity (hydrophilic and hydrophobic). Rheology and texturometry highlighted that the macroscopic properties of the three formulated emulsions were similar.

However, characterizations by optical microscopy, static light scattering and zetametry showed that their microstructures reflected the diversity of the incorporated NP surface properties. In order to use these model emulsions as tools for biological evaluations of the NP in use, they had to show the lowest initial microbiological charge and, specifically for the NPfree emulsion, the lowest bactericidal effect. Hence, formulae were developed preservativefree and a thermal sterilization step was conducted. Efficiency of the sterilization and its impact on the emulsion integrity were monitored. Results highlighted the effect of the NP surface properties: only the control emulsion and the emulsion containing hydrophilic NP fulfilled both requirements. To ensure the usability of these model emulsions as tools to evaluate the 'NP effect' on representative bacteria of the skin microflora (S. aureus and P. fluorescens), impact on the bacterial growth was measured on voluntary inoculated formulae.

Introduction

Titanium dioxide (TiO 2 ) NP are commonly used in Cosmetics, especially as UV filters in sunscreen emulsions. Their tiny size, enabling transparency in the visible range, makes it possible to obtain end products without any residual traces on skin in addition to a good UVprotection [START_REF] Serpone | Inorganic and organic UV filters: Their role and efficacy in sunscreens and suncare products[END_REF]). However, if compared to micrometric oxides, nanometric ones are more reactive and in particular photo-reactive. Therefore, such nano-objects receive a first coating with silica or alumina as it allows quenching photo-produced radicals [START_REF] Serpone | Beneficial effects of photo-inactive titanium dioxide specimens on plasmid DNA, human cells and yeast cells exposed to UVA/UVB simulated sunlight[END_REF]; in addition, a second coating is sometimes added in order to improve their dispersibility in formula. Whereas European Regulation (European Parliament, Council of the European Union 2014) clearly mentions the obligation to demonstrate the safety of even coated NP, very few studies took interest on their behavior in use. [START_REF] Santaella | Aged TiO 2 -based nanocomposite used in sunscreens produces singlet oxygen under long-wave UV and sensitizes Escherichia coli to cadmium[END_REF]) compared the eco-toxicity of NP with degraded coating in aqueous environment [START_REF] Auffan | Structural degradation at the surface of a TiO2-based nanomaterial used in cosmetics[END_REF] to that of pure TiO 2 . Other studies dealt with the impact of zinc oxide NP on bacteria according to their size [START_REF] Pasquet | Antimicrobial activity of zinc oxide particles on five micro-organisms of the Challenge Tests related to their physicochemical properties[END_REF]). However, aggregation state and physicochemical behavior of NP in formula have to be taken into account and their safety in emulsions when they are layered on skin remained a critical question. Moreover, these questions are closely related to the large variety of surface treatments commercially available (polar, non-polar, hydrophilic, hydrophobic…).

To evaluate the only effect of NP in formula, emulsions need to be developed in order to respect three main criteria:

-Being stable and exhibiting similar microstructure and macroscopic properties whatever the coating of the incorporated NP ; -Exhibiting the lowest initial microbiological charge ; -Giving the control emulsion (without NP) the lowest bactericidal effect.

Therefore emulsions need to be developed without preservative. Hence, in order to fulfill the second criterion, alternative process, like sterilization procedures, consisting in the elimination of germs and spores in a product, has to be used. However, cosmetic form does not allow the use of technics such as filtration of emulsion systems because it could lead to a demixing phenomenon [START_REF] Kocherginsky | Demulsification of water-in-oil emulsions via filtration through a hydrophilic polymer membrane[END_REF] or UV or gamma radiations because of presence of UV filters in this study. Sterilization by high temperature seemed the best solution especially as this kind of methods begins to be used on cosmetic emulsions [START_REF] Delaunay | Patent: (WO2013007755) Device and Method for Sterilizing an Emulsion at an Ultrahigh Temperature, Particularly a Cosmetic Emulsion for the Skin, Which Is Unstable at the Sterilization Temperature[END_REF] associated to an adapted packaging [START_REF] Devlieghere | A new protocol for evaluating the efficacy of some dispensing systems of a packaging in the microbial protection of water-based preservative-free cosmetic products[END_REF]. Nevertheless, high temperatures could accelerate aging (Waterman & Adami 2005) and, recently [START_REF] Rossano | Effects of aging on structure and stability of TiO2 nanoparticlecontaining oil-in-water emulsions[END_REF] demonstrated that aging of emulsions containing hydrophobic NP induced particles aggregation, adsorption of formula's compounds and coatings deterioration. Thus, as it is still relatively unexplored, it is essential to bring data concerning the effect of coatings and sterilization temperature on the emulsion properties. Our study focuses on the characterization of model emulsions in order to ensure the similarity in terms of microstructure and physicochemical properties of a control emulsion without nanoparticle and formulae containing either hydrophilic/hydrophobic NP.

Finally, microbiological evaluations were conducted on two skin bacteria strains to check the low initial microbiological charge of the model emulsions and the low bactericidal impact of the control one.

Experimental Section

Chemical and reference materials.

Water with a resistivity of 18 mΩ.cm used during the emulsion preparation and zetametry was purified, filtrated on 0.2 µm and treated by UV in order to eliminate germs, after described as ultrapure. Otherwise distilled water with a resistivity of 18 mΩ.cm was used for other emulsion characterizations. Two different grades of TiO 2 -NP were kindly given by Kobo and Merck; one hydrophilic, with a simple silica coating named N1 (EUSOLEX® TAVO, Merck Chimie SAS, France) and a second, hydrophobic, composed of alumina (and) triethoxycaprylylsilane named N2 (A10-TiO2-11S7, Kobo Products, France). Other cosmetic grade ingredients, introduced in the Table 1 were chosen to obtain simplified sunscreen, oil-in-water emulsions type, and to improve NP dispersions. Ingredients were used as received.

To achieve emulsions with the lowest bactericide impact, no preservative was added.

Emulsion preparation.

During preservative-free emulsion preparation, precautions were taken in order to minimize the risks of microorganisms contamination: all stainless steel materials were steamed for 45 min at 190°C and glass bottles were rinsed with ethanol prior to drying (INRS 2014). For technical reasons it was impossible to make all operation under laminar flow hood in sterile conditions. But controls revealed that aerial contaminants in final formulations remained under the detection limit.

Emulsion without nanoparticle.

Emulsion without NP was adapted from the protocol described by [START_REF] Rossano | Effects of aging on structure and stability of TiO2 nanoparticlecontaining oil-in-water emulsions[END_REF]. It was developed to obtain similar emulsions, with or without NP, and to improve the NP dispersion. The gelling agent was sprinkled in water and hydrated 20 min without stirring. This mix corresponding to the phase B and the phase A were separately homogenized at 80°C under mechanical stirring for 10 min. After water loss compensation, phase A was added to phase B off the heat and under vigorous stirring (11000 rpm for 1.5 min), by using a rotorstator type homogenizer (Ultra-Turrax, stator diameter 25 mm, rotor diameter 18 mm, IKA).

Then, the mixture was put under stirring (Turbotest, radial flow turbine of 55 mm diameter, VMI Raynerie) at 750 rpm and triethanolamin (TEA) was immediately added. After 5 min, the stirring rate was raised to 1000 rpm. A cold water bath was used once cooled below 38°C.

When the temperature of emulsion was close to 25°C, pH was adjusted at 6.6 -6.8 using HCl 0.4 M. The mixture was kept under stirring for 5 additional minutes. It was then put under vacuum to remove air bubbles incorporated during the previous process.

Nanoparticle dispersions.

Because rotor-stator type homogenization was not sufficient to obtain an acceptable dispersion in emulsion, NP powders were first pre-dispersed in one of the emulsion phases, leading to a NP paste. Hydrophilic NP and water with a ratio of 1/2 (w/w) were sonicated 10 min (Ultrasonic cleaner, VWR, 45 kHz, 120 W). Hydrophobic NP and cetearyl ethylhexanoate with a ratio of 2/3 (w/w) were mixed. Then, this pre-dispersion was grounded on a three-cylinder mill for three times to reduce the size of agglomerates.

Emulsions with nanoparticles.

The protocol remained the same as without NP except that just after the addition of TEA, NP paste was added under a stirring of 1000 rpm to reach the optimal dispersion in emulsion.

The end of the preparation was the same than without NP.

Sterilization.

Each emulsion was poured into several 50 mL glass autoclave-resistance bottles (VWR).

After maturation at 20 ± 2°C for 24h, a half of them was sterilized during 20 min at 120°C and 1 bar relative pressure in an autoclave (Laboster, Subtil-Crepieux, Chassieu, France). The second part was let at the same temperature. After heating and when the temperature cooled below 90°C, the autoclave was opened and the bottles were let at ambient air in order to accelerate the cooling. Thus, for instance, ST-F-0 was the autoclaved version of F-0.

Efficiency of the heat treatment can be evaluated by calculation of the sterility value (SV):

SV = ∫ ) )
 T(t) is the core temperature in the time interval,  T ref is the reference temperature,  Z is the difference of temperature required to accelerate tenfold the germs destruction reaction.

In the case of a sterilization step, T ref and Z are respectively equal to 121.1°C and 10°C [START_REF] Bimbenet | Génie des procédés alimentaires 2ème édition[END_REF]. A temperature probe (DataTrace, CMI) was introduced into a product bottle in order to obtain the core temperature essential for the SV calculation.

Emulsions characterization.

Once prepared, emulsions were characterized after a period of maturation for two days or if so, sterilization.

Microstructural characterization.

Optical Microscopy.

Qualitative observations were performed using an optical microscope (DMLP/DC 300, Leica Microsystems, Wetzlar, Germany) equipped with a camera and the IM1000 software (version 1.20 Release 19). In order to visualize emulsion microstructures, droplets sizes and quality of NP dispersion, a tip of emulsion plus a water drop were placed between slide and coverslip. Pictures were obtained at a magnification of x200.

Particle size measurement.

Particles sizes measurements were performed by using a laser diffractometer SALD 7500

Nano (405 nm, Shimadzu, Marne-la-Vallée, France) combined with the measure cell SALD BC 75 consisting in a 7 mL batch and a stirrer and equipped with the WingSALD II-7500 software (version 3.1). In order to reach an absorption value of the samples between 1.2 and 1.5 and the best quality of the obtained signal, particle dispersions had to be diluted and homogenized according to the following procedures. Approximately 0.1 g of NP pastes was pre-dispersed in a test tube with 5 mL of distilled water for coating 1 or of cetearyl ethylhexanoate for coating 2. This pre-dispersion was then sonicated 5 min (120 W). In the case of emulsion particle sizing measurement, approximately 0.1 g of emulsion were prediluted in a test tube with 5 mL of distilled water. The mix was vortexed 10 sec at 1000 rpm.

Finally, pre-dispersed samples were added by drop until reaching the expected absorption. All measurements were performed at least in triplicate on two different batches. D10, D50 and D90 were thereafter expressed in particle volume. By this way, for instance, D10 represented the threshold value in micrometer for which 10% of particles had a smaller size.

Zetametry.

Mean zeta potentials of emulsions were performed using a Zeta Sizer Nano-Z and folded capillary cells (Malvern instruments, Malvern, UK). Emulsions were diluted at 0.5% w/w in ultrapure water containing NaCl at 10 -4 M (conductivity of 1.70 µS/cm). Six samples of each emulsion were adjusted at pH 2, 4, 6, 8, 10 and 12, respectively, with using NaOH or HCl (0.4 or 0.04M).

Measurements were performed at least three times.

Macroscopic properties.

Rheology.

Rheological properties were achieved with a controlled-stress rheometer (Discovery HR1, TA instrument, Guyancourt, France) using an aluminum cone-plate geometry (40 mm diameter, cone angle of 1.994° and truncation of 47 µm). Data analyses were conducted with the software TRIOS® 3.0. Stress sweep was carried out at 25°C, temperature controlled by Peltier effect, with an imposed stress gradient from 0.01 to 150 Pa and a fixed frequency of 1

Hz. This analysis highlighted the linear viscoelastic region and permitted to obtain the viscoelastic parameters, independent of the applied stress as listed below:

 tan (δ), where δ is the phase shift between strain and stress and is equal to G"/G'.

 G', the elastic modulus (Pa),  G", the viscous modulus (Pa),  γ c , the critical strain, corresponds to the percentage strain at 90% of the plateau G' value.

All measurements were performed at least twice on two different batches.

Texture analyses.

Texture analyses were performed by using a TA.XT Plus (Stable Micro Systems, Cardiff, UK) and the software Texture Exponent 32 (version 5,0,6,0, 2010). It allows the characterization of mechanical properties of emulsions. The protocol was adapted from [START_REF] Gilbert | Predicting sensory texture properties of cosmetic emulsions by physical measurements[END_REF]. Effect of the compression was studied with the P/35 probe (35 mm diameter, aluminum). 750 µL of emulsion were deposited on the analyzer base by using a Microman® M250 (Gilson) and compressed by the probe until a gap of 0.5 mm with a rate of 1 mm/s. Three parameters were recovered:

 the positive area, corresponding to the work required during the product compression (kg.sec),

 the maximum force reached (kg),

 the negative area, corresponding to the work required during the probe removal (kg.sec).

Ability to spreading was lead with the friction modulus A/FR. A polypropylene (PP) sheet was fixed on the analyzer base and the probe was covered with a Helioplate TM HD 2 (Helioscreen, Creil, France) consisting of a PMMA plate. Four lines of each 50 µL of emulsion were deposited using a Microman® M250 (Gilson, Villiers-le-Bel, France) in the displacement direction. The probe was pulled on a distance of 120 mm with a speed of 3 mm/s. Pictures were taken from spreading traces on the PP sheet. In order to evaluate the corresponding traces thickness, the lightness evolution (L* factor in the CIE L*a*b* color model) was measured every 2 cm, starting to the beginning of the trace (Spectrophotometer CM-5, Konica Minolta, Nieuwegein, Netherlands).

All measurements were performed at least twice on two different batches.

2.4. Micrbiological assays.

Bacteria strain and culture conditions.

Pseudomonas fluorescens MFP05 and Staphyloccocus aureus MFP 03 are normal human skin bacteria strains collected by swabbing and characterized by metabolic, proteomic and genomic analysis [START_REF] Hillion | Comparative study of normal and sensitive skin aerobic bacterial populations[END_REF]. These strains are stored on cryobeads at -140°C.

Before use, they were sub-cultured over-night in 50 mL of Luria Bertani (LB) broth at 28 ± 2°C for MFP05 or 37 ± 2°C for MFP03. The cultures were then diluted again in LB broth to reach the appropriate initial bacterial density corresponding to an OD 580nm = 0.08. OD values were measured using a spectrophotometer (Helios Epsilon, ThermoSpectronics, Cambidge, UK). Bacteria were used when the cultures reached the end of the exponential growth phase.

Counting and detection of mesophilic aerobic bacteria.

Mesophilic aerobic bacteria in emulsions were counted and detected by using the norm NF EN ISO 21149:2006[START_REF] Afnor | NF EN ISO 21149:2006 -Cosmétiques -Microbiologie -Dénombrement et détection des bactéries aérobies mésophiles[END_REF]. It includes three culture tests. To base the study on solid grounds, we chose to correlate two of them. The first one, without enrichment, consists in diluting approximately 1 g of emulsion in ten times this exact mass in peptone water and then vortexing at 1000 rpm for 20 sec. 100 µL of this mix was spread on a TSA Petri dish and incubated at 30 ± 2 °C for four days. The second test, with enrichment, was the dilution of approximately 1 g (exactly weighed) of emulsion in ten times the mass in EUGON LT 100 broth. The liquid mix was incubated at least 20h at 30 ± 2 °C under orbital stirring at 180 rpm.

Then the mix was vortexed at 1000 rpm and 100 µL were spread on a EUGON LT 100 agar Petri plate. They were then incubated at 30 ± 2 °C for four days. Three Petri dishes were prepared for each condition and emulsion. The number of mesophilic aerobic bacteria (N) was determined by the equation:

 N = m / (V x d) where,
 m is the mean number of colonies on Petri dishes,  V is the inoculated volume on a Petri dish (mL),  D is the dilution of the emulsion (1/10 in this case).

Evolution of willingly inoculated bacteria in model emulsions.

Approximately 5 g of each emulsion were diluted to the third in physiologic water (NaCl 0.9%) or LB broth. Obtained solutions were inoculated with MFP03 or MFP05 at an initial concentration between 10 5 and 10 6 UFC/mL. The exact bacterial concentration was controlled by plating 100 µL of solutions and serial dilutions on TSA Petri Dishes immediately after bacterial inoculation. Inoculated solutions were incubated at 28 ± 2°C in the case of MFP05 or 37 ± 2°C for MFP03 under orbital agitation at 180 rpm. After different period of time (24h, 48h and 72h, respectively), 100 µL of solutions were spread on TSA Petri dishes in order to monitor bacteria growth evolution. All Petri dishes were incubated 24h at 28 ± 2°C for MFP05 or 37 ± 2°C for MFP03 before counting.

Data analyses.

Results were expressed as mean ± standard deviation (SD). Statistical analyses of collected data were performed on XLSTAT software (version 2012.1.01, Addinsoft, France). Singleway analyses of variance (ANOVA) were applied to data series in order to test significance of parameters. When a significant difference was revealed between emulsions (p<0.05), groups of emulsions were formed using Tukey multiple comparison test.

Results and Discussion

Comparative study of emulsions with or without nanoparticles.

Three emulsions were obtained by the lab-scale process depicted in the Material and Methods section: a control emulsion without NP (F-0), an emulsion (F-N1) with hydrophilic NP (N1) and an emulsion (F-N2) with hydrophobic NP (N2).

Microstructural characterization.

 Characterization of nanoparticles dispersions.

When working with NP, emulsions and their mixtures, an important issue is to characterize size distributions, making sure that the techniques used, the operating conditions and analyses parameters fitted well.

Both NP primary sizes given by suppliers were around 10 nm diameters which correspond to the dimension of the particles measured just after manufacturing. However, at the nanometric scale, attractive forces are intense and NP in powder are known to form aggregates-like structures of much larger dimensions (larger than 1 µm) [START_REF] Aldous | Conference: Titanium dioxide and zinc oxide nanoparticles in sunscreen formulations: A study of the post production particle size distribution of particles in a range of commercial emulsion variants[END_REF]. Thus, in the present study, several processes were tested to improve the particles dispersions in emulsions. The best one was obtained when pre-dispersion was prepared separately and added after the emulsification step. Size distributions of NP in pre-dispersion were monitored by laser granulometry as described in Materiel & Methods. Because of the thinness of the dispersions, determination of the refractive indexes (RIs) of the particles was required to use the Mie theory. Different parameters could have an impact on this RI and should be investigated. At the studied wavelength, calculation models showed that the crystallinity, the shape and the size did not impact the RI of TiO 2 -particles [START_REF] Auvinen | Refractive Index Functions of TiO 2 Nanoparticles[END_REF]). The coating could also have an effect on the particles refraction. Thus, calculations based on the RI of the TiO 2 or of the coating were made and gave the same results in term of size distribution.

Added to ester phase, N2 NP formed larger aggregates, with a D90 equal to 30 µm. Thanks to its high viscosity, the paste was refined using a three-cylinder mill. This method decreased the D90 under 4 µm. N1 NP in water were already well-dispersed and had a D90 already around 11 µm. Due to the low viscosity of the hydrophilic dispersion, an ultra-sonication treatment was used and reduced even more the D90 until 1.7 µm. Given in number of particles, the D50 of N2 was equal to 180 nm and the D50 of N1 to 60 nm. Consequently, N1

met the official definition of NP (Official Journal of the European Union 2011) while N2 was really close. However, in this study, size distribution was expressed in volume as it magnifies the impact of the biggest aggregates and allows a better evidencing of the dispersion heterogeneity.

 Characterization of emulsions.

For the same reasons as above, emulsions size distributions were also given in volume of colloids. All colloids present in emulsions, TiO 2 -particles as well as oil droplets, were detected during measurements. Since these colloids had various sizes and optical properties, the use of a RI was not applicable. For this reason, the Fraunhofer's approximation was chosen in term of calculation model. This model is generally applicable for colloids owning size larger than few micrometers; however [START_REF] Jones | Error contour charts relevant to particle sizing by forward-scattered lobe methods[END_REF] and Seville et al. (1984) demonstrated that the error made on the measured size is acceptable, when considering smaller and non-absorbing colloids like in our situation.

For F-0, even if this monomodal distribution was polydisperse (Figure 1), the D90 value, around 7 µm and the D50 value at 2.331 µm both proved that the blank emulsion was mostly composed of small droplets. F-N1 size distribution was very similar to the blank one: only the D10 was significantly lower. NP in the emulsion corresponded to a new population of colloids with a dimension around 0.6 µm. Size distribution obtained for F-N2 highlighted a large distance between the D10 and the D90 values and depicted a high polydispersity.

Moreover, the D50 was five time higher than the mean size value for F-0: F-N2 distribution was thus essentially formed of a new larger population around 10 µm.

As sampling required for the size distribution measurement induced a dilution of emulsions and, as a consequence potential physical modifications compared to the initial droplets sizes in bulk, the microstructure of emulsions was also controlled by optical microscopy. Indeed, the lowest dilution necessary to this method allowed observations closer to the reality even if limited by the resolution around 1 µm. Microscopic observations of F-0 (Figure 2A) confirmed the thinness and the homogeneity of the emulsion and allowed to identify droplets network, appearing as a sort of spider web visible on the picture by alternation of dark areas, build of droplets, and white areas consisting of blank. This phenomenon depicted a highly flocculated system. It justified the sample preparation for the granulometry measurement that reduce the aggregation phenomena and allowed the measurement of primary sizes. Optical microscopy also confirmed that only few aggregates were formed in F-N1 and that the majority of the NP was well-dispersed once in emulsion (Figure 2B). The important droplets network described for F-0 was no longer observed for F-N1. In this work, the high amount of tiny particles located in the continuous phase led to the formation of a new network, consisting of particles and droplets. Moreover TiO 2 -particles coated with silica had negative surface charges [START_REF] Junior | The behavior of zeta potential of silica suspensions[END_REF] that might lead to a repulsion between droplets and prevented flocculation. The optical micrograph of F-N2 (Figure 2C) showed large dark objects, around 10-20 µm, masking droplets and corresponding to the aggregates characterized by the D90 value. Added to an oil-in-water emulsion, hydrophobic particles could not be as well dispersed as N1 and formed large vesicles of cetearyl ethylhexanoate and NP. Here again, F-N2 was less flocculated than F-0 surely because the initial emulsion was less concentrated in oil droplets: vesicles contained almost 40% of the total oil of the emulsion. To conclude, the blank emulsion and the emulsion with hydrophilic NP were very similar in term of size distributions, while F-N2 containing hydrophobic NP was rougher. In both case, the presence of NP seemed to avoid the flocculation phenomenon.

Macroscopic properties.

Emulsions are viscoelastic materials that have both solid and viscous liquid properties. In this study, dynamic oscillatory tests did through a stress sweep (Table 2) bring out these behaviors. On the linear viscoelastic region, the G' of F-0 was much higher than the G" demonstrating the elastic behavior of the emulsions. This led to a tan δ equal to 0.2; this low value corresponds to a strong viscoelastic system. Approaching γ c , emulsion structure kept up until a strong decrease of both moduli. The observed G" overshoot is characteristic of a weak gel (not shown) usually obtained when any gelling agent is added in emulsion continuous phase.

With a similar G' value, F-N1 and F-N2 showed the same predominant elastic behavior as F-0. Even if it was demonstrated that a less flocculated networks or a higher mean size could lead to a lower elastic modulus (Tadros 2004;[START_REF] Pal | Effect of droplet size on the rheology of emulsions[END_REF], in this study, the presence of a solid dispersion offset this fall because it strengthened the network. However, for both emulsions containing NP, tan δ was significantly higher and the G" overshoot was no more observed.

The presence of NP in emulsion gave a system considered as a weaker gel compared to F-0, sign of a reduction of the network strength. Critical strains, γ c , were reduced showing a more polydispersed system: finally, F-N2 appeared less homogenous than F-N1 which was less than F-0.

Texture analyses were performed on emulsions and brought out the same behaviors for F-0, F-N1 and F-N2 during compression test (Table 2). Those results highlighted that the consistency of the three emulsions was the same. Spreading tests performed in order to mimic the application of sunscreen on skin, also highlighted the same behaviors of emulsions. The presence of UV filters only raised the lightness profile of traces (Figure 3) but did not modify the spreading quality. Finally, the slight differences observed during the microstructural characterizations did not impact the macroscopic properties of these emulsions.

Sterilization by high temperature of emulsions.

In this study, a classical sterilization batch method, imitating Appertisation, was chosen and consisted in high temperature sterilization by autoclaving emulsions directly in its storage packaging (glass bottles). The calculated SV was superior to 22 min whereas the minimum value recommended by the European Pharmacopoeia [START_REF] Delaunay | Patent: (WO2013007755) Device and Method for Sterilizing an Emulsion at an Ultrahigh Temperature, Particularly a Cosmetic Emulsion for the Skin, Which Is Unstable at the Sterilization Temperature[END_REF]) is only 15 min, meaning that the treatment was efficient. By this way, three new emulsions were obtained ST-F-0, ST-F-N1 and ST-F-N2. During sterilization, high temperatures decreased the viscosity procured by the gelling agent. After cooling, ST-F-0 and ST-F-N1 restructured spontaneously and recovered a normal emulsion aspect without demixing. Sterilized formula containing N2 was no longer homogenous: a compact block stood in the middle of the glass bottle and was surrounded by a clearer liquid. After few hours cooling down, the viscosities of both phases equilibrated and the batch looked more homogenous even if lumps were still observable.

Microscopic characterization.

The blank emulsion, ST-F-0, and the emulsion with hydrophilic NP, ST-F-N1, had droplet size distributions similar to their non-sterilized versions: only the D90 significantly increased after thermal treatment. Some larger droplets could be observed on the micrograph of ST-F-0 (arrows on Figure 4A) confirming the presence of a new larger droplets population around 10-100 µm (Figure 5). Large objects around 10 µm were present in F-N1 after sterilization (Figures 4B and5) and consisted in both droplets and particles aggregates.

Observations of ST-F-N2 phases (Figures 4C and4D) showed that the block part was constituted of around 500 µm long dark objects probably corresponding to large NP aggregates. The liquid part contained mostly fine droplets and air bubbles which appeared in the formula during the sterilization step as a consequence of the gas dilatation. The sterilized emulsion looked like after an accelerated aging already described by [START_REF] Rossano | Effects of aging on structure and stability of TiO2 nanoparticlecontaining oil-in-water emulsions[END_REF].

These observations highlighted the high heterogeneity of ST-F-N2 that constituted a real barrier to the physicochemical characterizations of the emulsion and particle size measurements were not conducted.

Zeta potential (ZP) measurement gave a mean value that took into account surfaces of both droplets and particles (curves not shown). At pH 7, corresponding to the pH in emulsions after maturation, ZP measured for F-0 was around -30 mV. Presence of NP in F-N1 and F-N2 slightly decreased the ZP intensity at respectively -23 mV and -17 mV. After heat treatment, the measured mean zeta potential in emulsions significantly amplified for F-0 (-43 mV) and F-N1 (-33 mV) indicating that surface rearrangements occurred and increased electrostatic repulsion forces, favorable to the emulsion stability. To sum up, the thermal treatment did not impact the integrity of microstructures of the blank emulsion and of the emulsion containing hydrophilic NP while emulsion with hydrophobic NP was no longer homogeneous.

Macroscopic properties.

As illustrated on Table 2, the G' value measured for ST-F-0 was significantly lower than the previous value obtained for F-0, while the G" value remained the same; thus, the solid behavior of the emulsion dramatically decreased. Moreover, the tan δ increased indicating that the gel behavior was considerably reduced. Complementary manipulations on polymer solutions showed that the G' value was divided by two whereas the G'' remained similar after the sterilization step. The viscoelastic evolutions of the emulsion during heat treatment may result from a decrease of the flocculated network but also from polymer degradation.

Compression tests did not allow differentiating F-0 and ST-F-0 (Table 2): therefore, apparent texture, like firmness of the emulsion, was not impacted by the heat treatment. For ST-F-N1, the G' value was the same as its non-sterilized version whereas the G" value was increased (Table 2). The rise of the tan δ reflected a reduction of the viscoelastic behavior of the emulsion. Moreover, γ c decreased indicating a loss of homogeneity, in relation with the rise of the D90 value.

The maximum force reached during the ST-F-N1 compression test (Table 2) was significantly higher when compared to other emulsions. This result revealed an high firmness for the sterilized emulsion containing hydrophilic NP [START_REF] Friedman | The Texturometer-A New Instrument for Objective Texture Measurement[END_REF], in line with the high value of the viscous modulus. As this phenomenon should be explained by water losses during sterilization inducing a concentration effect of both droplets and particles, the water content of emulsions was evaluated by desiccation before and after the sterilization step but no difference was observed. Then, the increase in consistency and changes in viscoelastic behaviors of ST-F-N1 should not be explained only by the slight changes in size distributions but also by the structural rearrangements as suggested by surface modifications evidence by zetametry.

However, the γ c remained identical in control and sterilized emulsions highlighted that their homogeneity was not impacted by the heat treatment (Table 2). Moreover, spreading tests highlighted that the important consistency of the ST-F-N1 did not affect the spreading quality as seen by comparing F-N1 and ST-F-N1 traces (Figure 2). Nevertheless, the lightness measurement was higher along the ST-F-0 trace than along the F-0 trace. Chantrapornchai et al. already measured the bulk lightness of low concentrated emulsions with monitored flocculation. They demonstrated that in these conditions flocculated droplets decreased the Lvalue [START_REF] Chantrapornchai | Influence of flocculation on optical properties of emulsions[END_REF]. In this present work, as ST-F-0 and F-0 had the same granulometry profiles, ST-F-0 seemed less flocculated than the F-0 formula.

Although ST-F-N2 was not homogeneous, its spreading quality was tested in order to evaluate the impact of lumps in use. Experiments confirmed the heterogeneity of this emulsion: at the beginning of the trace, lightness was similar to that of F-N2 but L* quickly decreased. Previous studies have shown that the lightness of an emulsion decreases with the size of droplets [START_REF] Chantrapornchai | Influence of droplet size and concentration on the color of oil-in-water emulsions[END_REF]. Then, the presence of larger droplets in ST-F-N2 than in the other emulsions could explain the low value of measured lightness. It is also interesting to note that the trace did not end before the maximum mobile displacement and was not well-spread (Figure 6). In fact, the emulsion seemed not to adhere on the hydrophobic surface of PP sheet but remained on the hydrophilic PMMA plate. This phenomenon should be regarded as a first sign of phase separation.

In conclusion, ST-F0 and ST-F-N1 showed macroscopic properties slightly different than their non-sterilized versions. Mainly, the gel behavior was considerably reduced for both However, these emulsions remained homogeneous and their qualities in term of consistency and spreading ability were acceptable. In addition to the presence of lumps in ST-F-N2, the emulsion was less covering compared to other studied emulsions and did not match the properties of sun protection cream. Hence, these results demonstrated that the initial emulsion microbiological charge of emulsions was acceptable and could not be an obstacle to future microbiological assays. For this purpose, only the non-sterilized versions were thereafter tested on bacteria strains.

Assessment of

Bactericidal effect of model emulsions.

Two normal human skin bacteria strains were used: Pseudomonas fluorescens MFP 05, a gram negative ubiquitous bacterium representing more than 90% of the microbial flora in humid skin areas [START_REF] Grice | A diversity profile of the human skin microbiota[END_REF], Staphyloccocus. aureus MFP03, a gram positive bacterium, usually considered as a member of transient human skin flora carried by 35 to 60%

of the human population [START_REF] Percival | Microbiology of the skin and the role of biofilms in infection[END_REF].

When the control emulsion was diluted in LB broth (Figure 7A) nutrient contribution let P.

fluorescens and S. aureus to quickly develop. The growth of MFP03 (slope of the linear part of the curve) appeared more rapid than for MFP05 but this difference should only reflect their optimal growth temperature (37°C and 28°C respectively) and metabolism. In this culture condition, no bactericidal effect of the control emulsion was observed.

Thus, it appears interesting to evaluate the bactericidal effect of the presence of NP. The results highlighted that the NP did not significantly affect the growth of both bacteria. Finally, both strains grew as well in all emulsions, with or without NP. One hypothesis could be that the high nutrient contribution of LB broth hided the effect of the nanoparticle presence as well as their coatings. Therefore, dilution in physiological water was conducted as it only brings nutrients already present in the emulsion. In this second condition, the evolutions of bacteria in the diluted control emulsion were also similar and there was no NP effect (Figure 7B). The growth kinetic of P. fluorescens MFP05 was reduced with a plateau, corresponding to the total biomass formed which was decreased of ten times. P. fluorescens MFP05 is resistant and highly adaptable to environment variations [START_REF] Chapalain | Comparative study of 7 fluorescent pseudomonad clinical isolates[END_REF] as previously shown, emulsions are not appearing as a favorable media for bacteria development (Teixeira et al. 2007). This was confirmed by the behavior of S. aureus MFP03 that was unable to grow in emulsion diluted in physiological water. This bacterium could not adapt and metabolize emulsion components. Its population was undergoing a progressive decay due to the lack of adapted resources.

To conclude, the control emulsion fulfilled the defined criteria in the introduction and this test following the bacteria growth in emulsions highlighted the absence of NP effect.

Conclusion

In this study, formulation development was conducted to provide lab-scale preservative-free sunscreen emulsions that should contain the same major ingredients and differed only by the presence and nature of TiO 2 -NP (NP), an inorganic UV-filter. This development phase showed that TiO 2 -NP could be difficult to implement, depending on their coating. While hydrophilic NP, coated with silica, was easily integrated into the continuous phase of the emulsion and did not change markedly the physicochemical characteristics of the emulsion, hydrophobic NP, coated with alumina (and) triethoxycaprylylsilane induced formation of large aggregates with the other ingredients during the emulsion preparation. Process had therefore to be carefully optimized and controlled in order to succeed in obtaining well dispersed and smooth emulsion. Emulsions containing NP as well as the control one had been characterized by microscopic analyses (laser diffraction granulometry and optical microscopy) and macroscopic analyses (rheology and texturometry) in order to ensure that they shared the same physicochemical properties. Even if these technics highlighted slight differences in terms of microscopic structures between emulsions, other macroscopic analyses such as rheology and texturometry did not showed bulk major differences. This should allow assessing the role of NP in use, regardless the global physicochemical properties of formulae.

We were able to show that the three emulsions fulfilled the main criteria for the evaluation of the effect of NP in formula on the microbiota. One interesting point is that non-sterile formula, developed with specific cautions taken with the formulation material and procedure, showed an initial bacterial charge well below the acceptable limit as defined by the European Pharmacopeia 8.0.

Applicability of high temperature sterilization to cosmetic emulsions and sunscreen emulsions containing inorganic UV-filters was as well evaluated. Although this treatment should be estimated sufficient to induce emulsion demixing, the control emulsion and the emulsion containing hydrophilic NP passed through it. For both emulsions, the change in gel behaviors could not be only explained by the presence of new larger but minor populations;

polymer deterioration and structural rearrangements may also explain these phenomena.

Unfortunately, for emulsions containing hydrophobic NP, sterilization induced the formation of two phases; one formed by aggregated NP and larger droplets, and the other by a liquid phase containing small droplets. Even starting from an emulsion with an optimized dispersion of hydrophobic NP, heat treatment induced dramatic changes for both the microstructure as well as the macroscopic properties of this emulsion. The polymeric coating of hydrophobic NP probably makes this kind of treatment more difficult to adapt and formulation development remains to be done. As an example, additional emulsifiers may be used to stabilize NP during the sterilization step. Nevertheless the high temperature sterilization appears applicable to cosmetic emulsions and sunscreen emulsions containing, or not, inorganic hydrophilic UV-filters.

Finally, in order to verify the usability of these model emulsions, an evaluation of the microbiological effect of NP was performed on non-sterile formula. It revealed that, when bacteria were voluntary inoculated in formulae, all emulsions had the same impact on the growth of two human skin bacteria representative strains, S. aureus and P. fluorescens.

The model emulsions developed in this article are well suited for the evaluation of the 'NP effect' in formula on microbiota, because they hide their other physicochemical aspects whatever the polarity of the coating.

  The efficiency of the sterilization process was evaluated following NFEN ISO 21149:2006[START_REF] Afnor | NF EN ISO 21149:2006 -Cosmétiques -Microbiologie -Dénombrement et détection des bactéries aérobies mésophiles[END_REF]. Culture without enrichment gave the real number of mesophilic aerobic bacteria in emulsions. It was around 100 and 170 UFC/g of product and appears acceptable in regard of the acceptation limit recommended by the European Pharmacopoeia for the nonsterile cutaneous products fixed at 200 UFC/g of product (European Pharmacopoeia 8.0 2014). Petri dishes spread with enriched non-sterilized emulsions were covered with bacteria colonies after incubation. On the contrary, in the case of sterilized emulsions, Petri dishes were devoid of any bacteria development thus proving the efficiency of the heat treatment.

  the same row means significant difference between emulsions for this parameter (p<0.05).

Figure 1 :

 1 Figure 1: Particle size distributions given in volume particle of fresh emulsions.

Figure 2 :

 2 Figure 2: Optical micrograph of; A: F-0, B: F-N1 et C: F-N2. (The scale bar corresponds to

Figure 3 :

 3 Figure 3: Evolution percentage of lightness (L*) along spreading traces.

Figure 4 :

 4 Figure 4: Optical micrograph of; A: ST-F-0, B: ST-F-N1, C: liquid part of ST-F-N2 and D:

Figure 5 :

 5 Figure 5: Particle size distributions given in volume particle of sterilized emulsions.

Figure 6 :

 6 Figure 6: Example of spreading traces, A: for F-N2, B: for ST-F-N2.

Figure 7 :

 7 Figure 7: Bacteria growth in different diluted emulsions (n=3); *emulsions diluted in

Acknowledgements

The authors would like to thank A. Garreau for her contribution in the microbiological part. This project is supported by the Région Haute-Normandie (GRR SéSa), the Conseil Général de l'Eure and EU (FEDER). LR is recipient of a Doctoral fellowship from the Région Haute-Normandie (GRR SéSa). This research program is realized in the framework of the PFMI Cosmétomique Haute-Normandie: Sureté, Innocuité des Produits Cosmétiques (PFMI cosmétomique SIP) supported by the network Cosmetic Valley and of the GDR CNRS Cosmactif.


O Ghaleb
email: omar.ghaleb@carleton.ca

B John Oommen
email: 2oommen@scs.carleton.ca

Learning Automata-based Solutions to the Single Elevator Problem

Keywords: Learning Automata (LA), Learning Systems, Single Elevator Problem (SEP), Elevator-like Problem (ELP), Parking Problem

The field of AI has been a topic of interest for the better part of a century, where the goal is to have computers mimic human behaviour. Researchers have incorporated AI in different problem domains, such as autonomous driving, game playing, diagnosis and security. This paper concentrates on a subfield of AI, i.e., the field of Learning Automata (LA), and to use its tools to tackle a problem that has not been tackled before using AI, namely the problem of the optimally scheduling and parking of elevators. In particular, we are concerned with determining the Elevators' optimal "parking" location. In this paper, we specifically work with the Single 1 Elevator Problem (SEP), and show how it can be extended to the solution to Elevator-like Problems (ELPs), which are a family of problems with similar characteristics. Here, the objective is to find the optimal parking floors for the single elevator scenario so as to minimize the passengers' Average Waiting Time (AWT). Apart from proposing benchmark solutions, we have provided two different novel LAbased solutions for the single-elevator scenario. The first solution is based on the well-known LRI scheme, and the second solution incorporates the Pursuit concept to improve the performance and the convergence speed of the former, leading to the P LRI scheme. The simulation results presented demonstrate that our solutions performed much better than those used in modern-day elevators, and provided results that are near-optimal, yielding a performance increase of up to 80%.

Introduction

Right from its infancy, the goal of the field of Artificial Intelligence (AI) has been to make computers respond intelligently in everyday and challenging situations. Turing, in his Turing test, suggested that a machine could be considered to possess AI if a human observer would not have been able to distinguish its behavior from the behavior of a real human being. This lofty goal has been achieved to a phenomenal degree. AI-based computer programs have challenged and beaten the best players in many games, including Chess and Go. This paper considers a field in which AI has not been applied much. Consider a tennis player who is moving within his side of the court. After he hits the ball, the question that he encounters is to know where he should place himself so as to best counter his opponent's response. This can be modelled as a parking problem, i.e., where should the player "park" himself so that when his opponent hits the ball, it is in the vicinity of where he has parked. We could refer to problems of this type as "Elevator-like" Problems (ELPs). They are, in fact, in a unidimensional domain, related to the problem of where an elevator within a building should be parked. Thus, if there is a building with n floors and if the elevator car is requested at floor i, where should the car move to after the passenger is dropped off? While this depends on the criterion function, it is expedient to work with the understanding that the car should be parked near to the next passenger call. We can extend the problem to consider where emergency vehicles should be parked so as to be readily available for the next call.

Although the above problems are, in general, transportation problems, their common facet can also be extended to other domains. For example, one could consider the problem of where the read/write disk head in a memory bank should be placed so that it can access data more expeditiously. Indeed, our problem model can be extended to consider improving underwater communication systems by determining where the underwater sensors should be located.

In this paper, we refer to all of these problems as ELPs, and this is the primary focus of the research. However, to render the problem to be non-trivial, we assume that "the world" in which we are operating is stochastic and that the underlying distributions are unknown. For example, in the case of the tennis player, we assume that there is a distribution for the place where the opponent will place his counter-shot, but that this distribution is unknown.

We will discuss the Single Elevator Problem (SEP) (also please see [START_REF] Das | Cross-scene trajectory level intention inference using Gaussian process regression and naive registration[END_REF] and [START_REF] Molina | An ACO model for a non-stationary formulation of the single elevator problem[END_REF]) and explain the solutions by which researchers have tackled it in the past. The SEP is a subclass of the Multi-Elevator Problem (MEP) in which we have a building with n floors and a single elevator, and where we are to design a policy for the elevator in order to operate in such a way so as to save energy, reduce the travel time or the waiting time for passengers. Moreover, the elevator policy should decide how the elevator should operate in order to achieve its goal. For example, one possible policy could require that the elevator picks the best route in order to pick passengers, while another could be to serve the longest queue first or to decide where the elevator should wait for the next call. As we shall see, the literature does not record several solutions for the SEP. The reason for this is that most papers deal with the more complex scenario, the MEP, which is a generalization of the SEP.

Prior Art for the SEP

Two notable contributions2 for the SEP were by Tanaka et al. in [START_REF] Tanaka | Dynamic optimization of the operation of single-car elevator systems with destination hall call registration: Part I. Formulation and simulations[END_REF][START_REF] Tanaka | Dynamic optimization of the operation of single-car elevator systems with destination hall call registration: Part II. The solution algorithm[END_REF]. In their papers, they studied the operation problem of a single-car elevator with so-called "destination hall call registration", where the system registers the passengers' destination floors before boarding the elevator. This is opposed to regular or "traditional" elevators, where passengers can only pick the direction of their trip before boarding, and thereafter, specify the destination floor after boarding.

In the first part of their study [START_REF] Tanaka | Dynamic optimization of the operation of single-car elevator systems with destination hall call registration: Part I. Formulation and simulations[END_REF], the authors tried to answer two questions, namely: (a)"Can (single) car operations be improved with destination hall call registration?", and (b)"How can it be realized?". They were able to formulate the operation problem as a Dynamic Optimization Problem (DOP) such that an objective function, which is the weighted average waiting time for passengers, is minimized. They showed that the DOP substantially improved the system's capability when compared to the conventional "selective collective" operation.

The authors of the second part [START_REF] Tanaka | Dynamic optimization of the operation of single-car elevator systems with destination hall call registration: Part II. The solution algorithm[END_REF] introduced a branch-and-bound algorithm to solve the formulation of the DOP problem [START_REF] Tanaka | Dynamic optimization of the operation of single-car elevator systems with destination hall call registration: Part I. Formulation and simulations[END_REF]. Here, the lower bound calculations for the subproblems generated in the course of the branch-and-bound algorithm, came from decomposing the problem into three subproblems, i.e., the passenger loading/unloading, car stops, and lastly the floor-to-floor travel. They then applied the Lagrangian relaxation method to solve the overall problem. Unfortunately, their algorithm was not fast enough for real-life situations.

After the studies in [START_REF] Tanaka | Dynamic optimization of the operation of single-car elevator systems with destination hall call registration: Part I. Formulation and simulations[END_REF], [START_REF] Tanaka | Dynamic optimization of the operation of single-car elevator systems with destination hall call registration: Part II. The solution algorithm[END_REF], the authors [START_REF] Tanaka | A study on objective functions for dynamic operation optimization of a single-car elevator system with destination hall call registration[END_REF] examined how one could improve the efficiency of a single elevator system with "destination hall call" by considering dynamically optimized objective functions. Here, they applied simulated annealing, and from their results they showed that the weighted average of two different objective functions, i.e., the weighted average service time and the maximum, yielded a better performance than by using only a single objective function. They also observed was that one had to choose the weights of the objective functions carefully because this significantly affected the results of the experiments. Moreover, these weights depended on the elevator's characteristics.

The authors of [START_REF] Sun | Estimation of optimal elevator scheduling performance[END_REF] addressed the goal of estimating the optimal values for the upper and lower bounds for the elevator scheduling problem, in which they assumed the availability of all the information about the passengers. To achieve this, they formulated the problem in two parts, a high level and a low level component. The high level component was a passenger-to-car assignment, and the low level component was the passenger-to-trip assignment. This then was used for the formulation of the low-level component. The authors obtained the upper bound, by finding a reasonable solution to the problem. They thus obtained the lower bound, by defining a lower bound for a newly-constructed problem using the Lagrangian Relaxation method. Their results were efficient and scalable.

Another work, presented by Xu and Feng [START_REF] Xu | Single Elevator Scheduling Problem with Complete Information : An Exact Model using Mixed Integer Linear Programming[END_REF], modelled the single elevator scheduling problem as a Mixed Integer Linear Program (MILP). They focused on using this model to improve the service time for passengers. Based on a prototype model obtained from the industry for the dynamics of a moving elevator, they linearized the nonlinear travel activities for the problem. They tested their "realtime" model under different circumstances and scenarios, and conjectured that their model could be extended and used as a benchmark.

Learning Automata (LA)

We now briefly review3 the field that we shall work in, namely, that of LA. The concept of LA was first introduced in the early 1960's in the pioneering research done by Tsetlin [START_REF] Tsetlin | On behaviour of finite automata in random medium[END_REF]. He proposed a computational learning scheme that can be used to learn from a random (or stochastic) Environment which offers a set of actions for the automaton to choose from. The automaton's goal is to pick the best action that maximizes the average reward received from the Environment and minimizes the average penalty. The evaluation is based on a function that permits the Environment to stochastically measure how good an action is, and to, thereafter, send an appropriate feedback signal to the automata.

After the introduction of LA, different structures of LA, such as the deterministic and the stochastic schemes, were introduced and studied by the famous researchers Tsetlin, Krinsky and Krylov in [START_REF] Tsetlin | On behaviour of finite automata in random medium[END_REF] and Varshavskii in [START_REF] Varshavskii | On the behavior of stochastic automata with a variable structure[END_REF].

The field of LA, like many of the Reinforcement Learning techniques, has been used in a variety of problems, mainly optimization problems, and in many fields of AI. Among other applications, it has been used in neural network adaptation [START_REF] Meybodi | New learning automata based algorithms for adaptation of backpropagation algorithm parameters[END_REF], solving communication and networking problems [START_REF] Obaidat | Learning automata-based bus arbitration for shared-medium ATM switches[END_REF], [START_REF] Oommen | Continuous learning automata solutions to the capacity assignment problem[END_REF], in distributed scheduling problems and in the training of Hidden Markov Models [START_REF] Kabudian | Applying continuous action reinforcement learning automata (carla) to global training of hidden markov models[END_REF].

In Figure 1, we have the general stochastic learning model associated with LA. The components of the model are the Random Environment, the Automaton, the Feedback received from the environment and the Actions chosen by the automaton. The Environment: We first define the stochastic Random Environment that the automaton interacts with. Initially, the automaton picks an action from the set of actions available to it and communicates it to the environment. The Environment evaluates that action according to a random function, and sends back a reward or a penalty feedback signal to the automaton.

The Automaton: The LA achieves its learning process as an iterative operation that is based on the Environment and the interaction between them. This process consists of two main steps, the first of which is policy evaluation, which is how the Environment evaluates the selected action. The second step involves policy improvement, where the LA improves the probability of selecting an action that would maximize the reward received from the Environment. Estimator and Pursuit Algorithms: The concept of Estimator Algorithms was introduced by Thathachar and Sastry in [START_REF] Thathachar | A new approach to the design of reinforcement schemes for learning automata[END_REF][START_REF] Thathachar | A class of rapidly converging algorithms for learning automata[END_REF] when they realized that the family of Absolutely Expedient algorithms would be absorbing, and that they possessed a small probability of not converging to the best action. Estimator algorithms were initially based on Maximum Likelihood (ML) estimates (and later on Bayesian Estimates), where they also used the estimates of the reward probabilities to update the actions' probabilities. By doing this, the LA converged faster to the actions that possessed the higher reward estimates. The original Pursuit Algorithms are the simplest versions of those using the Estimator paradigm introduced by Thathachar and Sastry in [START_REF] Thathachar | Estimator algorithms for learning automata[END_REF] and [START_REF] Thathachar | A new approach to the design of reinforcement schemes for learning automata[END_REF]. These algorithms are based on the pursuit strategy, where the idea is to have the algorithm pursuing the best-known action based on the corresponding reward estimates. The pursuit algorithms were proven to be ǫ-optimal. The concept of designing ǫoptimal discretized Pursuit LA was introduced by Oommen and Lanctot in [START_REF] Oommen | Discretized pursuit learning automata[END_REF]. The discretized Pursuit LA converged faster than their continuous counterparts.

Contributions of this Paper

The novel contributions of this paper are:

-We have surveyed a subfield of AI, i.e., the field of Learning Automata (LA),

and have concluded that it has not been used previously to solve the SEP. -We were able to identify two different models of computations for the elevator problem, where the first requires the calling distribution to be known a priori, and the second does not need such information.

-We were able to model the elevator problem in such a way that it can be solved using LA approaches. -We introduced a Linear Reward-Inaction (L RI )-based solution to tackle the single elevator problem. It has been referred to as SEP3. -We also presented an improvement on SEP3 by including the so-called Pursuit phenomenon into the LA solution. This led to the P L RI -based solution, referred to as SEP4, and this yielded better results and faster convergence than SEP3. -We have shown that LA-based solutions can solve elevator-like problems without requiring any knowledge of the underlying distributions. Amazingly enough, the results and solutions that they yielded are near-optimal.

2 The Single Elevator Problem (SEP)

The problem we are trying to tackle can be stated as follows: We have a specific building with n floors and a single elevator. The floors and passengers are characterized by distributions C = {c 1 , ..., c n } and D = {d 1 , ..., d n }, where c i is the probability of receiving a call from floor i, and d j is the probability that the elevator drops the passenger off at floor j. These distributions are unknown to the decision-making algorithm, and our goal is to design LA-based solutions such that they adaptively determine a set of floors for the e to park at during the idle period so as to minimize the passengers' waiting time.

The metric used as a performance measure in our study is the Average Waiting Time (AWT) of the passengers. This is, clearly, a function of the number of floors, and of also how close the converged solution is to the optimal floor.

Competitive Solutions

The "Do Nothing" Policy: SEP1 The "Do Nothing" policy, referred to as SEP1, is the one where the systems does nothing after dropping off a passenger. It is formally presented in [START_REF] Ghaleb | Novel Solutions and Applications to Elevator-like Problems[END_REF] and [START_REF] Ghaleb | On Solving Single Evelatior-like Problems using Learning Automata[END_REF]. As we can observe from the algorithm, the simulation starts by selecting a random initial parking floor for the elevator. This is done in an equiprobable manner. We then simulate a number of passenger calls that require the elevator to go to the calling floor, dropping-off the passenger at the destination floor, and parking the elevator car at the same destination floor while it waits for the next passenger.

To evaluate the AWT, we calculated the waiting time for each call which, as mentioned, is used to evaluate the performance of the policy. This is done by using the following equation:

W T = θ * |calling f loor -parking f loor|, (1)
where θ is a parameter characterizing the pace variable, and which differs from one system to another because of different shaft speeds and the associated accelerations/decelerations of the elevator shafts. Since this is constant for all the simulations, we ignored this pace parameter and focused on the travel distance from the parked location to the floor where the call is made, |calling f loorparking f loor|, which does not change for different elevator systems even if their paces are different. By doing this, the model would be generalized so that it can be applied to different elevator systems since it will be system independent. This yields the final waiting time equation to be:

W T = |calling f loor -parking f loor|, (2)
which is used to calculate the waiting time for all the policies studied here.

Myopic Policy: SEP2

The second policy, SEP2, was based on the model that was proposed in [START_REF] Parlar | Optimal parking of idle elevators under myopic and state-dependent policies[END_REF], referred to as a "Myopic Policy". The principle motivating it is that the model selects a predetermined floor that the elevator waits at for the next call. Using a complete knowledge of the relevant distributions, SEP2 pre-computes the best possible floor that the elevator should park at, so as to minimize the AWT. It uses the Call Distribution, C, to determine that floor.

The main simulation follows the same process as the SEP1 policy in Section 2.1, where the elevator receives a call and then picks the passenger up at that floor and drops the passenger off at the destination. It then moves to the predetermined floor to wait for the next call. The difference between SEP1 and SEP2 lies in the selection of the parking floor policy, where, instead of waiting at the drop-off floor, it moves to the pre-determined parking floor where it will wait for the next call. The corresponding formal algorithm is in [START_REF] Ghaleb | Novel Solutions and Applications to Elevator-like Problems[END_REF] and [START_REF] Ghaleb | On Solving Single Evelatior-like Problems using Learning Automata[END_REF], omitted here in the interest of space. This algorithm also shows the "optimal floor" selection scheme used to compute the optimal parking floor. This is done by exhaustively searching across all the floors so as to compute which floor produces the minimum expected waiting time as per:

T (f ) = n y=1 |y -f | * g(y), ( 3
)
where f is the floor selected as a parking floor, n is the number of floors in the building, g(y) is the probability of receiving a call from the floor y.

The main disadvantage of this policy is that it requires the a priori knowledge of C so as to calculate the expected waiting time for each floor.

LA-Based Solutions

In this section, we present our proposed LA-based solutions for the SEP. First, we will show how we have modelled the problem, and thereafter we present an L RI -based solution to the problem. Subsequently, we submit an enhancement to the L RI solution, in which we use the Pursuit concept for the L RI , and this yields the second and even better solution, which is the P L RI -based solution.

Problem Modelling

Before we present our proposed solutions, we need to explain how the problem was modelled so that it could be solved using an LA approach. As mentioned in Section 1.2, any LA structure consists of an Environment and the LA itself. The LA chooses one of the actions it is offered, i.e., one that is relevant to the problem domain, and then the Environment evaluates it and reacts based on the criterion it uses by responding with a reward or penalty feedback to the LA.

In the SEP, the modelling is much simpler than in the MEP investigated in [START_REF] Ghaleb | Learning Automata-based Solutions to the Multi-Elevator Problem[END_REF]. In the SEP, we modelled the floors as the actions of the LA, which, hopefully, will eventually converge to one that can be reckoned as the best parking floor. We then modelled the Environment so that it could provide us with the feedback about whether the selected floor was good or bad. In the former case the decision was rewarded, and in the latter, it was penalized.

To achieve this, we divided the SEP into two different parts, namely the controller and the evaluator. When it concerns the LA-based solution, we can say that the controller is the LA and the evaluator is the Environment, which evaluates the action selected by the controller or the LA.

The LA acts as the elevators' controller, which chooses one of the available floors or (actions) where the elevator will park at to wait for the next passenger call. On the other hand, the Environment evaluates the selected floor based on the objective or fitness function that is specified. One thing to note here is that the LA, or the controller, does not know anything about the distribution of passengers' calls, C, unlike the solutions presented earlier.

L RI -based Solution: SEP3

Our first proposed solution, referred to as SEP3, is based on the L RI scheme [START_REF] Narendra | Learning Automata: An Introduction Prentice-Hall[END_REF], in which the LA updates the actions probabilities when it receives a reward from the Environment, and it does nothing when it receives a penalty. Based on the theory of LA, the L RI scheme helps us to achieve a near-optimal solution by updating the action probabilities to converge towards the best possible solution, which, in our case, is the best parking floor.

The corresponding algorithm is presented in [START_REF] Ghaleb | Novel Solutions and Applications to Elevator-like Problems[END_REF] and [START_REF] Ghaleb | On Solving Single Evelatior-like Problems using Learning Automata[END_REF], omitted here in the interest of brevity. Initially, when the simulation of the experiments begins, the LA begins by selecting one of the available floors (actions) as the initial parking floor with equal probability and sends the selected floor to the Environment. Once the Environment receives the selected floor, it evaluates it based on the passengers' AW T from the start of the simulation until that time instance. If the Environment determines that the waiting time is less than or equal to the current AW T , it sends a reward feedback to the LA, informing it that it was a good choice. Otherwise, it sends a penalty feedback.

Once the LA receives the feedback, it checks whether it was a reward or a penalty. If it was a reward, the LA updates the probabilities of the floors according to the previously selected action by increasing the probability of the selected floor and decreasing the probabilities of the rest. On the other hand, if it was penalty feedback, the probabilities remain as they are -with no change.

The LA then selects a new parking floor based on the updated distribution, and repeats the process until it, hopefully, converges to the best parking floor.

P L RI -based Solution: SEP4

The second LA-based solution, referred to as SEP4, is an improvement on the L RI -based solution, SEP3, and it aimed to achieve an even better performance and faster convergence. To design it, we included the Pursuit phenomenon described in [START_REF] Oommen | Discretized pursuit learning automata[END_REF], [START_REF] Thathachar | A new approach to the design of reinforcement schemes for learning automata[END_REF] and [START_REF] Thathachar | Estimator algorithms for learning automata[END_REF]. This allowed the LA to pursue the action with the superior reward ratio rather than just updating the selected action. The formal algorithm is shown in [START_REF] Ghaleb | Novel Solutions and Applications to Elevator-like Problems[END_REF] and [START_REF] Ghaleb | On Solving Single Evelatior-like Problems using Learning Automata[END_REF], and omitted here due to space limitations.

One can observe that it executes in the same manner as SEP3, but it differed when achieving the task of updating the action probabilities. For every action, it accomplished this by keeping track of the ratio of the rewards obtained to the number of times the action was selected. Consequently, we introduced the vector of reward estimates to keep track of the reward ratio.

The simulation started by selecting each floor a small number of times (i.e., ten times each in our case), and recording the ratio of how many times each floor received a reward when compared to the number of times it was selected. Thereafter, the simulation proceeded also the same lines as the previous SEP3. The system first receives a call from a passenger, and it calculates the waiting time. It then requests the Environment for the feedback. The Environment evaluates the selected floor, and it uses the average waiting time to decide on the feedback sent to the LA.

If the LA gets a reward, it updates the probabilities of the parking floors. Instead of increasing the probability of selected floor, it pursues the one with the maximum reward estimate and increases its probability. This mechanism ensures that the algorithm converges towards the best solution faster than SEP3.

After that, the LA updates the rewards' estimates and again chooses a parking floor to be evaluated, and repeats the same cycle until it converges.

Results and Discussions

We now comparatively discuss the simulations results obtained from the previous solutions and our new solutions. The entires suite of results involve all the methods SEP1, SEP2, SEP3 and SEP4, and for many distributions, namely the Exponential, Inverse Exponential, Gaussian and Bimodal, represented by Exp, InvExp, Gaussian and Bimodal respectively. The results that we have obtained are extensive and involve all these distributions and for numerous settings. These are given briefly in [START_REF] Ghaleb | On Solving Single Evelatior-like Problems using Learning Automata[END_REF], but the more detailed set of results is included in the respective chapter and the Appendix of the thesis of the First Author [START_REF] Ghaleb | Novel Solutions and Applications to Elevator-like Problems[END_REF].

A very brief summary of the results are given in Table 1, where we submit all the results of the four solutions, for some typical settings and parameters. From the table, we see that SEP1, which we believe is the most popular policy currently used in buildings, performed very poorly in comparison to our proposed solutions. The improvement in the AWT was more than 50% and up to 80%. SEP1 is a lower bound for all schemes, as no solution should be worse than this.

SEP2 was able to give us the optimal parking floors from the beginning, but it required the a priori knowledge of C for each floor. However, our LA-based solutions were able to achieve a close-to-optimal AW T without this knowledge.

SEP3 yielded a performance that was better than SEP1 and recorded an AW T in 78.56% for the Exp, 77.85% for the InveExp, 32.49% for the Gaussian and 30.9% for the Bimodal . Moreover, the results were close to the values achieved by SEP2.

We attempted to improve SEP3 by proposing SEP4 that incorporated the Pursuit concept. Here, we achieved even better results than what we obtained for the SEP3. The algorithm helped the system to converge faster to the optimal locations. It also resulted in a better overall AW T . The improvements were 26.3% for the Exp, 29.5% for the InvExp, 3% for the Gaussian and 1% for the Bimodal distributions.

In Figure 2, we present the performance of each algorithm and show how our proposed LA-based algorithms were able to achieve an AWT that is very close to the optimal solution, SEP2, and how it significantly outperforms SEP1.

One result that we discovered is that the more equally-distributed the calls are, for example, for the Gaussian distribution, the higher the AW T will be. On the other hand, the importance of having a good policy shines when the calling distribution is skewed toward a specific region or a specific floor. Also, one can observe how the policies affected the AW T in the first and the second scenarios, i.e., for the Exp and InvExp distributions. 1: Simulation results for the previous and newly proposed solutions for the SEP for an ensemble of 200 experiments for the 12-floor settings. The results are given here as a tuple (α, β) where the first field, α, is the best optimal parking floor and the second field, β is the AW T in terms of number of floors travelled for the elevator to reach the passenger from the parked location.

Conclusions

In this paper, we considered the problem of determining the best parking location for the Single Elevator Problem (SEP). We surveyed various parking policies that are currently being used in various building settings, and for previously-reported solutions. We discussed two different solutions for the SEP and tabulated the experimental results that corresponded to various simulations settings. We then introduced our Learning Automata (LA)-based solutions, namely the L RI -based solution, SEP3, which achieved the optimal parking floor without knowing C a priori. We proceeded to introduce an improvement on SEP3 to incorporate the Pursuit concept in SEP4, a P L RI -based solution. From the simulation results for SEP4, we demonstrated that it performed even better than SEP3.

This paper included our findings and compared the results of our proposed solutions to the previous solutions, while extensive additional results are found in [START_REF] Ghaleb | Novel Solutions and Applications to Elevator-like Problems[END_REF] and [START_REF] Ghaleb | On Solving Single Evelatior-like Problems using Learning Automata[END_REF]. We showed that our LA-based solutions performed much better than SEP1 and converged to the optimal floor. They also reduced the AW T to be close to the optimal value with the advantage that they did not require us to know the distributions to determine the best floor.

The case in which the building use multiple elevators operating in the same environment will be presented in a future paper [START_REF] Ghaleb | Learning Automata-based Solutions to the Multi-Elevator Problem[END_REF].

Fig. 1 :

 1 Fig. 1: The Automaton-Environment Feedback Loop.

Fig. 2 :

 2 Fig. 2: The Average Waiting Time for SEP1, SEP2, SEP3 and SEP4 for an ensemble of 200 experiments for the case of the Exp distribution, given in the graph as P 1, P 2, L RI , P L RI respectively.

  InvExp (-, 5.377) (12, 0.751) (11, 1.191) (12, 0.839) Gaussian (-, 3.607) (6, 2.131) (7, 2.435) (6, 2.364) Bimodal (-, 3.707) (8, 2.251) (8, 2.560) (8, 2.533)

	Dist	SEP1	SEP2	SEP3	SEP4
	Exp	(-, 5.364) (1, 0.712) (2, 1.150) (1, 0.847)
	Table

The review presented here is necessarily brief. A more detailed survey of the field is included in[START_REF] Ghaleb | Novel Solutions and Applications to Elevator-like Problems[END_REF] and[START_REF] Ghaleb | On Solving Single Evelatior-like Problems using Learning Automata[END_REF].

We will not go through irrelevant details and/or the proofs of the LA-related claims. They can be found in[START_REF] Ghaleb | Novel Solutions and Applications to Elevator-like Problems[END_REF] and[START_REF] Ghaleb | On Solving Single Evelatior-like Problems using Learning Automata[END_REF], and the reference book by the pioneers[START_REF] Narendra | Learning Automata: An Introduction Prentice-Hall[END_REF].