Christel Pierlot

Hanyu Hu

Charles Reeb

Jordan Bassetti

Matthieu Bertin

David Lambertin

Catherine Davy

Véronique Nardello-Rataj

Selection of suitable surfactants for the incorporation of organic liquids into fresh geopolymer pastes

Geopolymers (GP) have recently emerged as suitable binders for the conditioning of radioactive organic liquids (OL). The condition for suitably incorporating high amounts of OL into GP is assessed in this research. A series of OL has been selected, comprising alkanes (dodecane and 2 paraffinic mineral oils), an industrial gear oil and tributylphosphate (TBP), in order to cover a wide range of viscosities (0.0018 to 2.33 Pa.s). The incorporation of viscous OL (> 0.2 Pa.s) into fresh GP results in a regular torque increase, indicating a suitable process with the dispersion of the OL in the form of fine micrometric droplets. When the torque remains stable, for OL of low viscosity (< 0.2 Pa.s) like TBP/Dodecane mixtures, the incorporation is not possible, or only partial in the form of large droplets quickly coalescing and leading to an organic supernatant above the hardened GP.

For low viscosity OL, surfactants are required. The use of quaternary ammoniums (QAs) like cetyltrimethylammonium bromide (CTAB) has been studied. Torque monitoring shows that the OL is incorporated similarly in the GP slurry, whether the surfactant is introduced at the beginning, through the course or at the end of the mixing process. The torque evolution also provides the optimum surfactant concentration, by evidencing that beyond a certain concentration, the medium hardens too quickly, preventing the incorporation of OL. Finally, torque monitoring highlights that QAs with shorter alkyl chains, such as hexamethyltrimethylammonium, are able to replace CTAB with the same efficiency. Therefore, the powerful action of CTAB on the incorporation of alkanes into GP is attributed to its positive charge rather than to its surfactant effect.

Introduction

Oil incorporation into geopolymer suspensions

Geopolymers (GP) are inorganic aluminosilicate materials synthesized at room temperature by the activation of a solid aluminosilicate precursor (e.g. metakaolin, fly ash F) in alkaline conditions (fresh GP) ((( [START_REF] Davidovits | Geopolymers: inorganic polymeric new materials[END_REF][START_REF] Provis | Geopolymers: Structure, Processing, Properties and Industrial Applications[END_REF]))) .The resulting mixture hardens to form a solid material (hardened GP). Mixing organic liquids (OL) with such aqueous suspensions [START_REF] Bai | Processing, properties and applications of highly porous geopolymers: A review[END_REF][START_REF] Davidovits | Geopolymers: inorganic polymeric new materials[END_REF][START_REF] Duxson | Geopolymer technology: The current state of the art[END_REF][START_REF] Provis | Geopolymers: Structure, Processing, Properties and Industrial Applications[END_REF] has been recently described for the synthesis of porosity-controlled media, thermal insulating foams or filtration supports [START_REF] Bai | High strength metakaolinbased geopolymer foams with variable macroporous structure[END_REF][START_REF] Barbosa | Preparation of mesoporous geopolymer using metakaolin and rice husk ash as synthesis precursors and its use as potential adsorbent to remove organic dye from aqueous solutions[END_REF][START_REF] Cilla | Geopolymer foams obtained by the saponification/peroxide/gelcasting combined route using different soap foam precursors[END_REF][START_REF] Glad | Highly Porous Geopolymers Through Templating and Surface Interactions[END_REF][START_REF] Medpelli | Geopolymer with Hierarchically Meso-/Macroporous Structures from Reactive Emulsion Templating[END_REF] -and for the treatment of radioactive wastes [START_REF] Cantarel | Geopolymer assembly by emulsion templating: Emulsion stability and hardening mechanisms[END_REF][START_REF] Cantarel | Solidification/stabilisation of liquid oil waste in metakaolin-based geopolymer[END_REF]. Vegetable oils [START_REF] Bai | High strength metakaolinbased geopolymer foams with variable macroporous structure[END_REF][START_REF] Cilla | Geopolymer foams obtained by the saponification/peroxide/gelcasting combined route using different soap foam precursors[END_REF] were also used as the organic phase. Because of the strong alkaline medium, the triglycerides contained in the vegetable oils are converted into glycerol and sodium salt of carboxylic acids, which act as surfactants during GP stirring. Therefore, unlike OL containing alkanes, incorporation of vegetable oils into aqueous geopolymer suspensions does not cause any process issue.

Two-steps oil incorporation process

With the aim of immobilizing radioactive organic wastes, alkanes [START_REF] Cantarel | Geopolymer assembly by emulsion templating: Emulsion stability and hardening mechanisms[END_REF][START_REF] Lambertin | Production of geopolymer composites comprising organic phase change materials[END_REF] and paraffin oil [START_REF] Liu | Foam geopolymer for tunnel composite lining and preparation method thereof[END_REF] have been used as model oils. In [START_REF] Cantarel | Geopolymer assembly by emulsion templating: Emulsion stability and hardening mechanisms[END_REF][START_REF] Lambertin | Production of geopolymer composites comprising organic phase change materials[END_REF], the two-steps incorporation process first requires the formulation of an oil in alkaline water emulsion at high shear rate in a first reactor, followed by its incorporation into an aqueous and viscous metakaolin suspension in a second reactor.

Even if this process allows an efficient incorporation up to 20%v of OL in GP, by providing oil droplets of about 10 to 100 μm, this two-steps protocol is hardly applicable at the industrial scale.

Moreover, the surfactant choice to form the initial emulsion is critical, since this conditions not only the stability of the emulsion, but also the easiness of incorporating the emulsion in the GP suspension. Among the different surfactants that have been tested, alkyl quaternary ammoniums, and in particular cetyl trimethyl ammonium bromide (CTAB), appears as a promising emulsifier. When using CTAB, the viscosity of the fresh hexadecane/geopolymer composite pastes at 20% by volume of hexadecane is between 100 and 1000 times greater than the viscosity of the fresh geopolymer free from alkane [START_REF] Cantarel | Etude de la synthèse de composites liquides organiques/géopolymère en vue du conditionnement de déchets nucléaires[END_REF].

One-step oil incorporation process

Former studies highlight that when paraffin oil is dispersed into a GP slurry, the suspension is not stable with time [START_REF] Thakur | Fabrication of interconnected macroporosity in geopolymers via inverse suspension polymerization[END_REF]. However, it is worth noting that it is possible to incorporate motor oils [START_REF] Cantarel | Solidification/stabilisation of liquid oil waste in metakaolin-based geopolymer[END_REF][START_REF] Davy | 3D structure of oil droplets in hardened geopolymer emulsions[END_REF] in a GP cement directly, in a one step process, without using any surfactant. Even if no details are given concerning the composition of these industrial oils, which probably contain additives like surfactants with some emulsifying and/or stabilizing properties, it was the first time that a one-step process had been successfully used. [START_REF] Cantarel | Solidification/stabilisation of liquid oil waste in metakaolin-based geopolymer[END_REF] noted an increase in viscosity during the incorporation of the oil.

Offline rheological measurements

Most of the time, rheological measurements are performed offline, which requires to transfer the fresh GP from the reactor to the measurement cell of the rheometer. The rheological properties of these fresh pastes are generally determined using rotational rheometers with a parallel plate [START_REF] Sun | 3D extrusion free forming of geopolymer composites: Materials modification and processing optimization[END_REF], a cone plate [START_REF] Dusserre | Rheology evolution of a geopolymer precursor aqueous suspension during aging[END_REF] or a bowl with an helical steel ribbon impeller [START_REF] Steins | Structural Evolution during geopolymerization from an Early Age to Consolidated Material[END_REF] type geometry. Offline rheological measurements can be carried out using several sample solicitations. As an example, by simply measuring the evolution of viscosity versus time on a scale of several hours, the setting time of geopolymers can be estimated [START_REF] Autef | Influence of metakaolin purities on potassium geopolymer formulation: The existence of several networks[END_REF].

Moreover, typical rheological shear stress/shear rate curves also provide apparent viscosity, yield stress and plastic viscosity. In Sun et al. [START_REF] Sun | 3D extrusion free forming of geopolymer composites: Materials modification and processing optimization[END_REF], these last two basic parameters have been characterized to determine to what extent geopolymers meet the requirements for extruding and molding processes.

In [START_REF] Dusserre | Rheology evolution of a geopolymer precursor aqueous suspension during aging[END_REF], thixotropic curves of fresh geopolymers have been studied using a cone plate geometry. From the interpretations of the rheological changes with time, a model for the dissolution mechanism of metakaolin followed by polycondensation is proposed.

Oscillatory rheological experiments allow the determination of viscoelastic parameters (G′, G″, and tan δ). With these information, Steins et al. [START_REF] Steins | Structural Evolution during geopolymerization from an Early Age to Consolidated Material[END_REF] have shown that a rigid percolating network in fresh GP occurs more quickly with alkali activator. Hussain et al. [START_REF] Hussain | Synthesis and thermal behavior of inorganic-organic hybrid geopolymer composites[END_REF][START_REF] Hussain | Investigation of thermal and fire performance of novel hybrid geopolymer composites[END_REF] have shown that the relaxation strength (tan δ ) is correlated to the glass transition temperature (Tg) of geopolymer-modified composites. In Rouyer et al. [START_REF] Rouyer | Evidence of a Fractal Percolating Network During Geopolymerization[END_REF], the elastic and viscous moduli are also measured to investigate the percolation and aggregation mechanisms during geopolymer paste hardening.

Online rheological measurement

To our knowledge, there are not many articles dealing with on line rheological measurement, and even less when it is related to oil incorporation in fresh geopolymer.

In some cases [START_REF] Autef | Influence of metakaolin purities on potassium geopolymer formulation: The existence of several networks[END_REF], rheological measurements are performed using an unconventional geometry to facilitate mixing, using a bowl and a helical steel ribbon impeller.

In that case, the torque measured during stirring can be converted to a viscosity thanks to a preliminary calibration, performed with standard oils [START_REF] Autef | Influence of metakaolin purities on potassium geopolymer formulation: The existence of several networks[END_REF]. However, geopolymer pastes may exhibit high stiffness and cohesion, which makes common shear-driven rheometers inappropriate. In that case, a homemade small-scale ramp-type rheometer has been used to simulate an extrusion process and to convert the applied pressure (i.e. the resulting torque) to the rheological behavior of the pastes [START_REF] Yunsheng | Impact properties of geopolymer based extrudates incorporated with fly ash and PVA short fiber[END_REF].

Online torque measurements have already been used to follow emulsification processes of viscous organic polymers such as epoxy resin [START_REF] Akay | Flow-induced phase inversion in the intensive processing of concentrated emulsions[END_REF], polyethylene [START_REF] Akay | Preparation of Colloidal Low-Density Polyethylene Latexes by Flow-Induced Phase Inversion Emulsification of Polymer Melt in Water[END_REF], polyester [START_REF] Catte | Catastrophic emulsion inversion process of highly viscous isosorbide biobased polyester monitored in situ by torque and light backscattering[END_REF][START_REF] Pierlot | Emulsification of viscous alkyd resin by catastrophic phase inversion with nonionic surfactant[END_REF], polyurethane [START_REF] Saw | Catastrophic phase inversion in region II of an ionomeric polymer-water system[END_REF], rosin resins [START_REF] Song | Role of operating conditions in determining droplet size and viscosity of tackifier emulsions formed via phase inversion[END_REF], silicon resin [START_REF] Galindo-Alvarez | Viscous Oil Emulsification by Catastrophic Phase Inversion: Influence of Oil Viscosity and Process Conditions[END_REF], bisphenol A-polyester [START_REF] Goger | In Situ Rheological Measurement of an Aqueous Polyester Dispersion during Emulsification[END_REF] or bitumen [START_REF] Edward | Preparation of highly concentrated bitumen emulsions by catastrophic phase inversion: Follow-up of the emulsification process[END_REF]. The torque may be converted to viscosity by calibration curves or by mathematical models [START_REF] Allouche | Simultaneous Conductivity and Viscosity Measurements as a Technique To Track Emulsion Inversion by the Phase-Inversion-Temperature Method[END_REF][START_REF] Edward | Preparation of highly concentrated bitumen emulsions by catastrophic phase inversion: Follow-up of the emulsification process[END_REF][START_REF] Song | Role of operating conditions in determining droplet size and viscosity of tackifier emulsions formed via phase inversion[END_REF].

In this article, the aim is to take advantage of the high viscosity of fresh geopolymers to perform online torque tracking into the reactor before and during organic liquid incorporation. The objective is to determine to what extent torque tracking provides relevant information on both the process of oil incorporation and its repartition into the GP. A series of organic liquids (OL)

have been selected for this study, comprising alkanes (dodecane, 2 paraffinic mineral oils (Finavestan and Nevastane)) as well as an industrial motor oil (Shellspirax) and tributylphosphate (TBP), in a view of covering a wide range of viscosity (0.0018 to 2.33 Pa.s).

The effects of different quaternary ammoniums on the oil incorporation have been studied, including the well-known cetyltrimethylammonium bromide (CTAB), with the aim of confirming the positive effect of CTAB on the organic liquid incorporation.

Materials and methods

Chemicals

Geopolymers with the following molar composition 3.8 SiO2 : 1 Al2O2 : 1 Na2O : 13 H2O were synthesized using a sodium silicate solution (Woellner, Betol® 39 T: 27.8 wt% SiO2, 8.3 wt% Na2O and 63.9 wt% H2O), sodium hydroxide (Sigma Aldrich, 99%) and an aluminosilicate source (metakaolin, ARGICAL-M 1000, Imerys). Metakaolin was chosen in this study for its stable chemical composition and easy supply, which is not the case of other raw materials used for geopolymer formulation (e.g. fly ash F, red mud).

Hexa, tetra, dode, de, octa, hexa-decylltrimethylammonium bromide and Sudan IV were purchased from Sigma-Aldrich. Mineral oils Nevastane EP100 and Finavestan A360B were obtained from Total, Shellspirax S2 A 80W-90 is a commercially available oil, Dodecane, and Tributyl phosphate were purchased from VWR with an announced purity of 99%. Chemicals were used as received without any further purification. Water was deionized using a Milli-QR©Water Purification System and was collected with a resistivity ≥ 18.2 MΩ•cm.

Propeller and reactor

The stirring system is a propeller (Table 1) with three contiguous blades (6.0 cm diameter) fixed on a stainless steel rod (0.6 cm diameter). The reactor is a cm beaker with an internal diameter of 8.0 cm. During the mixing step, the propeller is placed at 1 cm from the bottom of the beaker.

Geopolymer/oil preparation

First, 12.92 g of sodium hydroxide, 82.16 g of Betol® 39 T silicate solution and 11.04 g of deionized water are introduced into a 500 mL reactor to form the activating solution: The mixture was first stirred at 400 rpm with the Heidolph RZR 2051 control to ensure the complete dissolution of sodium hydroxide, and then allowed to cool down to room temperature during 1 hour. Afterwards, 72.12 g of ARGICAL-M 1000 are added during 2 min and the mixture is stirred at 800 rpm during 15 min until both visual homogeneity of the dispersion and torque stability. At this stage, the fresh geopolymer paste is obtained. The last step consists of introducing the organic liquid colored with Sudan IV into the reactor at a flow rate of 4.0 ml.min -1 with a Syringe pump Legato (kdScientific). The organic liquid is introduced into the geopolymer paste so that the point of impact is located at equal distance from the center and the periphery of the beaker. The fresh mixture (called fresh "geoil") is poured into graduated test tubes for liquid state characterization (discussed below) or directly transferred into open cylindrical polystyrene containers and aged at room temperature for 2 days before solid-state characterization.

Rheological experiments

3 mL of the liquid geopolymer suspension is placed in the gap of a cone-plate (6 cm, 4°)

Kinexus rheometer (Malvern). A shear stress gradient from zero to the maximum value (Pa), chosen in accordance to the consistency of the sample, is applied in 1 min followed by the return in the same conditions, in order to reach at least a shear rate of 100 s -1 . Experiments were carried out at a constant temperature of 25.0 °C. The linear profiles obtained allow the determination of the viscosity (shear stress divided by shear rate) in the studied scale.

Torque measurements

The torque is measured in function of time using a Heidolph RZR 2051 control at a selected stirring rate (100 to 800 rpm) and recorded via a specific software (Watch/Control 200) provided by Heidolph.

Optical microscopic analysis

Optical microscopic observations are carried out using an Axiostar microscope Zeiss in transmission-mode for fresh geopolymers. Due to their important opacity, geopolymer suspensions are directly diluted with water (1/10) on the small glass plate before observation.

A Keyence VHX-5000 microscope is also used in reflection-mode for hardened geopolymer observations..

Zeta potential

10 mg of metakaolin and an appropriate amount of alkyl trimethyl ammonium bromide are dispersed in 10 mL of aqueous solutions for 5 min using ultrasonic waves and left for 12 hours at rest. The pH of the aqueous solutions is adjusted by the addition of sodium hydroxide or hydrochloric acid and controlled before the Zeta Potentiel analysis, which is performed using a Nanosizer Malvern instrument.

Washburn tests

Washburn tests are carried out using a Krüss Force Tensiometer -K100 in combination with Krüss Windows based Laboratory Desktop software in the sorption mode. 1.0 g of (natural or hydrophobically modified) metakaolin is placed in an appropriate sample holder, packed for 30 seconds using 2 kg weight and finally suspended from the balance in the tensiometer. The liquid is raised until it just touches the bottom of the porous sample and mass 2 (w 2 ) versus time (t) data are then collected as liquid penetrates into the solid.

Hydrophobically metakaolin powders were obtained as follow: 5.0 g of natural metakaolin were introduced in 50 mL of 10 -2 M CTAB aqueous solution of and let for stirring during 2 hours.

The powders were then filtered on paper and dried in an oven at 60 °C overnight.

Results and discussion

Torque/Viscosity correspondence

The selected organic liquids (OL) of interest comprising alkanes (dodecane, 2 paraffinic mineral oils (Finavestan and Nevastane) as well as an industrial motor oil (Shellspirax) and tributylphosphate (TBP) have been first analyzed by shear stress/shear rate rheological curves.

Viscosity profiles are presented in Fig. 1. A. The maximum shear stress applied is selected in order to reach 100 s -1 that could be representative of a mean shear rate generated in the reactor during fresh geopolymer mixing.

The tested OLs are almost Newtonian since their viscosity is relatively constant with the shear rate. The small variations in viscosity for extremely flowing OLs (water, dodecane, TBP) at low speed gradients (< 10 s -1 ) are due to experimental conditions. The viscosity at 100 s -1 (Table 1) versus the torque obtained in the reactor at different speed rates have been plotted (Fig. 1.

B).

Table 1. Viscosity at 25 °C measured at 100 s -1 for different organic liquids, fresh geopolymer pastes with and without incorporated Shellspirax oil (20%v) and p ratio (viscosity of OL/ viscosity of fresh GP).

Organic liquids Viscosity at 100 s

Choice of geometry / transferability to other mixing geometries

Different stirring propeller (vortex, anchor, dispersing homogonising blades) have been tested without success. Only the propeller with three contiguous blades made it possible to incorporate organic liquids with a minimum speed of 800 rpm. Estimation of Reynold number (Re=23) has been done using Re=N.d 2 .ρ/µ with N=13.33 rounds/s (800 rpm), d=6.10 -2 m, ρ=1672 kg.m -3 , m=3.5 Pa.s. Estimation of the maximum shear rate at the extremity of the blade is 151 s -1 .

Given the high viscosity of the reaction medium, it is important to note that the mixture can be stirred homogeneously as long as the quantity of geopolymer is at a distance from the helix of the order of a few cm. Beyond this distance, the geopolymer at the surface of the reactor is no longer agitated. In the case of an industrial dimensioning, it would probably be necessary to choose more suitable systems with multi-blade systems for example.

Organic liquids incorporation into fresh geopolymer

The incorporation of OL (Fig. 2) was carried out by always following the same protocol. First, the geopolymer suspension is prepared by adding metakaolin to the activating solution. Once the torque value is stable, the OL is added at a constant flow rate of 4.0 mL per minute. The addition is stopped at 100 mL if the geopolymer is able to accept this volume (Fig. 1, Nevastane, Shellspirax). If not the addition is stopped when the organic liquid can no longer penetrate inside the geopolymer (Fig. 1, Finavestan, …, Dodecane)). At this stage, the mixture is removed from the reactor and a picture of the fresh GEOIL, fixed on the 6 cm diameter propeller is presented in Table 2.

All the points in Fig. 2 for which there is no OL added (v = 0.0 mL) correspond to the same fresh geopolymer composition, thus the small torque variations (35-40 N.mm) can be attributed to slight position change of the propeller in the reactor. In any case, this difference of 5.0 N.mm is negligible in comparison with the torque variation obtained during OL incorporation.

Three torque evolution profiles can be distinguished in Fig. 2 depending of the type of OL that is incorporated, whose corresponding GEOIL aspect (Type I, II or III) are presented in Table 2 before and after hardening. In the case of dodecane or TBP/dodecane, the torque remains constant. The torque recording was proceeded until introduction of 40 mL of OL, but in fact, the oil incorporation into the GP was not taking place. After stopping the agitation, the organic liquid directly coalesced at the surface of the GP. The small amount of OL that is being incorporated is in the form of big droplets of a few millimeters diameter, as can be seen on the picture of the paste (Table 2, first column). In the end, the paste corresponds practically to the fresh geopolymer in terms of viscosity and is defined as type I GEOIL.

In case of TBP and Finavestan, the torque increases regularly up to 54 N.mm and then stabilizes after 22 mL of TBP or 42 mL of Finavestan have been incorporated. This late stabilization results from the limit of incorporation and the appearance of big droplets of oil in the GEOIL that can be seen on the propeller (Table 2, second column). The fresh paste is still a bit too fluid to avoid coalescence of OL droplets on the surface of the geopolymer until the hardening phase.

GEOIL type II will characterize such systems for which the incorporation of oil is possible in a limited proportion and for which an oil resurgence appears before hardening.

In case of Nevastane and Shellspirax mineral oils, the torque increases constantly even after adding 100 mL of oil (55% oil incorporation) and the dispersion seems macroscopically very homogeneous as shown by the appearance of the GEOIL on the propeller (Table 2, third column). The microscopic state (picture not presented) shows the dispersion of numerous oil droplets ranging from 1.0 to 50 microns. GEOIL type III will characterize these fresh geoil with an adapted viscosity after oil incorporation that allows the organic liquid to remain trapped in the solid matrix until the hardening is completed. The green area in Fig. 2 is a schematic representation of torque variation versus volume of organic liquid that leads to GEOIL type III.

Finally, with regards to the OL tested, it seems that if its viscosity is greater than 0.13 Pa.s (at 100 s -1 ) like Nevastane or Shellspirax oils, the incorporation into geopolymer is properly done.

If the viscosity is about 0.13 Pa.s, such as the Finavestan oil, incorporation is possible but only at a limited extent of about 43% and the droplets are relatively big (100 microns). For OL with very low viscosities (lower than 0.004 Pa.) such as TBP, dodecane and their mixtures, incorporation is not possible. Observations have been carried out with preliminary dilution in water (1/10). Small colored (blue, green, red) parts correspond to solid particles that appear under polarized light. Circular oil droplets with sizes ranging from 1.0 to 50 microns are also visible (B).

Effect of the viscosity difference on the incorporation

The effect of the viscosity ratio (p = ηD /ηC) between the dispersed and the continuous phases on the ability of a dispersed liquid to be burst has been studied by Grace [START_REF] Grace | Dispersion phenomena in high viscosity immiscible fluid systems and application of static mixers as dispersion devices in such systems[END_REF]. This study concluded that there is a range of most favorable ratios (10 -2 < p < 3) to produce a high quality dispersion. It appears that these conclusions are also valid in the present study if the fresh geopolymer is considered as the continuous phase (ηC = 1.92 Pa.s) and the organic liquid as the dispersed phase. In fact, for Nevastane (p = 1.2) and Shellspirax (p = 0.17), the oil incorporation is successful (Table 1). For OL with lower p ratio such as Finavestan (p = 0.13) and TBP (p = 0.004), the incorporation of oil is still possible but at a limited amount, especially in the case of TBP for which the p ratio is outside the good interval conditions (10 -2 < p < 3).

As expected, dodecane, which does not incorporate into the geopolymer has indeed the lowest p ratio (p = 0.002). Organic liquids with a viscosity high enough to reach a p ratio higher than 3.0 have not been tested in this study. However, Grace's observations might be applicable to geopolymer suspensions with organic liquids having superior viscosities, as observed during the bad dispersion of excessively viscous epoxy liquids into fresh geopolymer resin [START_REF] Wang | High strength inorganic-organic polymer composites (IOPC) manufactured by mold pressing of geopolymers[END_REF].

Effect of CTAB added at different steps of the process

To see whether the addition of CTAB in geopolymer paste facilitates OL incorporation, Finavestan oil was selected for its moderate incorporation (Type II) without any additives.

In the protocol of the present study, the components are added in the following order: activating solution, metakaolin and oil. The CTAB is introduced at three different steps of the process:

directly in the activation solution, just after the addition of metakaolin, or at the end, just after the oil incorporation. The concentration of CTAB (10 -2 M) was calculated with respect to the volume of the fresh geopolymer. Without CTAB (black and green curves in Fig. 4), the torque typically increases from 35-40 to 55 N.mm during the addition of Finavestan, as already observed in Fig. 2. The optical microscopy visualization of the hardened GEOIL without oil are presented in Fig. 5. A Angular micron particles can be observed that probably corresponds to metakaolin (noted MK in the photo) trapped in the polymerized aluminosilcate matrix. (Fig.

5

. B) confirms the presence of large Finavestan oil droplets of more than 50 microns, the reflection of which is materialized by a small white disc in the center of the drop.

The addition of CTAB in the activating solution containing metakaolin already induces an increase in torque (50-55 N.mm) compared to the sample free of CTAB (25-40 N.mm). Oil addition then leads to a strong increase of the torque profile, up to 160 N.mm. After hardening, the microscopic analysis shows a homogeneous size distribution with droplets smaller than 10 microns that can be detected by the numerous small white points (Fig. 5. C). An enlargement of a central portion of the photo in fig. 5. C (see insert) allows the small drops to be seen better.

Even more surprising, when CTAB is added after the incorporation of Finavestan, the torque progressively increases during 10 minutes, leading to a similar torque value (150 N.mm) as when the CTAB is directly introduced in the activating solution. Again, the droplet size is about a few microns (Fig. 5. D). The huge increase of torque experienced when adding CTAB could be due to the adsorption of CTAB on the surface of MK particles. The polar head of CTAB is positively charged and is therefore likely to adsorb on the surface of MK particles, which are negatively charged in alkaline conditions (ZHANG and al. 2008). The hydrophobic forces arising from the interaction between hydrophobic tails of CTAB molecules cause the formation of agglomerates of MK particles [START_REF] Suzzoni | Interactions between kaolinite clay and AOT[END_REF]. These agglomerates are responsible for the viscosity increase due to higher effective volume. This phenomenon is even much more significant in the presence of oil, probably due to the fact that the mixture is getting close to the maximum packing fraction (E. [START_REF] Guazzelli | Rheology of dense granular suspensions[END_REF].

Effect of CTAB concentration

Since the positive effect of CTAB has been shown on the incorporation of a moderately viscous oil (Fig. 4), the incorporation of a very fluid oil mixture TBP/dodecane (7/3) will be tried using CTAB. The CTAB is introduced in the activation solution at different concentrations (0, 1, 3, 5 and 10.10 -3 M).

The black curve (Fig. 6) corresponds to the experience without CTAB. The correspondence between the torque and the viscosity was reported using the calibration curve (Fig. 1). Before oil addition, the viscosity of the GP increases with the amount of CTAB, from 1.7 Pa.s without CTAB to 3 Pa.s for 3.10 -3 M of CTAB and to 4.5 Pa.s if 10.10 -3 M of CTAB was used. For less important CTAB concentrations of 1.10 -3 M and 3.10 -3 , the torque strongly increases up to 5 mL of organic liquid added, then further increases but less quickly up to 20 mL. In the end, the quality of the dispersion is high and corresponds to a geoil type III. As previously observed, when the addition of 20 mL of oil is completed, the torque reaches a certain value (60-90 N.mm, ≈ 10 Pa.s) and further increases, leading to a successful incorporation. On the other hand, for the highest CTAB concentrations (5.10 -3 M and 10.10 -3 M), the torque increases too quickly up to 85 N.mm at just 5 mL of LO incorporated. At this stage, the viscosity is so high (around 20

Pa.s) that the rotation of the propeller can no longer drive the geopolymer from the outside towards the center of the reactor. Thus, the organic liquid can no longer penetrate inside the GP and accumulates at the surface of the slurry. In such a case, the torque signal is very noisy and the mixture heterogeneous. The torque tracking was therefore not reported on Fig. 6. A. for such high concentrations of CTAB (5.10 -3 M and 10.10 -3 M).

Effect of the alkyl chain length of trimethyl ammonium bromide surfactants

Several quaternary ammoniums (QA) with different chain lengths (C6, C8, C10, C12, C14, C16)

were tested in this study. Using the results of Fig. 6. A, a QA concentration of 3.10 -3 M was chosen for the incorporation of a TBP/dodecane mixture. Fig. 6. B shows that whatever the quaternary ammonium that is used, the oil incorporation is successful, providing with a final geoil type III, similar to the one previously obtained with CTAB (C16). The torque profiles are identical for each pairs of compounds C6-C8, C10-C12 and C14-C16-TAB. The profiles are convex for C6-C12-TAB and concave for C14-C16-TAB. Torque monitoring shows that quaternary ammonium with short alkyl chains such as hexamethyltrimethylammonium are able to replace CTAB for the incorporation of oil with the same efficiency. The positive action of CTAB on the incorporation of alkane type fluids is therefore due to its positive charge rather than its than its surfactant activity.

A B

Fig. 6. Torque (N.mm) variation versus TBP/Dodecane volume (mL) added at a flow rate of 4.0 mL.min -1 at a stirring speed of 800 rpm in 80 mL of geopolymer. (A): Different concentrations of CTAB (0, 1, 3, 5 and 10.10 -3 M) and (B): Different quaternary ammonium (C6, C8, C10, C12, C14, C16-TAB) at 3.10 -3 M are introduced in the activating solution.

It can be observed that the magnitude of the torque increase tends to be smaller with decreasing alkyl chain length. This is in accordance with the presence of hydrophobic forces between MK particles. Shorter alkyl chains decrease the range of attraction between MK particles, which decreases the size of particles agglomerates, the effective volume and ultimately the impact on the viscosity.

Zeta potential of metakaolin particles in the presence of QA

Quaternary ammonium salts (QAs) have been used to obtain high surface area nanoporous geopolymers [START_REF] Kang | Synthesis of mesoporous Al-MCM-41 materials using metakaolin as aluminum source[END_REF][START_REF] Pei | A geopolymer route to micro-and mesoporous carbon[END_REF][START_REF] Singhal | CTAB modified large surface area nanoporous geopolymer with high adsorption capacity for copper ion removal[END_REF][START_REF] Yu | Improved simultaneous adsorption of Cu (II) and Cr (VI) of organic modified metakaolin-based geopolymer[END_REF] improve the compressive strength of geopolymers [START_REF] Petlitckaia | Design of lightweight metakaolin based geopolymer foamed with hydrogen peroxide[END_REF] or to increase the hydrophobicity of mineral particles to improve the adsorption capacity of organic compounds [START_REF] Chen | Efficient removal of radioactive iodide anions from simulated wastewater by HDTMA-geopolymer[END_REF][START_REF] Falah | New composites of nanoparticle Cu (I) oxide and titania in a novel inorganic polymer (geopolymer) matrix for destruction of dyes and hazardous organic pollutants[END_REF][START_REF] Montgomery | Optimization of cement-based stabilization/solidification of organic-containing industrial wastes using organophilic clays[END_REF][START_REF] Siyal | A review on geopolymers as emerging materials for the adsorption of heavy metals and dyes[END_REF]. The interaction between metakaolin and QAs, responsible for an increase in viscosity of fresh geopolymers has already been observed when using CTAB (Cantarel, 2016b;[START_REF] Revathi | Evaluation of the role of Cetyltrimethylammoniumbromide (CTAB) and Acetylenicglycol (AG) admixture on fly ash based geopolymer[END_REF]. To demonstrate the interaction between metakaolin and QAs, suspensions of metakaolin in the presence of QAs with different chain lengths have been formulated and left at rest for 12 hours.

This period of rest allows the QAs species to fully adsorb on the surface of the particles. Despite the fact that geopolymer formation requires very basic conditions, the range of pH was extended from 2 to 12 in this study. In the absence of QAs, the Zeta potential of metakaolin (Fig. 7)

decreases from -10 mV to -60 mV with increasing pH value. In the presence of QAs, the Zeta potential/pH curves are strongly translated to higher values and the greater the length of the alkyl chain, the higher the value of the Zeta potential. When using CTAB, the Zeta potential is high and almost constant (60/50 mV) along de pH, even in highly alkaline conditions. These experiments demonstrate the affinity of QAs for metakaolin, and the interactions appear to be stronger with a longer alkyl chain like CTAB. The concentration of hydroxide ions in a geopolymer activation solution is about 6.75mol. L -1 .

This high concentration of NaOH is intended to accelerate the dissolution of the Metakaolin and thus the polymerization of the geopolymer. Since the most basic conditions used in zetametry never exceeded 10 -2 mol.L -1 , we considered that our basic conditions used did not influence the zeta potential measurement. Despite the fact that geopolymer formation requires very basic conditions, the range of pH was extended from 2 to 12 in this study.

Influence of the H2O/Na2O molar ratio (MR) without QAs

Surprisingly, it was observed that an increase in viscosity of the starting geopolymer slurry facilitates the incorporation of very fluid organic liquids such as TBP/dodecane mixtures, even in the absence of QAs. To increase the viscosity of a geopolymer slurry, the amount of water can be reduced while keeping other formulation parameters constant. The GP that serves as a reference in this study has a molar ratio (MR) H2O/Na2O of 13. Two others GPs with less water have been prepared (MR = 12 and MR = 11). The viscosities of the fresh geopolymers and the p ratio reported in Table 3 show that the viscosity of fresh GPs is increasing with decreasing amount of water. This is particularly observed for the formulation with the lowest amount of water (MR = 11, torque = 65 N.mm) which corresponds to a viscosity of about 10 Pa.s in comparison with 2.4 and 2.0 Pa.s for MR = 12 and 13 respectively.

During the addition of the organic liquid in the MR = 12 formulation, the torque remains slightly higher than the reference formulation MR = 13 (type I). The incorporation is therefore being slightly improved (type II) but the geoil that is obtained is still too fluid (3.5 Pa.s) to end with a high quality incorporation. In the case of the MR = 11 formulation, the torque measurement is linearly increasing with the amount of OL added and displaying a significant slope (1.7 N.mm. min -1 ) up to 100 N.mm. In the end, the incorporation is of quality and behave as a geoil type III.

Capillary forces

The interfacial tension between TBP/Dodecane and the activating solution is the same for the three experiments presented in Fig. 8, but there is a change in the viscosity of the aqueous metakaolin suspension. The p ratio ηoil/ηGeopolymer is always lower than 2.10 -3 . According to Grace's observations (((Grace, 1982))), it is logic to consider that TBP/Dodecane cannot be incorporated as droplets into geopolymer slurries, and it is indeed the case with geopolymers (MR=12 and 13). Such observations have also been confirmed in other studies 47 in which the incorporation of dodecane in geopolymers was only possible if p is higher than 10 -2 . Moreover, this rule seems to be verified as well for organic liquid polymers, for which the viscosity adjustment of the geopolymer slurries represents the key parameter in the process of incorporating the resins into geopolymers [START_REF] Colangelo | Preparation and characterization of new geopolymer-epoxy resin hybrid mortars[END_REF]. However, in the present study, the incorporation of TPB/dodecane in the MR = 11 geopolymer, for which the p ratio is the lowest (lower than 3.10 -3 ), the oil can be dispersed in a better way than for MR = 12 or 13 formulations.

To find an explanation, one has to consider that the system of interest is not just a liquid/liquid system, which would be governed by surface tension and viscosity forces only but also includes metakaolin particles. Ternary particle-liquid-liquid systems, composed of particles dispersed in two immiscible liquids can form a variety of structures depending on the ratio of the three components and their chemical affinity. Even when a tiny amount of the second immiscible liquid is added into the first liquid containing the particles (i.e. particles suspension), the rheological properties of the initial suspension can be dramatically altered from a fluid-like to a gel-like state or from a weak to a strong gel. In that case, the viscosity further increases as the volume fraction of the second fluid increases and is attributed to capillary forces between the two fluids and the solid particles. The so-called "capillary suspensions" are a typical phenomenon for these ternary systems, in which capillary or pendular states are building up depending on the wetting properties of each liquid towards the solid particles.

These capillary forces also depend on the interfacial tension between the two liquids and can be observed when the proportion of solid particles is important. This phenomenon could explain why the incorporation of TBP/dodecane is possible in a highly concentrated aqueous suspension (MR = 11) and not in a more diluted medium (MR = 12 or 13).

An experiment carried out with hydrophilic glass beads (SiO2) in water with small amount of diisononyl phthalate (DINP) as second immiscible fluid shows that the water preferentially wets the glass beads. However, oil droplets are also wetting the glass beads to a lesser extent, but high enough to create bridges joining beads with each other, thus forming the pendular state.

The same experiment with hydrophobic glass beads instead of hydrophilic ones changes the system from pendular to capillary state in which water, being the continuous phase, does not preferentially wet the treated beads [START_REF] Koos | Capillary suspensions: Particle networks formed through the capillary force[END_REF][START_REF] Koos | Capillary forces in suspension rheology[END_REF]. Such capillary associations can be observed by microscopy using fluorescent dyes.

Unfortunately, through the course of this study, it was not possible to show such water/oil/metakaolin associations by microscopy. However, to demonstrate these capillary forces, Washburn tests were performed. Water, dodecane and Finavestan were allowed to rise along conventional and hydrophobically modified (using CTAB) metakaolin. Fig. 9 presents the results of the Washburn methods that can be described as follow. After the contact of a glass tube filled with metakaolin powder has been made, the liquid is rising along the powder as a result of capillarity, inducing an increase in mass, whose square can be plotted in function of time. If considering the metakaolin powder as a bundle of capillaries, the liquid migration into the solid can be described thanks to the Washburn equation (Eq. 1) below:

= × γ × cos θ × ρ η ( )
where m, C, ρ, γ, η ,θ, t are respectively the mass of liquid adsorbed, the capillary constant of the powder, the density, the superficial tension and the viscosity of the liquid, the contact angle between the liquid and the solid and the time.

Table 4. Density (kg.m -3 ), Viscosity (Pa.s), Superficial tension (N.m -1 ) and ρ 2 .γ/η (kg 2 .m -3 .s -1 ) at 25 °C for water, dodecane and Finavestan oil. Slope (Δm 2 /Δt in kg 2 .s -1 ) and C.cos(θ) are extracted from the Washburn equation.

Liquid

(kg.m -3 ) (Pa.s = N.s.m -2 ) (N.m -1 ) (kg 2 .m -3 .s -1 ) (kg 2 .s -1 ) (m -3 ) Dodecane 749 0.0013 0.0254 3.7 10 2 3.8 10 -9 a 1.0 10 -7 a 1.6 10 -9 b 4.3 10 -8 b Finavestan 900 0.143 0.0314 6.2 10 0 1.9 10 -10 a 3.1 10 -11 a 1.9 10 -10 b 3.1 10 -11 b Water 1000 0.001 0.0725 5.3 10 3 3.2 10 -9 a 6.0 10 -7 a 3.5 10 -9 b 6.0 10 -7 b a Slope (Δm 2 /Δt) with metakaolin (MK), b with hydrophobic metakaolin (MK-CTAB) Dodecane and water show the same migration rate in untreated metakaolin. Concerning water, the speed of migration is almost identical on the two kinds of metakaolin particles, which is not the case for dodecane, which seems to be more retained by the hydrophobic powder. About Finavestan, there is no difference in behavior depending on the nature of the metakaolin, the migration speed being rather governed by the high viscosity of the oil (Eq. 1), as can be confirmed by the low value of ρ2.γ / η (Table 4). From the slopes measurements (Δm 2 /Δt) and the calculated ρ 2 .γ/η, C.cos(θ) values can be determined (Table 4). The value of C.cos(θ) is the highest for water, followed by dodecane and finally Finavestan, which indicates that water is better wetting natural or hydrophobically treated metakaolin than dodecane or Finavestan.

However, the most important observation to keep in mind for this study is that these three different liquids are capable of wetting metakaolin particles, since capillary rises are observed.

The capillary interactions seen in the Washburn experiments presented in Fig. 9 bring to light the importance of the interactions between metakaolin and the two immiscible liquids. When metakaolin is impregnated with CTAB, the behavior of dodecane towards the powder changes, whereas the behavior of water remains unchanged. Thus, by concentrating an aqueous suspension of metakaolin even more, the solid-liquid interactions with the introduced oil will increase, and the strength of capillary forces will be enhanced. This could explain why the incorporation of organic liquid is made possible in very high viscosity geopolymers, even if the p ratio is lower than 10 -3 (Fig. 8).

Conclusion

Torque monitoring, which is very convenient in the case of industrial applications, was used to follow the incorporation of OL in geopolymer. First, in-line torque monitoring during OL incorporation into geopolymer (GP) slurries provides information on the viscosity of the mixtures using a prior calibration. This avoids off-line rheological measurements and allows observing the impact of each components on the rheology of the mixtures. In addition, in-line torque monitoring allows having relevant information about the OL incorporation process. In particular, three types of torque evolution were identified upon OL addition, corresponding to three types of fresh GEOIL textures (Type I, II and III). When the OL is not incorporated and remains at the surface of the geopolymer slurry, the torque remains constant, to Type I GEOIL. On the contrary, effective OL incorporation processes are those displaying an increase in torque. If the torque increases at the start of OL incorporation, but remains constant from a specific amount of OL incorporated, Type II GEOIL is obtained, with an excess of OL at the surface of the mixture. In order to obtain a continuous increase in torque, corresponding to a full incorporation of the OL, the addition of quaternary alkylammonium (QAs) has proven to be very efficient (Type III). It was demonstrated that the effectiveness of QAs is due to the positive charge of the quaternary ammonium part rather than to the length of the alkyl chain. More specifically, QAs species increase the viscosity of fresh GP and act on the viscosity ratio (p) between the OL and the GP. This ratio is an important parameter to obtain a suitable OL incorporation (Type III). In accordance with Grace's observations, a ratio greater than 10 -2 constitutes favorable conditions. However, even at very low p ratio (< 10 -3 ), when highly viscous geopolymers are used (MR = 11), the incorporation of OL is possible. In that case, metakaolin particles (MK) are concentrated enough to consider the presence of strong particle/liquid interactions. These capillary forces were demonstrated using the Washburn method, showing different behavior between OL, water and MK particles, depending on whether the latter has been hydrophobically treated with QAs or not.

The alkali-activation of metakaolin, poor in Ca, leads to the formation of a 3D reticulated alumino-silicate gel (N-A-S-H), the so-called geopolymer structure. Other raw materials used for alkali-activation, rich in Ca (e.g. fly ash C or slag) lead to the formation of 2D calcium alumino-silicate hydrates (C-A-S-H) as in Portland cements. This type of material is less suitable for OL immobilization. On the contrary, the literature reports the very good compatibility of various organic liquids with geopolymer materials [START_REF] Reeb | Incorporation of organic liquids into geopolymer materials-A review of processing, properties and application[END_REF]. It would indeed be interesting to investigate other Ca-poor raw materials with the purpose of recycling industrial wastes (e.g. fly ash F, red mud). However, metakaolin (MK) was chosen in this study for its stable composition and easy supply.

In conclusion, the in-line torque monitoring is useful to ensure OL incorporation (i.e. when the torque increases) but also to maintain sufficient workability of the final GEOIL mixture. The addition of QAs surfactants is required for the incorporation of some OL in fresh GP. It can be noticed that QAs can be added directly in the activation solution or after the OL incorporation step with no significant differences. However, the addition of QAs leads to a strong viscosity increase of fresh GEOIL, which can be an issue at an industrial scale. In this work, only QAs surfactants were tested and other types should be considered to better control the rheology of GEOIL mixtures. Finally, in the case of successful OL incorporation (Type III), droplets are efficiently dispersed and smaller than 10 microns in the GEOIL composite.

( 5 .

 5 Fig. 1. A: Viscosity (Pa.s) versus shear rate (s -1 ) measured at 25.0 °C for water, different organic liquids, fresh geopolymers (with 0%v and 20%v of incorporated Shellspirax oil). Viscosity curve have been obtained during increase in shear stress from 0 to σmax during 1 min. σmax is fixed in order to obtain a maximum shear rate around 100 s -1 . B: Correspondence between viscosity at 100 s -1 and torque (N.mm) at different stirring speeds (100, 200, 400 and 800 rpm).

Fig. 2 .Fig. 3 .

 23 Fig. 2. Torque (N.mm) variation versus volume (mL) of different organic liquids incorporated (flow rate = 4.0 mL.min -1 and stirring speed = 800 rpm) into 80.0 mL of fresh geopolymer. Type I, II and III physical aspects of geopolymers are represented in Table2.

Fig. 4 .

 4 Fig.4. Torque (N.mm) variation versus time (min) at stirring speed = 800 rpm. The addition of Finavestan (60 mL/23%) happens during the first 15 min (flow rate = 4.0 mL.min -1 ) into 200 mL of geopolymer without CTAB (black), with introduction of 10 -2 M CTAB in the activation solution (AS, red), after the addition of metakaolin (AS+MK, blue) and after the oil incorporation (AS+MK+oil, green). In case of AS+MK+oil, the torque is monitored during 15 additional min.

Fig. 5 .

 5 Fig. 5. Optical microscopy in reflexion mode of cured geopolymers : (A) without Organic Liquid (OL) and with of 23% of incorporated of Finavestan OL as described in Fig. 4 : (B) without CTAB, (C) with addition of CTAB in the activating solution and (D) with addition of CTAB after Finavestan incorporation.

Fig. 7 .

 7 Fig. 7. Zeta potential (mV) versus pH of metakaolin suspensions without and with C6,C8,C12 and C16-TAB species at 3.10 -3 M.

Fig. 8 .

 8 Fig. 8. Torque (N.mm) variation versus TBP/Dodecane volume (mL) added at a flow rate of 4.0 mL.min -1 at stirring speed = 800 rpm in 80 mL of geopolymer. Formulated with water molar ratio of 11, 12 and 13. Table 3. Viscosity of geopolymer (Pa.s) at 25 °C at 100 s -1 for different H2O/Na2O molar ratio (MR) and corresponding p ratio (viscosity of TBP/Dodecane/ viscosity of geopolymer). Geopolymer MR =1 3 MR = 12 MR = 11 Viscosity of geopolymer (Pa.s) at 25 °C at 100 s -1 2.0 2.4 10 p ratio 1.8 10 -3 1.5 10 -3 0.36 10 -3

Fig. 9 .

 9 Fig. 9. Washburn tests representing the square of liquid mass (g 2 ) on metakaolin powders (MK) allowed to rise via capillarity in function of time. Liquids are water (blue circles), dodecane (braun circles) and Finavestan (red circles). Full circles corresponds to non-modified metakaolin and open circles to metakaolin impregnated with 10 -2 M of CTAB.

  In order to extend the viscosity/torque correlation curve with more consistent fluids, data obtained with fresh geopolymers (1.92 Pa.s) and with 20%v of Shellspirax oil incorporated

		(Pa.s)	-1 at 25 °C	p ratio (viscosity of OL/ viscosity of GP)
	Suspension of fresh geopolymer
	with 20%v Shellspirax	5.59
	alone	1.92
	Organic liquid
	Nevastane oil	2.33		1.214
	Shellspirax oil	0.32		0.167
	Finavestan	0.13		0.068
	Tributyl phosphate (TBP)	0.0036		0.002
	Dodecane	0.0018		0.001

Acknowledgments

Chevreul Institute (FR 2638), Ministère de l'Enseignement Supérieur et de la Recherche, Région Hauts de France, and FEDER are acknowledged for supporting and funding part of this work.


Philippe Rolet
email: rolet@lri.fr

Michèle Sebag
email: sebag@lri.fr

Olivier Teytaud
email: olivier.teytaud@inria.fr

Boosting Active Learning to Optimality: a Tractable Monte-Carlo, Billiard-based Algorithm

This paper focuses on Active Learning with a limited number of queries; in application domains such as Numerical Engineering, the size of the training set might be limited to a few dozen or hundred examples due to computational constraints. Active Learning under bounded resources is formalized as a finite horizon Reinforcement Learning problem, where the sampling strategy aims at minimizing the expectation of the generalization error. A tractable approximation of the optimal (intractable) policy is presented, the Bandit-based Active Learner (BAAL) algorithm. Viewing Active Learning as a single-player game, BAAL combines UCT, the tree structured multi-armed bandit algorithm proposed by Kocsis and Szepesvári (2006), and billiard algorithms. A proof of principle of the approach demonstrates its good empirical convergence toward an optimal policy and its ability to incorporate prior AL criteria. Its hybridization with the Query-by-Committee approach is found to improve on both stand-alone BAAL and stand-alone QbC.

Introduction

Active Learning (AL), a most active topic in supervised Machine Learning (ML) [START_REF] Kulkarni | Active learning using arbitrary binary valued queries[END_REF][START_REF] Cohn | Improving generalization with active learning[END_REF][START_REF] Schohn | Less is more: Active learning with support vector machines[END_REF][START_REF] Dasgupta | Analysis of a greedy active learning strategy[END_REF][START_REF] Castro | Faster rates in regression via active learning[END_REF][START_REF] Hoi | Batch mode active learning and its application to medical image classification[END_REF][START_REF] Hanneke | A bound on the label complexity of agnostic active learning[END_REF], aims at accurately approximating a target concept with a limited number of queries to an oracle; each query asks the oracle to label a point of the problem domain (instance) according to the target concept. Through a judicious selection of instances to query, the hope is to learn with a significantly smaller number of queries than in the standard ML setting using iid labeled instances. Prominent AL approaches (section 2) rely on the properties of the hypothesis space (VC dimension or covering numbers) and/or propose various criteria estimating the additional information provided by an instance; they mostly proceed by iteratively selecting the optimal instance in the sense of the considered criterion.

This paper presents a new perspective on AL, formalized as a finite time horizon reinforcement learning problem. Choosing an instance, i.e. making a query is viewed as an action taken by the learner. The reward associated to a sequence of actions is the generalization error of the hypothesis learned from the training set gathered from this sequence of actions. Under mild assumptions (Bayesian realizable setting) detailed in section 3, this paper offers a provably optimal AL strategy, parameterized by the initial training set and the finite horizon T , i.e. the maximum number of queries allowed.

As could have been expected, the formal derivation of this optimal AL strategy is intractable. A tractable approximation thereof is given by the Bandit-based Active Learner algorithm (BAAL, section 4). This tractable approximation involves two main ingredients. Firstly, the exploration of the AL search space relies on UCT, the tree-structured multi-armed bandit algorithm originally introduced by Kocsis and Szepesvari [START_REF] Kocsis | Bandit-based monte-carlo planning[END_REF]. The use of UCT is motivated as AL can be formulated as a single player game (section 4) and UCT has been shown quite efficient in learning game strategies [START_REF] Gelly | Combining online and offline knowledge in UCT[END_REF]. Secondly, an unbiased and frugal sampler of the hypothesis and instance spaces, based on billiard-based mechanisms [START_REF] Ruján | Playing billiards in version space[END_REF][START_REF] Herbrich | Bayes point machines[END_REF][START_REF] Warmuth | Support vector machines for active learning in the drug discovery process[END_REF] supports the Monte-Carlo simulations in UCT. A proof of principle of the BAAL algorithm is presented, showing its practical robustness for AL under bounded resource constraints. This bounded resource constraint is motivated by targeted applications, aimed at simplified models in Numerical Engineering: in this context, training sets are limited to a few dozen or hundred labeled instances due to the high computational cost of instance labeling.

The paper is organized as follows. Section 2 briefly introduces the notations used throughout the paper and reviews the state of the art. Section 3 formalizes AL as a reinforcement learning problem, and shows how to tackle it as a singleplayer game. The online learning algorithm BAAL, implementing this singleplayer game using the UCT algorithm, is described in section 4; it takes as input a training set and the available computational budget, and finds the best instance to label next. Section 5 details two extensions brought to UCT in order to match the AL context. Firstly, as UCT only addresses finite action/state spaces, it needs to be extended to explore the continuous instance space; the proposed approach relies on progressive widening [START_REF] Coulom | Efficient selectivity and backup operators in Monte-Carlo tree search[END_REF][START_REF] Chaslot | Progressive strategies for Monte-Carlo tree search[END_REF][START_REF] Wang | Algorithms for infinitely many-armed bandits[END_REF]. A second extension regards the hybridization of BAAL with prior knowledge, such as existing AL criteria. The proposed hybridization will be illustrated by embedding a variant of Queryby-Committee (QbC, [START_REF] Seung | Query by committee[END_REF][START_REF] Freund | Selective sampling using the query by committee algorithm[END_REF]) within the progressive widening heuristics. An experimental validation of the approach is presented in section 6, using passive learning and QbC as baselines, and the paper concludes with some perspectives for further research.

Background and State of the art

Notations and definitions used in the paper are as follows. Let s t = {(x i , y i ), x i ∈ X, y i ∈ Y, i = 1 . . . t} denote a t-size training set, with X the instance space and Y the label space; only the binary classification case (Y = {0, 1}) will be considered in the paper.

From s t , a learning algorithm A extracts some hypothesis h in hypothesis space H, mapping X onto Y . The learning performance most usually refers to the expectation of the loss ℓ(h(x), h * (x)) incurred by h over the target concept h * a.k.a. oracle, where the expectation is taken over the joint distribution P XY on the problem domain. Whereas the standard supervised learning setting assumes that training examples (x i , y i ) are independent and identically distributed (i.i.d.) after P XY , AL selects at time step i some instance x i in the instance space X (or in a pool of instances drawn from X), the label y i of which is determined by the oracle.

A sampler S is a mapping from t∈I N (X × Y ) t to X, also referred to as policy or strategy, selecting a new instance x to be labeled depending on training set s t . A learner A is a mapping from t∈I N (X × Y ) t to H associating a hypothesis h to any training set s t . The Version Space (VS) associated to a training set s t , noted H(s t ), is the set of hypotheses consistent with s t , i.e. such that h(x i ) = y i for 1 ≤ i ≤ t.

A first research direction focuses on the uncertainty region (set of examples where VS hypotheses disagree). Cohn et al. [START_REF] Cohn | Improving generalization with active learning[END_REF] reduce the VS by characterizing the uncertainty region via neural nets, and selecting new instances in this region. Well-known methods such as Query-by-Committee (QbC) algorithms [START_REF] Seung | Query by committee[END_REF][START_REF] Freund | Selective sampling using the query by committee algorithm[END_REF] directly reduce the VS volume; Freund et al. [START_REF] Freund | Selective sampling using the query by committee algorithm[END_REF] proved that these methods can lead to an exponentially smaller number of queries than the random querying strategy (passive learning), at least in the case of perceptrons. Dasgupta [START_REF] Dasgupta | Analysis of a greedy active learning strategy[END_REF] has established the quasi-optimality of these methods in the realizable classification case (i.e. when h * belongs to H) for a finite instance pool. More generally, when there exists a probability measure P H on H, usual AL strategies are concerned with reducing either the measure of the Version Space, or the variance of the VS hypothesis labels. Otherwise, the reduction of the Version Space can be expressed in terms of its diameter, that is the measure of points where hypotheses in the VS differ.

A related research direction focuses on error reduction, meant as the expected generalization error improvement brought by an instance. Many criteria reflecting various measures of the expected error reduction have been proposed [START_REF] Cohn | Active Learning with Statistical Models[END_REF][START_REF] Roy | Toward optimal active learning through sampling estimation of error reduction[END_REF][START_REF] Lindenbaum | Selective sampling for nearest neighbor classifiers[END_REF][START_REF] Dasgupta | Analysis of perceptron-based active learning[END_REF]; corresponding AL strategies proceed by greedily selecting the optimal instances in the sense of the considered criterion. Other approaches exploit prior, learner-dependent knowledge about what makes an instance informative, such as its margin [START_REF] Schohn | Less is more: Active learning with support vector machines[END_REF][START_REF] Cesa-Bianchi | Learning probabilistic linear-threshold classifiers via selective sampling[END_REF][START_REF] Florina Balcan | Margin based active learning[END_REF]. Hoi et al. [START_REF] Hoi | Batch mode active learning and its application to medical image classification[END_REF] consider batch active learning, querying a subset of instances that results in the largest reduction of the Fisher information. Some of these approaches however happen to face learning instabilities, which might require to mix the AL procedure with a uniform instance selection [START_REF] Xiao | Software testing by active learning for commercial games[END_REF]. Such instabilities suggest that in some cases an optimally efficient AL system can hardly be based on a greedy selection strategy, at least using the criteria considered so far.

On the theoretical side, significant results have been obtained in terms of lower and upper bounds on the reduction of the sample complexity brought by AL, e.g. depending on the complexity of the hypothesis search space measured through covering numbers or Kolmogorov complexity [START_REF] Kulkarni | Active learning using arbitrary binary valued queries[END_REF][START_REF] Vidyasagar | A Theory of Learning and Generalization, with Applications to Neural Networks and Control Systems[END_REF][START_REF] Castro | Faster rates in regression via active learning[END_REF]; the appropriateness of hypothesis spaces to AL has been studied [START_REF] Hegedüs | Generalized teaching dimensions and the query complexity of learning[END_REF][START_REF] Dasgupta | Coarse sample complexity bounds for active learning[END_REF][START_REF] Hanneke | A bound on the label complexity of agnostic active learning[END_REF] and an "almost" optimal (though intractable) algorithm has proposed by [START_REF] Dasgupta | Coarse sample complexity bounds for active learning[END_REF] in the realizable setting for finite VC-dimension.

An optimal active learning strategy: AL as a Markov decision process

This section presents a theoretically optimal strategy for Active Learning in finite time horizon T , where T corresponds to the total number of instances to be labeled along the AL process. As in [START_REF] Freund | Selective sampling using the query by committee algorithm[END_REF][START_REF] Dasgupta | Analysis of a greedy active learning strategy[END_REF], the proposed approach is built on a Bayesian setting [START_REF] Mackay | Bayesian interpolation[END_REF][START_REF] Haussler | Bounds on the sample complexity of bayesian learning using information theory and the VC dimension[END_REF]; prior knowledge about the target concept is accounted for by a probability distribution P H on the hypothesis space1 H. Further, the approach relies on the realizable assumption, i.e. the target concept h * to be learned is assumed to be deterministic and to belong to H (how to relax this assumption will be discussed in section 7).

Let us formalize AL as a Markov Decision Process (MDP). MDPs are classically described in terms of states, actions, reward, policy and transition functions [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. In the AL context, the state space S consists of all possible training sets s t . An action corresponds to the selection of a new instance to be labeled; the set of actions noted A thus coincides with the instance space X or a subset thereof.

The reward function associated to state s t corresponds to the generalization error of the hypothesis A(s t ) learned from s t by learner A. In many finite time horizon settings, and particularly so for the targeted applications, the reward at the final state of the learning is the only one that matters. Accordingly, the reward function will be defined for horizon states only, and assimilated to the value function of those states (see below).

The transition function p : S ×A×S → R + defines the probability of arriving at some state s t+1 by selecting action x in state s t (see below). Lastly, in the AL context a MDP policy is a sampler S, mapping a state s t onto an action, i.e. a new instance x t+1 .

Considering horizon T , let S T (h) denote the training set built by applying T times policy S when learning some target concept h, and let Err(A(S T (h)), h) denote the generalization error of the hypothesis learned from S T (h). It comes naturally that an optimal AL strategy is one minimizing the expectation of Err(A(S T (h)), h) when h ranges in H:

S * T = arg min S E h∼H Err(A(S T (h)), h). (1)
The main motivation behind the presented MDP framework is to build a policy with optimal behavior in the sense of Eq. ( 1). In order to do so, it remains to find the appropriate reward and transition probability functions, and such that the latter accounts for the actual behavior of the AL process.

Regarding the reward function, as the goal is to minimize the generalization error of the hypothesis learned from s T , the reward is only known at horizon T . After the realizable assumption, the target concept h is bound to be in the Version Space of s T , noted H(s T ). The reward function V (s T ) (value at horizon T ) therefore is the expectation of the generalization error of the learner A, taken over H(s T ):

V (s T ) = E h∼H(sT ) Err(A(s T ), h). (2)
By construction, the transition function p(s t , x, s t+1 ) is such that p is zero for all s t+1 except the ones satisfying s t+1 = (s t , (x t+1 = x, y t+1 )) for some y t+1 . In the latter case, p reflects the probability for the label of x to be y t+1 . p(s t , x, s t+1 ) is thus expressed as a function of the probability of the label of x, conditionally to the fact that the target hypothesis belongs to H(s t ):

p(s t , x, (s t , (x t+1 = x, y t+1 ))) = p(h(x) = y t+1 |h ∈ H(s t )) (3)
The above transition and value functions guarantee that the optimal MDP strategy achieves the AL goal and minimizes the expected generalization error (Eq. ( 1)):

Theorem 1 (Optimal AL policy). Let E denote the expectation operator defined after Eq. ( 3). Let value function V * ,T be recursively defined as follows, where |s| denotes the number of examples in training set s:

V * ,T (s) = E h∼H(s) Err(A(s), h) if |s| = T inf x∈X E s ′ ∼p(s ′ |x,st) V * ,T (s ′ ) otherwise (4)
Define S * ,T as S * ,T (s) = arg inf x∈X E s ′ ∼p(s ′ |x,st) V * ,T (s ′ ) (Bellman optimal strategy). Then S * ,T is optimal in the sense of Eq. ( 1).

The proof is given in [START_REF] Rolet | Boosting active learning to optimality: some results on a tractable Monte-Carlo, billiard-based algorithm[END_REF] due to space limitations. It essentially shows that the value function of a policy at the initial state coincides with the criterion in Eq. ( 1). The rest of the proof is a direct application of Bellman's optimality principle [START_REF] Bellman | Dynamic Programming[END_REF].

After this MDP formulation, AL can be seen as a one-player game. The active learner plays against the (unknown) target hypothesis h, belonging to the version space H(s 0 ) of the initial training set2 s 0 . Upon each move (selection of some instance x), oracle h provides the label y = h(x). At the end of the game -that is, after T examples have been picked, defining training set s T -the reward is the generalization error of the hypothesis A(s T ) learnt from s T . The real learning game is indeed played against oracle h.

It is however possible to train the AL player, and therefore devise a good AL policy beforehand, by mimicking the above game and playing against a "surrogate" oracle, made of a hypothesis h uniformly selected in the VS. The reward of such a game is computed as in the real game: it is the generalization error of the learned hypothesis w.r.t. the (surrogate) oracle. This reward implements a uniform draw of the random variable Err(A(s), h) for h ranging in H.

By construction, the average empirical reward collected by the AL player after many such games asymptotically converges toward the true expectation of this random variable, that is, the desired reward function (Eq. ( 2)). This section presents a consistent and tractable approximate resolution of the above MDP, the BAAL algorithm. BAAL relies on two main ingredients. Firstly the tree-structured multi-armed bandit UCT [START_REF] Kocsis | Bandit-based monte-carlo planning[END_REF] is extended to the one-player game of AL. Secondly, a fair and frugal billiard-based algorithm [START_REF] Ruján | Playing billiards in version space[END_REF][START_REF] Herbrich | Bayes point machines[END_REF] is used to sample the instance and hypothesis spaces.

Bandit-based Active Learning

UCT (Upper Confidence Tree) is a Monte-Carlo tree-search algorithm [START_REF] Kocsis | Bandit-based monte-carlo planning[END_REF][START_REF] Coulom | Efficient selectivity and backup operators in Monte-Carlo tree search[END_REF], where the selection of a child node is cast into a multi-armed bandit problem [START_REF] Auer | Using confidence bounds for exploitation-exploration trade-offs[END_REF]. UCT notably became famous in the domain of strategic games as it inspired the computer-Go program MoGo, first to ever win over professional human players [START_REF] Gelly | Combining online and offline knowledge in UCT[END_REF].

s0 x 1 g g g g g g g g g g g g g g g g g g x 2 ... x P s10 0 | | | | | s11 1 f f f f f s20 0 Ó Ó Ó Ó s21 1 Y Y Y Y Y Y Y Y sP 0 0 Ø Ø Ø Ø sP 1 1 T T T T x 1 g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g x 2 ... x P s2110 0 | | | | | | | | | | s2111 1 f f f f f sT O O O O Fig. 1.
Search Tree developed by BAAL, the Bandit Based Active Learner, in the binary classification case. The root of the tree is the initial training set s0. Each tree walk is based on selecting some hypothesis h in the Version Space of s0. The tree walk proceeds by iteratively selecting an instance x, whose label is set to h(x). Ultimately, the tree walk is assessed from the generalization error Err(A(sT ), h) of the learned hypothesis w.r.t. h.

The UCT-based game strategy provides the basis for BAAL (Fig. 4). Let T denote the "game" tree of AL; its root node is the initial training set s 0 . It is worth noting that BAAL actually explores a directed graph (a given node can be reached along different paths) although it is presented as a tree-search algorithm for simplicity.

Each tree-walk (aka game or simulation) is indexed by a surrogate hypothesis h (see below) and proceeds as follows. Each node corresponding to a training set s t is called a state node. Its child nodes, referred to as decision nodes, stand for the actions of the AL player, that is, the possible instances to be selected. A decision node thus is made of the training set s t (parent node), and the chosen instance x t+1 .

Every decision node has two child nodes (the possible labels of the instance x t+1 ). The one selected during the tree-walk indexed by h obviously corresponds to h(x t+1 ), thus leading to the next state node s t+1 = s t ∪ (x t+1 , h(x t+1 )).

More precisely, each tree-walk proceeds as follows (Fig. 4; BAAL pseudo-code is given in Alg. 1):

-Firstly, a surrogate hypothesis h is uniformly drawn in the version space H(s 0 ) of the initial training set s 0 ; -In each state node (i.e. some training set s t ), BAAL uses the famed Upper Confidence Bounds (UCB) criterion (Eq. ( 5), see below) to select some decision node, i.e. some instance x t+1 to be labeled. -From training set s T , a hypothesis ĥ is learned by BAAL3 .

-The reward of the tree walk is the generalization error of ĥ with respect to the surrogate hypothesis h: this reward is used to update the value of every relevant node.

The value μ(s t ) of state node s t is computed by averaging all rewards gathered for nodes s T such that s t ⊆ s T . These values are exploited by the UCB criterion [START_REF] Auer | Using confidence bounds for exploitation-exploration trade-offs[END_REF], determining which arm (instance aka decision node) should be selected. The arm selected is the one maximizing the sum of the empirical reward μi (exploitation term), plus an exploration term depending on the number n i of times arm i has been selected. The exploration vs exploitation tradeoff is adjusted using some tuned constant C:

arg max i∈child nodes μi + C log( j∈child nodes n j ) n i (5)
In the multi-armed bandit setting, this formula guarantees a provably optimal convergence towards the arms with maximal value, under the assumption that arm rewards are independent random variables. (Clearly this assumption does not hold in the AL game, no more than in the game of Go [START_REF] Wang | Modifications of UCT and sequence-like simulations for Monte-Carlo Go[END_REF]).

The memory-wise computational tractability of BAAL is ensured by gradually developing the tree, initially made of the root node and its child nodes only. After each tree-walk, the first randomly selected node is stored in memory with its child nodes: the current leaf node will no longer be a leaf node, and Algorithm 1 The BAAL algorithm: Input: measure P H on hypothesis space H; initial training set s 0 ; time horizon T ; number N of allowed tree-walks; Output: an instance x to be labelled by the oracle.

BAAL(PH, s0, T, N ) for i=1 to N do h = DrawHypothesis(PH , s0) Tree-Walk(s0, T, h) end for Return x = arg max x ′ ∈X {n(s

S {x ′ })} Tree-Walk(s, t, h) Increment n(s) if t==0 then Compute r = Err(A(s), h) else X (s) = ArmSet(s, n(s)) Select x * = U CB(s, X (s))) r = Tree-Walk(s S {(x * , h(x * ))}, t -1, h) end if r(s) ← (1 -1 n(s) )r(s) + 1 n(s) r Return r
the next time it is encountered, the selection among its child nodes will rely on the UCB formula. Thereby the tree is asymmetrically grown, developing more the subtrees where nodes have better values -since those will be selected more often.

BAAL (Alg. 1) implements the UCT scheme with two specific ingredients. The DrawHypothesis function selects the surrogate hypothesis h attached to each tree-walk using billiard algorithms (see below). The ArmSet function (section 5.1) extends UCT to deal with an infinite set of actions (the continuous instance space), using the so-called progressive widening heuristics.

Billiard algorithms

As already mentioned, each tree-walk in BAAL is indexed by a hypothesis h uniformly sampled in version space H(s 0 ). The most straightforward sampling algorithm is based on rejection: hypotheses are uniformly drawn in H and rejected if they do not belong to the VS (if they are inconsistent with the training set s 0 ). Unfortunately, the rejection algorithm, although sound, is hardly tractable. Alternative algorithms such as Gibbs sampling or more generally Monte-Carlo Markov Chains (MCMC) methods, involve quite a few free parameters and might scale poorly with respect to the dimensionality of the search space and the size of the training set s 0 . We thus used a billiard (a.k.a. ray-tracing) algorithm inspired from [START_REF] Ruján | Playing billiards in version space[END_REF][START_REF] Ruján | Computing the bayes kernel classifier[END_REF][START_REF] Herbrich | Bayes point machines[END_REF], assuming that the hypothesis space H can be parame-terized by IR d . Actually, the experimental validation (section 6) considers linear hypotheses; how to go beyond the linear case will be discussed in section 7.

Let Ω be a connected subset of IR d , defined by a set of constraints g 1 , . . . , g n : Ω = {x ∈ IR d s.t. g i (x) ≥ 0, i = 1 . . . n}. The billiard algorithm considers a point z ∈ IR d (not necessarily in Ω) and a direction v (v ∈ IR d , ||v|| = 1). The trajectory followed by z is such that: i) the set of constraints satisfied by z does not decrease; ii) z "bounces" when it meets an active constraint g, i.e. its direction v is changed to point inside the domain (see below); iii) the trajectory is stopped when the computational resources are exhausted, that is when its total length reaches some user-defined parameter L, and the final point is returned. Under a few regularity conditions on the constraints, a billiard trajectory is ergodic, i.e. it covers the whole domain when L goes to infinity [START_REF] Comets | Billiards in a General Domain with Random Reflections[END_REF]; the distribution of the final trajectory point converges toward the uniform distribution on Ω. Billiard algorithms have been successfully used in Machine Learning, e.g. to estimate the Bayes classifier in a (high-dimensional) kernel feature space [START_REF] Ruján | Playing billiards in version space[END_REF][START_REF] Ruján | Computing the bayes kernel classifier[END_REF].

Rebound policy

Let g i be the saturated constraint and z i the rebound point. The rebound policy sets the current direction v to a unit vector randomly drawn in the half sphere centered on z i and defined from the hyperplane tangent to g i in z i . Experimentally, this policy efficiently approximates the Knudsen law advocated by Comets et al. [START_REF] Comets | Billiards in a General Domain with Random Reflections[END_REF], which is more expensive to compute. This matter is however outside the scope of this work.

UCT with continuous domain and AL criteria

This section is devoted to specific adaptations of UCT involved in BAAL: how to deal with a continuous action set (the instances) and how to take advantage of existing AL criteria, such as Query-by-Committee [START_REF] Seung | Query by committee[END_REF].

Progressive Widening

UCB originally requires each arm to be selected at least once for Eq. ( 5) to be computed. When the number of arms is large with respect to the number N of simulations, UCB thus tends to degenerate into pure exploration. UCT faces the same limitation, and even more so since many arms need be considered at each node of the tree, strongly biasing the search towards exploration.

The progressive widening (PW) technique has been proposed by [START_REF] Coulom | Efficient selectivity and backup operators in Monte-Carlo tree search[END_REF] to handle such cases. Let n s denote the number of times node s has been visited so far; then the number of arms that can be considered from s is limited to a fraction m of n s . Empirical and theoretical studies suggest m = O(n 1/4 s ) [START_REF] Coulom | Efficient selectivity and backup operators in Monte-Carlo tree search[END_REF][START_REF] Chaslot | Progressive strategies for Monte-Carlo tree search[END_REF][START_REF] Wang | Algorithms for infinitely many-armed bandits[END_REF]. In practice, the ArmSet procedure implementing PW in BAAL (Algorithm 1), considers a finite set of options for each node (see below), and a new option is added to this set whenever ⌊n 1/4 s ⌋ is incremented by one. Typically, a single arm will be explored from the root node in the first fifteen tree-walks; an additional arm is considered in the sixteenth random walk; UCB will be used to select among both arms during random walks 16 to 80; a third arm is considered in random walk 81, etc. The rationale behind progressive widening is that the better (i.e. the more visited) s, the more careful the investigation of its subtrees should be.

Integrating AL criteria in BAAL

The selection of the actions (instances) considered at a given time step by ArmSet offers some room for the use of prior knowledge. The simplest selection procedure indeed is based on the uniform sampling of the instance space. This procedure reflects an agnostic viewpoint, making no assumption about the utility of instances. It must be noted however that BAAL convergence might be delayed (like other UCT-based algorithms) if the optimal options are considered during the search: instances introduced later on are clearly disadvantaged compared to the earlier, more investigated, instances.

Furthermore, the AL literature suggests that some selection criteria -although not optimal -offer generally sound indications in order to select informative instances (section 2). Such criteria can seamlessly be integrated in BAAL through the progressive widening procedure. Formally, a new instance is added to ArmSet when it satisfies the considered criterion, instead of being uniformly selected in the pool of instances4 .

QbC-BAAL: BAAL with Maximal Uncertainty

Maximal uncertainty (MU) (section 2) is a criterion stemming from Query-bycommittee (QbC) algorithms. QbC works by randomly sampling hypotheses in the Version Space, and choosing a given instance only if enough hypotheses disagree about its label. An analysis of this strategy with a committee of size 2 is for instance presented in [START_REF] Freund | Selective sampling using the query by committee algorithm[END_REF]. In the case of large-sized committees, one proceeds by ranking instances based on the committee disagreement: the topranked instances indeed correspond to those with maximal uncertainty.

The QbC approach is integrated within BAAL as follows. At each node (training set s t ) a committee of hypotheses is built by uniformly sampling the Version Space of s t . Whenever a new instance is to be added to ArmSet, one selects the one maximizing the committee disagreement.

MU is an aggressive criterion, thus holding a higher AL potential than random progressive widening. In counterpart it leads to a less diversified sample; whenever the criterion is under-optimal, the limited exploration will yield poorer performance. It is also more demanding computationally.

Experimental validation

This section reports on the experimental study of BAAL. After describing the experimental goal and setting, empirical results are discussed comparatively to ran-dom active learning, referred to as passive learning, and the Query-by-Committee (QbC) approach [START_REF] Seung | Query by committee[END_REF][START_REF] Freund | Selective sampling using the query by committee algorithm[END_REF].

Goal of experiments

The main questions investigated in the experiments regard the theoretical and computational performance of BAAL. Specifically: Question 1 (Optimality): Does the stand-alone BAAL, where the progressive widening heuristics involves a uniform instance selection, converge to an optimal strategy; does it match QbC results when these are known to be quasi-optimal after Dasgupta [START_REF] Dasgupta | Analysis of a greedy active learning strategy[END_REF] ? Question 2 (Flexibility): As shown in section 5.1, BAAL can embed existing AL criteria, e.g. QbC, within the progressive widening heuristics. Does the use of QbC within BAAL improve i) on stand-alone BAAL ? ii) on stand-alone QbC ?

Experimental setting

Following [START_REF] Freund | Selective sampling using the query by committee algorithm[END_REF][START_REF] Dasgupta | Analysis of perceptron-based active learning[END_REF][START_REF] Florina Balcan | Margin based active learning[END_REF], the instance space X considered in these experiments is the unit sphere of IR d . The hypothesis space H is restricted to the linear classifiers, a.k.a. separating hyperplanes. The choice of this search space is motivated as it has been thoroughly studied from a theoretical perspective [START_REF] Freund | Selective sampling using the query by committee algorithm[END_REF][START_REF] Dasgupta | Analysis of perceptron-based active learning[END_REF][START_REF] Florina Balcan | Margin based active learning[END_REF] although these results did not lead to experimental studies to our best knowledge. Accordingly, some upper and lower bounds on the optimal performance are available, and will be used to assess BAAL performance. The goal of this experimental study is to provide a proof of principle regarding the validity of this new AL approach; still, it must be emphasized that BAAL is not limited to linear hypothesis spaces (a kernelized extension will be discussed in section 7).

Two variants of BAAL will be considered. BAAL stand-alone (or BAAL for short), uses a uniform selection of instances within progressive widening, whereas QbC-BAAL uses a committee-based selection of instances. More precisely, in each node (training set s t ), a committee of 100 hypotheses uniformly selected in H(s t ) is built, and a set of 10, 000 instances uniformly drawn from X is sampled and ordered after the committee disagreement. Whenever a new action is to be considered (⌊n(s t ) 1/4 ⌋ increased by one), the first instance not yet considered in the ordered set is returned by ArmSet.

A hypothesis h is characterized as a unit vector w h (w h ∈ IR d , ||w h || 2 = 1), where h(x) is positive if and only if the scalar product < w h , x > is positive. The lack of prior knowledge is accounted for by setting distribution P H to the uniform distribution on the unit sphere of IR d .

A BAAL run proceeds as follows:

1. The target concept h * is uniformly selected in H. 2. For t = 0 to T , where T is the horizon time and the initial training set s 0 is empty: (a) The BAAL tree rooted on s t is constructed using N tree-walks, respectively using a randomly (a QbC-) ordered instance pool in the progressive widening heuristics for BAAL (resp. QbC-BAAL);

(b) The best instance, i.e. the most visited one at the first level of the s t rooted tree, noted x t+1 , is selected and labeled after the target concept (oracle) h * ; (c) s t+1 = s t ∪ {(x t+1 , h * (x t+1 )}. 3. From s T , the T -size training set built by BAAL (QbC-BAAL) and labeled after the current target concept h * , a hypothesis h T is learned (uniformly sampled in the Version Space H(s T )). 4. The performance of the run, that is, the generalization error Err(h, h * ) defined as P x∼X (h * (x) = h(x)), is computed as the dot product of the associated vectors w h and w h * .

BAAL and QbC-BAAL performances are averaged over 400 independent runs for each 3-uple {d,T,N} (dimension, horizon, simulation number). The number N of simulations, controlling the computational cost, ranges in {1, 2, 4, . . . N = 2 12 }. The reported experiments consider (d = 4, T = 15) and (d = 8, T = 20) to assess the scalability of the approach. The performance is plot against the number N of tree-walks, Fig. 2, indicating the average performance (plain line) and the standard deviation (vertical bars). The performance of BAAL standalone (respectively QbC-BAAL) is assessed comparatively to the passive learning (resp. the QbC greedy AL) baseline. The baseline performances correspond to those obtained for N = 1. Actually, BAAL can be viewed as an algorithm providing an "educated" sampling strategy, where the "educated sampler" is based on a computational budget of N simulations; the baseline, non-educated, sampler accordingly corresponds to N = 1. The competence of BAAL in terms of Active Learning is visible as stand-alone BAAL strongly outperforms the passive learning baseline (N = 1): the generalization error significantly decreases as N increases. For a given computational budget N , the improvement on passive learning is higher in small dimension, as could have been expected. Nevertheless, the improvement is significant both in dimensions 4 and 8 even for a small computational budget (N > 2 6 ).

Performance and Scalability

The performance of stand-alone BAAL is further assessed by comparison with that of a stand-alone QbC using a large-sized committee. After [START_REF] Dasgupta | Analysis of a greedy active learning strategy[END_REF][START_REF] Freund | Selective sampling using the query by committee algorithm[END_REF], the Maximum Uncertainty heuristics (approximated by a large-sized QbC selection) is almost optimal (up to logarithmic terms) in the linear setting. It is thus satisfactory to see that stand-alone BAAL steadily approaches the QbC reference performance (the fact that it can even outperform the QbC reference in dimension 8 is discussed below). These results suggest a positive answer to the Optimality question (section 6.1): stand-alone BAAL is a competent, criterionagnostic Active Learner, which might work well in the absence of prior knowledge about the instance selection, and which matches the known optimal performance in a simple hypothesis space. Another picture is provided by the performance of QbC-BAAL (Fig. 2.(c) and (d)), suggesting that QbC-BAAL can get the "best of both worlds". In dimensions 4 and 8, QbC-BAAL outperforms the QbC baseline in a statistically significant way; it overcomes the theoretical limitations of QbC with regards to optimality (i.e. multiplicative factors w.r.t. optimality logarithmic in the precision and linear in the dimension). These results thus suggest a positive answer to the Flexibility question (section 6.1): BAAL can be hybridized with the QbC criterion, and this hybridization can improve on both stand-alone BAAL and stand-alone QbC. Although BAAL flexibility remains to be confirmed by considering other AL criteria, QbC is among the most widely used criteria in the AL literature.

Due to space limitations, the tractability of BAAL is detailed in [START_REF] Rolet | Boosting active learning to optimality: some results on a tractable Monte-Carlo, billiard-based algorithm[END_REF]. Overall, billiard algorithms allow for a time complexity lower by several orders of magnitude than the rejection algorithm, with an outstanding scalability w.r.t. horizon and dimension of the instance space. For (d = 8, T = 20, N = 16, 000), the com-putational time is 17 seconds on Pentium PC vs > 3 hours for the rejection-based BAAL.

Conclusion and Perspectives

This paper focuses on Active Learning with a limited number of queries, motivated by application domains such as Numerical Engineering. In such domains, computing the response of an example might require several days of computation, limiting the training set size to a few dozen or hundred examples.

Within this bounded resource setting, Active Learning was tackled as a Reinforcement Learning problem: find the sampling strategy aimed at minimizing the overall generalization error for a finite time horizon. This ideal and utterly intractable formalization leads to the proposed BAAL algorithm: an approximation of the optimal sampling strategy is learned within a one-player game setting. BAAL is inspired from an earlier work devoted to Computer Go [START_REF] Gelly | Combining online and offline knowledge in UCT[END_REF], building upon the tree-structured bandit-based search algorithm UCT [START_REF] Kocsis | Bandit Based Monte-Carlo Planning[END_REF] and the progressive widening heuristics to deal with a continuous search space [START_REF] Coulom | Efficient selectivity and backup operators in Monte-Carlo tree search[END_REF][START_REF] Chaslot | Progressive strategies for Monte-Carlo tree search[END_REF][START_REF] Wang | Algorithms for infinitely many-armed bandits[END_REF].

The experimental validation of BAAL investigates three main questions: the convergence towards the optimal performance when it is known; the ability to take advantage of competent AL criteria, such as Maximum Uncertainty [START_REF] Freund | Large margin classification using the perceptron algorithm[END_REF], and improve on the greedy use of these criteria; the computational tractability of the overall AL scheme. A proof of principle, the experiments discussed in the paper show that the answer to all three questions is positive in the simple case of a realizable setting with linear hypotheses.

Further research aims at extending the proposed method beyond this simple setting. Getting rid of the realizable setting (considering label noise, or the case where the target concept does not belong to the hypothesis space H) will be investigated through relaxing the billiard-based exploration of the Version Space. The billiard-based sampling of the hypotheses will likewise be extended to accommodate the priors on the hypothesis space (set to the uniform density in the present paper).

Another perspective for further study is to extend BAAL to non-linear hypothesis spaces, thanks to the famed kernel trick. It must be noted that billiardbased algorithms have been investigated for kernel spaces [START_REF] Herbrich | Bayes point machines[END_REF][START_REF] Ruján | Playing billiards in version space[END_REF][START_REF] Ruján | Computing the bayes kernel classifier[END_REF], featuring good theoretical and computational results. The extension of BAAL to Active Learning with kernels thus defines a new and appealing objective.

Fig. 2 .

 2 Fig. 2.(a) and (c) display the overall performance of stand-alone BAAL and QbC-BAAL versus the computational budget N (in log scale).The competence of BAAL in terms of Active Learning is visible as stand-alone BAAL strongly outperforms the passive learning baseline (N = 1): the generalization error significantly decreases as N increases. For a given computational budget N , the improvement on passive learning is higher in small dimension, as could have been expected. Nevertheless, the improvement is significant both in dimensions 4 and 8 even for a small computational budget (N > 26 ).The performance of stand-alone BAAL is further assessed by comparison with that of a stand-alone QbC using a large-sized committee. After[START_REF] Dasgupta | Analysis of a greedy active learning strategy[END_REF][START_REF] Freund | Selective sampling using the query by committee algorithm[END_REF], the Maximum Uncertainty heuristics (approximated by a large-sized QbC selection) is almost optimal (up to logarithmic terms) in the linear setting. It is thus satisfactory to see that stand-alone BAAL steadily approaches the QbC reference performance (the fact that it can even outperform the QbC reference in dimension 8 is discussed below). These results suggest a positive answer to the Optimality question (section 6.1): stand-alone BAAL is a competent, criterionagnostic Active Learner, which might work well in the absence of prior knowledge about the instance selection, and which matches the known optimal performance in a simple hypothesis space.

Fig. 2 .

 2 Fig. 2. BAAL performances: Generalization error vs the number of simulations N in log scale. Top rows report the results obtained for instance space dimension d = 4 and time horizon T = 15; bottom rows report the results for d = 8, T = 20. Left rows report the performance of the stand-alone BAAL. Right rows report the performance of QbC-BAAL. The performance of the baseline correspond to the performance obtained for N = 1 (at the origin of the curve). The horizontal line reports the results obtained for the stand-alone QbC strategy.

  The associated label is set to h(x t+1 ), where h is the surrogate hypothesis. The next state node thus is s t+1 = s t ∪ (x t+1 , h(x t+1 )). -The tree-walk proceeds until arriving in a tree leaf, i.e. a state node s t0 not yet visited. At this point, Tt 0 additional instances are uniformly selected, labeled after h, and added to the training set s t0 , forming a T -size training set s T .

PH is set to the uniform distribution on H in the absence of prior knowledge.

For the sake of simplicity and when no confusion is to fear, the initial training set s0 is omitted and H is used instead of H(s0).

The learning algorithm actually is a parameter of BAAL. In the experiments, we used a uniform sampler of the version space of sT .

Let us emphasize that hybridizing BAAL with any such criterion does not depend on the true instance labels.

Acknowledgments

This work was supported in part by the Région Ile de France and Digiteo, and the PASCAL network of excellence. The authors acknowledge fruitful discussions with Jean-Marc Martinez, DM2S CEA, regarding applications of this work to Numerical Engineering, and Benoit Kloeckner, from the Mathematics Department of the University of Grenoble, regarding the theoretical properties of billiards.