Monique Lafon
email: monique.lafon@pasteur.fr25

Latent viral infections of the nervous system: role of the host immune response

Keywords: virus, brain, HSV-1, latency, neurons, TLR, CD8, IFN-γ, symbiosis

15, tel 33 (0)1 45 68 87 52, fax 33 (0)1 40 61 33 12 latentes du système nerveux. De récentes données illustrent le rôle paradoxal des lymphocytes cytotoxiques de l'hôte dans le système nerveux; leur présence serait requise pour que la latence virale s'établisse et persiste. Néanmoins, comme dans de nombreuses associations parasitaires, il se pourrait aussi que l'hôte infecté trouve quelques avantages à la présence de virus latents dans son système nerveux.

Résumé

En parasites obligatoires, les virus ont adopté des stratégies d'évitement des défenses cellulaires et des réactions immunitaires. Pendant la phase infection aiguë, les virus doivent se répliquer sans faire périr la cellule et l'hôte qui les hébergent et éviter de se faire éliminer trop rapidement par la réponse immunitaire de l'hôte infecté. Après quelques cycles et la dissémination des particules virales nouvellement formées, l'infection est généralement éliminée.

Dans certains cas, toutefois, le virus disparaît sous sa forme infectieuse, mais son génome reste hébergé par les cellules de l'organisme. Ces cellules constituent un réservoir où le virus persiste à l'état latent pendant toute la durée de vie de l'hôte. Les virus de la famille des Herpes (virus de Herpes simplex, virus de la varicelle) sont des exemples bien documentés d'infection latente du système nerveux central. L'état de latence virale semble résulter d'un équilibre subtil entre le virus, la machinerie cellulaire et la réponse immunitaire. Certains stimuli-comme un affaiblissement de la réponse immune-peuvent causer des phases de réactivation qui se traduisent par éruption de boutons de fièvre ou de zona. Les cellules infectées de façon latente sont souvent présentes dans les organes -comme le système nerveux-peu accessibles à la réponse immunitaire ce qui rend leur élimination difficile. L'infection par le virus du VIH réactive, à la faveur de l'immunodépression qu'il provoque, des infections virales du système nerveux responsables d'encéphalites (CMV, HSV) mais aussi de lymphomes (EBV). L'émergence de cas d'encéphalite leucocytaire causée par le polyoma virus JC après l'utilisation du Natalizumab (anticorps dirigé contre la chaine α4 de l'intégrine 4 dans le traitement de la sclérose en plaques) a mis en lumière le rôle de la réponse immune dans le contrôle des infections virales

Introduction

Neurotropic viruses cause serious neurological diseases in humans.

Development of the disease as an acute or latent infection results of several factors among which the strength of the viral load, the potency of the host immune response and the strategies of virus to escape from the immune response are important factors. In most experimental virus infections of the brain the rapid production of a local innate immune response, including the production of type I interferon (type I IFN) is paramount for host survival . In other instances, the virus burden is so high that clearance of infection by the immune system from the brain fails. This is for example the case of fatal encephalitic cases of West Nile virus encephalitis in mouse model and in humans, where death occurs despite accumulation of inflammatory infiltrates consisting predominantly of nodules of activated microglia, T and B cells, macrophages . Failure of the immune response to clear infection off the brain may also result of immuno-evasive strategies selected by viruses to evade the host immune response. A well characterized case is rabies. Rabies virus has developed sophisticated mechanisms to destroy or inactivate 'protective' T cells that migrate into the infected nervous system, as a result of the over expression of immunosubversive molecules such as FasL, HLA-G or B7-H1 in the infected nervous system (Lafon 2005, 2008, Baloul 2005). For other virus infection such as herpes simplex virus of type 1, HSV-1, evasion from the immune response can be in some how incomplete. In HSV-1 brain infection, the host immune constraints are sufficient to contribute to latency but they are not strong enough to clear infection. Weakness of the immune response leads episodically to reactivation of the infection. Nevertheless, the immune response in the brain is not always beneficial and can cause immunopathological conditions. An obvious illustration is multiple sclerosis, MS, for which an infectious etiology has been suspected suspected and where demyelination results both of inflammatory response and B cell activation. This could be also the case of the influenza associated encephalopathy, where neurological complications of influenza virus infection have been attributed to the inflammatory reactions in the brain rather to the dysfunction of the infected neurons .

Immune status of the nervous system -Innate immune response

Nervous parenchyma -as most tissues-has the capacity to sense viral infection. Origin of the tropism is not yet understood, nevertheless, inefficient innate immune response may play a role since HSVE development has been linked to TLR3 signalling deficiency, either through TLR3 polymorphism or to UNC-93B deficiency . UNC-93B-and TLR3-deficient patients appear to be specifically prone to HSVE, although clinical penetrance is incomplete.

Children with predisposition to HSVE carry an heterozygous mutation in TLR3 at the crucial site for dsRNA binding to TLR3 and TLR3 multimerization.

UNC93B1 binds the transmembrane domain of TLR3, 7/8 and 9. It is specifically involved in the trafficking of TLR3, 7/8 and 9. In its absence, TLR3, 7/8 and 9 cannot reach anymore the endolysosomes, and signalling is impaired . Humans with homologous germline mutation show impaired cytokine production upon TLR3, 7/8 and 9 stimulation.

-Adaptive immune response in the brain

If an allograft is implanted into the brain, rejection is delayed compared to grafts in other organs . This phenomenon has given rise to the concept that general rules of the immune system are not applicable to the central nervous system and that the brain-an organ with poor regenerative capacity-enjoys immunological privilege. The 'immune privilege' of the central nervous system is a longstanding notion which, over time, has acquired several misconceptions and a lack of precision in its definition . Different compartments composed the nervous system: parenchyma, ventricles containing the choroid plexus and filled with cerebrospinal fluid CSF and the meninges. The immune privilege is solely applicable to the nervous parenchyma and not to the other tissues such as meninges, choroid plexus, circumventricular organs and ventricles nor to CSF. Experimentally a virus infection confined to the parenchymal substance of the brain primed the immune system inefficiently or not at all . In contrast, infection in the CSF elicits a comparable immune response to intranasal infection, with an antiviral proliferative response in the draining lymph nodes .

These different situations result on the absence of lymphoid organs into the nervous system and the distinct capacities of antigens to be drained or not from the nervous system towards the cervical lymph nodes . Even if immune privilege of nervous parenchyma is well established, it is not a barrier for host defence, since all neurotropic viruses reach the brain after an initial entry in the peripheral tissues, such as muscle, skin or intestinal cells. In these conditions, detection of the pathogen by the peripheral immune system has already occurred and complete immune response has been triggered against the foreign pathogen. The so-called blood brain barrier is not a barrier either for access of T , B and macrophages into the brain from the periphery, since, once activated, immune expressing surface adhesion molecules have the capacity to enter the nervous system . In contrast, it is well-documented that after there entry, the migratory immune cells faced unfavourable conditions for survival. This Prevention of reactivation is controlled both by lytic granules and by IFN-γ.

Thus, as proposed by Hendricks and collaborators, altogether the virus, the neurons and CD8 + T cells are complementary actors in maintaining HSV1 latency .

In addition, HSV latency may confer a surprising benefit to the host. It appears that symbiotic protection could be offered form bacterial infection to the host harbouring HSV infection. Mice latently infected with a murine gamma herpes virus 68, were more resistant to bacterial infections than the non-latently virally infected mice. This effect is mediated by the prolonged production of the antiviral IFN-γ and the resulting systemic activation of macrophages . Thus, viral latency could be seen as a symbiotic relationship with immune benefit for the host. It remains to be shown whether this symbiotic protection do work in humans too.

Conclusion

Neurotropic virus and host immune responses can build intricate and complex interactions controlling viral pathology. Viral infections of the nervous system are powerful models to better understand how the nervous system controls inflammation and invading immune cells. Viral neuroimmunology studies may contribute to a better understanding of the harmful mechanisms leading to neurodegenerative diseases.

  The innate immune response triggered in situ by the entry of pathogen into the brain is characterized by the production of type I IFN [predominantly IFN-β in the brain ], chimiokines and inflammatory cytokines. Beside intrinsic antiviral property, type I IFN also controls the expression of a large number of genes involved in chemo attractive, antiviral and inflammatory responses which contribute to the host defence against brain invasion. Microglia, astrocytes and now neurons have been identified as main innate keepers of the brain . Cells of the NS, mainly glial cells, express receptors such as Toll-like receptors (TLR) or RIG-like (RLR) which allow them to recognize and respond to the presence of danger signals and Pathogen Associated Molecular Patterns (PAMPS) encoded by pathogens . Only recently neurons were found to express TLRs and RLRs, . TLR3 is strongly expressed by Purkinje neurons in the cerebellum of human brains affected by viral encephalitis, amyotrophic lateral sclerosis, stroke or Alzheimer disease . It is still unclear whether the innate immune responses of the brain are as efficient as those in periphery. Injection of the bacterial component LPS into the brain parenchyma elicits neutrophil and monocyte recruitment within 2h post-injection into the skin, whereas monocyte recruitment is observed only after 2 days when LPS is injected into the brain parenchyma . This reduced inflammatory response may result of the property of neurons to reduce inflammation and regulate microglial phenotype during infection or injury . Control of local glial inflammation occurs via the expression by neurons of receptors such as CD47, CD22, CD200 and by their ligands on glial cells .Nevertheless, the type I IFN in the infected nervous system is essential in controlling some neurotropic infections. This is the case for coronavirus clearance off the brain, for which type I IFN response is primordial irrespective of functional adaptive immune response . Neuronal expression of TLR3 seems to play a major role in the control of neurotropic viral infection, either decreasing viral replication in the case of West Nile virus infections or more surprisingly by promoting virus neuronal infection as shown in the case of rabies virus . Moreover, a role of TLR3 and the resulting innate immune response has been evoked in the susceptibility of children to encephalitis associated to HSV-1 infection (HSVE). When HSV-1 infects the temporal neurons, it causes severe HSVE, with 60% of fatality in absence of treatment.

  results of a series of parameters controlled by neurons that seriously dampen T cells activity. For example, secretion of several neuropeptides and neurotransmitters by neurons such as vasointestinal peptide, calcitonin gene related peptide, norepinephrin and alpha melanocyte stimulating hormone down regulate the activity of T cells . T cells can be subjected to apoptosis by encountering FasL . T cells can also be converted into regulatory T cells in presence of TGF-β secreted by neurons . Surprisingly despite this rigorous regulation, T cells can still participate to the immuno surveillance against brain infection as illustrated by the side effect attributed to the use of natalizumab. Natalizumab is a humanized recombinant monoclonal antibody that efficiently reduces inflammation of nervous system in MS patients. This mAb targeting the α4 subunit of α4β1 integrin expressed by T cells inhibits alpha (4) integrin-mediated adhesion of human T cells to the inflamed BBB . Nevertheless, a few MS patients treated with Natalizumab in clinical trials developed a progressive multifocal leukoencephalopathy caused by the polyomavirus JC, an opportunistic viral infection of the NS . This might be related to the property of Natalizumab to decrease the entry into the brain of protective lymphocytes allowing the infection of the nervous system by the JC virus a human neuroropic poyomavirus, a virus requiring strong suppression of the immune system in order to thrive. This opportunity occurs after Natalizmab treatment and as told by Igor Koralnik "Bad things may happen when rescuers are turned back at the gates" . This side effect illustrates the critical role of immune response-despite the immunosuppressive neuronal local environement-in protecting the brain against viral infection. Role of immune response in brain protection against invading pathogens, is also illustrated by HIV-infected subjects progressing into symptomatic AIDS and concomitantly their immune response deteriorating, they become vulnerable to opportunistic infections such as Cryptococcus neoformans (2-30% of cases), human cytomegalovirus, HCMV (9%), toxoplasmosis (4%), HSV-1 (4%) as well as Epstein-Barr virus infection (5-10%) the latest causing primary central nervous system lymphoma . Indeed T cells, are major actors in the clearance of viral infections . Nevertheless, in some cases, as illustrated below, T cells may have dual role in the fight against infection in the nervous system. Dual role of immune response in HSV-1 latency HSV-1 establishes a lifelong persistent infection of human peripheral nervous system. During primary infection, virus enters the nervous termini located in mouth-pharyngeal area, and then travels by retrograde axonal transport up to the bodies of the sensory nerves in sensory ganglia. Acute infection is replaced by a latent infection where viral genomes persist without viral particles production. Reactivation from latency can occur sporadically upon different triggering such as UV irradiation or stress. Reactivation results most of the times in benign cold sores and rarely in severe blinding immunopathological herpes stromal keratitis. The role of the immune responses in the control of acute HSV infection, the establishment of the latency and reactivation have been studied in a mouse model of infection, where corneal scarification in presence of virus result in the infection of trigeminal ganglia mimicking human infection. Acute infection. In the first 2-3 days after infection, an innate immune response is quickly triggered as soon as the virus starts replication in the ganglia. Both cellular arm of the innate immune response consisting in the triggering of IFN-γ secreting γ δ TCR + T cells or macrophages producing nitric oxide (NO) and tumour necrosis factor alpha (TNF-α) and humoral arm (type 1 IFN, chemokines, cytokines) are rapidly triggered. The recognition of HSV components is both TLRs (2 and 9) and RLRs-dependent. Despite the capacity of HSV-1 to impede the IFN response later in the virus cycle , this early competent innate immune response is sufficient to eliminate most-but not all-replicating virus from the infected ganglia. Plasmacytoid and conventional DCs are also triggered to produce type I IFN which is essential for the activation of CD8 + T cells and expansion of memory population . Indeed memory CD8 + T cells infiltrate the ganglion by day 6 post-infection, expand and settle in the trigeminal ganglion for the rest of life of the infected animals. In this period, CD8 + T cells participate to the clearance of virus. Establishment of latency Six days after corneal infection, latency was established in some sensory neurons which do not produce replicating viruses anymore. Molecular events that switch an acute HSV infection into a latent infection are not completely understood. Latency is characterized by circularization of the viral genome and the expression of latency associated transcripts (LATs) and viral transcripts corresponding to immediate early (α) early (β) and even late (γ1) viral genes . It is not excluded that the cell type provides a critical environment for establishment of latency; in the experimental ocular infection of mice, HSV-1 establishes latency preferentially within A5 neurons, a subset of sensory neurons expressing Galβ1-4GlcNAc-R epitopes . This subset of neurons could correspond to neurons where expression of lytic viral genes such as ICP0 a gene which prevents circularization of viral genome, is impaired . Intriguingly enough, establishment of latency may also require CD8 + T cells since mice genetically deficient in CD8 + T cells or depleted in CD8 + T cells failed to establish uniform latency . Resident CD8 + T cells located in the latentlyinfected ganglia are specific for the glycoprotein B and since they expressed the marker of activation CD69 they should result of recent viral antigen encountering and activation. Indeed transcripts and viral proteins can be detected in latently-infected mouse neurons . This observation challenges the concept that HSV-1 latency represents a silent infection that should be ignored by the host immune response and suggests instead the antigen direct retention of memory CD8 + T cells . CD8 + T lymphocytes play a critical role in preventing virus reactivation-a phenomenon which can occur in a small number of cells harbouring latent genomes, (1-5%) only, but sufficient enough to cause disease. It has been shown the reactivation is blocked by CD8 + T cells which produced IFN-γ . The CD8 + T cells produce IFN-γ which inhibits the expression of the viral gene, ICP0 that prevents genome circularisation . The IFN-γ receptor is constitutively expressed in many neuronal populations , nevertheless, not all the neurons expressed receptor for IFN-γ, prevention of reactivation by IFN-γ would be restricted to IFN-γ receptor positive cells only. In neurons lacking IFN-γ receptor, it is thought that some virus-mediated latency mechanisms operate. The CD8 + T cells also polarize their T cell receptor (TCR) to the junction with neurons forming immunological synapses where the lytic granules of perforine and granzymes accumulated . The function of lytic granules is not to trigger death of the surrounding HSV-infected neurons. Instead, lytic granules inhibit the expression of ICP4, a viral gene essential for further viral gene expression.


Cyrille Lenders
email: clenders@ulb.ac.be

Michaël Gauthier
email: michael.gauthier@femto-st.fr

Pierre Lambert
email: plambert@ulb.ac.be

‡ M Gauthier

Microbubble Generation Using a Syringe Pump

The context of this paper is to study the use of capillary microgripper in submerged mediums which requires the use of microbubbles. This paper presents a model and experimentations of the generation of bubbles. In the microsystems which uses liquid, gas bubbles can generate forces due to the surface tension at their interface. To use these bubbles, it is necessary to generate them in a controlled way. In this paper, we propose to study the generation of a bubble having a defined volume, using a syringe pump based device. We first build a mathematical model to predict the growth of the bubble in the liquid. Indeed, the compressibility of the gas and the effect of surface tension are of major importance at microscale, and our model will demonstrate the existence of an instability during the bubble growth. We proceed with a dimensionless study that will allow to predict the existence of the instability on the basis of a dimensionless number. Finally, we present experimental results to validate the mathematical model.

Introduction

The study of micromanipulations consists in producing methods able to position objects whose size is typically from 1 µm to 1 mm. Handling and positioning principles used for common industrial objects cannot be easily miniaturized because of the scaling effect. In fact, the behavior of the micro-objects is highly different from the macroscale because it mainly depends on surface forces. Current industrial handling methods are consequently not adapted to this particular behavior. Several microhandling strategies have been proposed in the literature, but their current reliabilities must be improved. One way to increase the reliability of microhandling tasks consists in micromanipulate the object in a liquid [START_REF] Gauthier | Analysis of forces for micromanipulations in dry an liquid media[END_REF][START_REF] Dejeu | Reduction of micro-object's adhesion using chemical functionnalisation[END_REF]. Submerged two-fingered grippers have been proposed

where the release of the object is controlled by dielectrophoresis [START_REF] Gauthier | Submerged robotic micromanipulation and dielectrophoretic micro-object release[END_REF] or chemical principle. Submerged ice gripper is also an original way to position micro-objects in a liquid [START_REF] Lopez-Walle | Principle of a submerged freeze gripper for micro-assembly[END_REF]. These submerged handling methods do not enable compliance which is usually useful in micro-assembly. In other hand, in air, capillary gripping is able to grasp object with some compliance (see Fig. 1) and performs micro-assembly tasks [START_REF] Lambert | Capillary Forces in Microassembly: Modeling, Simulation, Experiments, and Case Study, ser. Microtechnology and MEMS[END_REF]. In a general way, surface tension forces can be used as an efficient micro-actuation principle, since they become predominant at small dimensions. The frame of this paper deals with the study of capillary gripping in a submerged medium. To implement this principle for liquid environments, it is necessary to have a gas-liquid interface, and thus it requires generating bubbles. Consequently the first step of the study of submerged capillary handling consists in the study of the generation of single bubbles. This paper presents a model and experimentation of the bubble generation.

Some authors have already used bubbles in microsystems. For example, [START_REF] Suzuki | Integrated microfluidics system with electrochemically actuated on-chip pumps and valves[END_REF] use a bubble as a valve in a microfluidic system, and [START_REF] Lee | Deflection of microcantilever by growing vapor bubble[END_REF] propose to use bubbles as actuators. [START_REF] Tan | A non-spherical model for bubble formation with liquid cross-flow[END_REF][START_REF] Nahra | Prediction of bubble diameter at detachment from a wall orifice in liquid cross-flow under reduced and normal gravity conditions[END_REF] have studied the continuous generation of bubbles. [START_REF] Xiao | An improved model for bubble formation using the boundary-integral method[END_REF][START_REF] Yang | Numerical investigation of bubble growth and detachment by the lattice-boltzmann method[END_REF] proposed numerical methods to estimate the geometry of a growing bubble. Many of these authors are interested in the size of bubbles as function of parameters such as the fluid flow at the bubble generation point. The applications of these studies are most often in the field of chemistry and colloids.

In this article, we propose a method to generate only one bubble of a defined size in a liquid. The idea is very simple: we propose to push gas to an output channel by mean of a syringe system. It offers several advantages: the gas injected is independent of the fluid, contrary to hydrolysis, and parameters are easy to control compared to the study of droplet. One of the novelty of the study of bubbles is that their behavior depends on a coupling effect between surface tension and gas compressibility.

We present in section 2 the mathematical model used to predict the shape of the bubble. The resolution of the model is presented in section 3 and will show in some cases the existence of an instability during bubble growth. We propose in section 4 a study of dimensionless numbers, which lead to a criterion characterizing the existence of the instability. Finally, we describe in section 5 the experimental set up built to validate the model, and we present some results showing the good correspondence between the model and the experiment.

Bubble Growth Modeling

In this section, we present the model used to predict the generation of a gas bubble using a syringe pump. We first introduce the parameters, then the governing equations.

Parameters of the Model

The device consists of a gas tank (the syringe) and an exhaust channel (the needle) immersed in a liquid. The shape of the bubble is supposed to be a spherical cap. This means that the pressure inside the bubble is assumed to be uniform, i.e. we neglect the hydrostatic pressure due to gravity and dynamical effects, since this is an equilibrium model. The needle we used have diameters s ranging from 2 mm to 150 µm.

The parameters are shown in Table 1, and the device is illustrate on Fig. 2. The contact angle at the edge of the outlet channel is assumed to be variable, ranging from 0 • to 180 • , and the contact line is assumed to stay on the inner diameter of the exhaust channel. Displacement of the piston γ (N m -1 ) Surface tension at the liquid-gas interface P 0 (N m -2 ) Atmospheric pressure

P (N m -2 )
Pressure in the gas tank T (K) Temperature of the system R (N m mol -1 K -1 ) Universal gas constant n (mol)

Total number of gas mole in the system S (m 2 )

Section of the piston s (m)

Exhaust channel diameter r (m)

Radius of the spherical cap h (m) Height of the spherical cap

Governing Equations

We will build here the mathematical model of the system. The goal is to find the geometry of the bubble (defined by the height h and the radius r of the spherical cap, and the needle channel diameter s) as a function of the other parameters. The total amount of moles n in the gas is assumed to be constant. Since the surface tension in the spherical cap will induce across its interface a pressure gradient which depends on the size of the cap, the total volume of the gas is not constant.

The pressure P in the gas phase is supposed to be uniform in the entire system.

To build the governing equation, we use three mathematical relations: Laplace equation ( 1), the ideal gas law (2) and the equations describing the geometry of a spherical cap (3 and 4):

2γ r = P -P 0 (1)
P (V + V 0 -δ S) = nRT (2) r = s 2 8h + h 2 (3) V = πh 6 3s 2 4 + h 2 (4)
Laplace equation ( 1) links the pressure gradient P -P 0 across the gas-liquid interface with the mean curvature 2/r of this interface, and the surface tension γ. The ideal gas law (2) is used to take the compressibility of the gas in account. Finally, assuming the geometry of the interface to be a spherical cap [START_REF] Gauthier | Submerged robotic micromanipulation and dielectrophoretic micro-object release[END_REF][START_REF] Lopez-Walle | Principle of a submerged freeze gripper for micro-assembly[END_REF] allows to find a relation between the tip diameter, the curvature radius and the height of the bubble. In this model, we will assume that the number of mole n is fixed by V 0 and P 0 using the ideal gas relation. This means that the syringe is filled under atmospheric pressure.

The geometry of the bubble may be defined by two of the three variables r, s and h. Since s is set by the experimental setup, we will take it as one of the parameter. We will use h as the second parameter since it allows to set the resulting equation in a polynomial form.

Hence our governing equation will be in the form h = f (V 0 , P 0 , γ, T, R, n, S, s, δ ). The solutions of this equation give the possible heights of the bubble.

π 12 P 0 h 5 + π 3 γh 4 + π 12 P 0 s 2 h 3 + - 1 2 nRT + 1 2 P 0 (V 0 -δ S) + π 4 γs 2 h 2 + π 64 P 0 s 4 + 2γ(V 0 -δ S) h + 1 8 P 0 s 2 (V 0 -δ S) - 1 8 nRT s 2 = 0 ( 5
)
This equation is a fifth order polynomial with h as dependent variable. We cannot find analytical solution, but it can be easily solved using numerical methods. This expression will potentially give one, three or five real solutions.

Resolution of the Model

In this section, we will solve [START_REF] Lambert | Capillary Forces in Microassembly: Modeling, Simulation, Experiments, and Case Study, ser. Microtechnology and MEMS[END_REF] for two different sets of parameters and interpret the results. The liquid in which we create the bubble is supposed to be water and the gas is supposed to be air. Some of the parameters are common for both sets: atmospheric pressure P 0 = 101325 Pa, temperature T = 298.15 K and surface tension γ = 72 × 10 -3 Nm -1 . The universal gas constant is R = 8.314 N m mol -1 K -1 .

Two different kind of behavior are obtained: the bubble is growing continuously or not, as illustrated below.

First Behavior: Continuous Growth

Let us consider a first set of parameters (Table 2).

Calculating the roots of the polynomial for each value of piston displacement δ gives the curve illustrated on Fig. 3. The dashed curve represents the height of a spherical cap having a volume Sδ , i.e. an incompressible fluid behavior. This clearly shows the importance of coupling between the surface tension and the compressibility of the gas on the bubble behavior. Consequently, for small piston displacement δ , the height of the bubble h is defined by a physical equilibrium between the compressibility of gas and the capillary pressure. The growth of the bubble is not continuous: the arrow illustrates the sudden transition of its height at the inflection point I. Because of this instability, there is a range of bubble height that is not accessible From this figure, we may conclude that if we increase piston displacement δ , the height of the spherical cap will increase continuously. In this simulation, we assume that the bubble never lifts off. However, when the bubble is large enough, it will actually separate from the output channel and lift off, due to the Archimedes force.

Second Behavior: Discontinuous Growth

Let us now consider a second set of parameters (Table 2). When we plot the evolution of bubble height h as a function of piston displacement δ , we see the curve is "S"shaped (see Fig. 4).

The consequence is that when we push the piston of the syringe (i.e. we increase δ ), there is an inflection point I where a discontinuity appears. The size of the bubble will increase suddenly (arrow on Fig. 4). At this point, the bubble rapidly expands and the volume of the bubble increases rapidly (Fig. 5).

Figure 5: View of a bubble growth instability. The image has been recorded at 25fps on a CCD camera. The CCD sensor has captured in the same frame the geometry of the bubble before and after its expansion (left and right of point I on Fig. 4). The image shows the end of the glass tip used as outlet channel, the bubble generated (the halfsphere at the end of the tip), and the bubble after its sudden expansion (the shadow). This indicates that the expansion is a fast phenomenon This instability occurs when, in our polynomial model, there are three real roots instead of one.

We observe that the maximal height before the instability occurs is close to s/2, i.e. a half sphere. This configuration is particular because the interface curvature is minimal, which correspond to a maximal pressure inside the bubble.

Because of this instability, there is a range of h that is not accessible. Moreover, the increase of bubble volume lead to its uncontrolled lift off.

We will propose a criterion in the next section to predict the existence of this instability.

Dimensionless Study

Dimensionless numbers are often used to make a similarity study on a system, or to reduce the number of parameters in a model. In this section, we use dimensionless numbers and (5) to predict whether the parameters of the system will lead to an instability or not.

The idea is to find relevant dimensionless numbers for this problem, in order to define a criterion which characterizes the stability of the bubble growth.

Dimensionless Equation

In order to find the relevant dimensionless number, we could build the dimensional matrix for each parameters, and use Buckingham theorem to find a set of dimensionless numbers [START_REF] Szirtes | Applied Dimensional Analysis and Modeling[END_REF]. However, Buckingham theorem does not provide the most relevant set of dimensionless numbers. In fact, any combination of these numbers is also a dimensionless number.

A more efficient strategy is to set our governing equation ( 5) in a dimensionless form. Consequently, we normalize every parameters by a combination of s and γ.

Ẽ = nRT γs 2 S = S s 2 P = P 0 γ/s H = h s Ṽ = V 0 s 3 δ = δ s
The equation is then rewritten using the normalized parameters [START_REF] Suzuki | Integrated microfluidics system with electrochemically actuated on-chip pumps and valves[END_REF].

π 12 P H5 + π 3 H4 + π 12 P H3 + - Ẽ 2 + P 2 Ṽ -δ S + π 4 H2 + π 64 P + 2 Ṽ -δ S H + P 8 Ṽ -δ S - Ẽ 8 = 0 ( 6 )
It should be noticed that h is a function of 9 parameters in [START_REF] Lambert | Capillary Forces in Microassembly: Modeling, Simulation, Experiments, and Case Study, ser. Microtechnology and MEMS[END_REF], and H is a function of only 5 parameters in [START_REF] Suzuki | Integrated microfluidics system with electrochemically actuated on-chip pumps and valves[END_REF]. The reduction reaches 1 up to the number of dimensions in the model, four in this case.

In our case, we can even go one step further. Since the output of ( 6) is H, it can be rewritten as follow:

H = f P, Ẽ, Ṽ -δ S ( 7
)
The parameters have been grouped in only 3 sets of parameters: Ẽ, P and Ṽδ S. Since piston displacement δ appears in the set of parameters Ṽδ S, this set cannot influence the shape of the curve h (δ ) on Fig. 3 or Fig. 4.

Consequently, it cannot influence the existence of the instability. We can conclude that the existence of an instability when generating a bubble from a syringe pump device is only determined by two dimensionless numbers: Ẽ and P.

Determining Threshold of the Dimensionless Numbers

Since only two dimensionless numbers should define the existence of an instability, we have run a set of numerical simulations to determine the threshold values for these dimensionless numbers beyond which the system will be unstable.

The plot in Fig. 6 shows the results of the simulations. The crosses correspond to a set of parameters that led to an instability, and the dots where no instability has been observed. There is a clear borderline between the area of instability and the area of stability. The equation of the borderline allows to define the threshold for the two dimensionless parameters.

The equation of the borderline is well approached by (8) (the line in Fig. 6). log Ẽ = 2 log P (

Hence, the stability condition of the experimental setup is

Ẽ < P2 (9)
This condition can also be stated in terms of parameters with dimension:

σ = V 0 γ P 0 s 4 < 1 (10)
This simple criteria characterizes the fact that the complex equation ( 5) has only one solution for each value of piston displacement δ , and thus that the growth of the bubble is stable. This relation can be used to plot a chart in specific conditions, in order to ease the choice of the admissible initial volume V 0 regarding the diameter of the output channel s. We have plotted this chart for a standard pressure and different values of surface tension (Fig. 7). 1 Actually, the reduction is given by the rank of the dimensional matrix

Scale Effect on the Criterion

If we consider l as the typical size of the problem, the initial volume V 0 is proportional to l 3 , and s is proportional to l. The stability criteria σ is thus proportional to l -1 . Consequently, a miniaturization induces an increase of σ and can lead a non-continuous growing.

Let us consider the case where the tip diameter is identical to the piston diameter, and let L be the maximum height of gas in the syringe. The volume is V 0 = L π s 2 /4. Hence we can calculate the ratio of L as a function of a ratio of s:

(L 1 /L 2 ) = (s 1 /s 2 ) 2 .
If the diameter of the channel is reduced by a factor of 10, the maximum height of gas in the syringe is reduced by a factor of 100.

The non-continuous growth of bubble is therefore specific to microsystems.

Experimental Validation

To validate the mathematical model presented here, a set of experiments has been performed to correlate the growth of a bubble with the displacement of a piston in a syringe pump device.

Experimental Bench Description

The test bench (Fig. 8) mainly consists of:

• A syringe pump, composed of a syringe whose piston is actuated by a manual microtranslation stage;

• a glass micropipette, constituting the output channel. This kind of micropipette is normally used in in vitro fertilization;

• a visualization device (including a camera), to perform the image analysis and find the height of the bubble.

The experiments are performed in DI water, and the gas injected is air. Since the microtranslation stage is manual, we perform a quasi-static experiment by moving regularly the stage by a predefined quantity.

The images of the movie are analyzed using an edge detection function. The conversion from pixel to length is done by first calibrating the visualization device.

First Set of Experiments

The parameters for this experiment are given in Table 3.

We made two series of measurements on the same micropipette, that are compared to the theoretical value (Fig. 9). In this configuration, the dimensionless number σ = 67, agreeing with the fact that the bubble growth is not continuous.

The theoretical curve fits the general shape of the experimental curves h (δ ) in good approximation. However, we observe an horizontal offset that is due to the imprecise measurement of the zero height of the spherical cap, when starting the experiment. Figure 9: Experimental validation of the syringe pump bubble generator -First set of results. There is an horizontal offset because the height of the spherical cap is not zero when the experiment is started. The experiment is stopped when the bubble expands

Second Set of Experiments

We performed a second set of experiments with a different syringe. The parameters are shown in Table 4. In this configuration, σ = 783. The results of this set of experiments is shown in Fig. 10. The theoretical curve seems more steep than the trend of the experimental points. However, this can be explained by errors in the theoretical parameters. The dashed curve represent the model with a variation of 10% of its parameters (S, s, P 0 are reduced by 10% and V 0 is increased by 10%). Since the experimental points are located between the two curves, they are in agreement with the theoretical model. the model with a variation of 10% of the parameters (S, s, P 0 are reduced by 10% and V 0 is increased by 10%). The experimental points are located between the two curves, validating the model

Conclusions and Future Works

We have shown that controlling the generation of a bubble at microscale must be done taking into account the coupling between gas compressibility and surface tension . The developed model shows the existence of an instability under some circumstances, making the growth of the bubble discontinuous. Based on a dimensionless study of this model, a dimensionless number has been found that allows to predict the stable or unstable behavior of the system. Hence it is possible to find a threshold to design the bubble generator without instability. Finally, we have presented some experimental results to validate our model. The experimental results showed good agreement with the theoretical predictions. In micromanipulation application, this model is able to predict the growth of the bubble before the handling of the micro-object.

Future works will focus on the study of the interaction between the bubble and the grasped micro-objects. Model will be improved to take into account the microobject. Thus the generation of microbubble proposed in this paper will be applicable to microhandling tasks based capillary forces.

Figure 1 :

 1 Figure 1: Principle of capillary grippers: (a) current gripper in the air, (b) frame of the paper: submerged capillary gripper

Figure 2 :

 2 Figure 2: Schematic diagram of the bubble and the syringe

δ

  δ 0 → 25.5 × 10 -3 0 → 25.5 × 10 -3 (m) (m) -piston displacement h (m) -height of the spherical cap Incompressible fluid model Compressible fluid model Effect of compressibility and surface tension σ < 1

Figure 3 :

 3 Figure 3: Simulation of bubble height h as a function of piston displacement δ , with first set of parameter. The dashed curve represents the height of a spherical cap having a volume of Sδ , i.e. without taking compressibility and surface tension in account. The plain curve takes compressibility and surface tension in account. One should notice the growth of the bubble is continuous

Figure 4 :

 4 Figure4: Simulation of bubble height h as a function of piston displacement δ , with a second set of parameter. The dashed curve represents the height of a spherical cap having a volume of Sδ , i.e. without taking compressibility and surface tension in account. The growth of the bubble is not continuous: the arrow illustrates the sudden transition of its height at the inflection point I. Because of this instability, there is a range of bubble height that is not accessible

Figure 6 :

 6 Figure 6: Search of the dimensionless number threshold. Crosses (dots) represent experimental setups that led to an instable (stable) bubble generation

Figure 7 :

 7 Figure 7: Chart of maximum admissible initial volume as a function of tip diameter. If V 0 is too large, the bubble generator is likely to produce unstable bubbles

Figure 8 :

 8 Figure 8: View of the validation test bench

5 δ

 5 (m) -piston displacement h (m) -height of the spherical cap Test 1 (offseted) Test 2 (offseted) ModelModel with 10% variation of its parameters

Figure 10 :

 10 Figure 10: Experimental validation of the syringe pump bubble generator -Second set of results. The model (plain curve) do not fit the experimental points very well. However, this can be explained by errors in the parameters of the model. The dashed curvethe model with a variation of 10% of the parameters (S, s, P 0 are reduced by 10% and V 0 is increased by 10%). The experimental points are located between the two curves, validating the model

Table 1 :

 1

		Definition of model parameters
	Parameter	Description
	V 0 (m 3 )	Initial volume in the syringe
	V (m 3 )	Volume of gas in the spherical cap
	δ (m)

Table 2 :

 2 Sets of parameters for model interpretation

	Parameters First set	Second set	Units
	V 0	2 × 10 -6	5 × 10 -9	(m 3 )
	n	8.1748 × 10 -5	2.044 × 10 -7	(mol)
	S	7.8540 × 10 -5	1.9635 × 10 -7	(m 2 )
	s	2 × 10 -3	150 × 10 -6	(m)

Table 3 :

 3 First experiment parameters

	Parameters Value
	V 0	47.65 × 10 -9 m 3 (estimation including connections)
	T	between 298 K and 308 K (for theoretical curve, 298 K)
	S	9.62 × 10 -8 m 2
	s	150 × 10 -6 m

Table 4 :

 4 Second experiment parameters

	Parameters Value
	V 0	557.6 × 10 -9 m 3 (estimation including connections)
	T	between 298 K and 308 K (for theoretical curve, 298 K)
	S	1.96 × 10 -5 m 2
	s	150 × 10 -6 m

This work was supported by the French National Agency (ANR) under NANOROL contract ANR-07-ROBO-0003: Nanoanalyse for micromanipulate. The authors would like to thank also the BRIC (ULB) and the WBI-FNRS for their financial support.

† C.


Laëtitia Matignon

Guillaume J Laurent

Nadine Le Fort-Piat

Design of semi-decentralized control laws for distributed-air-jet micromanipulators by reinforcement learning

Recently, a great deal of interest has been developed in learning in multi-agent systems to achieve decentralized control. Machine learning is a popular approach to find controllers that are tailored exactly to the system without any prior model. In this paper, we propose a semi-decentralized reinforcement learning control approach in order to position and convey an object on a contact-free MEMS-based distributedmanipulation system. The experimental results validate the semi-decentralized reinforcement learning method as a way to design control laws for such distributed systems.

I. INTRODUCTION

In recent years there has been increased interest in decentralized approaches combined with machine learning techniques to solve complex real world problems [START_REF] Verbeeck | Exploring selfish reinforcement learning in repeated games with stochastic rewards[END_REF], [START_REF] Tumer | Distributed agent-based air traffic flow management[END_REF], [START_REF] Wolpert | Reinforcement learning in distributed domains: Beyond team games[END_REF]. A decentralized point of view offers several potential advantages in scalability and robustness, and can overcome the complexity constraints of conventional central control. Machine learning is a popular approach to find controllers that are tailored exactly to the system [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF].

We propose to use reinforcement learning (RL) control techniques in a semi-decentralized perspective as a way to design control laws for distributed manipulation systems. Distributed manipulation systems are based on MEMS 1 actuator arrays that can be used for positioning, conveying and sorting of small parts, or as bulk-fabricated (cheap), ultra-thin transport mechanisms, e.g. for paper in copy machines or printers. Distributed manipulation systems can be divided in two categories: contact systems and contactfree systems. Contact systems simulate cilia and can mainly perform high load capacity and accurate positioning [START_REF] Böhringer | Sensorless manipulation using massively parallel microfabricated actuator arrays[END_REF], [START_REF] Ataka | Design, fabrication and operation of two dimensional conveyance system with ciliary actuator arrays[END_REF]. Contact-free systems use air-flow levitation and have several advantages including higher velocity and no friction problems but they require a greater level of complexity for the control [START_REF] Fukuta | Design, fabrication and control of mems-based actuator arrays for air-flow distributed micromanipulation[END_REF], [START_REF] Konishi | A conveyance system using air flow based on the concept of distributed micro motion systems[END_REF].

In this paper, we aim at controlling a distributed-airjet MEMS-based micromanipulator designed by Chapuis et al. [START_REF] Fukuta | Design, fabrication and control of mems-based actuator arrays for air-flow distributed micromanipulation[END_REF] as part of a research project funded by the NRA (French National Research Agency). This project federates five French research teams and a Japanese one from five laboratories 2 . The device to control is an array of microelectro-valves able to produce controlled and directed microair-jets (cf. figure 1). The joint action of air-jets can achieve some positioning and conveyance tasks of a small part.

L. Matignon, G. J. Laurent and N. Le Fort-Piat are with FEMTO-ST/UFC-ENSMM-UTBM-CNRS, Université de Franche-Comté, Besanc ¸on, France, corresponding author: guillaume.laurent@ens2m.fr 1 Micro Electro Mechanical Systems 2 FEMTO-ST (Besanc ¸on, France), InESS (Strasbourg, France), LAAS (Toulouse, France), LIFC (Besanc ¸on, France) and LIMMS (Tokyo, Japan) The long-term objective is to design and develop a fully integrated distributed micro-manipulation system, that we call smart surface, for conveying, fine positioning and sorting of parts at meso-scale (µm to mm). Fully integrated means that the control must be embedded. However, a fully integrated approach still remains rigid and costly in micro-scale fabrication for research works on control. Consequently, in this paper, we experiment our control algorithms with a model of the distributed-air-jet micromanipulator.

'LVWULEXWHG $LU )ORZ 6XUIDFH $LU -HWV 2EMHFW D )URQW 6LGH 9LHZ E %DFN 6LGH 9LHZ 3QHXPDWLF 0LFURDFWXDWRU 1R]]OH (OHFWURGH IRU PLFUR YDOYH GLVSODFHPHQW WR ULJKW 0RYDEOH 0LFUR 9DOYH (OHFWURGH IRU PLFUR YDOYH GLVSODFHPHQW WR OHIW )OH[XUH
There are many challenging issues concerning the motion control and the increase of stability of the manipulated part. Finding coherent control laws for hundreds of independent air-jets is a complex problem. Fluidic effects are also very hard to control and actuators are not perfectly the same due to process dispersion factors. In the second part of this paper, we show that the open-loop control approach, usually proposed for contact distributed systems and called programmable vector field [START_REF] Bohringer | What programmable vector fields can (and cannot) do: Force field algorithms for mems and vibratory parts feeders[END_REF], [START_REF] Böhringer | Sensorless manipulation using massively parallel microfabricated actuator arrays[END_REF], is not satisfactory for this purpose. That's why we investigate decentralized RL control techniques.

The proposed control architecture is based on RL in cooperative multi-agent systems, where multiple agents are cooperating to solve a joint task [START_REF] Panait | Cooperative multi-agent learning: The state of the art[END_REF]. This framework allows to find local control laws which joint actions are optimal at the global level. Moreover, RL can find control laws without prior knowledge on the system. Thanks to this property, our approach could be applied to different distributed manipulation systems.

II. DISTRIBUTED-AIR-JET MICROMANIPULATOR

A. Real system

Figure 1 illustrates principles of the distributed-air-jet MEMS-based micromanipulator. It consists of an active surface based on an array of pneumatic micro-nozzles. Airflow comes through electrostatic micro-valves in the backside of the device. Each electrostatic micro-actuator works as a normally closed air-valve but has two opened positions, each generating opposite directed horizontal air-jet. When the valve is closed a slight vertical air-jet is produced to ensure levitation. In the front-side, the active surface is simply represented by holes where air-jets are produced. An object can be carried by actuating air-jets independently at each point. An overhead camera is used to get the position of the object's center. See [START_REF] Fukuta | Design, fabrication and control of mems-based actuator arrays for air-flow distributed micromanipulation[END_REF] for more details about MEMS's design and working.

B. Model

The aim of this section is to state a realistic model of the distributed-air-jet micromanipulator. A multi-domain simulation of this distributed-parameter system is a selfchallenging task. We make here some assumptions in order to keep only a finite set of differential equations based on fluidic forces exerted by an air-flow on a body.

1) Fluidic forces: Each nozzle is centered on a 1 mm 2 square. A nozzle generates :

• either a vertical air-jet v air = v air z when the microvalve is closed (the valve state a is 0), • or an oriented air-jet v air = v air x (nozzle normal to x)

or v air = v air y (nozzle normal to y) when the microvalve is opened (the valve state a is ±1).

By the combined action of vertical and oriented air-jets, forces and moments are exerted on the object. Two fluidic forces are defined : the levitation force F L applied in the vertical direction z and the thrust force F T which conveys the object on the surface ( x, y). A torque also applies leading to the object's rotation around z. 2) Hypotheses: For the conveyance model in two dimensions, we only model the thrust force F T so the hypotheses are :

• the levitation force F L is not evaluated; the object is supposed to be constantly in levitation, • effects of vertical air-jets on the object's motion are neglected, • we model only interactions between oriented air-jets and the object's edges. Oriented air-jets reaching the back side of the object are thus neglected, • air-jets are independent, i.e. there aren't any interactions between air-jets.

3) Thrust force:

To establish a model of the thrust force, we need some notations shown on figure 2. The object is represented by a N -faces polygon and is conveyed in ( x, y). Its center of gravity is noted G, G's coordinates are noted (x, y). All the air-nozzles are numbered between 1 to M . When an air-nozzle is less than 5 mm away to the object and when the air-jet is in the good direction, the produced air-flow reaches a small area of the object's edge, called wet area. We assume that the wet area of the face is just the normal projection of the nozzle in (O, y, z). This relation between a face n and a nozzle i is described using the following variables:

• u n is the normal vector of the face n, • s i,n is the surface of the normal projection of the wet area of the face n, • P i,n is the central point of the wet area of the face n, • d i,n is the distance between P i,n and the nozzle i. The elementary thrust force of the air-jet i on the face n is :

f i,n = 1 2 ρC x s i,n v 2 i,n u n (1)
where C x is the drag coefficient, ρ the density of the air and v i,n the relative speed of the air at the point P i,n . Each elementary thrust force f i,n is equal to 0 with a probability of 0.05 to imitate possible micro-actuator failures. The relative speed is given by :

v i,n = ( v air (d i,n , a i ) -v obj ) • u n (2)
where v obj is the speed of the object and v air is the speed of the air-jet. The direction of v air is normal to the direction of the nozzle. The magnitude of v air results from a finite element simulation [START_REF] Fukuta | Design, fabrication and control of mems-based actuator arrays for air-flow distributed micromanipulation[END_REF] that gives air-jet velocity decreasing with the distance d to the nozzle and according to the valve state a :

v air (d, a) =      0 if a = 0 5500 sgn(a)e -d 2 3.38 if a = 0 and d ≥ 0 -1000 sgn(a)e -d 2 0.08
if a = 0 and d < 0

(3) We can notice a negative air-jet velocity if d < 0. Indeed, when a nozzle generates an air-jet oriented along x, a slight opposite air-jet is produced alongx. That leads an object moving slowly along x to bounce of opposite air-jets.

Then, the model sums all elementary thrust forces generated by the M air-jets on the N object edges according to the surface geometry (cf. figure 3). The resulting thrust force F T is applied at its center of gravity G :

F T = M i=1 N n=1 δ i,n f i,n (4)
where δ i,n equals 1 if the air-jet i reaches the face n and zero else. δ i,n is also zero if the air-nozzle is under the object (see hypotheses). The resulting torque is :

M FT = M i=1 N n=1 δ i,n ( GP i,n ∧ f i,n ). ( 5
)
4) Viscous force: Friction forces due to the surrounding air are spread over the surface of the object. The resulting force is the viscous force :

F V = -Kη v (6)
where K is a geometric coefficient dependent on the shape of the object, η the coefficient of viscosity and v the speed of the object. 5) Conveyance model: The object's dynamic follows the equations :

   mẍ = F T • x -Kη ẋ mÿ = F T • y -Kη ẏ J θ = M FT • z (7)
where m is the mass of the object, J its inertia moment and θ its angular speed.

III. OPEN-LOOP CONTROL BY PROGRAMMABLE VECTOR FIELD

A. Control problem

In this section, we assume that no sensor is available (open-loop control). The problem is to address a direction to each air-jet in order to get a specific motion of the object. The magnitude of an air-jet is not adjustable. For instance, we focus here on a stabilization task; a cylinder (2 mm in diameter, 0.25 mm in height) must be maintained at a given place of the surface. The cylinder is modeled by a 21-faces polygon.

We aim to regulate the position in only one direction. For that, we use a restricted area of the active surface. This restricted area contains 19 columns per 2 rows of nozzles (cf. figure 4). All east-west air-jets are assigned to a constant air-jet direction (east, middle or west). North-south air-jets stay in middle position to ensure object's levitation. For this simple task, we try to use the programmable vector field method to determine the direction of each air-jet.

B. Programmable vector field

Programmable vector field was introduced by Böhringer et al. [START_REF] Bohringer | What programmable vector fields can (and cannot) do: Force field algorithms for mems and vibratory parts feeders[END_REF], [START_REF] Böhringer | Sensorless manipulation using massively parallel microfabricated actuator arrays[END_REF] to control contact-actuator arrays and transversely vibrating plates. Programmable vector field is sensor-less and may be employed to orient, sort, feed, assemble parts, etc.

The principle of programmable vector field is simple: actuators are assigned to a specific direction with regard to their position, then, when a part is placed on the device, the vector field induces a force and a torque upon it. Over time, the part may come to rest in a dynamic equilibrium state.

For the generation of manipulation plans with programmable vector fields, it is essential to be able to predict the motion of a part in the field and to determine the stable equilibrium poses a part can reach in which all forces and moments are balanced. To create a stable equilibrium point in the middle of the surface, we used the vector field shown by figure 5. The middle of the surface is located at y g = 9.5 mm.

C. Experimental results

Figure 6 shows the position of the cylinder's center versus the time. Three short perturbations at time 0, 5 and 10 s are done to test the robustness of the regulation. As we can see on the curve, the motion is very oscillatory and non-harmonic. Two oscillating phases can be distinguished. Moreover, in some cases due to a short perturbation, the object misses the target position and stays in another stable position (at time step 5 s). The reason is that slight counter-jets are produced in the opposite direction of the air-jets as described by equation 3. This phenomena was observed with the real system.

The first conclusion is that the vector field can create a stable equilibrium point. But the stable area is very small and the motion of the object is very oscillatory. The constant programmable vector field is acting as a simple on-off regulator with a high gain, so turning the velocity down might get rid of some of the oscillations and overshoots at the cost of slower convergence. This could be done by reducing the input pressure. However, as programmable vector field control is open-loop, it will not be able to reject perturbations.

The second conclusion is that the control of the airjet micromanipulator is not obvious. So, we get a strong benchmark to test various control approaches.

IV. CONTROL BY REINFORCEMENT LEARNING

A. Control problem

In this section, we assume that the manipulation area is supervised by a global sensor like a camera. Thanks to this sensor, the object's position is known at intervals of time T (sampling period). The position at time step k is noted (x k , y k ).

In order to reject perturbations or to control the object trajectory, a suitable combination of air-jets must be calculated at each sampling period in accordance with the position (x k , y k ) of the part. The control signal (air-jet direction) sent to the ith air-jet is noted a i,k (that can take one of the three possible discrete values). To control the entire surface, there are 96 control signals to process at each step! A fully centralized control architecture is not suitable due to processing complexity and the number of communication channels required (cf. figure 7a). Another solution is to use one controller per air-jet. The object's position is broadcasted to each independent controller so as to close the loop. Then, each controller sends a command to its associated air-jet. We call this architecture semi-decentralized because acting and decision-making are local but sensing is global (cf. figure 7b).

The control problem is to design local control laws that generate a satisfactory global behavior of the part. Under some assumptions, reinforcement learning techniques are able to find such controllers.

B. Reinforcement learning

RL methods are inspired by dynamic programming concepts and have been studied extensively in centralized framework [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. A controller, also called agent, learns by interactions with its environment, using a scalar reward signal called reinforcement as performance feedback. The studies about reinforcement learning algorithms in multi-agent systems are based on Markov game framework.

Definition 1: A cooperative Markov game 3 is defined as a tuple < m, S, A 1 , ..., A m , T, R > where : m is the number of agents; S is a finite set of states; A i is the set of actions available to the agent i (and A = A 1 × ... × A m the joint action space); T : S × A × S → [0, 1] is a transition function that defines transition probabilities between states; R : S × A → is the reward function.

This framework is equivalent to the semi-decentralized architecture we presented because all agents have access to the complete observable state s. Reinforcement function is determined by the task to achieve (see below). The transition function T is unknown from agent's perspective (learning hypothesis).

C. Think globally, act locally

The objective of the group (or the global objective) is to find a joint policy π that maximizes for all states s in S and joint actions a in A the expected sum of the discounted rewards in the future,

Q π (s, a) = E π    ∞ j=0 γ j r j+k+1 s, a    ( 8
)
Q π (s, a) is called the joint or global action-value function.

In the multi-agent system framework, independent learners (ILs) were introduced in [START_REF] Claus | The dynamics of reinforcement learning in cooperative multiagent systems[END_REF] as agents which don't know the actions taken by the other agents. The objective of an IL is then to find a local policy π i that maximizes the expected sum of the discounted rewards in the future for its own action a i in A i ,

Q πi i (s, a i ) = E πi    ∞ j=0 γ j r j+k+1 s, a i    (9)
Q πi i (s, a i ) is called the local action-value function. The IL approach brings the benefit that the size of the state-action space is independent of the number of agents. This choice is pertinent for the distributed-air-jet micromanipulator in order to avoid exponential growth of action space with the number of agents.

It is important to notice that it is necessary for each independent learner to find its local optimal action-value function, in order that the group achieves the global optimum [START_REF] Lauer | An algorithm for distributed reinforcement learning in cooperative multi-agent systems[END_REF]. [START_REF] Watkins | Technical note: Q-learning[END_REF] is one of the most used RL algorithm in single-agent framework because of its simplicity and robustness. That's also why it was one of the first to be applied to multi-agent environments [START_REF] Tan | Multiagent reinforcement learning: Independent vs. cooperative agents[END_REF]. Despite some difficulties as the coordination or the loss of theoretical guarantees [START_REF] Matignon | A study of fmq heuristic in cooperative multi-agent games[END_REF], it was successfully applied with ILs on some applications [START_REF] Sen | Individual learning of coordination knowledge[END_REF], [START_REF] Busoniu | Decentralized reinforcement learning control of a robotic manipulator[END_REF], [START_REF] Wang | Multi-robot box-pushing: Single-agent q-learning vs. team q-learning[END_REF], [START_REF] Guo | Dynamic correlation matrix based multi-q learning for a multi-robot system[END_REF].

D. Decentralized Q-Learning Q-learning
For an IL i, Q-Learning consists in getting a more and more accurate estimation of the optimal local action-value function using a recursive updating equation, that is:

Q i (s k-1 , a i,k-1 ) ← Q i (s k-1 , a i,k-1 ) + αδ ( 10
)
where

δ = r k + γ max b∈Ai Q i (s k , b) -Q i (s k-1 , a i,k-1 ), a i,k-1
is the individual action chosen by the agent i at time step k -1, α ∈]0; 1] is the learning rate and γ ∈ [0; 1[ is the discount factor. Q i (s, a i ) is the current value of the stateaction pair (s, a i ) for the agent i.

Q i (s, a i ) values are stored in a |S| × |A i | array.
We propose to combine decentralized Q-learning with eligibility traces to obtain a more efficient method. The eligibility trace e(s, a) is a measurement of the age of the last visit of the state-action pair (s, a). Action-value function is then globally updated according to eligibility trace, that is:

Q i (s k-1 , a i,k-1 ) ← Q i (s k-1 , a i,k-1 ) + αδe i (s k-1 , a i,k-1 ) (11)
Much more Q-values are then updated at each transition. This method is a decentralized version for ILs of the Watkins's Q(λ) algorithm4 [START_REF] Watkins | Learning from delayed rewards[END_REF].

At each time step, a new action a i,k is selected according to Q i (s k , * ) values and to an exploration/exploitation compromise. We use the -greedy action selection method 5 .

V. EXPERIMENTAL RESULTS

A. Part stabilization

As illustrated on the figure 4, we aim to control columns of east-west nozzles to regulate the position of the part in only one direction like in the previous section. All the eastwest nozzles of a column are controlled together according to the air-jet direction required. So the system requires as much controllers as east-west nozzles columns, i.e. 10 controllers. Possible actions of each controller are: directing the air-jet on the east or on the west or closing the valve, i.e. 3 actions (east, west, middle). So, the cardinal |A i | of the action space A i of the agent i is 3.

The state of the system s k at time step k is the object's current and previous positions, s k = (y k , y k-1 ). To apply RL in the form presented previously, time axis and continuous state must be discretized. The sample time of our simulation is 0.01 seconds (between step the integration method is ODE45). For object's position, a 41×41 spatial tile-coding is used. So, this yields Q i tables of size 41×41×3 for each 10 controllers, to compare with a Q table of size 41 × 41 × 3 10 in a centralized view.

According to [START_REF] Matignon | Improving reinforcement learning speed for robot control[END_REF] and in order to stabilize the object at the position y g with null speed, the chosen reward function is :

R(y k , y k-1 ) =      1 if (y k , y k-1 ) ∈ [y g -ρ, y g + ρ] 2 -1 if y k < y min or y k > y max 0 else ( 12
)
where ρ sets a margin, y min the minimal abscissa and y max the maximal abscissa. Independent controllers learn by decentralized Q-Learning during 300 trials. Each trial starts with the object in a random initial state (y 0 ∈ [START_REF] Wolpert | Reinforcement learning in distributed domains: Beyond team games[END_REF][START_REF] Matignon | A study of fmq heuristic in cooperative multi-agent games[END_REF] mm) and runs at the most 10 seconds. A trial ends if the object gets out from the restricted area (y k < y min or y k > y max ). All trials use α = 0.1, γ = 0.9, = 0.01 and ρ = 0.5 mm. Figure 8 shows the position of the object's center according to the time after learning. It takes around 0.2 seconds for the ILs to regulate the object's position with an oscillation range of 0.1 mm. The perturbations at time steps 0, 2 and 4 s are quickly rejected.

Semi-decentralized RL manages to stabilize the object with a high damp factor and with a good robustness to perturbations. This result demonstrates the potential capacity of RL control to regulate position and speed of a levitating part on distributed-air-jet micromanipulator.

B. Part conveying

The objective is to convey the object to the middle of the north border of the surface illustrated on the figure 3. The object is fed to the surface at the initial position (x 0 , y 0 ) = (7.5, 1.5) mm and with a random speed ( ẋ ∈ [-50, -10] mm/s). Open-loop control by programmable vector field is first applied to this task. The vector field shown by figure 9 is used to convey the object. The figure 10a shows the object's trajectories for various initial speeds of the object ( ẋ = {-10, -20, -30, -40, -50} mm/s). As illustrated on the figure, this vector field is not able to convey the object to its target position for all initial speeds.

We propose to test if semi-decentralized reinforcement learning control is robust to the different initial speeds of the object. This time, 96 independent controllers learn by decentralized Q(λ) during 1000 trials. Each trial starts with the object at the initial position and with a random speed and runs at the most 10 seconds. A trial ends if the object gets out from the surface. All trials use α = 0.1, γ = 0.9, λ = 0.7, = 0.01. The state of the system s k at time step k is the object's current and previous positions, s k = (x k , y k , x k-1 , y k-1 ). The sample time of our simulation is 0.01 seconds (between step the integration method is ODE45). For object's position, a 13 × 13 × 9 × 9 spatial tile-coding is used.

According to [START_REF] Matignon | Improving reinforcement learning speed for robot control[END_REF], we incorporate prior knowledge or bias to speed up RL. We choose to use a transient bias embedded in initial Q values. This bias advises controllers to generate the previous vector field (figure 9) at the beginning of the learning process. In order to realize the part conveying, the chosen reward function is :

R(x k , y k ) =        10e -(x-xmax 2 ) 2 2
if y ≥ y max -3 if x < x min or x > x max or y < y min 0 else. [START_REF] Watkins | Technical note: Q-learning[END_REF] where (x, y) are the object's coordinates. The reinforcement function rewards agents when the goal is reached, and punishes them when another border is crossed. At the end of the learning process, some conveying tasks with various initial speeds of the object ( ẋ = {-10, -20, -30, -40, -50} mm/s) are realized with controllers following their greedy policy 6 . The figure 10b shows the object's trajectories.

Semi-decentralized RL manages to convey the object near to the target position for all tested initial speeds. Controllers adjusted the initial bias so as to fit the speed of the object. This result confirms the potential capacity of semi-decentralized RL methods to control such distributed systems.

VI. CONCLUSIONS AND FUTURE WORKS

We first showed that the usual open-loop control for contact distributed-manipulation system, called programmable vector field, fails to damp object's motion and to reject perturbations on the contact-free distributed micromanipulator.

Then, a semi-decentralized reinforcement learning control approach has been investigated. This way of control has been validated in comparison with programmable vector field. Notably, it achieves good positioning and conveying performance and has high control stability. This result is a proof-of-concept of using decentralized RL control for distributed-manipulation systems. It is also a new successful application of decentralized Q-learning variant algorithms for independent agents.

We made here some strong assumptions in the model used for simulation. Although the behavior of the model is realistic, some assumptions as the independence of air-jets may be simplistic. Well, the purpose of this paper is precisely to propose a control approach by learning that does not need to state any model to find a controller. One major interest of this approach is to adapt itself to any systems. So, there is likelihood that decentralized RL control will achieve the control of a real contact-free distributed micromanipulator

Fig. 1 .

 1 Fig. 1. Distributed-air-jet MEMS-based micromanipulator [7].

Fig. 2 .

 2 Fig.2. Geometric notations in case of a nozzle close to the object, normal to x and orienting an air-jet with a velocity v air,i (top view).

Fig. 3 .

 3 Fig.3. Geometry of the distributed-air-jet micromanipulator. The device has a total of 96 actuators (air-jet).

Fig. 4 .

 4 Fig. 4. Task description.

Fig. 5 .

 5 Fig. 5. Vector field.

Fig. 6 .

 6 Fig.6.Position of the cylinder's center according to the time using programmable vector field open-loop control (solid line), the dashed line represents the target position. Short perturbations are done at time steps 0, 5 and 10 s.

Fig. 7 .

 7 Fig. 7. Control architectures.

Fig. 8 .

 8 Fig. 8. Position of the cylinder's center according to the time (solid line) after learning with the Q-Learning algorithm, the dashed line represents the target position. The control architecture is semi-decentralized.

Fig. 9 .

 9 Fig. 9. Vector field for the part conveying.

also called team game.

λ is the decay parameter for eligibility traces.

The probability of taking a random action for an agent i is and, otherwise, the selected action is the one with the largest Q i -value in the current state.

The selected action is the one with the largest Q i -value in the current state.

ACKNOWLEDGMENT

The authors gratefully acknowledge Joël Agnus and David Guibert from the FEMTO-ST Institute for their technical assistance. This work was supported in part by the Smart Surface NRA (French National Research Agency) project (ANR 06 ROBO 0009 03).

APPENDIX I NUMERICAL DATA OF DYNAMICAL MODEL