Miguel A Valbuena

Aránzazu Manzano

Joshua P Vandenbrink

Veronica Pereda- Loth

Eugenie Carnero Diaz

Richard E Edelmann

John Z Kiss

Dr Raúl Herranz
email: r.herranz@csic.es

Dr F Javier Medina
email: fjmedina@cib.csic.es

The combined effects of real or simulated microgravity and red light photoactivation on plant root meristematic cells

Keywords: Arabidopsis, Cell Growth, Cell Proliferation, Fractional Gravity, Spaceflight, Tropisms. Abreviations EMCS European Modular Cultivation System

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

The combined effects of real or simulated microgravity and red-light photoactivation on plant root meristematic cells

Introduction

Meristematic cells are undifferentiated highly proliferating totipotent cells, which sustain plant development. Ultimately, the basis of plant development is the balance between the cells actively proliferating in the meristems and the cells destined for differentiation [START_REF] Perrot-Rechenmann | Cellular responses to auxin: division versus expansion[END_REF]. The major functional parameters of meristematic cells are their proliferation rate (i.e. cell division) and their growth rate (i.e. protein synthesis). The strict coordination of these two parameters is called "meristematic competence" [START_REF] Mizukami | A matter of size: developmental control of organ size in plants[END_REF]. These important parameters are affected by environmental factors including light and gravity, which drive the two major plant tropisms, the directed growth movements by which environmental stimuli modulate plant development [START_REF] Vandenbrink | Light and gravity signals synergize in modulating plant development[END_REF].

Apart from its direct effect on photosynthesis and phototropism, light is a regulator of plant development through its effects on meristematic cell functions [START_REF] Griffiths | Plant development: light exposure directs meristem fate[END_REF][START_REF] Yoshida | Stem cell activation by light guides plant organogenesis[END_REF]). An important group of light receptors in plants are the phytochromes (PhyA to E), a family of proteins differing in the quality of light sensed by each member and the physiological responses they trigger, normally acting in combination. In addition, phytochromes regulate different mechanisms related to plant development by modulating changes in gene expression [START_REF] Molas | Phototropism and gravitropism in plants[END_REF].

From the earliest developmental stages, light is known to control seed germination and seedling morphogenesis (photomorphogenesis). The morphology of dark-grown (etiolated) and light-grown seedlings is quite different regarding, for example, the hypocotyl length, which is shorter under light irradiation [START_REF] Jiao | Light-regulated transcriptional networks in higher plants[END_REF]). Regarding roots, most reports indicate that illumination shortens roots [START_REF] Silva-Navas | D-Root: a system for cultivating plants with the roots in darkness or under different light conditions[END_REF], but other studies show higher rates of root elongation under light irradiation [START_REF] Laxmi | Light plays an essential role in intracellular distribution of auxine carrier PIN2 in Arabidopsis thaliana[END_REF], and, in other cases, no significant differences were observed [START_REF] Boucheron-Dubuisson | Functional alterations of root meristematic cells of Arabidopsis thaliana induced by a simulated microgravity environment[END_REF].

In meristems, cell cycle progression is controlled by photoreceptors, but there is increasing evidence of a different effect of light in the root meristems and in the meristems of the aerial part. In the shoot apical meristem, light irradiation activates growth by modulating the levels of the E2F transcription factor family resulting in enhanced rate of cell proliferation [START_REF] López-Juez | Distinct light-Initiated gene expression and cell cycle programs in the shoot apex and cotyledons of Arabidopsis[END_REF]. In young seedling plumules, red light irradiation was shown to produce increased mitotic index and nucleolin gene expression [START_REF] Reichler | Light differentially regulates cell division and the mRNA abundance of pea nucleolin during de-etiolation[END_REF]. In contrast, recent data show that light-grown roots have shorter meristems indicating a reduction in meristematic activity [START_REF] Silva-Navas | D-Root: a system for cultivating plants with the roots in darkness or under different light conditions[END_REF], 2016). The mechanism of this reduction involves its mediation by flavonols, metabolites whose production is induced by light irradiation [START_REF] Buer | The transparent testa4 mutation prevents flavonoid synthesis and alters auxin transport and the response of Arabidopsis roots to gravity and light[END_REF] and have an effect in the reduction of the root meristematic cell proliferation rate [START_REF] Silva-Navas | D-Root: a system for cultivating plants with the roots in darkness or under different light conditions[END_REF], 2016).

Previous experiments under real or simulated microgravity have shown that the signal of the lack of a defined gravity vector is sensed by root meristematic cells in young seedlings, and the result is the disruption of the meristematic competence (Matia et al. 2010). In these cells, this alteration of gravity produces an increase in the cell proliferation rate and a depletion of cell growth, estimated through the rate of ribosome biogenesis, and an increase in the root length [START_REF] Matía | Plant cell proliferation and growth are altered by microgravity conditions in spaceflight[END_REF][START_REF] Manzano | Meristematic cell proliferation and ribosome biogenesis are decoupled in diamagnetically levitated Arabidopsis seedlings[END_REF][START_REF] Boucheron-Dubuisson | Functional alterations of root meristematic cells of Arabidopsis thaliana induced by a simulated microgravity environment[END_REF]. Similar effects on cell functions have been described

in proliferating cells cultured in vitro [START_REF] Manzano | Early effects of altered gravity environments on plant cell growth and cell proliferation: Characterization of morphofunctional nucleolar types in an Arabidopsis cell culture system[END_REF]. Specific changes in the percentage of cells in each phase cycle have been found in response to gravity alteration, indicating changes in the regulatory processes of cell cycle progression taking place at cell cycle checkpoints.

There is general agreement in that the phytohormone auxin is the key factor mediating the transduction of the signals from light and gravity receptors to the effect on meristematic cells [START_REF] Vandenbrink | Light and gravity signals synergize in modulating plant development[END_REF]. It has been widely demonstrated that changes in light and gravity stimuli primarily result in alterations in the auxin polar transport. Furthermore, it has been shown that flavonoids, whose synthesis is induced by light and gravity stimuli, have an effect on auxin polar transport regulation and auxin accumulation in the root tip [START_REF] Buer | The transparent testa4 mutation prevents flavonoid synthesis and alters auxin transport and the response of Arabidopsis roots to gravity and light[END_REF].

In this context, we have conducted experiments in real and simulated microgravity aimed to understand how gravity and light responses influence each other and how the combined influence of light and gravity shape plant development. Specifically, we have focused on root development and the alterations found in the root meristematic cell growth and proliferation. For real microgravity and fractional gravity experiments, seedlings have been grown in the International Space Station (ISS); for simulated microgravity experiments, we have used specific well-characterized devices (Herranz et al. 2013), such as the Random Positioning Machine (RPM). Our results show that red light is able to compensate for most deleterious effects on the balance of root cell growth and cell proliferation in microgravity.

Materials and methods

Plant materials and the spaceflight experiment

In the space experiment Seedling Growth (SG), seeds of the Landsberg erecta ecotype of Arabidopsis thaliana (WT-Ler) as well as of phytochrome mutants phyA-201 and phyB-1 [START_REF] Kiss | Phytochromes A and B mediate red-light-induced positive phototropism in roots[END_REF] were uploaded to the International Space Station (ISS). Seeds were hydrated and germinated in the European Modular Cultivation System (EMCS) using the culture chambers (cassettes) previously developed for the "Tropi" experiments [START_REF] Correll | Ground-based studies of tropisms in hardware developed for the European Modular Cultivation System (EMCS)[END_REF]. After germination, all seedlings grew in these cassettes for 4 days under continuous white light illumination supplied by an array of LED lights placed in one of the walls of the cassette and 1g gravity, provided by in the EMCS centrifuge, with the gravity vector pointing to the opposite direction of the white light source [START_REF] Kiss | Phototropism of Arabidopsis thaliana in microgravity and fractional gravity on the International Space Station[END_REF][START_REF] Vandenbrink | A novel bluelight phototropic response is revealed in roots of Arabidopsis thaliana in microgravity[END_REF]. Then, seedlings grew for two additional days at the g-level of interest in six runs. Four of these runs, nominally providing µg (≤10 -4 g on ISS), 0.1g, 0.3g, and 1g (in flight-control) were carried out in the SG1 mission, flown in March-May 2013. The two last runs, nominally providing 0.5g and 1g (in flight-control) took place in the SG2 mission, flown in November-December 2014. In all cases, the g vector was obtained by the EMCS centrifuge, and the growth of seedlings occurred under continuous red light illumination supplied by a unilateral light source (Fig. 1).

Throughout all steps of the experiments, images were taken from samples and downlinked to the Earth for the analysis of tropisms and morphometric parameters [START_REF] Vandenbrink | A novel bluelight phototropic response is revealed in roots of Arabidopsis thaliana in microgravity[END_REF]. At the end of the 6-day growth period, cassettes were immediately removed from the EMCS and frozen at -80ºC in the MELFI freezer on the ISS. In this state, samples were downloaded to the Earth at -80ºC in the SpaceX capsule and retrieved in our laboratory for transcriptomic analysis.

On-ground reduced gravity experiment design using the RPM

In order to validate real microgravity findings (under photostimulation) and provide a comparison with previous experiments on ISS (in dark conditions), a complementary experiment on simulated microgravity (≤10 -4 g on RPM sample area), in a Random

Positioning Machine [model 2.0 (desktop RPM), Airbus Defense and Dutch Space] (Borst 2009) was performed using the same phytochrome mutants and WT-Ler. It consisted of growing seedlings for 4 days under white light photoperiod (16 h light / 8 h darkness), followed by 2 days under either continuous red light or darkness (Fig. 2).

RPM experimental hardware was provided to ensure similar illumination conditions to that provided by the TROPI hardware on the EMCS [START_REF] Millar | A novel phototropic response to red light is revealed in microgravity[END_REF][START_REF] Kiss | Phototropism of Arabidopsis thaliana in microgravity and fractional gravity on the International Space Station[END_REF]; namely the distance from LEDs to seedlings was approx. 40mm, photosynthetic photon flux were 40 µmol m -2 s -1 for White X4 Ultra Bright LED Lamps Reference AND420HWA X6 and 20 µmol m -2 s -1 for Red LED Lamps Reference AND155CRP

(Purdy Electronics Corporation). The gravity conditions (either simulated microgravity, or control 1g) were constant throughout the entire growth period of 6 days. Images of seedlings were taken at the end of the experiment. Then, a part of the samples were fixed for 2 h in 1.5% (v/v) glutaraldehyde or 4% (v/v) paraformaldehyde in PBS for microscopical observation, and the other part were frozen at -80ºC for transcriptomic analysis.

Sample processing for microscopy analyses

Seedlings fixed in aldehydes were processed for microscopical observation. were observed unstained by phase contrast microscopy (Leica DM2500 microscope with a CCD Leica DFC320 camera) and the differences in cell proliferation rate were estimated by counting the number of cells per millimeter in the cellular rows corresponding to future epidermis and cortex of the root meristem in each condition.

Sample processing for transcriptional analyses

Frozen samples were dissected and only the roots were processed for transcriptomic analysis with micropestles (Sigma #Z317314) using a plant specific RNA extraction column kit (Norgen Bioteck #48400). RNA quality was determined by Nanodrop2000

(Thermo) assays after DNAse-turbo (AmbioLife#1907) treatment. Quantitative RT-PCR (qPCR) was performed in a single step with a IQ5 Real Time Detection System (Biorad) with probes corresponding to selected genes involved in auxin polar transport, cell cycle regulation, and ribosome biogenesis (Suppl. Table S1). The nucleic acid stain used was SYBR green (Agilent Technologies #600886). RT-qPCR were performed in standardized conditions (35 cycles and 60ºC hybridization phase temperature) after two melting curve quality tests of the primer pairs. First, we tested that all probes combinations produced a single peak (amplicon) and second, the RNA/probe concentration ratios were in the optimal efficiency range of the qPCR reaction. The data were analyzed with IQ5 Optical System Software (Biorad) and processed with Microsoft Excel (Microsoft Office 2007).

Results

Seedling Growth space experiments (micro-g and fractional-g effect on photoactivated samples)

We determined the root length from images of the Seedling Growth-1 (SG1) experiment obtained by on-flight image [START_REF] Vandenbrink | A novel bluelight phototropic response is revealed in roots of Arabidopsis thaliana in microgravity[END_REF]) at the sixth day of growth (Fig. 1). The measurement of the root length showed a general increase in this parameter in all samples grown in micro-g. This increase was significant in all strains, WT-Ler, phyA and phyB mutant samples (Fig. 2a). In addition to the samples exposed to ISS microgravity environment, the SG experiment included the exposure of a similar set of samples to centrifugation to generate nominal 0.1g, 0.3g and 0.5g conditions. Root length showed a general increase in all partial gravity samples, but the intensity of the effect was reduced as the gravitational load increased, being conspicuous (although not significant for phyA) at 0.1g, remaining significant only in WT-Ler seedlings at 0.3g and only appearing as a small reduction trend in the 0.5g samples, irrespective of the strain used (Fig. 3). The roots of phyB mutant samples were longer than those of WT-Ler in all cases. This root elongation as a consequence of microgravity is in agreement with previous results in our spaceflight experiments [START_REF] Kiss | Phototropism of Arabidopsis thaliana in microgravity and fractional gravity on the International Space Station[END_REF][START_REF] Vandenbrink | A novel bluelight phototropic response is revealed in roots of Arabidopsis thaliana in microgravity[END_REF].

In a transcriptomic study, we included qPCR experiments performed from root RNA extracted from the SG1 and SG2 samples that were frozen on-board and recovered from the ISS. The analysis revealed the differences in the expression of three sets of genes, selected as markers of different functions. The first set was composed by two genes involved in auxin polar transport and perception (PIN-Formed 2, PIN2 and Transport

Inhibitor Response 1 TIR1); the second set was composed by genes involved in cell cycle regulation (Casein Kinase 2 A CK2A and Cyclin B1, CYCB1), indicative of cell proliferation; finally, the third set was composed by genes involved in the regulation of ribosome biogenesis (fibrillarin, FIB, and nucleolin 1, NUC1), indicative of cell growth (Fig. 4). Interestingly, the pattern of gene expression obtained in microgravity and partial gravity conditions was quite different. Under microgravity, in WT-Ler and phyB mutant, all gene markers were upregulated when compared to 1g control. Markers of auxin polar transport, cell cycle progression and ribosome biogenesis showed an enhanced expression in microgravity, especially for CYCB1 in WT-Ler. The enhancement was more significant for WT-Ler than for phyB. In contrast, phyA mutant, showed no significant variations in all the analyzed genes, with a slight decrease in expression in the cell growth marker genes (Fig. 4a).

Under partial gravity (Fig. 4b-d), the WT-Ler samples showed very small and nonsignificant alterations, except in the auxin polar transport genes. In contrast to the microgravity results, both phyA and phyB samples showed a similar trend under partial g, with significant decrease in the expression of at least one of the genes used to evaluate cell growth, cell proliferation and auxin polar transport. Noteworthy, those differences were proportional to the g-load, but always remaining relatively small (barely reaching a 2-fold decrease in the most variable sample). The samples exposed to 0.1g showed small variations, some of them statistically significant, which became more consistent and quantitatively much more evident in the 0.3g and 0.5g samples. A decrease in the expression of both nucleolar marker genes in the phyA and phyB mutants, but not in the wild type, and in the PIN2 in all strains, was clearly observed.

RPM simulated microgravity experiment (red light photostimulation effect)

With the purpose of extending the information obtained from the spaceflight experiments, including a broader range of methodological approaches, a complementary experiment was carried out on ground, using simulated microgravity (desktop RPM 2.0 in real random mode). We used the same set of genotypes as in the space experiment, and similar illumination conditions as described in the material and methods section.

The main difference with the space experiment was during the first four days of growth, when seedlings grew under simulated microgravity, combined with a 16h/8h photoperiod of white light illumination, whereas in space, growth during this period was at 1g under continuous white light. These conditions allow doing two complementary experiments: (i) The goal of the ISS experiment was to characterize the response of seedlings when they are exposed simultaneously to the two factors implicated: microgravity and red light. Thus, seedlings grew firstly with the same conditions (4 days under 1g, continuous light) in order to obtain a homogeneity population before exposure to migro-g or fractional-g and red light or darkness. (ii) The goal of the experiment performed on-ground was to characterize the response of the plants to the red light once they have already responded to the microgravity. Thus, these conditions allow analyzing the response of the plant to a supplementary factor. In this experiment, we chose the environmental factors that allow plant metabolism as close as possible to the physiological conditions: 4 days of photoperiod of 16h / 8h with white light.

In both cases, spaceflight and RPM, the conditions of the last two days included experimental gravity levels and either continuous red light, or darkness (Fig. 2). A control 1g experiment, applying the same parameters of illumination, was run in parallel.

This experimental setup allowed us to perform a preliminary test on whether or not the mutant line and the illumination conditions could produce any effect per se on the parameters under study, irrespective of the gravity alteration. The test was performed on the samples grown under 1g gravity control conditions, by comparing samples grown under different illumination conditions, and we measured the root length, the proliferation rate and the size of the nucleolus, a parameter indicative of cell growth, in root meristematic cells. Our initial analysis, showed very small changes in root length (Fig. 2). However, when quantified, roots were longer in the phyB mutant than in WT-Ler, reaching statistical significance only in non-photoactivated conditions (Fig. 5).

Red-light photoactivated seedlings were consistently longer than non-photoactivated ones, even though statistical significance was only reached in the phyA mutant (Fig. 5).

Cell proliferation rate was determined in the three cellular layers of the root meristem, from microscopy preparations of fixed samples (Suppl. Fig. S1). The result was that the cell proliferation rate in the root meristem was significantly enhanced by red light in WT-Ler and in phyB mutant (Fig. 6). In turn, cell growth rate was estimated in the same root meristematic cells by measuring the nucleolar size as detected by immunofluorescence studies with anti-fibrillarin antibodies (Suppl. Fig. S2), resulting in the detection of a similar enhancement in this parameter (Fig. 7). Therefore, red light illumination itself appeared to have an enhancing effect on these essential root developmental parameters of meristematic cells, except in the case of phyA.

RPM simulated microgravity experiment (microgravity effect)

When comparing the simulated microgravity samples with the corresponding 1g control, we found that seedlings were more disoriented to one another, as expected by the RPM randomization of the gravity vector. The longer and continuous simulated microgravity condition throughout the entire period of growth, from germination, can account for this feature, although the orientation of the seedlings is maintained due to the spatial reference of the source of light (Fig. 2). The simulated microgravity effects on WT-Ler and the phyB mutant were quite similar to each other and they showed marked differences with the results of the phyA mutant, consistently with the spaceflight SG1 data. Thus, simulated microgravity increased the root length on seedlings not photoactivated with red light of WT-Ler and phyB mutant, but not in the phyA mutant, and the red light treatment reversed this effect, producing roots of similar length, irrespective of the genotype and of the gravity condition (Fig. 8).

Regarding the proliferation rate, simulated microgravity enhanced cell proliferation in WT-Ler and phyB in comparison to the 1g control, and red light reversed this effect, even producing a decrease in this parameter (Fig. 9). The phyA mutant showed the opposite behavior, with a decrease of the cell proliferation rate under simulated microgravity, which was suppressed by red light irradiation (Fig. 9). The increase of the proliferation rate in WT-Ler and phyB mutant exposed to simulated microgravity, but not to red light photoactivation, was in agreement with the observed trend to overexpression of CYCB1 cell cycle regulation gene in these lines without photoactivation (Fig. 10). In contrast, the phyA mutant showed a significant depletion in the expression of the cell cycle genes (Fig. 10a). In general, the effects of simulated microgravity at the level of gene expression were completely reverted by red light photostimulation (Fig. 10b).

In terms of cell growth rate (estimated by the nucleolar size), a general trend was detected under simulated microgravity, consisting of the decrease of this parameter in all non-photostimulated samples. This decrease reached statistical significance in both phyA and phyB mutants (Fig. 11). Red light treatment did not alter this decrease except in the case of the phyA mutant and, interestingly, the WT-Ler showed the most intense effect in the simulated microgravity treatment (Fig. 11). These general inhibiting effects, estimated by microscopical methods, were in agreement with the downregulation of nucleolin and fibrillarin genes observed without red light photostimulation (particularly significant in phyB mutant, Fig. 10a). However, in terms of gene expression, the red light treatment was able to recover fibrillarin and nucleolin transcriptional levels (Fig. 10b).

Discussion

Our results involving microgravity (real or simulated), or partial g conditions (SG1/SG2), in combination with red light photostimulation, show complex responses in plants, particularly regarding the differential effects observed in the phytochrome mutants exposed to several low-gravity conditions. Given that the experiments involved a photoactivation treatment with red light during the two last days of seedling growth, it is worth noting that both phyA and phyB mutants have a compromised perception of the red light stimulus. Therefore, differential plant development responses could be expected under the combined light and gravity conditions that have been tested [START_REF] Kiss | Phytochromes A and B mediate red-light-induced positive phototropism in roots[END_REF], especially taking into account that phytochrome B is the main receptor specialized in red light sensing and perception under 1g conditions [START_REF] Molas | Phototropism and gravitropism in plants[END_REF].

Red light photoactivation compensates for the deleterious effects on microgravity on cell growth and proliferation

Taking together, our results show a positive effect of the red-light photoactivation treatment in the compensation of the alterations induced by the microgravity environment on cell growth and proliferation. As shown by our laboratories in previous papers [START_REF] Matía | Plant cell proliferation and growth are altered by microgravity conditions in spaceflight[END_REF][START_REF] Manzano | Meristematic cell proliferation and ribosome biogenesis are decoupled in diamagnetically levitated Arabidopsis seedlings[END_REF][START_REF] Boucheron-Dubuisson | Functional alterations of root meristematic cells of Arabidopsis thaliana induced by a simulated microgravity environment[END_REF]), the main alteration observed in root meristematic cells grown in a microgravity environment is the disruption of meristematic competence, that is, the coordination of cell proliferation and cell growth [START_REF] Mizukami | A matter of size: developmental control of organ size in plants[END_REF]. Whereas under microgravity, the cell proliferation rate is enhanced and the cell growth rate is inhibited, photoactivation by red light appears to restore the coordination, although important differences can be observed in different experimental conditions. In Table 1, we summarize the effects of the different environmental parameters (illumination/gravitational conditions) for each one of the genotypes tested (WT-Ler and the phyA and phyB mutants).

The results of the space experiment, incorporating red light photoactivation, showed enhanced expression of marker genes of cell cycle regulation and ribosome biogenesis in the WT-Ler and phyB under SG1 microgravity conditions. The enhancement was more significant for the WT-Ler than for phyB. Taken together, the results on the factors related to cell proliferation and cell growth show a concerted expression of these two sets of genes, which corresponds to the normal physiological pattern of meristematic cells, and should be more likely due to the red light photoactivation.

Furthermore, red light shows a general positive effect on the transduction of the light and gravity signals mediated by auxin [START_REF] Medina | Microgravity environment uncouples cell growth and cell proliferation in root meristematic cells: The mediator role of auxin[END_REF]) by inducing the normalization of the auxin polar transport, which is affected under microgravity conditions [START_REF] Herranz | Mechanisms of disruption of meristematic competence by microgravity in Arabidopsis seedlings[END_REF]. Also in this case, there are experimental differences depending on particular conditions.

Under simulated microgravity, the results showed a similar trend as reported in previous studies regarding the effects of the gravity alteration on the rates of cell growth and cell proliferation in the root meristem, and a partial reversion of these effects by the red light photoactivation, which occurs mainly through cell proliferation, but not cell growth as shown in our data. The result was that, after red light irradiation, both parameters appeared reduced in comparison to the values obtained in control 1g gravity. Since both microgravity (lack of gravitropic stimulus) and red light irradiation may be increasing the cell proliferation rate through similar pathways (e.g. those involving CK2 or PIF activity [START_REF] Herranz | Mechanisms of disruption of meristematic competence by microgravity in Arabidopsis seedlings[END_REF])), the absence of gravity should have minimal effects if the pathway is already activated by red light. The phytohormone auxin is the key factor mediating transduction of the signals from light and gravity receptors to the effect on meristematic cells [START_REF] Vandenbrink | Light and gravity signals synergize in modulating plant development[END_REF]. In turn, flavonoids, whose synthesis is also induced by light and gravity stimuli, have an effect on auxin polar transport regulation and auxin accumulation in the root tip [START_REF] Buer | The transparent testa4 mutation prevents flavonoid synthesis and alters auxin transport and the response of Arabidopsis roots to gravity and light[END_REF]. It is also known that auxin controls both cell cycle progression (through cyclinA-B and CDK-A gene expression) and cell elongation. Considering all these facts, we assume that the concerted action of the gravitropic and light stimuli could lead to the modification of the auxin polar transport and, consequently, to the re-distribution of this hormone. The effect of such a new distribution on the root meristem would be the induction of alterations in cell growth and proliferation, allowing a partial reversion of microgravity effects by red light photoactivation.

However, in terms of meristematic competence, our results suggest that red light irradiation restored the concert of the two parameters that define this feature. Actually, we have found different cases in which meristematic competence (coordination of cell growth and proliferation) is restored after its disruption caused by microgravity. We have defined in Table 1 with the term "fast" the case of enhancement of both parameters and upregulation of the marker genes, and with the term "slow" the situation of concerted reduction of the rates of both parameters and downregulation of the marker genes. In the case of "slow" meristematic competence, cells are indeed "competent" but they show a depleted level of activity (i.e. decelerated growth and proliferation).

An additional clarification of the role played by red light in the regulation of the functions of meristematic cells was obtained from the comparison of the 1g control runs of the RPM experiment, with and without the red light photoactivation pulse. These experiments are in agreement with the interpretation of the results of the space experiment, suggesting that red light irradiation is indeed a positive factor for restoration of meristematic competence disrupted by microgravity.

Differential microgravity responses in the two phytochrome mutants

While the results of phyB are very similar to WT-Ler under microgravity, phyA seems to be much more resilient to the absence of the gravity vector, showing quite normal meristematic competence parameters despite a longer root length is detected in these conditions. In fact, the phyA mutant keeps meristematic competence either in real or simulated microgravity in the dark, but showing reduced rates of cell proliferation and cell growth ("slow" meristematic competence). Both of these rates are increased by red light photoactivation, reaching values slightly higher than those reached by the in-flight or in-ground 1g controls. These results support once again that there is a contribution of the light/phototropic signal to the cell proliferation regulation, which can counteract some of the deleterious effects of the loss of gravitational signal in darkness. In turn, the phyB mutant is able to recover the meristematic competence, which was lost in real or simulated microgravity, as an effect of the photoactivation pulse with red light. This recovery is of the "fast" mode (i.e., the same as the WT-Ler, Fig. 4, Table 1).

The magnitude of the changes induced by red light photoactivation in the phyA mutant is not statistically significant in any real or simulated microgravity experiment.

Therefore, these changes should be only taken as general trends, better than real alterations severely affecting functional processes. Otherwise, if we analyze the effects of the genetic mutation irrespective of the environmental conditions, the phyA mutant shows a lower cell proliferation rate and higher cell growth activity compared to WT-Ler.

A higher activity of auxin polar transport is also observed in photoactivated WT-Ler and phyB mutant, but not in the phyA mutant. This observation is an additional indication that the link between the phototropic signal sensed by phyA and auxin polar transport and perception may be important to restore the functional balance that gives account of meristematic competence.

It is important to note that these effects of light and altered gravity on the WT-Ler and phytochrome mutants are accompanied by an altered phototropic behavior of these strains under spaceflight conditions, as we have found from our previous spaceflight experiments [START_REF] Millar | A novel phototropic response to red light is revealed in microgravity[END_REF][START_REF] Kiss | Phototropism of Arabidopsis thaliana in microgravity and fractional gravity on the International Space Station[END_REF]. In fact, all strains show a number of phototropic responses in space that are not observable under 1g conditions; particularly, the roots show a positive red-light phototropic response inversely correlated to the magnitude of the gravity vector [START_REF] Vandenbrink | A novel bluelight phototropic response is revealed in roots of Arabidopsis thaliana in microgravity[END_REF].

Differential gravitational responses at partial-g levels

We have found a decrease in both cell growth and cell proliferation rates (not leading to meristematic competence alteration but just to a "slower" activity level) at the 0.3g-0.5g level. These observed effects occur in the two phytochrome mutants, but not in the WT-Ler (see Table 1), suggesting that an optimal perception of the red stimulus could be enough to compensate the alterations related to the partial gravity environments. These observations depict an increasingly complex scenario for the plants, in which both a stimulating phototropic signal and a 0.3-0.5g load are pulling in opposite directions the developmental pattern of the cell. In this case, the auxin polar transport is decreased by the increasing partial gravity signal, and this effect also occurs in the wild type. It was shown that statoliths could still be sensing gravity in the root tips at 0.1g-0.5g, apparently triggering a normal gravitropism signal [START_REF] Perbal | Sensitivity to gravistimulus of lentil seedling roots grown in space during the IML 1 Mission of Spacelab[END_REF][START_REF] Kiss | Plant biology in reduced gravity on the Moon and Mars[END_REF]).

The importance of simulated microgravity experiments in ground-based facilities

This work also emphasizes the advantages of complementing space experiments performed in real microgravity with parallel experiment in ground-based simulated microgravity facilities [START_REF] Kraft | Plastid position in Arabidopsis columella cells is similar in microgravity and on a random-positioning machine[END_REF][START_REF] Herranz | Ground-based facilities for simulation of microgravity, including terminology and organism-specific recommendations for their use[END_REF][START_REF] Boucheron-Dubuisson | Functional alterations of root meristematic cells of Arabidopsis thaliana induced by a simulated microgravity environment[END_REF]. The results obtained in real microgravity (ISS experiment) were clearer, showing greater differences between experimental and control samples. However, ground-based facilities provided us with the possibility of testing more parameters and conditions, providing supporting data of high interest. In addition, the cross-comparison of real and simulated microgravity experiments allows improving the simulation technologies on ground. Further experimentation in partial g conditions should be considered in the near future, since the alterations observed in both the phototropic response [START_REF] Vandenbrink | A novel bluelight phototropic response is revealed in roots of Arabidopsis thaliana in microgravity[END_REF]) and the cell growth/cell proliferation balance may be important to define the Mars-related settings for plant breeding below the Earthnominal gravity level [START_REF] Kiss | Plant biology in reduced gravity on the Moon and Mars[END_REF]. In fact, particular mutants, as phyA, less sensible to the meristematic competence lost under low gravity and light conditions, could be good candidates to be added into extraterrestrial life support systems. Nevertheless, even considering the different effects observed under real or simulated microgravity, and even the partial gravity levels in the range close to Moon and Mars surface, most of parameters disturbanced were affected in a similar manner. Red light photoactivation is able to coordinate the rates of cell proliferation and cell growth, and, thus, red light causes the recovery of the meristematic competence after gravitational stress.

Figure Captions

  Miguel A Valbuena, Aránzazu Manzano, Joshua P Vandenbrink, Veronica Pereda-Loth, Eugénie Carnero-Diaz, Richard E Edelmann, John Z Kiss, Raúl Herranz, F. Javier Medina To cite this version: Miguel A Valbuena, Aránzazu Manzano, Joshua P Vandenbrink, Veronica Pereda-Loth, Eugénie Carnero-Diaz, et al.. The combined effects of real or simulated microgravity and red-light photoactivation on plant root meristematic cells. Planta, 2018, 248 (3), pp.691-704. 10.1007/s00425-018-2930-x. hal-04045460

Fig. 1

 1 Fig. 1 Images of 6-day-old seedlings from SG1 and SG2 experiments performed on the International Space Station. One culture chamber (cassette) per experimental condition is shown, taken at the end of the experiment (6 days after seed hydration). Seedlings of the different genotypes (WT-Ler, phyA and phyB mutants) were grown either under microgravity (µg), partial gravity (0.1g, 0.3g or 0.5g) or 1g FLIGHT control conditions. All samples were exposed to continuous white light for 4 days, from top of the cassette and with continuous red light (photoactivation) for the last two days, with the light source located laterally (left), as shown in the figure.

Fig. 2

 2 Fig. 2 Images of 6-day-old seedlings from the RPM experiment. a Samples with no photoactivation. b Samples with red light photoactivation the last 2 days of the experiment. The same genotypes used in the ISS experiment were exposed to simulated microgravity (RPM) or 1g control conditions. Seedlings were grown for six days after seed hydration, the first four days under light photoperiod, and the last two days seedlings were either subjected to red light photoactivation or remained in darkness. Scale bar indicates 3 mm.

Fig. 3

 3 Fig. 3 Length of the primary root in seedlings grown in the ISS (SG1 and SG2 experiments) at µg (a), 0.1g (b), 0.3g (c), 0.5g (d) next to the corresponding 1g FLIGHT control. Bars indicate standard deviation. Asterisks indicate significant difference (t test, P<0.05) between samples of the same genotype exposed to 1g FLIGHT gravity conditions (EMCS centrifuge).

Fig. 4

 4 Fig. 4 Relative gene expression estimated by quantitative PCR (qPCR) in samples of the SG1/SG2 space experiment [nominally 0g (µg), 0.1g, 0.3g and 0.5g]. Three sets of genes have been analyzed, which were two genes of the auxin polar transport perception (EIR1, also known as PIN2, and TIR1), two genes of the cell cycle regulation (CK2A and CYCB1) and two genes involved in the regulation of ribosome biogenesis (FIB and NUC1). Histograms indicate the difference in expression (Log2 Ratio between the same samples grown in µg (a), 0.1g (b), 0.3g (c), 0.5g (d) versus 1g FLIGHT control). Asterisks indicate significant values (t test, P<0.05).

Fig. 5

 5 Fig. 5 Primary root length in seedlings grown under control 1g conditions, in the context of the simulated microgravity experiment. Bars indicate standard deviation. Asterisks indicate significant difference (t-test, P<0.05) between samples of the same genotype in different illumination conditions. The symbol (#) indicates significant difference (t-test, P<0.05) with respect to WT-Ler under the same illumination regime.

Fig. 6

 6 Fig. 6 Cell proliferation rate in root meristematic cells of seedlings grown under control 1g conditions, in the context of the simulated microgravity experiment. Bars indicate standard deviation. Asterisks indicate significant difference (t-test, P<0.05) between samples of the same genotype in different illumination conditions. The symbol (#) indicates significant difference (t-test, P<0.05) with respect to WT-Ler under the same illumination regime.

Fig. 7

 7 Fig. 7 Cell growth rate (estimated by nucleolus cross-sectional area), in root meristematic cells of seedlings grown under control 1g conditions, in the context of the simulated microgravity experiment. Bars indicate standard deviation. Asterisks indicate significant difference (t-test, P<0.05) between samples of the same genotype in different illumination conditions. The symbol (#) indicates significant difference (t-test, P<0.05) with respect to WT-Ler under the same illumination regime.

Fig. 8

 8 Fig. 8 Primary root length in seedlings grown under simulated microgravity (RPM) and control 1g conditions. a Samples with no photoactivation. b Samples with red light photoactivation during the last 2 days of the experiment. Bars indicate standard deviation. Asterisks indicate significant difference (t-test, P<0.05) between samples of the same genotype in different gravity conditions.

Fig. 9

 9 Fig. 9 Cell proliferation rate in root meristematic cells of seedlings grown under simulated microgravity (RPM) and control 1g conditions. a Samples with no photoactivation. b Samples with red light photoactivation during the last 2 days of the experiment. Bars indicate standard deviation. Asterisks indicate significant difference (t-test, P<0.05) between samples of the same genotype in different gravity conditions.

Fig. 10

 10 Fig. 10 Relative gene expression estimated by quantitative PCR (qPCR) in samples of the RPM experiment. a Samples with no photoactivation. b Samples with red light photoactivation during the last 2 days of the experiment. Three sets of genes have been analyzed, namely two genes of the auxin polar transport and perception (EIR1, also known as PIN2, and TIR1), two genes of the cell cycle regulation (CK2A and CYCB1), and two genes involved in the regulation of ribosome biogenesis (FIB and NUC1). Histograms indicate the difference in expression (Log2 Ratio between the same samples grown in RPM (simulated µg versus 1g control). Asterisks indicate significant values (ttest, P<0.05).

Fig. 11

 11 Fig. 11 Cell growth rate (estimated by nucleolus cross-sectional area), in root meristematic cells of seedlings grown under simulated microgravity (RPM) and control 1g condition. a Samples with no photoactivation. b Samples with red light photoactivation during the last 2 days of the experiment. Bars indicate standard deviation. Asterisks indicate significant difference (t-test, P<0.05) between samples of the same genotype in different gravity conditions.

  Compared to 1g control (red light). (2) Compared to 1g -no red light. =: No clear variations. : Significant increase (P<0,05); : Low-significant increase (P>0,05, Log2Ratio>.5); : Significant decrease (P<0,05); : Low-significant decrease (P>0,05, Log2Ratio<-.5)

Table 1

 1 Summary of the combined effects of microgravity (real -ISS or simulated -RPM) or partial gravity (0.1g, 0.3g, 0.5g on ISS) and red light photoactivation on functional parameters of root meristematic cells. Note the similar strategy in WT-Ler and phyB to deal with microgravity environments while phyA/phyB mutants share a similar fractional gravity response.

	WT-Ler	Cell Proliferation Rate Genes	Cell Growth Nucleolus Genes	Meristematic Competence	Auxin	Root length
	ISS: Microgravity +Red Light (1)					YES, fast		
	RPM: 1g control +Red Light (2)					YES, fast		
	RPM: Sim µg No Red Light (2)					NO	=	
	RPM: Sim µg + Red Light (1)				=	YES		=
	ISS:Partial 0.1-0.5g + Red Light (1)		=		=	YES		
	phyB mutant	Cell Proliferation Rate Genes	Cell Growth Nucleolus Genes	Meristematic Competence	Auxin	Root length
	ISS: Microgravity +Red Light (1)					YES, fast		
	RPM: 1g control +Red Light (2)					YES, fast		
	RPM: Sim µg No Red Light (2)					NO	=	
	RPM: Sim µg + Red Light (1)		=		=	YES		=
	ISS:Partial 0.1-0.5g + Red Light (1)					YES, slow		
	phyA mutant	Cell Proliferation Rate Genes	Cell Growth Nucleolus Genes	Meristematic Competence	Auxin	Root length
	ISS: Microgravity +Red Light (1)		=			YES	=	
	RPM: 1g control +Red Light (2)					YES, fast		
	RPM: Sim µg
	No Red Light

Table 1

 1

Acknowledgments

Funding for this study was provided mainly by the Spanish National Plan for Research and Development (MINECO-ERDF co-funding) Grant ESP2015-64323-R to FJM. The access to ISS and RPM facilities was granted by ESA-ELIPS ILSRA-2009-0932 to FJM and GBF Program GIA Project (contract# 4000105761) to RH. This research was supported also by grants (NNX12A0656 and 80NSSC17K0546) from NASA to JZK and the French Space Agency-CNES to ECD and VPL. MAV and AM were recipients of grants of the Spanish National Program for Young Researchers Training (Refs. BES-2010-035741 and BES-2013-063933, respectively). We would like to thank the skillful technical assistance of Mrs. Mercedes Carnota (CIB-CSIC), the fine support of NASA's

Ames Research Center in the use of TROPI hardware, and the European Space Agency and the Norwegian User Support and Operations Center (N-USOC) for their continuous support throughout the entire "Seedling Growth" space project. Finally, we would like to thank the astronauts on board the ISS for their work, without whom these experiments would have not been possible.

Supplementary material online

Two figures and one table have been included as supplementary material to this article.

Author contribution statement

All authors participated in the design of the experiments and in the preparation of the spaceflight project, except VPL, who provided the simulated microgravity facility.

MAV, AM and JPV prepared and processed the spaceflight and RPM seedling samples. JZK, RH and FJM wrote the manuscript. All authors read and approved the manuscript.


Aurore Loisy
email: aurore.loisy@irphe.univ-mrs.fr

Robin A Heinonen

Deep reinforcement learning for the olfactory search POMDP: a quantitative benchmark

Keywords: olfactory search, source localization, POMDP, reinforcement learning, sniffer robots

The olfactory search POMDP (partially observable Markov decision process) is a sequential decision-making problem designed to mimic the task faced by insects searching for a source of odor in turbulence, and its solutions have applications to sniffer robots. As exact solutions are out of reach, the challenge consists in finding the best possible approximate solutions while keeping the computational cost reasonable. We provide a quantitative benchmarking of a solver based on deep reinforcement learning against traditional POMDP approximate solvers. We show that deep reinforcement learning is a competitive alternative to standard methods, in particular to generate lightweight policies suitable for robots.

Introduction

Partially observable Markov decision processes (POMDPs) provide an elegant mathematical framework to model decision-making in the face of uncertainty [START_REF] Åström | Optimal Control of Markov Processes with Incomplete State Information[END_REF][START_REF] Smallwood | The Optimal Control of Partially Observable Markov Processes over a Finite Horizon[END_REF][START_REF] Cassandra | Proceedings of the Twelfth AAAI National Conference on Artificial Intelligence[END_REF]. They generalize MDPs (Markov decision processes) to situations where the agent has only access to partial information about the state of the world, for example through sensors. In real life, partial observability is the rule rather than the exception, and an important application of POMDPs is robot navigation [START_REF] Cassandra | Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS '96[END_REF][START_REF] Thrun | Probabilistic Robotics[END_REF].

The olfactory search POMDP is a navigation problem where the agent must find a source of odor in a turbulent flow using information provided by odor detection events [START_REF] Vergassola | Infotaxis" as a strategy for searching without gradients[END_REF]. This task is faced by insects searching for food or mates using their sense of smell [START_REF] Murlis | Odor Plumes and How Insects Use Them[END_REF][START_REF] Vickers | Mechanisms of animal navigation in odor plumes[END_REF][START_REF] Cardé | Navigation Along Windborne Plumes of Pheromone and Resource-Linked Odors[END_REF], but also by sniffer robots used to locate gas leaks, land mines and explosives [START_REF] Russell | Odour Detection by Mobile Robots[END_REF]. Far from a toy problem, the olfactory search POMDP reproduces the key features of odor detection in turbulence: sparsity and stochasticity [START_REF] Celani | Odor Landscapes in Turbulent Environments[END_REF]. It can be used to assess and compare possible search strategies [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF] before implementing them in real robots [START_REF] Lochmatter | Bio-inspired and probabilistic algorithms for distributed odor source localization using mobile robots[END_REF][START_REF] Moraud | Effectiveness and robustness of robot infotaxis for searching in dilute conditions[END_REF][START_REF] Martinez | Frontiers in Neuroengineering[END_REF]. It also provides a tool to interpret the behavior of olfactory animals [START_REF] Calhoun | Maximally informative foraging by Caenorhabditis elegans[END_REF][START_REF] Voges | Reactive Searching and Infotaxis in Odor Source Localization[END_REF].

The POMDP framework rigorously models uncertainty that arises from partial observability. It allows the agent to compute a probability distribution over possible states of the world (possible source locations) and to update it as new sensory information (odor detections) arrives. This probability distribution, called the belief, is a sufficient statistic of the entire agent's history and completely describes the current uncertainty about the true state of the world (the true source location). Solving the POMDP means computing the optimal action to take as a function of the belief. This solution optimally balances exploration (acting to gain more information about the source location) and exploitation (acting to get closer to the current estimate of the source location).

The price to pay for this careful quantification of uncertainty is computational complexity. Finding the optimal strategy for a POMDP requires to solve a nonlinear functional equation, called the Bellman optimality equation, on the space of beliefs. This problem is computationally intractable and one must rely on approximate solvers [START_REF] Kurniawati | Partially Observable Markov Decision Processes and Robotics[END_REF][START_REF] Kochenderfer | Algorithms for Decision Making[END_REF]. Most popular solvers are "pointbased" and compute the solution by performing value iteration over a subset of beliefs [START_REF] Pineau | Anytime Point-Based Approximations for Large POMDPs[END_REF][START_REF] Shani | A survey of point-based POMDP solvers[END_REF]. A different approach was recently proposed in [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF], where deep reinforcement learning techniques were adapted to the POMDP framework and applied to the olfactory search problem. While this approach was novel, it has not been compared to well-established techniques, which are also able to obtain good approximate solutions for this problem [START_REF] Heinonen | Optimal policies for Bayesian olfactory search in turbulent flows[END_REF].

In this paper we benchmark deep reinforcement learning against standard point-based solvers on the olfactory search POMDP. The POMDP and its (formal) optimal solution are described in Section 2. The reinforcement learning approach to POMDP is presented in Section 3 and contrasted with existing point-based solvers. The benchmark methodologies and results are detailed in Section 4. Conclusions are drawn in Section 5.

The olfactory search POMDP

The olfactory search problem, illustrated in Fig. 1, is a POMDP in which the agent must find a source of odor hidden in a 2D Cartesian grid. At each step, the agent moves to one of the four adjacent cells. If the source is located in Fig. 1 Illustration of the olfactory search POMDP. In this POMDP, the agent must find a source of odor randomly hidden in a grid. At each step, the agent moves to a neighbor cell (action). It sniffs the air and has a small random chance of detecting an odor (observation). The closer it is to the source, the higher is the probability of detection. The search terminates when the agent enters the cell containing the source. The model used to generate odor detections (called "hits") is known, so the agent can maintain a belief (probability distribution over possible source locations) using Bayesian inference. We seek the policy (mapping from beliefs to actions) that minimizes the expected search duration.

this cell, the search is over. Otherwise, the agent receives a stochastic sensor measurement in the form of "hits". Hits represent odor particles detected by the agent. Their likelihood depends on the position of the agent with respect to the source. Therefore hits provide noisy information about the source location. The search continues until the agent enters the cell containing the source. We assume that the agent has a perfect memory and a perfect knowledge of the process that generates hits. The goal is to determine the strategy that the agent should follow in order to minimize the expected number of steps to find the source.

We now formally define this problem in the language of POMDPs. The state s is defined as the relative position of the agent with respect to the source, and belongs to the set S built from all possible combinations of the source's and the agent's positions within a finite-size grid. The special state s Ω = 0 is the terminal state where the agent is located in the cell containing the source. The action a is a move by the agent, and belongs to the set A ={'north,' 'south,' 'east,' 'west'}. As the agent executes an action a, it transitions deterministically to a new state s, receives a reward r = -1, and makes an observation o. Possible observations belong to the set O = {Ω, 0, 1, 2, . . . , h max }. If s = s Ω , the source is found: the agent receives the special observation o = Ω and the search terminates. If s = s Ω , the agent receives an observation o = h where h ∈ {0, 1, 2, . . . , h max } is a number of hits (the maximum number of hits h max will be specified later on). Hits represent odor detections and occur with conditional probability Pr(h | s), which is constructed from a physical model of odor dispersion and detection in turbulence [START_REF] Vergassola | Infotaxis" as a strategy for searching without gradients[END_REF] (cf. Appendix A).

The agent does not have access to its current state. Instead, it maintains a probability distribution over S, called belief and denoted b(s), as an estimate of its state. At each step, after making an observation o, the belief is updated from b to b using Bayes' rule

b (s) = Pr(o | s)b(s) s∈S Pr(o | s)b(s) . (1)
This update can made more explicit depending on the nature of the observation. If o = Ω, the source is found and the update then yields b (s) = δ(s Ω ).

We denote this special belief b Ω . Otherwise, o = h and

b (s) = Pr(h | s)b(s) s∈S Pr(h | s)b(s) ∀s = s Ω (2) with b (s Ω ) = 0, where Pr(h | s) is given in Appendix A.
The belief summarizes all the information brought by past observations and actions. It is a sufficient statistic over the agent's history: we can reason equally about beliefs as about agent histories without loss of information. The initial belief, b 0 , is called the "prior" in the language of Bayesian probabilities and is somewhat arbitrary. Here, b 0 is drawn from a set of initial beliefs B 0 . This set is generated by assuming that the search starts with a detection (h > 0) in an infinite domain with a uniform source distribution. The motivation behind this initialization procedure is that the start of the search is not arbitrary, but is triggered at the instant when the agent is informed that there is source (as opposed to nothing) in the neighborhood. It also drastically reduces artificial effects due to the finite size of the search domain. The details of the initialization protocol are provided in Appendix B.1 and [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF].

The search proceeds as follows:

• Initially -The initial belief b 0 is drawn randomly from B 0 . -The state s 0 is drawn randomly from the distribution b 0 .

• At the t th step of the search 1. The agent chooses an action according to some policy π: a t+1 = π(b t ).

2. The agent moves deterministically to the neighbor cell associated with a t+1 . This move is associated with a negative unit reward:

r t+1 = -1.
The state is updated to s t+1 . 3. The agent makes an observation o t+1 and the belief is updated to b t+1 .

-If b t+1 = b Ω (meaning that o t+1 = Ω and s t+1 = s Ω ), the source is found and the search terminates. -Otherwise, the search continues to step t + 1.

Each search (called an episode) is a sequence like this: and the cumulative reward of an episode is equal to minus the number of steps T to termination: T t=1 r t = -T . The agent's behavior is controlled by the policy, denoted π, which maps each belief to an action: a = π(b). The performance of a policy π is measured by E π [T ], the expected number of steps to reach the source. The expectation is taken over all possible sequences generated following policy π starting from all possible initial states s 0 . Solving the POMDP means finding the optimal policy π * that minimizes the expected duration of the search

b 0 , a 1 , r 1 , s 1 , o 1 , b 1 , a 2 , r 2 , s 2 , o 2 , b 2 , . . . , b T -1 , a T , r T , s Ω , Ω, b Ω .
π * = argmin π E π [T ]. (3)
The optimal policy can, at least formally, be determined from the solution of a recurrence equation known as the Bellman optimality equation as follows.

The optimal value function v * (b) of a belief b is defined as the maximum, over all policies, of the expected cumulated reward when starting from this belief b. Here, the reward is a simple unit penalty at each step, so v * (b) is equal to the expected number of steps remaining to find the source up to a minus sign:

v * (b) = max π v π (b) where v π (b) = -E π [T -t | b t = b]. (4)
The optimal value function satisfies the Bellman optimality equation:

v * (b) = -1 + max a∈A b ∈B(b,a) Pr(b | b, a)v * (b ) ∀b = b Ω (5)
The optimal value function cannot be computed exactly due to the size of the belief space. It can, however, be approximated, as explained in the next section.

Approximate POMDP solvers

Exactly solving the Bellman optimality equation of a POMDP is computationally intractable for any but the smallest problems, and a tremendous amount of effort has been devoted to the development of approximation methods [START_REF] Kurniawati | Partially Observable Markov Decision Processes and Robotics[END_REF][START_REF] Kochenderfer | Algorithms for Decision Making[END_REF]. The general idea is to solve the Bellman optimality equation only for a set of sampled beliefs rather than for the entire belief space, thereby substantially reducing the complexity of the problem. In order to obtain good policies, it is key to sample a sufficiently representative set of beliefs; therefore, the sampling strategy is critical to the performance of the method.

In the following we present the two generic approaches that are compared in this paper: model-based deep reinforcement learning and point-based POMDP solvers.

Model-based deep reinforcement learning

The deep reinforcement learning approach to solving a POMDP [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF] consists in approximating the optimal value function by a deep neural network, and training the network to minimize the error on the Bellman optimality equation (Fig. 3).

We denote by v(b; w) the neural network approximation of v * (b) parameterized by weights w. The Bellman optimality equation for the approximate value function reads

v(b; w * ) = -1 + max a∈A b ∈B(b,a) Pr(b | b, a)v(b ; w * ) ∀b = b Ω (7)
with v(b Ω ; w * ) = 0. The problem becomes that of computing the weights w * that minimize the residual error on Eq. 7. This residual error, called the Bellman optimality error, reads

L(w) = E b∼π   -1 + max a∈A b ∈B(b,a) Pr(b | b, a)v(b ; w) -v(b; w)   2 (8)
where the expectation is taken over beliefs b visited when following the policy π derived from v: Using neural network terminology, the functional L(w) is the "loss function" to minimize and "training" the network refers to the iterative update of the weights w using stochastic gradient descent. The intuition behind this deep reinforcement learning approach is the following. At the beginning of the training, v is initialized with random weights w. As the consequence, the policy π used to collect beliefs, which is derived from v, is random. At each training iteration, the weights are adjusted such that v becomes a better approximation of the true optimal value function on a collection of beliefs gathered by following π (Eq. 8). As v is improved, beliefs collected from π become more representative of the beliefs visited by the optimal policy. This allows to improve v even further. This iterative process continues until convergence to v ≈ v * and π ≈ π * .

π(b; w) = argmax a∈A b ∈B(b,a) Pr(b | b, a)v(b ; w). (9)
The training algorithm is a model-based version of DQN (Deep Q-Network) which relies on two stabilizing techniques to facilitate convergence (which in general is not guaranteed): experience replay and delayed target network [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF]. It is model-based because it takes advantage of model knowledge: since the probability of transitioning from a belief b to a successor belief b is known exactly, one can work directly with the value function rather than the actionvalue "Q function" in model-free reinforcement learning, and one can perform full backups (compute the sum over b in equations 7-9) rather than sample backups (estimates based on a single successor belief randomly sampled) in model-free reinforcement learning. The complete algorithm is provided in Algorithm 1.

Standard point-based POMDP solvers

The "classical" approach to approximating the optimal policy for POMDP is to use point-based value iteration (PBVI)1 [START_REF] Pineau | Anytime Point-Based Approximations for Large POMDPs[END_REF][START_REF] Shani | A survey of point-based POMDP solvers[END_REF][START_REF] Pineau | Proceedings of the Eighteenth International Joint Conference on Artifical Intelligence (IJCAI)[END_REF]. PBVI approximates the optimal value function as piecewise-linear and convex, i.e., v(b; Γ) = max

α∈Γ s∈S b(s) • α(s), (10)
for some collection Γ of α-vectors (it has been shown [START_REF] Sondik | The Optimal Control of Partially Observable Markov Decision Processes[END_REF] that v * can be arbitrarily well-approximated by such a function). The challenge consists in constructing the set Γ * such that the Bellman optimality equation

v(b; Γ * ) = -1 + γ max a∈A b ∈B(b,a) Pr(b | b, a)v(b ; Γ * ) ∀b = b Ω (11)
is solved "at best" for v. Note that here we have introduced a discount factor γ = 1 -with 0 < 1, which is required for PBVI. We refer to the Appendix C.2 for more details.

It can be shown that when viewed as an operator acting on v, the righthand-side of Eq. 11 is a contraction and v(•; Γ * ) is a fixed point of this operator. This property is the basis of value iteration, an iterative algorithm that proceeds as follows at the n th iteration:

v(b, Γ n+1 ) = -1 + γ max a∈A b ∈B(b,a) Pr(b | b, a)v(b ; Γ n ) (12)
for b in a collection of beliefs. The set Γ n+1 is built up by performing a "backup" operation on b: this generates a new α-vector which improves the approximation of that belief's value (and presumably that of other beliefs close to it). We refer the reader to the related technical literature [START_REF] Shani | A survey of point-based POMDP solvers[END_REF] for more information on backups. PBVI algorithms differ in how the beliefs to be backed up are chosen and the order in which they are backed up. Due to the size of the belief space, which is frequently very high-dimensional, choosing an efficient scheme is of critical importance.

Perseus [START_REF] Spaan | Perseus: Randomized Point-based Value Iteration for POMDPs[END_REF] constructs its set of beliefs by collecting them along trajectories generated using a heuristic policy. Then the beliefs are backed up, one by one, until every belief satisfies v(b; Γ n+1 ) ≥ v(b; Γ n ) (as a consequence of convexity, backups can only increase the estimated value of a belief). The order of the backups is either random or (our preference) in order of decreasing Bellman error [START_REF] Shani | Prioritizing point-based pomdp solvers[END_REF]. For very large POMDPs (as considered here) where the set of beliefs can only be a very small subset of the entire belief space, the quality of the heuristic used to sample beliefs is key: it must visit beliefs that are representative of the beliefs visited by the optimal policy. Sarsop [START_REF] Kurniawati | Proceedings of Robotics: Science and Systems IV[END_REF], on the other hand, interleaves belief sampling and backups which allows it to be more parsimonious in its selection of beliefs. It tries to construct a tree of beliefs which are reachable from the initial belief b 0 by taking sequences of quasi-optimal actions, pruning branches corresponding to provably suboptimal actions. An advantage of this approach is that it maintains lower and upper bounds on the exact optimal value which progressively tighten as the algorithm proceeds. The algorithm stops when the distance between the bounds for b 0 is sufficiently small. Perseus and Sarsop do not comprise an exhaustive list of PBVI algorithms for POMDPs. However, they are popular, and there is little interest in testing every available solver.

4 Quantitative benchmark

Methods

We consider four test cases which are described in Table 1. In the first two test cases, the problem is isotropic: the search domain is a square grid and the agent starts the search at the center of the domain. Two different problem sizes are considered: a 19 × 19 grid ("smaller") and a 53 × 53 grid ("larger"). In both cases the source emission rate is chosen such that occasional detections are likely to occur during the search. Possible hit values are integers between 0 and 2 or 3. The initial belief is drawn from a small set containing 2 or 3 initial beliefs corresponding with an initial nonzero hit. The setup is identical to that used in [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF]. In the last two test cases, the effect of a mean wind in the positive x-direction is accounted for: the domain is a rectangular grid (81 × 41) and the agent starts downwind at position (66, 21). Two different source emission rates are considered, such that the search is performed either with occasional detections ("with detections") or almost without any detections ("almost no detections"). Possible hit values are either 0 or 1. A single initial belief is used. This setup is similar to that used in [START_REF] Heinonen | Optimal policies for Bayesian olfactory search in turbulent flows[END_REF]. The probability distribution of possible source locations and the initial position of the agent are shown in Fig. 4 for each test case. Additional technical details on the setups are provided in Appendix B.

Approximately optimal policies have been computed using three different computational methods: deep reinforcement learning (DRL), Sarsop [START_REF] Kurniawati | Proceedings of Robotics: Science and Systems IV[END_REF] and Perseus [START_REF] Spaan | Perseus: Randomized Point-based Value Iteration for POMDPs[END_REF]. We briefly summarize our methodology in the remainder of this section; refer to Appendix C for further details.

The DRL method is very generic and has no theoretical restrictions on the type of POMDP it can solve, but involves a large number of hyperparameters. We found empirically that the quality of the solution shows very little sensitivity to most hyperparameters. Only the size of the neural network and the learning rate must be chosen in an appropriate manner (large enough and small enough, respectively). Based on extensive experiments by [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF], we use a learning rate of 10 -3 and a fully connected neural network with 3 hidden layers of 512 neurons for the small isotropic case, and of 1024 neurons for all other cases. We found that increasing the network size further does not yield further improvements to the learned policy.

Sarsop and Perseus are two standard PBVI solvers [START_REF] Shani | A survey of point-based POMDP solvers[END_REF]. In principle, these solvers can only deal with discounted POMDPs (the olfactory search POMDP is undiscounted). In practice we found that good policies can be obtained for the undiscounted POMDP (γ = 1) while solving for the discounted version of the problem (γ < 1). An additional limitation of Sarsop is that it requires a single initial belief, while there are several ones in our isotropic test cases. As one initial belief is much more likely than other ones (cf. Appendix B.1), Sarsop was used considering only this initial belief. The out-of-the-box version of Perseus is unable to obtain good approximate solutions. Nevertheless, using a good heuristic (we use infotaxis [START_REF] Vergassola | Infotaxis" as a strategy for searching without gradients[END_REF]) instead of a random policy to collect beliefs and using reward shaping allowed us to use Perseus for the olfactory search POMDP [START_REF] Heinonen | Optimal policies for Bayesian olfactory search in turbulent flows[END_REF].

Policy evaluation has been performed with OTTO, a software dedicated to the olfactory search POMDP and designed for this purpose [START_REF] Loisy | A Python package to simulate, solve and visualize the source-tracking POMDP[END_REF]. This software has been augmented with the windy setup, which was not present in the original version, and adapted so that it can use policies computed with PBVI solvers. The augmented version of OTTO used for this paper can be found at https://github.com/auroreloisy/otto-benchmark.

As policies, in general, do not guarantee that the source will be always found, a search may never terminate. We prescribe a maximum search time T max for each case (cf. Table 1), which is chosen much larger than the maximum time it takes a good policy to find the source. If the source is not found at t = T max , the episode is marked as 'failure'. The performance of a policy is defined based on two metrics: Mean(T), the mean time to find the source conditioned on the fact that the source is found, and Pr(failure), the probability that a search ends by a failure.

Results

Overall, all solvers are able to obtain decent policies for all test cases. Examples of quasi-optimal search trajectories are provided in Fig. 5. In all cases typical searches roughly consist of an exploratory phase followed by an exploitative phase after the first detections are made. At the beginning of the search, the source can be anywhere within a vast area, and the agent executes long straight moves to explore it efficiently (forming a spiral in the isotropic cases, or long upwind surges in the windy cases). When an odor is detected (nonzero hit), the belief suddenly narrows down to a much smaller area which is very likely to contain the source. The agent then restricts its moves to this area which is searched in a more exhaustive manner.

In the isotropic cases, the main difficulty is the lack of directionality. As the source is initially equally likely to be in any direction, and hits only inform about the distance to the source, the belief typically exhibits a high degree of symmetry around the agent. Committing to a given direction will necessarily incur a large penalty if the source is actually located in any other direction. In the windy cases, the main difficulties are the long distance that typically separates the source and the agent, resulting in longer search times compared to the isotropic cases, and the risk of "missing" the source (passing by it upwind) and leaving the odor plume (this explains why the agent goes back twice in Fig. 5, top right panel). The windy case with almost no detection is an extreme example where the search is very long but almost deterministic: unlike other cases, here the agent is almost sure to detect nothing at each step.

The quantitative performance of the various policies on the four test cases is reported in Fig. 6. Two metrics are considered: Pr(failure), the probability of never finding the source, and Mean(T), the mean time to find the source provided it is ultimately found. Three policies have been computed using numerical solvers: deep reinforcement learning (DRL), Sarsop and Perseus. For Fig. 6 Performance of various policies on the four test cases, measured by the mean search time to find the source (lower is better). The probability of never finding the source is color-coded. Four policies have been computed using numerical solvers: deep reinforcement learning (DRL) [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF], Sarsop [START_REF] Kurniawati | Proceedings of Robotics: Science and Systems IV[END_REF] and its light version (Sarsop-Light) where the number of αvectors is constrained such that the number of free parameters is comparable to DRL, and Perseus [START_REF] Spaan | Perseus: Randomized Point-based Value Iteration for POMDPs[END_REF]. Two heuristic policies are shown for comparison: infotaxis [START_REF] Vergassola | Infotaxis" as a strategy for searching without gradients[END_REF] and space-aware infotaxis (SAI) [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF].

comparison, two state-of-the-art heuristic policies are also shown: infotaxis [START_REF] Vergassola | Infotaxis" as a strategy for searching without gradients[END_REF] and space-aware infotaxis [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF].

DRL beats Perseus and Sarsop on 3 out of 4 test cases (the two isotropic cases, and the windy case with detections). The probability of never finding the source is negligible for the three solvers, and the mean time to find the source is lower with DRL. Perseus and Sarsop perform comparably well. These results also show that space-aware infotaxis is very close to the optimal performance on these three test cases. DRL, however, fails at obtaining a quasi-optimal policy for the windy case with almost no detections, where it performs worse than other solvers and than infotaxis (space-aware infotaxis is not a good policy in this case). This test case is particular, as the search is very long but essentially deterministic as odor detections are extremely rare. The failure of DRL in this scenario could be due to inappropriate hyperparameters, though we performed limited testing of those without success. We speculate that a possible explanation is that epsilon-greedy exploration, which we used in our implementation of DRL, is known to be deficient for problems with long time horizons and should be replaced with a form of "deep exploration" [START_REF] Osband | Advances in Neural Information Processing Systems[END_REF]. However, this is beyond the scope of the present work.

Discussion

While Perseus was found to perform well overall, it is worth reminding the reader that it requires a good heuristic to collect beliefs as well as a clever reward shaping. Here we used infotaxis as a heuristic, which is already close to optimality, and the reward shaping was based on trial and error. Therefore Perseus is actually inferior to DRL and Sarsop in general, which require no domain knowledge or human intuition. In our view, the sole advantage that Perseus enjoys over Sarsop is its applicability to problems with a broad distribution of initial beliefs. In this section we discuss further the pros and cons of Sarsop and DRL.

Beyond the raw performance of the policies on the task, it is interesting to consider the computational cost that each method entails. We will not provide quantitative metrics, as (i) the solver implementations are done in different languages (Python for DRL, C++ for Sarsop) and (ii) our tests have been performed on vastly different machines. But we will provide qualitative remarks on our experience with using these various approaches.

As a solver, DRL is painfully slow compared to Sarsop: DRL can take several days when Sarsop provides a solution within a couple hours. On the other hand, Sarsop provides very large policies which are exceedingly slow to execute. In comparison, the neural network policies obtained by DRL are much lighter and faster to execute, typically by a factor of roughly 10. To make Sarsop competitive with DRL with respect to execution time, we computed additional Sarsop policies, called "Sarsop-Light" in Fig. 6. Sarsop-Light policies are defined by a smaller number of α-vectors, chosen such that the number of free parameters that parameterize the policy is comparable to that of the corresponding DRL policies (cf. Table C4 andTable C6). We found that Sarsop-Light policies are typically inferior to DRL policies.

Conclusion

In this paper we have compared two computational methods for approximately solving large POMDPs: a recently proposed one based on deep reinforcement learning (DRL), and the standard one relying on point-based value iteration (PBVI). We benchmarked these methods on variants of the olfactory search POMDP, a Goal-POMDP where the agent must find a hidden odor source as quickly as possible using stochastic partial observations in the form of odor cues. As PBVI comes in a number of slightly different flavors, we chose for our benchmark two popular PBVI implementations: Sarsop and Perseus.

While all solvers performed well overall, DRL outperformed PBVI on 3 out of 4 test cases. DRL shines by its ability to provide lighter policies with significantly faster execution speed compared to the policies generated by PBVI. Constraining PBVI solvers to reduce the size of their policies considerably degrade their performance on the task.

On the other hand, obtaining good policies with DRL requires training a large deep neural network, which is a costly process. In comparison, PBVI can generate approximate solutions much more efficiently. DRL also involves a large number of hyperparameters. Even though hyperparameter optimization is not needed to obtain good solutions, some minimal tuning is needed, which further increases the cost of training. Finally, on the 4th test case (a very long, almost deterministic search), the performance of DRL is significantly worse than that of PBVI for reasons that remain to be elucidated.

To summarize, DRL is competitive with respect to traditional PBVI solvers, and the best choice depends on the use case. PBVI solvers are best if the computation time allowed to the solver is limited (and we recommend Sarsop over Perseus). DRL is best if the execution time or the policy size is limited, as is usually the case in robotics. Solutions to the olfactory search POMDP have applications to sniffer robots, and DRL offers interesting perspectives for the future of these robots.

DRL also has a strong advantage in its flexibility: it can be applied without convergence issue to undiscounted problems, whereas point-based solvers require a strictly less-than-unity discount factor. For this reason, DRL has potential to be applied to a risk-sensitive setting by appropriately generalizing the Bellman equation [START_REF] Marcus | Systems and control in the twenty-first century[END_REF][START_REF] Coraluppi | Risk-sensitive and minimax control of discrete-time, finite-state markov decision processes[END_REF], an idea which will be explored in future research.

The olfactory search POMDP we considered is model-based: the agent knows the process that generates odor detections (the "model"), which allows it to maintain a belief using Bayesian inference. This assumption can be relaxed by considering instead an agent that makes decisions based directly on its current observation and some internal memory state, typically using a recurrent neural network [START_REF] Singh | Emergent behaviour and neural dynamics in artificial agents tracking odour plumes[END_REF]. In principle, the model-free agent can attain the same performance on the task as the model-based one. In practice, recurrent neural networks are hard to train and it will be interesting to evaluate their performance in the light of the (near-)optimal performance that can be computed in the model-based setting.

Algorithm 1 Reinforcement learning algorithm used to train a neural network to approximate the optimal value function. C4.

For a search domain of size n x × n y , the belief is a two-dimensional array of probability values with size (2n x -1) × (2n y -1). Each entry corresponds to a possible position of the source with respect to the agent: if we denote the coordinates of the center of the array as (0, 0), the source's possible position relative to the agent ranges from -(n x -1) to (n x -1) along the x-direction, and from -(n y -1) to (n y -1) along the y-direction. Since the belief size is four times larger than the grid size, 3/4th of the entries are outside the search domain and are zeros (cf. Fig. 3). This two-dimensional array is flattened before being fed to the network. The pseudo-code of the reinforcement learning algorithm is provided in Algorithm 1. The hyperparameters we used are given in Table C5. They were not optimized (it would be too costly) but chosen based on [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF] and limited experiments which showed that the learned policy is essentially insensitive to hyperparameters (provided they are given reasonable values).

C.2 PBVI algorithms

The olfactory search POMDP is an example of POMDP with undiscounted rewards. More generally, one can introduce a discount factor γ ∈ (0, 1] and the cumulated reward for an episode is T t=1 γ t-1 r t . The original problem corresponds to γ = 1. Using r t = -1, we have:

E T t=1 γ t r t =        -E[T ] for γ = 1, - 1 -E[γ T ] 1 -γ for γ < 1.
(C7)

The discount factor helps regularize the problem by suppressing the influence of times far in the future on the policy. Formally, each choice of γ defines a different POMDP, but presently we will continue to treat the (undiscounted) mean arrival time as our objective function and consider γ to be a tunable [START_REF] Vickers | Mechanisms of animal navigation in odor plumes[END_REF]737,470) Table C6 Size of the PBVI policy, for each case. The number of free parameters is equal to the number of α-vectors multiplied by the number of states |S|.

hyperparameter, necessary for the functioning of most popular POMDP algorithms (Sarsop and Perseus included). Generally, we try to keep it as close to unity as possible; for Sarsop, we chose γ = 0.98, and for Perseus we chose one of 0.95, 0.96 or 0.98, tuned by hand to optimize performance.

In order to obtain competitive policies with Perseus, we usually found it necessary to transform the problem by introducing a potential shaping function [START_REF] Ng | Proceedings of the Sixteenth International Conference on Machine Learning[END_REF] We found that policies do not always improve monotonically under Perseus, so policies were evaluated empirically after every iteration, and the algorithm was terminated when the mean arrival time failed to improve for some selected number of iterations.

As written, the Sarsop algorithm takes as input a single prior which serves as the root of the belief tree; Perseus, in contrast, accepts an arbitrary distribution of priors. For the isotropic problems, where several priors are possible, we simply used the most likely prior, corresponding to a single detection (h 0 = 1) at time t = 0 (cf. Appendix B.1).

Fig. 2

 2 Fig. 2 Tree of possible successor beliefs b starting from a belief b and executing action a. Transitions from b to b are determined by the observation o.

  with v * (b Ω ) = 0, where B(b, a) is the set of successor beliefs b reachable from a belief b by executing action a and where Pr(b | b, a) is the probability of transitioning from b to b upon a. Possible transitions correspond to possible observations o, as illustrated in Fig. 2, and transition probabilities are given by Pr(o) = s∈S Pr(o | s)b(s). Once a function solution to the Bellman optimality equation is found, the optimal policy consists in selecting the action that maximizes the expected optimal value: π * (b) = argmax a∈A b ∈B(b,a) Pr(b | b, a)v * (b ).

Fig. 3

 3 Fig. 3 Neural network approximation of the optimal value function used in model-based deep reinforcement learning. The weights are optimized so as to minimize the error on the Bellman optimality equation (Eq. 7).

Fig. 4

 4 Fig.4Probability distribution of the source location for the four test cases. The agent's starting position is fixed, and is denoted by a circle. The source location is drawn randomly at the beginning of each search from this probability distribution. For the isotropic cases, the distribution shown here is actually the weighted sum of the different possible initial beliefs corresponding to the different possible initial hits. For the windy cases, a single initial belief is considered and corresponds to the probability distribution shown here.

Fig. 5

 5 Fig.5Examples of quasi-optimal search trajectories for the four test cases. The agent's starting point and the source's position are denoted by a circle and a cross, respectively. Nonzero hits (odor detections) are indicated by stars. The orange shading color-codes the likelihood of making a detection in each grid cell, given its position with respect to the source. The trajectories were obtained with the best policies for each case.

  Initialize replay memory to capacity memory size Initialize value function v with random weights w Initialize target value function v -with random weights w -= w it ← 0 repeat # Generate new experience epsilon ← max(epsilon init * exp(-it/epsilon decay), epsilon floor) # decaying m ← 0 episode complete ← True while m < new transitions per it do if episode complete then initialize belief b for a new episode episode complete ← False end if b ← apply random symmetry(b) # randomize over symmetries for all actions a, compute all b accessible from b store (b, a, b ) in replay memory m ← m + 1 with probability epsilon select a random action a, # -greedy exploration otherwise select action a = arg max a b Pr(b | b, a)v(b ; w) b ← make step in env(b, a) # execute action and transition to a new belief episode complete ← b = b Ω end while # Update weights by stochastic gradient descent for gd step = 1, gd steps per it do Sample minibatch size transitions (b, a, b ) from replay memory For each transition, compute targets y using the delayed target network: y = -1 + maxa b Pr(b | b, a)v -(b ; w -) Perform a gradient descent step on (y -v(b; w)) 2 w.r.t network weights w end for it ← it + 1 every update target network it iterations, reset v -= v until weights w have converged proportional to the size |S| of the input (the belief). Here we used either 512 or 1024 neurons per layer depending on the test case, cf. Table

  to the reward. Defining r(b, a) as the reward obtained when executing action a in belief b, one can show that replacing the original (constant) reward function r(b, a) = -1 by the shaped reward function r(b, a) = -1 + F (b, a) (C8) with F (b, a) a function of the form F (b, a) = φ(b) -γ b Pr(b | b, a)φ(b ), (C9)preserves the optimal policy. A clever choice can sometimes accelerate convergence of value iteration. We tookφ(b) = -c s∈S D(s) • b(s) (C10)with c constant and D(s) the Manhattan distance between the agent and the source in state s. The point of this choice is to incentivize the agent to move closer to the source. The hyperparameter c was tuned from problem to problem.

Table 1

 1 Description of the four test cases used for benchmarking. In the top table, we describe the size of the problem. The state is defined as the position of the agent relative to the source, so for a grid size nx ×ny the number of possible states is |S| = (2nx -1)(2ny -1). The action space A consists of the 4 possible moves to adjacent cells. The observation space O is the union of the terminal observation Ω and of the set {0, 1, . . . , hmax} of hit values that can be received at each step. B 0 is the set of initial beliefs. Tmax is the maximum time allowed for the search (if the source is not found at t = Tmax, the search is considered a 'failure'). In the bottom table, we provide information on search times and number of odor detections. Mean(T) is the mean number of steps to find the source.

	case	grid size	|S| |A|	hmax	|O| |B 0 |	Tmax
	isotropic, smaller domain	19 × 19	1369	4	2	4	2	642
	isotropic, larger domain	53 × 53 11025	4	3	5	3	2188
	windy, with detections	81 × 41 13041	4	1	3	1 10000
	windy, almost no detections	81 × 41 13041	4	1	3	1 10000
	case	Mean(T) P 99 (T) Mean(cum. hits)
	isotropic, smaller domain	13.2		79		1.7
	isotropic, larger domain	34	152		10
	windy, with detections		63	274		6
	windy, almost no detections	217	1331		1.6

P 99 (T) is number of steps after which 99 % of the sources are found. Mean(cum. hits) is the mean number of hits cumulated during the search. The reported values for Mean(T), P 99 (T) and Mean(cum. hits) correspond to the best policy we obtained for each case.

  TableC4Size of the fully connected neural network used for each case. We always use 3 hidden layers. The number of free parameters (weights) is N = H(I + 2H + 4) + 1 where I is the input size (number of states |S|) and H the number of neurons per hidden layer.

	case	input size hidden layers number of free parameters
	isotropic, smaller domain		1369	3×512	1,227,265
	isotropic, larger domain	11025	3×1024	13,390,849
	windy, with detections	13041	3×1024	15,455,233
	windy, almost no detections	13041	3×1024	15,455,233
	hyperparameter	value	description
	learning rate	0.001	for stochastic gradient descent (SGD)
	epsilon init	1.0	initial value of for -greedy exploration
	epsilon floor	0.1	final value of for -greedy exploration
	epsilon decay	20000 time scale for decay of
	memory size	1000	number of transitions stored in memory
	minibatch size	64	size of the mini-batch for SGD updates
	new transitions per it	192	transitions added to memory per iteration
	gd steps per it	12	number of SGD updates per iteration
	update target network it 1	frequency of target network updates

Table

C5

List of hyperparameters (as usually defined or defined in Algorithm 1) and the typical values we used. Actual values used for each case may slightly differ, refer to our implementation at https://github.com/auroreloisy/otto-benchmark.

  number of α-vectors (number of free parameters)[START_REF] Shani | Prioritizing point-based pomdp solvers[END_REF]152,746) 880 (1,204,720) 2,058 (2,817,402) isotropic, larger domain 10,523 (116,016,075) 1,207 (13,307,175) 4,392 (48,421,800) windy, with detections 9,314 (121,463,874) 1,149 (14,984,109) 3,880 (50,599,080) windy, almost no detections 9,509 (124,006,869) 1,149 (14,984,109) 670

	case	Sarsop	Sarsop-Light	Perseus
	isotropic, smaller domain	19,834

PBVI often refers to a specific algorithm introduced in Ref.[START_REF] Pineau | Proceedings of the Eighteenth International Joint Conference on Artifical Intelligence (IJCAI)[END_REF], the first of its kind. For our purposes, it refers to the broader class of point-based algorithms for value iteration.

Acknowledgments. We thank Luca Biferale, Antonio Celani, Massimo Vergassola, and Christophe Eloy for useful discussions. AL received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 834238). RAH received funding from the European Union's H2020 Program under grant agreement No. 882340. Centre de Calcul Intensif d'Aix-Marseille is acknowledged for granting access to its high performance computing resources.

Statements and Declarations

Code and data availability

The code used to generate DRL policies and to evaluate all policies on the olfactory search POMDP is available at https://github.com/auroreloisy/ otto-benchmark. The policies computed with the various solvers can be downloaded at https://doi.org/10.5281/zenodo.7586357. The data used to plot the results in Fig. 6 is available at https://doi.org/10.5281/zenodo.7586312. PBVI policies were obtained using the code available at https://github. com/rheinonen/PerseusPOMDP/ for Perseus, and at https://github.com/ rheinonen/sarsop/ for Sarsop. Deep reinforcement learning for the olfactory search POMDP The size of the PBVI policies is reported in Table C6.

Competing interests

The authors have no competing interests to declare.

Authors' contributions

AL and RAH designed the study, performed the research and analyzed the results. AL wrote the manuscript with the help of RAH.

Appendix A Observation model: odor dispersion and detection in turbulence

In this appendix we provide the model used to generate observations (hits), that is, we specify Pr(h | s). This model is based on a physical modeling of odor dispersion and detection in a turbulent flow. We consider a point source that emits, at a rate R, odor particles with a finite lifetime τ . They disperse in the turbulent medium, which is characterized by an effective diffusivity D and a mean wind speed V . The wind blows in the positive x-direction, denoted by the unit vector e x . The searcher is modeled as a sphere (disk) of diameter ∆x fully covered in receptors. A every step, it takes a "sniff" over a time ∆t, during which odor particles diffusing to its surface are absorbed. It then moves by one body length ∆x (which defines the size of a grid cell).

Based on these assumptions, one can derive the mean number µ of odor particles ("hits") detected by the searcher as a function of its position r with respect to the source [START_REF] Vergassola | Infotaxis" as a strategy for searching without gradients[END_REF][START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF]:

and where K 0 is the modified Bessel function of the second kind of order 0.

The number of hits is distributed according to a Poisson's law

with mean µ. This completes the description of the observation model Pr(h | s). In the main text, r is denoted s to be consistent with standard POMDP notations. The likelihood of observing h in state s corresponding to position r is given by

Appendix B Further specifications of the POMDP variants used as test cases

B.1 Search initialization

The search start by drawing a initial belief b 0 from the set of initial beliefs B 0 .

Here we provide further details about the construction of this set. Before the search starts, we assume that the searcher is located in an infinite domain containing a source which location is distributed uniformly. We calculate the probabilities of detecting an odor, Pr(h 0 ) for h 0 > 0, based on the observation model: Pr(h) = s Pr(h | s) Pr(s) where Pr(s) is a constant here. Then, we initialize the finite-size grid with a uniform prior b -1 (s). We draw a nonzero hit from the distribution Pr(h 0 ), and perform the Bayes update of the belief accordingly. This gives us b 0 . The search starts from this b 0 . Each initial nonzero hit value h 0 yields a different initial belief b 0 . Effectively, the initial belief is drawn randomly from the set of possible initial beliefs B 0 built from the possible values of h 0 . The reader is referred to [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF] for more details.

The set of initial beliefs and their probabilities depend on the test case, cf. Table B1. In the isotropic cases, it contains several elements. In the windy cases, hits are binary (h = 1 for a detection, h = 0 otherwise) so a single initial belief is considered.

B.2 Observation model

In the two isotropic cases, V = 0 and we use the 2D version of the observation model (Eq. A1a), which reduces to: with λ = √ Dτ . It is fully defined by specifying two dimensionless parameters, which we chose as λ/∆x and R∆t, as was done in [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF]. Their numerical values are given in Table B2.

In the two windy cases, we use the 3D version of the observation model (Eq. A1b). To fully specify the model, three dimensionless parameters must be set. For consistency with [START_REF] Heinonen | Optimal policies for Bayesian olfactory search in turbulent flows[END_REF] we choose

Their values are provided in Table B3.

Appendix C Methodological details on DRL, Sarsop and Perseus

C.1 Deep reinforcement learning

To approximate the optimal value function, we use a fully connected neural network with 3 hidden layers. Extensive experiments by [START_REF] Loisy | Searching for a source without gradients: How good is infotaxis and how to beat it[END_REF] suggest that as a rule of thumb, one should choose a number of neurons per layer roughly