Philippe Lagacherie

Inrae Lisah

Mieux connaître et prendre en compte les sols dans un contexte de changement climatique

Le sol

« Le sol est un volume qui s'étend depuis la surface de la Terre jusqu'à une profondeur marquée par l'apparition d'une roche dure ou meuble, peu altérée, ou peu marquée par la pédogenèse. » (AFES, 2018) Le sol : un acteur majeur du cycle de l'eau

Ø

  Gain de RU pour 10 cm de sol avec une augmentation de 1% en C sol 𝑆 = 𝑓 𝑠 , 𝑐 , 𝑜 , 𝑟 , 𝑝 , 𝑎 , 𝑛 + 𝜀 Données environnementales spatialisées en lien avec la formation des sols RRP Occitanie (1/250 000) Ø MNT IGN 25 m et 75m et variables dérivées Ø Images de télédétection (MODIS, Landsat) Transfert de méthode vers des opérateurs non académiques ( CRA Occitanie, Grand Est, Bureaux d'études) § En Occitanie : extension de l'application de la CSMS à toute la région (collaboration CRA Occitanie) § Mise à disposition des produits cartographiques via l'IDG régionale (OPenIG ) Perspectives à court terme § Cartographie des sols par modélisation statistique § Cartographie des sols participative Deux nouvelles démarches de cartographie des sols arrivant à des stades pré-opérationnels Tests sur plusieurs territoires agricoles § Lomagne (82) § Bassin Versant du Rieutort (34) § Périmètre irrigué de l'ASA Ouvèze Ventoux (84) § Village de Gopalpura (Sud de l'Inde) § Typologie d'agriculteurs moins détaillée que typologie de pédologues § 30 % de désaccords avec observations • Typologie pas stabilisée ou mal appropriée • Erreurs de localisations • Critères de comportement prépondérant pour agriculteurs Confrontation avec les observations pédologiques de terrain Généralisation cartographique par CSMS Carte pédologique Ø Carte des sols 1:100 000 feuille de Lodève Ø MNT IGN 25 m variables dérivées Ø Images de télédétection (Landsat) Données utilisées Algorithme d'apprentissage Ø arbre de classification § Améliorer la démarche collective de révision de typologies et de localisations après confrontation avec observations (atelier 3) § Elargir le panel de contributeurs après stabilisation de la typologie ( interface Web) § Faire le lien avec La cartographie des sols par modélisation statistique Perspectives § L'adaptation de l'agriculture au changement climatique passe par une meilleur prise en compte de la diversité des sols § Les bases de données sol actuellement disponibles ne sont pas assez précises pour être utilisées dans ce sens § Deux nouvelles approches sont à un stade susceptibles d'être transférées de la recherche aux opérateurs du développement agricole • Cartographie des sols par modélisation statistique • Cartographie des sols participatives § Un ré-investissement des organismes de développement agricole vers la connaissance et la cartographie des sols est indispensable En guise de conclusion

Bassin Versan de Roujan (Hérault) ORE OMERE


Jean Baptiste Doucet

S Pelloquin

Alexandre Lauvergne

E Daran

Sylvain Auge

Sylvain Auge A Uv-Nil

A UV-NIL resist developed for a transitional soft mold process and the subsequent dry etch transfer into hard materials

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Applications to research projects

Nanonex NX 2500

We designed a new UV nanoimprint resist adapted to the use of a transitional soft COP or COC mold to replicate or transfer submicro or nano features into hard materials such as silicon, silica or gallium arsenide. We have used this resist successfully to transfer submicro or nano features from a silicon mold into silicon, silica or gallium arsenide on small samples or full 4" wafers. This enables for instance the making of optical filters on transparent substrates on which direct stepper lithography can not be done easily. The knowledge of the resist content provides a good reliability of the process and enables adaptation of the resist to new applications (such as final optical index adjustment). Among currently investigatd and future works or those implying the use of the resist as an optical layer that will remain on the final device ( as it is, or with optical index adaptation) and it's use for nanoscale metal deposition by lift-off process.

  Photography of a GMRF (GuidedMode Resonance Filter) pattern transferred on the top layer of a full 4'' SiO2/a-Si/SiO2/a-Si/glass sample (respective thicknesses 61, 49, 120, 74 nm on 500 μm thick glass). Insert is an AFM micrograph. Red and green shapes come from diffraction of light


C Chan Yone

J L Milan

J M Rossi

J F Witz

M Brieu

P Chabrand

Model of cancellous bone adaptation considering hypermineralized bone tissue

Keywords: bone, remodeling, hypermineralization

Introduction

Aging and a low physical activity often involve a decrease in bone mass and a mechanical weakening of bone at global scale, increasing fracture risk. In particular, cancellous bone tissue which plays a mechanical role on the overall rigidity and strength of bone organs, is importantly degraded. Numerous computational models have been developed to simulate bone remodelling process and predict over decades the evolution of microstructure of cancellous bone and its mechanical behaviour [START_REF] Liu | Dynamic simulation of three dimensional architectural and mechanical alterations in human trabecular bone during menopause[END_REF][START_REF] Van Der Linden | A simulation model at trabecular level to predict effects of antiresportive treatment after menopause[END_REF]. These models were based on µCT reconstruction of bone samples and consider generally the tissue as a homogeneous material. However bone remodelling is spatially heterogeneous with concomitance of aged and new bone tissue. Moreover mineralization is a process occurring with time leading to heterogeneous mineral concentration and so mechanical properties. To determine the influence of a heterogeneous material on remodelling of cancellous bone, a computational study was performed here on a bone sample reconstructed in 3D from a µ-CT image stack. Heterogeneous material properties were defined depending on mineral density revealed by µCT gray value. The results show that hypermineralization of cancellous bone influences considerably local mechanical behaviour although overall mechanical behaviour was not affected. Besides, bone cells are mechano-sensitive and adapt bone microstructure by resorbing aged or damaged tissue and aposing new one. So, a iterative computation of bone degradation was then performed in case of a low physical activity. Sites which were underloaded were considered as possible sites of remodelling process. This study shows that heterogeneous material properties may influence localization of bone remodelling This model which analyzes alterations of cancellous bone microstructure may help in the prediction of fracture risk. It can be also implemented so as to predict effect of drug treatment on bone cell activity and the balance between apostition and resorption.

Methods

Trabecular bone sample and -CT imaging

We considered a bone sample from human femoral head (woman of 87 year-old). A cubic specimen of 10x10x10mm 3 was scanned with a -CT system of 20 m resolution. A phantom representing the mineral content of a normal mineralized bone tissue was placed under the sample. This latter gave a gray-value reference in order to identify locally material components and the related mechanical properties.

Finite element model

A first F.E. heterogeneous model (4x4x4 mm 3 ) extracted from cubic sample consisted of 2344910 elements converting voxels into equally sized eight-node brick. All elements were assumed to be linearly elastic and isotropic with Poisson's ratio set to 0.3 and the following power law relationship between Young's modulus and bone grey value :

Eelt=E0 . GVelt γ
where GVelt is the normalized gray-value element, γ a constant exponent set to 1.5 [START_REF] Homminga | Introduction and evaluation of a gray-value voxel conversion technique[END_REF], and E0 was taken to be equal to 19 GPa [START_REF] Hoc | Effect of microstructure on the mechanical properties of haversian cortical bone[END_REF]. In order to compare the influence of material heterogeneities, an homogeneous model with constant material properties was also considered. Young's Modulus and Poisson's ratio were equal to 19 GPa and 0.3 respectively. Uniaxial compressive loading of 0.1% was applied to the the cubic sample

Bone adaptation model

The finite element of cancellous bone is implemented in a algorithm modifying bone structure iteratively and under compressive loading by apposition or resorption. Some authors considered that when strain stimulus is lower than ε res =100µε bone matrix is resorbed. On the contrary, when strain stimulus exceeds εapp=3000µε new bone is apposed [START_REF] Frost | On our age-related bone loss: insights from a new paradigm[END_REF]. Here we considered that if strain energy is lower than wres, bone is resorbed and if it is greater than wapp new bone is apposed : wres =½ Ebone. εres² w app = ½ E bone . ε app ² with Ebone = 19GPa. When strain energy in a element of the model is outside [wres;wapp], elements are either deleted or created at bone surface to represent respectively resorption or apposition. New bone tissue was given Young modulus which increased iteratively from 0.5 GPa to 35Gpa to simulate mineralization.

Influence of hypermineralized bone tissue

Numerical compression tests were performed and Apparent Young's modulus and mean strain energy density were calculated for both previous models. Difference of 7.5% was noted between model with homogeneous material and model with heterogeneous material, which was in agreement with the literature [START_REF] Follet | The degree of mineralization is a determinant of bone strength: a study on human calcanei[END_REF]. Nevertheless a difference of 20% of over-constrained bone volume was identified (where strain exceed 3000µε). From a biological point of view beyond this threshold new bone tissue is apposed, or in contrary when strains are under 100µε bone tissue is removed [START_REF] Frost | On our age-related bone loss: insights from a new paradigm[END_REF].

Hypermineralized bone precursor of bone remodeling

Iterative compression tests were simulated on the cancellous bone model. Remodelling sites where identified: elements were removed or added respectively where strains are under 100 µε or exceed 3000 µε. Strain distribution in bone sample is due to a structure effect, but also a material heterogeneities distribution.

Conclusions

Geometrical bone degradation was simulated and overall stiffness of the sample was computed. The developed model took into account the hypermineralization of bone tissue. As alendronate treatment increased bone mineral density, but also degree of mineralization of bone, this model can simulate medical treatment on bone stiffness, and quantify fracture risk in particular in osteoporosis case.

Figure 1 :

 1 Figure 1 : Young's modulus heterogeneous distribution. Hypermineralized bone tissue in red

Figure 1 :

 1 Figure 1: Degradation of cancellous bone from initial to remodelled (after 5 year)


Richard Ghail

Colin F Wilson
email: colin.wilson@physics.ox.ac.uk

Thomas Widemann

Dmitry Titov

Lorenzo Bruzzone

Jörn Helbert

Ann Carine Vandaele

Emmanuel Marcq

Caroline Dumoulin

Pascal Rosenblatt

Envision

EnVision M5 Venus orbiter proposal

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Abstract

EnVision [START_REF] Ghail | EnVision M5 proposal[END_REF]2] is a Venus orbiter mission that will determine the nature and current state of geological activity on Venus, and its relationship with the atmosphere, to understand how and why Venus and Earth evolved so differently. Envision is a finalist in ESA's M5 Space Science mission selection process, and is being developed in collaboration with NASA, with the sharing of responsibilities currently under assessment. It is currently in Phase A study; final mission selection is expected in June 2021. If selected, EnVision will launch by 2032 on an Ariane 6.2 into a six month cruise to Venus, followed by aerobraking, to achieve a near-circular polar orbit for a nominal science phase lasting at least 4 Venus sidereal days (2.7 Earth years).

Introduction

Why are the terrestrial planets so different? Venus and Earth were probably very similar in the past, before their evolutionary paths diverged. Understanding how and why this occurred is crucial to understanding the diversity of terrestrial planets and their habitability.

The Envision Venus orbiter will undertake science investigations in three thematic areas:

1) Is Venus geologically active today? How?

2) How did Venus' surface reach its present state?

3) How did Venus' climate reach its present state?

Payload

EnVision hosts three instruments and a radio science investigation:

The Synthetic Aperture Radar, VenSAR, will:

Invitation to participate

EnVision will produce a huge dataset of geophysical data of a quality similar to that available for Earth and Mars, so will permit investigation across a large range of disciplines. Lab-based and modelling work will also be required to interpret results from the mission. We therefore invite scientists from across planetary, exoplanetary and earth science disciplines to participate in the analysis of the data.

The entire dataset obtained will be made publicly available; much of the SAR dataset will be made available in near-real-time to facilitate wide use of the data. We reiterate the opportunity for science experiments and target selection, and encourage researchers to contact the EnVision team.


Gabriel Capson-Tojo
email: gabriel.capson-tojo@inrae.fr

María Del Rosario Rodero

Renaud Escudié
email: renaud.escudie@inrae.fr

Jean-Philippe Steyer
email: jean-philippe.steyer@inrae.fr

Towards a gas-based biorefinery: purple photo-anaerobic processes for the valorization of fermentation effluents

Keywords: Resource recovery, Hydrogen, Purple phototrophic bacteria, Single Cell Protein, Nitrogen

 Biological processes can generate value-added products from gaseous effluents  Purple phototrophic bacteria can turn H 2 and CO 2 into proteins at high yields  Acetate-rich effluents can be used to fix N 2 by purple bacteria at high yields

Introduction

Processes based on purple phototrophic bacteria (PPB) are promising for resource recovery thanks to the high biomass yields (up to 1 g COD•g COD -1 ) that their phototrophic metabolism allows. While their photoheterotrophic capabilities for resource recovery from wastewater have been considerably studied in the last few years (Hülsen et al., 2022a(Hülsen et al., , 2022b)), their photoautotrophic metabolism has barely been researched for resource recovery. Autotrophic PPB growth allows the conversion of H 2 and CO 2 into single-cell protein (SCP), generating a pathogen-free value-added product applicable as feed/food. This process could be used to valorize gaseous effluents in a biorefinery (e.g. dark from fermentation), while also recovering remaining nutrients (e.g. from digestates). [START_REF] Spanoghe | Microbial Food From Light, Carbon Dioxide And Hydrogen Gas: Kinetic, Stoichiometric And Nutritional Potential Of Three Purple Bacteria[END_REF] realized the proof of concept of H 2 conversion by PPB using pure cultures, but yields and rates are still to be determined, particularly in enriched systems. PPB can also fix N 2 when there is limited soluble N in the media [START_REF] Capson-Tojo | Purple phototrophic bacteria for resource recovery: Challenges and opportunities[END_REF]. This capability could be used to produce sustainable SCP, as an alternative to the Haber-Bosch process, which is a fast and cheap process to generate ammonia, but relies on fossil fuels. With the prices of organic-N fertilizers being 2-4 times what they were 1-2 years ago (according to the European Commission), biological N fixation processes are starting to seem like a feasible alternative. In this study, we performed dedicated batch tests to evaluate the yields and kinetics of an enriched PPB consortia for the conversion of H 2 and CO 2 into SCP, evaluating the influence of environmental conditions (temperature, pH, light intensity). In addition, the possibility of producing SCP from N 2 was evaluated under both autotrophic and photoheterotrophic conditions.

Materials and Methods

A series of sequential batch autotrophic and heterotrophic enrichments (over 14) were performed in Schott flasks of 500 mL to obtain stable phototrophic PPB consortia. Each batch (which lasted for around 1 week) was inoculated with 20 mL from the previous culture and grown with 180 mL of fresh medium. The medium was the one proposed by [START_REF] Ormerod | Light-Dependent Utilization of Organic Compounds and Photoproduction of Molecular bHydrogen by Photosynthetic Bacteria; Relationships with Nitrogen Metabolism[END_REF] for the heterotrophic tests (using acetate as C source). It was modified for the autotrophic tests, adding H 2 and carbonate instead of acetate. The bottles were closed and the headspace was flushed with N 2 prior to incubation. The initial headspace pressure was adjusted to 1.3 bar in the H 2 -fed tests. The initial acetate concentration was 500 mg COD•L -1 in the heterotrophic tests. Illumination was provided as in [START_REF] Capson-Tojo | Purple phototrophic bacteria are outcompeted by aerobic heterotrophs in the presence of oxygen[END_REF]. The resulting stable enrichments were used to perform batch tests. Autotrophic growth was assessed at different environmental conditions (e.g. initial pH values of 6, 7 and 8.5, temperatures of 15, 25, 38 and 50 °C, and light intensities of 0, 5, 15, 30 and 50 W•m -2 ) using triplicate batch tests, as described above. The gas composition and pressure were monitored 4-5 times per day, and liquid samples were drawn to determine biomass yields and productivities, crude protein contents, amino acids profiles and microbial communities (16S rRNA gene sequencing). A mechanistic model based on Capson-Tojo et al. ( 2023) was used for estimating specific uptake rates. Tests for N 2 fixation were performed as the enrichments, but using N-depleted media to allow nitrogenase activity.

Results and Discussion

The obtained enriched PPB cultures grew autotrophically at yields always close to 1 g COD biomass ⸱g COD consumed -1 , corresponding to protein yields of around 4 g⸱g H 2 -1 , with protein contents of 51-64%

(Figure 1). These values are on the highest end of those reported for PPB pure cultures [START_REF] Spanoghe | Microbial Food From Light, Carbon Dioxide And Hydrogen Gas: Kinetic, Stoichiometric And Nutritional Potential Of Three Purple Bacteria[END_REF]. Optimal autotrophic specific uptake rates of 1.9-2.0 g COD•g COD -1 •d -1 were reached at pH 7, 25ºC and light intensities over 30 W•m -2 . Although these values are slightly lower than for photoheterotrophic growth (2.4-2.6 g COD•g COD -1 •d -1 ; [START_REF] Capson-Tojo | Expanding mechanistic models to represent purple phototrophic bacteria enriched cultures growing outdoors[END_REF]) and lower than those for hydrogen oxidizing bacteria (HOB; main competitors), they still allowed overall biomass productivities up to 378 mg COD•L -1 •d -1 , values that remain to be optimized in continuous reactors. Yields for HOB are around 0.2 g COD biomass ⸱g COD consumed -1 , confirming the high PPB yields as their main advantage (Hu et al., 2020). Analysis of the microbial communities via 16S rRNA gene sequencing showed that PPB represented over 85% of the 16S total copies, with Rhodobacter sp. and Rhodopseudomonas sp. as dominant genera (as in the inoculum). Regarding the N 2 fixation tests, preliminary results show that effective N 2 fixation can be achieved at reasonable rates in heterotrophic N-limited conditions (no more VFAs after 3-4 days), at yields of 1 g N•g N -1 and 0.88-0.92 g COD•g COD -1 . More research is needed to determined specific uptake rates and protein contents, and to elucidate why the biomass yields in terms of COD are lower than in soluble N rich media. Very likely, electrons from the substrate are not used for growth, but for the reduction of N 2 . Light intensity did not affect the yields (Figure 1), so energy availability was not responsible for the lower yields obtained. The non-energetic valorization of gases from fermentation (e.g. H 2 , and CO 2 ) could improve drastically the economic and environmental performances of environmental biorefineries. PPB grown heterotrophically could valorize mixed VFAs. The ammonia could be extracted (e.g. via electrodialysis) and used for autotrophic growth, while the N-depleted VFA-rich stream could be used for N 2 fixation by PPB, generating a sustainable, clean, SCP product.

Figure 1 .

 1 Figure 1. Biomass yields for the autotrophic and heterotrophic batch tests at different conditions. The specific uptake rates for the autotrophic tests are also shown. Temperatures (T) are expressed in Celsius and light intensities (I) in W•m -2 .


Gabriel Capson-Tojo
email: gabriel.capson-tojo@inrae.fr

Renaud Escudié
email: renaud.escudié@inrae.fr

Elie Desmond-Le Quéméner

Jean-Philippe Steyer
email: jean-philippe.steyer@inrae.fr

Ángel Robles
email: angel.robles@uv.es

Recommendations for the accurate modelling of food waste anaerobic digestion

Keywords: Syntrophic acetate oxidation, High-solids AD, Hydrogenotrophic methanogenesis, Ammonia

Highlights

 Syntrophic acetate oxidation and non-ideal ion speciation must be included  Stoichiometry must be corrected as per food waste composition and amino acids  Including chain elongation improved the butyrate and valerate predictions

Introduction

Efficient treatment and resource recovery from food waste (FW) is necessary for the implementation of sustainable and circular societies. Anaerobic digestion (AD) stands as an environmental-friendly and well-stablished technology to do so. Despite its potential, AD of highly concentrated substrates such as FW (20% total solids; TS) is a complex biological process prone to failure if not managed properly. Because of the fast biodegradability of FW, accumulation of volatile fatty acids (VFAs) has been frequently reported due to unbalance of the acidogenesis/acetogenesis and methanogenesis steps [START_REF] Capson-Tojo | Food waste valorization via anaerobic processes: a review[END_REF]. The high protein content of FW also leads to high concentrations of total and free ammonia nitrogen (TAN; FAN). FAN is toxic to microorganisms, mainly to strict acetoclastic methanogens (Capson-Tojo et al., 2018b[START_REF] Capson-Tojo | Unraveling the literature chaos around free ammonia inhibition in anaerobic digestion[END_REF]. Thus, to accurately represent the AD of concentrated substrates such as FW, mechanistic models must consider the dominant metabolic pathways occurring (e.g. syntrophic acetate oxidation (SAO) and hydrogenotrophic methanogenesis (HM)) [START_REF] Capson-Tojo | Considering syntrophic acetate oxidation and ionic strength improves the performance of models for food waste anaerobic digestion[END_REF]. Furthermore, the TAN-FAN acid-base equilibria, which is affected, by pH, temperature, and by the high ionic strength of the media, must also be properly depicted [START_REF] Capson-Tojo | Unraveling the literature chaos around free ammonia inhibition in anaerobic digestion[END_REF]. A recent article proposed a modified version of the widely used ADM1 [START_REF] Batstone | The IWA Anaerobic Digestion Model No 1 (ADM 1)[END_REF] to include SAO and to accurately predict FAN concentrations using a modified Davies equation [START_REF] Capson-Tojo | Considering syntrophic acetate oxidation and ionic strength improves the performance of models for food waste anaerobic digestion[END_REF]. Results showed that these modifications were crucial to predict the system's behavior, particularly the acetate kinetics at reasonable uptake rates, and the accumulation of propionate [START_REF] Capson-Tojo | Accumulation of propionic acid during consecutive batch anaerobic digestion of commercial FW[END_REF]. Nevertheless, the predictions for butyrate and valerate were far from perfect. This study aims at testing different alternatives that could further improve the predictions of these intermediates. More importantly, here we put together our previous knowledge to give recommendations that, in our opinion, are essential for an accurate modelling of FW AD.

Material and Methods

Batch results from Capson-Tojo et al. (2018a) were used to calibrate and validate the model. The different models compared were: (i) the unmodified ADM1 [START_REF] Batstone | The IWA Anaerobic Digestion Model No 1 (ADM 1)[END_REF], (ii) a modified ADM1 as in Capson-Tojo et al. ( 2021) (including SAO and non-ideal TAN-FAN equilibria) and including an accurate FW composition, (iii) model (ii) with uncoupled n-valerate and i-valerate consumption (as in [START_REF] Batstone | Kinetics of Thermophilic, Anaerobic Oxidation of Straight and Branched Chain Butyrate and Valerate[END_REF]), (iv) model (ii) considering chain elongation for valerate production (from acetate, H2 and propionate; assuming ethanol as non-detectable intermediate) and with a modified valerate stoichiometry from amino acid fermentation. To avoid biased comparisons, the modelling procedure was the same for all the approaches, as in [START_REF] Capson-Tojo | Considering syntrophic acetate oxidation and ionic strength improves the performance of models for food waste anaerobic digestion[END_REF]. The simulations were carried out using MATLAB R2021a.

Results and Discussion

Even after calibration, the unmodified ADM1 (red line in Figure 1) was not able to accurately predict the results, mostly because SAO (the predominant acetate-consuming pathway) is not considered. In addition, the obtained values for the kinetic parameters related to acetotrophic growth and those for FAN inhibition were not realistic [START_REF] Capson-Tojo | Considering syntrophic acetate oxidation and ionic strength improves the performance of models for food waste anaerobic digestion[END_REF]. The modified ADM1 improved the performance considerably, but the butyrate and valerate profiles could not be accurately predicted simultaneously (either one profile or the other could be accurate, depending on the calibration done). Uncoupling n-valerate and i-valerate as in [START_REF] Batstone | Kinetics of Thermophilic, Anaerobic Oxidation of Straight and Branched Chain Butyrate and Valerate[END_REF] did not enhanced the model performance (results not shown). The modified model considering chain elongation allowed to uncouple the kinetics of butyrate and valerate uptake. The methane kinetics and acetate and propionate concentrations were again accurately predicted (as for model (ii)), but in this case the kinetics of butyrate and valerate were uncoupled, improving the accuracy of the predictions (further calibration efforts are needed and are currently being done). In the case of butyrate, its more accurate prediction was related to: (i) an accurate substrate fractionation in protein, carbohydrates and lipid contents [START_REF] Capson-Tojo | Accumulation of propionic acid during consecutive batch anaerobic digestion of commercial FW[END_REF], and (ii) the uncoupling of the biomass responsible for butyrate an valerate degradation. Butyrate and valerate kinetics were very different, justifying this modification. Improving valerate predictions was more challenging. After trying several approaches, we found that including chain elongation in the model and reducing the amount of valerate initially produced from amino acid fermentation were necessary modifications to predict accurately the valerate profile. These modifications can be justified by: (i) the presence of traces of caproate in the reactors, confirming that chain elongation was indeed occurring, and (ii) the initial valerate kinetics differed totally from those of all the other VFAs, suggesting that its production was not related to the initial amino acid fermentation (the high soluble hydrogen concentrations in the first days would have inhibited valerate degradation, so no valerate consumption could occur). Chain elongation was included in the model assuming that the kinetics of ethanol production/consumption were much faster than those of valerate production, defining an overall reaction transforming acetate and hydrogen (ethanol precursors) and propionate (valerate precursor) into valerate. Summarizing, considering SAO and non-ideal FAN-TAN equilibrium is crucial to accurately predict the methane production rates and the concentrations of acetate and propionate. Furthermore, if the concentrations of butyrate and valerate are to be predicted accurately, the substrate must be characterized and fractionated extensively, and chain elongation processes could be accounted for. Normally, butyrate and valerate do not represent a high proportion of the overall COD in the mass balance, but as they can act as indicators for AD instability, their accurate prediction is relevant.

Figure 1 .

 1 Figure 1. Experimental results and modelling results for (A) methane production rates and concentrations of (B) butyrate, (C) acetate, (D) valerate, (E) propionate, and (F) total VFAs.


Etienne Ménager

Christian Duriez

Learning control strategy in soft robotics through a set of configuration spaces

Keywords: Modelling, Control, and Learning for Soft Robots. Soft Robot Applications

The ability of a soft robot to perform specific tasks is determined by its contact configuration, and transitioning between configurations is often necessary to reach a desired position or manipulate an object. Based on this observation, we propose a method for controlling soft robots that involves defining a graph of configuration spaces. Different agents, whether learned or not (convex optimization, expert trajectory, and collision detection), use the structure of the graph to solve the desired task. The graph and the agents are part of the prior knowledge that is intuitively integrated into the learning process. They are used to combine different optimization methods, improve sample efficiency, and provide interpretability. We construct the graph based on the contact configurations and demonstrate its effectiveness through two scenarios, a deformable beam in contact with its environment and a soft manipulator, where it outperforms the baseline in terms of stability, learning speed, and interpretability.

I. INTRODUCTION

Soft robots can be controlled using either physics-based or learning-based methods. In rigid robotics, various control methods are employed to handle contact, in planning and predictive control [START_REF] Pang | Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models[END_REF], [START_REF] Marcucci | Motion planning around obstacles with convex optimization[END_REF], [START_REF] Marcucci | Shortest paths in graphs of convex sets[END_REF], [START_REF] Ubellacker | Robust locomotion on legged robots through planning on motion primitive graphs[END_REF]. However, soft robots control based on Finite Element Method (FEM) is limited to one time step with optimization and inverse modelling [START_REF] Coevoet | Soft robots locomotion and manipulation control using fem simulation and quadratic programming[END_REF], except for simplified robot models [START_REF] Spinelli | A unified and modular model predictive control framework for soft continuum manipulators under internal and external constraints[END_REF], [START_REF] Lopez | Dynamic tube mpc for nonlinear systems[END_REF]. Although this method can handle contact [START_REF] Coevoet | Optimization-based inverse model of soft robots with contact handling[END_REF], it does not account for contact reconfiguration, as breaks in the optimization space occur due to contact. On the other hand, Reinforcement Learning (RL) has been successfully used in various applications, including robotics [START_REF] Pinto | Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours[END_REF], [START_REF] Levine | Learning handeye coordination for robotic grasping with deep learning and largescale data collection[END_REF], [START_REF] Agrawal | Learning to poke by poking: Experiential learning of intuitive physics[END_REF], [START_REF] Openai | Learning dexterous in-hand manipulation[END_REF], [START_REF] Openai | Solving rubik's cube with a robot hand[END_REF] and soft robotics [START_REF] Homberg | Robust proprioceptive grasping with a soft robot hand[END_REF], [START_REF] Wang | Motion planning based on learning from demonstration for multiple-segment flexible soft robots actuated by electroactive polymers[END_REF]. This approach can solve complex tasks that cannot be solved using optimization methods because of its 0th order approximation of the gradient and exploratory aspect [START_REF] Lidec | Leveraging randomized smoothing for optimal control of nonsmooth dynamical systems[END_REF]. However, it is generally less interpretable or less sample-efficient than model-based optimization approach.

Various methods can be employed to limit the gap between these two approaches. One approach is to modify the reward function [START_REF] Loftin | Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning[END_REF], [START_REF] Hu | Interactionaware decision making with adaptive strategies under merging scenarios[END_REF], such as by incorporating the system's physics [START_REF] Dang | Towards stochastic modeling for two-phase flow interfacial area predictions: A physics-informed reinforcement learning approach[END_REF] or using examples of task resolution [START_REF] Eysenbach | Replacing rewards with examples: Example-based policy search via recursive classification[END_REF]. Another approach is to focus on the initialization of the models, such as Transfer Learning and Imitation Learning [START_REF] Hua | Learning for a robot: Deep reinforcement learning, imitation learning, transfer learning[END_REF] where knowledge is learned from one task or an expert and transferred to the current task. Clustering methods and prototypical representation [START_REF] Akrour | Regularizing reinforcement learning with state abstraction[END_REF], [START_REF] Yarats | Reinforcement learning with prototypical representations[END_REF], [START_REF] Mannor | Dynamic abstraction in reinforcement learning via clustering[END_REF] can also be used to group similar states and facilitate learning. * Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 CRIStAL, F-59000 Lille, France etienne.menager@inria.fr Finally, Hierarchical Reinforcement Learning (HRL) [START_REF] Pateria | Hierarchical reinforcement learning: A comprehensive survey[END_REF], [START_REF] Ahilan | Feudal multi-agent hierarchies for cooperative reinforcement learning[END_REF] decomposes a task into different subtasks to accelerate learning. For example, in [START_REF] Veiga | Hierarchical tactile-based control decomposition of dexterous in-hand manipulation tasks[END_REF], an unlearned controller is used to stabilize an object, whereas RL algorithms are used for its manipulation. However, these methods may not always be straightforward to implement, lack interpretability, or training data may not always be available.

Contributions: In this paper, we present a novel approach for incorporating prior knowledge into the learning process, which is represented as a graph of configuration spaces. This graph divides the robot states into distinct hand-defined configuration spaces, and sequences of actions are used to navigate between them. It enables the integration of learning and convex optimization, enhances interpretability and reusability, and enables the use of expert trajectories in the learning process. We demonstrate the feasibility of this method through simulations of a beam in contact with its environment and a soft manipulator.

II. BACKGROUND AND NOTATIONS

We consider a Markov Decision Problem (MDP) with states s ∈ S, actions a ∈ A, transition distribution s t+1 ∼ P (s t , a t ) and reward function r t ∼ R(s t , a t ). Let π θ (a|s) be a stochastic policy with parameter θ. In this situation, the agent interacts with an environment by sampling an action a t ∼ π θ (.|s t ), receives a reward r t and a new state s t+1 . The objective of the agent is to maximize the expected sum of discounted rewards:

J(π) = E τ ∼π [Σ T t=0 γ t r(s t , a t )] (1)
where γ ∈ [0, 1) is a discount factor, τ = (s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ...) is a trajectory sampled with the policy π and T is the horizon. Different algorithms solve this problem, like the Soft Actor-Critic (SAC) algorithm [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF].

We add the definition of option [START_REF] Sutton | Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning[END_REF]. Options allow generalizing primitive actions, i.e. at one time step, to include sequences of actions. Options are composed of a strategy π, a termination condition β(s) and a starting set I ⊆ S. An option < I, π, β > is available in the state s if s ∈ I. If the option is taken, the actions are chosen according to π until the termination condition is reached according to the probability β. When an option is terminated, the agent can choose another option.

III. METHODS

We propose a modular method that combines the advantages of learning and prior knowledge (optimization-based or user-supplied data) in control problems. It is formalized as an HRL framework based on a user-defined knowledge graph, where the nodes represent configuration spaces of the robot. In this article, we assume that a configuration space corresponds to a set of robot configurations that share the same contact configuration. This method employs multiple agents to identify the current node in the graph (e.g., which contact configuration the robot is in), navigate through the graph (e.g., how to change the contact configuration), and solve the task in each node of the graph. This structure allows for the integration of various optimization methods. It enables also to reuse pre-trained agents and provides interpretability. The details of this approach are presented in the following sections.

A. General overview

The user defines a graph of N configuration spaces S i covering the state space of the robot S (see section III-B.1 for details). The use case limits the state space, and the covering ⊔ i≤N S i = S is defined according to the task. Each configuration space is identified with a vector h i ∈ R ds . The graph naturally induces a notion of neighbourhood V i of the configuration space S i . Two nodes are linked when it is possible to go from one configuration space to another by following a sequence of actions without going through a third different configuration space.

Given this graph, several agents are used to solve the task:

1) The Selector S determines from the state s t in which configuration space the robot is. To do this, it calculates the probability that the current state belongs to a given configuration space:

S(S i |s t ) = p(s t ∈ S i ) (2)
where S i |s t means "being in configuration S i knowing that the robot state is s t ". The Selector is deterministic when there is no uncertainty about the configuration spaces. Indeed, when the Selector is learned, there is uncertainty and the Selector's aim is to maximise the probability p(s t ∈ S i ). When the Selector is implemented using collision detection algorithms, there is only one possible choice and the probability is 1 or 0. The Selector is the agent that implicitly defines the knowledge graph. It associates each state with the configuration space that contains it, either by learning from data labelled according to the configuration spaces or by using an external criterion, such as a collisiondetection algorithm.

2) The Evaluator E determines if the robot should stay in the current configuration space to solve the task, or if it should change. It gives the probability of success p succ that the task can be solved in the configuration spaces:

E(S j |S j ∈ V i ∪ {S i }, s t ∈ S i ) = p succ (s t+1 |s t+1 ∈ S j , s t ∈ S i ) (3)
This probability corresponds to the action of the Evaluator (network output), and is trained to maximise its cumulative reward. Depending on the configuration space that maximises this probability, the Evaluator uses one of the two last node-specific agents to find the next action: the external agent or the internal agent. 3) An external agent A ext is used to switch from the current space S i to another S j̸ =i ∈ V i in the neighbourhood. This agent uses options. 4) An internal agent A int is used to solve the task inside the current configuration space. This agent uses primitive actions. Each agent can be either learned or not, and can share information with other agents. The Selector and the External agents are independent of the task to be performed, while the Evaluator and Internal agent are not. An overview of the method and the link between agents is shown in Figure 1.

B. Structure and Training of the different agents

In this section, we present all the trained agents, using off-policy learning and replay buffers.

1) Train the Selector: The configuration spaces S i are labelled according to the prior knowledge. The Selector is based on an attention mechanism [START_REF] Vaswani | Attention is all you need[END_REF]:

α = sof tmax( QK T √ d s ) (4)
where K = F (s t ) is the encoded current state, F a Multi-Layer Perceptron (MLP) network, Q is the matrix of the identifiers h i , and d s the dimension of the identifiers. It gives the similarity between the encoded current state and each identifier. The softmax function gives a probability α i = p(s t ∈ S i ) that the state s t belongs to the space S i . The current configuration space is given by the maximum of the probability. For the training, labelled data are collected, and the Selector has to predict the correct label in a supervised way according to:

∀s ∈ S i : α i = 1 and α j̸ =i = 0 (5)
2) Train the Evaluator: The Evaluator finds the probability of solving the task in V i ∪S i using an attention mechanism as well. For the Evaluator, K is the matrix of the identifiers and Q is the encoded current state. The result is given by α ∼ N (α mean , α std ) where α mean and α std are evaluated using Eq. 4 and N is the normal distribution. The rest of the learning is a classical RL learning process. Two different transitions are used, depending on whether the Evaluator chooses external or internal agents. They can be both express as a list:

(s t+k , a t , s t+T , t+T t ′ =t+k r t ′ , δ) (6)
with k an integer in [0, T -1], s t+k ∈ S i , s t+T ∈ S i for internal agents and s t+T ∈ S j̸ =i for external agents, T the duration of the option with T = 1 for internal agents (primitive action) and T ≥ 1 for external agents, and δ a flag to indicate the termination of the episode. Internal and external agents are used to explore the state space. The Evaluator can be penalized if it uses internal agents to move from one configuration space to another one.

3) Train the internal agents: They are trained with all the transitions starting from the current configuration space. The internal agents may not find solutions in this space and have to explore other spaces to do so. Although the Evaluator has to choose the external agents for this exploration, it may make mistakes or converge slower than the internal agents. To solve this problem, each internal agent has information about its neighbourhood. The idea is to update the Qfunction of each internal agent using the Q-function of its neighbouring internal agents for the boundary states (which allow switching from one configuration space to another in one action). This allows the internal agents to know if it is beneficial to visit adjacent configuration spaces or not.

4) Train the external agents: The implementation of external agents is not dependent on the task to be performed and is based on concepts from HER [START_REF] Andrychowicz | Hindsight experience replay[END_REF] as follows. The current state s t is augmented by the value of the identifier of the target configuration space h i to form the input of the external agent [s t |h i ] where | represents the concatenation. The idea is then to say that a trajectory τ allowing to reach a configuration space S i does not allow reaching S j̸ =i . [s t |h i ] following τ is rewarded while [s t |h j ] is penalized.

IV. MATERIALS: ROBOTS AND CREATION OF CONFIGURATION SPACES

We illustrate our method with two examples: the Cart-StemContact [START_REF] Schegg | Sofagym: An open platform for reinforcement learning based on soft robot simulations[END_REF] and the RodManipulator made of two-soft cylindrical robots called fingers [START_REF] Coevoet | Soft robots locomotion and manipulation control using fem simulation and quadratic programming[END_REF]. These robots and their associated configuration spaces graphs are shown in Figure 2. Both systems are simulated using SofaGym [START_REF] Schegg | Sofagym: An open platform for reinforcement learning based on soft robot simulations[END_REF], which is based on SOFA [START_REF] Faure | Sofa: A multi-model framework for interactive physical simulation[END_REF]. The learning process is implemented using an open-source implementation of SAC [START_REF] Keng | Slm lab[END_REF], [START_REF] Parker-Holder | Minimal pytorch soft actor critic (sac) implementation[END_REF]. Other RL algorithms were also used in [START_REF] Schegg | Sofagym: An open platform for reinforcement learning based on soft robot simulations[END_REF], and led to comparable performance. The hyperparameters are the same between the baseline method and our method. The code and the hyperparameters are open-source.

A. The CartStemContact example

The CartStemContact can be considered as a simplified soft robot. As illustrated in Figure 3, it highlights the control challenges that are associated with the contact configurations in soft robotics. Fig. 3.

Illustration of the limitations of optimization-based control approaches in the case of the CartStemContact robot. A deformable beam is fixed on a mobile base that can move horizontally. Two obstacles limit the movement of the beam. The objective is to minimize the distance between the end of the beam and a horizontal position behind one of the obstacles. (A) When the robot is not in contact with an obstacle, the use of optimization-based control leads to a local minimum. (B) solve the task, the robot must first be in contact with the opposite obstacle. The presence of the contact between the obstacle and the robot changes the optimization space.

The objective is to position the tip of a mobile beam at a specific horizontal location despite obstacles obstructing its horizontal movement. These obstacles can be used to bend the beam, and the solution to optimization-based control methods depends on the initial configuration of the beam's contacts. If no contacts exist at the beginning, the optimization falls into a local minimum. This task can be solved using reinforcement-learning algorithms; however, they require significant learning time [START_REF] Schegg | Sofagym: An open platform for reinforcement learning based on soft robot simulations[END_REF]. Because the solution to this example is known, the proposed approach can be compared with it.

An episode last at most 30 iterations, and the reward is equal to the horizontal distance between the tip of the beam and the goal. The position of the contact and the position of the mobile base are randomly initialized. The state s t is [x cart , x tips , x lef t , x right , l x , l z , x goal ] and contains the horizontal position of the mobile base, of the tips, of the obstacles and of the goal, and the dimension of the obstacles. The mobile base is controlled in position with a constraint on the maximum speed of the cart.

The state space is split according to the contact between the beam and the obstacles, as defined by the geometric conditions:

x cart >x lef t + 1/2 l 2 x + l 2 z x cart <x right -1/2 l 2 x + l 2 z ( 7
)
This partitioning enables the identification of areas that can be reached by the end effector of the robot. This process does not require an explicit expression of the physical model, but rather only the observation of the robot's current state.

B. The RodManipulator example

The method is illustrated by an example of object manipulation. The RodManipulator robot is used to catch and manipulate an object using two soft fingers. Each finger can bend in four directions and translate vertically. The object to be manipulated is a rod with a square cross-section placed on a table, and the goal is to rotate it using both fingers to reach a target vertical orientation. The state of the robot is defined by the positions of three points at the end of each finger, the position and orientation of the centre of the rod, and the target orientation. The reward is based on the normalized angle covered by the rod during its movement. Because there is no simple formulation for modelling the mechanical behaviour of the robot, a collision detection algorithm is used to identify the contact configuration. In this case, the Selector is deterministic and computed rather than learned. Contact configurations that are not useful for the task are gathered in the same configuration space, allowing to not have to define them in the graph and explore/train the corresponding agents.

V. RESULTS

A. Solving the CartStemContact and convex optimization

The Selector's training is achieved by randomly collecting states and using them to train the model. In supervised learning, the data set is generally separated into a training set and a validation set. In this work, the validation set consists of 20% of the labelled data, and the model has a success rate of over 99.7% for 50000 test samples. The Selector manages to associate each observation to its corresponding configuration space.

The learning results are shown in Figure 4. The proposed method can solve the task in 80000 iterations, whereas the baseline SAC requires 140000 iterations. This corresponds to a 42% reduction in the number of iterations required to achieve the same performance level. Both approaches have the same limitations, due to the definition of the reward: when the goal is close to the bounds of the obstacle, the cumulative reward is lowered by getting stuck to the obstacle.

The learned internal agents can be replaced by convex optimization algorithms. The training for the other agents Fig. 4. Learning results, reward as a function of the iterations. Results obtained with the SAC algorithm (orange), with our method (blue) and with our method with internal agents performed with convex optimization (purple) in the case of the CartStemContact. The learning conditions are the same in all three examples. The initial difference comes from the fact that the first results are obtained after 500 iterations, and that the method with internal agent performed with optimisation learns to solve the task faster than the other methods. Sliding average is used to facilitate the reading of the results. The size of the windows for the sliding average is approximately 2.5% of the number of iterations. remains the same. In this setup, the task is solved in 40000 iterations, which represents a 72% reduction in the number of iterations compared to the baseline. To validate the reusability of the agents, the fully trained internal agents of the network are replaced with convex optimization algorithms. When one or several internal agents change, the overall learning behaviour remains the same, as the value of the cumulative reward changes by only 0.64% compared with all learned internal agents. These features make the approach particularly easy to combine with the existing methods.

B. Manipulation of the rod and expert trajectories

We define two expert trajectories. They allow the fingers to: 1) move away from the rod, one finger being brought forward and the other backward; 2) reverse the position of the fingers, the one that was forward now being backward and vice versa; 3) bring the fingers in contact with the rod in a new contact configuration. An illustration of these expert trajectories corresponds to the four images of the first row in Figure 5. The expert trajectories are hand-defined to guide the system from one contact configuration to another, but they do not specify the target configuration space. In fact, as they are not designed for a specific state, the Evaluator cannot use them to precisely reach a given configuration. However, the Evaluator can use these expert trajectories (external agents) to move out of the configuration space and modify the contact configuration. As these trajectories involve moving the fingers away from the rod, a penalty is included in the reward to encourage the Evaluator to use them only when it would result in a significant increase in its long-term reward.

The cumulative reward for the RodManipulator is shown in Figure 6. The proposed method learns to use expert trajectories effectively to solve the task. An example of solving the task for a 280°target is shown in Figure 5. Starting without contact, the algorithm uses an external agent to create contact. The rod is then manipulated until it is impossible to move further without changing the contact con-Fig. 5. Example of RodManipulator task resolution for a target angle of 280°. Starting without contact, the algorithm first uses an expert trajectory to position the robot in contact with the rod. The rod is then manipulated until it reaches a configuration where it is not possible to move further without changing the contact configuration. The Evaluator then uses another expert trajectory to reconfigure the contacts and continues to rotate the rod until it reaches 280°. figuration. Finally, an expert trajectory is used to reconfigure the contacts and solve the task.

The results achieved with this algorithm can be compared to those obtained using a baseline SAC. When the robot starts without contact, this algorithm requires a long time to explore the space without finding actions to manipulate the rod. To ensure that the robot approaches the rod, a distance term can be added to the reward. In the proposed method, the reward is easier to design because part of the complexity is delegated to the expert trajectories and the definition of the contact configurations that the robot must visit to solve the task. Once the contact with the rod is made, the SAC algorithm is unable to reconfigure the contacts in less than 200000 iterations.

VI. DISCUSSION

A. Use of configuration spaces in the learning

The previous results demonstrate that it is possible to integrate prior knowledge into the learning process by defining a Fig. 6. Learning results, reward as a function of the iterations. Results obtained with our method in the case of RodManipulator. Sliding average is used to facilitate the reading of the results. The size of the windows for the sliding average is approximately 2.5% of the number of iterations.

knowledge graph based on configuration spaces. This graph allows for greater sample efficiency while maintaining the flexibility brought by learning.

This method provides interpretability during the learning process. Examining the results of the Selector and the choices of the Evaluator allows determining the agent's current configuration space and the targeted configuration space to solve the task. The CartStemContact first navigates to the space containing the goal and then solves the task within that configuration space. The Evaluator of the RodManipulator employs external agents to reconfigure the contacts.

B. Reusability

The Evaluator uses both external and internal agents to solve the task at hand. It employs internal agents to correct the results of external agents. The example of the Cart-StemContact demonstrates that using learned or not learned internal agents does not lead to a decrease in performance, as the entire network has the same behaviour. The two examples show the possibility of reusing and combining pre-learned, learned, or not learned agents. This validates the use of a task-independent structure as the basis of the learning process. The Evaluator can leverage the knowledge of external agents to solve a given task more efficiently.

C. Combine classical learning and prior knowledge

Optimization-based algorithms can only solve a task if the goal belongs to the same convex space as the effector's one. The CartStemContact shows that we can continue to use the advantage of optimization-based algorithms by learning the transition between two configuration spaces, even if they correspond to different convex spaces. This has two benefits: it can improve general sample efficiency and overcome the limitations of a convex approach. However, this study should be extended to include simultaneously learned and not learned internal agents. This would be particularly interesting in the case of the RodManipulator, where two opposite contact configurations can be handled using convex optimization, whereas the other configurations are learned.

Another prior knowledge is provided by the expert trajectories. Unlike Behaviour Cloning methods or other approaches that use entire trajectories, the definition of the knowledge graph and sub-part of the trajectories guide the learning by defining the passage point necessary to achieve the desired behaviour. In the RodManipulator example, the use of expert trajectories to reconfigure contacts is easier to obtain than Behaviour Cloning because it is independent of the task to be solved and does not correspond to an entire episode. Although this trajectory is not optimal, the algorithm can use it to solve the task and improve the efficiency of its own strategy.

VII. CONCLUSION

In this article, we present a new method for integrating prior knowledge into a learning model. By defining the configuration spaces, we can effectively partition the state space and facilitate the learning. These configuration spaces can be derived from the equations of the agent's behaviour, examples of the agent's configurations, or a desired sequence of behaviours. The use of multiple agents allows compensating the weaknesses of one agent by using others, integrating behaviours from other control methods such as convex optimization, and specialising agents in specific domains. As some parts of the method are independent of the task to be performed, they can be reused to speed up the learning.

In this article, we propose to use the contact configurations to create the configuration spaces, motivated by the importance of optimisation approaches in soft robotics. However, other approaches could also be considered, such as automatically defining the configuration spaces based on a robot's workspace in a specific contact configuration. This method is an intuitive way to incorporate prior knowledge into learning algorithms. Even though it can be applied to other fields than soft robotics, it is particularly relevant for the control of soft robots as traditional methods for path planning and predictive control with contact are limited.

One limitation of the method is the difficulty of defining different contact configurations. We illustrated the method with two examples with limited numbers of known contact configurations. The definition of these contact configurations in a more general case is not straightforward. In addition, the complexity of the general structure of the network has some limits. Each agent corresponds to one network, increasing the number of hyperparameters and update at each time step. Moreover, the definition of the configuration spaces can influence the convergence speed. In fact, agents share information, and an under-exploited agent takes more time to learn and thus to provide reliable information. There are several ways to improve this method. First, it would be interesting to extend the application domain of this method, for example, by using multitask and dynamic graph. With a method for dynamically learning configuration spaces, these spaces will be useful for the task and will be built to solve the task. In addition, the more multitask objectives the robot has to solve, the more general the configuration spaces will be. Modifications of the learning conditions could also be considered, such as local updates of the networks when the number of configuration spaces becomes large or the reward of the external agents depending on the length of the sequence of action.

Fig. 1 .

 1 Fig. 1. Schematic of the different elements constituting the knowledge graph. Each node in the graph corresponds to a configuration space, and each observation belongs to one node. Each node has an identifier, a neighbourhood (other configuration spaces), an internal agent (in red), and an external agent (in blue). The Evaluator (in pink) can either belong to a node or be shared between the nodes. The Selector (in orange) is global in the knowledge graph. Three steps are then performed. (1) The Selector determines in which configuration space the robot is. (2) The Evaluator decides in which configuration space to go. (3) The internal agent or external agent solves the task in the current space or moves to a different space.

Fig. 2 .

 2 Fig. 2.Splitting the state space into different configuration spaces (right) for two soft systems (left) based on the contact configuration. (A) CartStemContact. (B) RodManipulator. In this example, some contact configurations are gathered in one configuration space, not useful for the manipulation task.

ACKNOWLEDGMENTS

This work was supported by the TIRREX project, grant ANR-21-ESRE-0015.