16 ème Congrès Français d'Acoustique 11-15 Avril 2022, Marseille

The buffet of a pipe organ encloses a complex field created by thousands of pipes acting simultaneously as scatterers and point sources -one in the mouth and one in the passive end, when pipe is not stopped. The facade of the organ, otherwise called montre, couples the outside space in the nave with the aforementioned field inside. The main opportunity for field transmission is through the aperture offered by the separation between each of the pipes in the montre. While closed-form models and numerical simulations exist for the transmission through a single slit, the facade of an organ amounts to tens or hundreds of slits thus posing the questions of how to predict the transmitted field and how to group up its behaviour into smaller clusters. An empirically-based approach is followed to provide a description of the field outside the buffet. Measurements were carried out at the near-field and far-field radiation spaces with an experimental scaled model of a positive organ by changing conditions on the number of rigid walls and the density of scatterers.

Introduction

Modeling musical instruments has become a way to render accessible certain aspects of instruments that are otherwise bound to vanish from collective imagery. It entails an interest, on the one hand, for academic purposes to achieve a higher comprehension of its sounding mechanisms and, on the other hand, an augmentation of such instruments within the context of virtual reality. Pipe organs have been strongly associated with liturgical services and within the confinement of cult spaces. Needless to say, its lack of transportability to laboratories strengthens that prejudice.

In the frame of wind instruments, one seeks to find information along the following axes : the sound production mechanism, coupling mechanisms, the transfer function of the body of the instrument, and the radiation apertures -if the pressure supply at the beginning of the chain is to be left out. For the sake of simplicity, the following assumptions can be made : there is a predominance of flue ranks of pipes over reed ones and, similarly, with non-stopped over stopped pipes, that is, most of them having two open ends.

Pipe ranks inside the buffet are organised in several standard ways inside the buffet, which vary depending on the organ maker and the country and time of its construction. A first major concern to that variability relates to the subdivisions of the interior volume into smaller clusters, e.g. stops in swell box are bound to be found in an independent enclosure. A second one relates to the extent of cluttering, in other words, the density of pipes per squared meter. A last point to make about the internal pipe distribution is the symmetrical chromatic principle generally followed to locate adjacent semitones within the octaves : they are placed on alternating sides within the organ sections (great organ, pedal, positive, etc.) in order to provide spatial balance. Nevertheless, one can imagine that such inverted triangle-like distributions (taller to shorter pipes) will leave vast empty regions with taller pipes leaning towards the side walls of the buffet.

The scope of this study chiefly addresses contributions on the following aspects : the pressure field inside the buffet, the pipe radiation, the filtering effect of the facade, and the field outside the buffet in front of the facade. The eigenmodes of the rigid empty cavity, lossless and without sources, is a well-known consideration [START_REF] Kinsler | Fundamentals of Acoustics[END_REF], whereupon one adds the organ pipes with distinct heights and diameters. For the reasons above, these constitute a 2N p system of radiating monopoles (with N p eing the number of pipes), on the one hand, and to a great extent heterogenous media of cylinder-like scattering obstacles, on the other. Radiation impedance at the passive end of such cylinders is also well-known case of the unflanged pipe [START_REF] Chaigne | [END_REF] and research effort has also been put forward towards characterising the pipe's mouth impedance [START_REF] Ernoult | Window impedance of recorder-like instruments[END_REF][START_REF] Rucz | Finite element simulation of radiation impedances with applications for musical instrument design[END_REF]. From this point on, it is desirable to gain a better global understanding of the incident field on the slits between the pipes of the facade and the directivity and filtering transformations that it undergoes by propagating through and outwards into the room. For the current work, this has been approached by comparing data from measurements and simulations.

In Sec. 2 the test case and the setup are outlined. Sec. 3 puts forward analytical descriptions of the expected behaviour and compares those with matching simulations. The analysis is extended to acquisitions using the experimental organ in Sec. 4 to conclude with remarks on similarities between patterns observed and what is envisaged to find in further investigations. This study is to be considered complementarily with the companion publication "La boite expressive de l'orgue : étude acoustique de la densité de tuyaux et d'une facade a ouverture variable" by Dal Moro et al. [START_REF] Moro | La boîte expressive de l'orgue : étude acoustique de la densité de tuyaux et d'une façade à ouverture variable[END_REF], with further dimensioning details and an explicit coordinate system.

Test cases and setup

To begin, one considers reducing the problem complexity to two dimensions, notwithstanding that studies exist for the whole space with little loss of generality [START_REF] Park | Acoustic scattering from a rectangular aperture in a thick hard screen[END_REF][START_REF] Mellow | On the sound fields of infinitely long strips[END_REF]. To this end, an experimental homologue of an organ buffet was built and a scattering matrix of organ pipes was assimilated to an array of equal cross-section cylinders. The two-monopole model for one pipe was substituted by a cylindrical single-source strategy : a line array of in-phase acoustic sources was placed along the third dimension in the locus of one cylinder, while all other pipes were passive scatterers. It is obvious that the experimental buffet is not a real instrument, but a means of isolating elements under measurement. Details on the simulations are given in Sec. 3.2.

The conditions of interest were the presence and number of bounding walls in the buffet, the presence and density of cylinders in the matrix inside the cavity (also referred as the forest [START_REF] Chobeau | Modeling of sound propagation line matrix method[END_REF]), the single or multiple slit radiating system and the distance from the lines of acquisition to the interface between the inside and outside of the cavity. The aim was to investigate the strength and projection of low-frequency modal responses of the cavity found in the outer domain, pinpointing characteristics that appear across all conditions or only in response to some of them.

The array of scatterers is composed of 170 PVC hollow cylinders with a diameter of 40 mm cut to a height of 950 mm, embedded into horizontal top and bottom perforated plates made of plexiglass that support all the scatterers in the matrix in place : there are 11 rows and 16 columns available for density combinations, where odd and even rows appear staggered. The pipes are filled with acoustic foam and they serve also the purpose of supporting the acoustic source by clipping it onto any pipe. The source is a vertical array of 18 Aurasound NSW2-326-8A with dimensions 65 mm × 50 mm × 933 mm. The enclosing cavity around is made of plywood panels of 15 mm thickness supported by a structure of beams : 35 mm × 35 mm for the section of the four vertical pilar edges and 20 mm × 20 mm for the rest. The inside dimensions of the cavity are 1340 mm × 580 mm × 980 mm. Five of the cavity panels are removable, allowing for changes in conditions on the walls and for ease of access to the matrix of pipes inside. A window region was cut in the front panel so that interchangeable plexiglass panels can be attached with different slit conditions to the front end of the system. Finally, a small hole was drilled at the bottom of the rear panel in order to pass the wiring necessary for source driving and acquisitions inside the cavity. A Carver PM-175 amplifier was used for the entirety of the experiments to drive the array source.

Assessment of analytical descriptors 3.1 Expected responses from theoretical transfer functions point of view

We attempt to find source-to-system equivalents that resemble the behaviour of acoustic flows and we encapsulate these into a set of transfer functions. The filtering effects expected in the transmitted near-field outside the cavity can be split into three main contributions :

H buf f (ω) = H bw (ω) • H sw (ω) • H M X (ω) (1)
with H bw expressing the maxima and cancellations seen at a given observation point O( r) due to phase interference between the arrival of the direct wave-front and the reflected image-source on the back wall of the cavity, being a distance of 2L bw from each other, where |O| L bw . The parallel side walls of the cavity give rise to eigenmodes for the standing waves encapsulated in H sw . Lastly, H M X expresses the the effect of the forest matrix.

The back-wall response for an observation point in the middle front position takes the form :

H bw = P (k, r) P o = 1 √ r e -jk(r-L bw ) [1 + e jk2L bw ] |H bw | ≈ 1 (r) |1 + e -j2kL bw |e -jωτ (2)
where P o is the reference pressure amplitude, k (•) is the wavenumber and L bw is the distance from the source to the back-wall, yielding 478 Hz as the first null (for a sourceto-wall distance L bw = 176 mm). The minima shown by the phasor e -j2kL bw at which phase cancellation occurs are distributed in wave number as follows :

k n = (2n + 1)π 2L bw (3)
The presence of the side walls yields pressure maxima and zero velocity on the end-points of the cavity. Its transfer function can be expressed as the ratio Z = P/U between the acoustic flux exciting the cavity and the pressure response :

H sw = 1 jρc o n cos(k n x) n sin(k n x) (4) with x ∈ [0, L sw ], k n = nπ/L sw
, and L sw = 1340 mm being the wall-to-wall distance yields a periodicity in spectra of 128 Hz.

Lastly, a forest matrix under infinite extension assumption can be described in terms of wavenumber perturbation :

H M X = F (k ISA , r)H rsc (5)
where H rsc accounts for the scattering phenomena arising from the regularly-spaced cylinders, covered in Sec. 3.2.2.

The wavenumber k ISA has undergone a perturbation due to the presence of the forest, whose most salient events are an attenuation of the propagation velocity and pressure amplitude, ultimately shifting downwards the spectra. Some research literature is found to investigate this modified wavenumber, following the infinite-domain extension assumption, and a first order expression named independent scattering approximation (ISA) [START_REF] Norris | Multiple scattering by cylinders immersed in fluid : High order approximations for the effective wavenumbers[END_REF][START_REF] Rohfritsch | Numerical simulation of two-dimensional multiple scattering of sound by a large number of circular cylinders[END_REF] :

k 2 ISA = k 2 o -j4n o F (θ) (6)
where n o is the density of the scattering system and F (θ) is a scattering function.

The contributions of back-wall and side-wall are seen in Fig. 1, confronting the expected analytical transfer functions and those found in data from simulations. It is seen how the back-wall and side-walls show 478 Hz and 128 Hz respectively. The location of side-wall response shown is placed in the middle of the geometry, causing the response to mostly skip over the even-indexed modal frequencies.

Simulations

Numerical equivalents of the study cases in Sec. 2 are simulated using a hybrid FDTD regular hexagonal mesh, for the domain surrounding the scattering system at hand, with a FVTD mesh circumscribing the rigid bodies, treated as perfectly rigid. The two-dimensional volume cells are adjusted by Voronoi iteration over four buffers around each scatterer to achieve a stable mesh. The outermost buffer couples the FD regular mesh and the FV one. The perimeter of the domain is surrounded by a two-dimensional PML to avoid re-entring propagation from the periodic boundaries.

The excitation conditions injected in the simulation grid is chosen of the form of a scaled Ricker wavelet, this is supported by two reasons : a soft source term is readily differentiated in a wave-equation scheme, with pressure-like dimensions, and a DC contribution is avoided while still adhering to bandwidth and time compactness constraints [START_REF] Sheaffer | Physical and numerical constraints in source modeling for finite difference simulation of room acoustics[END_REF]. CFL is set to 0.95 below the stability condition for the finite volume part proposed in [START_REF] Bilbao | Passive volumetric time domain simulation for room acoustics applications[END_REF].

Data from simulation allows to obtain the field in anechoic conditions : being able to model any realisable condition, one can compute both the reference pressure and the scattered pressure at O j ( r). This has eased the computation of the arrival time difference between scattered and free propagating waves, which in turn allows comparison with perturbation of wave number model amid the forest media.

The following conditions were simulated :

• 3 walls with no cylinders (back-wall and side-walls)

• 3 walls with facade array of cylinders • 3 walls with full dense scattering matrix

Rigid empty cavity

Simulations are done to two degrees of detail : once for the sake of the comparability with analytical expressions where geometrical features are reduced to flat planes ; secondly with greater geometry detail, for comparability with measured results and analysis of the temporal evolution inside and outside the system. Time domain realisation of the entire domain surrounding the system without any scatterers shows the following (see Fig. 2) :

• First direct arrival wave-front (D) followed by the back-wall's first order reflection (BW) [a]

• First order reflection from the flat panels of side walls

Addition of facade and full scattering matrix

It has been shown above how one would expect to find the response of reflections on the main reflective panels (and corners) by means of assuming image-sources relevantly. Next we will see that the addition of the facade and the full dense matrix systems will constitute another direction for image-sources, in the case of the former, situated beyond the facade, and scattering phenomena due to regularly-spaced obstacles in the latter.

Similar issues were explored by D. L. Berry et al. [START_REF] Berry | Acoustic surface wave generation over rigid cylinder arrays on a rigid plane[END_REF] -under (in)finite array assumptions in 1D. They pointed out that some phenomena stronger than the predicted Bragg scattering were observed, referring to them as quarterwavelength frequencies (or gap frequencies) because of the similarity between the 3-wall acoustic trap and an organ pipe with open-closed ends -whose first eigenfrequency is ruled by four times its length.

The discussed trap is configured, first, by each twobounding cylinders where an incident front arrives and, second, by a rigid reflective plane, beyond which the imagesources or image-cylinders are distributed. The pressure terms of the resulting system are :

• Direct source

• Image source • Cylinder sources • Image-cylinder sources

In our setups we located source and receivers on opposite sides of the scattering system, thus discarding the image-terms, but the study case of the 1D pipe array facade remains comparable with the N-cylinders in [START_REF] Berry | Acoustic surface wave generation over rigid cylinder arrays on a rigid plane[END_REF]. In Sec. 4 and 5 numeric detail is provided while compared results from simulations and measurements. One can extend this scattering analysis to the case of the 50% and 100% forest densities without forgetting two aspects : our setup will become an NxM-cylinder system and wavenumber modification will start manifestating spectra shifts as mentioned above, but also delays in the arrival times.

The averaged arrival time differences between the free propagation of the source and the equivalent in the scattered pressure field show the influence of the matrix density and its prominence in terms of acquisition proximity to the interface. See Tab. 1, whereby one uses the lossless dispersion relation c o = ω/k o into Eq. 6 to link it with the propagation velocity :

c = α • c o = c o (1 -) = c o 1 - τ T (7)
Fig. 3 shows a difference in the integrity of a harmonic structure up to ∼1.5 kHz. Higher-frequency scattering originated by the reasons above is hinted in the regions of 2.5 kHz and 3.1 kHz. Closer examination of the lower frequency structure confirms the aforementioned frequency shift : the clear modal structure with two nulls centered at TABLEAU 1 -Time difference between the first direct arrival of the freely propagated pulse and that of the scattered field. Arrival times are considered by the -65 dB crossing of the pressure signal normalised by its peak maximum. T scatt is the arrival time of the perturbation to the interface in presence of scattering bodies. Values are averaged for all acquisition points along a line 20 cm in front of the interface. 520 Hz for the montre falls to 460 Hz when the full density of scatterers is included, which is in good agreement with experimental results. The dense case shows more clearly the formation of the spatially periodic scattered structure beyond 3 kHz.

Experiments

The measurements of the transmitted field are made by a finite line of acquisition points in front and parallel to the facade, covering all its width with resolving every 50 mm to a total of 1400 mm. One acquisition line was positioned at 200 mm from the facade, this is 500 mm from source, and another one at 2000 mm from the source. The excitation signal sent to the acoustic source for these experiments was a sinusoidal sweep sampled at 48 kHz ranging from 20 Hz to 20 kHz and lasting 30 s. Study cases measured in an anechoic room respond to the conditions specified in Sec. 2 and Sec. 3.2. Next, most prominent behaviours encountered are described. Bear in mind two setup factors : first, the alignment of the source position does not correspond with the center of the cavity, since it was clipped on the eighth cylinder from the back non-staggered row. Second, the acquisition locations limit the resolvable frequencies, which in the vicinity of the interface can make it unfeasible to capture some phenomena, especially when happening in non-stationary state.

Scattering effects due to regularly-spaced cylinders have a strong prominence in the case of 3 walls with facade cylinders (see Fig. 5) found within the vicinity of the 8 kHz third octave band. The integrity with which the ondulatory behaviour (with a spatial periodicity of ∼0.5 m) of the scatterered field in magnitude and phase is mostly disrupted by the presence of the back-wall but not with the presence of side walls. This behaviour appears in the band centerd at 15 849 Hz with the same spatial periodicity but with a sideways shift of the peak position, almost exchanging nulls with peaks. The phase behaviour shows the arrival delay for a cylindrical wave-front to reach a straight acquisition line. These results can be compared with the simulated results in Fig. 4 where the dashed line corresponds to the acquisition line of the measurements at 2 m : the energy in the stationary-state field clearly behaves accordingly to the experimental patterns. The facade setup and the 50% dense matrix share the same center-to-center distance between cylinders, therefore the same first quarter-wavelength gap frequency, estimated around ∼1.2 kHz if a length-correction of 0.35d is used (where the radiation equivalent for d here expresses the center-to-center distance). The acquisitions at 20 cm from the interface show a ∼7 dB peak in the band centered at 3136 Hz. This contribution appears also in the case of the 100% dense, matrix but is most prominently seen at an acquisition distance of 2 m, whereas at 20 cm another ∼7 dB peak appears, this time in the band centered around 2512 Hz. By introducing the rest of the scatterers in staggered disposition, the center-to-center distance between cylinders changed from 80 mm to 56.5 mm. This phenomena occur at the same frequency bands independently of wall conditions, where again the addition of the back-wall diminishes the congruence of the side lobes. It leads to believe that the common factor is the presence of the facade.

If we compare the measured and simulated responses (see Fig. 3 and6), we see that the modal structure in the low frequencies displays the shift behaviour of 530 Hz with only the facade falling to 470 Hz when the full scattering matrix is present. Similarly, the modal structure is more prominent in the case where the field is less perturbed, and the formation of scattering due to evenly-spaced cylinders is seen more clearly while the full staggered grid is present, with the smaller center-to-center spacing, as shown before.

Discussion and future work

The current study proposes first-order approximations to the responses of a cavity and a scattering system that behave like an organ buffet. Analytical, simulated, and measured responses show good agreement with respect to cavity responses. We have identified high-frequency patterns caused by regularly-spaced cylinders and showed that the modification of the wavenumber behaved as expected. Further simulations and analysis should be devoted to estimating transfer functions for the far-field directivity, a pseudo-infinite facade pipe array, and an (in)finite scattering matrix forest devoid of cavity. Dispersion analysis was carried out with simulated data but spatio-temporal frequency resolution methods have not yet yield meaningful results in the vicinity of the scatterers of the facade.

scattering effects analytically estimated, measured and simulated may have been overestimated by the 2D approach under the condition of homogeneously distributed, full-height scatterers. We need to assess how much of the attenuation differences between facade and full dense matrix is due to this overestimation. Further work is envisaged on numerically approximating the radiation from the facade as an array of spaced monopoles of the form :

P = 1 4 k o ρ o c o Np n
q n e -jφn H - 0 (k w r n )

with q n , φ n , and r n representing acoustic flow, phase shift and location for each of the N p monopoles.

Similar measurements were carried out in the emptybuffet and facade conditions in the tribune organ of Notre-Dame de Paris. Preliminary analysis yielded similar high-frequency scattering patterns as those described in this study. They prominently appeared at 5 kHz and 10 kHz center-frequencies. Scaling considerations need to be taken into account in order to make comparisons with measurements in the laboratory : size of the organ cavity, facade center-to-center spacings and pipe diameters.

FIGURE 1 -

 1 FIGURE 1 -Normalised magnitude of transfer functions of (a) back-wall responses of 3300 time-steps of P tot /P f ree-f ield using a Bartlett window and an FFT size of 2 15 samples. (b) Side-walls response over 10 3 time-steps of P tot /V tot using a Kaiser window (β = 4), and an FFT size of 2 17 samples. Red and blue solid lines correspond to analytic and simulated responses respectively at an observation point in the vicinity of the interface and slightly off-axis with respect to the source location.

  (SW) followed by reflections in the bounding inner corners (BW+SWc) [b] • Progressive structure of standing waves developing across the longest axis of the cavity [c,e,f] • Scattering from outer bounding edges back onto the back-wall [c] • Second reflection from BW after initial pulse and shedding of steeper-angle fronts [d] • Standing wave structure establishes and increases the steeper-angle of the wave fronts reaching outside the cavity [e,f]

FIGURE 2 -

 2 FIGURE 2 -Temporal evolution of the total pressure field around the cavity system without scatterers (pipes). Images correspond to times 1.76 ms, 3.39 ms, 3.94 ms, 6.29 ms, 8.41 ms, and 11.9 ms. Axes are within to the domain dimensions x ∈ [-2, 2]m and y ∈ [-0.5, 3.5]m.

FIGURE 3 -

 3 FIGURE 3 -Radiation spectra of simulated pressure, top to bottom : (upper) facade cylinders only and (lower) full dense cylinder matrix ; both in the presence of a 3-wall cavity. The radial axis represents frequency [Hz] and the polar axis expresses the radiation angle for an acquisition distance of 20 cm from the interface and a parallel extension of [-0.7, 0.7]m. The scales span 25 and 35 dB respectively.

FIGURE 4 -

 4 FIGURE 4 -Whole-domain scattered pressure field for 3 walls and facade density conditions. A Chebyshev Type 1 filter pass-band of 7.5 kHz to 8 kHz is applied to the time series at every grid point. Expressed in a dB scale within the range [-130, -70] after normalisation with the incident source pressure.

FIGURE 5 -

 5 FIGURE 5 -Measured pressure in front of the facade without forest in the interior. Conditions for each row are from top to bottom : (a) anechoic walls, (b) side walls, and (c) all three walls of the cavity. Data is extracted from third octave band analysis. Magnitudes appear in blue with corresponding dB axes on the LHS, superimposed to the phase information (orange) with axis on the RHS.

FIGURE 6 -

 6 FIGURE 6 -Radiation spectra of measured facade (upper) and full dense cylinder matrix (lower). Data shown (spanning 23 dB) are normalised by the free-field propagation reference pressure of the same setup in absence of any elements of the scattering system.

Acknowledgements

This work was supported by the Institute of Computing and Data Sciences (ISCD) in Paris.


Henrique Donâncio

Laurent Vercouter

Safety through Intrinsically Motivated Imitation Learning

Keywords: Learning from Demonstrations, Deep Reinforcement Learning, Pump Scheduling Optimization

Deep Reinforcement Learning methods require a large amount of data to achieve good performance. This scenario can be more complex, handling real-world domains with high-dimensional state space. However, historical interactions with the environment can boost the learning process. Considering this, we propose in this work an imitation learning strategy that uses previously collected data as a baseline for density-based action selection. Then, we augment the reward according to the state likelihood under some distribution of states given by the demonstrations. The idea is to avoid exhaustive exploration by restricting state-action pairs and encourage policy convergence for states that lie in regions with high density. The adopted scenario is the pump scheduling for a water distribution system where real-world data and a simulator are available. The empirical results show that our strategy can produce policies that outperform the behavioral policy and offline methods, and the proposed reward functions lead to competitive performance compared to the real-world operation.

Introduction

Over the past years, Reinforcement Learning (RL) approaches combined with function approximators have been applied in different tasks such as games [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF][START_REF] Lample | Playing fps games with deep reinforcement learning[END_REF] to control [START_REF] Yang | Recurrent deep multiagent q-learning for autonomous brokers in smart grid[END_REF][START_REF] Wei | Deep reinforcement learning for building hvac control[END_REF]. The appeal of this approach is the ability to support decision-making and leverage scalability for complex domains. However, exploration in large state spaces as found in the real world can be costly, inefficient, and even infeasible. For example, in scenarios such as healthcare systems and autonomous driving, trial and error methods are not an option due to safety constraints. A way to mitigate these problems is using historical interactions with the environment, an approach called Offline Reinforcement Learning 1 . In the Offline RL [START_REF] Levine | Offline reinforcement learning: Tutorial, review, and perspectives on open problems[END_REF] settings, the experiences of the agent are limited to collected data, without the possibility of further exploration. The reasons for this limitation include the complexity of building accurate simulators and safety con- 1 Some works uses the term Batch instead of Offline straints for exploring the environment. Even when online data collection is reasonable, the use of prior datasets capable of generalizing to efficient policies can be attractive due to the costs to interact with the environment. Offline RL methods such as [START_REF] Fujimoto | Off-policy deep reinforcement learning without exploration[END_REF][START_REF] Jaques | Way off-policy batch deep reinforcement learning of implicit human preferences in dialog[END_REF] rely upon the idea of constraining the policy to the dataset to mitigate overestimation caused when facing out-of-distribution state-action pairs. On the other hand, in Online RL, agents interact with the environment in an exploration-exploitation trade-off fashion. A widely applied exploration strategy is the epsilongreedy [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF], in which a given probability trade-off between exploring the environment or exploiting the policy learned. The exploration can also be encouraged by intrinsic motivation as curiosity or disagreement in the state's estimation by augmenting the reward function. For instance, some works [START_REF] Tang | #exploration: A study of count-based exploration for deep reinforcement learning[END_REF][START_REF] Abel | Exploratory gradient boosting for reinforcement learning in complex domains[END_REF][START_REF] Bellemare | Unifying count-based exploration and intrinsic motivation[END_REF] propose strategies based on counting occurrences of states/actions to encourage exploration of unfamiliar areas of the state-action space. In [START_REF] Pathak | Self-supervised exploration via disagreement[END_REF], the reward is augmented based on the disagreement of an ensemble of parametric Q-functions. Finally, methods to perform safe exploration are discussed in [START_REF] García | A comprehensive survey on safe reinforcement learning[END_REF]. In particular, incorporating demonstrations and restricting the exploration to meaningful states can produce safe policies. This work proposes the Safety through Intrinsically Motivated Imitation Learning (SIMIL) strategy that uses the distribution of historical interactions (demonstrations) as a guideline for action selection. The approach works as follows: given a current state, action selection depends on choosing the one that occurs most frequently in the most similar states found in the demonstrations. Later, we augment the immediate reward with an intrinsic motivation [START_REF] Singh | Intrinsically motivated reinforcement learning[END_REF] according to the state likelihood under some distribution of states. The underlying idea is to constrain the policy to state-action pairs found in expert demonstrations using k-Nearest Neighbors (k-NN) to avoid exhaustive exploration. Also, we encourage states that lie in high-density regions under the demonstrations distribution using Kernel Density Estimation (KDE) [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF]. We apply this imitation learning strategy in a scenario of pumping scheduling for water distribution systems (WDS). For that, it is available a dataset of three years of data collected in timesteps of one minute from a real-world operation. The pump scheduling is the process to decide when, and in some cases at which speed, the pump(s) should operate regarding the forecasting of the water demand. Yet, some requirements must be satisfied, including safety constraints of water level in the tanks and pressure in the network's nodes. Some works have addressed these questions through several methods, including linear optimization, evolutionary and branch-and-bound algorithms, and recently Deep RL [START_REF] Costa | A branch-and-bound algorithm for optimal pump scheduling in water distribution networks[END_REF][START_REF] Georgescu | Pumping station scheduling for water distribution networks in epanet[END_REF][START_REF] Abiodun | Pump scheduling optimization model for water supply system using awga[END_REF][START_REF] Hajgató | Deep reinforcement learning for real-time optimization of pumps in water distribution systems[END_REF]. This work uses a Deep Q-Networks (DQN) [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF][START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF]-based approach to handle the pump scheduling problem. The contributions presented in this work are the following:

• A formulation of the pumping scheduling problem using Partially Observable Markov Decision Process (POMDP) is presented, with definitions of system states/observations, actions, and reward function. These definitions allow the system to operate by achieving the constraints and minimizing the associated costs;

• An imitation learning strategy using real-world/offline data. The empirical results demonstrated that the obtained policies achieved competitive average cumulative rewards compared with fully-offline training.

• To evaluate the proposed scheduling, we compare the results with the real-world water distribution system regarding the electricity consumed, pumps use distribution and the tank level profile. The results showed that our approach achieved competitive performance with real-world operation.

The organization of the paper is as follows. Section 2 describes some related works. Section 3 introduces the pump scheduling problem and its formalization as a POMDP; Section 4 presents technical aspects. Section 5 describes the imitation learning strategy proposed. Section 6 describes the conducted experiments and shows the obtained results. Finally, are presented conclusions in Section 7.

Related Works

Offline RL methods rely on the capacity to exploit and generalize from static datasets to efficient policies. Although, leveraging the learning process using prior experiences can be challenging due to the distributional shift issue and overly optimism in the face of uncertainty [START_REF] Fujimoto | Off-policy deep reinforcement learning without exploration[END_REF][START_REF] Levine | Offline reinforcement learning: Tutorial, review, and perspectives on open problems[END_REF]. The Random Ensemble Mixture (REM) [START_REF] Agarwal | An optimistic perspective on offline reinforcement learning[END_REF] used in this paper addresses this problem by using a convex combination of Qvalues to mitigate overestimation under the assumption of a diverse and large dataset. Also, it can be adopted orthogonally to other sampling methods allowing further online data collection. Similarly, Jaques and colleagues [START_REF] Jaques | Way off-policy batch deep reinforcement learning of implicit human preferences in dialog[END_REF] handle the overestimation issue by applying a dropout-inspired Q-learning and penalizing divergence from prior data distribution through KL-control. Batch Constrained Deep Q-Learning (BCQ) [START_REF] Fujimoto | Off-policy deep reinforcement learning without exploration[END_REF][START_REF] Fujimoto | Benchmarking batch deep reinforcement learning algorithms[END_REF] also constrains the policy regarding actions found in the dataset using as a baseline a generative model. Some other works have used demonstrations as a pretraining step. In [START_REF] Hester | Deep q-learning from demonstrations[END_REF] is presented Deep Q-learning from Demonstrations (DQfD) that uses demonstrations as a pretraining and later improves the learned policy with selfgenerated experiences. The pre-training phase applies a supervised loss to ground values from unseen actions regarding the demonstrations. After the pre-training, DQfD interacts with the environment through the learned policy.

In [START_REF] Nair | Overcoming exploration in reinforcement learning with demonstrations[END_REF] the method incorporates the actor-critic algorithm DDPG, and a loss is applied to tie the policy to the offline data.

As an imitation learning strategy, this work interacts with the environment by performing a density-based action selection using the demonstrations as a distribution. Some approaches instead use this distribution to create models to perform exploration. For instance, Model-based Offline Policy Optimization (MOPO) [START_REF] Yu | Mopo: Model-based offline policy optimization[END_REF] builds a model using supervised learning and then penalizes the uncertainty in further interactions based on the model's error estimation. Therefore, MOPO balances the return and risk in collecting experiences out-of-distribution of the support data.

Similarly, the Model-Based Offline Reinforcement Learning (MOReL) [START_REF] Kidambi | Morel: Model-based offline reinforcement learning[END_REF] proposes to learn a policy for a pessimist MDP (P-MDP) using offline data. This P-MDP partitions the state space according to the uncertainty, applying a reward penalty to unknown areas. In [START_REF] Schrittwieser | Online and offline reinforcement learning by planning with a learned model[END_REF] the authors propose a model-based approach that can mix online data collection with prior offline data. For that, a model is built and incrementally improved through Monte Carlo Tree Search rollouts.

Imitation Learning approaches [START_REF] Hussein | Imitation learning: A survey of learning methods[END_REF] aims to mimic the behavior observed in demonstrations. In [START_REF] Bentivegna | Learning tasks from observation and practice[END_REF] is proposed a hierarchical method for action selection with selfimprovement over time. The first step is to select a primitive that corresponds to some behavior. The second step is selecting a sub-goal achieved by performing the chosen primitive. Finally, an action generator picks a policy to execute the primitive. The underlying idea is to improve the action generator by practicing. Similar to our work and [START_REF] Cardamone | Learning drivers for torcs through imitation using supervised methods[END_REF], this approach uses k-NN queries to retrieve a primitive from demonstrations given a current state. The difference lies in the fact that we propose a passive sampling approach, letting the evaluation of the state-action pairs to the learning method.

Our work has a basis on the wealth of literature on imitation learning and offline RL. However, the intersection between these branches remains underexplored. While offline methods rely upon methods to improve the exploitation of static datasets, the exploration often seeks to uncover areas in the state space. Yet, static datasets may not be large and diverse, and exploring unknown states can produce undesirable behaviors. Thus, the premise of our contribution is to increase the sample efficiency. We consider expert demonstrations as an underlying model for action selection and encourage policy convergence to high-density regions under the demonstration distribution through intrinsic motivation.

Modeling the pump scheduling problem

In water distribution systems, pump scheduling is a decision process about when operating pumps to supply water while limiting electricity consumption. Therefore some constraints must be respected, including a minimum pressure within the network, safety water level in the tanks, and avoiding frequent switches in pump operation to protect the assets. To that end, distinct strategies can be used according to the particularities of the system. For instance, pumping water in off-peak hours when the price of electricity has different tariffs throughout the day or reducing the tank level in periods of low consumption to preserve the water quality, and so on. The water distribution system used here is located in Worms, Germany, and supplies water for about 120000 citizens 2 . The composition of this system is one station with four pumps (NP1, NP2, NP3, NP4), with distinct settings and fixed speed (ON/OFF). The flow Q through those pumps is proportional to the electricity consumption kW , being NP1 > NP2 > NP3 > NP4. In other words, using pump NP1 supplies more water in the network than pump NP2 but also corresponds to higher electricity consumption. Also, two storage tanks with different capacities are placed and provide water for the end consumers. Among the constraints and requirements established in the operation settings for this system are the following:

• It is desirable to avoid frequent switches and distribute pump operations to protect the assets;

• It is imposed a boundary condition of the tank level and, once achieved, the minimum pressure is guaranteed;

• It is desirable to provide water exchange in the tank during one day of operation to keep the water quality.

The tank is located 47m above the pumps and has a 10m length. Thus, the tank levels considered are in the range of [47, 57]m. We consider only one tank once that the second has the level stable along with the operation. The specialists assume a safety operation guarantee with at least 3m filled with water. Besides this, the system does not have sensors measuring the water's quality. Thus, to ensure the exchange and preserve the water's quality, we assume that in one operation day, the level must decrease below half of the total capacity. Finally, the upper boundary constraint overlaps the physical limit.

As with many real-world tasks, the scenario of pumping scheduling is partially observable. In other words, the agent has a noisy or incomplete observation of the environment. For instance, most of the state's features are noisy once it has been gathering by sensors. Also, the water demand has variance along the hours, days, and seasons, even following a pattern. A POMDP is an extension of MDP that considers uncertainty regarding the current state of the environment. Formally, the POMDP can be defined as [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]:

P OM DP = < S, A, P, R, Ω, O, γ > (1)
where the set S correspond to the States of the environment; A is defined as the set of Actions available; P is the Transition Probability which defines the probability being in some state s t ∈ S, taking an action a t ∈ A, resulting a next state s ′ t+1 ∈ S; the Reward r t ∈ R is the return to be in some state s t and perform an action a t ; O is the set of conditional probabilities of take an action a t in some state s t and receive an Observation o t ∈ Ω about the next state s ′ t+1 ; Finally, γ is the Discount Factor ∈ [0, 1] which determines the relevance of immediate rewards over rewards in the future. The States S and the Observations Ω are interchangeable in the context of this work as adopted in [START_REF] Hausknecht | Deep recurrent qlearning for partially observable mdps[END_REF] and represented by:

• The water level in the tank and water consumption;

• The previous action performed (currently being applied);

• The cumulative time that the pumps operated in a horizon length of 24 hours, the month and time t;

• A binary value called water quality indicating whether on the current day of operation the system has reached a certain minimum in the tank level.

Actions A are defined by the set of binary values that represent if some pump is operating (value 1) or not (value 0) once the pumps have fixed speed. At each timestep, only one pump is running or none of them. Finally, two Reward functions are designed to choose the most efficient pump at a given time t, as well as respect the boundary conditions of the tank level, preserve the water quality, and make use of different pumps. The immediate rewards are defined by the Equations 2 and 3:

r t = e 1/(-Qt/kWt) -B * ψ + log(1/(P + ω)) (2)
r t = -e (-1/kWt) -B * ψ + log(1/(P + ω)) (3)
where at the time t, Q t is the flow rate through the active pump, and kW t is the respective electricity consumption; B is the achievement of lower/upper restrictions of the water level in the tank. These lower/upper values are defined by specialists in the system and in case of not achievement, B = 1 in case of overflow and B = abs(level_of _the_tank t -boundary_condition) ∈ (0, 1] in case of (near) shortage, being ψ = 10, otherwise B = 0. Also, B has an exception, being -1 strictly for the timestep when the tank level reaches the water quality condition. P is a penalty that increases with accumulated pump run time. The penalty P increases +1 at each timestep of cumulative operating time, and for the Equation 3 it also hold for the action (NOP). In the case of switching to a pump that has already been running throughout the day, ω equals 30 for the respective timestep of the switch, otherwise 1. If no pumps are running, neither -e (-1/kWt) nor e 1/(-Qt/kWt) are considered. Which differentiates the Equation 2 is efficiency regarding the pumps through Q t /kW t , when the Equation 3 directly penalize the electricity consumption through the term e -1/kWt . As a consequence, this leads to a different perception regarding the actions. As the agent tries to maximize these rewards along its trajectory, the result is the emergence of some behavior applying the policy learned through these distinct returns. Thus, by designing two reward functions, we aim to analyze the adequacy of those behaviors regarding the goals established.

Deep Reinforcement Learning

The DQN [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF][START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] combines Q-Learning [START_REF] Watkins | Q-learning[END_REF] with Deep Neural Networks. The state-input can be, for instance, a set of images or continuous values, and the output is an estimation of how good is be in that state s and perform an action a, called Q-value. During the learning process, DQN tries to approximate the optimal Q* for each stateaction pair performing updates through the Bellman equation. This approach achieves higher scalability compared to other methods once that is not necessary to keep a vast search space. Later, Hausknecht and Stone [START_REF] Hausknecht | Deep recurrent qlearning for partially observable mdps[END_REF] introduced long short-term memory (LSTM) in this structure to handle partially observable environments, and van Hasselt and colleagues adopt a Double DQN [START_REF] Van Hasselt | Deep reinforcement learning with double q-learning[END_REF] to tackle the optimistic nature of the original Q-Learning.

Learning Process

Using a simulator of the environment and real-world data of the water consumption at determined time t, the simulator can calculate at timestep t the values of flow Q, pressure H, and electricity consumed kW , as well the tank level at t + 1. The dynamic of this simulator is first to define the state s and then use some strategy to choose an action to be performed. Once this action is applied, a reward is given, and the next state is perceived, constituting a transition T = < state, action, reward, next state >.

During this process, new transitions feed the Experience Replay. The Experience Replay [START_REF] Lin | Self-improving reactive agents based on reinforcement learning, planning and teaching[END_REF] and the Target Network are two techniques applied in [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], to improve the performance of DQN. The former break the correlation of data, and the latter makes the learning process more stable.

Transitions stored in the replay memory consist of a batch. Then, this batch is split into mini-batches and shuffled to break the correlation between the data. Finally, the states of these mini-batches are inputs in the neural network, which aims to approximate Q*(s, a) through the Bellman Equation 4 [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF].

Q * (s, a) = E[R(s, a) + γmax a ′ Q ′ (s ′ , a ′ )], (4)
where an expectation is defined regarding the future returns, discounting it through the factor γ ∈ [0, 1]. The Q-Learning approach establishes a convergence for the optimality, updating the Bellman equation through Equation 5.

Q(s, a) = Q(s, a)+α[R(s, a)+γmax a ′ Q ′ (s ′ , a ′ )-Q(s, a)].
(5) In order to update the Q(s, a), every state-action pair is recorded and updated iteratively in the tabular form of Q-learning [START_REF] Watkins | Q-learning[END_REF]. This approach suffers from a problem called curse of dimensionality [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] as the number of possible states and actions grows. This can be even more complicated when considered continuous values, that must be discretized in some way. For that, DQN combines Qlearning with neural networks as a function approximator with weights θ to estimate the Q-values. This is accomplished by minimizing the loss δ at each time step i, as shown in Equation 6.

δ i (θ i ) = E[R(s, a)+γmax a ′ Q ′ (s ′ , a ′ , θ i-1 )-Q(s, a, θ i )] 2 , ( 6
)
where the weights θ i-1 are those fixed in the target network that in turn, are periodically updated copying weights θ. The frequency that the target network updates can be seen as a hyperparameter, being with the replay memory properties an object of study in the performance of DQN and variants [START_REF] Fedus | Revisiting fundamentals of experience replay[END_REF]. The problem of traditional Q-learning is that it tends to overestimate state-action pairs out of the distribution when exploiting a fixed dataset [START_REF] Fujimoto | Off-policy deep reinforcement learning without exploration[END_REF]. REM mitigates this using an ensemble of models to improve the generalization through the Equation 7.

δ i (θ i ) = E[R(s, a) + γmax a ′ X k α k Q ′ k (s ′ , a ′ , θ k i-1 )- X k α k Q k (s, a, θ k i )] 2 , (7)
where for each mini-batch, α is a set of weights randomly generated such that P k α k = 1. Thus, REM is a convex combination of Q-values, converging for itself [START_REF] Agarwal | An optimistic perspective on offline reinforcement learning[END_REF].

Sample Efficiency

The performance of the family of DQN-based approaches is strongly correlated with sample efficiency. This section describes the strategies adopted to provide richer observation of the current state, improve training performance, and make better use of samples.

State stacking

In the original approach of DQN, n last previous states (frames) are concatenated [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF]. Thus, the input provides a richer observation of the current state, such as the system's dynamic.

Training data scale

The state-input values have different ranges that differ substantially. It is applied normalization in both states and re-ward values for the range [0, 1] using Equation 8. The feature is the value x, and max/min was defined considering historical observations.

x ′ = x -min(x) max(x) -min(x) (8)

Prioritized Experience Replay (PER)

Schaul and colleagues present in [START_REF] Schaul | Prioritized experience replay[END_REF] an improvement regarding the Experience Replay, prioritizing samples more "unexpected". In other words, samples that provide the highest values |δ i | through the Equations 6 are those much to learn from [START_REF] Hessel | Rainbow: Combining improvements in deep reinforcement learning[END_REF]. Then, every transition in the mini-batch is associated with the correspondent magnitude of the loss, such as T = < state, action, reward, next state, |δ| >. Finally, to balance the bias introduced by the prioritization of samples, PER applies Importance Sampling (IS) weights.

Imitation Learning

The imitation learning strategy Safety through Intrinsically Motivated Imitation Learning (SIMIL) present in this work assumes that offline data is available and online data collection is feasible. The underlying idea is to use the offline dataset distribution as a model to constrain the action selection and enhance the sample efficiency while encouraging the policy's convergence to states that lie in high-density regions under the same prior distribution.

The imitation learning strategy works as a follows: given a current state s t and demonstrations D, select the action a mostly applied in the k-most similar states to s t in D. For that, we make use of k-Nearest Neighbors (k-NN), where the parameter k can be chosen such that minimizes the distance min P D d(τ, τ D ), regarding trajectories τ D ∈ D. The objective is to keep new transitions tied to the previously collected data, mitigating overestimation facing unseen state-action pairs. Finally, a reward bonus ρη(s t ) is added to the immediate reward according to the Kernel Density Estimation (KDE) for s t through Equation 9, being ρ the importance factor for the bonus. Thus, we encourage policy convergence to states with high density under prior dataset distribution.

η(s t ) = 1 N N X i=1 K s t -s D i h . (9)
In Equation 9, K(s t ) ≥ 0 is the kernel that estimates the density for the current state s t over the states s D found in the demonstrations. The parameter h is the bandwidth that trade-off the results between balance and variance. In this work, we adopt the k-NN based on Manhattan distance once it can provide suitable metric for real-values without parameter tuning and KDE with a gaussian kernel from Scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF]. The Algorithm 1 summarizes the strategy proposed.

In particular, we reduce the dimensionality of the state's representation for the meaningful features regarding the current status of the WDS and skip some of them for the k-NN queries. That is because the timesteps are strongly correlated, and skipping some of them reduces the computational overhead due to k-NN query. Thus, the state representation used to calculate the reward bonus and perform k-NN queries has the reduced form of ϕ(s t ) = < tank level, water consumption, current time, month >.

6 Policy Evaluation

Experimental Setup

In this work, we aim to evaluate if (1) the proposed imitation learning strategy can generate policies that outperform offline methods baselines; (2) the proposed POMDP can obtain policies that offer a competitive performance relative to that observed in the real world. To this end, we conducted the experiments using the real-world dataset divided into one year for the learning process and one year for the evaluation. Both Offline RL methods and SIMIL use the same amount of data for learning. For accurate comparisons, all samples interact with the simulator for both training and evaluation. This means that the evaluation of the offline dataset is done through interactions with the simulator. We compare the policies BCQ, REM, and SIMIL + REM using 5 models for each reward function due to the stochasticity in the learning process [START_REF] Henderson | Deep reinforcement learning that matters[END_REF].

Results

To analyze the performance, we call the set of policies obtained using the Equations 2 and 3 by Π tively. We show in Figure 1 the min, max, and average cumulative reward along with the episodes using the 5 policies obtained. The results show that SIMIL has lower variance and competitive performance relative to cumulative rewards compared to fully-offline policies. The lower peaks in performance are mainly due to not meeting the tank level safety constraints. The three sub-goals: electricity consumption, distribution of pump usage, and tank level are the counterparts of the policy. Thus, a suitable policy performs with lower electricity consumption/higher efficiency, reduces switches, and distributes the pump operation while respecting the tank level constraints. We show in Figure 2 the performance of policies π * with a better average cumulative reward for Offline RL and SIMIL. Tables 1 and2 present a comparison between the policies using as baseline the real-world statistics for the evaluation data. Table 1 compares the electricity consumption for Π regarding real-world operation while Table 2 shows the action distribution for π * . The results show that SIMIL policies achieve competitive results with real-world operations considering electricity consumption. Finally, generally, the policies presented an operation in the safety range of tank levels.

Conclusions

This work presents Safety through Intrinsically Motivated Imitation Learning (SIMIL), an imitation learning strategy using density-based action selection and intrinsic motivation to constrain policies to expert demonstrations. Our contribution lies in the idea that SIMIL, while retrieving expert demonstrations behavior, also allows the possibility of extrapolating it in favor of states that lies in high-density regions. That could represent a means to deploy safe deep RL approaches in real-world applications. Finally, the results show that SIMIL can lead to policies that could even outperform fully-offline methods. We present a real-world problem called pumping scheduling for water distribution utilities as an evaluation scenario. The contributions of this work extend to this domain. The proposed reward functions lead to policies that satisfy the safety constraints, protect the assets and lead to electricity savings. The authors hope that this representation of the pumping scheduling problem can help other researchers in different WDS settings.

Figure 1 :Offline RL π * 1 SIMIL + REM π * 1 Offline RL π * 2 SIMILFigure 2 :

 11122 Figure 1: (a) and (c) are respectively the min, max, and average cumulative reward for the set of policies Π 1 and Π 2 . (b) and (d) shows the average of the cumulative rewards along with the episodes for Π 1 and Π 2 . Offline RL π * 1

  Input : set of Q-Networks with weights θ Q , set of Target Q'-Networks with weights θ Q ′ ← θ Q , replay memory D ′ ,demonstrations D, Select action a t using k-NN(s t ) in D Play (s t , a t ), observe the reward r t and the next state s ′ Calculate η(s t ), sum it to a final reward r ′ t = r t + ρη(s t )

	Algorithme 1 : Safety through Intrinsically Motivated Imitation Learning (SIMIL)
	frequency which update target net λ, importance factor ρ; Output : Policy π 1 for t ∈ {1, 2, ...} do 2 Sample state s t
	3
	4
	t
	5
	6 7 8 end Store transition (s t , a t , r ′ t , s ′ t ) into D ′ s t ← s ′ t 9 for t ∈ {1, 2, ...} do 10 Sample a mini-batch of n transitions from D ′ 11 Calculate loss δ(θ Q )
	12 13 14 15 16 end Perform a gradient descent step to update θ Q if t mod λ = 0 then Update the set of weights θ Q ′ ← θ Q end

Table 1 :

 1 [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] and Π 2 respec-Average electricity consumption (%) ± standard deviation compared to real-world operation.

	Policy	Electricity Consumption (kW)
	REM Π 1 SIMIL + REM Π 1 BCQ Π 1 REM Π 2 SIMIL + REM Π 2 BCQ Π 2	-1.11 ± 9.78 -4.05 ± 1.97 -3.54 ± 2.71 4.08 ± 7.93 -3.33 ± 5.77 -1.40 ± 3.33
	Policy	NOP NP1	NP2	NP3	NP4
	Real-world REM π * 1 SIMIL + REM π * 1 BCQ π * 1 REM π * 2 SIMIL + REM π * 2 BCQ π * 2	30.47 8.30 11.38 4.93 17.05 0.17 22.87 17.79 8.13 43.42 8.31 0.87 82.82 0.0 9.50 28.54 5.29 48.95 51.09 0.12 32.64 25.85 0.04 41.47 0.0 28.08 3.12 36.04 4.89 27.87 37.11 37.48 0.06 25.35 0.0

Table 2 :

 2 Action distribution (%)

The dataset has been provided by the IoT.H2O project (IC4WATER JPI funding)

Acknowledgments

We would like to thank Harald Roclawski (TU Kaiserslautern), Aloysio P. M. Saliba (UFMG), Benjamin Dewals (ULiège), Anika Theis (TU Kaiserslautern), Thomas Pirard (ULiège), and Thomas Krätzig (Dr. Kraetzig) for their comments and suggestions regarding this work. The authors acknowledge FAPEMIG, Federal Ministry of Education and Research of Germany, Agence Nationale de la Recherce de France and Fonds de la Recherce Scientifique Belge, for funding this research by the project IoT.H2O (ANR-18-IC4W-0003) on the IC4Water JPI call.