Teresa S Barata

Sunil Shaunak

Ian Teo

Mire Zloh
email: mire.zloh@pharmacy.ac.uk

Steve Brocchini
email: steve.brocchini@pharmacy.ac.uk

Prof Steve Brocchini

Structural studies of biologically active glycosylated PAMAM dendrimers

Keywords:

The partial modification of carboxylic acid terminated polyamidoamine (PAMAM) dendrimers with glucosamine has been reported to give dendrimer glucosamine conjugates novel immuno-modulatory and anti-angiogenic properties. Experimental analysis of these glycosylated dendrimers showed that, on average, eight glucosamine molecules were covalently bound to each dendrimer. In order to better understand the surface loading and distribution of these glucosamine molecules, molecular reactivity was determined by evaluation of electronic properties using frontier molecular orbital theory (FMOT) and molecular dynamics simulations. It was shown that the surface loading and distribution of the zero length amide bond conjugated glucosamine molecules was determined by both electronic effects and by the different dynamic conformations adopted by the modified dendrimer during the incremental addition of glucosamine. Importantly, the structural features and the dynamic behavior of the partially glycosylated generation 3.5 PAMAM dendrimer showed that its flexibility and its polarity changed with the incremental addition of glucosamine. These peripheral glucosamine molecules remained available on the dendrimer"s surface for interaction with the biological target.

Introduction

The partial modification of carboxylic acid terminated polyamidoamine (PAMAM) dendrimers with glucosamine has been reported to give dendrimer glucosamine conjugates novel immuno-modulatory and anti-angiogenic properties [1]. Dendrimers are a class of polymeric compounds that can be distinguished from conventional linear polymers by their highly branched and symmetrical architecture [START_REF] Vögtle | Dendrimer Chemistry[END_REF]. Dendrimers are synthesized using a sequence of reactions, each time the sequence is conducted, the dendrimer is said to increase by one generation. The number of end groups double and the molecular weight of the dendrimer increases each time the sequence of reactions is conducted. The generation of a dendrimer is representative of both its size (i.e., diameter in angstroms) and its molecular weight [START_REF] Menjoge | [END_REF].Typically, PAMAM dendrimers are available in whole number generations (amine terminated). If the surface end groups are capped, the dendrimer is said to have increased by a half-generation. Hence for an amine terminated dendrimer of generation 3, capping it to have carboxylic acid terminated end groups will give a generation 3.5 dendrimer. The number of end groups does not change by simple capping of the existing endgroups. The PAMAM dendrimer used was a generation 3.5 with 64 carboxylic acid end groups. These were prepared by carbodiimide based coupling to form a stable zero length amide bond between the carboxylic acid terminated dendrimer and the aminosaccharide (Scheme 1). The divergent synthesis used does not allow for predetermined positioning of the glucosamine molecules on the dendrimer"s surface.

<Scheme 1>

Experimental analysis of the glycosylated dendrimers showed that, on average, eight glucosamine molecules were covalently bound to each dendrimer. In order to better understand the surface distribution of these glucosamine molecules, and its contribution to the biological properties of the dendrimer glucosamine molecule, we have developed a method to generate 3D models of these dendrimers in silico [4]. In this paper, these structures have been used to computationally study the loading and the distribution of glucosamine on the end groups of the PAMAM dendrimer, and to define the dynamic behavior of the whole molecule.

Molecular reactivity can be estimated by evaluation of electronic properties. The frontier molecular orbital theory (FMOT) focuses on valence electrons and the interaction between the highest occupied molecular orbital (HOMO) and the lowest unoccupied molecular orbital (LUMO) in the molecules involved in both the formation and the breaking of bonds [5].

Chemical reactions are favored at atoms in molecules where the maximum overlap between the HOMO of a reactant molecule and the LUMO of another reactant molecule leads to bond formation [6]. The FMOT has been widely used for the study of chemical reactions; e.g., Diels Alder reactions [7] and metal substitutions [8]. For dendrimeric structures, this theory has been used to study energy flow and the intensity of light emission from a nanostar dendrimer [9].

Although the electronic effects can have a determining effect on reactivity, molecular conformational and dynamic behaviors can also play important roles when macromolecules such as dendrimers are considered. Molecular dynamics studies of PAMAM dendrimers have already helped to elucidate the structural features and behavior of these molecules. A study varying the percentage acylation of a generation 5 PAMAM dendrimer showed that their overall shape was not dependent upon the percentage of acyl groups on their surface because intramolecular hydrogen bonds form and break during the simulation [10].

In this paper, frontier molecular orbital theory coupled with molecular dynamics simulations were used to understand the structural features of the experimentally determined loading and distribution of covalently bound glucosamine, and its impact on the surface properties of the PAMAM dendrimer.

Experimental section

Molecular dynamics simulation of saccharide modified dendrimers

The dendrimer structures were generated with X-PLOR using a recently described method and saved in pdb format [4]. VMD [11] was used to convert the structures into the mol2 file format and then imported into Maestro [12]. The dendrimer system was prepared for molecular dynamics simulations using Desmond with explicit solvent [13]. The dendrimer glucosamine system was built using the SPC solvation model and the size of the box was determined automatically by creating a 10 Å buffer zone around the dendrimer. The molecular dynamics simulation, structure minimization and relaxation steps were performed for 4.8 ns at 300K and 1.03 bar. Snapshot structures were recorded every 4.8 ps.

Trajectory analysis of the molecular dynamics simulation

The trajectory from Desmond output in ".cms" format was loaded into VMD where water and ions were removed and a new trajectory was exported as a mol2 file. These files were loaded in VegaZZ 2.2.0.54 [14] and the trajectory analysis tool used to determine gyration radius, polar surface area (PSA), surface area and RMSD. The resulting values were imported into Microsoft Excel and plotted. Both the distance from the core to the surface glucosamine and the number of hydrogen bonds were determined using the Maestro measuring toolkit.

Accelerys Discovery Studio 2.5 (http://accelrys.com/products/discovery-studio/) was used to generate the interpolated charge surfaces.

Determination of electronic properties by semi-empirical methods

The electronic properties of the dendrimers were determined by semi-empirical methods using Mopac 2009 9.303W. Input files for Mopac were created using VegaZZ 2.2.0.54 interface, choosing PM6 and the key words 1SFC, MMOK, GRAPHF and VECTORS. Due to computational restrictions, the molecules submitted for these calculations were a one quarter "slice" of the representative conformations of the generation 3.5 PAMAM dendrimer and dendrimer glucosamine conjugate as generated by molecular dynamics simulations. These calculations were conducted on a quarter of the generation 3.5 PAMAM dendrimer:-(a) without any modifications (qdend); (b) modified with one glucosamine (qd1s); (c) modified with two glucosamine molecules (qd2s); and (d) modified with three glucosamine molecules (qd3s). In each case, six representative conformations of each dendrimer structure were subjected to calculations. They were used to approximate electronic properties that reflected the electronic properties of the whole PAMAM dendrimer with different loadings of glucosamine.

Results and discussion

The three dimensional structures of five generation 3.5 PAMAM dendrimers with different loadings of glucosamine were generated and the resulting structures submitted to molecular dynamics simulations with 1000 structures saved in each trajectory. These were then clustered to select a significant number of representative structures to study the loading and distribution of glucosamine. Additionally, the whole trajectory was analyzed for a better understanding of the dynamic behavior of these dendrimers with and without conjugated glucosamine.

Loading and distribution of the glucosamine molecules conjugated to a generation 3.5

PAMAM dendrimer

The generation 3.5 PAMAM dendrimer displays 64 carboxylic end groups that are, in theory, all available for covalent modification with glucosamine. NMR studies of the glucosamine PAMAM dendrimer conjugates have shown that there are, on average, eight glucosamine molecules per dendrimer [1] & unpublished observations].

Frontier Molecular Orbital Theory (FMOT) postulates that the interaction between the HOMO and the LUMO of molecules undergoing a reaction can provide a good approximation of reactivity [5]. A reaction will occur with increasing and often maximal overlap of the HOMO and LUMO orbitals. For this study, the LUMO of the glucosamine molecule and the HOMO of the dendrimer with different patterns of glycosylation was determined. The difference in the energy between the dendrimer HOMO and glucosamine LUMO was then determined and analyzed.

The large size of the dendrimer (894 heavy atoms) meant that only partial segments of the structure could be readily used to determine the HOMO-LUMO electronic properties, and that semi-empirical methods would be applicable. The HOMO of the dendrimer was located on one of the terminal carboxylic acid groups. This would allow an interaction with the LUMO in the glucosamine, which was found to be located on the amine group (Fig. 1).

<Figure 1>

These electronic property calculations were applied to increasing sizes of dendrimer branches with and without glucosamine (Fig. 2). This approach enabled confirmation that the position of the HOMO was in the terminal carboxylic groups where the interaction with the LUMO of glucosamine was expected to take place. An important new observation was that when the first glucosamine molecule was covalently attached to the dendrimer, the HOMO was always located on the opposite branch to the one with the zero length amide bond between the dendrimer and the glucosamine.

<Figure 2>

The gap between the HOMO of the dendrimer and the LUMO of the glucosamine decreased as the size of the dendrimer molecule increased. According to the FMOT, the smaller the gap between the HOMO and LUMO the more likely is the reaction to take place. By analogy, it was expected that this gap would be even smaller if the whole dendrimer was considered.

Computational limitations meant that a quarter of the dendrimer structure (i.e., a "slice" displaying 16 terminal carboxylic groups) was selected for further detailed analysis.

Dendrimers are hyperbranched and flexible molecules with many rotatable bonds. Six representative conformations from a quarter section of the dendrimer without glucosamine, and the same number of conformations of the dendrimer with 1, 2 and 3 glucosamine molecules were considered in relation to the LUMO of the glucosamine molecule (-4.61 eV).

The averaged energy values of the HOMO for each dendrimer can be found in the table below (Table 1).

<Table 1>

It was found that the HOMO d /LUMO s gap increased as additional glucosamine molecules were covalently bound to the dendrimer. According to FMOT, the increase in the gap indicated that each addition of glucosamine decreased the overlap of the HOMO and the LUMO. This thereby decreased the probability that additional conjugation reactions would occur with the adjacent surface carboxylic acid groups of the dendrimer.

A detailed analysis of the dendrimer segments showed that the energies of both the HOMO and HOMO-1, with three glucosamine molecules attached, was almost double the LUMO energy value of the glucosamine molecule. This would make the addition of a fourth sugar to the dendrimer section under study very unlikely. Therefore, the theoretical maximum loading of a generation 3.5 PAMAM dendrimer with glucosamine was found to be 12. However, electronic effects on their own, are insufficient to completely understand the loading and distribution of the glucosamine molecules on the surface of these dendrimers.

Observation of the position of the HOMO, and defining both the location and availability of the terminal group, was necessary to better understand the loading and distribution of the glucosamine molecules (Fig. 3 and Table 2).

<Figure 3>

Both the HOMO and HOMO-1 positioning and availability were inspected for each structure. Fig. 3 shows an example of the HOMO position for each of the quarter segments of the dendrimers that were studied. The first structure is representative of the six conformations for the segment of the dendrimer with no glucosamine attached. The HOMO/HOMO-1 orbitals are located at the end groups of the branches of the dendrimer and they are always available for the conjugation reaction.

The second structure corresponds to one of the six conformations with one glucosamine conjugated to the surface of the dendrimer. It can also be considered representative of all six conformations in which the HOMO is always available. For these molecules, the HOMO, though not always located in the extremities or at the end groups of the branch, is always at a considerable distance from the conjugated glucosamine molecule. In this case, the HOMO-1 did not follow the trend described in that it was only available in half of the structures studied (Table 2). The third structure (Fig. 4) corresponds to one of the six conformations of the quarter segment of the dendrimer with two glucosamine molecules conjugated to its surface. This structure was also representative of what happens with these structures when the HOMO is located in the interior of the dendrimer"s branches. This affects its availability for reaction with glucosamine. The last structure shown (Fig. 4D) corresponds to one of the six conformations of the quarter of the dendrimer with three glucosamine molecules conjugated to its surface. In this case, the HOMO is never available on the surface and it appears on an end group at the extremity of the branch. Furthermore, it points inwards towards the core of the dendrimer. As a result, the adjacent branch from the whole dendrimer sterically interferes with the end group. The availability of the HOMO and HOMO-1 for all the representative structures under study are summarized in Table 2.

<Table 2>

The addition of the first glucosamine is favorable because the HOMOs of the carboxylic groups are available. After one saccharide has covalently bound to one of the end groups, the HOMO is again located on a peripheral carboxylic acid group making the addition of a second glucosamine favorable. When two sugars are bound, in a quarter segment of the dendrimer, it becomes much less likely that the addition of third glucosamine will take place because there is only a 50:50 chance that the HOMO will be available for interaction with a carboxylic acid end group. The addition of a fourth glucosamine molecule is very unlikely to occur within the same quarter segment of the dendrimer.

Taking both electronic and steric effects into account, the conjugation of 4 to 8 glucosamine molecules onto the available 64 carboxylic acid end groups that are present on the generation 3.5 PAMAM dendrimer is the most favorable energetic and steric outcome. The conjugation of additional glucosamine molecules requires overcoming increasingly higher energy barriers with available positions for conjugation being available for only half of the time. Additional conjugation reactions of glucosamine beyond twelve molecules on the whole dendrimer are very unlikely because the energy gap and the overlap between the LUMO of glucosamine and the HOMO of the newly formed conjugate is large, and the carboxylic acid end group is not available. These molecular modeling based observations are consistent with the experimental data that an average of 8 glucosamine molecules were conjugated to the surface of a generation 3.5 PAMAM dendrimer.

Molecular dynamics simulation analysis of glycosylated PAMAM dendrimers

To better understand the dynamic behavior of these dendrimers, molecular dynamics simulations were carried out for 4.8 ns. Molecular properties including the surface area, polar surface area, gyration radius and RMSD were estimated and plotted as a function of time using the VegaZZ trajectory analysis tool (Fig. 4).

<Figure 4>

The surface area graph (Fig. 4B) shows that there is an initial decrease in the surface area values. This is most likely to be the result of the starting structure being generated in vacuum and the use of nOe constraints to stretch the initial structure. After an initial decrease, the area oscillated around the average value of 13965 Å 2 . The polarity of the surface (Fig. 4C) followed a similar pattern along the trajectory.

The RMSD analysis, as a measure of flexibility and folding [15][16][17], was conducted for all of the PAMAM dendrimer structures. A plateau was seen after the first 1400 ps (Fig. 4D). This means that the molecule explored the conformational space with similar differences when compared to the initial structure set as reference for the RMSD calculations. However, further changes in the explored conformational space were observed as indicated by the increase of the RMSD values towards the end of the simulation. Although the RMSD reached a plateau after 1.4 ns, RMSD values of approximately 11 Å indicated a flexible structure. This plateau was consistent with the profile that was obtained for the gyration radius where, after only 1400 ps, limited fluctuations occurred. These fluctuations indicated that the overall shape of the structure was not undergoing major changes during the simulation period.

Closer inspection of the overlay of representative structures also indicated that the core and inner branching units had little mobility with most of the RMSD changes occurring because of movement of the outer sphere of the dendrimer (Fig. 5). The three superimposed structures, with overlapped dendrimer cores, at three different time points indicated that the first branching generation of the dendrimer did not change during the simulation. From the second generation onwards, and all the way to the end groups, the structures showed varying degrees of flexibility. The greatest flexibility was observed in the last 1.5 generations of the dendrimer. This was consistent with the observations described above.

<Figure 5>

To examine the influence of the conjugation of glucosamine on the conformation and the molecular properties of these surface modified PAMAM dendrimers, molecular dynamics simulations were performed with three different loadings of glucosamine of the generation 3.5 PAMAM dendrimer. The entire dendrimer was modified with seven, eight or nine glucosamine molecules and their molecular properties determined and compared as shown in Fig. 6.

<Figure 6>

These three glycosylated dendrimers displayed very similar gyration radius values.

Throughout the simulation, the dendrimer with nine glucosamine molecules was the most compact. For the surface area values, the three glycosylated dendrimers displayed profiles with similar values with the nine glucosamine dendrimer molecule having lower values than the eight glucosamine molecule for a significant period of time. This was consistent with the nine glucosamine dendrimer having a more compact structure, an observation that was not expected because this molecule had the highest number of atoms.

The surface polarity of these glycosylated dendrimers was proportional to the number of conjugated glucosamines. Finally, for the RMSD values, the dendrimer with eight glucosamines attached was clearly different. This suggested that this molecule explored a larger conformational space because of its even distribution of surface glucosamine molecules. However with longer simulation times, these properties reached a plateu and converged. The observed difference in behavior of the dendrimer with eight glucosamines on its surface from the other two dendrimer glucosamine conjugates was interesting because it was the only molecule with a even distribution of surface glucosamine molecules. It is the most flexible of the three molecules. This could be an indication that the uneven distribution of glucosamine molecules leads to a rigid dendrimeric structure with smaller conformational space.

For all of these glycosylated dendrimers, their molecular properties converged and stabilised after 4 ns. Therefore, a single dendrimer glucosamine could be used as a representative structure for all of these molecules with different loadings of glucosamine. Similarly, molecular dynamics studies of heterogeneously modified PAMAM dendrimers showed that the distribution of their surface modifying groups was dependent upon the conformation of the core and its branching monomers, and that its overall shape was not altered by the surface modifications [18]. Therefore, based on the computational HOMO-LUMO reactivity studies and the analytical studies of dendrimer glucosamine conjugates, the generation 3.5 PAMAM dendrimer with eight glucosamine molecules conjugated via a zero length amide bond was found to be a good representative structure for further studies and analysis.

Others have also studied the effects of terminal group modifications on the solution properties of dendrimers using molecular dynamics. For example, the flexibility and shape of glycodendrimers [19] and poly(L-lysine) dendrimers [20] were studied. These properties were also studied using different monomers, generations, and different surface monomers [19,20].

Our strategy was different in that we compared and contrasted biologically active (i.e., generation 3.5 PAMAM dendrimer modified with eight glucosamine molecules) and biologically inactive (i.e. generation 3.5 PAMAM dendrimer and a generation 3.5 PAMAM dendrimer modified with one glucosamine molecule) by analysing the flexibility and the shape of the molecule. Gyration radius, as a measure of molecular shape indicated that the presence of eight glucosamine molecules led to a less compact structure (Fig. 7A).

<Figure 7>

The RMSD values are a measure of flexibility in that any value greater than 4 Å is considered to be a different conformation. The dendrimer that has no glucosamine molecules conjugated to its end groups reaches a plateau rapidly during the 2 ns simulation (Fig. 7B). The units on the outer surface continued to change their conformation up to the end of the trajectory when it started to explore a different conformational space. The dendrimer with eight surface glucosamines presented several plateaus in its profile (before 2 ns, at 2.5 ns, and after 4 ns). This means that it passed through several significantly different conformations. With time, this dendrimer glucosamine explored an increasingly larger conformational space (Fig. 7B).

Considering the profiles of these biologically active and inactive molecules led to the conclusion that the dendrimer with eight surface glucosamine molecules was the most flexible molecule because it explored the largest conformational space. This indicated that flexibility was an important property for the biological properties of these molecules.

We have also studied the number of intermolecular hydrogen bonds that the dendrimer with eight glucosamine molecules can form with the interior of the dendrimer. In the whole trajectory, it was found that: (a) in 26% of the conformations, one hydrogen bond was observed within the branching units; (b) in 8% of the conformations, two hydrogen bonds were observed within the branching units; (c) in 1.7% of the conformations, three hydrogen bonds were observed within the branching units; and (d) in only one structure were four hydrogen bonds observed within the branching units (Fig. 8). This raised the possibility that the branches bearing glucosamine molecules could be folding inwards towards the center of the dendrimer. Such inward folding would lead to a diminished capacity of the glucosamine moieties to be available for intermolecular interactions. Visual inspection of the structures was therefore performed and the glucosamine molecules found to be mostly outward facing and accessible (Fig. 8).

<Figure 8>

Charge and its surface distribution were also studied. Interpolated charge surfaces were generated for the biologically inactive generation 3.5 PAMAM dendrimer and for the biologically active generation 3.5 PAMAM dendrimer with eight surface glucosamine molecules. The surfaces studied revealed a positively charged region throughout the branching dendrimer, and at the core of the dendrimer, where the protonated amine groups were located (Fig. 9). The negatively charged areas were found on the surface where the deprotonated carboxylic groups were present. The main differences between the PAMAM dendrimer and the glycosylated PAMAM dendrimer were seen on the surface of the molecule.

The presence of the eight glucosamine molecules reduced the negative charge on the surface (Fig. 9B). This allowed for a more relaxed structure, a higher surface area, and a more balanced overall charge distribution. This could be making an important contribution to the biological activity of the molecule. <Figure 9>

Conclusions

These electronic studies describe the relative roles, contribution and importance of both the peripheral carboxylic acid end groups and of the surface loading and distribution of glucosamine molecules on generation 3.5 PAMAM dendrimers to their observed biological activity. The structural heterogeneity of the molecule was a consequence of both electronic effects and of the different conformations that the molecule could adopt. Taken together, these studies showed that the dynamic flexibility of the generation 3.5 PAMAM dendrimer, modified with an average of eight evenly distributed surface glucosamine molecules, was an important property of this biologically active molecule. Additional factors identified as being important were the conformational space explored by the dendrimer glucosamine, its overall surface area, and its surface polarity. The availability of peripheral glucosamine molecules for interaction with the biological target was not impaired by intramolecular hydrogen bonds.

Tables

Table 1 Average values of the HOMO energy determined for each of the six representative conformations of each dendrimer studied, and differences between the HOMO of the quarter segment of the generation 3.5 PAMAM dendrimer and the LUMO of the glucosamine (-4.61 eV). HOMO d -dendrimer orbital; LUMO sglucosamine orbital

+ + + + + - + -+ -+ - 2 + + ---+ + + --+ + 3 + ----- ------

Figure captions

Scheme 1 Synthesis of dendrimer glucosamine Fig. 1 The HOMO orbital from a PAMAM dendrimer portion of a carboxylic acid end group (left) and the LUMO orbital of a glucosamine molecule (right). These electronic properties were calculated using PM6 method implemented in MOPAC

Fig. 2 Fig. 3 Fig. 4 Fig. 5 Fig. 6 Fig. 8 Fig. 9

 2345689 Fig. 2 Top -Increasing branches of PAMAM dendrimer modified with one glucosamine submitted to semiempirical calculations with Mopac to determine the HOMO energy value and its location (arrow). Bottom -HOMO d -LUMO s gap as defined by the HOMO d energy value determined for each dendrimer structure and the LUMO s of a glucosamine molecule (-4.61eV)

Table 2

 2 Availability of the terminal carboxylic groups where the HOMO and HOMO -1 are located for each conformation and molecule studied. + available; -not available for the reaction

	Number of sugars on the quarter of the gen. 3.5 PAMAM	Availability of the HOMO	Availability of the HOMO-1
	dendrimer
	0	+ -+ + + +	+ + + + + +
	1

Acknowledgments

We are grateful for funding from the US National Institutes of Health (NIH) (U01AI075351-02).


Lucie Daubigney

Milica Gašić

Senthilkumar Chandramohan

Matthieu Geist

Olivier Pietquin

Steve Young

Uncertainty management for on-line optimisation of a POMDP-based large-scale spoken dialogue system

Keywords:

The optimization of dialogue policies using reinforcement learning (RL) is now an accepted part of the state of the art in spoken dialogue systems (SDS). Yet, it is still the case that the commonly used training algorithms for SDS require a large number of dialogues and hence most systems still rely on artificial data generated by a user simulator. Optimization is therefore performed off-line before releasing the system to real users. Gaussian Processes (GP) for RL have recently been applied to dialogue systems. One advantage of GP is that they compute an explicit measure of uncertainty in the value function estimates computed during learning. In this paper, a class of novel learning strategies is described which use uncertainty to control exploration on-line. Comparisons between several exploration schemes show that significant improvements to learning speed can be obtained and that rapid and safe online optimisation is possible, even on a complex task.

Introduction

Spoken Dialogue Systems (SDS), and especially task-oriented SDSs, are now very common and are often encountered in everyday life (hotline, call routing etc.). A key requirement when building such a dialogue system is to design a dialogue manager that fulfils the user requests in a way which is robust, efficient and natural. To achieve optimality, dialogue strategies should handle speech and language processing errors robustly, exploiting implicit and explicit confirmation as well as disambiguation sub-dialogues as appropriate. Dialogue management can therefore be seen as a sequential decision making problem where decisions about which dialogue act should be output in a given context (dialogue state) should be taken so as to maximize some optimality criterion. Sequential decision making is often addressed by Reinforcement Learning (RL) [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] which is a statistical machine learning method that has been successfully applied to SDS during the last decade [START_REF] Singh | Reinforcement learning for spoken dialogue systems[END_REF][START_REF] Levin | A stochastic model of human-machine interaction for learning dialog strategies[END_REF][START_REF] Pietquin | A probabilistic framework for dialog simulation and optimal strategy learning[END_REF][START_REF] Young | The hidden information state model: A practical framework for POMDP-based spoken dialogue management[END_REF]. Optimizing dialogue management by means of RL requires casting the problem into a Markov Decision Process (MDP) [START_REF] Levin | A stochastic model of human-machine interaction for learning dialog strategies[END_REF] or a Partially Observable MDP (POMDP) [START_REF] Young | The hidden information state model: A practical framework for POMDP-based spoken dialogue management[END_REF]. In the latter case, decisions are not taken according to a single inferred context but according to a distribution over all possible contexts. This makes The work presented here has been done during the CLASSiC project (Grant No. 216594, www.classic-project.org) funded by the European Commission's 7th Framework Programme (FP7).

the learning process more robust to speech recognition and understanding errors but at the cost of significantly more complex dialogue policies.

Although the (PO)MDP framework is appealing for SDS optimisation, standard RL algorithms need large amounts of data to converge to an optimal strategy. Dataset expansion by means of user simulation has thus been intensively studied during the last 15 years [START_REF] Eckert | User modeling for spoken dialogue system evaluation[END_REF][START_REF] Pietquin | A probabilistic framework for dialog simulation and optimal strategy learning[END_REF][START_REF] Schatzmann | A survey of statistical user simulation techniques for reinforcementlearning of dialogue management strategies[END_REF]. Yet, user simulation induces a modeling bias which is hard to predict [START_REF] Schatzmann | Effects of the user model on simulation-based learning of dialogue strategies[END_REF] and can lead to suboptimal strategies after learning. In the meantime, RL research has made progress. In particular, batch algorithms that can learn from small amounts of fixed data have recently been shown to be applicable to SDS [START_REF] Pietquin | Sample-Efficient Batch Reinforcement Learning for Dialogue Management Optimization[END_REF]. Furthermore, the examples used for learning, whether simulated or real, may not represent all types of users. The ability of a system to adapt during actual use would greatly mitigate this. Recently, online learning with an efficient RL algorithm, namely GP-SARSA [START_REF] Engel | Reinforcement Learning with Gaussian Processes[END_REF], has been proposed to optimize dialogue management strategies [START_REF] Gašić | Gaussian processes for fast policy optimisation of POMDP-based dialogue managers[END_REF]. GP-SARSA makes use of Gaussian processes [START_REF] Rasmussen | Gaussian Processes for Machine Learning[END_REF] to approximate the state-action value function from which an optimal policy can be easily derived.

Several approaches to on-line policy optimisation can be envisioned depending on the way the policy is modified during learning. If the initial policy is modified after each interaction with a user, the approach is called on-policy. Whereas when a policy is improved by using interactions collected with another policy, the approach is called off-policy. While off-policy learning offers the possibility to use a non-optimal but acceptable policy during learning, it is not obvious how to identify the best moment to disable exploration and switch to the learnt policy. On the other hand, on-policy learning (as in [START_REF] Gašić | Gaussian processes for fast policy optimisation of POMDP-based dialogue managers[END_REF]) allows the policy to be updated immediately after each interaction but requires the user to suffer changes made to the policy during exploration, even if the changes are undesirable. Ideally, therefore, learning should be as quick as possible to minimise unacceptable changes to the policy affecting the users. However, this then raises the well-known exploration vs exploitation dilemma: should the system exploit its current knowledge and stop learning with the risk of being suboptimal or should it continue exploring different possible strategies to learn a better one with the risk of acting dangerously.

In this paper, we propose to use the uncertainty information relating to intermediate value estimates (namely the Qfunction) computed by the Gaussian Process for RL method to build a sample-efficient online learning algorithm. A key ad-vantage of the method is that exploring actions are selected not only because they are uncertain but also because they are expected not to be harmful. The results are compared with [START_REF] Gašić | Gaussian processes for fast policy optimisation of POMDP-based dialogue managers[END_REF].

The rest of the paper is organized as follows. In Section 2, the problem of casting SDS optimization into the POMDP framework is presented. The following section then proposes several exploration schemes based on the uncertainty of the value estimates. In Section 4 some results are presented using an operational dialogue system for tourist information. Finally, conclusions are drawn and perspectives offered.

Dialogue Management as a POMDP

Dialogue management (DM) can be seen as a sequential decision making problem. The decision maker is the dialogue manager. From user acts (observations), it should choose and perform a system act (actions) in order to satisfy the user in an efficient and natural way. This satisfaction is quantified by a reward provided at the end of a dialogue and computed as a mixture of objective measures (such as the task completion, the dialogue duration etc.). Framed like this, DM can be cast as a Partially Observable Markov Decision Process (POMDP) where each possible history is represented by a state and the decision is taken according to the distribution over all states. In practice, the exact implementation of a POMDP is intractable but there are efficient methods for approximating the state. In particular, in the hidden information state paradigm [START_REF] Young | The hidden information state model: A practical framework for POMDP-based spoken dialogue management[END_REF], the state distribution is reduced to a summary space which has the form of a continuous state MDP. In what follows, the term state refers to a state of this MDP.

DM decisions are defined by a policy π associating an action to each state. Its quality is quantified by a so-called Qfunction that gives the expected cumulative reward for starting with a given state-action pair and then following the policy π:

Q π (s, a) = E h X i≥0 γ i ri ˛s0 = s, a0 = a, π i ,
γ being a discount factor, (s, a) the state-action pair and (ri) i≥0 the set of obtained rewards. The optimal policy (π * ) is greedy wrt to its

Q-function (Q * (s, a)): π * (s) = arg max a Q * (s, a).
The problem of finding an optimal policy therefore reduces to learning the optimal Q-function. The Q-function not only allows two policies to be compared, but it also allows the possible actions in any given state to be compared under a fixed policy. Usually, the state-action space is too large to allow an exact computation of the Q-function and approximation is mandatory.

Here we are interested in learning an optimal control policy while interacting with the user i.e. we require on-line and onpolicy learning. There are usually two steps: estimating the Qfunction of the followed policy (with SARSA [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] for example) and choosing actions according to this estimated Q-function (the control part).

Given an appropriate Q-function approximation algorithm, the problem of choosing actions according to the current Qfunction estimate requires a choice between exploration and exploitation. At each interaction, the RL agent should choose between acting according to its current (imperfect) representation of the world (here the estimated Q-function) and performing some exploration action which although suboptimal according to the current estimate might in fact lead to a higher reward. A classical but crude scheme is the use of an ǫ-greedy policy whereby a greedy action is taken wrt to the estimated Qfunction (arg max a Q(s, a)) with probability (w.p.) 1-ǫ, and a random one w.p. ǫ. If learning from interactions with real users is envisioned, the choice between exploration and exploitation is crucial. First, learning should be fast: the DM should improve quickly and be able to adapt rapidly to users. Second, learning should be safe: the DM should not choose repetitively bad actions. This suggests that choosing actions purely randomly as done by the ǫ-greedy policy may not be wise.

Gaussian Processes and Exploration

The underlying idea of this paper is that if one has access to some uncertainty information about the estimated Q-function, then it should provide useful information for handling the choice between exploration and exploitation. For example, assume that two actions are possible for a given state, one (say a1) having a higher estimated Q-value than the other (a2). A greedy agent would choose a1. But assume also that confidence intervals for these estimates are available. If the uncertainty about a2 is high, this action may in fact be better than a1, and so it may be worth trying. On the other hand, if the confidence interval is small, one should choose a1. This is an instantiation of active learning where the utility function uses uncertainty information [START_REF] Deisenroth | Gaussian Process Dynamic Programming[END_REF]. In this paper, we propose exploration schemes which make this choice automatically.

In order to provide the required uncertainty information, we use an extension of GP-based temporal difference learning called GP-SARSA [START_REF] Engel | Reinforcement Learning with Gaussian Processes[END_REF]. The use of GP-SARSA for fast learning in spoken dialogue systems was first described in [START_REF] Gašić | Gaussian processes for fast policy optimisation of POMDP-based dialogue managers[END_REF] and the same system is used here. The underlying principle of GP-SARSA is to model the state-action value function as a Gaussian Process (that is a set of jointly Gaussian random variables, these random variables being the values of each stateaction pair). A generative model linking rewards to values via the sampled Bellman evaluation operator and an additive noise is set: ri = Q(si, ai)-γ Q(si+1, ai+1)+ni, ni being the additive noise. The Gaussian distribution of a state-action pair value conditioned on past observed rewards is computed by performing Bayesian inference and the value of this state-action pair is estimated as the mean of this Gaussian distribution. The associated variance quantifies the uncertainty. As in any Bayesian method, a prior must be defined. For GP-SARSA, this is done through the choice of a kernel function which defines the prior correlations of Q-function values.

The GP-SARSA algorithm provides at each step i an estimate Qi(s, a) of the Q-function as well as an associated standard deviation σi(s, a). The ǫ-greedy policy does not use this uncertainty information: the greedy action ai+1 = arg max a Qi(si+1, a) is chosen w.p. 1ǫ and a (uniform) random action otherwise. The obtained transition is then used to learn the new estimate Qi+1 (and σi+1). Under some conditions, notably a decaying exploration factor, it converges to the optimal policy. However, it can be rather slow and unsafe during the exploration stage.

In [START_REF] Gašić | Gaussian processes for fast policy optimisation of POMDP-based dialogue managers[END_REF] an informed exploration policy is introduced and this is used here as a baseline. It chooses the greedy action (wrt to the estimated Q-function) ai+1 = arg max a Qi(si+1, a) w.p. 1ǫ and another greedy action (wrt to the computed variance) ai+1 = arg max a

Therefore, the underlying idea is to act greedily wrt the estimated state-action value function, and to occasionally choose an exploratory action. By choosing the less certain action, this scheme actually chooses the action which provides more information. Moreover, the exploration parameter ǫ is decayed to zero as learning progresses, so this scheme tends to the greedy policy. This improves over the classical ǫ-greedy policy but with the possible drawback that the exploration stage does not take the estimated value into account. Therefore, an action with a very low estimated value (which is possibly a "dangerous" action) can be chosen repeatedly, as long as it has a higher uncertainty than other actions (even if the difference is slight). To avoid this drawback, we propose to use of a different form of exploration scheme.

The confident-greedy policy consists of acting greedily according to the upper bound of an estimated confidence interval [START_REF] Kaelbling | Learning in embedded systems[END_REF]. For a tabular representation (for which the confidence interval width is proportional to

1 √ n(s,a)
, where n(s, a)

is the number of visits to the considered state-action pair), some PAC (Probably Approximately Correct) guarantees can be provided [START_REF] Strehl | An Analysis of Model-Based Interval Estimation for Markov Decision Processes[END_REF]. In the case of continuous state spaces (which occur in SDS optimization), the state-action pairs are uncountable, this approach does not hold. Here, standard deviation is provided by a Bayesian method, the prior is given and the posterior is computed. The distribution being Gaussian by assumption, the confidence interval width is proportional to the estimated standard deviation (which is actually true for any distribution, according to the Bienaymé-Tchebychev concentration inequality). Let α be a free positive parameter. We define the confidentgreedy policy as:

ai+1 = arg max a ( Qi(si+1, a) + ασi(si+1, a)) (2)
This strategy favors less certain actions if they correspond to the upper bound of the confidence interval. On the other hand, if for a given state all standard deviations are equal, than the agent will act greedily wrt Qi. Notice also that the variance provided by GP-SARSA decreases as the prediction accuracy of rewards improves and as actions are more and more explored. A similar strategy has been shown to be efficient in model-based reinforcement learning using Gaussian Processes [START_REF] Deisenroth | Gaussian Process Dynamic Programming[END_REF].

The second approach we consider is the bonus greedy policy, inspired by [START_REF] Kolter | Near-Bayesian Exploration in Polynomial Time[END_REF]. Using a Bayesian argument, it acts greedily wrt the estimated Q-function plus a bonus, this bonus being proportional to the inverse of the number of visits to the stateaction pair of interest. As

1 √ n(s,a)
is proportional to the standard deviation in a frequentist approach, we interpret 1 n(s,a) as a variance. The proposed bonus-greedy policy therefore uses a variance-based bonus and is defined as:

ai+1 = arg max a ( Qi(si+1, a) + β σ2 i (si+1, a) β0 + σ2 i (si+1, a) ) (3)
where β0 and β are free parameters. This strategy also favors less certain actions and tends to be greedy wrt the estimated state-action value function if variances are close to each other.

Experiments

Experiments have been conducted using the Hidden Information State (HIS) dialogue manager [START_REF] Young | The hidden information state model: A practical framework for POMDP-based spoken dialogue management[END_REF]. The task concerns a tourist information system which assists users to find a venue in Cambridge using up to 12 attributes. The algorithm used for the dialogue management optimisation is GP-SARSA with a polynomial kernel for the summary state space and a Dirac kernel for the action space; it uses an identical set-up for the Q-function approximator as in [START_REF] Gašić | Gaussian processes for fast policy optimisation of POMDP-based dialogue managers[END_REF]. To obtain sufficient dialogues, the dialogue manager interacts at the intention level with a simulated user [START_REF] Schatzmann | Agenda-based user simulation for bootstrapping a pomdp dialogue system[END_REF]. The simulated user consists of a goal and an agenda. The goal ensures that the user simulator exhibits consistent, goal-directed behaviour. The role of the agenda is to elicit the dialogue acts needed to complete the goal. The speech understanding error rate is set to 10% by using an error simulator, which randomly confuses the semantic concepts that occur in user dialogue acts. A positive reward of +20 is given at the end of the dialogue if the DM managed to fulfil the user request and a penalty of -1 is applied per system turn to encourage short dialogues. The efficiency of the different exploration/exploitation schemes was compared with the informed exploration scheme described in Section 3.

The different schemes are first compared using the same approach as in [START_REF] Gašić | Gaussian processes for fast policy optimisation of POMDP-based dialogue managers[END_REF]. In each case, a policy is trained using a specific number of dialogues and the policy is tested (without exploration). The parameters used in the two new exploration schemes were set as follows: α = 12, β = 1000 and β0 = 100. These were chosen empirically to give good results but they are quite easy to tune since their order of magnitude can be deduced from the variance approximation of the Q function. The parameter ǫ0 = 0.1 was set at the start of training in the informed exploration scheme and then decreased during the learning phase according to the number of dialogues experienced.

Figure 1 shows the reward achieved as a function of the number of training dialogues. As can be seen, the confident greedy and bonus greedy algorithms both learn faster initially but eventually converge to similar values. However, the confident greedy scheme is consistently better than the bonus greedy scheme. To compute the average reward, each 100 dialogues, 100 policies were trained and tested in a greedy way 1000 times each.

The above results allow the performance of the trained policies to be assessed but they do not take account of the fact that during learning, exploration may result in some dialogues having very poor rewards. Whilst this is acceptable for off-line training, it may not be acceptable to users during on-line training. To assess the detrimental impact of poor rewards during on-line training, Figure 2 shows the rewards actually obtained during training as a function of the number of dialogues processed. As can be seen, the two newly proposed algorithms yield much better rewards during training. This is because they avoid subjecting the user to bad exploratory actions. A further advantage of the new on-line learning algorithms is that they avoid the need to decide when a trained policy is sufficiently good that exploration can be disabled. Indeed, they allow exploration to be resumed as needed if and when user behaviour changes as indicated by increasing confidence intervals. Thus, these algorithms allow for continuous adaptation during operation.

The above results show that improvements are obtained when the estimated Q values are augmented by uncertainty information when selecting actions. First, when learning online, the asymptotic performance of the new schemes is higher (around 12 v.s 10) because they do not suffer from the poor rewards resulting from random exploration.

Second, the number of samples needed to reach the same quality of policy is smaller (about 150 dialogues compared to 600). This is an advantage when interacting with real users, since the quicker a good policy is learnt the less the user will be annoyed by poor behaviour of the dialogue manager. Furthermore, the ability to learn policies quickly obviate the need for training using a user simulator. This is a significant benefit, since good user simulators are expensive to build.

Conclusion and Perspectives

In this paper, we have proposed two novel exploration schemes to learn on-line and on-policy an optimal dialogue management strategy using reinforcement learning. The key feature of the new schemes is that they augment the Q function value of each state-action pair with an estimate of its uncertainty. This has the effect that the dialogue manager will choose the action which leads to the highest reward when it is confident about its estimate of that reward, otherwise it will choose the action about which it is least certain. This does not mean that the dialogue manager will never choose bad actions, but it does reduce the frequency with which it does so. The net result is faster learning and a higher reward during learning. This makes the new schemes more suitable for continuous on-line adaptation compared to conventional ǫ-greedy schemes.

The new exploration schemes were evaluated in this paper using GP-SARSA. In the future, alternative sample-efficient reinforcement learning algorithms such as in [START_REF] Geist | Kalman Temporal Differences[END_REF][START_REF]Statistically Linearized Least-Squares Temporal Differences[END_REF][START_REF] Pietquin | Sample efficient on-line learning of optimal dialogue policies with kalman temporal differences[END_REF] will be tested with the aim of improving the learning speed further.

  σ2 i (si+1, a) w.p. ǫ: ai+1 = ( arg max a Qi(si+1, a) w.p. 1ǫ arg max a σ2 i (si+1, a) w.p. ǫ

Figure 1 :

 1 Figure 1: Performance of the trained policies (greedy setting).To compute the average reward, each 100 dialogues, 100 policies were trained and tested in a greedy way 1000 times each.

Figure 2 :

 2 Figure 2: Performances of the policies (online setting). The curves are obtained for an average over 1000 training trials. An average is made with a sliding window of 100 points width each 50 points.


A L Pitel
email: pitel@cyceron.fr

Objective:Todisentangle, among the structural damage observed in Korsakoff"s syndrome (KS), those related to alcoholism, and therefore shared between AL and KS, from thosespecific to KS. Methods: Magnetic resonance imaging (MRI) data were collected in11alcoholic patients with KS, 34 non-Korsakoff alcoholic patients (AL) and 25 healthy control subjects (CS). Gray and white matter volumes were compared in the three groups using a voxel-based approach.

Results: Aconjunction analysis indicated a large pattern of shared gray and white matter volume deficits in AL and KS. There were graded effects of volume deficits (KS<AL<CS) in the medial portion of the thalami, hypothalamus (mammillary bodies), left insula andgenu of the corpus callosum.Abnormalities in the left thalamic radiation wereobservedonly in KS.

Conclusions: Our results indicate considerable similarities in the pattern of gray and white matter damage in AL and KS. This finding confirms the widespread neurotoxic effect of chronic alcohol consumption. Only a few cerebral regions, including the medial thalami, mammillary bodies and corpus callosum, were more severely damaged in KS than in AL. The continuum of macrostructural damage from AL to KS is therefore restrictedto key brain structures. Longitudinal investigations are required to determine whether AL with medial thalamic volumes that are comparable to those of KS are at increased risk to develop Korsakoff"s syndrome.

Introduction

Korsakoff"s syndrome (KS) is characterized by a disproportionate impairment of episodic memory compared with other aspects of cognitive function [START_REF] Kopelman | The Korsakoff syndrome[END_REF].Itresults from the combinationof heavy alcohol consumptionand thiamine deficiency.

Macrostructural abnormalities were initially examined by neuropathological studies, which concluded that specific thalamic nuclei and mammillary bodies play a key role in the pathophysiology of KS [START_REF] Victor | The Wernicke-Korsakoff syndrome. A clinical and pathological study of 245 patients, 82 with post-mortem examinations[END_REF][START_REF] Sheedy | Size of mamillary bodies in health and disease: useful measurements in neuroradiological diagnosis of Wernicke's encephalopathy[END_REF].In vivo neuroimaging investigations confirmed reduced volume in thalami and mammillary bodies in KS [START_REF] Shimamura | Korsakoff's syndrome: radiological (CT) findings and neuropsychological correlates[END_REF][START_REF] Sullivan | In vivo mammillary body volume deficits in amnesic and nonamnesic alcoholics[END_REF]and revealed widespread cerebral damage with the nodes and connections of thefronto-cerebellar and limbic circuits being especially affected [START_REF] Pitel | Morphological and glucose metabolism abnormalities in alcoholic Korsakoff's syndrome: group comparisons and individual analyses[END_REF].These circuits are also classically described damaged in non-Korsakoffalcoholics (AL; [START_REF] Chanraud | Brain morphometry and cognitive performance in detoxified alcohol-dependents with preserved psychosocial functioning[END_REF]). A direct comparison of brain damage in KS and ALwas conductedin several regions of interest and revealed acontinuum of brain damage, from mild in AL to moderate or severe in KS, notably in the mammillary bodies and thalami [START_REF] Sullivan | Neuroimaging of the Wernicke-Korsakoff syndrome[END_REF]. However,the comparison between AL and KS has never been conducted throughout the whole brain.The goalof the present study was thusto compare gray and white matter abnormalities in patients with AL and KS using a voxelwise approach to disentangle brain damage related to chronic alcohol consumption, and therefore shared between AL and KS, from those selectively damaged in KS.

Method

Sample

Eleven patients with KS (5 women, age=56.00±10.55), 34 patients with AL (6 women, age=43.47±8.36) and 25 healthy control subjects (CS, 14 women, age=43.88±11.24) were enrolled. None of the participants were taking psychotropic [Nom de la société] 5 medication, or presented withhistory of psychiatricor medical problems (head injury, coma, epilepsy, depression, hepatic encephalopathy etc.),which might affect cognitive function.KS patients were older than AL and controls (F= 7.37, p=0.001;

Tukey"s test: KS>controls, p=0.003 and KS>AL, p=0.001). KS patients were diagnosed with reference to the DSM IVcriteria for ""Persisting Amnestic Disorder"". A detailed neuropsychological examination confirmed that all KSpresenteddisproportionately severe episodic memory disorders comparedwith other cognitive functions (see [START_REF] Pitel | Morphological and glucose metabolism abnormalities in alcoholic Korsakoff's syndrome: group comparisons and individual analyses[END_REF]for more details). All KS patients had a history of heavy drinking but it was difficult to gain an accurate picture of their drinking history because of the amnesia. AL patients were recruited by clinicians on the basis of the DSM-IV criteria for alcohol dependencewhile they were receiving treatment for alcohol dependence as inpatients at Caen University Hospital(see [START_REF] Berre | Study of metamemory in patients with chronic alcoholism using a feeling-of-knowing episodic memory task[END_REF]for more details). All patients were at an early stage of abstinence (12.67 ± 6.94 days of sobriety prior to inclusion). They were interviewed to specify the length of time they had drunk to excess (16.09 ± 10.29 years) and their usual daily alcohol consumption (23.36 ± 15.01 "standard drinks" per day, i.e. any drink that contains approximately 10 grams of pure alcohol). Controls were social drinkers as defined by the NationalInstitute on Alcohol Abuse and Alcoholism.

Standard Protocol Approvals, Registrations, and Patient Consents

The participants gave their informed consent prior to their inclusion in the study, which was conducted in line with the Declaration of Helsinki and was approved by the local ethical committee for human investigations.

[Nom de la société] 6

MRI acquisition and preprocessing

For each subject, a high-resolution T1-weighted volumeMRI scan was obtained, which consisted of a set of 128 adjacentaxial slices parallel to the AC-PC line, covering the whole brainand with a 1.5-mm slice thickness and a 0.

Statistical analyses

Preprocessed data were analyzed in SPM5 in twofull factorial designs (for gray and white matter independently) using age and gender as covariates.We first examinedregions with reduced volume in each patient group compared to controls.

We then conducted a conjunction analysis of the two previous comparisons (AL<CS andKS<CS)to reveal brain abnormalities shared betweenthe two patient groups.For these analyses, we used a p value of p<0.05 corrected for False Discovery Rate(FDR). We thenassessed whether there was any area of significantly greater brain damage in KS compared to AL (within the regions significantly damaged in KS [Nom de la société] 7 compared to controls). We thus assessed the contrast KS-AL with the contrast KS-CS as a mask and using a less stringent p value p<0.001 uncorrected for multiple comparisons. Indeed, the differences between these two pathologies are more subtle than the differences between each of these pathologies and the controls,so that a more permissive threshold should be employed to avoid false negative. When necessary, average signal within significant clusters were extracted and post-hoc tests (Tukey"s test) were conducted. For all analyses, we used an extended threshold of 200 voxels.

A confirmatory analysiswas also conducted in age-matched subgroups of participants with equal sample sizes (supplementary data e-1).

Results

Analyses of gray matter volume

As illustrated in Figure1A and B, the analysis revealed widespread gray matter damage in both AL and KS patientscompared tocontrols.The conjunction analysis indicated extended areas of shared gray matter volume deficitsbetweenAL and KS patients,notably bilaterally in the orbitofrontal cortex extending to the parietal lobe, and in the cingulate cortex, insula, medial temporal lobes, thalami,hypothalamus, and cerebellum (p<0.05 FDR corrected, Figure 1C).Graded effects of volume deficits(KS<AL<CS; p<0.01 in all cases)were found in the medial portion of the thalami, hypothalamus (mammillary bodies) and left insula (Figure 2A). Using a 6 mm Gaussian kernel yielded similar results.

Analyses of white matter volume

Figure 1D and E illustrates white matter volume deficits in AL and KS patients compared tocontrols.The conjunction analysis revealedsharedwhite matter volume deficits in AL and KS patients in the corpus callosum and cerebellar peduncles and bilaterally in the fornix and cingulum (Figure 1F).There were graded effects of brain damage in the corpus callosum (KS<AL<CS; p<0.01 in all cases) but the left thalamic radiation was exclusively affected in the KS (Figure 2B).

Discussion

The present voxel-based comparison of gray and white matter indicatedstriking similarities in the regional distribution and severityof brain damage in AL and KS. This finding confirms the widespread direct or indirect (dietary deficiencies) neurotoxic effects of chronic alcohol consumption on the structure of the brain.The frontocerebellar circuit was affected to the same extent in AL and KS, in agreement with neuropsychological data collected in other groups of AL and KS, whichindicatedsimilar pattern of working memory deficits and executive dysfunction in each of the two groups [START_REF] Pitel | Episodic and working memory deficits in alcoholic Korsakoff patients: the continuity theory revisited[END_REF]. Even though we found volume deficits in Papez"s circuit in both patient groups, this circuit was more severely damagedin KS than in AL in agreement with a previous neuroimaging investigation [START_REF] Sullivan | Neuroimaging of the Wernicke-Korsakoff syndrome[END_REF]. Moreover, the left thalamic radiation was the only brain region to be exclusively altered in KS (with the statistical threshold used here). These findings fit with the fact that episodic memory is impaired both in AL and KS but that the main neuropsychological feature that distinguishes AL and KS is the severity of amnesia [START_REF] Pitel | Episodic and working memory deficits in alcoholic Korsakoff patients: the continuity theory revisited[END_REF].

[Nom de la société] 9

Only a few cerebral regions, including the medial thalami, mammillary bodies and corpus callosum, were more severely damaged in KS than in AL, suggesting that the continuum of macrostructural damage from AL to KS is selective to key brain structures. Statistical cluster analyses (Supplementary data e2) indicated that alcoholics with medial thalamic volume inferior to 2 standard deviations from controls were included within the same cluster as KS patients. Longitudinal investigations are required to determine whether those AL are at risk to develop Korsakoff"s syndrome.

[Nom de la société] 10

  94x0.94mm pixel size,through the use ofthe spoiled gradient echo sequence (TR = 10.3 ms;TE =2.1 ms; FOV= 240x180 mm2; matrix= 256x192). All theMRI datasets were acquired on the same scanner (1.5T SignaAdvantage Echospeed; General Electric), and based onthe sameacquisition protocol. The MRI data were analyzed according tothe VBM5 toolbox implemented in Statistical Parametric Mapping 5 software (SPM5; Wellcome department of cognitive Neurology, Institute of Neurology, London, England). Briefly, the procedure included segmentation,registration (spatial normalization) of original MRI datasetsusing the default MNI template of SPM5 as priors,modulation (to correct only for non-linear warping so that values in the resultant images are expressed as volume corrected for global brain size) andsmooth(10 mm Gaussian kernel).

Figure 1

 1 Figure 1 about here

Figure 2

 2 Figure 2 about here.

Acknowledgement: AM Aupéefor data collection in the KS group,Vincent de La Sayette, François Vabret and FaustoViader for screeningpatients with alcoholism and Korsakoff"s syndromeandEric MacKenzie for improving the English style [Nom de la société] 13

We used a p value cut-off of p<0.05 corrected for False Discovery Rate (FDR, larger images) but also displayed the results using a restrictive p<0.05 corrected for Family-Wise Error (FWE, smaller images)to highlight the most significant regions.

Cluster size > 200 voxels


Bassem Khouzam
email: bassem.khouzam@supelec.fr

Hervé Frezza-Buet
email: herve.frezza-buet@supelec.fr

Keywords: Dynamical Systems, Recurrent Neural Networks, Self-Organization

This paper presents a self-organizing architecture made of several maps, implementing a recurrent neural network to cope with partial observations of the phase of some dynamical system. The purpose of self-organization is to set up a distributed representation of the actual phase, although the observations received from the system are ambiguous (i.e. the same observation may correspond to distinct phases). The setting up of such a representation is illustrated by experiments, and then the paper concludes on extensions toward adaptive state representations for partially observable Markovian decision processes.

I. INTRODUCTION

In the design of artificial agents evolving in some environment, one has to deal with information streams. Sensors provide input streams to the information processing system of the agent, and actuators actually produce a stream of actions performed in order to exploit the environment. Such an action stream is actually the output of the agent to the environment. The coupling of the agent and the environment through such information and energy streams is obvious for any biologist who analyzes the behavior of some animal. Nevertheless, in the field of Computer Science and Machine Learning, computation is often considered off-line, for technical reasons. The typical case is the use of data sets to train the models, before actually using the trained models on-line.

This paper is a contribution to the part of Computer Science that is rather involved in the design of situated systems, thus actually focusing on the process of streams of information. In that sense, it is related to reinforcement learning approaches, that deal with sequential decision making of an agent continuously interacting with its environment, as well as temporal systems like recurrent neural networks that handle sequences of input. Indeed, the model proposed here allows to extract the phase of some dynamical system from a sequence of observations computed from that phase. Let us illustrate the need for such a feature from a straightforward toy example.

Let us consider an animal perceiving the temperature T of the floor. Let us suppose that any temperature T > T 0 is dangerous to it. In our example, the temperature oscillates periodically between high and low values (for night and day). The whole solar system configuration is the phase of the environment, noted x t here. It evolves in a deterministic way, according to Newton's law. The phase evolution is thus driven by a transition function φ such as ∀t, x t+dt = φ(x t ). The temperature perceived by the animal is an observation of the solar system, that can be expressed as

T t = O (x t )
where O is the observation function, see figure 1. The sun position in the sky P t = O ′ (x t ) would have been another observation of the solar system phase. Let us now consider a time t for which the temperature T t = T 0 -ǫ is just below the threshold. Should the animal try to hide away from the sun? The answer depends on the phase x t , from which the animal could know if the temperature is currently decreasing or increasing. The decision would have been easier from the perception of sun position P t , since O ′ is a bijective function, and thus the values of P t allow to take the decision directly from the current perception. If only T t is perceived by the animal, an efficient behavior requires that the animal is able to represent internally a value xt from which it can take the right decision, since values of temperature may be ambiguous (similar temperatures are observed twice a day). T is thus said to be a partial observation of x. The current value xt is inferred and updated from the successive observations T t . It is not required that x be the exact representation of the phase x, i.e. the animal do not need to know where the planets are, but x has to be set up such that a bijective observation function implicitly exists from x to x.

Partially observed environments are of interest in reinforcement learning domain. While the general trend is to find xt before computing the corresponding value function of each state, other works like [START_REF] Frezza-Buet | A scalable model-free recurrent neural network framework for solving pomdps[END_REF] implement evaluation with a recurrent network, but without explicitly extracting some xt .

The neural architecture presented in this paper relies on self-organizing neural networks in order to build such an internal representation from ambiguous sequences of observations. Many works in the literature try directly or indirectly to find xt . Concerning the use of self-organizing maps, many enhancements on Kohonen basic map [START_REF]Self-Organizing Maps[END_REF] like in [START_REF] Chappell | The temporal kohonen map[END_REF], [START_REF] Varstal | A recurrent selforganizing map for temporal sequence processing[END_REF], [START_REF] Koskela | Time series prediction using recurrent som with local linear models[END_REF] consider the temporal dimension of input sequences but they deal with the recognition of manually extracted sequences, rather than on-line stream of inputs. Reservoir computing methods [START_REF] Lukoševičius | Reservoir computing approaches to recurrent neural network training[END_REF] are closer to that purpose, since they handle inputs from a stream one by one. Readout units then search the huge reservoir states in order to locate few significant states. The significant states represents the phases of the system that provide the inputs.

For the best of our knowledge, the present work is the first attempt to find a mapping of a dynamical system phase space using on-line self-organizing recurrent neural networks.

II. THE MODEL

As mentioned above, our goal is to design a system that generates an internal representation of the dynamical system phases from the stream of observations emerging from it. Our model is an effort in this direction, inspiring from biological information. It is based on the bijama model [START_REF] Ménard | Model of multi-modal cortical processing: Coherent learning in self-organizing modules[END_REF], proposed and developed in our team. It enables building computational cortical-like 2D-neural assemblies called maps, made of computational units representing cortical columns. Units allow to process in parallel external entry and internal signals carried out by connections. A schematic of the model is shown in Fig.s 2 and 3. The architecture comprises three maps, namely the input map, the delay map, and the associative map. The input map receives the external input stream. Its activity is expected to represent at time t the coding xt of the actual dynamical system phase x t .

The two other maps, the delay map and the associative map are auxiliary maps that play the role of intermediate structures for extracting xt from the input stream o t . Their purpose is to re-inject the delayed activity of the input map into its dynamics. This recurrent pathway actually reveals the temporal dimension of the input stream. Map activities are computed by a neural field. Each unit has lateral recurrent connections to other units within the map, implementing an on-center/off-surround connectivity [START_REF] Amari | Dynamics of pattern formation in lateral-inhibition type neural fields[END_REF], [START_REF] Alecu | Can selforganization emerge through dynamic neural fields computation?[END_REF]. The field performs lateral competition between units and computes the activity of each one, so that the global map activity has the shape of a bump (see dark meshes in Fig. 4). The bump positions are actually the response of the map to its input. Lateral competition, from which activity bumps emerge, is used in the model in order to guide the process of selforganization of inputs over the map surface. This is indeed difficult with neural fields as explained in [START_REF] Alecu | Can selforganization emerge through dynamic neural fields computation?[END_REF] from which the neural field process used in this paper is taken. The whole architecture evolution is controlled by successive time steps. A time step is a discrete time instance at which the activities of all units in all maps are evaluated once, using an asynchronous evaluation scheme [START_REF] Alecu | Can selforganization emerge through dynamic neural fields computation?[END_REF]. Another kind of connectivity in the model is the inter-map connectivity. A unit at the bi-dimensional position p in some local map can be connected to a whole strip-shaped region S p in some remote map. Then unit p handles connections from the units at positions q ∈ S p in the remote map, as shown in Fig. 2. Each connection in a strip between p and a remote unit q handles a weight whose current value is st pq , so that the strip S p owned by p handles a vector of weights St p = st pq q∈Sp . Let us note S the set of strips received by local map (S = A, I, D in Fig. 2). Inter-map connections are referred to as cortical connections. Strips are characterized by their width ρ S and direction ψ S relative to the horizontal axis connecting the centers of the local and remote map. Let us note the activity of the unit at position p at time t as u t p , and the vector of remote unit activities perceived at p through the strip S p as S t p = u t q q∈Sp . The computation of unit activity using bijama is achieved using a stack of modules (see Fig. 3). Each one handles a scalar value that may be received as input or computed from lower modules in the stack. The higher module (here, the neural field module) handles the unit output that is the one actually accessed through cortical connections.

Let us first describe the stack of modules used for the units in the input map, see Fig. 3 and4 while reading the definitions which follow. In general terms, the input map, in the one hand, receives external observations o t . In the other hand it also receives strips (noted A) from the associative map. It outputs an activity bump as a response. This activity represents xt as will be shown later. The lower module is referred to as the thalamic module. It handles the external input o t , and matches it against a stored prototype ω t p and computes the similarity θ t p .

θ t p = e -( o t -ω t p) 2 2σ 2 (1)
The second module is referred to as the cortical module. It handles the strip A p emerging from the associative map, and computes the matching c t p,A between the strip weight vector Āt p and the activity vector A t p . The matching is computed as follows, where B is a numerical constant:

c t p,A = A t p . Āt p max Āt p 2 , B (2)
The third module is referred to as cortico-thalamaic merging. It merges θ t p and c t p,A into one scalar ν t p as follows:

ν t p = θ t p .β + (1 -β).c t p,A (3)
Where β is a constant. The value of ν t p forms the final input ready to use by the neural field, i.e. the upper module, to compute the unit activity u t p . The unit activity is used to modulate the learning. Thalamic learning implies moving θ t p towards o t proportionally to u t p , as shown in [START_REF] Varstal | A recurrent selforganizing map for temporal sequence processing[END_REF].

ω t+1 p = ω t p + α ω .u t p .(o t -ω t p ) (4)
Where α ω is a fixed thalamic learning rate for all units.

On the other hand, cortical learning implies moving the weight āt pq of each connection included in the strip S p towards the cortical input u t q , as shown by 5.

āt+1 pq = āt pq + α S .u t p .(u t q -āt pq ) (5)
Where α S is a fixed cortical learning rate for all model connections. The previous rule means that learning occurs only in connections to active units in local maps.

The next map in the model is the associative map. Its receives the actual activity of the input map as well as its delayed activity exhibited by the delayed map, then it performs lateral competition via the neural field, and reinjects the result into the input map through the previously mentioned A strips. The first module of a unit q of this map handles the strip I q emerging from input map and computes the matching c t q,I . The second module handles the strip D q emerging from delay map and computes the matching c t q,D . Both matching values are computed similarly to what was shown in input map units, according to 2. The third module is referred to as cortico-cortical merging. It merges c t q,I and c t q,D in one scalar µ t q as follows:

µ t q = c t q,I .c t q,I + noise µ The purpose of noise is to boost the associative map activity in units receiving null cortical activity before being injected into the input map. The value of µ t q is actually the input to the associative map neural field module which computes the unit activity u t q . The last map in the model is the delay map. It receives a unit-to-unit copy of the input map activity and delays it for some period of T time steps, using a T -length FIFO queue. Units in this maps have two modules. The first is the copy module that copies u t p from the input map. The second is the FIFO module. Thus u t p = u t-T q where q is a position in the input map and p the same position in the delay map. There is no need for a neural field in this map.

As can be seen, the proposed architecture requires no prior conditions on the input stream, nor on the underlying dynamical system, it is thus a model free architecture. The model does not require to adjust any parameter during execution. Learning rates are thus constant. Moreover, there is no need for resetting output bump activities u when a new observation o τ +1 is presented.

III. EXPERIMENTS

In this section, the model is tested to validate its capability to find an internal state representation of some unknown dynamical system providing observations. A toy example of dynamical system is used here to test the capacity of spatio-temporal organization of the model. Let x ∈ C represent the system phase, and let the transition function be φ (x) = x.e iϕ . Let us consider that the system transition occurs at instant τ , thus we can write x τ +1 = φ (x τ ) with x 0 = 0.5. Let the partial observation fed to the model be O (x τ ) = 0.5 + ℜ(x τ ) + noise o . Its values are kept in [0, 1]. The sampled stream values are perturbed by noise to test the robustness to noisy observations.

The dynamical system phase can be thought of as the position of a point moving in a steady speed on a circle, and the observation is its noisy abscissa. This sinusoidal observation is obviously ambiguous. In this experiment, τ is incremented every T time steps, and each observation O (x τ ) is fed to the model during T time steps, i.e.

o t = o t+1 = • • • o t+T -1 = O (x τ ) and o t+T = o t+T +1 = • • • o t+2T -1 = O x τ +1 , etc.
The reason for that is to give enough time to the neural field to relax and form an appropriate bump, as well as for cortical and thalamic learning to influence significantly the weights. T value is the same as FIFO length used in delay map units, thus, it delays input maps activity until the next O (x τ ) is sampled.

The input stream value o t is presented to all the units in the input map as for Kohonen maps [START_REF]Self-Organizing Maps[END_REF]. The map response is computed as the barycenter position G τ of the u activity bump at the end of each chunk of T successive time steps. It is computed as follows:

G τ = p u t p .p/ p u t p : p ∈ [1, M ] 2 (7)
Where M is the dimension of the square surrounding the round map. Each time that l barycenters are computed, they are organized in a list

P τ = {G τ -l+1 , G τ -l+2 , • • • , G τ } of
positions forming l-length paths over the map as sketched in Fig. 5. Successive paths allow to track the evolution of the map state xt through time. Fig. 5 shows the evolution through time of the input map. At the experiment start, thalamic values ω t p are random and the activity is located in a limited regions on the map as illustrates Fig. 5(a). Activity bumps start to disperse in Fig. 5 Figure 5. Status of the input map during the system evolution. Grey-scaled values are the ω prototypes (white for 0, black for 1). P τ is represented with a poly-line, linking successive positions G τ -l+1 , G τ -l+2 expresses a bijective mapping between the map surface and the dynamical system phase space. This was possible because the model allowed the previous state of the input map to be considered in computing its new state, integrating this way, its state history. The added noise to the input stream did not affect the model capability to extract the mapping.

The experience was launched with numerical values for the dynamical system as follows: ϕ = 2π/15 and noise o is sampled from a uniform random noise U [-0.05, 0.05].

Model numerical values was initialized as follows: M = 30 for all maps, u 0 p = 0, ω 0 p and ā0 pq , ī0 pq , d0 pq are initialized to uniform random values from U [0, 1], σ = 0.07, α ω = α S = 0.0416, B = 10, β = 0.25, noise µ is sampled from a uniform random noise U [0, 0.1], ρ I = ρ A = ρ D = 5. ψ I = 90, ψ A = -90, ψ D = 0, T = 24, and l = 50.

IV. CONCLUSION AND FUTURE WORK

In this paper, a recurrent neural architecture is proposed for setting up a representation of the phases of a dynamical system from the stream of partial observations of that dynamical system. The phase extraction relies on three self-organizing modules, whose self-organizing processes are coupled via strip-like connections, according to the bijama model. Experiments show that this fully unsupervised architecture is able to self-organize so that the token of hidden phases of the dynamical system are explicitly built in the input map. Indeed, for each bump position in that map, a phase of the dynamical system can be assigned. Moreover, the topology preservation that is expected from self-organizing maps actually stands here, since the input map is still a continuous mapping of the space where the observation lives (here the interval [0, 1]).

In the one hand, seminal works by Elman and Jordan [START_REF] Elman | Finding structure in time[END_REF], [START_REF] Jordan | Attractor dynamics and parallelism in a connectionnist sequential machine[END_REF] have already addressed the learning of a dynamical system from the stream of observation, but this was obtained from a supervised approach. In the other hand, as mentioned, reservoir computing approaches relies on high dimensional representation spaces to build an a priori set of states, from which the ones corresponding to the actual phases of the system can be extracted by readout units. In both cases, the setting up of a phase representation is not explicit. Here, the whole architecture adapts for extracting explicit phase representation by self-organization.

Future work will investigate the potential of the model self-organization features in both space and time when applied to systems exhibiting non-stationary dynamics. The goal is to see if the model would be able to recruit new regions in the input map or release useless regions when the dynamic change. Future work consists also in using the representation built in the input map as a state space for taking decisions within Markovian decision processes. In a more integrated model, such cortical representations could indeed feed actor and critic neural modules, inspired from basal ganglia modeling [START_REF] Doya | What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?[END_REF], [START_REF] Leblois | Competition between feedback loops underlies normal and pathological dynamics in the basal ganglia[END_REF], with a Markovian state space representation that is updated from the current partial information provided by the robot sensors.

Figure 1 . 2 Figure 2 .

 122 Figure 1. Dynamical system phase extraction.

Figure 3 .

 3 Figure 3. Module stacks of the units in the three maps, and their cortical inter-connections.

Figure 4 .

 4 Figure 4. Activity of model maps modules.

  (b). At this stage, due to thalamic learning guided by the neural field, thalamic values start to exhibit spatial organization as the grey-scaled regions show. The reason is that ω t p mainly organize according to values of o t which are in [0, 1]. Fig. 5(c) shows a better organization of the thalamic values, and different regions appear, each handling some different range of the o t values. Besides, it exhibits a better dispersion of activity bumps in the each region. At the end of learning, Fig. 5(d) shows that different greyscaled regions can be distinguished, indicating the spatial self-organization of thalamic values. The poly-line in the figure is formed by l = 50 points, corresponding each to the representation xt of a dynamical system phase x t . As can be seen, points are clustered along Successive bump positions #Map activity τ = 1 time steps. Map activity after τ = 132 × 50 i.e. t = τ × T = 158400 time steps. Map activity after τ = 360 × 50 i.e. t = τ × T = 432000 time steps. Map activity after τ = 554 × 50 i.e. t = τ × T = 664800 time steps.

  , • • • , G τ , that are localized on the figures with red dots.

	the poly-line. One cluster is formed by repeated visits of the	representations marked A,B,C. When a range of observations
	same system state. This indicates the stable representations	is located in the middle of the input values range, points
	of states in the map space. Each thalamic region corresponds	(like D,E) express non-successive states corresponding to the
	to a range of close observations values. Within each region	same observation range, but in different temporal context.
	there exists the representation of 2 or more states. For
	example, the black region corresponds to observation values
	close to 1, nevertheless, it contains 3 distinct successive state

Such duplications of states, related to the same value of O, are progressively performed while the whole architecture gets organized. It removes observation ambiguity. Thus, the ensemble of state representations x (i.e. bump positions)

ACKNOWLEDGMENTS

This work is part of the InterCell project supported by INRIA, Région Lorraine and Supélec.

http://intercell.metz.supelec.fr

Discovering the phase of a dynamical system from a stream of partial observations with a multi-map self-organizing architecture