S Sahand

Mohammadi Ziabari
email: sahandmohammadiziabari@gmail.com

An adaptive Temporal-Causal Network Model for Stress Extinction Using Fluoxetine

Keywords: temporal-causal network model, cognitive, extreme emotion, drugtherapy, fluoxetine

In this paper, an adaptive temporal causal network model based on drug therapy named fluoxetine to decrease the stress level of post-traumatic stress disorder is presented. The stress extinction is activated by a cognitive drug therapy (here fluoxetine) that uses continuous usage of medicine. The aim of this therapy is to reduce the connectivity between some components inside the brain which are responsible for causing stress. This computational model aspires to realistically demonstrate the activation of different portions of brain when the therapy is applied. The cognitive model starts with a situation of strong and continuous stress in an individual and after using fluoxetine the stress level begins to decrease over time. As a result, the patient will have a reduced stress level compared to not using drug.

Introduction

Stress is a vital response to physical and emotional threats with strong roots in human evolution. Stress is important to protect humans from dangerous conditions, where in early history it could have life-or-death consequences. A particular situation might trigger a fight or flight reaction, which could result in unnecessarily avoiding certain (social) circumstances. As it has been described in [START_REF] Blazer | Mood Disorders: Epidemiology[END_REF] depression is one of the most grueling psychiatric sicknesses and may decrease life-time expectancy with up to 20%.

Recent literature [START_REF] Vetencourt | The antidepressant fluoxetine restores plasticity in the adult visual cortex[END_REF] shows that fluoxetine suppresses or decreases synaptic changes associated with stress. It has been also mentioned that fluoxetine relatively suppresses the impact of stress on the infusion of synaptic plasticity in the medial prefrontal cortex which is responsible for receiving direct fibers from the hippocampus. There are some previous temporal causal network-oriented modeling literatures for decreasing stress have been proposed [START_REF] Treur | An adaptive temporal-causal network model for decision making under acute stress[END_REF][START_REF] Mohammadi Ziabari | Computational analysis of gender differences in coping with extreme stressful emotions[END_REF][START_REF] Mohammadi Ziabari | A modeling environment for dynamic and adaptive network models implemented in Matlab[END_REF][START_REF] Mohammadi Ziabari | Integrative Biological, Cognitive and affective modeling of a drug-therapy for a post-traumatic stress disorder[END_REF][START_REF] Mohammadi Ziabari | An adaptive cognitive temporal-causal network model of a mindfulness therapy based on music[END_REF][START_REF] Mohammadi Ziabari | Cognitive Modelling of Mindfulness Therapy by Autogenic Training[END_REF][START_REF] Mohammadi Ziabari | A Temporal Cognitive Model of The Influence of Methylphenidate (Ritalin) on Test Anxiety[END_REF][START_REF] Mohammadi Ziabari | An adaptive cognitive temporal-causal network model of a mindfulness therapy based on humor[END_REF][START_REF] Mohammadi Ziabari | Integrative cognitive and affective modeling of deep Brain stimulation[END_REF][START_REF] Andrianov | Cognitive Modeling of Mindfulness Therapy: Effects of Yoga on Overcoming Stress[END_REF][START_REF] De Haan | Integrative Biological, Cognitive and Affective Modeling of Caffeine use on Stress[END_REF].

The paper is organized as follows. In Section 2 the underlying neurological principles concerning the parts of the brain involved in stress and in the suppression of stress are addressed. In Section 3 the integrative temporal-causal network model is introduced. In Section 4 the results of the simulation model are discussed, in section 5 the mathematical analysis of the model is presented and eventually in the last section a conclusion is presented.

Underlying Neurological Principles

In many recent research literature [START_REF] Kessal | Fluoxetine reverses stress-induced fimbria-prefrontal LTP facilitation[END_REF][START_REF] Rocher | Acute stress-induced changes in hippocampal/prefrontal circuits in rats: effects of antidepressants[END_REF] it has been proven that fluoxetine decreases or suppresses changes in synapses caused by stress. In many researches [55][56][57] it has been proven that repeated stressful conditions and experiences bring a remarkable effect on neural plasticity in many brain components, especially in limbic structures like hippocampal changes, prefrontal cortex (PFC), and Amygdala. As it has been clearly mentioned in [START_REF] Rocher | Acute stress-induced changes in hippocampal/prefrontal circuits in rats: effects of antidepressants[END_REF]:

'Acute stress inhibits long-term potentiation (LTP) at synapses from the hippocampus to prefrontal cortex in the rat, a model of the dysfunction in the anterior cingulate/orbitofrontal cortices which has been observed in human depression.

In major depressive disorder, decreased blood flow and metabolism have been regularly described in multiple areas of the prefrontal cortex (PFC) with occasional changes in the hippocampal region. Conversely, a beneficial response to antidepressants has been associated with reduced blood flow in the hippocampus and a return to baseline metabolism level or increase in blood flow in the anterior cingulate cortex. Plasticity at hippocampal to PFC synapses can be regulated up and down, as assessed by longterm potentiation (LTP) and long-term depression (LTD), depending on specific patterns of afferent activation and this circuit contributes to working memory processes. Antidepressant effects may be obtained by several mechanisms, such as inhibition of serotonin uptake, for fluoxetine.' [START_REF] Rocher | Acute stress-induced changes in hippocampal/prefrontal circuits in rats: effects of antidepressants[END_REF] Also, previous studies [START_REF] Cook | Chronic stress alters dendritic morphology in rat medial prefrontal cortex[END_REF][START_REF] Radley | Reversibility of apical dendritic retraction in the rat medial prefrontal cortex following repeated stress[END_REF] revealed that chronic stress changes dendritic morphology not just in the hippocampus, but also in the mPFC (medial Prefrontal Cortex).

'Depression is said to be caused by chronically low levels of serotonergic transmission. SSRIs interfere with the activity of the serotonin transporter (5-HTT), a reuptake molecule that removes serotonin from the synapses. The putative low levels of synaptic serotonin in the depressed patient are elevated, and depression is relieved. These manipulations of serotonin levels have little effect on mood except in individuals who are depressed or recently recovered from depression.' [11, pp.1] 'Studies of neurotransmitter release with microdialysis have demonstrated that acute olanzapine significantly increases both dopamine and norepinephrine levels in rat prefrontal cortex, nucleus accumbens, and striatum, and the combination of olanzapine plus fluoxetine produces a greater increase in levels of dopamine and norepinephrine in the rat prefrontal cortex than fluoxetine alone.' [START_REF] Kuroki | Effects of antipsychotic drugs on extracellular dopamine levels in rat medial prefrontal cortex and nucleus accumbens[END_REF], pp.776.

The functionality of chronic stress on brain parts particularly on the Hippocampus is mentioned in [START_REF] Bremner | MRI-based measurement of hippocampal volume in patients with combat-related posttraumatic stress disorder[END_REF][START_REF] Kodama | Chronic Olanzapine or Fluoxetine administration increases cell proliferation in hippocampus and prefrontal cortex of adult rat[END_REF].

'The volume of the hippocampus is decreased in patients with depression or posttraumatic stress disorder.' [START_REF] Bremner | MRI-based measurement of hippocampal volume in patients with combat-related posttraumatic stress disorder[END_REF], pp. 975 'The reduction in hippocampal volume is inversely proportional to the amount of time a patient is medicated with an antidepressant, and reduced hippocampal volume is partially reversed after antidepressant treatment.' [START_REF] Kodama | Chronic Olanzapine or Fluoxetine administration increases cell proliferation in hippocampus and prefrontal cortex of adult rat[END_REF], pp.577 'In the striatum, there was a tendency for an increase in the number of BrdU-positive cells that is similar in magnitude to that in hippocampus. This effect is consistent with highly significant and robust induction of cell proliferation reported in a recent study, and the greater increase could be due to the higher dose of olanzapine used (10mg/kg) relative to the current study (2 mg/kg).

In the current study, we found that the combination of olanzapine plus fluoxetine did not produce a greater increase in the number of BrdU-positive cells than either drug alone. This suggests that fluoxetine alone would be sufficient to produce a maximum response and therefore could not account for the augmentation that has been observed clinically; however, the clinical approach has been to add olanzapine after a patient has failed to respond to an SSRI like fluoxetine' [START_REF] Kodama | Chronic Olanzapine or Fluoxetine administration increases cell proliferation in hippocampus and prefrontal cortex of adult rat[END_REF], pp.577.

Many researches illustrate that the hippocampus and other sections in the medial temporal lobe are interfered with detection of novelty [START_REF] Radley | Reversibility of apical dendritic retraction in the rat medial prefrontal cortex following repeated stress[END_REF][START_REF] Treur | Network-Oriented Modeling: Addressing Complexity of Cognitive, Affective and Social Interactions[END_REF]. In other research [START_REF] Treur | Verification of temporal-causal network models by mathematical analysis[END_REF] it has been proved that using of antidepressant drugs (Ads) will enhance the levels of extracellular epinephrine and serotonin.

'The ability to detect unusual events occurring in the environment is essential for survival. Several studies have pointed to the hippocampus as a key brain structure in novelty detection, a claim substantial by its wide access to sensory information through the entorhinal cortex and also distinct aspects of its intrinsic circuity.' [START_REF] Cook | Chronic stress alters dendritic morphology in rat medial prefrontal cortex[END_REF], pp.18286

In [START_REF] Han | Single fluoxetine treatment before but not after stress prevents stress-induced hippocampal long-term depression and spatial memory retrieval impairment in rats[END_REF] it has been shown that small amounts of fluoxetine might block stress-facilitated hippocampal LTD and eventually helps in memory retrieval impairment.

'Chronic fluoxetine treatment reinstates ocular dominance plasticity in the primary visual cortex of adult rats, a form of developmentally regulated plasticity that is significantly reduced in the mature brain, and enhances long-term potentiation (LTP) in the dentate gyrus of adult mice. Results show that chronic fluoxetine treatment suppresses LTP in the primary auditory cortex and hippocampus of adult rats. It has been well documented that exposure to acute stress impairs LTP and facilitate LTD in rats, as well as to produce learning and memory impairment in rats and monkeys. A single systematic injection of fluoxetine is able to reverse the impairment in LTP at synapses from the hippocampus to prefrontal cortex in the rats, caused by stress on elevated platform.' [START_REF] Han | Single fluoxetine treatment before but not after stress prevents stress-induced hippocampal long-term depression and spatial memory retrieval impairment in rats[END_REF], pp.1

In [10, pp.7] the influences of antidepressant agents such as fluoxetine is clearly mentioned:

'The chronic effects of antidepressant agents including fluoxetine, are involved in the regulation of intracellular transduction pathways, implicating changes in the cyclic adenosine monophosphate (cAMP) second messenger system, cAMP response element binding protein (CREB) and brain-derived neurotrophic factor (BDNF) in antidepressant action.'

In [START_REF] Czeh | Chronic social stress inhibits cell proliferation in the adult medial prefrontal cortex: Hemispheric Asymmetry and Reversal by Fluoxetine Treatment[END_REF] explicitly the effect of chronic stress on medial Prefrontal Cortex and Amygdala has been mentioned:

'Chronic stress significantly suppressed cytogenesis in the mPFC and neurogenesis in the dentate gyrus, but had minor effect in nonlimbic structures. Fluoxetine treatment counteracted the inhibitory effect of stress. Hemispheric comparison revealed that the rate of cytogenesis was significantly higher in the left mPFC of control animals, whereas stress inverted this asymmetry, yielding a significantly higher incidence of newborn cells in the right mPFC. Fluoxetine treatment abolished hemispheric asymmetry in both control and stressed animals. Structural alterations including suppressed dentate neurogenesis may contribute to the pathogenesis of depression. Antidepressant treatment with fluoxetine or electroconvulsive seizure modulates cell proliferation not only in the dentate gyrus, but also in the medial PFC (mPFC) in adult rats. (pp.1490)'

The Temporal-Causal Network Model

First the Network-Oriented Modelling approach used to model the integrative overall process is briefly explained. As discussed in detail in [START_REF] Treur | Network-Oriented Modeling: Addressing Complexity of Cognitive, Affective and Social Interactions[END_REF]Ch 2,[START_REF] Treur | Verification of temporal-causal network models by mathematical analysis[END_REF][START_REF] Treur | The Ins and Outs of Network-Oriented Modeling: from biological networks and mental networks to social networks and beyond[END_REF] this approach is based on temporal-causal network models which can be represented at two levels: by a conceptual representation and by a numerical representation. A conceptual representation of a temporal-causal network model in the first place involves representing in a declarative manner states and connections between them that represent (causal) impacts of states on each other, as assumed to hold for the application domain addressed. The states are assumed to have (activation) levels that vary over time. In reality, not all causal relations are equally strong, so some notion of strength of a connection is used. Furthermore, when more than one causal relation affects a state, some way to aggregate multiple causal impacts on a state is used. Moreover, a notion of speed of change of a state is used for timing of the processes. These three notions form the defining part of a conceptual representation of a temporal-causal network model:

• Strength of a connection wX,Y Each connection from a state X to a state Y has a connection weight value wX,Y representing the strength of the connection, often between 0 and 1, but sometimes also below 0 (negative effect) or above 1.

• Combining multiple impacts on a state cY(..) For each state (a reference to) a combination function cY(..) is chosen to combine the causal impacts of other states on state Y.

• Speed of change of a state hY For each state Y a speed factor hY is used to represent how fast a state is changing upon causal impact.

Combination functions can have different forms, as there are many different approaches possible to address the issue of combining multiple impacts. Therefore, the Network-Oriented Modelling approach based on temporal-causal networks incorporates for each state, as a kind of label or parameter, a way to specify how multiple causal impacts on this state are aggregated by some combination function. For this aggregation a number of standard combination functions are available as options and a number of desirable properties of such combination functions have been identified. Fig. 1 represents the conceptual representation of the temporal-causal network mode. The components of the conceptual representation shown in Fig. 1 are explained here. The state wsc shows the world state of the contextual stimulus c. The states ssc and ssee are the sensor state for the context c and sensor state of the body state ee for the extreme emotion. The states srsc and srsee are the sensory representation of the contextual stimulus c and the extreme emotion, respectively. The state srsee is a stimulus influencing the activation level of the preparation state. Furthermore, psee is the preparation state of an extreme emotional response to the sensory representation srsc of the context c, and fsee shows the feeling state associated to this extreme emotion. The state esee indicates the execution of the body state for the extreme emotion. All these relate to the affective processes. The (cognitive) goal state shows the goal for absorbing fluoxetine in the body. The (cognitive) state pspil is the preparation state of taking a pill (here fluoxetine). The state espil is the execution state of taking pill (fluoxetine). The other states relate to biological brain parts (Norepinephrine, Hippocampus, Thalamus, Serotonin, Prefrontal Cortex, Amygdala, Lateral Cerebellum, Striatum) which are involved in the stress condition, and in the influence of the fluoxetine applied.

Table 1 Explanation of the states in the model

The connection weights ωi in Fig. 1 are as follows. The sensor states ssee, sscc have two incoming connections from wsee and wsc (weights ω1, ω2). The world state of extreme emotion wsee has one arriving connection from esee, ω11 as a body-loop with weight. The sensory representation state of an extreme emotion srsee has an incoming connection weights ω8 from state preparation state of an extreme emotion psee. The feeling state fsee has one outgoing connection weight ω5 from srsee. The preparation state of an extreme emotion psee has two incoming connection weights ω36, ω37 from states Striatum and esact, respectively. The preparation state of an extreme emotion psee has three outgoing connection weights, esee, Thalamus, and the connection weight between states Hippocampus and Prefrontal Cortex, (ω10, ω15, ω20) respectively.

The goal has one arriving connection weight from the sensory representation srsee (ω27) and preparation state pspil an entering connection from the goal with weight ω28. The execution of taking the drug (here fluoxetine) is named espil, and has an entering connection weight w29 from preparation state of taking pspil. The state Thalamus has three entering connection weights ω12, ω14 and ω22 from preparation state of extreme emotion psee, Hippocampus and Amygdala, respectively. The Norepinephrine of brain has an arriving connection weight ω16 from espil. The Hippocampus in brain has four incoming connection weights, ω17, ω24, ω15 and ω13 from Serotonin, Prefrontal Cortex, Norepinephrine and Thalamus. Note that the connection weight between states Prefrontal Cortex and Hippocampus is adaptive and using Hebbian learning means through time will be changed. The state Serotonin has an arriving connection from espil (ω19). Prefrontal Corte state has an incoming connection weight from amygdala with ω25 and three outgoing connection weights to Amygdala, Lateral Cerebellum and psact, ω26, ω30, and ω31.

X21 psact

Preparation of action inside the brain psee named ω35, ω36. Finally, the state psact has an outgoing connection weight to psee named ω37.

Y(t+Dt) = Y(t) + hY [aggimpactY(t) -Y(t)] Dt or dY(t)/dt = hY [aggimpactY(t) -Y(t)]
• Thus, the following difference and differential equation for Y are obtained:

Y(t+Dt) = Y(t) + hY [cY(wX 1,Y X1(t), …, wX k,Y Xk(t)) -Y(t)] Dt dY(t)/dt = hY [cY(wX 1,Y X1(t), …, wX k,Y Xk(t)) -Y(t)]
For states the following combination functions cY(…) were used, the identity function id(.) for states with impact from only one other state, and for states with multiple impacts the scaled sum function ssuml(…) with scaling factor l, and the advanced logistic sum function alogistics,t(…) with steepness s and threshold t.

id(V) = V ssuml(V1, …, Vk) = (V1+ …+ Vk)/l alogistics,t(V1, …, Vk) = [(1/(1+e -σ(V 1 + … + V k -t ) )) -1/(1+e σ t )] (1+e -σ t )

Example Simulation

The simulation results of the cognitive temporal causal network model, which was constructed based on the neurological science which contains qualitive empirical information (such as fMRI) both for the mechanism by which the brain components work and for emerging result of the processes, has been shown in Figure 2. Therefore, one can imply that the best option for declining the stress level has been chosen, given the usage of fluoxetine. The model used the Matlab codes which have been implemented in [START_REF] Mohammadi Ziabari | A modeling environment for dynamic and adaptive network models implemented in Matlab[END_REF]. Using appropriate connections weights make the model numerical and adapted to qualitative empirical information. Table 2 illustrates the connection weights that has been used, where the values for are initial values as these weights are adapted over time. The time step was Dt = 1. The scaling factors li for the nodes with more than one arriving connection weights are mentioned in Table 2. At first, an external world state of an extreme emotion-stimuli context c (represented by wsc) will influence the affective internal states of the individual by influencing the emotional response esee (via ssc, srsc, and psee) conducted to manifest the extreme emotion by body state wsee. As a consequence, the stressed individual senses the extreme emotion (and at the same time all the biological brain components increased over time), so as a cognitive process, as a next step the goal becomes active to decrease this stress level by using fluoxetine at time around 300. As a biological process, the goal and in further steps, execution of taking drugs triggers the changes and suppression of execution of stress at the first state and this affects other brain components to be less active around time 300 and for stress level around 600. However, this effect is just temporary, and as the stressful context c still is present all the time, after a while the stress level goes up again, which in turn again leads to activation of the goal and performing another desire or prescription of eating fluoxetine, and so on and on repeatedly until the person or the doctor decides to stop taking drugs. The fluctuation in Figure 2 shows how in real life the repeated usage of medicine (here fluoxetine) decreases the stress level over each intake. It is worth to tell that all of this fluctuation is produced internally by the model; the environment is constant, external input for the model is only the constant world state wsc. Therefore, based on the simulation results it is illustrated that the model for the drug therapy (fluoxetine) works as expected.

In Figure 3, the situation where there is not any active goal (intake) has been shown. Based on this figure, when there is no intake (the goal is blocked in an artificial manner here), the stress level and activity of brain parts go up and stay high.

Mathematical Analysis

Emerging dynamic properties of dynamical models can be analyzed by simulation experiments, but some types of properties can be found by calculations in a mathematical manner using the WIMS Linear Solver1 . For verification of the proposed temporalcausal network model, stationary points are investigated.

To analyze the model mathematically, the solutions of linear equations of each state of the model are achieved and by comparing the outcome with the simulation results of the model using Matlab [START_REF] Mohammadi Ziabari | A modeling environment for dynamic and adaptive network models implemented in Matlab[END_REF] one can verify the model. To compare mathematical results with simulation results, in particular the ones illustrated in Fig. 1 and2, the parameter values for X3 =1 and the value of the X10 (goal) = 0.5 were used, due to the fact that the other states that are goal dependent are not able to go up and would not reach equilibrium. Some of the comparisons among simulation and mathematical analysis of states is depicted in Table 3.

Conclusion

In this paper a cognitive temporal causal network-oriented model of therapy by using drug (fluoxetine) for individuals under stress is introduced in which usage of medicine is used. The proposed model can be used to test different hypothesis and neurological principles about the impacts of the brain and the effects that different brain areas have the extinction of stress, but also on other processes. Some simulations have been implemented, one of which was presented in the paper. This model can be used as the basis of a chatbot, a virtual agent model and to get insight in such processes and to bring up a certain cure or treatment of individuals to perform the therapies of extreme emotions for post-traumatic disorder individuals.

Figure 1 .

 1 Figure 1. Conceptual representation of the temporal-causal network model This conceptual representation was transformed into a numerical representation as follows [17, Ch 2, 18, 19]: • at each time point t each state Y in the model has a real number value in the interval [0, 1], denoted by Y(t) • at each time point t each state X connected to state Y has an impact on Y defined as impactX,Y(t) = wX,Y X(t) where wX,Y is the weight of the connection from X to Y • The aggregated impact of multiple states Xi on Y at t is determined using a combination function cY(..): aggimpactY(t) = cY(impactX 1,Y (t), …, impactX k,Y (t)) = cY(wX 1,Y X1(t), …, wX k,Y Xk(t)) where Xi are the states with connections to state Y • The effect of aggimpactY(t) on Y is exerted over time gradually, depending on speed factor hY:

Figure 2 .

 2 Figure 2. Simulation results for temporal-causal network modeling of the therapy by fluoxetine

Figure 3 .

 3 Figure 3. Simulation results for equilibrium state without eating drug

Figure 4 .

 4 Figure 4. Simulation results for adaptivity connection weight between Prefrontal Cortex and Hippocampus

  The state Lateral Cerebellum has two incoming connection weights from Prefrontal Cortex and Amygdala ω30, ω32, respectively and it has an outgoing connection weight to Striatum ω34. The state Striatum has two outgoing connection weights to psact,

	X1	wsee	World (body) state of extreme emotion ee
	X2	ssee	Sensor state of extreme emotion ee
	X3	wsc	World state for context c
	X4	ssc	Sensor state for context c
	X5	srsee	Sensory representation state of extreme emotion ee
	X6	srsc	Sensory representation state of context c
	X7	fsee	Feeling state for extreme emotion ee
	X8	psee	Preparation state for extreme emotion ee
	X9	esee	Execution state (bodily expression) of extreme emotion ee
	X10	goal	Goal of using fluoxetine
	X11	pspil	Preparation state of using pill
	X12	espil	Execution of using pill
	X13	Norepinephrine	Brain part
	X14	Hippocampus	Brain part
	X15	Thalamus	Brain part
	X16	Serotonin	Brain part
	X17	Prefrontal Cortex Brain part
	X18	Amygdala	Brain part
	X19	Lateral Cerebellum Brain part
	X20	Striatum	Brain part

Table 2

 2 Connection weights and scaling factors for the example simulation

	Connection weight	w1	w2	w3 w4 w5	w6	w7	w8	w10	w11	w12	w13
	Value	1	1	1	1	1	1	1	1	1	1	1	1
	Connection Weight	w14	w15 w16 w17 w18 w19 w20 w21	w22	w23	w24	w25
	Value	1	1	1	1 -0.7	1	0.7	1	1	0.4	0.4	1
	Connection Weight	w26	w27 w28 w29 w30 w31 w32 w33	w34	w35	w35	w35
	Value	1	1	1	1	1	1	1	1	1	1	1	-0.9
	Connection Weight	w37
	Value	-0.9

Table 3 .

 3 Comparing Analysis and Simulation

		wsc	ssc	srsc	goal	pspil	espil	Norepine
	State	X3	X4	X6	X10	X11	X12	X13
	Simulation 1,0000	1.0000	1.0000	0.5000	0.5000	0.5000	0.5000
	Analysis	0.9999	0.9999	0.9999	0.4900	0.4900	0.4900	0.4900
	Deviation 0.0001	0.0001	0.0001	0.0100	0.0100	0.0100	0.0100

https://wims.unice.fr/wims/wims.cgi?session=K06C12840B.2&+lang=nl&+mod-ule=tool%2Flinear%2Flinsolver.en


Abdolreza Shirvani

B John Oommen

The Power of the "Pursuit" Learning Paradigm in the Partitioning of Data

Keywords: Object Partitioning, Learning Automata, Object Migration Automaton, Partitioning-based Learning

Traditional Learning Automata (LA) work with the understanding that the actions are chosen purely based on the "state" in which the machine is. This modus operandus completely ignores any estimation of the Random Environment's (RE's) (specified as E) reward/penalty probabilities. To take these into consideration, Estimator/Pursuit LA utilize "cheap" estimates of the Environment's reward probabilities to make them converge by an order of magnitude faster. This concept is quite simply the following: Inexpensive estimates of the reward probabilities can be used to rank the actions. Thereafter, when the action probability vector has to be updated, it is done not on the basis of the Environment's response alone, but also based on the ranking of these estimates. While this phenomenon has been utilized in the field of LA, until recently, it has not been incorporated into solutions that solve partitioning problems. In this paper 3 , we will submit a complete survey of how the "Pursuit" learning paradigm can be and has been used in Object Partitioning. The results demonstrate that incorporating this paradigm can hasten the partitioning by a order of magnitude.

Introduction

The Pursuit Concept in LA: Absolutely Expedient LA are absorbing and there is always a small probability of them not converging to the best action. Thathachar and Sastry realized this phenomenon and proposed to use Maximum Likelihood Estimators (MLEs) to hasten the LA's convergence. Such an MLEbased update method would utilize estimates of the reward probabilities in the update equations. At every iteration, the estimated reward vector was also used to update the action probabilities, instead of updating it based only on the RE's feedback. In this way, the probabilities of choosing the actions with higher reward estimates were increased, and those with lower estimates were significantly reduced, using which they proposed the family of estimator algorithms.

The Pursuit strategy of designing LA is a special derivative of the family of estimator algorithms. Pursuit algorithms "pursue" the currently-known best action, and increase the action probability associated with this action. The pursuit concept was first introduced by Thathachar et al, and the corresponding LA was proven to be ǫ-optimal. Its discretized version was proposed by Lanctot et al in [START_REF] Lanctot | Discretized estimator learning automata[END_REF], who also discretized the original estimator algorithm. Agache et al [START_REF] Oommen | Continuous and discretized pursuit learning schemes: various algorithms and their comparison[END_REF] then analyzed all the four linear combinations, i.e., the L RI and L RP paradigms. The Object Partitioning Problem (OPP): Consider the problem of partitioning a set

A = {A 1 , • • • , A W } of W physical objects into R groups Ω = {G 1 , • • • , G R }.
We assume that the true but unknown state of nature, Ω * , is a partitioning of the set A into mutually exclusive and exhaustive subsets

{G * 1 , G * 2 , • • • G * R }.
The composition of {G * i } is unknown, and the elements in the subsets fall together based on some criteria which may be mathematically formulated, or may even be ambiguous. These objects are now presented to a learning algorithm, for example, in pairs or tuples. The goal of the algorithm is to partition A into a learned partition, Ω + . The hope is to have Ω + converge to Ω * . In most cases, the underlying partitioning of Ω * is not known, nor are the joint access probabilities by which the pairs/tuples of A are presented to the learning algorithm known. This problem is known to be NP-hard [START_REF] Oommen | Stochastic automata solutions to the object partitioning problem[END_REF]. Clearly, if we increase the number of objects, the number of partitions increases, and in addition to this quantity, the problem's complexity grows exponentially. To resolve this, it is possible to explore all partition combinations, use a ranking index, and to thereafter, report the best plausible partition. The goal of the OPP is to identify the best or most likely realizable partitioning. This requires the AI algorithm to perceive the semantic physical world aspects of the objects, and to then make local decisions based on the best partition in the abstract domain [START_REF] Freuder | The object partition problem[END_REF][START_REF] Gale | Improvements to an algorithm for equipartitioning[END_REF]. Real versus Abstract Objects: If there exists a mutual relation between the real objects in the semantic domain A, and a domain of abstract objects O = {O 1 , • • • , O W }, we define the partitioning of O in such way that the corresponding partitions of O map onto the partitions of the real objects in A so as to mimic the state-of-nature. Thus, while we operate on the abstract objects in O, the objects in A are not necessarily moved because they constitute reallife objects which cannot be easily moved. A special case of the OPP is the Equi-Partitioning Problem (EPP) in which all the partitions are equi-sized. The Object Migrating Automation (OMA): Due to the poor convergence of prior OPP/EPP solutions, they were never utilized in real-life applications. The introduction of an LA-based partitioning algorithm, the OMA, (explained in Section 2) made real-life applications possible. The OMA resolved the EPP both efficiently and accurately. This solution is regarded as a benchmark for the EPP. Indeed, since 1986 4 , it has been applied to variety of real-life problems and domains which include keyboard optimization, image retrieval, distributed computing, graph partitioning, the constraint satisfaction problems, cryptanalysis, reputation systems, parallel and distributed mapping etc.

The Intent of this Paper: Although the "Pursuit" learning paradigm has been utilized in the theory and applications of LA as fundamental machines, until recently, it has not been incorporated into solutions that solve partitioning problems. The goal of this paper is to submit a comprehensive survey of how this paradigm can be used in Object Partitioning, and to optimize various versions of the OMA. We also include simulation results on benchmark environments that demonstrate the advantages of incorporating it into the respective machines.

The Object Migration Automata

The OMA is a fixed structure LA designed to solve the EPP. It is defined as a quintuple with R actions 5 , each of which represents a specific class, and for every action there exist a fixed number of states, N . Every abstract object from the set O resides in a state identified by a number, and can move from one state to another, or migrate from one group to another. If the abstract object O i is in state ξ i belonging to a group α k , we say that O i is assigned to class k.

If two objects O i and O j happen to be in the same class and the OMA receives a query A i , A j , they are jointly rewarded by E, the Environment. Otherwise, they will be penalized. We formalize the movements of {O i } on reward/penalty.

We shall formalize the LA as follows: For every action α k , there is a set of states {φ k1 , • • • , φ kN }, where N is the fixed depth of the memory, and where 1 ≤ k ≤ R represents the number of desired classes. We also assume that φ k1 is the most internal state and that φ kN is the boundary state for the corresponding action. The response to the reward and penalty feedback are as follows:

-Reward: Given a pair of physical objects presented as a query A i , A j , if both O i ,and O j happen to be in the same class α k , the reward scenario is enforced, and they are both moved one step toward the actions's most internal state, φ k1 . This is depicted in Figure 3.2 (a) in [START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF] 6 . -Penalty: If, however, they are in different classes, α k and α m , (i.e., O i , is in state ξ i where ξ i ∈ {φ k1 , • • • , φ kN } and O j , is in state ξ j where ξ j ∈ {φ m1 , • • • , φ mN }) they are moved away from φ k1 and φ m1 as follows:

1. If ξ i = φ kN and ξ j = φ mN , we move O i and O j one state toward φ kN and φ mN , respectively, as shown in Figure 3.2 (b) in [START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF]. 2. If ξ i = φ kN or ξ j = φ mN but not both (i.e., only one of these abstract objects is in the boundry state), the object which is not in the boundary state, say O i , is moved towards its boundary state as shown in Figure 3.2 (c) in [START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF]. Simultaneously, the object that is in the boundary state, O j , is moved to the boundary state of O j . Since this reallocation will result in an excess of objects in α k , we choose one of the objects in α k (which is not accessed) and move it to the boundary state of α m . In this case, we choose the object nearest to the boundary state of ξ i , as shown in Figure 3.2 (c) in [START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF].

3. If ξ i = φ kN and ξ j = φ mN (both objects are in the boundary states), one object, say O i , will be moved to the boundary state of α m . Since this reallocation, will again, result in an excess of objects in α m , we choose one of the objects in α m (which is not accessed) and move it to the boundary state of α k . In this case, we choose the object nearest to the boundary state of ξ j , as shown in Figure 3.2 (d) in [START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF].

To asses the partitioning accuracy and the convergence speed of any EPP solution, there must be an "oracle" with a pre-defined number of classes, and with each class containing an equal number of objects. The OMA's goal is to migrate the objects between its classes, using the incoming queries. E is characterized by three parameters: (a) W , the number of objects, (b) R, the number of partitions, and (c) a probability 'p' quantifying how E pairs the elements in the query.

Every query presented to the OMA by E consists of two objects. E randomly selects an initial class with probability 1 R , and it then chooses the first object in the query from it, say, q 1 . The second element of the pair, q 2 , is then chosen with the probability p from the same class, and with the probability (1 -p) from one of the other classes uniformly, each of them being chosen with the probability of 1 R-1 . Thereafter, it chooses a random element from the second class uniformly. We assume that E generates an "unending" continuous stream of query pairs.

The results of the simulations are given in Table 1, where in OM ApX , X refers to the probability specified above, W , is the number of objects, W/R is the number of objects per class, and R is the number of classes. The results are given as a pair (a, b) where a refers to the number of iterations for the OMA to reach the first correct classification and b refers to the case where the OMA has fully converged. In all experiments, the number of states of the OMA is set to 10. Also, the OMA's convergence for a single run and for an ensemble of runs display a monotonically decreasing pattern (with time) for the latter. 3 Developing the Pursuit Concept: The Environment

In an "un-noisy" Environment, we can denote the actual value of the relation between A i and A j (for k ∈ {1, • • • , R}) by the quantity µ * (i, j), expressed as:

µ * (i, j) = P (R k ) • P (A j |A i ) • P (A i ), ∀i, j if A i , A j ∈ R K , = 0 otherwise,
where P (R k ) is the probability that the first element, A i , is chosen from the group R k , and P (A j |A i ) is the conditional probability of choosing A j , which is also from R k , after A i has been chosen. Since E chooses the elements of the pairs from the other groups uniformly, with a possible re-numbering operation, the matrix M * = [µ * (i, j)] is a block-diagonal matrix given by Eq. ( 1).

M * =           M * 1 0 . . . 0 0 M * 2 . . . . . . . . . . . . 0 • • • . . . M * R ( 1
)
where 0 represents a square matrix containing only 0's.

Theorem 1. The matrix M * r , (1 ≤ r ≤ R), is a matrix of probabilities of size W R × W R
possessing the following form:

M * r =       0 R W (W -R) • • • R W (W -R) R W (W -R) 0 • • • R W (W -R) . . . . . . . . . R W (W -R) • • • R W (W -R) 0       (2)
Proof. The proof of the theorem is omitted here. It is found in [START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF].

⊓ ⊔

In a real-world scenario where E is noisy, i.e., the objects from the different groups can be paired together in a query, the general form for M * is:

M * =           M * 1 θ . . . θ θ M * 2 . . . . . . . . . . . . θ • • • . . . M * R , (3)
where θ and M * r s are specified as per Equations ( 4) and ( 5).

Theorem 2. In the presence of noise in E, the entries of the pair A i , A j can be selected from two different distinct classes, and hence the matrix specifying the probabilities of the accesses of the pairs obeys Equation (3), where:

θ = θ o •    1 1 • • • 1 . . . . . . . . . . . . 1 • • • • • • 1    , (4)
M * r = θ d •      0 1 • • • 1 1 0 • • • 1 . . . . . . . . . . . . 1 1 • • • 0      , (5)
where, 0 < θ d < 1 is the coefficient which specifies the accuracy of E, and θ o is related to

θ d as θ d = 1-θo(W -W R ) W R -1 .
Proof. The proof of the theorem is omitted here and found in [START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF].

The Design and Results of the POMA

In a real world scenario, since E's true statistical model is unknown, the expressions in Equations ( 1) and ( 2) can only be estimated through observing a set of queries. In the presence of noise though, we need to devise a measurable quantity which makes the algorithm capable of recognizing divergent pairs. Observe that whenever a real query A i , A j appears, we will be able to obtain a simple ML estimate of how frequently A i and A j are accessed concurrently. Clearly, by virtue of the Law of Large Numbers, these underlying estimates will converge to the corresponding probabilities of E actually containing the elements A i and A j in the same group. As the number of queries processed become larger, the quantities inside M * i will become significantly larger than the quantities in each of the θ matrices. From the plot of these estimates [START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF], one will observe that the estimates corresponding to the matrix M * i have much higher values than the off-diagonal entries. This implies that these off-diagonal entries represent divergent queries which move the objects away from their accurate partitions.

Intuitively, the pursuit concept for the OPP can be best presented by a matrix of size W × W where every entry will capture the same statistical measure about the stream of the input pairs. For the sake of simplicity, we use a simple averaging and denote this matrix by P. Every block represents a pair and the height of the block is set to the frequency count of the reciprocal pair. To obtain the average frequency of each pair, we let the OMA iterate for a sufficient time, say J iterations, and at every incident we update the value of the matrix P respectively. In this way, at the end of the J-th iteration, we have simply estimated the frequency of each pair. At this point, by observing the values of the matrix, the user can determine an appropriate threshold (τ > 0) to be adopted as the accept or reject policy for any future occurrence of this particular pair of objects. If we permit the algorithm to collect a large enough number of pairs, we see that ∃θ * | ∀θ o ≤ θ * , and that ∀ i, j : µ i,j ≫ θ * i,j . If we utilize a user-defined threshold, τ , (which is reasonably close to 0), we will be able to compare every estimate to τ and make a meaningful decision about the identity of the query. In other words, by merely comparing the estimate to τ we can determine whether a query pair A i , A j should be processed, or quite simply, be ignored. This leads us to algorithm POMA on Page 74 of [START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF] in which every query which is inferred to be divergent is ignored. Otherwise, one invokes the Reward and Penalty functions of the original OMA algorithm. The issue of determining the parameters of the POMA algorithm are detailed in [START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF], and omitted here in the interest of space.

In the initial phase of the algorithm, the estimates for the queries are unavailable. Thus, it only makes sense to consider every single query and to process them using the OMA's Reward and Penalty functions. Since the objects in each class are equally-likely to happen and the classes are equi-probable,

k ≥ W R 2 -W R × R
, is chosen as the lower-bound of the number of iterations for any meaningful initialization.

We have compared our results with those presented in [START_REF] Gale | Improvements to an algorithm for equipartitioning[END_REF] and those reported for the original OMA for various values of R and W . The number of states in every action was set to 10, and the convergence was expected to have taken place as soon as all the objects in the POMA fell within the last two internal states. The results (specified using the same notation as in Table 1) obtained are outstanding and are summarized in Table 2. The simulation results are based on an ensemble of 100 runs with different uncertainty values, (i.e., values of p). To observe the efficiency of the POMA, consider an easy-to-learn Environment of 6 groups with 2 objects in each group and where p = 0.9. It took the OMA 599 iterations to converge. As opposed to this, the POMA converged in only 69 iterations, which represents a ten-fold improvement. On the other hand, given a difficult-to-learn Environment with 12 objects in 2 groups, the OMA needs 6, 506 iterations to converge. The POMA required only 2, 112 iterations to converge, which is more than a three-fold improvement.

The learning of an enhanced LA proposed by Gale et al. [START_REF] Gale | Improvements to an algorithm for equipartitioning[END_REF], is based on the same principles of the OMA and leads to the Enhanced OMA (EOMA). They introduced three enhancements to improve its efficiency and speed as below:

1. Initial Boundary State Distribution: All of the objects are initially distributed at the respective boundary states of their respective classes; 2. Redefinition of Internal States: They diminished the vulnerability of the convergence criterion of the OMA by redefining the internal state to include "the two innermost states of each class", rather than a single innermost state. The simulation results obtained by introducing all of the three above-mentioned modifications are given in Table 3. In this table, we have reported the results from various Environments, with probabilities p = 0.9, 0.8 and 0.7, where p = 0.9 is the near-optimal Environment. Such an Environment is easy to learn from. On the other hand, for the case where p = 0.7, we encounter a difficult-to-learn scenario. Our results have also compared our own implementation of the OMA algorithm described in [START_REF] Gale | Improvements to an algorithm for equipartitioning[END_REF] with the EOMA's simulation results. These are the results displayed in Table 3. All the simulations reported were done on an ensemble of 100 experiments to guarantee statistically stable results. From the table we clearly see that as the value of p increases, the queries are more informative, and the convergence occurs at a faster rate. As before, the complexity of the classification problem has two criteria which are both observable in the tables, i.e., the number of objects in every group, W R , and the number of groups, R. As the number of objects and groups increase, the problem becomes increasingly complex to solve. The reader will easily observe the advantages gleaned by the above three modifications in the EOMA, by comparing Tables 1 and3.

The convergence of the EOMA with respect to time starts with a large number of objects which are located in random partitions. This number steadily decreases with time to a very small value. This graph is not monotonic for any given experiment. But from the perspective of an ensemble, the performance is much more monotonic in behavior. 5 Enhancing the EOMA with a Pursuit Paradigm

The methodology by which we incorporated the pursuit concept into the OMA required us to formally model noise-free and noisy queries in Section 3. However, this is the precise dilemma that the EOMA faces. On the one hand, it would be advantageous, from a partitioning perspective, to have a noise-free Environment. However, it is precisely such noise-free Environments that lead to deadlock situations. Consequently, an attempt to elevate a noisy Environment to become noise-free would only defeat the purpose by exaggerating deadlock scenarios. However, rather than seeking to make the Environment noise-free, we will again accept or reject queries from the incoming stream. To accomplish this, we again apply the same "Pursuit" paradigm, explained below for this specific setting.

To design the PEOMA, as before, we again incorporate the pursuit principle by estimating the Environment's reward/penalty probabilities. This could, of course, be done based on either a ML or Bayesian methodology. As these estimates become more accurate, we force the LA to converge to the superior actions at the faster rate. In all brevity, the PEOMA utilizes the exact same Pursuit principles explained in Section 3.1. Essentially, the EOMA which mitigates the "deadlock" situation is now augmented with the Pursuit concept, and thus:

-The stream of queries is processed using an estimation phase; -The divergent queries are filtered using a thresholding phase, that serves as a filter for the above estimates; -The deadlock scenarios are resolved using the enhancements of the EOMA over the OMA; -The convergence criterion of making the two most internal states of every group to report convergence, makes the entire process converge even faster.

The experimental results compared the PEOMA with the EOMA, and we were able to show how the PEOMA out-performed the EOMA and the OMA. The results are given in Table 4 which uses the same notation and for the same settings as in the Tables 1 and2. One can also compare its performance with the results presented in [START_REF] Gale | Improvements to an algorithm for equipartitioning[END_REF] and those reported in Table 3 for various values of R and W and in different Environments. The number of states in every action was set to be 10 as in Table 4. The convergence condition was also identical to the one specified in Section 4, and was assumed to have taken place when all the objects in the PEOMA fell within the last two internal states. Further, the query probability approximations were updated after receiving every single query.

The performance significance of the PEOMA is, really, not noticeable for easy problems where we had a small number of objects and groups, and where the noise level was low. But this becomes invaluable when we encounter a large number of actions as well as a stream of divergent queries (when p is "smaller") throughout the simulation, especially if we factor in the number of iterations used to obtain an estimate of τ . the PEOMA's performance can be up to more than two times better than the EOMA. But if we compare the results with the original OMA, the immense performance gain leads to about forty times less number of iterations for a complete convergence -which is by no means insignificant. It is fascinating to note that the reduction in the number of iterations required by the PEOMA can again be seen to be a consequence of a Pursuit-like filtering phase in all problem domains.

Table 1 .

 1 Experimental results for the OMA done for an ensemble of 100 experiments in which we have only included the results from experiments where convergence has occurred.

	W W/R R	OMAp9	OMAp8	OMAp7
	4	2	2	(2, 26)	(2, 36)	(2, 57)
	6	2	3	(3, 44)	(4, 62)	(4, 109)
	-	3	3	(22, 66)	(20, 88)	(26, 153)
	9	3	3	(44, 110)	(43, 144)	(70, 261)
	12 2	6	(10, 101)	(12, 146)	(15, 285)
	-	3	4	(82, 172)	(84, 228)	(128, 406)
	-	4	3	(401, 524)	(252, 405)	(256, 552)
	-	6	2 (2240, 2370) (1151, 1299) (1053, 1486)
	15 3	5	(152, 265)	(155, 325)	(191, 607)
	-	5	3 (1854, 2087) (918, 1136) (735, 1171)
	18 2	9	(17, 167)	(24, 252)	(29, 582)
	-	3	6	(180, 319)	(202, 413)	(288, 839)
	-	6	3 (5660, 5786) (1911, 2265) (1355, 2111)
	-	9	2 (11245, 11456) (6494, 7016) (3801, 4450)

Table 2 .

 2 Experimental results for the POMA approach done for an ensemble of 100 runs.

	W W/R R POMAp9	POMAp8	POMAp7
	4	2	2	(2, 25)	(3, 30)	(3, 38)
	6	2	3	(4, 44)	(4, 52)	(5, 67)
	-	3	2	(20, 65)	(22, 77)	(24, 106)
	9	3	3 (44, 105)	(70, 148)	(85, 169)
	12 2	6	(10, 88)	(12, 103)	(20, 173)
	-	3	4 (77, 166)	(105, 205)	(292, 462)
	-	4	3 (328, 417)	(228, 372)	(202, 487)
	-	6	2 (1563, 1836) (945, 1091) (1088, 1395)
	15 3	5 (112, 213)	(142, 274)	(179, 315)
	-	5	3 (1534, 1655) (766, 998)	(556, 931)
	18 2	9 (20, 151)	(26, 161)	(29, 566)
	-	3	6 (245, 410)	(198, 417)	(226, 395)
	-	6	3 (3146, 3270) (2182, 2371) (1145, 1542)
	-	9	2 (5500, 5621) (5064, 5523) (4104, 4711)

  [START_REF] Fayyoumi | Achieving microaggregation for secure statistical databases using fixed-structure partitioning-based learning automata[END_REF]. Breaking the Deadlock: The original OMA possesses a deadlock-prone infirmity (please see Section 4.3 of[START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF]) in which the machine can cycle between two identical configurations by virtue of a sequence of query pairs. Thus is especially evident in noise-free Environments. The EOMA remedies this as follows. Give a query pair of objects O i , O j , let us assume that O i , is in the boundary state, and O j is in a non-boundary (internal) state of another class. If there exists an object in the boundary state of the same class, we propose that it gets swapped with the boundary object O j to bring both of the queried objects together in the same class. Simultaneously, a non-boundary object has to be moved toward the boundary state of its class. Otherwise, if there is no object in the boundary state of the class that contained O j , the algorithm performs identically to the OMA.

Table 3 .

 3 Experimental results for the Enhanced OMA (EOMA) done for an ensemble of 100 runs.

	W W/R R EOMAp9 EOMAp8 EOMAp7
	4	2	2 (2, 26)	(2, 30)	(3, 60)
	6	2	3 (4, 46)	(4, 65)	(5, 106)
	-	3	2 (6, 50)	(8, 74)	(11, 127)
	8	2	4 (6, 64)	(7, 95)	(8, 158)
	-	4	2 (14, 75) (20, 110) (32, 185)
	9	3	3 (18, 91) (24, 132) (35, 233)
	10 5	2 (8, 85)	(10, 118) (13, 226)
	-	2	5 (25, 106) (33, 153) (70, 277)
	12 2	6 (10, 102) (12, 154) (17, 291)
	-	3	4 (43, 136) (56, 207) (81, 380)
	-	4	3 (54, 150) (66, 196) (99, 388)
	-	6	2 (40, 133) (64, 208) (105, 405)
	15 3	5 (65, 187) (92, 284) (134, 554)
	-	5	3 (75, 191) (108, 295) (192, 617)
	18 2	9 (19, 170) (26, 253) (36, 630)
	-	3	6 (106, 258) (140, 389) (242, 827)
	-	6	3 (114, 255) (167, 392) (261, 857)
	-	9	2 (112, 246) (142, 363) (311, 854)

Table 4 .

 4 Experimental results for the PEOMA approach done for an ensemble of 100 runs.

	W W/R R PEOMAp9 PEOMAp8 PEOMAp7

The second author gratefully acknowledges the partial support of NSERC, the Natural Sciences and Engineering Council of Canada.

The bibliography in this paper is necessarily limited. The majority of the present results very briefly summarize the results in the Ph.D. thesis of the First Author.

To be consistent with the terminology of LA, we use the terms "action", "class" and "group" synonymously.

The OMA's algorithms/figures are in[START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF], and omitted here in the interest of space.

The first issue that we encounter when we want to advance the field of resolving the EPP is to see if we can use new criteria to identify which objects belong to the same partition. We intend to investigate how this can be inferred without considering the issues that have been analyzed earlier. It is easy to see that all the objects within an underlying partition should be strongly and directly related to each other, and that they should frequently co-appear in the queries. Such structural patterns are, in turn, based on so-called casual propositions which should lead towards relational "interactions" between the objects themselves. This is the avenue that we now investigate.

Structural relations that are imposed by the Environment can orient the objects towards a uniformity when there is an "interaction" between a pair of objects. Such relations may be "transmitted" through intermediaries even when two objects are not explicitly examined at any given time instant. This interconnection is directly associated with the relational bonds that these objects possess. We shall now investigate whether this property, which already relates subgroups and not just pairs, can be quantified by various specific properties that can be extracted from the Environment. They can be seen to be:

1. The frequency of objects co-occurring; 2. The relative frequency of the objects in a pair belonging to distinct partitions; 3. The symmetric property of the queries in any pair presented by E; 4. The reachability of the objects in a partition within the graph representing the set of all objects.

We first formalize the partitioning problem's symmetry and transitivity properties proven in [START_REF] Shirvani | Novel Solutions and Applications of the Object Partitioning Problem[END_REF]. Since E is transitive, our aim is now to have the LA infer this transitivity and to further enhance the PEOMA. Indeed, if the pursuit matrix is appropriately thresholded, the entries become unity and zero, which allows us to demonstrate transitivity and thus, invoke reward/penalty operations even while the environment is dormant and not generating any new queries. Without going into the explicit details (omitted due to space limitations), this is essentially done by invoking the assertion:

This leads us to the Transitive PEOMA (TPEOMA). The experimental results for the PEOMA are given in Table 5 for the same settings as in the previous tables.

By way of example, if the TPEOMA is compared with the previously bestreported algorithm, the PEOMA reported in Section in 5, one can see that the PEOMA can solve the partitioning problem with p = 0.9 and 3 groups with 3 objects in each group, in 85 iterations. For the same problem, the TPEOMA required only 65 iterations to converge. For a difficult-to-learn Environment (p = 0.7) and a more complex partitioning problem with 18 objects in 3 groups, the PEOMA needed 472 iterations to converge. The TPEOMA required only 244 iterations to converge, which is nearly two times better than the PEOMA. It is certainly the fastest partitioning algorithm reported to date, and its behavior is monotonically decreasing for an ensemble of many experiments. The reader should observe the considerable performance that is gained by a very little additional computational cost. Again, by comparing Tables 4 and5, one observes that although the gain is not significant for simple problems and easy Environments, it becomes remarkably high for complex partitioning experiments.

Conclusions

In this paper we have shown how we can utilize the "Pursuit" concept to enhance solutions to the general problem of partitioning. Unlike traditional Learning Automata (LA), which work with the understanding that the actions are chosen purely based on the "state" in which the machine is, the "Pursuit" concept has been used to estimate the Random Environment's (RE's) reward probabilities and to take these into consideration to design Estimator/Pursuit LA. They, utilize "cheap" estimates of the Environment's reward probabilities to make them converge by an order of magnitude faster. This is achieved by using inexpensive estimates of the reward probabilities to rank the actions. Thereafter, when the action probability vector has to be updated, it is done not on the basis of the Environment's response alone, but also based on the ranking of these estimates.

In this paper we have shown how the "Pursuit" learning paradigm can be and has been used in Object Partitioning. The results demonstrate that incorporating this paradigm can hasten the partitioning by a order of magnitude. This paper comprehensively describes all the Object Migration Automaton (OMA)-related machines to date, including the Enhanced OMA [START_REF] Gale | Improvements to an algorithm for equipartitioning[END_REF]. It then incorporates the Pursuit paradigm to yield the Pursuit OMA (POMA), the Pursuit Enhanced OMA (PEOMA) and the Pursuit Transitive Enhanced OMA (PTOMA).

Apart from the schemes themselves, the papers reports the experimental results that have been obtained by testing them on benchmark environments.


E Ploux

L Gorisse

I Radzishevsky

H Wolosker

T Freret

J-M Billard

D-serine is

#

  Extracellular recording in CA1 stratum radiatum hippocampal slices from either 3--4 or 10--12 month--old mice (aEer electrical sFmulaFon of Schaffer collaterals) • High frequency (HFS)--induced long--term potenLaLon (LTP): --TesFng sFmulaFon (average of 3 sFmulaFons at 0,1 Hz) for 15min baseline and then resumed for 60 min aEer condiFoning --CondiFoning : 1 electrical train of sFmulaFon at 100 Hz for 1 sec • Isolated NMDAr--mediated fEPSPs (field Excitatory PostSynapLc PotenLals): --PresynapFc Fiber Volley (PFV) and field Excitatory PostsynapFc PotenFals (fEPSPs) were recorded and fEPSP/PFV raFo were calculated for Input/Output curves --Recording were performed (low magnesium supplemented medium with the non--NMDAr antagonist NBQX, 10µM) either before and 15 min aVer addiLon of D--serine (100 µM) D--SERINE IS INVOLVED IN THE β--AMYLOID--RELATED PATHOPHYSIOLOGY IN ALZHEIMER'S DISEASE 1 Ploux E., Gorisse--Hussonnois L., 2 Radzishevsky I., 2 Wolosker H., 1 Freret T., 1 Billard J--M. 1 Normandie Univ, UNICAEN, INSERM, COMETE, GIP CYCERON, 14000 Caen, France 2 Department of biochemistry, Technion, Israel InsItute of Technology, Haifa, Israel AcFvaFon of N--methyl--D--aspartate subtype of glutamate receptors (NMDAr), key regulators of funcFonal brain plasFcity and memory process, requires the binding of the co--agonist D--serine. The homeostasis of these receptors are affected by soluble oligomers of the beta--amyloid pepLde (Aßo) in Alzheimer´s disease (AD). Aßo toxic effects possibly pass through mechanisms involving D--serine since Aßo sFmulate in vitro the producFon of the co--agonist. However, the actual in vivo contribuFon of D--serine in the funcFonal NMDAr--related deregulaFons mediated by Aßo is sFll unknown. Spontaneous alternaFon test --12 month--old mice were tested for working memory performances --Mice were let free to explore Y-maze for 8--min and successive entry of the 3 arms of the maze was considered as an alternaFon --The percentage of alternaLon was calculated as follows: NB of alternaFons / (total NB of arms visited -2) x 100 Hippocampal phenotype of the experimental groups 5xFAD and 5xFAD/SR --/--display similar hippocampal level of Aßo (only traces were observed in WT mice, used as negaIve control) 5xFAD/SR --/--mice at both ages show markedly reduced hippocampal level of D--serine (either compared to WT mice or 5xFAD) Func+onal plas+city (HFS--induced LTP) SR deleFon (5xFAD/SR --/--) reverses LTP deficits displayed in 5xFAD mice, both in 3--4 and 10--12 month--old animals NMDAr ac+va+on (isolated synap+c poten+als) No genotype difference of NMDAr acFvaFon is observed in basal condiFons (A) Only WT and 5xFAD/SR --/--mice show significant D--serine--induced increase in NMDAr acFvaFon (B) Our results show that deleLon of serine racemase prevents, at least parLally, memory--related behavioral deficits observed in mice with prominent features of amyloïdogenesis. Besides, impairment of NMDAr-dependent funcLonal plasLcity, indicated a significant contribuLon of D--serine in NMDAr--dependent β-amyloid related pathophysiology of AD.

3

  5xFAD/SR --/--) parFally reverses reference memory deficits displayed by 5xFAD mice aEer a second spaFal learning Spa+al learning, Flexibility, Reference memory Contact : eva.ploux@unicaen.fr 5xFAD and 5xFAD/SR --/--mice display reduced spaFal learning performances (compared to WT mice) 5xFAD mice display flexibility deficits (compared to WT mice), fully reversed in 5xFAD/SR --/--mice Learning Relearning Probe--test1 Probe--test2 Unaltered reference memory in 5xFAD and 5xFAD/SR --/--mice Morris Water Maze --12 month--old mice were trained to learn locaFon of hidden plaporm (through disFnct visual cues) --Learning (spaIal learning performance) and relearning (flexibility) session: 4 trials/day during 5 days (60 sec inter--trial interval) --Probe tests (spaFal reference memory performance): performed 48 aEer last trial of (respecFvely, learning and relearning session). Plaporm was removed and mice were free to explore the maze during 60 sec In this study, we wonder if D--serine contributes in vivo to NMDAr deregulaFons mediated by Aßo. Behavioral analysis combined to extracellular electrophysiological recordings at hippocampal synapses is conducted in the 5xFAD transgenic mice model of amyloïdogenesis displaying marked increase in Aßo rates. They are compared to 5xFAD animals in which the homozygous gene of the serine racemase (-s e r i n e + D --s e r i n e + D --s e r i n e ; ANOVA repeated measures * p< 0.05 ; *** p< 0.001 ; ### p< 0.001 Mean + sem ; Univariate t--test : $ p<0.05 vs 50% Mean + sem ANOVA one--way No group effect Univariate t--test : $ p<0.05 vs 25%


Marion David

Mathieu Lavandier

Nicolas Grimault

Andrew J Oxenham

Binding of speech syllables when segregation occurs

have investigated the segregation of speech syllables made of a fricative consonant and a vowel, referred to as CV tokens. The first study explored the segregation of such syllables based on fundamental frequency differences. The second study explored the segregation of the CV tokens based on localization cues, especially the spectral cues in the median plane. Both studies found that segregation can be observed based on F0 and on spectral cues. Interestingly, it was found that the whole CV token remains grouped even when segregation occurs based on cues that affect only one part of the CV: F0 differences affect mostly the vowel part, whereas coloration in the median plane is effective mostly at high frequencies, selectively affecting the consonant part. The mechanisms that allow the CV to remain grouped under such circumstances remain unclear. The present manuscript reviews the results of these two studies and provides some suggestions as to how such binding might occur.

Introduction

In order to understand a target voice amid a background noise (e.g., in a cocktail party; [START_REF] Cherry | Some experiments on the recognition of speech, with one and with two ears[END_REF], the target must be grouped into one single auditory stream and segregated from the other sound sources. Such auditory object formation relies on our ability to organize competing sound sources into coherent streams [START_REF] Bregman | Auditory Scene Analysis: The Perceptual Organization of Sounds[END_REF].

Auditory stream segregation and integration have been studied using both speech and non-speech sounds (for reviews, see [START_REF] Moore | Factors influencing sequential stream segregation[END_REF][START_REF] Moore | Properties of auditory stream formation Properties of auditory stream formation[END_REF]. It has been shown that differences in fundamental frequency (F0) provide important segregation cues for pure and complex tones [START_REF] Miller | The masking of speech[END_REF][START_REF] Van Noorden | Temporal Coherence in the Perception of Tone Sequences[END_REF][START_REF] Vliegen | Sequential stream segregation in the absence of spectral cues[END_REF]. In addition, sound localization cues, including interaural time and level differences (ITDs and ILDs, respectively) and monaural spectral (or coloration) differences [START_REF] Blauert | Spatial hearing : the psychophysics of human sound localization[END_REF][START_REF] Wightman | The dominant role of low-frequency interaural time differences in sound localization[END_REF], can induce stream segregation of pure and complex tones [START_REF] Gockel | Context dependence of fundamental-frequency discrimination: lateralized temporal fringes[END_REF][START_REF] Sach | Some characteristics of auditory spatial attention revealed using rhythmic masking release[END_REF][START_REF] Stainsby | Sequential streaming due to manipulation of interaural time[END_REF], as well as frozen speech-shaped noises. With some exceptions [START_REF] Gaudrain | Streaming of vowel sequences based on fundamental frequency in a cochlear-implant simulation[END_REF][START_REF] Gaudrain | Effect of spectral smearing on the perceptual segregation of vowel sequences[END_REF][START_REF] Hartmann | Stream Segregation and Peripheral Channeling[END_REF], studies of auditory streaming have tended to use sequences of single repeated sounds, making it difficult to generalize the results to real-world situations, where sounds are complex, constantly changing, and not fully predictable. The aim of the studies reviewed in this manuscript (David et al., 2017a, b) was to apply a streaming paradigm to stimuli that are closer to real-world speech, by using random sequences of consonant-vowel (CV) syllables with natural spectro-temporal variability. Such stimuli could be segregated based on differences in either F0 or simulated spatial position, despite variations along other spectro-temporal dimensions.

Rationale

Two sequences of sounds can be perceived as integrated -if the perceptual acoustical difference between the sequences is small enough -or as segregated -if the perceptual difference between the sequences is large enough. Auditory scene analysis is governed by both voluntary and obligatory mechanisms, depending on the task the listeners have to complete. Voluntary streaming corresponds to tasks where the listeners try to hear out a target sound from a mixture. Conversely, obligatory streaming corresponds to situations where the listeners attempt to bind the sounds into a single stream but fail to do so [START_REF] Bregman | Auditory Scene Analysis: The Perceptual Organization of Sounds[END_REF].

Two interleaved sequences, A and B, were presented to the listener (cf. Fig. 1). The difference between the sequences was based either on fundamental frequency or on simulated spatial position in the horizontal or median plane. Here we discuss only the results in the median plane, as these are the ones where the spectral cues are superimposed on the inherent spectro-temporal variability of the speech tokens.

In both studies, one of the tasks consisted of attending to the whole interleaved sequence (i.e., grouping the two sequences together) regardless of the difference in F0 or in simulated positions between tokens, and indicating whether or not a repeated token was introduced. For good performance in this task, listeners should perceptually integrate the A and B sequences into a single stream, so that a repetition is heard within this stream, making it a measure of obligatory stream segregation [START_REF] Micheyl | Objective and Subjective Psychophysical Measures of Auditory Stream Integration and Segregation[END_REF]. Since a difference in F0 would only influence the segregation of vowels, and since a spectral difference induced by a difference in elevation would only influence the segregation of the consonants, the analysis of the results focused on assessing whether consonants were separately grouped from vowels or whether they nevertheless remained grouped due to their occurrence in one syllable.

3. Stimuli and procedure David et al. (2017a, b) used differences in spatial cues and F0, respectively, to study streaming of speech sounds. The stimuli used in both studies were naturally uttered pairs of voiceless fricative consonants and voiced vowels. They were recorded by a male native speaker of American English as a whole, so that they included a fricative part (the consonant), a formant transition part (the vocalic part still containing some consonant information) and a voiced part (the vowel). In the first study (David et al., 2017a), a set of 45 stimuli were recorded (five voiceless fricative consonants - [uː]). In the second study [START_REF] David | Sequential stream segregation of voiced and unvoiced speech sounds based on fundamental frequency[END_REF], 36 stimuli were recorded (four consonants -[f], [s], [Ɵ] and [ʃ] -combined with the same nine vowels). Each single recorded stimulus will be referred to as speech token in the rest of the manuscript.

[f], [s], [Ɵ], [ʃ] and [h] -combined with nine vowels -[ae], [e], [iː], [I], [Ə], [Ɛ], [Ʌ], [ɑ] and
The stimuli had to be short enough to potentially produce obligatory stream segregation [START_REF] Van Noorden | Temporal Coherence in the Perception of Tone Sequences[END_REF], but long enough to contain the information from both the consonant and the vowel. The duration of each speech token was therefore limited to 160 ms with 40-ms inter-token intervals, leading to an onset-to-onset time of 200 ms, which is close to the upper limit for observing obligatory stream segregation. The consonant and the vowel parts had approximatively the same length. The pitch contour of the tokens were flattened (110 Hz, unless specified otherwise) using Praat Software [START_REF] Boersma | Praat a system for doing phonetics by computer[END_REF], where the stimuli were resynthesized using a pitch synchronous overlap-add technique (PSOLA).

In the first study, the ΔF0s tested were 0 semitones (F0 A = 110 Hz, F0 B = 110 Hz), 3 semitones (104 and 123 Hz), 5 semitones (98 and 131 Hz), 7 semitones (92 and 139 Hz), 9 semitones (87 and 147 Hz), and 13 semitones (78 and 165 Hz). In the second study, the differences in elevation angle separating the sequences were 0º, 10º, 50º and 70º.

In 50% of the trials, a repetition of a full token (consonant and vowel, "full repeat") was presented. In 25% of the trials, a repetition of only the consonant was presented and in the last 25%, a repetition of only the vowel was presented. According to this paradigm, the Hits (Hs) correspond to the proportion of full repeats that were correctly reported and the False Alarms (FAs) correspond to the proportion of trial in which a repetition was reported when only a half-repeat was presented. Hence, it was possible to calculate separately the FAs for the consonant-only and the vowel-only repeats. The repeat, if present, occurred always at the penultimate position and the length of the sequence varied between 8 and 14 pairs of tokens. Figure 1 represent the tokens in the across-sequence task in the three conditions.

Separate contributions of vowels and consonants to repetition detection

It was found that obligatory stream segregation could be observed based on a F0 difference [START_REF] David | Sequential stream segregation of voiced and unvoiced speech sounds based on fundamental frequency[END_REF]) and coloration differences in the median plane (David et al., 2017a). Fig. 2 represents the results of the first and second studies in terms of mean rates for the Hits and False Alarms.

In both cases, the Hit rate decreased, and thus segregation increased as the difference between the two sequences increased (a ΔF0 or a difference in spectral coloration in the median plane). There was no significant difference between the FAs due to a repeat of the consonant only or of the vowel only.

Therefore, the authors then concluded that the listeners based their judgments on the whole CV token.

This result is striking because the CV token seems to remain grouped, even when segregation occurs based on cues that affect only one part of the CV: F0 differences affect mostly the vowel (voiced) part, whereas coloration in the median plane, effective mainly at high frequencies, affects mostly the consonant part.

Discussion

The findings of the above studies suggest that in real life, when two sequences of sounds are perceptually segregated, all the syllables from one sequence are grouped together and segregated from the second sequence. From the studies discussed, it seems that the consonants and the vowels belonging to one sequence are never streamed apart. One possible interpretation of this result might be a top-down process. When learning a language, we acquire knowledge of what syllables and words exist. In the present studies, the CV tokens were common syllables used in the American English language and uttered by a human native speaker of American English. The listeners were also native speakers of American English. The knowledge of the language might have helped the listeners to group the tokens even when segregation occurred. To support this hypothesis, Jusczyket al. (1994) showed that infants learn the "rules" governing permissible sequences of phonemes in a given language allowing to compose words. Each language permits different sequences, for instance the syllable / zb / is not found in English but is common in Polish. Infants are able to use the transitions probabilities between adjacent tokens to detect words. They learn by being exposed to the right lexicon inherent to their native language (for a review, see [START_REF] Kuhl | Early language acquisition: cracking the speech code[END_REF]. Another possible explanation might come from the formant transition. For example, [START_REF] Whalen | Perception of the English /s/-/∫/ distinction relies on fricative noises and transitions, not on brief spectral slices[END_REF] showed that a whole fricative noise and the formant transition are needed for the listener to identify the fricative. Another example, taken from [START_REF] David | Sequential stream segregation of voiced and unvoiced speech sounds based on fundamental frequency[END_REF], showed that the voiced portion of a CV token does not carry enough information about the consonant to enable an accurate identification required in a streaming task (see experiment 3). It has also been shown that the recognition of monosyllabic speech sounds requires the adjacent formant transition [START_REF] Lindblom | On the Rôle of Formant Transitions in Vowel Recognition[END_REF]. [START_REF] Stachurski | The verbal transformation effect and the perceptual organization of speech: Influence of formant transitions and F0-contour continuity[END_REF] investigated the importance of formant transition and continuity in F0 contours on the binding of speech sounds. They found that both cues help to keep the perceptual coherence of speech sounds. In addition, [START_REF] Cole | Perception of temporal order in speech: the role of vowel transitions[END_REF], in a task where listeners had to repeat the order of a sequence of speech sounds, found that vowel transitions help to preserve the temporal order of the sequences, so to preserve the coherence of natural speech. These findings concern mostly speech understanding. However, we can assume that stream segregation requires speech identification, and thus these results can be extended to streaming.

The two explanations, top-down process and formant transitions, are not exclusive. Indeed, [START_REF] Wagner | Formant transitions in fricative identification: The role of native fricative inventory[END_REF] investigated the use of formant transition as a function of the spectral content of the fricatives present in the native language of the listeners. They tested five different languages: Dutch, English, German, Polish and Spanish. The participants had to recognize a fricative target in pseudowords, the fricatives being preceded and followed by the vowels / a i u /. The results showed that depending on the native language, the listeners paid or did not pay attention to the formant transitions.

Indeed, for the languages where there are no spectral similarities in the fricatives (e.g., Dutch and German), the listeners were not affected by a misleading formant transition. However, for the languages presenting spectral similarities in some fricatives (e.g., / f / and / th / in English and Spanish), the listeners appeared to pay attention to formant transitions. Further investigation would be required to understand the binding of speech in languages where listeners do not pay attention to the formant transitions (e.g., Dutch or German).

When using synthesized vowels, [START_REF] Gaudrain | Streaming of vowel sequences based on fundamental frequency in a cochlear-implant simulation[END_REF] observed that different formants of the same vowel were segregated to form separated auditory streams. This outcome might be explained by the micro-modulations of frequency and amplitude present in the natural utterance, which is missing in the synthesized vowels. Besides, for naturally uttered sequences of vowels, where the spectral cues varied across time, it was found that segregation was not based on the spectral formant cues (see [START_REF] Gaudrain | Streaming of vowel sequences based on fundamental frequency in a cochlear-implant simulation[END_REF][START_REF] Gaudrain | Effect of spectral smearing on the perceptual segregation of vowel sequences[END_REF]. Indeed, if segregation was based on vowels' spectral similarities, it would lead to a grouping unfavourable for a coherent speech perception (one stream of /a /, another stream of / i / etc… instead of different streams of meaningful syllables). The same is true for the organization of speech syllables like the CV tokens tested in the previously detailed studies. In this situation, the auditory system seems to be able to ignore the spectral variations inherent to speech, and thus grouping in speech may have rely on top-down information.

Fig. 1 :

 1 Fig.1: Schematic diagram of the tokens in the across-sequence task. From left to right the diagrams represent the full repeat, the consonant-repeat and the vowel-repeat.

Fig. 2 :

 2 Fig. 2: Mean FA and Hit rates for the across-sequence task for the first study (left panel) and for the second study (right panel).

Acknowledgments

This work was supported by NIH grant R01 DC007657 and R01 DC016119 (AJO), Erasmus Mundus Auditory Cognitive Neuroscience travel award 22130341 (MD) and LabEX CeLyA ANR-10-LABX-0060/ANR-16-IDEX-0005 (ML, NG). We thank Steven van de Par for helpful discussions.