Pramod Kaushik

Jérémie Naudé

Surampudi Bapi Raju

Frédéric Alexandre
email: frederic.alexandre@inria.fr

A VTA GABAergic computational model of dissociated reward prediction error computation in classical conditioning

Classical Conditioning is a fundamental learning mechanism where the Ventral Striatum is generally thought to be the source of inhibition to Ventral Tegmental Area (VTA) Dopamine neurons when a reward is expected. However, recent evidences point to a new candidate in VTA GABA encoding expectation for computing the reward prediction error in the VTA.

In this system-level computational model, the VTA GABA signal is hypothesised to be a combination of magnitude and timing computed in the Peduncolopontine and Ventral Striatum respectively. This dissociation enables the model to explain recent results wherein Ventral Striatum lesions affected the temporal expectation of the reward but the magnitude of the reward was intact. This model also exhibits other features in classical conditioning namely, progressively decreasing firing for early rewards closer to the actual reward, twin peaks of VTA dopamine during training and cancellation of US dopamine after training.

Introduction

The phasic firing activity of midbrain dopamine neurons is believed to encode a reward prediction error, which can guide learning and serve as an incentive signal. In his famous experiment, Pavlov observed that if food follows the ring of a bell, a dog comes to salivate after the bell is rung. This process is called classical (or pavlovian) conditioning: an unconditioned response (salivation) originally associated with an Unconditioned Stimulus (US, the food) becomes conditionally elicited by a Conditioned Stimulus (CS, the bell ring). Schultz and collaborators examined the activity of midbrain dopamine neurons in primates, during a classical conditioning task similar to Pavlov's. They observed that originally, midbrain dopamine neurons responded with a burst of spikes to unexpected primary rewards (juice/water dripped in the mouth of thirsty primates), i.e. an US. After the primates learned the association between a tone CS and the reward US, dopamine cells responded to the reward-predicting CS. Moreover, these neurons stopped responding to US whose arrival was expected, being predicted by the CS. When cued rewards failed to be delivered, a large percentage of dopamine neurons revealed a brief pause with respect to their background firing at the moment of expected reward.

This predictive behavior of VTA Dopamine neurons was linked to the Temporal Difference Learning algorithm in Reinforcement Learning [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. The TD algorithm predicts a future reward that occurs at a specific state ahead of time after a given number of trials. This prediction is arrived through the computation of a reward prediction error (RPE) that enables learning the value of each state. It is represented by the equation:

= + ̄ +1 -̄ (1)
where is the reward (return) on time step . Let ̄ be the correct prediction that is equal to the discounted sum of all future rewards. The discounting is done by powers of factor of such that reward at distant time step is less important as compared to recent rewards.

The TD algorithm is powerful in its heuristic value, as it links the activation of dopamine cells with classical conditioning. However, some properties of TD models are inconsistent with the electrophysiological data, and it lacks the biological realism needed to explain the mechanisms through which RPE-like activity emerges in dopamine cells. Most inconsistencies are concerned with the way time is represented in the TD model. Whereas the TD error signal travels back in time from US to CS when learning progresses, it has been reported in [START_REF] Schultz | A neural substrate of prediction and reward[END_REF] that US-related activation of VTA slowly decreases while the CS-related one increases. Ramping dopamine activity during the CS-US interval has been proposed to reflect back-propagating error signals averaged over trials [START_REF] Niv | Characterization of oculomotor and visual activities in the primate pedunculopontine tegmental nucleus during visually guided saccade tasks[END_REF] but this ramping can be observed in individual trials and is actually related to reward uncertainty [START_REF] Fiorillo | Discrete coding of reward probability and uncertainty by dopamine neurons[END_REF] [START_REF] Fiorillo | Evidence that the delay-period activity of dopamine neurons corresponds to reward uncertainty rather than backpropagating TD errors[END_REF]. More importantly, when US is delivered earlier than predicted, VTA dopaminergic neurons are activated at the actual time of the US but not at the usual time of reward [START_REF] Hollerman | Dopamine neurons report an error in the temporal prediction of reward during learning[END_REF], contrarily to what is predicted by TD. Critically, when striatum is lesioned, experiments [START_REF] Takahashi | Temporal specificity of reward prediction errors signaled by putative dopamine neurons in rat VTA depends on ventral striatum[END_REF] show that VTA dopaminergic neurons signal a RPE when reward magnitude changes, but not when time of the reward is modified. This suggests that learning about reward timing is computationally and anatomically separated from learning about reward magnitude, i.e. a completely different implementation of reinforcement learning than is usually considered [START_REF] Joel | Actor-critic models of the basal ganglia: New anatomical and computational perspectives[END_REF].

Here we sought a system-level account of how the CS-US interval duration and the value of the reward are separately learned, and how these two features are combined to give rise to dopaminergic phasic activity. In particular, we considered two components of the meso-limbic loops that have been overlooked in previous models: VTA GABA cells and neurons from the Pedunculopontine Nucleus (PPN). We propose that VTA GABA neurons provide inhibitory drive onto VTA DA neurons [START_REF] Eshel | Arithmetic and local circuitry underlying dopamine prediction errors[END_REF] , display persistent firing during the CS-US interval [START_REF] Cohen | Neuron-type specific signals for reward and punishment in the ventral tegmental area[END_REF] and are necessary to compute the RPE in DA cells [START_REF] Eshel | Arithmetic and local circuitry underlying dopamine prediction errors[END_REF]. However, here we only include positive reward predictions and exclude omissions as VTA GABA neurons have been found to be too weak for reducing baseline dopamine firing [START_REF] Eshel | Arithmetic and local circuitry underlying dopamine prediction errors[END_REF] and we hypothesize that they are being handled by the Ventral Pallidum and the Rostromedial Tegmental Nucleus (RMTg) pathway. Neurons from the PPN project to both VTA DA and GABA neurons and have been found necessary for appetitive conditioning [START_REF] Yau | Pontomesencephalic tegmental afferents to VTA non-dopamine neurons are necessary for appetitive Pavlovian learning[END_REF]. PPN neurons are classically believed to signal the delivery of actual reward [START_REF] Vitay | Timing and expectation of reward: a neuro-computational model of the afferents to the ventral tegmental area[END_REF]. However, during conditioning, two types of response are recorded from neurons from the PPN : neurons responding to the actual reward magnitude, and a persistent neuronal firing during the CS-US interval, reflecting the prediction or expected reward magnitude [START_REF] Niv | Characterization of oculomotor and visual activities in the primate pedunculopontine tegmental nucleus during visually guided saccade tasks[END_REF] [START_REF] Okada | Reward prediction-related increases and decreases in tonic neuronal activity of the pedunculopontine tegmental nucleus[END_REF]). Here we show that a system-level computational model can account for these yet-unexplained physiological and lesion data. We propose that the Ventral Striatum learns the reward timing and the Amygdala the reward magnitude, which is then transferred to the PPN. Expected reward timing and magnitude are subsequently combined at the level of VTA GABA cells to compute the expectation term needed by VTA DA neurons to generate a prediction error. We furthermore provide testable predictions for future experiments, on the role of PPN and VTA GABA in classical conditioning.

Results

We built a system-level network in which reward prediction error (RPE) emerges during learning, from the interaction of the neruronal ensembles repersenting RPE computation. To illustrate how RPE is computed in the network, we subjected it to in silico experimental scenarios similar to a conditioning task. This analogue of classical conditioning consists in the repeated pairing of a CS with the US, separated by a fixed interval duration. The in silico experiments below examine how the value (magnitude) of the US reward and the duration of the CS-US interval are learned separately, and combined by VTA GABA cells. In turn, VTA GABA neurons provide the expectation term used to cancel the excitation of dopamine cells by the US, at the time of the expected reward [START_REF] Cohen | Neuron-type specific signals for reward and punishment in the ventral tegmental area[END_REF] [START_REF] Eshel | Arithmetic and local circuitry underlying dopamine prediction errors[END_REF]. Finally, we show how this new model provides a better description of experimental data that have been left unexplained yet, i.e. how dopamine cells respond to rewards delivered earlier than expected [START_REF] Fiorillo | The temporal precision of reward prediction in dopamine neurons[END_REF], and how value and timing are dissociated by VS lesions [START_REF] Takahashi | Temporal specificity of reward prediction errors signaled by putative dopamine neurons in rat VTA depends on ventral striatum[END_REF].

Model Architecture

The system-level computational model attempts to explain how the dopamine reward prediction error is computed in appetitive conditioning in the VTA through understanding the roles of VTA GABA and Peduncolopontine (PPN) neurons. The model is shown in Figure 1. The model focuses on the computation inside the VTA carried out by two populations viz., the VTA DA and VTA GABA neurons. Lateral Hypothalamus (LH) projects to PPN RD and VS and these in turn project to VTA DA and VTA GABA respectively. When reward is delivered, it is reported to fire the Lateral Hypothalamus (LH) and activates the LH → PPN RD (Reward delivery) → VTA Dopamine pathway resulting in US dopamine firing prior to any sort of learning [START_REF] Semba | Afferent connections of the laterodorsal and the pedunculopontine tegmental nuclei in the rat: a retro-and antero-grade transport and immunohistochemical study[END_REF][START_REF] Lokwan | Stimulation of the pedunculopontine tegmental nucleus in the rat produces burst firing in A9 dopaminergic neurons[END_REF]. Basolateral Amygdala (BLA) learns the magnitude of the US through the projections from VTA DA to BLA, which signal a reward prediction error that modulates synaptic plasticity. A pathway from LH to BLA learns that BLA firing for CS has the same amplitude as the US-induced firing [START_REF] Sah | The amygdaloid complex: anatomy and physiology[END_REF]. The immediate firing in response to the CS occurs through the BLA recognizing the cue encoded by the Infero-temporal cortex (IT), with BLA activating the VTA DA through the BLA→ CE → PPN RD → VTA DA pathway. The CS has been assumed to be encoded through the IT in this model though there could be other candidates and this connectivity is not central to the functioning of the model. The Central Nucleus of the Amygala (CE) has excitatory projections on PPN FT(Fixation Target) [START_REF] Okada | Reward prediction-related increases and decreases in tonic neuronal activity of the pedunculopontine tegmental nucleus[END_REF] [START_REF] Kobayashi | Reward prediction error computation in the pedunculopontine tegmental nucleus neurons[END_REF] which displays persistent activity unless strongly inhibited. The projections from PPN to VTA are functional in nature in the model with empirical evidences lacking so far. The LH also projects to the VS, which is also modulated by VTA DA neurons. Finally, excitatory projections from PPN FT and inhibitory projections from VS neurons to VTA GABA enable the final reward cancellation of VTA DA observed in electro-physiological experiments.

Control Scenario

In the following, we show how a reward prediction error emerges in VTA Dopamine cells with learning, as a consequence of network dynamics and plasticity.

• Initial Trial

The arrival of an unexpected reward induces a firing in the LH neurons. This LH firing subsequently activates VTA Dopamine neurons, through the PPN RD neurons (Figure 2 This corresponds to the activity of the Ventral Striatum reaching its minimum at the exact moment of reward. Hence, the VS does not exert any inhibition at the end of the interval, which results in the inhibition of VTA DA neurons at the expected time of the US. However, as the US magnitude is not fully learned yet, the activity of PPN FT to VTA GABA pathway only results in partial cancellation of VTA DA activity upon US delivery. This is consistent with experimental results on Partial conditioning [START_REF] Pan | Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network[END_REF], and designed as a partial expectation in the TD framework. (Figure 2 B bottom panel Trial 7). This partial expectation consists in a twin peak of VTA firing, at the respective times of the CS and the US.

• Complete Reward Cancellation

The final state of the circuit, after 16 CS-US pairings, is given in Figure 2 A. The magnitude of expectation originates from the CS firing in the Central Amygdala (CE) and is maintained in the PPN FT through a self-sustaining mechanism [START_REF] Okada | Reward prediction-related increases and decreases in tonic neuronal activity of the pedunculopontine tegmental nucleus[END_REF]) [START_REF] Kobayashi | Reward prediction error computation in the pedunculopontine tegmental nucleus neurons[END_REF]. At the end of learning, the BLA neurons have reached an asymptote in their firing, which encodes for reward magnitude. Hence, PPN FT neurons display maximum tonic activity after the presentation of the CS. In parallel, as in partial conditioning, the presentation of the CS, which fires the IT and thereby the Orbitofrontal Cortex (OFC) [START_REF] Carmichael | Sensory and premotor connections of the orbital and medial prefrontal cortex of macaque monkeys[END_REF], activates the VS that encodes the interval timing. It acts similar to a negative integrator and progressively lowers the inhibition that VS exerts on PPN FT, to reach zero inhibition at the expected time of the reward. The magnitude of expectation originates from the CS firing in the Central Amygdala (CE) and maintained in the PPN FT through a self sustaining mechanism (Okada and Kobayashi, 2013) [START_REF] Kobayashi | Reward prediction error computation in the pedunculopontine tegmental nucleus neurons[END_REF]. The GABA firing in the VTA is reflective of this [START_REF] Yau | Pontomesencephalic tegmental afferents to VTA non-dopamine neurons are necessary for appetitive Pavlovian learning[END_REF] and the PPN FT integrates the magnitude from the Central Amygdala (CE) and timing information from the VS to achieve the ramping signal that encodes both time and magnitude of the reward delivery.

Variability in Magnitude and Time

1. Variability in Timing When a reward is delivered earlier than expected (i.e. with a shorter delay than the CS-US interval that has been learned), US firing is observed in VTA dopamine neurons. More precisely, in this case of earlier-than-expected US reward, VTA dopamine neurons fire less than the initial (before learning) firing observed at US delivery. This is consistent with the US being expected, albeit not at this precise timing. An interpretation would be that partial expectations are generated during the CS-US interval, hence the reward prediction decreases with time until the expected timing of US delivery. In our model, we observed that the earlier the reward was delivered, the higher was the VTA DA firing (Figure 3 B). VTA DA firing in Figure 3 A (middle left panel) shows a reward delivered before the half-way-point (at the 100 ℎ time step) evoking a dopamine firing, but less than the firing initially induced by an unpredicted reward. By contrast, Figure 3 B (middle left panel) shows an early reward delivered after the half-way-mark (at the 300 ℎ time step), which induces lesser firing in VTA DA cells. The model is consistent with physiological data in primates, where earlier-than-expected rewards evokes progressively less firing as the reward delivery time increases [START_REF] Fiorillo | The temporal precision of reward prediction in dopamine neurons[END_REF]. Our model provides a mechanistic explanation for this data.

Variability in Magnitude

VTA Dopamine firing reflects the difference between the actual reward and the expected reward. For instance, VTA Dopamine neurons fire on US arrival if a reward is larger than expected (Figure 4 C middle panel). This corresponds to a positive reward prediction error, in accordance with physiological data. The subtractive nature of inhibition [START_REF] Eshel | Arithmetic and local circuitry underlying dopamine prediction errors[END_REF] encodes the difference of magnitude between the actual magnitude of reward and the expected magnitude of reward ( Figure 4C middle panel).

Dissociation of time and magnitude prediction errors following VS Lesions

VS lesion affects time prediction error

When an experimental lesion is made in the VS from rats [START_REF] Takahashi | Temporal specificity of reward prediction errors signaled by putative dopamine neurons in rat VTA depends on ventral striatum[END_REF] , earlier-thanexpected US does not trigger firing in VTA Dopamine cells. Accordingly, our model reproduces this lack of timing prediction error, as the virtual lesion of the VS in the model abolishes the VTA Dopamine response to earlier-than-expected US. Moreover, VTA GABA cells, instead of displaying a ramping signal, provide a constant tonic inhibition, similar to PPN FT, throughout the duration of the trial (Figure 4 A middle right panel). Indeed, the model posits that lack of dopamine firing for the VS-lesioned scenario compared to the control scenario is due to the higher inhibition from the VTA GABA neurons (Figure 4 A middle right panel). The VS inhibitory signal acts as a "when" signal providing VTA GABA neurons with information enabling computation of reward expectation at a given point in time. Due to this information being lost because of lesion, VTA GABA neurons do not have a ramping signal that peaks at the right moment. Instead, they have a constant inhibition throughout the interval.

VS lesion does not alter magnitude prediction error

The rewards with higher magnitude induce firing in VTA DA cells even if the VS is lesioned. A reward that is double the magnitude triggers the same effect on US VTA Dopamine as in the control scenario (Figure 4 B middle left panel).

Since the VS encodes only temporal information, its lesion does not affect the magnitude encoding of the stimuli. This results in VTA DA firing showing the same subtractive effect as in the control scenario(Figure 4 B middle left panel). This is due to the VTA GABA encoding the magnitude from the BLA through the PPN FT neurons. A lack of VTA GABA ramping does not prevent the VTA DA from having the magnitude encoded within its population, and provides the same RPE at usual reward time. This behavior of VTA DA neurons in the case of a VS lesion with larger magnitude is also in accordance with experimental results [START_REF] Takahashi | Temporal specificity of reward prediction errors signaled by putative dopamine neurons in rat VTA depends on ventral striatum[END_REF].

Discussion

Calculation of RPE in the model

The model aims to provide an alternative view of how RPE is calculated in the brain. Looking at the model with a functional lens, the VTA GABA signal of the model looks very much like the expectation signal in Temporal Difference (TD) Learning but differs from TD in how this final expectation is GABA is no longer able to integrate the timing information coming from VS and just reflects the projection from PPN FT neurons. Hence, early rewards are treated as any other reward that comes at the usual time, reflecting no firing (left panel). But when the reward is of a higher magnitude (middle panel), it is able to indicate the prediction error just as in the control scenario (right panel) where VS is not lesioned and VTA GABA integrates both magnitude and timing.

calculated. Unlike TD, the expectation signal is a resultant computation derived from calculation of time and calculation of magnitude. The calculation of time involves learning a single parameter, the slope, that learns an interval between two points (here the CS and the US) and calculation of magnitude involves learning the amplitude of the US (encoded as persistent firing). The final expectation calculated in VTA GABA of the model is the interaction between a slope like signal of time and the persistent amplitude of magnitude giving rise to a TD like expectation signal.

Role of Timing in conditioning

The importance of CS-US interval timing is one of the key postulates of the proposed theory underlying the model. In this model, the interval timing mechanism uses the dopamine signal since the onset of the CS to learn the underlying temporal distribution to predict the arrival of the reward, however it needn't be the case. Learning the interval time separating the CS and the reward could happen without dopamine with a phenomenon called sensory-preconditioning [START_REF] Sadacca | Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework[END_REF]. Our model predicts that since time and magnitude are separate signals in the brain, the learning of time precedes the learning of magnitude for reward prediction error to take place. Interval timing learning in animals has been observed to happen in very few trials and sometimes even one. The model posits that interval timing learning is an integral part of reward prediction error computation in appetitive learning and the learning of timing happens before the magnitude of the stimulus so as to construct an inhibitory signal that has a ramping nature before it ramps to its minimum amplitude.

The timing mechanism used in the model is very simplistic, reflecting the learning of a single parameter to get the final slope of inhibition. Here, the ability of the model to learn the timing arises from learning the slope of the interval timing function using a geometric learning rule (refer to Ventral Striatum and OFC section in Methods for more details) and this learning rule is robust for different intervals (Rivest and Bengio, 2011). The slope is modelled as a negative ramp in the striatum acting as disinhibition on the VTA GABA neurons. Emmons et al. (2017) report such ramping activities in the striatum for the control of temporal processing. It is possible that more complex timing mechanisms are incorporated in the striatum to accommodate stochasticity in the temporal interval. This is consistent with the original Temporal Difference Reinforcement Learning (TDRL) representation of dopamine where the cue is tracked since the onset of its presentation, state by state until reward is delivered as in the complete serial compound (CSC) stimulus representation [START_REF] Schultz | A neural substrate of prediction and reward[END_REF]. Here too, the state is tracked at each time point using the timing signal. However, unlike the TD model , which is model-free and tracks value across states, this model learns a separate value signal for state and reward and state representation are computed separately unlike TD-Learning. This dissociation enables the model to learn changes in magnitude independent of the state and learn the state independent of the magnitude.

Dual pathway model

The dual pathway models of classical conditioning posit that the mechanism with which the CS firing occurs at the onset of a trained cue is dissociated from the expectation that inhibits the reward signal at the time of the reward. Our model too, largely follows the same pattern but with some deviations (O'reilly et al., 2007). This model replicates the observation in Fiorillo et al. (2008) about early rewards delivered at different time points exhibiting different firing. The interpretation of this model is that, this dopamine error is indicating a mismatch in state rather than a mismatch in magnitude. Since magnitude is a separate signal and does not require updating, the early reward firing is indicative of a violation of a belief of the state the animal is in. This prediction error is predicted to happen between the striatal inhibition and VTA GABA. This inhibition is also predicted to play a role in the lesion experiments done by Takahashi et al. (2016) where VS lesions hamper the ability of the animal in tracking state. This lesion inturn has an effect on VTA GABA which is no longer inhibited and only serves the magnitude component of the reward prediction error. Hence, early reward prediction errors no longer happen, because the animal does not recognize state prediction errors and expects the reward to happen all the time. It was also found in the study that VTA non dopamine neurons have a higher firing after the VS lesions consistent with our model and our model predicts, VTA GABA to have higher firing when VS is lesioned. This model hypothesis a continuous ramping signal that is active throughout the duration of the CS and the US, peaking at the time of the US and speculates that this inhibition signal and the CS firing could have the same source.

Redundancy in RPE signals

Though the model architecture shows a circuit-wise strict delineation between expected magnitude and expected timing, there could be a degree of mixing between these signals. Indeed, on a general note, we relied on the assumption that each structure constitutes a module that computes a variable (though not necessarily a term of the classical RPE equation). However, the way RPE is computed in dopamine cells seems more mixed [START_REF] Tian | Distributed and mixed information in monosynaptic inputs to dopamine neurons[END_REF]. The multiple areas that send inputs to dopamine neurons do not encode "pure" variables (reward, expectation etc. . . ) needed to compute to the RPE, but rather a mixture of these variables (and/or mixed with a RPE signal) suggesting some redundancy in RPE computation [START_REF] Hunt | A distributed, hierarchical and recurrent framework for reward-based choice[END_REF]. At face value, this would suggest that instead of VS purely encoding expected timing and PPN purely encoding expected magnitude, these circuits would encode heterogeneous signals of both reward and timing expectations. However, more neurons in the LH seem to encode reward and not expectations (as in our model), while striatal cells encode more expectations and less reward (as in our model) [START_REF] Tian | Distributed and mixed information in monosynaptic inputs to dopamine neurons[END_REF]. Hence, the functional separation between timing and magnitude might not be as stringent as in our model, but there would still exist a strong bias in VS and PPN signals towards encoding timing and magnitude, respectively.

Heterogeneity of PPN

Though much of PPN's anatomical and chemical characteristics are unknown, studies have shown functional differences between populations inside PPN. Specific populations within PPN exist which fire phasically for rewards and others which have sustained firing from reward prediction to delivery of reward, and the activation sustaining till the delivery of the reward even if the arrival of the reward is delayed [START_REF] Niv | Characterization of oculomotor and visual activities in the primate pedunculopontine tegmental nucleus during visually guided saccade tasks[END_REF]. Moreover these neurons seem to encode amount of reward firing higher for rewards with larger magnitude [START_REF] Niv | Characterization of oculomotor and visual activities in the primate pedunculopontine tegmental nucleus during visually guided saccade tasks[END_REF] [START_REF] Hong | Pedunculopontine tegmental nucleus neurons provide reward, sensorimotor, and alerting signals to midbrain dopamine neurons[END_REF] portraying a graded firing signal capable of differentiating reward amounts consistent with this models characteristics. Studies have also pointed out a growing role for PPN projections to VTA non-dopamine neurons as necessary for appetitive pavlovian conditioning [START_REF] Yau | Pontomesencephalic tegmental afferents to VTA non-dopamine neurons are necessary for appetitive Pavlovian learning[END_REF] and activation of PPN glutamate neurons to be reinforcing [START_REF] Yoo | Activation of pedunculopontine glutamate neurons is reinforcing[END_REF]. The hypothesis of this model is that a subset of PPN neurons (PPN FT) convey reward magnitude information to the VTA GABA neurons and any optogenetic silencing of these neurons can interfere with the computation of reward prediction error, though isolating these neurons could prove difficult owing the structures heterogeneity.

VTA GABA theory of computing RPE in classical conditioning

Previous studies have implicated the striatum [START_REF] Usuda | Efferent projections of the nucleus accumbens in the rat with special reference to subdivision of the nucleus: biotinylated dextran amine study[END_REF] as the source of the inhibitory signal cancelling the dopamine. But, recent projection-specific activation by optogenetic studies among others have shown that the inhibition from striatum has weak to no inhibitory effects on DA neurons when stimulating direct striatal inputs on DA neurons in the VTA [START_REF] Keiflin | Dopamine prediction errors in reward learning and addiction: from theory to neural circuitry[END_REF] [START_REF] Bocklisch | Cocaine disinhibits dopamine neurons by potentiation of GABA transmission in the ventral tegmental area[END_REF] [START_REF] Chuhma | Functional connectome of the striatal medium spiny neuron[END_REF] [START_REF] Xia | Nucleus accumbens medium spiny neurons target non-dopaminergic neurons in the ventral tegmental area[END_REF]

(Klein-Flügge et al., 2011).

There have been a number of recent results, suggesting an alternate pathway within VTA which might be responsible for this inhibitory signal. Moreover, optogenetic studies done on VTA [START_REF] Cohen | Neuron-type specific signals for reward and punishment in the ventral tegmental area[END_REF] [START_REF] Eshel | Dopamine neurons share common response function for reward prediction error[END_REF] have pointed out not only does the VTA GABA neurons exert enough inhibition to cancel VTA DA neurons but the inhibition is also subtractive in nature and hence suitable for computation of reward prediction error [START_REF] Eshel | Arithmetic and local circuitry underlying dopamine prediction errors[END_REF]. This model hypothesises that the ramping nature of expectation which encodes both magnitude of the stimulus and time of arrival is encoded in the VTA GABA which acts as the site of integration between these two different dimensions of the reward. One possible explanation of this distributed nature of reward prediction error could be that this is what allows for rapid recomputation of values (either of time or magnitude) and allows the animal to exhibit and sometimes fast, adaptive behavior. Parallels could be drawn with the literature of reinforcement learning that the animals are not purely engaged in model-free reinforcement learning and that the dopamine signal itself could be not just performing reward prediction errors and differences in timing could elicit an error from the dopamine system for state prediction errors. For example, dopamine firing for early reward delivery could be interpreted as a state prediction error where the animals has to reevaluate the time of the reward rather than the magnitude of the reward. The precise interpretation of the dopamine prediction error could be handled by the upstream areas to determine what computations are to be done to reflect a changed scenario. Here we did not consider a role for VTA GABA in reward omission nor in the extinction of conditioning. Indeed, VTA GABA firing does not seem to code for the absence of reward [START_REF] Cohen | Neuron-type specific signals for reward and punishment in the ventral tegmental area[END_REF]. Instead, there may be an asymmetrical computation for omissions compared to reward, with the former likely driven by the Lateral Habenula, thus forming a separate circuit [START_REF] Baker | The lateral habenula circuitry: reward processing and cognitive control[END_REF].

This model examines the role of VTA GABA in computing the reward prediction error along with a few other substrates based on some of the results provided by Takahashi et al. (2016). This paper adopts a semi-markov approach to explain the findings while the model given here attempts to provide a system level model of how the underlying neuronal substrates might act. There are a few other behaviors that is observed in the model. The authors note that removing VS does not remove A Bound function is used when the firing of a population is described with an upper and a lower limit in certain populations.

( ) = ⎧ ⎪ ⎨ ⎪ ⎩ 0 if x < 0 x if 0 < x < 1 1 if x > 1 (6)
A threshold function is also used in some populations and it outputs 1 when the input exceeds a threshold Γ, 0 otherwise:

Δ Γ ( ) = 0 if x < Γ 1 otherwise (7)
The learning rules defined in the model are based on the Hebbian learning rule and a DA modulated learning rule in the case of BLA like the multiplicative three factor learning rule. The evolution over time of the weight ( ) of a synapse between the neuronal population (presynaptic neurons) and the neuronal population (postsynaptic neurons) is governed by:

( ) = ( . ( ). ( )) (8)
where is the weight term, the learning rate, ( ) and ( ) are indicating the firing rates of the presynaptic and postsynaptic neuronal populations, respectively.

Population definitions

Representations of inputs

The sensory inputs of the CS and the reward input of the US are encoded by the inferotemporal cortex (IT) and the lateral hypothalamus (LH), respectively, simply as square wave signals:

( ) = ( ) + (9)
where ( ) is an external input resulting either from a stimulus or from a reward.

Basolateral Amygdala

The BLA receives inputs about the CS from the IT, the US from the LH, as well as VTA DA output. This allows the BLA to learn to associate the CS with the US, thus providing a magnitude expectation. The equation below is the same equation as in Equation 2 without the inhibitory component and with the presence of a tonic to phasic conversion.

.

( ) = (-( ) + , ( (
)) + ( )) (10)
( ) = ( ( )) + with = 10ms, = 10ms, = 1. The CS is learned by updating the synaptic weights between IT and BLA and the learning rule is given by:

( ) = . . ( ).( - ( )) + (11)
Here indicates the presence of the US corresponding to the dopaminergic neuronal modulation from the VTA, is the learning rate equal to 0.003, is the magnitude of LH firing, and are the firing rates of presynaptic and postsynpatic neurons, respectively.

where indicates the presence of dopamine is the learning rate equal to 0.4. The first term decreases the weight based on and the weight keeps decreasing until the bound is reached when Δ Γ ( ( )) becomes greater than 0 at the time of the reward. The correcting update is the second term of the weight updating and the slope is increased with a weight increase encoding the duration of the interval. As a result of this learning process, the weight between OFC and VS will result in VS predicting the occurrence of US and disinhibiting VTA GABA accordingly.

It should be noted that the model postulates that the learning of time happens before the learning of value of the stimulus, i.e., its magnitude.

VTA

The VTA comprises two major neuronal populations, dopaminergic (DA) and gabaergic (GABA), glutamatergic cells representing less than 3 percent of VTA cells. VTA GABA neurons locally inhibit VTA DA neurons and participate in the computation of reward prediction error in VTA DA cells [START_REF] Cohen | Neuron-type specific signals for reward and punishment in the ventral tegmental area[END_REF]. More precisely, VTA GABA neurons display a sustained, slowly-increasing ramping activity during the CS-US interval [START_REF] Eshel | Arithmetic and local circuitry underlying dopamine prediction errors[END_REF] but only significantly affect phasic DA activity (i.e. the RPE) rather tonic DA activity during the interval. We thus modeled the two populations from the VTA as follows.

VTA Dopamine

The VTA dopaminergic (DA) neurons receive excitatory inputs from the PPN RD population, which conveys actual reward and reward prediction from the amygdala; and inhibitory inputs from the VTA GABA cells, which signal reward expectation. The difference between these excitatory and inhibitory inputs constitutes the reward prediction error (RPE) [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] [START_REF] Glimcher | Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis[END_REF]. VTA DA neurons broadcast this RPE to the system. During learning, VTA DA neurons initially fire upon US reward delivery. This US activity progressively gets canceled by VTA GABA signaling the reward expectation, and at the same time, phasic firing upon CS arrival develops with learning. This model is implemented in Python, and uses the DANA library for neuronal computation [START_REF] Rougier | DANA: distributed numerical and adaptive modelling framework[END_REF]. Description of all the other model parameters is detailed in Table 1

  B top panel Trial1, Figure 2 C top panel Trial 1). Hence, the model reproduces that VTA Dopamine neurons fire upon the delivery of an unexpected reward. No activity in BLA and Ventral

Figure 1 .

 1 Figure 1. Model diagram illustrating the neuronal structures and their connections involved in Reward Prediction Error (RPE) computation. Pointed arrows represent excitatory connections, while rounded arrows represent inhibitory projections. Dashed lines represent learnable connections, while solid lines represent fixed connections in the model. The red color indicates the VTA population and the yellow color indicates the PPN population.The blue and green color indicates the source of magnitude and temporal expectation respectively. Ventral Tegmental Area has Dopamine (DA) and GABA populations which play a central role in computing RPE. This RPE is transmitted to Basolateral Amydala (BLA) and the Ventral Striatum (VS) where the modulatory connections undergo change depending on this signal. Both the VTA structures receive inputs from distinct Peduncolopontine (PPN) neural populations. The PPN Reward Delivery (RD) neurons deliver reward to VTA DA neurons from the Lateral Hypothalamus (LH) which receives the unconditioned stimulus (US). The PPN Fixation target (FT) neurons receive their projections from the Central Nucleus (CE) of the Amygdala and these PPN neurons along with Ventral Striatum (VS) project to VTA GABA forming the inhibitory signal that cancels the VTA DA upon reward delivery. The BLA is regarded to learn the association between unconditioned stimulus (US) and the conditioned stimulus (CS) and produces the anticipatory firing in VTA DA through the BLA->CE->PPN RD->VTA DA pathway. The VS is posited to learn the timing of the interval and it has inhibitory projections on VTA GABA.

  Both signals are combined by VTA GABA,the activity of which peaks at the time of the reward (Figure 2 B bottom panel Trial 16), cancelling VTA dopamine, which no longer shows firing at the time of the US when reward arrives through the LH (Figure 2 B top panel Trial 16).

Figure 2 .

 2 Figure 2. (A) The figure represents the firing of the neuronal populations after training following the same color conventions as the model architecture diagram. IT and LH indicate the stimulus and the reward respectively. PPN RD represent the phasic signals received from BLA and the LH at stimulus onset and reward delivery. VTA DA neurons show firing only at the arrival of the CS and VTA GABA neurons have a ramping expectation signal that fully cancels the VTA DA signal at reward delivery. BLA and VS have encoded magnitude and timing of the reward respectively. (B) This figure portrays the evolution of the VTA and PPN sub-populations throughout the duration of trials. VTA DA shows firing only at the reward delivery during initial trials while VTA GABA shows no firing due to absence of any expectation at this stage. At Trial 7, VTA DA shows twin peaks reflecting both a partial encoding at stimulus arrival and partial cancellation of reward signal. VTA GABA on the other hand has a ramping nature at the precise time of the arrival of the reward indicating encoding of timing and partial encoding of expectation. The final trial diagram shows the complete cancellation of the VTA DA signal at reward delivery and a complete encoding of stimulus at the CS arrival. Correspondingly, VTA GABA has a larger expectation signal that ramps that has completely encoded the magnitude of the reward signal. (C) shows the evolution of the PPN sub-populations during the sequence of trials with PPN RD signalling reward delivery from LH to VTA DA during the initial trials and PPN FT not showing any sign of expectation at the same time. In the middle of training, PPN RD reflects a partial firing for stimulus onset and the same reward signal firing as during the initial trials. PPN FT also reflects this partial firing with a tonic nature of firing that passes onto VTA GABA. During the final trial stages, PPN RD firing peaks for its stimulus arrival firing while a bigger tonic firing signal is observed for PPN FT. (D) portrays the evolution of VTA DA CS and US signals and that of VTA GABA. Both VTA GABA and VTA DA CS show increased firing across the trials while VTA DA US progressively reduces ultimately to the background firing rate of VTA DA. All the figures are averaged for 10 runs.

Figure 3 .Figure 4 .

 34 Figure 3. These figures indicate the early firing scenarios on VTA and PPN sub-populations. LH indicates the reward delivered earlier than usual and IT the stimulus. (A) denotes the arrival of the reward at 100 ℎ time step after training where VTA DA shows some firing compared to no firing after training when reward is delivered at the usual time. (B) This firing for an earlier reward is still larger than a later arrival of early reward at 300 time steps where VTA DA barely shows any firing. (C) indicates the progressively later "early" rewards fire less as early reward delivery times get closer to usual reward arrival times in accordance with data on[START_REF] Fiorillo | The temporal precision of reward prediction in dopamine neurons[END_REF]

  reward delivery from VTA dopamine to VS or LH to VS with Γ threshold function reaching 1 for any value greater than 0. () is a bounded function.

Figure 5 .

 5 Figure 5. The slope is decreased at every iteration until it exceeds the duration (the red line) enabling exact correction of the weight encoding the duration to be found (the black line). The colors indicate the progressive iterations

  is the baseline firing rate of the VTA Dopamine equal to 0.2 VTA GABA VTA GABA neurons combine inputs from the VS, which encodes the expected time of reward, and from the PPN, which signals expected reward magnitude. VTA GABA neurons thus encode reward expectation and inhibit VTA DA neurons. The equation for membrane potential is the same as in Equation 2 without baseline firing and population dynamics follows Equation 3 with = 20ms.

Table 1 .

 1 . The model can be accessed in the following link : https://github.com/palladiun/Pavlovian-Conditioning-VTA-GABA Table describing network architecture and parameters used in activation and learning rules.

		Architectural parameters
	Parameter	Meaning	Value
	US input_size	size of input vectors from LH	1
	CS input_size	size of input vectors from IT	4
	VTA Dopamine_size	number of neurons in VTA Dopamine	10
	VTA GABA_size	number of neurons in VTA GABA	5
	BLA_size	number of neurons in BLA	1
	CE_size	number of neurons in CE	1
	OFC_size	number of neurons in OFC	1
	PPN RD_size	number of neurons in PPN RD	4
	PPN FT_size	number of neurons in PPN FT	4
	PPN Magnitude_size	number of neurons in PPN Magnitude	4
		Equation parameters
	BLA_CE	constant weights from BLA to CE	0.15
	LH_PPN_RD	constant weights from LH to PPN_RD	1.2
	LH_BLA	constant weights from LH to BLA	1
	IT_OFC	constant weights from IT to OFC	.25
	CE_PPN_RD	constant weights from CE to PPN_RD	2
	CE_PPN_Mag	constant weights from CE to PPN_Mag	.3
	PPN_RD_PPN_Mag	constant weights from PPN_RD to PPN_Mag	0.8
	PPN_RD_VTA_Dop	constant weights from PPN_RD to VTA_Dop	1
	PPN_Mag_PPN_Rel	constant weights from PPN_Mag to PPN_Rel	0.2
	VS_PPN_Rel	constant weights from VS to PPN_Rel	1
	PPN_Rel_VTA_GABA	constant weights from PPN_Rel to VTA_GABA	0.25
	VTA_Dopamine_BLA	constant weights from VTA_Dopamine to BLA	1
	VTA_Dopamine_BLA	constant weights from VTA_Dopamine to VS	1
	VTA_GABA_VTA_Dopamine constant weights between VTA_GABA and VTA_Dopamine	0.2
	OFC_VS	initial weights between OFC and VS	0.006
	IT_BLA	initial weights between OFC and VS	0.01

Acknowledgments

We would like to acknowledge the following grants which have been a major support to this research.

• Indo-French CEFIPRA Grant for the project Basal Ganglia at Large (No. DST-INRIA 2013-02/Basal Ganglia dated 13-09-2014) • Internships programme at INRIA, 6 month Internship with Team Mnemosyne at INRIA Bordeaux -Sud-Ouest

We also thank Maxime Carrere for helpful discussions.

expectation and the animal in effect expects reward all the time, this could be the VTA GABA signal we observe in the model when VS is lesioned. VTA GABA loses its ramping functionality and has a flat tonic firing pattern carrying on its earlier peak expectation thoughout the duration of the trial, The authors also note that "non-dopaminergic" neurons show significantly higher baseline firing rate when VS is lesion. Our model hypothesises that it is indeed the VTA GABA neurons that are now exhibiting a higher flat expectation due to the VS being lesioned. Thus, the model proposes that it is the VS input to VTA GABA that gives its expectation signal the temporal specificity that the authors mention in their paper.

Methods and Materials

Evaluation of the model

The paradigm used to evaluate the model is a simple CS-US associative learning task and also considers how the expectation cancels out the dopamine peak at the time of the reward. The trial duration is 500 time steps with each time step corresponding to 1ms. The stimulus is presented at the 10 ℎ time step and is kept switched on till the arrival of the reward at the 400 ℎ time step (400ms). The reward and the stimulus have by default a magnitude of 1. The number of trials for the entire conditioning to happen was set at 14 trials (i.e. trials required for the learning algorithm to converge).

Model Description Computational principles

The system-level model is composed of mean-field description of neuronal populations representing distinct, interconnected brain structures. Population dynamics is described by its average firing frequency across time ( ), which is taken as the positive part of a membrane potential ( ), represented by the following equations:

Here is the time constant of the cell, is the baseline firing rate and ( ) is the additive noise term chosen randomly at each time step from a uniform distribution between -0.01 and 0.01. The incoming afferent synaptic currents and ℎ represent the weighted sum of excitatory and inhibitory firing rates, respectively, the weight representing the synaptic weights between the populations.

Some of the neuronal populations extract a short-term phasic activity from their incoming inputs, by removing out the tonic component of the input. This is done by the following equations:

Here ( ) integrates the incoming input ( ) with a time constant and thus represents the tonic component of the input, while , ( ( )) represents the positive part of the difference between ( ) and ( ). Hence, The constant controls how much of the original input is kept, a value of 0 indicates the entire synaptic input is to be preserved and a value of 1 outputs the phasic component only, i.e. the entire tonic component has been entirely removed.

Central Amygdala

The CE is the output nuclei of the amygdala and it projects to both the PPN nuclei, relaying information from the BLA. The CE projects to the PPN RD neurons that convey US and CS firing to the VTA dopamine neurons and PPN FT neurons that convey reward expectation.

The equations for the membrane potential and the firing rate are the same as Equation 10 and Equation 3, respectively, with = 20ms, = 5ms, = 1.

Peduncolopontine nucleus

The PPN has two distinct populations in this model for reward and expectation. The PPN is a heterogeneous structure both in terms of neuronal populations and of responses during classical conditioning [START_REF] Niv | Characterization of oculomotor and visual activities in the primate pedunculopontine tegmental nucleus during visually guided saccade tasks[END_REF]. Hence, we modeled two distinct subpopulations reflecting the two major classes of responses found experimentally: FT (Fixation Target) population, which activates briefly upon CS or US presentation, and RD (Reward Delivery) population, which display sustained activity during the CS-US interval.

PPN RD

The PPN Reward Delivery neurons signal the occurrence of the CS and the US from the CE and the LH, respectively. It also contains a sub-population of inhibitory neurons that inhibits the PPN FT neurons.

The equations for the membrane potential and the firing rate are the same as Equation 10and Equation 3 respectively, with = 5ms, = 5ms, = 1.

PPN FT

The PPN FT neurons encode the magnitude expectation delivered to the VTA GABA neurons. The PPN FT neurons receive information from the CE and are inhibited by the PPN RD neurons. They serve to maintain a constant magnitude that is conveyed to the VTA GABA neurons for final reward prediction error computation. The equations for the membrane potential and the firing rate are the same as Equation 2 without baseline firing and Equation 3 respectively with = 5ms.

Ventral Striatum and OFC

It has long been thought that the Ventral Striatum (VS) is responsible for the reward prediction term in RPE calculation. In our model, the VS encodes the duration of the CS-US interval only. The VS is composed of inhibitory cells, and signals the timing of expected reward to VTA GABA cells through a decrease in activity. The OFC (Orbitofrontal Cortex) relays the presence of the US from the IT to the VS. Then, a simplified timing model comprising a negative integrator similar to the timing algorithm in Rivest and Bengio (2011) signals the interval duration through a slowly decreasing activity until the expected timing of the US. To do so, the integrator here has an amplitude of 1 at the beginning of the trial and after weight updating, decreases its firing to 0 at the precise time of reward delivery. In this framework, learning the CS-US interval consists in adjusting the slope of the slowly-decreasing activity.

Mechanism of timing

The timing mechanism in the VS transforms a tonic excitatory input from the OFC (which is the same as the signal from IT, cf fig 2.A) and transforms it into a decreasing sustained activity, whose slope depends upon the weights between OFC and VS.


R Kilani
email: rim.kiilani@gmail.com

A Zouinkhi
email: ahmed.zouinkhi@enig.rnu.tn

E Bajic
email: eddy.bajic@univ-lorraine.fr

M N Abdelkrim
email: naceur.abdelkrim@enig.rnu.tn

Socialization of Smart Communicative Objects in Industrial Internet of Things

Keywords: Smart object, Internet of Things, Interaction mechanisms, Social Internet of Industrial Things, Social community

In the recent past the Internet of Things (IoT) is moving from interactive objects to smart objects. The number of smart devices or sensors built-in becomes larger. The increase in the number of services exchanged between connected objects has led to the creation of a new concept called Social IoT (SIoT), in which heterogeneous or homogeneous objects can interact and request or exchange services, which makes it possible to include social or human actions in the IoT platform. The use of technologies based on the concept of the Internet of Things in applications and industrial sectors has built a new concept of research called the Industrial Internet of Things (IIoT). The integration of the SIoT concept into the Industrial Internet of Things (IIoT) plays an important role in improving system performance in Industry 4.0 and gives rise to the Social Internet of Industrial Things (SIoIT). In this context, we have revealed a new socioinspired interaction model between industrial communicating objects inspired by sociological approaches that transform objects into socialized industrial communicating objects. These objects form a community, autonomously and dynamically, by exchanging messages to know each other perfectly, and service requests between objects are realized in an adaptive way according to the principles of social interaction governed by socio-inspired strategies and conditions. The model is evaluated using Netlogo multi-agent system simulation environment.

INTRODUCTION

The Internet of Things (IoT) is recent and it is growing exponentially. Various real-world objects are connected through the IoT which facilitates machine-to-machine (M2M) communication through the medium of the Internet [START_REF] Yugha | A survey on technologies and security protocols: Reference for future generation IoT[END_REF]. The importance of IoT that it processes huge amount of data which is detected from various smart devices. The Internet of Things (IoT) combines many diverse technologies and interconnects the various objects of our periphery. In this regard, by using various protocols each object receives its specific address, this object becomes more capable of establishing a relationship with the other objects that surround it [START_REF] Barbin | Navigation in the social internet-of-things (SIoT) for discovering the influential service-providers using distributed learning automata[END_REF]. In these connections, objects can cooperate to achieve a common goal [START_REF] Nitti | The social internet of things (siot)-when social networks meet the internet of things: Concept, architecture, and network characterization[END_REF].

The integration of the concepts of human interactions within the IoT gives rise to SIoT (Social Internet of Things). SIoT is an active research axis in which we find social relationships are established between smart objects and their users [START_REF] Roopa | Dynamic management of traffic signals through social IoT[END_REF]. Social IoT is an emerging IoT paradigm in which smart objects have the power to establish sociable relationships to achieve common or individual interest and perform service exchanges autonomously [START_REF] Lye | User Trajectory Analysis within Intelligent Social Internet-of-things (SIoT)[END_REF]. In addition, SIoT should improve service discovery, information composition and management, and service reliability management (Atzori et al., 2014). To advance and improve the operation of machines in industry, the SIoT concept could be integrated with the Industrial Internet of Things to provide human-to-machine (H2M) communications and machine-tomachine (M2M) interactions. The integration of SIoT in industry is called the Social Internet of Industrial Things (SIoIT) [START_REF] Roopa | Social Interaction-Enabled Industrial Internet of Things for Predictive Maintenance[END_REF].

In this paper, we present sociologically inspired IoT interactions in which, a synthesis of sociological approaches is shown. So, we introduce a new socio-inspired model of the principles of interaction between social industrial objects. The addition of a communicating object to an already stable community is introduced in the following section, presenting the exchange of messages between industrial social objects to determine relations and perform services. The paper ends with a conclusion in which we synthesize our work and identify future work.

SOCIAL INTERACTION BETWEN OBJECTS

Several works in the field of sociology and anthropology have focused on interactions between individuals. The work presented in [START_REF] Sun | A domain based approach to social relation recognition[END_REF], [START_REF] Wagener | Le système interactionnel : connexions sémantiques et contextique relationnelle[END_REF] shows that to achieve their own interests, people act together to achieve common goals through the principle of cooperation. In [START_REF] Foa | Resource theory of social exchange[END_REF], the authors emphasized the principle of authority rank, which translates into the relationship of respect and the principle of sharing in which people share information. In [START_REF] Tischler | Introduction to Sociology[END_REF], the principle of competition is well discussed. It is a form of interaction in modern applications not only in the sports field, but in the market, the educational system, and the political system. In the works presented in [START_REF] Pincus | Interpersonal theory of personality[END_REF] and (Fischer, 2010) we find that with the principle of resistance, the authors show where one does not make concessions to the other. While with the principle of the market price Fiske also introduce the notion of cost [START_REF] Mitchell | Social exchange theory, exchange resources, and interpersonal relationships: A modest resolution of theoretical difficulties[END_REF], in which the interactions are organized according to a common scale of values of the elements exchanged.

After an in-depth study of research work in sociology, we have synthesized 6 sociological principles which make it possible to describe most of the relationships and social interactions between individuals. These principles are: Cooperate (Co), Dominate (Do), Compete (Com), Share (Sha), Resist (Re) and Business (Biz). From these interactions, we propose a transposition of these interactions towards communicating industrial objects. We propose a model that consists of six principles of interactions between smart objects as shown in Figure 1. This model is named codocomshare.biz, in which all objects can establish relationships and services can be performed to other objects according to one of these six principles.

Principles of interaction between objects

From these interactions, we propose a transposition of these interactions towards communicating industrial objects.

Cooperate: All industrial communicating objects cooperate, i.e. they share services and perform them for a common purpose. Each communicating object has well-defined missions. To achieve a common goal, the services are performed unconditionally. We have three cooperation strategies:

-Cooperation between communicating objects without conditions. -Cooperation between communicating objects, in which it is necessary to take into consideration the previous cooperation. In this case, the trust between these communicating objects can be very important factor.

-Cooperation between communicating objects to access a direct or indirect advantage.

Dominate: Industrial communicating objects have a rank of authority. According to the rank of authority, relationships of subordination are imposed and respected in the interactions between communicating objects. For example: the master/slave model can satisfy this type of interaction because it can be seen that only the master can request a service from the slaves who are forced to operate.

Compete: According to the compete principle; there are two ways in which Objects compete to achieve a service. There are two modes for this principle: Several applicants / one executor or One applicant / several executors.

Share: According to the principle of sharing, the sharing of services between all the objects of a community is accepted by all. This exchange can be carried out unconditionally or sometimes with the reciprocity of exchange of services. Sharing strategies are:

-Unconditionally, i.e. fully open.

-Conditioned by the reciprocity of service sharing actions, for example: equivalent to Fiske's Equality Matching.

Resist: Depending on the resistance strategy, a communicating object resists to service execution, it can resist with several strategies: -Delaying service execution.

-Giving partial information.

-Giving obfuscated information.

-Refusing services totally.

Business: According to the business principle, each service has a predefined cost through a common rating system (bit coin, non-fungible NFT token, etc.), so the notion of monetization of the execution of services between communicating objects is introduced. Through the application of Block Chain smart contracts, monetization of services between objects can be automated and decentralized with security and trust.

OPPORTUNISTIC INTERACTIONS BETWEEN INDUSTRIAL OBJECTS

In the interactions between industrial objects, we propose a paradigm shift from deterministic relations to opportunistic relations between more mobile objects. Then each object will be able to have its own interaction strategy n which becomes suitable according to the environment and the strategies of the other intelligent objects that surrounded it.

As shown in Figure 2, the classic M2M interaction is determined with a service request/response. This type of classic exchange is not suitable, so the emergence of the concept of autonomous smart objects is very important. These objects can simulate human behaviour to interact with each other through interaction principles according to conditions and strategies. This interaction is achieved by applying the SIoT concept, i.e. the service response becomes smarter. This communication evolves from classic to innovative.

SOCIAL INDUSTRIAL COMMUNICATING OBJECT

The social interaction between Communicating Industrial Sociable objects must have mechanisms, characteristics, attributes, and rules of behaviour to establish social relations and manage social interactions as shown in Figure 3. To individually differentiate an object from all other objects, an object's attributes determine its own characteristics. Four attributes or identifiers are proposed in [START_REF] Kilani | A unified social behaviourism model for interactions management in Industrial Internet of Things[END_REF]; Object Identifier (ID), Owner ID (OID), Family Code (FC), Application Identifier (AID). These identifiers define the social relations between communicating objects. They are used during social interaction with each other.

Determination of the types of relationships between communicating objects

The approach inspired by human "social relations" and "relational models" shows that the interaction between humans is based on the type of relationship between them. So, for objects to interact effectively with each other, they use human behaviour. In (Atzori et al., 2014) The POR is established between similar objects belonging to the same production batch as shown in Figure 4. The POR can be used to create robust links in the SIoT. CLOR is established between objects residing in the same location. The CWOR is established whenever objects periodically collaborate or cooperate to provide a common goal. The OOR is established between heterogeneous or homogeneous objects belonging to the same user. The SOR is established continuously or discretely, as long as the owners are in contact during the lifetime of these objects.

Community Table :

In our work, an intelligent communicating object keeps a table called community table in which it memorizes the characteristics (ID, AID, OID and FC) and the relations (POR, CWOR, SOR, CLOR, OOR) which are established between socialized industrial objects. Also, a communicating object can remember objects that it has already seen. Objects are not stable, so the number of objects in a community is variable, resulting in a dynamic community table.

List of Services:

According to a distributed approach, a communicating object memorizes the list of the services of the other objects, which are in its interpersonal distance. On the other hand, according to the centralized approach, the object does not remember the services of the other objects that it has encountered previously.

Social community:

By analogy with humans, each object has a list of services that can execute them according to well-defined rules. In the decentralized approach, by the discovery of services of the same community, each object requests and knows the services of all the other communicating objects. The notion of distance is important in communication, so that the contact is effective and comfortable. Each contact is characterized by a distance that must be respected; this distance is called the interpersonal distance. To accommodate this distance, Hall determined the existence of four areas in human's zones as shown in Table I: which are intimate, personal, social, and public zone [START_REF] Rios-Martinez | From proxemics theory to socially aware navigation: A survey[END_REF].

CREATION OF COMMUNITIES OF SOCIALIZED INDUSTRIAL COMMUNICATING OBJECTS

When an object is located within the interpersonal distance of another communicating object, therefore, a community will be created or updated. The overlap of interpersonal distances shown in Figure 5 describes two cases: a) the overlap is not sufficient to bring object 1 into the community of object 4; b) the overlap is total so the two objects enter their respective communities. In the case where all the objects are located within the interpersonal distance of all the other objects, we can say that these communicating objects belong to the same community as it is presented in figure 6. So, each object has the same community stored in its inner memory. Each communicating object has its own community table as shown in table II and a specific list of services which are available for all the other objects of its community. 1, 2, 3, 4, 5

NETLOGO SIMULATION

The simulation of our model CoDoComShaRe.Biz was performed by the NetLogo simulator. NetLogo is a programming and simulation language for a multi-agent environment. The exchange of messages under NetLogo is done using a "messaging.nls" extension [START_REF] Kilani | A Socio-Inspired Methodology and Model for Advanced and Opportunistic Interactions between Industrial IoT Objects[END_REF].

In this simulator each object has fixed characteristics like the attributes which are: (ID, AID, OID and FC), the community table and the list of services.

After the creation of a unique stable community and following the appearance of a new communicating object. The latter announces his presence by the broadcast of an announcement message. The community's industrial communicating objects respond with a relationship-type proposal message. The new object responds with an acceptance or rejection message for the proposed relationship type. If the relationship is established, each object then sends its service list to this object and the latter responds with its service list.

The announcement and the relationship proposal constitute the community messages. The service list exchange messages form the service presentation messages. Two types of messages are exchanged between industrial communicating objects: community messages and service presentation messages.

As a result of a NetLogo simulation, by adding an object to a unique community of objects, the number of community messages exchanged is represented by figure 7. The estimated equation of the curve in Figure 7 is:

𝑦 = 2𝑥 + 1 (1)
Equation ( 1) gives the number of community messages exchanged when a Nth new object enters a community already consisting of (N-1) objects. Thus, for an existing community of 5 objects, the integration of a 6th object in this community requires the transmission of 11 community messages.

The estimated equation of the curve in Figure 8 is:

𝑦 = 2𝑥 (2)
Equation ( 2) gives the number of service presentation messages exchanged when a Nth new object enters a community already made up of (N-1) objects. Thus, for an existing community of 5 objects, the integration of a 6th object in this community requires the transmission of 10 service presentation messages. Thus, the incremental creation of a community of objects generates a number of messages that can be easily calculated by the approximated formula (2) for service presentation messages, and equation ( 1) for community messages. The total number of messages is fully supportable and manageable by standard wireless communication technologies used in the IIoT such as ZigBee, Bluetooth, Z-Wave, WiFi. When a community of objects is stable and already exists, ie there is no new incoming object, then the exchanges are limited only to service requests between objects according to their needs. The message exchange flow is then equivalent to the traffic generated by a classic request/response method, with the addition of the socialized management of interactions between objects.

In the case of the complete creation of a community of N objects "from scratch", the total number of messages generated is given by the following equation:

𝑦 = * 4𝑥 + 1 ! "#$ (3)
The proposed methodology of socialized interaction between communicating objects is totally decentralized because each object implements the same interaction algorithms as described in the previous paragraphs and does not require any centralized management of services or object characteristics.

The method is robust and adaptive because it allows manage communities of objects in an opportunistic way, with a zeroconfiguration approach. Community interactions are totally spontaneous and deterministic. The complete sharing of the characteristics of industrial communicating objects and their services gives complete visibility for the entire community to each of its members.

CONCLUSION

In this work, we have described how industrial communicating objects interact together through a sociological mechanism. This interaction mechanism is based on social behaviour's inspired by sociology research on human interaction. These social behaviours' are introduced in our innovative model called CoDoComShaRe.Biz which is based on six principles of interaction. It makes it possible to manage sociality between objects. The model is dedicated to industrial communicating objects in the development of Industry 4.0. The interaction of industrial objects is carried out by applying the SIoT concept, i.e., according to the principles of social interaction (strategies and conditions) the objects accept or refuse the execution of the requested services. The sociability of objects makes it possible to decentralize decisions and to self-regulate conflicts between objects.

The addition of an object to a preset community is done in an autonomous and dynamic way. These industrial objects exchange two types of messages which are the community messages and the service presentation messages that give a complete description for the characteristics and services of all industrial communicating objects. These messages enable adaptive and dynamic sharing of the characteristics and service description of all objects in object communities. Each object in a community dynamically adapts to new incoming objects.

Netlogo simulation proves that our model is efficient and workable by computing all messages exchanged between objects. messages are used for adding an object to a preestablished community. The simulation results demonstrate that messaging traffic between socialized industrial communicating objects is deterministic and can be estimated a priori by a simple equation. This messaging can be easily implemented by current wireless communication technologies used in industrial applications.

As a perspective of our work, a study of industrial application scenarios of different principles of interaction will be proposed, as well as a practical validation on a real platform of industrial Internet of Things.

Figure 1 .

 1 Figure 1. Five principles of interactions

Figure 2 .

 2 Figure 2. Deterministic / opportunistic industrial object interactions

Figure 3 .

 3 Figure 3. Interaction sequence between communicating objects

  the authors propose five types of relationships: Parental Object Relationship (POR), Co-location Object Relationship (CLOR), Co-work Object Relationships (CWOR), Ownership Object Relationships (OOR) and Social Object Relationships (SOR).

Figure 4 .

 4 Figure 4. Flowchart of the type of relationship between objects

Figure 5 .

 5 Figure 5. Community made by 2 objects 1 and 4.

Figure 6 .

 6 Figure 6. A unique community of Objects.

Figure 7 .

 7 Figure 7. Community messages exchanged by adding a new object

Figure 8 .

 8 Figure 8. Service messages exchanged by adding a new objectFigure 9 represents the two types of messages, updated community messages and updated service presentation messages according to the number of objects forming a single community.

Figure 9 .

 9 Figure 9. Number of messages between objects by adding a new object in the community

Table 1 . Physical distance zones

 1

	Distance zones	Mode
	Intimate zone	15-40 cm
	Personal zone	75 -125 cm
	Social zone	2,10 -3,60 m
	Public zone	7,50 m and beyond

Table II . Individual community for each object

 II

	Object	Object's Community
	1	2, 3, 4, 5, 6
	2	1, 3, 4, 5, 6
	3	1, 2, 4, 5, 6
	4	1, 2, 3, 5, 6
	5	1, 2, 3, 4, 6
	6


Ziad Tlaiss

Anomaly root cause diagnosis from active and passive measurement analysis

Keywords: troubleshooting, active probe, passive probe, TCP, QUIC, Congestion Control algorithm, time series data

Diagnosis demands a deep analysis of data to identify the root cause of an anomaly and still mostly relies on human experts. The increase of Internet traffic combined with the arrival of the encrypted protocol QUIC which invalidates many troubleshooting methods, urges to automate this process. To this effect, both domain familiarity and analysis skills are required. In this work we present our methods and strategies to detect the root cause of anomalies from active and passive network measurement and we share our plan towards an automatic root cause diagnosis. We focus on four root causes : transmission, congestion, application limited and packet delay variation, and present the building blocks of classification methods.

I. INTRODUCTION

Troubleshooting is essential to the operators to offer good network services to their clients. Telecommunication operators have made substantial investment to ensure high QoS (Quality of Service) and QoE (Quality of Experience) to their customers, this has been made possible thanks to relentless tracking of network degradation. In this article we focus on troubleshooting that rely on transport layer and more precisely on congestion control algorithm behaviour. Operators use active and passive probes to perform network measurement. This troubleshooting method is considered as one of the best solutions to detect root causes that directly impact the end user. In this purpose we should distinguish two types of network measurement : active and passive measurement.

Active measurement consists in observing test traffic generated by active probes, sometimes called robots. These probes will generate packets that are sent over the network to perform different measures. The main role of active probes is to act as an end-user client connected to the Internet through an access network. In passive measurement, probes are deployed in the network with the objective of tracking and collecting real users data.

The collected data from active and passive measurement is presented as a time series data. For confidential matters all the data presented is this work will be extracted with active probes only. However the analysis and diagnosis methods of the data collected with passive probes still remain the same while using TCP. Sequence number, acknowledgment, receiving window, selective acknowledgment, round trip time and bytes in flight represent the different metrics captured by probes and extracted or calculated from TCP packet headers.

II. METRICS DESCRIPTION

Active and passive probes timestamp each captured packet which leads to a time series. In this section we describe the most important metrics that we collect and analyse to detect the root cause of an anomaly.

• Sequence (SEQ): The sequence number identifies the first data byte in a packet. [1]. To measure the BIF, we calculate the bytes difference between the sequence number and the last received ACK. If the congestion control algorithm has the sack option then we take this information into consideration for better BIF estimation. • Round-Trip-Time (RTT) : Round trip time is the delay between the emission of a packet and the reception of the corresponding acknowledgment. Like the BIF, the RTT is not included in the TCP header. To measure the RTT we compute the time difference between the observation of the sequence number and its acknowledgment.

Identifying the root cause of an anomaly is essential for an efficient troubleshooting. This requires a deep knowledge of the congestion control algorithm's (CCA) behaviour; this behaviour can be caught by observing the time series of CCA states (e.g. : Slow-Start, Congestion Avoidance, Fast Retransmit, Fast Recovery [START_REF] Stevens | TCP Slow Start, Congestion Avoidance, Fast Retransmit, and Fast Recovery Algorithms[END_REF]). CCA states are easily readable in the sender stack, typically a server in an active measurement scenario. On the contrary, internal CCA states are out of reach for a mid-point observer, typical of a passive measurement scenario. In that case, human experts typically derive CCA state time series from the analysis of the previously defined metrics (BIF, SEQ, etc.). For example, detecting Slow-Start state exit time and determining if it's an early exit (or not) is the first necessary step to investigate an anomaly. This detection is crucial because in this state the CCA estimates the bottleneck capacity, and overestimating or underestimating the bottleneck capacity will lead to a bad connection throughput.

III. TIME SERIES ANALYSIS FOR ROOT CAUSE

IDENTIFICATION

In this section we analyse time series captured by active probes so as to classify them into root causes. We focus on four of the most frequent root causes in Internet networks. In each case we expose our data and explain the reason underlying our diagnosis. The presented data are collected from RAN and fully wired network with various TCP versions.

A. Transmission loss

Transmission losses occur when packets are dropped without any congestion in a network. It mostly affects Radio Access Networks (RAN). Radars, weather and RAN interference can be the origin of this issue. We can identify it by analysing packet sequence numbers and their acknowledgements. Transmission loss appears as individual and isolated packet losses as shown on Fig. 1 (where we have three individual losses at 13.61s, 13.82s and 14.03s). A loss can be identified by a sequence number decrease due to packet retransmission. This loss is considered isolated as we have only one packet lost from a burst of packets. It's harmful when it is repeated many times and more precisely at the beginning of the connection. Indeed it causes the CCA to exit the slow-start state early, because it considers these losses as a sign of a congestion, thus underestimating the bottleneck.

B. Network congestion

Network congestion can occur when the emitted traffic throughput is greater than the network capacity. It may affect any network, wireless or wireline and mostly depends on link capacity and buffer sizing. Contrarily to transmission loss, to detect congestion we look for a burst of packet losses as shown on Fig. 2 (from 1.88s to 1.94s). These lost packets will be accompanied by a significant increase in RTT as shown on Fig. 3 at 1.90s where RTT value starts to rise from 200ms at 1.90s to 625ms at 2.30s.

C. Application limited

Beside network issues, the performance of a TCP transfer can be affected by the endpoints' ability to send or receive fast enough. This problem only depends on the server and terminal. Servers are typically sized not to be overwhelmed by normal traffic, so the "slow sender" case is rare. On the contrary, clients are typically battery-powered devices with limited resources, and are fairly often the limiting factor. We thus focus on this case here.

This "slow receiver" situation is defined by the RWIN limiting the BIF, as shown on Fig 4 . Here we can notice the BIF's exponential growth in the slow-start phase, ending at 1.4s by reaching the RWIN's limitation. This indicates that the receiver's buffer cannot grow beyond that size; accordingly, the source's sending rate cannot increase any further without overwhelming the receiver.

D. Packet delay variation impact

Packet delay variation, also called jitter, is a network issue that normally affects wireless users. Jitter causes bad performance due to early Slow Start exit when the CCA A misinterprets it as a sign of congestion. To verify if jitter is the root cause of bad performance we first compute the slow-start exit time. After that, we correlate with other time series (e.g. SEQ, ACK, RTT) looking for packet loss or RTT increase. If no loss is detected and high RTT variance is observed, then we can conclude that CCA has exited slow-start due to jitter.

IV. TOWARDS AN AUTOMATIC TROUBLESHOOTING!

Finding the root cause of degradation within the network is time consuming for human experts. With the skyrocketing growth of Internet traffic, this task becomes intractable without automation. The faster we can detect the heart of the problem in networks, the quicker we can solve it before impacting the experience of customers.

A. Slow-start exit automatic detection

Slow-start exit can be detected by searching the exponential part in BIF time series. For this purpose, we distinguish two steps : the first one with traffic with headers in clear (TCP), the second one with deeply encrypted traffic (QUIC).

1) Detection with readable headers: Our work mainly focuses on slow-start automated detection in TCP where BIF series is at hand. We first tried to identify the Slow-Start phase by fitting the BIF series to an exponential, but the results were disappointing with bursty traffic which typically exhibits an on-off pattern. We then designed a promising innovative smoothing method which will be published soon.

2) Detection with encrypted headers: Strictly applying the previous slow-start detection method on passive measurements with encrypted traffic is impossible. Indeed, only packets timestamp and length are reachable in QUIC, thus BIF and RTT are out of reach.

To overcome this difficulty, we propose to use machine learning algorithms trained on clear-headers samples: thus, we first classify TCP captures, then turn them into QUIC-like captures, erasing all header information. We then feed them into the ML algorithm, using their previously determined class as supervision. We then expect the algorithm to extract the proper regularities, assuming it is possible.

B. Labelled dataset

We gathered more than 400 captures from active probes in Orange networks and manually labelled them into root causes using human expertise. This dataset relies on probes in a broad range of conditions: 4G, wireline, various TCP versions, various RTT. This dataset will be used both to evaluate analytical methods and to train machine-learning algorithms. The next step is to "QUIC-ize" this dataset so as to mimic encrypted traffic, so as to verify the consistency of our results;

V. CONCLUSION This thesis contributes to the research and industry in the field of network monitoring and troubleshooting. Automatic troubleshooting is not simple, especially with the arrival of QUIC that invalidates passive troubleshooting methods due to packets headers encryption. In this thesis, typical degradation root causes are analysed, descriptors are defined and new classification methods are proposed. These methods -coupled with new learning solutions such as Machine Learningseem promising; they potentially allow a high automation of processes while being compatible with deeply encrypted flows such as QUIC.

Fig. 1 .

 1 Fig. 1. transmission loss detection using seq & ack evolution

Fig. 2 .

 2 Fig. 2. congestion detection using seq & ack evolution

Fig. 4 .

 4 Fig. 4. application limited detection using rwin & bif evolution

Fig. 5

 5 Fig. 5. bif evolution

ACKNOWLEDGEMENTS I would like to express my sincere gratitude to my supervisors Mr. Alexandre Ferrieux and Ms. Isabelle Hamchaoui in Orange Labs and Ms. Sandrine Vaton and Ms. Isabel Amigo in IMT Atlantique for their advice on this research and on my PhD project.