text:
Anna Castrioto

Odeya Marmor

Marc Deffains

Dafna Willner

Eduard Linetsky

Hagai Bergman

Zvi Israel

Renana Eitan

David Arkadir
email: arkadir@hadassah.org.il

Anesthesia reduces discharge rates in the human pallidum without changing the discharge rate ratio between pallidal segments

Keywords: Basal ganglia, human, anesthesia, dystonia, Parkinson's disease

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

 [START_REF] Albin | The functional anatomy of basal ganglia disorders[END_REF][START_REF] Alexander | Basal ganglia-thalamocortical circuits: parallel substrates for motor, oculomotor, "prefrontal" and "limbic" functions[END_REF][START_REF] Bergman | Reversal of experimental parkinsonism by lesions of the subthalamic nucleus[END_REF]. Today, more than 25 years later, it is still the most widely accepted model linking dopamine imbalance in the striatum, rate of discharge in the different nuclei of the basal ganglia, and motor expression of hypo-and hyper-kinetic movement disorders. Deep brain stimulation (DBS), an established treatment for both dystonia and Parkinson's disease, enables researchers to test some of the predictions of this model in humans (Deffains et al., 2014a;[START_REF] Starr | Spontaneous pallidal neuronal activity in human dystonia: comparison with Parkinson's disease and normal macaque[END_REF][START_REF] Tang | Neuronal firing rates and patterns in the globus pallidus internus of patients with cervical dystonia differ from those with Parkinson's disease[END_REF].

In this work we tested the classical basal ganglia model prediction that increased discharge rates in the external segment of the globus pallidus (GPe) lead to reduction of discharge rates in its internal segment (GPi) and vice versa. We analyzed data from three different groups of human patients who were undergoing DBS surgery: awake patients with dystonia, awake patients with Parkinson's disease (PD) and dystonic patients under general anesthesia (GA). This enabled us to dissociate the striatal dopaminergic state and the general changes in neuronal discharge rates induced by anesthesia. We used an automatic offline spike detection and sorting method to objectively select only well isolated single neurons to test the rate model with greater accuracy.

Materials and Methods

Patients, surgical procedure and recording of neuronal activity

Neuronal activity was collected intra-operatively from 45 patients (39 with dystonia and 6 patients with PD) during surgical implantation of a DBS lead in the ventroposterior-lateral GPi. Surgeries were performed between 2003 and 2014 at the Surgery was performed using a CRW stereotactic frame (Radionics, Burlington, MA) and planning software (Framelink 4 or 5, Medtronic, Minneapolis, Minnesota). Target coordinates were calculated by direct targeting with either 1.5 or 3 Tesla T2 magnetic resonance imaging (MRI). Trajectory angles varied from 50 to80 degrees from the axial AC-PC plane and 0-10 degrees from the mid-sagittal plane, adjusted for the cortical, ventricular, and blood vessel anatomy (based on gadolinium-enhanced T1 MRI images) of individual patients. On the day of the surgery the stereotactic frame was attached to the patient and a high resolution computerized tomography (CT) was performed. In patients operated under general anesthesia this procedure was also performed under general anesthesia (see below). The T1, T2 and other MRI images were fused and registered with the CT images (Framelink 4 or 5, Medtronic, Minneapolis, Minnesota) and the target coordinates and trajectory angles were calculated. The final location of the permanent electrode was based on the results of intraoperative micro-recordings and (in PD patients) the results of macro-stimulation.

General anesthesia was induced with intravenous propofol (1.5-2mg/kg) and alfentanil (10 µg/kg). Neuromuscular blockade was achieved with vecuronium (0.1 mg/kg) to facilitate endotracheal intubation. Anesthesia was then maintained with nitrous oxide in oxygen (70%:30%) and isoflurane (0.4-0.5% end-tidal concentration). Before the skin incision, a local anesthetic (0.5% bupivicaine) was applied. At least ten minutes prior to the micro-electrode recording, the isoflurane dose was reduced to the point where single neuronal units could be observed. In the surgeries that were performed Electro-physiological recording was performed using polyamide-coated tungsten microelectrodes (Impedance at 1000Hz -0.3-0.9Mohm). The position of the electrode was controlled through the MicroGuide system (AlphaOmega Engineering, Nazareth, Israel). The signal was amplified by 10000, filtered with a four-pole Butterworth filter (250-6000 Hz) and sampled at 24-48 kHz by a 15+1-bit A/D converter (using ± 5V input range).

Recording involved two parallel microelectrodes 2 mm apart that were simultaneously moved toward the estimated target (ventral border of the lateral-posterior portion of the GPi). Recording started 15 mm above the estimated target (as per imaging). A third microelectrode penetration was only necessary in a few cases. Microelectrodes were advanced simultaneously in steps of 50-400 µm (the shorter steps while inside the pallidum).

Spiking activity was classified on-line by an expert neurophysiologist as belonging to the striatum, GPe, GPi, or border cells on the basis of the signal characteristics and the spatial display of the normalized root mean square value of the microelectrode recordings [START_REF] Zaidel | Delimiting subterritories of the human subthalamic nucleus by means of microelectrode recordings and a Hidden Markov Model[END_REF]. For the purpose of our analysis, recordings that could not be classified were discarded.

Offline data analysis

Recording stability was assessed (Deffains et al., 2014b) and only stable recording segments exceeding 10 seconds were used for further analyses. An automatic spike detection and sorting method was then implemented to isolate single units [START_REF] Deffains | Higher neuronal discharge rate in the motor area of the subthalamic nucleus of Parkinsonian patients[END_REF]) from the stable recording segments. An isolation score was calculated for each unit that was identified by the spike sorter software [START_REF] Joshua | Quantifying the isolation quality of extracellularly recorded action potentials[END_REF]. This score ranges from 0 (i.e. highly noisy spiking activity) to 1 (perfect isolation). Only units with an isolation score equal or greater to 0.7 were further analyzed. For each unit the peakto-peak duration of the spike waveform was calculated as the difference between the maximal and the minimal amplitude on the time axis. The coefficient of variation of the inter spike intervals (ISI CV, standard deviation/mean), and the power spectrum as a measure of the discharge pattern (regularity, Poisson like, bursting and periodic oscillations) of spiking activity were also computed. The raster plots of units with an ISI CV larger than 2.5 were inspected visually to remove possible unstable segments.

Data are expressed as the mean ± standard error of the mean (S.E.M.). All analyses were conducted with Matlab 7.5 2007 (Mathworks, Natickm MA).

Results

Patients and neuronal database:

During DBS surgery we recorded the neuronal extracellular activity from 39 patients with dystonia and 6 patients with PD. In 19 patients with dystonia (14 with isolated generalized dystonia and 5 with segmental dystonia) recordings were performed in the awake state, without any sedation or anesthesia. In the other 20 dystonic patients (16 with isolated generalized dystonia, 4 with secondary generalized dystonia and 2 with focal cervical dystonia) recordings were performed under generalized anesthesia.

Recordings in all PD patients were performed in the awake state, at least 12 hours Table 1. Recordings were made in 8430 sites. These included 3806 sites in awake dystonic patients, 3536 sites in anesthetized dystonia patients and 1088 sites in patients with PD.

We then used an offline automatic sorting system to isolate single units. 1560 pallidal (GPe and GPi) cells fulfilled our stringent criteria (isolation score, IS > 0.7, stable recording time >10s) and were used for further analysis. This relatively small fraction (<20%) of units reflected our strategy to use only highly-isolated neurons (recorded in the noisy environment of an operating theater). Details regarding our database are presented in Supplementary Table 2. An example of 2 seconds of raw recording and the aligned spike waveforms in the different structures and conditions can be seen in Supplementary Fig. 1. This short report focuses on mean firing rates. We carried out further analyses of the discharge patterns and oscillations, but failed to find robust results (see supplementary text).

Mean firing rates of pallidal neurons:

Awake dystonia vs. awake PD -Our analysis database was composed of 222 GPe neurons from awake patients with dystonia and 79 units from awake patients with PD.

There were no significant differences in the frequency of the GPe discharge rate of these two groups of patients (mean ± S.E.M, dystonia: 58.60 ± 1.77 Hz, PD: 53.79 ± 3.2 Hz, t-test p =0.17, t=1.37, df=299).

GPi analysis covered 415 units from awake patients with dystonia and 63 units from awake patients with PD. The frequency of discharge was significantly lower in dystonia (dystonia: 56.89 ± 1.20 Hz, PD: 81.24 ± 3.43 Hz, t-test p < 0.001, t=7.26, df=476). The frequency of discharge was significantly higher in GPi than in GPe in PD (t-test p < 0.001, t=5.83, df=140), but did not differ in dystonia (t-test p =0.41, t=0.82, df=635).

These results are consistent with non-human primate studies (Bergman et al. 1994;[START_REF] Heimer | Dopamine replacement therapy reverses abnormal synchronization of pallidal neurons in the 1methyl-4-phenyl-1,2,3,6-tetrahydropyridine primate model of parkinsonism[END_REF].

Awake dystonia vs. dystonia under general anesthesia -We analyzed the activity of 244 GPe and 415 GPi neurons in dystonic patients under general anesthesia. These were compared to the GPe and GPi neurons (n=222 and 415, respectively) characterized above that were recorded in awake dystonic patients. The analysis revealed that the discharge rates were significantly lower in the GPe (dystonia anesthetized: 40.1 ± 1.9 Hz, t-test p < 0.001, t=6.97, df=464) and the GPi (dystonia anesthetized: 40.7 ± 1.0 Hz, t-test p < 0.001, t=10.43, df=950) under general anesthesia and that in this population, as was the case in awake dystonic patients, mean firing rates in the GPe and the GPi did not differ significantly.

Relationships between GPe and GPi firing rates:

The classical rate model of the basal ganglia predicts that increased firing rates in the GPe should lead to a decrement in the firing rates in the GPi (and vice versa) through the GPe-STN-GPi indirect path. The recently discovered GPe to GPi GABAergic projections [START_REF] Hazrati | Evidence for interconnections between the two segments of the globus pallidus in primates: a PHA-L anterograde tracing study[END_REF][START_REF] Sato | Single-axon tracing study of neurons of the external segment of the globus pallidus in primate[END_REF] suggests there may be a complementary mechanism for the reciprocal relationships between the GPe and the GPi discharge rates. We showed above that in awake patients with PD the firing rates in the GPi were significantly higher than in awake dystonic patients, although the firing rates in the GPe were similar in both conditions (Fig. 1). This finding is in line with previous observations [START_REF] Starr | Spontaneous pallidal neuronal activity in human dystonia: comparison with Parkinson's disease and normal macaque[END_REF][START_REF] Tang | Neuronal firing rates and patterns in the globus pallidus internus of patients with cervical dystonia differ from those with Parkinson's disease[END_REF], but contrasts with the prediction of the classical model of the basal ganglia network.

To further test these findings, we calculated the mean firing rate in the two pallidal segments for each individual patient and for each hemisphere separately. We then plotted these mean firing rates in each of the GPi against the mean values of the ipsilateral GPe (Fig. 2A-C). Since we used stringent criteria to filter our database (see above), this analysis yielded only 63 ipsilateral GPe-GPi pairs (dystonia awake 30, PD awake 8 and dystonia anesthetized 25). Surprisingly, and in contrast to the predictions of the classical model, in all 3 states increased GPe firing rates were associated with increased GPi firing rates. The slope of fitted linear least squares line was positive in all 3 states (dystonia awake: 0.96, 95% interval 0.85-1.07, PD awake: 1.40, 1.12-1.67, dystonia anesthetized 1.01, 0.90-1.12).

The seemingly steeper slope for GPe-GPi firing rates in awake PD patients led us to calculate the ratio of the GPi mean firing rate to the GPe mean firing rate for each of these GPe-GPi pairs (Fig. 3). This ratio was similar in both awake and anesthetized patients with dystonia (dystonia awake: 1.13 ± 0.11, dystonia anesthetized 1.12 ± 0.05 p=0.92, df=53, t=0.1) but significantly higher in awake PD patients (1.49 ± 0.11, significantly different from dystonia in the awake [t-test p < 0.05, t=2.08, df=36] and anesthetized state [t-test p=0.006, t=2.92, df=31]).

Discussion

In this work we used an automatic offline sorting system to analyze only highly isolated human pallidal units (<20% of our recorded units). The analysis revealed major differences in firing rates in the GPi between awake dystonic and PD patients although the firing rates in the GPe was similar in these two populations. Firing rates in dystonic patients under general anesthesia were lower in both nuclei. In contrast to the prediction of the classical basal ganglia model, in each population, increased GPe firing rates were associated with increased firing rates in the ipsilateral GPi. The firing rate ratio of ipsilateral GPi/GPe pairs was similar in awake and anesthetized patients with dystonia but significantly higher in PD.

By using an automatic offline spike detection and a sorting system and analyzing only well-isolated neurons, our analysis confirmed that the firing rate of GPi neurons was significantly lower in the awake patients with dystonia than in the awake PD patients. This finding is in line with the classical model of the basal ganglia [START_REF] Albin | The functional anatomy of basal ganglia disorders[END_REF][START_REF] Alexander | Basal ganglia-thalamocortical circuits: parallel substrates for motor, oculomotor, "prefrontal" and "limbic" functions[END_REF][START_REF] Bergman | Reversal of experimental parkinsonism by lesions of the subthalamic nucleus[END_REF], physiological studies in the MPTP primate models of PD [START_REF] Heimer | Dopamine replacement therapy reverses abnormal synchronization of pallidal neurons in the 1methyl-4-phenyl-1,2,3,6-tetrahydropyridine primate model of parkinsonism[END_REF] and with previous work in awake patients [START_REF] Starr | Spontaneous pallidal neuronal activity in human dystonia: comparison with Parkinson's disease and normal macaque[END_REF][START_REF] Tang | Neuronal firing rates and patterns in the globus pallidus internus of patients with cervical dystonia differ from those with Parkinson's disease[END_REF][START_REF] Alam | Globus pallidus internus neuronal activity: a comparative study of linear and non-linear features in patients with dystonia or Parkinson's disease[END_REF]. Unlike the difference observed in the GPi, the discharge rates in the GPe were similar in the PD and dystonia patients [START_REF] Starr | Spontaneous pallidal neuronal activity in human dystonia: comparison with Parkinson's disease and normal macaque[END_REF][START_REF] Tang | Neuronal firing rates and patterns in the globus pallidus internus of patients with cervical dystonia differ from those with Parkinson's disease[END_REF]. Thus small changes in discharge rates in the GPe may lead to pronounced changes in discharge rates in the GPi. This may stem from the converging nature of the basal ganglia; i.e., the gradual reduction of the neuron population between the striatum, GPe and the output structures of the basal ganglia [START_REF] Goldberg | Computational physiology of the neural networks of the primate globus pallidus: function and dysfunction[END_REF].

General anesthesia with isoflurane depresses firing in both pallidal segments in dystonia. This is congruent with previous work reporting a depressed firing rate in the GPi or STN under anesthesia [START_REF] Hutchison | Pallidal neuronal activity: implications for models of dystonia[END_REF][START_REF] Moll | Asymmetric pallidal neuronal activity in patients with cervical dystonia[END_REF][START_REF] Raz | Propofol decreases neuronal population spiking activity in the subthalamic nucleus of Parkinsonian patients[END_REF] and contrasts with the unchanged firing rates in the STN found by others [START_REF] Kim | Influence of propofol and fentanyl on deep brain stimulation of the subthalamic nucleus[END_REF][START_REF] Lettieri | Deep brain stimulation: Subthalamic nucleus electrophysiological activity in awake and anesthetized patients[END_REF]. This difference might be related to the anesthesia protocol, the site of recordings or the strict selection of the units included in the analysis. In any case this state enabled us to test the relationship between GPe and GPi firing rates while dissociating dopaminergic state and discharge rates.

We found that for each of the conditions, increased firing rates in the GPe were associated with increased firing rates in the GPi, and vice versa for decreased discharge rates. This finding conflicts with the prediction of the classical model and is even more striking given recent discoveries regarding the direct GPe to GPi strong inhibitory GABAergic input [START_REF] Hazrati | Evidence for interconnections between the two segments of the globus pallidus in primates: a PHA-L anterograde tracing study[END_REF][START_REF] Sato | Single-axon tracing study of neurons of the external segment of the globus pallidus in primate[END_REF]. Since we did not use electroencephalograms to monitor the level of arousal of the patients, we can only speculate that this finding may reflect the effect of different arousal states during recording. This speculation is based on our finding of decreased firing rates in both pallidal nuclei under general anesthesia, and on previously demonstrated decreased firing rates in both segments of the pallidum in non-human primates with decreased arousal levels [START_REF] Adler | Neurons in both pallidal segments change their firing properties similarly prior to closure of the eyes[END_REF].

The main finding is that the firing rate ratio GPi/GPe was not affected by the discharge rates in the GPe but rather was determined by the type of disease. Importantly, this ratio was not affected by differences in the discharge pattern (burstiness and oscillations) between the awake and anesthetized states (see supplementary analysis).

Based on these results we suggest that the firing rates in the GPe and in the GPi are modulated by two apparently independent processes (Fig. 4). The first is dopamine deficiency that changes the ratio between the GPi and GPe firing rates (probably by increasing the GPi discharge rate with no major change in GPe), and the second is anesthesia (possibly the arousal state) that modulates the firing rates in both of these nuclei without changing the ratio between their discharge rates. Testing this hypothesis in PD patients under general anesthesia is crucial to proving this hypothesis. However, we did not have access to such data. Our findings may suggest a more complex view of the basal ganglia network, where the striatum and the STN constitute the input stage of the basal ganglia, and the activity of the GPi (output stage) is affected by the projections from the GPe, STN and striatum [START_REF] Goldberg | Computational physiology of the neural networks of the primate globus pallidus: function and dysfunction[END_REF]. Arousal level, as well as dopamine level, affects the different components of this network, leading to the complementary effects of arousal and dopamine on the discharge rate of pallidal neurons. Coherence analysis: We calculated the coherence between the stable segments of the background multi-unit spiking activity and the activity of single units. To clean the trace of multi-unit activity from the dominant spikes, we removed the spikes detected by the offline sorter and replaced them with a 2 ms trace taken from the middle of the closest spike free segment exceeding 10 ms (Moran et al., 2008). The background clean multi-unit activity was rectified by an absolute operator and the single unit activity was low-passed filtered (cut-off frequency 40 Hz) to obtain a rate-like function of the single unit. The coherence was calculated using a 1.5 second Hamming window (50% overlap) and a spectral resolution of 1/3 Hz. Frequencies below 3 Hz were removed to avoid possible physiological artefacts (i.e. cardiac activity or respiration). The coherence of each pairs was smoothed and z-transformed, averaged and then backtransformed to calculate the mean coherence in each target.

A coherence analysis of the background activity and single unit activity was conducted for 218 and 73 GPe pairs in awake dystonic and awake PD patients, respectively. No coherence peaks were found in dystonia whereas in PD there was a possible indication of a peak around 15Hz. The same analysis was also conducted for 413 and 62 pairs in the GPi of dystonia and PD patients, respectively. This analysis suggested there may have been two peaks at low frequency (5-10Hz) in the GPi but only in PD.

  IntroductionThe classical rate model of the basal ganglia was consolidated in the late 1980's (Albin

  Hadassah Medical Center in Jerusalem. Written informed consent was obtained from all patients and the study was approved by the local Institutional Review Board (reference code: 0168-10-HMO).

  , the patients' state of full arousal was continuously monitored and confirmed by a team member.

  dopaminergic medication. The demographic characteristics of the patients are reported in Supplementary

FiguresFig. 1 .Fig. 2 .

 12 Figures

Fig. 3 .

 3 Fig.3. GPe-GPi firing ratio. Ratio of mean firing rates of ipsilateral GPe-GPi pairs. For

Fig. 4 .

 4 Fig. 4. Effects of dopamine depletion and anesthesia on the GPe and GPi discharge rate.

European Journal of Neuroscience

Acknowledgements

This work was supported in part by research grants from the MAGNET program of the office of the chief scientist (OCS) at the Israel Ministry of Economy, the Israel Science Foundation (ISF), the German Israel Science Foundation (GIF), the Adelis, Rosetrees and Vorst foundations, the Simone and Bernard Guttman Chair in Brain Research, to HB. AC was supported by a grant from the Edmond and Lily Safra Foundation. DA was supported by the Hebrew University and the Hadassah joint fund and by a Prusiner-Abramsky Research Award.

Supplementary Tables

 Supplementary Table 1
Supplementary Analysis ISI CV:

The discharge rate and coefficient of variation of the inter-spike intervals (ISI CV, standard deviation/mean) is a measure of the discharge pattern (regularity, Poisson like, bursting) of spiking activity. There were no significant differences in the ISI CV between awake patients with dystonia and PD in the GPe (1.44 ± 0.05 and 1.25 ± 0.05 respectively) or the GPi (1.08 ±0.02 and 1.06 ± 0.07 respectively). Under general anesthesia the ISI CV was significantly higher both in the GPe (1.7 ± 0.1, p<0.001) and the GPi (1.2 ± 0.02, p<0.001), indicating more bursty spiking activity under general anesthesia in both segments. In all conditions, the ISI CV was significantly higher in the GPe than in GPi (p<0.001) and probably reflected the presence of pauses in the GPe.

Power spectral density (PSD):

To calculate the power spectral density (PSD) of the population spiking activity, a Z-score normalization was performed on the stable selected sections. The z-normalized signal was rectified by an absolute 'operator' to enable detection of periodic oscillations with an oscillation frequency below the operating room high-passed filter (250 Hz) [START_REF] Raz | Propofol decreases neuronal population spiking activity in the subthalamic nucleus of Parkinsonian patients[END_REF]. The PSD was calculated using the Welch method with a Hamming window of 1.5 seconds (50% overlap) and a spectral resolution of 1/3 Hz [START_REF] Deffains | Higher neuronal discharge rate in the motor area of the subthalamic nucleus of Parkinsonian patients[END_REF]. All DC components were removed by subtracting the mean of every windowed segment of the rectified signal.

PSD was performed on 1269, 2537 and 426, 662 multiunit activity traces in GPe and GPi of awake dystonic and awake PD patients respectively. PSD in awake dystonia patients showed a peak in the low frequency (5-10 Hz) in both GPe and GPi. In awake PD patients, PSD showed a peak in the low beta frequency (around 15 Hz) in both GPe and GPi. A 5Hz peak was also found in 5 trajectories in patients with PD and resting tremor. In contrast to the awake state, PSD analysis on 1242 and 2294 GPe and GPi multiunit traces in dystonic patients under general anesthesia did not show any peaks, either in the GPe or in the GPi. The PSD of single unit activity (with an isolation score > 0.7) was also calculated and was similar to that of the multiunit activity.


Joseph Doumit

Marie-Josépha Youssef

Abdel Charbel

Nour

Joumana Farah

Catherine Douillard

Resource Allocation in Full-Duplex Uncoordinated Communication Systems with NOMA

Keywords: orthogonal multiple access, uncoordinated system, relays, multi-armed bandits, full duplex, power allocation

The Internet of Things (IoT) is a major driving use case of future communication systems. 5G and beyond networks should be properly designed to meet the quality of service (QoS) requirements of IoT devices in terms of massive access, data rates, latency and reliability. In this paper, we study the uncoordinated spectrum and power allocation problems in an uplink IoT network. In the considered setting, IoT devices aim at organizing their transmissions, without any coordination, on the available frequency channels and power levels. To enhance performance, we propose the deployment of full-duplex relays as well as the use of non-orthogonal multiple access and queuing at the level of the relays. A new algorithm, based on the multiplayer multi-armed bandits framework, is proposed with the aim of reducing the transmit power of IoT devices. Simulation results validate the convergence of the proposed algorithm. Results also confirm the superior performance of the proposed technique in terms of satisfying QoS requirements and reducing the needed user transmit power when compared to two baseline schemes.

I. INTRODUCTION

I N recent years, Internet of Things (IoT) devices have become a main component in our daily lives. Because of their dense deployment, machine-type devices (MTDs) or IoT devices produce a large amount of data to be transmitted. Moreover, the mobile traffic of IoT devices mostly consists of short packets concentrated in the uplink. Hence, to ensure successful transmissions and due to the heterogeneous nature of the underlying applications, a communication network with low-latency and low complexity features [START_REF] Balevi | ALOHA-NOMA for Massive Machine-to-Machine IoT Communication[END_REF] is required.

Non-orthogonal multiple access (NOMA) has recently emerged as a promising multiple access technology. Powerdomain NOMA consists of multiplexing the signals of multiple users on the same resource block by exploiting the power domain [START_REF] Zhang | Uplink Non-orthogonal Multiple Access in 5G Systems[END_REF], [START_REF] Choi | NOMA-Based Random Access With Multichannel ALOHA[END_REF], [START_REF] Kilzi | Mutual Successive Interference Cancellation Strategies in NOMA for Enhancing the Spectral Efficiency of CoMP Systems[END_REF], [START_REF] Saito | Non-Orthogonal Multiple Access (NOMA) for Cellular Future Radio Access[END_REF], [START_REF] Farah | Power Minimization in Distributed Antenna Systems Using Non-Orthogonal Multiple Access and Mutual Successive Interference Cancellation[END_REF]. By doing so, NOMA enhances system spectral efficiency and increases the achieved data rates [START_REF] Benjebbour | Nonorthogonal multiple access (NOMA): Concept, performance evaluation and experimental trials[END_REF] with respect to orthogonal multiple access (OMA). At the receiver side, successive interference cancellation (SIC) is performed to retrieve the superimposed signals. Although most studies related to power-domain NOMA focus on the downlink setting, some work recently considered the application of NOMA in uplink scenarios. In [START_REF] Zhang | Uplink Non-orthogonal Multiple Access in 5G Systems[END_REF] and [START_REF] Choi | NOMA-Based Random Access With Multichannel ALOHA[END_REF], power allocation schemes are proposed. A backoff step between power levels is introduced in [START_REF] Zhang | Uplink Non-orthogonal Multiple Access in 5G Systems[END_REF], then system outage probability and achievable sum rate are shown to outperform the OMA model. In [START_REF] Choi | NOMA-Based Random Access With Multichannel ALOHA[END_REF], a new power allocation scheme that aims at ensuring the user rate requirements is proposed. Other studies have also shown that NOMA helps achieve massive connectivity and low latency communications [START_REF] Xiao | Downlink MIMO-NOMA for Ultra-Reliable Low-Latency Communications[END_REF], making it a promising candidate to support IoT communications [START_REF] Mazin | Comparison of Slotted Aloha-NOMA and CSMA/CA for M2M Communications in IoT Networks[END_REF].

To reduce the signaling overhead in IoT networks, uncoordinated spectrum access has received significant interest in recent literature [START_REF] Li | A Frequency Hopping Spread Spectrum Transmission Scheme For Uncoordinated Cognitive Radios[END_REF], [START_REF] Bande | Adversarial Multi-user Bandits for Uncoordinated Spectrum Access[END_REF]. That said, few works have considered the uncoordinated spectrum access in an uplink NOMA system. In fact, most of the available literature body on NOMA considers fully coordinated systems. Only a few recent works considered semi-grant free communication systems [START_REF] Ding | Simple Semi-Grant-Free Transmission Strategies Assisted by Non-Orthogonal Multiple Access[END_REF] and fully uncoordinated ones [START_REF] Jiang | Distributed Layered Grant-Free Non-Orthogonal Multiple Access for Massive MTC[END_REF].

In addition to NOMA, the deployment of fixed or mobile relays has gained attention. In massive communication networks, adding relays could be a solution to alleviate network congestion. Relays can also increase the coverage area and the achieved throughput of wireless networks. In addition to that, they can restore communication links in damaged networks [START_REF] Youssef | Backhaul-Constrained Resource Allocation and 3D Placement for UAV-Enabled Networks[END_REF], [START_REF] Youssef | Full-Duplex and Backhaul-Constrained UAV-Enabled Networks Using NOMA[END_REF]. In order to ensure low-latency and high data rates in massive networks, full-duplex capabilities at the relay level can be leveraged. By allowing the simultaneous transmission and reception of information on the same frequency resource, full-duplex links [START_REF] Youssef | Full-Duplex and Backhaul-Constrained UAV-Enabled Networks Using NOMA[END_REF] reduce both the experienced latency and the need to store information at the relay.

The use of reinforcement learning and game theory for uncoordinated resource allocation has also recently garnered an increasing attention. Several studies [START_REF] Youssef | Stochastic Multi-Player Multi-Armed Bandits with Multiple Plays for Uncoordinated Spectrum Access[END_REF], [START_REF] Adjif | Joint Channel Selection and Power Control for NOMA: A Multi-Armed Bandit Approach[END_REF], [START_REF] Zhou | Budget-Constrained Multi-Armed Bandits with Multiple Plays[END_REF] have considered reinforcement learning, and especially the multiarmed bandits (MAB) framework, to enable users to organize their transmissions on the available channels [START_REF] Youssef | Stochastic Multi-Player Multi-Armed Bandits with Multiple Plays for Uncoordinated Spectrum Access[END_REF]. In the MAB framework, a set of players compete to find the actions, also called arms, that maximize their expected gain or welfare [START_REF] Bistritz | Distributed Multi-Player Bandits -a Game of Thrones Approach[END_REF]. Since the arm rewards are previously unknown to the players, there is a need to achieve a trade-off between exploring the arms to learn their associated rewards, and exploiting the optimal arms to maximize the achieved gain. Multiple algorithms have been proposed to solve problems modeled using the MAB framework, such as the -greedy method, the -decreasing technique and the Upper Confidence Bound (UCB) algorithm [START_REF] Adjif | Joint Channel Selection and Power Control for NOMA: A Multi-Armed Bandit Approach[END_REF]. These algorithms have been widely used to study, among others, the opportunistic spectrum access problem in cognitive radio systems [START_REF] Liu | Distributed Learning in Multi-Armed Bandit With Multiple Players[END_REF], and the uncoordinated spectrum access problem [START_REF] Youssef | Stochastic Multi-Player Multi-Armed Bandits with Multiple Plays for Uncoordinated Spectrum Access[END_REF].

To prevent data loss in case of system congestion, relays can be equipped with queuing capabilities. These capabilities were mostly considered in coordinated systems where users have information about the buffer status [START_REF] Li | Queuing Analysis of Two-Hop Relay Technology in LTE/LTE-A Networks With Unsaturated and Asymmetric Traffic[END_REF]. However, no previous work has considered the use of full-duplex relays in an uncoordinated uplink NOMA system with transmit power limits and queuing capabilities at the relays. In this paper, we consider the uplink of an uncoordinated spectrum access system where IoT devices aim at organizing their transmissions with the deployed relays using MAB frameworks without communicating with each other. Each IoT device has a power budget and is allowed to connect to one relay only. In addition to that, the devices adopt continuous transmissions. Leveraging full-duplex communications, the deployed relays forward the received messages to a base station (BS). Also, queuing is applied, when needed, at the relay level.

The rest of this paper is organized as follows. In section II, the system model is described and the problem is modeled using the MAB framework. Section III presents the proposed solution for the joint distributed spectrum and power allocation, while section IV considers queuing at the level of the relays. Simulation results are presented in section V and conclusions in section VI.

II. SYSTEM MODEL

Consider a hexagonal cell with one BS located at the cell center, relays and IoT devices. Let D = { 1 , 2 , . . . , } and R = { 1 , 2 , . . . , } be the sets of devices and relays respectively. The IoT devices are randomly distributed in the cell. Moreover, the relays are distributed uniformly at a middistance between the BS and the cell edges.

We assume an uncoordinated or grant-free uplink communication system where no communication occurs between the MTDs. A two-hop communication network, where relays forward the received signals from the IoT devices to the BS, is considered. We assume that IoT devices cannot communicate directly with the BS due to large distances between them and the BS. As shown in Figure 1, each IoT device is allowed to communicate with one relay at a time using one channel. The relays operate in full-duplex, i.e., they receive signals from the MTDs and transmit them to the BS by simultaneously using the same resources. Thereby, relays may suffer from self-interference, possibly impeding the successful decoding of the received signals. Each relay has access to a separate set of subbands which will be used simultaneously in the access and the backhaul links. Hence, relays do not suffer from interrelay interference. Also, clustering the MTDs over the channels using NOMA allows their clustering between the relays. For multiple access, NOMA is considered between the devices and the relays, while OMA is applied between the relays and the BS.

A. Power Allocation Scheme

To enable SIC decoding at the level of the relays, devices transmitting on the same channel should have their signals received with different power levels. To this aim, we generalize the uplink NOMA power allocation scheme proposed in [START_REF] Choi | NOMA-Based Random Access With Multichannel ALOHA[END_REF] to the case of full-duplex transmissions.

We consider available received power levels per channel at the relay side and let Γ be the signal-to-interference-plusnoise ratio (SINR) requirement for each MTD. Inspired by [START_REF] Choi | NOMA-Based Random Access With Multichannel ALOHA[END_REF], it can be shown that the ℎ received power level , = 1, ..., ( 1 > 2 > ... > ), at the relay side, to guarantee the target SINR, is given by:

= 2 + × × Γ (Γ + 1) -. (1)
2 is the additive Gaussian noise power, the residual selfinterference factor, and the maximum transmit power of relay on each of its channels. Hence, × is the maximum self-interference power experienced at the relay.

In this work, MTDs are assumed to perform a perfect channel estimation so as to estimate the channel gains of their links with the relays. Moreover, these channel gains are supposed to depend only on the experienced path loss and shadowing, i.e., only long-term variations of the channel gains are considered. To reach a received power level at the ℎ relay level, each device calculates its needed transmit power according to:

, = ℎ 2 , , (2)
where ℎ , is the channel gain between user and relay . As previously said, to ensure successful SIC, the signals of devices transmitting on the same channel must be received with different power levels at the relay side. Therefore, if two or more devices, associated with the same relay and transmitting on the same channel, choose the same power level , a collision occurs, leading to a failed signal decoding. In uplink NOMA, SIC is performed in decreasing order of channel gains. Therefore, signals of devices having chosen weaker received power levels than the collided signals are also undecodable. The collision can even cause upper-level signals to be undecodable if their target SINR is not reached.

B. Uncoordinated Joint Channel and Power Allocation

To allow devices to organize their transmissions in an uncoordinated manner, the problem of joint channel and power allocation is modeled using the multi-player MAB framework with zero-reward on collision [START_REF] Bistritz | Distributed Multi-Player Bandits -a Game of Thrones Approach[END_REF]. The set of players is the set of IoT devices D and the set of arms is the set of available channel-power level pairs. Having a transmit power budget , each device builds its action profile A . The latter consists of channel and power level pairs, or arms, on which device can transmit without violating its power budget. Let A ={ 1 , 2 , ... , }, with representing the number of available arms for user .

Upon choosing an action, each device receives a reward or a utility from the relay. In this work, the reward is taken as the device achieved rate. It is given by:

, = log 2 (1 + Γ , ), (3)
where is the channel bandwidth and Γ , is the achieved SINR of device , given by:

Γ , = = +1 + + 2 , (4)
with = +1 being the interference of the NOMA signals not canceled with SIC, and the transmit power of relay per channel. In [START_REF] Kilzi | Mutual Successive Interference Cancellation Strategies in NOMA for Enhancing the Spectral Efficiency of CoMP Systems[END_REF], is the power level chosen in action . If two or more devices choose the same action, a collision occurs, resulting in a null utility for some users. In this case, the relay returns a zero reward to the concerned devices. Let represent the collision indicator for device on arm :

, = 0, in the case of a collision, 1, otherwise. (5)
Therefore, the utility , received by device when picking arm can be expressed as:

, = , × , . (6)
In the considered MAB framework, no communication is allowed between players. Hence, each player can only observe its own chosen actions and achieved rewards. The objective of the following study is to maximize the welfare of all devices, which is expressed as:

= =1 , . (7)

C. Relay-BS Communication

In full-duplex communications, the relay receives signals from the IoT devices and transmits them to the BS simultaneously. That said, at timeslot , the relay transmits the signals received at timeslot -1 while receiving new signals from the devices. At timeslot = 1, the relay has not received any messages beforehand, hence does not communicate with the BS. Starting at = 2, the relay transmits the signals received at = 1 to the BS, while the devices transmit new messages to the relay, and so on.

To guarantee the users QoS requirements when forwarding the device messages to the BS, the transmit power of the relay should be optimized. Let denote the number of correctly decoded signals at the level of relay at timeslot . The minimum rate required for the relay to be able to transmit all successfully decoded signals is equal to:

= × , (8)
where stands for the required rate per device and is related to Γ through (3). Each relay is assumed to transmit its data over orthogonal channels in an OMA manner. Hence, the rate achieved by relay on channel is given by:

, = × log 2 1 + ℎ 2 , 2 . ( 9
)
ℎ , is the channel gain between relay and the BS. It is common to all channels since only large-scale channel variations are considered. Assuming an equal rate repartition among the channels, to reach its rate requirement expressed in [START_REF] Xiao | Downlink MIMO-NOMA for Ultra-Reliable Low-Latency Communications[END_REF], the transmit power of relay over a channel should satisfy:

= 2 ℎ 2 , 2 × × -1 , . (10)

III. PROPOSED ALLOCATION TECHNIQUE

Classical MAB algorithms for uncoordinated resource allocation such as the -greedy, -decreasing, and UCB techniques [START_REF] Adjif | Joint Channel Selection and Power Control for NOMA: A Multi-Armed Bandit Approach[END_REF] generally lead to system convergence. However, the solutions proposed in the literature do not take into consideration the transmit power limit of devices. In MTD applications, there is a need to save power in order to maximize the battery life of devices. To fulfill this objective, we propose to modify the utility function of the UCB algorithm to account for the devices power budget.

Each IoT device aims at achieving a rate larger than or equal to its requested rate, while consuming the least amount of transmit power. Thus, two parameters should be taken into consideration in the design: the transmit power of device ,

, , and its achieved rate, , . As shown in (2), , depends on both the received power level chosen by and the channel gain between the relay and . An increase in the channel gain reduces the amount of required transmit power. Hence, the minimum transmit power value is reached when the device chooses its best radio channel, i.e., the one with the largest gain. However, having each device choose its best arm in terms of channel gain might not always lead to system convergence. Constrained by its power budget, user might not be able to transmit on all arms associated with channel . In such a case, the choice of all users must be adapted in a way that system welfare is maximized.

IoT devices should choose their channels and corresponding received power levels in a way to strike a trade-off between the achieved rate and the consumed transmit power. Therefore, instead of choosing the arm that maximizes the achieved rate, each device chooses arm * ∈ A that satisfies:

* = argmax ∈A , - , . (11)
In [START_REF] Bande | Adversarial Multi-user Bandits for Uncoordinated Spectrum Access[END_REF], and are positive weight parameters relative to the rate and power respectively, with

+ = 1. = log 2 (1 + 1
2 ) is the maximum achievable rate by all users. The proposed technique to solve the uncoordinated joint channel and power allocation problem is summarized in Algorithm 1, where is the time horizon and ( , ) the number of times arm is played by device i until timeslot .

( * , + 1) = ( * , ) + , * ( * , + 1) = ( * , ) + 1 ( , * ) = ( * , +1) ( * , +1)
end end

IV. QUEUING AT THE LEVEL OF THE RELAYS

When the relay successfully decodes the signals received from the MTDs, it forwards them to the BS. Occasionally, in the case of congestion or bad channel states between the relay and the BS, the relay may not be able to transmit all the received data to the BS within a unique timeslot. To avoid data loss in such a case, buffering the unsent signals at the relay can be used. In fact, this is necessary when the required power to transmit the received signals from the devices, , exceeds the relay power budget per channel, i.e., when:

≥ . ( 12
)
This can be translated as the number of correctly decoded signals at the relay level, , being greater than some threshold value 0, . The latter is deduced from (10) after setting the transmit power value equal to the relay power budget. 0, relates to the maximum number of received IoT signals that a relay can forward to the BS per timeslot without resorting to buffering. It is expressed as:

0, = × log 2 1 + ℎ 2 , 2 . ( 13
)
When buffering occurs, a certain number of MTDs should be incited to connect to another, less congested, relay. This is necessary to enable each relay to forward all received messages to the BS during one timeslot, thereby satisfying the QoS requirements of connected devices. The system being uncoordinated, very little information is shared between the relay and the connected devices. Thus, MTDs must learn on their own the need to connect to another relay. To do so, the relay having reached its overflow threshold, i.e., a full buffer possibly entailing information loss, returns a zero reward to all its connected devices for a certain period. This period is referred to as Timeout period which is determined in practice depending on the system parameters.

During the Timeout period of relay , devices associated with are encouraged to perform a new exploration phase. In other words, those devices may try to access arms associated with another relay, if their power budgets allow. By exploring new arms associated with less congested relays, IoT devices aim at reaching a new stable state leading to system convergence. Note that this solution could lead to some devices remaining silent, i.e., refraining from transmitting, for multiple timeslots if they are not able to connect to another relay.

To represent the zero utility in the case of a buffering overflow, the collision indicator is modified as follows: , = 0 in case of collision or buffer overflow, and 1 otherwise.

To avoid returning a zero reward for a long period by the congested relay, i.e., to help devices react faster in the Timeout period, they may take into consideration the total transmit power that has been spent on each specific arm , reflected by , × ( , ). Since the congested relay is replying with zero rewards, the decision function on this relay depends mostly on the consumed power, leading to free some arms and to try new ones. Therefore, to choose their actions, instead of using [START_REF] Bande | Adversarial Multi-user Bandits for Uncoordinated Spectrum Access[END_REF], IoT devices will adopt the following metric:

* = argmax ∈A , - , × ( , ) . (14)
To summarize, Algorithm 2 is proposed to account for buffering needs at the relays. V. NUMERICAL RESULTS Extensive simulations of the proposed allocation technique were conducted to validate its performance. First, we considered an uplink communication system with one BS at the cell center, = 50 transmitting IoT devices, and one relay. The cell radius is 250 m. A total of = 10 channels and = 5 power levels is considered, leading to a maximum of × = 50 arms. The MTDs are scattered around the relay according to a Poisson distribution [START_REF] Afshang | Poisson Cluster Process Based Analysis of HetNets With Correlated User and Base Station Locations[END_REF]. The channel propagation model corresponds to a path loss exponent of 3.76, and an 8 dB zero mean lognormal shadowing. The noise power spectral density is 0 = 4 × 10 -21 W/Hz. Other design parameters are presented in Table I. The performance of the proposed method is analyzed over = 200 timeslots and compared to that of two baseline schemes. The first is the UCB method in [START_REF] Adjif | Joint Channel Selection and Power Control for NOMA: A Multi-Armed Bandit Approach[END_REF] that does not take into consideration the power budget. The second baseline method is referred to as Random Access NOMA (RA-NOMA), where devices choose their channels and power levels uniformly at random.

A. Convergence and Performance Assessment of the Proposed Solution

In Figure 2, the instantaneous successful transmission probability is shown. While both UCB-based algorithms converge to a success probability of 1, our proposed method reaches convergence slightly faster than the baseline UCB technique. Both methods greatly outperform RA-NOMA. Figure 3 shows the successful transmission percentage for a varying IoT requested data rate, averaged over 2000 simulations. For the UCB-based methods, the successful transmission percentage is evaluated during both the exploration and the exploitation phases. Figure 3 shows that both UCB-based algorithms converge to an almost 90% successful transmission probability for all considered values of the IoT requested rate. Moreover, the UCB-based methods greatly outperform RA-NOMA.

In Figure 4, the instantaneous sum transmission power of all IoT devices is shown. The proposed method greatly decreases the needed transmit power when compared to the baseline UCB technique that does not account for power minimization and outperforms RA-NOMA regarding Peak to Average Power Ratio (PAPR).

Figure 5 shows the average transmit power per device for different values of the required rate. Here also, the proposed method outperforms the baseline UCB technique by reducing the average needed transmit power, with a performance gap that increases with the required rate. To conclude, using the proposed algorithm, the devices can limit their transmit power while reaching their rate requirements.

B. Performance Assessment for a Varying Residual Self-Interference Factor

In the following, the effect of self-interference on system performance is evaluated by varying the value of the residual self-interference factor . A system with = 4 full-duplex relays and a total of = 40 transmitting devices is now considered.

= 2 channels and = 5 power levels are set for each relay, leading to a maximum of × = 10 arms per relay. Having four relays, the MTDs are now uniformly distributed over the cell.

Figure 6 shows the successful transmission percentage with a varying factor. By keeping constant the power budget and increasing , the performance of the proposed method with full-duplex relays decreases when compared to the baseline UCB technique. In fact, when increases, the effect of the self-interference becomes dominant, thereby lowering system performance. It should be noted that the performance of the baseline UCB technique is not affected by an increasing value since it does not account for the devices power budgets. That said, the baseline UCB technique requires a much larger amount of power when increases (around a 450% increase in the power consumption of the baseline UCB compared to the proposed UCB, for = -80 dB). This may lead to unacceptable power levels for such type of devices.

A suitable alternative solution for high values could be reached by introducing additional channels to the system. Table II shows the effect of an increasing number of channels per relay on the performance of our proposed method for = -80 dB. The results show that, to reach a success percentage of 90%, the number of channels should be increased to 7 per relay, resulting in a total of 35 × 4 = 140 available arms per device.

C. Performance Assessment with Queuing Needs at the Relays

In the following, we consider a system with = 4 relays and = 160 uniformly distributed MTDs. = 10 channels and = 5 power levels are considered per relay, leading to a maximum of × × = 200 arms per IoT device. The purpose of this setting is to validate the role of queuing at the relays. As explained in Section IV, to avoid the need for queuing, relay can support a maximum of 0, connected devices. The value of 0, , estimated using [START_REF] Jiang | Distributed Layered Grant-Free Non-Orthogonal Multiple Access for Massive MTC[END_REF], for one simulation of the considered setting is shown in Table III. Table III shows that there is no need for queuing at the level of relays 2 and 4. In fact, the number of users that can connect to relays 2 and 4 per timeslot, i.e., the number of users able to choose unique arms, is lower than the maximum number of supported users. This is not the case for relays 1 and 3 where a queue may be needed if the number of connected devices exceeds the maximum supported number 0, .

In Figure 7, the instantaneous number of successfully connected devices per relay is shown for this setup. The devices start by exploring the available arms until reaching convergence. Relay 3 reaches its overflow requirements since the number of connected devices to this relay when reaching the first convergence state is larger than its maximum supported number 0, . Hence, relay 3 enters a timeout period between timeslots 40 and 90, in which a zero reward is returned to all devices trying to associate with it. During this period, users previously associated with relay 3 attempt to connect to other less congested relays. Having succeeded to do so, a first convergence is reached within 100 timeslots and then full convergence is reached within 200 timeslots, as shown in Figure 7. Upon reaching convergence, the rate requirements of all relays and IoT users are satisfied.

VI. CONCLUSION

In this paper, uncoordinated joint channel and power allocation in a NOMA uplink system equipped with full-duplex relays was studied. The considered problem was modeled using the multi-armed bandits framework with zero-reward on collision. A solution based on the UCB algorithm that does not require any communication between devices and that satisfies the QoS requirements of all devices was proposed. Simulation results confirm the convergence of the proposed algorithm. Moreover, they show that, compared to two baseline schemes, the proposed allocation technique increases the number of satisfied IoT users whilst decreasing the needed transmit power. The proposed algorithm was then extended to account for buffering requirements in case of congestion at the relay level. In the future, we will extend this study to the case of heterogeneous systems and multiple-input multiple-output relays.

Figure 1 :

 1 Figure 1: System model with full-duplex relaying.

Algorithm 1 :

 1 UCB Power Minimization Initialization: Each device generates a random permutation of A . During the first |A | iterations, plays the arms iteratively and updates the utilities accordingly. for t=1:T do for i=1:N do // Best arm identification: * = argmax ∈A (

Algorithm 2 :Figure 2 :

 22 Figure 2: Instantaneous success probability.

Figure 3 :Figure 4 :Figure 5 :Figure 6 :

 3456 Figure 3: Average success percentage for a varying requested rate.

Figure 7 :

 7 Figure 7: Number of connected devices per relay.

Table I :

 I Design parameters values

		0.3 Mbps	23 dBm
	×	5W	-120 dB
		156 kHz	50 ms
		0.3	0.5

Table II :

 II Success percentage of the proposed technique for a varying number of channels per relay, with = 5 power levels.

	Number of channels 2	3	4	5	6	7
	Success Percentage 15.98 17.54 25.05 69.77 79.91 89.54

Table III :

 III Maximum number of supported IoT users per relay,

	0, .
	Relay	1	2	3	4
	Arms	50	50	50	50
	0,	43	87	36	75

ACKNOWLEDGMENT

This work was partly carried out in the context of Beyond5G, a project funded by the French government as part of the economic recovery plan, namely "France Relance", and the investments for the future program.