Riaz Shah

Faisal Bahdur

Iqbal Umer

Muhammad Javed
email: javedsosm2@gmail.com

Implementation of multiple thread pools based on distribution of service times

Keywords: Thread Pool Allocator, Slave Listener, Request Listener, Concurrency, Thread Pool, Multithreading, Threads

The demand of internet is increasing day by day and web server receives millions of hits per day.

To manage these raising demands high performance web servers are required. Multithreading is the elementary approach used in web server to achieve high performance and handle number of requests from clients. The existing scheme which is distributed frequency based optimization strategy DFBOS cannot distribute thread pools on the basis of service times due to this starvation occurs. When starvation occurs it decreases response time and increases wait time. The research aims is to explore the implementation of multiple thread pools based on distribution of service times to avoid starvation and achieve concurrency in server site. For comparing both strategies we have used a simulator named as Thread Pool Tester TPT which is a JAVA based simulator and it has shown that proposed strategy is much better than the existing DFBOS. The analysis shows that proposed scheme is increases the response time and reduces the wait time.

Introduction

To achieve concurrency at server side application we use multi threading. In similar programs various activity can be processed in multithreading. Multi threading execute multiple process in multiple CPUs, therefore provide enhance application retort.

Mostly used techniques to increase throughput of CPUs is multithreading. Thread-pool administration is considered one of the challenging job in this method since it require sufficient numbers of thread in the thread-pool tuning system which can provide better responsive time and soaring utilizations of scheme possessions [START_REF] Kang-Lyul | A novel predictive and self-Adaptive Dynamic Thread Pool management[END_REF].

Multithreading construction used systems resource competently and as well as provides multi processor architecture hence it has become the supreme option for execution of server side applications. Although require run times process of memory allotment and de allotment for threads creation and distortion [START_REF] Ling | Analysis of optimal thread pool size[END_REF].

Thread pool and thread per request is the most commonly used multithreading architecture. The both architecture has its own advantage and disadvantage. Now we are explaining it briefly as following.

In thread per request architectures for every client request a thread is created. After the completion of request the thread is destroyed. The advantages of these architectures are that it is easily to implement as compare to other architecture. This strategy can be used for task which has extensive running time like database query which is coming from a number of users on different servers. The drawbacks of this strategy are that it consume system properties if continue numbers of customer are convey demand to server.

In thread-pool architectures thread are generated in the initial stage of thread-pool-system. That structural design is beneficial for O-R-Bs in which numbers of resource can be utilized by OS and kept initially. In this strategy request is processed continuously although if the numbers of request on server surpass then the existing thread in the pool then the exceeding task has to stay for execution in the queue. The request may be either in ready or in waiting condition. The benefit of this strategy in multi-threading architectures is that it is simple to execute. The drawbacks of this strategy are unnecessary framework changing and synchronization complexity.

Although threads pool as compare to thread-per-request architecture enhance system throughput and decrease reaction times for users [START_REF] Schmidt | Evaluating Architectures for Multi-threaded CORBA Object Request Brokers[END_REF].

Consequence of Implement Multiple Thread Pools Based On distribution of Service

Times for DFBOS

The existing strategy DFBOS equally is distributed the request among the different node's Increase the node's performance through the load balance mechanism. In DFBOS they cannot distribute thread pools on the basis of service time. In DFBOS starvation can occur which decrease the performance of system.

In Our proposed strategy we can implement Multiple Thread Pools Based On distribution of Service Times for DFBOS, Which can avoid the starvation and also improve the performance of the system. Research question for the proposed scheme is as under.

i. How to remove starvation of short tasks in a Multiple Thread pool environment?

Assessing Effect of Implementation of Multiple Thread-Pools Based On distribution of

Service-Times for DFBOS

For using the effect of implementation of multiple thread pools based on distribution of service times for DFBOS, we use the following methodology. ii.

Request counter can be used for CAS code arrangement for shared resources.

iii.

Code can be constructed for Implementation of Multiple Thread Pools Based On distribution of Service Times.

iv.

For performance association we use the simulator named as thread pool tester which is based on java applications for our proposed strategy that is the implementation of Multiple Thread Pools Based On distribution of Service Times with existing strategy DFBOS .

Related Work

[1] D. C Shmist and S Vinosky, developed a thread pool system which consists of stable number of threads in the pool. In this thread pool at the start the thread per request replica are used. They consist of queue in which the incoming requests from clients are inserted in first in first out order (FIFO). The threads which are exist in the thread pool pick up these request and when the number of request are increased then the available threads in the pool, then more threads are generated in the thread pool until it reached to the soaring water mark , and then no more threads can be created.

In [START_REF] Yue-Shan | Design and Implementation of Multi-Threaded Object Request Broker[END_REF] the completion of five tasks, and the increase the thread pool size after the execution of five tasks then the average ideal time is increased by one percent. Without any justification they can used the different values in algorithm.

In [START_REF] Ogasawara | Dynamic Thread Count Adaptation for Multiple Services in SMP Environments[END_REF] T. Ogasawara, proposed a thread pool model in which the threads are borrowed for other pools, and they can use the heuristic computations. The measured the average ideal time for those jobs which have in waiting state after about 20ms, then he compare the average ideal time for all threads which are in waiting queues and then from the less thread pool system they can get borrow. Here they can measure the average ideal time of the waiting threads but they show the lower performance.

In [START_REF] Chen | A Dynamic Adjustment Mechanism with Heuristic for Thread Pool in Middleware[END_REF] N. Chen and P. Lin, proposed a thread pool system for online server applications. For proficient resource consumption they can use some extra factors in the thread pool system to adjust the pool size. They can calculate the average wait time of the queue request and increases the thread pool size at the extend when the wait time become turn down. The drawbacks of this paper is lower response time recorded by clients and when they can measure the average ideal time then the ready queue become padlock.

In [START_REF] Ling | Analysis of optimal thread pool size[END_REF] Y. Ling et al.., proposed the most favorable size for thread pool model. For efficient threads organization they can used the numerical examination and the can calculate the most favorable pool size by using the arithmetical and numerical calculations. To measure the favorable pool size they can use the thread background control time and creation time of thread. The drawbacks of this paper are it is difficult to measure the current parameters at kernel level and still it can't be measured by any web server.

In [START_REF] Hellerstein | Configuring resource managers using model fuzzing: A case study of the .NET thread pool[END_REF] J. L. Hellerstein , presented a method which can be used for analyzing and assigning the resource management called model fuzzing. They can also calculating the output criteria at different stages, and also analyze the concurrency level for different numbers of threads. They can use the model for analyzing the concurrency level by using different parameters. They can record the different alternatives of the current working system and the remaining alternatives and then compare with each other and provide the performance of the system. They can predict that the workload of the incoming and outgoing requests demonstrate diverse kind of attribute at diverse times, which is concluded a wrong prediction.

In [START_REF] Kim | Prediction-based Dynamic Thread Pool Management of Agent Platform for Ubiquitous Computing[END_REF] J. H. Kim et al.., proposed gaussion division for manipulating prospect guess. the request are coming the thread pool creating the threads automatically and when the numbers of request are decreased then it delete the threads automatically means that the thread pool system work dynamically. The prediction is unsuccessful at high and low request rate.

In [START_REF] Kang-Lyul | A novel predictive and self-Adaptive Dynamic Thread Pool management[END_REF] Kang Lyul Lee et al.., presented a guessing system named as narrative tendency exponential moving average system (T.E.M.A) to elaborating the E.M.A. They want to avoid the duplication of threads from E.M.A, by using arithmetical hypothesis they can calculate the rate of change of threads in the thread pool by prediction.

In [START_REF] Ramisetti | Design of hierarchical Thread Pool Executor[END_REF] S. Ramisetti and R. Wanker, proposed a thread pool model for cluster atmosphere. They can use for calculating the concentrated requests. In this model we have to add more machines in the cluster, there is no need of cluster alteration. For the efficiency of thread pool they can use the load balancing algorithm. The drawbacks of this system are to load balance between the different nodes, and also added the new nodes.

In [START_REF] Bhattacharya | Mobile Agent Based Elastic Executor Service[END_REF] A. Bhattacharya, proposed a thread pool model for disperse cluster atmosphere. The give the idea of perpetrator service which is disperse and which process a submitted task and control it by using mobile controller. The thread pool system consists of threads which is independent of each other and executed within pool. Here they can use load balancing by using mobile controller. The drawbacks of the paper are there in no archetypal confirmation.

In [START_REF] Bahadur | FBOS: Frequency Based Optimization Strategy For Thread Pool System[END_REF] In [START_REF] Ahmad | Load Balancing in Distributed Framework for Frequency Based Thread Pools[END_REF] Sheraz. A et all.., proposed a thread pool system which can distribute the load among different node by using the load balancer, the drawbacks of this paper is that they cannot distribute the thread pool on the basis of service time due to which starvation occurred.

The drawbacks of the existing system

i. The can only distribute the work load among the different.

ii. They cannot distribute the thread pool on the basis of service times.

iii. Starvation occurs which can decrees the performance of the system. iv.

Context switch and synchronization overheads.

To overcome the drawbacks of the existing DFBOS we implement multiple thread pools based on distribution of service times to avoid starvation and achieve concurrency in server site.

Material and Methods

In this section we will discuss about the proposed system named as implementation of multiple thread-pools based on distribution of service-times.

Assumption for Thread Pool System

i. Threads have the same precedence in the thread pool. In thread pool system each thread consumes the same amount of CPU time. In thread pool system there is no classification of clients i.e. client or supervisor, every clients have the same priority in the thread pool system.

ii. In Our proposed strategy we can implement Multiple Thread-Pools Based on distribution of Service-Times for DFBOS. This can avoid the starvation and improve the performance of the system.

Existing DFBOS Scheme

In figure 1 detect the new jobs then they can check it in workload profiling table for their service times and then send it to the required thread pool. In proposed scheme we can used two nature of jobs i.e.

100ms and 200ms, and also we can used the two pools one for 100ms and the other for 200ms.The proposed scheme improves the performance of system.

Figure 2 Block Diagram For Proposed Scheme

The proposed strategy is consisting of numerous classes. Here we can introduce the factors that are used in our proposed strategy one by one.

Offline Workload Profiling

d. Slave Pool Algorithm Flowchart

At the start of the slave node is connect by giving the IP of the main server. Then it can listens the request from the TP allocator if they cannot detect request it can listens continuously for request if detect the request then they check the stability if stability is no then send stability off to main server. And check the stability continuously until the node become stable and then send stability on message to the main server and listen for new node. If the node is stable then execute jobs.

Figure 6 Slave Pool Algorithm Flowchart

e. Thread Pool Allocator Algorithm Flowchart

The main object of our proposed scheme is the thread pool allocator. In this section we will discuss these performance metrics one by one below.

a. Throughput

The number of jobs which can be completed in one second are called throughput.

b. Response Time Of Jobs

The duration from the submission of jobs is to the completion of jobs and provides the required outputs are called the response time of jobs.

c. Thread Pools

Thread pools represent the number of running threads in the pool; in our proposed strategy we use the default thread pool, high service times and low service times thread pools.

d. Wait time

The interval which jobs spent in ready queue, the proposed scheme will minimize the wait time to achieve maximum performance.

Analyses

In this section we will discuss about the imitation atmosphere that we have used for validating our proposed thread pool system.

Thread -Pool -Tester

We use JAVA based simulator for examining dissimilar requests from clients to achieve the required performance in our proposed scheme. We load the proposed scheme i.e. implementation

Imitation Atmosphere

For validating our strategy we are performing simulation based environment which consist of three machines, one for server and one for client. 9, we can use the same load generation for existing and proposed strategy and the consequences are plotted from simulation which can be discussed one by one in the next section for analysis.

Analysis & Results

This sector we will discuss the contrast of existing scheme with proposed strategy. We can compare the proposed scheme that is implementation of multiple thread pools based on distribution of service times with existing strategy DFBOS on the basis of response time and wait time statistics.

Our proposed strategy can divide the thread pools based on distribution of service time which cannot be considered in DFBOS they can only balance the load on nodes. In proposed strategy we can divide the thread pool in low service time and high service time thread pools accordingly.

We can use the workload by Poisson distribution that is 100 λ. The client node can send two type of job that is 1kb and 10kb. The response time for 1kb job is 100ms and 10kb job is 200ms. We can store the 100ms job in low service time thread pool and 200ms job in high service time thread pool. After dividing the thread pool on the basis of service times the simulator can generate the graphs for wait time and response time and then we can compare the both strategies. For comparing both strategies we have used a simulator named as Thread Pool Tester TPT which is a JAVA based simulator and it has shown that proposed strategy is superior then DFBOS.

From our analysis we have concluded that the response time of our research work is worse than the existing DFBOS i.e. for 50 percentile the response time is 14% and for 90 percentile the response time is 13%. The analysis can also conclude that the wait time of proposed strategy is less as compare to the existing DFBOS i.e. wait time of proposed strategy is 1ms and existing scheme is 6ms. In future we can elaborate our research work to sensor thread level in distributed environments

  i. Reuse the current link queue of JAVA 5 for Implementation of Multiple Thread Pools Based On distribution of Service Times for DFBOS.

Figure 1 3 . 3 .

 133 Figure 1 Task Distribution without considering service time in DFBOS3.3. Implementation of Multiple Thread-Pools Based on Distribution of Service-TimesBefore the start of proposed scheme, first we discuss the limitation of existing strategy that they can distribute the work load through load balancer among the different nodes without considering service times in the result of which the starvation occurs. Figure2shows the implementation of multiple thread-pools based on distribution of service-times for DFBOS in which we can distribute the thread pool based on distribution of service times. In this scheme we can use the different nature jobs that is light and heavy weight jobs e.g. if we have a jobs which processing times is 100ms and the other one which have 200ms if the 200ms job acquires the CPU then the 100ms job will wait for 200ms for other jobs and 100ms for itself processing time then here the starvation will occur. To avoid starvation we can divide the thread pool in to low service time thread pool (100ms) and high service time thread pool (200ms). For validating our strategy we can use a simulator named as thread pool tester (TPT). TPT consist of two main agents that are client tire and the server tire. In server tire we can install the simulator and then embed our proposed thread pool. Before the addition of any clients we can run the main server switch for offline workload profiling table. And store they require workload in workload

Figure 3

 3 Figure 3 Main Server Algorithm Flowchart b. Slave Listener Algorithm Flowchart It starts the slave listener launch for detecting the new slave nodes; if they can't find the new slave node then it can search for it thoroughly. If they can detect the node then it can then it can check that is new are old. If they are not new then it can read the signal and update the stability

Figure 4

 4 Figure 4 Slave Listener Algorithm Flowchart

Figure 5

 5 Figure 5 Request Listener Algorithm Flowchart

Figure 7

 7 Figure 7 Thread Pool Allocator Algorithm Flowchart 3.4 Performance Metrics For analyzing our proposed strategy i.e. implementation of multiple thread pools based on distribution of service times with existing strategy DFBOS we use various performance metrics.

  of multiple thread pools based on distribution of service times for LDFBOS on a simulation environment named as Thread -Pool -Tester[START_REF] Bahadur | FBOS: Frequency Based Optimization Strategy For Thread Pool System[END_REF] generally known as TPT. Two agents combine together in Thread Pool Tester that is client and server. On different machine we run these both agents the machines have Intel core i5 which have four cores. The basic diagrammatical representation of our thread pool tester is shown in figure8.

Figure 8 .

 8 Figure 8. Graphical representation of simulator a. Production of Load The server has assembled according to the load generation having Poisson division 100λ. We have generated the two types of jobs that are 1kb and 10kb accordingly. Frequencies of each job are 50% that is 50 % for 1kb and 50% for 10kb. Response time of 1kb job is 100ms and 10kb job is 200ms. The cumulative load on the server as shown in figure 9.

Figure 9

 9 Figure 9 represents the load generation for both scheme. X -Axis represents time in seconds and Y -Axis represents number of requests. We can use the same load generation of 100 λ. We can perform the load generation for one minute.

Figure 9 .

 9 Figure 9. Load generation on server

Figure 10

 10 Figure 10 Performance comparisons of response times Figure 11 represents the comparisons of response time statistics for both strategies in percentile. X -Axis represents the response time in percentile and Y -Axis represents the values in milliseconds. Proposed scheme has increased the response time for 50 percentile is 14% and for 90 percentile is 13% as compare to existing scheme.

Figure 11

 11 Figure 11 Performance comparisons of response times statistics Figure 12 represents the comparisons of wait time for both schemes. X-Axis represents the responses / jobs and Y-Axis represents the wait time in milliseconds. Graph shows that proposed scheme has better wait time than existing scheme.

Figure 12

 12 Figure 12 Performance comparisons of wait times Figure 13 represents the comparisons of wait time statistics for both strategies. X -Axis represents the average wait time and Y -Axis represents the value of wait time in milliseconds.Graph shows that the wait time of proposed strategy is 1 millisecond and the existing scheme is 6 milliseconds.

Figure 13

 13 Figure 13 Performance comparison of wait time statistics 5. Conclusion and future work The proposed strategy can implement multiple thread pool based on distribution of service times for distributed frequency based optimization strategy DFBOS to avoid starvation and achieve concurrency in server site. In our research work we can divide thread pool on the basis of service time into low service time and high service time thread pool in distributed environments.

  and turnaround time of the request. The drawback of this paper is that they can use the locks which can slow the performance of the system.

F. Bahadur et al.., proposed a thread pool model called FBOS. Which compute the incoming client's requests, and it can decrease or increase the thread pool system automatically when the requests are coming. They can calculate different parameters such as response time, wait time

  shows the existing scheme i.e. DFBOS (Load Balancing in Distributed Frequency Based Optimization strategy) it consists of job in queue which stockpiles requests coming from user side and hold by a dynamic job in queue. This strategy is used for first in first out (F.I.F.O) data structures in which the requests are entered from one site and taken from other site of the queue after the execution. After the job in queue the request is hold by load balancer thread pool consists of threads which are used to fulfill the client's requests. Load balancer thread pool is connected to distributed slave server thread pool. When the requests are coming from users the load balancer thread pool divides the load equally to itself and also to the distributed slave serve thread pool. It can distribute the same workload among the slave nodes. After the completion of task it can be store in job out queue. Through the load balancer mechanism every node shows its performance. Load balancer divides the workload among the different nodes through the help of global table, listener and register.Listener is the pathway from loader balancer to the other slave node, and also it detects the new incoming task which is coming from user. It also sends demand to load-balancer for the addition or removing of new node. If new node is added to the load-balancer then it added them to the register and calls to global table.Register are used for the registration of new node when the listener listen about the addition of new to the load balancer. Then it call to the global-table for the verification of new node added or not, if the new node is added to the load balancer then the global table generates a specific IP for the already added node.

Global table consists of IP's of all nodes that are connected to the loader balancer, if the nodes are added or removed from load balancer it can maintain the IP's of that nodes. It can update the IP's of nodes.

The drawbacks of the existing method are that they can only distribute the work load between the various nodes without considering service times, in the results of that it occurs starvation .i.e. if we have different nature of jobs such that light and heavy weight if the heavy weight job acquire the CPU then the light weight job may wait until the processing are finished. To avoid starvation we can introduce a new technique in the next section i.e. implementation of multiple threadpools based on distribution of service-times.

  checking the new node if they can detect the new node then they can update the thread pool table. Initially we have only one thread pool in TP table i.e. local pool, the slave listener work continuously for detection of new node. After the TP table we can initiate the request listener and it work continuously for detection of new request from nodes, if they can

table. And store they require workload in workload profiling table. The status of jobs is non active initially. After WP table we can initiate the slave listener for

Table :

 : WP table is used to store the service time, status and the thread pool of jobs. Before the addition of any node we can run our main switch for storing the service times of jobs in WP table. Initially, the status of every job is non active and they can reside in local pool. Initially we have only two types of jobs that are 100ms and 200ms. When they request listener detect the jobs from the node, then they can check it in WP table for their

	Jobs	Service Times Status	Thread Pool
	Job1	100ms	Non active	1
	Job2	200ms	Non active	2

service time and then sent it to the required pool. After the detection of jobs its status will become active and also its thread pool will be changed in WP table. The jobs are in sorted order in WP table, Workload Profiling table as shown bellow.

Table 1

 1 Offline Work Load Profiling TableThread Pool Table:TP table consists of pools. Initially it consists of one local thread pool and when the new server are add to the system it can also added a separate pool for it. We can also check that thread pool is stable or not. TP table is demonstrated as follows.

	Pool IP	Stability
	1	Local host	Yes
	2	192.48.45	Yes

Table 2 Thread

 2 It can reallocate the thread pool in work load profiling table. Second it can add a new thread pool in thread pool table when the new slave nodes are detected. The last one it can send a job to a specific thread pool when the incoming requests are detected. The object of this class is initiated first. Our overall thread pool system is represented by this class. The object of this class initializes a whole system which is important for our thread pool system. When scheme starts it can create an instant of job-in-queue which can store the incoming user's request. When the user requests are coming for the first time it can process it in the default thread pool and also store its value in log table for further use, when the requests are for the first time from user it can also check it in log table for finding its processing time. If the same job is processed, before it can then send it to the required thread pool for further processing.When the main switch server are started for validating the proposed scheme it can initiate the work load profiling table for storing the service time, status and pool of the jobs. Then it can initiate the thread pool table. When the new node is added to the main server, it can add the new thread pool for it in TP table. The TP table updates continuously, it inserts new slave server. It starts the thread pool table which has only one local thread pool, which relates to the main server switch. After TP table it can launch slave listener for detecting the new slave node thoroughly. If they detect the new slave server then it can add a new thread pool for it in TP table. At second

	Local Thread -Pool -System:

Pool Table

Slave listener: Slave listener listens the port thoroughly for new node if the new node is detected then it update the thread pool table and then passes it to the thread pool allocator. It can add new thread pool in the TP table.

Request Listener: Request listener listens the incoming jobs from the clients thoroughly. When the jobs are detected it can update the work load profiling table and pass it to the thread pool allocator. Then the TP allocator allocates the thread pool for the new incoming jobs on the basis of service times.

Thread Pool Allocator:

The main agent of our proposed scheme is the TP allocator. It can call from two sides that is slave listener and from request listener. The TP allocator performs three main functions. last it can launch the request listener for detecting the new request from clients thoroughly and can update the. At last the main server can initiate the local pool. WP table and then passes it to the TP allocator.

  It starts the TP allocator called by slave listener. If they can detect the new node then they can update the work load profiling table and can also add a new pool in thread pool table, and then it can divide the incoming request from the node by using the formula (Compute n = Active Jobs / Numbers of Pools ). If new node is not detected then thread pool allocator is called by request listener if the new incoming jobs are not new means the jobs which are coming from nodes are already stored in work load profiling table. Then it select the job by indexing and then check the stability of the node if the node is not stable show the message that server is busy. If the node is stable then send job to the pool and exit. If the request listener detects the new job then they can distribute it by using the formula (Compute n = Active Jobs / Numbers of Pools). If the work load profiling table has more numbers of jobs then the thread pools, they can select the n number of jobs from WP table and send it to the next pool in TP table. If the entry is null in TP table then it goes back pointer in TP table. If the entry is not null then it checks that all jobs are sorted if yes then exit otherwise they can select the n number of jobs from WP table and can assign it to the next thread pool in thread pool table.

  We have used Microsoft Window 10 in server machine and Microsoft Window 7 on remaining two systems. The server node is Intel ® Core TM i7 and the remaining two nodes are Intel ® Core TM i5 processors. The main memory of the server node is 8GB and the other two nodes are 4GB. We have performed simulation for one minute according to the load generation having Poisson distribution 100λ. We have generated the two

	types of jobs that are 1kb and 10kb accordingly. Frequencies of each job are 50% that is 50 % for
	1kb and 50% for 10kb. Response time of 1kb job is 100ms and 10kb job is 200ms. The
	cumulative load on the server as shown in figure


David Doukhan
email: ddoukhan@ina.fr

Jean Carrive
email: jcarrive@ina.fr

Investigating the Use of Semi-Supervised Convolutional Neural Network Models for Speech/Music Classification and Segmentation

Keywords: Speech/music discrimination, Audio segmentation, Convolutional Neural Networks, Music Information Retrieval, Multimedia Indexation

A convolutional neural network architecture, trained with a semi-supervised strategy, is proposed for speech/music classification (SMC) and segmentation (SMS). It is compared to baseline machine learning algorithms on three SMC corpora and demonstrates superior performances, associated to perfect medialevel speech recall scores. Evaluation corpora include speechover-music segments with durations varying between 3 and 30 seconds. Early SMS results are presented. Segmentation errors are associated to musical genres not covered in the training database, and/or with close to speech acoustic properties. These experiments are aimed to help the design of novel speech/music annotated resources and evaluation protocols, suited to TV and radio stream indexation.

I. INTRODUCTION

Speech/Music classification (SMC) task consists in predicting if a given audio excerpt contains speech or music. The excerpts are supposed to be pure and contain either speech or music. Classified excerpts may have variable durations. Longer excerpt durations are known to make classification tasks easier. Speech excerpts are generally defined as containing spoken speech: this includes speech alone, superposed voices, and speech over music. Music excerpts are defined as containing instruments, instruments mixed with lyrics, or a cappella vocals. These technologies attracted much interest for the management of large multimedia collections: selection of optimal audio compression strategy at Swedish radio [START_REF] Ericsson | Automatic speech/music discrimination in audio files[END_REF], as well as tag correction in Deezer's catalog [START_REF] Royo-Letelier | Detection and characterization of singing voice using deep neural networks[END_REF].

Speech/Music segmentation (SMS) task consists in splitting audio streams into pure speech and pure music segments [START_REF] Lavner | A decision-tree-based algorithm for speech/music classification and segmentation[END_REF]. SMS algorithms are often based on frame-level SMC procedures, followed by a post-processing step (mean filtering, dynamic programming, etc.). SMS is a pre-processing stage required for several higher level indexation tasks such as speech and speaker recognition, song and musical genre recognition. Consequently, their development has received considerable attention from speech analysis and music information retrieval communities, illustrated by several evaluation campaigns, e.g. ESTER [START_REF] Galliano | The ester 2 evaluation campaign for the rich transcription of french radio broadcasts[END_REF], Albayzín-2014 [START_REF] Castán | Albayzín-2014 evaluation: audio segmentation and classification in broadcast news domains[END_REF] or MIREX 2015 [START_REF]Mirex 2015 music/speech classification and detection and challenge[END_REF].

This paper presents the ongoing research on SMC and SMS tasks carried out at French National Institute of Audiovisual (Ina). Ina is a public institution in charge of the preservation, digitization, distribution and dissemination of the French audiovisual heritage. Ina's archives represent 70 years of radio and 60 years of TV programs, for a total of 15 million hours. The integration of SMS technologies in Ina workflows would allow a fast localization of interest areas within audio recordings, and address several identified needs. SMS may help speeding up media descriptions processes, which are performed manually by professional archivists. Manual media description is expensive, and associated to variable levels of detail: TV broadcast news are described with greater details than early radio collections. Consequently, SMS may ease the browsing and exploitation of under-documented archive contents. Latest identified use-case is music track segmentation, aimed at detecting and measuring the duration of musical tracks, in order to calculate the amount of royalties to be paid to rights collection societies.

The work presented in this paper is a preliminary study on SMC and SMS issues, using Convolutional Neural Networks (CNN). It is motivated by the excellent results reported with these architectures on MIREX 2015 SMC and SMS tasks, consisting in classifying 30-seconds long pure music and pure speech audio files. MIREX 2015's best SMC results were obtained using CNNs trained using fully supervised procedures: a MFC-based model with 1 convolutionnal layer [START_REF] Lidy | Spectral convolutional neural network for music classification[END_REF], and a CQT-based model with 3 convolutionnal layers [START_REF] Royo-Letelier | Mirex 2015 music/speech classification[END_REF]. The main contribution of this paper is the description of MFCbased CNN's performances on publicly available datasets, using shorter audio segments (from 3 to 30 seconds), as well as speech-over-music excepts. Another major contribution is the proposal of an unsupervised SMC training strategy used in the first layer of the network, allowing to obtain visually relevant audio classification features.

This paper is structured as follows. Section II describes the audio feature extraction process. Section III describes the proposed CNN architecture and training strategy. Section IV and V describe the corpora used for SMC evaluation, and the corresponding results. Section VI describes the early results obtained on the SMS task. Section VII provides a summary of the results obtained, and introduces our future work.

II. FEATURE EXTRACTION AND NORMALIZATION

Audio excerpts are downsampled to 16KHz mono signals. Mel-frequency cepstrums (MFC) corresponding to 40 Mel bands are extracted from 20 ms frames, sampled with a 10 ms time step. Adjacent frames are concatenated using a contextual length varying between 1 (no context, 40 dimensions) and 50 (context of 500 ms, 2000 dimensions). The resulting features are firstly normalized at the media level through a cepstral mean subtraction and a standard deviation division process. Similarly to patch normalization procedures, mean and standard deviation are also computed for each feature vector, and used to perform local mean subtraction and standard deviation division process. III. SEMI-SUPERVISED CONVOLUTIONAL NEURAL NETWORK MODEL CNNs are a type of feed-forward artificial neural network, in which the response of neurons to stimuli is triggered by convolution operations [START_REF] Lecun | Convolutional networks for images, speech, and time series[END_REF]. The weights are organized into a set of filters allowing the detection of localized spatio-temporal patterns, behaving like task-oriented feature extractors. Pattern shift and space invariance is achieved through the use of pooling operations.

In the scope of this study, several CNN architectures and training strategies were implemented using Tensorflow [START_REF] Abadi | Tensorflow: Large-scale machine learning on heterogeneous distributed systems[END_REF] and compared using variable number of convolutional and densely connected layer, filter shapes, pooling strategies, regularization methods. Figure 1 shows the structure of the CNN model associated to the best performances, reported in the next sections.The model input is composed of 50 MFC frames, corresponding to a 500 milliseconds temporal resolution. The frames are first processed by two convolutional layers (128 10 × 10 and 32 5 × 5 filters) reduced by max-pooling layers (5 × 5 and 2 × 2). These layers are followed by two densely connected layers of 128 neurons, which are associated to dropout rates of 0.75 and 0.5 [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF]. Rectified linear unit (ReLU) activation functions are used between layers. The output layer is composed of two neurons, normalized using a softmax function, corresponding to the detection of music or speech. The first convolutional layer of the network is trained using an unsupervised procedure based on Spherical K-Means and ZCA whitening [START_REF] Coates | Learning feature representations with kmeans[END_REF], with K being the number of filters of the first layer. Filters obtained through this procedure are associated to unit l 2 norm, which avoids irregular neural response magnitude, and prevents overfitting. First layer's filters are associated to visually relevant features (vertical, horizontal and diagonal patterns) illustrated by figure 2. Visual relevance of these shapes is also a desirable property (see [START_REF] Royo-Letelier | Detection and characterization of singing voice using deep neural networks[END_REF] for filter shapes associated to fully supervised regularization strategies). The first layer stays constant during the training procedure, and remaining layers are trained using supervised Adaptive Moment Estimation (Adam) gradient descent optimization algorithm [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], for a maximal amount of 30 epochs, and are stopped when the accuracy on the training set reaches 99.9%.

IV. SPEECH/MUSIC CLASSIFICATION CORPORA Three publicly available SMC corpora were used, each of them being associated to specific speaking styles, musical genres, and track durations. Music, Speech, and Noise Corpus (MUSAN) [START_REF] Snyder | Musan: A music, speech, and noise corpus[END_REF] contains about 60 hours of speech and 42 hours of music. 20 hours of the speech material are full audiobook chapters, read in 12 languages, and correspond to clean read speech. The 40 re-maining hours of speech material are composed of US government hearings, committees and debates, containing background noise (noisy spontaneous and prepared speech). Music material is provided with annotations related to performers and musical genres. 3 seconds long segments were extracted from MUSAN corpus through a procedure aimed at measuring the ability of the models to discriminate speech versus music on small segments and providing controlled variations on the training and testing examples. MUSAN's speech material is composed of 2 randomly selected segments per track in US government material, and 3 segments per audiobook, amounting to 1024 segments, for a duration of 9 minutes for each speech category. Music material is composed of some 6 randomly extracted segments per artist, amounting to 924 segments obtained from 252 different artists. Automatic energy-based procedures were designed to discard empty segments, as well as segments with less than one second of activity.

V. SPEECH/MUSIC CLASSIFICATION EVALUATION CNN performances are estimated on SMC corpora and compared to baseline machine learning algorithms provided in Scikit-learn toolbox [START_REF] Pedregosa | Scikit-learn: Machine learning in python[END_REF]: Support Vector Machines (SVM) with RBF kernel, and Gaussian Mixture Models (GMM) with diagonal covariance matrices and varying number of gaussians (32 to 512). Models are evaluated using input vectors corresponding to concatenated MFC frames, with contextual length varying between 1 (no context) and 50 (context of 500 milliseconds). Best results are obtained using the highest contextual length, which was limited to 50 in order to allow acceptable temporal resolution in following segmentation tasks (10 milliseconds time step and 500 milliseconds context).

Effectiveness is reported at the frame level and at the media level. Frame-level decisions are based on the raw instantaneous predictions of SMC models obtained for each input vector (500 milliseconds context). The frame-level decision correspond to the class (speech or music) with highest probability. Medialevel decisions are based on the products across all framelevel probabilities per class obtained for an audio excerpt (100 frame-level predictions per second). Best instantaneous framelevel estimates does not necessarily produce best media-level estimates, nor best segmentations. SMC performances reported are those associated to the models achieving the best medialevel performances, which is more representative of the final use-case.

Models are compared using a 5-fold cross-validation process. Effectiveness is described using speech, music and mean recalls, providing a description of the most frequent classification errors. MUSAN corpus evaluations are carried out using constraints on cross-validation, in order to group subtracks corresponding to a same performer in the same folds. These constraints avoid testing a model on a track produced by a performer found in the model's training set. Similar constraints are applied for segments obtained from the same audio-book or us-government track.

Tables I, II and III report the results obtained on GTZAN, Scheirer-Slaney and MUSAN corpora. All classification algorithms achieve 100% correct media-level classification rate on GTZAN corpora. This better media-level recall is probably due to the duration associated to GTZAN samples (30 seconds, instead of 15 in Scheirer-Slaney and 3 in MUSAN). For all other tasks, CNN models achieve better classification results than GMM and SVM, both at the frame and media level. For all of these methods, frame-level recall is higher for speech than for music, as several music frames are incorrectly predicted as speech. CNN predictions are associated to a perfect media-level speech recall score for all corpora. CNN produces only 2 classification errors on Scheirer-Slaney corpus, and 5 on MUSAN corpus. Manual analysis of these errors shows vocal singing excerpts without instrumental accompaniment, or singing style close to regular speech, that are predicted as speech. An additional error found in Scheirer-Slaney corpus corresponds to a pure music track incorrectly classified as speech. This is explained by the presence of 3 training samples containing the same music track with superposed speech, learned as speech excerpts. Frame-level results obtained on Scheirer-Slaney corpus are worse than those obtained on other corpora, since this corpus contains speech-over-music excerpts, harder to discriminate than pure speech or pure music segments. Best frame-level results are obtained on MUSAN corpus; this may be explained by the low variability observed within its speech material.

VI. SPEECH/MUSIC SEGMENTATION EVALUATION

This section reports the early results obtained on the SMS task, using the CNN architecture associated to the best SMC performances. CNN model was trained using GTZAN, Scheirer-Slaney and MUSAN corpora described in the last sections, corresponding to about 3 hours of annotated material. Raw frame-level music and speech probabilities are obtained using a step size of 10 milliseconds. Viterbi algorithm is used to infer the most likely state sequence (speech or music) from these raw estimates, allowing to increase the robustness of CNN's predictions.

MIREX 2015 speech/music detection training examples material [START_REF]Mirex 2015 music/speech classification and detection and challenge[END_REF] was used for this evaluation (the training examples material is different from MIREX evaluation material which is not public). It contains about 5 hours of radio streams, corresponding to 4 hours of music, 1 hour of speech, 7 minutes of speech-over-music, and 6 minutes of other phenomena (pauses, applause, etc.). It is split into 7 tracks associated VII. DISCUSSION AND FUTURE WORK This study presents the use of a CNN architecture on several SMC tasks and on a SMS task. The proposed model shows clear advantages over SVM and GMM, both at the frame level and at the media level. All speech segments used in classification evaluations, with to durations between 3 and 30 seconds, are correctly classified. The proposed semi-supervised training procedure allows to obtain slightly better SMC results than fully supervised approaches on MUSAN corpus (not reported in last sections), resulting in a reduction of two media-level errors over 1948 samples. This trend needs to be confirmed using harder SMC evaluation protocols, as well as larger evaluation corpora.

Errors related to music excerpts, recognized as speech, are mostly associated to a cappella or predominant vocals, and singing styles close to regular speech (hip hop, shamanic psalms, country vocals, opera recitative style, etc.). Scheirer-Slaney corpus is the only classification corpus used in this study including speech-over-music samples, resulting in the lowest frame-level classification results. These findings suggest to constitute training and evaluation corpora containing these difficult musical genres, and to systematically integrate speechover-music samples in our future evaluation procedures.

Work in progress covers the constitution of a representative music segment dataset, with annotated variations related to genre, singing style, performer, and track identification. The use of speech databases containing speaker identities, and speaking styles annotations is required to improve control over our next evaluations. This issue may be partly addressed using a 2290-speaker corpus realized at Ina [START_REF] Salmon | An effortless way to create large-scale datasets for famous speakers[END_REF]. The hardest issue is the design of models able to discriminate speech-overmusic and music, especially for music having vocal acoustic properties close to regular speech (hip hop, eletro, etc.). This will be addressed through data augmentation strategies, here consisting in artificially superposing speech to music [START_REF] Razik | Comparison of two speech/music segmentation systems for audio indexing on the web[END_REF]. The last issue is the constitution of a music track segmentation corpus, including early challenging archive documents representative of the diversity of Ina's collections [START_REF] Doukhan | Simple neural representations of speech for voice activity detection and speaker tracking in noisy archives[END_REF].

Figure 1 .

 1 Figure 1. Proposed convolutional neural network architecture for frame-level speech/music classification

Figure 2 .

 2 Figure 2. 10 × 10 Filters used in the first layer of the CNN

TABLE I .

 I SPEECH/MUSIC CLASSIFICATION RESULTS OBTAINED ON GTZAN CORPUS[START_REF] Tzanetakis | Musical genre classification of audio signals[END_REF] USING 120 30-SECONDS LONG EXCERPTS

	Algorithm	Frame-level Recall	Media-level Recall
		Speech	Music	Mean	Speech	Music	Mean
	GMM	89,18	84,05	86,61	100	100	100
	SVM	95,64	91,06	93,35	100	100	100
	CNN	98,16	96,25	97,21	100	100	100

TABLE II

 II

	. SPEECH/MUSIC CLASSIFICATION RESULTS OBTAINED ON
	SCHEIRER-SLANEY CORPUS [15] USING 240 15-SECONDS EXCERPTS
	Algorithm	Frame-level Recall	Media-level Recall
		Speech	Music	Mean	Speech	Music	Mean
	GMM	85,28	81,03	83,15	99,29	98,00	98,64
	SVM	93,01	89,62	91,32	98,54	98,00	98,27
	CNN	93,79	91,79	92,79	100	98,00	99,00

TABLE III .

 III SPEECH/MUSIC CLASSIFICATION RESULTS OBTAINED ON MUSAN CORPUS[START_REF] Snyder | Musan: A music, speech, and noise corpus[END_REF] USING 1948 3-SECONDS LONG EXCERPTS

	Algorithm	Frame-level Recall	Media-level Recall
		Speech	Music	Mean	Speech	Music	Mean
	GMM	97,32	95,04	96,18	99,41	97,17	98,29
	SVM	97,17	94,5	95,84	99,8	97,41	98,6
	CNN	99,36	98,85	99,11	100	99,46	99,73

TABLE IV .

 IV SPEECH/MUSIC SEGMENTATION RESULTS OBTAINED ON MIREX 2015 TRAINING EXAMPLES MATERIAL Three files contain ethnological recordings, including shamanic singing and psalms. One contains country music and blues, including a cappella singing. Last track contains Irish folk music. TableIVreports frame-level results obtained for all of these genres. Results for frames not associated to speech or music are not reported. Perfect results are obtained for material associated to instrumental classical music, which is the easiest category, well represented in our training database. Errors found in ethnological recordings are mostly associated to shamanic psalms, not covered in our training set, which are detected as speech. Errors found in the country music material are mostly associated to a cappella singing detected as speech. This source of error is coherent with studies reporting similar acoustic properties in country singer's speech and singing[START_REF] Stone | Formant frequencies in country singers' speech and singing[END_REF].

	Radio Stream	Raw Frame-level Recall	Frame-level Recall after
	Genre				Viterbi post-processing
		Speech	Music	Mean	Speech	Music	Mean
	Classical	91.30	97.66	94.48	100	100	100
	Country	92.94	73.45	83.19	96.82	88.55	92.69
	Ethnic	95.15	71.49	83.32	99.18	79.26	89.22
	Irish	95.08	88.32	91.70	99.38	96.50	97.94
	to specific musical genres. Two tracks contain instrumental
	classical music.


Meropi Topalidou

Daisuke Kase

Thomas Boraud

Nicolas P Rougier
email: nicolas.rougier@inria.fr

One critic, two actors Evidence for covert learning in the basal ganglia

This paper introduces a new hypothesis concerning the dissociated role of the basal ganglia in the selection and the evaluation of action that has been formulated using a theoretical model and confirmed experimentally in monkeys. To do so, prior to learning, we inactivated the internal part of the Globus Pallidus (GPi, the main output structure of the BG) with injections of muscimol and we tested monkeys on a variant of a two-armed bandit task where two stimuli are associated with two distinct reward probabilities (0.25 and 0.75 respectively). Unsurprisingly, performance in such condition are at the chance level because the output of basal ganglia is suppressed and they cannot influence behaviour. However, the theoretical model predicts that in the meantime, values of the stimuli are nonetheless covertly evaluated and learned. This has been tested and confirmed on the next day, when muscimol has been replaced by a saline solution: monkeys instantly showed significantly improved performances (above chance level), hence demonstrating they have covertly learned the relative value of the two stimuli. This tends to suggest a competition takes place in the Cortex-BG loop between two actors, one of whom being sensitive to criticism and the other not. Ultimately, the actual choice is valuated, independently of the origin of the decision.

Introduction

The now classical actor-critic model of decision making elaborated in the 1980s posits that there are two separate components in order to explicitly represent the policy independently of the value function. The actor is in charge of choosing an action in a given state (policy) while the critic is in charge of evaluating (criticizing) the current state (value function). This classical view has been used extensively for modelling the basal ganglia [START_REF] Suri | A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task[END_REF][START_REF] Suri | TD models of reward predictive responses in dopamine neurons[END_REF][START_REF] Doya | Reinforcement learning: Computational theory and biological mechanisms[END_REF][START_REF] Glimcher | Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis[END_REF][START_REF] Doll | The ubiquity of model-based reinforcement learning[END_REF] even though the precise anatomical mapping of these two components is still subject to debate and may diverge from one model to the other [START_REF] Redgrave | What is reinforced by phasic dopamine signals?[END_REF][START_REF] Niv | Reinforcement learning with Marr[END_REF]. However, all these models share the implicit assumption that the actor and the critic are acting in concert, i.e. the actor determines the policy exclusively from the values estimated by the critic, as in Q-Learning or SARSA. Interestingly enough, [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] noted in their seminal work that one could imagine intermediate architectures in which both an action-value function and an independent policy would be learned. One legitimate question is thus to wonder whether the role of the BG critic is restricted to the exclusive evaluation of the BG actor or if this role might be considered to be more general. In other words, can the BG critic role extend beyond the basal ganglia and evaluate any actor, independently on its origin?

We support this latter hypothesis based on a decision-making model that is grounded on anatomical and physiological data and that identify the cortex-basal ganglia (CBG) loop as the actor. The critic -of which the Substantia Nigra pars compacta (SNc) and the Ventral Tegmental Area (VTA) are essential components -interacts through dopamine projections to the striatum [START_REF] Leblois | Competition between feedback loops underlies normal and pathological dynamics in the basal ganglia[END_REF]. Decision is generated by symmetry breaking mechanism that emerges from competitions processes between positives and negatives feedback loop encompassing the full CBG network [START_REF] Guthrie | Interaction between cognitive and motor cortico-basal ganglia loops during decision making: a computational study[END_REF]. This model captured faithfully behavioural, electrophysiological and pharmacological data we obtained in primates using implicit variant of two-armed bandit tasks -that assessed both learning and decision making -but was less consistent with the explicit version (i.e. when values are known from the beginning of the task) that focus on the decision process only. We therefore upgraded this model by adding a cortical module that is granted with a competition mechanism and Hebbian learning [START_REF] Doya | Complementary roles of basal ganglia and cerebellum in learning and motor control[END_REF]. This improved version of the model predicts that the whole CBG loop is actually necessary for the implicit version of the task, however, when the basal ganglia feedback to cortex is disconnected, the system is still able to choose in the explicit version of the task. Our experimental data fully confirmed this prediction [START_REF] Piron | The globus pallidus pars interna in goal-oriented and routine behaviors: Resolving a long-standing paradox[END_REF] and allowed to solve an old conundrum concerning the pathophysiology of the BG which was that lesion or jamming of the output of the BG improve parkinsonian patient motor symptoms while it affects marginally their cognitive and psychomotor performances.

An interesting prediction of this generalized actor-critic architecture is that the valuation of options and the behavioural outcome are segregated. In the computational model, it implies that if we block the output of the basal ganglia in a two-armed bandit task before learning, and because reinforcement learning occurred at the striatal level under dopaminergic control, this should induce covert learning when the model chooses randomly. The goal of this study is thus twofold: i) to present a comprehensive description of the model in order to provide the framework for an experimental paradigm that allow to objectivize covert learning and ii) to test this prediction in monkeys.

Material and methods

Behavioral experiments

Experimental procedures were performed in accordance with the Council Directive of 20 October 2010 (2010/63/UE) of the European Community. This project was approved by the French Ethic Comity for Animal Experimentation (#50120111-A). Data were obtained from two female macaque monkeys that were previously used in a related set of experiments. All the details concerning animal care, experimental setup, surgical procedure, bilateral inactivation of the GPi and histology can be found in [START_REF] Piron | The globus pallidus pars interna in goal-oriented and routine behaviors: Resolving a long-standing paradox[END_REF]. Raw data is available from [START_REF] Kase | Covert learning in the basal ganglia: raw data[END_REF] Computational modeling

Code and statistical analysis are available from [START_REF] Rougier | Covert learning in the basal ganglia: code[END_REF].

Architecture

The model is an extension of previously published models [START_REF] Leblois | Competition between feedback loops underlies normal and pathological dynamics in the basal ganglia[END_REF][START_REF] Guthrie | Interaction between cognitive and motor cortico-basal ganglia loops during decision making: a computational study[END_REF]. The model by [START_REF] Leblois | Competition between feedback loops underlies normal and pathological dynamics in the basal ganglia[END_REF]) introduced an action selection mechanism which derives from the competition between a positive feedback through the direct pathway and a negative feedback through the hyper-direct pathway in the cortico-basalthalamic loop. The model has been extended in [START_REF] Guthrie | Interaction between cognitive and motor cortico-basal ganglia loops during decision making: a computational study[END_REF] in order to explore the parallel organization of circuits in the BG. This model includes all the major nuclei of the basal ganglia (but GPe) and is organized along three segregated loops (motor, associative and cognitive) that spread over the cortex, the basal ganglia and the thalamus [START_REF] Alexander | Parallel organization of functionally segregated circuits linking basal ganglia and cortex[END_REF][START_REF] Albin | The functional anatomy of basal ganglia disorders[END_REF][START_REF] Alexander | Preparation for movement: neural representations of intended direction in three motor areas of the monkey[END_REF][START_REF] Parent | Functional anatomy of the basal ganglia. I. The cortico-basal ganglia-thalamo-cortical loop[END_REF]. It incorporates a two-level decision making with a cognitive level selection (lateral prefrontal cortex, LPFC) based on cue shape and a motor level selection (supplementary motor area, SMA, and primary motor cortex, PMC) based on cue position (see figure 2). In this latter model [START_REF] Rougier | Covert learning in the basal ganglia: code[END_REF], the cortex was mostly an input/output structure under the direct influence of both the task input and the thalamic output resulting from the basal ganglia computations. Consequently, this cortex could not take a decision of its own. In the present work, and to cope with our main hypothesis, we added a lateral competition mechanism in all three cortices (motor, cognitive, associative) based on short range excitation and long range inhibition and connections [START_REF] Coombes | Neural Fields: Theory and Applications[END_REF] between the associative cortex and cognitive (resp. motor ones) to allow for the cross talking of these structures. This competition results in the capacity for the cortex to make a decision, although with a slower dynamic when compared to the BG.

Dynamics

The dynamic of a decision in the model is illustrated on the bottom part of figure 3 before any learning has occurred. The left panel shows the dynamics of the unlesioned model where a decision occurs a few milliseconds after stimulus onset. However, in the lesioned model (right panel), the suppression of the GPi output slows down considerably the decision process compared to the intact model. This means that the decision is initially driven by the basal ganglia as it has been suggested in [START_REF] Pasupathy | Different time courses of learning-related activity in the prefrontal cortex and striatum[END_REF][START_REF] Turner | Basal ganglia contributions to motor control: a vigorous tutor[END_REF] Learning Dopamine modulates learning using reinforcement learning (RL) between the cognitive cortex and the cognitive striatum such that the decision made at the cognitive level can be used to bias the decision at the motor level. Hebbian learning (HL) occurs just after a motor action has been selected and carried out and modifies the connections (LTP) between the cognitive cortex and the associative cortex. It does not depend on reward but only on the actual cognitive and motor choices. It is to be noted that the cortical selection (resulting from lateral competition in the cortex) is slower than the cortico-basal selection such that the cortex is initially driven by the basal ganglia output (GPi), hence it learns from the statistics provided by the BG selection.

Lesion

Lesion in the model is made through the removal of all the connections between the motor (resp. cognitive) GPi and the motor (resp. cognitive) thalamus (red crosses on figure 3). This prevents any communication from the basal ganglia to the model but keeps intact the communication from the cortex to the basal ganglia.

Statistical Analysis

Theoretical and experimental data were analyzed using Kruskal-Wallis rank sum test between the two conditions (muscimol or control) for the 6 samples (10 first trials of D0 (control), 10 last trials of D0 (control), 10 first trials of D1; 10 last trials of D1; 10 first trails of D2; 10 last trials of D2) with posthoc pairwise comparisons using Dunn's-test for multiple comparisons of independent samples. P-values have been adjusted according to the false discovery rate (FDR) procedure of Benjamini-Hochberg. Results were obtained from raw data using the PMCMR R package [START_REF] Pohlert | The Pairwise Multiple Comparison of Mean Ranks Package (PMCMR). R package version 1.0. h ttp. CRAN. R-project[END_REF]. Significance level was set at P< 0.01.

Results

We designed a simple two-armed bandit task where two stimuli A and B are associated with different reward probability as explained on figure 1. The goal for the subject is to choose the stimulus associated with the highest reward probability, independently of its position. The protocol is split over two days. On the first day, a novel set of stimuli (with respective reward probability 0.25 and 0.75) is used while the GPi output is suppressed and a session is made of 60 consecutive trials. On the second day, GPi suppression is removed and the same set of stimuli from day 1 is used for another 60 consecutive trials. A control has been performed using a different set of stimuli to check for the learning curve in saline condition. Note that control and two-days session uses a different set of stimuli.

Theoretical results. We tested our hypothesis on a computational model using 12 different sessions (corresponding to 12 different initializations of the model). The upper part of figure 3B shows individual trials for all the different sessions. On day 1, we suppressed the GPi output by cutting the connections between the GPi and the thalamus. When the GPi output is suppressed on day 1, the performance is random at the beginning as shown by the average probability of choosing the best option (expressed in mean±SD) in the first 10 trials (0.408 ±0.161) and remain so until the end of the session (0.525 ±0.164). Statistical analysis revealed no significant difference between the 10 first and the 10 last trials. On day 2, we re-established connections between the GPi and the thalamus and the model has to perform the exact same task as for day 1 using the same set of stimuli. Results shows a significant change in behavior: the model starts with an above-chance performance on the first 10 trials (0.717 ±0.241) and this change is significant as compared to the beginning of D1 and as compared to the end of D1, confirming our hypothesis that the BG have previously learned the value of stimuli even though they were unable to alter behavior.

Experimental results. We tested the prediction on two female macaque monkeys which have been implanted with two cannula guides into the left and right GPi (see Materials and Methods section for details). In order to inhibit the GPi, we injected bilaterally a GABA agonist (muscimol, 1µg) 15 minutes before working session (see Materials & Methods) on day 1. The two monkeys were trained for 7 and 5 sessions respectively, each session using the same set of stimuli. Results on day 1 shows that animals were unable to choose the best stimulus in such condition from the start (0.433 ±0.236) to the end (0.492 ±0.250) of the session. Statistical analysis revealed no significant difference between the 10 first and the 10 last trials on day 1. On day 2, we inject bilaterally a saline solution 15 minutes before working session and animals have to perform the exact same protocol as for day 1. Results shows a significant change in behavior: animals start with an above-chance performance on the first 10 trials (P=0.667 ±0.213, as compared to the beginning of D1, as compared to the end of D1), confirming our hypothesis that the BG have previously learned the value of stimuli.

Discussion

These results reinforce the classical idea that the basal ganglia architecture is based on an actor critic architecture where the dopamine serves as a reinforcement signal. However, the proposed model goes beyond this classical hypothesis and propose a more general view on the role of the BG in behaviour and the entanglement with the cortex. Our results, both theoretical and experimental, suggest that the critic part of the BG extends its role beyond the basal ganglia and makes it de facto a central component in behavior that can evaluate any action, independently of their origin. This hypothesis is very congruent with the results introduced in [START_REF] Charlesworth | Covert skill learning in a cortical-basal ganglia circuit[END_REF] where authors show that the anterior forebrain pathway in Bengalese finches contributes to skill learning even when it is blocked and does not participate in the behavioural performance. To understand such hypothesis, it is important to reconsider how the basal ganglia forms a series of parallel loops (motor, associative, limbic) with the cortex and the thalamus. In higher order mammals, such as primates, the overall process starts in the sensory cortex, where stimuli are encoded, and ends up preferentially in the motor cortex from where an actual action is sent to the medullar motor neurons. Accordingly, in the previous versions of our model, the cortex was considered as a single input/output excitatory layer without intrinsic dynamic properties other than the I/O function of the populations. We then added a thalamic loop which allowed positive feedback but the different channels/populations were still independent. This limited autonomy is reasonable to mimic non-mammal vertebrate 3-layers dorsal mantle (aka pallium), but it is too rudimentary for the more complex architecture of the 6layers mammal cortex. The latter is remarkable for its organization in functional columns that are able to provide themselves positive feedback and to exert lateral inhibition on their neighbors. This architecture grants the cortex with dynamic properties that are far beyond what we were able to capture with our previous versions of the model: the balance between activation and inhibition allowed to toggle into various states that could be segregated enough to trigger different decisions. We therefore decided to add a cortical module encompassing these properties. We also grant it with the capacity to perform Hebbian learning based on the consensual hypothesis that this property is shared by most of the cortical structures. This new model captures the very essence of animal behavior in our two-armed bandit like task and provides a non-intuitive prediction about the occurring of covert learning in the striatum while the output of the BG is disrupted. It also reinforces the idea we proposed in [START_REF] Piron | The globus pallidus pars interna in goal-oriented and routine behaviors: Resolving a long-standing paradox[END_REF]) that a behavioral decision results from both the cooperation (acquisition) and the competition (expression) of two distinct but entangled actors under the evaluation of a single critic. However, if the theoretical model proposes the cortex to be a potential actor, we have no experimental evidence yet that this is the case for the monkeys. From the experiments, we conducted, we can only deduce that the monkeys are able to initiate random choices following injection of muscimol in the internal part of the Globus Pallidus and that they learn the value of the different stimuli in the meantime. If the theoretical model proposes that the motor cortex is able to make such move thanks to lateral competition, it is yet to be confirmed at the experimental level.

Legends

Table 1. Results from the Kruskal-Wallis rank sum test with posthoc pairwise comparisons using Dunn's-test for multiple comparisons of independent samples. Pvalues have been adjusted according to the false discovery rate (FDR) procedure of Benjamini-Hochberg. Results were obtained from raw data using the PMCMR R package [START_REF] Pohlert | The Pairwise Multiple Comparison of Mean Ranks Package (PMCMR). R package version 1.0. h ttp. CRAN. R-project[END_REF].

Figure 1.

A trial is made of the simultaneous presentation of two cues at two random positions associated with a fixed reward probability. The monkey has to choose a stimulus at the go signal and maintain this choice for one second. Reward is delivered according to the reward probability associated with the chosen stimulus. For all the experiments in this study, we used fixed reward probability set respectively to 0.75 and 0.25. Figure from [START_REF] Rougier | Covert learning in the basal ganglia: figures[END_REF], CC-BY.

Figure 2. The computational model is made of 12 neural groups organized along three segregated loops (motor, associative and cognitive) that spread over the cortex, the basal ganglia and the thalamus. Blue lines represent excitatory pathways; red lines represent inhibitory pathways and dashed lines represent emulated pathways (they are not physically present in the model but their influence is taken into account). Red crosses represent lesion sites emulating the muscimol injection in the GPi of the monkeys. The color of the different units has only an illustrative purpose and does not represent actual activation. Figures from [START_REF] Rougier | Covert learning in the basal ganglia: figures[END_REF], CC-BY. At the start of day 2, the performance for both the model and the monkeys is significantly higher compared to the start and end of day 1, suggesting some covert learning occurred during day 1 even though performances are random during day 1. B & C panels show all the individual trials (n=2x60) for all the sessions (n=12) for the theoretical model (panel B) and the experiments with monkeys (panel C). A black dot means a successful trial (the best stimulus has been chosen) and an outlined white dot means a failed trial (the best stimulus has not been chosen). Measure of success is thus independent of the actual reward received after having chosen one of the two stimuli. The bottom part shows the mean success rate over a sliding window of ten consecutive trials and averaged across all the sessions. The thick black line is the actual mean and the gray-shaded area represents the standard error of the mean (STD) over sessions. These results show that both the model and the monkeys are unable to choose the best stimulus when the GPi is disabled on day 1. This can be seen on the left part of both panels with a mean success rate oscillating around 0.5 (i.e. random choice). However, on day 2, when the GPi is re-enabled, there is an immediate effect (right part of both panels) and the mean success rate is instantly above 0.5 even if it isn't instantly optimal. Our hypothesis is that the model benefits from the covert learning happening inside the basal ganglia during day 1 even though the BG cannot influence behavior during this period. The other consequence of this covert learning is a better estimation of both stimuli value during day 1 because there is no selection bias in favor of one or the other stimulus. They are both sampled uniformly, leading to a quantitatively uniform estimation of their respective value. * = p < 0.01. Figure from [START_REF] Rougier | Covert learning in the basal ganglia: figures[END_REF], CC-BY.

Illustrations and Tables

Figure 3 .

 3 Figure3. A. Histograms show the mean performance at the start and the end of a session in day 1 and day 2 conditions for both the model (left part) and the monkeys (right part). At the start of day 2, the performance for both the model and the monkeys is significantly higher compared to the start and end of day 1, suggesting some covert learning occurred during day 1 even though performances are random during day 1. B & C panels show all the individual trials (n=2x60) for all the sessions (n=12) for the theoretical model (panel B) and the experiments with monkeys (panel C). A black dot means a successful trial (the best stimulus has been chosen) and an outlined white dot means a failed trial (the best stimulus has not been chosen). Measure of success is thus independent of the actual reward received after having chosen one of the two stimuli. The bottom part shows the mean success rate over a sliding window of ten consecutive trials and averaged across all the sessions. The thick black line is the actual mean and the gray-shaded area represents the standard error of the mean (STD) over sessions. These results show that both the model and the monkeys are unable to choose the best stimulus when the GPi is disabled on day 1. This can be seen on the left part of both panels with a mean success rate oscillating around 0.5 (i.e. random choice). However, on day 2, when the GPi is re-enabled, there is an immediate effect (right part of both panels) and the mean success rate is instantly above 0.5 even if it isn't instantly optimal. Our hypothesis is that the model benefits from the covert learning happening inside the basal ganglia during day 1 even though the BG cannot influence behavior during this period. The other consequence of this covert learning is a better estimation of both stimuli value during day 1 because there is no selection bias in favor of one or the other stimulus. They are both sampled uniformly, leading to a quantitatively uniform estimation of their respective value. * = p < 0.01. Figure from[START_REF] Rougier | Covert learning in the basal ganglia: figures[END_REF], CC-BY.

  Figure1