Justine Fouillé
email: justine.fouille@etudiant.univ-rennes1.fr

Thi Lan

Huong Nguyen
email: thi-lan-huong.nguyen@etudiant.univ-rennes1.fr

Baptiste Alix
email: baptiste.alixetudiant@univ-rennes1.fr

Brett Becker
email: brett.becker@etudiant.univ-rennes1.fr

Matthieu Rochard
email: matthieu.rochard@etudiant.univ-rennes1.fr

Hélène De Ribaupierre
email: deribaupierreh@cardiff.ac.uk

Laurent D'orazio
email: laurent.dorazio@univ-rennes1.fr

ASSIST: Article eXtraction and statIstical AnalysiS

Keywords: Data Extraction, Data Analytic, Statistical Analysis

There are fewer female authors than male authors in the field of scientific research. However, there is not yet a system that provides a way to analyze the data that is available, and to backup that claim. This paper illustrates the upgrade of a tool previously made, in order to make it more efficient and add new features. Such new features are the keywords cloud or the new statistical functionality. Sources, references and other information on the article will be displayed for each articles retrieved. Genders of the authors will be determined using a database linking first names to genders, to be able to get accurate statistics on a large number of gathered articles.

I. INTRODUCTION

Publish or perish illustrates quite well the pressure researchers feel to succeed an academic career. Researchers are thus encouraged to promote their progress writing papers submitting it to international workshops, symposia, conferences or journals.

Some studies have recently demonstrate that there are fewer female authors than male authors in the field of scientific research 1 . However, there is not yet a system that provides concrete data to prove this, making it possible for example to know the ratio of women for a given domain (for example medicine or computer science), a sub-domain (databases or computer human interaction for instance), a a research field (like data integration or query optimization), or according to the different venues and with respect to their rank (A*, A, B, C or unranked). It is thus impossible to see the evolution of the place of women during the time.

Several plateform exist to anaylse some aspects of scientific publications such as Scopus2 , altmetric 3 , Plumx4 , Dimen-sions 5 . These plateforms are either focusing on the performances of the scientists (such as the h-index), the citations networks for the articles or the ranking of the journal (such as the impact factors). Solutions such as Core6 aggregate the open access articles from different resources, and offer an API to access the raw text of the articles. However, to the best of our knowledge, there is no open platform that make it possible to gather and to analyse the data contained in the scientific papers. One possible use case of this platform we are presenting is to analyse the evolution of the place of women within the scientific community.

In this paper, we introduce our web application, AXIS, that stands for Article eXtraction and statIstical AnalysiS. The aim is to draw a portrait of the evolution of the research world and to highlight the differences of gender in this field. AXIS therefore makes it possible to centralize several bibliographic sources and to analyze data behind scientific articles. The data concerning the articles (title, list of authors, references, citations, publisher, origin...) are therefore retrieved from multiple Application Programming Interfaces or APIs (DBLP, Core, etc.) to vary the sources of information. The gender of the authors is determined by their first names. The application then allows statistics to be compiled on the number of authors by genre per publisher.

The structure of this article is as follows. First, we present how we improve the analyses made with the articles. We have developed new functionalities allowing more precise analyses such as the number of female authors according to the years with a publisher. We have also developed a keyword cloud for each article that displays the main themes it addresses. To finish, we show you how we solve the data retrieval problem we have changed the system for retrieving data from the API's.

II. MOTIVATION

Bibliographic data extracted from scientific articles have many purposes, the authors in [START_REF] Di Iorio | Exploring scholarly papers through citations[END_REF], [START_REF] West | The role of gender in scholarly authorship[END_REF] propose a list of the different tasks on bibliographies possible depending on the different classes of users. In this paper, we will discuss two tasks (building a bibliography and analysing the impact of a researcher) and expand the last one into a new task (analysing the evolution of the publications landscape and the number of women). The following is not an exhaustive list of all the possible tools and research that are be available to achieve these tasks, but the ones that are often used by scientists to achieve these tasks.

The user wants to build a bibliography on a given topic. For example, the user wants to build a bibliography on the subject of "nanopublications" and get all the relevant publications on this subject. Tools such as Google Scholar 7 could help them to start their task. Nanopublications being a very specific term, the result of the query in google scholar will be only 973 results. If the scientist wants to explore the full set of results it will already take them some time. However, study shows that scientists often don't explore all the results [START_REF] Renear | Strategic reading, ontologies, and the future of scientific publishing[END_REF]. The task becomes already more tricky for the scientist as the first paper presented to the user is a paper that has 286 citations. Does the user want to explore all the different citations from this paper, or only a subset of these 286 citations? And this is only for the first publication presented to the scientist. Google scholar offers some possibility of filtering, such as searching through the 286 citations of the first paper, but is not offering the possibility to filter on a deeper level or to seek for all the articles that could have been cited commonly by the set of 973 articles. The user will start to go article by article, and look at the different lists of citations, open the articles that seem interesting, and look for the references or the citations of these articles. The task becomes very time consuming and difficult as the number of papers grow in an exponential way. The scientist could then change for a tool that could be more appropriate, and offer more filtering such as Microsoft Academic 8 . The interface does allow you to follow the thread of citations a little bit longer, but if the scientist wants to download a batch of related articles, this is not feasible either. In addition, it is not possible to give the number of iterations the scientist wants to achieve in their research.

The second task is to analyse the impact of a scientist. This is often done by the number of citations on the publications of the author, or new metrics such as the popularity of this article on social media (Tweeter, researchGate, etc). Several tools can help to achieve this task such as Web Of Science 9 , Google Scholar, Microsoft Academic, PlumX 10 , Altmetric 11 , Sci2Tool 12 . Most of these tools are giving only a partial view of the impact of a scientist, and will use different methods.

7 scholar.google.com 8 academic.microsoft.com 9 https://app.webofknowledge.com 10 https://plumanalytics.com 11 https://www.altmetric.com 12 https://sci2.cns.iu.edu/user/index.php For example, Google scholar seems to count in the h-index of the scientist the self-citations, when other tools such as WebOfScience is only counting the publications accessible to them. The researcher that wants to analyse their own impact or the impact of someone else will navigate (when they have access) through different web sites and gather the different information to be able to have a complete view.

The third task is to analyse the participation of the different minorities in science. Numerous research shows that the percentage of women authors is lower than men [START_REF] Danell | Career prospects for female university researchers have not improved[END_REF], however this research is often done in a very manual way, where the scientist will collect manually a subset of bibliographic data and analyse the percentage of women authors in this subset. This is very time costly, and in addition, it shows only a very small image of the big picture. The comparison between the different areas of research is difficult, and to be able to analyse on a larger scale, the data need to be available and to be processed in an automatic way. This task present several challenges. First, not all the data are freely available; Second, not all the data are located in the same places; and third, some data need to be inferred such as the gender of the author as they are not available from the publication data itself.

III. OVERVIEW

Our proposal, article Extractions and Statistical Analysis (AXIS), consists of two main functionalities: (1) Data Management (Data extraction and Processing) which recovers articles and responds to queries for articles for users; (2) Data Analysis will allow the user to gather statistics on the stored articles and display them. The data extraction system is based on the combination of numerous API's (such as the DBLP API, CrossRef and the Core API). These API's allow access to research papers from different sources. Indeed, since each API does not provide the same information, or provides incomplete information on the papers, combining several API's allows a maximum amount of information to be gathered.

IV. DATA MANAGEMENT

A. Data extraction

For each article its title, date of publication, abstract, DOI13 and the full text are collected. For the authors: their names, genders, positions (first author, second author...) and affiliation are also stored. Finally, the origin of the article ( a conference or a journal ) as well as citations of the article and references made by the article are also collected.

Once the articles are retrieved, a gender-defining application (such as gender.io) will allow the database to associate a gender to each author's name.

B. Challenges

The main challenges are managing the different formats of information sent by these APIs. Some of them use XML, others JSON or even Bibtex. One way to avoid this problem is by using only APIs which use the same format or can return different formats. A second way is converting the data into a single format once it is received.

The lack of DOI for older articles can also pose a problem for recovering citations and references and thus alternative APIs may be required. Homonyms is another issue as many authors could share the same name and the challenge will be to distinguish between them. Another challenge will be the change of first name or last name during the career of the author.

Even if this will not be the ultimate answer to these challenges, we can use the ORCID id of an author. Currently, none of the API that have been chosen return the ORCID id of the author, but we can imagine that it will soon be the case, considering that its use is tending to increase.

V. DATA ANALYTIC

Having gathered much data it is possible to analyse it and uncover interesting information. By cross-referencing one or more data points it will be possible to obtain numerous statistics.

A. Visualization

Using the data stored in the database, statistics can be calculated from a variety of options: author's gender, author's position on the article, number of citations, the number of autocitations, number of references, publishers, article's origin and the date of publication. This feature allows for the analysis of different issues concerning the field of scientific research and in particular the place of women in it. The user can display the number of authors by gender by year for a previously chosen publisher or conference. The result is displayed in the form of a table with the total number of papers, the number of men authors, the number of women authors and the number of authors whose gender is unknown. This makes it possible to retrieve raw data for use in various analyses. Thanks to this feature, it is possible to compare the number of authors per genre from one year to the next or from several publishers in a given year.

B. Keywords cloud

The tag cloud's purpose is to display the keywords of a scientific article but also to retrieve those that match that keyword. This allows the user to view the different topics that match a specific term. This feature allows you to discover new scientific articles by offering a list of articles corresponding to the keyword selected by the user. This tag cloud would also allow to know on which topics women publish the most. We would therefore have even more accurate statistics than the proportion of female authors in a field.

C. Challenges

There are several aspects that can influence the statistics, in particular for the gender. First of all, it is not always possible to attribute a gender to a name. For this reason, a threshold of 85% probability from genderize.io has been decided upon. It seems preferable to build statistics on less data, rather than building on false ones. In some cases, it is also difficult to determine the gender of the authors, since they only provide the initial of the first name.

VI. CONCLUSION

This paper presents AXIS, a platform relying on multiple APIs to gather and store large amounts of scientific publications. This platform should help young researcher to determine a bibliography in a much easier way, as the recursive process will allow the user to gather a precise and exhaustive list of scientific publications. This platform will also help any scientists interested in analysing the scientific publications landscape to have a more comprehensive data set to analyse.

In addition the large amount of data allows for the analysis of Statistics and the studying of trends in the scientific community. Facilitating access to the vast quantities of information available is paramount.

With the vast amounts of data however comes different constraints such as the constraint of homonyms. It is most likely that among the authors inserted into the database some will have the same names. There is no current way to distinguish between them. Moreover, should an author change names, they will be recorded as two different people. In addition, the plateform is currently only analysing the sex of the authors using the first name of the author, this way of analysing the sex of the authors will certainly lead to the exclusion of other minorities, and should be investigate further.

This platform could be very useful for conducting new research in the field of scientometry, or simply for referencing a large number of scientific articles and their authors.

Figure 1

 1 Figure 1 illustrates the data flow diagram of AXIS, from the data management perspective.The data extraction system is based on the combination of numerous API's (such as the DBLP API, CrossRef and the Core API). These API's allow access to research papers from different sources. Indeed, since each API does not provide the same information, or provides incomplete information on the papers, combining several API's allows a maximum amount of information to be gathered.For each article its title, date of publication, abstract, DOI 13 and the full text are collected. For the authors: their names, genders, positions (first author, second author...) and affiliation are also stored. Finally, the origin of the article ( a conference or a journal ) as well as citations of the article and references made by the article are also collected.Once the articles are retrieved, a gender-defining application (such as gender.io) will allow the database to associate a gender to each author's name.

Fig. 1 .

 1 Fig. 1. Data flow diagram

In this article, we are only considering female and male, and further research should be done to include everyone

https://www.scopus.com/home.uri

https://www.altmetric.com

https://plumanalytics.com/

https://www.dimensions.ai

https://core.ac.uk

A digital object identifier is a persistent identifier or handle used to identify objects uniquely, standardized by the International Organization for Standardization


Chenxiao Wang
email: chenxiao@ou.edu

Le Gruenwald
email: ggruenwald@ou.edu

Laurent D'orazio
email: laurent.dorazio@univ-rennes1.fr

Eleazar Leal
email: eleal@d.umn.edu

Cloud Query Processing with Reinforcement Learning-based Multi-Objective Re-optimization

Keywords: Query Optimization, Cloud Databases, Reinforcement Learning, Query Re-optimization

Query processing on cloud database systems is a challenging problem due to the dynamic cloud environment. The configuration and utilization of the distributed hardware used to process queries change continuously. A query optimizer aims to generate query execution plans (QEPs) that are optimal meet user requirements. In order to achieve such QEPs under dynamic environments, performing query re-optimizations during query execution has been proposed in the literature. In cloud database systems, besides query execution time, users also consider the monetary cost to be paid to the cloud provider for executing queries. Thus, such query re-optimizations are multi-objective optimizations which take both time and monetary costs into consideration. However, traditional re-optimization requires accurate cost estimations, and obtaining these estimations adds overhead to the system, and thus causes negative impacts on query performance. To fill this gap, in this paper, we introduce ReOptRL, a novel query processing algorithm based on deep reinforcement learning. It bootstraps a QEP generated by an existing query optimizer and dynamically changes the QEP during the query execution. It also keeps learning from incoming queries to build a more accurate optimization model. In this algorithm, the QEP of a query is adjusted based on the recent performance of the same query so that the algorithm does not rely on cost estimations. Our experiments show that the proposed algorithm performs better than existing query optimization algorithms in terms of query execution time and query execution monetary costs..

Introduction

In a traditional database management system (DBMS), finding the query execution plan (QEP) with the best query execution time among those QEPs generated by a query optimizer is the key to the performance of a query. In a cloud database system, minimizing query response time is not the only goal of query optimization. As hardware usages are charged on-demand and scalability is available to users, monetary cost also needs to be considered as one of the objectives for optimizing QEPs in addition to query response time. In order to do that, the query optimizer evaluates the time and monetary costs of different QEPs in order to derive the best QEP for a query. These time and monetary costs are estimated based on the data statistics available to the query optimizer at the moment when the query optimization is performed. These statistics are often approximate, which may result in inaccurate estimates for the time and monetary costs needed to execute the query. Thus, the QEP generated before query execution may not be the best one.

To solve the problem, researchers have developed learning-based algorithms to adjust the data statistics to get more accurate cost estimations [START_REF] Kabra | Efficient mid-query re-optimization of sub-optimal query execution plans[END_REF][START_REF] Park | Database learning: Toward a database that becomes smarter every time[END_REF]. These methods are heuristic-based and the adjustment of QEP is not adaptable to the dynamic environment. More recently, machine learning-based algorithms are introduced [START_REF] Markl | Robust query processing through progressive optimization[END_REF][START_REF] Wu | Towards a learning optimizer for shared clouds[END_REF]. More accurate cost estimations are made by the data statistics estimated by the machine learning models. The optimizer uses these cost estimations to adjust the QEP. Again, even those methods improved the accuracy of data statistics estimation such as cardinalities, the overall performance is not improved much. This is usually because updating data statistics for the optimizer to use is a very expensive operation by itself. This becomes the main source of negative impacts on the overall performance. To eliminate this problem, in this paper we propose an algorithm, called Re-OptRL, for query re-optimization that makes use of reinforcement learning (RL) to find the best QEP without relying on data statistics.

RL is about taking suitable actions to maximize reward in a particular situation. The decision of choosing actions follows the trial-and-error method and is evaluated by the reward. The learning model is adjusted with the reward after each action has been performed. We choose RL instead of supervised learning methods [START_REF] Murphy | Machine Learning: A Probabilistic Perspective[END_REF] because unlike supervised learning, RL does not require a labeled dataset of past actions to be available to train the learning model in advance before it can be used to predict future actions. Query optimization in the database system can be regarded as a series of actions and the best actions can be learned from evaluating the historical optimizations. In the proposed algorithm ReOp-tRL, the QEP of a query is adjusted during the query execution independently of data statistics. The adjustment is based on the performance of historical queries that are executed on the system recently. Evaluating these historical executions is reasonable especially on the cloud database system. As a large number of different queries are executed frequently, for the same query, the time gap between its incoming query and its historical query is short. The algorithm monitors and keeps learning from these executions while more incoming queries are executed. Some algorithms used reinforcement learning in adjusting their QEPs also, but these adjustments are only focusing on the JOIN order of queries [START_REF] Kipf | Learned cardinalities: Estimating correlated joins with deep learning[END_REF][START_REF] Marcus | Deep reinforcement learning for join order enumeration[END_REF]. Our algorithm extends these features to not only adjust the QEP itself but also to find the best allocations to execute different operators in the query, i.e, to find which machines should be used to execute which operator. In a cloud database system, each operator can be executed on a different machine, and different machines can have different hardware configurations with different usage prices. This means that for the same query operator, not only different execution times but also different monetary costs can incur depending on which machine is used to execute the operator. An operator executed on machine A may have a lower execution time but may cost more money, on the other hand, the same operator executed on machine B may have a higher execution time but may cost less money. Those differences in execution times and monetary costs for individual operators eventually affect the overall performance in accumulation for the entire query. Our goal is to fulfill both user's execution time and monetary cost requirements of the overall query performance. Due to this reason, optimal allocations of operator executions to appropriate machines on the cloud are as important as optimal QEPs themselves. Our contributions in this work are the following:

-We propose an algorithm that uses RL to perform multi-objective query processing for an end-to-end cloud database system. -Our algorithm employs a new reward function designed specifically for query re-optimization. -We present comprehensive experimental performance evaluations. The experimental results show that our algorithm improves both query execution time and monetary costs when comparing to existing query optimization algorithms.

The remaining of this paper is organized as follows: Section 2 discusses the related work; Section 3 presents the proposed reinforcement learning-based query re-optimization algorithm; Section 4 discusses the experimental performance evaluations; and finally, Section 5 presents the conclusions and discusses future research directions.

Related Work

The problem of query re-optimization has been studied in the literature. In the early days, heuristics were used to decide when to re-optimize a query or how to do the re-optimization. Usually, these heuristics were based on cost estimations which were not accurate at the time when query re-optimization takes place. Also, sometimes, a human-in-the-loop was needed to analyze and adjust these heuristics [START_REF] Kipf | Learned cardinalities: Estimating correlated joins with deep learning[END_REF]. These add additional overheads to the overall performance of queries. Unfortunately, these heuristic solutions can often miss good execution plans. More importantly, traditional query optimizers rely on static strategies, and hence do not learn from previous experiences. Traditional systems plan a query, execute the query plan, and forget that they ever optimized this query. Because of the lack of feedback, a query optimizer may select the same bad plan repeatedly, never learning from its previous bad or good choices.

Machine learning techniques have been used recently in query optimizations for different purposes. In earlier works, Leo [START_REF] Stillger | Leo -db2's learning optimizer[END_REF] learns from the feedback of executing past queries by adjusting its cardinality estimations over time, but this algorithm requires human-tuned heuristics, and still, it only adjusts the cardinality estimation model for selecting the best join order in a query. More recently, the work in [START_REF] Wu | Towards a learning optimizer for shared clouds[END_REF] presents a machine learning-based approach to learn cardinality models from previous job executions and these models are then used to predict the cardinalities in future jobs. In this work, only join orders, not the entire query, are optimized. In [START_REF] Wang | Database meets deep learning: Challenges and opportunities[END_REF], the authors examine the use of deep learning techniques in database research. Since then, reinforcement learning is also used. The work proposes a deep learning approach for cardinality estimation that is specifically designed to capture join-crossing correlations. SkinnerDB [START_REF] Trummer | Skinnerdb: Regretbounded query evaluation via reinforcement learning[END_REF] is another work that uses a regret-bounded reinforcement learning algorithm to adjust the join order during query execution. None of these machine learningbased query optimization algorithms is designed for predicting the action that should follow after one query operator has been executed. Recently, there have been several exciting proposals in putting reinforcement learning (RL) inside a query optimizer. As described in [START_REF] Wiering | Reinforcement Learning: State-of-the-Art[END_REF] and shown in Fig. 1, reinforcement learning describes the interaction between an agent and an environment. The possible actions that the agent can take given a state S t of the environment are denoted as A t ={a 0 , a 1 , . . . , a n }. The agent performs an action from the action set A t based on the current state S t of the environment. For each action taken by the agent, the environment gives a reward r t to the agent and the environment turns into a new state S t+1 , and the new action set is A t+1 . This process repeats until the terminal state is reached. These steps form an episode. The agent tries to maximize the reward and will adjust after each episode. This is known as the learning process. Ortiz et al. [START_REF] Ortiz | Learning state representations for query optimization with deep reinforcement learning[END_REF] apply deep RL to learn a representation of queries, which can then be used in downstream query optimization tasks. Liu [START_REF] Liu | Cardinality estimation using neural networks[END_REF] and Kipf [START_REF] Kipf | Learned cardinalities: Estimating correlated joins with deep learning[END_REF] use DNNs to learn cardinality estimates. Closer to our work is Marcus et al.'s work of a deep RL-based join optimizer, ReJOIN [START_REF] Marcus | Deep reinforcement learning for join order enumeration[END_REF], which orders a preliminary view of the potential for deep RL in this context. The early results reported in [START_REF] Marcus | Deep reinforcement learning for join order enumeration[END_REF] give a 20% improvement in the query execution time of Postgres. However, they have evaluated only 10 out of the 113 JOB queries.

Deep Reinforcement Learning based Query Re-Optimization

In this section, we present our query re-optimization algorithm that makes use of reinforcement learning, ReOptRL. First, we discuss our query processing procedure in the cloud database system. Then we describe how the deep reinforcement learning algorithm is used in our query processing to select the best actions for the performance of queries. In this algorithm, a query will be converted into a logical plan by a traditional query parser. Then for each logical query operator, we use a deep reinforcement learning model to select what physical operator and which machines (containers) should be used to execute this logical operator so that each operator execution is optimized to gain the maximum improvement on overall performance. These selections are learned from the same operator executed previously in the system. Besides that, as in large applications, there will be a large number of queries running frequently. For a query operator, it is reasonable to utilize the performance of its previous executions of the same query to predict the performance of its current execution because the time between the two executions is short.

The Environment of Our Cloud Database System

In this section, we first briefly describe our environment for query processing. Our applications mainly focus on processing queries in a mobile cloud environment. Fig. 2 shows the process flow of query processing in the mobile-cloud database system that we have developed [START_REF] Wang | Adaptive time, monetary cost aware query optimization on cloud database systems[END_REF]. In this architecture, the mobile device is for the user to access the database and input queries, the data owner is a server onsite that contains private data, and the cloud provider owns the cloud database system. Executing a query incurs three different costs: the monetary cost of query execution on the cloud, the overall query execution time, and the energy consumption on the mobile device where the query might be executed. These three costs constitute the multi-objectives that the query optimizer needs to minimize to choose the optimal query execution plan (QEP). Different QEPs are available due to the elasticity of the cloud which considers multiple nodes with different specifications. In this paper, we focus on query processing on the cloud provider's part. We consider both the query execution time and monetary cost, but not the energy cost on mobile devices. Different users have different preferences for choosing a suitable QEP for their purposes. In an application scenario where many queries are executed per day, organizations may want to minimize the monetary cost spent for query execution to fit their budget. They may also want to minimize query execution time to meet customers' query response time requirements and to optimize employees' working time. In order to incorporate these user's preferences into the query optimization algorithm, we use the Normalized Weighted Sum Model we have developed in [START_REF] Helff | Weighted sum model for multi-objective query optimization for mobile-cloud database environments[END_REF] to select the best plan. In this model, every possible QEP alternative is rated by a score that combines both the objectives, query response time, and monetary cost, with the weights defined by the user and the environment for each objective, and the user-defined acceptable maximum value for each objective. These weights defined by the user are called Weight Profile (wp), which is a two-dimensional vector and each dimension is a number between 0 to 1 which defines the preferences. The following function is used to compute the score of a QEP:

A W SM -SCORE i = n j=1 W j a ij m j (1)
where a ij is the value of QEP alternative i (QEP i ) for objective j, m j the user-defined acceptable maximum value for objective j, and w j the normalized composite weight of the user and environment weights for objective j, which is defined as follows:

w j = uw j * ew j (uw * ew) (2)
where uw j and ew j describe the user and environmental weight for objective j, respectively. These weights are user-defined. Since the different objectives are representative of different costs, the model chooses the alternative with the lowest score to minimize the costs. In this work, we use a policy gradient deep RL algorithm for query re-optimization. This algorithm uses a deep neural network to help the agent decide the best action to perform under each state. In this work, the agent is the query optimizer, an action is a combination of a physical operator to execute a logical operator and a machine to execute this operator, and a state is a fixed-length vector encoded from the logical query plan parsed from a traditional query optimizer. The input of the neural network is the vector of the current state. The input is sent to the first hidden layer of the neural network whose output is then sent to the second layer, and so on until the final layer is reached, and then an action is chosen. The policy gradient is updated using a sample of the previous episodes, which is an operator execution in our case. Once an episode is completed (which means a physical operator and execution container are selected in our case), the execution performance is recorded and a reward is received where a reward is a function to evaluate the selected action. The details of the reward function will be explained in Section 3.3. The Weights of the neural network is updated after several episodes using existing techniques, such as back-propagation [START_REF] Liu | Cardinality estimation using neural networks[END_REF]. Fig. 3: Procedure of the proposed algorithm, ReOptRL Fig. 3 shows the major steps in query processing when ReOptRL is incorporated for query re-optimization. First, the optimizer receives a query and then the query is compiled into a QEP (logical plan). Secondly, the first available operator is converted into a vector representation and is sent to the RL model. The RL model will select the optimal action, which is the combination of a physical operator and a container to execute this operator. Then this operator is sent to execution and the execution time and monetary costs of this execution are recorded to update the reward function. Once the reward function is updated, the weights of this RL model are adjusted according to the updated reward function. Then the updated RL model is ready for future action selections of the same operator.

There are various kinds of reinforcement learning algorithms. Q-Learning is one of the popular value-based reinforcement learning algorithms [START_REF] Wiering | Reinforcement Learning: State-of-the-Art[END_REF]. In Q-Learning, a table (called Q-table) is used to store all the potential state-action pairs (S n , a n ) and an evaluated Q-value associated with this pair. When the agent needs to decide which action to perform, it looks up the Q-value from the Q-table for each potential action under the current state and selects and performs the action with the highest Q-value. After the selected action is performed, a reward is given and the Q-value is updated using the Bellman equation [START_REF] Ohnishi | Constrained deep q-learning gradually approaching ordinary q-learning[END_REF].

Q(S t , a t ) ← Q(S t , a t ) + α(R t + γQ(S t+1 , a t+1 ) -Q(S t , a t )) (3)
In Equation (3), Q(S t , a t ) is an evaluated value (called Q-value) of executing Action a t at State S t . This value is used to select the best Action to perform under the current state. To keep this value updated with accurate evaluation is the key to reinforcement learning. α is the learning rate and γ is the discount rate. These two values are constant between 0 and 1. The learning rate α controls how fast the new Q-value is updated. The discount rate γ controls the weight of future rewards. If γ = 0, the agent only cares for the first reward, and if γ = 1, the agent cares for all the rewards in the future [START_REF] Kipf | Learned cardinalities: Estimating correlated joins with deep learning[END_REF]. R is the reward which is described in Section 3.3 Fig. 4 shows the pseudo-code of the proposed algorithm. First, the optimizer receives a query and then the query is compiled into a QEP (logical plan) (Line 4). Then, the QEP is converted into a vector representation. By doing this, the current QEP represents the current state and can be used as the input of a neural network. We use a one-hot vector to present this vector and this technique is adapted from the recent work [START_REF] Marcus | [END_REF] (Line 5). This vector is sent to the RL model, which is a neural network as described in Section 3.1. The RL model will evaluate the Q-values for all the potential actions to execute the next available query operator (Line 6). Each of these actions includes two aspects, the optimal physical operator and the best container to execute the next available query operator. Then the action with the best Q-value will be selected and performed by the DBMS (Line 7). After that, the executed operator is discarded from the QEP and the time and monetary costs to execute this operator are used to compute the reward for this action (Line 8). The reward is updated with the time and monetary cost to execute the operator and then the expected Q-value is updated by the Bellman Equation (3) with the updated reward (Line 9-11). The weights of the neural network are updated accordingly by the backpropagation method (Line 12). This process repeats for each operator in the QEP and terminates when all the operators in the QEP are executed. The query results are then sent to the user (Line 15

Reward Function

In ReOptRL, after an action is performed, the reward function is used to evaluate the action. This gives feedback on how the selected action performs to the learning model. The performed action with a high reward will be more likely to be selected again under the same state. The reward function plays a key role in the entire algorithm. In our algorithm, we would like the actions with low query execution time and monetary cost to be more likely chosen. To reflect this feature, here we define the reward function as follows:

Reward R = 1 1 + (W t * T q op ) + (W m * M q op ) (4)
where W t and W m are the time and monetary weights provided by the user, T q op and M q op is the time and monetary costs for executing the current operator op in query q. According to this reward function, the query is executed based on the user's preference which is either the user wants to spend more money for a better query execution time or vice versa. We call these preferences Weights. These weights defined by the user are called Weight Profile (wp), which is a two-dimensional vector and each dimension is a number between 0 to 1. Notice that, the user only needs to specify one dimension of the weight profile, the other dimension is computed with 1-Weight automatically. For example, if a user demands fast query response time and is willing to invest more money to achieve it, the weight profile for this user probably would be < W t = 0.9, W m = 0.1 >. The detail can be found in our previous work [START_REF] Helff | Weighted sum model for multi-objective query optimization for mobile-cloud database environments[END_REF]. This reward function is a monotonic decreasing function. With the increase of (W t * T q op ) + (W m * M q op ), which is the total costs of executing a query operator, the reward decreases. Notice that, as (W t * T q op ) + (W m * M q op ) approaches zero, the reward approaches positive infinity. When this situation happens, if an action A is performed with small total costs, then A will always be selected and performed, and all the other actions will be ignored. This is not desirable, and to keep the relationship of reward and total costs close to linear, we use 1 + (W t * T q op ) + (W m * M q op ) as the denominator in the reward function. In summary, if performing an action takes high costs, this action will be less likely to be chosen in the future.

Performance Evaluation

In this section, we first describe the hardware configuration, database benchmark, and parameters we used in our experimental model. We also introduce the algorithms we compared with our algorithm. . The data are distributed among these VPSs. The queries and database tables are generated using the TPC-H benchmark [START_REF] Barata | An overview of decision support benchmarks: Tpc-ds, tpc-h and ssb[END_REF]. There are eight database tables with a total size of 1,000 GB. We run 50,000 queries in total and these queries are generated by the query templates randomly selected from the 22 query templates from the TPC-H benchmark [START_REF] Barata | An overview of decision support benchmarks: Tpc-ds, tpc-h and ssb[END_REF]. In the experiments, to update the Q-value using the Bellman equation as shown in Equation (3) discussed in Section 3.2, we set the learning rate α as 0.1 and discount rate γ as 0.5.

Evaluation of ReOptRL

In this section, we compare the query processing performances obtained when the following query re-optimization algorithms are incorporated into query processing: 1) our proposed algorithm (denoted as ReOptRL); 2) the algorithm where a query re-optimization is conducted automatically after the execution of each stage in the query is completed (denoted as ReOpt), which we developed based on the state-of-art works [START_REF] Wang | Adaptive time, monetary cost aware query optimization on cloud database systems[END_REF][START_REF] Bruno | Continuous cloud-scale query optimization and processing[END_REF]; 3) the algorithm where a query re-optimization is conducted by a supervised machine learning model decision (denoted as Re-OptML); 4) the query re-optimization algorithm which uses sampling-based query estimation (denoted as Sample) proposed in [START_REF] Wu | Sampling-based query re-optimization[END_REF]; and 5) the query processing algorithm that uses no re-optimization (denoted as NoReOpt).

We use NoReOpt as the baseline and the other algorithms are compared to the baseline. The two figures, Fig. 5 and Fig. 6, show the improvement of each algorithm over the baseline. From Fig 5, we can see that, from the query execution time perspective, ReOptRL performs 39% better on average than NoReOpt, while ReOptML performs 27%, ReOpt 13% and Sample 1% better than NoRe-Opt. There are 15 out of 22 query types that have better performance if our algorithm is used. The best case is query 9 (Q9), for which our algorithms perform 50% better than NoReOpt. From Fig 6, from the monetary cost perspective, query processing uses ReOptRL performs 52% better on average than NoReOpt, while ReOptML performs 27%, ReOpt 17%, and Sample 5% better than NoRe-Opt. There are 14 out of 22 query types that have better performance if our proposed algorithm is used. The best case is also Q9, for which our algorithms perform 56% better than NoReOpt.

The above results show that overall our proposed algorithm improves more time and monetary cost than the four algorithms, ReOpt, ReOptML, Sample, and NoReOpt. Especially, the monetary cost has a significant improvement (56% better than NoReOpt); however, for simple query types [START_REF] Barata | An overview of decision support benchmarks: Tpc-ds, tpc-h and ssb[END_REF][START_REF] Bruno | Continuous cloud-scale query optimization and processing[END_REF][START_REF] Helff | Weighted sum model for multi-objective query optimization for mobile-cloud database environments[END_REF][START_REF] Kabra | Efficient mid-query re-optimization of sub-optimal query execution plans[END_REF][START_REF] Liu | Cardinality estimation using neural networks[END_REF][START_REF] Marcus | Deep reinforcement learning for join order enumeration[END_REF][START_REF] Murphy | Machine Learning: A Probabilistic Perspective[END_REF][START_REF] Ohnishi | Constrained deep q-learning gradually approaching ordinary q-learning[END_REF] which are 8 query types out of 22 TPC-H query types, our algorithm does not improve their performance due to the overhead involved. In these experiments, the Reinforcement Learning part of the query processing contributes to these improvements and it is beneficial in the following aspect. In all the three algorithms, ReOpt, ReOptML, and Sample, query re-optimization requires a lot of overhead data statistics that need to be accessed and updated frequently. In our proposed algorithms, no data statistics are needed, and re-optimization is based on the results of learning which can be decided quickly.

Impacts of Weight Profiles

Our algorithm allows users to input their weight profiles and this feature is reflected in the reward function. We study how our proposed algorithm works under different weight profiles submitted by users. Fig. 7 shows the performance of the monetary cost of each algorithm and the percentage of improvement of the monetary cost of each algorithm compared to the baseline on different weights. Notice that the Weight of Money in our application is (1-Weight of Time). From Fig. 7, we can see that, when the weight of money increases, the monetary cost to execute the query decreases accordingly. Even when the weight of money is low, our algorithm still outperforms other algorithms on the monetary cost (by 10%). This happens because by using the learning model, our algorithm always chooses the right physical operator and container to execute the operator. Similarly, from the monetary cost perspective, our algorithm performs better on average than the other three algorithms.

The above results show that our algorithm can adapt to the user's weight profile very well. When the user has a high demand on either query execution time or money cost, our algorithm still outperforms the other algorithms.

Conclusion and Future Work

This paper presents an algorithm called ReOptRL for processing queries in a cloud database system taking both query execution time and money costs to be paid to the cloud provider into consideration. The algorithm uses a reinforcement learning-based model to decide the physical operator and machines to execute an operator from a query execution plan (QEP) of a query. The experimental performance evaluations using the TPC-H benchmark show that our proposed algorithm, ReOptRL, improves the query response time (from 12% to 39%) and monetary cost (from 17% to 56%) over the existing algorithms that use either no re-optimization, re-optimization after each stage in the query execution plan (QEP) is executed, supervised machine learning-based query re-optimization, or sample-based re-optimization.

While our studies have shown that reinforcement learning has positive impacts on the optimization and execution of each operator in a query, our algorithm currently does not consider the Service Agreement Level (SLA), which is an important feature of cloud database systems. For future work, we will investigate techniques to incorporate this feature into our algorithm by adjusting the reward function to consider SLA.

Fig. 1 :

 1 Fig.1: General Procedure of Reinforcement Learning[START_REF] Wiering | Reinforcement Learning: State-of-the-Art[END_REF]

Fig. 2 :

 2 Fig. 2: Mobile Cloud Database Environment

Fig. 4 :

 4 Fig. 4: Pseudo code of Proposed algorithm (ReOptRL)

4. 1

 1 Hardware Configuration, Database Benchmark, and Parameter ValuesThere are two sets of machines that are used in our experiments. The first set consists of a single local machine used to train the machine learning model and to perform the query optimization. This local machine has an Intel i5 2500K Dual-Core processor running at 3 GHz with 16GB DRAM. The second set consists of 10 dedicated Virtual Private Servers (VPSs) that are used for the deployment of the query execution engine. 5 of these VPSs, called small containers, have one Intel Xeon E5-2682 processor running at 2.5GHz with 1 GB of DRAM. The other 5 VPSs, called large containers, each has two Intel Xeon E5-2682 processors running at 2.5GHz with 2 GB of DRAM. The query optimizer and the query engine used in this experiment are modified from the open source database management system,PostgreSQL 8.4 [14]

Fig. 5 :Fig. 6 :

 56 Fig. 5: Time cost performance of executing query using different algorithms

Fig. 7 :

 7 Fig. 7: Impacts of the weight of query execution time on the performance improvement of the algorithm over the baseline algorithm, NoReopt

  ). Update Rt=R (wp, Actiont.time, Actiont.money));

	Algorithm 1: Query Processing with Reinforcement Learning Based
	Re-Optimization (Re-OptRL)
		Data: SQL query, Weight Profile wp, Reward Function R (), Learning rate α,
		Discount rate γ
		Result: The query result set of the input query
	1 t=0;
	2 Result = ∅;
	3 Qt= 0;
	4 while QEP = ∅ do
	5	State St = convert QEP to a state vector;
	6	Actiont=RunLearningModel (St, wp);
	10	Obtain Q-value of next state Qt+1 from the neural network;
	13	t=t+1;
	14 end
	15 return Result;

7

Result=Result ∪ execute (Op, Actiont); 8 QEP=QEP-Op; 9 11 Update Q-value of current state Qt = Bellman(Qt, Qt+1, Rt, α, γ); 12 Update Weights in the neural network;

Acknowledgement

This work is partially supported by the National Science Foundation Award No. 1349285.