Aitor Sanchoyerto
email: aitor.sanchoyerto@ehu.eus1

Ruben Solozabal
email: ruben.solozabal@ehu.eus1

Bego Blanco
email: bego.blanco@ehu.eus1

Elisa Jimeno
email: elisa.jimeno@atos.net

Endika Aldecoa
email: endika.aldecoa@ehu.eus1

Estrella Basurto
email: estrella.basurto@ehu.eus1

Fidel Liberal
email: fidel.liberal@ehu.eus1

Orchestration of Mission-Critical services over an NFV architecture

Keywords: NFV, Management and orchestration, MC

In the race towards 5G, NFV (Network Functions Virtualization) arises as one of the enabler technologies. The intelligent orchestration of the network becomes a key element to achieve the demanded network slicing for an efficient allocation of the available shared virtualised resources. In this paper we propose an intelligent orchestration process of mission critical services over an NFV architecture. Mission critical services have tight requirements in terms of latency and high-availability that must be met in an end-to-end basis. Our proposal includes a monitoring system that collects performance data from the VNF (Virtual Network Function) instances in order to feed the decisionmaking process of the orchestrator and then elastically assign resources to the network service. The software components that compose our deployment are presented as well as the validation scenario in which the features of the test-bed are exposed.

Introduction

In the recent years, the Public Safety (PS) field has shown an increasing interest towards different forms of network sharing models in contrast to building out dedicated legacy PS networks. This interest has led 3GPP to include, from Release 11 onwards, the requirements of Mission Critical (MC) communications as a central topic to address the key requirements of the next generation broadband PS networks.

In order to support the demanding set of requirements of the PS vertical, NFV offers a new way to design, deploy and manage networking services. As a key technology for the development of the 5G ecosystem, NFV decouples the network functions from proprietary hardware appliances in order to be run in a virtualized infrastructure. This architecture drives the rapid development of new network services with elastic scale and automation. Additionally, the use of end-to-end network slicing mechanisms will allow sharing the infrastructure among other vertical industries or services and customising its capabilities on a per-tenant basis.

This paper presents the intelligent orchestration process of a Mission Critical Push-To-Talk (MCPTT) service over an NFV architecture. MCPTT compels tight requirements that include, among others, high availability and reliability, very low latency, support for one-to-one and group calls, talker identification and high audio quality for clear interchange of information. In this context, the network orchestrator must provide the tools to share edge computing capabilities between mission critical and commercial users.

MCPTT is a PS mission-critical voice communication type, aimed at the coordination of emergency teams that are organized in groups [START_REF]Technical Specification Group Services and SystemAspects; Mission Critical Push to Talk (MCPTT) over LTE[END_REF]. It provides an arbitrated method by which two -or more-users may engage in communication. Users may request permission to transmit (e.g., traditionally by means of a press of a button) and the MCPTT service provides a deterministic mechanism to arbitrate between requests that are in competition (i.e., Floor control). When multiple requests occur, the determination of which user's request is accepted, and which users' requests are rejected -or queuedis based upon a number of characteristics (including the respective priorities of the users in contention).

The challenge consists on transparently and elastically allocating the available resources to the variety of actors requiring different services with different priorities in space and time. To that aim, the network slicing based on virtualization techniques must be used to modify the network behaviour by changing functions or reconfiguring parameters. At this point, the intelligent orchestration process of the network slices, including commercial and MCPTT users, involves the collection of performance data through a monitoring system that processes them and generates the corresponding alerts.

This paper is organized as follows. Next section provides a brief overview on the current state of network management and orchestration towards the definition of service slices. Then, Section 3 describes the deployment of an MCPTT service over NFV architecture, including the technical specifications of the platform that will host the service. Section 4 describes the validation scenario to later, in Section 5, propose an intelligent orchestration process that includes a monitoring module to feed the decisionmaking engine. Finally, Section 6 summarizes the main contributions and poses new research challenges that will be addressed in the future.

Related Work

The previous section remarks the importance of network slicing in NFV architectures to simultaneously provide a multitude of diverse services over a common underlying physical infrastructure. The concept of slicing in 5G and next-to-5G environments has been widely discussed in the recent years.

Authors in [START_REF] Foukas | Network Slicing in 5G: Survey and Challenges[END_REF] define the concept of slicing as the separation of a single physical network into multiple isolated logical networks. Another definition in [START_REF] Martínez | Network Slicing Resource Allocation and Monitoring over Multiple Clouds and Networks[END_REF] states that a network slice instance is formed by a set of network functions and logical resources enabling the deployment of a complete logical network infrastructure capable of accommodating the requirements for a specific service. Then, the network slicing in an NFV architecture includes the allocation of the common virtualised resources among the existing services.

Work in [START_REF] Ordoñez-Lucena | Network Slicing for 5G with SDN/NFV: Concepts, Architectures and Challenges[END_REF] shows an example of a network slicing use case over NFV with two tenants, each of them operating several service slices. The authors describe the multiple challenges to be addressed in order to make the slicing process possible. These challenges include the design of adequate resource management mechanisms that enable resource sharing among slices when necessary, without violating their required performance levels. The work in [START_REF] Han | Application-Driven End-to-End Slicing: When Wireless Network Virtualization Orchestrates With NFV-Based Mobile Edge Computing[END_REF] proposes an orchestration system that realises application-aware end-to-end slices on demand, each of which contains not only guaranteed end-to-end bandwidth resources (i.e., in the forms of virtual links, virtual switches, and virtual access points) but also isolated Information Technology (IT) resources (i.e., in the form of virtual network functions) to carry specific applications with quality of service (QoS) guarantees. These examples reflect the interest on network slicing, but the challenge of the elastic resource allocation that dynamically distributes the available virtualized resources among the existing slices instances still remains unsolved, especially when related to the dynamic prioritization of services.

One fundamental issue when addressing the dynamic allocation of resources for network slicing in NFV is related to the possibility of scaling the VNFs. There are many algorithms that employ machine learning or other Artificial Intelligence (AI) methods to decide the optimal number of instances of a VNF needed to provide the requested service. The study in [START_REF] Wang | Online VNF Scaling in Datacenters[END_REF] targets the dynamic provisioning of network services expressed as one or multiple service chains in cloud datacentres and designs efficient online algorithms without requiring any information about future traffic rates. Authors in [START_REF] Fei | Adaptive VNF Scaling and Flow Routing with Proactive Demand Prediction[END_REF] seek a proactive approach to provision new instances for overloaded VNFs ahead of time based on the estimated flow rates. The authors formulate the VNF provisioning problem in order that the cost incurred by inaccurate prediction and VNF deployment is minimized. The work in [START_REF] Wang | Online Learning-Assisted VNF Service Chain Scaling with Network Uncertainties[END_REF] addresses a dynamic VNF service chain deployment and scaling by a novel combination of an online provisioning algorithm and a multi-armed bandwidth optimization framework, which exploits online learning of the available bandwidths to enable the optimal deployment of a scaled service chain. This kind or methods solve the analytical problem of deciding when and how to scale a VNF, but must jet be introduced into a complete orchestration system in order to be integrated with the management and orchestration (MANO) operation.

The existing literature shows that, although network slicing has gained a great interest, the proposed solutions are still in process of being fully developed. Among the many challenges that arise, one fundamental issue is the automation of the resource allocation for the elastic dimensioning of the slice. The following sections of this paper introduce the concept of service monitoring, that involves the collection and processing of performance data from the NFV architecture. The monitoring system is an intelligent decision-making module that generates the necessary alerts to trigger the orchestration processes, such as VNF scaling, in order to perform the re-allocation of resources as needed.

Deployment of Mission Critical Push-To-Talk over an NFV architecture

To evolve the MCPTT into an NFV-ready service it must be defined as a Network Service and, as such, be described by a Network Service Descriptor (NSD). Each VNF that composes the service chain, for its side, is defined by a VNF Descriptor (VNFD) plus the image that implements its correspondent functionality. When a MC service is instantiated, the orchestrator must create the network slice that accommodates critical services ensuring the high availability, reliability, and low latency expected for these communications. This implies not only the allocation of the necessary resources to instantiate and run the VNF chain, but also the monitoring and scaling of the service in order to meet the elastic demand. The MC slice must have a prioritized access to resources in the cloud infrastructure, which may result in a low-priority service deallocation, if required, in order to meet the service level agreements (SLAs).

The specifications of the platform and the virtualized service are described in the following sections.

Platform specifications

The platform selected to perform the testbed complies the ETSI NFV standard [START_REF]Network Functions Virtualization (NFV); Virtual Network Functions Architecture[END_REF]. The orchestrator selected is Open Source MANO (OSM) [10], an ETSI-hosted initiative for the development of open-source NFV Management and Orchestration software stacks that is aligned with the ETSI reference architecture. OSM works in conjunction with Canonical's Juju to configure the VNFs. The Virtual Infrastructure Manager (VIM) used in the project is OpenStack.

Service specifications

Presenting the service as a single VNF provides little flexibility for scaling possibilities. The service must be separated into functional blocks that belong to different VNFs composing the whole network service. This allows an efficient orchestration of the service according to the demand and at the same time it enables the provision of redundancy to the service.

Key to define the VNFs that compose the service is to detect which ones belong to the signalling plane and which ones to the data plane. Bear in mind that data plane is susceptible to scale up proportionally to the number of users while the control play may follow a different scaling ratio. Being able to separate these planes, enables us to replicate the data plane near the end-user easily, following the demand in specific locations.

Figure 1 shows how MCPTT service has been divided in several VNFs following the user/control plane separation criteria, as well as considering the most efficient way to scale the service. The first elements being isolated are control elements as the DNS (Domain Name System) server, the HSS (Home Subscriber Server) and the controlling CSC (Common Services Core) servers. The CSC is composed by the services: IDMS (Identification Management System), KMS (Key Management System), CMS (Configuration Management System) and GMS (Group Management System). This enables the redundancy of these critical control components. The MCPTT application server is also separated into a functional block that manages the Floor Control responsible for the arbitration of talk turns, and the Media Distribution VNF, in charge of processing the media encryption, recording and media multiplexing tasks. This way the system achieves a complete Control and User Plane Separation. The service descriptors are defined in OSM following the Yang information model.

Validation Scenario

To demonstrate that this architecture provides a solution for deploying efficient and elastic MC services over a network slice, we will describe a validation scenario [START_REF]5G ESSENCE Deliverable D2.2, Overall System Architecture and Specifications[END_REF]. It is composed by three stages with an increasing emergency level, that are described next. The objective of this validation scenario is how the MCPPT service behaves with the increase in each scenario of the number of PS users that will use the MCPTT service when increasing the alarm level. Monitoring and orchestration will ensure continuity and quality of service.

Stage 1: Under normal circumstances, the platform provides different network slices. Some slices correspond to PS organizations running respectively a MC service and while other slices correspond to legacy end users that have subscribed to the classical communications. Each network slice is composed of an allocation of cloud resources. In addition to the QoS guarantees for each tenant, the deployment owner has to assure the required levels of isolation in the provisioning of the network slices.

Stage 2:

In case of emergency, the MANO will be able to react to the new service requirements. For instance, the MCPTT communications provider may require additional service in order to cope with an increased number of first responders. Based on service scaling policies, the MANO will implement new elastic resource allocation schemes, giving priority access to first responders and taking into cloud resources (for deploying more resource-consuming cloud services). In this situation, the commercial slices may suffer a service degradation in favor of the PS slices in order to guarantee the quality of service of mission-critical applications.

Stage 3:

In case that ICT infrastructure is damaged during a natural disaster or a terrorist attack, the infrastructure should be able to be operative to dynamically instantiate required services to ensure local service continuity until the infrastructure is restored.

5

Monitoring and Orchestration Telemetry is the automated process of collecting measurements from remote or inaccessible equipment and transmit them to receivers. Monitoring is a continuous assessment that is in charge of the analysis of the data extracted from telemetry and use it to manage the service. The telemetry and monitoring in an NFV system are mainly in charge of managing information that the hypervisor or the VNFs itself expose during execution. These systems are expected to enable additional non-functional requirements such as: scalability, non-intrusiveness and service continuity.

In order to guarantee the quality of the service, the monitoring system must be continuously analyzing the metrics received. These metrics can be oriented towards re-sources or service performance indicators. With the first ones the system extracts information of the hardware infrastructure on which the services are deployed. And with the second, it ensures that it complies with the agreed SLAs.

Each service requires a specific capacity to perform efficiently, minimizing processing times and delays. The quality of the services offered to customers must comply with the service level contracted, for this reason, the monitoring system must be provided with intelligence. So that depending on the metrics received by telemetry, it can predict the degradation of the quality of service and notify the system orchestrator to perform the most appropriate decision.

Figure 2 represents how the resource metrics are collected using the Open-Stack Monitoring Frameworks [START_REF]OpenStack Telemetry[END_REF]. Ceilometer [START_REF]OpenStack Celiometer[END_REF] is focused on collecting data from physical and virtual resources available to the cloud deployment. These data can be stored by a Gnochi [START_REF]OpenStack Gnocchi[END_REF] database, and actions or alarms can be triggered according to predefined conditions set by the user on Aodh [START_REF]OpenStack Aodh[END_REF].

On the other hand, service metrics are exported using Juju´s charms, preconfigured with the VCA (VNF Configuration and Abstraction) on the orchestration phase. With this procedure, the VNF can be accessed to gather any information from the running services, in this case scenario these metrics are being collected by the mcptt_exporter service.

Then, the service metrics are collected using Prometheus [START_REF]Prometheus[END_REF], which is a white box monitoring and alerting system designed for large and scalable environments that includes built-in and active scraping, storing, querying, graphing, and alerting based on time series data. Prometheus covers the domains of instrumentation (using so called "exporters" to instrument platform or applications) and monitoring (providing mechanisms for the data collection, alerting, etc.). It features:

• A multi-dimensional data model, where data can be sliced and diced along multiple dimensions like host, service, endpoint and method. • Operational simplicity to set up monitoring anywhere without being an expert through configuration files. • Scalable and decentralized, for independent and reliable monitoring.

• A powerful query language that uses the data model for meaningful alerting and visualisation.

As mentioned, an exporter must be introduced into the monitored service. It is in charge of publishing all the information related to the users, calls and from the MCPTT Server itself. These service metrics are collected and stored by Prometheus, while Grafana is used for graphic visualization.

The data collected during monitoring can be used to trigger events for automated resource re-provisioning, in cooperation with the orchestrator OSM. One of the main methods to guarantee the fulfillment of the SLAs is autoscaling. This solution needs the allocation of a load balancer to monitor and distribute the loads across all the VMs on the scaling group. Hence, autoscaling can increase, decrease, and replace instances without manual intervention even across thousands of instances. This should not be confused with scale-in and scale-out of the Network Service; NS scaling refers to the addition or removal of VNFs. OSM supports scale-out and scale-in operations on running services.

Conclusions and Future Work

This paper presents an NFV architecture that deals with the tight requirements of a Mission-Critical service. It is capable of maintaining the high availability and reliability required for the service, while sharing the infrastructure in an elastic manner between mission critical and commercial users thanks to the supervision of a network orchestrator. To that aim, the network slicing based on virtualization techniques is key to maintain isolation between clients.

The intelligent orchestration process of the network slices involves the collection of performance data through a monitoring system that processes them and generates the corresponding action on the infrastructure. Being the most significant the scaling of the service VNFs. There are many algorithms that employ machine learning or other AI methods to decide the optimal number of instances of a VNF needed to provide the requested service. This paper is focused on presenting the architecture, foreseen for the future the definition of the metrics or the development of an intelligent decision-making system that cooperates with the orchestrator.

Figure 1 :

 1 Figure 1: MCPTT network service

Figure 2 :

 2 Figure 2: Service Monitoring and Orchestration

Acknowledgements. This work has been partly funded by the EU funded H2020 5G-PPP project ESSENCE (Grant Agreement Nº 761592) and the Spanish Government's MINECO project 5GRANVIR (TEC2016-80090-C2-2-R).


Elias Dritsas
email: eldritsas@gmail.com

Gerasimos Vonitsanos
email: mvonitsanos@ceid.upatras.gr

Ioannis E Livieris
email: livieris@teiwest.gr

Andreas Kanavos
email: kanavos@ceid.upatras.gr

Aristidis Ilias

Christos Makris
email: makri@ceid.upatras.gr

Athanasios Tsakalidis

Pre-processing Framework for Twitter Sentiment Classification

Keywords: Classification, Microblogging, Pre-processing, Sentiment Analysis, Supervised Machine Learning, Twitter

Twitter Sentiment Classification is undergoing great appeal from the research community; also, user posts and opinions are producing very interesting conclusions and information. In the context of this paper, a pre-processing tool was developed in Python language. This tool processes text and natural language data intending to remove wrong values and noise. The main reason for developing such a tool is to achieve sentiment analysis in an optimum and efficient way. The most remarkable characteristic is considered the use of emojis and emoticons in the sentiment analysis field. Moreover, supervised machine learning techniques were utilized for the analysis of users' posts. Through our experiments, the performance of the involved classifiers, namely Naive Bayes and SVM, under specific parameters such as the size of the training data, the employed methods for feature selection (unigrams, bigrams and trigrams) are evaluated. Finally, the performance was assessed based on independent datasets through the application of k-fold cross validation.

Introduction

The rapid development of modern computing systems along with Internet access and high communication capabilities, has turned these schemes into an integral part of human everyday life. Nowadays, users can express their personal opinion on any matter whenever they wish as well as share their thoughts and feelings. It is no coincidence that most websites encourage their users to review their services or products while social media accounts have significantly increased.

In particular, a user through websites can be informed, express their personal views on a variety of topics and simultaneously interact with other users. Hence, this kind of interaction produces a large amount of data that is of particular interest for further process and analysis. Companies have started to poll these microblogs to get a sense and understand the general sentiment towards their product. Often, these companies study user replies and in following reply on the corresponding microblogs. The challenge is to build tools that can detect and summarize an overall sentiment, so that valuable conclusions and information on various kinds of issues can be drawn. For example, one can consider demographic and social conclusions, information of economic nature such as the prevailing view of a product or service, or even results of political content.

Sentiment analysis constitutes a subtle field in information mining. It is considered a computational analysis and categorization of opinions, feelings and attitudes that are drawn up in text format. The use of Natural Language Processing techniques is sought through polarity performance, while polarity can be characterized by many different classes. Specifically, the positive and negative terms, which correspond to the positive or negative view that a user has, are utilized in terms of a specific event or topic.

It is widely noted that the emotional analysis has many applications, as an individual may have a view on a huge range of issues of a different nature, such as economic, political, religious, and so on. For this reason, the positive and negative classes are not the only ones used, as aforementioned. It constitutes a basic way of studying and analyzing such data. Indeed, in recent years, the great volume of raw information generated by Internet users has increased the interest in processing such data. In other words, it is the process of mining and categorizing subjective data so as to extract information about the polarity and overall emotion expressed in natural language in text form [?].

Data mining is considered an important part of the data analysis [?]. It largely consists of collection, processing and modeling of data and is aimed at the objectives shown in Figure ??. Its characteristics are the export of information from a dataset, its following transformation in a comprehensible structure with the use of various techniques (machine learning, statistics, etc.) which facilitate analysis, and finally the conclusions export as well as the decision-making. In this present work, the main contributions concern the following aspects. Concerning data pre-processing techniques, tweets face several new challenges due to the typical short length and irregular structure of such content. Hence, data pre-processing constitutes a crucial step in the corresponding field of sentiment analysis, since selecting the appropriate pre-processing methods, the correctly classified instances can be increased [?]. To properly implement the data analysis process, it is necessary to process the raw data collected in a variety of ways. Initially, it should be decided what data will be used depending on the purpose of the analysis. In following, it is necessary to eliminate the abnormalities and deal with incorrect values and/or incomplete inputs. Subsequently, the processed data are modified in a form suitable for mining them. Therefore, we focus on designing an efficient pre-processing tool which facilitates the sentiment analysis conducted based on supervised machine learning algorithms. Another contribution is the application of Latent Dirichlet Allocation (LDA) based probabilistic system to discover latent topics from the conversations in connection to the event we take into account.

The proposed research is organized as follows: Section ?? describes the background topics and the challenges faced, while Section ?? explains the process of information retrieval from Twitter platform. The same section analyzes the tool for data pre-processing, which is the main contribution of this work. Furthermore, Section ?? presents the experimental results and main conclusions extracted from the study and the analysis. Finally, Section ?? portrays conclusions and draws directions for future work.

Related Work

Sentiment Analysis can be considered as a field of data mining, which intends to determine the polarity of a subjective text, through the use of various algorithms. In the recent years, this particular branch of science has started to gain increasing interest both from academic and organizations perspective; users started to study the emotions around a subject for a variety of reasons. This interest is boosted by the rapid growth of users participating in social media, which is a modern world phenomenon. Some researches that have played a key role in the evolution and importance of emotional analysis are presented below.

Pang and Lee [?] introduce techniques and challenges in the application of emotional analysis, while at the same time a plethora of datasets are widely used. Also, the authors show how both the frequency of the terms and the ngram feature selection affect the result. In addition, within the framework of the system developed, the incidence of the individual condition and in following the posts splitting into n-grams was taken into account.

Of particular interest is the application of emotional analysis to different hierarchical levels. Pang et al [?] study the effectiveness of document-level for a large number of critically acclaimed films from the popular website IMDB. Turney et al [?] examine the polarity of a file through its proposals. Phrases that include adjectives and/or adverbs, or other features and parts of speech that are highly likely to express the author's emotion, are selected. Concretely, user reviews on a variety of topics such as movies, travel destinations, banks and cars, were utilized. Wilson et al [?] analyzed the views of the MPQA corpus along with a set of data containing journal articles from different sources that have been rated in terms of their emotion. An important part of their work was the separation of neutral phrases in polar and objective and then, the polarity analysis of subjective phrases to extract the overall feeling of the text.

Furthermore, the emotional analysis can also be applied in the field of economic interest, in sets of journalistic content and critical films [?,?,?], as well as for political aspects [?,?]. Initially, authors in [?] studied reviews for mobile applications and in following exported important features for their nature. Then, in [?], product reviews to identify consumer's sentiment in terms of certain characteristics and the products themselves were analyzed. Authors in [?] constructed a model for emotional analysis based on travelers reviews about the destinations they visited. In a similar way, a system that analyzes the sentiment of publications of real-time Twitter users to predict the results of the 2012 presidential election in the United States of America was created [?]. Finally, in [?], researchers used messages (not real-time data) that included reference to German political parties in 2009.

Previous works regarding emotional content are the ones presented in [?,?,?], in which the authors presented various approaches for the automatic analysis of tweets and the recognition of the emotional content of each tweet based on Ekman emotion model; based on Ekman emotion model, the existence of one or more out of the six basic human emotions (Anger, Disgust, Fear, Joy, Sadness and Surprise) is specified. Finally, a novel distributed framework implemented in Hadoop as well as in Spark for exploiting the hashtags and emoticons inside a tweet is introduced in [?]. Moreover, Bloom filters are also utilized to increase the performance of the proposed algorithm.

Tools and Environment

The pre-processing methods evaluated by the current research are three different data representations, namely unigrams, bigrams and trigrams. Two well-known machine learning algorithms were selected for classification, namely Naive Bayes and Support Vector Machine (SVM) as shown in the following section.

Twitter

The platform that is being studied in this work is Twitter. This is a platform for posting posts, exchanging messages between users, and modifying their private profiles according to their needs. There is the possibility of communicating links, images and audiovisual material to the posts. Twitter has gained considerable interest on a global scale, due to the services it provides its users with.

A special feature that makes emotional analysis quite difficult in its context is the small length of suspension that it allows its users. It is therefore perceived that studying the polarity of users' publications, beyond the general challenges and difficulties faced, is even more difficult due to their limited length.

Publications Mining Tools

The mining of posts was done using the Tweepy library which through the Twitter interface allows managing a user's profile, collecting data by optionally using certain search keywords, and finally creating and studying a stream of posts over a specific time interval.

In this work, posts were stored in a CSV file, where rows contain the posts that were extracted, while the columns contain the values of the different attributes of each post (e.g., date, text, username, etc.).

Useful tools in the context of this work were the following:

1. The Natural Language Tool Kit is a natural language processing library, which offers classification, parsing, tagging, and clipping stemming possibilities. 2. Scikit-Learn is a library that addresses the implementation and development of machine learning techniques and text editing tools. This library interacts with the Python, NumPy and SciPy computational and scientific libraries enhancing its efficiency and speed significantly. 3. Pickle is a library that converts objects in a form understandable only by the Python language, in order to limit the space they occupy in memory; this is due to certain features (e.g. JSON format) to be stored and reused whenever necessary. In this work, this library was used to allow the usage of objects returned after the classifier training, without retraining, whenever necessary.

Pre-processing Scheme

In order to facilitate the mining process of the collected data, it is necessary to apply several pre-processing steps [?]. The main parts of this process are the following .

Pandas Library: It was utilized to facilitate the management of input files having the components of the publications. Then, a dataframe was created only with the most important components for our analysis, i.e. records were removed, either with incorrect values or the ones not rated.

Regular expressions: They were utilized to remove the urls and references to other users' username. Also, it was possible to find and replace/remove alphanumeric characters that match a predefined search pattern and remove unnecessary spaces. The repetition of suffix characters that were used for emphasis reason and numeric characters, which do not facilitate Sentiment Analysis, were removed as well.

Emoticons: Emoticons are characters, such as punctuation and parentheses, which in turn form representations of expressions of the human face, e.g. cheerful person {:-)}, but also different representations that play an important role in analyzing the feelings of each publication. Emoticons are widely used in social media, especially on the Twitter, to express feelings and impressions in a short way. Therefore, a set of regular expressions containing a large part of these representations was created. Additionally, a set of widely used unofficial abbreviations was generated in order to replace words that users make up. For example, the lol expression, which was replaced by its equivalent full form, namely, laugh out loud.

Autocorrect library: This library uses a list of words found in recognized dictionaries, and given an input word, compares its similarity to the words on that list. If the input word is correctly spelled, it is returned as a whole. If it is not correctly spelled, then its similarity is checked with the words in the list; if its similarity is greater than a certain threshold, then it is replaced by the word in the list. Otherwise, it is returned as a whole, without any changes.

Pycontractions library: It detects a set of successive characters in which the apostrophe is contained and replaces it with the full form of expression. Expressions with the {'s} special character complicate emotional analysis since they express two or more words, which makes tokenization, as well as, normalization, particularly difficult. For effective mining, contractions are written in their original form no matter how complicated it is. In case the set has a single possible replacement, then the expression is transformed into its original full form. Using a grammatical controller and Word Mover's Distance [?], a method of calculating the distance between the original text and the texts produced, called "compatibility" metric, is derived. The substitution applied is the one with the highest value.

Emoji library: They are Unicode characters in the form of an icon representing face expressions, and many kinds of objects. This part differentiates our work from others, as in most other studies, emoticons and emojis are not taken into account. However, they are used in most publications in opinion mining. This library uses a mapping list, as created by the Unicode Consortium. Unicode characters included in a text are reviewed, and if they are in this list, they are replaced with the text form of their representation.

Part-of-Speech Tagging: Each word of the text is tagged according to the part of the speech that it constitutes (adverb, verb, object). This process uses the context of the text to be analyzed as well as a set of aggregated elements (corpus), to evaluate and attribute the part of speech to the particular term being studied.

Lemmatization: It is the process in which lexical and morphological analyses of words are taken into account in order to remove complex suffixies and to retrieve the lexical form of the term. It is applied after POS tagging and facilitates the emotional analysis through the application of machine learning algorithms. In the context of this work, POS tagging labels are Penn Treebank format.

Tokenization: It is the separation of sentences into a list of symbol terms that can be used to reform the original sentence. Both emoticons and emojis that have now been converted to text characters are taken as tokens, without being divided into individual characters or punctuation marks. The tokenization process is applied to all sentences, and their terms are stored in the same token list. Essentially, the tokens list is created for each post. Once the publication's details are now in a list in the order they appear in it, some more conversions are made to optimize and improve the viewing process.

Punctuation: These are also tokens of the list. Generally, punctuation marks do not attach any emotional significance to the publication and thus, they are removed.

Stopwords: These are words appearing very often, without expressing some form of feeling. The reason why such words are removed is because the whole attempt is to examine meaningful words in order to determine the overall emotion expressed in publication.

Features

N -grams are one of the most common structures used in text mining and natural language processing fields. They constitute a set of co-occuring words within a given window. In addition, as already known, a Markov assumption is the assumption that the probability of a word depends only on the previous word. So, Markov models are these probabilistic models that with their use, the probability of a future aspect without looking too far into the past can be predicted. The most popular ones are the bigrams, which search for one word from the past, the trigrams, which search for two words from the past, and the n-grams, which search for n1 words from the past.

The "bag-of-words" approach is considered a very simple and flexible representation of text that describes the occurrence of words within a document. It involves a vocabulary of known words as well as a metric of the presence of these known words. The model considers only the existence of the known words in the document, and not the exact place where to be found in the document. The intuition is that documents are similar if they have similar content.

The Summed Term Frequency constitutes the sum of all the term frequencies in the documents. In the proposed paper, it is utilized as

SummedT ermF requency = d∈D T F n-gram (1)
In addition, the "Apply Features" method has been taken into consideration in order to obtain a feature-value representation of the documents. Concretely, this method is used in order to apply a "positive" or a "negative" label to each feature of the training data.

Topic Modeling

One other aspect we want to take into consideration in our proposed work is the verification of whether all the posts discuss the specific topic. Topic modeling considers a document as a "bag-of-topics" representation, and its purpose is The results of our work are presented in the following Tables ?? to ??. The Accuracy, in terms of percentage, is used as the evaluation metric of the two different algorithms (Naive Bayes and SVM) for the different setup (Unigrams, Bigrams and Trigrams). Also, the percentage of training and test set is taken into account when considering the two datasets.

In Table ??, the results of the RapidMiner platform3 are presented. We have used RapidMiner as a baseline to emphasize the improvement of our proposed methodology. Furthermore, in RapidMiner we cannot include features that are utilised in our paper, such as emojis and emoticons, etc. We observe that SVM performs better than Naive Bayes for the three different setups and for both datasets. Secondly, in both datasets, Unigrams and Bigrams achieve better accuracy than Trigrams; this is expected as tweets usually have small length due to the number restriction that characters have and thus, Trigrams cannot be considered as a qualitative metric. In following, Table ?? presents the results for different ratio of training versus test set. We have utilised three different cases, with training set having values equal to 70, 75 and 80 whereas test set has values equal to 30, 25 and 20 respectively. Worth noting is the fact that our proposed methodology outperforms the results from RapidMiner as regarding Selfdriving Cars Dataset, for both classifiers and the three different setups, the accuracy has lower value equal to 72% and higher equal to 80%. On the other hand, for the Airlines dataset, we observe high rate fluctuations as the higher and the lower percentages are de-picted. Concretely, Naive Bayes (Trigrams) achieves the lowest accuracy with almost 60% and Naive Bayes (Unigrams) achieves the highest accuracy with 85% (for training-test ratio equal to 80-20). Furthermore, we notice that for all cases, as the percentage of training set increases, so does the accuracy. This is something that we expect as higher values of training set increase the classifier's results. Finally, Table ?? presents the Accuracy results when splitting with 10-Fold Cross-Validation. The concept of using this technique is that important information can be removed from the training set. In addition, this method is simple to understand and generally results in a less biased or less optimistic estimate of the model skill than other methods, such as a simple train/test split. As in Table ??, Naive Bayes outperforms SVM, while it achieves the higher percentage with value equal to 85%. What is more, for both datasets, Naive Bayes has values close to 79% whereas SVM has different values for the three setups. In this paper, we proposed a pre-processing framework for Twitter sentiment classification. We chose Twitter because of tweets' short length and content's diversity. We used supervised machine learning techniques for the analysis of the raw data in the user posts and incorporated emojis and emoticons if order to enrich our features. Furthermore, we applied the Latent Dirichlet Allocation (LDA) based probabilistic system to discover latent topics from the conversations. Two popular classifiers (Naive Bayes and SVM) were used for three different data representations (unigrams, bigrams and trigrams) in order to perform our experiments in two datasets.

In the near future, we plan to extend and improve our framework by exploring more traits that may be added in the feature vector and will increase the classification performance. Moreover, we plan to compare the classification performance of our solution with other classification methods. Another future consideration is the adoption of other heuristics for handling complex semantic issues, such as irony that is typical of messages on Twitter.

Fig. 1 .

 1 Fig. 1. KDD process for knowledge mining from data [?]

Table 1 .

 1 Datasets Details

	Sentiment	Selfdriving Cars Dataset Airlines Dataset
	Positive	1262	2363
	Slightly Positive	1452
	Neutral	4245	3099
	Slightly Negative	1498
	Negative	1076	9178
	Total Number of Tweets	9533	14640

Table 2 .

 2 RapidMiner Results -Accuracy

	Setup	Selfdriving Cars Dataset Airlines Dataset
	Naive Bayes (Unigrams)	62.13	64.95
	Naive Bayes (Bigrams)	64.37	62.40
	Naive Bayes (Trigrams)	58.89	50.66
	SVM (Unigrams)	69.56	75.50
	SVM (Bigrams)	71.78	73.20
	SVM (Trigrams)	71.49	70.95

Table 3 .

 3 Accuracy for different Training -Test Set ratios

		Selfdriving Cars dataset	Airlines dataset
	Setup	70 -30 75 -25 80 -20 70 -30 75 -25 80 -20
	Naive Bayes (Unigrams) 71.99	72.73	74.04	82.60	83.72	85.18
	Naive Bayes (Bigrams)	77.83	79.49	80.76	77.78	78.55	80.94
	Naive Bayes (Trigrams)	76.57	77.93	79.44	59.45	59.70	60.72
	SVM (Unigrams)	73.50	74.45	74.15	77.45	77.57	78.44
	SVM (Bigrams)	76.43	77.77	78.76	68.90	69.43	69.56
	SVM (Trigrams)	75.70	77.27	78.67	64.94	65.28	65.82

Table 4 .

 4

	Setup	Selfdriving Cars Dataset Airlines Dataset
	Naive Bayes (Unigrams)	78.66	85.38
	Naive Bayes (Bigrams)	79.92	79.99
	Naive Bayes (Trigrams)	79.29	74.26
	SVM (Unigrams)	73.04	77.26
	SVM (Bigrams)	76.59	69.32
	SVM (Trigrams)	75.93	65.25
	5 Conclusions and Future Work

https://www.kaggle.com/c/twitter-sentiment-analysis-self-driving-cars

https://www.kaggle.com/crowdflower/twitter-airline-sentiment

https://rapidminer.com/

Acknowledgement

Elias Dritsas was funded by General Secretariat for Research and Technology (GSRT) and Hellenic Foundation for Research and Innovation (HFRI) and supported by University of Patras. Andreas Kanavos, Aristidis Ilias and Christos Makris are co-financed by the European Union (European Social Fund) and Greek national funds through the Operational Program Research and Innovation Strategies for Smart Specialisation -RIS3 of "Partnership Agreement (PA) 2014-2020".

to cluster each term in each post into a relevant topic. Variations of different probabilistic topic models [?], [?] have been proposed and LDA [?] is considered to be a well known method.

Concretely, the LDA model extracts the most common topics discussed that are represented by the words most frequently used, by simply taking as input a group of documents. The input is a term document matrix, and the output is composed of two distributions, namely document-topic distribution θ and topicword distribution φ. EM [?] and Gibbs Sampling [?] algorithms were proposed to derive the distributions of θ and φ. In this paper, we use the Gibbs Sampling based LDA. In this approach, one of the most significant steps is updating each topic assignments individually for each term in every documents according to the probabilities calculated using Equation 2.

where z i = k shows that the i th term in a document is assigned to topic k, z i signifies all the assignments of topic except the i th term, n -i (k,m,•) is the number of times that the document d contains the topic k, n -i (k,•,wi) is the number of times that term v is assigned to topic k, V represents the size of the vocabulary as well as α and β are hyper-parameters for the document-topic distribution and topic-word distribution respectively.

The number of the Gibbs sampling iterations performed for every terms in the corpus is N ; after this component, the document-topic θ and topic-word φ distributions are estimated using Equations 3 and 4 respectively.

Evaluation

In Table ??, the two datasets studied as well as their characteristics are presented. There are 5 different categories whereas the first dataset contains tweets for all of them and the second dataset contains tweets for the 3 of them. The total number of tweets studied per each dataset is also considered. The first dataset consists of tweets about self-driving cars 1 . The sentiment is categorized into 5 categories, ranging from very positive to very negative. The second dataset consists of the feelings that travelers have in February 2015 towards the problems of each major U.S. airline 2 . The sentiment in the tweets of this dataset is categorized as positive, neutral or negative for six US airlines.