Eleana Tsiara

Christos Tjortjis
email: c.tjortjis@ihu.edu.gr

Using Twitter to Predict Chart Position for Songs

Keywords: Social Media Analytics, Prediction, Classification, Regression

With the advent of social media, concepts such as forecasting and nowcasting became part of the public debate. Past successes include predicting election results, stock prices and forecasting events or behaviors. This work aims at using Twitter data, related to songs and artists that appeared on the top 10 of the Billboard Hot 100 charts, performing sentiment analysis on the collected tweets, to predict the charts in the future. Our goal was to investigate the relation between the number of mentions of a song and its artist, as well as the semantic orientation of the relevant posts and its performance on the subsequent chart. The problem was approached via regression analysis, which estimated the difference between the actual and predicted positions and moderated results. We also focused on forecasting chart ranges, namely the top 5, 10 and 20. Given the accuracy and Fscore achieved compared to previous research, our findings are deemed satisfactory, especially in predicting the top 20.

Introduction

Social media have penetrated everyday life to the point that they constitute an integral part of our daily routines. The vast amount of data available through these services can be utilized in many domains including finance, marketing, and politics [START_REF] Rousidis | Social Media Prediction A Literature Review[END_REF]. Social media can be exploited in a variety of cases, such as forecasting the commercial success of movies [START_REF] Asur | Predicting the Future with Social Media[END_REF], election result predictions etc. [START_REF] Oikonomou | A Method for Predicting the Winner of the USA Presidential Elections using Data Extracted from Twitter[END_REF], [START_REF] Beleveslis | A Hybrid Method for Sentiment Analysis of Election Related Tweets[END_REF].

In this paper, we chose Twitter to generate predictions for the Billboard chart. After gathering chart data, including titles, artist names and rankings, as well as tweets related to the top 10 songs for each week, results showed a moderate correlation between the number of mentions of a song and its future performance, but no relation between the number of mentions of an artist and their imminent success.

This work has the following objectives: a) to acquire data from the Billboard chart, including rank, artist and song title of the top 100 songs at the current time. For that purpose we developed a method which extracts these parameters from the official site and saves them in a .json and a .csv file. b) to collect Twitter posts concerning the top 10 songs utilizing the Twitter Search API. c) to preprocess data into a homogenous, structured format removing redundant information. d) to perform sentiment analysis on tweets, categorizing each post as positive, negative or neutral. e) to assess the contribution of the features, extracted from the Billboard chart and collected posts, to forecast the chart for the week to come.

The aim is to predict the top N songs for the following week and evaluate the efficiency of the process. This requires a classifier to generate predictions. The optimal number of N, the highest performing classification algorithm and the best feature combination can be determined by experimentation and result comparison.

The remaining of this paper reviews background in section 2, describes our approach in section 3, discusses results in section 4 and concludes with directions for future work in section 5.

Background

This section reviews forecasting related literature. Twitter was employed to predict the commercial success of movies [START_REF] Asur | Predicting the Future with Social Media[END_REF]. The choice of film was based on the number of discussions and the difference of opinion, as well as by obtaining financial information about it. After collecting about 3 million tweets, the authors used a linear regression model and performed sentiment analysis to conclude that there is a correlation between the fame of a movie prior to its release and the revenue it produces.

Sales prediction was targeted in [START_REF] Boldt | Forecasting Nike's Sales using Facebook Data[END_REF]. The aim was to estimate the impact of each variable such as posts, comments and likes on a group of Nike's Facebook page, examining each variable and each page individually, as well as a combination of all variables and all pages. Moreover, the predictive impact of search query data and the relation between Nike's events and the subsequent Facebook activity was explored. Simple regression scored as high as Bloomberg forecasts in terms of accuracy for predictions pertaining to the near future.

Another study that used Twitter data aimed at predicting the last presidential elections in the USA [START_REF] Oikonomou | A Method for Predicting the Winner of the USA Presidential Elections using Data Extracted from Twitter[END_REF]. Its first objective was to retrieve data from Twitter, while organizing the timing of data collection according to important dates, such as debates. It then used sentiment analysis on 277,509 tweets. Each tweet was given a polarity and subjectivity score, depending on being negative, positive or neutral. Naïve Bayes was used for classification. The results were accurate and in fact predicted the right candidate, in contrast to most polls.

With regards to the prediction of the Billboard chart. Yekyung et al. focused on two tasks: a) the relationship between Twitter activity regarding music and forthcoming sales and b) predicting hit songs for the next Billboard chart [START_REF] Yekyung | #nowplaying the Future Billboard: Mining Music Listening Behaviors of Twitter Users for Hit Song Prediction[END_REF]. The authors gathered over 30 million tweets, searching for the keywords #nowplaying, #np and #itunes, presumably because these hashtags are used to indicate the song a Twitter user is currently listening to. They also collected information from previous Billboard charts over the span of 10 weeks, which resulted in a dataset of songs, each with its own title, artist, rank and the time period it stayed on the chart. Altogether, the information retrieved related to 178 songs and 134 artists. 3 distinct metrics were established: a) song popularity, i.e. the number of tweets related to a specific song, b) artist popularity, i.e. the number of tweets mentioning this artist, and c) the number of weeks a song appeared on the Billboard chart. Forecasting targeted predictions for the top 10 songs, since this was the range with the highest accuracy achieved. Using the Pearson correlation, artist popularity and number of weeks on the chart, were not found to be strong predictors for a song's ranking. However, considering song popularity if all 3 metrics are combined, it is possible to predict the imminent success of a song quite accurately.

Koenigstein et al. approached Billboard chart prediction through the exploitation of Gnutella peer-to-peer network [START_REF] Koenigstein | Predicting Billboard Success Using Data-Mining in P2P Networks[END_REF]. In total, 185,598,176 query strings, originating from the USA, were gathered during a 30-week period. They used both M5 and C4.5 algorithms to predict the Billboard Hot 100 and the Billboard Digital Songs charts. Moreover, in some cases, they also considered a song's debut rank on the chart, while predicted positions were usually either the top 10 or 20. They verified that queries used in peer-to-peer services are strong predictors for a song's success. For the Billboard Hot 100 chart, precision surpassed 86%, while for the Billboard Digital Songs accuracy exceeded 89%.

Finally, Zangerle et al. examined the relationship between song-related tweets and their ranking on the Billboard Hot 100 [START_REF] Zangerle | Can Microblogs Predict Music Charts? An Analysis of the Relationship Between #Nowplaying Tweets and Music Charts[END_REF]. Their objective was to investigate the resemblance of the amounts of Twitter data referring to Billboard tracks to the state of the actual chart, along with the temporal offset between them, so that it can be determined if tweets have a predictive value for the chart and not vice versa. The authors used an existing dataset which consists of 111,260,925 tweets that included the term #nowplaying, gathered during 2014 and 2015. They also collected data from the Billboard Hot 100 chart for the same time period, corresponding to 886 distinct songs. They observed that a song remains on the chart for an average of 11.74 weeks and may stay on the top 100 for up to 58 weeks. To determine the correlation of rankings, they calculated three metrics per song, based on tweets: the median of play-counts per week, the mean play-counts per day and the total play-counts for a whole week. The median achieved the highest correlation (0.5), for 481 songs or 54.29% of the dataset.

The temporal relationship between tweets and charts was investigated through cross-correlation analysis that increased the mean correlation to 0.57 compared to the value of 0.5 that was previously found. 89.23% of all the examined tracks appear to have a temporal lag in relation to the Billboard chart, while 41.09% of all tracks have a negative lag. This means that the last percentage is exploitable for providing predictions about the chart's ranking for the weeks to come. Moreover, the authors followed the same process for songs that were first noticed in Twitter data and appeared on the chart at a later point (619 tracks in total). Similarly, 42.64% of them featured a negative lag thus facilitating future chart forecasts.

Regarding the predictive capability of Twitter data, they compared 3 models: one based exclusively on Billboard chart data, one relying only on tweets and the last one combining both. In terms of RMSE, the Twitter-based model had the worse score of 116.1, while the first one achieved an RMSE of 26.8. However, the multivariate model displayed a notable performance of 14.1. In conclusion a combination of data originating from both the Billboard chart and Twitter can significantly reduce forecasting error, thus song-related tweets can be useful for increasing the accuracy of ranking predictions. Summing up, about 41% of the collected tweets could be used for the prediction of the Billboard chart ranking if handled properly.

Having considered the literature, it can be hypothesized that the more attention a song gathers in Twitter, the more likely it is for people to listen to it and possibly end up purchasing it, further adding to its total popularity. Similarly, the whole publicity an artist gets and the image she presents, considering both her career and personal life, will probably motivate the public to check her work and increase her commercial success.

Approach

The methodology we followed is depicted in Fig. 1 and described below. The Search API was utilized for interacting with Twitter, using Python scripts for searching for tweets, storing posts and performing sentiment analysis. Database (DB) management tasks, including storing and retrieving Twitter data, were implemented using SQL Server. All useful information extracted from DB records was preprocessed and input to Weka [START_REF] Witten | Data Mining: Practical machine learning tools and techniques[END_REF] for mining.

Twitter Search API

Regarding song searching, tweets related to specific words were collected mainly to estimate their airplay count, either from radio stations or from individual users. These queries incorporate the term #nowplaying, which is joined via the AND operator with the titles of all songs, in brackets. Each title is placed in quotes and brackets, if it consists of multiple words. Quotes are necessary to ensure that the post contains all the words in the same order, and they are not just spread in the text. This would lead to the collection of many irrelevant tweets and would add noise to the dataset. Moreover, similarly to the queries about artists, all words of each title relate to the removal of spaces, and the hashtag symbol is inserted at the beginning of each concatenated word.

Figure 1: Proposed methodology

The strings for searching for songs do not include artist names, since they might exceed the 500-character limit and unnecessary complexity would be added to queries. Instead, artist names and titles are matched, when retrieving data from the DB where all tweets are stored. This ensures that the posts refer to the songs that need to be analyzed.

Sentiment Analysis

Overall, the VADER lexicon comprises useful scripts [START_REF] Hutto | VADER: A Parisomonious Rule-based Model for Sentiment Analysis of Social Media Text[END_REF]. Compared to VADER, sentiment analysis with SentiWordNet [START_REF] Baccianella | SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining[END_REF] is not easy to implement, since it includes only the lexicon without any ready-to-use code. Therefore, it is up to the programmer to develop her own code to manipulate the lexicon and make the appropriate configurations to get the result she needs. The task is rather cumbersome because each word needs to be treated separately. It needs to be tagged based on the part of speech they represent, which can be noun, verb, adjective, adjective satellite (a subcategory of adjective according to WordNet) and adverb, and also each term has to be assigned with a score indicating its usage frequency.

For that reason, a handy script from [13] was utilized, featuring a class with methods for calculating different sentiment parameters for a given word or phrase. Most importantly, a procedure for estimating the total sentiment score of a sentence was used. This was the metric of interest we used to make comparisons with the respective result of VADER's sentiment analysis. The scoring method considers the following negation words: (not, n't, less, no, never, nothing, nowhere, hardly, barely, scarcely, nobody, none). Furthermore, there are 3 options for the generation of the sum of scores: average, geometric and harmonic.

The connection to the DB and the execution of queries was implemented in a similar way. However, in this script, there is a need for an extra query to fetch all the tweets that have not been analyzed yet. The table is updated with the compound column receiving the sentiment score through the compound dimension of the polarity_scores method with the extracted tweet text as a parameter. If the compound column was not specified, the method would also return all other metrics, meaning the positive, negative and neutral scores. In order to set the sentiment column, a new method has been defined, which takes a chunk of text as a parameter and returns its semantic orientation (-1 for negative, 0 for neutral and 1 for positive) based on the common thresholds of the compound value.

Mining Information from the Database

In terms of preprocessing, tweet replication is treated with the removal of all records that have the same tweet_id leaving only one copy. Any records with a black text field are also redundant and therefore deleted. Twitter typically does not allow tweeting blanks and all tweets captured by the Search API are retrieved using specific keywords, which should be part of a post's text. Discarding posts with an empty string is just an added form of protection against unexpected behavior.

Duplicate tweets with a different tweet_id field, but identical text, were not removed because they were perceived as adding information and depicting extra attention towards a topic. Moreover, especially in the case of songs that are gathered using the #nowplaying term, the content of the posts is mainly fixed, as it usually includes the artist and title of the track, sometimes followed by a link.

Twitter's Search API captures all tweets, including retweeted posts. These can be sometimes recognized from their textual content, which begins with the pattern "RT @[username]", where [username] refers to the user being quoted. Nevertheless, this is not an official feature and it is not necessarily inserted in every retweet, so there is no accurate way to determine that a post captured via the API is definitely a retweet.

In order to produce a tradeoff between the number of occurrences for each distinct text value (that is the number of posts that have the exact same text) and the number of retweets, the greatest value of the two is chosen. The idea behind this is to get the biggest sample possible and exclude the difference between the two parameters, which is probably tweets that coincide. For retrieving song tweets from the DB, the song title is used with and without spaces among words. For most tracks, the artist must also appear in the text. This is a measure to filter out any records with different songs that happen to have the same title or phrases which are absolutely irrelevant to the tracks under examination.

Attribute Description

In order to use Weka, all the information extracted from the Billboard chart and gathered tweets should be organized in attributes and formatted properly to create an .arff file. The file consisted of 80 instances in total with no missing values. Each instance describes a specific track for one week, during which the track was at one of the top 10 positions of the chart. So, the data set consists of 10 instances per week.

Since our hypothesis is that most of the attributes are positively correlated to the success of a song, a song will get to a higher position when attribute values increase. The Billboard chart ranks songs in ascending order, starting with the most popular ones. So, mainly for harmonization with the rest of the attributes, as it was seen in [START_REF] Yekyung | #nowplaying the Future Billboard: Mining Music Listening Behaviors of Twitter Users for Hit Song Prediction[END_REF], the position of each track is inverted by subtracting it from 101. For instance, number 1 song will be in position 100, while number 10 in 91 and so on. This also applies to previous-position and position attributes.

Each instance of the data represents a different song for a single week and it has to be compared to all other instances, which are the remaining 9 tracks from the top 10 positions of the chart for the same week and 10 more tracks for every week that has been monitored. Data gathering took place daily in a scheduled manner, but the total number of tweets captured on a weekly basis would differ each time, despite requesting a consistent number of items. Therefore, measuring every instance against all the others required some transformation on the specimen of tweets. For each week, the total number of posts for the top 10 songs was summed and the percentage of each track in comparison to the other 9 was calculated. Attributes song-play-count, artist-playcount and artist-tweets were tuned as described above.

The attributes selected are described below: • Previous-position: The song position one week before.

• Chart-weeks: The number of weeks that the song has been in one of the 100 positions of the chart since its first entry. This value is set to 1 as soon as the song makes its chart debut. • Top-weeks: The number of weeks that the song was in number 1.

• Song-play-count: The tuned number of song related tweets, including retweets, with the #nowplaying keyword. • Artist-play-count: The tuned number of tweets, including retweets, which are related to the artist who sings this particular track, along with the #nowplaying keyword, not necessarily referring to this specific song. This is an estimation of the total number of play-counts for the artist, irrespective of the song being played. If a song belongs to more than one artist, the artist with the maximum number of tweets associated with her, represents the song. • Artist-tweets: The tuned number of tweets, including retweets, which are related to the artist who sings this track. If a song belongs to more than one artist, the artist with the maximum number of tweets associated with her represents the song. • Artist-sentiment-analysis-1: The average value of the score, produced by VADER via sentiment analysis on all tweets associated with the artist of this song. If there are many artists, the choice is made similarly to the previous cases. • Artist-sentiment-analysis-2: This attribute is calculated like artist-sentiment-analysis-1 and the only difference is using SentiWordNet for score generation. • Position: The position of the song for that week. This is the class attribute to be predicted.

Table 1 shows type and value restrictions for each attribute.

Results and Discussion

This section presents and discusses results

Results

Table 2 presents the attributes in Pearson's correlation value descending order. Based on this output, the attributes can be divided in 3 categories with regards to their correlation (r) with the predicted class (position):

• Moderately correlated attributes (0.2 < r < 0.7): These are: chart-weeks, song -playcount and previous-position. • Weakly correlated attributes (0.1 < r < 0.2): Artist-play-count, artist-sentiment-analysis-2, artist-sentiment-analysis-1 and top-weeks are the attributes in this category. • Unrelated attributes (r~0): Artist-tweets is the only attribute in this category with its correlation coefficient close to 0. Regarding regression analysis, the following results were generated using 10-fold cross-validation. Support Vector Regression had the smallest Mean Absolute Error (MAE): 4.0515 and Random Forest the lowest Root Mean Square Error (RMSE): 8.8117. Bagging also achieved decent results with RMSE: 8.961 and the highest correlation coefficient: 0.5697.

Hit prediction refers to forecasting whether a song is going to be within a specific range of positions or not. This is a classification problem, as each instance should be matched to one of the two available classes. For this purpose, a new attribute should be created through Weka's preprocessing functionality. After experimenting with all available algorithms using 10-fold cross-validation, J48 and PART appeared to be the classifiers with the best performance in terms of both accuracy and F1 scores.

The best scoring algorithms in the case of top 5 hits are Filtered Classifier and Decision Table, achieving 90% accuracy and the same for precision, recall and F1 score. F1-scores was 0.875 for hits and 0.917 for non-hits. For the prediction of the top 20 hits of the following week, the classifiers that achieved high accuracy and F1 values were LMT and Simple Logistic.

Discussion

This study investigated the forecasting strength of social media, focusing on the prediction of the Billboard's Hot 100, based on data extracted from the chart and music-related Twitter posts referring to artists and songs on the top ten each week. In total, more than one million tweets were gathered, and data collection lasted for about 2 months, during October and November in 2018.

Twitter data appear to be quite representative when it comes to the number of total play-counts for a song. Considering the generally accepted ranges, there is a moderate correlation (0.2917) between the number of tweets that include the title of a song and the #nowplaying terms as a subset and the imminent success of the song on the Billboard Hot 100 chart for the following week. On the contrary, tweets that provide an estimation for an artist's total play-counts in general are not adequate to give an accurate picture of the future performance of a particular song, as the correlation coefficient was shown to be weak.

There appears no relation between an artist's publicity, positive or negative, expressed by the total number of tweet mentions, and his ranking on the chart, since the value of the correlation coefficient was close to 0 (0.0134). Furthermore, there is no evidence that the positive attention an artist gets, represented by a value that estimates how favorable the posts related to her are, is significantly correlated to the positioning of her tracks on the Billboard chart, at least in the short term. Findings concerning sentiment analysis cannot be generalized for other domains other than this particular chart, as many other studies have ascertained [START_REF] Asur | Predicting the Future with Social Media[END_REF], [START_REF] Oikonomou | A Method for Predicting the Winner of the USA Presidential Elections using Data Extracted from Twitter[END_REF].

Features derived from tweets combined with chart data, can provide results of noteworthy accuracy, but this happens only in the investigation of specific aspects of the problem. Regarding regression analysis, the best performance was achieved by Support Vector Regression with MAE 4.0515 and Random Forest with RMSE 8.8117. MAE value means that the position predictions were on average 4 ranks away from the actual values. Nevertheless, RMSE is, in most cases, considered to be more reliable, as it penalizes outliers. According to RMSE, the average derivation is approximately 8 positions.

Result evaluation depends on the type of problem that needs to be addressed. For example, if the goal is to predict all the Billboard chart for the following week, the values could be considered useful. On the other hand, if it is desired to predict a particular range of chart positions, especially if this is limited, such as top 10 hits for instance, then this model would not be able to provide an accurate prediction. The highest correlation coefficient was 0.5697, with Random Forest, and the squared correlation coefficient was approximately 0.325, which is low compared to the best result found in [START_REF] Yekyung | #nowplaying the Future Billboard: Mining Music Listening Behaviors of Twitter Users for Hit Song Prediction[END_REF], with a squared correlation of 0.57, using Support Vector Regression and 5-fold validation.

Hit prediction, implemented via classification, yielded some promising results: top 10 songs could be predicted with an accuracy of 85% utilizing J48 and PART. The F-scores for PART was 0.906 for hits and 0.625 for non-hits, while the same values for J48 were 0.909 and 0.571. In [START_REF] Yekyung | #nowplaying the Future Billboard: Mining Music Listening Behaviors of Twitter Users for Hit Song Prediction[END_REF], Random Forest with 5-fold cross-validation, achieved 90% accuracy and 0.901 and 0.899 F-scores for hit and non-hits, respectively.

Experimenting with different ranges for hits, specifically the top 5 and top 20, scores were higher than for the top 10 range. Filtered Classifier and Decision Table achieved an accuracy of 90% and F-scores of 0.875 for the top 5 hits and 0.917 for non-hits. For the prediction of the songs in the 20 highest positions Simple Logistic and LMT achieved accuracy of 96.25%. Specifically, the F-scores for hits and non-hits were 0.980 and 0.727, respectively. In [START_REF] Yekyung | #nowplaying the Future Billboard: Mining Music Listening Behaviors of Twitter Users for Hit Song Prediction[END_REF], the accuracy for the same range (with the Random Forest Classifier and a 5-fold validation) was 88.2% and the F-score for hit songs was 0.885. However, they achieved a better F-score for non-hits, which was 0.879.

Conclusions and Future Work

This study gathered data from two sources: titles, artist names and rankings from the Billboard Hot 100 chart and tweets related to songs in the top 10, in order to test how their integration could be used for forecasting future charts. Our findings suggest that there is a moderate correlation of 0.2917, between the number of mentions of a song and its chart performance. No significant relationship was observed between the attention an artist gets on Twitter, even via emotionally charged tweets, and their tracks' success.

Regarding regression analysis, the best score was a mean square error of 4.0515 and RMSE 8.8117. Both metrics show the average number of positions between the actual and predicted values, and their evaluation depends on the range of positions we want to predict. With the best scoring algorithm, hit prediction could be achieved with an accuracy of 80% and F-scores of 0.881 and 0.385 for hits and non-hits, for the 10 highest ranks. Similarly, for the top 5 hits these values were 90%, 0.917 and 0.875, respectively. Finally, for the top 20 songs, the results were competitive to previous research, achieving top accuracy of 96.25% and F-scores 0.980 for hits and 0.727 for non-hits.

Overall, the study has shown that mining Twitter data, extracting specific information and handling them properly can provide useful conclusions regarding the next Billboard chart, although it is not yet capable of perfect predictions.

Limitations and Future Research

This research gathers and examines Twitter and Billboard chart data for hit songs at the top 10. Tracks at lower ranks were not considered, thus it is impossible to predict their way up. So, future research could investigate the chart at a larger scale and consider all songs on the chart, and create a more comprehensive dataset, consisting of chart data for a period of one or more years, like the ones used in [START_REF] Yekyung | #nowplaying the Future Billboard: Mining Music Listening Behaviors of Twitter Users for Hit Song Prediction[END_REF] and [START_REF] Zangerle | Can Microblogs Predict Music Charts? An Analysis of the Relationship Between #Nowplaying Tweets and Music Charts[END_REF].

From a technical point of view, researchers are encouraged to try to capture tweets incorporating the #nowplaying term in general, without targeting specific songs or artists. Finding Twitter data referring to a set of 100 songs for each week would require a complicated methodology if the Search API is used, due to the search query limitations that were discussed in section 3.1. On the other hand, a more generalized approach would demand a lot of space for storing data and would capture a lot of irrelevant tweets that would need to be excluded at a later stage. Thus, there should also be extra preprocessing effort. Nevertheless, collecting tweets indiscriminately ought to give a better insight regarding airplay trends and could probably aid the discovery of songs that are not at the chart at the given moment, but would be introduced in the weeks to come.

Undoubtedly, an exhaustive study of the optimized way to predict the Billboard chart should rely on a model that considers all parameters that affect song ranking, such as streaming activity and song sales. During data gathering and experimentation, it was observed that many tracks made their debut into the chart by occupying one of the first 10 positions, consequently they could not have been monitored even if all 100 songs were accounted for. They also tended to follow a specific pattern: usually they dropped significantly after their successful entrance into the chart and their play-counts seemed to be quite low at that time. It is, therefore, speculated that their steep rise to the top is basically owing to music sales and streaming data. In order to get a better idea about this phenomenon, research should be extended beyond the limits of the Billboard chart and the airplay counts and consider the other dimensions that contribute to the formation of the chart each week.

With regards to sentiment analysis, the degree to which tweets constitute a trustworthy sample for determining the positive or negative disposition of the population towards an artist is still vague. There may be a relationship, but it may not be directly visible. This fact raises new questions about the predictive strength and limits of sentiment analysis in social media and is something worth investigating in the future. It is suggested that researchers explore the impact of sentiment analysis in a longer term context. This essentially means, monitoring sentiment orientation for an artist and track the performance of her songs for many weeks, possibly considering some time lag, as the public's response to a likeable or unlikeable person may be revealed gradually.

Different angles can also be investigated, for instance, the relationship between a positive or negative bias towards an artist and the number of tracks belonging to him that are on the chart at the present time or the rate at which these tracks ascend and descend in the chart. Researchers could, additionally, consider improving the mechanism of sentiment analysis and attempt to enhance existing lexicons in order to make them more suitable for music-related data. For instance, some terms referring to music, like radio, album, concert, listen, etc., which would otherwise be characterized as neutral, could infuse a positive tone into a tweet, when the name of an artist is included in the text.

Table 1 :

 1 Attribute type and values

	Attribute	Type	Range
	Previous-position	Integer	[0 , 100]
	Chart-weeks	Integer	>= 1
	Top-weeks	Integer	>= 0
	Song-play-count	Real	[0 , 100]
	Artist-play-count	Real	[0 , 100]
	Artist-tweets	Real	[0 , 100]
	Artist-sentiment-analysis-1	Real	[-1, 1]
	Artist-sentiment-analysis-2	Real	[-1, 1]
	Position	Integer	[0 , 100]

Table 2 :

 2 Pearson's correlation per attribute

	Attribute	Pearson's Correlation Coefficient
	Chart weeks	0.377
	Previous position	0.370
	Song playcount	0.292
	Artist playcount	0.147
	Artist sentiment analysis 2	0.134
	Artist sentiment analysis 1	0.133
	Top weeks	0.123
	Artist tweets	0.013


Dimosthenis Beleveslis

Christos Tjortjis
email: c.tjortjis@ihu.edu.gr

Promoting Diversity in Content Based Recommendation using Feature Weighting and LSH

Keywords: Content-Based (CB) Recommendation Systems (RS), eCommerce, Minhash, Locality-Sensitive Hashing (LSH), Decision Support

This work proposes an efficient Content-Based (CB) product recommendation methodology that promotes diversity. A heuristic CB approach incorporating feature weighting and Locality-Sensitive Hashing (LSH) is used, along with the TF-IDF method and functionality of tuning the importance of product features to adjust its logic to the needs of various e-commerce sites. The problem of efficiently producing recommendations, without compromising similarity, is addressed by approximating product similarities via the LSH technique. The methodology is evaluated on two sets with real e-commerce data. The evaluation of the proposed methodology shows that the produced recommendations can help customers to continue browsing a site by providing them with the necessary "next step". Finally, it is demonstrated that the methodology incorporates recommendation diversity which can be adjusted by tuning the appropriate feature weights.

Introduction

A Recommendation System (RS) refers to an intelligent system that produces suggestions about items to users. Its role is to predict items that are likely to be of interest to the user. They provide personalized information by learning the user's interests through his/her interaction with items. The way that these recommendations are produced depends on the specific domain and the desired results. In general, they are based on item similarities, user preferences, past purchases and actions related to items and users.

Recommended items can potentially be anything that a user is looking for, such as products, movies, songs, services etc. [START_REF] Gerogiannis | Personalised Fuzzy Recommendation for High Involvement Products[END_REF][START_REF] Nalmpantis | The 50/50 Recommender: A Method Incorporating Personality into Movie Recommender Systems[END_REF][START_REF] Schoinas | A Product Recommendation System Based on Multi-source Implicit Feedback[END_REF]. E-commerce is a domain that RS are used extensively [START_REF] Schafer | Recommender Systems in E-Commerce[END_REF][START_REF] Karimova | A Survey of e-Commerce Recommender Systems[END_REF]. Online stores need to offer customers a similar or even better shopping experience than that of an actual store.

A customer can see product recommendations in several parts of an e-commerce site including the 'product page', showcasing a specific product, along with its detailed description and features [START_REF] Han | Feature-based recommendation system[END_REF]. The user can add the product to their cart or visit an alternative product page that better matches their preferences. The aim of the system is to display relevant items on such a page and help customers to continue browsing the site by providing them with the necessary "next step". The recommended products must be similar to the target product, by considering the importance of their features. However, it is important to achieve diversity in recommendations because customers do not need a list of almost identical products. For example, when a user views a laptop of a specific brand and receives recommendations corresponding only to similar laptops of the same brand. In this case, the user will not be able to visit the product page of a laptop of an alternative brand by navigating through the recommendations. Furthermore, the logic of such a RS relies on the idea of finding similar products by calculating similarities among them [START_REF] Charikar | Similarity estimation techniques from rounding algorithms[END_REF]. This is computationally heavy and time-consuming, for sites with thousands of products. Calculating the similarity between a product and other products, in order to find the most similar ones, is usually inefficient. So, it is important to apply more sophisticated methods that reduce the complexity and computational cost without compromising accuracy. The scope of this research is to implement a Content-Based (CB) RS suitable to the 'product page' of an e-commerce site [START_REF] Lops | Content-based Recommender Systems: State of the Art and Trends[END_REF]. Specifically, we address the problem of product representation by incorporating a weighted method that offers the functionality to customize the importance of product features. In that way, the logic of the methodology is adjusted to the needs of the site. In addition, recommendation diversity is accomplished without compromising similarity. Furthermore, this work addresses the problem of efficient product similarity calculations for sites that dispose thousands of products. This problem is tackled by incorporating a Locality-Sensitive Hashing (LSH) method [START_REF] Indyk | Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality[END_REF][START_REF] Gionis | Similarity Search in High Dimensions via Hashing[END_REF]. Finally, our implemented heuristic approach is being evaluated based on real data from two e-commerce sites. The remainder of the paper discusses related work and the data used in sections 2 and 3 respectively. The proposed methodology are detailed in section 4. Section 5 presents results and evaluation. Section 6 concludes the paper. [START_REF] Barranco | A Method for Weighting Multi-valued Features in Content-Based Filtering[END_REF]. They argued that a user considers some features as more important than others. This represents an implicit feature weighting, which can be subjective. The weight of each feature is computed according to: (i) the entropy or amount of information provided (the more entropy the higher the weight ), and (ii) the correlation between items chosen by the user in the past and the values of some features of the set of items. K. Bradley argued that recommendation diversity is important, and that traditional CB RS suffer from potentially poor diversity [START_REF] Bradley | Improving Recommendation Diversity[END_REF]. He also questioned the blind faith in the similarity assumption. A diversity preserving similarity-based retrieval algorithm is proposed. Multiple strategies for retrieving k items from a collection, each focusing on a different way of increasing diversity is presented. Finally, he concluded that an over emphasis on diversity can result in a corresponding drop in similarity.

Related Work

Data

Two datasets were used to evaluate the proposed RS. The first one was created by scraping a real e-commerce site and comprises information for thousands of products.

The second one was created by preprocessing an existing dataset that has been used for similar purposes in the past [START_REF] Schoinas | A Product Recommendation System Based on Multi-source Implicit Feedback[END_REF]. These datasets are described next.

Bestprice dataset

Bestprice is a new dataset created for the purposes of this research. Data were gathered by scraping BestPrice.gr, a commercial site, where a customer can compare product prices across numerous e-shops [START_REF]Bestprice Homepage[END_REF]. A huge variety of product categories and subcategories is available. We used a subset consisting of data regarding 29.541 products that belong to 6 main technology categories, each comprising several subcategories, 22 in total. Additionally, there are 515 different product brands. A product title is also available for each item, along with a set of 10 product recommendations for each product, provided by the site in each product page, which were used for evaluation.

The dataset was preprocessed in order to assign a textual representation to each product. This text holds information that characterizes each product and is produced by concatenating its title, category, subcategory and brand. Each of these features were preprocessed before being added to the final text. First, the textual data of each feature were transformed to lowercase. Then categories, subcategories and brands were discriminated from other words by adding the suffixes '_cat', '_subcat' and '_brand' respectively. This helps to assign different weights to these specific words. Specific symbols that do not add any value were removed from the title. Finally, a string that consists of the above textual features was created for each product. This set of strings was used to create the TF-IDF matrix [START_REF] Ramos | Using TF-IDF to determine word relevance in document queries[END_REF]. The information available and the final textual representation for a product is presented in table 1.

Retailrocket dataset

Retailrocket dataset is a dataset published by Retail Rocket that has been used in RS [START_REF]Retailrocket dataset[END_REF]. The data were collected from a real-world e-commerce website. We selected a subset of 28.241 products that belong to 6 main categories and 37 subcategories to produce a dataset with customer sessions. A session refers to a group of user interactions with an e-shop that take place within a given time frame. For example, a session may contain the information that a customer first viewed several products, added some to their cart and finally purchased them. This session-based dataset was used to determine the product pages that users visit after viewing each product. An important aspect we considered was the number of consecutive events related with each other. We call this window size. Window size equal to 1 means that a product that a user viewed is linked only to the product that they visited next. Respectively, window size equal to 2 means that a product that a user viewed is linked to the next two products they visited. Furthermore, a representative text, that consists of the title, category and subcategory, has been assigned to each product. This set of texts was used in order to calculate the TF-IDF matrix.

Methodology

The heuristic approach for the proposed CB RS is based on feature weighting and LSH. The methodology consists of three parts. The first part refers to the method that is used in order to represent the set of products as a weighted matrix. The second is the weighted Minhash method that is used to create a compressed representation of each product by its Minhash signature and to approximate the Jaccard similarity [START_REF] Bawa | LSH forest: Self-tuning indexes for similarity search[END_REF] of two sets. The last part is the efficient production of recommendations based on LSH. The implementation of the methodology based on the Bestprice dataset is presented in the following three subsections and in Figure 1.

Fig. 1. Methodology diagram

Weighted method

The first part involves creating a TF-IDF matrix for the set of 29.541 products. Typically, different features carry different amount of information. A word in a product title may be more informative than the rest, and this must be considered when calculating similarity. In our case, each product is represented by a text that is the concatenation of the product title and three specific features, as described in section 3.1. The terms that correspond to the three specific product features are initially considered to be simple terms. These features are the category, subcategory and brand of each product. The total number of products is 29.541 and the respective corpus consists of 35.405 unique terms.

The target is to create a matrix that represents how important each term is for each product. A weight is assigned to each unique term for each product text. This weight corresponds to the significance of the term for each specific product. The more texts a term appears in, the less informative it is considered to be; thus it gets a smaller weight.

Respectively, the significance of a term in a product text increases with the occurrences of it in that specific text. In particular, the TF-IDF matrix [START_REF] Ramos | Using TF-IDF to determine word relevance in document queries[END_REF] has been created by using the corresponding functions of the 'scikit-learn' python library [START_REF]scikit-learn, Machine Learning in Python, scikit-learn[END_REF]. Furthermore, specific terms of the corpus have been given extra weight. These terms correspond to the three product features that are parts of each product text. The target is to have a functionality with which we can easily adjust the importance of each feature in the calculation of product similarities and enhance recommendation diversity, without compromising similarity or efficiency. Assuming a products' textual representation that consists of N terms and that n of them correspond to specific product features, we assign a weight wi (w1,w2,...,wN) to each of the N terms by applying the TF-IDF method. Then we adjust the importance of those n product features by multiplying their weights by small constants, while the rest of the weights remain the same. By multiplying a feature's weight with a constant greater than 1, this is considered as more important during the calculation of product similarities. The multiple scenarios that were implemented are presented in section 5.1.

Weighted Minhash

The second part of the methodology aims to create a compressed representation for each product by applying the Minhash approach. Specifically, a method that incorporates real number weights has been used since the set of products is represented by a TF-IDF matrix. The weighted Minhash algorithm [START_REF] Wu | A Review for Weighted Minhash Algorithms[END_REF][START_REF] Wu | Consistent Weighted Sampling Made More Practical[END_REF][START_REF] Manasse | Consistent weighted sampling[END_REF][START_REF] Ioffe | Improved Consistent Sampling, Weighted Minhash and L1 Sketching[END_REF] that is available in 'datasketch' python library [20] is used. In practice, a Minhash signature has been created for each product based on the corresponding TF-IDF array. Hence, each product is represented by a much smaller array than before. Specifically, while the TF-IDF array consists of 35.405 elements, the Minhash signature consists of only 128 elements. The length of the signature corresponds to the 'sample_size' parameter that can be adjusted accordingly as by increasing the number of samples, a better accuracy is accomplished, at the expense of slower speed.

LSH Forest and Recommendations

In the third phase, the methodology uses the Minhash signatures in order to approximate the Jaccard similarity between products by applying the LSH approach. In particular, the LSH approach is used in order to find the most similar products to each product based on their Minhash signatures. In particular, we search for the k-top similar products that correspond to k recommendations. For this reason, a variation of LSH that is known as LSH Forest [START_REF] Bawa | LSH forest: Self-tuning indexes for similarity search[END_REF] is used. Minhash LSH Forest, uses the Minhash representation of the query product and returns the k-top matching products that have the approximately highest Jaccard similarities with the query product. Hence, it is not necessary to pre-define a specific threshold for the Jaccard similarity score. In this way, we produced k recommendations for each of the 29.541 products in the Bestprice dataset.

Evaluation

The evaluation of the RS consists of two parts. The first part is based on the Bestprice dataset that was created by scraping a real e-commerce site and is presented in paragraph 3.1. The aim is to compare the recommendations of our methodology with those that were available on the site, concerning certain aspects. Specifically, we compare the diversity of the recommendations and we examine whether we can achieve different results by tuning the weights of the product features. The second part is based on the Retailrocket dataset that consists of real sessions as described in paragraph 3.2. The aim is to examine whether the recommendations that the proposed methodology produces would help users navigate an e-shop and prevent them from leaving the site without finding products that meet their preferences.

Recommendation diversity

In this part of the evaluation process, the recommendation diversity of our methodology is examined. We evaluate the recommendations that our methodology produced for the Bestprice dataset. Finally, we compare the recommendations that our methodology produced with those that were scraped from the respective site. The dataset is presented in section 3.1. A subset of 1.182 products has been used in the evaluation process.

Having applied the methodology that is presented in detail in section 4, we have produced a set of ten recommendations for each product page. This number matches the number of recommendations that are available for each product in the Bestprice dataset.

The experiment was conducted for 11 scenarios that are presented in table 2. In each scenario we consider the importance of the product features (brand, category and subcategory) to be different, by tuning the corresponding weights (W_brand, W_cat and W_subcat). The average number of different brands, categories and subcategories in those sets of 10 recommendations is presented in table 2.

In the first 3 scenarios, we see that the number of different brands decreases by increasing the corresponding weight. This means that considering the product brand as more important in the similarity calculations, results in recommendations that include more of the same brand for each product page. Respectively, in scenarios 4 and 5, the number of different subcategories decreases by the increase of the corresponding weight. The same happens for the product category feature in scenarios 6 and 7. Furthermore, in the rest of the scenarios, the weights of different combinations of the product features are tuned. The characteristics of the recommended products are affected by the weights that are assigned to each product feature. In general, by observing table 2, we see that the methodology incorporates diversity as it does not recommend products that are almost the same with each other. Specifically, products from more than one brand are present in each set of 10 recommendations. Also, by assigning the appropriate weights, we have product recommendations from more than one subcategory. There are also cases in which the recommended products belong to more than one category. The recommendation diversity is very important for the quality of the system [START_REF] Bradley | Improving Recommendation Diversity[END_REF]. In CB RS, diversity can be as important as similarity [START_REF] Smyth | Similarity vs. Diversity[END_REF]. Similarity assures that the recommended products are similar to the target product. Diversity means that the recommended products are not very similar to each other. The importance of the recommendation diversity can be explained through the following example. Assume that the target product is a Dell laptop and the system recommends 10 different Dell laptops. This might probably mean that the recommended products are very similar to the target product. However, the user will not have the option to move to a laptop of a different brand through the recommendations. Similar problems will occur in a case when all the recommended products are of the same subcategory or category.

In the last row of table 2, we see the respective statistics in the product recommendations that were scraped from the site (BestPrice.gr). It is obvious that there is diversity in the recommendations concerning the brand feature. There is an average of more than 5 different brands in each set of 10 product recommendations. However, there is no diversity in the cases of subcategory and category. The system recommends products that belong only to the category and subcategory of the target product. As a result, a user does not have the option to move to a product of a similar subcategory through clicking one of the recommendations.

Session based

By applying our methodology, we produced a set of ten recommendations for each product page in the Retailrocket dataset. The aim is to determine whether the recommendations would help customers navigate through the e-commerce site, by offering them the required next step in order to move from a product page to another. For this purpose, we compare the recommendations with the product pages that users visited after viewing each single product. The test was conducted for multiple scenarios in which the importance of specific product features was tuned. These product features are the product category and subcategory. In addition, each scenario was tested for a window size up to 3. Table 3 presents the percentage of cases that at least one recommended product matches with one product view, for the scenarios where subcategory weights were tuned. It turns out that the percentage of cases that at least one recommended product matches the real product views is between 94% and 97%, regardless the feature importance. The percentage increases with the increase of the window size. Similar results were accomplished by tuning the category weights. Hence, in most of the cases, the recommendations that our methodology produces would potentially help users move to a similar product page by clicking one of the recommended products.

Conclusions

In this paper we proposed a heuristic approach for CB RS based on feature weighting and LSH with the aim to promote recommendation diversity and make similarity calculations efficient. The proposed methodology creates a textual representation for each item based on its features. This is used in order to create a weighted representation of the set of items based on TF-IDF scheme. The importance of specific features is adjusted according to our needs by tuning the corresponding weights. Furthermore, the methodology incorporates the weighted Minhash algorithm in order to create a compressed representation of each item. Finally, the Minhash signatures are used by LSH forest in order to make the similarity calculations efficient and produce the recommendations.

Based on the evaluation of our proposed RS and the results that were produced by conducting several experiments, interesting conclusions were drawn. Regarding the recommendation diversity, results show that the characteristics of the produced recommendations can easily be adjusted to the desired results by tuning the product feature weights appropriately. The methodology incorporates recommendation diversity without compromising similarity. This aspect turns out to be very important because most of the CB approaches lack diversity [START_REF] Bradley | Improving Recommendation Diversity[END_REF]. Considering the product brand as less important in the similarity calculations, results in recommendations that include products of different brands. Similar results can be achieved by tuning other product features. In that way, customers see recommendations that are similar to the target product, but not very similar to each other. Furthermore, our approach incorporates a hashing method that makes the product similarity calculations much faster and efficient than traditional systems. The representation of product profiles as compressed signatures, by applying the Minhash method, turns out to be effective. Finally, the approximation of product similarities by applying the LSH method results in a system that can handle thousands of products efficiently without compromising similarity. In that way, our proposed approach outperforms against others, as it combines recommendation diversity with calculation efficiency.

Regarding the quality of the produced recommendations, results show that in a very high percentage (94-97%) of cases, at least one recommended product matches with one product view that was generated based on real customer actions. Hence, our proposed approach was shown to offer recommendations that could potentially help users move to a similar product page instead of leaving the e-commerce site.

Table 1 .

 1 Product Details for Bestprice dataset.

	Title	Category	Subcategory	Brand	Text
	Omega	Laptop_pc Laptop_bases Omega omega_brand ice box laptop_pc_cat
	Ice Box				laptop_bases_subcat

Table 2 .

 2 Average # of brands, categories, subcategories in the sets of 10 recommendations

		W_brand W_cat W_subcat	Brands	Categories Subcategories
	Scenario1	0.5	1	1	3.07	1.16	1.30
	Scenario 2	1	1	1	2.54	1.16	1.30
	Scenario 3	1.5	1	1	2.16	1.16	1.31
	Scenario 4	1	1	0.5	2.47	1.21	1.40
	Scenario 5	1	1	1.5	2.58	1.14	1.25
	Scenario 6	1	0.5	1	2.51	1.20	1.33
	Scenario 7	1	1.5	1	2.63	1.12	1.29
	Scenario 8	0.5	0.5	1	2.94	1.20	1.40
	Scenario 9	1.5	1.5	1	2.19	1.14	1.26
	Scenario 10	0.5	1	0.5	3.01	1.20	1.32
	Scenario 11

Table 3 .

 3 Percentage of cases that at least one recommended product matches with views.

	Window		Subcategory Weight
	Size	0.5	1	1.5	2.0
	1	94.74 % 94.79 % 94.84 %	94.86 %
	2	96.90 % 97.01 % 96.98 %	97.03 %
	3	97.68 % 97.73 % 97.68 %	97.73 %


Foteini Filippidou

Lefteris Moussiades

Α Benchmarking of IBM, Google and Wit Automatic Speech Recognition Systems

Keywords: Automatic Speech recognition, WER, Hper, Rper 1

As the requirements for automatic speech recognition are continually increasing, the demand for accuracy and efficiency is also of particular interest. In this paper, we present most of the well-known Automated Speech Recognition systems(ASR), and we benchmark three of them, namely the IBM Watson, Google, and Wit, using the WER, Hper, and Rper error metrics. The experimental results show that Google's automatic speech recognition performs better among the three systems. We intend to extend the benchmarking both to include most of the available Automated Speech Recognition systems and increase our test data.

Introduction

Speech is probably the primary means of man communication. Therefore, acquiring speech from computers is reasonable to contribute to more effective man-machine communication. Two primary technologies have developed concerning speech: the Automatic Speech Recognition or ASR, for short, and the Text to Speech or TTS. ASR refers to the conversion of speech into text [START_REF] Anusuya | Speech Recognition by Machine, A Review[END_REF], while TTS, as its name suggests, is the reverse process [START_REF] Sharma | A Speech Recognition and Synthesis Tool : Assistive Technology for Physically Disabled Persons[END_REF]. Although man naturally acquires speech in early growth [START_REF]Using Statistical Methods in a Speech Recognition System for Romanian Language -ScienceDirect[END_REF], speech production and recognition by computers is a complicated process that has extensively been addressed by the research community.

As early as 1952, the first system was built that could identify digits with high precision. This early system required users to pause after each digit [START_REF]Automatic Recognition of Spoken Digits[END_REF]. Raj Reddy constructed the first recognition system of continuous speech as a student at Stanford University in the late 1960s [Wikipedia -Speech Recognition]. Shortly after, T.K. Vintsyuk presented an algorithm [START_REF]Applications and potential of speech recognition[END_REF] that can recognize speech, creating a sequence of words that contained in continuous and connected speech. Later, in 1981, Logica developed a real-time speech recognition system based on the original project of the Joint Speech Research Unit in the United Kingdom. This userfriendly system is one of the first steps in improving human-machine communication [START_REF]Britain scores in easier man-machine communication[END_REF]. Today, ASR applications are not just confined to human-machine communication for personal use but include industrial machine guidance with voice commands [START_REF] Upton | Speech recognition for computers in industry[END_REF] [START_REF]Applications and potential of speech recognition[END_REF], automatic telephone communication [START_REF]B2 -Speech recognition system and method -Google Patents[END_REF], communication with automotive systems, military vehicles, and other equipment, communication with health care, aerospace and other systems [START_REF] Alyousefi | Digital Automatic Speech Recognition using Kaldi[END_REF]. Also, ASR systems are utilized to provide more specialized services, such as training in pronunciation and vocabulary [START_REF] Srikanth | Automatic Pronunciation Scoring And Mispronunciation Detection Using CMUSphinx[END_REF].

As the number of ASR systems is continuously increasing, it is quite challenging to select the most appropriate for a particular application. Our immediate plans are to build an artificial vocabulary learning assistant. In particular, this assistant will be able to contribute to the learning of vocabulary by entering into dialogues with the trainee. In this context, it is necessary to select the appropriate ASR. Thus, this work first attempts a presentation of known ASR systems and then proceeds to benchmark three wellknown systems, namely IBM Watson, Google, and Wit. Therefore, we set the foundations for a more general assessment of most ASR systems with the ultimate goal of choosing the most appropriate vocabulary learning assistant.

The rest of this review is organized as follows. In section two, we briefly present analyze the Architecture of ASR systems. In section three, some of the most wellknown ASR systems are introduced. In section four, we describe criteria and metrics used for system evaluation. Next, in section five, we analyze the data and the methodology we use for the experimental procedure. Finally, section six closes the review with final remarks and conclusions.

Architecture

An ASR application accepts the speech signal as input and converts it into a series of words in text form. During the speech recognition process, a list of possible texts is created, and finally, the most relevant text to the original sound signal is selected [4] [START_REF]Britain scores in easier man-machine communication[END_REF]. A typical ASR consists of an acoustic front-end that process the speech signal to extract useful features [START_REF] Karpagavalli | A Review on Automatic Speech Recognition Architecture and Approaches[END_REF]. Then, a feature vector is generated. Several feature extraction methods are used including, Principle Component Analysis (PCA), Linear Discriminant Analysis (LDA), Independent Component Analysis (ICA), Linear Predictive Coding (LPC), Cepstral Analysis, Mel-Frequency Scale Analysis, Filter-Bank Analysis, Mel-Frequency Cepstrum Coefficients (MFCC), Kernal Based Feature Extraction, Dynamic Feature Extraction, Wavelet-based features, Spectral Subtraction and Cepstral Mean Subtraction (CMS). Generally, some spectra temporal analysis of the signal generates features that usually transform into more compact and robust vectors [START_REF] Karpagavalli | A Review on Automatic Speech Recognition Architecture and Approaches[END_REF]. At the processing step, the acoustic lexicon and language model are used by a decoder (search algorithm) to produce the hypothesized word or phoneme, as shown in Fig. 1. The acoustic model contains acoustic features for each of the distinct phonetic units and typically refers to the process of establishing the statistical representations for the feature vector sequences computed from the speech waveform. The Hidden Markov Model (HMM) is one of the most commonly used to build acoustic models. Other acoustic models include segmental and super segmental models, neural networks, maximum entropy models, and conditional random fields. An acoustic model is a file that contains statistical representations of each of the distinct sounds that makes up a word [START_REF] Karpagavalli | A Review on Automatic Speech Recognition Architecture and Approaches[END_REF]. The lexicon includes terms of the vocabulary of the current application [START_REF] Karpagavalli | A Review on Automatic Speech Recognition Architecture and Approaches[END_REF]. The language model consists of the limitations associated with the sequence of words that are accepted in a given language. Two popular tools for language modelling are the CMU Statistical Language Modeling (SLM) Toolkit and the Stanford Research Institute Language Modeling Toolkit [START_REF] Karpagavalli | A Review on Automatic Speech Recognition Architecture and Approaches[END_REF]. The decoder aims to find the most likely sequences of words that could match the audio signal by applying appropriate models. Most decoding algorithms produce a list of possible word sequences called the n-best list [4] [START_REF]Britain scores in easier man-machine communication[END_REF]. 3

Well-Known ASR Systems

Many international organizations, such as Microsoft SAPI, Dragon-Natural-Speech, and Google, as well as research teams, are working in the area of ASR systems [START_REF]Using Statistical Methods in a Speech Recognition System for Romanian Language -ScienceDirect[END_REF].

Next, we describe in short some of the most well-known ASR systems. CMU-Sphinx was introduced in 1986 and is a set of open-source speech recognition libraries. It supports English but not Greek. Contains a series of packages for different tasks and applications:

Pocketsphinx: Lightweight library with C for written recognition [START_REF] Shmyrev | CMUSphinx Open Source Speech Recognition[END_REF]. It is fast, runs locally, and requires relatively modest computational resources. It provides nbest lists and lattices and supports incremental output. It provides voice activity detection functionality for continuous ASR and is also fully customizable and trainable [START_REF] Morbini | Which ASR should I choose for my dialogue system?[END_REF].

Sphinxbase: support for the libraries required by Pocketsphinx [START_REF] Shmyrev | CMUSphinx Open Source Speech Recognition[END_REF]. CMUclmtk: language model tools [START_REF] Shmyrev | CMUSphinx Open Source Speech Recognition[END_REF]. Sphinxtrain: acoustic model tool. It includes a set of programs and documenta-tion for making and building audio models for any language [START_REF] Shmyrev | CMUSphinx Open Source Speech Recognition[END_REF].

Sphinx3: voice recognition decoder is written in C [START_REF] Shmyrev | CMUSphinx Open Source Speech Recognition[END_REF]. Sphinx4: the latest Java-based speech recognition tool made by Carnegie Mellon University (CMU), Sun Microsystems Laboratories, and Mitsubishi Elec-tric Research Laboratories (MERL). It is flexible, supporting all types of acoustic models based on the Hidden Markov model (HMM), all basic types of language models, and multiple search strategies [START_REF] Lamere | The CMU Sphinx-4 Speech Recognition System[END_REF].

CMU also attempted to develop an Amazigh speech recognition system used in a vast geographical area of North Africa [START_REF] Satori | Investigation Amazigh speech recognition using CMU tools[END_REF].

Kaldi is also an open-source speech recognition tool, written in C ++. Kaldi's goal is to have a modern and flexible code that is easy to understand, modify, and expand. Kaldi supports conventional models (GMMs) and Subspace Gaussian Mixture (SGMMs) but can be easily extended to new types of models [START_REF] Povey | The Kaldi Speech Recognition Toolkit[END_REF].

Google uses closed-source speech recognition technology based on deep learning achieving an error rate of 8% in 2015, down by more than 23% from 2013.

Dragon Naturally Speaking launched in 1975 by Dr James Baker supports ASR and TTS [START_REF]Dragon NaturallySpeaking[END_REF].

Microsoft Speech API (SAPI) is a closed-source library developed that supports ASR and TTS within Windows applications. SAPI 5 is the latest major version released in 2000 [START_REF] Sharma | A Speech Recognition and Synthesis Tool : Assistive Technology for Physically Disabled Persons[END_REF].

DragonDictate (TM)-30K an ASR containing an extensive vocabulary that allows interactive users to create printed text faster through speech than they would have done by hand [START_REF] Baker | DRAGONDICTATETM-30K: NATURAL LANGUAGE SPEECH RECOGNITION WITH 30,000 WORDS[END_REF].

Amazon Transcribe is an ASR service used in many typical applications, including phone customer service systems, and automatic subtitling for audio and video. Amazon Transcribe is continuously learning and improving to keep up with the development of language [START_REF]Amazon Transcribe -Automatic Speech Recognition -AWS[END_REF].

Microsoft Azure provided as a service, converts speech to text in real-time. It includes an API Speech SDK and a REST API. The latter can be used only for queries containing up to 10 seconds recorded audio [START_REF] Erhopf | Speech-to-text with Azure Speech Services -Azure Cognitive Services[END_REF].

Wit is free even for commercial use. It supports over 130 languages, and it applies the applicable EU data protection laws, including GDPR. Wit supports Node, Python, and Ruby programming languages [START_REF][END_REF].

Twilio includes vocabulary to support the industry. It recognizes 119 languages and dialects. It is paid but also has a free trial version [START_REF]Speech Recognition API -Twilio[END_REF].

Houndify is a platform that allows anyone to create intelligent communication interfaces with voice capability. It supports both ASR and TTS on Android, iOS, C ++, Web, Python, Java, and C Sharp. It is available for a fee but also has a free version [START_REF]Houndify | Add voice enabled, conversational interface to anything[END_REF].

IBM® (Speech to Text) allows users to add up to 90 thousand out-of-vocabulary (OOV) words to a custom language model. The service utilizes machine learning to combine the knowledge of grammar and language structure as well as the synthesis of sound and voice signals to transform the human voice accurately. It continually renews and improves its function as it receives speech. It can support continuous speech. Programming languages that supports are Node, Java ™, Python, Ruby, Swift, and Go SDKs [25]. The service offers three interfaces: A WebSocket and synchronous HTTP interface, each being able to transfer up to 100 MB of audio data with one request. It also offers an asynchronous HTTP interface, which with one request can pass up to 1 GB of audio data [25].

AT&T Watson, created in 1995 by AT & T Research [START_REF] Sharp | The Watson speech recognition engine[END_REF]. It allows users to create real-time multilingual voice and speech recognition applications [START_REF] Goffin | The AT&T WATSON Speech Recognizer[END_REF]. It is a cloudbased service that can be accessed through HTML POST requests.

Web Speech API, developed by the W3C Speech API team, it is open but seems to be driven by two companies: Google and Openstream. To upload the audio data, the user needs to open an HTTPS connection with the network service through a POST request [START_REF] Adorf | Web Speech API[END_REF].

NaturalReader. It can read any text, such as Microsoft Word files, Web pages, PDFs, and emails, and convert them into audio files. It is available in free and paid versions [29].

iSpeech is an online free solution available for mobile applications. It has a unique multithreaded and multicore method for text-to-speech conversion, which allows for a simple text-to-speech approach with an unlimited number of processors simultaneously. Thus, it is speedy. It also provides an SDK for java applications [START_REF]Text to Speech | TTS SDK | Speech Recognition (ASR)[END_REF].

WaveSurfer is an open-source model made in KTH's Center for Speech Technology. An essential feature of WaveSurfer is its versatile configuration [START_REF] Sjölander | WAVESURFER -AN OPEN SOURCE SPEECH TOOL[END_REF].

Julius is open-source written in C with high performance. It incorporates significant cutting-edge speech recognition techniques, achieving LVCSR, real-time vocabulary, and real-time speech recognition. It works on Linux, Windows, Mac OS X, Solaris, and other Unix variants and also runs on the Apple iPhone. It supports several languages [START_REF] Lee | Recent Development of Open-Source Speech Recognition Engine Julius[END_REF].

Evaluation Metrics

There are three types of errors that occur in speech recognition [START_REF] Errattahi | Automatic Speech Recognition Errors Detection and Correction: A Review[END_REF]: If a word in the reference sequence is transcribed as a different word, it is called substitution. When a word is completely missing in the automatic transcription, it is characterized as deletion. The appearance of a word in the transcription that has no correspondent in the reference word sequence is called insertion.

The performance accuracy of a system is usually rated by the word error rate (WER) (1), a popular ASR comparison index [START_REF] Gaikwad | A Review on Speech Recognition Technique[END_REF]. It expresses the distance between the word sequence that produces an ASR and the reference series. Defined as: WER=(S+D+I)/N1=(S+D+I)/(H+S+D) [START_REF] Anusuya | Speech Recognition by Machine, A Review[END_REF] where I = the total number of entries, D = total number of deletions, S = total number of replacements, H = total number of successes, and N1 = total number of reference words [START_REF] Errattahi | Automatic Speech Recognition Errors Detection and Correction: A Review[END_REF]. Despite being the most commonly used, WER has some cons. It is not an actual percentage because it has no upper bound. When S=D=0 and we have two insertions for each input word, then I=N1 (namely when the length of the results is higher than the number of words in the prompt), which means WER=200%. Therefore it does not tell how good a system is, but only that one is better than another. Moreover, in noisy conditions, WER could exceed 100%, as it gives far more weight to insertions than to deletions [START_REF] Errattahi | Automatic Speech Recognition Errors Detection and Correction: A Review[END_REF].

Match Error Rate (MER) is the proportion of I/O word matches, which are errors, which means that is the probability of a given match being incorrect. The ranking behaviour of MER ( 2) is between that of WER and WIL [START_REF] Morris | From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition[END_REF].

MER=(S+D+I)/(N=H+S+D+I) =1-H/N (2)
WIL is a simple approximation to the proportion of word information lost, which overcomes the problems associated with the RIL (relevant information lost) measure that was proposed years ago [START_REF] Morris | From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition[END_REF]. The Relative Information Lost (RIL) measure was not widely taken up because it is not as simple as WER, and it measures zero error for any one-one mapping between input and output words, which is unacceptable [START_REF] Morris | From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition[END_REF].

The evaluation of an ASR system also correlates with PER (Position-Independent word Error Rate) automatic metric. PER compares the words in the two sentences (hypothesis and reference) and is always lower than or equal to WER. Hypothesis per (Hper) (similar to precision) (3) refers to the set of words in a hypothesis sentence which do not appear in the reference sentence. Reference PER (Rper) (similar to recall) (4), denotes the set of words in a reference sentence which do not appear in the hypothesis sentence. In other words, the main goal of Hper and Rper is to identify all words in the hypothesis which do not have a counterpart in the reference and vice versa [START_REF] Seljan | Combined Automatic Speech Recognition and Machine Translation in Business Correspondence Domain for English-Croatian[END_REF].

WER = S+D+I N 1 = S+D+I H+S+D (3) MER = S+D+I N=H+S+D+I = 1 - H N (4)

Experimentation

Data and Methodology

From the tools described above, IBM Watson, Wit, and Google were selected to be tested and compared to the accuracy of the results they produce. Three speakers recorded specific sentences in English. The recorded sentences are typical in dialogues between a teacher and a student while learning English as a second language. The used sentences contain words, numbers, and names.

Speakers are not native English speakers, and their native language is Greek. Speakers A and B are women, and speaker C is a man. We intend to evaluate ASR systems for an application that will train the student in vocabulary learning. Recording took place in an ideal environment with no noise except for speaker C, whose recording has a low level of noise. Each of the three speakers recorded twenty sentences.

Experimental measurements were made using audio files, recorded with Audacity 2.3.1. The WAV files were stereo with two recording channels, 44100Hz, 32 bit, and they range in size from about 350KB to 0.99MB. Generated texts were compared to the corresponding correct texts that had been given to the speakers. The Word error rate (WER), Hper, and Rper were measured using an Open Source Tool for Automatic Error Classification of Machine Translation Output available on GitHub [START_REF] Madl | Hjerson: An Open Source Tool for Automatic Error Classification of Machine Translation Output: cidermole/hjerson[END_REF].

Results

The text files were created with Uberi, available at https://github.com/Uberi/speech_recognition. Uberi uses the Python Speech Recognition library and supports several ASR engines.

Comparing the resulting texts with the reference texts, we calculated the WER error measurements as a percentage with an accuracy of two decimal places. WER measurements were made using the tools that are available on pages https://github.com/belambert/asr-evaluation and https://github.com/cidermole/hjerson. In the end, the average error of each tool per speaker was calculated and is presented in Table 1. Taking into account the three error values, we calculate the average error for each system. The overall mean error was 38.1% for ΙΒΜ, 20.63% for Google, and 36% for Wit. We observe that among the three, the smallest average error is reported by the Google tool (20.63%). Based on these measurements, it seems that Google's system responds better regardless of the speaker's type.

We then recorded how many phrases out of a total of 60 (20 sentences for each of the three speakers that each system processed) were fully recognized, i.e., in how many cases the error was 0. In Table 2, we present the total percent of correct transcribes per As we see in Table 2, Google presents a higher percentage of accurate forecasts (43.33%) than the other systems. Wit tool is the second in the rankings and the IBM the last. These measurements reveal the ability of ASR systems to recognize a sentence without any errors. This ability is essential if the ASR system is to be used in teaching English as a foreign language, as the application we intend to build. Our app must be able to fully recognize the pupil's speech to teach him the correct pronunciation, grammar, and pension, so we do not want to have an approximation but an accurate recognition.

To get a better and more accurate estimate of the three systems, we measured the Hper και Rper errors using the available tool on https://github.com/cidermole/hjerson. In Table 3, we present the results of these measurements. Observing the values in Table 4, we find out that Google's ASR shows the lower average error for both metrics (Rper and Hper).

Conclusions

This paper presents some of the most widely known speech recognition tools currently in use. Three of them were then selected to study and compare their performance. The tools selected are IBM Watson, Google API, and Wit. All three tools were tested on the same audio data files and not in a continuous speech stream. The audio files involved texts that occur very often when teaching English as a foreign language. The WER, Hper, and Rper error metrics were measured to evaluate the tools. The results showed that the Google tool is superior to the other two. The Google tool was able to predict most of the sentences accurately, and it presented the smallest error percent. In the future, the size of the data set could be increased by adding more sentences to increase the reliability of the measurements and conclusions. Other tools could also be tested to allow us to decide which one is most suitable for our application.

Fig. 1 .

 1 Fig.1. Speech Recognition Architecture[START_REF] Karpagavalli | A Review on Automatic Speech Recognition Architecture and Approaches[END_REF]

Table 1 .

 1 Average WER (%) measurements

		Speaker A			Speaker Β			Speaker C
	IBM	Google	Wit	IBM	Google	Wit	IBM	Google	Wit
	30.10 16.60	25.87 47.73	20.45	23.28	36.51 24.85	58.87
	According to Table I, we note that Google has the smallest average error in any case.
	Specifically, Google system presents WER error 16.60%, 20.45%, and 24.85% for
	Speaker Α, B, and C respectively.

Table 2 .

 2 Percent (%) accurate prediction of all 60 sentence

	IBM	Google	Wit
	26.67	43.33	36.67

Table 3 .

 3 Rper και Hper error percentage (%)According to Table2, the Google ASR system presents the lower Rper and Hper errors for each speaker. Knowing that Rper and Hper errors are associated with adding or removing words in recognized speech, the measurements confirm that the Google system presents the fewest distortions. Finally, we calculated the total mean errors per tool, which are shown in Table4.

	Speaker A		Speaker Β	Speaker C
	IBM Google Wit	IBM	Google	Wit IBM Google Wit
	Rper 28.45 16.81 24.21	42.24	20.44	26,64 30.66 26.94 61.78
	Hper 28.60 13.35 19.79	42.09	19.53	24,17 32.97 22.29 52.99

Table 4 .

 4 Total Mean Errors (%)

		IBM	Google	Wit
	Rper	23,90	21,40	37,54
	Hper	34,55	18,39	32,32

Acknowledgements. This work is partially supported by MPhil program "Advanced Technologies in Informatics and Computers", hosted by Department of Computer Science, International Hellenic University.