Alexandros Zervopoulos

Aikaterini Georgia Alvanou

Konstantinos Bezas

Asterios Papamichail

Manolis Maragoudakis

Katia Kermanidis

Hong Kong Protests: Using Natural Language Processing for Fake News Detection on Twitter

Keywords: Fake news detection, Natural Language Processing, Machine Learning, Twitter, Hong Kong protests

The automation of fake news detection is the focus of a great deal of scientific research. With the rise of social media over the years, there has been a strong preference for users to be informed using their social media account, leading to a proliferation of fake news through them. This paper evaluates the veracity of politically-oriented news and in particular the tweets about the recent event of Hong Kong protests, with the aid of a dataset recently published by Twitter. From this dataset, Chinese tweets are translated into English, which are kept along with originally English tweets. By utilizing a language-independent filtering process, relevant tweets are identified. To complete the dataset, tweets originating from valid sources are used as the real portion, with journalists rather than news agencies being considered, which constitutes a novel aspect of the methodology. Well-known Machine Learning algorithms are used to classify tweets, which are represented by a feature value vector that is extracted, selected and preprocessed from the datasets and mainly revolves around language use, with word entropy being a novel feature. The results derived from these algorithms highlight morphological, lexical and vocabulary differences between tweets spreading fake and real news, which are for the most part in accordance with past related work.

Introduction

Social media, popular or not, allow both borderless communication and a plethora of information to be spread at a dizzying speed around the world, justifying the choice of the largest percentage of them to keep up to date with domestic and global news events, via Facebook, Twitter and so on. However, the validity of the news is not guaranteed, as it may be hampered by conspiracy, political expediencies and interests. By extension, the spread of fake news contributes to a common and everyday phenomenon, which can undermine values and ideals and, thus, needs addressing.

The spread of fake news is particularly prevalent in politically oriented content, especially so on Twitter, where it has been found that the rate of dissemination of fake news is higher than that of real news [START_REF] Vosoughi | The spread of true and false news online[END_REF][START_REF] Parmelee | Politics and the Twitter revolution: How tweets influence the relationship between political leaders and the public[END_REF]. In this context, computer science can also be used as the primary asset and tool for false detection in news releases from Twitter user accounts, helping to counteract and eliminate this phenomenon. Since the process of falsehood detection by conventional methods, such as the involvement of certified journalists, is a costly process, due to the financial costs and the lengthy periods of time required to complete it, technological approaches, starring Artificial Intelligence, have gained ground [START_REF] Oshikawa | A survey on natural language processing for fake news detection[END_REF], instead. The recent (June 2019) events of the Hong Kong protests related to political controversy have been of great concern to the public because of the violent turn and the high turnout of citizens inside and outside China's borders [START_REF] Purbrick | A report of the 2019 hong kong protests[END_REF]. As a result, a plethora of tweets were triggered, raising the question of the validity of their content. It is therefore important to study the extent of the fake news spread on Twitter about this event.

In this paper, the problem of automatically distinguishing between tweets spreading fake and real news is tackled through the use of Machine Learning (ML) algorithms. In order to accomplish this, an initial dataset published by Twitter3 regarding the Hong Kong protests is used to represent the fake portion of the data used for classification. This dataset contains tweets in a multitude of languages, though only English and Chinese tweets are utilized, with the aid of machine translation. Relevant tweets are pinpointed through a filtering process that is language-independent, making selective use of machine translation. A collection of tweets is gathered to represent the real portion of the dataset, which are considered trustworthy based on the account posting the tweet. News agency and journalist accounts are considered trustworthy sources for the purposes of this study. The assembled dataset is publicly available for research purposes in Humanistic and Social Informatics Laboratory's website 4 . From the assembled dataset, a plethora of linguistic features are extracted, preprocessed and selected to be used as inputs for a variety of well-established ML algorithms, namely Naive Bayes, Support Vector Machines (SVMs), C4.5 and Random Forest. Twitter text has idiosyncrasies that render its linguistic processing quite interesting and that have been tackled in various contexts, the TraMOOC system being one of them [START_REF] Sosoni | Translation crowdsourcing: Creating a multilingual corpus of online educational content[END_REF]. The derived models indicate significant differences in morphological, lexical and vocabulary features between tweets spreading fake and real news. In contrast to previous studies, journalists are investigated regarding trustworthiness, rather than just news agencies, and word entropy is used as a novel feature, which plays an important role in classification.

The rest of this paper is organized as follows. An overview of related literature is presented in Section 2, while the applied methodology is described in Section 3. Section 4 specifies the produced results and finally, conclusions are drawn in Section 5.

Related Work

The detection of fake news, and, in particular, those that are spread through social media, has been extensively researched by the scientific community. Specifically, in [START_REF] Oshikawa | A survey on natural language processing for fake news detection[END_REF][START_REF] Shu | Fake news detection on social media: A data mining perspective[END_REF], the technical challenges in automating fake news detection using Natural Language Processing (NLP) tactics are presented, while a comparison between the used datasets, features, models and their respective performances is provided, with the aim of facilitating future studies.

On the other hand, Ahmed et al. [START_REF] Ahmed | Detection of online fake news using n-gram analysis and machine learning techniques[END_REF] approach the issue with the help of text analysis with N-gram attributes (up to 4-gram size) and by using 6 different machine learning techniques for classification. The model that reached a standing out performance is the linear SVM with the use of unigram attributes. Conroy et al. [START_REF] Conroy | Automatic deception detection: Methods for finding fake news[END_REF] focused on the detection of fake news, with the aim of presenting a hybrid approach based on a combination of linguistic and network-analysis techniques. For both categories, machine learning tools that ultimately contribute to successful detection are described.

In addition, Buntain and Golbeck [START_REF] Buntain | Automatically identifying fake news in popular twitter threads[END_REF] are occupied with automating the detection of fake news on Twitter via 2 existing datasets to analyze the structure and behavior of potentially fake Twitter threads, assessing their proximity to the thread with the help of the Buzzfeed dataset 5 . The aim is to determine the appropriate characteristics of training capable models for predicting falsehood.

While multiple well-established datasets exist, experimentation focusing on specific events regularly takes place. This poses an array of challenges, primarily due to the fact that expertly-annotated data is hard to come by. As such, attempts have been made to circumvent the need for experts' opinions by utilizing data-driven techniques. One such example is the work by Helmstetter [START_REF] Helmstetter | Weakly supervised learning for fake news detection on twitter[END_REF], who consider the credibility of a tweet's source as a proxy for the trustworthiness of the tweet itself, achieving high prediction scores.

In [START_REF] Volkova | Separating facts from fiction: Linguistic models to classify suspicious and trusted news posts on twitter[END_REF], authors deliberate the classification of fake and verified news and the promotion of 4 categories of fake news: propaganda, satire, hoaxes and clickbait. The analysis and experimentation is based on Twitter and, in fact, the data collection takes place over a period of 2 weeks, during the terrorist attacks in Brussels in 2016. For classification, linguistically-infused neural network models are created, based on the content of tweets and social network interactions. Finally, they conclude that morphological and grammatical features are not efficient.

Methodology

Fake News Dataset

The initial fake news dataset is retrieved from Twitter's Election Integrity Hub 6 , where three sets were disclosed in August and September 2019. In greater detail, this dataset consists of 13,856,454 tweets in total and includes 31 fields, which represent tweet-related features about both the tweet's text and the user. In the present study, Twitter is regarded as a reliable source and they have deemed these accounts to be "deliberately and specifically attempting to sow political discord in Hong Kong,"7 ; thus, this dataset is considered as ground truth with respect to the fake news portion of the assembled dataset.

However, as per Twitter's description of the dataset, the accounts involved tend to be fake, post spam and act in a coordinated manner, which has also been investigated in the literature [START_REF] Shu | Fake news detection on social media: A data mining perspective[END_REF]. Hence, not all tweets are relevant to the spread of fake news related to the Hong Kong protests, deeming mandatory an initial preprocessing step. Furthermore, due to the specificities of the events, which take place in China, it is assumed that most of the relevant tweets would be worded either in Chinese or English, as the latter is more prevalent in the Twitter platform.

To better visualize and understand the contents of the dataset, the tweets' text is preprocessed, from which word clouds are constructed. Namely, hashtags, mentions and URLs are removed from the texts in English and in Chinese. Those in Chinese are also translated into English, using Google's Translation API available through the Google Cloud8 platform. Afterwards, a frequency word cloud is created for each language, with the aid of Python's word cloud 9module, containing at most 400 words. It should be noted that these word clouds are derived only from the first two out of the three of Twitter's datasets, as described at the beginning of this subsection. The resulting word clouds are depicted in Fig. 1a, Fig. 1b. Chinese tweets are evidently more relevant to the events than their English counterparts.

To precisely identify the tweets spreading false information regarding Hong Kong protests, the ensuing filtering methodology is followed, which is largely based on the assumption that a tweet's hashtags also indicate the content of a tweet's text. It is worth pointing out that this process is language-independent, which is particularly advantageous in this case, as it is impractical to translate millions of Chinese tweets into English. Moreover, the presented filtering process is overall fairly efficient, requiring a short amount of time, typically a few minutes using commodity hardware.

The methodology can be broken down in the subsequent manner. First of all, a list of hashtags related to Hong Kong protests is manually constructed, comprising both English and Chinese hashtags. Afterwards, hashtags appearing in tweets along with at least one of the previously mentioned hashtags are kept track of, and their co-occurrence counts are calculated across the entire dataset. Finally, tweets are deemed relevant if they contain a hashtag with a co-occurrence count higher than an arbitrary threshold (in this case, 100). It is evident that, due to this approach, tweets without hashtags cannot be considered relevant. Upon the completion of the filtering steps, only 3,908 tweets, worded in English or Chinese, are considered to be relevant to the spread of fake news related to Hong Kong protests. The word cloud constructed by the collection of these relevant tweets is depicted in Fig. 1c.

Real News Dataset

As previously mentioned, the dataset corresponding to real news has not been retrieved in the same manner as the fake news, i.e. no real news dataset has been made publicly available by Twitter. Additionally, well-established datasets are not appropriate for this study, due to the fact that they do not specifically deal with events surrounding the Hong Kong protests. To address these challenges, the real news dataset has to be constructed from the ground up here. In more detail, since expert labeling is a costly process, news posts about Hong Kong protests appearing in generally well-regarded news agencies are often assumed to be reliable sources 10 . Therefore, tweets from the accounts of news agencies are retrieved, rather than articles.

Moreover, one would expect news agencies' tweets to differ in style from those of personal accounts, e.g. more formal speech, fewer replies to other users, more references to the agency's articles, etc. As such, in this study, tweets from journalists of well-regarded news agencies are also considered as real news. The search and retrieval of relevant twitter accounts of news agencies and journalists employed by them takes place manually and, in this case, the following news agencies are considered: BBC News, Reuters, Bloomberg, BuzzFeed, Channel News Asia, CNN, Agence France-Presse, South China Morning Post, Wall Street Journal, The New York Times, The Associated Press, The Washington Post and Quartz. Having completed the search and retrieval of the aforementioned twitter accounts, 13 accounts of news agencies and 107 accounts of journalists are gathered. Using Twitter's user timeline API, 41,996 tweets from news agencies' accounts and 103,359 tweets from journalists' accounts are collected.

The retrieved data seem to be supporting the assumption that the tweets contained in the fake news dataset are more similar to those of journalists than those of news agencies. A few notable statistics derived from the unfiltered data and the first two fake news datasets are listed to showcase some differences and the aforementioned notion of 'similarity': On average, a tweet contains approximately 0.22 hashtags in the fake news dataset, 0.23 hashtags when posted by a journalist and 0.1 hashtags when posted by a news agency. Additionally, the mean number of urls in a tweet is 0.3 in the fake news dataset, 0.35 when posted by a journalist, and 0.82 when posted by a news agency. Lastly, on average, each of the accounts posting tweets have 4.1 followers in the fake news dataset, 15.14 in the journalist dataset, and 11,509.08 in the news agency dataset.

Frequency word clouds are also derived from relevant tweets found in these two datasets and are shown in Fig. 1d, Fig. 1e. While both are evidently relevant to Hong Kong events, the more objective, news-based narrative of the news agency dataset differs from the journalist and fake news dataset. Thus, it becomes clear that the tweets collected from journalist accounts are more similar to those in the fake news dataset, when considering both tweet content and account characteristics.

Using the filtering process described previously, 5,388 and 666 of the tweets posted by journalists and news agencies, respectively, are considered relevant, with the latter being fewer due to the fact that the filtering process relies purely on hashtags, which news agencies don't use as often, as was already showcased. Due to both the low number of tweets and dissimilarity to the fake news dataset, the news agency dataset is entirely dropped and not further studied. All in all, the assembled dataset consists of 3,908 and 5,388 tweets spreading fake and real news, respectively.

Features

The selected features are purely linguistic in nature and they represent a single tweet. While the literature indicates that network-related features are worth investigating, most of the information about the fake news dataset has been made unavailable by Twitter and is no longer accessible on the platform, as the accounts involved in the disclosed datasets have been banned. Regarding the derivation of features, various preprocessing stages are necessary in some cases, which are mentioned when appropriate. The features add up to 38 in total, including the class label, and their Pearson correlation heatmap is depicted in Fig. 2. Even at this early stage, the features of tweet entropy, tweet length and type to token ratio (TTR) are highly correlated with the class label, so they are likely to be important in classification. Label The TTR is calculated after the text has been tokenized as an indication of the tweet's richness in vocabulary. A plethora of features are used in the morphological level. These include a large assortment of counts regarding the tweet's: length in characters, tokens, included URLs, hashtags, certain punctuation marks (' ?', ' !', ' ?!') and total punctuation marks, emojis, periods, stopwords, words, vowels, consonants, upper and lower case characters, digits and letters. Since these counts are expected to be correlated with the tweet's length, they are normalized by dividing them with the latter. Furthermore, the average number of characters per word and average number of words per period are calculated, along with the length of the longest sequence of vowels and consonants in a word. Finally, if a character's consecutive appearance takes place more than thrice, it is marked in the form of a boolean value. Last but not least, the entropy of a tweet is calculated through the equation: S =i P i log P i , where P i is the probability of word i, which has been stemmed and converted to lowercase, ap-pearing in the dataset. Entropy has been used in this case as an indicator of word importance for a tweet, but other similar metrics, such as word weighted frequency could be considered instead.

A number of Part of Speech (PoS) features are included, which keep track of the corresponding occurrences in the tweet: verbs, entities, pronouns, determiners, adverbs, as well as the proposition "to". These features are calculated with the aid of the NLTK module's [START_REF] Loper | Nltk: the natural language toolkit[END_REF] recommended PoS tagger and, much like the morphological features representing counts, they are normalized by the tweet's length.

A wide array of semantic features are used to provide higher-level information about the tweet. These include: positive, neutral and negative sentiment derived from text and emojis. The sentiment of emojis is calculated based on the list provided by [START_REF] Novak | Sentiment of emojis[END_REF] and summed up for each emoji found in the tweet text. Similarly, the text sentiment is calculated according to the AFINN word list [START_REF] Nielsen | A new anew: Evaluation of a word list for sentiment analysis in microblogs[END_REF], which is also available as a Python package 11 .

Algorithms

In the sequel, the ML algorithms, feature preprocessing and selection methods are considered. Literature has deemed effective the use of Naive Bayes, SVMs and Decision Trees for predicting the veracity of news. As such, four different algorithms are used for the training and evaluation of classification models: Naive Bayes, SVM, C4.5 and Random Forests [START_REF] Breiman | Random forests[END_REF] of C4.5. The rather popular Scikit-Learn Python module [START_REF] Pedregosa | Scikit-learn: Machine learning in python[END_REF] implements these algorithms and is being used for the purposes of this study. Regarding SVMs, the Radial Basis Function kernel is made use of and the tweaking of parameters gamma and C is optimized through the use of the grid search hyper parameter tuning technique.

Furthermore, certain algorithms require feature preprocessing or selection to become more effective. In all cases, feature selection can significantly reduce training time, and aids in the prevention of overfitting. As such, for all algorithms except Random Forest, all features are ranked according to their mutual information and the top 10 of those are selected. Due to the fact that decision trees are harder to understand if preprocessing is applied to the data, it is avoided in this study. However, in the case of Naive Bayes and SVMs, all feature values v have been normalized to values v norm lying in the [0, 1] range through the transformation v norm = (v -v min )/(v max -v min ), where v min , v max are the minimum and maximum values across all values of a given feature, respectively.

Results

Feature Selection

Having completed the feature selection process for the collected datasets, the top 10 most significant features according to the mutual information metric are: tweet length, TTR, number of punctuation marks, number of periods, average number of characters per sentence, number of adverbs, number of "to", number of verbs, number of entities and tweet entropy.

Since the features are mostly linguistic, the goal is to identify patterns in language use between tweets spreading fake and real news. Thus, one would perhaps expect language to be more formal and structurally sound in tweets disseminating real news than those disseminating fake ones. The most significant features derived from the feature selection process seem to be in accordance with that theory. For instance, low TTR indicates repetition of words, which may be inversely associated with conceptual variance, while adverbs constitute a hardto-interpret source of information [START_REF] Conlon | Can computers handle adverbs?[END_REF].

Machine Learning Models

The ML algorithms utilized for the classification of tweets spreading fake and real news are trained using the collected datasets and the corresponding evaluation results are presented. The dataset consists of 3,910 and 5,388 tweets spreading fake and real news, respectively, with the majority class baseline being 57.9%. In all cases, 5-fold cross validation is used to increase reliability of results. As can be observed from Table 1, although all algorithms perform fairly well, the most highly performing algorithm is Random Forest, achieving a macro average F1 Score of 92.4%. A noteworthy observation from these results is that all algorithms tend to score higher in precision in the Fake rather than the Real class, whereas the opposite is true for recall; recall scores are noticeably lower for the Fake class when compared to Real. Since the impact of a tweet spreading fake news could be considered significant, one could argue that models achieving higher recall scores would be preferred, even if precision scores suffered somewhat. Nevertheless, both scores are adequately high, such that no additional performance concerns are raised. In order to gain a better understanding of the factors that affect classification, an indicative decision tree resulting from C4.5 is depicted in Fig. 3. It is evident that tweet entropy contributes to an exceedingly high degree in the classification, as 75.7% and 96.7% of the tweets belonging to the Fake and Real class, respectively, are correctly identified within the first split. This would indicate that tweets in the Fake class contain more infrequent words than those in the Real class, which could be an aftereffect of the tweets being translated. Other notable features include tweet length, number of punctuation marks and a few of the morphological features, such as number of verbs. All things considered, according to such a model, a tweet spreading fake news would exhibit the following traits: unconventional vocabulary, longer length, fewer punctuation marks and shorter sentences.

The obtained results are compared to those found in the literature. Granik and Mesyura [START_REF] Granik | Fake news detection using naive bayes classifier[END_REF] scored an accuracy of 74% employing Naive Bayes. Ahmed et al. [START_REF] Ahmed | Detection of online fake news using n-gram analysis and machine learning techniques[END_REF] scored an accuracy of 92% utilizing Linear SVM. Rubin et al. [START_REF] Rubin | Fake news or truth? using satirical cues to detect potentially misleading news[END_REF] achieved 87% F-score using SVM model. Sentiment features do not seem to be very important, as is also highlighted by literature [START_REF] Rashkin | Truth of varying shades: Analyzing language in fake news and political fact-checking[END_REF]. Unlike the results presented here, Volkova et al [START_REF] Volkova | Separating facts from fiction: Linguistic models to classify suspicious and trusted news posts on twitter[END_REF] conclude that morphological and grammar features are not important, although they do find that their results contradict previous work.

The overall high classification scores can be attributed to a number of factors resulting from the methodology. For one, even though journalists' tweets may be more similar to the ones in the fake dataset than those of news agencies, they may still differ somewhat noticeably, making classification easier. Furthermore, the employed methodology relies purely on linguistic traits, with the ones in the fake news dataset being mostly translated to English from Chinese. Therefore, there is significant risk that the model may distinguish traits originating from the translation, which could explain the exceedingly high importance of the tweet's length and entropy. This is further accentuated if one considers the intricacies of the Chinese language, such as the fact that a single symbol corresponds to an English word. The consequences of such intricacies are evident in the translated dataset: a Chinese tweet that is originally 125 characters long is translated to 585 characters in English, which would normally be too long for a tweet.

Conclusions and Future Work

This study focused on detecting fake news in tweets related to Hong Kong protests, using ML algorithms. It used an initial dataset with fake news that has been publicized by Twitter, from which English and Chinese tweets are taken into account, with the latter being translated into English. Relevant tweets were identified through a language-independent process, utilizing hashtag information, enabling the selective use of machine translation. A whole new dataset with real news was built as well, which comes from Twitter accounts of worldwide news agencies. Interesting (in comparison to other studies) is the fact that real news were also retrieved from the personal Twitter accounts of the journalistic team of these agencies. The assembled datasets were used to train and evaluate ML algorithms, once the necessary feature extraction, preprocessing and selection was accomplished, to represent each tweet as a feature value vector. The obtained results achieved high classification scores and indicate that tweets spreading fake news and real news differ noticeably in linguistic features and most notably in tweet length, vocabulary. Word entropy was also deemed very important in classification, a feature not commonly used in similar studies.

Even though the obtained results are promising, there is still much room for improvement and experimentation in future work: (i) study the impact translation has on the results; (ii) include and compare different kinds of features, especially ones related to user and network characteristics; and (iii) utilize more modern ML algorithms, such as deep neural networks.

Fig. 1 .

 1 Fig. 1. Frequency word clouds formed from tweets: (a) In the fake news dataset worded in English. (b) In the fake news dataset worded in Chinese. (c) Resulting from the filtering process. (d) In the news agency dataset. (e) In the journalist dataset.

Fig. 2 .

 2 Fig. 2. Pearson correlation heatmap of features.

Fig. 3 .

 3 Fig.3. A decision tree resulting from the C4.5 algorithm with maximum depth set to four.

Table 1 .

 1 Evaluation results of the ML algorithms used in the classification process.

	Algorithm	Class Precision Recall F1 Score
		Fake	90.1%	85.4% 87.6%
	Naive Bayes	Real	89.7%	92.8% 91.2%
		Average 89.9%	89.1% 89.4%
		Fake	96.0%	84.0% 89.6%
	SVM	Real	89.4%	97.5% 93.3%
		Average 92.7%	90.8% 91.4%
		Fake	94.7%	84.7% 89.3%
	C4.5	Real	89.8%	96.6% 93.0%
		Average 92.3%	90.6% 91.2%
		Fake	97.5%	84.3% 90.3%
	Random Forest	Real	89.7%	98.4% 93.8%
		Average 93.6%	91.3% 92.1%

https://transparency.twitter.com/en/information-operations.html

https://hilab.di.ionio.gr/index.php/en/datasets/

https://github.com/BuzzFeedNews/2016-10-facebook-fact-check

https://transparency.twitter.com/en/information-operations.html

https://tinyurl.com/y3ffrblt

https://cloud.google.com/translate/

https://github.com/amueller/word_cloud

https://www.4imn.com/news-agencies/

Acknowledgments

This project has received funding from the GSRT for the European Union's Horizon 2020 research and innovation programme under grant agreement No 644333.

Tweet length Number of tokens TTR Number of URLs Number of hashtags Number of punctuation marks Number of "?!" Number of "?" Number of "!" Number of emojis Negative emoji sentiment sum Neutral emoji sentiment sum Positive emoji sentiment sum Number of periods Number of stopwords Number of words Average # of chars per word Average # of chars per sentence Overall text sentiment Number of vowels Number of consonants Number of uppercase chars Number of lowercase chars Longest sequence of consecutive vowels Longest sequence of consecutive consonants Has repetition of >3 identical consecutive chars Number of digits Number of letters

Ratio of letters to digits Number of pronouns Number of determiners Number of nouns Number of adverbs Number of "to" Number of verbs Number of entities Tweet entropy Label Tweet length Number of tokens TTR Number of URLs Number of hashtags Number of punctuation marks Number of "?!" Number of "?" Number of "!" Number of emojis Negative emoji sentiment sum Neutral emoji sentiment sum Positive emoji sentiment sum Number of periods Number of stopwords Number of words Average # of chars per word Average # of chars per sentence Overall text sentiment Number of vowels Number of consonants Number of uppercase chars Number of lowercase chars Longest sequence of consecutive vowels Longest sequence of consecutive consonants Has repetition of >3 identical consecutive chars Number of digits Number of letters Ratio of letters to digits Number of pronouns Number of determiners Number of nouns Number of adverbs Number of "to" Number of verbs Number of entities Tweet entropy


Orestis Lampridis
email: lorestis@csd.auth.gr

Athanasios Kefalas
email: kefalasa@csd.auth.gr

Petros Tzallas
email: ptzallas@csd.auth.gr

Greek Lyrics Generation

Keywords: Lyrics Generation, Natural Language Processing, Machine Learning

This paper documents the efforts in implementing lyric generation machine learning models in the Greek language for the genre of Éntekhno music.

To accomplish this, we used three different Long Short-Term Memory Recurrent Neural Network approaches. The first method utilizes word-level bi-directional network models, the second method expands on the first by learning the word embeddings on the initial layer of the network, while the last method is based on a char-level network model. Our experimental procedure, which utilized a high sample of human judges, shows that texts of lyrics generated by our models are of high quality and are not that easily distinguishable from actual lyrics.

Introduction

Natural Language Generation (NLG) is the process of generating text or speech with the use of structured data. Although precisely defining NLG has been proven difficult, a definition often used has been given in [START_REF] Reiter | Building applied natural language generation systems[END_REF]. "NLG is a subfield of artificial intelligence and computational linguistics that is concerned with the construction of computer systems than can produce understandable texts in English or other human languages from some underlying non-linguistic representation of information". NLG as a whole and more specifically the specialized task of creative writing is a task that humans can be quite effective at, while computational intelligence may find it rather difficult to generate creative and good quality text [START_REF] Colton | Computational creativity: The final frontier?[END_REF]. To this end, many attempts have been made in the literature to try and mimic human creativity, including automatic generation of poetry [START_REF] Das | Poetic Machine: Computational Creativity for Automatic Poetry Generation in Bengali[END_REF][START_REF] Gervás | An expert system for the composition of formal spanish poetry[END_REF][START_REF] Oliveira | Automatic generation of poetry: an overview[END_REF], metaphors [START_REF] Veale | A fluid knowledge representation for understanding and generating creative metaphors[END_REF], slogans [START_REF] Tomašic | Implementation of a slogan generator[END_REF] and others.

The domain of song lyrical writing has not been explored as much [START_REF] Oliveira | Tra-la-lyrics 2.0: Automatic generation of song lyrics on a semantic domain[END_REF]. This problem can be more apparent due to the fact that song lyrics also have a musical aspect and may have a different style based on the genre of the song as well as secondary constraints of the task such as rhyming words and thematic tone definition.

To evaluate the performance of our models, we used human judges in order to evaluate the fidelity and credibility of our model when it comes to accurately imitating the lyrics of a real Éntekhno song. In other words, how likely it would be for the text outputted by our model to be the lyrics of an actual song. This paper is structured as follows: In section 2, we introduce various computational approaches which have been used to successfully generate either poetry or song lyrics. In section 3, we demonstrate our dataset. In section 4, we present our different approaches to solve this problem. In section 5, we evaluate our different approaches with the help of human judges. Finally, in section 6, we offer our conclusions and discuss several future directions for our work.

Related Work

While the study of text generation for creative writing purposes, such as poetry and song lyrics, is of great interest in academic areas such as linguistics and music, it is also of great importance for many subfields of computer science. Relevant literature can be found in research areas such as computational creativity, information extraction, natural language processing and machine learning.

In the work of Graves [START_REF] Graves | Generating sequences with recurrent neural networks[END_REF], Recurrent Neural Networks (RNNs) [START_REF] Rumelhart | Learning representations by back-propagating errors[END_REF] were used for text generation and their high effectiveness was showcased. Graves used a variation of RNNs called Long Short-Term Memory (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF] architecture to create a language model in the character level, which has a higher success at text generation than a regular RNN model. The results are impressive, as the network created was able to learn various grammatical rules, while also being able to accurately reproduce a considerable amount of vocabulary of words in the English language.

As already mentioned, automatic text generation for poetry and song lyrics, has also been explored in the literature. Language models have been used to generate poetic text, constrained by both a target style and a predefined form. These include Markov models as in [START_REF] Barbieri | Markov Constraints for Generating Lyrics with Style[END_REF] and models based on Support Vector Machines (SVMs) as in [START_REF] Das | Poetic Machine: Computational Creativity for Automatic Poetry Generation in Bengali[END_REF]. As far as rap lyrics are concerned, Wu et al. [START_REF] Addanki | Unsupervised rhyme scheme identification in hip hop lyrics using hidden Markov models[END_REF] present a system generating rap lyrics that outputs a single sequence of rap lyrics that are a response to a particular input. A different approach was given by Malmi et al. [START_REF] Malmi | Dopelearning: A computational approach to rap lyrics generation[END_REF] by using a Deep Neural Network (DNN) and the RankSVM [START_REF] Joachims | Optimizing search engines using clickthrough data[END_REF] algorithm. By utilizing full lines of lyrics from rap songs, they created 16-line verses.

The work which is most similar to ours is that of Potash et al. [START_REF] Potash | Ghostwriter: Using an lstm for automatic rap lyric generation[END_REF]. Given a sequence of rap lyrics and a specific rapper, an LSTM RNN model is trained and used to predict the next word. Their goal is to produce rap lyrics that are similar in style but not identical to already existing rap lyrics; a task known as ghost writing in the music industry.

Dataset

The task of generating song lyrics using machine learning algorithms requires the use of a proper dataset of song lyrics for training these algorithms. The dataset was downloaded mainly from three sources, stixoi.info, greeklyrics.gr and kithara.to by using a web scraper utility program. The scraper utility targeted the URL of the song details web page of each source by injecting an id that was generated from a randomly shuffled sequence of Long type numbers. The Long datatype was selected, because Long is an appropriate type for mapping a numeric primary key id in most widely used relational databases and therefore would have the highest probability of targeting an existing id in the source's database. The scrapper was setup to use a random back-off scheme with a 20 second minimum await time, to avoid interfering with the website's normal traffic as much as possible.

In cases where the scrapper requested a URL that was not a miss, the retrieved HTML document was queried using XPath to get the textual content of HTML DOM elements that contained the song title, lyrics, artist and other optional information such as the songs composer or lyricist. The raw textual data was then sanitized by removing special characters, duplicate whitespace characters, new line characters and symbols. Finally, after gathering an appropriately large amount of songs for the scope of our research, the dataset was exported as a csv.

The size of the raw dataset we have gathered for training the text generating language model, consisted of about 18000 entries of unprocessed songs of various artists and genres. Each song in the dataset consists of the song title and lyrics for each song and optionally in cases where it the data were available, the information of the artist, the composer, the music producer and lyricist. All entries in the raw dataset are strictly Greek songs and short poems or limericks that had been used in some traditional folk songs. The very first preprocessing step was to prune all entries in the dataset that had mixed language lyrics because the context switch between different languages along with the small number of samples with language context switches would impede the text generation model from effective learning and would negatively affect characterbased models.

As different genres of music can have different styles of lyrics and specific phraseology, rhythm and even vocabulary, using the entirety of the dataset would only produce incoherent results regardless of the quality of the text generation model. For the purposes of our research, we selected songs of various artists from only the Greek Éntekhno genre. After dropping songs from all other genres the dataset was then pruned to a final size of 1150 songs.

The entries in the dataset were preprocessed to replace all whitespace characters in the lyrics of song with a space character. Furthermore, all remaining symbols and nonalphanumeric characters expect for punctuation were removed from all songs in the dataset. Additionally, alternate types of single and double quote characters were replaced by the respective standardized English counterpart. Finally, the last preprocessing step for the dataset was to correct all the disfluencies and typographical errors within the songs by hand and additionally expand words with apostrophes to their full variant form. Shortened word forms contained in the song lyrics as localisms and idioms were replaced by hand with the full variant or normalized form. Lastly, some explicit or offensive words were replaced in the corpus by alternates that had either a similar meaning or a similar phonetic rhythm or letter similarity to retain rhyme within the song.

Implementation

As seen in the literature, RNNs are appropriate for the modeling of natural language sequences as the RNN cells are able to retain contextual information about a sequence of tokens. As basic RNN units can present a variety of issues like the exploding or vanishing gradient problem [START_REF] Pascanu | Understanding the exploding gradient problem[END_REF], we settled on using LSTM units as the primary component of the neural networks. Sequences of word tokens may have long running dependencies in the context of a sentence that may be directed either from the beginning to the end or vice versa. The use of bidirectional LSTM units allows for the modeling of these forward and backward dependencies of a token in a given sequence. Deriving from the baseline architectural components succinctly described above we followed three different approaches that all utilize the bidirectional LSTM -RNN models and compared their results. For the first two models, we based our approach in [START_REF]Word-level LSTM text generator[END_REF][START_REF]Automatic song lyrics generation with Word Embeddings[END_REF], while for the final model we drew influence from [START_REF] Ma | AI Generates Taylor Swift's Song Lyrics[END_REF]. The code that was developed is available on our GitHub repository 1 .

Word level Bi-LSTM

The RNN LSTM model used in the first approach is based on the idea of training it with a large sequence of words and then trying to predict the next word by using in a one-hot vector representation of the sequences. As a first step, we had to read our lyrics in text form, convert to lowercase, to have fewer words and split the sentences into tokens. We chose to treat the newline as an individual word. The thought process behind this is that we are giving the LSTM the capability to decide when to start a new line. On this first approach, we didn't use any further pre-processing (i.e. punctuation removal), because we wanted to see if the network could also learn from these features and apply them when creating the new lyrics. Before building the LSTM RNN model, we split our data into 98% for the training set and 2% for the test set. Several different architectures were explored for our network. After carefully examining the quality of the resulting lyrics and taking into account metrics such as accuracy and validation accuracy we ended up on using the following architecture: The first layer in the network consisted of a bidirectional RNN layer with 256 LSTM units. The second layer was a dropout layer with dropout=0.2. Finally, there is a dense layer with softmax as activation function. The output of this layer is a vector of size equal to the number of words in the corpus which contains the probabilities for each available word in the corpus. The next word is then predicted by using the multinomial distribution to sample on these probabilities.

In order to fit the model, we had to use a data generator. We do this because in this approach we used one-hot representation of the sentences. One-hot vectorization results in vectors of 0s with a single 1 in the column of the used word. Thus, each sentence is represented by an array of size the length of the sequences times the number of unique words in our corpus. Our total training set would have size equal to the number of sentences times the length of the sequences times the number of unique words. This number ends up being enormous. Using data generators, we feed the model with chunks of the training data, one for each step, instead of feeding everything at once. The generator function gets the list of sentences, the list of next words, and the size of the batch. For training the model, a shuffled set of the training sequences was used. The loss function used in this model is the categorical cross entropy function. For validation, we send another generator with the test data, so it gets evaluated every epoch. Finally, the optimizer used was adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF].

Word level Bi-LSTM with trainable embeddings

The model used in this approach uses a trainable Embedding layer before feeding the embeddings of a word sequence to a bidirectional LSTM. The raw text of the dataset is further preprocessed by removing all punctuation from the song lyrics and all tokens were converted to lowercase. By applying case folding to lowercase, we limit the size of the vocabulary the embedding layer will have to learn. After processing the raw text of a song, the lyrics were segmented in the word level and were consequently converted to sequences of tokens. Sequences with size smaller than the pre-configured sequence size were padded with a synthetic padding token. Along with the synthetic word used for padding an additional two synthetic words were introduced to the sequences, one marking the beginning of a song and one denoting the end of a song. The inputs of the embedding layer and the model as a whole are non one-hot encoded words, meaning that instead of vectors representing words the model accepts integers that map to a specific token in a predefined vocabulary that was extracted from the dataset. The first layer in the model's neural network was a trainable embedding layer. The size of the output vector was set to 1024 dimensions. The next layer is a bidirectional RNN layer with 256 LSTM units. The output of this layer is a collection of logit values that are intercepted by a dropout layer. The final layer is a dense or regression layer with a softmax activation. The output of this layer is a vector with a number of dimensions equal to the size of the vocabulary. The next token is predicted by using the multinomial distribution to sample on the probabilities contained in the output vector.

For training the above model a shuffled set of all available sequences was used. In contrast with the word level approach, the loss function used in this model is the sparse categorical cross entropy function that is used on sparse categorical data. For validation, the entirety of available sequences were also used, in essence forcing the model to overfit on all of the sequences, while simultaneously increasing the dropout rate to 0.2. This method essentially forces the model to overfit on the dataset but still retain some generalization when predicting sequences. In later epochs of training, some of the data in the train set should be held out to increase the generalization of the model's predictions. The validation function used was sparse categorical accuracy while the optimizer used was adamax [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] for the first few epochs and later manually switched to adam. The use of adamax in the first few epochs is a minor optimization on training because that specific optimizer is the best suited for training models with embedding layers.

Character level Bi-LSTM

In this approach, the data are left in their raw form. The vocabulary now consists of all the characters in the text data, including punctuation. Since we are working at character level there is no point in finding the most common words. Additionally, the data are not converted to lowercase in the final approach, because we noticed that the results were worse in this scenario. The only data processes are the mapping of the characters to integer numbers and the creation of sequences to feed the RNN model. The input value in each sequence is a 100-character string and the target value is simply the next character which is encoded by the One Hot Encoding method. This way we can give RNN a greater power to learn a probability for each possible target value. When a one hot encoding is used for the output variable, it may offer a finer set of predictions than a single label. The input is reshaped from a one-dimensional array to a two-dimensional array to feed the first layer of the RNN.

The model uses a trainable Embedding layer which works in the same fashion as the one in the Word level Bi-LSTM with Embeddings. It then feeds the embedding sequences of characters to the first layer of the RNN, which a bidirectional CuDNNLSTM layer with 512 nodes. The CuDNNLSTM is the same as a simple LSTM layer, the only difference is that it uses GPU for training, which makes it much faster. The hidden layer is another bidirectional CuDNNLSTM, again with 512 nodes. There is a Dropout layer set to 0.2 to avoid overfitting. Finally the output layer is a dense layer and the activation function is the softmax function as we want a distribution over the outputs. We use the adam optimize and the categorical cross-entropy loss.

Callbacks are made monitoring the train and validation accuracy. We train the model for 50 epochs. We use the validation split set to 0.2, which randomly splits up the data into a training set and test set and evaluates the model. The maximum validation accuracy achieved by this model is around 50%, which is an acceptable value for an NLG model. In the generation process we use both a random seed, sampling the sequences made before and a specific set seed. The seed is then mapped and reshaped and a prediction is made using the argmax function.

Evaluation

In evaluating the results of our approaches, we decided to use human judges for extrinsic evaluation of the three proposed neural language models. We were inspired by the work of Belz and Reiter [START_REF] Belz | Comparing automatic and human evaluation of NLG systems[END_REF] to set up our experimental procedure. We created a survey that contained 10 different texts in total. Out of these texts, 9 of them were from our 3 different models (3 texts for each model) and the last text was from an actual song. We did this to see how humans would rate an actual (i.e. not automatic generated) Éntekhno song. The participants didn't know that a text from an actual song would be shown. We expected this song to have the highest score and if this was the case, we would be certain about the validity of our survey. Then, we asked 76 volunteers to participate in the survey. A high percentage of them (96.1%) stated that they were familiar with the genre of Éntekhno.

The texts were presented in a random fashion and after each text, the volunteers were called to rate the accuracy of the generated lyrics on a scale of [START_REF] Addanki | Unsupervised rhyme scheme identification in hip hop lyrics using hidden Markov models[END_REF][START_REF] Das | Poetic Machine: Computational Creativity for Automatic Poetry Generation in Bengali[END_REF] when it comes to imitating the lyrics of a real song. In other words, how likely it would be for the text they were reading to be the lyrics of an actual Éntekhno song. We call this accuracy the fidelity of our models. An example of the texts that were shown to the participants is shown in Table 1. To make the results more understandable to the international community, we tried to offer the best possible translation in the English language for the texts in Table 2. Note, that these translations do not accurately represent the capability of our models since each language has different ways to express creativity and our models were only trained using the Greek language. Lastly, average fidelity for each model according to the survey and after taking into account the three texts for each of our models is reported in Table 3.

Actual

From the same again to fill the head. Bring me to drink so that my weight will go away because tonight the Reaper will die. A woman has made me stop drinking. She wanted to hurt me but she will pay dearly. Bring me a drink As we can see in Table 3, the word level model with pre-trained embeddings is the one with the highest fidelity, while the word level model is a close second. This is mostly because the embeddings model uses a more complex model than the word level model one, as it uses an additional embedding layer. The character level approach suffers from grammatical and syntax errors; therefore, it has been rated lower. Also, we noticed that the longer the lyrics generated the more difficult it is for each model to generate realistic lyrics.

Conclusions and future work

With regards to future work, a larger dataset could be very beneficial. In our approach, we used a dataset of only 1150 songs, which is a compensatory number, but a larger dataset may produce better results. To this end, the Greek Music Dataset [START_REF] Makris | The greek audio dataset[END_REF] could also be explored as it includes songs of the Éntekhno category. Also, even if we tried many different combinations of layers and tunings in our models, there are a lot more that can be used and bring a better output. Furthermore, an expansion to the whole process could be attempted. One idea could be to add a function to compute the style and tempo of each song and use that information to the generation process, this could produce more realistic lyrics imitating the style of actual songs. In addition, more complex features could be added to the RNN, such as the part of speech of each word, the phonetic representation of each word or the frequency of each word or character. In the character level approach, a function could be made to correct the syntax and grammatical errors or investigate if the words outputted from the network are correct by crosschecking them with a Greek dictionary of words. Finally, another approach to help with the rhyme of the song generated could be to create two RNN models, one that generates the song from start to finish and another one that does the opposite and takes as seed the end of the previous lyric. Concluding, we are obligated to acknowledge the difficulty of the lyrics generation task. With all the progress that is made it can only be used for inspiration by a lyricist and not for immediate commercial use. Poetry and music are arts and have a high degree of creativity that emanate from each artist's soul and it is quite difficult to train a machine learning algorithm to copy them.

2 .

 2 embeddings να αράξω και τον καημό για να σε δω να σε αγκαλιάσω και θα μου πεις φοβάμαι μη σε χάσω δεν είσαι εδώ το όνειρο μου ξανά ζήσω και να φοβάμαι μη χαθείς όταν ξυπνήσω char_level Χρόνια γυρίζεις θεατής στου κόσμου τις κερκίδες μα πες μου τι αγκάλιασες σφιχτά από όσα είδες. Χρόνια καρδιές πολλές και σε χρόνια να παίζεις το παραθείο και στην καρδιά μου ποιημούρα αγάπη μου είπες μου πονάς τα μάτια σου κι εσύ το χρώμα της καρδιάς σου μια κορφή εκεί που πάει εσύ είσαι εγώ εγώ είσαι εσύ. Πάρε με μια καρδιά μου που σε αγαπώ σε αγαπώ το χρόνο να παίζει το φως μου και με παγιδεύεις Actual Από τα ίδια πάλι να γεμίσω το κεφάλι. Φέρε να πιω για να μου φύγει το βάρος γιατί απόψε θα πεθάνει ο χάρος. Μια γυναίκα με έχει κάνει το πιοτό να μη με πιάνει. Θέλησε να με πληγώσει μα πικρά θα το πληρώσει. Φέρε να πιω Table Sample texts shown to the participants translated in the English language text shown word_level how it happened and we changed course and the dream stays in my soul again. tide, tide, you came in and my life. drink fire from my eyes, and burn my soul. like a sudden tide, you came into my life. drink fire from my eyes, and burn my soul. like a sudden tide, you came into my life embeddings to seize the misery to see you hug you and you will tell me i'm afraid i will lose you you're not here my dream live again and i'm afraid you get lost when i wake up char_level For years you have been watching the stands in the world, but tell me what you hugged tightly from what you saw. Years many hearts and in years to play the paratheo and in my heart my love poem you said you hurt my eyes and you the color of your heart a peak where it goes you are me I am you. Take me with a heart that I love you I love you time to play my light and you trap me

Table 1 .

 1 Sample texts shown to the participants during our experiment text shown word_level πως έγινε και αλλάξαμε πορεία και μένει το όνειρο ξανά στην ψυχή μου. παλίρροια, παλίρροια, μπήκες και η ζωή μου. πιες από τα μάτια μου φωτιά, και κάψε την ψυχή μου. σαν ξαφνική παλίρροια, μπήκες και τη ζωή μου. πιες από τα μάτια μου φωτιά, και κάψε την ψυχή μου. σαν ξαφνική παλίρροια, μπήκες μες τη ζωή μου

Table 3 .

 3 The average fidelity for each model Looking at the highest fidelity score in Table4, it was interesting to see that the actual song scored a fidelity of 4.092, not that close to the perfect score of 5. Also, its score was not that far ahead from the second highest scoring text, which was generated by our word level embeddings model and had a fidelity score of 3.815. The third highest scoring text was also not that far behind with a score of 3.671.

		word_level embedings char_level
	fidelity	3.022	3.053	2.338

Table 4 .

 4 The highest fidelity achieved by each model along with the fidelity of the actual lyrics

		word_level embeddings char_level	actual
	fidelity	3.671	3.815	3.026	4.092

https://github.com/orestislampridis/Greek-Lyrics-Generation

Acknowledgments. We would like to express our gratitude to prof. Grigorios Tsoumakas for assigning us this project and for his valuable help and guidance. We would also like to thank all 76 volunteers for the time and they took to participate in the survey used for evaluating our three proposed models along with the comments and suggestions they provided about possible improvements. This work is supported by the Data and Web Science MSc program of the School of Informatics at the Aristotle versity of Thessaloniki.


Rui Ribeiro
email: rui.ribeiro@ccg.pt

André Pilastri
email: andre.pilastri@ccg.pt

Carla Moura

Filipe Rodrigues

Rita Rocha

José Morgado

Paulo Cortez
email: pcortez@dsi.uminho.pt

Keywords: Textile Fabrics, Regression, Machine Learning

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Predicting Physical Properties of Woven Fabrics via Automated Machine Learning and Textile Design and Finishing Features

The introduction of the Industry 4.0 concept is transforming diverse industry sectors due to the adoption of Information Technology (IT), such as Internet of Things (IoT), Big Data, Cloud Computing and Artificial Intelligence (AI) [START_REF] Drath | Industrie 4.0: Hit or hype?[END_REF][START_REF] Wang | Implementing Smart Factory of Industrie 4.0: An Outlook[END_REF]. In particular, the Industry 4.0 transformation can enhance the textile industry by improving the production efficiency (e.g., reducing costs) and assisting in the design of woven fabrics.

In this work, we address a textile company that is being transformed by the Industry 4.0. The company produces custom made woven fabrics for diverse clients. Currently, the fabric design is mostly based on the designer experience and intuition, which results in the execution of several trial-and-error production experiments that require resources (production materials, machines, human labour) and time. Each new fabric production attempt also requires laboratory quality tests, to verify if the produced fabric complies with quality standards and the client requirements. If a fabric is not approved, a new design attempt is set, resulting in an additional production time and costs. All these production steps generate data that can be explored by AI tools, namely Data Mining (DM) and Machine Learning (ML), to support the design of new woven fabrics.

In this paper, we report the implementation of a CRoss-Industry Standard Process for DM (CRISP-DM) [START_REF] Wirth | Crisp-dm: Towards a standard process model for data mining[END_REF] project for the prediction of the final fabric physical properties, as measured by nine laboratory quality tests (e.g., abrasion, pilling). The goal is to use a ML model as an "oracle", providing estimates of the fabric real physical properties for several input design options, thus aiding the textile design experts and reducing the number of fabric production attempts. To better focus on input feature selection and transformation, we adopt an Automated Machine Learning (AutoML) procedure during the modeling stage of CRISP-DM, allowing to automatically select and tune the hyperparameters of the predictive ML models [START_REF] Le | Scaling tree-based automated machine learning to biomedical big data with a feature set selector[END_REF]. In particular, we focus on input variables that can be set during the textile design phase, namely based on fabric design (e.g., composition, amount of finished threads) and finishing (e.g., washing, drying, singeing) features. In total, we executed two major CRISP-DM iterations, in which we explored different input feature engineering strategies.

This paper is organized as follows: Section 2 introduces the related work; Section 3 presents the two CRISP-DM iterations; Section 4 details the obtained results; and finally Section 5 presents the main conclusions.

Related Work

The creation of a new woven fabric is composed of several phases (e.g., design, production, testing). In particular, fabric testing has a crucial role in assessment product quality and performance, ensuring regulatory compliance and it provides information about the properties of the fabrics [START_REF] Hu | Fabric testing[END_REF]. The overall process of new fabric creation generates large amounts of data, which under the Industry 4.0 concept can be used by AI tools (DM and ML) to extract valuable knowledge [START_REF] Han | Data Mining: Concepts and Techniques[END_REF]. Following the increasing interest in DM, the CRISP-DM was proposed as a standard methodology to support the execution of real DM projects [START_REF] Wirth | Crisp-dm: Towards a standard process model for data mining[END_REF]. The methodology involves interactions between business domain and DM/ML experts and several iterations that can include up to six main phases: business understanding, data understanding, data preparation, modeling, evaluation and deployment. Regarding the textile domain, use of DM techniques is more recent, involving mainly classification tasks, such as defect detection and estimating the quality of yarns [START_REF] Yildirim | Data mining and machine learning in textile industry[END_REF].

In what concerns a data-driven modeling of textile quality tests, the research is more scarce. Fan and Hunter [START_REF] Fan | A worsted fabric expert system: Part ii: An artificial neural network model for predicting the properties of worsted fabrics[END_REF] used a backpropagation Neural Network (NN) with one single hidden layer with 30 inputs based on fibre, yarn, and fabric constructional parameters to predict nine fabric properties (e.g., abrasion, seam slippage). A similar NN model was adopted in [START_REF] Beltran | Predicting the pilling propensity of fabrics through artificial neural network modeling[END_REF] to predict the pilling propensity of fabrics. In other study, Support Vector Machines (SVM) with 17 input features related to fiber and yarn were used to predict 8 different rates of pilling [START_REF] Yap | Prediction of wool knitwear pilling propensity using support vector machines[END_REF]. In the same study, backpropagation NN were also proposed to predict other textile properties, such as seam strength and elongation. In a more recent study [START_REF] Eltayib | The of tear strength of plain weave fabric using linear regression models[END_REF], a simple multiple regression model was used to estimate the relationship between fabric tear strength and other input variables, such as yarn tensile strength, yarn count and fabric linear density. In previous work [START_REF] Ribeiro | Predicting the Tear Strength of Woven Fabrics via Automated Machine Learning: An Application of the CRISP-DM Methodology[END_REF], we performed an initial exploration of ML algorithms, such as Random Forests (RF) and Gradient Boosting Machines (GBM), to predict two fabric properties (tear strength in warp and weft directions). While interesting results were achieved, the study explored a very limited set of inputs (e.g., no finishing data was used).

The performance of a ML algorithm is dependent on a correct feature engineering and ML model selection [START_REF] Domingos | A few useful things to know about machine learning[END_REF]. Most of the mentioned state-of-the-art works adopt a simple and fixed set of input variables (defined a priori) when predicting the textile physical properties. Moreover, the prediction models were obtained by using an empirical trial-and-error process that often requires a substantial effort from the ML expert [START_REF] Gibert | Which method to use? an assessment of data mining methods in environmental data science[END_REF]. In contrast with the related works (e.g., [START_REF] Fan | A worsted fabric expert system: Part ii: An artificial neural network model for predicting the properties of worsted fabrics[END_REF][START_REF] Beltran | Predicting the pilling propensity of fabrics through artificial neural network modeling[END_REF][START_REF] Yap | Prediction of wool knitwear pilling propensity using support vector machines[END_REF]), we employ in this paper an AutoML that automatically selects the best among several state-of-the-art ML algorithms. The adoption of the AutoML procedure allowed us to better focus on feature engineering, which is a non-trivial task in this domain. For instance, finishing features were considering a future challenge and thus were excluded from the predictive study performed in [START_REF] Fan | A worsted fabric expert system: Part ii: An artificial neural network model for predicting the properties of worsted fabrics[END_REF]. Woven fabric feature engineering is a complex task due to two main reasons. First, the design and finishing processes of the fabric creation includes a variable number of input features (e.g., yarns, finishing operations) that can influence the targeted textile physical properties. Second, most of these features are nominal and often present a high cardinality. Since most regression ML algorithms work only work with numeric values, a nominal to numeric transform is needed.

CRISP-DM Methodology

First CRISP-DM Iteration

Business Understanding: The creation of a new woven fabric starts with the definition desired characteristics. The fabric developer uses its experience and intuition, taking into account the textile requirements and starts to analyse the most similar fabrics already produced. Then, a several design elements are initially set, such as the type and number of fibers and the pick count. Some of these design elements involve a single value per fabric (e.g., number of picks), while others involve a variable number of choices (e.g., which and how many yarns to use). Next, a physical sample is produced using several materials (e.g., yarns) and machines (e.g., loom). The final production stage includes a variable sequence of finishing operations (e.g., washing, drying). Then, the produced fabric is tested via laboratory instruments, allowing to infer the physical properties and check if it meets the desired characteristics. If the fabric does not comply with the quality standards or client requirements, then the whole fabric creation process is repeated. In practice, several iterations are executed until a quality fabric is achieved, which results in additional production time and costs. The analyzed textile company expressed the need to get a fast and cheap estimate of the true fabric physical properties by adopting a ML approach. The goal is to use the predictive ML models as "oracles", quickly checking some fabric design and finishing alternative choices, thus reducing the number of attempts necessary to produce a woven fabric.

In total, the company identified nine target properties: abrasion, seam slippage (warp and weft directions), elasticity (warp and weft directions), dimensional stability to steam (warp and weft directions), bias distortion and pilling. All these nine properties are measured using numeric values. In this work, each property is measured as a separate regression task.

Data Understanding and Preparation: The data was collected from two main data sources: the company Enterprise Resource Planning (ERP), with fabric production records, and the laboratory testing database, with fabric quality tests performed between February 2012 to March 2019. The ERP data included 90,034 examples with 2,391 features per row. Using a manual analysis and domain expert knowledge, the ERP features were filtered into a total of 805 potentially relevant attributes. The laboratory dataset had 149,388 examples with the results for the nine selected physical tests. To aggregate all data, a Data Warehouse system was implemented, using an Extraction, Transform, Load (ETL) process to merge and preprocess the two data sources. During the ETL, some records were discarded since they had missing features (e.g., no yarns or no composition values).

When analysing the obtained historical data, we identified a small fraction of laboratory database entries (around 1%) that included slight different physical test values for the same fabric. After consulting the laboratory analysts, it become clear that the differences were due to the execution of laboratory tests at different fabric finishing procedures (e.g., before or after drying). Since the laboratory database did not include when such tests were executed, we opted to compute average values, in order to get a single number per fabric and test.

The initial set of input features explored in this CRISP-DM iteration is presented in Table 1. Figure 1 exemplifies how some of these features are related with the textile fabric. The first 11 rows of the table are related with a fixed set of design attributes that are defined for all fabrics. Each fabric is composed by two main elements warp and weft, each including a variable mixture of yarns, from 1 (minimum) to a maximum of 21 (in our database). Moreover, each yarn has four main characterizing features plus the number of its repetitions in the warp or weft (these features are shown in the last five rows of Table 1). Thus, the proper preprocessing of yarn data, to feed the regression models, is a nontrivial issue. In this work, we propose the following yarn representation. For each fabric, we use a sequence that has a maximum of max y yarns for warp and then another sequence of max y yarns for weft. Each sequence is thus composed by the elements < y 1 , ..., y maxy >, where y i denotes the i-th yarn representation data. In this work, we adopt the threshold of max y =6 yarns per warp and weft. This value allows the representation of 99.7% of the fabrics without any information loss, while using a larger threshold would increase the sparseness of the input space, increasing the complexity of the predictive models. When a fabric does not have 6 yarns, we use a zero padding to fill the "empty" yarn values, which is a popular text preprocessing technique that adds null values (e.g., 0) to non-existent features. Finally, we explore two yarn alternative representations: A, use of the code (unique value) and its number of repetitions, where y i is set as the tuple (code i , repetitions i ); and B, use of all yarn characterizing elements except the code, where y i is set as (composition i , folds i , count i , repetitions i ). In total, the A representation assumes 35 input variables (11+2×6×2), while the B encoding results in 59 input features (11+2×6×5). Before feeding the ML algorithms, all the numeric inputs were standardized to a zero mean and one standard deviation. As for the nominal variables, several of them contain a high cardinality. For instance, the analyzed database includes 6,265 distinct types of yarns. A popular nominal to numeric transform is the one-hot encoding, which assigns one boolean value per nominal level. However, this transformation would highly increase the input space, resulting in a very sparse representation that would prejudice the learning of the regression models, also enlarging the computational memory and effort. To handle this issue, in this work we transform all nominal attributes with the Inverse Document Frequency (IDF) function:

IDF (l) = ln(n/n l ) ( 1
)
where n is the total number of examples in the training set and n l is the number of examples that contain the level l in the analyzed attribute [START_REF] Campos | On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study[END_REF]. The advantage of this transform is that is encodes a nominal attribute into a single numeric value, with the most frequent levels being set near the zero (but with a larger "space" between them), and the less frequent ones being more close to each other and near a IDF (l) maximum value. Table 2 presents the nine output targets. The last column (Range) shows the admissible range values for each target, as defined by the textile company. All examples outside such range were considered outliers (e.g., uncommon military fabrics) and thus removed from the dataset. Since different quality tests can be assigned to different fabrics (depending on the client requirements), a variable number of examples is presented for each output (column Examples).

Modeling:

The experiments were conducted in a personal computer using two different computational environments: the R statistical tool and its rminer package for data manipulation and evaluation of ML algorithms [START_REF] Cortez | Modern optimization with R[END_REF], and H20 software which implements a AutoML procedure [START_REF] Cook | Practical machine learning with H2O: powerful, scalable techniques for deep learning and AI[END_REF]. As previously discussed, during the first iteration of CRISP-DM we explored the issue of yarn representation, thus two main strategies as compared: A and B. During this modeling stage, to find the best ML algorithm we adopt an AutoML procedure.

The AutoML was configured to automatically select the regression model and its hyperparameters based on the best Mean Absolute Error (MAE), using a internal 5-fold cross-validation applied over the training data. We adopted the same AutoML configuration executed in [START_REF] Ferreira | An Automated and Distributed Machine Learning Framework for Telecommunications Risk Management[END_REF]. The computational experiments were executed on a desktop computer and each ML algorithm was trained using a maximum running time of 3,600 seconds. After selecting the best ML algorithm, its best set of hyperparameters are fixed and the ML is retrained with all training data. A total of 6 different regression families are searched by the Au-toML tool: RF, Extremely Randomized Trees (XRT), Generalized Linear Models (GLM), GBM, XGBoost (XG) and Stacked Ensemble (SE). RF is an ensemble method that typically combines a large set of tree predictors, such that each tree depends on a random sample of features and training examples [START_REF] Breiman | Random forests[END_REF]. XRT is another tree ensemble that consists of randomizing both attribute and cut-point choices when splitting a tree node [START_REF] Geurts | Extremely randomized trees[END_REF]. GLM estimates regression models for outcomes following exponential distributions (e.g., Gaussian, Poisson, gamma) [START_REF] Nelder | Generalized linear models[END_REF]. GBM performs an ensemble of weak successive decision trees, sequentially building regression trees for all data features [START_REF] Natekin | Gradient boosting machines, a tutorial[END_REF]. XG is another popular boosting decision tree algorithm [START_REF] Chen | Xgboost: A scalable tree boosting system[END_REF]. Finally, the SE combines the predictions of the previous individual ML algorithms by using a second-level ML algorithm [START_REF] Breiman | Stacked regressions[END_REF]. The H20 tool sets RF and XRT with their default hyperparameters, performs a grid search to set the hyperparameters for GLM (1 hyperparameter), GBM (9 hyperparameters) and XG (10 hyperparameters), and uses GLM as the second-level learner for SE.

Evaluation: An external 3-fold cross-validation was executed to evaluate the regression models. Several metrics were selected to measure the quality of the predictions: MAE, Normalized MAE (NMAE), Adjusted R2 (Adj.R 2 ) and classification accuracy for a given tolerance T (Acc@T ). Regarding MAE and NMAE, the lower the values, the better are the predictions. The NMAE measure normalizes the MAE by the range of the output target on the test set, thus it provides a percentage that is easy to interpret and is scale independent. In the case of Adj.R 2 and Acc@T (from 0 to 1), higher values indicate better predictions. The Acc@T value is based on the Regression Error Characteristic (REC) curves and it measures the percentage of correctly classified examples when assuming a fixed absolute error tolerance (T ) [START_REF] Bi | Regression error characteristic curves[END_REF]. In this work we use T ∈ {5%, 10%, 20%}.

We note that the percentage of error tolerance is computed by considering the range of the target values. The first CRISP-DM iteration results are discussed in Section 4.

Second CRISP-DM Iteration

After showing the Section 4 results to the textile experts, it was decided to perform a second CRISP-DM iteration to check the utility of finishing features.

During a new business understanding phase, it become clear that the finishing process should influence the final fabric properties. The finishing consists of a predefined sequence of operations that are applied to a fabric with the goal to increase the attractiveness or serviceability of the textile product [START_REF] Hall | Finishing of technical textiles[END_REF].

At the new data understanding and preparation stages, we identified that the company had a total of 61 different types of finishing operations. Moreover, the sequence of finishing operations can be different for each fabric and it can include repetitions of the operations (e.g., several wash and dry cycles). In the analyzed database, the number of executed finishing operations ranged from 1 (minimum) to 39 (maximum), with an average of 6.82. Table 3 presents the top ten most used types of finishing and the respective number of usages (column Examples). In the table, we added the special value "Others" to represent a merge of distinct finishing operations for which there was no description data. Similarly to the yarn encoding strategy, in this work we will assume a sequence with a maximum of max f finishing operations to represent the finishing process: < f 1 , ..., f max f >, where f i denotes the i-th finishing operation. In this work, we set max f =10 as a reasonable value that represents around 85% of all fabrics without information loss, helping to reduce the number of inputs that are fed into the ML models. To encode each finishing operation (nominal attribute) we adopt the same IDF transform (Equation 1). In the modeling phase, the best previous input encoding (A) is compared with the new encoding C that merges all A inputs with the finishing features, resulting in 45 (35+10) input variables. The evaluation phase was executed similarly to the first CRISP-DM iteration.

Results

In all experiments performed the AutoML always selected the GBM or SE algorithms. GBM provided the best overall results (lowest NMAE averaged over the external 3 cross-validation iterations), while for some targets and specific folds (e.g., Bias Distortion and third fold experiment), the selected model was SE.

To compare the feature strategy results, we always assume the best algorithm (GBM or SE) per external fold validation, denoting this as the AutoML model. Table 4 summarizes the predictive performance results, in terms of the 3-fold average NMAE values for the best AutoML model that were obtained during the first and second CRISP-DM iterations. For the first iteration, it becomes clear that A is the best yarn representation strategy. It provides the lowest NMAE results for eight of the nine fabric targets and it also obtains the lowest average value, over all output tasks (difference of 0.66 percentage points when compared with B). Moreover, A has the additional advantage of producing less inputs (35 and not 59), leading to predictive models that require less computational memory and fitting effort. Following these results, we adopted the A encoding to represent the yarns. In the second iteration, the usage of fabric finishing features (C) improves the prediction results for eight of the nine targets. Overall, C provides the lowest average NMAE, with a 0.25 percentage point improvement when compared with A. Table 5 complements the results by showing the other predictive measures for A and C (represented in column Str.). In general, when C obtains the lowest MAE error, it also outperforms the A strategy for the other measures (Acc@T and Adj. R 2 ). For demonstrative purposes, Figure 2 shows the AutoML elasticity (warp) predictions (x-axis) versus the target values for a particular external 3-fold iteration. The plot includes the tolerance ranges of the T = 5% and T = 10%, showing that an interesting percentage of the values are correctly predicted within those ranges (e.g., 43% of accuracy for T = 10%).

Conclusions

This paper addresses a textile company that is being transformed under the Industry 4.0 concept and that identified the need to reduce the number of production attempts when new woven fabrics by using a Machine Learning (ML) approach. To handle this goal, we implemented two iterations of the CRoss-Industry Standard Process for Data Mining (CRISP-DM) methodology.

Each iteration focused on a feature engineering task, aiming to check the value of input fabric yarn and finishing feature representations. During the modeling stage of CRISP-DM, an Automated ML (AutoML) was used to select the best among six state-of-the-art ML algorithms. The best results were achieved by an input set of features that includes a fixed sequence with a simple yarn code representation and another fixed sequence with fabric finishing operations (strategy C). Interesting predictive results were achieved for nine targeted fabric properties, with an average NMAE error that ranges from 2% to 7%. The results were shown to the textile company, which considered them valuable to reduce the number of fabric creation attempts, thus having a potential to save the production time and costs. In future work, we intend to apply a similar approach in the prediction of other fabric quality tests, such as residual extension and traction.

  Ribeiro, André Pilastri, Carla Moura, Filipe Rodrigues, Rita Rocha, José Morgado, Paulo Cortez To cite this version: Rui Ribeiro, André Pilastri, Carla Moura, Filipe Rodrigues, Rita Rocha, et al.. Predicting Physical Properties of Woven Fabrics via Automated Machine Learning and Textile Design and Finishing Features. 16th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2020, Neos Marmaras, Greece. pp.244-255, 10.1007/978-3-030-49186-4_21. hal-Predicting Physical Properties of Woven Fabrics via Automated Machine Learning and Textile Design and Finishing Features 1 Introduction

Fig. 1 .

 1 Fig. 1. Visualization of some woven fabric features.

Fig. 2 .

 2 Fig. 2. Predicted versus real elasticity (warp) values.

Table 1 .

 1 Initial list of fabric design input features.

	Attribute	Description (data type)	Range
	composition	Composition of the fabric (nominal code )	660 levels
	t pol	Number of finished threads per centimeter (numeric) [18,1321]
	p pol	Number of finished picks per centimeter (numeric) [7,510]
	weight/m 2	Weight (in grams) per square meter (numeric)	[22,1690]
	finished width	Width in centimeters (numeric)	[122,168]
	weave design	Weave pattern of the fabric (nominal code)	20 levels
	reed width	Width of the reed in centimeters (numeric)	[30,242]
	denting	Number of the reed dents per centimeter (numeric) [5,252]
	ends/dent	Number of yarns per dent (numeric)	{0,1,...,8}
	n picks	Number of picks on loom per centimeter (numeric) [0,3450]
	warp total ends Total number of threads on the warp (numeric)	[1026,6862]
	yarn code	Identification code of the yarn (nominal code)	6,265 levels
	yarn composition Composition of the yarn (nominal code)	88 levels
	n folds

Number of single yarns twisted (numeric) {0,1,...,12} yarn count Mass per unit length of the yarn (numeric) [0,268] yarn repetitions Number of yarn repetitions in warp or weft (numeric) {0,1,...,8}

Table 2 .

 2 List of output target variables.

	Test	Examples Range
	Abrasion	456	[5000,30000]
	Seam Slippage (warp)	10,605	[1,20]
	Seam Slippage (weft)	10,279	[1,20]
	Elasticity (warp)	7,901	[5,55]
	Elasticity (weft)	12,698	[5,70]
	Dimensional Stability to Steam (warp) 8,773	[-4,2]
	Dimensional Stability to Steam (weft) 8,871	[-4,2]
	Bias Distortion	15,141	[1,14]
	Pilling	11,912	[1,4.5]

Table 3 .

 3 Ten most used fabric finishing operations.

	Rank Finishing Examples Rank Finishing	Examples
	1	Dry	145,008	6	Finish Fixate 53,717
	2	Wash	122,513	7	Dyeing	44,162
	3	Sanforization 73,401	8	Others	31,704
	4	Finish	67,774	9	Shear Right 21,600
	5	Singeing	58,875	10	Decatizing	20,685

Table 4 .

 4 AutoML predictive results (average NMAE test set values, in %; best results per CRISP-DM iteration in bold).

	First	Second
	Iteration Iteration

Table 5 .

 5 AutoML predictive results (other regression measures; best results in bold).

	Regression Metrics

Acknowledgments

This work was carried out within the project "TexBoost: less Commodities more Specialities" reference POCI-01-0247-FEDER-024523, co-funded by Fundo Europeu de Desenvolvimento Regional (FEDER), through Portugal 2020 (P2020).