Aliaksandr Barushka
email: aliaksandr.barushka@student.upce.cz

Petr Hajek
email: petr.hajek@upce.cz

Review Spam Detection using Word Embeddings and Deep Neural Networks

Keywords: review spam, skip-gram, word2vec, word embedding, neural network

Review spam (fake review) detection is increasingly important taking into consideration the rapid growth of internet purchases. Therefore, sophisticated spam filters must be designed to tackle the problem. Traditional machine learning algorithms use review content and other features to detect review spam. However, as demonstrated in related studies, the linguistic context of words may be of particular importance for text categorization. In order to enhance the performance of review spam detection, we propose a novel content-based approach that considers both bag-of-words and word context. More precisely, our approach utilizes ngrams and the skip-gram word embedding method to build a vector model. As a result, high-dimensional feature representation is generated. To handle the representation and classify the review spam accurately, a deep feed-forward neural network is used in the second step. To verify our approach, we use two hotel review datasets, including positive and negative reviews. We show that the proposed detection system outperforms other popular algorithms for review spam detection in terms of accuracy and area under ROC. Importantly, the system provides balanced performance on both classes, legitimate and spam, irrespective of review polarity.

Introduction

Review spam (fake review) can be defined as unwanted and misleading messages that can be published on multiple platforms, such as forums or online shops. User base of those online platforms is steadily growing over the years. For instance, TripAdvisor is the world largest travel site with over 455 million average monthly unique visitors and 600 million reviews and opinions covering 7.5 million accommodations, airlines, attractions, and restaurants [1]. Many travelers rely on reviews before choosing a hotel to stay. Online reviews have become a concern of the industry, since review spam may mislead purchasers and eventually lead to a lawsuit against the seller. According to recent statistics, every third review is spam on TripAdvisor [2]. By writing positive or negative review spam, the seller may gain a competitive advantage or disadvantage, respectively. To guarantee fair competition, it is therefore crucial for shopping portals to identify and block review spam and ban fraud users. Review spam can be detected either manually or automatically. Compared to automatic detection, manual review detection is slow, expensive and relatively inaccurate [START_REF] Harris | Detecting deceptive opinion spam using human computation[END_REF]. Therefore, over the past decade, researches have explored ways to improve automatic review spam detection. Machine learning approaches, such as neural networks, support vector machines or Naïve Bayes, have a reputation of effective methods in detecting review spam [START_REF] Heydari | Detection of review spam: A survey[END_REF]. Such methods utilize content-based and other features of reviews to filter review spam accurately. Review spam detection is usually considered as a binary classification problem, in which each review is classified either as legitimate (trustworthy) or spam (fake). Besides overall high classification accuracy, low false positive rate is of particular importance because, otherwise, users of shopping portals would not be able to see legitimate reviews and trustworthy users would get offended and may lose motivation to submit reviews on the particular portal. The main idea behind content-based machine learning models is to build a word (phrase) list and assign a weight to each word or phrase (bag-of-words) or word category (part-of-speech tagging or psycholinguistic) [START_REF] Crawford | Survey of review spam detection using machine learning techniques[END_REF]. However, such features suffer from sparsity, which makes it difficult to capture semantic representation of reviews. To address this issue, Ren and Ji [START_REF] Ren | Neural networks for deceptive opinion spam detection: An empirical study[END_REF] proposed a gated recurrent neural network model to detect opinion spam. This approach utilized word embeddings obtained by using the CBOW (continuous bag-ofwords) model [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Le | Distributed representations of sentences and documents[END_REF] so that words are mapped to vectors based on their context. Thus, global semantic information can be obtained, and, to certain degree, the problem of scarce data is overcome. This approach was reportedly more effective than traditional bag-of-words or part-of-speech tagging.

Inspired by these recent findings, here we utilize the word embeddings to obtain the semantic representation of online reviews. Word2vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Le | Distributed representations of sentences and documents[END_REF] is a popular method to produce word embeddings (vector space model) from a corpus of text data. The word representation can be obtained by two alternative model architectures, namely CBOW or skip-gram. Unlike earlier literature, here we use a skip-gram model for this task, which exploits word context more effectively and thus generates a more generalizable context when compared with the CBOW model [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. To train the skip-gram model, we use the hierarchical softmax algorithm, a computationally effective version of the softmax algorithm. To further enhance the detection performance, we combine the generated word embeddings with bag-of-words in the second stage and train a deep feed-forward neural network (DNN) to classify spam / legitimate reviews. DNN is used to capture complex features hidden in high-dimensional data representations [START_REF] Barushka | Spam filtering using regularized neural networks with rectified linear units[END_REF][START_REF] Barushka | Spam filtering using integrated distribution-based balancing approach and regularized deep neural networks[END_REF][START_REF] Barushka | Spam filtering in social networks using regularized deep neural networks with ensemble learning[END_REF].

The rest of the paper has the following structure. In Section 2, the literature review of the recent advances in review spam detection is introduced. Section 3 describes the corpora of hotel reviews that are used in experiments. In Section 4, our model for review spam detection is introduced. Section 5 presents the results of the experiments and the final section summarizes our findings and suggests future research directions.

Review Spam Detection -A Literature Review

Review spam has been increasingly subject to scrutiny owing to the outstanding importance of product reviews that are used for purchase and business decisions. To influence the decisions and thus make profit, spam reviews are produced to promote (positive reviews) or demote (negative reviews) some products [START_REF] Ren | Neural networks for deceptive opinion spam detection: An empirical study[END_REF]. To detect those deceptive reviews (opinions), machine learning methods have been used because, as shown in earlier literature [START_REF] Harris | Detecting deceptive opinion spam using human computation[END_REF][START_REF] Heydari | Detection of review spam: A survey[END_REF], human readers have limited capacity to detect spam reviews. This task is typically handled as a classification problem, with reviews categorized as spam or legitimate class. This annotation (class label) is provided by people and the aim of the machine learning-based detection system is to automatically classify reviews into these classes.

One of the first published effort to detect review spam utilized the fact that spammers duplicate their reviews, either on the same or different product [START_REF] Jindal | Analyzing and detecting review spam[END_REF]. Similarly, spam scoring proposed in [START_REF] Lim | Detecting product review spammers using rating behaviors[END_REF] was based on the cosine similarity between reviews. Furthermore, Wang et al. [START_REF] Wang | Review graph based online store review spammer detection[END_REF] developed a review graph to capture the interactions among reviews, reviewers and stores. Thus, the honesty of reviews could be calculated. Interestingly, this approach did not use any review text information. In contrast, the approach proposed in [START_REF] Lau | Text mining and probabilistic language modeling for online review spam detecting[END_REF] was based on text features only. Li et al. [START_REF] Li | Learning to identify review spam[END_REF] examined the effect of several feature categories on review spam identification, including content, sentiment, product or profile features. Review metadata were integrated with relational features in SpEagle, a unified framework to rank reviews [START_REF] Rayana | Collective opinion spam detection: Bridging review networks and metadata[END_REF]. Unusual temporal patterns of correlated review ratings were also used to detect spam attacks [START_REF] Xie | Review spam detection via temporal pattern discovery[END_REF]. These patterns make real-time detection of abnormal events possible [START_REF] Ye | Temporal opinion spam detection by multivariate indicative signals[END_REF][START_REF] Li | Bimodal distribution and co-bursting in review spam detection[END_REF]. Spatial patterns (user IPs) were utilized together with temporal patterns in [START_REF] Li | Analyzing and detecting opinion spam on a large-scale dataset via temporal and spatial patterns[END_REF].

Most existing approaches focus on extracting informative features from the texts of reviews to enhance detection performance. Traditional features include bag-of-words, part-of-speech tagging or psycholinguistic word lists [START_REF] Crawford | Survey of review spam detection using machine learning techniques[END_REF][START_REF] Ott | Estimating the prevalence of deception in online review communities[END_REF][START_REF] Liu | A unified framework for detecting author spamicity by modeling review deviation[END_REF]. For example, Ott et al. [START_REF] Ott | Negative deceptive opinion spam[END_REF] identified n-gram features in the texts and then employed support vector machines (SVMs) to perform the classification of reviews. As noted above, the semantic representation of reviews can be captured by word embeddings. In addition to their use in [START_REF] Ren | Neural networks for deceptive opinion spam detection: An empirical study[END_REF], the CBOW model was also combined with network features in a semi-supervised approach developed by [START_REF] Yilmaz | SPR2EP: A semi-supervised spam review detection framework[END_REF]. A sentence weighted convolutional neural network was used to obtain the document representations for review spam detection [START_REF] Li | Document representation and feature combination for deceptive spam review detection[END_REF].

Dataset

In this study, we used two datasets from Cornell University1 , namely positive hotel review spam [START_REF] Ott | Estimating the prevalence of deception in online review communities[END_REF] and negative hotel review spam [START_REF] Ott | Negative deceptive opinion spam[END_REF]. Since the details on these datasets can be found in [START_REF] Ott | Estimating the prevalence of deception in online review communities[END_REF][START_REF] Ott | Negative deceptive opinion spam[END_REF], we provide only their brief description in this study. The positive hotel review spam dataset contained 400 legitimate and 400 spam positive reviews from TripAdvisor (20 legitimate and 20 spam reviews for each of the 20 selected hotels). The spam reviews were gathered using Amazon Mechanical Turk. Only a single review per Turker was allowed, and unreasonably short or plagiarized reviews were rejected. For the positive dataset, only 5-star reviews were included.

A similar procedure was used to collect the negative hotel review spam dataset. Again, Turkers were employed to provide spam reviews on 20 popular hotels, such as such as Affinia Chicago or Ambassador East Hotel, and corresponding legitimate reviews were obtained from several online review communities, such as Expedia, TripAdvisor or Hotels.com. For the negative dataset, only 1-or 2-star reviews were used. The average review length for both datasets was 116 words. The datasets included the following types of information: message content, spam label, hotel information, polarity of the message, and travel agency aggregator name.

To pre-process the content of the reviews in the datasets, several tools were applied. First, tokenization was performed using standard delimiters: ".,;:'"()?!". Second, the stopwords were removed using the Rainbow stopwords list to reduce the noise in the data. Third, all numbers, punctuation and some special symbols were stripped off and all tokens were transformed to lowercase letters.

For the bag-of-words representation, tf.idf weighting scheme was used, in which the weight a term is obtained as follows:

vij = (1 + log(tfij)) × log(N/dfi), (1)
where vij is the weight of the i-the term in the j-th review, tfij is term frequency, dfi is document frequency, and N is the number of reviews. This weighting scheme was selected because it takes into account review lengths and term rareness.

Methods

The proposed architecture for spam review detection is depicted in Fig. 1. To avoid overfitting, the datasets were divided into training and testing data using 10 times repeated stratified 10-fold cross-validation.

In the n-gram model, we used the bag-of-words representation as defined in eq. ( 1). In this model, text is represented as the bag of its words, disregarding grammar and even word order but keeping multiplicity. In bag-of-words, string attributes are converted into a set of numeric attributes representing word occurrence information from the text contained in the strings. Note that only most relevant terms (attributes) were selected according to their weights vij. In agreement with previous studies [START_REF] Barushka | Spam filtering using integrated distribution-based balancing approach and regularized deep neural networks[END_REF][START_REF] Barushka | Spam filtering in social networks using regularized deep neural networks with ensemble learning[END_REF], top 2000 terms were retained, including bigrams and trigrams as suggested in [START_REF] Li | Document representation and feature combination for deceptive spam review detection[END_REF]. An example of features with the highest information gain is presented in Table 1.

To obtain word embeddings, the skip-gram model was employed. This is a language modelling and feature learning technique that maps words or phrases from the vocabulary to vectors of numerical values. Word embeddings are unsupervisedly learned word representation vectors whose relative similarities correlate with semantic similarity. The skip-gram model, one of the word2vec methods, includes the following steps [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Le | Distributed representations of sentences and documents[END_REF]:

• obtain a training dataset (sequences of words) w1, w2, … , wT;

• train the classifier and embedding function parameters;

• process each word wt in the vocabulary by applying embedding function to generate digital representation for every word in the vocabulary in high-dimensional space; • map every word in the vocabulary to digital representation of the word.

Negative dataset

Positive dataset feature IG feature IG "chicago" 0.123 "chicago" 0.090 "at the" 0.046 "location" 0.062 "luxury" 0.044 "floor" 0.052 "location" 0.043 "bathroom" 0.044 "-" 0.038 "on the" 0.041 "when i" 0.036 "small" 0.037 "chicago hotel" 0.034 "reviews" 0.037 "smell" 0.033 "luxury" 0.037 "my room" 0.033 "2" 0.037 "recently 0.030 "priceline" 0.035

The skip-gram model aims to find word representations that can be used to predict the context words in a sentence. The objective function of the skip-gram model is defined as follows:

𝐸 = 1 𝑇 ∑ ∑ log𝑝(𝑤 𝑡+𝑗 |𝑤 𝑡 ) -𝑐≤𝑗≤𝑐 𝑇 𝑡=1 , (2)
where w1, w2, … , wT is a sequence of training words, c is the size of context, and p(wt+1|wt) is defined using the hierarchical softmax (a binary tree representation of the output layer) as follows [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]:

𝑝(𝑤|𝑤 𝐼 ) = ∏ 𝜎(⟦𝑛(𝑤, 𝑗 + 1) = ch(𝑛(𝑤, 𝑗)⟧𝑣´𝑛 (𝑤,𝑗) 𝑇 𝑣 𝑤 𝐼 ) 𝐿(𝑤)-1 𝑗=1 , (3)
where wI are input words, vw and v'w are the input and output vector representations of word w, respectively, n(w,j) is the j-th node in the tree, L(w) is the length of the path from root node to word w, ch(n) is a child node of n chosen arbitrarily, ⟦𝑥⟧=1 if x is true, otherwise ⟦𝑥⟧=-1, and σ(x) is a sigmoidal function. Given the vocabulary size V, the computational complexity per training example per context word is O(log(V), which is a substantial improvement over the original softmax (O(V)).

The size of the word vectors (embeddings) was set to 500 and context size c=5 [7] to generate a complex representation. The average values of the vector were used to represent each review. Thus, the input attributes (features) for the subsequent supervised learning included 2000 n-grams and 500 embeddings.

Deep feed-forward neural network (DNN) was used to classify reviews into spam / legitimate categories. DNN enables processing the complex sparse representations of documents just like online reviews [START_REF] Barushka | Spam filtering using integrated distribution-based balancing approach and regularized deep neural networks[END_REF]. To reduce the risk of overfitting and avoid poor local error minima, we used dropout regularization and rectified linear units, respectively, as suggested in [START_REF] Barushka | Spam filtering using regularized neural networks with rectified linear units[END_REF]. To train the DNN, we used the mini-batch gradient descent algorithm that ensures stable convergence. This algorithm updates the synapse weights as follows: st+1 = st -η∇θJ(wt; x (i:i+n) ; y (i:i+n) ), [START_REF] Heydari | Detection of review spam: A survey[END_REF] where s is synapse weight, t is the iteration index, η is learning rate, J denotes an objective function, x i is the input, y i is the output of the i-the training example, and n is size of the mini-batch. The DNN was trained using different numbers of neurons in hidden layers = {10, 20, 50, 100} and different numbers of hidden layers = {1, 2, 3}. A grid search procedure was used to find the optimal DNN structure. Dropout rate for the input and hidden layers were set to 0.2 and 0.5, respectively, and the number of iterations was 1000, η = 0.1 and n = 100.

Experimental Results

While evaluating the experimental results, we took into consideration four evaluation measures: accuracy, area under ROC (receiver operating characteristic) curve (AUC), FN (false negative) rate and FP (false positive) rate. Accuracy represents the percentage of reviews correctly classified, FP rate stands for the percentage of spam reviews incorrectly classified as legitimate, while FP rate is the percentage of legitimate reviews misclassified as spam. Hereinafter, we present the averages and standard deviations of 100 experiments (10 times repeated stratified 10-fold cross-validation) performed on the two datasets. The n-gram and skip-gram models, as well as the experiments with the DNN method were performed in Deeplearning4j program environment.

To demonstrate the effectiveness of the proposed detection system, we also compared the results with several state-of-the-art approaches:

• Convolutional neural network (CNN) was used in [START_REF] Ren | Neural networks for deceptive opinion spam detection: An empirical study[END_REF][START_REF] Li | Document representation and feature combination for deceptive spam review detection[END_REF] as a basic CNN with unigrams and bigrams as inputs. Here we trained the CNN model using the mini-batch gradient descent algorithm with patch size 5×5 and max pool size 2×2, the remaining parameters were the same as for the DNN model; • Naïve Bayes represents a baseline classifier used in earlier research [START_REF] Li | Learning to identify review spam[END_REF];

• Support vector machine (SVM) represents another popular method used in previous literature on review spam detection [START_REF] Harris | Detecting deceptive opinion spam using human computation[END_REF][START_REF] Ott | Estimating the prevalence of deception in online review communities[END_REF][START_REF] Ott | Negative deceptive opinion spam[END_REF]. It was also used as a baseline method in recent studies [START_REF] Li | Bimodal distribution and co-bursting in review spam detection[END_REF]. In this study, SVM was trained using the SMO algorithm with varying complexity C = {2 0 , 2 1 , … , 2 6 } and polynomial kernel function; • Random Forest (RF) is another well-performing benchmark method used in several comparative studies [START_REF] Barushka | Spam filtering using integrated distribution-based balancing approach and regularized deep neural networks[END_REF][START_REF] Barushka | Spam filtering in social networks using regularized deep neural networks with ensemble learning[END_REF]. Here we used it with 100 random trees.

Note that all the baselines are suitable for review spam detection due to their qualities in handling sparse and high-dimensional data [START_REF] Barushka | Spam filtering using integrated distribution-based balancing approach and regularized deep neural networks[END_REF]. All the experiments with the comparative methods were run in Weka 3.8.2 program environment. The results of the conducted experiments are provided in Tables 2-5. To show the improvement in performance, we also compared the results with those obtained for the traditional n-gram model.

The results in Table 2 show that the n-gram+skip-gram approach performs well in terms of accuracy rate. All machine learning algorithms benefit from this combination except RF. It is important to note that the DNN approach achieves the highest accuracy rate and that the proposed method helps improve the performance of the n-gram model for both positive and negative polarity by about 2%. Compared with the baseline methods, the DNN method performed best for the negative dataset but CNN and SVM performed similarly at P<0.05 using the Wilcoxon signed rank test. For the positive dataset, DNN performed significantly better than the compared counterparts. What is interesting in this dataset is that the performance of DNN is 4% better than the baselines. These results corroborate those achieved in the original paper [START_REF] Ott | Negative deceptive opinion spam[END_REF] where the highest accuracies were 86.0% and 89.3% for the negative and positive dataset, respectively. However, this comparison must be interpreted with caution because different crossvalidation procedure was used in [START_REF] Ott | Negative deceptive opinion spam[END_REF]. In summary, Table 2 shows the superiority of DNN in review spam detection and substantial improvement achieved using the skipgram model over the baseline n-gram model.

The results in Table 2 demonstrate that the n-gram+skip-gram outperforms the ngram model in terms of FN rate except the RF method that, however, performed relatively poorly considering FP rate. In fact, FP rate is usually preferable to FN rate in the related literature [START_REF] Zhang | Binary PSO with mutation operator for feature selection using decision tree applied to spam detection[END_REF] due to the importance of retaining legitimate reviews and trustworthy users. From this perspective, DNN and CNN are the best methods. The most remarkable result to emerge from Table 3 and 4 is the balanced performance of DNN on both classes, legitimate and spam. As can be seen from Table 5, the DNN method performs well on both classes even when only the n-gram model is used. However, the additional semantic representation provided by the skip-gram model leads to further improvement. Among the compared baselines, CNN performs well on the negative dataset but only when the skip-gram model is included, suggesting that this method is more sensitive to both the polarity of the reviews and word context. In contrast, RF provides good performance on the positive dataset even with the baseline n-gram model. This can be explained by the fact that RF tends to apply features with many distinct values (embeddings, in this case), rather than those with few distinct values (unigrams and bigrams). As a result, the information contained in n-grams is not fully utilized in the n-gram+skip-gram model. In that sense, DNN achieves a more robust performance, with high AUC for both datasets. In addition, the DNN models integrating the n-gram and skip-gram features consistency outperforms those with n-grams only.

Conclusion

In this study, we demonstrated that using word embedding methods help achieve better results in review spam detection across state-of-the-art classification methods. The results show that the proposed content-based approach based on the DNN method performed best in terms of accuracy, FN and FP rate and AUC for the review spam dataset with positive polarity. Moreover, the proposed system demonstrated decent and balanced performance also for the dataset with negative polarity. Therefore, the proposed model can be recommended for review spam, irrespective of reviews' polarity. The results of the experiments suggest that DNN with word embeddings might be used in alternative document categorization tasks with sparse high-dimensional data, such as e-mail and social network spam detection. In further research, it would be interesting to include additional features, including the content of the other reviews for the same hotel. Moreover, it would be worth using the proposed approach in different review domains, rather than only hotel reviews. Additional important features could be extracted using attention-based recurrent neural networks that have recently been reported to be effective in social network spam filtering [START_REF] Chen | Call attention to rumors: Deep attention based recurrent neural networks for early rumor detection[END_REF]. Different languages represent another challenge for future research.

Fig. 1 .

 1 Fig. 1. The proposed architecture for spam review detection.

Table 1 .

 1 Top 10 features from the n-gram model in terms of information gain (IG).

Table 2 .

 2 Results of the experiments -accuracy.

		Negative dataset	Positive dataset
		n-gram	n-gram+skip-gram	n-gram	n-gram+skip-gram
	NB	80.36±3.12	81.75±3.02	82.88±3.12	84.38±2.96
	SVM	84.00±4.36	86.50±2.99*	81.63±5.04	84.50±2.71
	DNN	86.88±4.09*	88.38±3.12	87.25±2.99*	89.75±3.05
	RF	85.25±4.67	84.50±3.59	86.50±3.76	84.00±4.71
	CNN	82.25±3.67	88.13±4.57*	81.75±11.70	85.75±12.87
	* significantly similar performance as the best at P<0.05 (Wilcoxon signed rank test)

Table 3 .

 3 Results of the experiments -FN rate.

		Negative dataset	Positive dataset
		n-gram	n-gram+skip-gram	n-gram	n-gram+skip-gram
	NB	0.198±0.061	0.208±0.047	0.158±0.053	0.170±0.054
	SVM	0.155±0.028	0.138±0.041*	0.170±0.079	0.145±0.052
	DNN	0.140±0.047	0.115±0.038	0.128±0.048	0.103±0.038
	RF	0.118±0.044*	0.170±0.048	0.110±0.058*	0.135±0.050
	CNN	0.190±0.058	0.123±0.066*	0.130±0.074	0.180±0.290

* significantly similar performance as the best at P<0.05 (Wilcoxon signed rank test)

Table 4 .

 4 Results of the experiments -FP rate.

		Negative dataset	Positive dataset
		n-gram	n-gram+skip-gram	n-gram	n-gram+skip-gram
	NB	0.195±0.064	0.158±0.056	0.185±0.056	0.143±0.044
	SVM	0.165±0.068	0.133±0.049*	0.198±0.066	0.165±0.043
	DNN	0.123±0.056*	0.118±0.033*	0.128±0.049	0.103±0.034
	RF	0.178±0.077	0.140±0.036	0.160±0.050	0.185±0.068
	CNN	0.165±0.054	0.115±0.047	0.235±0.246	0.105±0.055*

* significantly similar performance as the best at P<0.05 (Wilcoxon signed rank test)

Table 5 .

 5 Results of the experiments -AUC. significantly similar performance as the best at P<0.05 (Wilcoxon signed rank test)

		Negative dataset	Positive dataset
		n-gram	n-gram+skip-gram	n-gram	n-gram+skip-gram
	NB	0.833±0.044	0.880±0.036	0.886±0.029	0.895±0.032
	SVM	0.840±0.044	0.865±0.030	0.816±0.050	0.845±0.027
	DNN	0.946±0.020*	0.956±0.013*	0.950±0.023*	0.956±0.025
	RF	0.924±0.033	0.925±0.025	0.943±0.023*	0.932±0.026*
	CNN	0.915±0.026	0.958±0.017	0.877±0.155	0.923±0.113

*

http://myleott.com/op-spam.html

Acknowledgments. This article was supported by the by the grant No. SGS_2019_17 of the Student Grant Competition.


Augusto J Guimarães
email: augustojunioguimaraes@gmail.com

Vinicius Araujo
email: vinicius.j.s.a22@hotmail.com

Vanessa S Araujo

Lucas O Batista
email: lobatista@outlook.com.br

Paulo Campos Souza
email: v.souzaaraujo@yahoo.com.br

Vinicius J Silva Araujo

Vanessa S Araujo

Paulo V De Campos Souza

A Hybrid Model Based on Fuzzy Rules to Act on the Diagnosed of Autism in Adults

Keywords: fuzzy neural network, autism in adults, fuzzy rules, extreme learning machine

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

A hybrid model based on fuzzy rules to act on the diagnosed of autism in adults.

Introduction

Autistic Spectrum Disorder (ASD) is characterized by deficiencies related to social communication and restricted, repetitive and stereotyped patterns of behaviors, interests or activities. Even present in people of the community, this disorder does not prevent that the person has an average life compared with the other people. Aspects of ASD can be noted in children with a delayed speech in isolated words or simple phrases, and 25% of children lose previously acquired language skills (regression) [START_REF] Goldberg | Language and other regression: assessment and timing[END_REF],even with the advancement of diagnosis in very young children, there is occasionally the recognition of the disorder in children at the beginning of their school life or deep reflexes during adolescence or in adulthood [START_REF] Le Couteur | Diagnosing autism spectrum disorders in pre-school children using two standardised assessment instruments: The adi-r and the ados[END_REF]. Since there is so little information, families fear that growth will bring new problems and can be high anxiety about how the child will cope when you can no longer care for them [START_REF] Eaves | Young adult outcome of autism spectrum disorders[END_REF]. Diagnosis can occur in people with early ages, adolescents or even adults, and may present symptoms and in several cases, these adults live with ASD years without specialized monitoring.

With the objective of identifying and producing techniques in the diagnosed of autism in children, adolescents and adults, Thabtah presents a review of techniques to facilitate the collection of information to present them and, consequently, a mobile application [START_REF] Thabtah | Asdtests. a mobile app for asd screening[END_REF] was developed to facilitate and disseminate diagnostic techniques for autism. The application that can be downloaded via mobile devices contains questions that help identify the characteristics of autism, becoming a popular form of data collection to form bases for the training of algorithms and more usual for detection of this type of diagnosis. The application allows parents to see if the traces of this mental illness are in their children. In order to use the techniques of diagnosed of autism in adults, they were used with the use of a decision tree, reaching between 95.85% and 95.73% accuracy on how the algorithm correctly classifies a patient with or without autism [START_REF] De Campos Souza | Using fuzzy neural networks for improving the prediction of children with autism through mobile devices[END_REF].

Many studies have been conducted to identify traits of autism in people through mobile devices, where the work done by [START_REF] Akyol | A study on autistic spectrum disorder for children based on feature selection and fuzzy rule[END_REF][START_REF] Büyükoflaz | Early autism diagnosis of children with machine learning algorithms[END_REF][START_REF] De Campos Souza | Using fuzzy neural networks for improving the prediction of children with autism through mobile devices[END_REF][START_REF] Demirhan | Performance of machine learning methods in determining the autism spectrum disorder cases[END_REF][START_REF]A machine learning based approach to classify autism with optimum behavior sets[END_REF][START_REF] Satu | Early detection of autism by extracting features: A case study in bangladesh[END_REF][START_REF] Thabtah | An accessible and efficient autism screening method for behavioural data and predictive analyses[END_REF][START_REF] Thabtah | A new computational intelligence approach to detect autistic features for autism screening[END_REF][START_REF] Thabtah | A new machine learning model based on induction of rules for autism detection[END_REF]. These approaches allow intelligent systems to be created to aid in the diagnosis of people who have such cognitive definitions. These original structures facilitate the decision-making of specialists in the correct diagnosis to the patients, with a decrease in risk of failure and taking into account the time provided to the patient in a future medical follow-up. The negative point was that studies that use decision trees (knowledge base), where it should be modeled each time a new factor is inserted in the evaluation of the problem. Other factors that can stand out as elements that do not contribute to the stability of the decision trees are the linear, and perpendicular limits to the axes and the high sensitivity to small perturbations in the training set generate very different networks.

In order to overcome these problems, specifically in the set of techniques in the diagnosed of autism in adults, this work proposes the use of fuzzy neural networks (FNN) in order to extract diffuse rules and maintain a high level of precision for the evaluated results , allowing the interpretability of the database to be extracted, comparing its results with machine learning algorithms that are provided by a tool developed in Java called WEKA [START_REF] Hall | The weka data mining software: an update[END_REF]. They can perform efficient training, extract information from the problem data and maintain high accuracy results. Do they effectively act in solving problems of various natures of science as in the pattern classification [START_REF] De Campos Souza | Regularized fuzzy neural networks based on nullneurons for problems of classification of patterns[END_REF][START_REF] De Campos Souza | Uninorm based regularized fuzzy neural networks[END_REF][START_REF] De Campos Souza | Data density-based clustering for regularized fuzzy neural networks based on nullneurons and robust activation function[END_REF][START_REF] Souza | Regularized fuzzy neural networks for pattern classification problems[END_REF][START_REF] Souza | Redes neurais nebulosas para problemas de classificação[END_REF], linear regression [START_REF] De Campos Souza | Fuzzy neural networks based on fuzzy logic neurons regularized by resampling techniques and regularization theory for regression problems[END_REF], time series forecasting [START_REF] De Campos Souza | Regularized fuzzy neural network based on or neuron for time series forecasting[END_REF], industry issues [START_REF] Lu | Generalized predictive control using recurrent fuzzy neural networks for industrial processes[END_REF],and also for problem-solving in healthcare [START_REF] Guimaraes | Using fuzzy neural networks to improve prediction of expert systems for detection of breast cancer[END_REF][START_REF] Guimarães | Using fuzzy neural networks to the prediction of improvement in expert systems for treatment of immunotherapy[END_REF] and software effort [START_REF] Souza | Regularized fuzzy neural networks to aid effort forecasting in the construction and software development[END_REF] and SQL Injection attack [START_REF] Batista | Fuzzy neural networks to create an expert system for detecting attacks by sql injection[END_REF][START_REF] Batista | Utilização de redes neurais nebulosas para criação de um sistema especialista em invasões cibernéticas[END_REF]. We can find the model used by the fuzzy neural networks concerning the diagnosed of autistic children [START_REF] De Campos Souza | Using fuzzy neural networks for improving the prediction of children with autism through mobile devices[END_REF] also with ASD. The paper is organized as follows: In section 2 are present the concepts of the literature that make the study carried out in this work, highlighting aspects of autism, a review of the concepts of intelligent hybrid systems. In section 3, the main concepts of the model used for the tests are presented. In section 4 the tests and the configurations of the same are presented to the reader so that finally, in section 5 the conclusion about the experiments and future works are highlighted in the paper.

2 Literature Review

Autistic Spectrum Disorder (ASD)

Autistic Spectrum Disorder (ASD) encompasses different syndromes marked by neurodevelopmental disorders with fundamental characteristics, which can be manifested together or in isolation. The difficulty of communication due to lack of mastery in speech, difficulty in social interaction and a pattern of restrictive and repetitive behavior [START_REF] Goldberg | Language and other regression: assessment and timing[END_REF]. A large proportion of individuals of all ages who require ASD treatment can still be found. When performing the early diagnosis, the following impacts on the life of this young person can be controlled with the accompaniment and treatment of professionals. However, it has always been difficult to obtain accurate data so that health professionals could efficiently diagnose the patients [START_REF] Thabtah | Machine learning in autistic spectrum disorder behavioral research: A review and ways forward[END_REF][START_REF] Wall | Use of machine learning to shorten observation-based screening and diagnosis of autism[END_REF]. Already in the adult phase, the diagnosis can be considered a rare event [START_REF] Lehnhardt | Psychosocial functioning of adults with late diagnosed autism spectrum disorders-a retrospective study[END_REF], however, it generates concern, associated with it can lead to other problems, such as the search for psychiatric consultations, which reveals a lifetime consultation rate of 78 %, more often with depression.

Fuzzy Neural Networks: Principal concepts

The fuzzy neural networks are a group of hybrid models that can unify in a model the interpretability characteristics of fuzzy systems and the capacity of training and generalization of parameters provided by artificial neural networks. These models can work harmonically to solve problems of linear regression, classification of patterns, diagnosed of time series and other problems of a specific nature with an excellent ability to interpret aspects of the user database of the problem, identifying existing patterns that make up a set of fuzzy rules capable of transforming numerical information into linguistic contexts that can be interpreted by people who do not necessarily know about the main techniques of artificial intelligence. This hybrid technique provides a solid knowledge base on the evaluated database, allowing fuzzy rules of the TSK-type [START_REF] Won | Parameter conditions for monotonic takagisugeno-kang fuzzy system[END_REF] to be obtained through the model. These techniques generally have at least one hidden layer, an output layer, which is also known as the defuzzification layer and the input layer, also called the fuzzification layer. In this fuzzy process, the data submitted to the model are sampled in the space of characteristics a through fuzzification techniques they are transformed into representations of the problem in the sample space. Usually, these approaches are made by cluster or grid algorithms [START_REF] Pedrycz | Fuzzy systems engineering: toward human-centric computing[END_REF].

3 Pruning Fuzzy neural network for detection of autistic persons

Network architecture

The architecture of the model follows the same definitions proposed by the paper [START_REF] Vitor De Campos Souza | Pruning fuzzy neural networks based on unineuron for problems of classification of patterns[END_REF]. It has three layers, where the first layer is responsible for the fuzzification process through the grid technique with equally spaced membership functions (which may be Gaussian or Triangular) [START_REF] Jang | Anfis: adaptive-network-based fuzzy inference system[END_REF]. Already second layer logical neurons type III do the aggregation of all the fuzzy neurons of the first layer. This logical neuron can extract fuzzy rules from the database, besides facilitating the aggregation and determination of more relevant rules. To filter the leastneeded rules for the model, a pruning approach is used in the second layer of the model. The concepts of f-scores applied in [START_REF] Gao | A novel approach for lie detection based on f-score and extreme learning machine[END_REF] make the selection of the most important rules for the model, making the model simpler and with a more cohesive architecture. In this model, the first two layers represent a fuzzy inference system, and the third layer is represented by a single artificial neuron capable of performing binary pattern classification tasks, which is the present context in this classification of adults with characteristics of autism. This single neuron can be seen as a singleton, and its weights are obtained through least squares parameterization techniques. The Extreme Learning Machine [START_REF] Huang | Extreme learning machine: theory and applications[END_REF] is present in the model to perform merely the definition of the synaptic weights that will compose the artificial neural network of aggregation in the third layer. The first layer is composed of neurons whose activation functions are membership functions of fuzzy sets defined for the input variables. This means that the creation of neurons in the first layer is an exponential relation of high computational cost since it is related between the number of membership functions chosen and the number of features of the base. As the base of autistic adult people has a high number of dimensions to be evaluated, a technique was applied that delimits the relation of neurons in the first layer in 500, where the relation of the number of functions of membership and dimensions is established in a probabilistic way, where not always for each dimension has precisely the number of membership functions chosen but rather the maximum value of M (in this case, for example, if M = 3 was chosen, it may be that in the first dimension only two membership functions and so on up to the limit of 500 relations). So for each input variable x ij , L fuzzy neurons (or membership functions) are defined A lj , l = 1, ... L whose membership functions are the activation functions of the corresponding neurons. Thus, the outputs of the first layer are the membership degrees associated with the input values, i.e., a jl = µ A l for j = 1... N and l = 1 ... L, where N is the number of inputs and L is the number of fuzzy sets for each input results by ANFIS [START_REF] Vitor De Campos Souza | Pruning fuzzy neural networks based on unineuron for problems of classification of patterns[END_REF] which through IF-THEN fuzzy rules incorporates the knowledge of an expert in a computational system through the database used in the research. It is implemented through a distributed parallel architecture in such a way that the learning paradigms can be harnessed allowing the integration of implicit (dataset) and explicit knowledge (common sense, preliminary knowledge or a specialist) for the resolution and adequate explanation of the problem [START_REF] Jang | Anfis: adaptive-network-based fuzzy inference system[END_REF].

The second layer is composed by L fuzzy orneurons, a fuzzy neuron composed of a fuzzy input, in this case, the neurons of the first layer and a numerical weight and bias, allowing the aggregation of several inputs with specific weights using s-norm and t-norm type fuzzy operators. Each neuron performs a weighted aggregation of some of the first layer outputs. This aggregation is performed using the weights w il (for i = 1... N and l = 1... L). For each input variable j, only one first layer output a jl is defined as input of the l -th neuron. So that w is sparse (that is, with many weights equal to zero), each neuron of the second layer is associated with an input variable. Finally, the output layer is composed of one neuron whose activation functions are ReLu [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF]. The output of the model is:

y = l j=0 f ReLU (z l v l ) ( 1
)
where z 0 = 1, v 0 is the bias, andz j and v j , j = 1, ..., l are the output of each fuzzy neuron of the second layer and their corresponding weight, respectively. Figure 1 presents an example of FNN architecture proposed in this paper. We can see that the first layer has fuzzy neurons produced through the relation between the dimensions and the number of membership functions chosen. In the second layer, the orneuron is highlighted and finally in the third layer a neuron of type ReLU. The architecture of this model is very similar to the one presented in [START_REF] Vitor De Campos Souza | Pruning fuzzy neural networks based on unineuron for problems of classification of patterns[END_REF], highlighting the changes in the second and third layers. ReLU is the most widely used activation function when treating nonlinearity in artificial neural networks today. First, the ReLU function is non-linear, which means that we can easily copy the errors back and have multiple layers of neurons activated by the ReLU function. In this case, only the third layer of the fuzzy neural network uses a neuron with the activation function [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF]. The main advantage of using the ReLU function over other activation functions is that it does not activate all neurons at the same time. This facilitates the application of the neurons of the second layer, discarding the activation of neurons not so significant for the responses of the model. This function acts in the process of defuzzification the model coherently and precisely [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF]. ReLU can be represented by:

f ReLU (z) = max(0, z) (2)
The logical neurons used in the second layer of the model are of the orneuron type what is a functional units able to perform multivariate nonlinear operations in unit hypercubes, where the input signals are individually combined with the weights and performed the subsequent global combining. The orneuron used in this paper can be expressed as [START_REF] Pedrycz | Neurocomputations in relational systems[END_REF]:

z = OR(w; z) = S n i=1 (w i tz i ) (3)
where T are t-norms (product), s is a s-norms (probabilistic sum). Fuzzy rules 4 can be extracted from orneurons according to the following example:

Rule 1 : If x i1 is A

Pruning method-F-score

Given the ith feature vector (in the case of the FNN the z-vector representing orneurons) with the number of positive instances n + , negative instances n + and the number of all the instances N, the F-score value of the i -th feature is defined by [START_REF] Gao | A novel approach for lie detection based on f-score and extreme learning machine[END_REF]:

F (i) = (x (+) i -x i ) 2 + (x (-) i -x i ) 2 1 n+-1 n+ k=1 (x (+) k,i -x (+) i ) 2 + 1 n--1 n- k=1 (x (-) k,i -x (-) i ) 2 , (5)
Where

x (+) i , x (-) i
and x i are the mean of the positive, negative and whole samples, respectively, and is the k -th feature value in the i -faith feature vector.

The numerator means the differentiation between the positive and negative sets, and the denominator is the sum of the deviation within each set of resources. This may even imply that the higher the F-score indicates that the resource has more discriminative power. We adopted the F-score method in this study due to the simplicity of its use and mainly in the pure determination of relevance of the fuzzy rules present in the second layer of the problem. This method is nonparametric, making it fast and less complicated to use in complex databases [START_REF] Gao | A novel approach for lie detection based on f-score and extreme learning machine[END_REF].

Training of the model based on Extreme Learning Machine.

Subsequently, following the determination of the network topology, the diagnoseds of the evaluation of the vector of weights' output layer are performed. The Moore-Penrose pseudo-Inverse [START_REF] Golub | Calculating the singular values and pseudo-inverse of a matrix[END_REF] estimates the weight vector was proposed by [START_REF] Huang | Extreme learning machine: theory and applications[END_REF] to facilitate the definition of intelligent model parameters through a single step. Extreme Learning Machine (ELM) concepts can be adapted in this FNN model as:

v = Z + y (6)
where Z + is the Moore-Penrose pseudo Inverse of z, which represents the outputs of the orneurons of the second layer. This approach is interpreted by the minimum norm of the least squares solution for the output weights.

This solution allows that the weights for the third layer aggregation neural network can act as a facilitator for the defuzzification process, allowing the transformed fuzzy representation in numerical outputs to be compared with the supervised training process, with the expected labels for the samples evaluated.

Tests

This section presents the settings for the identification test for adults with autism. Aspects related to the database, configurations, evaluation metrics and tools used in the tests will be presented below.

Assumptions and Initial Test Configurations

For the evaluation of the model presented in this paper,, 70 % of dataset is destined for the training and 30 % of the samples were used for validation with all combinations of M = [START_REF] Batista | Utilização de redes neurais nebulosas para criação de um sistema especialista em invasões cibernéticas[END_REF][START_REF] Büyükoflaz | Early autism diagnosis of children with machine learning algorithms[END_REF][START_REF] Vitor De Campos Souza | Pruning fuzzy neural networks based on unineuron for problems of classification of patterns[END_REF][START_REF] De Campos Souza | Using fuzzy neural networks for improving the prediction of children with autism through mobile devices[END_REF][START_REF] De Campos Souza | Fuzzy neural networks based on fuzzy logic neurons regularized by resampling techniques and regularization theory for regression problems[END_REF] in a cross-validation process evaluating the result that maximizes the training accuracy. The result of the cross-validation test evaluated the best value of accuracy, selecting the best value of M which will be used in comparative tests with other intelligent models. The outputs of the dabatase were normalized to -1 and 1 to aid the correct calculations. The factors evaluated in this paper are as follows: In order to evaluate the effectiveness of the FNN model, we performed tests in the WEKA [START_REF] Hall | The weka data mining software: an update[END_REF] Software, with the main classifier algorithms: MLP (Multilayer Perceptron) [START_REF] Rumelhart | Parallel distributed processing: explorations in the microstructure of cognition[END_REF], J48 (J48) [START_REF] Witten | Data Mining: Practical machine learning tools and techniques[END_REF], Naive Bayes (NB) [START_REF] Rish | An empirical study of the naive bayes classifier[END_REF], Zero Rule (ZR) [START_REF] Witten | Weka: Practical Machine Learning Tools and Techniques with Java Implementations[END_REF] and Random Tree (RT) [START_REF] Ho | Random decision forests. In: Document analysis and recognition[END_REF]. These models were applied to adult autism classification tests with the initial WEKA settings, also using 70% of the samples to train the model and 30% of them for validation. All samples were resampled, and a total of 30 replicates were also applied in each model.

Dataset used in the tests.

In sequence to extract the desired classifications of the models being tested, the database collected and made available by [START_REF] Thabtah | Asdtests. a mobile app for asd screening[END_REF], Is composed of tests of users aged 17 and over -adults -the database is formed by information and the result of 10 multiple choice queries, depending on the use of the mobile application, information from people from various countries of the world were collected. The data set gathers information from 701 people. As its characteristic, the FNN model, corresponds only to numerical type attributes, when necessary, literal attributes were changed to numeric. Database uses the following dimensions3 :

-Result Test = minimum of 0 and a maximum of 10 with mean in 4.89 and standard deviation of 2.49; -Age = minimum of 17 years and a maximum of 64 years with mean in 29. [START_REF] Hall | The weka data mining software: an update[END_REF] and standard deviation of 9.71 years; -Sex = 366 male and 49 female; -Ethnicity = 232 White-European, [START_REF] Lehnhardt | Psychosocial functioning of adults with late diagnosed autism spectrum disorders-a retrospective study[END_REF] In the tests, we can verify that the neural network models MLP and J48 present excellent results in the identification of patients with autism. Closely followed is the model proposed in this paper that has statistically equivalent results to the models that obtained the best results. The exciting thing is that the proposal of this paper can achieve statistically equal results and aggregate the answers to knowledge extraction from the database used in the test.

In general, the only model that can not be identified as a suitable classifier for this type of problem is the ZR. The others presented good results, but without highlighting a knowledge obtained from the problem.

The FNN model of this article worked with very good execution time, especially compared to MLP that has its same layered architecture structure. The weights found by ELM allow an evaluation of the importance of the rule for the evaluation of a patient. The following fuzzy rules 1 were derived from the model to exemplify the ability to abstract knowledge from a basis that the FNN possesses.

Conclusion

After performing the tests and extracting the fuzzy rules, we can conclude that the fuzzy neural network model presented in this paper can act as a tool to help physicians who diagnose adults with autism. Of course, the role of the specialist is crucial, but the more tools he has, the faster the diagnosis can be given, and the treatment can be started. Future work may address other fuzzy neural network architectures, broaden the assessment of people of different ages, and seek confirmation of the rules drawn from the problem with a specialist in the field.

Fig. 1 .

 1 Fig. 1. Fuzzy Neural Network Architecture

  accuracy =T P + T N T P + F N + T N + F P P = true positive, T N = true negative, F N = false negative and F P = false positive.

Table 1 .

 1 Latino, 43 Black, 123 Asian, 92 Middle-Eastern, 11 Pasifika, 36 South Asian, 13 Hispanic, 6 Turkish and 121 Other; -Jundice = 632 not have jundice and 69 have the symptoms of the disease; -Autism in Family = 610 not and 91 yes; -Country = 67 countries, with a predominance of people in United States,The table 1 the pattern classification results for the adult autism dataset. Also unique are the tables that present the accuracy and final neurons after pruning and time is in seconds. The results highlighted in the table are the best results obtained in the test or equivalents statistically within the range of standard deviation analyzed. Accuracies of the model in the tests performed.

	New Zealand, United Arab Emirates, United Kingdom and India;
	-Used App Before = 689 not and 12 yes;
	-Relationship With The Patient = 521 Self, 50 Parent, 4 Health profes-
	sional, 28 Relative, 98 Others;
	-The outcome of the treatment = 512 results of people without ASD and
	189 people who present the characteristics of the disorder.

The original database can be viewed in https://archive.ics.uci.edu/ml/datasets/Autism+Screening+Adult


Siva Krishna Dasari

Abbas Cheddad

Petter Andersson

Random Forest Surrogate Models to Support Design Space Exploration in Aerospace Use-case

Keywords: machine learning, random forests, hyperparameter tuning, surrogate model, meta-models, engineering design, aerospace

In engineering, design analyses of complex products rely on computer simulated experiments. However, high-fidelity simulations can take significant time to compute. It is impractical to explore design space by only conducting simulations because of time constraints. Hence, surrogate modelling is used to approximate the original simulations. Since simulations are expensive to conduct, generally, the sample size is limited in aerospace engineering applications. This limited sample size, and also non-linearity and high dimensionality of data make it difficult to generate accurate and robust surrogate models. The aim of this paper is to explore the applicability of Random Forests (RF) to construct surrogate models to support design space exploration. RF generates meta-models or ensembles of decision trees, and it is capable of fitting highly nonlinear data given quite small samples. To investigate the applicability of RF, this paper presents an approach to construct surrogate models using RF. This approach includes hyperparameter tuning to improve the performance of the RF's model, to extract design parameters' importance and if-then rules from the RF's models for better understanding of design space. To demonstrate the approach using RF, quantitative experiments are conducted with datasets of Turbine Rear Structure use-case from an aerospace industry and results are presented.

Introduction

In aerospace industry, engineers develop highly complex composite and metallic engine structures, fan cases and exhaust systems. The early design phase of these components and their performance estimation is a complex multidisciplinary problem, and it involves analyzing the effects of aero performance, mechanical functions and producibility aspects. For this, simulations play a vital role to help to have a better understanding of the functional behaviour and to predict possible failure modes in design concepts. However, high-fidelity simulations can take significant time to compute. Typically, a large system has millions, if not billions of possibilities to explore in the design space [START_REF] Kang | An approach for effective design space exploration[END_REF]. Thus, it is impractical to explore the design space by conducting simulations for all possible design concepts due to the time constraints. Hence, to minimize the number of simulations which is needed to investigate a certain design space, different approaches are used. One of the approaches is surrogate modelling which is also known as response surface modelling and metamodeling [START_REF] Queipo | Surrogate-based analysis and optimization[END_REF] [START_REF] Mack | Surrogate model-based optimization framework: a case study in aerospace design[END_REF].

Surrogate models are used to approximate the time-consuming simulations by mimicking the complex behaviour of the underlying simulation model. This provides a great opportunity to explore as many as design concepts without needing more computationally expensive simulations. The generation of surrogate models requires a dataset of inputs and known outputs. Since simulations are expensive to conduct, datasets are usually small in real-world context. The size of the data sets and the complexity of the underlying simulation model make it difficult to generate accurate enough and robust surrogate models. Thus, the challenge is to generate a model as accurate as possible with a small size of datasets for design space exploration. This motivates to study surrogate modelling and methods that can be helpful to construct surrogate models as accurate as possible. For instance, in many studies, the applicability of widely used methods such as Support Vector Machines (SVM), Artificial Neural Networks, Radial Basis Function, Kriging methods, and Linear Regression (LR) have been investigated to construct surrogate models [START_REF] Amouzgar | Radial basis functions with a priori bias as surrogate models: A comparative study[END_REF] [5] [START_REF] Gorissen | A surrogate modeling and adaptive sampling toolbox for computer based design[END_REF] [17] [START_REF] Simpson | Comparison of response surface and kriging models in the multidisciplinary design of an aerospike nozzle[END_REF].

Furthermore, the performance of two tree methods (Random Forests and M5P) have been investigated for surrogate modelling and concluded that these tree models performed as similar to the widely used methods such as LR and SVM [START_REF] Dasari | Tree-based response surface analysis[END_REF]. The authors of this study stated that tree model could provide comprehensibility (if-then rules) that could be used to interpret model behaviour. Though tree models have been suitable and have shown to have a better performance in other applications, they have been rarely used for surrogate modeling in design engineering. Furthermore, Random Forests (RF) generate ensembles of decision trees, and they are capable of fitting highly non-linear data given quite small samples [START_REF] Biau | A random forest guided tour[END_REF] [START_REF] Menze | On oblique random forests[END_REF]. Therefore, we choose a tree method, RF, to explore its applicability for surrogate modeling. The aim of this paper is to support design space exploration of Turbine Rear Structure (use-case) using RF generated surrogate models. For this, we present an approach that aims (1) to construct surrogate models by RF (2) to improve RF's model performance using hyperparameter tuning method (3) to extract design parameter importance and rules from RF's generated surrogate models. We present related work in Section 2, our approach in Section 3, experiments in Section 4, results and analysis in Section 5, and conclusions in Section 6.

Related Work

In this section, we present related work on several methods that have been used to construct surrogate models. We found that Kriging and Polynomial methods are commonly used to construct surrogate models [START_REF] Simpson | Comparison of response surface and kriging models in the multidisciplinary design of an aerospike nozzle[END_REF] [20] [START_REF] Carley | Response surface methodology[END_REF]. Mack et al. have used Polynomial response surface approximation for multi-objective optimization of a compact liquid-rocket radial turbine in aerospace application [START_REF] Mack | Surrogate model-based optimization framework: a case study in aerospace design[END_REF].

Thorough investigations of Polynomial Regression, Multivariate Adaptive Regression Splines, Radial Basis Functions, Kriging and Bayesian Neural Networks were conducted, and some observations were presented in the following studies [START_REF] Zhao | A comparative study of metamodeling methods considering sample quality merits[END_REF] [START_REF] Jin | Comparative studies of metamodelling techniques under multiple modelling criteria[END_REF]. For instance, a drawback of Polynomial models is that they have limited flexibility and need prior knowledge of underlying functions [START_REF] Sathyanarayanamurthy | Metamodels for variable importance decomposition with applications to probabilistic engineering design[END_REF].

In another study, the authors used Random Forests (RF) to construct surrogate models for design optimization of a car and concluded that RF is performing as well as Kriging method [START_REF] Preuss | High-dimensional model-based optimization based on noisy evaluations of computer games[END_REF]. Though Kriging is another well studied method for surrogate model generation, the use of Kriging method is not trivial to construct surrogate models due to its global optimization process [START_REF] Wang | Review of metamodeling techniques in support of engineering design optimization[END_REF] for non-experts who are unaware of Kriging methodology. On the other hand, the construction of RF is not as difficult as Kriging but still shows an equal performance. RF was also compared with Support Vector Machines, M5P and Linear Regression and the authors observed that RF is performing as well as these methods [START_REF] Dasari | Tree-based response surface analysis[END_REF]. Though RF models have been shown a better performance for these two studies and also other applications, they have been rarely used for surrogate modeling in design engineering. These observations motivate us to study RF applicability for our design study problem.

Furthermore, we focused on related work on how to improve the performance of RF, and we found that hyperparameter tuning of RF could help achieve that [START_REF] Biau | A random forest guided tour[END_REF] [8] [START_REF] Liaw | Classification and regression by randomforest[END_REF]. Regarding this, Oshiro et al. analysed the performance of Random Forests by tuning the number of trees (Ntree) for classification tasks [START_REF] Oshiro | How many trees in a random forest?[END_REF]. The authors provided insights about Ntree (threshold configurations). Diaz et al. also mentioned about hyperparameters of RF that increasing Mtry can lead to a small decrease in error rate and vice versa [START_REF] Díaz-Uriarte | Gene selection and classification of microarray data using random forest[END_REF]. Similar to these studies, we focus on tuning hyperparameters of RF for regression instead of classification tasks.

Terminology: A design concept (also called an instance) is a set of design parameters (also called variables or inputs) and design objectives (also called outputs) that explain the design of Turbine Rear Structure (TRS) component. In every design concept, the design variables and objectives are the same, but the values vary. Design space exploration is investigating alternative design concepts in order to find an optimal design concept that fulfill design requirements and constraints prior to implementation. The term hyperparameter refers to the characteristic of a method. For instance, Ntree and Mtry are the hyperparameters of the RF method.

3 An Approach to Explore Design Space using Random Forests Surrogate Models

In this section, we present our approach, that is shown in Fig. 2, to explore the design space of Turbine Rear Structure through RF generated surrogate models.

Use-case: A Design Study of Turbine Rear Structure

The geometry model of Turbine Rear Structure (TRS) is shown in Fig. 1, and it is a component of aircraft engine. TRS is a complex component of engine which involves multidisciplinary studies that address the behaviour of the design from mechanical, aerothermal and producibility aspects. For better understanding of design space of TRS, engineers divide design space into several design studies incrementally to evaluate different aspects of the proposed design, piece by piece unveiling behaviour and constraints. In this study, we use data from one of TRS design studies that focus on investigating four design parameters of hub configuration (inputs) such as hub rear stiffener height, forward hub wall angle, hub knee point radial position and bearing flange axil position shown in Fig. 1, to get better understanding about them. These design parameters are studied together with 17 other parameters related to thickness groups (engine mount, hub surface, mount sectors etc), turbine vanes and outer case. In total, 21 design parameters are used to create computer aided design (CAD) models which are analysed using finite element analysis. The outputs (design objectives) from this analysis are shown in Table 1. For this design study, a total of 56 design concepts were generated from finite element analysis. These 56 design concepts are too small to understand the design space of parameters. At the same time, it is time-consuming to perform finite element analysis on more design concepts. Therefore, surrogate models are constructed to analyse more design concepts in order to reduce the number of simulations that are needed for design space exploration.

Surrogate Model Generation

In surrogate modelling, the aim is to determine a continuous function f (model) of a set of design variables x = x 1 , x 2 , ..., x n from a limited amount of available data D (shown in Fig. 2 (a)). The available data D represent the exact evaluations of the function f , and in general cannot carry sufficient information to uniquely identify f . Thus, surrogate modelling deals with two problems which are constructing a model f from the available data D, and evaluating the error ε of the model [START_REF] Mack | Surrogate model-based optimization framework: a case study in aerospace design[END_REF]. The prediction of the simulation based model output using surrogate modelling approach is formulated as follows:

f (x) = f (x) + ε(x) (1)
Where f (x) is the predicted output and ε(x) is the error in the prediction. The construction of response surface model involves several steps (shown in Fig. 2 We used Random Forest to construct surrogate models and the brief description of it is as follows: Random Forests (RF) is an ensemble method that is a combination of multiple methods, and can handle nominal, categorical and continues data, thus, it is used for both classification and regression [START_REF] Breiman | Random forests[END_REF]. RF contains several decision trees. Each tree in the forest represents a model. It is built using a deterministic algorithm by selecting a random set of variables and random samples from the training set. Two hyperparameters of RF are needed to build a forest: Ntreethe number of trees to grow in the forest based on a bootstrap sample of observations, and Mtry -a number of features which are randomly selected for all split in the tree. The training procedure of RF is as follows (a brief visual representation is shown in Fig. 2 (b)):

1. From a dataset D, draw a bootstrap sample D randomly with replacement for each tree construction. 2. Build a tree T using the bootstrap sample, at each node, choose the best split among a randomly selected subset of Mtry descriptors. The tree is constructed until no further splits are possible or reaching given Node size limit. 3. Repeat the 2 nd step until user defined number of trees is reached.

When choosing the bootstrap samples to build a tree, some of the training data may be repeated and some of the samples are left out. These left out data samples are called out-of-bag samples. For RF model generation, two thirds of all training samples are used for deriving the regression function whereas one third forms the out-of-bag samples. A regression tree is built using randomly selected training samples, and out-of-bag samples are used to test for the accuracy of the tree.

Improve Random Forests Model Performance

We found that one of the ways to improve the performance of a model is hyperparameter tuning [START_REF] Biau | A random forest guided tour[END_REF] [8] [START_REF] Liaw | Classification and regression by randomforest[END_REF]. We use the same method to improve the performance of RF models, and the procedure is as follows. Firstly, two hyperparameters and a set of parameter configuration values are selected for parameter tuning of RF. The detailed descriptions of parameters and configurations are explained in section 4.2 (Hyperparameters and Configuration Selection). Secondly, experiments are conducted to construct surrogate models using our design study data with those parameter configurations of RF. Finally, Root Mean Square Error (RMSE) is calculated for the prediction error rate of the surrogate model. The RMSE is calculated as the sum of squared differences of the predicted values and the actual values of the regression variable divided by the number of predictions as shown in equation 2 [START_REF] Witten | Data Mining: Practical machine learning tools and techniques[END_REF]. Where ŷ is the predicted value and y is the actual value.

RM SE

= 1 N n i=1 (ŷ -y) 2 (2)

Parameters Importance and Rules from RF Surrogate Models

As mentioned earlier in Section 3.2, numerical simulations use a black-box function to evaluate design points. Hence, the mapping procedure from input to output parameters are hard to interpret. Hence, if we can provide some information regarding this mapping procedure, it could help to understand the design space better and make informed decisions about design parameters. This is one of the reasons to select the RF method to construct surrogate models, because it provides parameter importance. Furthermore, it can provide rules that could help to understand the design space. For instance, we can observe where the first split is on design space and which design parameter is used for that split. In order to extract information from RF, we use the InTrees framework which is presented in [START_REF] Deng | Interpreting tree ensembles with intrees[END_REF]. The InTrees framework extracts information from the trees in the form of if-then rules, and prunes redundant rules and leaves the non-redundant rules in the forest. These rules may provide some information regarding the prediction behaviour to understand the relationship between input and output parameters.

An example of the if-then rule is shown in Fig. 2 (c).

Decision Support

The end goal of our design study application is to analyse design parameters (inputs) by exploring design concepts, and then to get some insights about them.

For this, we use Random Forest surrogate models as a support in decision making by [START_REF] Amouzgar | Radial basis functions with a priori bias as surrogate models: A comparative study[END_REF] predicting the value of design objectives for as many design concepts as possible to reduce computational expensive simulations needed to explore the design space ( 2) getting insights about the importance of design parameters towards the design objectives, and when needed, reduce number of parameters to narrow down its space for further analysis, and (3) extracting if-then rules that could help to better understand the reasoning of prediction behavior of the RF surrogate model.

Experimental Design

This section presents the experimental design to construct surrogate model by Random Forests.

Dataset Descriptions

We have a dataset D1 from our design study which contains 56 design concepts (instances). Each design concept describes a possible design of TRS which is a component of an aircraft engine. The design concepts are selected from the design space of a turbine vane using Latin hypercube sampling technique. Later, simulations were conducted using selected design concepts to get their design objectives (outputs). The 56 concepts contain 21 design variables (Section 3.1) and 14 design objectives with continuous values. We use these simulated design concepts to construct surrogate models. Since we have 14 outputs, we divided the dataset into 14 sub-datasets as single output design concept datasets. This means that we have an independent RF model for each output, hence 14 models.

We have another dataset D2 related to TRS with 410 design concepts with three more outputs (D2-15, D2-16 and D2-17). Table 1 (second row) shows the description of design objective or outcome of TRS for all datasets. We used in total, 17 datasets for experiments, and we named them as D1-1, D1-2. . . , D2-17.

Hyperparameters and Configuration Selection

We selected Mtry and Ntree hyperparameters to tune RF. The motivation is, these parameters have an effect on model performance. For instance, a previous study states that increasing the size of tree (Ntree) can decrease the forest error rate, and decreasing number of random features (Mtry) can reduce the strength of a tree (increases the error rate) [START_REF] Oshiro | How many trees in a random forest?[END_REF]. Threshold range was provided for Ntree which is 1 to 128 to increase accuracy for classification tasks [START_REF] Oshiro | How many trees in a random forest?[END_REF]. We also selected a narrower range of Ntree configuration values between 10 and 130 with step size 10. We select Mtry configuration values from minimum to maximum number of input features from datasets with step size 2. For instance, for n = 5 input features, the Mtry configurations are 1, 3 and 5. We automatized hyperparameter tuning using the following procedure to get a model with a better RMSE.

1. Set configuration values for Mtry and Ntree. The maximum Mtry value should be n-1 where n is the number of features in the datasets. 2. Generate possible configurations using Mtry and Ntree.

3. Construct a model with single output dataset using RF. 4. Repeat step 3 for each possible hyperparameter configuration value. 5. Evaluate the performance of model for each hyperparameter configuration value. 6. Select the model, which has least RMSE. 7. Repeat steps 1-6 for every single output dataset.

Experiment

We used the selected hyperparameter configuration values (Ntree 10 to 130 with step size 10 and Mtry from 1 to 21 with step size 2) to generate possible combinations of configurations for all selected datasets. We have 143 possible parameter configurations for datasets D1-1 ..., D1-13 and 130 for D2-14 ..., D2-17. Later, we conducted the experiment to construct surrogate models using RF by tuning the selected hyperparameter's configurations (Ntree and Mtry) with 10 fold cross-validation [START_REF] Kohavi | A study of cross-validation and bootstrap for accuracy estimation and model selection[END_REF]. We measured RMSE to evaluate the prediction accuracy. We used R software environment and RF packages3 to conduct experiments.

Results and Analysis

In this section, we present the prediction error (RMSE) of RF surrogate models in order to predict the value of design objective (output) for as many design concepts (inputs) as possible, and to reduce the number of expensive simulations that are needed for design space exploration. Table 1 shows RMSE for RF with default parameter configurations (Ntree = 100 and Mtry = 5), and tuned RF. The results show that tuned RF yields the best results compared to RF with default configurations for all datasets. Furthermore, we compared our study results to those reported in [START_REF] Dasari | Tree-based response surface analysis[END_REF] for the same datasets, and we found that tuned RF performs better than Support Vector Machines, Linear Regression and M5P.

In this paper, we included RMSE results (Table 1) for one Mtry configuration which gave the best results compared to the other configurations. Due to the page limit, the RMSE results for the rest of Mtry configurations are available on the online web link 4 . These RMSE results show that the prediction error is decreased by hyperparameter tuning, and we observed that Mtry has influence on the prediction error. We can support our observation on Mtry with Diaz et al.'s study observations on Mtry that shows similar effects [START_REF] Díaz-Uriarte | Gene selection and classification of microarray data using random forest[END_REF], however, for classification tasks. Regarding Ntree, we observed that it has little influence on the performance. Hence, the threshold values of Ntree which are provided for classification tasks [START_REF] Oshiro | How many trees in a random forest?[END_REF] and also for our configurations (Ntree: between 10 and 130) can be recommended for regression tasks too.

In order to test if Mtry has significant influence on the prediction error, we conducted the Friedman statistical test [START_REF] Sheskin | Handbook of parametric and nonparametric statistical procedures[END_REF] to test the null hypothesis which is there is no significant difference between Mtry parameter values and predictive performance. From the Friedman statistical test, we got p-value which is less than 0.05 significant level, hence, rejecting the null hypothesis. Thus, there is a significant performance difference between different Mtry values. In addition, we conducted the post hoc Nemenyi test for pair wise comparison to see individual differences. These pair wise statistical results indicate that there is significant difference between larger Mtry configurations and smaller Mtry configurations. The post hoc Nemenyi test results are also available on the same online link 4 .

As a summary of the analysis, we see that automated hyperparameter tuning helped us to get a model with better performance. Also, it only took between 2 to 3 minutes to run all hyperparameter configurations. The system we used is a 64-bit Windows 7 Operating System with 2.7 GHz Intel Core i7-4600U CPU and 8 GB RAM. Since we have small sample size-and also in general, surrogate models usually are built using small size of samples-of datasets, we believe this automated procedure has a reasonable time complexity. Thus, this procedure is recommended for design engineers when they construct surrogate models with small sample sizes using RF for hyperparameters tuning to get a model with better prediction accuracy.

Parameters Importance and Rules Extraction

In this section, we present how to get variable importance and rules from RF surrogate models. For this, we focus on one of the design objectives of the turbine rear structure (TRS) which is TRS mass (D1-11 dataset in Table 1). The measure that we used to extract parameter's importance is the increase in mean squared error (MSE) as a result of a parameter permutation (more details on parameters importance can be found in [START_REF] Breiman | Random forests[END_REF]). Hence, the higher value indicates that the parameter has a high importance on the model performance (building model without this parameter causes an increase in the prediction error). We have used a total of 21 design parameters (inputs) and the mass of TRS (design outcome or output) to build the surrogate model by RF, and the importance extracted from the surrogate model is plotted in Figure 3. The x-axis represents the number of design parameters and the y-axis represents the importance of each of these variables. Due to the confidentiality of design parameters, we did not include their names in the plot. Fig. 3 provides an idea on different design parameters 'contribution on model performance, and this analysis could help to further explore the space of design parameters. For example, we observed that parameters that are related to forward hub wall angle and hub cone mount have high importance, and inner ring reg and mount have low importance compared to other parameters. Further analysis of this example would be tune parameter configurations of high or low importance and then predict the outcomes for a better understanding of these parameters.

Furthermore, we extracted if-then rules from surrogate models for TRS mass. For a better understanding, we discretize the outcome of TRS mass into three levels with intervals (low, medium and high). There are 7 rules for low mass, 9 rules for medium and 8 rules for high mass of TRS. By looking at these rules, we can see the reasoning for the predictions that cause the low or high mass of TRS. For example, if we want to focus on minimising the mass of TRS, we use the 7 rules (1) to understand design parameters and their configurations' contribution in predictions, (2) to narrow down the design regions (3) and then to generate more design concepts within that region to analyse the design outcome. We believe that this analysis (extracting if-then rules from surrogate models) helps to understand the design space better. Consequently, it helps to find the design concepts that minimise or maximise the mass of TRS.

Conclusions and Future work

In this paper, we investigated the performance of Random Forests (RF) for surrogate modeling to support the design space exploration of Turbine Rear Structure. In order to get accurate surrogate models, we conducted hyperparameter tuning for RF. The use-case results show that the prediction performance of the model is improved by hyperparameter tuning. Additionally, we observed in our use-case results, Mtry hyperparameter has an influence on increasing the performance. Hence, our automatized hyperparameter tuning method and threshold values are recommended when using RF to get accurate surrogate models. Furthermore, we extracted design parameters' importance and if-then rules from RF generated surrogate models to better understand the design parameters and design space of Turbine Rear Structure. In future work, we will focus on investigating RF's applicability for other tasks such as sensitivity analysis and design optimization.

Fig. 1 .

 1 Fig. 1. Left: Turbine Real Structure case (TRS), Middle: Hub configuration of TRS (with other aircraft engine parts), and Right: Four design parameters of hub configuration (a more detailed from middle picture)

3 .Fig. 2 .

 32 Fig.2. An approach to explore design space using Random Forest (RF) surrogate models: (a) and (b) show the construction of surrogate models and a brief visual representation of RF, and (c) shows an example of an if-then rule extracted from surrogate models.

  )[START_REF] Queipo | Surrogate-based analysis and optimization[END_REF]: (1) Design of experiments: A set of design variables (x = (x 1 , x 2 , ..., x d ) T , T means transpose of vector) and their values are selected (2) Numerical simulations: Let f be the black-box function (simulations), evaluate f on design points y i = f (x i ) where x i ∈ R d and y i ∈ R (3) Construction of response surface model: Consider the data D = {(x 1 , y 1 ), . . . , (x n , y n )}, given the data, a continuous function f is determined to evaluate new design point ŷ = f ( xi ) (4) Model validation: Assessing the predictive performance of f from the available data D.

Fig. 3 .

 3 Fig. 3. Parameters Importance from RF model

Table 1 .

 1 Description of output from TRS design study, and RMSE performance comparison of default RF and tuned RF on 17 datasets

	Data Set	Output Description	RMSE Default RF Tuned RF
	D1-1	Related to Angles for Turbine Hub and	2.0553	0.0296
	D1-2	Cone	10.4724	3.5116
	D1-3		0.3155	0.2185
	D1-4		0.0171	0.0061
	D1-5		0.2720	0.1658
	D1-6	Related to Measure Points for the	0.0151	0.0056
	D1-7	Tolerance Stability to Weld	0.1714	0.1297
	D1-8		0.0459	0.0118
	D1-9		0.1067	0.0791
	D1-10		0.0464	0.0121
	D1-11	TRS Mass	5.5322	3.4871
	D1-12	Related to Fan Blade out	0.0254	0.0223
	D1-13	Load Cases	0.0482	0.0298
	D1-14	Welding Life	0.1248	0.1169
	D2-15	Related Stress at Leading Edge and	0.0602	0.0462
	D2-16	Trailing Edge	0.0757	0.0466
	D2-17	Time Step Trailing Edge	1.2752	0.7535

RF: https://cran.r-project.org/web/packages/randomForest/ randomForest.pdf

https://github.com/dasSiv/Mtry-analysis.git

Acknowledgments This work is part of the research project "Model Driven Development and Decision Support" funded by the Knowledge Foundation (grant: 20120278) in Sweden.