Nestoras Georgiou

Andreas Konstantinidis

Harris Papadopoulos
email: h.papadopoulos@frederick.ac.cy

Malware Detection with Confidence Guarantees on Android Devices

Keywords: Malware Detection, Android, Security, Inductive Conformal Prediction, Confidence Measures, Multilayer Perceptron

The evolution of ubiquitous smartphone devices has given rise to great opportunities with respect to the development of applications and services, many of which rely on sensitive user information. This explosion on the demand of smartphone applications has made them attractive to cybercriminals that develop mobile malware to gain access to sensitive data stored on smartphone devices. Traditional mobile malware detection approaches that can be roughly classified to signature-based and heuristic-based have essential drawbacks. The former rely on existing malware signatures and therefore cannot detect zero-day malware and the latter are prone to false positive detections. In this paper, we propose a heuristic-based approach that quantifies the uncertainty involved in each malware detection. In particular, our approach is based on a novel machine learning framework, called Conformal Prediction, for providing valid measures of confidence for each individual prediction, combined with a Multilayer Perceptron. Our experimental results on a real Android device demonstrate the empirical validity and both the informational and computational efficiency of our approach.

Introduction

The widespread deployment of smartphone devices has brought a revolution in mobile applications and services that span from simple messaging and calling applications to more sensitive financial transactions and internet banking services. As a result, a great deal of sensitive information, such as access passwords and credit card numbers, are stored on smartphone devices, which has made them a very attractive target for cybercriminals. More specifically, a significant increase of malware attacks was observed in the past few years, aiming at stealing private information and sending it to unauthorized third-parties.

Mobile malware are malicious software used to gather information and/or gain access to mobile computer devices such as smartphones or tablets. In particular, they are packaged and redistributed with third-party applications to inject malicious content into a smartphone and therefore expose the device's security. While the first one appeared in 2004 targeting the Nokia Symbian OS [START_REF]Cabir, smartphone malware[END_REF], in the fourth quarter of 2015 G DATA security experts reported discovering 8,240 new malware applications on average per day and a total of 2.3 million new malware samples in 2015, in just the Android OS [START_REF] Data | [END_REF]. When malware compromises a smartphone, it can illegally watch and impersonate its user, participate in dangerous botnet activities without the user's consent and capture user's personal data.

Mobile malware detection techniques can be classified into two major categories: static analysis [START_REF] Christodorescu | Static analysis of executables to detect malicious patterns[END_REF][START_REF] Shabtai | Detection of malicious code by applying machine learning classifiers on static features: A state-of-the-art survey[END_REF] and dynamic analysis [START_REF] Egele | A survey on automated dynamic malware-analysis techniques and tools[END_REF][START_REF] Rieck | Learning and classification of malware behavior[END_REF][START_REF] Moskovitch | Detection of unknown computer worms based on behavioral classification of the host[END_REF]. The former aim at detecting suspicious patterns by inspecting the source code or binaries of applications. However, malware developers bypass static analysis by employing various obfuscation techniques and therefore limiting their ability to detect polymorphic malware, which change form in each instance of the malware [START_REF] Moser | Limits of static analysis for malware detection[END_REF]. Dynamic analysis techniques on the other hand, involve running the application and analyzing its execution for suspicious behavior, such as system calls, network access as well as file and memory modifications. The main drawback of these techniques is that it is difficult to determine when and under what conditions the malware malicious code will be executed.

Both static and dynamic analysis techniques are typically implemented following two main approaches: signature-based approaches, which identify known malware based on unique signatures [START_REF] Griffin | Automatic generation of string signatures for malware detection[END_REF], and heuristic based approaches, which identify malicious malware behaviour based on rules developed by experts or by employing machine learning techniques [START_REF] Jacob | Behavioral detection of malware: from a survey towards an established taxonomy[END_REF][START_REF] Shabtai | Detection of malicious code by applying machine learning classifiers on static features: A state-of-the-art survey[END_REF][START_REF] Moskovitch | Detection of unknown computer worms based on behavioral classification of the host[END_REF][START_REF] Menahem | Improving malware detection by applying multi-inducer ensemble[END_REF]. Even though signaturebased techniques have been successfully adopted by antivirus companies for malware detection in desktop applications, this is not a preferred solution in the case of mobile devices due to their limited available resources in terms of power and memory. Additionally, signature-based techniques cannot detect zero-day malware (not yet identified) or polymorphic malware, something that is not an issue for heuristic based techniques. On the other hand, unlike signature-based techniques, heuristic based techniques are prone to false positive detections (i.e. wrongly identifying an application as malware).

Most recent research studies focus on extending the idea of heuristic-based approaches by employing machine learning techniques. For example, Sahs and Khan [START_REF] Sahs | A machine learning approach to android malware detection[END_REF] use a one-class Support Vector Machine to detect malicious applications based on features extracted from Android Application Packages (APKs) of benign applications only. Demertzis and Iliadis [START_REF] Demertzis | Bio-inspired hybrid intelligent method for detecting android malware[END_REF] propose a hybrid method that combines Extreme Learning Machines with Evolving Spiking Neural Networks using features extracted from the behaviour of applications when executed on an emulated Android environment. In [START_REF] Demertzis | Same: An intelligent anti-malware extension for android art virtual machine[END_REF] the same authors propose an extension to the Android Run Time Virtual Machine architecture that analyses the Java classes of applications using the Biogeography-Based Optimizer (BBO) heuristic algorithm for training a Multilayer Perceptron to classify applications as malicious or benign. Abah et. al. [START_REF] Joshua | A machine learning approach to anomaly-based detection on android platforms[END_REF] present a detection system that uses a k -Nearest Neighbour classifier to detect malicious applications based on features extracted during execution. To the best of our knowledge, none of the machine learning based methods proposed in the literature provides any reliable indication on the likelihood of its detections being correct.

This paper proposes an approach that addresses the uncertainty arising from the use of machine learning techniques. Since there is no way of guaranteeing 100% accuracy with any machine learning technique, this study aims at providing a reliable indication of how probable it is for a prediction to be correct. The provision of such an indication for applications identified as possible malware, would be of great value for the decision of the user on whether to remove an application or not, depending on the risk he/she is willing to take. The proposed approach utilizes a novel machine learning framework, called Conformal Prediction (CP) [START_REF] Vovk | Algorithmic Learning in a Random World[END_REF], for providing provably valid measures of confidence for each individual prediction without assuming anything more than that the data are generated independently from the same probability distribution (i.i.d.). The confidence measures produced by CP have a clear probabilistic interpretation, thus enabling informed decision making based on the likelihood of each malware detection being correct.

Specifically, we combine the inductive version of the CP framework [START_REF] Papadopoulos | Inductive confidence machines for regression[END_REF][START_REF] Papadopoulos | Inductive Conformal Prediction: Theory and application to neural networks[END_REF], which is much more computationally efficient than the original version, with a Multilayer Perceptron (MLP), which is one of the most popular and wellperforming machine learning techniques. We evaluated the proposed approach on a realistic malware dataset using a prototype developed for a real Android device (LG E400). Our experimental results demonstrate the empirical validity of our approach as well as its efficiency in terms of accuracy, informativeness and computational time.

The rest of the paper starts with an overview of the Conformal Prediction framework and its inductive counterpart in Section 2. The next section (Section 3) details the proposed approach. Section 4 describes the dataset used for evaluating our approach, while Section 5 presents our experiments and the obtained results. Finally, Section 6 gives our conclusions.

Conformal Prediction

The Conformal Prediction (CP) framework extends conventional machine learning algorithms into techniques that produce reliable confidence measures with each of their predictions. The typical classification task consists of a training set {(x 1 , y 1 ), . . . , (x l , y l )} of instances x i ∈ IR d together with their associated classifications y i ∈ {Y 1 , . . . , Y c } and a new unclassified instance x l+1 . The aim of Conformal Prediction is not only to find the most likely classification for the unclassified instance, but to also state something about its confidence in each possible classification.

CP does this by assigning each possible classification Y j , j = 1, . . . , c to x l+1 in turn and extending the training set with it, generating the set {(x 1 , y 1 ), . . . , (x l , y l ), (x l+1 , Y j )}.

(

) 1
It then measures how strange, or non-conforming, each pair (x i , y i ) in ( 1) is for the rest of the examples in the same set. This is done with a non-conformity measure which is based on a conventional machine learning algorithm, called the underlying algorithm of the CP. This measure assigns a numerical score α i to each pair (x i , y i ) indicating how much it disagrees with all other pairs in (1).

In effect it measures the degree of disagreement between the prediction of the underlying algorithm for x i after being trained on (1) with its actual label y i ; in the case of x l+1 , y l+1 is assumed to be Y j .

To convert the non-conformity score α (Yj ) l+1 of (x l+1 , Y j ) into something informative, CP compares it with all the other non-conformity scores α (Yj ) i , i = 1, . . . , l. This comparison is performed with the function

p((x 1 , y 1 ), . . . , (x l , y l ), (x l+1 , Y j )) = |{i = 1, . . . , l + 1 : α (Yj ) i ≥ α (Yj ) l+1 }| l + 1 . (2)
The output of this function, which lies between 1 l+1 and 1, is called the p-value of Y j , also denoted as p(Y j ), as this is the only unknown part of [START_REF]Cabir, smartphone malware[END_REF]. If the data are independent and identically distributed (i.i.d.), the output p(y l+1 ) for the true classification of x l+1 has the property that ∀δ ∈ [0, 1] and for all probability distributions P on Z, P l+1 {((x 1 , y 1 ), . . . , (x l+1 , y l+1 )) : p(y l+1 ) ≤ δ} ≤ δ;

(

for a proof see [START_REF] Nouretdinov | Pattern recognition and density estimation under the general i.i.d. assumption[END_REF]. Therefore all classifications with a p-value under some very low threshold, say 0.05, are highly unlikely to be correct as such sets will only be generated at most 5% of the time by any i.i.d. process.

Based on the property (3), given a significance level δ, or confidence level 1 -δ, a CP calculates the p-value of all possible classifications Y j and outputs the prediction set

{Y j : p(Y j ) > δ}, (4)
which has at most δ chance of not containing the true classification of the new unclassified example. In the case where a single prediction is desired (forced prediction) instead of a prediction set, CP predicts the classification with the largest p-value, which is the most likely classification, together with a confidence and a credibility measure for its prediction. The confidence measure is calculated as one minus the second largest p-value, i.e. the significance level at which all but one classifications would have been excluded. This gives an indication of how likely the predicted classification is compared to all other classifications. The credibility measure on the other hand, is the p-value of the predicted classification. A very low credibility measure indicates that the particular instance seems very strange for all possible classifications.

Inductive Conformal Prediction

The transductive nature of the original CP technique, which means that all computations have to start from scratch for every new test example, makes it too computationally demanding for a mobile phone application.

p(Y j ) = |{i = m + 1, . . . , m + q, l + 1 : α i ≥ α (Yj ) l+1 }| q + 1 . (5)
Notice that the steps that need to be repeated for each test example have almost negligible computational requirements.

Proposed Approach

In this study a Multilayer Perceptron (MLP) was used as underlying algorithm of the ICP. The MLP used was a 2-layer fully connected feed-forward network with a sigmoid activation function in all units. It was implemented using the Multilayer Perceptron class of the WEKA data mining software libraries [START_REF] Hall | The weka data mining software: An update[END_REF]. When given an instance x i to classify, the trained MLP produces a probabilistic value P (Y j |x i ) for each classification Y j . In this work the classification task is binary, therefore Y j ∈ {0, 1} and two probabilistic values are produced by the MLP: P (0|x i ) and P (1|x i ).

The nonconformity measure used for the proposed MLP-ICP was

α Yj i = 1 -P (y i |x m+i ), i = 1, . . . , q, (6a)
α Yj l+1 = 1 -P (Y j |x l+1 ), (6b)
where P (y i |x i ) is the output of the MLP for the true classification of x i and P (Y j |x l+1 ) is the output of the MLP for the assumed class Y j ∈ {0, 1} of x l+1 . The complete process followed by the proposed approach is detailed in Algorithm 1. Lines 1 to 5 correspond to the training phase that needs to be performed only once. This phase trains the MLP on the proper training set (line 1) and calculates the nonconformity score of each calibration example x m+i , i = 1, . . . , q, by inputing it to the trained MLP to obtain the probabilistic outputs P (0|x m+i ) and P (1|x m+i ) (line 3) and using them in (6a) to calculate α m+i (line 4). The testing phase, lines 6 to 11, is the only part that needs to be repeated for every new instance. This phase obtains the probabilistic outputs of the trained MLP for the new instance (line 6) and then for each possible class, it calculates the corresponding nonconformity score with (6b) in line 8 and uses it together with the nonconformity scores of the calibration examples in (5) to calculate the pvalue of the new instance belonging to that class (line 9). Finally, in line 11, it outputs the prediction set (4).

Data Used

For evaluating the proposed approach a malware dataset1 created by B. Amos [START_REF] Amos | Applying machine learning classifiers to dynamic android malware detection at scale[END_REF] was used. The data was collected using a shell script to automatically record the behavior of Android application files (.apk ) by installing and running them on an Android emulator. After each installation the emulator simulated random user interaction with the application using the Android "adb-monkey" tool. The data collected for each application included Binder, Battery, Memory, CPU, Network, and Permission information. The battery related features were removed in our experiments since they all had the same value across all instances, as they were collected through an emulator. Table 1 presents all the features used in our experiments.

The dataset consisted of 6832 data samples in total, with 5121 benign samples and 1711 malicious samples described by 40 features (after the removal of the battery related data). In our experiments the data samples were divided randomly into a training set consisting of 6165 samples (4617 benign and 1548 malicious) and a test set consisting of 667 samples (504 benign and 163 malicious). For MLP-ICP the training set was further divided (also randomly) into the proper training set with 5466 samples (4097 benign and 1369 malicious) and the calibration set with 699 samples (520 benign and 179 malicious). All experiments were repeated 10 times with different random divisions of the data so as to ensure that the obtained results are not dependent on a particular division.

Experimental Results

In order to evaluate the proposed approach in the environment it is intended for, both the MLP-ICP and its underlying algorithm were implemented on the Android OS using the WEKA data mining software libraries and all experiments were performed on a LG E400 Android device. Before each experiment all input features were normalized to the range [0,1], based only on the training set of each of the 10 random divisions of the data and the resulting transformation was applied to the test set.

Accuracy

Our first set of experiments evaluates the proposed technique in terms of the accuracy of forced predictions. That is when the MLP-ICP predicts the most likely classification together with a confidence and credibility measure to that classification. We compare the accuracy of the proposed approach to that of its underlying algorithm. Note that the aim of the proposed approach is not to Table 2. Forced prediction performance of the proposed approach compared to its underlying algorithm. The standard deviation of each value is given in the brackets.

Accuracy (%) Mean Confidence (%) MLP-ICP 94.24 (0.0116) 98.83 (0.0020) Conventional MLP 94.17 (0.0117) improve performance in terms of accuracy, but rather to provide probabilistically valid additional information about the likelihood of each prediction being correct, which will aid user decision making. Therefore our aim here is to examine if any degree of accuracy is sacrificed by the proposed approach in order to provide this additional information.

Table 2 reports the mean accuracy of the proposed approach and of the conventional MLP over 10 experimental runs with different random divisions of the data. For MLP-ICP we also report the mean of the confidence values produced. The values in brackets are the standard deviation of the corresponding values over the 10 experiments. The reported results show that there is no significant difference in terms of the accuracy of the two approaches; in fact the MLP-ICP gives higher accuracy, but the difference is insignificant. This shows that there is no "price to pay" in terms of accuracy for obtaining the much more informative outputs of the proposed approach. Furthermore, the high mean confidence produced by the proposed approach gives a first indication of the usefulness of its outputs. Finally, the small standard deviation of both accuracy and mean confidence over the 10 experimental runs shows that the presented mean values are a reliable indication of performance.

Empirical Validity

Next we evaluate the empirical validity of the prediction sets, and indirectly of the p-values and confidence measures, produced by the proposed approach. Figure 1 plots the error percentages of these prediction sets as the significance level changes. The left pane displays the error percentages for all significance levels δ ∈ [0, 1], while the right pane displays the lower left corner of the first plot, for δ ∈ [0, 0.1], which is the most important part of the plot as we are generally interested in high confidence levels. The solid line represents the actual error percentages observed, while the dashed line represents the diagonal, i.e. where the error is equal to the required significance level δ. The closeness of the actual errors to the diagonal confirms the empirical validity of the proposed approach, i.e. the errors made are always extremely near the required significance level; the very small deviation of about 0.01 visible in the left pane is due to statistical fluctuations. This confirms empirically the guaranteed validity of the ICP.

Quality of p-values

evaluate the quality of the resulting p-values we examine the percentage of prediction sets that contained both classifications at different confidence levels. This is the percentage of instances for which the ICP was not able to reject one of the two possible classes at the required confidence level, we call these uncertain prediction sets. The lower these values are the more the instances for which the prediction sets are informative at the given confidence level and the higher the quality of the p-values used to compute them.

Figure 2 plots the percentage of uncertain prediction sets for the range of significance levels δ ∈ [0, 0.1], i.e. confidence levels between 100% and 90%. The plot shows that the prediction sets produced by the proposed approach are quite informative. With a confidence level as high as 99%, i.e. guaranteeing only 1% of errors, we can be certain in 63.7% of applications on whether they are malware or not. If we decrease the required confidence level by 1% (to 98%) we obtain certain predictions for 80.28% of the cases. Further lowering the confidence level to 95%, which is still rather high, increases the percentage of certain predictions to 95.53%. While for confidence levels 93% and 92% the number of certain predictions rises to more than 99% and becomes 100% from 91% confidence and below. This means that we can have a certain prediction for the large majority of applications with a very small risk, while for applications with uncertain predictions further investigation might be the best option.

The informativeness of the obtained prediction sets also shows the quality of the p-values used for constructing them, and as a result of the confidence measures produced based on the same p-values in the prediction case. In this case, the user can decide based on the confidence measure for a given malware prediction and on the risk he/she is willing to take on whether to remove it or not.

Computational Efficiency

Finally, since in a mobile device environment the amount of recourses needed are an important consideration, we examine the execution time of the proposed approach on a LG E400 Android device and compare it to that of its underlying algorithm. The training time for the MLP-ICP was 21 minutes, while the training time for the conventional MLP was 29 minutes. The reason for the lower training time of the ICP is the removal of the calibration examples from the proper training set. This works in favour of the proposed approach in terms of computational efficiency, while it has no negative impact on its accuracy as shown in Subsection 5.1. Although the training time needed seems somewhat long, it should be kept in mind that training only needs to be performed once and this can even be done on a desktop device. The testing time for both methods was 0.005 seconds, which is very computationally efficient for a mobile device.

Conclusions

This work proposed a machine learning approach for Android malware detection, which unlike traditional machine learning based malware detection techniques, produces valid confidence measures in each of its predictions. Such measures enable the user to take informed decisions on whether to remove an application identified as a possible malware, knowing the risk associated with each decision. The proposed approach is based on the Conformal framework, which produces provably valid confidence measures that have a clear probabilistic interpretation without assuming anything more than i.i.d. data.

Our experimental results on a real Android device have shown that the proposed approach gives high accuracy, which is equal to (or even better than) that of the conventional machine learning approach it is based on. Furthermore, they demonstrate the empirical validity and high informational efficiency of the produced prediction sets and confidence measures. Finally in terms of computational efficiency, the training time needed by our approach is smaller than that of the conventional MLP on which it bases its predictions. While the time needed for classifying a new instance is adequately small for the limited resources of mobile devices.

The particular CP used in this work is based on MLP, which is one of the most popular machine learning techniques. However, the same general approach proposed here can be followed with other high performing machine learning techniques preserving the desirable properties observed in our experiments. The evaluation and performance comparison of CPs based on different conventional machine learning techniques for the particular task is our immediate future plan. Additionally the collection of a dataset from actual mobile devices rather than in an emulated environment in order to study the performance of the proposed approach on more realistic data is another future goal.

  For this reason the proposed approach follows the inductive version of the framework, called Inductive Conformal Prediction (ICP), which only performs one training phase to generate a general rule with which it can then classify new examples with minimal processing. Specifically, ICP divides the training set (of size l) into the proper training set with m < l examples and the calibration set with q := l -m examples. It then uses the proper training set for training the underlying algorithm (only once) and the examples in the calibration set for calculating the p-value of each possible classification of the new test example. In effect, after training the underlying algorithm on the proper training set the non-conformity scores α m+1 , . . . , α m+q of the calibration set examples are calculated. Then to calculate the p-value of each possible classification Y j ∈ {Y 1 , . . . , Y c } of a new test example x l+1 , ICP only needs to calculate the non-conformity score of the pair (x l+1 , Y j ) using the already trained underlying algorithm and compare it to the non-conformity scores of the calibration set examples with the function

Algorithm 1 : 3 { 4 αm+i ← 1 - 9 p 11

 1341911 Binary MLP-ICP Input: proper training set {(x1, y1), . . . , (xm, ym)}, calibration set {(xm+1, ym+1), . . . , (xm+q, ym+q)}, test example x l+1 , significance level δ 1 h ← train the MLP on {(x1, y1), . . . , (xm, ym)};2 for i = 1 to q do P (0|xm+i), P (1|xm+i)} ← h(xm+i); P (ym+i|xm+i); 5 end 6 { P (0|x l+1 ), P (1|x l+1 )} ← h(x l+1 ); 7 for j = 0 to 1 do 8 α j l+1 ← 1 -P (j|x l+1 ); Prediction set R ← Yj : p(Yj) > δ ∪ arg max k=1,...,c p(Y k ) .

Fig. 1 .

 1 Fig.1. Empirical validity of the proposed approach. The error percentage of the ICP is plotted with a solid line and the diagonal (exact validity) with a dashed line.

Fig. 2 .

 2 Fig. 2. Percentage of uncertain prediction sets for significance levels δ ∈ [0, 0.1].

Table 1 .

 1 Features used in this work divided into categories.

	Category	Features
	Binder	Transaction, Reply, Acquire, Release, ActiveNodes, TotalNodes,
		ActiveRef, TotalRef, ActiveDeath, TotalDeath, ActiveTransaction,
		TotalTransaction, ActiveTransactionComplete,
		TotalTransactionComplete, TotalNodesDiff, TotalRefDiff,
		TotalDeathDiff, TotalTransactionDiff, TotalTransactionCompleteDiff
	CPU	User, System, Idle, Other
	Memory	Active, Inactive, Mapped, FreePages, AnonPages, FilePages,
		DirtyPages, WritebackPages
	Network	TotalTXPackets, TotalTXBytes, TotalRXPackets, TotalRXBytes,
		TotalTXPacketsDiff, TotalTXBytesDiff, TotalRXPacketsDiff,
		TotalRXBytesDiff
	Permissions TotalPermissions

Available on-line at: https://github.com/VT-Magnum-Research/antimalware


Terence Fusco

Yaxin Bi
email: bi.y@ulster.ac.uk

A Cumulative Training Approach to Schistosomiasis Vector Density Prediction

The purpose of this paper is to propose a framework of building classification models to deal with the problem in predicting Schistosomiasis vector density. We aim to resolve this problem using remotely sensed satellite image extraction of environment feature values, in conjunction with data mining and machine learning approaches. In this paper we assert that there exists an intrinsic link between the density and distribution of the Schistosomiasis disease vector and the rate of infection of the disease in any given community; it is this link that the paper is focused to investigate. Using machine learning techniques, we want to accumulate the most significant amount of data possible to help with training the machine to classify snail density (SD) levels. We propose to use a novel cumulative training approach (CTA) as a way of increasing the accuracy when building our classification and prediction model.

Introduction

The resurgence of epidemic disease breakouts in regions of Asia and South America in the past decade has given local governments and health organisations cause for much concern. The devastating impact that these diseases can have on many aspects of human, cattle and crop life incurs huge financial and social cost. This rationale makes research into the prevention and preparation for future outbreaks, a problem that requires immediate attention and one that is crucial to supporting the locally affected municipalities [START_REF] Ross | Schistosomiasis in the people's republic of china: prospects and challenges for the 21st century[END_REF]. The epidemic disease Schistosomiasis is detrimental to many sections of society in China. Schistosomiasis is the second most widely affected disease in the world as stated by the World Health Organisation [2]. It is a disease, which is transmitted through water infected by parasites known as Schistosomes. The intermediate host of the disease is the Oncomelania Hupensis snail. Humans are affected mainly through freshwater used for washing clothes and household items as well as through infected crops and cattle. The affect it can have on many areas of human, cattle, crop life both in terms of health and financially is a valid cause for concern [START_REF] Ross | Schistosomiasis in the people's republic of china: prospects and challenges for the 21st century[END_REF]. To combat Schistosomiasis can be very difficult due to the fact that there is no vaccine available against the disease and therefore it can only be treated once the patient has been infected. Currently, the most effective way of dealing with the disease is by trying to establish areas that are of high risk of the disease and putting in place preventative measures such as chemical treatment to specific freshwater areas [START_REF] Ma | The analysis of east dongting lake water change based on time series of remote sensing data[END_REF] in order that the disease is addressed before the vectors multiply or increase in density and distribution. An alternative solution is to plant poplar trees, which would disturb the natural vegetation and moisture factors that encourage snail life and breeding habitat [START_REF] Sun | Effect of poplar forest on snail control in dongting lake area[END_REF]. Whichever method is applied to at-risk areas will have a time and financial cost incurred therefore the concerned municipalities require the most informed data available before acting and addressing the area in question. The local governments will also need to prepare those areas for any panic or influx of patients that may occur.

The environment features present in Schistosomiasis areas of interest can be shown to be intrinsically linked to the disease infection rates [START_REF] Wu | Identification of optimum scopes of environmental factors for snails using spatial analysis techniques in dongting lake region, china[END_REF]. By using data mining methods we can assess the corollary relations between the environment feature values and the SD and distribution values. We aim to identify the environment conditions which make the Oncomelania Hupensis snail most suitable for transmission of the Schistosomiasis disease. We know that for the Oncomelania Hupensis reproduction and for life to flourish, it requires specific environment conditions. We also know the snails will not survive in strong currents and that during early years in their lives the Oncomelania Hupensis snail will live only in water. Once they are adults they then must move from the water usually to moist soil above the water line as the snail activity increases with soil moisture and that the optimum temperature for breeding is around 20 • C [START_REF] Seto | The use of remote sensing for predictive modeling of schistosomiasis in china[END_REF]. The Oncomelania Hupensis snail flourishes and breeds particularly well in areas with high levels specific environment features such as soil moisture (NDMI) and vegetation (NDVI) therefore we can deduce that areas which meet these specific environment conditions have a greater likelihood of high snail vector density.

By analyzing and assessing this information we can achieve greater success from our classification accuracy. With the implementation of this research approach we can make the most informed prediction on which to base information to provide to those concerned. We believe that the most promising approach to detect high-risk areas of disease outbreak is to use vector density classification techniques based on environmental features that exist in each area of interest.

Using our proposed Cumulative Training Approach (CTA), we can enhance the training potential of our limited dataset. This will help to provide a larger pool of relevant training data which we hope will increase the classification accuracy during the testing process. The process involved uses the data from a combination of collective years' data as a training set to train the machine for classifying SD based on the environment information given. Particularly the CTA also involves the pre-processing of segmenting the SD into the three or five point categories, handling of missing values, environment feature selection and correlation analysis between environment features and attributes.

This paper provides the description, rationale and results of preliminary experiments that examine the correlation and influence levels between environmental features and SD present in the Dongting Lake area of China. This lake represents a very relevant study area with which to examine the moisture and vegetation levels required for snail life to flourish. The datasets used in this paper were derived from remotely sensed image extraction information together with manually collected field survey data provided by our Chinese project partners at Academy of Opto-Electronics the and the European Space Agency (ESA). The datasets have been analysed quantitatively and results are illustrated with this paper. The aim of these studies is to discover if there exists strong correlation between individual or component environment features and the Schistosomiasis disease vector (Oncomelania Hupensis snail) density and distribution. If we can identify this, then we can make future SD classifications using our prediction models based on previously collected datasets. The resulting prediction models will be capable of making informed assumptions on future SD levels and therefore provide likelihood of outbreaks of the disease occurring based on environmental feature values on a larger scale and with greater efficiency than is currently available.

The CTA proposed in this paper is a framework we use to enhance the training potential of real-world sparse datasets. This approach is conducted using a range of pre-processing methods together with attribute ranking and data analysis, the results of which we take into consideration when building our classification and prediction models to determine the density and distribution of epidemic disease vectors. We aim to enrich our training set by investigating the various methods discussed in this paper to deduce if they can have a positive effect when applied for classification of SD in terms of accuracy performance. This includes using combined years of data instances for training against alternative testing sets. We aim to take into account and apply the optimal test conditions as a training paradigm to discover whether those criteria perform better than standard datasets for classification purposes.

Experiment Data

The datasets used in this report are derivative of remotely sensed satellite images ranging from between 2003-2009 in the Dongting Lake area of China. The images were processed and feature extraction was carried out by our Chinese partners to provide values for the environmental features present for each year. While we can access vast amounts of data from satellite imaging, the primary field survey data of which we can be sure is a much more time consuming process so this is why we have such a limited dataset in terms of instances. The number of common features from each year was seven with the collective number of instances being 180. While the dataset is relatively small in data mining terms, it provides a basis on which to form initial opinions and observations as to which attributes or combination of attributes have the strongest influence on SD levels. When we deduce which feature subsets are most influential on the SD levels, we can make assertions on future SD classification and therefore provide important information to those concerned for preventative measures to be put into place.

During initial assessment of the dataset, we looked at how we would categorize the SD values in terms of whether the raw value provided would constitute the label of high SD. To this end, the data was preprocessed by normalizing the values in order to gain arbitrary values into a predefined range which could then be labelled in terms of density level. We subsequently assessed whether we could achieve better classification success by using a 3 or 5 point scale of SD as in Low, Medium or High as opposed to Very Low, Low, Medium, High and Very High. We must categorize the density level in this way for classification purposes otherwise we will be restricted to using either statistical or regression models. The data was initially normalised to achieve a range from 0 to 1 then discretised into the 3 or 5 point scale with even distribution. To discretise the data, data binning was used. This method takes a set of continuous values and turns them into a set of bins which are nominal values. The number of bins was set to five for 'very low', 'low', 'medium', 'high' and 'very high'; this resulted in five data intervals which the data was then split into. It was also used for three bins, representing 'low', 'medium' and 'high'.

In addition, we used two regressive methods on our data to make initial assessment on how well the data fits and therefore how well each year fits for classification purposes. While using the linear regression and support vector regression on our unprocessed SD data, we can assess the accuracy of each year of data when predicting new instances of SD. With linear regression we assume environment features to be independent in a dataset; in this case the environment factors are in relation to the dependent variable which is the SD value.

These regressive methods do not provide specific classification percentage accuracy results as the SD value has not been pre-processed or classified to a selected scale that can be used for prediction. We can instead use the coefficient of determination calculation to give the R value which tells us how good of a fit the data is that we are experimenting with. We can assess the results of the coefficient of determination with results ranging from 0 to 1. The closer to 1 the result is, the better the fit therefore the higher likelihood of predicting a new instance of SD.

Equation 1 involves taking the average of the entire SD actual values for each year, then subtracting the average value from each individual actual SD value to the power of two and the same with each predicted value from that year. We then take the sum of results from each instance of predicted and actual values and divide the total value from the predicted SD calculation by the total actual value calculation with the value ranging between 0 and 1 with 0 being the least well-fitting data and 1 being the best fitting data.

We can see from table 1b that the best fitting data is from 2008 training and testing data using linear regression. It scored 0.8 which is the closest result to 1 making it the most promising data combination for classifying future instances of SD. We can also see that in 2007, both the linear regression and support vector regression classifiers performed well with similar performance which can also be an indicator of potentially generating good classification models.

Once we had made initial assessment of the datasets, they were preprocessed by normalizing and then discretizing the SD information from each year. This enabled us to have more options for using different algorithms for classification purposes. The SD data was separated at the beginning into 5 categories and the This is what we would expect to see once the data has been preprocessed but it does not provide the entire picture so we will have to explore the dataset further and assess the relative SD levels and in conjunction with environmental features.

Methods

In earth observation research, weather conditions directly affect the quality of satellite imagery which causes some values of environment variables to be missing in the set and discontinuity in terms of fully recorded data relationships. These partially complete datasets are caused by anomalies in the remotely sensed image extraction process and by issues such as weather clarity from satellite imagery. One of the major issues faced when using the data provided was that specific data particularly from 2007 was only partially complete. This problem correlates directly to the weather conditions present at the time of acquiring data from the satellites therefore we are interested in providing a resolution that can be Fig. 1: Classification of SD over Time applied to any future incomplete datasets provided by satellite images. This issue highlighted the need for an approach capable of imputation of the values that were incomplete from the dataset in order to be able to use the 2007 dataset and any other incomplete data for future temporal assessment of SD levels.

The rationale behind this imputation process is to find a solution for replacement of partially complete data that could potentially be scalable for much larger datasets with a variety of different features. The process of removing known values from our dataset then providing replacement using the following methods is documented below, where V represents the feature value of an instance.

-The Weka replace missing value filter replaces missing values with the mean and modal values from the remaining set for data imputation. -The Single Pre-Succession Method uses the previous and following values to replace the missing value.

v i = v i-1 + v i+1 2 (1)
-The Mean Single Pre-Succession Method uses the previous and following values to replace the missing value together with the mean of the entire set.

v i = v i-1 + v i+1 + v 3 (2)
-The Double Pre-Succession Method uses the two previous and following values to replace the missing value.

v i = v i-2 + v i-1 + v i+1 + v i+2 4 (3)
-The Mean Double Pre-Succession Method uses the two previous and following values to replace the missing value together with the mean of the entire set.

v i = v i-2 + v i-1 + v i+1 + v i+2 + v 5 (4)
We can see from Table 2 that the most accurate performing method is the Mean Double PreSuccession method with an average percentage difference of 32.58% while the lowest performance is of the PreSuccession method which has an average percentage difference of 333.36% from the original value that was replaced. These results can now be analysed and used for future incomplete datasets to verify the accuracy of value replacement over more extensive datasets.

Feature Assessment

-We want to evaluate the dataset to discover the relevance of each attribute to SD levels individually and as subsets of features. -To assess and rank the features of the data yearly to gain a deeper understanding of the value of the environment features to the data as a whole.

-Selection of an efficient, well performing method to handle replacement of missing values in the data as this is an ongoing issue with RS images that will be required for application in any future data that may be accessed. -To distinguish the most effective category of SD to move forward with for future classification purposes.

Information Gain

To assess the attribute values in relation to SD, Information Gain attribute ranker was applied to the data and documented in a table for each years' data.

Information Gain is a feature ranking approach that uses entropy to identify which feature in the dataset gains the most information relative to the class. This is beneficial when carrying out analysis of a dataset to extract the most influential features in relation to their corresponding SD values. It can be of significant value in order to identify any corollary inferences with regards environment features to SD levels. Once we identify which attributes have most significant influence on the SD value, then it can be established for future experiments that these specific attributes are closely connected to high levels of SD. The results in Table 3 show relative consistency with each year having similar positions for each of the attributes. We can see certain attributes consistently trending such as the Normalised Difference Water Index (NDWI) and the Tasseled Cap Greenness (TC G) which indicate that these attributes are of significant value in relation to SD of each of the particular years in the dataset.These results will now form an element of consideration for our CTA model.

The information Gain calculation used is shown in equation 5 where entropy (H) is given of the class (C) given the attribute (A) [START_REF] Blake | UCI Repository of machine learning databases[END_REF]. Entropy and information gain are intrinsically linked as the decrease in the entropy of the class is a direct reflection of the added information about the class provided by the attribute and this is referred to as the information gain and therefore entropy is a pre-requisite for information gain to be calculated [START_REF] Quinlan | C4.5: Programs for Machine Learning[END_REF].

H(C|A) = - a∈A p(a) - c∈C p(c|a) log 2 p(c|a) (5)
Correlation analysis was applied to the combination of each of the attributes with the SD temporally. In terms of relationships, we used Pearson's r approach, which uses the covariates X and Y, this is then divided by the standard deviation of X and of Y to give a correlation value of each individual attribute and SD value. The results are shown in figure 2b and they indicate that data from 2008 is not in correlation with the alternate years as the trend lines show us. The combination of the SD and environmental attributes (X, Y) does not show correlate with the dataset from each year. The corollary relationship results between SD levels and environment features is an integral component of our CTA framework below for future classification of SD levels based on environment factors. As the environment feature values increase towards 1.0 in Table 4 it shows the impact factor that is present when compared with SD levels with TCG and NDVI from 2005 showing good correlation as opposed to NDWI in the same year. This method of training will make up another component of our proposed idea referred to throughout this paper as the CTA. By applying this proposed CTA, we are combining the most promising experiments and test results from a variety of relevant areas of our data pertaining to the classification of the Schistosomiasis disease vector. Using this data we can then build a model for application with any future spatio-temporal epidemic disease environment data, which is a different approach from the standard application of classification methods.

P (x, y) = cov (x, y) σxσy (6)
In addition carried out testing on three ensemble learning methods of Bagging, Boosting and Stacking as the ensemble methods have been shown to provide better classification accuracy than single classifiers [START_REF] Pan | Impact of accuracy, spatial availability, and revisit time of satellite-derived surface soil moisture in a multiscale ensemble data assimilation system[END_REF]. Using these three ensemble methods we can get a varied range of results based on training model performance (Adaboost), equal sized training set sampling (Bagging) and combined classifier prediction (Stacking). Results were recorded in table 5.

Conclusion

From the correlation analysis graph, we can see that each of the years data with the exception of 2008, follow together in a trend which shows that the correlation values of each combination of attributes together with SD, can be predictable which is of high value to this particular research area looking at future distribution and density predictions.

By handling and assessing missing value replacement in the data, we can identify the success of replacing these values based on the mean and mode of the existing data. These results can be applied to future RS data that will be accessed for research and experimentation. By testing effectiveness of replacement methods, we can identify confidence in future replacement of data.

All experiments and collective research to date have become part of the CTA for Schistosomiasis vector density and distribution prediction. This approach has

  (a) 2003Train -2005Test CTA Data (b) Pearson's Correlation Co-Efficient

Fig. 2 :

 2 Fig. 2: CTA and Correlation figures

Table 1 :

 1 Statistics for yearly SD Categories and R 2 Values results are recorded in figure 1 and Table 1a. They show on average that density and distribution of the Oncomelania Hupensis snail during the time period from 2003-2009 were predominantly very low and low.

	Year AVG SD Normalised Category
	2003	0.727	0.107	V. Low
	2005	0.879	0.129	V. Low
	2007 2008	2.633 1.396	0.388 0.206	Low Low	2003 2005 2007 2008 2009
	2009	1.056	0.156	V. Low	LR 0.325 0.590 0.734 0.808 0.699
	Collective 1.014	0.149	V. Low	SVR 0.052 0.221 0.732 0.691 0.506
		(a) Average SD Values		(b) R 2 Values for Linear Reg. and
					Support Vector Reg.

Table 2 :

 2 Data Imputation from 2007

	Original	Weka	PreSucc. Method	Mean PreSucc.	Double PreSucc.	Mean Dbl PreSucc.
	0.0348	0.228	0.251	0.169	0.012	0.061
	0.128	0.201	0.100	0.254	0.123	0.069
	-0.084	-0.084 0.236	0.052	0.401	0.064
	0.024	0.201	0.097	0.106	0.027	0.050
	-0.660	-0.471 0.026	-0.156	0.080	-0.083
	0.242	0.228	0.386	0.214	0.103	0.072
	-0.521	-0.622 0.387	-0.091	0.346	-0.062
	0.410	0.657	-0.006	0.232	-0.141	0.112
	0.400	0.344	0.025	0.123	-0.025	0.064
	0.35545	0.657	0.007	0.236	-0.119	0.117
	Avg.% Diff. 145.13% 333.36% 226.81% 132.04% 32.58%

Table 3 :

 3 Information Gain feature ranking

	2003	2005	2007	2008	2009 Collective
	1 NDWI NDWI NDWI TC G MNDWI TC G
	2 TC W TC W TC W NDWI NDVI	NDWI
	3 TC G	TC G	TC G	NDVI NDWI	TC W
	4 NDMI NDMI NDMI MNDWI TC G	NDMI
	5 MNDWI MNDWI MNDWI NDMI NDMI MNDWI
	6 NDVI	NDVI	NDVI TC W TC W	NDVI
	7 TC B	TC B	TC B	TC B	TC B	TC B

Table 4 :

 4 Correlation analysis between SD vs. FeaturesGiven the limited amount of data and the pre-processing results, we consider how to construct prediction/classification models. A caveat to address with the SD classification is the fact that for years 2003, 2005 and 2007 we have 18/19 attributes partially complete labelled whereas with years 2008 and 2009 we have eight/nine attributes given to experiment with. The most beneficial approach to dealing with this issue is to use those attributes which are common to each year in order to make a comparable dataset for training and testing purposes. It was decided to use the initial year's collected research information to build a training model, which is then used as a benchmark against future data for testing purposes. This approach will enable us to enrich the dataset with variable subsets of the data being used to discover temporal relationships within the dataset. The method was used tested with five classification methods to assess accuracy. We can see from figure2athat year 2003 training data with 2005 testing yields highly accurate results as the accuracy during training and prediction accuracy are in close proximity to each other, this indicates good classification performance.

	TCB TCG TCW NDMI NDVI MNDWI NDWI
	2003 -0.069 0.352 0.352 0.101 0.359 -0.289	-0.339
	2005 0.308 0.517 -0.301 0.332 0.519 -0.4999 -0.521
	2007 0.287 0.192 -0.17 0.194 0.227 -0.07	-0.195
	2008 -0.32 -0.132 0.163 0.194 -0.179 0.397	0.289
	2009 0.208 0.333 -0.193 0.139 0.418 -0.462	-0.387

Table 5 :

 5 CTA Ensemble Results

	Training	Testing Boosting	Bagging	Stacking

Train Predict Train Predict Train Predict 2003/05 2009 0.740 0.483 0.753 0.5 0.712 0.467 2003/05/07 2009 0.530 0.467 0.504 0.467 0.556 0.467 2003/05/07/08 2009 0.558 0.483 0.509 0.533 0.491 0.467 2003/05 2008 0.740 0.478 0.753 0.348 0.712 0.326 2003/05/07 2008 0.530 0.326 0.504 0.348 0.556 0.326 2003/05

2007 0.740 0.295 0.753 0.318 0.712 0.295 provided us with a better understanding of our datasets and the classification results which it provides. In combining each aspect of the training process we have a greater understanding of the research area and we can apply this knowledge to future data obtained for classification and prediction purposes.

From the results to date, we can deduce that specific environmental attributes such as TC G and NDWI have more influence on the SD and distribution than others. This information will be further analysed and implemented into a cluster ensemble algorithm for optimum accuracy classification for future work [START_REF] Elshazly | Ensemble classifiers for biomedical data: Performance evaluation[END_REF].


Yao Jean

Marc Pokou

Philippe Fournier-Viger

Chadia Moghrabi
email: chadia.moghrabi@umoncton.ca

Using Frequent Fixed or Variable-Length POS Ngrams or Skip-grams for Blog Authorship Attribution

Authorship attribution is the process of identifying the author of an unknown text from a finite set of known candidates. In recent years, it has become increasingly relevant in social networks, blogs, emails and forums where anonymous posts, bullying, and even threats are sometimes perpetrated. State-of-the-art systems for authorship attribution often combine a wide range of features to achieve high accuracy. Although many features have been proposed, it remains an important challenge to find new features and methods that can characterize each author and that can be used on non formal or short writings like blog content or emails. In this paper, we present a novel method for authorship attribution using frequent fixed or variable-length part-of-speech patterns (ngrams or skip-grams) as features to represent each author's style. This method allows the system to automatically choose its most appropriate features as those sequences being used most frequently. An experimental evaluation on a collection of blog posts shows that the proposed approach is effective at discriminating between blog authors.

Introduction

Authorship analysis is the process of examining the characteristics of a piece of work to identify its authors [START_REF] Koppel | Authorship attribution: What's easy and what's hard? Available at[END_REF]. The forerunners of authorship attribution (AA) are Medenhall who studied the plays of Shakespeare in 1887 [START_REF] Mendenhall | The characteristic curves of composition[END_REF], and Mosteller and Wallace who studied the disputed authorship of the Federalist Papers in 1964 [START_REF] Mosteller | Inference and disputed authorship: The federalist[END_REF]. Beside identifying the authors of works published anonymously, popular applications of AA techniques include authenticating documents, detecting plagiarism, and assisting forensic investigations.

In recent years, an emerging application of AA has been to analyze online texts, due to the increasing popularity of Internet-based communications, and the need to find solutions to problems such as online bullying, and anonymous threats. However, this application raises novel challenges, as online texts are written in an informal style, and are often short, thus providing less information about their authors. The accuracy achieved with well-established features for AA like function words, syntactic and lexical markers suffers when applied to short-form messages [START_REF] Grant | Text messaging forensics: Txt 4n6: idiolect free authorship analysis?[END_REF], as many of the earliest studies have been tested on long formal text such as books. It is thus a challenge to find new features and methods that are applicable to informal or short texts.

Studies on AA focused on various types of features that are either syntactic or semantic [START_REF] Stamatatos | Automatic text categorization in terms of genre and author[END_REF]. Patterns based on syntactic features of text are considered reliable because they are unconsciously used by authors. Syntactic features include frequencies of ngrams, character ngrams, and function words [START_REF] Koppel | Authorship attribution: What's easy and what's hard? Available at[END_REF]. Baayen, van Halteren, and Tweedie rewrote the frequencies rules for AA based on two syntactically annotated samples taken from the Nijmegen corpus [START_REF] Baayen | Outside the cave of shadows: Using syntactic annotation to enhance authorship attribution[END_REF]. Semantic features take advantage of words' meanings and their likeness, as exemplified by Clark and Hannon whose classifier quantifies how often an author uses a specific word instead of its synonyms [START_REF] Clark | A classifier system for author recognition using synonym-based features[END_REF]. Moreover, state-of-the-art systems for AA often combine a wide range of features to achieve higher accuracy. For example, the JStylo system offers more than 50 configurable features [START_REF] Mcdonald | Use fewer instances of the letter i: Toward writing style anonymization[END_REF].

Syntactic markers, however, have been less studied because of their languagedependent aspect. This paper studies complex linguistic information carried by part-of-speech (POS) patterns as a novel approach for authorship attribution of informal texts. The hypothesis is that POS patterns more frequently appearing in texts, could accurately characterize authors' styles.

The contributions of this paper are threefold. First, we define a novel feature for identifying authors based on fixed or variable length POS patterns (ngrams or skip-grams). A signature is defined for each author as the intersection of the top k most frequent POS patterns found in his or her texts, that are less frequent in texts by other authors. In this method, the system automatically chooses its most appropriate features rather than have them predefined in advance. Second, a process is proposed to use these signatures to perform AA. Third, an experimental evaluation using blog posts of 10 authors randomly chosen from the Blog Authorship Corpus [START_REF] Schler | Effects of age and gender on blogging[END_REF] shows that the proposed approach is effective at inferring authors of informal texts. Moreover, the experimental study provides answers to the questions of how many patterns are needed, whether POS skip-grams or ngrams perform better, and whether fixed length or variable-length patterns should be used.

The rest of this paper is organized as follows. Section 2 reviews related work. Section 3 introduces the proposed approach. Section 4 presents the experimental evaluation. Finally, section 5 draws the conclusions.

Related Work

The use of authorship attribution techniques goes back to the 19th century with the studies of Shakespeare's work. This section concentrates on those using POS tagging or informal short texts.

Stamatatos et al. exclusively used natural language processing (NLP) tools

for their three-level text analysis. Along with the token and analysis levels, the phrase level concentrated on the frequencies of single POS tags. They analyzed a corpus of 300 texts by 10 authors, written in Greek. Their method achieved around 80% accuracy by excluding some less significant markers [START_REF] Stamatatos | Computer-based authorship attribution without lexical measures[END_REF].

Gamon combined syntactic and lexical features to accurately identify authors [START_REF] Gamon | Linguistic correlates of style: authorship classification with deep linguistic analysis features[END_REF]. He used features such as the frequencies of ngrams, function words, and POS trigrams from a set of eight POS tags obtained using the NLPWin system. Over 95% accuracy was obtained. However, its limitation was its evaluation using only three texts, written by three authors [START_REF] Gamon | Linguistic correlates of style: authorship classification with deep linguistic analysis features[END_REF].

More recently, Sidorov et al. introduced syntactic ngrams (sngrams) as a feature for AA. Syntactic ngrams are obtained by considering the order of elements in syntactic trees generated from a text, rather than by finding n contiguous elements appearing in a text. They compared the use of sngrams with ngrams of words, POS, and characters. The 39 documents by three authors from Project Gutenberg were classified using SVM, J48 and Naïve Bayes implementations provided by the WEKA library. The best results were obtained by SVM with sngrams [START_REF] Sidorov | Syntactic n-grams as machine learning features for natural language processing[END_REF]. The limitations of this work are an evaluation with only three authors, a predetermined length of ngrams, and that all 11,000 ngrams/sngrams were used.

Variations of ngrams were also considered. For example, skip-grams were used by García-Hernández et al. [START_REF] García-Hernández | Finding maximal sequential patterns in text document collections and single documents[END_REF]. Skip-grams are ngrams where some words are ignored in sentences with respect to a threshold named the skip step. Ngrams are the specific case of skip-grams where the skip step is equal to 0. The number of skip-grams is very large and their discovery needs complex algorithms [START_REF] Sidorov | Syntactic n-grams as machine learning features for natural language processing[END_REF]. To reduce their number, a cut-off frequency threshold is used [START_REF] García-Hernández | Finding maximal sequential patterns in text document collections and single documents[END_REF].

POS ngrams have also been used for problems related to AA. Litvinova et al. used the frequencies of 227 POS bigrams for predicting personality [START_REF] Litvinova | Using part-of-speech sequences frequencies in a text to predict author personality: a corpus study[END_REF].

Unlike previous work using POS patterns (ngrams or skip-grams), the approach presented here considers not only fixed length POS patterns but also variable length POS patterns. Another distinctive characteristic is that it finds only the k most frequent POS patterns in each text (where k is set by the user), rather than using a large set of patterns or using a predetermined cut-off frequency threshold. This allows the proposed approach to use a very small number of patterns to create a signature for each author, unlike many previous works that compute the frequencies of hundreds or thousands of ngrams or skip-grams.

The Proposed Approach

The proposed approach takes as input a training corpus C m of texts written by m authors. Let A = {a 1 , a 2 , .....a m } denote the set of authors. Each author a i (1 ≤ i ≤ m) has a set of z texts T i = {t 1 , t 2 , . . . t z } in the corpus. The proposed approach is applied in three steps.

Preprocessing. The first step is to prepare blogs from the corpus so that they can be used for generating author signatures. All information that does not carry an author's style is removed, such as images, tables, videos, and links. In addition, each text is stripped of punctuation and is split into sentences using the Rita NLP library (http://www.rednoise.org/rita/). Then, every text is tagged using the Standford NLP Tagger (http://nlp.stanford. edu/software/) as it produced 97.24 % accuracy on the Penn Treebank Wall Street Journal corpus [START_REF] Toutanova | Feature-rich part-of-speech tagging with a cyclic dependency network[END_REF]. Since the main focus is analyzing how sentences are constructed by authors rather than the choice of words, words in texts are discarded and only the patterns about parts of speech are maintained. Thus, each text becomes a set of sequences of POS tags. For example, Table 1 shows the transformation of six sentences from an author in the corpus. Signature extraction. The second step of the proposed approach is to extract a signature for each author, defined as a set of part-of-speech patterns (POSP) annotated with their respective frequency. Signature extraction has four parameters: the number of POS patterns (ngrams or skip-grams) to be found k, the minimum pattern length n, the maximum length x, and the maximum gap maxgap allowed between POS tags. Frequent patterns of POS tags are extracted from each text. The hypothesis is that each text may contain patterns of POS tags unconsciously left by its author, representing his/her writing style, and could be used to identify that author accurately. For each text t, the k most frequent POS patterns are extracted using a general-purpose sequential pattern mining algorithm [START_REF] Fournier-Viger | TKS: Efficient mining of top-k sequential patterns[END_REF]. Let P OS denote the set of POS tags. Consider a sentence w 1 , w 2 , . . . w y consisting of y part-of-speech tags, and a parameter maxgap (a positive integer). A n-skip-gram is an ordered list of tags w i1 , w i2 , . . . w in where i 1 , i 2 , . . . i n are integers such that 0

< i j -i j-1 ≤ maxgap + 1(1 < j ≤ n).
A n-skip-gram respecting the constraint of maxgap = 0 (i.e. no gaps are allowed) is said to be a ngram. For a given text, the frequency of a sequence seq is the number of sequences (sentences) from the text containing seq. Similarly, the relative frequency of a sequence is its frequency divided by the number of sequences in the text. For example, the frequency of the 5-skip-gram N N P, V BP, V BP, P RP, RB is 1 in the transformed sentences of Table 1, while the frequency of the 2-skip-gram DT, N N is 4 (this pattern appears in the second, third, fifth, and sixth transformed sentences).

In the following, the term part-of-speech pattern of a text t, abbreviated as (P OSP t) k n,x , or patterns, is used to refer to the k most frequent POS patterns extracted from a text, annotated with their relative frequency, respecting the maxgap constraint, and having a length no less than n and not greater than x.

Then, the POS patterns of an author a i in all his/her texts are found. They are denoted as (P OSP a i ) k n,x and defined formally as the union of the POS patterns found in all of his/her texts: (P OSP a i ) k n,x = t∈Ti (P OSP t) k n,x . Then, the signature of each author a i is extracted. The signature s ai of author a i is the intersection3 of the POS patterns of his/her texts T i . The signature is formally defined as: (s ai ) k n,x = t∈Ti (P OSP t) k n,x . For instance, the part-of-speech patterns (P OSP 1029959 ) 41,5 of the blogger 1029959 from our corpus for maxgap = 3 are shown in Table 2. In this table, POS patterns are ordered by decreasing frequency. It can be seen that the patterns Noun (NN), and Determiner -Noun (DT-NN) appear in four of the six sentences shown in Table 1. Note that the relative frequency of each pattern is calculated as the relative frequency over all texts containing the pattern.

Moreover, the POS patterns of an author a i may contain patterns having unusual frequencies that truly characterize the author's style, but also patterns representing common sentence structures of the English language. To tell apart these two cases, a set of reference patterns and their frequencies is extracted to be used with each signature for authorship attribution. Extracting this set of reference patterns is done with respect to each author a i by computing the union of all parts of speech of the other authors 4 . This set is formally defined as follows. The Common POS patterns of all authors excluding an author a i is the union of all the other P OSP a, that is: (CP OSa i ) k n,x = a∈A∧a =ai (P OSP a) k n,x .

For example, the common POS patterns computed using authors 206953 and 2369365, and excluding author 1029959, are the patterns DT, IN and PRP. The relative frequencies (%) of these patterns for author 206953 are 67.9%, 69.6% and 79.1%, while the relatives frequencies of these patterns for author 2369365 are 64.2%, 63.0% and 58.7%. Note that the relative frequency of each pattern in CPOS is calculated as the relative frequency over all texts containing the pattern.

The revised signature of an author a i after removing the common POS patterns of all authors excluding a i is defined as:

(s ai ) k n,x = (s ai ) k n,x \ (CP OSa i ) k n,x .
When the revised signature of each author a 1 , a 2 , ...a m has been extracted, the collection of revised signatures s ,k n,x = {(s a1 ) k n,x , (s a2 ) k n,x , . . . , (s am ) k n,x } is saved. Authorship attribution. The third step of the proposed approach is to use the generated signatures to perform authorship attribution, that is to identify the author a u of an anonymous text t u that was not used for training. The algorithm takes as input an anonymous text t u , the sets of signatures s ,k n,x and the parameters n, x and k. The algorithm first extracts the part-of-speech patterns in the unknown text t u with their relative frequencies. Then, it compares the patterns found in t u and their frequencies with the patterns in the signature of each author using a similarity function. Each author and the corresponding similarity are stored as a tuple in a list. Finally, the algorithm returns this list sorted by decreasing order of similarity. This list represents a ranking of the most likely authors of the anonymous text t u . Various metrics may be used to define similarity functions. In this work, the Pearson correlation was chosen as it provided better results in initial experiments.

Experimental Evaluation

Experiments were conducted to assess the effectiveness of the proposed approach for authorship attribution using either fixed or variable-length ngrams or skipgrams of POS patterns. Our Corpus consists of 609 posts from 10 bloggers, obtained from the Blog Authorship Corpus [START_REF] Schler | Effects of age and gender on blogging[END_REF]. Blog posts are written in English and were originally collected from the Blogger website (https://www.blogger. com/). The ten authors were chosen randomly. Note that non-verbal expressions (emoticons, smileys or interjections used in web-blogs or chatrooms such as lol, hihi, and hahaaa were not removed because consistent part-of-speech tags were returned by the tagger for the different blogs. The resulting corpus has a total of 265, 263 words and 19, 938 sentences. Details are presented in Table 3.

Then, each text was preprocessed (as explained in section 3). Eighty percent (80%) of each text was used to extract each author's signature (training), and the remaining 20% was used to perform the unknown authorship attribution by comparing each text t u with each author signature (testing). This produced a ranking of the most likely author to the least likely author, for each text. Recall that our proposed approach takes four parameters as input, i.e. the minimum and maximum length of part-of-speech patterns n and x, the maximum gap maxgap, and k the number of patterns to be extracted in each text. The influence of these parameters on authorship attribution success was first evaluated. For our experiment, parameter k was set to 50, 100, and 250. For each value of k, the length of the part-of-speech patterns was varied from n = 1 to x = 4. Moreover, the maxgap parameter was set to 0 (ngrams), and from 1 to 3 (skip-grams). For each combination of parameters, we measured the success rate, defined as the number of correct predictions divided by the number of predictions. Tables 4,5, 6, and 7 respectively show the results obtained for maxgap = 0, 1, 2, 3, for various values of k, n and x. Furthermore, in these tables, results are also presented by ranks. The row R z indicates the number of texts where the author was predicted as one of the z most likely authors, divided by the total number of texts (success rate). For example, R 3 indicates the percentage of texts where the author is among the three most likely authors as predicted by the proposed approach. Since there are 10 authors in the corpus, results are shown for R z varied from 1 to 10. The first observation is that the best overall results are obtained by setting n = 2, x = 2, k = 250 and maxgap = 0. For these parameters, the author of an anonymous text is correctly identified 73.3% of the time, and 86.6% as one of the two most likely authors (R 2 ).

The second observation is that excellent results can be achieved using few patterns to build signatures (k = 250). Note that our approach was also tested with other values of k such as 50, but it did not provide better results than k = 250 (results are not shown due to space limitation). This is interesting as it means that signatures can be extracted using a very small number of the k most frequent POS patterns (as low as k = 100) and still characterize well the writing style of authors. This is in contrast with previous works that generally used a large number of patterns to define an author's signature. For example, Argamon et al. have computed the frequencies of 685 trigrams [START_REF] Argamon-Engelson | Style-based text categorization: What newspaper am i reading[END_REF] and Sidorov et al. computed the frequencies of 400/11,000 ngrams/sngrams [START_REF] Sidorov | Syntactic n-grams as machine learning features for natural language processing[END_REF]. By Occam's Razor, it can be argued that models with less patterns (simpler) may be less prone to overfitting.

Third, it can be observed that good results can also be obtained using POS skip-grams. This is interesting since skip-grams of words or POS have received considerably less attention than ngrams in previous studies. Moreover, to our knowledge no studies had previously compared the results obtained with ngrams and skip-grams for authorship attribution of informal and short texts. Fourth, it is found that using skip-grams of fixed length (bigrams) is better than using patterns of variable length. This provides an answer to the important question of whether fixed length POS patterns or variable-length POS patterns should be used. This question was not studied in previous works.

Influence of parameters n, x and k on authorship attribution for each author

The previous subsection studied the influence of parameters n, x and k on the ranking of authors for all anonymous texts. This subsection analyzes the results for each author separately. Tables 8a and 8b respectively show the success rates attributed to each author (rank R 1 ) for maxgap = 0, k = 100 and k = 250, when the other parameters are varied. It can be observed that for most authors, at least 66.7% of texts are correctly attributed. For example, for n = 2, x = 2 and k = 250, four authors have all texts correctly identified, four have 66.7% of their texts correctly classified, and and that it is unnecessary to create signatures with a large number of patterns. Moreover, it was found that using fixed length POS bigrams provided better results than using POS ngrams or using a larger gap between POS tags, and that using fixed length patterns is preferable to using variable-length patterns. To give the reader a feel of the relative efficacy of this approach on traditional long texts, we present a summary table of results on a corpus of 30 books by ten 19th century authors (c.f. Table 11 for comparative results) [START_REF] Pokou | Authorship attribution using small sets of frequent part-of-speech skip-grams[END_REF][START_REF] Pokou | Authorship attribution using variable-length part-of-speech patterns[END_REF]. In future work, we plan to develop other features and also evaluate the proposed features on other types of texts.

Acknowledgements. This work is financed by a National Science and Engineering Research Council (NSERC) of Canada research grant, and the Faculty of Research and Graduate Studies of the Université de Moncton.

Table 1 :

 1 An example of blog text transformation. Live From Ohio Its Youth Night. JJ IN NNP PRP$ NNP NNP. 2 So the Youth Rally was tonight. IN DT NNP NNP VBD NN.

	# Original Sentence	Transformed Sentence into POS Sequences
	1 3 I think it was a success.	PRP VBP PRP VBD DT NN.
	4 Maybe not.	RB RB.
	5 The activity looked complicated
	and confusing but luckily Tim

and I didn't have to do it instead we got to (...) right. DT NN VBD JJ CC JJ CC RB NNP CC PRP VBP VBP TO VB PRP RB PRP VBD TO (...) JJ. 6 And finally the skit. CC RB DT NN.

Table 2 :

 2 Part-of-speech patterns and their relative frequencies.

	Pattern	Relative Frequency (%) Part-of-speech Description
	NN	66.6	Noun, singular or mass
	DT-NN	66.6	Determiner -Noun
	NNP	50.0	Proper noun, singular
	VBP-VBP-PRP 16.6	Verb, non-3rd person singular-verb, non-3rd person singular -Personal pronoun
	NNP-VBP-VBP-PRP-RB	16.6	Proper noun, singular -Verb, non-3rd per-son singular-verb -Verb, non-3rd person singular-verb -Personal pronoun-Adverb

Table 3 :

 3 Corpus statistics.

	Author Id	Post Count	Word Count	Sentence Count
	1029959	83	29,799	2,314
	2069530	125	40,254	2,409
	2369365	119	24,148	1,692
	3182387	87	30,375	1,752
	3298664	34	26,052	2,417
	3420481	63	19,063	1,900
	3454871	37	16,322	1,722
	3520038	45	21,312	1,698
	3535101	1	24,401	1,865
	3701154	15	33,537	2,169
	Total	609	265,263	19,938

4.1 Influence of parameters n, x, k, and maxgap on overall results

Table 4 :

 4 Overall classification results using ngrams (maxgap = 0)

	(a) Top-k, for k=100.	(b) Top-k, for k=250.
	Success rate in %	Success rate in %
	n, x 1, 2 1, 3 1, 4 2, 2 3, 3	n, x 1, 2 1, 3 1, 4 2, 2 3, 3
	R1 56.7 53.3 56.7 63.3 40.0	R1 60.0 60.0 56.7 73.3 46.7
	R2 73.4 60.0 63.4 76.6 70.0	R2 73.3 76.7 73.4 86.6 70.0
	R3 80.1 73.3 73.4 76.6 73.3	R3 83.3 86.7 86.7 86.6 83.3
	R4 83.4 83.3 83.4 79.9 80.0	R4 86.6 86.7 90.0 86.6 86.6
	R5 86.7 90.0 90.1 79.9 86.7	R5 86.6 93.4 90.0 89.9 89.9
	R6 90.0 90.0 90.1 86.6 90.0	R6 93.3 93.4 93.3 89.9 93.2
	R7 90.0 90.0 90.1 86.6 90.0	R7 93.3 93.4 96.6 89.9 96.5
	R8 96.7 96.7 96.8 89.9 96.7	R8 96.6 96.7 96.6 89.9 99.8
	R9 96.7 96.7 96.8 96.6 100.0	R9 96.6 96.7 96.6 96.6 99.8
	R10 100.0 100.0 100.0 100.0 100.0	R10 100.0 100.0 100.0 100.0 100.0

Table 10 :

 10 Summary of Best Success Rate Results (Best R 1 Rank) Nb. of words 2,615,856 (30 books, 10 authors) idem (30 books, 10 authors) 265,263 (609 posts, 10 authors)

	Markers		Ngrams		Skip-grams	Ngrams vs Skip-grams
		k	50	k	250	k	250
	Parameters	n, x	1, 2	n, x maxgap	3, 3 1	n, x maxgap	2, 2 0
		R1	73.3	R1	70.0	R1	73.3
	Results in %	R2	83.3	R2	76.6	R2	86.6
		R3	90.0	R3	86.6	R3	86.6

A less strict intersection could also be used, requiring occurrences in some or the majority of texts rather than all of them.

A subset of all other authors can also be used if the set of other authors is large.

two have 33.3% of texts correctly classified. Overall, it can be thus found that the proposed approach performs very well.

It can also be found that some authors were harder to classify (author 3420481 and 3454871). After investigation, we found that the reason for the incorrect classification is that both authors have posted a same very long blog post, which represents about 50% of the length of their respective corpus. This content has a distinct writing style, which suggests that it was not written by any of these authors. Thus, it had a great influence on their signatures, and led to the poor classification of these authors.

In contrast, some authors were very easily identified with high success rate and high accuracy (cf. Table 9 for accuracies). For example, all texts by authors Table 7: Overall classification results using skip-grams with maxgap = 3 (a) Top-k, for k=100 Success rate in % n, x 1, 2 1, 3 1, 4 2, 2 3, 3


Jan Boril
email: jan.boril@unob.cz

Miroslav Jirgl
email: miroslav.jirgl@phd.feec.vutbr.cz

Rudolf Jalovecky
email: rudolf.jalovecky@unob.cz

Use of Flight Simulators in Analyzing Pilot Behavior

Keywords: Flight Simulators, Pilot Training, Human Behavior Model, Time Constants, MATLAB®, X-Plane

établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

In modern aviation flight simulations play an irreplaceable role both in pilot training and maintaining their needed habits. The level of safety in transport as well as individual aviation now considerably depends on simulation technologies that are used not only for training but also for investigation of aircraft accidents, studying airplane designs, or better understanding of ergonomic relations [START_REF] Boril | Aviation Simulation Training in the Czech AirForce[END_REF]. The importance of flight simulators is increasing proportionally with growing complexity of modern aviation systems, so training on simulators has become an inseparable part of pilot training, their professional growth and research focused on adaptability of aircraft environment to human capabilities and limits. As a result of the constant development of pilot natural working environment, flight simulators are becoming an invaluable instrument. Indispensable part of training is currently being transferred from flight training on planes to simulators for the reasons of undisputable advantages that this type of training offers. The growing significance of flight simulations is also evidenced in the latest international regulations stipulating requirements for aviation synthetic training equipment. Training efficiency can be achieved with use of synthetic equipment of a low reliability for some types of missions. Thanks to the interconnection of flight simulations with the rapidly growing field of computer technologies, the synthetic methods offer more and more opportunities in running effective training of flying crews and maintenance personnel [START_REF] Allerton | The impact of flight simulation in aerospace[END_REF].

As a result of progress made in the field of avionics, the complexity of civilian and military airplanes have increased too, which puts higher demands on crew training and strengthens the dependency on flight simulations. For the reasons of crew training and retaining their licenses many civilian airlines have established and are running large aviation training centers. Not only have flight simulation radically changed flight training methods in terms of reducing risks and increasing quality of training, but they have also resulted in a significantly higher flight safety, decreased traffic density, and positive effect on environment, when all of that is achieved in reducing cost of flight training. These trends will continue in the foreseeable future [START_REF] Allerton | The impact of flight simulation in aerospace[END_REF][START_REF] Foyle | Human Performance Modeling in Aviation[END_REF].

The main goal of this paper is to verify the potentiality of using flight simulators and subsequent application of mathematical methods to evaluation of pilot training experience. Currently, the authors are in the phase of repeated measurements running under the equal conditions and with the same pilots. The aim of which is to verify the accuracy of all analyses depending on repeatability with an emphasis laid on changes resulting from continuous training.

Mathematical Equation of Human Behavior

The literature offers us several works dealing with modelling of human behavior, including pilots. Most of these papers are based on the description (1) first presented by D.T.McRuer in the 1970s which models human behavior primarily at the level of the feedback regulator as described above. It is a linear model (transfer function) of a proportionally-derivative regulator with a second order lag and time (reaction) delay, in which each of the constants has a certain physiological or neurological interpretation [START_REF] Mcruer | Mathematical Models of Human Pilot Behavior[END_REF][START_REF] Lone | Development of a Pilot Model Suitable for the Simulation of Large Aircraft[END_REF][START_REF] Hess | Analytical Assessment of Flight Simulator Fidelity Using Pilot Models[END_REF]:

     s e s T s T s T K s X s Y s F        1 1 1 ) ( ) ( ) ( 2 1 3 . (1)
Where K Pilot gain representing a pilot´s habits in response to a certain action. It is also connected with the input and output signal ratio. T1

Neuromuscular lag time constant expressing a pilot´s delayed reaction resulting from his neuromuscular system. It ranges from 0.05 to 0.2 s and is not dependent on the training intensity. T2

Lag time constant characterizing a pilot´s promptness and agility. Thus it is related to performing acquired stereotypes and routine procedures. Ranges from 0.1 s to 5 s.

T3

Lead time constant connected with a pilot´s experience. It reflects a pilot´s capability to predict a situation that may occur. This capability is developed through training and experience and ranges from 0.2 s to 15 s. τ Time constant expressing a delayed reaction in pilot´s brain to a motional and visual perception. As a result of fatigue this constant can be extended and the pilot´s regulating capabilities may subsequently fail, thus the regulating system becomes unstable. This constant most frequently ranges from 0.3 to 1s. s Laplace operator.

The model in the form of equation ( 1) is a general model that may be used in a wide spectrum of activities connected with driving [START_REF] Havlikova | Diagnostic of Systems with a Human Operator[END_REF] or piloting. Individual constants most frequently reach the values within the ranges as shown above, and depend on pilots´ abilities to adapt to the controlled dynamics (adaptability).

Validity of this model was verified on the basis of many experiments, and primarily thanks to its simplicity it turned out to be a very efficient tool for description and modelling of a pilot´s reactions [START_REF] Hess | Analytical Assessment of Flight Simulator Fidelity Using Pilot Models[END_REF], [START_REF] Glodek | Fusion Paradigms in Cognitive Technical Systems for Human-Computer Interaction[END_REF].

The identification algorithm as written in MATLAB environment is used to identify model parameters. This algorithm is based on applying the library function fminsearch resulting from Nelder-Mead simplex algorithm [START_REF] Lagarias | Convergence Properties of the Nelder-Mead Simplex Method in Low Dimensions[END_REF]. The criterial function for searching for parameter is then in the form of the sum of squared deviations, see equation ( 2):

      n i m i y i y KRIT 1 2 ) ( ) ( . (2)
Where y real (measured) value of the output quantity (pilot´s intervention), ym modelled (estimated) value of the output quantity (pilot´s intervention), n length of data.

The Best fit [%] parameter is used for expressing the accuracy of identification because it gives an opportunity for making an easy comparison of identification results with those obtained through MATLAB -System Identification Toolbox. The equation for calculating the Best fit parameter is as follows [START_REF] Ljung | Experiments with Identification of Continuous Time Models[END_REF]:

            ) ( 1 100 y mean y y y fit m . (3)
Where y real (measured) value of the output quantity (pilot´s intervention), ym modelled (estimated) value of the output quantity (pilot´s intervention).

Description of Experimental Workplace and Measurement Procedure

Most of armed forces also use training centers equipped with flight simulators designed for basic training, transitions to another type of aircraft, and tactical training on the platform of both fixed-wing and rotary-wing aircraft. Beyond doubt simulations provide a wide range of true advantages, therefore a growing number of synthetic equipment is being introduced into modern air forces to be used in a broad spectrum of activities, mainly for training of aviation personnel [START_REF] Allerton | Principles of Flight Simulation[END_REF]. In addition, flight simulators are used by aircraft and airplane equipment manufacturers, system developers, research organizations and academies whose effort is aimed at proving the validity of studies, designs, developments and calculations of flight systems.

Flight Simulator at the University of Defence

The Department of Air Electrical Systems strives to keep up with these above mention trends, therefore the flight simulator based on X-Plane-10 software was built up to raise the quality of training and create a capacity for scientific research. The mission of the flight simulator at the University of Defence does not lie in competing with or replicating modern sophisticated simulators designed for pilot basic or improvement training. Using this simulator the authors prepare their procedures and methods for data gathering and evaluation. Currently, as described in the chapter Introduction, the authors are in the phase of repeated measurements running under the equal conditions and with the same pilots. The aim of which is to verify the accuracy of all analyses depending on repeatability with an emphasis laid on changes resulting from continuous training. Based on the knowledge obtained in such a manner, they will be able to implement the methodology of measurement and data evaluation on commercial full flight simulators. Throughout the whole world this simulator program is currently rated as a comprehensive, complex and highly efficient PC flight simulator that offers most advanced flight models and tools. This engineering tool has been designed for an easy estimate of a flight model of all categories of aircraft and their construction types. The information acquired from mathematical and physical calculations in real time are relatively precise. The basic version of X-Plane software offers dynamics of 30 airplanes, such as Bell 206 JetRanger helicopter, Cessna 172, King-Air C90 etc. From the previous experience the King-Air C90 airplane (Fig. 1) was chosen for testing a pilot´s response to an unexpected flight situation. The simulator is capable to record the flight trajectory (selected parameters) and safe them into a text file in 20 Hz frequency.

Experimental Flight Task

A flight mode was defined -altitude of 2900ft, speed 170 mph, angle of attack and pitch angle, including their change, was approximately zero (note 1 ft = 0.305 m and 1 mph = 0.447 m.s -1 ). At a certain time the altitude was step-changed to 2600 ft and the task of the pilot was to correct the altitude back to the original flight level 2900 ft. A total of 6 trainee pilots were tested in this flight mode, all of them had about 60-80 flight hours of experience in the Zlin Z-142C AF aircraft. Each pilot had the altitude changed 10 times one after another, always after putting the aircraft back into the initial flight state. The whole experiment was repeated after 6 months. The tested subjects were the same pilots. These pilots were trained within this time and the experience was increased by about 20 hours.

Measurement and Data Analysis

For the purpose of demonstrating the results, an analysis of pilot no. 1 was made, as shown below. The evaluation of other flights was carried out in a similar manner. In order to evaluate the level of pilot training, i.e. compare changes in their responses, it is advisable to create and define the so-called average pilot behavior model. The parameters of an average pilot behavior model as described by the equation (1) were identified from the average responses -red curves in Fig. 2 using the identification algorithm, as described above.

s s s s s F B         Best fit = 45.62 %
The approximation of the original responses (measured data) with use of the responses of identified models F1A(s) and F1B(s) is shown in Fig. 3. The testing measurements and data analysis were carried out similarly with other pilots. The below-stated table summarizes the results achieved from the parameter identification of pilot behavior model [START_REF] Boril | Mathematical Analysis of Human Factors Using Experimental Parameter Identification of Human Behaviour Model[END_REF][START_REF] Boril | Experimental Identification of Pilot Response Using Measured Data from a Flight Simulator[END_REF]. Letter A stands for the measurement taking place in spring 2015, whereas B for the measurement in fall 2015.

Table 1. Parameters of the identified transfer functions for pilots 1 -6. The results as shown in Table 1 clearly indicate that the values of neuromuscular lag time constant T1 remain in repeated measurements practically unchanged, which confirms the premise that their value is independent of the training intensity and is characteristic for all pilots. All pilots, except for pilot no. 3, display decreased value in response time (transport delay) τ, which means that pilots are able to response to a changed situation faster. The ratio between regulating constants T2 and T3 determines adaptability to controlled dynamics. Changes were mostly recorded even in this case.

A K.10 -4 [-] B K.10 -4 [-] A T1 [s] B T1 [s] A T2 [s] B T2 [s] A T3 [s] B T3 [s] A τ [s] B τ [s] Pilot no

Conclusion

The aim of this paper was to verify the potentiality of using flight simulators and subsequent application of mathematical methods in order to carry out an objective assessment of pilot training experience in terms of intensity and status. Two sets of measurements focused on pilots´ response (reaction) to a visual stimulus with use of a stationary flight simulator were simultaneously performed at the University of Defence in Brno. Both measurements were always running under identical conditions, i.e. with pilots who had flown around 60 -80 hours in real airplane before the first measurement. Another measurement took place after six months when pilots completed another training phase and their recorded number of flying hours increased approximately by 20 hours.

Based on the measured data the authors made mathematical analyses needed to generate models of pilot behavior. On the grounds of these models authors were then able to create the so-called average pilot behavior model. Having compared the average pilot behavior model for individual measurements (before and after training), the authors, in most cases, registered a change, which specifically means that pilots´ transport delay and the ratio between regulating constants, featuring the pilots´ adaptability to controlled dynamics, changed too. The achieved results outline a potentiality of a further development in this field in the future. Hence more sets of measurement are needed to be performed in a foreseeable time horizon to verify the applied methodology.

Fig. 1 .

 1 Fig.1. The simulator at the Department of Air Electrical Systems, University of Defence, built on the basis of X-Plane 10 simulation program.

Fig. 2

 2 presents data measured on the simulator by pilot´s no. 1 containing individual measurements performed a) in spring 2015, b) in fall 2015 (after the training). On the left side are always altitude records by airplane H depending on the stick deflection (pilot´s reaction) dv, as shown on the right side. The red curve indicates the arithmetic average of individual measurements, i.e. average pilot´s response and flight altitude reached by the plane calculated out of 10 measurements.

Fig. 2 .

 2 Fig. 2. Simulator measured data -pilot no.1: a) measured in spring 2015, b) fall 2015. Thus, the identification results in the average pilot behavior model in the following form:  spring 2015

Fig. 3 .

 3 Fig. 3. The result achieved from parameter identification of pilot behavior model (pilot no.1) : a) measured in 2015, b) fall 2015. Apparently the results indicate that practically all parameters of the model were changed. Both lead time constant T3 and lag time constant T2, being in relation with pilot´s experience and length of flying practice, increased. On the contrary, the value of transport delay τ dropped, which means that average pilot reaction is faster by almost 0.1 s. Neuromuscular lag time constant T1 remained unchanged. This constant represents dynamics of spreading impulses throughout human neuromuscular system.Its numeric value, i.e. approximately 0.13 s, also conforms to the theory. The testing measurements and data analysis were carried out similarly with other pilots. The below-stated table summarizes the results achieved from the parameter identification of pilot behavior model[START_REF] Boril | Mathematical Analysis of Human Factors Using Experimental Parameter Identification of Human Behaviour Model[END_REF][START_REF] Boril | Experimental Identification of Pilot Response Using Measured Data from a Flight Simulator[END_REF]. Letter A stands for the measurement taking place in spring 2015, whereas B for the measurement in fall 2015.

Acknowledgments. The paper was written within project Technology Agency of the Czech Republic n.TA04031376 research / development methodology training aviation specialists L410UVP E20 and grant No. FEKT-S-14-2429 -"The research of new control methods, measurement procedures and intelligent instruments in automation", and the related financial assistance was provided from the internal science fund of Brno University of Technology.