Iaroslav E A N Omelianenko
email: yaric@newground.com.ua

Applying Deep Machine Learning for psycho-demographic profiling of Internet users using O.C

In the modern era, each Internet user leaves enormous amounts of auxiliary digital residuals (footprints) by using a variety of on-line services. All this data is already collected and stored for many years. In recent works, it was demonstrated that it's possible to apply simple machine learning methods to analyze collected digital footprints and to create psycho-demographic profiles of individuals. However, while these works clearly demonstrated the applicability of machine learning methods for such an analysis, created simple prediction models still lacks accuracy necessary to be successfully applied for practical needs. We have assumed that using advanced deep machine learning methods may considerably increase the accuracy of predictions. We started with simple machine learning methods to estimate basic prediction performance and moved further by applying advanced methods based on shallow and deep neural networks. Then we compared prediction power of studied models and made conclusions about its performance. Finally, we made hypotheses how prediction accuracy can be further improved. As result of this work, we provide full source code used in the experiments for all interested researchers and practitioners in corresponding GitHub repository. We believe that applying deep machine learning for psycho-demographic profiling may have an enormous impact on the society (for good or worse) and provides means for Artificial Intelligence (AI) systems to better understand humans by creating their psychological profiles. Thus AI agents may achieve the human-like ability to participate in conversation (communication) flow by anticipating human opponents' reactions, expectations, and behavior. By providing full source code of our research we hope to intensify further research in the area by the wider circle of scholars.

Introduction

By using various on-line services, modern Internet user leaves an enormous amount of digital tracks in the form of server logs, user-generated content, etc. All these information bits meticulously saved by on-line service providers create the vast amount of digital footprints for almost every Internet user. In recent research [Lambiotte, R., and Kosinski, M., 2014], it was demonstrated that by applying simple machine learning methods it was possible to find statistical correlations between digital footprints and psycho-demographic profile of individuals. The considered psycho-demographic profile comprise of psychometric scores based on five-factor O.C.E.A.N. model of personality [START_REF] Goldberg | [END_REF] and demographic scores such as Age, Gender and the Political Views. The O.C.E.A.N. is an abbreviation for Openness (Conservative and Traditional -Liberal and Artistic), Conscientiousness (Impulsive and Spontaneous -Organized and Hard Working), Extroversion (Contemplative -Engaged with outside world), Agreeableness (Competitive -Team working and Trusting), and Neuroticism (Laid back and Relaxed -Easily Stressed and Emotional).

In this work we decided to test whether applying advanced machine learning methods to analyze digital footprints of Internet users can outperform results of previous research conducted by M. Kosinski: Mining Big Data to Extract Patterns and Predict Real-Life Outcomes [Kosinski et. al, 2016]. For our experiments we used data corpus comprising of psycho-demographic scores of individuals and their digital footprints in form of Facebook likes. The data corpus used in experiments kindly provided by M. Kosinski through corresponding web site: http://dataminingtutorial.com.

We started our experiments with building simple machine learning models based on linear/logistic regression methods as proposed by M. Kosinski in [Kosinski et. al, 2016]. By training and execution of simple models we estimated basic predictive performance of machine learning methods against available data set. Then we continued our experiments with advanced machine learning methods based on shallow and deep neural network architectures.

The full source code of our experiments provided in form of GitHub repository: https://github.com/NewGround-LLC/ psistats

The source code is written in R programming language [R Core [START_REF] Core | R: A language and environment for statistical computing[END_REF] which is highly optimized for statistical data processing and allows to apply advanced deep machine learning algorithms by bridging with Google Brain's TensorFlow framework [Google Brain Team, 2015]. This paper is organized as follows: In Section 2, we describe data corpus structure, and necessary data preprocessing steps to be applied. It is followed in Section 3 by details about how to build and run simple prediction models based on linear and logistic regression with results of their execution. In Section 4, we provide details how to create and execute advanced prediction models based on artificial neural networks. Finally, in Section 6 we compare the performance of different machine learning methods studied in this work and draw conclusions about the predictive power of studied machine learning models.

Data Corpus Preparation

In this section, we consider the creation of input data corpus from the publicly available data set, and it's preprocessing to allow further analysis by selected machine learning algorithms.

Data Set Description

The data set kindly provided by M. Kosinski and used in this work contains psycho-demographic profiles of n u = 110 728 Facebook users and n L = 1 580 284 of associated Facebook likes. For simplicity and manageability, the sample is limited to U.S. users [Kosinski et. al, 2016]. The following three files can be downloaded from corresponding web sitehttp://dataminingtutorial.com:

1. users.csv: contains psycho-demographic user profiles. It has n u = 110 728 rows (excluding the row holding column names) and nine columns: anonymised user ID, gender ("0" for male and "1" for female), age, political views ("0" for Democrat and "1" for Republican), and scores of five-factor model of personality [START_REF] Goldberg | [END_REF].

2. likes.csv: contains anonymized IDs and names of n L = 1 580 284 Facebook (FB) Likes. It has two columns: ID and name.

3. users-likes.csv: contains the associations between users and their FB Likes, stored as user-Like pairs. It has nu L = 10 612 326 rows and two columns: user ID and Like ID. An existence of a user-Like pair implies that a given user had the corresponding Like on their profile.

Data pre-processing

The raw data preprocessing is an important step in machine learning analysis which will significantly reduce the time needed for analysis and result in better prediction power of created machine learning models.

The detailed description of data corpus preprocessing steps applied during this research given hereafter.

Construction of sparse users-likes matrix and matrix trimming

To use provided data corpus in machine learning analysis it should be transformed first into optimal format. Taking into account properties of provided data corpus (user can like specific topic only once and most users has generated small amount of likes) its natural to present it as sparse matrix where most of data points is zero (the resulting matrix density is about 0,006% -see Table 1). The sparse matrix data structure is optimized to perform numeric operations on sparse data and considerably reduce computational costs compared to the dense matrix used for same data set.

After users-likes sparse matrix creation, it was trimmed by removing rare data points. As a result, the significantly reduced data corpus was created, imposing even lower demands on computational resources and more useful for manual analysis to extract specific patterns. The descriptive statistics of users-likes matrix before and after trimming present in Table The users-likes matrix can be constructed from provided three comma-separated files with the help of accompanying script written in R language: src/preprocessing.R. To use this script make sure that input_data_dir variable in the src/config.R points to the root directory where sample data corpus in the form of .CSV files were unpacked.

To start preprocessing and trimming, run the following command from terminal in the project's root directory: $ R s c r i p t . / s r c / p r e p r o c e s s i n g . R \\ -u 150 -l 50 where: -u is the minimum number of users per like u L , and -l is the minimum number of likes per user L u to keep in resulting matrix.

The values for the minimum number of users per like u L and the minimum number of likes per user L u was selected based on recommendations given in [Kosinski et. al, 2016]. We have experimented with other set of parameters as well (u L = 20 and L u = 2), but accuracy of trained prediction models degraded as result.

Data imputation of missed values

The The descriptive statistics for data imputation applied to political variable using LDA method with number of multiple imputations equals to m = 5. The plausibility of applied multivariate imputation can be confirmed by low values in column fmi and lambda. The column fmi contains the fraction of missing information as defined in [Rubin DB, 1987], and the column lambda is the proportion of the total variance that is attributable to the missing data λ = B+ B m T ).

Dimensionality reduction with SVD

After two previous steps, the resulting users-likes sparse matrix still has a considerable number of features per data sample: 8 523 of feature columns. To make it even more maintainable, we considered applying singular value decomposition [Golub, G. H., and Reinsch, C. 1970], representing eigendecomposition-based methods, projecting a set of data points into a set of dimensions. As mentioned in [Kosinski et. al, 2016], reducing the dimensionality of data corpus has number of advantages:

• With reduced features space we can use fewer number of data samples, as it is required by most of the machine learning analysis algorithms that number of data samples exceeds the number of features (input variables) • It will reduce risk of overfitting and increase statistical power of results • It will remove multicollinearity and redundancy in data corpus by grouping related features (variables) in single dimension • It will significantly reduce required computational power and memory requirements • And finally it makes it easier to analyze data by hand over small set of dimensions as opposite to hundreds or thousands of separate features

To apply SVD analysis against generated users-likes matrix run the following command from project's root directory: $ R s c r i p t . / s r c /svd_varimax . R \\ --svd_dimensions 50 --apply_varimax t r u e where: -svd_dimensions is the number of SVD dimensions for projection, and -apply_varimax is the flag to indicate whether varimax rotation should be applied afterwards.

Factor rotation analysis

The factor rotation analysis methodology can be used to further simplify SVD dimensions and increase their interpretability by mapping the original multidimensional space into a new, rotated space. Rotation approaches can be orthogonal (i.e., producing uncorrelated dimensions) or oblique (i.e., allowing for correlations between rotated dimensions).

In this work during data preprocessing we applied one of the most popular orthogonal rotations -varimax. It minimizes both the number of dimensions related to each variable and the number of variables related to each dimension, thus improving the interpretability of the data by human analysts.

For more details on rotation techniques, see [Abdi, H., 2003].

Regression analysis

There is an abundance of methods developed to build prediction machine learning models suitable for analysis of large data sets. They ranging from sophisticated methods such as Deep Machine Learning [START_REF] Goodfellow | [END_REF], probabilistic graphical models, or support vector machines [START_REF] Cortes | [END_REF], to much simpler, such as linear and logistic regressions [Yan, Su, 2009].

Starting with simple methods is a common practice allowing the creation of good baseline prediction model with minimal computational efforts. The results obtained from these models can be used later to debug and estimate the quality of results obtained from advanced models.

Regression model description and model-specific data preprocessing

In our data corpus, we have eight dependent variables with psycho-demographic scores of individuals to be predicted. Among those variables, six have continuous values, and two has categorical values. To build the prediction model for variables with continuous values we applied linear regression analysis and for variables with categorical valueslogistic regression analysis.

Hereafter we describe the rationale for selection of appropriate regression analysis methods as well as a description of model-specific data preprocessing needed.

Linear regression analysis

The linear regression is an approach for modeling the relationship between continuous scalar dependent variable y and one or more explanatory (or independent) variables denoted X. The case of one explanatory variable is called simple linear regression. For more than one explanatory variable, the process is called multiple linear regression [David A. Freedman, 2009]. In linear regression, the relationships are modeled using linear predictor function y = Θ T X whose unknown model parameters Θ estimated from the input data. Such models are called linear models [Hilary L. Seal, 1967].

We used linear regression to build prediction models for analysis of six continuous dependent variables in given data corpus: Age, Openness, Conscientiousness, Extroversion, Agreeableness, and Neuroticism

Logistic regression analysis

The logistic regression is a regression model where the dependent variable is categorical [David A. Freedman, 2009].

It measures the relationship between the categorical dependent variable and one or more independent variables by estimating probabilities using a logistic function σ(x) = 1 1+e -x , which is the cumulative logistic distribution [Rodriguez, G., 2007].

We considered only specialized binary logistic regression because categorical dependent variables found in our data corpus (Gender and Political Views) are binominal, i.e. have only two possible types, "0" and "1".

Cross-Validation

We applied k-fold cross-validation to help with avoiding model overfitting when evaluating accuracy scores of prediction models. In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k -1 subsamples are used as training data. The cross-validation process is then repeated k times (the folds), with each of the k subsamples used exactly once as the validation data. The k results from the folds can then be averaged to produce a single estimation. The advantage of this method is that all observations are used for both training and validation, and each observation is used for validation exactly once. The 10-fold cross-validation is commonly used, but in general, k remains an unfixed parameter [Kohavi, Ron, 1995].

Dimensionality reduction

To reduce the number of features (input variables) in the data corpus was applied singular value decomposition (SVD) with subsequent varimax factor rotation analysis. The number of the varimax-rotated singular value decomposition dimensions (K) has a considerable impact on the accuracy of model predictions. To find an optimal number of SVD dimensions, we performed analysis of relationships between K and accuracy of model predictions by creating series of regression models for different values of K. Then we plotted prediction accuracy of regression models against chosen number of K SVD dimensions. Typically the prediction accuracy grows rapidly within lower ranges of K and may start decreasing once the number of dimensions becomes large. Selecting value of K that marks the end of a rapid growth of prediction accuracy values usually offers decent interpretability of the input data topics. In general, the larger K values often results in better predictive power when preprocessed data corpus further analyzed with specific machine learning algorithm [START_REF] Zhang | Singular value decomposition and its visualization[END_REF]. See Figure 1 for results of our experiments.

To start SVD analysis run the following command from terminal in the project's root directory: $ R s c r i p t . / s r c / a n a l y s i s . R

The resulting plots will be saved as "Rplots.pdf" file in the project root and include two plots:

• the plot with relationships between the accuracy of prediction models for each dependent variable and the number of the varimax-rotated SVD dimensions used for dimensionality reduction (Figure 1). With this plot, The relationship between the accuracy of predicting psycho-demographic traits and the number of the varimax-rotated singular value decomposition dimensions used for dimensionality reduction. The results suggest that selecting K = 50 SVD dimensions might be a good choice for building models predicting almost all dependent variables, as it offers accuracy that is close to what seems like the higher asymptote for this data. But for Openness, Extroversion, and Agreeableness dependent variables prediction results can be slightly improved with higher numbers of K SVD dimensions selected.

it's easy to visually find an optimal number of K SVD dimensions to maximize predicting power of regression model per particular dependent variable. • the heat map of correlations between scores of digital footprints of individuals projected on specific number of varimax-rotated SVD dimensions and each dependent variable (Figure 2). This plot can be used to find most correlated dependent variables visually. Later, it will be shown that predictive models for dependent variables with higher correlation have better prediction accuracy.

Implementing simple prediction models and its accuracy evaluation

The given data corpus has eight dependent variables for which to build prediction models. The simple machine learning methods such as regression analysis mostly applied to estimate single dependent variable. But in case when multiple dependent variables need to be estimated the specialized methods of multivariate regression analysis can be used. Taking into account that our dependent variables have different types (continuous and nominal) which require different regression analysis methods to be applied, we decided to build separate regression models per each dependent variable. The metric to evaluate accuracy of prediction model is related to the regression method used in the model. In this research we have considered following metrics:

• the accuracy of prediction model applied to the continuous dependent variable will be measured as Pearson product-moment correlation [START_REF] Gain | The frequency distribution of the product moment correlation coefficient in random samples of any size draw from non-normal universes[END_REF] • the accuracy of prediction model applied to the bi-nominal dependent variable will be measured as area under the receiver-operating characteristic curve coefficient (AUC) [Sing, Sander, Beerenwinkel, Lengauer, 2005] Before executing models make sure that data corpus already preprocessed as described in Subsection: "Construction of sparse users-likes matrix and matrix trimming"

When data corpus is ready, the following command can be executed to start linear/logistic regression models building and its predictive performance evaluation (run command from terminal in the project's root directory): $ R s c r i p t . / s r c / r e g r e s s i o n _ a n a l y s i s . R

The results will be saved into the file "out/pred_accuracy_regr.txt". The prediction accuracy of regression models for data corpus trimmed to contain 150 users-per-like and 50 likes-per-user and varimax-rotated against K = 50 SVD dimensions presented in Table 3. The heat map suggests that Age, Gender, and the Political view dependent variables have maximum correlation with a maximal number of SVD dimensions. The higher correlation will result in higher prediction power of regression model for particular dependent variable (which will be shown later).

From the Table 3 it can be seen that prediction accuracy of linear/logistic regression models differs per each dependent variable. Furthermore for most variables the accuracy is too low to be applied in real-life predictions. The most accurate predictions made for Gender, Age, and Political view with Openness following after. That correlates well with our previous analysis of SVD correlations heat map (see Figure 2). In general only prediction model for Gender is accurate enough to be useful in real-life applications. Thus, simple linear/logistic regression models can not be used to accurately estimate psycho-demographic profiles of Internet users based only on their Facebook likes.

In following sections, we will test if applying advanced deep machine learning methods can improve prediction accuracy any further.

Fully Connected Feed Forward Artificial Neural Networks

In this work, we considered multilayer fully connected feed-forward neural networks (NN) for building simple (shallow) and deep machine learning NN models. The fully connected NN characterized by interconnectedness of all units of one layer with all units of the layer before it in the graph. The feed-forward NN is not allowed to have cycles from latter layers back to the earlier.

Hereafter we will describe Artificial NN architectures evaluated and prediction accuracy results obtained.

The Shallow Feed Forward Artificial NN evaluation

A Shallow Neural Network (SNN) is an artificial neural network with one hidden layer of units (neurons) between input and output layers. Its hidden units (neurons) take inputs from input units (columns of input data matrix) and feeds into the output units, where linear or categorical analysis performed. To mimic biological neuron, hidden units in the neural network apply specific non-linear activation functions. One of the popular activation functions is ReLU non-linearity that we considered as activation function for the units of hidden layers in studied network architectures [START_REF] Nair | [END_REF]. It improves information disentangling and linear separability producing efficient variable size representation of model's data. Furthermore ReLU activation is computationally cheaper: there is no need for computing the exponential function as in case of sigmoid activation [Glorot, Bordes, Bengio, 2011].

To reduce overfitting was applied dropout regularization with drop-probability 0.5, which means that each hidden unit if randomly omitted from the network with the specified probability. This helps to break the rare dependencies that can occur in the training data [Hinton, G. et al., 2012].

The NN architecture was build using Google Brain's TensorFlow library -an open source software library for numerical computation using data flow graphs. [Google Brain Team, 2015] Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. The resulting two layer (one hidden layer) ANN's architecture graph depicted in Figure 5 As the loss function to be optimized was selected Mean Squared Error (MSE) with Adam optimizer (Adaptive Moment Estimation) to estimate it's minimum. The Adam optimizer was selected for it's proven advantages some of them are that the magnitudes of parameter updates are invariant to rescaling of the gradient, its step sizes are approximately bounded by the step size hyper-parameter, it does not require a stationary objective, it works with sparse gradients, and it naturally performs a form of step size annealing [Kingma, Ba, 2014]. The batch size was selected to be 100. We also tested training with batch size 10 but found no statistically relevant prediction accuracy difference between runs with either batch size but reducing batch size considerably increased training time of NN models.

It was found that optimal number of SVD dimensions for Shallow ANN is K = 128, with number of units in the hidden layer 512, and learning rate γ = 0.0001. See Table 4. The optimal learning rate was selected by comparing ratio of survived hidden units with non zero ReLU activation and monitoring loss function over iterations plot (Figure 3). With presented hyper-parameters it was achieved maximum ratio 0.57 of zero ReLU activations for largest learning rate value, which is acceptable taking into account tendency of ReLU to saturate at zero during gradient back propagation stage when strong gradients applied, due to high learning rates [START_REF] Nair | [END_REF]. The maximal number of iterations (50 000) and correspondingly number of training epochs was selected based on loss function plot (Figure 3). With series of experiments it was selected as optimal the learning rate value γ = 0.0001, which gives smooth loss function, stable acceptable number of "dead" neurons after ReLU activation (ReLU zero activations ratio is about 0.5), and best prediction scores among runs (see Table 5). The accompanying launch script provided to conduct experiments under Unix:

$ . / eval_mlp_1 . sh u l _ s v d _ m a t r i x _ f i l e

where: ul_svd_matrix_file the path to the preprocessed users-likes matrix with dimension of feature columns reduced as described in Construction of sparse users-likes matrix and matrix trimming

The source code of shallow ANN implementation used for experiment can be found in src/mlp.R of accompanying GitHub repository.

Feed Forward Deep Learning Networks Architecture Evaluation

A Deep Neural Network (DNN) is an artificial neural network with multiple hidden layers of units between the input and the output layers. The first hidden layer will take inputs from each of the input units and the subsequent hidden layer will take inputs from the outputs of previous hidden layer's units [Christopher M. Bishop, 1995]. Similar to shallow, deep neural network can model complex non-linear relationships. But added extra layers enable composition of features from lower layers, giving the potential of modeling complex data with fewer units than a similarly performing shallow network [Bengio, Yoshua, 2009]. Deep learning discovers intricate structure in large data sets by using the backpropagation algorithm to indicate how a machine should change its internal parameters that are used to compute the representation in each layer from the representation in the previous layer [LeCun, Bengio, Hinton, 2015].

As with shallow ANNs, many issues can arise with training of the deep neural networks, with two most common problems -are overfitting and computation time [Tetko, Livingstone, Luik, 1995].

Hereafter we will consider DNN architectures studied in this research.

The three-layer Deep Learning Network Architecture Evaluation

Our experiments with deep learning networks we started with simple DNN architecture comprising of two hidden layers with ReLU activation and dropout after each hidden layer with keep probability of 0.5. The experimental network graph depicted in Figure 7.

Prediction accuracy, γ = 0.0001 We started with learning rate γ = 0.0001 which resulted in best prediction performance for shallow ANN and attempted series of experiments to estimate optimal value of K SVD dimensions. The optimal prediction accuracy of DNN model was achieved with K = 256 SVD dimensions and two hidden layers comprising of [512,256] units correspondingly. Similar prediction accuracy can be achieved with K = 1024 SVD dimensions and [2048,1024] units per layer with learning rate γ = 10 -5 . But we have not considered later set of hyper-parameters due to its extra computational overhead while giving statistically same results as former set. The results of experiments presented in Table 6.

K = 50 K = 128 K = 256 K = 512 K = 1024
After finding optimal values of K SVD dimensions and number of units per hidden layer, we have conducted series of experiments to determine optimal initial learning rate value for found hyper-parameters. It was experimentally confirmed that initial learning rate value γ = 0.0001 is the optimal one. See In our experiments, we applied exponential learning rate decay with the number of steps before decay 10 000 and decay rate of 0.96. Such scheme has positive effect on network convergence speed due to the learning rate annealing effect, which gives a system the ability to escape from poor local minima to which it might have been initialized [START_REF] Kirkpatrick | [END_REF]. We selected batch size 100 as optimal for this experiment.

The accompanying launch script provided to conduct experiments under Unix:

$ . / eval_dnn . sh u l _ s v d _ m a t r i x _ f i l e where: ul_svd_matrix_file is the path to the preprocessed users-likes matrix with dimension of feature columns reduced as described in Construction of sparse users-likes matrix and matrix trimming

The source code of DNN with two hidden layers implementation used for experiment can be found in src/dnn.R of accompanying GitHub repository.

The four-layer Deep Learning Network Architecture Evaluation

This architecture comprise of three hidden layers with ReLU activation and one output linear layer. All network layers are fully connected and network architecture is feed-forward as in all previous NN experiments. The experimental network graph depicted in Figure 8.

We have tested two dropout regularization schemes: (a) dropout applied after each hidden layer with keep-probability 0.5; (b) dropout applied after each second hidden layer with keep probability calculated by formula: p d = i 2n , (where: n -number of dropouts, i -current dropout index).

It was found that former scheme gives better results than the last one. Thus for final evaluation run, we applied dropout regularization after each hidden layer.

Based on our previous experiments with more shallow networks we decided to start with following hyper-parameters: learning rate γ = 0.0001, learning rate decay step 10 000 with decay rate 0.96, K = 128 SVD dimensions, and hidden layers configuration - [256,128,128]. To find the optimal number of K SDV dimensions and hidden layers configurations we have conducted series of experiments trying various combinations. The heuristic applied to select the number of units per hidden layer is rather naive and assumes that with dropout probability of 0.5 half of the units will be saturated to zero at ReLU activation. Thus we decided to have the number of units in the first hidden layer to be twice as much as the number of features in input data (K). The results of experiments present in Table 8 Despite the fact that best accuracy was achieved with K = 1024 SDV dimensions and the number of units in hidden layers [2048,1024,1024] respectively, it was detected slight model overfitting during training/validation with these hyper-parameters applied. So, we have decided to stop increasing K and number of hidden units as it will give no further gain in prediction accuracy against validation data set and even may lead to worsening of validation accuracy with greater overfitting level. (See Figure 6)

K = 128 K = 256 K = 512 K =
The accompanying launch script provided to conduct experiments under Unix:

$ . / eval_3dnn . sh u l _ s v d _ m a t r i x _ f i l e where: ul_svd_matrix_file is the path to the preprocessed users-likes matrix with dimension of feature columns reduced as described in Construction of sparse users-likes matrix and matrix trimming

The source code of DNN with three hidden layers implementation used for experiments can be found in src/3dnn.R of accompanying GitHub repository.

Future Work

With conducted experiments, we have found that prediction accuracy differs considerably among machine learning methods studied and best results was achieved by using advanced methods based on neural networks architectures. At the same time, the prediction accuracy per individual dependent variable also differs per particular prediction model and selected set of hyper-parameters. From experimental results it can be seen that specific combination of NN architecture with given set of hyper-parameters are best suited for one dependent variable but worsened predictive power for some of the others.

In future studies, it is interesting to investigate this dependency and build separate NN models per each dependent variable as it was done in case of simple machine learning methods (see Section: 'Regression analysis') Also, it seems promising to apply methodology described in [START_REF] Ba | [END_REF] which provide evidence that shallow networks are capable of learning the same functions as deep learning networks, and often with the same number of parameters as the deep learning networks. In [START_REF] Ba | [END_REF] it was shown that with wide shallow networks it's possible to reach the state-of-the-art performance of deep models and reduce training time by the factor of 10 using parallel computational resources (GPU).

Conclusion

From our experiments we found that weak correlation exists between most of O. Among all studied machine learning prediction models the best overall accuracy was achieved with Shallow Neural Network architecture. We hypothesize that this may be the result of its ability to learn best parameters space function within an optimal number of SVD dimensions applied to the input data set (users-likes sparse matrix). Adding extra hidden layers either leads to model overfitting when the number of SVD dimensions is too high, or underfitting when the number of SVD dimensions is too low. Also, it's interesting to notice that performance of shallow networks and deep learning networks with two hidden layers are comparable, while with introducing of third and more hidden layers it drops significantly. Thus we can conclude that no further improvements can be gained with extra hidden layers introduced. See Table 9.

Gathered experimental data confirms that advanced machine learning methods based on variety of studied artificial neural networks architectures outperform simple machine learning methods (linear and logistic regression) described in previous research conducted by M. Kosinski [Kosinski et. al, 2016]. We believe that further prediction accuracy improvements can be achieved by building separate advanced machine learning models per dependent variable, which is the subject of our future research activities.

A. The SNN evaluation plots

The following pages provide plots and diagrams related to evaluation of two layer (shallow) feed forward artificial neural network with one fully connected hidden layer and linear output layer. By examining weights histograms it may be noticed that middle has widest base with sharp peak which means that layer converged, but search space was widest among all runs. The left one has narrower base and sharp peak, which means that layer converged withing narrower search space and as result has better prediction power. The right one has narrow base but wide plateau at the top, which means that search space is narrow but algorithm still failed to converge. The left and middle histograms has sharp peaks compared to right one, which may be a signal that their learning rate values has more relevance for algorithm convergence and as result we have better predictions for those learning rates.

Figure 1 :

 1 Figure 1:The relationship between the accuracy of predicting psycho-demographic traits and the number of the varimax-rotated singular value decomposition dimensions used for dimensionality reduction. The results suggest that selecting K = 50 SVD dimensions might be a good choice for building models predicting almost all dependent variables, as it offers accuracy that is close to what seems like the higher asymptote for this data. But for Openness, Extroversion, and Agreeableness dependent variables prediction results can be slightly improved with higher numbers of K SVD dimensions selected.

Figure 2 :

 2 Figure 2:The heat map is presenting correlations between scores of digital footprints of individuals projected on K = 50 varimax-rotated singular value decomposition dimensions and scores of psycho-demographic traits of individuals. The heat map suggests that Age, Gender, and the Political view dependent variables have maximum correlation with a maximal number of SVD dimensions. The higher correlation will result in higher prediction power of regression model for particular dependent variable (which will be shown later).

Figure 3 :

 3 Figure 3: The training process evaluation based on loss values and ReLU zero activations per number of iterations. With higher learning rate (γ = 0.001 -orange) we have fast convergence but ratio of ReLU-zero activations is higher than 0.5 and quickly rising with relatively low evaluated prediction accuracy, which implies that optimum was missed. With medium learning rate (γ = 0.0001violet) we have smooth loss function plot with ratio of ReLU-zero activations bellow 0.5, giving best prediction scores among all three runs. With lowest learning rate (γ = 0.00001 -purple) we can see that learning struggled to find global minimum, reduced speed of convergence, and despite the lowest ReLU zero activations rate -worst prediction accuracy among all runs due to high loss values.

Figure 4 :

 4 Figure 4: The histograms of various tensors collected during three runs within hidden layer (Left: γ = 0.0001, middle: γ = 0.001, right: γ = 0.00001).By examining weights histograms it may be noticed that middle has widest base with sharp peak which means that layer converged, but search space was widest among all runs. The left one has narrower base and sharp peak, which means that layer converged withing narrower search space and as result has better prediction power. The right one has narrow base but wide plateau at the top, which means that search space is narrow but algorithm still failed to converge. The left and middle histograms has sharp peaks compared to right one, which may be a signal that their learning rate values has more relevance for algorithm convergence and as result we have better predictions for those learning rates.

Figure 5 :

 5 Figure 5: The tensor network graph for multi layer perceptron with one hidden layer (fully_connected), and one linear output layer (fully_connected1). The input layer presented as input tensor placeholder. The hidden layer has ReLU activation nonlinearity. The loss function is MSE (mean squared error). The train optimizer is Adam (for Adaptive Moment Estimation).

Figure 7 :

 7 Figure 7: The tensor network graph for DNN with two hidden layers. The input layer presented as input tensor placeholder. The hidden layers has ReLU activation nonlinearity. The loss function is MSE (mean squared error). The train optimizer is Adam (for Adaptive Moment Estimation).

Figure 8 :

 8 Figure 8: The tensor network graph for DNN with three hidden layers. The input layer presented as input tensor placeholder. The hidden layers has ReLU activation nonlinearity. The loss function is MSE (mean squared error). The train optimizer is Adam (for Adaptive Moment Estimation).

  .

	Descriptive statistics Raw Matrix Trimmed Matrix
	# of users	110 728	19 742
	# of unique Likes	1 580 284	8 523
	# of User-Like pairs	10 612 326	3 817 840
	Matrix density	0,006%	2,269%
	Likes per User
	Mean	96	193
	Median	22	106
	Minimum	1	50
	Maximum	7 973	2 487
	Users per Like
	Mean	7	448
	Median	1	290
	Minimum	1	150
	Maximum	19 998	8 445

Table 1 :

 1 The descriptive statistics of raw and trimmed users-likes matrix with minimum users per like threshold set to u L = 150 and minimum likes per user L u = 50

  raw data corpus has missed values in column with 'Political' dependent variable data. Before building the prediction model for this dependent variable, it is advisable to impute missed values. In this work, we applied multivariate imputation using LDA method with number of multiple imputations equals to m = 5 to fill missed values as described in[van Buuren, Groothuis-Oudshoorn, 2011].

	The data imputation performed by the same src/preprocessing.R script, as part of users-likes matrix creation routine.
	The summary statistics for data imputation applied to political variable, presented in Table 2.
		est	se	t	df Pr(>|t|) lo 95 hi 95 nmis fmi lambda
	(Intercept)	1.39 0.01 102.29 1240.53	0.00	1.36	1.41	NA 0.06	0.05
	gender	-0.02 0.01	-2.80	25.27	0.01 -0.04 -0.01	0 0.44	0.40
	age	0.00 0.00	-0.73	577.61	0.47	0.00	0.00	0 0.09	0.08
	ope	-0.23 0.00 -68.10 2446.16	0.00 -0.23 -0.22	0 0.04	0.04
	con	0.05 0.00	10.92	20.28	0.00	0.04	0.06	0 0.49	0.44
	ext	0.03 0.00	6.44	14.40	0.00	0.02	0.04	0 0.58	0.53
	agr	0.02 0.00	6.72	189.32	0.00	0.02	0.03	0 0.15	0.14
	neu	-0.01 0.00	-2.07	95.30	0.04 -0.02	0.00	0 0.22	0.20

Table 2 :

 2

Table 3 :

 3 The predictive accuracy of linear and logistic regression models per depended variable (for u L = 150, L u = 50, and K = 50 SVD dimensions).

  Prediction accuracy, γ = 0.0001Trait Variable K = 50 K = 128 K = 256 K = 512

	Gender	gender	93.76% 93.61%	93.00%	93.82%
	Age	age	85.00% 85.07%	85.03%	83.59%
	Political view	political 66.65% 66.86%	67.28%	65.32%
	Openness	ope	47.22% 51.24%	47.26%	44.81%
	Conscientiousness con	28.56% 28.45%	29.65%	28.26%
	Extroversion	ext	32.53% 32.87%	30.44%	30.32%
	Agreeableness	agr	25.21% 27.66%	24.58%	24.80%
	Neuroticism	neu	33.53% 36.24%	33.09%	31.96%
		Mean 51.58% 52.75%	51.29%	50.36%

Table 4 :

 4 The predictive accuracy results of SNN per K SVD dimensions with learning rate γ = 0.0001 and 512 units in the hidden layer.

Table 5 :

 5 The predictive accuracy results of SNN per learning rate with K = 128 SVD dimensions and 512 units in the hidden layer.

Table 6 :

 6 The predictive accuracy results of three layer DNN per K SVD dimensions with learning rate γ = 0.0001, with sizes of hidden layers presented in third table header row as [hidden 1 , hidden 2 ].

Table 7

 7 Prediction accuracy, K = 256 SVD

	Trait	Variable γ = 0.001 γ = 0.0001 γ = 0.00001
	Gender	gender	90.89%	92.53%	82.75%
	Age	age	79.60%	80.67%	79.44%
	Political view	political	67.33%	67.06%	59.69%
	Openness	ope	44.73%	47.68%	39.48%
	Conscientiousness con	21.41%	25.68%	22.86%
	Extroversion	ext	24.62%	34.35%	24.07%
	Agreeableness	agr	17.27%	26.48%	16.34%
	Neuroticism	neu	27.72%	35.94%	28.53%
		Mean	46.70%	51.30%	44.14%

Table 7 :

 7 The predictive accuracy results of three layer DNN per learning rate with K = 256 SVD dimensions, and with sizes of hidden layers 512 and 256 correspondingly.

Table 8 :

 8 The predictive accuracy results of three layer DNN per K SVD dimensions with the learning rate γ = 0.0001. The number of units in hidden layers differ per configuration and presented as [hidden 1 , hidden 2 ] in third table header row

  C.E.A.N. psychometric scores of individuals and collected Facebook likes associated with them. Either simple or advanced machine learning algorithms that we have tested provided poor prediction accuracy for almost all O.C.E.A.N. personality traits. It seems not feasible yet to use machine learning models to accurately estimate psychometric profile of an individual based only on Facebook likes. But we believe that by complementing Facebook likes of user with additional data points, it is possible to greatly improve accuracy of machine learning prediction models for psychometric profile estimation.At the same time, we have found a strong correlation with demographic traits of individuals such as Age, Gender, and the Political Views with their Facebook activity (likes). Our experiments confirmed that its possible to use advanced machine learning methods to build the correct demographic profile of an individual based only on collected Facebook likes.

						DNN
			Regression	SNN	512,256 2048,1024,1024
	Trait	Var.	K = 50	K = 128 K = 256	K = 1024
	Gender	gen.	93.65%	93.61%	92.53%	91.03%
	Age	age	61.17%	85.07%	80.67%	84.96%
	Political view	polit.	68.36%	66.86%	67.06%	66.52%
	Openness	ope	44.02%	51.24%	47.68%	44.86%
	Conscientious. con	25.72%	28.45%	25.68%	28.72%
	Extroversion	ext	30.26%	32.87%	34.35%	26.85%
	Agreeableness agr	23.97%	27.66%	26.48%	20.02%
	Neuroticism	neu	29.11%	36.24%	35.94%	28.97%
		Mean	47.03%	52.75%	51.30%	48.99%

Table 9 :

 9 The comparison of prediction accuracy for best prediction models found. The best prediction accuracy demonstrated by shallow neural network (SNN) followed by three-layered deep neural network(DNN [512,256]). The further increase of K SVD dimensions and adding of extra hidden layers lead to model overfitting and degradation of accuracy over validation data set.

B. Deep NN evaluation plots

The following pages provide plots and diagrams related to evaluation of studied deep neural networks.


Artur H Kronbauer
email: arturhk@gmail.com

Celso A S Santos

Vaninha Vieira
email: vaninha@ufba.br

Smartphone Applications Usability Evaluation: A Hybrid Model and Its Implementation

Keywords: Usability Evaluation, Smartphone Application, Remote Usability Evaluation, Usability Testing

Evaluating the usability of smartphone applications is crucial for their success, so developers can learn how to adapt them considering the dynamicity of mobile scenarios. The HCI community recommends considering different requirements when evaluating those applications, such as quantitative data (metrics), subjective evaluation (users' impressions) and context data (e.g. environment and devices conditions). We observed a lack in the literature of approaches that support those three requirements combined into a single experiment; generally one or a pair of them is used. Besides, performing usability evaluation on real mobile scenarios is hard to achieve and most proposals are based on laboratory-controlled experiments. In this paper, we present our proposal for a hybrid usability evaluation of smartphone applications, which is composed by a model and an infrastructure that implements it. The model describes how to automatically monitor and collect context data and usability metrics, how those data can be processed for analysis support and how users' impressions can be collected. An infrastructure is provided to implement the model allowing it to be plugged into any smartphone Android-based application. To evaluate our proposal, we performed a field experiment, with 21 users using three mobile applications during a 6-month period, in their day-to-day scenarios.

Introduction

In recent years, the introduction of several technologies has revolutionized society's methods of communication, entertainment, and the execution of daily tasks. Simultaneously, the process of digital convergence has generated a great number of devices (PDAs, smartphones, tablets) capable of gathering different forms of usercomputer interaction in an integrated way and with reasonable processing power.

Considering that the interaction of the user with this kind of device is extremely sensitive to the context in which the application is running, what constitutes the most adequate scenario for executing usability tests is a topic of great discussion. The central question is whether these experiments should be performed in the field, or in the laboratory [START_REF] Queiroz | A Multidimensional Approach for the Evaluation of Mobile Application User Interfaces[END_REF].

Another important question has to do with the kind of data that must be present in a usability evaluation. Besides quantitative data used to determine the quality of the interfaces and the usability of the applications, subjective data should also be considered in order to draw conclusions about the users' satisfaction.

The main problem when evaluating mobile applications is the development of approaches whose evaluations integrate quantitative, subjective and context data in the same experiment. Nowadays, those three points of interest are investigated in isolation, which makes the relationship between those results difficult to determine [START_REF] Zhang | Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications[END_REF].

Faced with this challenge, the objective of this study is to assess the new applications usability, their functionalities, how the users adapt to them to do their daily tasks, the satisfaction level provided and the level of external interferences.

In order to achieve evaluations with different kinds of information, we propose an approach to support the evaluation of smartphone applications based on the Android platform. This approach blends two strategies for data capture: the first one, known as Logging [START_REF] Ivory | The state of the art in automating usability evaluation of user interfaces[END_REF], is based on the data collection related to the user interaction with an application, allowing statistical analysis regarding usability; it allows the use of sensors available in the smartphones for contextual data collection, such as luminosity intensity and the device´s position [START_REF] Froehlich | MyExperience: A System for In Situ Tracing and Capturing of User Feedback on Mobile Phones[END_REF]. The second one, known as Experience Sampling Method (ESM) [START_REF] Meschtscherjakov | Utilizing Emoticons on Mobile Devices within ESM studies to Measure Emotions in the Field[END_REF], is based on the collection of users' feelings towards a specific product through questions.

The main contribution of this work is to propose a new approach that is able to relate different types of data to evaluate the usability of embedded applications on smartphones. The proposed approach can contribute to the implementation of field and laboratory experiments, whether they are supervised or not.

The remainder of this article is divided into six sections. Section 2 describes the proposed model. Section 3 presents an infrastructure that implements the model, to capture and evaluate usability data from Android applications. Section 4 presents a case study performed to evaluate the proposal and analyzes the results found on the experiments. Section 5 discusses related works to reinforce our contributions. Finally, in Section 6, we present conclusions and future work.

The UEProject: A Model for Usability Evaluation on Smartphone Applications

In our work, we propose the Usability Evaluation Project (UEProject), which is composed by a model and an infrastructure to support a hybrid approach for usability evaluation on Smartphone Applications. The proposed model enhances the capture of quantitative and subjective data, contextualized within the evaluation scenario and in conjunction with the identification of the users' profiles.

The intention is to go beyond the collection of statistical data, such as identifying the frequency of use, the error count or speed of execution of a task. The goal is to provide an infrastructure that enables the contextual factors associated with usability problems, and identify the user perception regarding the application.

The model uses a component-based architecture to support the reuse of the implemented resources and its redefinition according to evaluators' needs. Figure 1 presents the three high level components that represent the model and their relations: Mapping Unit, Traceability Unit and Assessment Unit. Arrows indicate information transfer between the components.

Figure 1. Main components of the model

The Mapping Unit is responsible to provide the necessary resources for mapping the tasks that will be investigated later in the application under evaluation. It generates the Mapped Tasks and Reference Values that provide information about the context in which the task was mapped and the perceptions of the person responsible for mapping it. The Traceability Unit provides a library of metrics that uses the mapped tasks, enabling an instrumentation of the application's source code with metrics for the capture of quantitative, contextual, and subjective data concerning the user profile. The values obtained during user interactions are sent to the Assessment Unit. The Assessment Unit is responsible for receiving the transmitted data, transferring it to a database and enabling evaluations. Next subsections present the internal components of the model's units.

Mapping Unit

This unit is subdivided into three components responsible for providing the taskmapping functionalities. In Figure 2 the components diagram can be seen, with its interfaces and doors for data transfer from one component to another.

Figure 2. Mapping Unit Architectural Overview

Initially it has been planned that the applications' source code be made available. The objective of this action is to allow the Source Code Analyzer component to identify which classes refer to the treatment of users' interactions. These classes are identified and provided as a requirement for the Mapping Code Generator component so that it can attach information that enables the identification of tasks to the original source code. The result of this component's action is to perform the instrumentation of the application's original code, thereby allowing the tasks to be mapped. After the source code is prepared for mapping, it must be forwarded to the Mapping Device component where the tasks will be mapped. This device must enable the mapping of tasks and make them available to be used by other units in the model.

Figure 3. Steps for application instrumentation for task mapping

In Figure 3, the diagram portrays the sequence that enables the exchange of messages and data between the Mapping Unit components.

In Figure 4 the diagram of a simplified use case is presented, where the only two existing interactions are shown. In the first interaction, the Evaluation Team is responsible for providing the application's source code and, in the second, for performing the tasks' mapping for later assessment. It is worth noting that the tasks' mapping can be done by a test engineer, an experienced user, or even average users who are chosen according to their profile, thereby allowing future comparisons.

Traceability Unit

This unit is responsible for collecting different types of information: (i) user profile (e.g. education and age); (ii) data referring to the user's interaction, such as hits and time to complete a task; (iii) contextual data like luminosity and noise; and (iv) subjective data, related to the users' feelings regarding the applications' usage.

For the instrumentation of the source code to include the code for capture metrics and data collection, the Traceability Unit was divided into three internal components, as can be seen in Figure 5: Metrics Library, Metrics Generation and Interaction Device. The Metrics Library is the component that provides the structure to all capture metrics. It uses the mapped tasks to bring together the structure of the metrics with information about the application source code. It generates, as result, the metrics' structures adapted to the application under evaluation. In the development of the Metrics Library, the incorporation of three types of structure was expected: (i) the ones used to capture quantitative data from users interactions; (ii) the ones responsible for the subjective data (direct interactions with users) and; (iii) the metrics that allow instrumentation of the available sensors. The Metrics Generation component receives as input the application's source code and the adapted metrics structure to be attached to this code. As result it provides a new source code with the inserted metrics, the Traceability Code. The component Interaction Device requires the installation of the application's new code containing the metrics that will be used for data capture. This device should enable the user's interaction while allowing data to be captured.

Figure 6. Steps for the instrumentation of an application with the metrics for capturing data

In Figure 6, the diagram presents the sequence of actions that permits the exchange of messages and data between the Traceability Unit components.

Figure 7 presents two use case diagrams concerning the Traceability Unit. The first actor (Evaluation Team) is responsible for providing files with the mapped tasks and the application's source code, enabling the creation of a new application with metrics for capturing data. The second actor (User) uses the application while data concerning usability, context, profile, and feeling about application usage are captured.

Assessment Unit

This unit was structured in four different components that are responsible for infrastructure where data to be evaluated will be stored over time. Figure 8 presents an architectural overview of its components. To illustrate better its behavior, the sequence diagram in Figure 9 presents the exchange of messages and data between the components.

Initially, the Data Receiver component collects the data captured during the execution of the user's interactions within the application under evaluation. A second component, the Populate Database, performs the verification of new data arrival and populates it into the database. The Database component represents a Database Management System (DBMS) that stores persistent data. The Analysis Component has the responsibility to access existing data in the database and to provide resources that enable the extraction of usability information relevant to the evaluators.

Figure 8. Assessment Unit Architectural Overview

To make the Assessment Unit easier to understand, the diagram that follows (Figure 9) illustrates the exchange of messages and data between components.

Figure 9. Actions performed by the components of the Assessment Unit

The only direct interaction of an actor with the devised components in this unit is at the moment of the assessments done by the Evaluation Team. This team is responsible for defining data, parameters, and configurations so the information can be extracted according to their needs.

The UEProject Infrastructure

In this section, we describe the infrastructure implemented to verify the feasibility of the UEProject model. The technological aspects and techniques used in the implementation of the infrastructure are discussed. Additionally, the section describes the tools created, or used from the market, to cover all the components according to the model proposal.

We defined four premises related to how technical choices will be made when planning the construction of the infrastructure, as follows.

1) To propose that the data can be collected anywhere the user is interacting. To facilitate this, the technique Logging was chosen. This strategy consists of automatically monitoring and gathering data concerning the applications' use.

2) To define that all used software licenses should be free. To this end, we decided to develop the components using the Java Language, since it is platform-independent and free. The components obtained from the market were also chosen according to this premise. The objective is to make the project economically feasible.

3) To determine that all collected data might be gathered from smartphone applications. For this reason, the Android platform was chosen because it uses Java and Android is widely used nowadays.

4) To suggest that the applications' source code to be instrumented should not be modified. Therefore the Aspect Oriented Program (AOP) [START_REF] Kiczales | Aspect-oriented programming[END_REF] was chosen. AspectJ1 , an AOP plugin for Java is used to instrument the applications with codes considered as transversals. In this case, the codes are needed to map the tasks and the code related to the metrics for data gathering.

In this implementation of the infrastructure, the proposed way for task mapping was to capture the methods executed in the application after the interaction of the Evaluation Team. The group of interactions for a task execution is translated into a group of methods executed in the application. The mapped methods are used as Join Points so the metrics can be introduced in an application, without modifying the original source code. In this sense, Point Cuts are used to define the rules that validate the execution of the metrics built in the Advices.

The following subsections describe the tools used to implement the components of the three units defined in the Model.

Tools used in the Mapping Unit

The first developed tool looks at the Source Code Analyzer and the Mapping Code Generator. This tool was called the Mapping Aspect Generator (MAG). Figure 10 presents a workflow showing all the necessary steps for the preparation of an application to be mapped.

Figure 10. Workflow for the insertion of mapping aspects into an application

Initially the MAG tool must receive the application's source code to be mapped (step 1). After that, the tool goes through all the classes and creates a list of the classes that deal with users' interactions (step 2). Afterwards the tool uses the list of classes and generates an Aspect to implement the onUserInteraction2 method in the classes that do not yet have it (step 3). This action enables the detection of users' interactions. The next action is to compile the application's source code with the generated Aspect, using AspectJ (step 4). The result will be that the application is ready to be mapped (step 5). To end the process, the application will be embedded on a device that allows the interactions to take place (step 6).

To allow the task mapping and the association of reference values and context during the interaction of the Evaluation Team, another tool was developed, called the Automatic Task Description (ATD). This tool must be embedded on the device on which the application will be mapped and executed simultaneously with the application (Device Mapping component). Therefore, as the Evaluation Team interacts with the application, the executed methods are automatically captured as the concluding steps of a task.

The main point of this tool is to act as a filter that identifies when a user interaction occurs. Also, the filter identifies which classes, methods, and application parameters were used.

Tools used in the Traceability Unit

The tool proposed to implement the Metrics Library and the Metrics Generation components was named Usability Metrics Generation. This tool contains a library that has the structure of the metrics that are available to perform measurements. Figure 12 shows a workflow of necessary actions for the automatic generation of the Aspects and its association with the source code of the application.

Figure 12. Workflow for the insertion of the aspects of traceability in an application

The tool receives as input the XML file generated in the Mapping Unit (step 1). Next, the existing methods in the XML, along with the information contained in the Metrics Library (step 2), will be used to generate the Aspects responsible for the capture, transmission, and persistency of the data (step 3). Afterwards, the source code of the application (step 4) and the generated Aspect (step 3) must be compiled using AspectJ (step 5). The result is the generation of an application with the metrics (step 6). Later, the Application will be loaded onto the device with which the user will interact.

The efficiency and effectiveness of the users' interactions are measured through usability metrics described in Table 1. All the measurements take into consideration the mapping of a task, where all the steps to its conclusion are described. Thus, the errors, for instance, are identified every time the user interacts with the application that is not in the previous mapping. The sensors used to capture the contextual data were the accelerometer (to capture the horizontal or vertical position), GPS (to capture movement), Luminosity Sensor (to capture the environment's luminosity), and the microphone (to capture noise in the environments).

It is important to stress that the infrastructure uncovers new metrics to be incorporated in the Metrics Library, which increases its adoption in different scenarios, contexts, and with the use of other sensors. For example, specific metrics could be incorporated in an application for the spatial orientation of people. Therefore, such metrics could be associated with the data provided by a mobile device's GPS, enabling the comparison of the user's interactions with information regarding positioning, speed, and route taken. In that case, the usability analysis would take into account not only how a user performs a task but also how the environment influences the user's interactions.

The subjective metrics are used to measure the emotional state of the users during their experiences with an application. The approach chosen for representing these metrics is known as the Experience Sampling Method (ESM) [START_REF] Meschtscherjakov | Utilizing Emoticons on Mobile Devices within ESM studies to Measure Emotions in the Field[END_REF].

ESM was chosen because of the following aspects: (i) it is appropriate for use on devices with relatively small screens; (ii) it is intuitive and doesn't take much mental effort to be interpreted; and (iii) it is capable of being answered with entry modes provided by different mobile devices.

ESM measures two dimensions, the kind of emotion (positive or negative) and the intensity of the emotion. To do this, a group of pictures is displayed indicating emotional states associated with a question, as can be seen in Figure 13. The sequence of the pictures represents varying degrees of emotional intensity and can be interpreted from left to right as: very displeased, displeased, indifferent, pleased, very pleased. These questions are defined by the evaluators during the execution of the Usability Metrics Generation Tool.

Figure 13. Example of an ESM

Tools used in the Assessment Unit

To devise the components defined in the Assessment Unit, the following processes were performed: (i) to create and configure a FTP and database server (DB) and to make them available on the internet; (ii) to model a database capable of storing the captured information about the usability of the instrumented applications; (iii) to create tools for detecting the presence of new files on the FTP server and populate the DB; and (iv) to choose an Online Analytical Processing (OLAP) tool to analyze the data.

In order to transmit and store data on the mapping of tasks and the results of users' interactions, it is necessary to have a FTP and a DB available. For this, a micro instance of the service known as Amazon Elastic Compute Cloud (Amazon EC2) was contracted with Amazon, running on the Ubuntu operating system, version 11.10 Oneiric Server, with 613 Mbyte RAM memory and a 16 Gbytes hard drive.

To study the Data Receiver component, the FTP server ProFTPd, version 1.3.4, was installed to manage and control the transfer of files.

To devise the Database component, the DBMS MySql Community Server, version 5.1.58 was installed to store the data.

To devise the Populate Database component, a tool called Data Loader was developed. The steps executed by this tool are: (i) to detect the arrival of new files on the FTP server; (ii) to extract the data; and (iii) to load the data into the database.

To devise the Analysis component, the Pentaho Analysis Services, version 3.9.0, was chosen. It offers two possibilities for data analysis currently used in usability evaluations, OLAP and data mining tools.

Case Studies

This section describes the case studies performed to evaluate the functionalities and identify the potentials of the UEProject Model and the Infrastructure with the tools that give support to its implementation.

Methodology

The first step taken to carry out the experiments was the completion of an exploratory research on the internet with the objective to find applications with attractive functionalities that could be easily introduced into people's lives. Also, it was taken into consideration in the applications selection the compliance to the Android platform and the use of software modularization, allowing the application's source code to be instrumented with AOP.

Following those requirements, three applications were chosen: Mileage, ^3 (Cubed), and Shuffle 3 .

The use of these applications in the case studies is based on two important aspects: (i) to show that the infrastructure can be used in usability experiments for Android applications; and (ii) to show that, although the originating process of the application has no connection with the proposed usability evaluation model, the aspects of mapping the tasks and the metrics can be easily integrated into its source code. To enable the investigations, tasks were chosen for each application that would be interesting to evaluate and that would offer more possibilities for interaction. Thus, the model can be tested with a varied group of actions.

The purpose of Mileage is to help users to control the money used on fuel and automotive maintenance services. Eight tasks were mapped in this application to complete the experiments. Some of the mapped tasks for the execution of the Mileage experiments were, for example: to register a vehicle, to add a new maintenance control, and to visualize the graph showing fuel price variation.

^3 Cubed is a music and video clip manager. In this application, six tasks were mapped. Some of the mapped tasks in the experiments were: to choose a song from a playlist, to change the appearance of the application, and to activate the equalizer.

The Shuffle application is an activity scheduler that can manage daily tasks. Ten tasks were used from this application in completion of the experiments. Some examples of mapped tasks in this application are: to insert a new activity, to download a copy (backup) of the data, and to create a new project so that new activities can be linked to it.

It is important to note that, independent of the tasks chosen for the evaluation, the user should not perceive any change in the functionality or appearance of the application. The only change implemented is the presentation of subjective questions to the user.

The applications instrumented with the traceability metrics were made available for download on the internet by users who are interested in collaborating with the experiments 3 .

The experiments started in December 2011. About thirty two users downloaded the applications; however, only twenty one of them are frequent and contribute effectively to the investigations. They were categorized according to age, educational grade, field of academic study, work type and social class. The context data being collected to associate to the users' experiences are: environment luminosity, device position (vertical, horizontal and mixed) and user´s movement. In addition to these data, the screen size and resolution of the used device are identified.

The first results of the analyses indicate the possibility for different methods of analysis, such as: to verify the applications usability, to analyze the experiences provided for the users and to relate their interactions to the context of use.

To exemplify, we would like to answer three questions in this preliminary study: 1) What can be concluded regarding the users´ performance over time?

2) What kind of information the context can offer to improve the usability analysis? 3) What is the users´ satisfaction level with the applications proposed in the experiment?

Analyzing Users' Performance on Interacting with the Applications

To analyze users' performance over the period when the experiments were carried out, the average speed to execute the tasks was evaluated. Figure 14 presents the average time, in seconds, used to conclude the tasks available in the three applications. It is possible to see that, in the first month, there is a discrepancy in the average time when set against the other months. In December, we can see that the users need more time to conclude the tasks than in the following months. This draws to the conclusion that, after the first month, the users show they are more familiar with the available resources.

Analyzing the Potential of Considering Contextual Information

It was observed in the analyzed data that the occurrence of errors in the low-income class is greater than in the other classes. To find an explanation to this result, the type of device used by those people was investigated.

It was found that the error rate was not related to the users' purchasing power, but to the low resolution of the devices used. So, as the low resolution devices are more frequently used in the low-income class, a wrong result might be found without the evaluation of a contextual factor.

In Figure 15, the relation between the purchasing power and the smartphones screen resolution is presented. It shows that the errors are more frequent when low resolution smartphones are used, regardless the purchase power. Another relevant factor regarding the applications' context of use is that approximately 70% of the interactions occur when the users are still, having the device in a single position and under normal environment luminosity. However, when those contextual factors change, the users make more mistakes and take longer to execute the tasks. This information suggests that the applications should, for example: (i) make the interaction impossible in positions where there is greater probability of errors, forcing the user to interact in the proper position; (ii) detect the external luminosity and try to balance the luminosity radiated by the device in order to guarantee a good visualization; and (iii) identify the user movement and make the most usual functionalities available, decreasing the visual pollution.

Analyzing Users' Feelings Related to Using the Applications

In order to cover the last question proposed in this study, the ESM technique was used, presenting the questions in Table 2. The aim was to collect subjective data from the users' feelings related to using each application. The analysis of such kind of information allows to detect problems that generate either discomfort or dissatisfaction to users, as well as to learn what attracts users the most to insert applications into their daily routine. How satisfied are you with using this application in your daily routine? 4

How satisfied are you with the application functionalities? 5

How will you feel when using this application again?

Figure 16 presents the percentages concerning users' answers to the questions made in the experiment for the application Cubed, according to the description and identification (ID) presented in Table 2.

Figure16. ESM answers associated to the application Cubed.

A fact that deserves to be highlighted is that the distribution of satisfaction level is maintained constant and increasing from very displeased to very pleased. This trend is observed in all questions submitted to users in the three applications.

Another relevant aspect is that the group of pleased and very pleased users is higher than 50% in all items evaluated. This reveals one of the reasons for the success of the experiment in maintaining the users utilizing the applications during the six months. It is essential that the applications are interesting, useful, attractive, rewarding and easy to interact with. After the findings presented, it is possible to say that the users are satisfied with the applications used in the experiments, answering, thus, the last question asked.

Related Works

In this section, we describe and discuss some works related to our research that uses at least one of the chosen collecting data used in our proposal to support evaluating mobile applications that is Logging, Devices' Sensors and Online Interviewing with users.

The systems evaluation through Logging is not a new strategy as it has been used for over two decades [START_REF] Ivory | The state of the art in automating usability evaluation of user interfaces[END_REF]. However, it has gained strength over the last years with the use of AOP to develop it [START_REF] Moldovan | Automatic Usability Evaluation using AOP[END_REF][8] [START_REF] Bateman | Interactive usability instrumentation[END_REF] and the possibility to be applied in smartphones [START_REF] Froehlich | MyExperience: A System for In Situ Tracing and Capturing of User Feedback on Mobile Phones[END_REF][10] [START_REF] Shepard | LiveLab: measuring wireless networks and smartphone users in the field[END_REF].

The data collection through Logging and sensors in smartphones is the focus of many tools [START_REF] Falaki | SystemSens: a tool for monitoring usage in smartphone research deployments[END_REF][13] [START_REF] Shin | AnonySense: A System for Anonymous Opportunistic Sensing[END_REF]. These tools, in general, aim to identify the users' behavior and to associate contextual information to its use. The captured data are sent to available servers in the Web for later analysis. The tools LiveLab [START_REF] Shepard | LiveLab: measuring wireless networks and smartphone users in the field[END_REF], SystemSens [START_REF] Falaki | SystemSens: a tool for monitoring usage in smartphone research deployments[END_REF] e AnonySense [START_REF] Shin | AnonySense: A System for Anonymous Opportunistic Sensing[END_REF] are examples of systems that encompass uncountable logs records, with the objective of monitoring the incoming calls, network traffic, data sharing, battery level, user location, frequency of application use, and CPU and memory use. MobileSens [START_REF] Guo | MobileSens: A Framework of Behavior Logger on Android Mobile Device[END_REF] differentiates itself from the previous tools because it uses log records to optimize some smartphones resources in real time such as, reconfiguring the system in order to minimize the battery waste when it is low and indicate more appropriate networks to be used based on the user location. Although the described tools are useful to evaluate users' behavior, they do not enable Online Interviewing with users to identify their satisfaction with the used applications.

The MyExperience [START_REF] Froehlich | MyExperience: A System for In Situ Tracing and Capturing of User Feedback on Mobile Phones[END_REF] stands out in this sense since it uses direct interactions to obtain subjective data that refers to users' satisfaction. The study proposed by Ickin et al [START_REF] Ickin | Factors Influencing Quality of Experience of Commonly Used Mobile Applications[END_REF] presents a mixed methodology to evaluate the user experience, using three different resources: (i) a tool named CSS Application, to capture the log records related to the system use; (ii) ESM, to identify the satisfaction and obtain information about the user interaction context, such as location, social context and mobility level (still, walking, motorized) and (iii) DRM (Day Reconstruction Method), a questionnaire that helps identifying the relationships and results of the Quality of Experience (QoE). Although those tools allow evaluating the user satisfaction, they do not enable identifying specific usability problems, as they don't encompass particular tasks of available applications.

In this context, Paterno et al. [START_REF] Paterno | Remote evaluation of mobile applications[END_REF] proposed a methodology and environment that allows the remote evaluation of mobile applications. The system features a Mobile Logger that collects the information from the mobile device, and a tool called MultiDevice RemUsine that processes the logged information and provides the necessary visualizations to analyze the application usability.

Another similar approach by Au et al [START_REF] Au | Automated Usability Testing Framework[END_REF] was done to test usability of handheld applications. Different aspects considered in testing the usability of handheld device applications and a proposed a list of functional requirements that automated usability testing tools should have in order to be effective were discussed. The Handheld device User Interface Analysis (HUIA) testing framework was then developed and meets most of the requirements proposed.

The EvaHelper [START_REF] Balagtas-Fernandez | A methodology and framework to simplify usability analysis of mobile applications[END_REF] framework is another proposal for usability evaluation. Its structure is divided in three aspects: preparation of applications to be tracked, extraction of data obtained and results evaluation. Whilst its structure is similar to the one proposed in this paper, many differences can be highlighted: (i) it does not use AOP, which generates direct intrusion in the application original source code; (ii) it does not have a tool to automatically map the tasks; (iii) it does not have a tool to instrument the code automatically, without the knowledge of APIs and programming in the Android platform and, finally, (iv) it is restricted to statistical evaluations, that is, it does not make the contextual and subjective data capture possible.

Conclusions

This article presented a hybrid model to capture and assess data from smartphones applications. The units of the model were structured in components that can be redefined according to evaluators' needs. Basically, the model can be described in three units: Mapping Unit, Traceability Unit, and Assessment Unit.

The proposal has as main contributions the possibility to evaluate the smartphone applications' usability by integrating, in the same experiment, metrics data, contextual data and users' feelings according to the used applications. In the literature, there is a lack of proposals that relate those three types of data and that support the performance of field experiments with users using the applications in their daily lives.

Other contributions that can be highlighted are:

• The implementation of the components that interfere in the source code of the applications using AOP, so the aspects can be inserted and removed according to evaluators' needs.

• The creation of tools that enable the use of the infrastructure and the instrumentation of applications without the need for programming.

• The provision of a database that allows continuous assessments in order to identify usability problems with a large number of users and for long periods of time.

For future research, new metrics must be developed, prioritizing common attributes of certain groups of applications. Moreover, there is an intention to incorporate to the infrastructure interactions with users that permit to identify their location and social context. Other studies will be published to present the results of the experiments carried out with the objective of contributing to new findings in the Mobile HCI area.

Figure 4 .

 4 Figure 4. Use Case of the Mapping Unit

Figure 5 .

 5 Figure 5. Traceability Unit Architectural Overview

Figure 7 .

 7 Figure 7. Use Cases Diagram of the Traceability Unit

Figure 14 .

 14 Figure 14. Average time to execute tasks (in seconds)

Figure 15 .

 15 Figure 15. Relation between the error rate and the social class and screen resolution

Table 1 .

 1 Usability Metrics

	Usability Metrics	Objectives
	Metrics for the capture of assertiveness
	Actions in conformity	Quantify the difficulty of interaction for the
	Actions in violation	user in a task using the interface.
	Metrics for the search and use of tasks
	Unfinished task	An unfinished task can mean that the user is
	Finished tasks	lost or searching for a desired task.
	Finished and used tasks	The use/nonuse of a task indicates if the user
	Finished and unused tasks	found what s/he was searching for.
	Metrics related to performance
	Average time for task completion Average number of actions for task completion Number of tasks concluded without errors	Indicate the need for improvements that minimize the time and actions for the execution of tasks.
	Metrics for the use of help
	Frequency of the use of help	Measure the level of difficulty faced by the
		user. Identify if the help available provides
	Violations made after the use of help	the desired effect.

Table 2 .

 2 Questions submitted to users.

	ID	Questions
	1	How satisfied are you with the interface?
	2	How satisfied are you with learning new functionalities?
	3

Available in http://www.eclipse.org/aspectj/

Available in http://developer.android.com/reference/android/app/Activity.html

Available in ueproject.no-ip.org