Bo Yang

Zuopeng Zhao

Xinzheng Xu

Convolutional Neural Networks Optimized By Logistic Regression Model

Keywords: Convolution neural network, Logistic regression, Softmax regression

In recent years, convolutional neural networks have been widely used, especially in the field of large scale image processing. This paper mainly introduces the application of two kinds of logistic regression classifier in the convolutional neural network. The first classifier is a logistic regression classifier, which is a classifier for two classification problems, but it can also be used for multi-classification problems. The second kind of classifier is a multiclassification logistic regression classifier, also known as softmax regression classifier. Two kinds of classifiers have achieved good results in MNIST handwritten digit recognition.

Introduction

In recent years, since the convolutional neural network is proposed [START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF], it is widely used in pattern recognition [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF], image processing, especially achieved good results in the large field of image processing [START_REF] Krizhevsky | ImageNet classification with deep convolutional neural networks[END_REF].This paper [START_REF] Bouvrie | Notes on convolutional neural networks[END_REF] makes a detailed theoretical analysis of the convolution neural network, then the various classification algorithms and models have been proposed. In the paper [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] proposed the multilayer perceptron as a convolutional neural network classifier, the paper [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] also used k nearest neighbor algorithm as a convolutional neural network classifier, the paper [START_REF] Dreiseitl | Logistic regression and artificial neural network classification models: a methodology review[END_REF] used support vector machine (SVM) as the convolutional neural network classifier, both of them have achieved good results in the handwritten numeral recognition experiment. The paper [START_REF] Vittinghoff | Regression methods in biostatistics: linear, logistic, survival, and repeated measures models[END_REF] mainly introduces the model of linear regression, logistic regression and so on. The paper [START_REF] Graves | Supervised sequence labelling with recurrent neural networks[END_REF] introduces the softmax regression model, and the detailed formula derivation of the algorithm combined with the back propagation algorithm. The structure of this paper is divided into five parts. The second part mainly introduces the convolutional neural network structure. The third part mainly introduces two kinds of classification model. The first is a logistic regression model and how to use logistic regression to solve multi classification problem, on the other is the Soft regression model. The forth part of the thesis is the experiment and the result analysis. The last part of the paper is the summary.

2

Structure of convolutional neural network

Convolutional neural network can be functionally divided into two parts, one is the image feature extraction section, the other part of the classifier. In our experiment, convolutional neural network in the structure can be divided into seven layers including an input layer, convolutional layer C1, sub-sampling layer S2, convolutional layer C3, sub-sampling layer S4, sub-sampling layer S4 unfolded layer (not included in the number of layers), fully connected layer and output layer. The feature extraction part includes C1, S2, C3, S4 layer, the classifier part includes the sub-sampling layer S4 expansion layer (also can be used as the input layer of the classifier), fully connected layer and output layer.

Convolutional neural network structure has many variants, this article uses a classic convolution neural network. In this paper, we mainly introduce the application of two kinds of logistic regression classifiers in the convolutional neural network, so we simply introduce the convolution neural network. We can more detailed understanding of convolutional neural networks in this classic paper [START_REF] Bouvrie | Notes on convolutional neural networks[END_REF]. Figure 1 is the structure of the convolution neural network used in this paper. This paper carries out a handwritten numeral recognition experiment on the MNIST data set.

Logistic Regression Model

The two classification problem is divided into 𝐶 1 and 𝐶 2 two categories, then the output layer has only one neuron and its output is y. Assuming that the number of weight parameters of the output layer is n, then w 𝐿 = [𝑤 1 , 𝑤 2 , … , 𝑤 𝑛 ]. Since the output value y is processed by the sigmoid function, and the output range of the sigmoid function is 0 to 1, then we can put the output value y as the probability value which the input vector 𝑥 1 belongs to the first category𝐶 1 . Conversely, using one minus the output y gives the probability value that the input vector x 1 belongs to the second category𝐶 2 .

p(𝐶 1 |𝑥 1 ) = 𝑦 (1)
p(𝐶 2 |𝑥 1 ) = 1 -𝑦 (2)
For the two classification problem with N training samples, the cost function E can be expressed as:

E = 1 𝑁 ∑ [-𝑧 (𝑖) log(𝑦 (𝑖) ) -(1 -𝑧 (𝑖) )log (1 -𝑦 (𝑖) )] 𝑁 𝑖=1 (3)
If we use logistic regression to solve the multi classification problem, we need to train a logistic regression classifier for each category. For the K classification problem, then the output layer has K neural units, the use of logistic regression to solve the problem of K classification need to train K logical regression classifier. For a specific category, the rest of the category as a category, then it becomes a two classification problem. For the K classification problem with N training samples, the cost function E can be expressed as:

E = 1 𝑁 ∑ ∑ [-𝑍 𝑘 (𝑖) log(𝑦 𝑘 (𝑖) ) -(1 -𝑍 𝑘 (𝑖) )log (1 -𝑦 𝑘 (𝑖) )] 𝐾 𝑘=1 𝑁 𝑖=1 (4)
In order to prevent over fitting in the training process of neural network, we can add the regularization in the cost function. Regularization controls the model's complexity and makes the model more generalizable to unseen data. In order to simplify the problem, it is assumed that the neural network model is only the input layer and the output layer. We assume that the number of neural units in the input layer is M, while the output layer has K neural units. The weights between the input layer and the output layer are w, then the cost function is as follows:

E = 1 𝑁 ∑ ∑ [-𝑍 𝑘 (𝑖) log(𝑦 𝑘 (𝑖) ) -(1 -𝑍 𝑘 (𝑖) )log (1 -𝑦 𝑘 (𝑖) )] 𝐾 𝑘=1 𝑁 𝑖=1 + 𝜆 2 ∑ ∑ 𝑤 𝑚𝑘 2 𝐾 𝑘=1 𝑀 𝑚=1 (5)
In the above model, we only add the L2 regularization. The 𝑤 𝑚𝑘 represents the connection weights of neural unit m and neural unit k.

Softmax Regression Model

We need to use the softmax function instead of the activation function in solving the K classification problem and obtain the class probabilities:

p(𝐶 𝑘 |𝑥 1 ) = 𝑦 𝑘 = 𝑒 𝑢 𝑘 ∑ 𝑒 𝑢 𝑖 𝐾 𝑖=1 (6)
This is called as a multinomial logit model。For the target vector z to meet the relationship ∑ 𝑧 𝑖 = 1 𝐾 𝑖=1

.Assuming K = 5 , if the correct classification of a training sample is 𝐶 3 , then the target vector should be 𝑧 (𝑖) = [0,0,1,0,0] 𝑇 . Then the probability formula of the input vector 𝑥 1 and the target output vector z is:

p(z|𝑥 1 ) = ∏ 𝑦 𝑘 𝑧 𝑘 𝐾 𝑘=1 (7)
For the multi classification problem with N training samples, the cost function of the Softmax regression model is:

E = - 1 𝑁 ∑ ∑ 𝑧 𝑘 (𝑖) 𝑙𝑜𝑔𝑦 𝑘 (𝑖) 𝐾 𝑘=1 𝑁 𝑖=1 (8)
Because the model has the redundant weight parameters, we need to add a weight decay term which penalizes large values of parameters to modify the cost function, our cost function is now:

E = - 1 𝑁 ∑ ∑ 𝑧 𝑘 (𝑖) 𝑙𝑜𝑔𝑦 𝑘 (𝑖) 𝐾 𝑘=1 𝑁 𝑖=1 + 𝜆 2 ∑ ∑ 𝑤 𝑚𝑘 2 𝐾 𝑘=1 𝑀 𝑚=1 (9)

Back propagation

The "errors" which we propagate backwards through the network can be thought of as "sensitivities" of each unit with respect to perturbations of the bias [16]. That is to say,

δ = 𝜕𝐸 𝜕𝑏 = 𝜕𝐸 𝜕𝑢 𝜕𝑢 𝜕𝑏 (10)
As a result of ∂u ∂b = 1, the sensitivity of the output layer can be obtained.

δ 𝐿 = 𝜕𝐸 𝜕𝑢 𝐿 = 𝑦 (𝑖) -𝑧 (𝑖) (11)
By deriving the formula, we can obtain the sensitivity of the previous layer as follows:

δ 𝑙 = (𝑤 𝐿 ) 𝑇 𝛿 𝐿 • 𝑓 ′ (𝑢 𝑙 ) (12)
Due to the use of the back propagation algorithm, we also need to know the gradient in the process of updating the weight parameters. The gradient formula is as follows:

𝜕𝐸 𝜕𝑤 𝑙 = 𝜕𝐸 𝜕𝑢 𝑙 𝜕𝑢 𝑙 𝜕𝑤 𝑙 = 𝛿 𝑙 (𝑥 𝑙 ) 𝑇 + 𝜆 𝑁 𝑤 𝑙 (13)
Δ𝑤 𝑙 = -𝜂 𝜕𝐸 𝜕𝑤 𝑙 (14)

Experiment and Result analysis

We have made a good effect on the handwritten digit recognition based on MNIST data set with two kinds of logic regression classifier combined with convolutional neural network. We use three kinds of classifiers to do the contrast experiment, one is based on the logistic regression classifier, one is the softmax regression classifier, and the last one is the multilayer perceptron. We find that whether the classifier uses the hidden layer has a great influence on the convergence speed and the test results. The experimental results of the classifier without using the hidden layer are as follows: In this experiment, there are 60000 training samples, each of the 50 samples to update a weight parameter, then an iteration will update the weight 1200 times.

Conclusions

In this thesis, we mainly study the use of the classifier based on logistic regression in the convolutional neural network, and introduce two kinds of classifier models based on logistic regression. There are many factors that affect the classifier performance, such as whether to use the hidden layer, the number of hidden layer neurons, and the learning rate of the regular term and so on. We can improve the convergence speed and test results by increasing the depth of the convolutional neural network. In practical applications, the classifier is selected according to the classification of the data set.

In a word, it is a relatively simple and good method to apply logistic regression to convolutional neural network.

Fig. 1 .

 1 Fig. 1. Network structure of convolutional neural network

Table 1 .

 1 Comparison of experimental resultsThe experimental results of the classifier using the hidden layer are as follows:

	Iterations	Method	CNN+MLP	CNN + LRM	CNN + Softmax
		1	11.15%	14.96%	5.71%-7.49%
		5	4.34%	8.01%	2.67%-3.26%
		10	2.73%	5.89%	2.37% -2.75%
		50	1.51%	3.24%	1.69%-2.68%
	100	1.21%	2.27%	1.13%-1.61%
	Iterations	Method	CNN + MLP	CNN + LRM	CNN + Softmax
		1	88.65%	5.46%	7.94%
		5	5.17%	1.80%	1.86%
		10	3.25%	1.37%	1.44%
		20	1.76%	1.09%	1.12%
		50	1.23%	0.86%	0.93%
		80	1.00%	0.85%	0.94%
	100	1.02%	0.74%	0.82%

Table 2 .

 2 Comparison of experimental results


Jiangping Wang
email: jpwang@xsyu.edu.cn

Tengfei Duan
email: tfduan@163.com

Lujuan Lei

Application of Manifold Learning to Machinery Fault Diagnosis

Keywords: Manifold Learning, Fault Diagnosis, Bearing

The essence of machinery fault diagnosis is pattern recognition. Extracting the fault pattern contained in the vibration signal is the frequently used method to diagnose mechanical fault. Manifold Learning is widely used to extract the non-linear structure within the data and could do the dimensionality reduction of high-dimensional signal. Therefore manifold learning is employed to diagnose the machinery fault. The feature space is constructed by characters in time-frequency domain of vibration signal firstly, and then the manifold learning method named as sparse manifold clustering and embedding is used to extract the essential nonlinear structure of feature space. Afterwards, the fault diagnosis is implemented with spectral clustering and support vector machine. The experiment demonstrates that the approach can effectively diagnose the fault of Machinery.

Introduction

With the increment of precision and operation complexity in industry, the techniques of machinery fault diagnosis, which is used for the safe guarantee in operation of mechanical equipments, has gained more and more attention. At present, data-driven methods play a important role in the mechanical fault diagnosis. The vibration signal contains a wealth of information which indicate the condition of machinery, therefore it is widely used in fault diagnosis that the method based on vibration signal processing [START_REF] Wang | The fault diagnosis technology of mechanical equipment and its application[END_REF].The characters of vibration signal in time-frequency domain that is used to establish the initial feature space can be extracted by methods such as Fourier transform or wavelet transform etc. And then, the classifier is constructed with the further processing of initial feature space [START_REF] Wang | The fault diagnosis technology of mechanical equipment and its application[END_REF]. The dimensionality reduction method is taken to get a low-dimensional feature representation since the initial feature space is generally high-dimensional. The frequently used dimensional reduction methods, such as principal component analysis [START_REF] Wold | Principal component analysis[END_REF] or linear discriminant analysis [START_REF] Leo | Fault diagnosis based on fisher discriminant analysis and support vector machines[END_REF], is the linear dimensional reduction method. For the nonlinear signal, the linear method is not the best choice.

Manifold learning has been applied in many fields since three articles published in Science [START_REF] Seung | The manifold ways of perception[END_REF], [START_REF] Roweis | Nonlinear dimensionality reduction by locally linear embedding[END_REF], [START_REF] Tenenbaum | A global geometric framework for nonlinear dimensionality reduction[END_REF]. Manifold learning is a nonlinear data dimensional reduction method, which is used to extract low-dimensional manifold structure embed in high-dimensional space. Therefore there is a new way to diagnose the machinery fault. Generally speaking, the operation data of machinery with the same condition lie on the same manifold and the different condition lie on different manifold [START_REF] Jiang | Machinery fault diagnosis using supervised manifold learning[END_REF]. Based on this setting, manifold learning can be applied to machinery fault diagnosis.

A typical manifold learning method includes Local Linear Embedding (LLE) [START_REF] Roweis | Nonlinear dimensionality reduction by locally linear embedding[END_REF], Isometric Mapping(ISOMAP) [START_REF] Tenenbaum | A global geometric framework for nonlinear dimensionality reduction[END_REF] etc. A proper choice of neighborhood size is critical in manifold Learning. Using a fixed neighborhood size is inappropriate, such as in LLE and ISOMAP, because the curvature of the manifold and the density of the sample points may be different in different regions of the manifold in the engineering applications. Sparse Manifold Clustering and Embedding (SMCE) is proposed in [START_REF] Elhamifar | Sparse manifold clustering and embedding[END_REF]. In SMCE, the automatic choice of the neighborhood size is done by solving a sparse optimization problem. Thus SMCE is more suitable for engineering applications. In this paper, we briefly introduce the principle of SMCE and discuss its application in feature extraction and diagnosis of mechanical fault.

The principle of SMCE

SMCE can be used for manifold clustering and dimensionality reduction for multiple nonlinear manifold simultaneously. The key difference between SMCE and other manifold learning methods is that SMCE can automatically search neighbors by solving a sparse optimization problem and weight matrix is established by calculating weights between each points and its neighbors. And then, spectral clustering [START_REF] Ng | On spectral clustering: Analysis and an algorithm[END_REF] and Laplacian eigenmap [START_REF] Belkin | Laplacian eigenmaps and spectral techniques for embedding and clustering[END_REF] can be used for manifold clustering and dimensional reduction respectively.

Given a set of N samples {x i ∈ R D } N i=1 lying in n different manifolds {M l } n l=1 with intrinsic dimensions {d l } n l=1 . For each sample point x i ∈ M l consider the smallest ball B i ⊂ R D contains that the d l + 1 nearest neighbors of x i . Namely, the affine subspaces with intrinsic dimensions d l is spanned by the d l + 1 neighbors. This is the fundamental assumption of SMCE. That is,

|| j∈Bi k ij (x j -x i )|| 2 and j∈Bi k ij = 1 (1)
Where x j ∈ B i and j = i, ( 0) is the upper bound of error, k ij is coefficient. It is hard to know the diameter of B i , therefore the equation (1) can not be solved directly. The diameter of B i is selected according to empirical rules in the LLE and ISOMAP which are local and global manifold learning algorithm respectively. In SMCE, this challenge is to be solved by a sparse optimization problem. Consider a sample point x i and a sample set

{x j |j = i, j = 1, 2, • • • , N }, the column vector c i with dimension N -1 is obtained by solved Eq.(2). ||[x 1 -x i , • • • , x N -x i ]c i || 2 and 1 T c i = 1 (2)
Where the solution c i is sparse and the non-zero elements are corresponding to several sample points lying in the same manifold that x i belongs to. It is the key difference between SMCE and other manifold learning algorithm.

In the case of densely sampled set, the affine subspace coincides with the d l dimensional tangent space of M l at x i . In other words, the sample point corresponding to non-zeros elements of c i may no longer be the closest points to x i in M l . Therefore, the vectors {x j -x i } j =i is normalized and let

X i := x 1 -x i ||x 1 -x i || 2 , • • • , x N -x i ||x N -x i || 2 ∈ R D×(N -1) (3)
Thus the (2) has the following form:

||X i c i || 2 and 1 T c i = 1 (4)
In this way, among all the solutions of ( 4), the one that uses a few closest neighbors of x i is searched by considering the following 1 optimization problem.

min ||Q i c i || 1 s.t. ||X i c i || 2 , 1 T c i = 1 (5)
Where the proximity inducing matrix Q i , which is a positive definite diagonal matrix, is defined as

Q i = ||X j -X i || 2 t =i ||X t -X i || 2 ∈ (0, 1] (6)
Another optimization problem which is related to ( 5) by the method of Lagrange multipliers is

min λ||Q i c i || 1 + 1 2 ||X i c i || 2 2 s.t. 1 T c i = 1 (7)
Where the parameter λ sets the trade-off between the sparsity of the solution and the affine reconstruction error. By solving the optimization problem above for each sample point, it is obtained that the solution c T

i := [c i1 , • • • , c iN ]. Thus, the weight vector w T i = [w i,1 , • • • , w iN ] ∈ R N is defined as w ii = 0, w ij := c ij /||X j -X i || 2 t =i c it /||X t -X i || 2 , j = i (8)
The similarity graph G = (V, E) whose nodes represent sample points is built. Node i, corresponding to x i , connects the node j, corresponding to x j , with an edge whose weight is equal to |w ij |. Each node i connects only a few other nodes named sparse neighbors that correspond to the neighbors of x i in the same manifold.Hence, the similarity matrix W is constructed with weight vector w i and reflect the distance from sparse neighbors to x i . The samples are clustered by applying spectral clustering to W [START_REF] Ng | On spectral clustering: Analysis and an algorithm[END_REF],or by applying dimensionality reduction for original samples with Laplacian eigenmap [START_REF] Belkin | Laplacian eigenmaps and spectral techniques for embedding and clustering[END_REF], and then the classifier is constructed effectively.

The method of feature extraction based on SMCE

The mathematical principles of SMCE adapt to engineering applications because the high-dimensional Euclidean space in SMCE is corresponding to the feature space that is spanned by feature vectors of machinery. In practice, the manifolds embedded in feature space have the following characters: the density of sample points in manifold is different in different regions in the manifold, and curvature of each point in manifold is also different. Therefore, it is inappropriate that the fixed neighborhood size is used to extract the manifold structure within mechanical operation data.

The initial feature space should be constructed firstly before that the nonlinear dimensionality reduction is done by manifold learning. In the engineering application, the initial feature space can be spanned by feature vectors that should be composed by characters in time domain or characters in frequency domain, or characters in both time and frequency domain of vibration signal [START_REF] Wang | The fault diagnosis technology of mechanical equipment and its application[END_REF]. In this paper, we combine characters in time domain and frequency domain of vibration signals as an initial feature vector. The term of characters in time domain is shown in Tab.1, and the sub-band energy method is employed to establish frequency feature vector. The energy distribution in different frequency sub-bands of the vibration signals indicates the operation condition of machinery. Thus the sub-band energy vector could regard as the frequency feature vector. Divide vibration signal into m sub-bands with a constant bandwidth and the frequency range of i-th subband is shown in [START_REF] Ng | On spectral clustering: Analysis and an algorithm[END_REF].

f i = f i-1 + ∆f (9)
Where i = 1, 2, • • • , m, f i and f i-1 is the upper and low limiting frequency of i-th sub-band, ∆f is the bandwidth of i-th sub-band. Through Fast Fourier Transform(FFT), the number of frequency lines n i in each sub-band is equal, that is

n i = f i -f i-1 ξ ( 10
)
Where ξ is frequency resolution. Hence, all frequency lines can be divided into m equal partitions related to m sub-bands. Suppose that the sequence index of frequency lines in i-th sub-band is corresponding to k i-1 and k i , the energy of i-th sub-band is obtained by [START_REF] Loparo | Bearing vibration data set[END_REF] according to Parseval's theorem.

E i = ki k=ki-1 |A k | 2 ∆f ( 11
)
Where A k is the amplitude of frequency line whose sequence index is k. In this paper, the vibration signal is divided into 16 equal sub-band, thus we can obtain a feature vector y f of dimension 16. And y f is normalized by [START_REF] Zhang | Multiscale morphology analysis and its application to fault diagnosis[END_REF]. In a similar way, the feature vectors in time domain that is shown in Tab.1 can be normalized, and we obtained a normalized feature vector in time domain Y t with dimension 12. And then, the feature vectors Y = [Y f , Y t ] with dimension 28 is used to span the initial feature space.

Y f = y f -min(y f ) max(y f ) -min(y f ) ( 12
)
The presented method of machinery fault diagnosis based on manifold learning is done with following stage.

(1)For each vibration signal, the feature vector with dimension 28 is constructed firstly.

(2)Using appropriate parameter λ to build similarity matrix W . Due to there is not mature theory to guide us to choose the best parameter λ, therefore the choice of best parameter λ is done by equal step length searching. Namely, the diagnosis accuracy is different under different λ, and we fixed the λ corresponding to best accuracy.

(3) Clustering of the samples data by applying spectral clustering [START_REF] Ng | On spectral clustering: Analysis and an algorithm[END_REF] to W . The samples data from the same machinery condition is in one set and from the different condition is in different set. This is the unsupervised fault diagnosis approach. In addition, we apply laplacian eigenmap [START_REF] Belkin | Laplacian eigenmaps and spectral techniques for embedding and clustering[END_REF] to do dimensionality reduction and the low-dimensionality representation of original high-dimensionality feature space is obtained. And then, the support vector machine(SVM) is employed to do the classification of the low-dimensionality space. Because of SVM is one of supervised classifier, this approach of fault diagnosis is named supervised method of fault diagnosis.

Experiment

Bearing Date Center(BDC) [START_REF] Loparo | Bearing vibration data set[END_REF] of Case Western Reserve University has been used by many researchers [START_REF] Zhang | Multiscale morphology analysis and its application to fault diagnosis[END_REF], [START_REF] Yang | Fault diagnosis of rolling element bearings using basis pursuit[END_REF]. The availability of BDC has been proved, and this database has become a standard database of vibration signal of bearing defect. In this paper, The deep groove ball bearings SKF 6205-2RS-JEM-SKF with 9 rolling elements made by Swedish is employed as the analysis object. In experiment, the shaft speed is 1750RPM and the sampling frequency is 48KHZ. The vibration signal that collected by a accelerometer is shown in Fig. 1. Considering four condition of bearing, that is, normal, inner-race defect, ball defect, outer-race defect, 200 samples were chosen for each condition. Thus there are 800 samples in all and the sample length is 2048 of all. The feature space is spanned by 800 feature vectors, denoted by x 1 , x 2 , • • • , x 800 ∈ R 28 , and the manifold structure in the feature space is extracted by the method described previously. Similarity matrix W ∈ R 800×800 is obtained with fixed parameter λ. And then, the unsupervised fault diagnosis can be achieved by spectral clustering [START_REF] Ng | On spectral clustering: Analysis and an algorithm[END_REF]. For the supervised fault diagnosis, Laplacian eigenmap [START_REF] Belkin | Laplacian eigenmaps and spectral techniques for embedding and clustering[END_REF] is applied to the W , and original feature space is mapped to three dimensional space. And then, the samples of each condition in three dimensional space is divided into 2 equal partition. One of the half samples are taken as training samples, and the remaining are testing samples. The problem we faced is to classify 4 conditions, but SVM can classify only 2 condition at a time. Generalizing method to solve this challenging is by decomposing the 4 class problem into 2 class problem. Each time take one class of the training samples as positive class and take remaining class as negative class, thus classification of 4 class is done by doing SVM 4 times. The parameter λ play a important role for diagnosis, we make the parameter λ to traverse [START_REF] Belkin | Laplacian eigenmaps and spectral techniques for embedding and clustering[END_REF]200] with step 5. Under different λ, the diagnosis accuracy of spectral clustering and support vector machine is shown in Fig. 2. As the Fig. 2 show, the best accuracy of diagnosis with spectral clustering is up to 97.125% when the λ is set as 30. In the other way, three dimensional representation of origin feature space with dimension 28 is obtained under different λ. The best accuracy of diagnosis with SVM is up to 98.75% when the λ is set as 130. And the three dimensional space is shown in Fig. 3. Spectral clustering do the classification using the geometric structure of space spanned by sample point and without the tag of samples. This is the reason that it is named unsupervised diagnosis. Conversely, SVM do the classification with tags of samples, thus the diagnosis based on SVM is named supervised diagnosis. From Fig. 3, the diagnosis accuracy of SVM is higher than spectral clustering and the accuracy of both has remained stable under different λ. The accuracy difference between maximum and minimum is 0.625% and 1% with spectral clustering and SVM respectively. Namely, the parameter λ is easily fixed. It is demonstrated that the presented method is suitable for engineering applications. 5 Conclusion

(1)The method of feature extraction based on SMCE could extract low dimensional manifold structure that indicates the nature of mechanical condition embed in high-dimensional feature space.

(2)The feature vector that consist of time-domain characters and sub-band energy can be used to diagnose the condition of machinery.

(3)The diagnosis accuracy of presented method in this paper is less affected by the parameters, therefore it is more suitable for engineering applications thanks to the convenience of parameter choice.

Fig. 1 .

 1 Fig. 1. The original signal of bearing.(a)Normal condition. (b) Inner-race defect. (c) Ball defect. (d)Outer-race defect.

Fig. 2 .Fig. 3 .

 23 Fig. 2. The diagnosis accuracy with different λ

Fig. 4 .

 4 Fig. 4. The diagnosis accuracy with LLE and SVM

Table 1 .

 1 Characters in time domain

	k	y k	k	y k	k	y k	k	y k
	1	Peak value	4 Root mean square value 7	Kurtosis	10 Impulsion index
	2 Peak to peak value 5	Variance	8 Peak index 11 Wave index
	3 Absolute mean value 6 Degree of skewness 9 Kurtosis index 12 Tolerance index

  To reveal the outstanding characters of presented method, the fault diagnosis is done with LLE [5] instead of SMCE. The classifier is established with SVM. The accuracy of diagnosis is shown in Fig.4. It is illustrated that the diagnosis accuracy vary in a large range with different parameter. The comparison of diagnosis accuracy between SMCE and LLE is shown in Tab.2. The diagnosis method with SMCE is more stable than LLE. The satisfactory result is reached with nearly almost arbitrary parameter with SMCE, but the diagnosis result with LLE is more affected by the parameters. Therefore the convenience of parameter choice is one of the advantages of presented method.

Table 2 .

 2 The comparison of accuracy between SMCE and LLE

	Diagnosis method	minimum	maximum	mean	standard variance
	SMCE	97.75%	98.75%	98.12%	0.32%
	LLE	61.38%	97.88%	89.66%	8.57%


Bo Zhang
email: zhangb@ics.ict.ac.cn

Gang Ma

Xi Yang
email: yangx@ics.ict.ac.cn

Zhongzhi Shi
email: shizz@ics.ict.ac.cn

Jie Hao
email: haojie@xzmc.edu.cn

Automatic Image Annotation Based on Semi-supervised Probabilistic CCA

Keywords: probabilistic CCA, semi-supervised method, automatic image annotation

We propose a novel semi-supervised method for building a statistical model that represents the relationship between images and text labels (tags) based on a semi-supervised variant of CCA called SemiPC-CA, which extends the probabilistic CCA model to make use of the labelled and unlabelled images together to extract the low-dimensional latent space representing topics of images. Real-world image tagging experiments indicate that our proposed method improves the accuracy even when only a small number of labelled images are available.

Introduction

Automatic image annotation has become an important and challenging problem due to the existence of semantic gap. The state-of-the-art techniques of image auto-annotation can be roughly categorized into two different schools of thought. The first one defines auto-annotation as a traditional supervised classification problem , which treats each word (or semantic concept) as an independent class and creates different classifiers for every word. This approach computes similarity at the visual level and annotates a new image by propagating the corresponding words. The second perspective takes a different stand and treats images and texts as equivalent data. It attempts to discover the correlation between visual features and textual words on an unsupervised basis, by estimating the joint distribution of features and words. Thus, it poses annotation as statistical inference in a graphical model. Under this perspective, images are treated as bags of words and features, each of which are assumed generated by a hidden variable. Various approaches differ in the definition of the states of the hidden variable: some associate them with images in the database, while others associate them with image clusters or latent aspects (topics) .

As latent aspect models, PLSA [START_REF] Hofmann | Unsupervised learning by probabilistic latent semantic analysis[END_REF] and latent Dirichlet allocation (LDA) [START_REF] Blei | Latent dirichlet allocation[END_REF] have been successfully applied to annotate and retrieve images. PLSA-WORDS [START_REF] Monay | Modeling semantic aspects for crossmedia image indexing[END_REF] is a representative approach, which achieves the annotation task by constraining the latent space to ensure its consistency in words. However, since standard PLSA can only handle discrete quantity (such as textual words), this approach quantizes feature vectors into discrete visual words for PLSA modeling. Therefore, its annotation performance is sensitive to the clustering granularity. GM-PLSA [START_REF] Li | Modeling continuous visual features for semantic image annotation and retrieval[END_REF] deals with the data of different modalities in terms of their characteristics, which assumes that feature vectors in an image are governed by a Gaussian distribution under a given latent aspect other than a multinomial one, and employs continuous PLSA and standard PLSA to model visual features and textual words respectively. This model learns the correlation between these two modalities by an asymmetric learning approach and then it can predict semantic annotation precisely for unseen images.

Canonical correlation analysis (CCA) is a data analysis and dimensionality reduction method similar to PCA. While PCA deals with only one data space, CCA is a technique for joint dimensionality reduction across two spaces that provide heterogeneous representations of the same data. CCA is a classical but still powerful method for analyzing these paired multi-view data. Since CCA can be interpreted as an approximation to Gaussian PLSA and also be regarded as an extension of Fisher linear discriminant analysis (FDA) to multi-label classification [START_REF] Bach | A probability interpretation of canonical correlation analysis[END_REF], learning topic models through CCA is not only computationally efficient, but also promising for multi-label image annotation and retrieval.

However, CCA requires the data be rigorously paired or one-to-one correspondence among different views due to its correlation definition. However, such requirement is usually not satisfied in real-world applications due to various reasons. To cope with this problem, several extensions of CCA have been proposed to utilize the meaningful prior information hidden in additional unpaired data. Blaschko et al. [START_REF] Blaschko | Semi-supervised laplacian regularization of kernel canonical correlation analysis[END_REF] proposes semi-supervised Laplacian regularization of kernel canonical correlation (SemiLRKCCA) to find a set of highly correlated directions by exploiting the intrinsic manifold geometry structure of all data (paired and unpaired). SemiCCA [START_REF] Kimura | Semicca: Efficient semi-supervised learning of canonical correlations[END_REF] resembles the manifold regularization, i.e., using the global structure of the whole training data including both paired and unpaired samples to regularize CCA. Consequently, SemiCCA seamlessly bridges CCA and principal component analysis (PCA), and inherits some characteristics of both PCA and CCA. Gu et al. [START_REF] Gu | Localization with incompletely paired data in complex wireless sensor network[END_REF] proposed partially paired locality correlation analysis (PPLCA), which effectively deals with the semi-paired scenario of wireless sensor network localization by virtue of the combination of the neighbourhood structure information in data. Most recently, Chen et al. [START_REF] Chen | A unified dimensionality reduction framework for semi-paired and semi-supervised multiview data[END_REF] presents a general dimensionality reduction framework for semi-paired and semi-supervised multi-view data which naturally generalizes existing related works by using different kinds of prior information. Based on the framework, they develop a novel dimensionality reduction method, termed as semi-paired and semi-supervised generalized correlation analysis (S2GCA), which exploits a small amount of paired data to perform CCA.

We propose a semi-supervised variant of CCA named SemiPCCA based on the probabilistic model for CCA. The estimation of SemiPCCA model parameters is affected by the unpaied multi-view data (e.g. unlabelled image) which revealed the global structure within each modality. Then, an automatic image annotation method based on SemiPCCA is presented. Through estimating the relevance between images and words by using the labelled and unlabelled images together, this method is shown to be more accurate than previous publish methods.

This paper is organized as follows. After introducing the framework of the proposed SemiPCCA model briefly in Section 2, we formally present our automatic image annotation method based on SemiPCCA in Section 3. Finally Section 4 illustrates experiments results and Section 5 concludes the paper.

Framework

In this section, we first review a probabilistic model for CCA. Then armed with this probabilistic reformulation of CCA, we present our semi-supervised variant of CCA named SemiPCCA based on the probabilistic model for CCA. The estimation of SemiPCCA model parameters is affected by the unlabelled multi-view data which revealed the global structure within each modality.

Probabilistic Canonical Correlation Analysis

In [START_REF] Bach | A probability interpretation of canonical correlation analysis[END_REF], Bach and Jordan propose a probabilistic interpretation of CCA. In this model, two random vectors x 1 ∈ R m1 and x 2 ∈ R m2 are considered generated by the same latent variable z ∈ R d (min (m 1 , m 2 ) d 1) and thus the "correlated" to each other.

In this model, the observations of x 1 and x 2 are generated form the same latent variable z (Gaussian distribution with zero mean and unit variance) with unknown linear transformations W 1 and W 2 by adding Gaussian noise ε 1 and ε 2 , i.e.,

P (z) ∼N (0, I d ) , P (ε 1 ) ∼ N (0, Ψ 1 ) ,P (ε 2 ) ∼ N (0, Ψ 2 ) , x 1 = W 1 z + µ 1 +ε 1 , W 1 ∈ R m1×d , x 2 = W 2 z + µ 2 +ε 2 , W 2 ∈ R m2×d . (1)
From [START_REF] Bach | A probability interpretation of canonical correlation analysis[END_REF], the corresponding maximum-likelihood estimations to the unknown parameters

µ 1 , µ 2 , W 1 , W 2 , Ψ 1 and Ψ 2 are μ1 = 1 N N i=1 x 2 1 ,μ 2 = 1 N N i=1 x 2 2 , Ŵ1 = Σ 11 U 1d M 1 , Ŵ2 = Σ 22 U 2d M 2 , Ψ1 = Σ 11 -Ŵ1 Ŵ T 1 , Ψ2 = Σ 22 -Ŵ2 Ŵ T 2 , (2)
where Σ 11 , Σ 22 have the same meaning of standard CCA, the columns of U 1d and U 2d are the first d canonical directions, P d is the diagonal matrix with its diagonal elements given by the first d canonical correlations and M 1 , M 2 ∈ R d×d , with spectral norms smaller the one, satisfying M 1 M T 2 = P d . In our expectations, let

M 1 = M 2 = (P d ) 1/2
. The posterior expectations of z given x 1 and x 2 are

E (z|x 1 ) = M T 1 U T 1d (x 1 -μ1 ) , E (z|x 2 ) = M T 2 U T 2d (x 2 -μ2 ) . (3)
Thus, E (z|x 1 ) and E (z|x 2 ) lie in the d dimensional subspace that are identical with those of standard CCA.

Semi-supervised PCCA

Consider a set of paired samples of size N p ,

X P 1 = {(x i 1 )} N p i=1 and X P 2 = {(x i 2 )} N p i=1 , where each sample x i 1 (resp. x i 2
) is represented as a vector with dimension of m 1 (resp. m 2 ). When the number of paired of samples is small, CCA tends to overfit the given paired samples. Here, let us consider the situation where unpaired samples

X U 1 = {(x j 1 )} N 1 j=N p +1 and/or X U 2 = {(x k 2 )} N 2
k=N p +1 are additional provided, where X U 1 and X U 2 might be independently generated. Since the original CCA and PCCA cannot directly incorporate such unpaired samples, we proposed a novel method named Semi-supervised PCCA (SemiPCCA) that can avoid overfitting by utilizing the additional unpaired samples. See Figure 1 for an illustration of the graphical model of the SemiPCCA model.

The whole observation is now

D = {(x i 1 , x i 2 )} N p i=1 ∪{(x j 1 )} N 1 j=N p +1 ∪{(x k 2 )} N 2 k=N p +1
. The likelihood, with the independent assumption of all the data points, is calculated as

L(Θ) = N p i=1 P (x i 1 , x i 2 ; Θ) N 1 j=N p +1 P (x j 1 ; Θ) N 2 k=N p +1 P (x k 2 ; Θ) (4)
In SemiPCCA model, for paired samples {(x i 1 , x i 2 )} N p i=1 , x i 1 and x i 2 are considered generated by the same latent variable z i and P (x i 1 , x i 2 ) is calculated as in PCCA model, i.e. Whereas for unpaired observations

P (x i 1 , x i 2 ; Θ) ∼ N µ 1 µ 2 , W 1 W T 1 + Ψ 1 W 1 W T 2 W 2 W T 1 W 2 W T 2 + Ψ 2 . ( 5
) x 2 z 2  z x 1 x 2 1  2  1  N p 2  1 W 2 W 1  2  x 1 N 1 -N p z 1  N 2 -N p
X U 1 = {(x j 1 )} N 1 j=N p +1 and/or X U 2 = {(x k 2 )} N 2 k=N p +1 , x j
1 and x k 2 are separately generated from the latent variable z 1 j and z 2 k with linear transformations W 1 and W 2 by adding Gaussian noise ε 1 and ε 2 . From Eq.(1),

P (x j 1 ;Θ) ∼ N µ 1 , W 1 W T 1 + Ψ 1 , P (x k 2 ;Θ) ∼ N µ 2 , W 2 W T 2 + Ψ 2 . (6)
For means of x 1 and x 2 we have

μ1 = 1 N 1 N1 i=1 x i 1 , μ2 = 1 N 2 N2 i=1 x i 1 , (7)
which are just the sample means. Since they are always the same in all EM iterations, we can centre the data X P 1 ∪ X U 1 , X P 2 ∪ X U 2 by subtracting these means in the beginning and ignore these parameters in the learning process. So for simplicity we change the notation x i 1 , x i 2 , x j 1 and x k 2 to be the centred vectors in the following.

For the two mapping matrices, we have the updates

Ŵ1 = ( Np i=1 x i 1 z i T + N1 j=Np+1 x j 1 z 1 j T )( Np i=1 z i z i T + N1 j=Np+1 z 1 j z 1 j T ) -1 (8) Ŵ2 = ( Np i=1 x i 2 z i T + N2 k=Np+2 x k 2 z 2 k T )( Np i=1 z i z i T + N2 k=Np+1 z 2 k z 2 k T ) -1 (9)
Finally the noise levels are updated as

Ψ1 = 1 N 1 {( Np i=1 (x i 1 -Ŵ1 z i )(x i 1 -Ŵ1 z i ) T + N1 j=Np+1 (x j 1 -Ŵ1 z 1 j )(x j 1 -Ŵ1 z 1 j ) T )} (10) Ψ2 = 1 N 2 {( Np i=1 (x i 2 -Ŵ2 z i )(x i 2 -Ŵ2 z i ) T + N2 k=Np+1 (x k 2 -Ŵ2 z 2 k )(x k 2 -Ŵ2 z 2 k ) T )} (11)

Projections in SemiPCCA Model

Analogous to the PCCA model, the projection of a labelled image (x i 1 , x i 2 ) in SemiPCCA model is directly given by Eq.( 3).

Although this result looks similar as that in PCCA model, the learning of W 1 and W 2 are influenced by those unpaired samples. Unpaired samples reveal the global structure of whole the samples in each domain. Note once a basis in one sample space is rectified, the corresponding bases in the other sample space is also rectified so that correlations between two bases are maximized.

Annotation on unlabelled image

Now, we presents an automatic image annotation method based on the SemiPC-CA, which estimating the association between images and words by using the labelled and unlabelled images together.

Let X P 1 = {(x i 1 )} N p i=1 and X P 2 = {(x i 2 )} N p i=1 be the set of labelled images and its corresponding semantic features with m 1 and m 2 dimensions of size N p , and X U 1 = {(x j 1 )} N 1 j=N p +1 be a set of unlabelled images. The first step is to extracts image features and labels features of training samples, and generates the essential latent space by fitting SemiPCCA.

In the context of automatic image annotation, X U 1 only exists, whereas X U 2 is empty. So, for the mapping matrices W 2 and the noise levels Ψ 2 , we have to change the updates as follows, Ŵ2 = (

Np i=1 x i 2 z i T )( Np i=1 z i z i T ) -1 (12) Ψ2 = 1 N p {( Np i=1 (x i 2 -Ŵ2 z i )(x i 2 -Ŵ2 z i ) T )} (13)
Using this model, we derive the posterior probability of a sample in the latent space. When only an image feature x 1 is given, the posterior probability P (z 1 |x 1 ) of estimated latent variable z 1 becomes a normal distribution whose mean and variance are,

µ z1 = Ŵ1 T ( Ŵ1 Ŵ1 T + Ψ1 ) -1 (x 1 -μ1 ) , Ψ z1 = I -Ŵ1 T ( Ŵ1 Ŵ1 T + Ψ1 ) -1 , (14)
respectively. Also, when both an image feature x 1 and semantic feature x 2 are given, the posterior probability P (z|x 1 , x 2 ) becomes,

µ z = Ŵ T ( Ŵ Ŵ T + Ψ ) -1 x 1 x 2 -μ , Ψ z = I -Ŵ T ( Ŵ Ŵ T + Ψ ) -1 Ŵ . ( 15
)
The second step is to map labelled training images {T

(P ) i = (x i 1 , x i 2 )} N p i=1
and unlabelled images {(Q

(U ) j = x j 1 } N 1 j=N p +1
to the latent space with posterior probability P (z|x 1 , x 2 ) and P (z|x 1 ) separately, and K-L distance is used for measuring the similarity between two images.

We define the similarity between two samples as follows. When two labelled images T

(P ) i = (x i 1 , x i 2 ) and T (P ) j
= (x j 1 , x j 2 ) are available, then similarity is defined as,

D T (P ) i , T (P ) j = µ i z -µ j z T Ψ z -1 µ i z -µ j z , (16)
which measuring essential similarities both in terms of appearance and semantics. Furthermore, when labels feature of one of two samples is not available, e.g. one labelled image T

(P ) i = (x i 1 , x i 2 )
and one unlabelled image Q (U ) j = x j 1 , which is the usual case in automatic image annotation, our framework also enables measuring similarities with semantic aspects even in the absence of labels features, and their similarity becomes:

D T (P ) i , Q (U ) j = µ i z -µ j z1 T Ψ z -1 + Ψ z1 -1 2 µ i z -µ j z1 (17)
As we described, we can formalize a new image annotation method. Let x new demote a newly input image. To annotate x new with some words, we calculate the posterior probability posterior probability of a word w given by x new , which is represented as

P (w|x new ) = Np i=1 P w|T (P ) i P T (P ) i |x new . (18)
The posterior probability P T (P ) i

|x new of each labelled image T (P ) i using the above similarity measurement is defined as follow, P T

(P ) i |x new = exp -D T (P ) i , x new Np i=1 exp -D T (P ) i , x new , (19)
where, the denominator is a regularization term so that corresponds to the sample-to-label model, which is defined as P w|T

(P ) i = µδ w,T (P ) i + (1 -µ) N w N W , (20)
where N w is the number of the images that contain w in the training data set, δ w,T (P ) i = 1 if word w is annotated in the training sample T (P ) i

, otherwise δ w,T (P ) i = 0. µ is a parameter between zero and one. N W is the number of the words.

The words are sorted in descending order of the posterior probability P (w|x new ). The highest ranked words are used to annotate the image x new .

Experiments

This section describes the results for the automatic image annotation task.

We use Corel5K and Corel30K to evaluate the performance of the proposed method. Corel5K contains 5,000 pairs of the image and the labels. Each image is manually annotated with one to five words. The training data has 371 words. 260 words among them appear in the test data.

Corel30K dataset is an extension of the Corel5K dataset based on a substantially larger database, which tries to correct some of the limitation in Corel5k such as small number of examples and small size of the vocabulary. Corel30K dataset contains 31,695 images and 5,587 words.

We follow the methodology of previous works, 500 images from the Corel5K are the test data. The other 1500, 2250 and 4500 images are selected from the Corel5K as the training data respectively, alone with the remaining training images in Corel5K and 31,695 images in Corel30K which acted as the unlabelled image to estimate the parameters of SemiPCCA together.

Feature Representation

As the image feature, we use the color higher-order local auto-correlation (Color-HLAC) features. This is a powerful global image feature for color images. Generally, global image features are suitable for realizing scalable systems because they can be extracted quite fast. Also, they are well suited for unconstrained image level annotation.

The Color-HLAC features enumerate all combinations of mask patterns that define autocorrelations of neighboring points and include both color information and texture information simultaneously. In this paper we use at most the 2nd order correlations, whose dimension is 714. The 2nd order Color-HLAC feature is reduced by PCA to preserve the 80 dimensions.

We extract Color-HLAC features from two scales (1/1, 1/2 size) to obtain robustness against scale change. Also, we extract them from edge images obtained by using the Sobel filter as well as the normal images. In all, the final image features are 320 dimensions.

As for labels feature, we use the word histogram. In this work, each image is simply annotated with a few words, so the word histogram becomes a binary feature.

Evaluation and results

In this section, the performance of our model (SemiPCCA) is compared with several models. Image annotation performance is evaluated by comparing the captions automatically generated for the test set with the human-produced ground truth. For evaluation of annotation performance of our method, we follow the methodology of previous works. We define the automatic annotation as the five semantic words of largest posterior probability, and compute the recall and precision of every word in the test set. For a given semantic word, recall = B/C and precision = B/A, where A is the number of images automatically annotated with a given word; B is the number of images correctly annotated with that word; C is the number of images having that word in ground truth annotation. The average word precision and word recall values summarize the system performance.

Table 1 shows the results obtained by the proposed method and various previously proposed methods --CRM [START_REF] Jeon | Automatic image annotation and retrieval using crossmedia relevance models[END_REF],MBRM [START_REF] Feng | Multiple bernoulli relevance models for image and video annotation[END_REF], PLSA-WORDS [START_REF] Monay | Modeling semantic aspects for crossmedia image indexing[END_REF], GM-PLSA [START_REF] Li | Modeling continuous visual features for semantic image annotation and retrieval[END_REF], using Corel5K. In order to compare with those previous models, we divide this dataset into 2 parts: a training set of 4,500 images and a test set of 500 images. We report the results on two sets of words: the subset of 49 best words and the complete set of all 260 words that occur in the training set. From the table, we can see that our model performs significantly better than all other models. We believe that using SemiPCCA to model visual and textual data by labelled and unlabelled images respectively is the reason for this result.

Conclusions

This paper presents an automatic image annotation method based on the SemiPC-CA. Through estimating the association between images and words by using the labelled and unlabelled images together, this method is shown to be more accurate than previous publish methods. Experiments on the Corel dataset prove that our approach is promising for semantic image annotation. In comparison to several state-of-the-art annotation models, higher accuracy and superior effectiveness of our approach are reported.

  This work is supported by the National Program on Key Basic Research Project(973 Program)(No. 2013CB329502), National Natural Science Foundation of China(No.61035003), National High-tech R&D Program of China(863 Program)(No.2012AA011003), National Science and Technology Support Program(No.2012BA107B02), Natural Science Foundation of Jiangsu Province(No.BK20160276).

Fig. 1 .

 1 Fig. 1. Graphical model for Semi-supervised PCCA. The box denotes a plate comprising a data set of Np paired observations, and additional unpaired samples.

Table 1 .

 1 Performance comparison of different automatic image annotation models on Corel5k dataset.

	Models	CRM MBRM PLSA WORDS GM PLSA Semi PCCA
	#Words with recall > 0 107	122	105	125	151
		Results on 49 best words,
	MR	0.70 0.78	0.71	0.79	0.94
	MP	0.59 0.74	0.56	0.76	0.77
	F1	0.64 0.76	0.63	0.77	0.85
		Results on all 260 words,
	MR	0.19 0.25	0.20	0.25	0.32
	MP	0.24 0.24	0.14	0.26	0.24
	F1	0.21 0.24	0.16	0.25	0.27


Zhenzhou Shao
email: zshao@cnu.edu.cnyqu3

Zhiping Shi

Ying Qu

Yong Guan
email: guanyong@cnu.edu.cnyqu3

Hongxing Wei
email: weihongxing@buaa.edu.cn

Jindong Tan
email: tan@utk.edu

A Confidence Weighted Real-Time Depth Filter for 3D Reconstruction

Keywords: Depth sensor, depth filter, image processing, 3D reconstruction

3D reconstruction is an important technique in the environmental perception and rehabilitation process. With the help of active depth-aware sensors, such as Kinect from Microsoft and SwissRanger, the depth map can be captured at the video frame rate together with color information to enable the real-time reconstruction. Particularly, it features prominently in the activity recognition and remote rehabilitation. Unfortunately, the coarseness of the depth map make it difficult to extract the detailed information in 3D reconstruction of the scene and tracking of thin objects. Especially, geometric distortions occur around the edge of an object. Therefore, this paper presents a confidence weighted real-time depth filter for the edge recovery to reduce the extra artifacts due to the uncertainty of each depth measurement. Also the intensity of depth map is taken into account to optimize the weighting term in the algorithm. Moreover, the GPU implementation guarantees the high computational efficiency for the real-time applications. Experimental results are shown to illustrate the performance of the proposed method by the comparisons with the traditional methods.

Introduction

3D reconstruction technique is usually used for the rehabilitation purposes, such as activity recognition in remote rehabilitation [START_REF] Díaz Rodríguez | Ubiquitous Computing and Ambient Intelligence[END_REF], facial muscle and tongue movement capture in speech recovery sessions [START_REF] Suciu | On the Move to Meaningful Internet Systems[END_REF], joint kinematics supervision [START_REF] Tuan Dao | Knowledge and Systems Engineering[END_REF] and so on. In recent years, the emergence of depth-aware sensors, such as Flash Lidar [START_REF] Amzajerdian | Lidar systems for precision navigation and safe landing on planetary bodies[END_REF], Time-of-Flight (ToF) [START_REF] Mutto | Accurate 3D Reconstruction by Stereo and ToF Data Fusion[END_REF] and Kinect from Microsoft, provides a potential solution for real-time 3D reconstruction. This kind of sensor can capture the depth image at video frame rate, and is quite suitable for 3D rendering of the dynamic scenes by integrating with the color camera. It can be used for environmental perception, autonomous navigation, data visualization, robotics and 3D entertainment. Detailed applications based on depth sensors are reviewed in [START_REF] Lun | A survey of applications and human motion recognition with microsoft kinect[END_REF].

However, the depth map generated by depth sensors is interfered by the random noise and systematic errors. Especially, some geometric distortions with invalid depth measurements are introduced around the edge of an object. The coarseness of the depth measurements make it difficult to extract the detailed information for further processing such as segmentation, measurement and human pose estimation. Therefore, edge recovery is necessary to guarantee the high fidelity of final 3D rendering. Another problem is the real-time requirement for some applications. The effectiveness and computational cost of the filter implementation have to be considered.

There exist some algorithms to reduce the noise level of the depth map. A few denoising approaches are proposed based on a bilateral filter, which is an edge-preserving and noise reducing smoothing filter [START_REF] Kornprobst | Bilateral filtering: Theory and applications[END_REF]. In this paper, we call it standard bilateral filter to distinguish from the algorithm below. In [START_REF] Kopf | Joint bilateral upsampling[END_REF], joint/cross bilateral filter (JBF) is presented using the color information to enhance the depth map. It assumes that the depth edges are consistent with the color edges. Unfortunately, the guide of color information runs the risk of the introduction of the new artifacts from the color image. Therefore, Chan et al. [START_REF] Chan | A noise-aware filter for real-time depth upsampling[END_REF] proposed a noise-aware filter incorporating the depth intensity to balance the influence from the color image. However, the introduction of the noise-contained depth intensity also brings extra uncertainty. Another alternative way to denoise the depth map is called Non-Local Means (NL-Means) filter [START_REF] Huhle | Robust non-local denoising of colored depth data[END_REF], which is the extension of bilateral filter. Considering the noise in the depth map, patches surrounding the filtered pixel and neighborhood are taken into account instead of comparing the single pixel value. Although the accuracy is higher, it is not suitable for the real-time applications due to the heavy overhead.

This paper presents a real-time Confidence Weighted Depth Filter (CWDF) for 3D reconstruction to deal with the geometric distortions around the edges of objects. Following the concept of the noise-aware filter, a confidence map of the depth sensor is estimated as the weight of the depth intensity to reduce the influence of the noise from depth measurements, which also solves the problem NL-means filter can deal with. To speed up the filter implementation, the filter is decomposed into a number of constant time spatial filters. And with the GPU support, the proposed algorithm can be performed in a real-time manner.

CWDF: Confidence weighted depth filter

Depth measurements are interfered by the random noise and artifacts, so that the neighbors for averaging the gray-scale depth produce undesirable artifacts near strong discontinuities especially. In order to reduce the effect on the weighted result, a confidence weighted depth filter is proposed as a modified noise-aware depth filter, where con-fidence estimation C(q) is introduced into the computation of the weighting term, as shown in Eq. 1.

I C (p) = 1
K(p) ∑ q∈Ω f s ( p -q )[α( Ω ) f r ( Ĩ(p) -Ĩ(q) ) + (1 -α( Ω ))C(q) f r ( I(p) -I(q) )]I(q) (1)
where f s , f r , f r are all Gaussian kernels. Compared with JBF, f r is a new range term. K(p) is a normalizing factor. Confidence measurement is denoted by C(q), which will be estimated using the method in Sect. 2.1. And C(q) features the proposed method to optimize the noise-aware filter in this paper. α( Ω ) is used to balance the influence of depth and color intensity, as shown in Eq. 2.

α( Ω ) = 1 1 + e -ε( Ω -τ) (2)
where the parameters ε and τ are chosen to adjust the rate of change in transition area and the blending influence at the minimal min-max difference, respectively. These values depend on the characteristics of the sensor, and can be obtained by the empirical experiments. Ω is the difference between the maximum and minimum gray-scale value in the neighbors Ω of the pixel in depth map. When α is higher, the filter behaves like a standard bilateral filter with the C(q).

Confidence estimation

In this paper, Kinect is chosen to construct the depth map based on the light coding technique. The emissive light is organized using the speckle pattern, which is projected onto the surface of an object. The speckle pattern can change along with the different distance. Through comparing with the reference pattern, the depth map can be constructed.

Confidence estimation is mainly used to determine the reliability of each value in the depth map. In this paper, the confidence measurement is taken into account following the truth that depth values that are more reliable should make more contributions to the final result. To estimate the confidence, absolute difference is employed as the cost. In [START_REF] Sing | Handling occlusions in dense multi-view stereo[END_REF], the depth d 0 with the lowest cost is considered as the true value, although the depth perturbed by the noise and artifacts. We assume the cost of each pixel is subjected to Gaussian distribution. Let c(d, x) be the cost for depth d at pixel x. The probability is proportional to e -(c(d,x)-c(d 0 ,x)) 2 /σ 2 , where d 0 is the ground truth for a specific depth, and σ 2 denotes the strength of the noise.

For each specific depth d i , the confidence C (x, d i ) is defined as the inverse of the sum of these probabilities for all possible depths. In order to estimate the confidence better, C (x, d i ) is calculated with the N truth values. Then the final confidence C(x) can be derived by averaging the estimations at the different depth, as shown in Eq. (3).

C (x, d i ) = ( ∑ d =d i e -(c(d,x)-c(d i ,x)) 2 /σ 2 ) -1 C(x) = 1 N N ∑ i=1 C (x, d i ) (3)

Determination of the filtered region

To avoid introducing extra artifacts and improve the ability of real-time processing, the filtered region is determined firstly. Generally, the significant noise occurs around the object's edges according to the empirical measurement. In this paper, we assume that discontinuities in depth maps is the center line of the filtered region. Canny edge detector is applied to the gray-level depth image to obtain the location of the filtered region. Then, object's edge is dilated using 3 × 3 rectangular structuring element. To facilitate further processing using the method proposed in this paper, the pixels in the filtered region are labeled.

Approximation of the CWDF

According to Eq. 1, it is also signal-related filter, including f r ( Ĩ(p) -Ĩ(q) ) and f r ( I(p) -I(q) ) terms. To employ the recursive implementation of the Gaussian filter, The CWDF is approximated to approach the evolutional expression as shown as follows.

I C (p) = α( Ω ) K 1 (p) ∑ q∈Ω f s ( p -q ) f r ( Ĩ(p) -Ĩ(q) )I(q) + 1 -α( Ω ) K 2 (p) ∑ q∈Ω f s ( p -q ) * C(q) f r ( I(p) -I(q) )I(q) (4)
where K 1 (p) and K 2 (p) are normalizing factors. The proposed filter is decomposed into a joint bilateral filter and a standard bilateral filter related to confidence map C(q). The CWDF can be expressed as Eq. 5. B J I(p) (p) and B S I(p) (p) are defined for the joint bilateral filter and standard bilateral filter respectively. In practice, min(I) and max(I) in two bilateral filters are usually different, so we set [min(min(I), min( Ĩ)), max(max(I), max( Ĩ))] as the range of intensity value in order to guarantee the completeness of LUT.

I C (p) =CW I(p) (p) =α( Ω )B J I(p) (p) + (1 -α( Ω ))B S I(p) (p) (5)
Only N intensity values are chosen from [min(min(I), min( Ĩ)), max(max(I), max( Ĩ))] to be implemented the proposed filter. The remaining can be obtained by the linear interpolation after looking up the both the nearest intensity values from LUT.

In addition, to ensure the synchronous implementation of the proposed method, the capturing and filtering procedure are run in two separate threads in a thread-safe way based on interlock mechanism, which prevents more than one thread from using the same variable simultaneously. And in practice, the filter is implemented on a Nvidia's graphics card to achieve the real-time denoising performance. As a data-parallel computing device, the massive stream processing is executed using a high number of threads in parallel. In the experiments below, the CUDA programming framework [START_REF]Nvidia[END_REF] is employed to port the filter processing onto graphics hardware only. And the selection of the appropriate granularity is a trade-off between the runtime and memory.

Experimental results

In this section, Kinect from Microsoft is chosen to evaluate the proposed algorithm for its simple setup and depth map at a granularity of 640 × 480 pixels, which is higher than 128 × 128 of Flash Lidar and 176 × 144 of Mesa Swissranger T M . In addition, the resolution of color image is the same with the depth map, so that we can align them easily. Especially, both of them can record scenes at up to 30 fps in real time.

To demonstrate the effectiveness and accuracy of the proposed method, the algorithm was applied to the real-world sequences and scenes from the Middlebury stereo benchmark data set [13] and Advanced Three-dimensional Television System Technologies (ATTEST) [START_REF] Redert | Attest: Advanced three-dimensional television system technologies[END_REF]. In the following, two main aspects are discussed in details, including the resultant comparison with the joint bilateral filter and noise-aware filter in [START_REF] Chan | A noise-aware filter for real-time depth upsampling[END_REF]. All computations in the experiments were performed on two Intel Core (TM) P8700 CPUs with an Nvidia GeForce GT 120M.

Quality and effectiveness analysis

Several different scenes were captured to test the proposed algorithm. One scene shows two boxes with severely noise-affected edges, as shown in Figure . 1(b) in the red rectangular region. Particularly, in Figure . 4(a), the color image is modified by drawing a red rectangle across the edge of one box to evaluate the performance of the joint bilateral filter specifically.

Figure . 2 shows the results using the proposed filter, comparing with that using joint bilateral filter and noise-aware filter. As visible, all algorithms above can reduce the noise level. However, the intensity texture pattern of the red mark in the color image is introduced as 3D structure into the filtered depth by the joint bilateral filter, as shown in right red rectangle in Figure . 2(c). Although texture copying is avoided using the noiseaware filter, there are still some noise around the bottom edge due to the low confidence measurement, shown in Figure . 2(d). In contrast, confidence weighted depth filter gets the edges across the boxes straight, and prevents the texture copying effectively. Therefore, the proposed method can sharp details in the original depth image and improve the reconstruction quality. The detailed view warped by color information in Figure . 3. The noise across the edges is almost removed, while the depth measurements are severely interfered in the original 3D view. The point cloud contains more than 300, 000 points, although some points with invalid depth are eliminated. Due to the average filtering time 47ms with GPU support, the capturing rate of the device is set to 20 fps. Then the resultant colored depth map is transmitted to a server for 3D reconstruction using SURF (Speeded Up Robust Feature) feature descriptor.

Quantitative accuracy

To evaluate the performance of our filter, Cones scene from Middlebury stereo data set and video-plus-depth sequence "Interview" from ATTEST are considered. In order to illustrate the effectiveness of the confidence estimation and quantitative accuracy, confidence based noise is superimposed on the original depth image, which is considered as the ground truth. Then the filtered depth map will be compared with the ground truth. Supposed the confidence measurement C depends on the distance from the position of current pixel p to the image center c, and the noise n is subject to Gaussian distribution with mean 0, the noise model is defined as follows.

C = 1 - p -c w n ∼ N(0, 100 • (1 -C)) (7)
where w is a factor to balance the influence of the additional noise. To evaluate the numerical accuracy, peak signal-to-noise ratio (PSNR) is employed. Given two gray-scale images I, I , usually the ratio is defined using the mean squared error (MSE).

MSE = 1 hw h-1 ∑ i=0 w-1 ∑ j=0 I(i, j) -I (i, j) 2 PSNR = 10 • log 10 ( MAX 2 MSE ) (8)
where h and w is the height and width of input image, and MAX is the maximum of the pixel value, which means MAX = 255, if both input are gray images, where the pixel is indicated by 8 bits. The PSNR of the depth with noise and original depth image is 24.16 dB. To speed up the implementation, only N intensity values are selected to perform the proposed algorithm. Figure . 5 shows the result of the PSNR accuracy using different number of intensity levels. When 8 levels are used, the noise level is reduced greatly, and an acceptable PSNR value is achieved, although there exists visible difference. Figure . 4(f) shows the filtered result using 8 intensity levels. Note that as the level number increases, we have to compromise the computational time. Moreover, video-plus-depth sequence "Interview" from ATTEST is used with the yuv420 video format for comprehensive test. The sequence is decoded based on the leading audio/video codec library in FFmpeg. Every frame is in the size of 720 by 576 for both color and depth sequence. The depth is indicated by 8 bits, the brighter, the closer. The first 60 frames are chosen to implement our method, joint bilateral filter and noise-aware filter respectively. Similarly, noise superimposition is repeated based on the same noise model above. Then the PSNR accuracy of each frame is calculated, as shown in Figure . 6. the proposed method has the best performance with 39 dB on average. Due to the resolution gets bigger, the GPU implementation is prolonged to the averaging 55 ms.

Conclusion

The accurate 3D reconstruction is required for the environmental perception and rehabilitation process. This paper presented a confidence estimation based depth filter to recover the edges around the object in 3D reconstruction of a scene with color and depth images. The proposed method takes into account both the inherent depth nature of real-time depth data and the color information from video camera to reconstruct a multi-lateral filter. Compared with the existing joint bilateral filter and noise-aware filter, the experimental results show that the proposed method obtains more detailed information around the edge of an object and reduces the geometric distortions. Using the parallel data processing based on GPU implementation, the real-time high-quality 3D reconstruction is achieved.

In future, the adaptive parameter selection for the kernel size in the range term will be achieved, while these parameters are set manually according to the different scenes in current phase. In addition, multiple depth sensors will be employed for dynamic environment. A corresponding distributed system will be set up, and more studies will concentrate on the time synchronization of multiple data streams in order to ensure the temporal stability.

  Corresponding author. This paper is supported by Beijing Advanced Innovation Center for Imaging Technology, Development and application of domestic robot embedded real-time operating system (No. 2015BAF13B01) and Training young backbone talents personal projects (No. 2014000020124G135).

  For instance, for I(p) ∈ [I(p 1 ), I(p 2 )], I C (p) =(I(p 2 ) -I(p))CW I(p 1 ) (p) +(I(p) -I(p 1 ))CW I(p 2 ) (p)

Fig. 1 .Fig. 2 .

 12 Fig. 1. Color image and depth image with two boxes.

Fig. 3 .

 3 Fig. 3. Comparison of the 3D view.

  Figure. 4(e) shows the depth image with Gaussian noise, where the image distortion only exists across the edge, as shown in Figure. 4(d).

Fig. 4 .

 4 Fig. 4. Algorithm evaluation using the cones scene from the Middlebury data set.

Fig. 5 .

 5 Fig. 5. PSNR accuracy using different number of intensity values.

Fig. 6 .

 6 Fig.6. PSNR accuracy with respect to different methods.


Bo Yang
email: yangbo32@iie.ac.cn

Jianguo Jiang

Ning Li

Towards Discovering Covert Communication through Email Spam

Keywords: Email, spam, digital forensics, LDA

Recently, email spam has been noticed as a covert communication platform for criminals. However, investigators tend to overlook this kind of evidence during an investigation, and searching for incriminating information from unstructured textual data is one of the most cumbersome missions due to characteristics of email spam. This paper is the first work that presents a unified text mining solution to detect digital evidence from spam emails. It is helpful in the initial stage of investigation, in which investigators often have little information on the collection of spam emails. Our proposed solution applies a topic modeling technique, Latent Dirichlet Allocation, and a text visualization technique to discover various suspicious emails based on different camouflage methods. We present experimental results on a data set collected by the Spam Archive, which comprises 100 random spam emails. The results suggest that the proposed method is able to identify potential evidence.

Introduction

As one of the main communication forms, email has been gaining popularity, both for business and individuals. Because of its efficiency, convenience, and low cost, email allows users to communicate with each other at will, and also to manage user's personal information in a convenient way [START_REF] Whittaker | Email in personal information management[END_REF]. However, similar to other communication methods, suspects or adversaries use email for illegitimate purposes as well. They employ email to facilitate their schemes. For example, an abundance of evidence indicates email is used in terror plots during the events of 9 / 11 investigation [START_REF]on Terrorist Attacks Upon the United States and U. S. of America[END_REF]. Consequently email has been demonstrated as a very important source of evidence in investigations [START_REF] Casey | Digital evidence and computer crime[END_REF].

Email spam, which is sent to recipients with unsolicited and unrelated content, has been becoming more capricious with the advance of techniques [START_REF] Wang | A study on evolution of email spam over fifteen years[END_REF]. Email spam has been a persistent problem, since it has occupied most share of email traffic nowadays [START_REF] Group | Email metrics report[END_REF]. However, digital investigators usually pay no attention to this kind of seemingly irrelevant information. Commercial advertisement is the main category of email spam message. Email spam is always related to identity theft, phishing and malware distribution for illegitimate purposes. Moreover, crucial incriminating information may be placed in the email spam message. Along with hundreds of bona fide spam emails hidden in the spam folder, it is an effective camouflage, partly due to the rampant problem of email spam.

In any case involving email evidence, practitioners have to search crucial incriminating communication between or among suspects from high volume unstructured textual messages. In the current practice practitioners use modern computer forensics tools perform keyword searches at first, and then read flagged mails one by one for incriminating information. Detailed and thorough analysis is needed in this manual process. Moreover, the difference of investigator's experience or expertise may influence the investigation. Nevertheless, these tedious analysis tasks still miss crucial evidence frequently. How to improve effectiveness of text analysis has been studied by lots of researchers. For instance, in order to discover criminal networks, Al-Zaidy et al. [START_REF] Al-Zaidy | Mining criminal networks from unstructured text documents[END_REF] make use of a modified Apriori algorithm to extract hidden clues from email message. An implementation of customized associative classification techniques is proposed by Schmid et al. [START_REF] Schmid | E-mail authorship attribution using customized associative classification[END_REF] to address the problem of email author-ship attribution. The methods referred above apply to investigation involving email message except email spam. In contrast to messages from the inbox folder, the sending of spam message is in batch, and the message contents are different to each other. It is impossible to discover direct or indirect related clues not only based on the sending behavior but also analyzed by their content patterns. Moreover, researchers in forensics always tend to ignore the importance of processing spam emails until recently [START_REF] Yu | Covert communication by means of email spam: A challenge for digital investigation[END_REF].

This paper is motivated by an article [START_REF] Yu | Covert communication by means of email spam: A challenge for digital investigation[END_REF] named "Covert communication by means of email spam: a challenge for digital investigation", which illustrate seemly irrelevant messages might contain crucial incriminating information. We propose a unified text mining solution to detect convert communication in email spam. The detection method employs the topic modeling technique, Latent Dirichlet Allocation (LDA), and a visualization and information-retrieval technique to extract clues from the content of a spam email.

Email spam Characteristics

It is difficult for investigators to detect covert information from hundreds of spam emails. Because most of spam message are commercial advertisements and poorly correlated with any specific case, crucial evidence hidden in email spam could be overlooked. In this section, email spam features are generalized at first, and then we make an introduction of covert communication methods used in email spam and our detection strategies.

Email spam characteristics

The sending behavior of spammers and the spam message content are important features to detect email spam [START_REF] Tang | Email mining: tasks, common techniques, and tools[END_REF]. In our solution, we first use these features to detect non-spam messages. These non-spam messages appearing in the spam folder are most suspicious obviously so that we can acquire important clues immediately by searching for them.

■Spammers post email spam emails in bulk to spread out. Moreover, spammers employed spoofing-the-sender-address techniques as their camouflage. We can look into headers of message to study the sending behavior.

■Features selection from messages poses special challenges due to its characteristics of content. Email spam messages are informal in style and often do not obey established syntax or grammar rules. Commercial advertisement in the spam messages are irrelated information to investigations. Furthermore, spam messages include plenty of URL links, HTML web pages or images. With the development on URL camouflage techniques, spammers raise the proportion of URL links sharply. Based on Wang et. al's [START_REF] Wang | A study on evolution of email spam over fifteen years[END_REF] results, spammers decrease the percentage of spam message containing image to less than 5%.

Covert communication methods

In the section, we introduce the covert communication methods briefly. Yu [START_REF] Yu | Covert communication by means of email spam: A challenge for digital investigation[END_REF] summarizes five scenarios in according to real digital investigations.

1. Computer-aided-encryption message: Encryption technology is employed by criminals to produce encrypted messages through internet. The encrypt-decrypt algorithm is required to master by the two communicating parties in advance. In these encrypted messages, some seemly random text without meaning is pro-vided for specific purpose actually. Since criminals use this method to deliver specific information, this kind of text representing particular meaning appears unique in the collection of email spam. Therefore, we can find suspicious messages by searching for these unique words.

2. Manual-encryption message: In order to make encrypted message less noticeable, criminals create their own encrypted algorithm by hand. Although this kind of method is more flexible, it requires more creativity. Usually, this kind of encryption is a challenge for investigators. However, criminals also face difficulties for interpreting the algorithm themselves. Consequently, detectable patterns or some specific context is required for the manual encryption. Our strategy is to extract named entities from text in order to find encryption clues. Named (per-son, location, organization, misc), numerical (money, number, ordinal, percent), and temporal (date, time, duration, set) items constitute named entities in general. It contributes to find clues hidden in an encrypted message based on specific items.

3. Link-to message: The recipient receives a link from the sender, and the recipient acquires the actually location where the message is. Due to the elusiveness of the website, it requires digital forensics practitioners examine the address that the link leads to at once. Therefore, our method search for every link from spam messages firstly, and then passes these links to specific investigators who responsible for examining websites.

4. Steganography message: images from attachments are applied to deliver covet information by steganography techniques. Our solution concentrates only on semantic analysis of email spam. We leave problems relating to this method to future work.

5. Direct-message: The sender inserts message into a real spam email for hiding. The solution is analogous to the second scenario.

Methodology

In this section, we introduce TF-IDF algorithm that is used to measure the important of a word in the collection at first. Second we review how LDA works. Then we de-scribe the text visualization techniques employed. Finally, our proposed method is described.

TF-IDF

As one of the most popular algorithms, TF-IDF [START_REF] Salton | Introduction to modern information retrieval[END_REF] is employed in information retrieval and text mining extensively. In a collection, TF-IDF algorithm computes a numerical statistic for each word in order to present its important degree to a document. The times a word occurs in a given document are related to the importance directly, but are inversely proportional to the frequency of the word. The TF-IDF weight of 𝑤𝑜𝑟𝑑 𝑗 is computed as follows:

𝑇𝐹 -𝐼𝐷𝐹( 𝑤𝑜𝑟𝑑 𝑗 ) = 𝑇𝐹( 𝑤𝑜𝑟𝑑 𝑗 ) × 𝐼𝐷𝐹( 𝑤𝑜𝑟𝑑 𝑗 ) Term Frequency (TF) represents the occurrences a specific word. However, there are many words in practice, such as stopwords, that actually do not help to the meaning of a document. Jones [START_REF] Jones | A statistical interpretation of term specificity and its application in retrieval[END_REF][START_REF] Robertson | Understanding inverse document frequency: on theoretical arguments for idf[END_REF] proposed Inverse Document Frequency (IDF) to remove the influence of words that occur frequently in a collection. The IDF of 𝑤𝑜𝑟𝑑 𝑗 is computed as follows:

𝐼𝐷𝐹( 𝑤𝑜𝑟𝑑 𝑗 ) = log 𝑁 𝐷𝐹( 𝑤𝑜𝑟𝑑 𝑗 )
where Document Frequency (DF) is defined to be the number of documents containing a given word.

TF-IDF assigns to 𝑤𝑜𝑟𝑑 𝑗 a high weight when 𝑤𝑜𝑟𝑑 𝑗 occurs frequently in a few documents, whereas it assigns to 𝑤𝑜𝑟𝑑 𝑗 a low weight when 𝑤𝑜𝑟𝑑 𝑗 appears in many documents. Usually, each document is regarded as a vector with TF-IDF weight corresponding to each word, and a collection is regarded as a TF-IDF matrix.

In our solution, we make transformation for our dataset between word-document cooccurrence matrixes into a TF-IDF matrix as input of LDA topic model in the next step.

Latent Dirichlet Allocation

Latent Dirichlet Allocation (LDA) [START_REF] Blei | Latent dirichlet allocation[END_REF] proposed by Blei et al. is an unsupervised machine learning technique. A topic model is a generative model for documents: it specifies a simple probabilistic procedure by which documents can be generated [START_REF] Steyvers | Latent semantic analysis: a road to meaning, chapter probabilistic topic models[END_REF]. As a kind of probabilistic topic model, LDA has been used to model and discover underlying topic structures of any kind of discrete data, such as text data.

LDA assumes that documents exhibit multiple latent topics, where each topic is a multinomial distribution over a fixed vocabulary. The topics are shared by all documents in the collection, but the topic proportions vary stochastically across documents, as they are randomly drawn from a Dirichlet distribution. There are three level parameters to the LDA representation. The Dirichlet priors α and β over the document and topic respectively distributions are corpus-level parameters. The multinomial random parameter θ over topics is document-level parameter, and the Z and W are word-level variables. The graphical model representation of LDA is shown in Figure1. By introducing the Dirichlet priors α and β over the document and topic distributions respectively, the generative model of LDA is generalized to process unseen documents. The generative process of the topic model specifies a probabilistic sampling procedure that describes how words in documents can be generated based on the hidden topics. It can be described as follows:

Given the parameters 𝛼 and , the LDA model expression is described as the joint probability distribution of a topic mixture 𝜃, a set of N topics 𝒛, a set of N words 𝒘 :

𝑝(𝜃, 𝒛, 𝒘|𝛼, 𝛽) = 𝑝(𝜃|𝛼) ∏ 𝑝(𝑍 𝑛 |𝜃)𝑝(𝑤 𝑛 |𝑍 𝑛 , 𝛽) 𝑁 𝑛=1
Gibbs sampling, the most commonly used sampling algorithm for topic modeling, is used to approximate the posterior probability distribution of hidden topic variables with the collected sample for solving LDA. LDA provides a powerful method for discovering the hidden thematic structure in large collections of documents. In light of that, our method employs the LDA model for finding incriminating information from spam emails.

Text Visualization Techniques

In the field of information retrieval, there is a classic problem that is how to display and refine search results. The "keyword in context" (KWIC) technique [START_REF] Fischer | The kwic index concept: A retrospective view[END_REF] has been studied by lots of researchers, which is employed to present the keyword enclosed by a part of the paragraph in which it appears. It is difficult to realize which single word in the context helps to the meaning of a paragraph. More details about texts can be represented by keywords with context. It is easy to rapid query bodies of text by applying text visualization techniques [START_REF] Wattenberg | The word tree, an interactive visual concordance[END_REF]. The combination of both above techniques allows viewer to find out the implication of given words in a document. In our solution, a modified KWIC method is proposed to present context flagged by given items. Two ideas contribute to our text visualization. First, each email message flagged with given items is presented. Second, since it is possible that detached terms are useless to forensic practitioners at times, KWIC solution helps to discover valuable clues. Different colors are applied to concentrate given terms in context.

Our method

Our purpose is to discover suspicious messages in the spam folder through text mining techniques. Fig. 2 shows our schematic representation. Firstly, folders named "spam" is α θ z w N M β searched in the directory of email and all messages contained in them are read as input. The second step is to find emails that are regarded as non-spam emails based on features of sending behavior and content. Then URL links are extracted by regular expressions. Thereafter the content of spam email is preprocessed by regular expressions and NLTK suite, such as filtering HTML tags, searching for named entity terms, tokenizing messages into items, removing stopwords from the tokenized documents. Next, a TF-IDF matrix is acquired. In the next step, the LDA topic model is built according to TF-IDF results to identify latent information from the contents of spam emails. Finally, we search for clues from flagged emails.

Dataset description

A set of spam emails is selected at random and incriminating messages are inserted into them, which are two computer-aided encryption messages (target 1 message and target 2 message) and one manual-encryption message (target 3 message), to simulate email examination in a case. Extracting URL links from messages is to address the Link-to message problem. Our dataset, which consists of a total of 100 spam emails, is split into two subsets for detecting computer-aided-encryption messages. Each subset contains one computer-aided-encryption email respectively. Half are used to training and half to test. The manual-encryption message hidden among other spam emails is used to find clues by the KWIC technique. Every message in our dataset is a bona fide spam email, which is from spam archive collected by Bruce Guenter [START_REF]untroubled website[END_REF]. This project still continues to update new spam emails monthly, so we can use the latest spam email data to study.

Fig. 3. Target messages for searching

The original computer-aided-encryption message, which is from one of cases in Yu [START_REF] Yu | Covert communication by means of email spam: A challenge for digital investigation[END_REF], is modified into two ones for our study (see Fig. 3). In that scenario, the criminal delivery messages to their intended recipients. "TGGTCGCCTTTGCTTCGCCTGT" in the subject line implies that the intended recipient can notice this message. The sender's fake email address provides a username "Laysex" and a login address "tinyurl.com / wuincha" In the body, "CGCCAATCCATTCGTTTCGAGGTTACATATTAGCGGGATTTTGTCGTAACC GAG" is an encrypted message that means "hollow soul", which is the password actually. This kind of encrypted message is encoded and decoded according to the format of DNA coding and its meaning is decided by the sequence of four letters, which is A, C, T and G. In our case, we only focus on the body part. These two messages can be considered as two diff erent ones, because we insert two diff erent paragraphs of text from other random spam emails into each message separately, and the encrypted part in target 2 is modified into "CGCCAATCCATTCGTTTCGAGGTTACATATTA" for further camouflage and distraction. In the manual-encryption message, target 3 appears like other spam emails that introduces a value-added service at first glance. In fact, the sender and recipient were brewing terrorist plot, this message contains a date and GPS coordinates to Subject:superb pleasure enchancement TGGTCGCCTTTGCTTCGCCTGT Sender: Laysex@tinyurl.com/wuincha Miracle impr0vent on pennnis size http://xesuwerj.o-f.com/amixine.html pass CGCCAATCCATTCGTTTCGAGGTTACATATTAGCGGGATTTTGTC GTAACCGCG My wife and I have been wanting to go 9328717 here for quite a while now, and so we took advantage of it being our 21st anniversary (yes, we're finally at a year... ) Subject:superb pleasure enchancement TGGTCGCCTTTGCTTCGCCTGT Sender: Laysex@tinyurl.com/wuincha Miracle impr0vent on pennnis size http://xesuwerj.o-f.com/amixine.html pass CGCCAATCCATTCGTTTCGAGGTTACATATTA review Wash got doubt even death the As french name got violet EGGS hands remove lending tail. passing got cream bar I myself im traveling find indicator favorite mess indicator be higher because guide If stirredaccording always fire eyes. onion, ye slantidly stooke quarter substance medium improved chief blue grew other indicator york a despenseme the if spicy that square ask the coloring librarian liquor. confirm. In order to enhance applicability of our study, we insert a URL link into it for further camouflage and distraction. The detail of cases is also introduced in Yu's paper.

Extracting URL links

In this section, we present how effective our method of extracting URL links is. The results are shown in Fig. 4. Because of space limit, part of our results is presented. The last line in Fig. 4 demonstrates the processing time of extracting URL links from 100 spam emails. The result suggests the efficiency of our method. Usually, investigators can visit webpages provided by URL links from the spam emails in a short duration. It turns out that extracting speed is the key to acquire useful leads from websites where inks are located. The result demonstrates that our scheme contributes to search for clues of links-to-messages email.

Identifying clues by LDA

In the second experiment, the first subset is trained to identify latent topic information using LDA described above, and the second is used to discover other messages containing related suspicious topic information. The terms mostly used in expressing topics, which is LDA results, are divided into the six topics as shown in Table 1. The "CGCCAATCCATTCGTTTCGAGGTTACATATTAGCGGGATTTTGTCGTAACC GCG" can be noticed in the topic 4 by experienced investigators at one glance. Furthermore, the LDA topic model provides a clustering of the messages of our dataset by associating them to topics. It is evident form Table 2, where the distribution over topics is listed, that the two target messages are located in the same topic 4. It also helps to find crucial clues in a smaller range.

Text Visualization

Our implementation of text visualization techniques is presented in the third experiment. Fig. 5 shows that multiple valuable named entities terms (keywords, times, numbers) high-light with colors in target 3 message. Investigators may be confused by the results, so we select to flag one kind of terms each time. The date and GPS coordinates information are highlighted in the Fig. 6. Our results suggest that our text visualization solution contributes to discover important clues hidden in target 3 message. It applies to identify suspicious computer-aided-encryption email, manualencryption email and direct-messages email.

Conclusions

In this paper, a unified text mining solution is proposed for discovering suspicious messages hidden in spam emails. The method employs regular expressions for extracting named entities and URL links. Experimental results show that extracting URL links from all the messages is high efficient and completely, and it contributes to identify evidence where is placed in the linked webpage. Our method calculates a TF-IDF matrix for LDA topic model. We can identify computer-aided-encryption messages hidden in spam by LDA topic modeling technique. At last, we discover manual-encryption messages by text visualization technique, and it also applies to discover Direct-message.

In future work, we tend to establish a spam feature database for filtering non-spam emails. We are also interested in refining our method in extracting terms in wider scale. Finally, we plan on testing this method over a larger email spam data set containing more covert communication behavior spam emails. Only when testing wider data set of covert communication spam emails can we fully evaluate our method.

Fig. 1 .

 1 Fig. 1. Graphical model representation of LDA

Fig. 2 .

 2 Fig. 2. Covert communication detection solution

Fig. 4 .

 4 Fig. 4. Results of extracting URL links

Fig. 5 .Fig. 6 .

 56 Fig. 5. Highlighting target message with colors

Table 1 LDA Topics

 1

	Topic1	Topic2	To pic3	Topic4	Topic5	Topic6
	presented	heavy	upMy	herpes	Weston	E2
	babes	Offer	hook	hearing	food	A6
	Welcome	tag	ckFriends	21st	herpes	shrimp
	ready	watches	F	xesuwerj	good	pork
				CGCCAATCCATTCGTTTCGAGGTTACA
	gorgeous	Duplicate	TTYL		beef	strong
				TATTAGCGGGATTTTGTCGTAACCGCG

Table 2

 2 Topics-specific rank

		topic1	Topic2	Topic3	Topic4	Topic5	Topic6
	target	0.0348823878901	0.0348992465435	0.0348847963338	0.825551413019	0.0348953830001	0.034886773214
	1
	target	0.029848100117	0.332788949524	0.029749466089	0.547678235257	0.0299929641262	0.0299422848864
	2