Emmanuel Maggiori
email: emmanuel.maggiori@inria.fr

Guillaume Charpiat

Yuliya Tarabalka

Pierre Alliez

Learning Iterative Processes with Recurrent Neural Networks to Correct Satellite Image Classification Maps

whether they are published or not. The documents may come   L'archive ouverte pluridisciplinaire

Introduction

One of the most explored problems in remote sensing is the pixelwise labeling of satellite imagery. Such a labeling is used in a wide range of practical applications, such as precision agriculture and urban planning. Recent technological developments have substantially increased the availability and resolution of satellite data. Besides the computational complexity issues that arise, these advances are posing new challenges in the processing of the images. Notably, the fact that large surfaces are covered introduces a significant variability in the appearance of the objects. In addition, the fine details in high resolution images make it difficult to classify the pixels from elementary cues. For example, the different parts of an object often contrast more with each other than with other objects [START_REF] Akc | Building detection using directional spatial constraints[END_REF]. Using high-level contextual features thus plays a crucial role at distinguishing object classes.

Convolutional neural networks (CNNs) [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF] are receiving an increasing attention, due to their ability to automatically discover relevant contextual features in image categorization problems. CNNs have already been used in the context of remote sensing [START_REF] Mnih | Machine learning for aerial image labeling[END_REF][START_REF] Vakalopoulou | Building detection in very high resolution multispectral data with deep learning features[END_REF], featuring powerful recognition capabilities. However, when the goal is to label images at the pixel level, the output classification maps are too coarse. For example, buildings are successfully detected but their boundaries in the classification map rarely coincide with the real object boundaries. We can identify two main reasons for this coarseness in the classification: a) There is a structural limitation of CNNs to carry out fine-grained classification. If we wish to keep a low number of learnable parameters, the ability to learn long-range contextual features comes at the cost of losing spatial accuracy, i.e., a trade-off between detection and localization. This is a well-known issue and still a scientific challenge [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF][START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF].

b) In the specific context of remote sensing imagery, there is a significant lack of spatially accurate reference data for training. For example, the OpenStreetMap collaborative database provides large amounts of free-access maps over the earth, but irregular misregistrations and omissions are frequent all over the dataset. In such circumstances, CNNs cannot do better than learning rough estimates of the objects' locations, given that the boundaries are hardly located on real edges in the training set.

Let us remark that in the particular context of highresolution satellite imagery, the spatial precision of the classification maps is of paramount importance. Objects are small and a boundary misplaced by a few pixels significantly hampers the overall classification quality. In other application domains, such as semantic segmentation of natural scenes, while there have been recent efforts to better shape the output objects, a high resolution output seems to be less of a priority. For example, in the popular Pascal VOC semantic segmentation dataset, there is a band of several unlabeled pixels around the objects, where accuracy is The are two recent tendencies to overcome the structural issues that lead to coarse classification maps. One of them is to use new types of CNN architectures, specifically designed for pixel labeling, that seek to address the detection/localization trade-off. For example, Noh et al. [START_REF] Noh | Learning deconvolution network for semantic segmentation[END_REF] duplicate a base classification CNN by attaching a reflected "deconvolution" network, which learns to upsample the coarse classification maps. Another tendency is to use first the base CNN as a rough classifier of the objects' locations, and then process this classification using the original image as guidance, so that the output objects better align to real image edges. For example, Zheng et al. [START_REF] Zheng | Conditional random fields as recurrent neural networks[END_REF] use a fully connected CRF in this manner, and Chen et al. [START_REF] Chen | Semantic image segmentation with task-specific edge detection using cnns and a discriminatively trained domain transform[END_REF] diffuse the classification probabilities with an edge-stopping function based on image features.

The first scheme seems unfeasible in the context of largescale satellite imagery, due to the lack of large amounts of precisely labeled training data. Even if an advanced architecture could eventually learn to conduct a more precise labeling, this is not useful when the training data itself is inaccurate. We thus here adopt the second strategy, reinjecting image information to an enhancement module that sharpens the coarse classification maps around the objects. To train or set the parameters of this enhancement module, as well as to validate the algorithms, we assume we can afford to manually label small amounts of data. In Fig. 1a we show an example of imprecise data to which we have access in large quantities, and in Fig. 1b we show a portion of manually labeled data. In our approach, the first type of data is used to train a large CNN to learn the generalities of the object classes, and the second to tune and validate the algorithm that enhances the coarse classification maps outputted by the CNN.

An algorithm to enhance coarse classification maps would require, on the one hand, to define the image features to which the objects must be attached. This is datadependent, not every image edge being necessarily an object boundary. On the other hand, we must also decide which enhancement algorithm to use, and tune it. Besides the efforts that this requires, we could also imagine that the optimal approach would go beyond the algorithms presented in the literature. For example we could perform different types of corrections on the different classes, based on the type of errors that are often present in each of them.

Our goal is to create a system that learns the appropriate enhancement algorithm itself, instead of designing it by hand. This involves learning not only the relevant features but also the rationale behind the enhancement technique, thus intensively leveraging the power of machine learning.

To achieve this, we first formulate a generic partial differential equation governing a broad family of iterative enhancement algorithms. This generic equation conveys the idea of progressively refining a classification map based on local cues, yet it does not provide the specifics of the algorithm. We then observe that such an equation can be expressed as a combination of common neural network layers, whose learnable parameters define the specific behavior of the algorithm. We then see the whole iterative enhancement process as a recurrent neural network (RNN).

The RNN is provided with a small piece of manually labeled image, and trained end to end to improve coarse classification maps. It automatically discovers relevant datadependent features to enhance the classification as well as the equations that govern every enhancement iteration.

Related work

A common way to tackle the aerial image labeling problem is to use classifiers such as support vector machines [START_REF] Camps-Valls | Kernel-based methods for hyperspectral image classification[END_REF] or neural networks [START_REF] Mas | The application of artificial neural networks to the analysis of remotely sensed data[END_REF] on the individual pixel spectral signatures (i.e., a pixel's "color" but not limited to RGB bands). In some cases, a few neighboring pixels are analyzed jointly to enhance the prediction and enforce the spatial smoothness of the output classification maps [START_REF] Fauvel | Advances in spectral-spatial classification of hyperspectral images[END_REF]. Handdesigned features such as textural features have also been used [START_REF] Lloyd | A comparison of texture measures for the per-field classification of mediterranean land cover[END_REF]. The use of an iterative classification enhancement process on top of hand-designed features has also been explored in the context of image labeling [START_REF] Tu | Auto-context and its application to high-level vision tasks[END_REF].

Following the recent advent of deep learning and to address the new challenges posed by large-scale aerial imagery, Penatti et al. [START_REF] Penatti | Do deep features generalize from everyday objects to remote sensing and aerial scenes domains[END_REF] used CNNs to assign aerial image patches to categories (e.g., 'residential', 'harbor') and Vakalopoulou et al. [START_REF] Vakalopoulou | Building detection in very high resolution multispectral data with deep learning features[END_REF] addressed building detection using CNNs. Mnih [START_REF] Mnih | Machine learning for aerial image labeling[END_REF] and Maggiori et al. [START_REF] Maggiori | Fully convolutional neural networks for remote sensing image classification[END_REF] used CNNs to learn long-range contextual features to produce classification maps. These networks require some degree of downsampling in order to consider large contexts with a reduced number of parameters. They perform well at detecting the presence of objects but do not outline them accurately.

Our work can also be related to the area of natural image semantic segmentation. Notably, fully convolutional networks (FCN) [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] are becoming increasingly popular to conduct pixelwise image labeling. FCN networks are made up of a stack of convolutional and pooling layers followed by so-called deconvolutional layers that upsample the resolution of the classification maps, possibly combining features at different scales. The output classification maps being too coarse, the authors of the Deeplab network [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF] added a fully connected conditional random field (CRF) on top of both the FCN and the input color image, in order to enhance the classification maps.

Zheng et al. [START_REF] Zheng | Conditional random fields as recurrent neural networks[END_REF] recently reformulated the fully connected CRF of Deeplab as an RNN, and Chen et al. [START_REF] Chen | Semantic image segmentation with task-specific edge detection using cnns and a discriminatively trained domain transform[END_REF] designed an RNN that emulates the domain transform filter [START_REF] Gastal | Domain transform for edge-aware image and video processing[END_REF]. Such a filter is used to sharpen the classification maps around image edges, which are themselves detected with a CNN. In these methods the refinement algorithm is designed beforehand and only few parameters that rule the algorithm are learned as part of the network's parameters. The innovating aspect of these approaches is that both steps (coarse classification and enhancement) can be seen as a single end-to-end network and optimized simultaneously.

Instead of predefining the algorithmic details as in previous works, we formulate a general iterative refinement algorithm through an RNN and let the network learn the specific algorithm. To our knowledge, little work has explored the idea of learning an iterative algorithm. In the context of image restoration, the preliminary work by Liu et al. [START_REF] Liu | Learning PDEs for image restoration via optimal control[END_REF][START_REF] Liu | Toward designing intelligent pdes for computer vision: An optimal control approach[END_REF] proposed to optimize the coefficients of a linear combination of predefined terms. Chen et al. [START_REF] Chen | On learning optimized reaction diffusion processes for effective image restoration[END_REF] later modeled this problem as a diffusion process and used an RNN to learn the linear filters involved as well as the coefficients of a parametrized nonlinear function. Our problem is however different, in that we use the image as guidance to update a classification map, and not to restore the image itself. Besides, while we drew inspiration on diffusion processes, we are also interested in imitating other iterative processes like active contours, thus we do not restrict our system to diffusions but consider all PDEs.

Enhancing classification maps with RNNs

Let us assume we are given a set of score (or "heat") maps u k , one for each possible class k ∈ L, in a pixelwise labeling problem. The score of a pixel reflects the likelihood of belonging to a class, according to the classifier's predictions. The final class assigned to every pixel is the one with maximal value u k . Alternatively, a softmax function can be used to interpret the results as probability scores: P (k) = e u k / j∈L e uj . Fig. 2 shows a sample of the type of fuzzy heat map outputted by a CNN in the context of satellite image classification, for the class 'building'.

Our goal is to combine the score maps u k with information derived from the input image (e.g., edge features) to sharpen the scores near the real objects in order to enhance the classification.

One way to perform such a task is to progressively en- hance the score maps by using partial differential equations (PDEs). In this section we first describe different types of PDEs we could certainly imagine to design in order to solve our problem. Instead of discussing which one is the best, we then propose a generic iterative process to enhance the classification maps without specific constraints on the algorithm rationale. Finally, we show how this equation can be expressed and trained as a recurrent neural network (RNN).

Partial differential equations (PDEs)

We can formulate a variety of diffusion processes applied to the maps u k as partial differential equations. For example, the heat flow is described as:

∂u k (x) ∂t = div(∇u k (x)), (1)
where div(•) denotes the divergence operator in the spatial domain of x. Applying such a diffusion process in our context would smooth out the heat maps. Instead, our goal is to design an image-dependent smoothing process that aligns the heat maps to the image features. A natural way of doing this is to modulate the gradient in Eq. 1 by a scalar function g(x, I) that depends on the input image I:

∂u k (x) ∂t = div(g(I, x)∇u k (x)). (2)
Eq. 2 is similar to the Perona-Malik diffusion [START_REF] Perona | Scale-space and edge detection using anisotropic diffusion[END_REF] with the exception that Perona-Malik uses the smoothed function itself to guide the diffusion. g(I, x) denotes an edge-stopping function that takes low values near borders of I(x) in order to slow down the smoothing process there. Another possibility would be to consider a more general variant in which g(I, x) is replaced by a matrix D(I, x), acting as a diffusion tensor that redirects the flow based on image properties instead of just slowing it down near edges:

∂u k (x) ∂t = div(D(I, x)∇u k (x)). (3)
This formulation relates to the so-called anisotropic diffusion process [START_REF] Weickert | Anisotropic diffusion in image processing[END_REF]. Alternatively, one can draw inspiration from the level set framework. For example, the geodesic active contours tech-nique formulated as level sets translates into:

∂u k (x) ∂t = |∇u k (x)|div g(I, x) ∇u k (x) |∇u k (x)| . (4)
Such a formulation favors the zero level set to align with minima of g(I, x) [START_REF] Caselles | Geodesic active contours[END_REF]. Schemes based on Eq. 4 could then be used to improve heat maps u k , provided they are scaled so that segmentation boundaries match zero levels.

As shown above, many different PDE approaches can be devised to enhance classification maps. However, several choices must be made to select the appropriate PDE and tailor it to our problem.

For example, one must choose the edge-stopping function g(I, x) in Eqs. 2, 4. Common choices are exponential or rational functions on the image gradient [START_REF] Perona | Scale-space and edge detection using anisotropic diffusion[END_REF], which in turn requires to set an edge-sensitivity parameter. Extensions to the original Perona-Malik approach could also be considered, such as a popular regularized variant that computes the gradient on a Gaussian-smoothed version of the input image [START_REF] Weickert | Anisotropic diffusion in image processing[END_REF]. In the case of opting for anisotropic diffusion, one must design D(I, x).

Instead of using trial and error to perform such design, our goal is to let a machine learning approach discover by itself a useful iterative process for our task.

A generic classification enhancement process

PDEs are usually discretized in space by using finite differences, which represent derivatives as discrete convolution filters. We build upon this scheme to write a generic discrete formulation of an enhancement iterative process.

Let us consider that we take as input a score map u k (for class k) and, in the most general case, an arbitrary number of feature maps {g 1 , ..., g p } derived from image I. In order to perform differential operations, of the type

{ ∂ ∂x , ∂ ∂y , ∂ 2 ∂x∂y , ∂ 2 ∂x 2 , .
..}, we consider convolution kernels {M 1 , M 2 , ...} and {N j 1 , N j 2 , ...} to be applied to the heat map u k and to the features g j derived from image I, respectively. While we could certainly directly provide a bank of filters M i and N j i in the form of Sobel operators, Laplacian operators, etc., we may simply let the system learn the required filters. We group all the feature maps that result from applying these convolutions, in a single set:

Φ(u k , I) = M i * u k , N j l * g j (I) ; ∀i, j, l . (5)
Let us now define a generic discretized scheme as:

∂u k (x) ∂t = f k Φ(u k , I)(x) , (6)
where f k is a function that takes as input the values of all the features in Φ(u k , I) at an image point x, and combines them. While convolutions M i and N j i convey the "spatial"

reasoning, e.g., gradients, f k captures the combination of these elements, such as the products in Eqs. 2 and 4. Instead of deriving an arbitrary number of possibly complex features N j i * g j (I) from image I, we can think of a simplified scheme in which we directly operate on I, by considering only convolutions: N i * I. The list of functionals considered in Eq. 6 is then

Φ(u k , I) = M i * u k , N j * I ; ∀i, j (7)
and consists only of convolutional kernels directly applied to the heat maps u k and to the image I. From now on, we here stick to this simpler formulation, yet we acknowledge that it might be eventually useful to work on a higher-level representation rather than on the input image itself. Note that if one restricts functions f k in Eq. 6 to be linear, we still obtain the set of all linear PDEs. We consider any function f k , introducing non-linearities. PDEs are usually discretized in time, taking the form:

u k,t+1 (x) = u k,t (x) + δu k,t (x), (8)
where δu k,t denotes the overall update of u k,t at time t. Note that the convolution filters in Eqs. 5 and 7 are classagnostic: M i , N j and N j l do not depend on k, while f k may be a different function for each class k. Function f k thus determines the contribution of each feature to the equation, contemplating the case in which a different evolution might be optimal for each of the classes, even if just in terms of a time-step factor. In the next section we detail a way to learn the update functions δu k,t from training data.

Iterative processes as RNNs

We now show that the generic iterative process can be implemented as an RNN, and thus trained from labeled data. This stage requires to provide the system with a piece of accurately labeled ground truth (see e.g., Fig. 1b).

Let us first show that one iteration, as defined in Eqs. 6-8, can be expressed in terms of common neural network layers. Let us focus on a single pixel for a specific class, simplifying the notation from u k,t (x) to u t . Fig. 3 illustrates the proposed network architecture. Each iteration takes as input the image I and a given heat map u t to enhance at time t. In the first iteration, u t is the initial coarse heat map to be improved, outputted by another pre-trained neural network in our case. From the heat map u t we derive a series of filter responses, which correspond to M i * u t in Eq. 7. These responses are found by computing the dot product between a set of filters M i and the values of u k,t (•) in a spatial neighborhood of a given point. Analogously, a set of filter responses are computed at the same spatial location on the input image, corresponding to the different N j * I of Eq. 7. These operations are convolutions when performed Conv.

MLP

Concat. densely in space, N j * I and M i * u t being feature maps of the filter responses. These filters are then "concatenated", forming a pool of features Φ coming from both the input image and the heat map, as in Eq. 7, and inputted to f k in Eq. 6. We must now learn the function δu t that describes how the heat map u t is updated at iteration t (cf. Eq. 8), based on these features.

N j * I M i * u t u t u t +1 δ u t
Eq. 6 does not introduce specifics about function f k . In ( 1)-( 4), for example, it includes products between different terms, but we could certainly imagine other functions. We therefore model δu t through a multilayer perceptron (MLP), because it can approximate any function within a bounded error. We include one hidden layer with nonlinear activation functions followed by an output neuron with a linear activation (a typical configuration for regression problems), although other MLP architectures could be used. Applying this MLP densely is equivalent to performing convolutions with 1 × 1 kernels at every layer. The implementation to densely label entire images is then straightforward.

The value of δu t is then added to u t in order to generate the updated map u t+1 . This addition is performed pixel by pixel in the case of a dense input. Note that although we could have removed this addition and let the MLP directly output the updated map u t+1 , we opted for this architecture since it is more closely related to the equations and better conveys the intention of a progressive refinement of the classification map. Moreover, learning δu t instead of u t+1 has a significant advantage at training time: a random initialization of the networks' parameters centered around zero means that the initial RNN represents an iterative process close to the identity (with some noise). Training uses the asymmetry induced by this noise to progressively move from the identity to a more useful iterative process.

The overall iterative process is implemented by unrolling a finite number of iterations, as illustrated in Fig. 4, under the constraint that the parameters are shared among all iterations. Such sharing is enforced at training time by a simple modification to the back-propagation training algorithm where the derivatives of every instance of a weight at dif-ferent iterations are averaged [START_REF] Werbos | Backpropagation through time: what it does and how to do it[END_REF]. Note that issues with vanishing or exploding gradients may arise when too many iterations are unrolled, an issue inherent to deep network architectures. Note also that the spatial features are shared across the classes, while a different MLP is learned for each of them, following Eq. 6. As depicted by Fig. 4 and conveyed in the equations, the features extracted from the input image are independent of the iteration.

The RNN of Fig. 4 represents then a dynamical system that iteratively improves the class heat maps. Training such an RNN amounts to finding the optimal dynamical system for our enhancement task.

Implementation details

We first describe the CNN used to produce the coarse predictions, then detail our RNN. The network architecture was implemented using Caffe deep learning library [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF].

Our coarse prediction CNN is based on a previous remote sensing network presented by Mnih [START_REF] Mnih | Machine learning for aerial image labeling[END_REF]. We create a fully convolutional [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] version of Mnih's network, since recent remote sensing work has shown the theoretical and practical advantages of this type of architecture [START_REF] Kampffmeyer | Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks[END_REF][START_REF] Maggiori | Fully convolutional neural networks for remote sensing image classification[END_REF]. The CNN takes 3-band color image patches at 1m 2 resolution and produces as many heat maps as classes considered. The resulting four-layer architecture is as follows: 64 conv. filters (12 × 12, stride 4) → 128 conv. filters (3 × 3) → 128 conv. filters (3 × 3) → 3 conv. filters (9 × 9). Since the first convolution is performed with a stride of 4, the resulting feature maps have a quarter of the input resolution. Therefore, a deconvolutional layer [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] is added on top to upsample the classification maps to the original resolution. The activation functions used in the hidden layers are rectified linear units. This network is trained on patches randomly selected from the training dataset. We group 64 patches with classification maps of size 64 × 64 into minibatches (following [START_REF] Mnih | Machine learning for aerial image labeling[END_REF]) to estimate the gradient of the network's parameters and back-propagate them. Our loss function is the cross-entropy between the target and predicted class probabilities. Stochastic gradient descent is used for optimization, with learning rate 0.01, momentum 0.9 and an L2 weight regularization of 0.0002. We did not however optimized these parameters nor the networks' architectures.

. . . + Image ... + ... ... N j * I u t =0 u t =1 u t =2 u t =3
We now detail the implementation of the RNN described in Sec. 2.3. We unroll five RNN iterations and learn 32 M i and 32 N j filters, both of spatial dimensions 5 × 5. As explained in Sec. 2.3, an independent MLP is learned for every class, using 32 hidden neurons each and with rectified linear activations. Training is performed on random patches and with the cross-entropy loss function, as done with the coarse CNN. The employed gradient descent algorithm is AdaGrad [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF], which exhibits a faster convergence in our case, using a base learning rate of 0.01 (higher values make the loss diverge). All weights are initialized randomly by sampling from a distribution that depends on the number of neuron inputs [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]. We trained the RNN for 50,000 iterations, until observing convergence of the training loss, which took around four hours on a single GPU.

Experiments

We perform our experiments on images acquired by a Pléiades satellite over the area of Forez, France. An RGB color image is used, obtained by pansharpening [START_REF] Wang | A comparative analysis of image fusion methods[END_REF] the satellite data, which provides a spatial resolution of 0.5 m 2 . Since the networks described in Sec. 3 are specifically designed for images with a 1m 2 resolution, we downsample the Pléiades images before feeding them to our networks and bilinearly upsample the outputs.

From this image we selected an area with Open-StreetMap (OSM) [START_REF] Haklay | Openstreetmap: User-generated street maps[END_REF] coverage to create a 22.5 km 2 training dataset for the classes building, road and background. The reference data was obtained by rasterizing the raw OSM maps. Misregistrations and omissions are present all over the dataset (see Fig. 1a and further examples in suppl. material). Buildings tend to be misaligned or omitted, while many roads in the ground truth are not visible in the image (or the other way around). This is the dataset used to train the initial coarse CNNs.

We manually labeled two 2.25 km 2 tiles to train and test the RNN at enhancing the predictions of the coarse network. We denote them by enhancement and test sets, respectively. Note that our RNN system must discover an algorithm to refine an existing classification map, and not to conduct the classification itself, hence a smaller training set should be sufficient for this stage.

In the following, we report the results obtained by using the proposed method on the Pléiades dataset. Fig. 5 provides closeups of results on different fragments of the test dataset. The initial and final maps (before and after the RNN enhancement) are depicted, as well as the intermediate results through the RNN iterations. We show both a set of final classification maps and some single-class fuzzy probability maps. We can observe that as the RNN iterations go by, the classification maps are refined and the objects better align to image edges. The fuzzy probabilities become more confident, sharpening object boundaries. To quantitatively assess this improvement we compute two measures on the test set: the overall accuracy (proportion of correctly classified pixels) and the intersection over union (IoU) [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF]. Mean IoU has become the standard in semantic segmentation since it is more reliable in the presence of imbalanced classes (such as background class, which is included to compute the mean) [START_REF] Csurka | What is a good evaluation measure for semantic segmentation?[END_REF]. As summarized in the table of Fig. 6(a), the performance of the original coarse CNN (denoted by CNN) is significantly improved by attaching our RNN (CNN+RNN). Both measures increase monotonously along the intermediate RNN iterations, as depicted in Fig. 6(b).

The initial classification of roads has an overlap of less than 10% with the roads in the ground truth, as shown by its individual IoU. The RNN makes them emerge from the background class, now overlapping the ground truth roads by over 50%. Buildings also become better aligned to the real boundaries, going from less than 40% to over 70% overlap with the ground truth buildings. This constitutes a multiplication of the IoU by a factor of 5 for roads and 2 for buildings, which indicates a significant improvement at outlining and not just detecting objects.

Additional visual fragments before and after the RNN refinement are shown in Fig. 7. We can observe in the last row how the iterative process learned by the RNN both thickens and narrows the roads depending on the location.

We also compare our RNN to the approach in [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF] (here denoted by CNN+CRF), where a fully-connected CRF is coupled both to the input image and the coarse CNN output, in order to refine the predictions. This is the idea behind the so-called Deeplab network, which constitutes one of the most important current baselines in the semantic seg- mentation community. While the CRF itself could also be implemented as an RNN [START_REF] Zheng | Conditional random fields as recurrent neural networks[END_REF], we here stick to the original formulation because the CRF as RNN idea is only interesting if we want to train the system end to end (i.e., together with the coarse prediction network). In our case we wish to leave the coarse network as is, otherwise we risk overfitting it to this much smaller set. We thus simply use the CRF as in [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF] and tune the energy parameters by performing a grid search using the enhancement set as a reference. Five iterations of inference on the fully-connected CRF were preformed in every case.

To further analyze our method, we also consider an alternative enhancement RNN in which the weights of the MLP are shared across the different classes (which we denote by CNN+RNN = ). This forces the system to learn the same function to update all the classes, instead of a classspecific function.

Numerical results are included in the table of Fig. 6(a) and visual fragments are compared in Fig. 8.

The CNN+CRF approach does sharpen the maps but this often occurs around the wrong edges. It also makes small objects disappear in favor of larger objects (usually the background class) when edges are not well marked, which explains the mild increase in overall accuracy but the decrease in mean IoU. While the CNN+RNN = outperforms the CRF, both quantitative and visual results are beaten by the CNN+RNN, supporting the importance of learning a class-specific enhancement function. The suppl. material includes the visualization of different filters learned by the RNN and visual results over a larger surface.

To validate the importance of using a recurrent architecture, and following Zheng et al. [START_REF] Zheng | Conditional random fields as recurrent neural networks[END_REF], we retrained our system considering every iteration of the RNN as an independent step with its own parameters. After training for the same number of iterations, it yields a lower performance on the test set compared to the RNN and a higher performance on the training set. If we keep on training, the nonrecurrent network still enhances its training accuracy while performing poorly on the test set, implying a significant degree of overfitting with this variant of the architecture. This provides evidence that constraining our network to learn an iterative enhancement process is crucial for its success.

Concluding remarks

In this work we presented an RNN that learns how to refine the coarse output of another neural network, in the context of pixelwise image labeling. The inputs are both the coarse classification maps to be corrected and the original color image. The output at every RNN iteration is an update to the classification map of the previous iteration, using the color image for guidance.

Little human intervention is required, since the specifics of the refinement algorithm are not provided by the user but learned by the network itself. For this, we analyzed different iterative alternatives and devised a general formulation that can be interpreted as a stack of common neuron layers. At training time, the RNN discovers the relevant features to be taken both from the classification map and from the input image, as well as the function that combines them.

The experiments on satellite imagery show that the classification maps are improved significantly, increasing the overlap of the foreground classes with the ground truth, and outperforming other approaches by a large margin. Thus, the proposed method not only detects but also outlines the objects. To conclude, we demonstrated that RNNs succeed in learning iterative processes for classification enhancement tasks.

Figure 1 :

 1 Figure 1: Samples of reference data for the building class. Imprecise OpenStreetMap data vs manually labeled data.

Figure 2 :

 2 Figure 2: Sample classification of buildings with a CNN.

Figure 3 :

 3 Figure 3: One enhancement iteration represented as common neural network layers.

Figure 4 :

 4 Figure 4: Modules of Fig. 3 are stacked (while sharing parameters) to implement an RNN.

Figure 5 :

 5 Figure 5: Evolution of fragments of classification maps (top rows) and single-class fuzzy scores (bottom rows) through RNN iterations.

Figure 6 :Figure 7 :Figure 8 :

 678 Figure 6: Quantitative evaluation on Pléaiades images test set over Forez, France.

Acknowledgment

All Pléiades images are c CNES (2012 and 2013), distribution Airbus DS / SpotImage. The authors would like to thank the CNES for initializing and funding the study, and providing Pléiades data.


Quang-Vinh Dang
email: quang-vinh.dang@inria.fr

Claudia-Lavinia Ignat

Measuring Quality of Collaboratively Edited Documents: the case of Wikipedia

Wikipedia is a great example of large scale collaboration, where people from all over the world together build the largest and maybe the most important human knowledge repository in the history. However, a number of studies showed that the quality of Wikipedia articles is not equally distributed. While many articles are of good quality, many others need to be improved. Assessing the quality of Wikipedia articles is very important for guiding readers towards articles of high quality and suggesting authors and reviewers which articles need to be improved. Due to the huge size of Wikipedia, an effective automatic assessment method to measure Wikipedia articles quality is needed.

In this paper, we present an automatic assessment method of Wikipedia articles quality by analyzing their content in terms of their format features and readability scores. Our results show improvements both in terms of accuracy and information gain compared with other existing approaches.

I. INTRODUCTION

Today Wikipedia is the largest and most common reference source on the Internet. It contains around 40 million articles, among which more than five million articles belong to English Wikipedia. According to Wikimedia Foundation, Wikipedia in general and English Wikipedia in particular get about 14 billion and seven billion page views per month, respectively 1 . A study [START_REF]Wikipedia: Page one of google uk for 99% of searches[END_REF] investigated 1,000 randomly generated search terms in Google and measured the rankings for Wikipedia.org. This study found that Wikipedia is ranked in the first five positions by Google for 96% of queries. This leads to higher probability for Internet users to check the content of Wikipedia [START_REF]The value of google result positioning[END_REF]. Consequently, it is very important to provide high quality Wikipedia articles.

However, concerns about the quality of Wikipedia have been raised [START_REF] Denning | Wikipedia risks[END_REF], [START_REF] Simonite | The decline of wikipedia[END_REF]. For instance, the information presented on Wikipedia is not accepted as a reliable source for research by many professors and researchers [START_REF] Dooley | Wikipedia and the two-faced professoriate[END_REF]. The main problem is that, while many articles are of high quality, many others did not receive the desired attention from authors to improve their quality [START_REF] Lam | Is wikipedia growing a longer tail?[END_REF], [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF].

In order to improve quality of Wikipedia pages, several collaborative projects, such as Collaboration of the Week (CotW), WikiCup and Wikipedia Education Program (WEP), were organized. The success and failure of these projects are discussed in [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF]. Wikipedia development team also implemented different kinds of bots 2 to execute several automatic tasks, such as checking if a submitted revision damages a particular Wikipedia page or not [START_REF] Segall | The illiterate editor: metadata-driven revert detection in wikipedia[END_REF]. These bots were proved to be efficient in preventing flaws in Wikipedia articles [START_REF] Halfaker | Don't bite the newbies: how reverts affect the quantity and quality of wikipedia work[END_REF]. Nonetheless, measuring the quality of Wikipedia articles is more difficult. A text that does not contain harmful content is valid but might not be of a good quality.

Quality assessment on Wikipedia is being performed by human judgement, based on a small group of experts. In order to assess the quality of a Wikipedia page, several reviewers have to read, review and discuss what quality label should be assigned to this particular page. The process indeed requires a lot of time and effort. Moreover, the process needs to be repeated if the particular page is updated. Currently, the average number of edits per second that are performed on Wikipedia3 is ten. A manual quality assessment method may not scale well for this editing frequency [START_REF] Hu | Measuring article quality in wikipedia: models and evaluation[END_REF], [START_REF] De La Robertie | Measuring article quality in wikipedia using the collaboration network[END_REF]. An automatic approach for assessing quality is therefore required to support collaboration on Wikipedia. This automatic approach would provide an immediate guidance for readers and search engines to choose high-quality articles, and an immediate feedback for writers and reviewers to have a plan for quality improvement.

In this paper we address the challenge of automatically rating the quality of a Wikipedia articles. We use the following quality class labels defined by Wikipedia ordered from low to high quality: Stub, Start, C, B, GA, FA [START_REF] Hu | Measuring article quality in wikipedia: models and evaluation[END_REF], [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF] 4 . The description of these quality classes is provided in Table I 5 . Similar to [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF], [START_REF] Wu | Classifying wikipedia articles using network motif counts and ratios[END_REF], [START_REF] Warncke-Wang | Tell me more: an actionable quality model for wikipedia[END_REF] we removed the quality labels A and Bplus from our analysis as the number of articles belonging to these categories is very small, even less than the number of FA articles.

Several research works were proposed for classifying the quality of Wikipedia articles using machine learning algorithms such as random forest [START_REF] Warncke-Wang | Tell me more: an actionable quality model for wikipedia[END_REF], [START_REF] Wu | Classifying wikipedia articles using network motif counts and ratios[END_REF]. These approaches

Professional, outstanding, and thorough; a definitive source for encyclopedic information.

GA

Useful to nearly all readers, with no obvious problems; approaching (but not equalling) the quality of a professional encyclopedia.

B

Readers are not left wanting, although the content may not be complete enough to satisfy a serious student or researcher.

C

Useful to a casual reader, but would not provide a complete picture for even a moderately detailed study.

Start

Provides some meaningful content, but most readers will need more.

Stub

Provides very little meaningful content; may be little more than a dictionary definition. Readers probably see insufficiently developed features of the topic and may not see how the features of the topic are significant. [START_REF] Warncke-Wang | Tell me more: an actionable quality model for wikipedia[END_REF] and using same machine learning techniques, we present a new classification model to predict the quality of Wikipedia articles. In addition to the features presented in [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF] that refer uniquely to the structure of an article, we introduced contentbased features of the text such as readability scores. Our model achieves a higher accuracy and information gain compared with other approaches. We showed that our added features play an important role in the performance of the classifier. The paper is organized as follows. Section II discusses related approaches on measuring quality of Wikipedia articles. Section III presents the set of features of Wikipedia articles that we selected for our model. Section IV presents the data set and the accuracies obtained by the classification algorithms that we used in combination with the proposed feature set. Evaluation of our model using the proposed feature set and the random forest approach are discussed in Section V. Conclusions and directions for future work are presented in Section VI.

II. RELATED WORK

Due to the importance of Wikipedia, many approaches on classifying the quality of Wikipedia articles were proposed. These approaches can be divided into two main categories [START_REF] Warncke-Wang | Tell me more: an actionable quality model for wikipedia[END_REF]: using editor-based information such as about article authors, or using article-based information such as about the content or the format of the articles themselves.

A. Using editor-based information

Approaches that used editor-based information analysed information that cannot be computed uniquely from the current content of Wikipedia pages, such as the authors of a particular article, their contributions and the duration of each contribution.

Using the hypothesis that the more reputable an author is, the higher the quality of the articles this author produces, Adler et al. [START_REF] Adler | Assigning trust to wikipedia content[END_REF] and Javanmardi and Lopes [START_REF] Javanmardi | Statistical measure of quality in wikipedia[END_REF] used reputation of authors to determine the quality of Wikipedia articles. The result was confirmed in German Wikipedia [START_REF] Stein | Does it matter who contributes: a study on featured articles in the german wikipedia[END_REF]. The social capital of the editors could also affect the quality of the articles they contributed [START_REF] Nemoto | Social capital increases efficiency of collaboration among wikipedia editors[END_REF]. Recently, Suzuki applied the idea of using h-index on academic ranking for assessing the quality of an article [START_REF] Suzuki | Quality assessment of wikipedia articles using h-index[END_REF].

Another criterion used for assessing the quality of a text is the period of time the text remains stable or is modified by other authors/reviewers. If an article has not been modified significantly for a long time, this article can be considered as mature and of high quality. For instance, Calzada and Dekhtyar [START_REF] Calzada | On measuring the quality of wikipedia articles[END_REF] used the idea of stable article to determine the quality of Wikipedia articles. Wohner and Peters [START_REF] Wöhner | Assessing the quality of wikipedia articles with lifecycle based metrics[END_REF] also claimed that a good article should not be modified for a long enough period of time. Biancani [START_REF] Biancani | Measuring the quality of edits to wikipedia[END_REF] showed that there is a strong relationship between the number of words that were not modified for a long period of time and the quality of Wikipedia articles.

Some other research works presented the idea that the quality of Wikipedia articles can be determined based on the interaction between authors and reviewers [START_REF] De La Robertie | Measuring article quality in wikipedia using the collaboration network[END_REF], [START_REF] Hu | Measuring article quality in wikipedia: models and evaluation[END_REF], [START_REF] Wu | Classifying wikipedia articles using network motif counts and ratios[END_REF]. For instance, Wilkinson and Huberman [START_REF] Wilkinson | Cooperation and quality in wikipedia[END_REF] showed that a large number of authors and reviewers with an intensive cooperation should lead to high quality articles. Arazy and Nov [START_REF] Arazy | Determinants of wikipedia quality: the roles of global and local contribution inequality[END_REF] showed that inequality of editors' local contribution on a particular article, inequality of their global contribution on overall Wikipedia activity levels as well as their coordination affect document quality. Liu and Ram [START_REF] Liu | Who does what: Collaboration patterns in the wikipedia and their impact on article quality[END_REF] classified editors based on their roles in editing individual Wikipedia articles and identified collaboration patterns among these contributors that are preferable or detrimental for article quality. Li et al. [START_REF] Li | Automatically assessing wikipedia article quality by exploiting article-editor networks[END_REF] analysed the article-editor network to assess quality of Wikipedia articles. Ruvo and Santone [START_REF] Ruvo | Analysing wiki quality using probabilistic model checking[END_REF] analysed the network of articles in private enterprise wiki systems in order to assess their quality.

B. Using article-based information

The second main approach of assessing quality of Wikipedia articles is to analyse directly the content of Wikipedia articles.

One of the simplest solutions is to measure the length of Wikipedia articles [START_REF] Blumenstock | Size matters: word count as a measure of quality on wikipedia[END_REF]. This solution achieved a very high accuracy in separating between FA and non-FA articles. Other works considered the writing styles, such as how editors vary the words they used, for assessing articles quality [START_REF] Lipka | Identifying featured articles in wikipedia: writing style matters[END_REF], [START_REF] Xu | Measuring article quality in wikipedia: Lexical clue model[END_REF].

Dalip et al. [START_REF] Dalip | Automatic quality assessment of content created collaboratively by web communities: a case study of wikipedia[END_REF] analyzed the effect of the feature set comprising text, review and network on the quality of Wikipedia articles. A correlation between this feature set and the quality of Wikipedia articles was performed. Authors claimed that, using the error term of linear regression, the features that describe the structure and style of the articles are the best to distinguish between articles of different quality classes.

Similarly, using content, structure, network and edit history features, Anderka et al. [START_REF] Anderka | Predicting quality flaws in usergenerated content: the case of wikipedia[END_REF] built a binary classifier to predict quality flaws in Wikipedia. They based their approach on the cleanup tags, which are given by the reviewers who detected the flaws but do not have enough time / expertise to fix them.

Focusing on the feature set that describes the content of Wikipedia articles, Warncke-Wang et al. [START_REF] Warncke-Wang | Tell me more: an actionable quality model for wikipedia[END_REF] presented and analyzed a feature set including 17 features. Authors claimed that among these features only a set of 11 should be considered to evaluate the quality of Wikipedia articles. The result is improved in [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF].

Based on the work of [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF] and [START_REF] Warncke-Wang | Tell me more: an actionable quality model for wikipedia[END_REF], Wikimedia Foundation 6 built an online API to predict the quality class of Wikipedia articles called ORES (Objective Revision Evaluation Service) [START_REF] Halfaker | Artificial intelligence service gives wikipedians 'x-ray specs' to see through bad edits[END_REF].

Editor-based approaches are characterised by a high time complexity as they require processing the whole history associated to an article. Moreover, editor-based approaches are indirect predicting methods that rather than considering content information they take into account authors and reviewers related information. For instance, it is not necessary that good authors always write good articles.

We applied the article-based approach, which is faster than the editor-based approach as it uniquely requires processing the current document content. In addition, article-based approaches are direct predicting methods where the quality of a Wikipedia article is determined by its current content, not by its edit history.

We extended the model presented in [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF] by adding readability scores to the feature set. Our hypothesis is that the quality of an article depends not only on the structure of an article, but also on how well the article is written. The experiments showed that using readability scores as a part of the feature set can improve the performance of the predicting model. Although some readability scores have been previously studied in [START_REF] Warncke-Wang | Tell me more: an actionable quality model for wikipedia[END_REF], [START_REF] Dalip | Automatic quality assessment of content created collaboratively by web communities: a case study of wikipedia[END_REF], [START_REF] Stvilia | Assessing information quality of a community-based encyclopedia[END_REF], they were used in the context of different techniques for measurement of the effect of readability scores on Wikipedia articles quality. Moreover, we propose using some readability scores such as difficult words and Dale-Chall readability score that have not been investigated by any other study. Later in this paper we show that the proposed readability scores as well as the chosen classification algorithm play a critical role in the performance of the predicting model.

III. FEATURE SELECTION

In this section we present the features included in our model for assessing the quality of Wikipedia articles. Our hypothesis is that the writing style matters for measuring the articles quality.

We based our model on the one presented in [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF]. In addition to the features presented in [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF] related to the structure of an article (e.g, does the article has infobox or not, or how many references the article has), we added content-based features of the text. The complete set of features for our model is presented in what follows. 6 Structure-based features of our model refer to the structure of the document and they are the same as those proposed in [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF]. These features are listed below, where the terms inside parentheses represent the variable names used in our model.

B. Content-based features

We added to the model the following content-based features. The variables used in the computation of the content-based features are explained in Table II.

1) Flesch reading score (flesch reading ease): Flesch reading score, or Flesch reading ease [START_REF] Kincaid | Derivation of new readability formulas for navy enlisted personnel[END_REF], is a measure to test how difficult a reading text in English is to understand. Flesch reading ease for a given text is a number between 100 and 0, where higher scores indicate text that is easier to read while lower numbers mark text that is more difficult to read. flesch reading ease = 206.835 -(1.015 × avg sentence len) -(84.6 × avg syllables per word )

2) Flesch-Kincaid grade level (flesch kincaid grade): Flesch-Kincaid grade level [START_REF] Kincaid | Derivation of new readability formulas for navy enlisted personnel[END_REF] for a given English text is a number corresponding to the US grade level required to understand the text. For example, if the score is 9.3, it means that the reader of the text should be ninth grader or higher. Although Flesch reading ease and Flesch-Kincaid grade level use both word length and sentence length as core measures, they have different weighting factors. These measures are inversely correlated: a text with a high score on the reading ease test should have a low score on the grade-level test.

flesch kincaid grade = 11.8 × avg syllables per word + 0.39 × avg sentence len -15.59

(2)
3) Smog index (smog index ): Smog index [START_REF] Mclaughlin | Smog grading: A new readability formula[END_REF] of a text estimates the years of education a person needs to understand a given text in English.

(3)

smog index = 3 + polysyllable count
The polysyllable count is defined as the number of words with more than two syllables. Automated readability index (ARI) [START_REF] Senter | Automated readability index[END_REF] is another readability score to detect the readability of a given text in English in terms of the US grade level similar to Flesch-Kincaid grade and Coleman -Liau index. ARI and Coleman-Liau index rely on a factor of characters per word, instead of syllables per word as the other listed measures. automated readability index = 4.71 × avg word len + 0.5 × avg sentence len -21.43

(5)

6) Difficult words (difficult words):

The difficult words score [START_REF] Chall | Readability revisited: The new Dale-Chall readability formula[END_REF] of a given English text is calculated based on how many difficult words appear in a text. A word is considered difficult if it does not appear in a list of 3000 common English words that groups of fourth-grade American students could reliably understand.

7) Dale-Chall score (dale chall readability score): Dale-Chall readability score [START_REF] Dale | A formula for predicting readability: Instructions[END_REF] is another measure for comprehension difficulty when reading a text. This score takes into account the percentage of difficult words in the text as well as the ratio between the number of words and the number of sentences. [START_REF] Lam | Is wikipedia growing a longer tail?[END_REF] dale chall readability score = 0.1579 × percentage of difficult words + 0.0496 × avg sentence len 8) Linsear write formula (linsear write formula): Linsear Write Formula is a readability score initially designed for the United States Air Force to compute the readability of their technical manuals [START_REF] Chen | How to use readability formulas to access and select english reading materials[END_REF]. This score corresponds to the US grade level of a text sample based on sentence length and the number of words used that have three or more syllables.

More precisely, based on a sample of 100 words from the text, where the number of words with two syllables or less is denoted by n 1 and the number of words with three syllables or more by n 2 , Linsear Write Formula is calculated as n1+3×n2 number of sentences×2 if n1+3×n2 number of sentences > 20 and as n1+3×n2 number of sentences×2 -1 in other cases. 9) Gunning-Fog index (gunning fog): Gunning-Fog index [START_REF] Gunning | The fog index after twenty years[END_REF] is another readability score to measure the difficulty of a given text in terms of the years of formal education needed to understand the text on a first reading. It is a weighted average of the number of words per sentence, and the number of long words per word. [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF] gunning fog = 0.4 × (avg sentence len + percentage of difficult words)

Our proposed model comprises the above readability scores in addition to the 11 features from the original model of [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF]. Due to the nature of the above readability scores, we only can apply our model to English Wikipedia.

Several readability scores seem related but this is not a problem for the classification method we chose, i.e. the random forest algorithm, as it can cope with multi-collinearity [START_REF] Matsuki | The random forests statistical technique: An examination of its value for the study of reading[END_REF]. Indeed, also other approaches on classification of Wikipedia articles according to their quality used a set of related features. For instance, ORES method used a feature set including the length of the article and the number of images, but also the division of number of images by length, which is derived from the first two features. Furthermore, relationship between features does not necessarily lead to collinearity. For instance, the correlation between dale chall readability score and difficult words in our data set is only -0.4.

IV. PREDICTING MODELS

In this section, we present the data set we used in our experiments and the performances obtained by different classification techniques applied to the set of features presented in the previous section.

A. Data set

We used a set of Wikipedia articles generated through several quality improvement projects ran by Wikipedia as mentioned in Section I. The data set was provided by the authors of [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF]

B. Data preparation

The data preprocessing program is written in Python. In order to retrieve the content of a Wikipedia page, we used Wikipedia API 8 . We used the open-source project wikiclass9 to compute structure-based features. The open-source project textstat10 was used to compute content-based features. We collected the content of Wikipedia pages in the data set corresponding to the revision for which the quality labels were assigned.

We applied different classification methods with 5-fold cross-validation techniques to compare the performance of these algorithms on evaluating the quality of Wikipedia articles. 5-fold cross validation is considered as a good practical technique for bias-variance trade-off in evaluating machine learning algorithms [START_REF] James | An Introduction to Statistical Learning: with Applications in R[END_REF]. In 5-fold cross validation the entire data set (20,489 articles) is divided into five equal parts (5fold): four parts are used as a training set and the remaining part as the testing set. This process is repeated five times, each part being used as a testing set alternately.

C. Classification using regression model

In this subsection we present our model using a multiple regression approach [START_REF] Wilkinson | Symbolic description of factorial models for analysis of variance[END_REF]. Our dependent variable is the quality class, and the independent variables are the features described above. The target of a multiple regression model is to build a linear function of the dependent variable on other independent variables which best fits to the training data and then use this function to predict the unknown data.

As regression models can be applied only for integer-based values, we converted the quality class to an integer: Stub to 0, Start to 1, C to 2, B to 3, GA to 4 and FA to 6. After using the regression model for predicting the quality class of test data set, we converted back the quality level by rounding.

We achieved an accuracy of 25%, which is not a surprising result as the linear regression is not expected to perform well in classification.

D. Classification using multinomial logistic regression

Multinomial logistic regression [START_REF] Böhning | Multinomial logistic regression algorithm[END_REF] is the extended version of logistic regression to classify the data set with more than two possible outputs. Multinomial logistic regression does not require that the dependent variables are continuous.

We achieved an accuracy of 60% on our data set with 5-fold cross-validation.

E. Classification using kNN

k-Nearest Neighbors (kNN) algorithm [START_REF] Altman | An introduction to kernel and nearest-neighbor nonparametric regression[END_REF] has been widely used in classification problems. The principle of kNN algorithm is to determine a class of an element (in our case an article) by a majority vote of its neighbors, with the element being assigned to the most common class among its k nearest neighbors. For instance, with k = 3, in order to classify the quality label of an article in the testing data set, first we find 3 nearest articles of this test article in the training data set. If among the 3 nearest neighbors of an article, 2 of them are assigned the quality class FA and 1 of them GA, the article will be predicted as FA. The distance between articles is calculated by using an Euclidean metric in n-dimension space, where n is the number of features.

In order to apply kNN algorithm for classification of our Wikipedia articles according to their quality, firstly we converted the variable has infobox to a numeric value, namely value 0 is assigned to articles without information box, and value 1 to the articles with information box.

The accuracy of cross-validation with kNN is 55%.

F. Classification using CART

In this subsection we present the model using classification and regression tree (CART) [START_REF] Breiman | Classification and Regression Trees[END_REF]. The idea of CART is to build a series of if -else decision points to classify the data set, with the goal to minimize the entropy of the training set. A simple example of CART is shown in Figure 1. In this example for instance, to classify the quality label of a particular article, first of all we consider the length of the article: if the content length < 1, 000, we classify this article as Stub, otherwise we consider the variable num references. If num references ≥ 25, we consider the variable difficult words, and if difficult words ≥ 892 we classify the article as FA, otherwise we classify the article as GA.

Fig. 1: An example of CART Using cross -validation with optimized CART, we achieved the accuracy of 48% on our data set.

G. Classification using Support Vector Machine

Support Vector Machine (SVM) is a supervised machine learning algorithm, which was designed for classification [START_REF] Cortes | Support vector machine[END_REF]. The idea of Support Vector Machine is to build a hyper plane to classify the data in training set, and then use this hyper plane to predict the quality labels of articles in the testing set.

SVM-based solution achieved an accuracy of 61% on 5-fold cross-validation.

H. Classification using random forest model without readability scores

We tested the performance of the algorithm presented by Warncke et al. [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF] based on the source code provided by the authors. In [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF], the random forest classification [START_REF] Ho | Random decision forests[END_REF] was applied uniquely on the structure-based feature set. Random forest is an improved model of CART where multiple CART are built and used to vote for the output class of a new article. We note that in [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF] the content-based features including the readability scores were not used by the random forest model.

By using a 5-fold cross-validation technique on the data set, the algorithm of [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF] achieved an accuracy of 58%.

I. Classification using random forest model with the complete set of features

In this subsection, we present the prediction model with random forest with the complete set of features including all structure-based and content-based features. We used a 5-fold cross validation to obtain the optimized parameters for the random forest model, and found that the optimized parameter set is 450 trees with a node size of 4.

This model achieved an accuracy of 64% with 5-fold cross validation on our data set, which is the highest accuracy compared with other existing approaches.

V. EVALUATION

In this section we present the performances of our model using the random forest approach. As shown in the previous section, random forest approach provides the best accuracy for our model. We compare performances of our model with other existing approaches by using the following three popular metrics: accuracy, AUC and NDCG. We used NDCG score as research approaches that aim ranking articles according to their quality rather than classifying them into quality classes, do not report on accuracy scores.

A. Accuracy & AUC

As our data set is relatively balanced, accuracy is a useful metric to measure classifier performance.

The accuracy score is calculated as:

(8) accuracy = correct prediction number total prediction number Approaches concerned with the classification of quality of Wikipedia articles can be divided according to their selected classes for the classification. Some of them distinguish between FA and Start articles, others between FA-GA and C-Start articles, and others between all classes.

1) Binary classification: As a fundamental classification, Xu et al [START_REF] Xu | Measuring article quality in wikipedia: Lexical clue model[END_REF] classified between two article classes FA and Start, and they achieved an accuracy of 84%. Lex et al [START_REF] Lex | Measuring the quality of web content using factual information[END_REF] classified the set FA-GA classes versus all other classes, and achieved an accuracy of 84%. Wu et al [START_REF] Wu | Classifying wikipedia articles using network motif counts and ratios[END_REF] presented two classification results: for the classification between FA and Start with an accuracy of 85.8% and between two sets of classes FA-GA and C-Start with an accuracy of 66.4%. For these binary classifications, our method, i.e. random forest with the complete set of features, achieved a very high accuracy: 99.8% on classifying FA vs Start, 92.7% on classifying FA-GA vs C-Start, and 91.1% on classifying FA-GA vs all other classes.

2) All classes classification: We report the comparison between accuracy and Area Under Curve (AUC ) value of different techniques in Table V. The full confusion matrix of our method is displayed in Table IV.

Receiver operating characteristic (ROC ) AUC is the measurement of how well does the method behave when the discrimination threshold varies. In order to calculate ROC AUC , firstly the ROC curve is built by plotting true positive rate against false positive rate when the threshold varies and then the area under the curve is computed.

In general, ROC AUC is considered as a more robust measure than accuracy [START_REF] James | An Introduction to Statistical Learning: with Applications in R[END_REF]. However, the accuracy metric is important from users' point of view because in applications, a classifier needs to assign one quality label for a particular article without varying the threshold [START_REF] Japkowicz | Evaluating Learning Algorithms: A Classification Perspective[END_REF]. Therefore, we present both metrics.

AUC is generally defined for binary classification and there does not yet exist a standard way to calculate AUC for multiclass classification [START_REF] Fawcett | An introduction to roc analysis[END_REF]. However, the method proposed by Han and Till [START_REF] Hand | A simple generalisation of the area under the ROC curve for multiple class classification problems[END_REF] is widely used. We reported the AUC values calculated by this method in Table V.

Table V showed that our method achieved both higher accuracy and AUC compared to other existing methods. To confirm the improvement in term of statistical significance, we performed McNemar test [START_REF] Eliasziw | Application of the McNemar test to nonindependent matched pair data[END_REF] on our method and the method presented by Warncke et al. [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF], which is the second best method in terms of both accuracy and AUC . The McNemar test confirmed that our method is significantly better than the method of Warncke et al. [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF], with p -value < 0.001. In fact, the difference of 6% on accuracy score between two methods means that we can correct the quality label prediction for about 300,000 English Wikipedia articles.

While random forest was applied for the classification of Wikipedia articles according to their quality, to our knowledge, no prior studies applied some of the previously mentioned machine learning techniques such as multinomial logistic regression and SVM on this purpose. As displayed in Table V, multinomial logistic regression and SVM obtained better accuracy than random forest applied in [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF], although the AUC scores are lower. Classification results with good accuracies, but low AUC scores are quite common in the literature [START_REF] Millard | Rate-constrained ranking and the rate-weighted AUC[END_REF].

B. Normalized Discounted Cumulative Gain

In this section, we present the evaluation of our model with the Normalized Discounted Cumulative Gain (NDCG) score. NDCG metric proposed by Jarvelin et al. [START_REF] Järvelin | Cumulated gain-based evaluation of IR techniques[END_REF] to evaluate ranking systems was used by several studies on the quality assessment of Wikipedia articles [START_REF] Hu | Measuring article quality in wikipedia: models and evaluation[END_REF], [START_REF] Qin | Assessing the quality of wikipedia pages using edit longevity and contributor centrality[END_REF], [START_REF] Suzuki | Mutual evaluation of editors and texts for assessing quality of wikipedia articles[END_REF]. The ranking is done according to the quality of the articles from high to low, i.e. from FA to Stub. N DCG for the classification of n articles is calculated as:

NDCG = DCG iDCG (9)
DCG = r 1 + n i=2 r i log 2 (i) (10)
where r i is the corresponding grade (or gain) of the article which was ranked i th and iDCG (ideal DCG), is DCG score for an ideal classification where all the articles are classified correctly. NDCG assigns more importance to the items which were predicted at higher classes such as FA than at lower classes such as Stub. NDCG score ranges between 0 and 1, with higher values corresponding to a better prediction.

In order to calculate NDCG value for our model, we need to assign a gain score for each article. Using a similar scoring system with [START_REF] Hu | Measuring article quality in wikipedia: models and evaluation[END_REF], we assigned a score of 6 for FA articles, 4 for GA, 3 for B, 2 for C, 1 for Start and 0 for Stub, meaning that Stub articles do not contribute to the gain.

Because of the formula of NDCG score, the articles ranked at low level actually have no contribution to the score. For this reason, we usually calculate NDCG score for first k items denoted as NDCG@k. For all articles in the test data set, our Fig. 2: NDCG@k score of our model NDCG score [START_REF] Hu | Measuring article quality in wikipedia: models and evaluation[END_REF] [START_REF] Hu | Measuring article quality in wikipedia: models and evaluation[END_REF] 0.84 Suzuki (2015) [START_REF] Suzuki | Quality assessment of wikipedia articles using h-index[END_REF] 0.84 Robertie et al (2015) [START_REF] De La Robertie | Measuring article quality in wikipedia using the collaboration network[END_REF] 0.90 Our method 0.987 The fact that our method achieved a very high NDCG score while the accuracy is not very high is explained by the reason that the NDCG value depends mostly on the correct classification of higher ranked items than lower ranked items. In Table IV, we can observe that our model achieved a low error rate in classifying FA articles, and the error rate increases for the classification of lower quality articles, except for Stub articles. The NDCG score is calculated mostly based on the classification result of FA articles, but does not take into account the high error rate we have in classifying other quality classes, such as GA, B, C and Start. For this reason, a model that classifies FA articles with a certain level of accuracy could achieve high NDCG score, despite the fact that it fails classifying other quality labels.

C. Discussion 1) Over-fitting problem: Over-fitting is a critical problem in machine learning. Over-fitting occurs when a model can predict very well on the training set but fails to predict with the testing set.

We used 5-fold cross validation to test the over-fitting problem in our model, as suggested by [START_REF] James | An Introduction to Statistical Learning: with Applications in R[END_REF]. As we achieved an accuracy of 64% on both training and testing set we concluded that the over-fitting problem is avoided.

2) Contributions of adding features: We showed that our model improved the accuracy of Wikipedia quality prediction models by adding readability scores. In this section we discuss on how these readability scores contributed to the performance of the model. Three experiments were performed to measure the effect of adding readability scores into the baseline model. Among all features of our model, the number of difficult words in the content (difficult words) had the highest contribution to the accuracy of the model. This is an expected result as difficult words are mostly present in detailed and knowledge intensive articles than in no quality articles. High quality articles require authors to present an in-depth knowledge on the topic. In many cases this leads to the use of a technical language where frequency of occurrence of difficult words is higher.

The variable with the second highest contribution is the length of Wikipedia articles. Even though it is a very simple feature, it plays an important role to determine the quality of the text, as suggested in [START_REF] Blumenstock | Size matters: word count as a measure of quality on wikipedia[END_REF].

The next almost equally important variables are: the number of references (num references) and the number of links to other Wikipedia pages (num page links). Both of these variables show how authors support their content by means of different information sources, which increase the reliability of the article.

The next important variable is Dale-Chall readability score (dale chall readability score) that measures how difficult words and sentences are distributed throughout the document. This metric is slightly different than difficult words. As previously discussed in Section III, in machine learning it is common that using a set of closely related features provides a better accuracy than separately using these features.

In the second experiment, we measured the contribution of added features in a different way: starting from the baseline model, i.e. the model with eleven features [START_REF] Warncke-Wang | The success and failure of quality improvement projects in peer production communities[END_REF], readability scores are successively added in a cumulative way to the list of features to study how our model performance changes. The results are presented in Figure 4. The model always performs better when a new readability score is added, although the performance increases are different.

In the third experiment, starting from the baseline model, each readability score was added to the model and then the process was repeated with a different readability score. Each time the experiment was performed, the feature set comprised the baseline structure-based features and one readability score. The results are presented in Figure 5. We observe that adding any readability score improves the performance of the baseline model.

Overall, we can claim that readability scores are important factors to determine the quality of Wikipedia articles and they contributed as much as structure-based scores to the performance of the model. As we discussed in Section I, the readability scores for measuring quality of Wikipedia articles have been studied by [START_REF] Dalip | Automatic quality assessment of content created collaboratively by web communities: a case study of wikipedia[END_REF], [START_REF] Warncke-Wang | Tell me more: an actionable quality model for wikipedia[END_REF], [START_REF] Stvilia | Assessing information quality of a community-based encyclopedia[END_REF]. However, these approaches considered the correlation of readability scores with the quality classes by using linear regression or CART. As we observed in Table V, performances of linear regression and CART on the same feature set and same data set are not as good as random forest. A reason for the difference between performances of linear regression or CART in comparison with random forest is that linear regression and CART fail to deal with collinearity variables [START_REF] Matsuki | The random forests statistical technique: An examination of its value for the study of reading[END_REF], [START_REF] James | An Introduction to Statistical Learning: with Applications in R[END_REF], [START_REF] Montgomery | Introduction to linear regression analysis[END_REF]. It is important to notice that we used different readability scores than the ones proposed in [START_REF] Dalip | Automatic quality assessment of content created collaboratively by web communities: a case study of wikipedia[END_REF], [START_REF] Warncke-Wang | Tell me more: an actionable quality model for wikipedia[END_REF], [START_REF] Stvilia | Assessing information quality of a community-based encyclopedia[END_REF], we investigated the combination of various readability scores and we reported on their importance in the prediction model.

3) Implications for design: In this subsection we present some implications for design for authors to improve the quality of Wikipedia articles and for Wikimedia Foundation on how to provide feedback to authors.

Figure 3 displayed the contribution of each variable to the classification of Wikipedia articles according to their quality classes. However, it is not necessary true that using more difficult words will lead to higher quality articles.

Based on our findings, here are some general suggestions to Wikipedia authors for generating high quality articles:

• Do not hesitate to use technical terms and difficult words if needed. • Elaborate your ideas: some authors tend to write as concise as possible as they assume that some fundamental knowledge is already known by all other people, which usually is not the case. • Provide references to support your content.

• Separate the text into small sections rather than using long paragraphs.

Our proposed automatic quality assessment method on Wikipedia articles can serve for several purposes:

• The method achieves very high accuracy on binary classification, so it can notify authors with a high reliability whether their articles belong to a low quality class and they should be improved. • The method can serve as a measurement for contribution of different authors to a Wikipedia article. For instance, if a Wikipedia article was rated as Stub and after the modification of a user the quality of this article becomes GA, the contribution of this particular user can be highly rated. 4) Limitation: Our model can be applied uniquely for English Wikipedia articles, because the readability scores we used have been designed particularly for English. In order to extend the model to other languages, we need to adapt the readability scores.

VI. CONCLUSION

Wikipedia can be considered as one of the most successful user-generated content projects. However, there is a serious concern related to the quality of information in Wikipedia articles. At the time of writing, among more than five million articles of English Wikipedia, only 4,775 (less than 0.1%) articles have been ranked FA 11 . If the quality of Wikipedia articles can be automatically computed, we can provide a guidance to readers to select the high quality information and a signal to writers to improve their content.

In this paper, we presented a model to classify the quality class of Wikipedia articles. We showed that in addition to studying the structure-based features by analyzing the content of the articles in terms of their readability, a higher accuracy and information gain can be obtained compared with other approaches. As a future direction of our work we plan to combine manual feature design with automatic feature extraction from deep learning techniques [START_REF] Dang | Quality assessment of wikipedia articles without feature engineering[END_REF] in order to improve classification performance.

Fig. 3 :

 3 Fig. 3: Feature importance in random forest model

Fig. 4 :

 4 Fig. 4: Accuracy when readability scores are successively added in a cumulative way to the list of features.

Fig. 5 :

 5 Fig. 5: Accuracy when each readability score is added to the baseline model.

TABLE I :

 I Description of Wikipedia's quality labels used each different feature sets for the classification. In this paper, by introducing new nine features to the set presented by

  4) Coleman-Liau index (coleman liau index ): Coleman-Liau index, or Coleman-Liau readability formula [38] is a linguistic test that measures as Flesch-Kincaid grade the US grade level thought necessary to comprehend a text. As opposed to Flesch-Kincaid grade, Coleman -Liau index relies on characters instead of syllables per word.

	coleman liau index = 5.88 × avg word len -29.6	(4)
		× avg sentence len -15.8
	5) Automated	readability	index
	(automated readability index ):

  7 and it includes the content of 20,489 Wikipedia articles, with corresponding quality labels assigned by Wikipedia reviewers. The distribution of articles within different quality classes is displayed in TableIII. As there is no dominating

	Number of FA articles	2,415
	Number of GA articles	3,160
	Number of B articles	3,209
	Number of C articles	3,322
	Number of Start articles	4,110
	Number of Stub articles	4,273
	Total	20,489

TABLE III :

 III Distribution of the data set within different quality classes quality class in the data set, a naive prediction that predicts every output as the major class, i.e. Stub, achieves a low accuracy of 20.8%.

TABLE IV :

 IV Confusion matrix of our method on testing data with cross-validation. Gray cells are correct predictions. Rows are actual quality class. Columns are predicting values of the model. For example, there are 1,816 articles which are predicted correctly as FA, and 640 articles which are GA and are predicted as FA. The last column is the error predicting rate for each class. Because 5-fold cross validation is used, the confusion matrix contains the entire dataset.

	Algorithm	RS	Accuracy	ROC AUC
	Linear regression	Yes	25%	0.53
	CART	Yes	48%	0.70
	kNN	Yes	55%	0.75
	Multinomial logistic regression	Yes	60%	0.78
	SVM	Yes	61%	0.78
	Warncke et al (2015)	No	58%	0.87
	Our method	Yes	64%	0.91

TABLE V :

 V Comparison of accuracy and AUC value. RS column indicates whether the corresponding feature set includes readability scores or not.

TABLE VI :

 VI NDCG score of different models model achieved a NDCG score of 0.987. The full NDCG score is displayed in Figure 2. Table VI displays the NDCG scores obtained by our model as well as various existing approaches in the literature.

https://reportcard.wmflabs.org/

https://en.wikipedia.org/wiki/Wikipedia:Bots

https://en.wikipedia.org/wiki/Wikipedia:Statistics

An alternative rating system was proposed by Wikipedia users according to four dimensions, i.e. complete, trustworthy, well-written, objective[START_REF] Flekova | What makes a good biography?: multidimensional quality analysis based on wikipedia article feedback data[END_REF], but we do not discuss this rating system in this paper.

https://en.wikipedia.org/wiki/Template:Grading scheme

The data set is available at http://figshare.com/articles/English Wikipedia Quality Asssessment Dataset/1375406.

https://www.mediawiki.org/wiki/API:Main page

https://github.com/nettrom/Wiki-Class

https://pypi.python.org/pypi/textstat

VII. ACKNOWLEDGEMENTS

This research was partially supported by PSPC Open-PaaS::NG project funded by the Investissements d'Avenir French government program managed by the Commissariat général à l'investissement (CGI).

The authors would like to thank Morten Warncke-Wang, University of Minnesota for sharing the dataset and for the valuable discussions.