Florence Lai
email: florence.lai@student.ecp.fr

Frédéric Magoulès
email: frederic.magoules@hotmail.com

Fred Lherminier

Vapnik's learning theory applied to energy consumption forecasts in residential buildings

Keywords: statistical learning theory, data mining, optimization, predictive modeling, energy efficiency, energy conservation

For the purpose of energy conservation, we present in this paper an introduction to the use of Support Vector (SV) Learning Machines used as a data mining tool applied to buildings energy consumption data from a measurement campaign. Experiments using a SVM-based software tool for the prediction of the electrical consumption of a residential building is performed. The data included one year and three months of daily recordings of electrical consumption and climate data such as temperatures and humidities. The learning stage was done for a first part of the data, the predictions were done for the last month. Performances of the model and contributions of significant factors were also derived. The results show good performances for the model. Besides the second experiment consists in model re-estimations on a one-year daily recording dataset lagged at one-day time intervals in such a way that we derive temporal series of influencing factors weights along with model performance criteria. Finally we introduce a perturbation in one of the influencing variable to detect a model change.

Comparing contributing weights with and without the perturbation, the sudden contributing weight change could have diagnosed the perturbation. The important point is the ease of the production of many models. This method announces future research work in the exploitation of possibilities of this "model factory".

Introduction

Large scale energy efficiency has become crucial for clear economical and environmental reasons, namely as a primary means to cope with demand side energy issues. On one side, individual equipment energy performance is a long debated issue, with numerous references and well known teams [START_REF] Barnaby | The Residential Heat Balance Method for Heating and Cooling Load Calculations[END_REF][START_REF] Barnaby | Development of the Residential Load Factor Method for Heating and Cooling Load Calculations[END_REF][START_REF] Stene | Residential CO2 heat pump system for combined space heating and hot water heating[END_REF]. Simulta-neously, there has been significant work on global grid optimization strategies, primarily for gas, oil and electricity networks. Yet the gap between the local scale equipment control loop and the large scale energy distribution network has yet to be filled. Indeed there is only little consideration for the wealth of data collected at the equipment level for control purposes, which can now be used to a much broader extent, thanks to readily available networking and processing technology.

As a result, although automated data analysis has become widely used in different fields such as Finance, Marketing, Telecommunication, it has been less used in the field of Energy Management. Nevertheless research has been performed for the purpose of energy conservation in the residential, the industrial, the service sectors and different approaches can be used. We either can build a parametrical model of the studied system or the subsystem to account for the variation of significant physical values such as the electrical consumption, the heating load, or use statistical tools from classical or modern statistics. The second choice has the drawback of lacking physical meaning but has the significant asset to provide, with a reduced cost of calculation, a range of models, with satisfying performances and properties.

In this paper we first introduce in Section 2 the concept of Energy Intelligence. Energy Efficiency means analyzing existing energy data to unveil the underlying structural exceptions or drifts which may cause abnormal energy consumptions. We also explain to what extent the statistical learning theory preconized by Vapnik can improve the methods to get fast results and thus can be a good approach for real time monitoring. In Section 3 we present the issues raised by the data mining tools of Vapnik's statistical learning theory. Then in Section 4 we describe the context of our work, the resources used and the process of our numerical experiments. Then in Section 5 we present the results. Finally, Section 6 presents the conclusions and as an open issue, we discuss about possible applications as an enhancement of the "model factory" allowed by Vapnik's implementation of the statistical learning theory and the possibility to compare with other data mining tools.

Energy Intelligence

Energy Intelligence aims at identifying new Energy Savings Opportunities from readily available data, by deploying large scale processing techniques to produce energy savings oriented results. Energy Intelligence relies on a number of problem classes such as classification, predictive modeling, system identification, regression.

Typology of Energy Intelligence issues

Below is a typology of the specific problems addressed by Energy Intelligence.

Modeling Strategy Analysis:

The goal is to benchmark various model construction schemes to select a robust modeling scenario. Questions addressed here refer to the availability of a physical system model, state of the art in engineering models, actual dynamics and accuracy required, available field measurements, and relevance of non parametric models. This step is instrumental to all other problem solving approaches described herein. An example could be to obtain and validate a model for the thermal behavior of an entire residence, with input variables such as outside air temperature, number and type of in-house equipment, and individual workload; parameters such as total air volume, thermal exchange coefficients; output variables such as air conditioning system electrical power.

Load Profile Analysis:

Individual systems have their own Energy Consumption patterns; the goal of Load Profile Analysis is to determine a set of typical Energy Use patterns for a given system or subsystem, along with explanatory variables, underlying trends and cycles. This includes a possibility to "teach" (i.e. specify) certain patterns known to the industry, therefore constraining the search for patterns. An example could be to identify a set of electrical power need patterns for residential usage, then derive standard consumption targets for each pattern, associated with relevant scaling factors (such as building volume, exposure, surface, etc).

Multi-site Load Patterns Classification:

This term means to perform a multi-site load pattern classification to determine subsystem families, with comparable energy use patterns, prior to performing multi-site efficiency benchmarking. This implies being able to determine a distance function between actual load profiles.

Performance Rating:

This term means to derive performance indices from target outputs and accepted models, in order to rate performance of actual systems.

Single Site Performance Drift Detection: This term means to detect a single site drift from prior behavior by continuously applying the initially acceptable model to actual inputs.

Multi-site benchmarking:

This term means to perform a multi-site performance rating, based on a priori classification and modeling, applied to actual field data, in order to detect single sites abnormal events or behaviors.

Energy Consumption Forecasts:

This term means to perform predictive modeling for strategic energy consumption variables from numerous candidate explanatory variables. Industrial applications range from peak shaving and supply risk management, to suppliers' portfolio optimization, contract negotiation and spot market lump energy purchase strategies. Residential applications include energy efficiency ratings and drift detection.

The following part deals with different strategies for energy forecast modeling, and their respective advantages and drawbacks.

Model derivation strategies

In order to deal with energy efficiency in buildings, we need to have a general approach to the characterization of energy systems. In [START_REF] Shipper | Explaining residential energy use by international bottom-up comparisons[END_REF] are two antagonist points of view in representing a building and its performances: either from its most simple component to the whole building or the other way around. How can we provide a model of an energy system for monitoring purposes? Several models are proposed, many equations are already edited in handbooks, among them Ashrae's transactions for thermal modeling [START_REF] Ashrae | Energy estimating and modeling methods chapter 32[END_REF][START_REF] Ashrae | Energy estimating and modeling methods[END_REF] and in other papers as [START_REF] Lee | Developing a simplified model for evaluating chiller-system configurations[END_REF][START_REF] Lowry | Modelling the passive thermal response of building using a sparse BMS data[END_REF][START_REF] Tzikopoulos | Modeling energy efficiency of bioclimatic buildings[END_REF].

Different types of data can be used in energy performance assessment [START_REF] Richalet | HELP (house energy labeling procedure): methodology and present results[END_REF]. They either are the result of energy consumption calculation from energy auditors who collect the physical properties and dimensions of the system under control assumptions and proceed to calculation from predefined parametrical models or simply assessed by the energy bill or by real-time or consumption measurements campaigns. In this paper we use data analysis based on a residential building energy consumption measurement campaign.

The two fundamental options for predictive modeling of energy consumption are physical modeling and empirical modeling. A physical model considers each energy source and sink, from conductive losses to natural heating and cooling, radiant and human loads (latent and sensible), along with artificial means (heating, ventilation, air conditioned). It will rely on such calculations as degree-days and solar equivalent surface [START_REF] Ashrae | Energy estimating and modeling methods chapter 32[END_REF]. Physical modeling strategies yield parametric models. They can achieve fairly high accuracy, and are mostly used at design stage, for equipment selection and sizing, materials validation, or architectural design options benchmarking in terms of energy performance. They also prove to be extremely useful for design optimization schemes, which are only possible with parametric models. Yet their derivation is time intensive since they require, by nature, a precise definition of each subsystem, each physical parameter such as window and wall size, material, and exposure, and therefore the individual cost of model makes them impractical for large scale automated model construction. Examples of such tools are given in [START_REF] Sowella | Efficient solution strategies for building energy system simulation[END_REF]: SPARK,IDA, HVACSIM+, TRNSYS. They can tackle detailed building simulation with large set of non-linear algebraic differential equations. The two first tools consists in a set of mathematical models of individual physical component models together with set of coupling equations. As TRNSYS and HVACSIM+ are concerned, they are iterative component-based simulations which consist in subroutines with algorithmic models to account for the physical representation of the building.

The second family of models is the so-called "black box" models from the artificial intelligence field, otherwise referred to as empirical, or non parametric modeling techniques. These models do not assume any preliminary knowledge of the specifics of the system, yet they can achieve reasonable accuracy. The challenge here is in the trade-off between accuracy and human know-how in terms of modeling strategy. Theses include ARMA, State Space, Grey forecasting, Neural Networks. Extensions are also made to polynomials and other basic operations (square roots). They allow information extraction from real data to assess the state of the actual system. Previous research has been performed in [START_REF] Loveday | Artificial intelligence for buildings[END_REF] where the significance of control tools for energy monitoring in buildings is underlined. In this paper predictive control is defined as control actions based on forecasts from a mathematical model of the system behavior which is exactly the motivation of our paper. Different artificial intelligence tools were used as stated in [START_REF] Kalogirou | Applications of artificial neural-networks for energy systems[END_REF] where a chronological review on the first applications of artificial neural networks in the energy field reported models with good performances, with the use of multiple hidden-layer, for the heating-loads of buildings. Besides, Grey Forecasting predicted the performance of a particular chiller [START_REF] Jiang | Applying grey forecasting to predicting the operating energy performance of air cooled water chillers[END_REF] from the variation ratio of its COP, coefficient of performance with high accuracy.

Simultaneously papers have been written on energy consumption forecasts from climate data regression. A regression model of energy is given in [START_REF] Tzikopoulos | Modeling energy efficiency of bioclimatic buildings[END_REF] efficiency as a function of climate and location data, building features. Similarly [START_REF] Valor | Daily Air Temperature and Electricity Load in Spain[END_REF] studies models based on simple regressions with the use of engineering known indicators such as the heating and cooling degree-days and their correlations with the electrical load. In the same way regression of the heating loss with outdoor temperature is performed in [START_REF] Ghiaus | Experimental estimation of building energy performance by robust regression[END_REF].

In the control field, namely in fault detection diagnosis of energy systems, recent research [START_REF] Liang | Model-based Fault Detection and diagnosis of HVAC systems using Support Vector Machine method[END_REF][START_REF] Seem | Using intelligent data analysis to detect abnormal energy consumption in buildings[END_REF] shows a particular interest in data mining tools. The first article introduces a method for the detection of abnormal running condi-tions in a HVAC (Heating Ventilating and Air Conditioning) system with the following process: first a comparison between reference temperature signals in normal conditions, calculated from a thermal modeling, and real signals, then a classification of the transformed residual signals with a multi-layer SVM classifier. In case of fault detection, control actions to come back to normal working conditions are implemented according to the specific abnormal supposed running conditions. As an example the tested abnormal conditions were as follows: the recirculation damper stuck, cooling coil fooling and supply fan speed decreasing and the fault indicators were found to be for example the supply air temperature, mixed air temperature, outlet water temperature. In the second paper [START_REF] Seem | Using intelligent data analysis to detect abnormal energy consumption in buildings[END_REF], the same kind of work is performed. The paper's aim consists in preventing peak loads. The algorithm introduced detects outliers from past energy consumption thanks to relevant indicators. As a result, exploiting real past cases with abnormal operation data, a peak load could have been predicted several days ahead. One important point in detection fault diagnosis is the need for fast and simplified calculations. Of course approaches with different complexity degrees are possible in issues dealing with modeling energy systems, and very detailed and precise models are possible but, in real environment the most complicated model often turns out to be inefficient simply because the calculation resource, namely machines, men and time, can be too heavy. Moreover data for only a few factors are available, sensors can not be installed wherever needed due to cost issues. Suitable and easily produced models, even relevant to ill-posed problems, have to be found. This explains the choice of KXEN software which implements the SVM in order to obtain satisfactory models with minimal or no human tuning required. The SVM are part of the increasing trend in the research field as shown in [START_REF] Daehyon | Prediction performance of support vector machines on input vector normalization methods[END_REF], because of its high performances.

The Vapnik Theory aims at producing such models by first optimizing the model structure according to a certain risk function, then computing the model itself. The overall modeling scheme may therefore be summarized as follows:

(i) Select the target variable and gather the candidate explanatory variables (ii) Select a learning phase and a validation phase in order to consolidate the modeling algorithm (iii) Derive the predictive model (iv) Analyze the model quality criteria (v) Perform predictive modeling analysis to detect energy efficiency drifts by comparing computed and actual energy consumption (vi) Monitor the model coefficients to detect and explain energy performance instability

In the following Section we explain the mathematical substance of Vapnik's learning theory.

3 Statistical learning theory

Learning theory issues

The issues of data analysis, and learning problems are to construct a model from a sample set of observations. That is to say, to estimate a function given the information of the measurement, submitted to some noise due to the measurement process, of the value of this function for a few points, a few observations: from empirical data how can we estimate this function as much as possible, how can we find a robust approximation of it for the entire input space? The problem of choosing a few relevant data to describe a system, model it, has been handled in [START_REF] Bakar | Propositional Satisfiability Algorithm to Find Minimal Reducts for Data Mining[END_REF] where data mining is at the core of the problem.

Classical statistical methods are multidimensional regression estimation, logistic regression. They are based on first the minimization of the empirical risk on a given set of functions with definite structure. But here we will have a different approach based on the structural risk minimization of Vapnik's learning theory.

Vapnik [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF] introduced the SRM concept, the structural risk minimization as a better objective function than the ERM, empirical risk minimization. The SRM takes into account the empirical error but also the complexity of the class of functions to which the searched function belongs. He presents a resolution method that can be used to solve the function estimation problem for a large amount of points in a high dimensional space. Beyond the SRM concepts, the growth function and the VC dimension are introduced in order to quantify a bound on the real error. To explain in more details we will give us n observations (X i , Y i ) i.i.d. random variables of unknown probability P . As already said, the problem consists in estimating Y given X and we will deal with the case Y ∈ {-1, 1} as explained in [START_REF] Bousquet | Introduction to statistical learning theory[END_REF].

Resolution

3.2.1

Structural risk minimization. The SRM, structural risk minimization is the minimization of the sum of a risk related to the choice of the class of functions F in which the candidate f will be searched, the approximation error (biase error) and the risk linked to the estimation from the observations, in the sample set, to the best possible candidate f in F , the variance, the estimation error (variance error). The SRM is a trade-off between the approximation and the estimation error.

The more complex a class of function is, the higher become the first risk, since the risk of over-learning increases. We can formalize this here for f ∈ F .

The structural risk of f , R(f ) is defined by:

R(f ) = R n (f ) + R c (f ) R n f = E(Y i = X i ) = 1 {f (xi) =Yi} dP
where R c (f ) is the risk related to the choice of the function class, the approximation error, and R n (f ) is the empirical risk, the estimation error. Our aim is to find an approximation of:

f * = argminR(f )
on a restricted class of functions F which will be:

f n = argminR f ∈F (f )
As a summary, we have two kinds of risks: R the real risk, R n the empirical risk and we consider two function spaces: the whole function space and F in which, our algorithm picks the approximated function f n . Since P is unknown, we can not directly find R(f n ) so using this decomposition:

R(f n ) = R n (f n ) + [R(f n ) -R n (f n )] = R n (f n ) + R c (f n )
we will work on bounds over R c (f n ).

3.2.2

Capacity measures. We will find the bounds over R c (f ) and this thanks to Capacity Measures, indicators that account for the complexity of the class of function F and can be used in the calculation of the bounds over R c (f n ). In the next two paragraphs we will define two of them, the growth function and the VC dimension. Definition 3.1 The growth function is an attribute of a class of functions F and it represents for n observations X i the number of different nuplets formed by (f (X 1 ), ...., f (X n )) for f ∈ F :

S f (n) ≤ 2 n Definition 3.2 The Vapnik Chervonenkis (VC) dimension is also a character- istic figure of a class of function F , it is the maximal n such that S f (n) = 2 n
, which is the maximal number of points the functions f in the class of function F can classify or shatter. Or in other words, it is the maximal size of an nuplet for which the functions in F can produce the whole possible combinations of values in {-1, 1} which will form (f (X 1 ), ...., f (X n ). We will call this VC dimension h.

3.2.3

Bounds for the approximation error. In order to calculate the bounds over R c (f ) we have to distinguish two cases: the case where the number of functions in F is infinite and the case where it is finite.

Infinite Case. The bound on the real risk of a function f in class F is function of the VC dimension (characteristic of F ) and the size of the sample set n as follows.

Theorem 3.3 Theorem Vapnik and Chervonenkis For any δ ≥ 0, with probability at least 1 -

δ ∀f ∈ F, R(f ) ≤ R n (f ) + 2 2 logS F (2n) + log( 2 δ ) n Then R c (f ) = R(f ) -R n (f ) ≤ 2 2 logSF (2n)+log( 2 δ ) n
. Thus the searched bound depends on the growth function S F . Since this value is not reachable we need a bound on it, which will be possible thanks to the VC dimension. The following lemma gives the needed bound:

∀f ∈ F, R(f ) ≤ R n (f ) + 2 2 hlog 2en h + log( 2 δ ) n (1)
And we have with probability at least 1δ:

R c (f ) = R(f ) -R n (f ) ≤ 2 2 hlog 2en h + log( 2 δ ) n Finite Case.
If the number of functions in F is N , that is to say if F = N then with probability at least 1δ:

sup f ∈F (R(f ) -R n (f )) ≤ logN + log 1 δ 2n (2)
Finally the searched bound for R c (f ), with f ∈ F such as |F | = N is:

R c (f ) = R(f ) -R n (f ) ≤ logN + log 1 δ 2n 3.2.4
Speed of convergence. Here are the different dependencies of sup f ∈F (R(f ) -R n (f )) with n which accounts for the speed of convergence of the empirical risk to the real risk.

Infinite Case. From equation (1) we can find the following dependance:

sup f ∈F (R(f ) -R n (f )) ≈ 2 2
hlogn n

Finite Case. In a same way, from equation (2) the dependance of the bound R c (f ) with the size of the sample set n and the size of the function class F is:

sup f ∈F (R(f ) -R n (f )) ≈ 2 log N n 3.3 Support Vector Machines
Two key values for constructing the learning algorithm have to be kept in mind: the empirical risk and the VC dimension that characterizes the class of functions F . Vapnik introduced the Support Vector Machines technique with which we can control the VC dimension and the empirical risk. Here we explain the principles of the SVM technique.

3.3.1 SVM definition. Support vector machines (SVM) are made from particular vectors from the input space which are the nearest observations to the discrimining hyperplan H, the so-called Support Vectors. One key component of the learning machine will be a linear combination of them. Non linear classification is also possible. Instead of directly taking the observations and the support vectors associated, we will modify the observations to get a transformation of them in an intermediate space and then find the support vectors related to this modified sample set. In the following paragraph we will explain how we can find these support vectors [START_REF] Piquepé | Techniques d'analyse de données[END_REF].

3.3.2 SVM resolution.
The underlying problem is an optimization problem resolved with the Lagrange method, that is why we will first deal with the primal form of the problem, and then work on the dual form of the problem which consists in deriving the Lagrange coefficients of the objective function.

We will also first deal with the specific case where all observations can be separated and then more generally when the problem faced belongs to the non separable case we will introduce slack variables as additional elements of the problem.

Primal form. Suppose that all the points can be classified by f a discrimining linear learning machine such as:

(X i , Y i ) → R d × {-1; 1} if f (X i ) = W ′ X i + b ≥ 1 then Y i = 1 if f (X i ) = W ′ X i + b ≤ -1 then Y i = -1
In order to maximize the distance between the points in H and the nearest point, among the sample set, to H, since we are in a separable case, where in the nearest point

Y i ∈ {-1; 1} and f (X i ) = Y i so f (X i ) × Y i = 1, and since the distance to H is d = Yi(W ′ Xi+b) W = 1
W , the problem is the same as:

M in W W 2 2 under the constraints: Y i (W ′ Xi + b) ≥ 1.
Following the Lagrange method we have to maximize:

L(W, b, α 1 , ..., α n ) = W 2 2 - n i=1 α i (Y i (W ′ X i + b) -1)
which leads to

W - n i=1 α i Y i X i = 0 n i=1 α i Y i = 0
Dual form. The problem is equivalent to maximizing:

L(α 1 , ..., α n ) = 1 2 ( n i=1 α i Y i X i ) ′ ( n i=1 α i Y i X i ) -W ′ ( n i=1 α i Y i X i ) - n i=1 α i L(α 1 , ..., α n ) = - 1 2 n i=1 n j=1 α i α j Y i Y j X ′ i X j + n i=1 α i
under the constraints: n i=1 α i Y i = 0 and ∀i α i ≥ 0. As a result we get:

W * = n i=1 α * i Y i X i
where W * is a linear combination of the nearest observations X i to the hyperplan, which are the support vectors. And thus the support vector machine is:

f (X) = W * ′ X = n i=1 α * i Y i X ′ i X + b
General case. When it is impossible to find a linear function that can separate the points, which constitute most of the cases, we introduce slack variables ξ i in order to allow a misclassification of X i . The objective function becomes:

M in W ( W 2 2 + C n i=1 ξ i ) under the constraints: Y i (W ′ Xi + b) ≥ 1 -ξ i .
This problem is equivalent to maximizing:

L(α 1 , ..., α n ) = - 1 2 n i=1 n j=1 α i α j Y i Y j X ′ i X j + n i=1 α i (3)
under the constraints: n i=1 α i Y i = 0 and ∀i0 ≤ α i ≤ C.

3.3.3

Use of kernels. Instead of the direct use of the observations X i , we can transform them in an intermediate space G:

Φ : R d → G X i → Φ(X i )
Transposing equation (3) the problem is then maximizing:

L(α 1 , ..., α n ) = - 1 2 n i=1 n j=1 α i α j Y i Y j Φ(X i ) ′ Φ(X j ) + n i=1 α i (4)
under the constraints: n i=1 α i Y i = 0 and ∀i0 ≤ α i ≤ C. If we define a kernel function as: k(X 1 , X 2 ) = Φ(X 1 ) ′ Φ(X 2 ), equation (4) becomes

L(α 1 , ..., α n ) = - 1 2 n i=1 n j=1 α i α j Y i Y j k(X i , X j ) + n i=1 α i
And thus the support vector machine is:

f (X) = W * ′ Φ(X) = n i=1 α * i Y i Φ(X i ) ′ Φ(X) + b
If the kernel is well chosen then, there will be no need to represent the points in the intermediate space and learning machines can be considerably improved.

Setting of the problem

The essential idea is: from an analysis of experimental electric consumption data of a residential building, to extract knowledge and derive energy conservation drifts. Or, as reformulated, to find the knowledge of the indicators/factors attesting to an energy consumption in normal running conditions and to analyze their behavior in case of abnormal consumption. We are willing to conduct this resolution using predictive models of this target variable, of whom reliability can be asserted. In the experiments we want to show the link between the fluctuation of influencing factors and model change. In order to do that the strategy is first, deriving a methodology for the derivation of predictive models, secondly to exploit these models and study their variation in experimental conditions and finally study the model behavior in reaction to the introduction of an abnormal running condition. To hope satisfactory results this supposes that the influencing factors must be among those measured which obviously can not always be the case. Thus the possibility for us to get a good model depends first on the quality of our dataset. We found a good dataset candidate which is described in the following part.

Dataset description

The data come from a measurement campaign in 2002 in residences located in different regions of Japan and were released at a domestic architectural conference. Depending on the considered house, energy consumption and climate data, physical properties and dimensions of the building are available. We selected one specific residence in the Tohoku region. For this residence the following data are available:

• Hourly recordings of energy consumption data for the exhaustive set of energy consumption sinks (namely energy consumption for lighting household appliances, HVAC systems ...) • Daily recordings of climate data (temperatures, humidity) • Building features Building features are shown in Table 1. Household appliances description including model, nominal power, and the corresponding energy end-use appliance and measurement points are also available. The dataset is submitted to a preprocessing before the statistical learning step. Here are the dataset construction steps.

(i) Data aggregation to get daily recordings (ii) Electrical consumption calculation from all energy consumption points

For the experiments we finally get a year and three months of daily recordings dataset with variables as shown with their alias in the experiments in Table 2. Time series of temperature, humidity and electrical consumption are plotted in Figures 1,2. For numerical simulations we use the KXEN [START_REF] Kxen | Available online at www[END_REF] software implementing Vapnik's SVM (support vector machines). This tool offers several statistical processing, among which classification, segmentation, robust regression. More interesting to our present discussion temporal series forecasts are calculated. For this particular case applied to energy systems, statistical learning from empirical energy consumption data leads to temporal series models. In this situation energy consumptions predictions and the calculation of characteristic variables influencing these forecasts. The learning process is as follows:

(i) Specify the learning dataset (ii) Run calculation and obtain the model (iii) Apply the model on the desired dataset (iv) Derive the contributions and the performances of the model For the learning process the learning data set must include complete temporal series of potentially influencing factors, which are also called 'explanatory variables', as well as a complete dataset for the forecast variable (or the 'target variable'). We can then apply the model to datasets including the explanatory variables.

We can display the features of the model such as the weight of influencing factors in the model, the performance indicators such as the Pearson coefficient and the robustness coefficient which is the ratio between the residues, difference between the real and predicted signal, variance over the real signal variance.

Numerical experiments

The numerical experiments were divided into two parts.

First approach.

We first start with a global learning step for the first year and a half data followed by the application of the model to the learning dataset plus one month ahead. The different steps of the first approach were as indicated in Figure 3 and were gathered in the "Model-production" module. We first start by specifying the settings of the learning step. For a chosen dataset, the variable to predict, and the number of learning data, or learning phase, are specified. Then the model is derived after the learning step. This model has to be applied to an applying dataset, for which the predictions are calculated after the application step. The predictions are derived within a 95% confidence interval. Results also include the contribution weights of influencing factors, these values are within the interval [0, 1] and the sum of contributions for all influencing variables is one, provided a model is derived. Included as well, the model quality criteria include among others the Pearson coefficient giving the correlation between the predicted and the experimental signal, and the robustness coefficient which is the ratio between residues variance and experimental data variance [START_REF] Kxen | Available online at www[END_REF].

4.3.2

Second approach. The second experiment consists in learning from datasets steaming from parts of the same dataset. Each dataset has a constant time duration and are lagged from one to the other by a preset time delay. For our experiment we take a duration of one year with a time lag of one day as shown in Figure 4. Our aim is to analyze the evolution of occurences of models derived from datasets which are sliding windows of the same dataset taken at regular time intervals. The indicators attesting to the model variation are the factors con-tribution weights. Remembering the process in the first approach with the 'Model-production' module in Figure 3, the logical steps of our second experiment are as in diagrams in Figures 5,6. The 'Model-deviation-detection' module manages the process step. Settings consist in defining the variable to predict V p , the reference learning dataset R 0 which is the dataset from which each subdataset will be sampled, the size of each dataset or number of learning data #ld which for us means the number of recordings and finally the reference time lag T l 0 which is the constant time delay between each sampled dataset. This last value is also equivalent to the number of recordings in this time delay. Then the launch of the 'Data-split' module provides with the datasets dataset(n). For the experiment we set V p as the electrical consumption, #ld = 364, T l 0 = 1.

4.3.3 Third approach. We now decide to introduce a perturbation in one of the contributing variable time series to investigate the changes in contributing weights distribution. We altered the bedroom temperature time series from 2003.12.21 until 2004.01.15. This variable was a good candidate for perturbation since its contribution weights were quite steady during December 2003 as shown in the next section. The alteration consisted in the reproduction of one part of the signal as described in Figure 7. We chose this part of the signal as a perturbation because the sudden decrease shown by that signal could have been the consequence of abnormal running conditions. The experiment was aiming at deriving the impact of such a change in the predictive model of the electrical consumption.

5 Results and discussion

First experiment

Experimental electrical consumption is plotted in Figure 8 along with the signal generated by the model during the learning phase. The predictions are given for the last month within a 95% confidence interval. Weights of influencing factors, part of the model characteristics, and indicators of the quality of the model are given in Tables 3,4. The more contributive variables are plotted in Figure 9. Following are the definitions of the model quality coefficients, the corresponding nomenclature is given in Table 5:

P 2 = (x i -x)(y i -ȳ) (n -1)S x S y Definition 5.2
The robustness coefficient is the ratio between residues variance and experimental data variance. This coefficient is defined by:

U 2 = (x i -x) 2 (y i -ȳ) 2
During the learning phase, the learning dataset is divided into two parts: the estimation and the validation dataset. A first step consists in estimation dataset learning and second in applying the model on the validation set in order to confirm whether or not the model has quite the same performances in both sets. This process is done in order to check that the model derived is reliable. The automated process obtains quite satisfactory predictions. The values of the contribution of temperature and humidity variables are as in Table 3.

The most contributing factor is the outdoor temperature with a contribution weight of 0.63. All the other variables have a weight inferior to 0.1. The quality of the model is quite satisfactory for the Pearson coefficient and the robustness coefficient both around 0.91 for the estimation dataset and around 0.88 for the validation dataset as shown in Table 4.

Time series of the most important influencing variables are as in Figure 9 among them the outdoor temperature is the more contributive one. We can notice that the variation on the outdoor temperature during December had an impact on the energy consumption at the same period. We now can derive the predictive model at a definite instant from a given number of learning recordings. Let us use that methodology to investigate the study of a fluctuating model, of which we could measure the variation. And that, in order to prevent a model change accounting for abnormal operation conditions as shown in the Subsection 5.2.

Second experiment

The purpose of this experiment was to investigate whether or not, the structure of models derived previously was stable and what was the behavior of the contributing variables. How do they vary? How do their variations affect the electrical consumption time series?

Resulting time series of the weights of contributing variables and performance indicators are plotted along with the experimental electrical consumption, and the contributing variables in Figures 10,[START_REF] Richalet | HELP (house energy labeling procedure): methodology and present results[END_REF] The period considered is a three months-period from mid-December to mid-January 2008. One model was built for each day from 2005.25.10 to 2004.02.22. For the sake of readability only the time series of influencing factors with contribution weight superior to 0.2 are considered, which by decreasing mean weights are: the bedroom temperature, the outdoor temperature, the water temperature of cold tap water. The electricity consumption is increasing, whereas the three temperatures decreases. The contributions time series are not continuous, at some interval they could not have been calculated. Indeed, due to the implementation of Structural Risk Minimization (SRM) the models derived can be different depending on the dataset since the SRM principle The performance indicators are remarkably high, at least for the middle of the considered period. The Pearson coefficient and the robustness one are almost equal for the estimation dataset and the validation data set. The coefficients on the validation dataset are curiously low at the beginning and the end of the period. We still do not have the clues to explain those trends.

A comparison is done between the model derived for the reference dataset, and the ones for the sample-datasets of that reference in Figure 19, 15. The bar graphs show descriptive statistics of the contributing weight time series for each variable. An other one is also displayed for the performance indicators time series. The contribution ranking between the mean contributions for the samples is substantially different from the one derived for the reference datasets.

But this could be explained by the fact that the models are fluctuating; they can not really be compared using simple means. The high standard deviation of the contributions attests to those high fluctuations. That is why, one model calculated on the reference dataset may not exclude significant gap with the mean. Nevertheless, regarding the reference dataset, the contributions derived from the model are still in the ± one standard deviation range. Moreover regarding the performance factors, the Pearson and the robustness coefficients are in good adequation between the reference dataset and the samples although the similarity is better for the estimation stage than for the validation one.

Third experiment

In the experiments, we want to derive links between specific running conditions, which for us are interpreted as a given set of influencing variables time series and the alteration of the weights of influencing factors. The contribution weights time series are plotted in Figure 16. A comparison is done with contributions derived without the perturbation in Figure 17. We can clearly notice the change caused by that perturbation which means that the model change was efficiently detected. This looms successful experiments using relevant perturbations. By relevant perturbations, we mean artificial perturbation introduced from signals of which, the origin was known abnormal operating conditions, and for which we could conclude that a specific real faulty condition was detected. This kind of experiment would validate one more time our methodology. Figure 18 shows the performance indicators time series. These indicators do not outline significant changes with the performances for the experiment without perturbation. The performances on the validation dataset after the perturbation introduction are quite low, that was already the case for the previous experiment.

Finally Figures 19,[START_REF] Bakar | Propositional Satisfiability Algorithm to Find Minimal Reducts for Data Mining[END_REF] show bar graphs of the contributions and performance coefficient with their standard deviations. Performances are quite the same as in Subsection 5.2. We can compare the mean contribution weight values obtained for the two experiments in Figure 19. We notice that with the bedroom temperature altered, the contribution weights ranking changes the bedroom temperature is not the most contributing variable, it is second behind the outdoor temperature but the other variables means remain quite the same.

Conclusion and future development

As a conclusion, for the purpose of energy conservation, we seek for an automated implementation of predictive models and their impact on the detection of factors that have an influence on the energy system behavior. The results of our experiments outline that the data analysis of the electrical consumption of a residential building, attests that the SVM-based algorithmic tool used for the experiment is satisfactory. Since its implementation is simple, we can easily use it to make predictive modeling. We introduce the concept of models continuous production which could as well be called "models time series" to account for changes in an energy system. The idea is based on model reestimation, the model is re-learnt at each step on a sliding window of data, which is relevant for real-time monitoring. We achieve a set of experiments that bring to light a model alteration. Since data measurement campaigns are time-expensive and thus data hardly reachable, we used here artificial synthesized perturbation, to study the influence on the model changes, which also means a contribution change of explanatory variables.

For further research our idea is to synthesize altered signal from physical perturbation, in order to detect physical comprehensible abnormalities. We dealt earlier with the SRM (structural risk minimization) method which prevents us from deriving models from the same structure. Nevertheless, given the fact that we are studying the behavior of contribution variables, and that only the same sets of contributions have to be analyzed for the sake of an analysis method that makes sense, structures of models have to be restricted. The issues here are a rapid means of calculations for results at a lower cost. This shows the possibility of achieving good improvement of existing energy systems operation with relevant data exploitation.

Lemma 3 . 4 h

 34 Vapnik and Chervonenkis, Sauer, Shela Let F be a class of functions with finite VC dimension h.Then for all n ∈ N , Then for all n ≥ h, S F (n) ≤ en hFinally with Theorem 3.3 and Lemma 3.4, we obtain the required bound with probability at least 1δ:

Figure 1 .

 1 Figure 1. Temperature time series.

Figure 2 .

 2 Figure 2. Humidity time series.

Figure 3 .

 3 Figure 3. Processing steps for model derivation by the 'Model-production' module.

Figure 4 .

 4 Figure 4. Datasets construction for learning at regular time intervals.

Figure 5 .

 5 Figure 5. Processing steps for the derivation of contributions and performance time series accounting for the model variation by the 'Model-deviation-detection' module.

Figure 6 .

 6 Figure 6. Auxiliary data processing for the obtention of datasets with preset number of recordings lagged with a preset time interval by the 'Data-split' module.

Figure 7 .

 7 Figure 7. Perturbation introduction in the reference dataset.

Definition 5 . 1

 51 The Pearson coefficient represents the correlation between the experimental electrical consumption and the predicted one. This coefficient is

Figure 8 .

 8 Figure 8. Experimental and predicted signal for electrical consumption.

Figure 9 .

 9 Figure 9. Contributing factors to the electrical consumption prediction

  ,12 13.

Figure 10 .

 10 Figure 10. Experimental signal for the electrical consumption.

Figure 11 .

 11 Figure 11. Significant influencing factors time series.

Figure 12 .

 12 Figure 12. Contribution time series.

Figure 13 .

 13 Figure 13. Performance factors time series.

Figure 14 .

 14 Figure 14. Contribution comparisons.

Figure 15 .

 15 Figure 15. Performance comparisons.

Figure 16 .

 16 Figure 16. Contribution time series with the introduction of a perturbation.

Figure 17 .

 17 Figure 17. Contribution comparison in the specific perturbation area.

Figure 18 .

 18 Figure 18. Performance coefficients time series with the introduction of a perturbation.

Figure 19 .

 19 Figure 19. Contribution comparisons with the introduction of a perturbation.

Figure 20 .

 20 Figure 20. Performance comparisons with the introduction of a perturbation.

Table 1 .

 1 Building physical properties and dimensions

	Name	Value
	Location	Tohoku region
	Construction date	1999.11
	Total surface (first floor/second floor) 153.44m 2 (79.62/73.82)
	Overall heat transfer coefficient	1.79W/m 2 • K
	Ratio of open surfaces	0.93cm 2 /m 2

Table 2 .

 2 Variables for each recording in dataset

	Name	Alias
	Date	date
	Outdoor temperature	outdoor temperature
	Bedroom temperature	bedroom temperature
	Living temperature	living temperature
	Living humidity	living room humidity
	Bedroom humidity	bedroom humidity
	Outdoor humidity	outdoor humidity
	Water temperature	water temperature
	Electrical consumption	total electric
	4.2 KXEN software tool

Table 3 .

 3 Contribution weights of factors influencing the electrical consumption.

	Name	Value
	Outdoor temperature	0.636013
	Living room temperature	0.151169
	Bedroom temperature	0.118585
	Living room humidity	0.0677057
	Bedroom humidity	0.0151545
	Outdoor humidity	0.0113732
	Water temperature	0

Table 4 .

 4 Pearson and robustness coefficients of the model for the estimation and validation datasets.

	Performance	Value
	Robusness Estimation	0.916178
	Pearson Estimation	0.917233
	Robustness Validation	0.880781
	Pearson Validation	0.883079
	defined by:

Table 5 .

 5 Index

	x	real signal
	y	forecasted signal
	s	standard deviation
	n size of the sample
	x	average value


Christine Largeron
email: christine.largeron@univ-st-etienne.fr

Christophe Moulin
email: christophe.moulin@univ-st-etienne.fr

Mathias Géry
email: mathias.gery@univ-st-etienne.fr

Entropy based feature selection for text categorization

In text categorization, feature selection can be essential not only for reducing the index size but also for improving the performance of the classifier. In this article 1 , we propose a feature selection criterion, called Entropy based Category Coverage Difference (ECCD). On the one hand, this criterion is based on the distribution of the documents containing the term in the categories, but on the other hand, it takes into account its entropy. ECCD compares favorably with usual feature selection methods based on document frequency (DF ), information gain (IG), mutual information (IM ), χ 2 , odd ratio and GSS on a large collection of XML documents from Wikipedia encyclopedia. Moreover, this comparative study confirms the effectiveness of selection feature techniques derived from the χ 2 statistics.

INTRODUCTION

Text categorization (or classification) is a supervised task for which, given a set of categories, a training set of preclassified documents is provided. Given this training set, the task consists in learning the class descriptions in order to be able to classify a new document in one of the categories ( [START_REF] Yang | A re-examination of text categorization methods[END_REF][START_REF] Sebastiani | Machine learning in automated text categorization[END_REF][START_REF] Feldman | The text mining handbook : Advanced approaches to analysing unstructured data[END_REF]).

Irrespective of the categorization method applied, the documents must be represented by a set of features. Several models can be used, such as the boolean model, the probabilistic model or the vector space model which described documents as a bag of words. As each term belonging to at least one document can be considered as a feature, this can lead to an index size (i.e. the feature space dimension) very large, even for a small collection composed of short documents like news articles. Moreover, all these words are not equally useful for the categorization, like for instance stop words, synonymous, etc. Their distribution must also be studied. For example, words that appear in a single document or in all the documents are not relevant for the categorization task. So, we need to extract a more effective set of features from the text, that can be used to efficiently represent the documents for their categorization. This preprocessing step, can be essential on textual data for improving the performance of the categorization algorithm ( [START_REF] Yang | A comparative study on feature selection in text categorization[END_REF][START_REF] Forman | An extensive empirical study of feature selection metrics for text classification[END_REF]).

Among the many methods that can be used, two strategies can be distinguished: on the one hand, the dimension reduction which consists in creating new synthetic features that are a combination of the original ones, like for instance the Latent Semantic Analysis (LSA) which uses a Singular Value Decomposition (SVD) to extract concepts from the words [START_REF] Deerwester | Indexing by latent semantic analysis[END_REF] and, on the other hand, the feature subset selection that is based on non useful features elimination [START_REF] Han | Data Mining: Concepts and Techniques[END_REF]. In this last case, the selection can be embedded, in other words integrated in the categorization algorithm itself. The feature selection can also be based on a wrapper approach which tries different subsets of features as input and takes the subset that produces the best results. Finally, the selection can be done before the categorization, using a criterion to filter the features.

The common feature selection criteria proposed in the literature ( [START_REF] Wiener | A neural network approach to topic spotting[END_REF][START_REF] Caropreso | A learner-independent evaluation of the usefulness of statistical phrases for automated text categorization[END_REF][START_REF] Ng | Feature selection, perceptron learning, and a usability case study for text categorization[END_REF][START_REF] Galavotti | Experiments on the use of feature selection and negative evidence in automated text categorization[END_REF]) notably, document frequency (DF ), information gain (IG), χ 2 or mutual information (IM ) consider the distribution of the documents containing the term between the categories but they do not take into account the frequency of the term between the categories. However, we can note that a term which is characteristic of a category must appear in a greater number of documents belonging to this category than into the other categories but it should also appear more frequently. For this reason, we propose the Entropy based Category Coverage Difference, denoted (ECCD), which exploits also the entropy of the term.

In this article, this ECCD criterion is also compared to usual feature selection methods mentioned above (IG), χ 2 and mutual information (IM )) on a large collection of XML documents. Indeed, previous works have already shown that removing up to 90% of terms can improve the classification accuracy measured by precision [START_REF] Yang | A comparative study on feature selection in text categorization[END_REF]. However, these experiments have been performed on corpora composed of short documents such as the well known Reuters Collection, composed of short news articles ( [START_REF] Lewis | Feature selection and feature extraction for text categorization[END_REF][START_REF] Lewis | A comparison of two learning algorithms for text categorization[END_REF][START_REF] Wiener | A neural network approach to topic spotting[END_REF][START_REF] Moulinier | Applying an existing machine learning algorithm to text categorization[END_REF][START_REF] Yang | A comparative study on feature selection in text categorization[END_REF][START_REF] Dumais | Inductive learning algorithms and representations for text categorization[END_REF][START_REF] Galavotti | Experiments on the use of feature selection and negative evidence in automated text categorization[END_REF][START_REF] Caropreso | A learner-independent evaluation of the usefulness of statistical phrases for automated text categorization[END_REF]). The second aim of this work is to verify the performance of these feature selection methods on the large INEX XML Mining collection composed of heterogeneous XML documents extracted from the Wikipedia encyclopedia [START_REF] Denoyer | The Wikipedia XML corpus[END_REF].

ECCD proposed in this article is defined in section 2 while the usual criteria are detailed in section 3. The experiments and the obtained results are presented respectively in sections 4 and 5.

FEATURE SELECTION BASED ON EN-TROPY

Textual document representation

In text categorization, the vector space model (VSM) introduced by Salton et al. [START_REF] Salton | A vector space model for automatic indexing[END_REF] is widely used as well for flat documents as for semi structured documents written in markup languages like HTML or XML. In this model, documents are represented as vectors which contain term weights. Given a collection D of documents, an index T = {t1, t2, ..., t |T | }, where |T | denotes the cardinal of T , gives the list of terms (or features) encountered in the documents of D. A document di of D is represented by a vector di = (wi,1, wi,2, ..., w i,|T | ) where wi,j represents the weight of the term tj in the document di. In order to calculate this weight, the TF.IDF formula can be used [START_REF] Salton | A vector space model for automatic indexing[END_REF]:

wi,j = ni,j l n i,l × log |D| |{di : tj ∈ di}|
where ni,j is the number of occurrences of tj in document di normalized by the number of occurrences of all terms in document di, |D| is the total number of documents in the corpus and |{di : tj ∈ di}| is the number of documents in which the term tj occurs at least one time.

However, even for limited collections, the dimension of the index can be exceedingly large. For example, in INEX collection, 652,876 non trivial words have been identified. In a small collection of 21,578 documents extracted from Reuters news, more than 40,000 non-trivial words could be identified [START_REF] Feldman | The text mining handbook : Advanced approaches to analysing unstructured data[END_REF].

Moreover, in the context of categorization, the terms belonging to this bag of words are not necessarily discriminant features of the categories. For this reason, non useful words must be removed, in order to extract a subset T ′ from T more suited for the categorization task. For that purpose, the local approach consists in filtering a specific subset for each category in such a way that the indexes used to represent documents belonging to different categories are not the same, while the global approach, adopted in this work, uses the same subset T ′ extracted from T to represent all the documents of the collection ( [START_REF] Sebastiani | Machine learning in automated text categorization[END_REF][START_REF] How | An examination of feature selection frameworks in text categorization[END_REF]).

In this article, we introduced the ECCD criterion in order to select a subset T ′ from T , providing a more efficient description of the documents. ECCD considers not only the number of documents belonging to the category but also the number of documents belonging to the other categories. Moreover, this criterion also takes into account the entropy of the term. Thus, ECCD exploits two hypotheses to evaluate if a term is a characteristic feature for a category. According to the first one, the major part of the documents containing this term must belong to this category. According to the second one, its number of occurrences must be important in the documents of the category and, on the contrary, it must be lower in the other categories.

Entropy based Category Coverage Difference criterion (ECCD)

Let n k j be the number of occurrences of tj in the category c k and, tf k j the frequency of tj in this category c k :

tf k j = n k j k n k j
The Shannon entropy E(tj) of the term tj is given by [START_REF] Shannon | A mathematical theory of communication[END_REF]:

E(tj) = - r k=1 (tf k j ) × (log2(tf k j ))
The entropy is minimal, equals 0, if the term tj appears only in one category. We consider that this term might have a good discriminatory power in the categorization task. Conversely, the entropy is maximal, equals Emax, if tj is not a good feature to represent the documents i.e. if tj appears in all the categories with the same frequency. ECCD(tj, c k ) is defined by:

ECCD(tj , c k ) = (P (tj|c k ) -P (tj| ck )) × Emax -E(tj) Emax (1)
with P (tj|c k ) (respectively P (tj| ck )) the probability of observing the word tj in a document belonging to the category c k (respectively the other categories):

P (tj|c k ) = |{di ∈ c k : tj ∈ di}| |c k | P (tj| ck ) = |{di ∈ c k : tj ∈ di}| |D| -|c k |
where |c k | represents the number of documents in the category c k and (|D| -|c k |) the number of documents in the rest of the collection .

We can note that the probabilities of observing the word tj in c k and in ck could also be compared with a ratio but this ratio is undefined when the denominator is null. For this reason, it is better to use a difference as done in the formula 1. It is obvious that P (tj|c k ) increases with the number of documents in c k containing tj. Consequently, the higher the number of documents in c k containing tj and the lower the number of documents in the other categories containing tj, the higher the first part of the formula 1. So, tj is a characteristic feature of the category c k if the value of ECCD(tj, c k ) is high, in other words, when the major part of the documents containing this term belongs to this category and simultaneously, its number of occurrences is higher in c k than in the other categories.

Given a term tj and a category c k , ECCD(tj, c k ) can be computed from a contingency table. Let A be the number of documents in the category containing tj; B, the number of documents in the other categories containing tj; C, the number of documents of c k which do not contain tj and D, the number of documents in the other categories which do not contain tj (with

N = A + B + C + D): c k ck tj A B tj C D
Using this contingency table, equation 1 can be estimated by:

ECCD(tj , c k ) ≈ AD -BC (A + C)(B + D) × Emax -E(tj) Emax

COMPARATIVE STUDY OF FEATURE SE-LECTION CRITERIA

In the next section, we compare the criterion defined in the previous section with other usual selection feature methods which have proved good performances in text categorization. These methods are the document frequency (DF ), information gain (IG), χ2 , mutual information (IM ) and odd ratio.

In our experiments mutual information (IM ) and odd ratio have been less effective. Consequently, only the definitions of the first ones, for which we will detail the results in the next section, are given.

Document frequency Thresholding (DF)

This method exploits the hypothesis according to which a term belonging to a few number of documents is not a good feature for the categorization task [START_REF] Wiener | A neural network approach to topic spotting[END_REF].

So, only the terms, that appear in a number of documents higher to a defined threshold, are selected. This threshold can be determined using a training set.

Given a term tj , this criterion can be computed globally on the collection (DF G(tj)) or on each category c k (DF L(tj , c k )):

DF G(tj ) = P (tj) ≈ A + B N DF L(tj, c k ) = P (tj|c k ) ≈ A A + C
One usual way to apply this method consists in eliminating all the words which appear in less than x documents, x varying between 1 and 3 ( [START_REF] Wiener | A neural network approach to topic spotting[END_REF][START_REF] Dumais | Inductive learning algorithms and representations for text categorization[END_REF][START_REF] Li | Classification of text documents[END_REF]). Frequently, this technique is used with another feature selection method.

Information Gain (IG)

Given a term tj and a category c k , the information gain IG(tj, c k ) is defined by [START_REF] Caropreso | A learner-independent evaluation of the usefulness of statistical phrases for automated text categorization[END_REF]:

IG(tj, c k ) = P (tj, c k ) log( P (tj, c k ) P (tj)P (c k ) ) +P ( tj, c k ) log( P ( tj, c k ) P ( tj)P (c k ) ) IG(tj, c k ) ≈ - A + C N log( A + C N ) + A N log( A A + B ) + C N log( C C + D )
Only the words for which the value of the criterion is the most important are considered as characteristics for c k .

The χ 2 and its extension GSS

χ 2 is used to measure the independence between a term tj and a category c k . It is originally defined by:

χ 2 (tj , c k ) =
N.[P (tj , c k ).P ( tj, ck ) -P ( tj, c k ).P (tj, ck )] 2 P (tj).P ( tj) -P (c k ).P ( ck )

χ 2 (tj, c k ) ≈ N.(AD -BC) 2 (A + B)(A + C)(B + D)(C + D)
This criterion equals 0 when tj and c k are independent. On the contrary, tj is considered as a characteristic feature for c k if the value of χ 2 (tj, c k ) is high.

Ng et al. [START_REF] Ng | Feature selection, perceptron learning, and a usability case study for text categorization[END_REF] have observed that the power of 2 at the numerator must be eliminated because it gives the same importance to the probabilities that indicate a positive correlation between tj and c k and to those that indicate a negative correlation. Galavotti et al. have also proposed to eliminate on the one hand N , that is constant, and on the other hand, the probabilities at the denominator that emphasize very rare features and very rare categories [START_REF] Galavotti | Experiments on the use of feature selection and negative evidence in automated text categorization[END_REF]. By removing these factors, Galavotti has introduced the GSS criterion: GSS(tj, c k ) = P (tj, c k ).P ( tj, ck ) -P ( tj, c k ).P (tj, ck )

GSS(tj, c k ) ≈ AD -BC N 2
Both χ 2 and GSS criteria will be presented in the experiments even if the latest is less popular.

EXPERIMENTS

In order to evaluate and compare previously defined selection feature criteria, we performed several experiments on the INEX XML Mining collection [3][4]. After describing this collection, we detail the experimental protocol and we present the obtained results.

Collection description

The different presented selection feature criteria were subject of several studies in the past. The main used collection for these studies is the well known Reuters-21578 collection proposed by Lewis [14]. This collection is composed of 21,578 documents published on the Reuters newswire in 1987 and manually labelled by Carnegie Group, Inc. and Reuters, Ltd. This collection is often used, although it suffers from many problems. The main drawback is its size. Indeed, the number of documents that composed the collection is small with only 21,578 documents. On top of that, these documents are short with on average 137 words per document. This collection is also disputed in reason of the famous blah blah blah [START_REF] Lewis | RCV1: A new benchmark collection for text categorization research[END_REF] that composes some documents. At last, different subsets were used for training and testing in the different studies, that make the comparison tough between obtained results.

In this article, we aim to verify that feature selection, frequently evaluated on small collections of short documents, is also efficient on a bigger collection composed of heterogeneous XML documents. The collection, used in the experiments, is the INEX XML Mining collection from the competition INEX 2008 [START_REF] Denoyer | Overview of the INEX 2008 XML Mining Track[END_REF] 2 . This INEX XML Mining collection is composed of 114,366 documents extracted from the well known Wikipedia encyclopedia [START_REF] Denoyer | The Wikipedia XML corpus[END_REF]. The number of words per document is 423 on average, which is three times more than for the documents of Reuters-21578. Moreover, the index size is greater for the INEX XML Mining collection with 160,000 words compared to 40,000 words for the Reuter-21578 collection.

Documents of the INEX XML Mining collection are categorized into 15 categories that correspond to specific topics, such as sociology, sport, fiction, europe, tourism, united states, etc.. Each document belongs to exactly one category.

The training set, given by the competition INEX, is created using 10% of the whole collection.

Preprocessing, feature selection and categorization

Before performing the categorization task, the index of words that is used to represent documents is preprocessed. In order to perform the indexing, the LEMUR software has been used3 . Without preprocessing, the original index size is 652,876. In order to reduce this index, we perform a common stemming step using the Porter algorithm [START_REF] Porter | An algorithm for suffix stripping[END_REF] that aims to reduce each word to its root form. Thanks to this stemming, the index is reduced to 560,209 distinct words. A lot of words that are not discriminant for the categories are useless. Thus, we remove numbers and words that contains figures (7277, -2311, 0b254c, etc.). The obtained index T after removing all useless words is composed of 161,609 words over the whole collection and 77,706 words in the training set.

However, as explained in the previous section, the terms of T are not necessarily appropriated for the categorization task inasmuch they are not discriminatory for the categories. Thus, thanks to the feature selection criteria, we create a subset T ′ from T that is more representative than T .

All criteria have been used in the same way: firstly, a number n of words that will be selected for each category, is set. Secondly, given a category, terms are ranked by decreasing order in function of the selection feature criteria values and, the first n words are selected. Finally, the index T ′ is composed of the union of the first n selected words for each category. Experiments have been done with values of n varying between 10 to 5,000.

In our experiments, the categorization task is performed using SVM algorithm [START_REF] Fan | Liblinear: A library for large linear classification[END_REF] available in the Liblinear library 4 . SVM has been introduced by Vapnik [START_REF] Vapnik | The nature of statistical learning theory[END_REF] for solving two class pattern recognition problems using Structural Risk Minimization principal. Many studies showed the efficient of SVM in text categorization [START_REF] Joachims | Text categorization with support vector machines: learning with many relevant features[END_REF]. The results provided by this approach on the INEX XML Mining collection are presented in the next section.

RESULTS

In our experiments mutual information (IM ) and odd ratio have been less effective. Consequently, only the results obtained with ECCD, DF , IG, χ 2 and GSS are given.

Two different measures are used to evaluate the performance of the categorization. The first one is the classification rate which corresponds to the number of correct labelled documents divided by the total number of documents. The second measure is the reduction rate obtained using the index T ′ relatively to the original index T (77,706 words) and it is computed by |T

|-|T ′ | |T | .
If we consider the original index T composed of 77,706 words and if we perform the categorization, we obtain a classification rate of 78.79%. Figure 1 still quite good results. Except for the DF G and the χ 2 , for an index T ′ of 3,000 words the classification rate is around 75% which corresponds to a loss of around 5% on the classification rate with an important reduction rate of about 96%. By decreasing order of performance, ECCD criterion obtains the best results, followed by GSS, IG, CC, DF L, DGL and χ 2 . Globally, these results confirm the different comparative studies performed on smaller collections. We can notice that the χ 2 criterion leads to unsatisfactory results when, according to [START_REF] Sebastiani | Machine learning in automated text categorization[END_REF], those results should have been similar to those obtained with IG. The ECCD criterion leads to the best results and this improvement is more significant for a small index size. For an index size of 5,495 words, which corresponds to a reduction rate of 92.93%, the classification rate is 77.86%, which corresponds to a loss of only 1.18%.

CONCLUSION

This study, realized on the INEX XML Mining 2008 collection, confirms the importance in the use of selection feature for the categorization of XML documents and emphasizes the efficiency of criteria derived from the χ 2 such as GSS. It also shows the interest of using the words frequency with the entropy. Indeed, we observed that taking into account the occurrence of terms within the different classes using the entropy of the terms lets us improve significantly classification rate. We can also note that the term occurrence could be integrated into the other criteria. To this extend, we could calculate these criteria with the number of term per class instead of the number of document that contain the term.

  represents the obtained classification rate in function of the size of the index T ′ that depends on the number n of words selected by category with the different selection feature criteria. As shown by this figure, reducing significantly the index size leads to

Figure 1 :

 1 Figure 1: Classification rate obtained for different index size using different feature selection criteria.

This work has been partly funded by the Web Intelligence project (région Rhône-Alpes:http://www. web-intelligence-rhone-alpes.org)

INEX competition: http://www.inex.otago.ac.nz

Lemur Project: http://www.lemurproject.org

Liblinear: http://www.csie.ntu.edu.tw/~cjlin/ liblinear/ -L2 loss support vector machine primal