Sébastien Eskenazi
email: sebastien.eskenazi@univ-lr.fr

Petra Gomez-Krämer

Jean-Marc Ogier
email: jean-marc.ogier@univ-lr.fr

Evaluation of the stability of four document segmentation algorithms

The importance of having stable information extraction algorithms for security related applications and more generally for industrial use cases has been recently highlighted.

Stability is what makes an algorithm reliable as it gives a guarantee that the results will be reproducible on similar data. Without it, security criteria such as the probability of false positives cannot be quantified. As a consequence, no security application can be built from an unstable algorithm. In a document verification framework, the probability of false positives indicates the probability that two different results are given for two copies of the same document. This paper builds on our previous work about a stable layout descriptor to study the stability of four segmentation algorithms. We consider that a segmentation algorithm is stable if it produces the same layout for all copies of the same document. The algorithms studied are two versions of PAL, Voronoi, and JSEG. We compare the stability of the different algorithms and study the factors influencing their stability.

I. INTRODUCTION

Many documents need to be secured, ideally by the means of an electronic signature. Typically, the electronic signature is obtained by computing a hash code on the document's pixels values. If two documents have the same signature, then they are authentic copies of each other and if their signatures are different, one of the two documents is fraudulent or at least different from the other one. This concept works well for naturally born digital documents. However, nowadays a document, the so-called hybrid document, is often used in electronic or paper form according to the need. Hence, the hybrid document undergoes a life cycle of printing and scanning and thus different degraded versions of the document exist as the printing and scanning process introduces specific degradations, such as print and scan noise, in the document [START_REF] Kanungo | Global and local document degradation models[END_REF]. Thus, the concept of electronic signature can not be applied. For this reason, our work intends to develop an advanced electronic signature for the field of securing hybrid documents, the so-called hybrid security. To our knowledge, the closest work is a signature [START_REF] Garcià | Secure Imprint Generated for Paper Documents (SIGNED)[END_REF] based on pixel-wise signal analysis techniques which is not satisfactory because of its large size. Other signatures such as the 2D-Doc [START_REF] Ants | Spécifications techniques des Codes à Barres 2D-Doc[END_REF] require the emitter of the document to provide the information used for signature computation, e.g. the name and the address of the beneficiary, and thus secures only partially the document (the provided information, but for instance not the layout).

Our idea is to extract the layout, the text and the images from the document to compute a stable signature that will be the same for all the authentic copies of the document. Here, we consider that an authentic copy should have the same layout as the original. We are aware about the fact that the layout is not sufficient to embed the total information of a document, but it is a important step to characterize its content. A smaller signature than [START_REF] Garcià | Secure Imprint Generated for Paper Documents (SIGNED)[END_REF] will result as the amount of information to be secured is smaller since the image is not processed pixel-wise. However, this signature will still allow to secure the whole information of the document. In consequence, this requires a segmentation algorithm that is stable with respect to print and scan noise so that two copies of the same document can have the same signature. This paper studies the suitability of current document segmentation algorithms for such a task.

A lot of work has been presented for evaluating segmentation accuracy [START_REF] Antonacopoulos | Competition on Historical Newspaper Layout Analysis (HNLA 2013)[END_REF], [START_REF]Competition on Historical Book Recognition (HBR 2013)[END_REF]. Anyhow, the concept of accuracy does not apply to our security context. The two concepts of accuracy and stability should not be confused. Accuracy requires a ground truth to evaluate how close a result is to this ground truth. Accuracy can be evaluated with only one result as long as there is also a ground truth. Stability does not require a ground truth. Stability requires at least two results with similar inputs to see how close these results are together compared to how close the inputs were. In our case, similar inputs are two photocopies of the same document. A consequence of this is that an algorithm can be very stable and yet not be accurate. For instance, this can be an algorithm that always makes the same mistakes or in the case of a segmentation algorithm, an algorithm producing always one region covering the whole image. Such an algorithm would have an absolute stability and zero accuracy. The contrary is not true. An algorithm with an absolute accuracy will always produce results that are identical to the ground truth and hence identical between each other. A perfect segmentation algorithm will also be perfectly stable.

Furthermore, stability should also not be confused with robustness. A robust algorithm is an algorithm capable of providing a relevant output even when a certain amount of noise is contained in its input. The variation of this output is not constrained as it is for stability.

This raises the question of defining similar segmentation results. The first criteria is that they should have the same number of regions. Then, these regions should have a similar relative position. This is what we call the layout. To illustrate this definition Figure 1c shows a set a of precise segmentation results. They all match the overall shape appearance of the original document. On the contrary figure 1b presents some segmentation results that are less precise but stable. They all have two regions side by side at the top of the document and two regions on top of each other at the bottom. In order to have a precise criteria, we will use the Delaunay Layout Descriptor that we have created specifically for this purpose [START_REF] Eskenazi | The Delaunay document layout descriptor[END_REF]. This definition is quite minimal as there is a strong tradeoff between the precision of describing the layout and its stability. Moreover, the content of a region can be, for instance, represented by its textual content instead of its size.

We will now present the studied algorithms and the evaluation protocol in sections II and III respectively. Then we will present the results of our study in section IV before concluding this paper in section V.

II. STATE OF THE ART

There are two kinds of works related to this paper: the work on segmentation evaluation and the work on document image segmentation algorithms.

A. Evaluation of segmentation algorithms

Evaluations of document segmentation algorithms are regularly published either as a dedicated papers [START_REF] Shafait | Performance evaluation and benchmarking of six-page segmentation algorithms[END_REF], [START_REF] Stamatopoulos | Goal-oriented performance evaluation methodology for page segmentation techniques[END_REF] or as a competition report [START_REF] Antonacopoulos | Competition on Historical Newspaper Layout Analysis (HNLA 2013)[END_REF], [START_REF]Competition on Historical Book Recognition (HBR 2013)[END_REF], [START_REF] Antonacopoulos | ICDAR2015 Competition on recognition of documents with complex layouts[END_REF]- [START_REF] Antonacopoulos | ICDAR2007 Page segmentation competition[END_REF]. These papers have investigated how to evaluate a single segmentation result based on a given ground truth. A corollary is the question of the quality, relevance and bias of the ground truth [START_REF] Stamatopoulos | Goal-oriented performance evaluation methodology for page segmentation techniques[END_REF]. To take a trivial example, a line-wise ground truth will favor line segmentation algorithms over paragraphs segmentation algorithms. Yet, both kinds of algorithms could actually be of nearly equal use and of equal quality.

Regarding the comparison of several results with each other, i.e. stability, to our knowledge, the only paper treating this topic is the one by Guo et al. [START_REF] Guo | A multi-stage approach to curve extraction[END_REF]. They analyze the stability of an algorithm for curve line extraction from natural images. To this intent, they use several similar images of the same scene and analyze the performance variation. However, this work still requires a ground truth to evaluate the performance of the algorithm on each image.

B. Segmentation algorithms

The latest document segmentation competitions provide a good state of the art of document segmentation algorithm. They can be completed with [START_REF] Nagy | Twenty years of document image analysis in PAMI[END_REF]- [START_REF] Kise | Page segmentation techniques in document analysis[END_REF].

Out of these, we selected the best segmentation algorithm in the latest competition at the time of writing this paper. The PAL [START_REF] Chen | Hybrid page segmentation with efficient whitespace rectangles extraction and grouping[END_REF] algorithm won the two ICDAR 2013 competitions on historical document layout analysis [START_REF] Antonacopoulos | Competition on Historical Newspaper Layout Analysis (HNLA 2013)[END_REF], [START_REF]Competition on Historical Book Recognition (HBR 2013)[END_REF]. It is a top down algorithm working on binary images. Furthermore, Shafait et al. [START_REF] Shafait | Performance evaluation and benchmarking of six-page segmentation algorithms[END_REF] did an analysis of document segmentation algorithms and found that Voronoi segmentation [START_REF] Kise | Segmentation of page images using the area voronoi diagram[END_REF] was the best.

To complete our panel we added JSEG [START_REF] Deng | Unsupervised segmentation of colortexture regions in images and video[END_REF]. JSEG is originally made for segmenting natural images and is a reference algorithm. It has the particularities of being quite robust and of using color textures as well as a multiscale analysis. All this makes it an interesting natural image segmentation algorithm. It will allow us to study the impact of color textures and multiscale analysis on stability and the applicability of natural image segmentation algorithms to document image segmentation.

PAL was published by Chen et al. at ICDAR 2013 [START_REF] Chen | Hybrid page segmentation with efficient whitespace rectangles extraction and grouping[END_REF]. The algorithm works by identifying text line chains based on connected components. Then it detects the white spaces along each chain and filters them according to some heuristics. It produces two outputs: line rectangles and block rectangles. Hence we separated it into two algorithms: PALL for the identification of the lines and PALB for the blocks.

The Voronoi algorithm was published in 1998 by Kise et al. [START_REF] Kise | Segmentation of page images using the area voronoi diagram[END_REF]. It is a versatile segmentation algorithm that works for any document layout. It creates edges between points sampled on each connected component. Then it removes the superfluous edges that are too short or that connect connected components of sizes that are too different.

JSEG was described by Deng and Majunath in 2001 [START_REF] Deng | Unsupervised segmentation of colortexture regions in images and video[END_REF]. It first quantizes the image colors. Then it segments/clusters the image based on the J value of each region. The J value is an indicator of uniformity of the region. Because it works at several scales, it can segment non-uniform textures. We used the implementation available on the project website 1 .

III. EVALUATION PROTOCOL

In order to provide a proper analysis of the algorithms we performed a cross-validation of the algorithms on a dataset of 990 document images.

A. Dataset

Our dataset is based on the PRImA dataset used in 2009 [START_REF] Antonacopoulos | ICDAR2009 Page segmentation competition[END_REF]. It has the advantage of having documents with a very varied content. We printed its 55 pages with three times at 300 dpi and three times at 600 dpi on several scanners as shown in Table I.

For each document there are 3 printers × 3 scanners × 2 resolutions = 18 copies. Thus the total dataset contains 18 × 55 = 990 document images.

Because of the scanning process there are no perspective distortions and hardly any skew on the images. Hence the main degradations are print and scan noise, printing scale and image translation. The printing scale can vary slightly depending on the settings for printing an image. We used the defaults settings for all the printers.

B. Evaluation process

Considering the requirements of each algorithm, we resized the original 300 dpi and 600 dpi dataset (denoted HR) to 60 dpi (denoted LR). Then we tested all algorithms separately with high-resolution images and with low-resolutions images except for JSEG which cannot use HR images. This allows us to evaluate the influence of the resolution on the stability of the results. In order to have enough images for testing, we decided to perform a 5-fold cross-validation. The dataset is first split randomly into 5 datasets. The first dataset is used for testing in the first fold and the rest for training. The second dataset is used for testing in the second fold and so on.

C. Metrics

We use two metrics to evaluate the stability of the segmentation algorithms: the probability of false positives and the normalized standard deviation of the number of segmented regions.

1) Probability of false positives: The probability of false positives (PFP) is widely used for evaluating the performance of a binary classifier such as a medical test or an authentication application. For the specific case of layout authentication/comparison, it is the probability that the algorithm will produce two different layouts when given two inputs with the same layout. A stable algorithm should have a PFP below 5%.

As stated in the introduction, we consider that two segmentation results produce the same layout if the DLDs that they produce can be matched according to the method described by Eskenazi et al. [START_REF] Eskenazi | The Delaunay document layout descriptor[END_REF]. We accepted an angle error of 5 • and a maximum number of 2 simultaneous instabilities as defined in [START_REF] Eskenazi | The Delaunay document layout descriptor[END_REF]. The DLD works by triangulating the centroids of the segmented regions. This triangulation is seen as a graph whose nodes are then ordered. The adjacency matrix of this ordered graph makes the DLD. As a side note, we reimplemented and accelerated the computation of the DLD approximately by a factor 30 and its matching by a factor 30 × n where n is the number of simultaneous instabilities.

The probability of false positives is then computed imagewise. Let us consider one segmentation algorithm and one image I of an input dataset. Let ts be the number of the images including the first image I that have the same layout (copies of the same document). Then we count how many of these images yield a different segmentation result than that of image I. Let t be the number of failed matches. The probability of false positives for image I is given by:

P F P I = t ts (1)
The PFP for the algorithm is the average of all the PFPs for the images of the input dataset. During the cross validation, it can occur that there is only one copy of a document. In that case we will ignore it for the computation of the PFP.

D. Normalized standard deviation of the number of regions

Two layouts will never be identical if they do not have the same number of regions. Hence having a stable number of regions is a necessary condition to having stable results and the standard deviation (StD) of this number of regions should be as low as possible.

However an StD of 5 is high if there are 10 regions but low if there are 50 regions. Hence we normalize it by the number of regions produced for each document by each algorithm.

Let us consider a given algorithm and a given image I of a document from an input dataset. We list the number of regions of each segmentation result of the copies of the same document. Let n i be the number of segmented regions in the i th copy.

The normalized standard deviation of the number of regions S R for image I is computed as

S R (I) = σ(n i ) n i (2)
where σ(n i ) is the standard deviation of n i and n i is the average of n i .

Similarly to the PFP, the S R for the algorithm is the average of all the S R for the images of the input dataset. Once again, we ignore the single copies of a document for the computation of the S R . To be able to analyze the values of S R in more details, we also compute the average number of regions produced by each algorithm, here denoted as n R .

E. Algorithm specifities

The algorithms have different parameters which call for specific training strategies in order to evaluate each algorithm in the best conditions. As a rule of thumb we decided to study the parameter values around their default value.

1) PAL: Both versions of PAL take a color image as input and do not use any parameter so we ran them on the complete dataset and evaluated them without training.

2) Voronoi: Voronoi takes binary images as input and has five parameters. We binarized the original images with Otsu's method [START_REF] Otsu | A threshold selection method from gray-level histograms[END_REF]. We chose the parameter sampling rates in order to achieve the best trade-off between an exhaustive sampling of the parameters and a small number of combinations to provide the best possible training within a feasible time.

The first parameter is the sampling rate of the contour of the connected components which is used to select the vertices that will be the ends of the Voronoi tesselation. It can take an integer value above 1. The default value is 7 which means that 1 out of 7 pixels on the contours of the connected components will be selected. We consider the values between 5 and 10 included. A smaller value would result in a clear oversampling and bigger value could prevent a proper sampling of the contours of small connected components. The second parameter is the maximum length of the contour of a noise connected component. The default value is 20 pixels and we consider values between 10 and 50 pixels by increments of 5 pixels. Because these two parameters are pixel values, they need to be adapted to the image resolution. They are related to a length at a resolution of 300 dpi so we scale them linearly with the image resolution. The remaining parameters influence the removal of superfluous Voronoi edges based on the processing of the distribution of the distances of these edges. The interested reader can refer to [START_REF] Kise | Segmentation of page images using the area voronoi diagram[END_REF] for a detailed explanation. The third parameter is the ratio of the value of the second highest peak which defines the maximum length of an edge. Its default value is 0.5. We make it vary between 0.2 and 0.8 by increment of 0.1. Considering the explanation of [START_REF] Kise | Segmentation of page images using the area voronoi diagram[END_REF] a lower value would not be selective enough and a higher value would be too selective. The fourth parameter is the maximum area ratio between two regions linked by an edge. If the ratio is bigger than this value, the edge is deleted. The default value is 40 and we make it vary between 20 and 60 by increments of 5. The last parameter is the size in pixels of the smoothing window of the distance distribution. Its default value is 2. After performing exhaustive testing, the only relevant values are between 1 and 3.

This makes 6 × 9 × 7 × 9 × 3 = 10206 parameter combinations. Considering the size of the dataset and the processing time, a brute force optimization would be inefficient. This is why we settled for the use of a genetic algorithm with a population of 20 individuals, 20 generations, a mutation probability of 0.05 and a mating probability of 0.6.

3) JSEG: JSEG requires a small resolution color image as input (60 dpi maximum) and has three parameters. The requirement for a small resolution image is related to the convergence of the algorithm and its implementation. The color quantization threshold and the number of scales used for multiscale processing have an automatic mode which chooses the best parameter on an image per image basis. Because this is a finer grained tuning than fixing these parameters for the whole dataset, we used the automatic mode for them. The last parameter is the threshold to merge regions which can take a value between 0 (all regions are merged) and 1 (no regions are merged). Its default value is 0.4. Since the range of values is between 0 (no merging) and 1 (everything merged) we chose to study the values from 0.1 to 0.9 with a step of 0.1. Our preliminary tests have shown that a finer step does not result in a significant improvement. That makes 9 values to test so that a brute force training can easily be performed and ensures the best training.

IV. RESULTS

Before comparing the algorithms we will first analyze each of them. In all the following "LR" indicates that we used the low-resolution dataset (60 dpi) and "HR" the high-resolution one (300 and 600 dpi).

A. PAL

Table III summarizes the different results of both versions of PAL. As we can see it is not stable. The use of low-resolution images improves it a bit. This is mostly due to the fact that PAL is designed for text line extraction and over-segments the graphical areas. We also noticed that PALL and PALB tend to produce similar segmentation results on graphical areas. Thus, an improvement could be to remove the regions that are common to PALB and PALL results.

B. Voronoi

Table II summarizes the results of the Voronoi segmentation algorithm for each fold of the cross-validation. The numbering of the parameters is the same as that of section III-E2. From a global point of view, the algorithm performs better with lowresolution images although it is nearly as unstable as PAL. It has a slightly lower PFP, a lower S R and it produces a lower number of regions. The fifth fold of the low-resolution testing is a clear outlier.

Let us study the influence of each parameter. Table IV presents the maximum variation of the metrics when only one parameter is changed. We can already notice that the ranking of the influence of the parameters is the same for the PFP and for S R . This comes in support of the hypothesis that these metrics are related.

The contour sampling rate (parameter n • 1) has roughly the same value for HR and LR images. This means that our scaling of this parameter with the image size is adequate. Its best value seem to be 7 and it has an average influence on the algorithm performance.

The maximum size of a noise contour (parameter n • 2) has an optimal value in HR that is double of that in LR. It also has a strong influence on the algorithm performance. Its best value is 10 in LR and 25 in HR.

The ratio of the value of the second highest peak (parameter n • 3) has a similar behavior between HR and LR. This can be explained by the fact that because the LR images have a smaller size, the peaks of the distribution of the distances of the Voronoi edges will be steeper. In order to keep enough edges, we need to have a lower threshold in LR than in HR. This parameter has a significant influence on the algorithm performance in LR and should be carefully tuned. This explains its stability across the folds. Its best value is 0.2 in LR and 0.4 in HR.

The maximum area ratio between two regions (parameter n • 4) is approximately the same in LR and HR, around 30, and has a low influence on the algorithm performance.

Finally, the optimal size of the smoothing window for the distribution of the distances of the Voronoi edges (parameter n • 5) is bigger in LR than in HR. This is coherent with the fact that the peaks are steeper in LR than in HR and thus need more smoothing. This parameter has an average influence in LR and a very low influence in HR on the algorithm performance. It best value is 3 in LR and 1 in HR.

To summarize, parameter 2 (respectively 3) has the highest impact in HR (respectively LR). Parameters 4 and 5 have the lowest impact. Anyhow all parameters have a negligible impact in comparison to the improvement required to make the algorithm stable.

C. JSEG

Table V summarizes the testing results of JSEG for each fold. The RMT is the region merging threshold. The results do not vary much and are surprising because JSEG has a very good S R and a very bad PFP. This is due to the fact that the layout is not just related to the number of regions.

When looking deeper in the training data, we see that the PFP (respectively S R ) decreases (respectively increases) when the region merging threshold increases. JSEG actually produces a stable number of regions when there are many regions and this number becomes more unstable when their number decreases. This suggest that the merging of JSEG process isunstable. When visually examining the quality of the segmentation results we notice that an RMT of 0.1 oversegments the document and an RMT of 0.9 undersegments it very clearly. In our opinion, the value of 0.3 achieves the best tradeoff with a PFP of 0.989, an S R of 0.270 and proper segmentation results.

The PFP on the training data goes from 0.73 to 1 with an average of 0.875, S R goes from 0.2 to 0.61 with an average of 0.44 and n R goes from 3 to 22 with an average of 10. testing parameters. We can see the influence of the resolution on the algorithm results and their instability. Except for JSEG, they all tend to create artificial regions in place of the picture. Considering all the above, JSEG has a better performance in terms of the stability of the number of regions it produces. It is followed by PALL. Voronoi and PALB both perform honorably on low-resolution images but are heavily unstable on high resolution images. Regarding the probability of false positives, JSEG is also the only one that has one below 0.99. Texture features are known for being robust to noise and adding a multiscale analysis allows a global view of the layout which helps produce more stable results.

D. Comparison of the algorithms

V. CONCLUSION

This paper treats a new field of evaluating algorithms : stability. To this end, we have provided a set of metrics and an evaluation methodology for studying the stability of segmentation algorithms. We have applied it on four state of the art algorithms on document images. Finally we have analyzed these algorithms in detail in order to understand what Unfortunately, all the algorithms studied have a poor stability. The best one, JSEG, achieves only a probability of false positives of 99% which is far from the objective of being below 5%. We have also found that using texture features, multiscale analysis and low resolution images may help achieve a better stability.

We will need to evaluate more algorithms and to combine the evaluation of stability with more traditional metrics in order to ensure that the algorithms produce relevant results.

Fig. 1 .

 1 Fig. 1. Difference between precise and stable segmentation results

Figure 2

 2 Figure2shows two results of each algorithm for each tested resolution on different copies of the same document with their

Fig. 2 .

 2 Fig. 2. Segmentation results of the algorithms

TABLE II VORONOI

 II TESTING RESULTS. ALL VALUES SHOULD BE AS LOW AS POSSIBLE.

	Version Resolution	PFP	S R	n R
	PALB	LR HR	0.998 0.32 26 1.000 0.9 72
	PALL	LR HR	0.999 0.20 89 1.000 0.34 149

TABLE III PAL

 III TESTING RESULTS. ALL VALUES SHOULD BE AS LOW AS POSSIBLE.

TABLE V JSEG

 V TESTING RESULTS (ON LOW-RESOLUTION IMAGES). ALL VALUES SHOULD BE AS LOW AS POSSIBLE.

http://vision.ece.ucsb.edu/segmentation/jseg/software/

ACKNOWLEDGMENT

This work is financed by the French National Research Agency (ANR) project SHADES referenced under ANR-14-CE28-0022.


Jie Liu

Enrico Zio
email: enrico.zio@ecp.fr

SVM hyperparameters tuning for recursive multi-step ahead prediction

Keywords: recursive multi-step ahead prediction, support vector machine, hyperparameters tuning, single-objective optimization, bi-objective optimization

SVM hyperparameters tuning for recursive multi-step-ahead prediction

Introduction

Prognostics and Health Management (PHM) is becoming an important topic of interest for industrial applications [START_REF] Vichare | Prognostics and health management of electronics[END_REF][START_REF] Lei | Fault diagnosis of rotating machinery based on multiple ANFIS combination with GAs[END_REF][START_REF] Wu | Prediction of rainfall time series using modular soft computingmethods[END_REF][START_REF] Zhao | A Profust Reliability Based Approach to Prognostics and Health Management[END_REF][START_REF] Javed | Enabling health monitoring approach based on vibration data for accurate prognostics[END_REF][START_REF] Lei | Health condition identification of multi-stage planetary gearboxes using a mRVM-based method[END_REF]. Prognostics includes the prediction of the health condition of the equipment of interest in the future [START_REF] Jardine | A review on machinery diagnostics and prognostics implementing condition-based maintenance[END_REF][START_REF] Palm | Multiple-step-ahead prediction in control systems with Gaussian process models and TS-fuzzy models[END_REF][START_REF] Saha | Prognostics Methods for Battery Health Monitoring Using a Bayesian Framework[END_REF][START_REF] Zio | Prognostics and health management of industrial equipment[END_REF]. This can be done with reference to the next time instant, i.e. one-step ahead prediction or multiple time instants, i.e. multi-step ahead prediction [START_REF] Jardine | A review on machinery diagnostics and prognostics implementing condition-based maintenance[END_REF]Wang and Han, 2015). The latter is definitely more difficult than the former.

Support Vector Machine (SVM) [START_REF] Drucker | Support vector regression machines[END_REF][START_REF] Ding | A fast fuzzy support vector machine based on information granulation[END_REF][START_REF] Wang | An efficient Kernel-based matrixized least squares support vector machine[END_REF] is a popular data-driven approach, also widely used for prognostics. The inputs are features extracted from measurements directly or indirectly related to the prediction of interest.

SVM has been used for example in Muller et al. (1997), [START_REF] Min | Bankruptcy prediction using support vector machine with optimal choice of kernel function parameters[END_REF], [START_REF] Asefa | Multi-time scale stream flow predictions: The support vector machines approach[END_REF], [START_REF] Widodo | Intelligent prognostics for battery health monitoring based on sample entropy[END_REF], [START_REF] Zio | Fatigue crack growth estimation by relevance vector machine[END_REF], Benkedjouh et al. (2013b), [START_REF] Liu | A hybrid WA-CPSO-LSSVR model for dissolved oxygen content prediction in crab culture[END_REF], [START_REF] Namdari | Incipient fault diagnosis using support vector machines based on monitoring continuous decision functions[END_REF]. Multi-step ahead prediction, e.g. to compute the remaining useful life, i.e. the time at which the component degradation state reaches a predefined failure threshold, has been done for example in [START_REF] Asefa | Multi-time scale stream flow predictions: The support vector machines approach[END_REF], [START_REF] Lin | Using support vector machines for long-term discharge prediction[END_REF], [START_REF] Sorjamaa | Methodology for long-term prediction of time series[END_REF], Sapankvych and Sanka (2009), [START_REF] Nuhic | Health diagnosis and remaining useful life prognostics of lithium-ion batteries using data-driven methods[END_REF], Benkedjouh et al. (2013a), [START_REF] Bao | Multi-step-ahead time series prediction using multiple-output support vector regression[END_REF]. Recursive (iterative) prediction is a popular strategy for multi-step ahead prediction of time series data. Recursive SVM achieves the multi-step ahead prediction on the basis of the predictions of previous steps, which are used as in inputs for the prediction in the following step. For example, in Benkedjouh et al. (2013a), SVM is used to assess the current status and predict the Remaining Useful Life (RUL) of bearings. In [START_REF] Nuhic | Health diagnosis and remaining useful life prognostics of lithium-ion batteries using data-driven methods[END_REF], SVM is used to recursively predict the future capacity of lithium-ion batteries to estimate the time left for reaching 80% of the nominal capacity.

There are three types of hyperparameters in SVM, i.e. the regularization parameter which balances the prediction accuracy and the flatness of the model, the hyperparameters related to the kernel function which define the high-dimensional feature space and the hyperparameters related to the loss function which controls the number of data points in the model.

Tuning the hyperparameters of SVM for application to prognostics is a challenge. For the same dataset, different hyperparameters values may give either very good or very bad prediction results. The best hyperparameters values are those which give best prediction on the new data. How to find the optimal hyperparameters values effectively and efficiently and how to guarantee the prediction performance without overfitting on the dataset used for SVM training are critical problems.

Cross-Validation is used in [START_REF] Nuhic | Health diagnosis and remaining useful life prognostics of lithium-ion batteries using data-driven methods[END_REF] to find the best hyperparameters of SVM considering the Mean Squared Error (MSE). A hybrid of Particle Swarm Optimization (PSO) and Differential Evolution (DE) is used to search the optimal parameter combination in the search space in [START_REF] Zhang | Support vector machine with parameter optimization by a novel hybrid method and its application to fault diagnosis[END_REF] with minimal prediction error. PSO is employed also in Aich and Banerjee (2014) for the purpose of finding the SVM hyperparameters that minimize the Mean Absolute Percentage Error (MAPE) on the training dataset. Adaptive PSO is used in Wang et al. (2015) for tuning hyperparameters. Hyperparameters tuning in [START_REF] Chapelle | Choosing multiple parameters for support vector machines[END_REF] is done by minimizing some estimates of the generalization error of SVM using a gradient descent algorithm over the parameters space. In [START_REF] Liu | Optimizing the hyper-parameters for SVM by combining evolution strategies with a grid search[END_REF], Grid Search (GS) is combined with evolutionary strategies for optimizing hyperparameters in SVM, aiming at minimizing the cross-validation accuracy. In [START_REF] Wu | A real-valued genetic algorithm to optimize the parameters of support vector machine for predicting bankruptcy[END_REF], a geneticbased SVM is developed to determine automatically the hyperparameters values with the highest predictive accuracy and generalization ability simultaneously. A fast messy GA is used in [START_REF] Chou | Optimizing parameters of support vector machine using fast messy genetic algorithm for dispute classification[END_REF] to find the best hyperparameters values with a minimum number of support vectors and optimal SVM parameters to preserve an acceptable level of prediction accuracy. In [START_REF] Igel | Multi-objective model selection for support vector machines[END_REF], different bi-objective optimizations considering model complexity and prediction accuracy for hyperparameters tuning are proposed and tested. [START_REF] Aydin | A multi-objective artificial immune algorithm for parameter optimization in support vector machine[END_REF] uses the percentage of support vector and prediction accuracy as the objectives for tuning hyperparameters in SVM. A comprehensive learning PSO-based memetic algorithm that evolves feature selection and parameter optimization simultaneously is proposed in [START_REF] Hu | Comprehensive learning particle swarm optimization based memetic algorithm for model selection in short-term load forecasting using support vector regression[END_REF] to minimize the MAPE on the training dataset. [START_REF] Liu | A hybrid WA-CPSO-LSSVR model for dissolved oxygen content prediction in crab culture[END_REF] hybridizes a SVM model with an adaptive Genetic Algorithm (GA) search and the seasonal index adjustment to forecast daily flow of holiday tourist.

The maximum-margin principle and the jackknife technique are used in [START_REF] Chang | Tuning of the hyperparameters for L2-loss {SVMs} with the {RBF} kernel by the maximum-margin principle and the jackknife technique[END_REF] for tuning hyperparameters of L2-loss SVM. Fruit fly optimization algorithm is used in [START_REF] Shen | Evolving support vector machines using fruit fly optimization for medical data classification[END_REF] for tuning hyperparameters.

As discussed above, tuning hyperparameters in SVM is normally treated as an optimization problem, single-objective or bi-objective. Most methods developed in the literature, then, focus on faster processes for optimization. The objective is always prediction accuracy for single-objective optimization or prediction accuracy combined with some metrics of model generalization ability for bi-objective optimization. These objectives are computed on the one-step ahead prediction, even for recursive multi-step ahead prediction. Indeed, recursive SVM for multi-step ahead prediction is based on the one-step ahead prediction and that is why the prediction accuracy on the one-step ahead prediction is the objective of the optimization problem to find the best hyperparameters during the training process. This, however, does not always guarantee satisfactory prediction results at multistep ahead prediction, as pointed out in [START_REF] Ben Taieb | Multiple-output modeling for multi-step-ahead time series forecasting[END_REF].

To the authors' knowledge, there are many good reviews and original research on tuning hyperparameters in SVM for one-step ahead prediction, whereas there has not been much published research reporting specifically on the problem of the objectives for hyperparameters tuning in SVM multi-step predictioin. In this paper, we try to address this by considering the prediction accuracy at all steps, also including the last step, during the training process for multi-step ahead prediction. Various possibilities for defining the objectives for tuning hyperparameters of recursive SVM are compared. Considering prediction accuracy, the objectives could be the accuracy (MSE or MAPE) on the first step, on all the steps and on the last step of the multi-step ahead prediction on the training dataset.

Model complexity is also considered as proposed in [START_REF] Igel | Multi-objective model selection for support vector machines[END_REF] to quantify the generalization ability: the smaller the model complexity is, the stronger its generalization ability is. The Mean of the Absolute value of the Derivatives (called MAD, in this paper) of the estimate function in SVM on all training data points is used to describe the robustness of the model under small perturbations in the inputs. Smaller MAD means stronger robustness of the SVM model.

Single-objective optimization for tuning hyperparameters considering only the prediction accuracy (for multi-step ahead prediction, as explained previously) and bi-objective optimization considering the prediction accuracy (on multi-step ahead prediction) and the model inherent characteristics (model complexity and MAD) are compared on the multisteps ahead prediction of one synthetic dataset and two real time series datasets.

One real series relates to the prediction of the leakage from the first seal of the Reactor Coolant Pump (RCP) in a Nuclear Power Plant (NPP). The other real series regards predicting wind speed. The experimental results obtained in these three cases show that 1) optimization considering the prediction accuracy on all the steps of the prediction horizon as objective of the optimization for tuning hyperparameters gives better results than the one of considering only the prediction accuracy on the first or last step of the prediction horizon; 2) the bi-objective optimization gives better results than the single-objective optimization.

The rest of the paper is structured as follows. Section 2 describes briefly the hyperparameters to tune in SVM and the objectives in the associated optimization. The case studies on one synthetic dataset and two real time series datasets are carried out in Section 3 to compare the different objectives. Based on the results obtained, some discussions are also given in this Section. Conclusions are drawn in Section 4.

Methodology

In this Section, the basic strategy for recursive multi-step ahead prediction by SVM is introduced at the beginning. Different objectives that can be used for optimizing the hyperparameters in SVM for one-step ahead prediction are presented. Their extension to multi-step ahead prediction are also presented in this Section.

Recursive multi-step ahead prediction

In recursive multi-step ahead prediction, the model is trained for one-step ahead prediction and the predicted value is appended to previous predicted values in input to predict the value for the next step. For simplicity, we consider a single-variable time series whose values until time 𝑇 are {𝑥 1 , 𝑥 2 , … , 𝑥 𝑇 }. The aim is to predict the future values of 𝑁 steps ahead. The inputs are 𝑀 previous values, i.e. the input-output pairs (𝒙 𝑡 , 𝒚 𝑡 ) are

([𝑥 𝑡-𝑀+1 𝑥 𝑡-𝑀+2 … 𝑥 𝑡 ], [𝑥 𝑡+1 𝑥 𝑡+2 … 𝑥 𝑡+𝑁 ]
). For a time series data with 𝑇 values, the number of constructed data points is 𝐾 = 𝑇 -𝑁 -𝑀 + 1. In Figure 1, 𝑥 ̂𝑡+𝑖 , 𝑖 = 1,2, … , 𝑁 are the predicted values for each step. Figure 1 shows the schematic illustration of the recursive prediction process. It is seen that the predictions on all the steps, except the first step, use the values predicted at the previous steps.

Support Vector Regression

In this paper, we consider regression by SVM with 𝜖-insensitive loss function. The basic idea of SVM is to map the training data points into a high-dimensional space, i.e. a feature space, where the relation between the training data points becomes linear.

Suppose we have 𝐾 data points of the form (𝒙 𝑡 , 𝑦 𝑡 ), 𝑡 = 1,2, … , 𝐾 and that 𝑘(𝒙 𝑖 , 𝒙) is the kernel function that represents the inner product of two training data vectors (points) in the feature space. SVM tries to estimate the function of interest 𝑓(𝒙) as the linear combination (as in Equation (1) below) that best approximates the underlying relation between the input and output, as represented by the data:

𝑓(𝒙) = ∑ (𝛼 𝑖 -𝛼 𝑖 * )𝑘(𝒙 𝑖 , 𝒙) 𝐾 𝑖=1 + 𝑏 . (1)
In Equation ( 1), the unknown Lagrange multipliers 𝛼 𝑖 and 𝛼 𝑖 * , for 𝑖 = 1,2, … , 𝐾 , can be calculated by solving the following optimization problem:

𝑀𝑎𝑥𝑖𝑚𝑖𝑧𝑒 - 1 2 ∑ (𝛼 𝑖 -𝛼 𝑖 * )(𝛼 𝑗 -𝛼 𝑗 * )𝑘(𝒙 𝑖 , 𝒙 𝒋 ) 𝐾 𝑖,𝑗=1 -𝜖 ∑ (𝛼 𝑖 + 𝛼 𝑖 * ) 𝐾 𝑖=1 + ∑ 𝑦 𝑖 (𝛼 𝑖 -𝛼 𝑖 * ) 𝐾 𝑖=1 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 ∑ (𝛼 𝑖 -𝛼 𝑖 * ) 𝐾 𝑖=1 = 0 𝑎𝑛𝑑 𝛼 𝑖 , 𝛼 𝑖 * ∈ [0, 𝐶] (2)
with 𝐶 being the penalty hyperparameter and 𝜖 the hyperparameter related to the loss function. There are also hyperparameters related to the kernel function 𝑘(𝒙 𝑖 , 𝒙 𝒋 ), e.g. 𝜎 in a Radial Basis Function (RBF), with 𝑘(𝒙 𝑖 , 𝒙 𝒋 ) = 𝑒^(-‖𝒙 𝑖 -𝒙 𝑗 ‖ 2 /(2𝜎 2 )). The values of these hyperparameters are critical for the prediction performance of the SVM model and, thus, hyperparameters tuning is very important for implementing SVM.

The unknown 𝑏 in Equation ( 1) can be calculated by imposing the so-called Karush-Kuhn-Tucker (KTT) conditions [START_REF] Karush | Minima of functions of several variables with inequalities as side constraints[END_REF][START_REF] Kuhn | Nonlinear Programming[END_REF].

Objectives for tuning hyperparameters

Normally the hyperparameters introduced in Section 2.2 are tuned by solving an optimization problem. As shown in Figure 1, recursive prediction is achieved by recursively applying one-step ahead prediction. In most of the published papers, the objective of the optimization problem for tuning hyperparameters in SVM for recursive multi-step ahead prediction is the same as for one-step ahead prediction, i.e. to minimize the prediction error of the one-step ahead prediction on the training dataset.

In this section, the possible objectives for hyperparameters tuning of SVM for one-step ahead prediction are considered and some of them are extended for multi-steps ahead prediction. In Section 3, experiments are carried out comparing these different objectives in the case studies to draw some practical conclusions.

Accuracy-related objectives

In the published papers, two widely used accuracy-related objectives are MSE and MAPE on the one-step ahead prediction:

𝑀𝑆𝐸 = 1 𝐾 ∑ ‖𝑥 ̂𝑡+1 -𝑥 𝑡+1 ‖ 2 𝐾 𝑡=1
(3)

𝑀𝐴𝑃𝐸 = 1 𝐾 ∑ ‖𝑥 ̂𝑡+1 -𝑥 𝑡+1 ‖ ‖𝑥 𝑡+1 ‖ 𝐾 𝑡=1 (4)
where 𝐾 is the number of training data points, 𝑥 ̂𝑡+1 and 𝑥 𝑡+1 are the predicted value and true value of the one-step ahead prediction, respectively and ‖∎‖ is the absolute value.

In the problem of multi-step ahead prediction, other accuracy-related objectives can be used. In order to distinguish these different accuracy-related objectives, the previous MSE and MAPE on one-step ahead prediction in Equations ( 3) and ( 4) are named MSE_one and MAPE_one.

For the multi-step ahead prediction, the accuracy on the first step is not the only objective;

actually, the prediction performance on the following steps is the most important. Thus, four other prediction accuracy-related objectives are proposed in this paper for optimizing the hyperparameters of SVM, which are the MSE and MAPE on the last step, i.e. MSE_last and MAPE_last, and the MSE and MAPE on all the steps, i.e. MSE_all and MAPE_all:

𝑀𝑆𝐸_𝑙𝑎𝑠𝑡 = 1 𝐾 ∑ ‖𝑥 ̂𝑡+𝑁 -𝑥 𝑡+𝑁 ‖ 2 𝐾 𝑡=1 , ( 5
) 𝑀𝐴𝑃𝐸_𝑙𝑎𝑠𝑡 = 1 𝐾 ∑ ‖𝑥 ̂𝑡+𝑁 -𝑥 𝑡+𝑁 ‖ ‖𝑥 𝑡+𝑁 ‖ 𝐾 𝑡=1 , ( 6
) 𝑀𝑆𝐸_𝑎𝑙𝑙 = 1 𝐾 ∑ ∑ ‖𝑥 ̂𝑡+𝑖 -𝑥 𝑡+𝑖 ‖ 𝑁 𝑖=1 2 𝐾 𝑡=1 , ( 7
) 𝑀𝐴𝑃𝐸_𝑎𝑙𝑙 = 1 𝐾 ∑ ∑ ‖𝑥 ̂𝑡+𝑖 -𝑥 𝑡+𝑖 ‖ ‖𝑥 𝑡+𝑖 ‖ 𝑁 𝑖=1 𝐾 𝑡=1 . ( 8
)
The previous four accuracy-related objectives for hyperparameters tuning give the information of interest on the trained SVM model for multi-step ahead prediction and remaining useful life estimation.

Model-characteristics-related objectives

The accuracy measures in Equations (3-8) can be used separately as objectives of singleobjective optimization or jointly with other model-characteristics-related objectives for tuning hyperparameters in a bi-objective optimization framework. One widely used modelcharacteristics-related objective is the model complexity of SVM, which describes the complexity (or simplicity) of the model. The simpler the model is, the stronger the model generalization ability is. The model complexity of SVM is calculated as [START_REF] Igel | Multi-objective model selection for support vector machines[END_REF]:

𝑀 𝑐𝑜𝑚𝑝 = 𝑅 2 ∑ ‖𝛼 𝑡 -𝛼 𝑡 * ‖ 𝐾 𝑡=1 , ( 9
)
where 𝛼 𝑡 and 𝛼 𝑡 * are the Lagrange multipliers, and R denotes the radius of the smallest sphere in the feature space containing all training data points. .

Then, MAD can be calculated as:

𝑀𝐴𝐷 = ∑ ‖ 𝑑𝑓(𝒙) 𝑑𝒙 | 𝒙 𝑡 ‖ 𝐾 𝑡=1 = ∑ ∑ ‖𝛼 𝑖 -𝛼 𝑖 * ‖ 𝐾 𝑖=1 ‖ 𝑑𝑘(𝒙 𝑖 ,𝒙) 𝑑𝒙 | 𝒙 𝑡 ‖ 𝐾 𝑡=1 . ( 13
)
As an example, for the RBF kernel function we have:

𝑀𝐴𝐷 = ∑ ∑ ‖𝛼 𝑖 -𝛼 𝑖 * ‖ 𝐾 𝑖=1 ‖ 𝒙 𝑖 -𝒙 𝑡 𝜎 2 ‖ 𝑒 - ‖𝒙 𝑖 -𝒙 𝑡 ‖ 2 2𝜎 2 𝐾 𝑡=1 . ( 14
)

Possible objectives for tuning hyperparameters in SVM

Considering the objectives introduced in Sections 2. In Section 3, experiments on real time series data are carried out to compare these different objectives.

Pareto-optimal solution set for bi-objective optimization

For tuning hyperparameters in SVM, the bi-objective optimization problem can be formulated considering the prediction accuracy and the characteristics of the SVM model, as introduced in Section 2.4. A Pareto-front results from the bi-objective optimization and solutions of compromise of the two objectives can be identified [START_REF] Kim | Adaptive weighted-sum method for bi-objective optimization: Pareto front generation[END_REF][START_REF] Blasco | A new graphical visualization of ndimensional Pareto front for decision-making in multiobjective optimization[END_REF]. For example, the "Min-Max" method provides a widely accepted way to find a compromised solution (Belegundu and Chandrupatla, 1999). Suppose 𝒈 𝑛 = (𝑔 𝑛,1 , 𝑔 𝑛,2 ), 𝑛 = 1,2, … , 𝑃 are the solution points in the bi-dimensional Pareto front and 𝑔 1 𝑚𝑎𝑥 , 𝑔 2 𝑚𝑎𝑥 denote the maximal value of the first and second objective functions on this front, respectively, i.e. the maximal accuracy-related objective value and modelcharacteristics-related objective value, respectively. The relative deviation of each objective of each solution (𝑔 𝑛,1 , 𝑔 𝑛,2 ), 𝑛 = 1,2, … , 𝑃 is calculated as 𝑧 𝑛,𝑖 = (𝑔 𝑖 𝑚𝑎𝑥 -𝑔 𝑛,𝑖 )/𝑔 𝑖 𝑚𝑎𝑥 𝑖 = 1, 2 and 𝑧 𝑛 = min (𝑧 𝑛,1 , 𝑧 𝑛,2 ) represents the solution (𝑔 𝑛,1 , 𝑔 𝑛,2 ) . The optimal compromised solution 𝒈 for the bi-objective optimization is arg(max (𝑧 𝑛 )) , 𝑛 = 1,2, … , 𝑃. Such a solution represents the "center" of the Pareto-front (Figure 2).

Numerical experiments

Case studies of one synthetic dataset and two real time series datasets are considered with respect to the optimization problem of tuning hyperparameters in recursive SVM. One real case study considers the leakage from the first seal of RCP in NPP and the other one the prediction of wind speed.

The procedure of the experiments is shown in Figure 3. The time series data 𝑎(𝑡) is reconstructed into data points that can be treated with SVM. Partial autocorrelation analysis [START_REF] Rodgers | Thirteen Ways to Look at the Correlation Coefficient[END_REF]

Prediction of one synthetic time series data

In this section one synthetic dataset is used to test different objectives for optimizing SVM hyperparameters. The synthetic data, as shown in Figure 4 is a time series data of sinc function added with a certain noise. The results are shown in Tables 1 and2.

In all the Tables in Sections 3

Prediction of the leakage from the RCP of a NPP

RCP is a very important component of a NPP, as it must provide sufficient coolant to transport the heat from the reactor core to the steam generator. Radioactive coolant leaked from RCP may endanger the personnel and equipment in NPP. In the extreme case of loss of coolant accident, the reactor core could also melt down if proper safety actions are not taken. Thus, the prediction of leakage is critical for the operation of a NPP. 3 and4, using different objectives for optimizing the SVM hyperparameters during the training process.

Wind speed prediction

As more wind power generation is connected to the electrical network, prediction of wind speed becomes more and more important for effective electricity generation planning and management, while guaranteeing network stability and end-users service satisfaction [START_REF] Ak | NSGA-IItrained neural network approach to the estimation of prediction intervals of scale deposition rate in oil & gas equipment[END_REF].

In this case study, we consider the wind speed data measured of Regina, Saskatchewan, a region of central Canada (Canadian Weather Office, 2012) over a period of two months, from 1st of February 2012 to 31st of March 2012. According to a partial autocorrelation analysis, wind speed in the last two 1-hour time steps are used as inputs to predict the wind at the next five 1-hour time steps. The total data set includes 1438 values (as shown in Figure 6) [START_REF] Ak | NSGA-IItrained neural network approach to the estimation of prediction intervals of scale deposition rate in oil & gas equipment[END_REF]. The first 800 of the total constructed 1432 data points are used as training dataset and the rest as test dataset. The prediction results are compared by using different objectives for optimizing the SVM hyperparameters during the training process, as shown in Tables 5 and6.

Discussions

Several insights can be drawn from the results of the three case studies. (1) Single-objective optimization based on MSE for tuning SVM hyperparameters for leakage prediction, leads to prediction results that are not satisfactory, as the MSE during the search for optimal hyperparameters values is strongly influenced by the bad predictions of only few training data points. This is one drawback of using MSE as accuracy measure, because the MSE can be dominated by relatively big errors on only a small number of data points and the optimization efforts are, then, specifically dedicated to improving these errors. Figure 7 is an example of this drawback using MSE as the objective for tuning SVM hyperparameters. The solid line without marker is the output of the training data points. If the MSE on the training data points is the sole objective for tuning SVM hyperparameters, the first set of hyperparameters with a MSE of 0.0013 is slightly better than the second set of hyperparameters with a MSE of 0.0018. But the fact is that the second set of hyperparameters gives better results than the first set, except for the really bad prediction on the 15 th and 24 th data points. In such case, the MSE as objective does not choose the best set (the second set of hyperparameters). (2) For predictions of the three case studies, Figures 8,9,10,11,12 and 13 show the comparisons of the prediction results of each line of Tables 1, 2, 3, 4, 5 and 6, i.e. the prediction accuracy obtained by SVM when tuning the hyperparameters calculated on different time horizons as explained in Section 2.3.1 e.g. comparing the prediction accuracy obtained by using bi-objective optimization with model complexity and MSE with singleobjective MSE at the first step, at all the prediction steps or at the last step as objective, for tuning hyperparameters. It is observed that compared to using only the prediction accuracy on the one-step ahead prediction, for most of the cases the prediction accuracy on the test dataset can be improved by using the prediction accuracy (MSE or MAPE) on all the steps or on the last step as optimization objective for tuning the SVM hyperparameters. This 8, 9, 10, 11, 12 and 13, it can also be observed that in the experiments considered, using the prediction accuracy on all steps as objective in the optimization problem for tuning the SVM hyperparameters leads, in general, to better results than using the prediction accuracy only on the last step as objective of the optimization. (3) Figures 14,15,16,17,18 and 19 show the comparisons of the prediction results on the test datasets using single-objective and bi-objective optimization for the SVM hyperparameters tuning, with the same prediction-accuracy-related objective, e.g. MSE (i.e. MSE on all steps, MSE on all steps and model complexity, MSE on all steps and MAD). Generally, the bi-objective optimization gives better results on the test datasets than the single-objective optimization, as simply minimizing the prediction accuracy on the training dataset can fall into overfitting of the training dataset. For the two bi-objective optimizations, the combination of prediction accuracy with MAD, which characterizes the robustness of the model, gives more stable and accurate results than the combination with model complexity. This can be explained by the inherent meaning of the two model characteristics: MAD characterizes the robustness of the model against small noises, which can decrease the influence of the prediction error that may cumulate during the recursive prediction process; model complexity characterizes the generalization ability of the model, which guarantees the prediction performance for different inputs. (4) From the discussions above, we can draw the conclusion that for recursive multi-step ahead prediction by SVM, the combination of model complexity or MAD with prediction accuracy on all the steps as objectives of the optimization for hyperparameters tuning give the best results, with MAD guaranteeing more robustness than model complexity.

Conclusions

In this paper, we have considered the objectives for optimizing the hyperparameters of SVM for recursive multi-step ahead prediction. The candidate objectives have been presented and discussed, distinguishing between accuracy-related and modelcharacteristics-related objectives, and considering different step horizons (single step, all steps and last step). Experiments on one synthetic time series data and two real time series data (leakage from RCP in NPP and wind speed) show that the bi-objective optimization combining the MAD and the prediction accuracy (MSE or MAPE) on all the steps is a preferable choice for SVM hyperparameters tuning for recursive multi-step ahead prediction.

Fig. 1

 1 Fig. 1 Schematic illustration of recursive multi-steps ahead prediction using SVM.

  3 and 2.4, the possible optimization for tuning hyperparameters in SVM during the training process can be formed as singleobjective or bi-objective. Single-objective optimization considers only the prediction accuracy, which includes MSE_one, MSE_last, MSE_all, MAPE_one, MAPE_last and MAPE_all. Bi-objective optimization combines the prediction accuracy and the characteristics of the model to find compromised hyperparameters values, i.e. MSE_one and model complexity, MSE_last and model complexity, MSE_all and model complexity, MSE_one and MAD, MSE_last and MAD, MSE_all and MAD, MAPE_one and model complexity, MAPE_last and model complexity, MAPE_all and model complexity, MAPE_one and MAD, MAOE_last and MAD, MAPE_all and MAD.

Fig. 2

 2 Fig. 2 Best compromised solution 𝒈 selected from the bi-dimensional Pareto-front using "Min-Max" method.

  Fig. 3 Flow chart of the experiment.

  .1, 3.2 and 3.3, the first column and the first line list the possible objectives that can be used for tuning hyperparameters during training. The second line shows the prediction results on the test dataset using only accuracy-related objectives, and the third and fourth lines show the prediction results on the test dataset using biobjective optimization, i.e. combining the prediction accuracy on the training dataset with the model complexity or the MAD.

Fig. 1

 1 Fig. 1 Normalized values of synthetic time series data.

Figure 5

 5 Figure 5 is a time series dataset which contains 1553 values, measured every four hours. The values are normalized for confidentiality reasons. The objective of the analysis is the ten-step ahead prediction. The leakage in the last 10 4-hour time steps are used to predict the leakage values in the next 10 4-hour time steps. A set of 1534 data points are constructed as explained in Section 2.1. The first 1234 data points are used as training dataset and the remaining 300 data points form the test dataset. The prediction results, including MSE and MAPE of the prediction on the last step for the test dataset are shown separately in Tables3 and 4, using different objectives for optimizing the SVM

Fig. 5

 5 Fig. 5 Normalized values of the leakage measured at time steps of 4 hours.

Fig. 6

 6 Fig. 6 Normalized values of the wind speed with a time step of 1 hour.

Fig. 7

 7 Fig. 7 Illustration of the possible drawback of using MSE as objective for tuning SVM hyperparameters.

  seems to confirm the observation in Taieb et al. (2010) that the relation between the input and the one-step ahead prediction cannot guarantee the relation between the input and the multi-step ahead prediction. If the training focuses only on minimizing the prediction error of the one-step ahead prediction, the trained model can be trapped in overfitting such prediction, and the prediction accuracy on the following steps may decrease. From Figures

Fig. 8

 8 Fig. 8 Comparison of MSE on the test dataset of synthetic data prediction considering the prediction accuracy on different time horizons, i.e. one step, all steps and last step as optimization objective for tuning hyperparameters.

Fig. 9

 9 Fig. 9 Comparison of MAPE on the test dataset of synthetic data prediction considering the prediction accuracy on different time horizons, i.e. one step, all steps and last step as optimization objective for tuning hyperparameters.

Fig. 10

 10 Fig. 10 Comparison of MSE on the test dataset of leakage prediction considering the prediction accuracy on different time horizons, i.e. one step, all steps and last step as optimization objective for tuning hyperparameters.

Fig. 12

 12 Fig. 12 Comparison of MSE on the test dataset of wind speed prediction considering the prediction accuracy on different time horizons, i.e. one step, all steps and last step as optimization objective for tuning hyperparameters.

Fig. 13

 13 Fig. 13 Comparison of MAPE on the test dataset of wind speed prediction considering the prediction accuracy on different time horizons, i.e. one step, all steps and last step as optimization objective for tuning hyperparameters.

Fig. 14

 14 Fig. 14 Comparison of MSE on the test dataset of synthetic data prediction considering accuracy-related measures in combination with model complexity and MAD, as optimization objectives for tuning SVM hyperparameters.

Fig. 16

 16 Fig. 16 Comparison of MSE on the test dataset of leakage prediction considering accuracy-related measures in combination with model complexity and MAD, as optimization objectives for tuning SVM hyperparameters.

Fig. 17

 17 Fig. 17 Comparison of MAPE on the test dataset of leakage prediction considering accuracy-related measures in combination with model complexity and MAD, as optimization objectives for tuning SVM hyperparameters.

Fig. 18

 18 Fig. 18 Comparison of MSE on the test dataset of wind speed prediction considering accuracy-related measures in combination with model complexity and MAD, as optimization objectives for tuning SVM hyperparameters.

Fig. 19

 19 Fig. 19 Comparison of MAPE on the test dataset of wind speed prediction considering accuracy-related measures in combination with model complexity and MAD, as optimization objectives for tuning SVM hyperparameters.

  𝑅 = √∑ 𝛽 𝑖 𝑘(𝒙 𝑖 , 𝒙 𝑖 )The estimate function in Equation (1) is differentiable if the kernel function is differentiable, which is the case most times, e.g. RBF, polynomial kernel function etc. Thus, if the kernel function is differentiable, the derivative of the estimate function on each training data point characterizes the robustness of the trained SVM model on the training data point under small noise. The Mean Absolute Derivative (MAD), proposed in this paper for the first time, can be used to characterize the robustness of the model under small noise.

			𝐾 𝑖=1	-∑	𝐾 𝑖,𝑗=1	𝛽 𝑖 𝛽 𝑗 𝑘(𝒙 𝑖 , 𝒙 𝑗 )	,	(10)
	where the values of 𝛽 𝑖 for 𝑖 = 1,2, … , 𝐾 are calculated by solving the quadratic
	optimization problem:
	𝑀𝑎𝑥𝑖𝑚𝑖𝑧𝑒 ∑ 𝛽 𝑖 𝑘(𝒙 𝑖 , 𝒙 𝑖 ) 𝐾 𝑖=1 𝑆𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 ∑ 𝛽 𝑖 𝐾 𝑖=1 = 1 𝑎𝑛𝑑 𝛽 𝑖 > 0, 𝑖 = 1,2, … , 𝐾 -∑ 𝛽 𝑖 𝛽 𝑗 𝑘(𝒙 𝑖 , 𝒙 𝑗 ) 𝐾 𝑖,𝑗=1	(11)
	𝑑𝑓(𝒙) 𝑑𝒙	=	𝑑(∑ (𝛼 𝑖 -𝛼 𝑖 * )𝑘(𝒙 𝑖 ,𝒙) 𝐾 𝑖=1 +𝑏) 𝑑𝒙	= ∑ (𝛼 𝑖 -𝛼 𝑖 * ) 𝐾 𝑖=1	𝑑𝑘(𝒙 𝑖 ,𝒙) 𝑑𝒙

The minimal radius R can be calculated by:

The derivative of the estimate function on the training data point (𝒙, 𝑦) is:

Table 1

 1 MSE of the prediction results on the last step for the test dataset with respect to the different objectives used for optimizing the SVM hyperparameters for synthetic data prediction.

		MSE_one MSE_all MSE_last MAPE_one MAPE_all MAPE_last
	Single-objective	0.0658	0.0062	0.0056	0.0415	0.0072	0.0071
	Bi-objective with model	0.0415	0.0055	0.0055	0.0168	0.0066	0.0127
	complexity
	Bi-objective with MAD	0.0307	0.0055	0.0054	0.0070	0.0066	0.0127

Table 2

 2 MAPE of the prediction results on the last step for the test dataset with respect to the different objectives used for optimizing the SVM hyperparameters for synthetic data prediction.

		MSE_one MSE_all MSE_last MAPE_one MAPE_all MAPE_last
	Single-objective	0.4316	0.0319	0.0333	0.2711	0.0326	0.0323
	Bi-objective with model	0.2711	0.0328	0.0328	0.1021	0.0314	0.0833
	complexity
	Bi-objective with MAD	0.1941	0.0328	0.0326	0.0034	0.0314	0.0833

Table 3

 3 MSE of the prediction results on the last step for the test dataset with respect to the different objectives used for optimizing the SVM hyperparameters for leakage prediction.

		0.9
		0.8
		0.7
	Normalized leakage values	0.4 0.5 0.6
		0.3
		0.2
		0.1	0	200	400	600	800	1000	1200	1400	1600
							Time steps

Table 4

 4 MAPE of the prediction results on the last step for the test dataset with respect to the different objectives used for optimizing the SVM hyperparameters for leakage prediction.

		MSE_one MSE_all MSE_last MAPE_one MAPE_all MAPE_last
	Single-objective	2.5034	4.9544	8.2861	0.1067	0.1014	0.1014
	Bi-objective with model	0.1212	0.1173	0.1281	0.1073	0.1085	0.1082
	complexity
	Bi-objective with MAD	0.1204	0.1152	0.1252	0.1067	0.1014	0.1000

Table 5

 5 MSE (10^-4) of the prediction results on the last step for the test dataset with respect to the different objectives used for optimizing the SVM hyperparameters for wind speed prediction.

		MSE_one MSE_all MSE_last MAPE_one MAPE_all MAPE_last
	Single-objective	9.5748	5.1991	5.2952	9.5748	6.2335	6.2335
	Bi-objective with model	7.7724	6.0290	6.3404	5.9548	5.6379	5.6379
	complexity
	Bi-objective with MAD	5.8448	5.2072	5.2816	6.0042	5.3199	5.3199

Table 6

 6 MAPE of the prediction results on the last step for the test dataset with respect to the different objectives used for optimizing the SVM hyperparameters for wind speed prediction.

		MSE_one MSE_all MSE_last MAPE_one MAPE_all MAPE_last
	Single-objective	0.0062	0.0048	0.0049	0.0062	0.0050	0.0050
	Bi-objective with model	0.0059	0.0055	0.0072	0.0052	0.0058	0.0058
	complexity
	Bi-objective with MAD	0.0050	0.0049	0.0050	0.0050	0.0047	0.0047