Guy-Michel Cloarec

John Ringwood

Modelling of a Four-Effect Evaporator using Hybrid Neural Networks

Keywords: System Identification, Neural Network Modelling, Statistical Methods, Multi-step Ahead Prediction, Principal Component Analysis, Industrial Evaporator

This paper demonstrates the power of neural networks models when used in conjunction with statistical theory techniques. A modelling methodology is presented and illustrated by an application to an industrial four-effect evaporator used in dairy industry. The hybrid neural network model will enable multi-step ahead prediction for multivariable (MIMO) systems, essential to perform industrial process control such as Model Predictive Control (MPC). The modelling methodology based on a probabilistic approach will implement techniques such as statistical bootstraps, principal component analysis and network committees.

I. INTRODUCTION

The purpose of this paper is to demonstrate the performance of a hybrid neural network modelling methodology by applying concepts and techniques issued from the statistical theory field. The underlying probabilistic approach is considered to address the critical issues such as topology determination and generalization performance. The method will be illustrated and validated on a real life nonlinear multivariable (MIMO) industrial process, a four-effect (stage) evaporator. The system and the corresponding data sets has been chosen since it has already been used as a benchmark problem for regression modelling methods [START_REF] Zhu | Comparison of Three Classes of Identification Methods[END_REF].

In dairy industry [START_REF] Bylund | Dairy Processing Handbook[END_REF], evaporators are used to concentrate products (such as milk) before the spray drying process. The basic processing unit is the evaporator represented on Figure 1.

The basic principle of the four-effect evaporator (see Figure 1) [START_REF] Bylund | Dairy Processing Handbook[END_REF] consists in preheating the product before entering the effect (using the heat of the extracted vapor of the previous effect). The product flows down as a film obtained after a distribution plate. The film flows down on the evaporation tubes heated by condensing vapour on the outside of the tubes. A two-phase flow of concentrate and vapour reaches the separator where product and vapour are separated. To reduce energy consumption and heat damage, the evaporation is performed under vacuum.

In multiple effect (stage) systems, represented on Figure 2 [2], the evaporators are operated in series at increasing level of vacuum (decreasing temperatures). The concentrated product is pumped into the next distribution plate. The extracted vapour is compressed in thermocompressors and is used as a heat medium in the next evaporator.

The cascading of the evaporators leads to potential energy savings but also increases the performance of the evaporation while reducing the risk of damaging the product. Once the milk is concentrated, the product is directed to a dryer which can operate only if the product is concentrated enough.

The four-effect evaporator is a complex nonlinear system. One of the cause of nonlinearity is the cascading of the evaporators and their capacity to hold quantities of product. By doing so, they can act as buffers leading to nonlinear behaviour, especially for the flow of product. The nonlinearities and the complexity of the input and outputs make the realization of optimal control more difficult. Modern control methods such as Model Predictive control (MPC) are particularly suitable to perform multicriteria control and optimize operation costs while ensuring safeguards for critical parameters. MPC requires multi-step ahead prediction modelling techniques that can treat nonlinear multivariable industrial processes. The core of MPC (and the main challenge) is to obtain a reliable multi-step ahead prediction model. The nonlinear nature of the problem lead to use nonlinear model such as neural network models and particularly a hybrid neural network model developed for multistep ahead prediction problems.

The data set corresponding to a four-effect evaporator of the Netherlands Institute for Diary Research NIZO and was obtained from the DAISY data base [START_REF]DAISY: Database for the Identification of Systems[END_REF]. The process is modeled as a three-input three-output process with disturbances. The input signals are PRBS and are used to ensure that the input space span the entire operation modes. The sample rate is 30 seconds and the training and validation sets contain 3000 samples each.

II. HYBRID NEURAL NETWORK MODEL

The modelling methodology is based on the use of committees of network. In multi-step ahead prediction, the use of the previous predictions leads to highly correlated prediction error. As the prediction horizon increases, the prediction error grows exponentially. The committees of network are used to produce a set uncorrelated predictions. These predictions are combined to produce a more accurate single-step ahead prediction but also will improve the multi-step ahead prediction as the prediction error will no longer be correlated.

The rational for that approach is the statistical concept that a set of uncorrelated models will produce predictions centered on the desired answer. This requires uncorrelated model that are obtained by training each individual with different training sets. In that sense, it is a different solution proposed for the conditioning approach considered in Bayesian methods [START_REF] Mackay | Bayesian Methods for Adaptive Models[END_REF].

The first statistical technique used to obtain uncorrelated models is the so called statistical bootstrap [START_REF] Shao | The Jacknife and Bootstrap[END_REF], [START_REF] Cloarec | Multistep Ahead Prediction of the Sunspot Series using Hybrid Neural Network Models[END_REF]. It is a resampling method used to decrease the dependencies on the parameter optimization and the model performance estimates. The rational for its use is the same that justifies PRBS for system identification. It can be used to overcome limited training data sets [START_REF] Cloarec | Incorporation of Statistical Methods in Multi-step Neural Network Prediction Models[END_REF] or in our case, to reduce the computation times. A bootstrap set is obtain by random drawing with replacement on the training set.

The bootstraps are used at all steps of the modeling. They will be particularly useful to provide generalization estimates. It has been shown [START_REF] Hansen | Early Stop Criterion from the Bootstrap Ensemble[END_REF] that a performance criterion evaluated over several other bootstraps can be considered as a good estimate of the generalization. With such an estimate, it is no longer required to divide the data into the classical training, test and validation sets. With bootstraps, the former training and test sets can both be used for training. In that case, by providing larger data sets, the bootstraps contribute to improve the overall generalization performance of the model.

Another technique implemented will be the Principal Component Analysis (PCA) [START_REF] Jobson | Applied Multivariate Data Analysis[END_REF]. Mathematically similar to a singular value decomposition (SVD), the results will be used to perform the orthogonalization, the feature extraction and the reduction of the input space. All these characteristics lead to feed the network with more relevant information, lead to better generalization (noise filtering) and computation times.

The PCA is not applied to PRBS input because of their uncorrelated nature. We will see that although the use of PRBS inputs is relevant in linear models, they can unefficient in nonlinear models especially when preprocessing methods like PCA are used.

The PCA are also used to analyze the output of each layer. The optimal topology is then the topology that would perform the compression of the inter-layer signals.

Further statistical methods will be used in the effective input determination. Statistical inference will be applied to decision making based on the performance of linear regression models but also in the weight elimination process.

While the previous techniques can be applied to various identification methods, the key feature of our methodology is the use of committees. The concept of network committees, also called ensembles or redundant networks [START_REF] Cloarec | Multistep Ahead Prediction of the Sunspot Series using Hybrid Neural Network Models[END_REF], [START_REF] Mammone | When Networks Disagree : Ensemble Methods for Hybrid Neural Networks[END_REF], comes from the field of statistical theory and is a natural extension of the estimates accuracy improvement through statistical data analysis. The committees are collections of finite numbers of neural networks trained on different sample sets, with different initial conditions or learning criteria. Their use is known to improve the generalization performance by decreasing the effect of spatial and temporal dependencies.

Once trained, the network committees are used to improve the decision making process of the structural determination. By enabling statistical inference, the model selection process was made on the basis of confidence limits. Furthermore, since the committees prediction errors are uncorrelated, they can be combined using a network combiner to give better predictions but The use of committees for single-step ahead predictions gives good results (see Table I) and the same principle can be used to obtain multi-step ahead predictions. Such predictions are implemented by the committees for increasing horizon of predictions. The statistical analysis enable to determine the optimal horizon of prediction. The combiner network is then trained using the multi-step ahead prediction as inputs. We have already shown in [START_REF] Cloarec | Multistep Ahead Prediction of the Sunspot Series using Hybrid Neural Network Models[END_REF], [START_REF] Cloarec | Incorporation of Statistical Methods in Multi-step Neural Network Prediction Models[END_REF] that the use of neural networks to perform the combination considerably improves the final multi-step ahead predictions.

III. RESULTS AND COMPARISONS

The three outputs systems is modeled by three MISO models, noted DMC-model, FP-model and TP-model for the dry matter content model, flow of product model and temperature of product model, respectively. Although it would be possible to implement MIMO models, the separation enable to identify clearly the different dynamics of the system. To perform multistep ahead prediction, the one-step ahead prediction of each output is calculated and used to implement the next prediction.

The first step of the modelling is the determination of the causal inputs. This is achieved by combining correlation and spectral analysis to linear model performance based statistical decision making processes. The inputs corresponding to the output variables are processed using PCA and only the most significant components are used as inputs in the networks.

The structure of the DMC-model input space can be expressed by :

Input(1, 2, 3) = P CA (1,2,3) (DM C(k -1, k -2, • • • , k -13)) Input(4, 5, 6) = P CA (1,2,3) (F P (k -1, k -2, • • • , k -5)) Input(7, 8) = P CA (1,2) (T P (k -1, k -2, • • • , k -5)) Input(9, 10, 11, 12) = F F (k -4, k -5, • • • , k -8)) Input(13, 14, 15) = V F (k -6, k -7, k -8) Input(16, 17, 18) = CW F (k -1, k -2, k -3)
for the DMC-model, the first three inputs are the three first principal scores of the set made of the past 13 dry matter content of the product.

Similarly, we have the input spaces of the FP-model : [START_REF] Cloarec | Incorporation of Statistical Methods in Multi-step Neural Network Prediction Models[END_REF][START_REF] Cloarec | Probabilistic Approach for Hybrid Neural Network Modeling[END_REF][START_REF] Hansen | Early Stop Criterion from the Bootstrap Ensemble[END_REF][START_REF] Jobson | Applied Multivariate Data Analysis[END_REF][START_REF] Mammone | When Networks Disagree : Ensemble Methods for Hybrid Neural Networks[END_REF][START_REF] Ljung | System Identification : Theory for the User[END_REF][START_REF] Van Overschee | N4SID Two Subspace Algorithms for Identification of Combined Deterministic-Stochastic Systems[END_REF]

Input(1, 2) = P CA (1,2) (DM C(k -1, k -2, • • • , k -8)) Input(3, 4, 5, 6) = P CA (1,2,3,4) (F P (k -1, k -2, • • • , k -8)) Input
= F F (k -2, k -3, • • • , k -8)
For the TP-model, the input space is :

Input(1, 2) = P CA (1,2) (T P (k -1, k -2, • • • , k -6)) Input(3, 4, 5, 6) = F F (k -2, • • • , k -5) Input(7, 8, 9, 10) = CW F (k -2, • • • , k -6)
The analysis of the input space indicates that the product spends between 2 and 5 minutes in the system. The two values indicate that the four stages act like buffers. The transit time is minimal when the evaporators are full while it is minimal when the systems is empty. The accuracy of that transit times are downgraded by the use of PRBS with unsufficient low frequencies. This explains the small correlation observed between the flow of feed and the output variables. Since the flow of feed plays an important causal role for the three outputs, and because the PRBS are uncorrelated inputs, the flow of feed input has to be used used with a higher order of regression.

Other input signal types such as ramps or steps may be more suitable to identify some characteristics of the buffer effect. Another possibility would be to evaluate the quantity of product in the system by balancing the input and output flows. Once identified, such variable could make a rich in information input. In any case, whatever the rational for general method such as PRBS, it is essential to adapt the modelling to the specificities of the system, in other words, to implement the understanding of the system.

As a consequence of the use of PRBS functions, the statistical decision methods used for topological, bootstrap, neuron pruning and weight elimination determination processes had to operate with poor confidence limits.

Another consequence of the lack of accuracy on the inputs is that the modelling will tend to emphasize correlated inputs over causal inputs. This can be a problem for multistep ahead forecasting since very good performance can be obtained in single-step ahead prediction only relying on inertial correlation. However, as the prediction horizon H increases, the model performance collapses.

Despite all the drawbacks of the data sets, a first network committee of 50 networks was trained. Form these, two subcommittees of networks optimized for error variance and error bias was selected. A network combiner was trained to perform the combination of the first step ahead prediction of the subcommittees, forming the hybrid neural network. The combination of the prediction enabled to improve by about 10 % the prediction MSE. Moreover, the statistics on the subcommittees predictions (variance) were used as upper bounds for the confidence on the combined predictions. The initial 10 % gain proved to be even more important for the multi-step ahead prediction.

Once the three hybrid neural network models were obtained, the multistep ahead prediction process was implemented. For each prediction horizon, three combiners were trained. It is important to have different combiners for each horizon of prediction since the performance of the subcommittee changes as the inertial correlation decreases. As H increases, the networks optimized for causal inputs perform better requiring to change their respective weight in the predictions combination.

The multi-step ahead prediction was implemented to prediction horizons corresponding to the highest regression order of the predicted outputs. In other words, at the exception of the inputs, a H ∞ predictor was obtained.

The results of the single-step ahead predictions are presented in Table I with comparisons with the results obtained on three classes of identification methods [START_REF] Zhu | Comparison of Three Classes of Identification Methods[END_REF]. In [START_REF] Zhu | Comparison of Three Classes of Identification Methods[END_REF], three linear discrete time identification methods were compared and the application to the evaporator were performed by the authors. The prediction error method (PEM) is the most classical identification method (ARX, ARMAX, ...) [START_REF] Ljung | System Identification : Theory for the User[END_REF]. N4SID refers to the numerical algorithms for Subspace State Space System Identification [START_REF] Van Overschee | N4SID Two Subspace Algorithms for Identification of Combined Deterministic-Stochastic Systems[END_REF]. ASYM is the asymptotic method based on the asymptotic theory of Ljung [START_REF] Zhu | Comparison of Three Classes of Identification Methods[END_REF] and uses frequency domain criterion. The order denotes the McMillan degree of model.

In Table II, the H ∞ prediction models performances are represented. To enable comparisons, the performance criterion is the normalized mean square error (MSE). The hybrid models perform significantly better that the three methods presented. Moreover, while the three methods were in turn most efficient for different outputs, the hybrid model performs better for any of the output.

The comparisons of the H ∞ models, although they show better performance, are not as much significant as the single step ahead prediction. This is probably due to the uncertainty in the identification of the causal inputs. Although the method have shown a good ability to perform multi-step ahead predictions, the improvements are slightly under expectations. It is important to notice however that the poor performance of the multi-step ahead predictions, whatever the method used, are mainly explained by the prediction error on the flow of product. As already mentioned, an improvement in the input signals would lead to far better multi-step ahead predictions.

In order to illustrate the prediction performance, we can examine the predictions for one model. Figure 4 represents the single step ahead prediction on the Dry Matter Content of product (DMC-model) while Figure 5 represents the H ∞ predictions.

IV. CONCLUSION

The hybrid neural network modelling methodology have been applied successfully to the four-effect evaporator. The H ∞ is accurate enough to implement a Model Predictive Control strategy. Moreover, the availability of upper bounds of the confidence limits make possible to ensure that some critical parameters, such as excessive temperature that would damage the product, will stay within safe limits.

The modelling does not only provides prediction capabilities but also some understanding of the system behaviour. The insight gained should be used as a feedback to improve the iterative modelling process. Therefore, the next step would be to perform another data acquisition campaign with signals more suitable to provides valuable information for the identification process. Concerning the modelling method itself, the results would suggests the need for improvement in the causal input selection. Although the use of linear model based criterion is a rational choice, the use of multi-step ahead prediction criteria would probably help to differentiate further correlation from causality.

In any case, the angular stone of modeling is the human understanding of the system and the implementation of that knowledge in the modelling process.

Fig. 1 .

 1 Fig. 1. Single-effect Evaporator.

Fig. 2 .

 2 Fig. 2. Four-effect Evaporator with Mechanical Vapour Compression.

• input 1 :

 1 Flow of Feed (FF) • input 2 : Vapour Flow to First Effect Evaporator Stage (VF) • input 3 : Cooling Water Flow to Condenser (CWF) • output 1 : Dry Matter Content of the Product (DMC) • output 2 : Flow of Product (FP) • output 3 : Temperature of Product (TP) Other variables such as dry matter content of feed, temperature of feed, temperature of cooling water entering the condenser and surrounding temperatures are considered as disturbances.

Fig. 3 .

 3 Fig. 3. Multi-step Prediction Neural Network Structure.

Fig. 4 .

 4 Fig.[START_REF] Mackay | Bayesian Methods for Adaptive Models[END_REF]. Observed (solid) and Prediction (dashed) and 90 per cent Confidence Limits (dotted).

Fig. 5 .

 5 Fig. 5. Observed (solid) and Prediction (dashed) and 90 per cent Confidence Limits (dotted).

TABLE II COMPARISONS

 II OF THE NORMALIZED MSE FOR H∞ FORECASTING ON THE THREE OUTPUTS.

	Method	PEM	N4SID ASYM	ASYM	Hybrid NN
				(MISO) (reduced MISO)
	Order	13	13	22	12	13
	Dry Matter Content	0.3464 0.3987	0.3700	0.3718	0.3205
	Flow of Product	0.4033 0.4048	0.4115	0.4102	0.3812
	Temperature of Product	0.2939 0.3003	0.3581	0.3582	0.2365


Furong Ye
email: f.ye@liacs.leidenuniv.nl

Carola Doerr
email: carola.doerr@lip6.fr

Thomas Bäck

Leveraging Benchmarking Data for Informed One-Shot Dynamic Algorithm Selection

Keywords: Genetic Algorithms, Dynamic Algorithm Selection, Black-Box Optimization, Evolutionary Computation

A key challenge in the application of evolutionary algorithms in practice is the selection of an algorithm instance that best suits the problem at hand. What complicates this decision further is that different algorithms may be best suited for different stages of the optimization process. Dynamic algorithm selection and configuration are therefore well-researched topics in evolutionary computation. Two different settings are classically considered: hyper-heuristics and parameter control studies typically assume a setting in which the algorithm needs to be chosen and adjusted during the run, without prior information, other approaches such as hyper-parameter tuning and automated algorithm configuration assume the possibility of evaluating different configurations before making a final recommendation. In practical applications of evolutionary algorithms we are often in a middle-ground between these two settings, where one needs to decide upon the algorithm instance before the run ("oneshot" setting), but where we have (possibly lots of) data available on which we can base an informed decision.

We analyze in this work how such prior performance data can be used to infer informed dynamic algorithm selection schemes for the solution of pseudo-Boolean optimization problems. Our specific use-case considers a family of genetic algorithms.

CCS CONCEPTS

• Theory of computation → Bio-inspired optimization.

INTRODUCTION

It is very well known that different algorithms or different instantiations of the same algorithm are best suited for different problems and even for different stages of the optimization process. Automated algorithm selection [START_REF] Kerschke | Automated Algorithm Selection: Survey and Perspectives[END_REF] as well as dynamic parameter selection [START_REF] Karafotias | Parameter Control in Evolutionary Algorithms: Trends and Challenges[END_REF] are therefore intensively studied meta-optimization problems in evolutionary computation. However, the former has a strong requirement on being able to run different algorithms (or algorithm configurations) prior to making a decision which algorithm to apply to the problem at hand. Parameter control and related concepts (including hyper-heuristics, adaptive operator control, etc.), in contrast, assume that the selection has to be made on the fly, without leveraging existing data from previous or related runs. With the rise of artificial intelligence methods, evolutionary computation is currently facing a paradigm shift, in that we aim to actively exploit existing performance data to select which algorithms to apply, and how to possibly adjust them during the run. We are, however, still far from achieving a fully automated informed online selection.

We study in this work how well we can predict from existing performance data which algorithm instances to combine for a given problem at hand. While we do allow for switching between different algorithms, the decision when to switch has to be made prior to the run, and depends, in our case, on the solution quality of the evaluated solution candidates. More precisely, we use the benchmarking data from our study [START_REF] Ye | Benchmarking a (𝜇 + 𝜆) Genetic Algorithm with Configurable Crossover Probability[END_REF] as starting point to investigate, for each of the 25 individual problems, how well we can predict which single-switch algorithm combinations would show good performance. For some functions we easily obtain algorithm combinations that outperform the best static algorithms. For other functions the results are rather mixed. On three functions, none of the 100 tested single-switch algorithm combinations was able to outperform the best static solver. The prediction quality of the approach suggested in [START_REF] Vermetten | Towards dynamic algorithm selection for numerical black-box optimization: investigating BBOB as a use case[END_REF] varies a lot between the different functions. While for LeadingOnes, for example, the performance predictions are rather accurate, large discrepancies between predicted and actual performance can be observed for more complex function. In particular for multi-model functions the approach can get trapped by a first algorithm that is very efficient in converging to a local optimum from which the second algorithm cannot escape easily.

Data availability: Our data is available at [START_REF] Ye | Data Sets for the study "Leveraging Benchmarking Data for Informed One-Shot Dynamic Algorithm Selection[END_REF].

INFORMED 1-SWITCH DYNAMIC ALGORITHM SELECTION

We take as input the benchmarking data from [START_REF] Ye | Benchmarking a (𝜇 + 𝜆) Genetic Algorithm with Configurable Crossover Probability[END_REF], which comprise detailed performance records for 80 genetic algorithms on the 25 functions suggested in [START_REF] Doerr | Benchmarking discrete optimization heuristics with IOHprofiler[END_REF]. We focus on expected running time (ERT) as performance measure, i.e., the average time needed by an algorithm to reach a given solution quality, the target value, denoted by 𝜙 𝑓 in the following. More precisely, the ERT(𝐴, 𝑃, 𝜙) of an algorithm 𝐴 on problem 𝑃 for target 𝜙 is computed as 𝑟 𝑖=1 min{𝑡 𝑖 (𝐴,𝑃,𝜙),𝐵 } 𝑟 𝑖=1 1{𝑡 𝑖 (𝐴,𝑃,𝜙) <∞} , where 𝐵 is the total budget of function evaluations that the algorithm can perform, 𝑡 𝑖 (𝐴, 𝑃, 𝜙) is the number of function evaluations that were needed in the 𝑖-th run to reach the target value (𝑡 𝑖 (𝐴, 𝑃, 𝜙) = ∞ if none of the evaluated solutions satisfies this

ACKNOWLEDGMENTS

Our work is supported by the Chinese scholarship council (CSC No. 201706310143) and by the Paris Ile-de-France region.

Relative ERT values of 100 1-switch combinations (𝐴 1 , 𝐴 2 , 𝜙 𝑠 ) (𝑑𝐸𝑅𝑇 ) for 23 out of 25 IOHprofiler problems in dimension 𝑑 = 100, compared to the ERT of the best static GAs according to [START_REF] Ye | Benchmarking a (𝜇 + 𝜆) Genetic Algorithm with Configurable Crossover Probability[END_REF] (𝑠𝐸𝑅𝑇 ). Each black dot represents one ERT value. The relative deviation is calculated by (𝑑𝐸𝑅𝑇 -𝑠𝐸𝑅𝑇 )/𝑠𝐸𝑅𝑇 so that negative values (below the red line) correspond to an advantage of the dynamic combination over the best static algorithm. We only display values between -0.5 and 0.5 so that the results of F24-F25 are missing here with values larger than 1. All ERT values are based on 100 independent runs. quality constraint), and 1(E) = 1 if event E is true and 1(E) = 0, otherwise.

Following the approach suggested in [START_REF] Vermetten | Towards dynamic algorithm selection for numerical black-box optimization: investigating BBOB as a use case[END_REF] we compute a "theoretical" ERT value for all combinations (𝐴 1 , 𝐴 2 , 𝜙 𝑠 ), where 𝐴 1 is the first algorithm, 𝐴 2 the second, and 𝜙 𝑠 the target value at which we switch from 𝐴 1 to 𝐴 2 . To this end, we simply compute ERT(𝐴 1 , 𝑃, 𝜙 𝑠 ) + ERT(𝐴 2 , 𝑃, 𝜙 𝑓 ) -ERT(𝐴 2 , 𝑃, 𝜙 𝑠 ), where all these ERT values are based on the performance recodes provided in [START_REF] Ye | Benchmarking a (𝜇 + 𝜆) Genetic Algorithm with Configurable Crossover Probability[END_REF]. In total, we consider 42 possible switching points 𝜙 𝑠 , which we select within the interval [𝜙 𝑚 , 𝜙 𝑓 ] between the smallest fitness value 𝜙 𝑚 of the problem and the best found target 𝜙 𝑓 according to [7, Table 1]. We consider evenly spaced targets, for the original and for the log-scaled interval, respectively. For each problem, we consider only algorithms that hit the final target value with probability at least 80% according to the data from [START_REF] Ye | Benchmarking a (𝜇 + 𝜆) Genetic Algorithm with Configurable Crossover Probability[END_REF]. Using this approach, we select for each problem the 100 best combinations (𝐴 1 , 𝐴 2 , 𝜙 𝑠 ) and we then run the combination 100 independent times on the problem that they have been selected for.

In Figure 1 we compare the so-obtained ERT values with the best ERT value reported in [START_REF] Ye | Benchmarking a (𝜇 + 𝜆) Genetic Algorithm with Configurable Crossover Probability[END_REF], which we refer to as the best static algorithm (BSA). For combinations (𝐴 1 , 𝐴 2 , 𝜙 𝑠 ) for which the parent population sizes 𝜇 1 of 𝐴 1 is larger than the parent population size 𝜇 2 of 𝐴 2 we selected the best 𝜇 2 points to initialize the parent population of 𝐴 2 . Where 𝜇 1 < 𝜇 2 , the new parent population comprises all 𝜇 1 points, as additional ⌊𝜇 2 /2⌋ -𝜇 1 copies of the best points, and ⌈𝜇 2 /2⌉ randomly added individuals.

For some of the problems (e.g., F1, F2, F7, F11-14, F16-23)), the ERT of several combinations (𝐴 1 , 𝐴 2 , 𝜙 𝑠 ) outperform that of the BSA. For other functions, and in particular for F10, F24, and F25, none of the combinations (𝐴 1 , 𝐴 2 , 𝜙 𝑠 ) is able to outperform the BSA. A few reasons and more detailed analyses can be found in the extended version of this poster, available at [START_REF] Ye | Leveraging Benchmarking Data for Informed One-Shot Dynamic Algorithm Selection[END_REF].

FUTURE WORK

We have investigated in this work possibilities to leverage existing benchmark data to derive switch-once dynamic algorithm selection policies. While for some cases the "theoretical" approach suggested in [START_REF] Vermetten | Towards dynamic algorithm selection for numerical black-box optimization: investigating BBOB as a use case[END_REF] could indeed predict combinations that outperformed the best static solver, the results are less positive for others. One obstacle that hinders an accurate performance prediction are local optima: when the first algorithm is very good at converging to a local optimum, it is likely to be chosen as 𝐴 1 . It is then important, however, to continue the search with an algorithm that has a good enough exploration power to escape the local optimum. This ability, however, seems hard to infer from the pure performance profiles, and may require a "human in the loop".

Going forward, our long-term goal is the automated detection of situations in which switching from one algorithm to another one can be beneficial. To this end, we will further investigate efficient strategies to warm-start the algorithms by actively using the information accumulated thus far. In the here-presented study, we have used ERT values as performance measure and as indicator to select which algorithm combinations to execute. In future work we will consider other performance measures, and in particular those that measure the anytime performance of the algorithms.


Amine Aziz-Alaoui

Carola Doerr

Towards Large Scale Automated Algorithm Design by Integrating Modular Benchmarking Frameworks

We present a first proof-of-concept use-case that demonstrates the efficiency of interfacing the algorithm framework ParadisEO with the automated algorithm configuration tool irace and the experimental platform IOHprofiler. By combing these three tools, we obtain a powerful benchmarking environment that allows us to systematically analyze large classes of algorithms on complex benchmark problems. Key advantages of our pipeline are fast evaluation times, the possibility to generate rich data sets to support the analysis of the algorithms, and a standardized interface that can be used to benchmark very broad classes of sampling-based optimization heuristics.

In addition to enabling systematic algorithm configuration studies, our approach paves a way for assessing the contribution of new ideas in interplay with already existing operatorsa promising avenue for our research domain, which at present may have a too strong focus on comparing entire algorithm instances.

Introduction

When confronted with an optimization problem in practice, one of the major challenges that we face is the selection (and the configuration) of an algorithm that corresponds well to the given problem structure, optimization objective(s), and the available resources (compute, possibility to parallelize computations, accessibility of the problem, etc.). A vast amount of different optimization techniques exist, which renders this algorithm selection problem non-trivial.

In practice, algorithm selection is often biased by personal preferences and experiences, as well as by practical aspects such as the availability of ready-to-use implementations. Supporting practitioners in making more systematic choices is one of the key objectives of our research domain. A key tool for deriving such recommendations is algorithm benchmarking, i.e., the analysis of empirical performance data and search trajectories of one or several algorithms on one or several optimization problems [BDB + 20, HAR + 20]. Several important benchmarking tools and software frameworks have been developed by our community to ensure sound and meaningful data extraction. These platforms address different stages of the algorithm selection process. They cover, for example, instance selection and generation [SB15, WW18, ZR20], feature extraction [START_REF] Kerschke | The r-package FLACCO for exploratory landscape analysis with applications to multi-objective optimization problems[END_REF], algorithm configuration [HHLB11, BBFKK10, LDC + 16, LJD + 16], experimentation [HAR + 20, RT18, DWY + 18, WKB + 14], data analysis [CSC + 19, EPK20, FGLP11], and performance extrapolation [START_REF] Kerschke | Automated algorithm selection: Survey and perspectives[END_REF]. However, most of these tool are developed in isolation, paying little attention to building compatible interfaces to other benchmarking modules. This significantly hinders their wider adoption.

With this work we demonstrate the benefits of a fully modular benchmarking pipeline design, which keeps the different steps of the benchmarking study in mind. We see our work as a proof of concept for better compatibility between benchmarking software. On the practical side, our pipeline paves a way for assessing the benefits of new algorithmic ideas in the context of and in interplay with other operators and ideas that our community has to offer.

Our Contribution

Concretely, we propose in this work a benchmarking pipeline that integrates the modular algorithm framework Paradiseo [START_REF] Maarten Keijzer | Evolving objects: A general purpose evolutionary computation library[END_REF][START_REF] Cahon | Paradiseo: A framework for the reusable design of parallel and distributed metaheuristics[END_REF] with the algorithm configuration tool irace [LDC + 16], the experimental platform IOHexperimenter [DWY + 18], and the data analysis and visualization module IOHanalyzer [WVY + 20]. We test our pipeline on tuning a family of genetic algorithms, inspired by [START_REF] Ye | Benchmarking a (µ + λ) genetic algorithm with configurable crossover probability[END_REF], on the so-called W-model problem instances suggested in [START_REF] Weise | Selecting a diverse set of benchmark instances from a tunable model problem for black-box discrete optimization algorithms[END_REF].

Quality of the results: We show that irace is capable of finding algorithm instances which outperform all baseline algorithms selected by hand, and this for each of the 19 problem instances that we consider. The relative advantage of the best out of 15 irace suggestions over the best baseline algorithm, measured in terms of volume under the discretized Empirical Attainment Function (see Sec. 3.1), varies between 1% and 30%, with a median gain of 13%.

Scalability: Targeting per-instance algorithm design on synthetic benchmark, our algorithmic framework is capable of generating large set of solvers, up to several millions of unique configurations. We show that it is possible to tackle such spaces thanks to fast computations. For instance, we give irace a budget of 100 000 target runs for each of 19 problems, and it completes the full task in approximately 3 hours on a laptop. In our experience, our C++ pipeline is at least 10 times faster than heavily optimized counterparts in Python, not mentioning that most of the available modular frameworks are not always heavily optimized.

Take-away for instance selection: As a side result, we observe that similar algorithm instances can be suggested by irace for some problems, suggesting that the diversity in performance profiles sought in [START_REF] Weise | Selecting a diverse set of benchmark instances from a tunable model problem for black-box discrete optimization algorithms[END_REF] may be weaker than intended. Our work suggests that an approach like ours may result in a more reliable instance selection, since it will be less biased by a small set of baseline algorithms, but rather be built on a large and diverse set of possible algorithm instances.

Extendability: Our pipeline is ready to perform large benchmark studies, covering large classes of continuous and discrete optimization algorithms. For example, local searches, particle swarm optimization, estimation of distribution algorithms and using numerical or bitstring encodings. Similarly, the pipeline gives direct access to all problems collected in IOHprofiler, which comprises in particular the BBOB functions from the COCO framework [HAR + 20], the Nevergrad problem suites [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF], the W-model instances [START_REF] Weise | Difficult features of combinatorial optimization problems and the tunable w-model benchmark problem for simulating them[END_REF], the PBO suite [DYH + 20], etc. Additionally, any benchmark or solver which would be plugged into IOHprofiler would be easily used to further extend this study.

Comparison to Previous Works

Our work is a top-down approach for automatic algorithm design [START_REF] Mascia | Grammar-based generation of stochastic local search heuristics through automatic algorithm configuration tools[END_REF], which uses a parametrized algorithmic framework to instantiate many algorithm instances.

Following [START_REF] López-Ibánez | Automatic design of hybrid metaheuristic from algorithmic components[END_REF], we observe that this differs from bottom-up "grammar-based" approaches [START_REF] De | Automatic grammar-based design of heuristic algorithms for unconstrained binary quadratic programming[END_REF][START_REF]An automatically designed recombination heuristic for the test-assignment problem[END_REF][START_REF] Pagnozzi | Automatic design of hybrid stochastic local search algorithms for permutation flowshop problems[END_REF] or Grammatical Evolution [START_REF] Ryan | Grammatical evolution: Evolving programs for an arbitrary language[END_REF][START_REF] Lourenço | Evolving Evolutionary Algorithms[END_REF], which allow for easily designed algorithm space, but complicates algorithm instantiation and optimization. In our case, the width of the design space is already large and we target fast algorithm instantiation. We thus favor the top-down approach. In this first study, we only consider categorical parameters, for the sake of implementation simplicity.

A similar approach to ours was suggested in [LS12, BLIS20, BLIS16] for multi-objective optimization. Those studies also use irace, but the authors implemented their own modular algorithm frameworks that are restricted to multi-objective optimization. Our work significantly scales up this kind of study, by leveraging larger algorithm design spaces, larger sets of benchmarks, with more problems and allowing a more detailed analysis of the results.

Few other studies consider a bi-objective measurement of performance for automated algorithm design. Most notably, [START_REF]Automatically improving the anytime behaviour of optimisation algorithms[END_REF] introduced the use of the hypervolume for given quality and time budgets. In our case, we use the volume under the curve of the empirical cumulative histogram of quality and time attainments [START_REF] Grunert Da Fonseca | Inferential performance assessment of stochastic optimisers and the attainment function[END_REF]. This should behave as the sum of hypervolumes defined for a set of thresholds covering the whole domain, instead of a single one.

Structure of the Paper

Sec. 2 briefly introduces the individual modules of our algorithm design pipeline and how they interplay with each other. The use-case on which we apply this pipeline, as well as the experimental setup are summarized in Sec. 3. The results of our empirical analysis are described in Sec. 4. We conclude our paper in Sec. 5 with a discussion on promising avenues for future work.

Availability of Code and Data

The code and data used for this study have been archived at [START_REF] Aziz-Alaoui | Towards Large Scale Automated Algorithm Design by Integrating Modular Benchmarking Frameworksdata and analysis scripts[END_REF]. Up-to-date versions of the code are available at https://github.com/jdreo/paradiseo and https://github.com/ IOHprofiler/IOHexperimenter.

The Modular Benchmark Pipeline

Figure 1 summarizes our automated algorithm design pipeline for the concrete use-case that will be studied in Sec. 3. The pipeline links an algorithm configurator with an algorithm generator and a benchmark platform. The algorithm configurator asks the algorithm generator to instantiate an algorithm, which then solves a problem of the benchmark platform while being observed by a logger. After this run, the logger's data are summarized as a scalar performance measure, which is sent back to the algorithm configurator. We briefly present in this section the different components of our pipeline, and explain the reasons behind our choices.

Algorithm Framework: Paradiseo Many evolutionary algorithms share similar design patterns, and are often composed of similar operators. This has given rise to several platforms which aim at supporting their users in designing evolutionary heuristics by compiling a set of readily-available operators within a standardized software environment. Given the substantial work that has been put into these frameworks, we decided to build our pipeline around one of the most powerful toolboxes. To this end, we have ranked 39 frameworks among the ones easily available on the web, based on an adhoc metric combining rapidity, activity, features and license, e.g., [NDV15, SL19, GP06, Jen, ECF, FDG + 12, CEP08], to name only a few. Since speed is a major concern for our pipeline, we favor frameworks written in C++. To select an up-to-date framework and to ensure availability of support in case of technical issues, we also checked the contribution activity in recent years. These two criteria reduced our choices to Paradiseo [START_REF] Maarten Keijzer | Evolving objects: A general purpose evolutionary computation library[END_REF][START_REF] Cahon | Paradiseo: A framework for the reusable design of parallel and distributed metaheuristics[END_REF], OpenBeagle [START_REF] Gagné | Genericity in evolutionary computation software tools: Principles and case study[END_REF], and ECF [ECF]. Among these three, Paradiseo covers the largest portfolio of algorithm families, which are composed in the framework by assembling atomic functions (called operators). Paradiseo is also the most actively maintained framework among the three, so that we decided to use it for our work.

The upper part of Figure 1 shows the core classes of Paradiseo involved in our setting. After the run of the algorithm, a statistic is computed on the ECDF data (in cyan), which is then returned to irace as performance metric (i.e., this is the "fitness value" that the evaluation associates to the configuration under evaluation). Involved classes are represented using the UML convention. For the sake of clarity, the IOHprofiler prefix is written as IOH and the type of the eoAlgoFoundryFastGA slots are indicated as {double} instead of eoOperatorFoundry<double>.

Algorithm Configuration: irace Several algorithm configuration tools have been developed in the last decade. Among the most common ones used in our community are irace [LDC + 16], SMAC [START_REF] Hutter | Sequential model-based optimization for general algorithm configuration[END_REF], SPOT [START_REF] Bartz-Beielstein | SPOT: A toolbox for interactive and automatic tuning in the R environment[END_REF], GGA [AMS + 15], and hyperband [LJD + 16]. We have chosen irace1 for this study, for practical considerations (previous experience, availability of documentation, support from development team).

Experimental Environment: IOHexperimenter The IOHprofiler project [DWY + 18] is a modular platform for algorithm benchmarking of iterative optimization heuristics (IOH). Within this project, IOHexperimenter provides synthetic benchmarks which are very fast to execute and a standardized way of observing algorithms behavior through so-called loggers. We have chosen this platform, because it is fast and its modular design made it particularly easy for us to integrate the algorithm design framework (being written in C++, as Paradiseo). IOHprofiler is also actively maintained, and provides access to broad ranges of different optimization processes. Compared to Nevergrad [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF], we particularly like the detailed logging options, which provide information about the anytime behavior of the algorithms-information that is currently not available in Nevergrad. Compared to the COCO [HAR + 20] environment, IOHprofiler makes it considerably easier to test algorithms' performance on our own benchmark problems or suites. Finally, the project also supports interactive performance analysis and visualization module, IOHanalyzer, which we used for the interpretation of our data.

The lower part of Figure 1 shows the classes related to the loggers and the problems that are used in our experimental study in Sec. 3. Data Records: fast ECDF Logger In our use-case, we decide to tune algorithms for good anytime performance, and to use volume under the approximated empirical cumulative density function (ECDF) curve as objective. To this end, we implement within IOHexperimenter an efficient way of computing these values. This "ECDF logger" will be described in Sec. 3.1.

Data Analysis and

Use-Case and Experimental Setup

Our use-case is the optimization of the anytime performance of a genetic algorithm on selected instances of the W-model problem. Our performance measure (Sec. 3.1), the algorithmic framework (Sec. 3.2), and the problems (Sec. 3.3) are introduced in the first three subsections. We then summarize the experimental setup of the whole pipeline in Sec. 3.4. Our objective is to find the best algorithm for each instance, which would be the first step of a per-instance, landscape-aware algorithm selection, for instance. We thus do not consider training versus test sets.

Anytime Performance Measure: AUC

In order to allow for large scale experiments, we implement a fast logger within IOHexperimenter, which essentially stores a histogram of the two-dimensional distribution of the number of runs having reached a quality/time target. The time dimension is given as the number of calls to the objective function, linearly discretized between zero and the allowed budget. The quality dimension is given as the absolute value of the best solution found during the run, linearly discretized between zero and the known V max bound (see Table 1). This is essentially a discrete version of the Empirical Attainment Function [START_REF] Grunert Da Fonseca | Inferential performance assessment of stochastic optimisers and the attainment function[END_REF], which is related itself to the multivariate Empirical Cumulative Distribution Function [START_REF] Grunert Da Fonseca | A link between the multivariate cumulative distribution function and the hitting function for random closed sets[END_REF].

Figure 2 shows two examples of such histograms, arbitrarily chosen. The matrix defines the considered quality/time targets (v, t). The color of each cell corresponds to the probability that the algorithm has identified, within the first t function evaluations, a solution of quality at least v. The darker a cell, the larger the fraction of runs that could successfully meet the quality/time target. Note here that we assume minimization as objective.

Using the histogram of the performance ECDF instead of its continuous counterpart allows to keep the data in-memory, in compact data structures, without having to rely on slow disk accesses.

The performance of the considered algorithm is computed as a statistic on this histogram. In our study, we use the volume under the curve (3D counterpart of the area under the curve, AUC) of the discretized ECDF, approximated as the sum of the EAF histogram. This allows for a compromise between quality and time, which is easily available because we consider synthetic benchmarks with known bounds.

The (µ + , λ) "Fast" GA Family

We chose for our use-case a family of (µ + , λ) GAs, which is to a large extend inspired by the study [START_REF] Ye | Benchmarking a (µ + λ) genetic algorithm with configurable crossover probability[END_REF]. Algorithm 1 summarizes the framework, called "FastGA" in the implementation.

Essentially, given a parent population of µ points, each of the λ offspring is created by first deciding which variation operator is applied (line 9): with probability p c the offspring is generated by first recombining two search points from the parent population (lines 11-13) and then randomly deciding (with probability p m ) whether or not to apply a mutation operator to Algorithm 1: A Configurable Family of (µ + , λ) Genetic Algorithms.

1 Input: Budget B, configuration (µ, λ, p c , p m ), choice of the operators and conditional parameters. Note that P and P are multi-sets, i.e., the same point may appear multiple times; 2 Initialization: y (i,1) , y (i,2) ← SelectC(P ); 12 y (i,1) , y (i,2) ← Crossover y (i,1) , y (i,2) ; 13 Sample z (i,1) ∈ y (i,1) , y (i,2) u.a.r. ; the so-created offspring (lines 15-18). When crossover was not selected in line 10, the offspring is created by mutation (lines 20-21). When all λ offspring have been created, the iteration is completed by a replacement step (line 25).

14 Sample r m ∈ [0, 1] u.a.r. ; 15 if r m ≤ p m then 16 z (i,2) ← Mutation z (i,1) ; 17 else 18 z (i,2) ← z (i,1) ; 19 else 20 z (i,1) ← SelectM(P ); 21 z (i,2) ← Mutation z (i,

Implementation of this Family in Paradiseo:

We implement this family of GAs through Paradiseo's "foundries", which allow to register a set of operators (e.g., several kind of mutations) within a "slot" (e.g., the step at which mutation is called within the algorithm). Before each call, it is possible to instantiate a specific operator among the registered ones, for each slot, thus assembling one of the algorithm instance among all the possible combinations of operators. Note that operators can be simple numbers, like a probability. Operators are referenced within slots by their indices.

Most of the operators we use were already available in Paradiseo, to the exception of mutations operators with indices 1-5 (see below), which we implemented for this study. We also implemented the algorithm 1 as the eoFastGA class 2 , in which to plug the operators.

We consider the following operators and parametrizations, which result in a (large) total number of 1 630 475 different configurations of Algorithm 1. Numbers in brackets indicate the indices of the corresponding operators within its slot. Crossover rate p c (6 options). We consider p c ∈ {0, 0.2, 0.4, 0.5, 0.6, 0.8}. Being only able to use the integer and categorical interface for irace.

SelectC(P ): Selection of two points for the crossover operation (7 options). Note that in the implementation, the selection operator (line 11) is called twice to select the two candidate points.

[0] eoRandomSelect(): Uniformly select a point from P (without removing the first selected individual from the set P used by the second selection). (1 option).

[1] eoStochTournamentSelect(k): Select a point from P with tournament selection, i.e., we select uniformly at random k different points in P and the best one of these is selected. k denotes the tournament size as percentage of population (i.e., k ∈ [0, 1]). (1 option, k = 0.5).

[2] eoSequentialSelect(): Select the best point from P (with respect to the objective function value). This operator is sometimes referred to as elitist selection or truncation selection. When called twice, it selects the two distinct best points from P . (1 option).

[3] eoProportionalSelect(): Select a point from P with so-called fitness-proportional selection, i.e., point x ∈ P is chosen with probability f (x)/ y∈P f (y). (1 option).

[4--6] eoDetTournamentSelect(k): Like eoDetTournamentSelect, but k is deterministic.

(3 different options, each one for k ∈ [2, 6, 10]).

Crossover(x, y): Bivariate Variation Operators (11 options).

[0--4] eoUBitXover(b c ): Uniform crossover with bias (or "preference" in ParadisEO) b c , setting (independently for each position i ∈ [1..n]) z i = x i with probability b c and setting z i = y i otherwise. z denotes the offspring element coming from the crossover of x and y.

(5 different options, b c ∈ [0.1, 0.3, 0.5, 0.7, 0.9]).

[5--9] eoNPtsBitXover(k) : k-point crossover, which selects i 1 , . . . , i k uniformly at random and without replacement from [1.

.n] and sets

z i = x i for i ∈ [1..i 1 ] ∪ [i 2 + 1..i 3 ] ∪ . . . and sets z i = y i for i ∈ [i 1 + 1..i 2 ] ∪ [i 3 + 1..i 4 ] ∪ . . . (5 different options, k ∈ [1, 3, 5, 7, 9]
).

[10] eo1PtBitXover(): Classic 1-point crossover. (1 option). We mistakenly added this option even if it is the same as the previous crossover with k = 1.

Mutation probability p m (6 options): We consider p m ∈ {0, 0.2, 0.4, 0.5, 0.6, 0.8}.

Mutation(x): Univariate Variation Operator (11 options) All mutation operators are unary unbiased in the sense proposed in [START_REF] Per | Black-box search by unbiased variation[END_REF]. For a compact representation, we follow the characterization suggested in [START_REF] Doerr | Optimal parameter choices via precise black-box analysis[END_REF] and define the mutation operators via the distributions that they define over the possible mutation strengths k ∈ [0.

.n]. After sampling k from the operator-specific distribution, the k-bit flip operator, flip k (•), is applied; it flips the entries in k uniformly chosen, pairwise different bits (i.e., the k bits are chosen u.a.r. without replacement).

[0] eoUniformBitMutation(): The "uniform" mutation operator, which samples k uniformly at random in the set [0.

.n]. (1 option).

[1] eoStandardBitMutation(p = 1/n): This is the standard bit mutation with mutation rate p. It chooses k from the binomial distribution B(n, p). (1 option).

[2] eoConditionalBitMutation(p = 1/n): A conditional standard bit mutation operator with mutation rate p. It chooses k from B(n -1, p) and applies the flip k (•) operator with k = k + 1. (1 option).

[3] eoShiftedBitMutation(p = 1/n): The "shifted" standard bit mutation with mutation rate p, suggested in [START_REF] Carvalho | Towards a more practice-aware runtime analysis of evolutionary algorithms[END_REF]. It samples k from the binomial distribution B(n, p). When k = 0, it uses k = 1 and it uses k = k otherwise. (1 option).

[4] eoNormalBitMutation(p,σ 2 ): The "normal" mutation operator suggested in [START_REF] Ye | Interpolating local and global search by controlling the variance of standard bit mutation[END_REF]. It samples k from the normal distribution N (pn, σ 2 ). When k > n, k is replaced by a value chosen uniformly at random in the set [0..n]. (1 option, p = 1/n and σ 2 = 1.5).

[5] eoFastBitMutation(β): The "fast" mutation operator suggested in [START_REF] Doerr | Fast genetic algorithms[END_REF]. It samples k from the power-law distribution SelectM(P ): Selection of one point for the mutation operation if crossover was not chosen (7 options) We essentially have the same selection operators as for crossover. The only difference is that we select only one point instead of two.

P[L = k] = (C β n/2 ) -1 k -β with C β n/2 = n/2 i=1 i -β . When k is larger than n, it
Replace(P, P , µ): Replacement of population (11 options)

[0] eoPlusReplacement(): The best µ points of the multiset P ∪ P are chosen. (1 option).

[1] eoCommaReplacement(): The best µ points of the offspring multiset P are chosen. (1 option).

[2] eoSSGAWorseReplacement(): The min(λ, µ) points of the offspring multiset P replace the worst points in P . (1 option). [6--10] eoSSGADetTournamentReplacement(k): The µ points are selected through tournament selection. Each tournament involves k uniformly chosen points in P ∪P and the best ones of these k points is selected. This procedure is repeated µ times, each time removing an already selected point from the multi-set P ∪P . (5 different options, k ∈ [2, 4, 6, 8, 10]).

This concludes our description of the high level operators of our family of (µ + , λ) GAs. The set of all combinations generates the algorithm design space on which we let irace search for the configuration(s) that best solve a given problem instance.

Baseline Algorithms We consider four baseline algorithms, against which we compare the results of the automated design. They were manually chosen without particular justification.

(1) (λ + λ) EA: no crossover, plus replacement, standard bit mutation, random selector for mutations. (2) (λ + λ) fEA: no crossover, plus replacement, fast bit mutation, random selector for mutations.

The W-Model Problems

We evaluate our automated algorithm design pipeline on the W-model functions originally suggested in [START_REF] Weise | Difficult features of combinatorial optimization problems and the tunable w-model benchmark problem for simulating them[END_REF]. In a nutshell, the W-model is a benchmark problem generator, which allows to tune different characteristics of the problems, see below for a description. We selected from this family of benchmark problems the 19 instances suggested in [START_REF] Weise | Selecting a diverse set of benchmark instances from a tunable model problem for black-box discrete optimization algorithms[END_REF], which are summarized in Table 1. Note here that the description differs from that given in [START_REF] Weise | Selecting a diverse set of benchmark instances from a tunable model problem for black-box discrete optimization algorithms[END_REF], since we used the implementation within IOHexperimenter, which was made available in the context of the work [DYH + 20]. The problem instances listed in Table 1 are identical to those suggested in [START_REF] Weise | Selecting a diverse set of benchmark instances from a tunable model problem for black-box discrete optimization algorithms[END_REF], it is only the representations that differ.

It was suggested in [DYH + 20] to superpose the W-model transformations to different optimization problems. The instances selected in [START_REF] Weise | Selecting a diverse set of benchmark instances from a tunable model problem for black-box discrete optimization algorithms[END_REF], however, were only selected from transformations applied to the OneMax problem OM : {0, 1} -→ [0.

.n], x -→ n i=1 x i . The One-Max problem has a very smooth and non-deceptive fitness landscape. Due to the well-known coupon collector effect [START_REF] Flajolet | Birthday paradox, coupon collectors, caching algorithms and self-organizing search[END_REF], it is relatively easy to make progress when the function values are small, and the probability to obtain an improving move decreases considerably with increasing function values. The complexity of the OneMax problem can be considerably increased through the following W-model transformations.

(1) Neutrality W (., µ W , ., .): The bit string (x 1 , ..., x n ) is reduced to a string (y 1 , ..., y m ) with m := n/µ W , where µ W is a parameter of the transformation (we use the subscript W to indicate that these are parameters of the W-model problem generator). For each i ∈ [m] the value of y i is the majority of the bit values in the size-µ substring (x (i-1)µ W , x (i-1)µ W +1 , ..., x iµ W ) (2) Epistasis W (., ., ν W , .): Epistasis introduces local perturbations to the bit strings. It first "cuts" the input string (x 1 , ..., x n ) into subsequent blocks of size ν W . Using a permutation e ν W : {0, 1} ν W -→ {0, 1} ν W , each substring (x (i-1)ν W +1 , x (i-1)ν W +2 , ..., x iν W ) is mapped to another string (y (i-1)ν W +1 , y (i-1)ν W +2 , ..., y iν W ) = e ν W ((x (i-1)ν W +1 , x (i-1)ν W +2 , ..., x iν W )). The permutation e ν W is chosen in a way that Hamming-1 neighbors are mapped to strings of Hamming distance at least ν W -1, see [START_REF] Weise | Difficult features of combinatorial optimization problems and the tunable w-model benchmark problem for simulating them[END_REF] for examples.

(3) Ruggedness and Deceptiveness W (., ., ., γ W ): This layer perturbs the fitness values, by applying a permutation σ(γ W ) to the possible fitness values [0..n]. The parameter γ W can be thought of as a parameter which controls the distance of the permutation to the identity. The permutations σ(γ W ) are chosen in a way such that the "hardness" of the instances monotonically increases with increasing γ W , see [START_REF] Weise | Difficult features of combinatorial optimization problems and the tunable w-model benchmark problem for simulating them[END_REF] for details.

We convert these functions into a minimization problem by multiplying all values by -1.

Experimental Setup

Our test bed is the automated design of Algorithm 1 with the options specified in Sec. 3.2 and with the objective to maximize the AUC as defined in Sec. 3.1, and this for each of the 19 problems listed in Table 1. These instances of the W-model problem were suggested in [START_REF] Weise | Selecting a diverse set of benchmark instances from a tunable model problem for black-box discrete optimization algorithms[END_REF] based on an empirical study using clustering of algorithm performance data, with the goal to select a diverse collection of benchmark problems. Note here that we tune the algorithms for each problem individually. That is, we apply our algorithm design pipeline 19 independent times.

For the sake of simplicity, we fix the population sizes to λ = µ = 5, for the search performed by irace and for our baseline algorithms.

For each use-case, we set the budget of the algorithms to 5n function evaluations (FEs). To compute the AUC, we evaluate the performance at 100 linearly distributed budgets b 1 , . . . , b 100 ∈ [1, 5n] and at 100 linearly distributed target values v 1 , . . . , v 100 ∈ [0, V max ]. Linearization computes the bucket index i = (x -x min )/(x max -x min ) • 100 for both budgets and targets.

To find the best algorithm design, we allow irace a budget of 100 000 target runs. We ensure that irace performed at least 50 independent runs for the first ranked elite configuration, adding additional runs if needed, and keeping all runs if irace conducted more than 50 runs. We do not consider the other elite configurations ranked by irace, even in draw cases. We run irace 15 independent times, to check the robustness of its selection. We compare performance to the four baseline algorithms, which we run 50 independent times each on each of the 19 test problems.

In total, these experiments took around 15 × 3 hours on a computer with four Intel CPU cores i5-7300HQ at 2.50GHz and Crucial P1 solid-state disks. 1 compares the AUC values of the best out of the four baseline algorithms against that of the elite configuration suggested by irace. We observe that, for each of the 19 functions, the elite configurations suggested by irace perform better than the best baseline algorithm. We report in Table 1 the average values, but the differences between the individual irace runs are very small, less than 2.1% difference in AUC value between the best and the worst elite configuration for all 19 problems, and less than 1% performance difference for 9 out of the 19 functions. The relative advantage of the irace recommendations over the best baseline algorithms varies between 1% and 30%. When looking at each of the 15 elite configurations suggested per function, the best relative advantage is 31% for F17, whereas two of the irace elites performed worse than the best of the four baseline algorithms on function F3. For all other functions, all 15 irace elites have a better AUC value than the best of the four baseline algorithms. However, although we see a clear advantage of the irace configurations, we should keep in mind that the irace configurations are specifically tuned for each function, whereas the configurations of the four baseline algorithms are identical for all 19 W-model functions.

Experimental Results

Comparison of AUC Values by Function Table

Unfortunately, our pipeline does not yet allow to tune a single best solver, i.e., a single configuration that maximizes the AUC under the aggregated ECDF curve. Adding this functionality is a straightforward extension of our framework, which we plan to address in future work. The key challenge here is that Paradiseo does not have the feature to easily reset on the fly the states of solvers between two runs on different problems.

Comparison of the Configurations. Table 2 summarizes the best of the 15 elite configurations that were suggested by irace and compares them against the four baseline algorithms. Table 3 shows the distribution of operators chosen by the 15 irace runs. For the latter, we have chosen problem 17 as an example because we observed here the largest relative gain (see Table 1). We have added problem 5 for comparison, because the distribution of operators suggested for it is very distinct from that of problem 17.

It is worth noting that each operator is selected at least once in the 19×15 elite configurations suggested by irace (Table 3, right), which seems to confirm that i) different operators work well on different problems and ii) that irace searches the full design space, giving some indication that it is not too large or too complex for automated tuning approaches.

We can see that, among all the best configurations proposed by irace across 15 runs, none are similar to one of the baseline algorithms. The probability of mutation p m is most frequently set to higher values and the most often chosen mutation is deterministic bit flip with larger number of bits (index 10 in the Mutation slot, which is the flip 9 mutation operator). This indicates that larger mutation strengths could have been worth investigating, a result that has surprised us, since in most benchmark studies we see small mutation rates as defaults (albeit we do not consider various population sizes in this study). The results confirm the superiority of the plus replacement (id. 0 in the Replace slot) and support the use of an elitist selection for the crossover candidates (id. 2 in SelectC). We can also see that the uniform crossover with b c = 0.5 (id. 2 in Crossover) is more often chosen, like a small probability of performing a crossover (id. 1 in p c ).

For some problems, irace almost always suggest a similar algorithm. On problem 17, for example, it often selects a GA with a large probability of applying uniform crossover in combination with deterministic bit flip mutations. For some other problems, a larger variance on the selected operators can be observed. For instance on problem 5, irace selects a high mutation probability along with an elitist mutation selection, but does not show a clear preference for the other slots.

These results support the idea that there is not always a single best solver (i.e., "No Free Lunch"), even when considering limited design and benchmarking spaces. We also see that some problems seem to require certain design choices, whereas others can be solved well by a broad range of configurations. A more detailed analysis of how these preferences correlate with the characteristics of the problems should offer plenty of interesting insights, but is left for future work. evaluations performed, for each baseline algorithm and for the best elite configuration selected by irace. Problems 5 and 17 are chosen to allow for comparison with Table 2.

Fixed-budget solution qualities

We observe that the elite configuration on problem 17 is largely more efficient than any of the baseline algorithms. However, on problem 5, the elite configuration is only the most efficient until 140 evaluations. It is selected nonetheless, because we consider the AUC of the 2D ECDF, which takes into account the average performance (across all budgets and targets) rather than the terminal budget of the best target. We believe that, whatever the performance metric we choose, there will always exists such artifacts, where some algorithm would be the best, had we chosen another metric. It is clear, however, that even in this plot the elite configuration performs better most of the time.

Conclusions and Future Work

By interfacing the three state-of-the-art benchmarking modules from the evolutionary computation literature, irace [LDC + 16], Paradiseo [START_REF] Maarten Keijzer | Evolving objects: A general purpose evolutionary computation library[END_REF], and IOHprofiler [DWY + 18], we have introduced in this work a powerful pipeline for the automated design of sampling-based optimization algorithms. We have demonstrated its efficiency on the use-case of tuning a family of genetic algorithms on instances of the W-model [START_REF] Weise | Difficult features of combinatorial optimization problems and the tunable w-model benchmark problem for simulating them[END_REF] suggested in [START_REF] Weise | Selecting a diverse set of benchmark instances from a tunable model problem for black-box discrete optimization algorithms[END_REF].

Our results supports the idea that automated algorithm design can lead to increased performances and that there are efficient designs which wait to be studied more thoroughly. We believe that efficient pipelines like the one we introduce has the potential to help raising the level of abstraction at which researchers are working. Using automated algorithm design, it becomes possible to check if a newly designed operator can actually be useful in some algorithms/problems coupling [START_REF] Xu | Evaluating component solver contributions to portfolio-based algorithm selectors[END_REF]. We also believe that such studies can help deriving generic rules about algorithm design and could probably help theoretical researchers by suggesting where to look for interesting structures.

The modular design of the pipeline and its components makes our approach very broadly applicable. It is not restricted to particular types of problems nor to specific algorithms. In particular, extensions to continuous or mixed-integer problems are rather straightforward. Indeed, the Paradiseo framework is designed to separate operators which are independent of the encoding (selection, replacement, etc.) from operator which depends on it (mutation, crossover, etc.), allowing for easy reuse of components and extensions to other algorithmic paradigms (estimation of distribution, local search, multi-objective, etc.). Additionally, the IOHexperimenter provides loggers for vectorial encodings and benchmarks for both numerical and bitstring encodings.

Our work is partially motivated by an industrial application that requires an automated configuration of hardware products. However, we believe that our pipeline is not only interesting for such practical purposes. For researchers, our pipeline offers an elegant way of assessing new algorithm operators and their interplay with already existing ones.

In terms of further development, we plan to add the necessary features which would i) allow for running the same algorithm on multiple problems, while using a single logger that aggregates the results and would ii) support irace's interface for numerical parameters (additionally to categorical and integer ones).

We then plan to test the approach on different algorithms families, with a possible extension to generic "bottom-up" hybridization grammars [START_REF] Marmion | Towards the Automatic Design of Metaheuristics[END_REF] and studies on the most efficient algorithms design (e.g., on the correlations between elite algorithms' operators).

We also plan to extend the framework by integrating feature extraction methods that use algorithm trajectory data [DLV + 19, BPRH19] and/or samples specifically made for exploratory landscape analysis [MBT + 11, KT16] to couple the algorithm design to such information, similar to the per-instance configuration approaches made in [START_REF] Hutter | Performance prediction and automated tuning of randomized and parametric algorithms[END_REF][START_REF] Belkhir | Per instance algorithm configuration of CMA-ES with limited budget[END_REF].

Our long-term vision is a pipeline for the automated design of algorithms which adjust their behavior during the optimization process, by taking into account information accumulated so far, similar to the dynamic algorithm configurations studied under the notion of parameter control [START_REF] Karafotias | Parameter control in evolutionary algorithms: Trends and challenges[END_REF]. In contrast to the static designs considered in this work, the automated design of dynamic algorithms requires to select suitable update rules (e.g. based on time, on progress, on self-adaption, etc.).

Finally, we also consider interesting the idea to provide a user-friendly front-end which allows users to assemble a benchmark study by selecting (e.g. through a graphical user interface) one or more algorithms and problems, the budget, etc. and then passing on this study to an automated interface which tunes (if desired) and runs the algorithm(s) and then automatically directs its users to the data summary and visualization platform IOHanalyzer, where the results of the empirical study can be analyzed. We believe that such a pipeline would greatly improve the deployment of evolutionary methods in practice.

A Comparison of frameworks

Table 4: Comparison of software frameworks for evolutionary computation. Rank is based on an adhoc aggregation of subjective metrics based on performance of the programming language, activity of the project (number of contributors), breadth of features (number of modules, number of lines of code), and ease of integration in industrial projects (license). Data has been gathered in 2019.

B Average AUC values

D Convergence plots for all problems

The following 19 figures shows the convergence plots of the baseline algorithms against the best elite selected by irace. Algorithms are denoted in the legend by the set of indices for each slots, using the following code (see Table 2 for the corresponding algorithms):

Figure 1 :

 1 Figure1: Summary diagram of the FastGA evaluation pipeline involving the Paradiseo (upper part, red colors) and IOHexperimenter (lower part, blue colors) frameworks along with the irace entry point. The execution starts from the irace run command on the left, goes through the Paradiseo modules, which call the IOHexperimenter problem (in blue) and loggers (in green). After the run of the algorithm, a statistic is computed on the ECDF data (in cyan), which is then returned to irace as performance metric (i.e., this is the "fitness value" that the evaluation associates to the configuration under evaluation). Involved classes are represented using the UML convention. For the sake of clarity, the IOHprofiler prefix is written as IOH and the type of the eoAlgoFoundryFastGA slots are indicated as {double} instead of eoOperatorFoundry<double>.

  Visualization with IOHanalyzer Data analysis and visualization is performed via IOHanalyzer [WVY + 20], another module of the IOHprofiler project [DWY + 18].

3P←

  InitialSampling(µ); 4 evaluate the µ points in P ; 5 Evals ← µ; 6 Optimization: while Evals < B do 7 P ← ∅; 8 for i = 1, . . . , λ do 9 Sample r c ∈ [0, 1] u.a.r.; 10 if r c ≤ p c then 11

P

  ← P ∪ z (i,2) ; 25 P ← Replace(P, P , µ);

Figure 2 :

 2 Figure 2: Example of two ECDF histograms. Abscissa shows the time dimension and the ordinate shows the target quality. The colormap shows the probability for the considered algorithm to reach a quality/time target falling in each bucket of the histogram. The figures show ECDF histograms for the 1ptGA algorithm (see Sec. 3.2) after 50 independent runs on problem 2 (left) and of problem 5 (right), respectively, using 20 buckets on each axis.

  samples a uniform value k in [0..n], and it uses k = k otherwise. (1 option, β = 1.5). [6--10] eoDetSingleBitFlip(k): Deterministically applies flip k (•). (5 different options, k ∈ [1, 3, 5, 7, 9]).

[ 3 -

 3 -5] eoSSGAStochTournamentReplacement(k): Like eoSSGADetTournamentReplacement, k being the the tournament size as percentage of population. (3 different options, k ∈ [0.51, 0.71, 0.91]).

  (3) (λ+λ) xGA: sequential selections, uniform crossover, standard bit mutation, plus replacement, p c = 0.4, b c = 0.4. (4) (λ+λ) 1ptGA: sequential selections, 1-point crossover, standard bit mutation, plus replacement, p c = 0.4, b c = 0.4.

  Figure 3: Convergence plots for the baseline algorithms and the elite configurations suggested by irace, on problem 5 (top) and problem 17 (bottom).

Figure 4 :Figure 5 :

 45 Figure 4: Distances between AUCs of elite algorithms and baseline algorithms.

P

  : population size (always 5 in this study), C: crossover probability, s: crossover selector, c: crossover, a: selector after crossover (always 0 in this study), M: mutation probability, u: mutation selector, m: mutation, r: replacement, O: stopping criterion (always 0 in this study).

Figure 6 :Figure 7 :Figure 8 :Figure 9 :Figure 10 :

 678910 Figure 6: Convergence plot of baseline algorithms and elite, for problem 1.

Figure 11 :Figure 12 :Figure 13 :Figure 14 :Figure 15 :

 1112131415 Figure 11: Convergence plot of baseline algorithms and elite, for problem 6.

Figure 16 :Figure 17 :Figure 18 :Figure 19 :

 16171819 Figure 16: Convergence plot of baseline algorithms and elite, for problem 11.

Figure 20 :Figure 22 :

 2022 Figure 20: Convergence plot of baseline algorithms and elite, for problem 15.

Table 1 :

 1 Test problems on which the pipeline is evaluated, taken from[START_REF] Weise | Selecting a diverse set of benchmark instances from a tunable model problem for black-box discrete optimization algorithms[END_REF]. In column "best" we list the baseline algorithm with largest average AUC value, reported in column AUC b . AUC i is the average of the 15 mean AUC of elite configurations, as suggested by 15 independent runs of irace with default hyperparameters. AUC-values are w.r.t. to at least 50 validation runs and "rel." indicates the relative gain (AUC i -AUC b )/ AUC b .FID dim µ W ν W γ W V max best AUC b AUC i

	rel.

of x. That is, y i = 1 if and only if there are at least µ W /2 ones in this "block". When n/µ W / ∈ N , the last bits of x are copied to y.

Table 2 :

 2 Configuration of the best out of the elite recommendation suggested by 15 independent runs of irace, for each of the 19 benchmark problems specified in Table1, and compared against the configuration of the four baseline algorithms. The "op." column gives the number of options per operator. All other integer values correspond to the indices with which the different options are listed in Sec. 3.2 and "-" indicates a non-applicable element (e.g., no crossover operator is used when p c = 0).

	Operator	op. 1 2 3 4 5 6	7	8	9 10 11 12 13 14 15 16 17 18 19 EA fEA xGA 1ptGA
	pc	5 1 4 1 2 4 0	3	0	2	4	3	2	3	1	2	2	3	4	4	0	0	2	2
	SelectC	7 2 5 3 1 2	-	0	-	2	2	2	2	6	5	5	2	2	2	2	-	-	2	2
	Crossover	11 1 2 8 1 2	-	3	-	2	2 10	5	2	9	2 10	2	2	2	-	-	2	5
	pm	5 2 3 2 2 4	-	4	-	4	4	4	4	4	4	4	4	4	4	4	-	-	2	2
	SelectM	7 2 4 6 6 3 2	3	2	5	5	2	3	1	2	6	6	5	1	6	0	0	2	2
	Mutation	11 8 9 3 9 7 6 10 10 10	9 10	9 10	8	8 10 10	8	9	1	5	1	1
	Replace	11 8 9 2 0 0 0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0

Table 3 :

 3 Distribution of operators variants recommended by 15 runs of irace, for problems 5 (left), problem 17 (center) and all problems (right). The most selected indices are highlighted in bold and the darker the background color, the more often the operator instance is selected. Empty cells indicates that irace never selected the operator instance, cells with a "-" entry marks indices which are not defined for this operator.

						Problem 5								Problem 17									All problems
	Op. index	0	1	2	3	4	5	6	7	8	9 10 0	1	2	3	4	5	6	7	8	9 10 0	1	2	3	4	5	6	7	8	9 10
	Pc	5	6	1	3		-	-	-	-	-	-				1 14 -	-	-	-	-	-	18 74 56 57 65 -	-	-	-	-	-
	SelectC	1		5	2		4	3 -	-	-	-		15				-	-	-	-	18 16 141 19 18 30 28 -	-	-	-
	Crossover	1	5	2	2	1	1				1	2		15								11 29 78 24	6	8 11 19 18 23 43
	Pm			2	3 10 -	-	-	-	-	-				15 -	-	-	-	-	-	4	9 25 49 183 -	-	-	-	-	-
	SelectM			14				1 -	-	-	-	6	2	1	2	2	2 -	-	-	-	26 17 61 36 27 41 62 -	-	-	-
	Mutation	2	5		1	1	2	4											1	9	4	1	2	7		3	1	2	6 10 62 81 96
	Replace	15										15										245	2	3	6	2	1	2	5	1	3

Table 5 :

 5 Average AUC values of the elites returned by irace in each of the 15 independent runs and of the four baseline algorithms on each of the 19 W-model instances

	FID	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15 1ptGA	EA	fEA	xGA
	1	8,696	8,700	8,716	8,728	8,731	8,732	8,735	8,742	8,748	8,749	8,753	8,753	8,755	8,759	8,804	8351.2	8340.6	8369	8377.8
	2	8,704	8,734	8,738	8,739	8,740	8,742	8,754	8,754	8,760	8,761	8,765	8,769	8,770	8,779	8,794	8265.6	8318.2	8402.4	8311.4
	3	8,298	8,350	8,354	8,359	8,368	8,386	8,395	8,401	8,411	8,411	8,426	8,437	8,441	8,443	8,472 8300.48	8341.5 8352.14 8328.02
	4	8,862	8,882	8,886	8,886	8,888	8,892	8,909	8,912	8,929	8,930	8,930	8,940	8,943	8,949	8,970 8179.12 8298.52 8290.12 8252.38
	5	8,478	8,478	8,480	8,488	8,491	8,498	8,498	8,501	8,502	8,507	8,533	8,546	8,547	8,548	8,554	7923.6	7975.6 8002.88 7983.12
	6	7,241	7,261	7,276	7,276	7,281	7,288	7,304	7,315	7,316	7,325	7,330	7,342	7,367	7,373	7,374 7055.06 6666.08	6534.9 7054.04
	7	8,120	8,155	8,159	8,161	8,173	8,175	8,181	8,186	8,188	8,188	8,195	8,195	8,201	8,214	8,252 6833.16	6828.8 6827.14 6698.18
	8	8,451	8,468	8,474	8,480	8,482	8,486	8,491	8,503	8,506	8,506	8,509	8,510	8,520	8,524	8,574 6796.86	6884.5 6851.04 6750.42
	9	8,758	8,761	8,771	8,774	8,779	8,783	8,787	8,790	8,793	8,794	8,796	8,796	8,796	8,807	8,807 8075.14 8089.02 8040.36 8153.66
	10	8,062	8,079	8,090	8,096	8,102	8,108	8,108	8,110	8,124	8,137	8,146	8,155	8,162	8,167	8,177 7091.52 7187.64 7216.32 7139.88
	11	9,117	9,120	9,123	9,125	9,127	9,132	9,139	9,140	9,143	9,144	9,149	9,156	9,158	9,158	9,159 8178.84 8314.34	8289.5 8183.16
	12	8,690	8,696	8,717	8,718	8,721	8,728	8,733	8,734	8,735	8,739	8,744	8,746	8,747	8,749	8,759	7960.4 8033.76 8022.48 8000.08
	13	9,304	9,324	9,333	9,337	9,338	9,347	9,348	9,351	9,352	9,353	9,353	9,356	9,359	9,360	9,368 8001.24 8068.56 8076.36 7969.56
	14	7,610	7,628	7,634	7,636	7,640	7,665	7,670	7,688	7,691	7,693	7,695	7,702	7,705	7,724	7,729 6064.98 6137.96 6173.42 6138.36
	15	8,255	8,265	8,272	8,279	8,279	8,284	8,288	8,295	8,297	8,298	8,302	8,306	8,315	8,318	8,330 6703.18 6768.46 6797.36 6765.32
	16	8,552	8,565	8,573	8,580	8,588	8,591	8,594	8,600	8,601	8,601	8,602	8,603	8,609	8,609	8,619 7164.98 7237.94 7273.42 7238.36
	17	8,995	8,997	9,012	9,013	9,015	9,024	9,024	9,026	9,028	9,030	9,031	9,037	9,057	9,064	9,067 6838.68	6910 6873.06 6935.24
	18	7,042	7,047	7,057	7,069	7,070	7,078	7,091	7,094	7,103	7,103	7,104	7,105	7,108	7,133	7,134 5764.78 5957.88 5956.34 5885.52
	19	8,685	8,698	8,700	8,701	8,702	8,706	8,709	8,718	8,720	8,722	8,727	8,733	8,734	8,751	8,756 7330.28	7398.6 7359.56 7287.74

AUC of the elite returned by irace run number Baseline Algorithms

Version 3.4.1 of https://cran.r-project.org/web/packages/irace/, ran with R 3.6.3.

All our code is contributed to the Paradiseo project.

Acknowledgments. We thank the GECCO ECADA workshop reviewers for very constructive feedback and for pointers to [LS14].

This work has been financially supported by the Paris Ile-de-France region.