Anja Jankovic

Gorjan Popovski

Tome Eftimov

Carola Doerr

The Impact of Hyper-Parameter Tuning for Landscape-Aware Performance Regression and Algorithm Selection

Automated algorithm selection and configuration methods that build on exploratory landscape analysis (ELA) are becoming very popular in Evolutionary Computation. However, despite a significantly growing number of applications, the underlying machine learning models are often chosen in an ad-hoc manner.

We show in this work that three classical regression methods are able to achieve meaningful results for ELA-based algorithm selection. For those three models -random forests, decision trees, and bagging decision trees -the quality of the regression models is highly impacted by the chosen hyper-parameters. This has significant effects also on the quality of the algorithm selectors that are built on top of these regressions.

By comparing a total number of 30 different models, each coupled with 2 complementary regression strategies, we derive guidelines for the tuning of the regression models and provide general recommendations for a more systematic use of classical machine learning models in landscape-aware algorithm selection. We point out that a choice of the machine learning model merits to be carefully undertaken and further investigated.

INTRODUCTION

A vast majority of real-world optimization problems are too complex to be defined by an explicit mathematical model, and they require (sometimes very expensive) evaluations to assess and to compare the quality of different alternatives. Classical examples for such problems are typically found in settings where the relationship between decision variables and solution quality cannot usually be established without a computer simulation or a physical experiment, both of which require a significant amount of resources, such as crash tests in automotive industry or clinical trials in medicine. A line of research known as black-box optimization (BBO) studies solving those types of problems. They can only be solved using so-called iterative sampling-based heuristics, which are algorithms that guide the search towards the optimal solution of a certain problem by sampling candidate solutions, evaluating them (assessing their quality) and using gained information to select the new candidate solutions for the next step. They proceed in iterations until converging to an estimated optimum.

In the last decades, many iterative optimization heuristics have been designed and improved upon [START_REF] Bäck | Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms[END_REF][START_REF] Eiben | Introduction to Evolutionary Computing[END_REF][START_REF] Hansen | Completely Derandomized Self-Adaptation in Evolution Strategies[END_REF][START_REF] Kennedy | Particle swarm optimization[END_REF][START_REF]Estimation of Distribution Algorithms[END_REF][START_REF] Mühlenbein | From recombination of genes to the estimation of distributions I. Binary parameters[END_REF][START_REF] Storn | Differential Evolution -A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces[END_REF], but the quest for good algorithm designs is far from over. Quite the contrary, the design of efficient iterative sampling-based heuristics is a very active area of research, which conceives considerable attention at the moment due to the important role that black-box optimization techniques play in artificial intelligence. While often being advertised as universal optimization techniques, it cannot be neglected that different algorithms can show very different performances on different types of problems (and for different performance measures). Some -and often a substantial amount of -customization is therefore needed to obtain peak performance.

Consequently, a very important and challenging task is to select the most efficient and appropriate algorithm every time one is faced with a new, previously unseen problem instance. This research problem, formalized as the algorithm selection problem (ASP) [START_REF] Rice | The Algorithm Selection Problem[END_REF], has been classically tackled by relying on expert knowledge in both the problem and the algorithm domain. However, significant progress in machine learning (ML) in recent years has allowed for a shift to be made towards automated selection (AS) [START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF][START_REF] Kerschke | Leveraging TSP Solver Complementarity through Machine Learning[END_REF][START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning[END_REF][START_REF] Muñoz | Algorithm selection for black-box continuous optimization problems: A survey on methods and challenges[END_REF] and configuration (AC) [START_REF] Belkhir | Per instance algorithm configuration of CMA-ES with limited budget[END_REF], both being important parts of a more general AutoML research framework [START_REF]Automated Machine Learning -Methods, Systems, Challenges[END_REF], where ML techniques (typically based on supervised learning approaches, such as regression and classification) are used to design and train models to predict the performance of different black-box algorithms on unseen problem instances as accurately as possible. These performance predictions would then allow for selecting and/or configuring the best algorithm for the problem at hand.

One of the challenges for such supervised learning methods to be useful is the identification of convenient representations of the problem instances, which can be used by the AutoML techniques to derive a predictive model. Such representations are usually expressed as vectors of numerical values, each quantifying a relevant characteristic of the instance through an appropriate measure. These numerical representations serve to identify different problem instances and allow to discriminate between them. In evolutionary computation (EC) terms, these quantified measures, known as features, thus describe the fitness landscape of a problem instance.

Fitness landscapes have been and still are extensively studied in EC. For a long time, the only accessible landscape properties were high-level and intuitive (e.g., degree of multimodality, separability, number of plateaus), and their major drawback is that they all require prior expert knowledge and thus cannot be automatically computed. More recently, however, a line of research under the umbrella term of exploratory landscape analysis (ELA) [START_REF] Mersmann | Exploratory Landscape Analysis[END_REF] has come up with new ways of proper numerical feature extraction (or more precisely approximation, as we have at our disposal only the samples that we have evaluated and to which a solution quality has been assigned). In addition to the original features sets proposed in [START_REF] Mersmann | Exploratory Landscape Analysis[END_REF], many new features sets, each quantifying some relevant problem characteristic, have since been incorporated [START_REF] Derbel | New features for continuous exploratory landscape analysis based on the SOO tree[END_REF][START_REF] Kerschke | Detecting Funnel Structures by Means of Exploratory Landscape Analysis[END_REF][START_REF] Lunacek | The dispersion metric and the CMA evolution strategy[END_REF][START_REF] Muñoz | Exploratory Landscape Analysis of Continuous Space Optimization Problems Using Information Content[END_REF]. These insights have fast given rise to nowadays very active research questions around the landscape-aware algorithm selection and/or configuration.

Since the EC community itself is traditionally predominantly focused on the development or the assessment of different landscape features and search heuristics, the relevance of selecting an appropriate machine learning model needed to provide information for the algorithm selection (AS) is often neglected. With the exception of selected communities such as AutoML, most authors, following common practices, typically apply easily accessible, off-the-shelf techniques such as default implementations of random forests (RF), support vector machines (SVM), or decision trees (DT) -available in the scikit library [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF], for example. As suggested in [START_REF] Hutter | Algorithm runtime prediction: Methods & evaluation[END_REF], random forests are considered to be the preferred technique when it comes to landscape-aware algorithm performance prediction, as it was empirically demonstrated that they outperformed other frequently used ML models (e.g., ridge regression, abovementioned SVMs, Gaussian processes, neural networks) in the context of combinatorial and mixed-integer problems. However, typically only a single ML model is trained to perform the task.

With that being said, hyper-parameter tuning of regression models might have a potential of largely improving the regression quality. In this work, we thus raise the question of the magnitude of impact that hyper-parameter tuning can have on both regression quality and algorithm selection, as the effects of different ML frameworks and their respective strengths and weaknesses have yet to be investigated in detail in the scenario of landscape-aware AS. We aim to highlight said potential of hyper-parameter tuning on a selected set of tree-based regression models, while arguing that the model's performance quality is in fact highly dependent on the problem set and the algorithm portfolio, which are case-specific.

Our Contribution and Results. We compare in this work the performance of 30 different regression models on the algorithm selection task suggested in [START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning[END_REF]. After a preliminary step of testing the quality of the following seven families of regression models with different hyper-parameter values tested via iterative grid search: Random Forests [START_REF] Breiman | Random forests[END_REF], Decision Trees [START_REF] Breiman | Classification and regression trees[END_REF], Bagging decision trees [START_REF] Breiman | Bagging predictors[END_REF], Lasso [START_REF] Tibshirani | Regression Shrinkage and Selection via the Lasso[END_REF], ElasticNet [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF], KernelRidge [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF], and PassiveAggressive [START_REF] Crammer | Online passive aggressive algorithms[END_REF], we retained only the models that largely outperformed the rest in terms of regression quality. The 30 selected models are all tree-based, and they are configurations of random forest, decision tree and bagging decision tree regression techniques. Note that, due to the rather small size of our data set, we have not considered techniques such as Neural Networks which are powerful for a huge quantity of data. We have not considered any classification techniques either (including Bayes classifier), as we are interested in predicting numerical performance values for each algorithm.

We train the 30 models on different data sets, taking as input landscape (ELA) features of a problem and outputting the fixedbudget performance of an algorithm. Two distinct ELA feature representations (based on different sample sizes for feature computation) describe each of 120 problem instances belonging to 24 problem classes that were used in our paper. Fixed-budget performances for a portfolio of 12 algorithms were recorded for different budgets of function evaluations (we consider budgets of 250, 500, and 1 000 function evaluations), measuring the target precision (i.e., the distance to the optimum). On top of that, two complementary regression approaches were adopted for each model and each data set: one that predicts true (unscaled) target precision values and another that predicts log 10 target precision, as suggested in [START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF].

We then use stratified 5-fold cross-validation to ensure that all problem instances were used in the test phase of our models, and build for each regression model and each data set an automated algorithm selector, which takes as input the predicted target precision of each of the twelve algorithms and which returns the algorithm with the best predicted performance. The true target precision of this selected algorithm is then compared to that of the actual best algorithm for that problem instance, which defines the loss that we associate to the regression model. Following the approach in [START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF], for each regression model and each data set, we also build an algorithm selector which combines the regression for the unscaled target precision with that for the logarithmic precision, favoring the latter for small target precision values, and favoring the former otherwise. In total, we evaluate for each of the 30 regression models, three different algorithm selectors (unscaled, logarithmic, combined), on 6 different data sets (3 different budgets of function evaluations and 2 different feature sample sizes).

The results of our study clearly indicate a need for appropriate tuning of the regression techniques, but more importantly, they raise an important question of how the chosen ML model can lead to highly varying results in the algorithm selection step. We argue that the untapped potential of a careful choice of the relevant ML model and its hyper-parameter configuration can be only fully exploited when taking into account some preliminary knowledge about the problem classes and the algorithms. This sub-explored area of research merits further investigation. We see differences of up to several orders of magnitude in the Root Mean Squared Error values of the different models, not aggregated across optimization algorithms. Even if we aggregate the errors, the differences between them are still as high as 60%. We further notice that different models perform differently on different types of problems, making it difficult to derive a general recommendation for which model to favor in which scenario. This, however, does not limit the relevance of our work, since training the different regression models is of negligible cost, in particular when compared to the efforts required for setting up the whole algorithm selection pipeline. In practice, the use of several ML techniques at the same time ("ensembles") is not uncommon -quite the contrary, in fact [START_REF] Mendes-Moreira | Ensemble Approaches for Regression: A Survey[END_REF]. We therefore also hope that our work motivates further investigations of ensemble learning techniques in the context of landscape-aware algorithm selection and configuration.

Related Work. Existing research in automated algorithm selection and configuration can be roughly positioned on one of the two main axes, depending on whether machine learning techniques utilized are supervised or unsupervised. In terms of unsupervised learning, reinforcement learning is the most predominant (see [START_REF] Biedenkapp | Towards White-box Benchmarks for Algorithm Control[END_REF][START_REF] Kerschke | Automated Algorithm Selection: Survey and Perspectives[END_REF] and references mentioned therein). When it comes to the the supervised approaches, approaches building upon exploratory landscape analysis (ELA) [START_REF] Mersmann | Exploratory Landscape Analysis[END_REF] are most frequent in the field. In particular, ELA-based regression have been applied to assess the effect problem features have on algorithm performance [START_REF] Liefooghe | Landscape-Aware Performance Prediction for Evolutionary Multiobjective Optimization[END_REF], to configure algorithms' parameters (per-instance algorithm configuration, PIAC) [START_REF] Belkhir | Per instance algorithm configuration of CMA-ES with limited budget[END_REF], as well as to select algorithms from a given portfolio [START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning[END_REF][START_REF] Muñoz | A Meta-learning Prediction Model of Algorithm Performance for Continuous Optimization Problems[END_REF]. See [START_REF] Kerschke | Automated Algorithm Selection: Survey and Perspectives[END_REF][START_REF] Muñoz | Algorithm selection for black-box continuous optimization problems: A survey on methods and challenges[END_REF] for comprehensive surveys of automated AS state-of-the-art methods and results. With regards to dynamic algorithm selection and configuration, search behavior or algorithms' state parameters can also influence the model recommendations, see [START_REF] Bajer | Gaussian Process Surrogate Models for the CMA Evolution Strategy[END_REF][START_REF] Derbel | New features for continuous exploratory landscape analysis based on the SOO tree[END_REF] and references mentioned therein.

PERFORMANCE REGRESSION 2.1 Experimental Setup

Regression Models. Both regression and classification as supervised learning techniques have been studied in the context of landscape-aware algorithm selection. The advantage of regression models over classification ones is in keeping track of the magnitude of differences between performances of different algorithms, as they predict numerical performance values.

Following best practices in this line of work, as mentioned above, for our analysis we have selected three different classes of regression models, namely Decision Tree [START_REF] Breiman | Classification and regression trees[END_REF], Random Forest [START_REF] Breiman | Random forests[END_REF] and Bagging Decision Tree [START_REF] Breiman | Bagging predictors[END_REF]. For each model class, the hyper-parameter configurations used are shown in Table 1.

Since the basic component unit of all considered models is a decision tree (both Random Forests and Bagging DTs are based on them), the hyper-parameter crit value can be one of the following three: mse (mean squared error), mae (mean absolute error) and so-called friedman mse -Friedman mean squared error. The minsplit hyper-parameter represents the minimum number of data instances a tree node has to contain in order to become a splitting node. Lastly, the nest hyper-parameter defines the number of decision trees needed to build a Random Forest or Bagging DT model. In total, we end up with 30 different regression models, with 6 different configurations for Decision Tree, 12 for Random Forest, and 12 for Bagging DT. Table 1 summarizes the chosen hyper-parameter values for different regression classes.

Table 1: Hyper-parameter values for the regression models

Model

Hyper-parameters DecisionTree

• 𝑐𝑟𝑖𝑡 ∈ {"𝑚𝑠𝑒", "𝑚𝑎𝑒", "𝑓 𝑟𝑖𝑒𝑑𝑚𝑎𝑛_𝑚𝑠𝑒"} (6 configs.)

• 𝑚𝑖𝑛𝑠𝑝𝑙𝑖𝑡 ∈ {4, 5} RandomForest • 𝑐𝑟𝑖𝑡 ∈ {"𝑚𝑠𝑒", "𝑚𝑎𝑒"} (12 configs.) • 𝑚𝑖𝑛𝑠𝑝𝑙𝑖𝑡 ∈ {4, 5} • 𝑛𝑒𝑠𝑡 ∈ {3, 6, 9} BaggingDT • 𝑐𝑟𝑖𝑡 ∈ {"𝑚𝑠𝑒", "𝑚𝑎𝑒"} (12 configs.) • 𝑚𝑖𝑛𝑠𝑝𝑙𝑖𝑡 ∈ {4, 5} • 𝑛𝑒𝑠𝑡 ∈ {3, 6, 9}
Benchmark problems. As commonly used in the continuous optimization community, our benchmark set of choice when measuring different algorithm performances was the noiseless BBOB testbed [START_REF] Hansen | Real-Parameter Black-Box Optimization Benchmarking: Experimental Setup[END_REF] on the COCO platform [START_REF] Hansen | COCO: a platform for comparing continuous optimizers in a black-box setting[END_REF], which is a dedicated environment for comparison of algorithm performance in continuous black-box optimization. It contains 24 different single-objective functions (noted here as FID 1-24), and different instances can further be generated based on those 24 initial definitions by rotation and translation in the objective space. We hence consider the first 5 instances of each function (IID 1-5), for a total of 120 problem instances. Lastly, we restrict our analysis to dimension 5.

Algorithm Portfolio. The algorithm portfolio we chose for this work was suggested in [START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning[END_REF] for its diversity. It consists of the following 12 algorithms: BrentSTEPqi [START_REF] Posík | Dimension Selection in Axis-Parallel Brent-STEP Method for Black-Box Optimization of Separable Continuous Functions[END_REF], BrentSTEPrr [START_REF] Posík | Dimension Selection in Axis-Parallel Brent-STEP Method for Black-Box Optimization of Separable Continuous Functions[END_REF], CMA-ES-CSA [START_REF] Atamna | Benchmarking IPOP-CMA-ES-TPA and IPOP-CMA-ES-MSR on the BBOB Noiseless Testbed[END_REF], HCMA [START_REF] Loshchilov | Intensive Surrogate Model Exploitation in Self-adaptive Surrogate-assisted Cma-es (Saacmes)[END_REF], HMLSL [START_REF] Pál | Benchmarking a hybrid multi level single linkagealgorithm on the bbob noiseless testbed[END_REF], IPOP400D [START_REF] Auger | Benchmarking the local metamodel CMA-ES on the noiseless BBOB'2013 test bed[END_REF], MCS [START_REF] Huyer | Benchmarking of MCS on the noiseless function testbed[END_REF], MLSL [START_REF] Pál | Benchmarking a hybrid multi level single linkagealgorithm on the bbob noiseless testbed[END_REF], OQNLP [START_REF] Pál | Comparison of multistart global optimization algorithms on the BBOB noiseless testbed[END_REF], fmincon [START_REF] Pál | Comparison of multistart global optimization algorithms on the BBOB noiseless testbed[END_REF], fminunc [START_REF] Pál | Comparison of multistart global optimization algorithms on the BBOB noiseless testbed[END_REF], and BIPOP-CMA-ES [START_REF] Hansen | Benchmarking a BI-population CMA-ES on the BBOB-2009 Function Testbed[END_REF]. Note that, due to the unavailability of the raw performance data for one of the algorithms in the original study, the BIPOP-CMA-ES was added instead of the missing one. The performance data of all twelve algorithms can be downloaded at [START_REF] Hansen | Data from the BBOB workshops[END_REF], but for our setting it was more convenient to extract the relevant figures from IOHprofiler [14]. Throughout this work, we focus on the fixed-budget performance, which is a scenario most often seen in real-world application where we can allow ourselves only a limited budget of function evaluations, and where the performance metric used is target precision of an algorithm, i.e., the distance between the best solution reached after a certain budget of function evaluations and the estimated optimal solution. Concretely, we consider 3 different budget sizes of 250, 500 and 1000 function evaluations across all algorithms from the portfolio for purpose of sensitivity analysis, and we restrict ourselves to a single algorithm run per problem instance. We show in Figure 1 the portfolio's target precisions reached after 1000 evaluations. We note that the algorithm performances are significantly less diverse for functions 15 through 20, 23 and 24 than for the other problems.

Problem features. Predictor variables for our regression models are vectors of ELA feature values, which quantify the landscape properties of each problem instance. Feature computation is done using the flacco package [START_REF] Kerschke | The R-Package FLACCO for exploratory landscape analysis with applications to multi-objective optimization problems[END_REF] for two distinct sets of uniformly sampled points and their evaluations. Samples are of sizes 50𝑑 (250) and 400𝑑 (2000) respectively for purpose of sensitivity analysis. 50 independent feature computations were performed for each sample size, as some of them can show low robustness on certain features [START_REF] Renau | Expressiveness and Robustness of Landscape Features[END_REF], and a median feature value was taken for each one. Following suggestions from [START_REF] Belkhir | Per instance algorithm configuration of CMA-ES with limited budget[END_REF][START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning[END_REF], we choose only those feature sets that do not require additional sampling during their computation; this way, we end up with a total of 56 feature values per problem instance, belonging to the following sets: classical ELA (y-Distribution, Levelset and Meta-model) [START_REF] Mersmann | Exploratory Landscape Analysis[END_REF], Dispersion [START_REF] Lunacek | The dispersion metric and the CMA evolution strategy[END_REF], Information Content [START_REF] Muñoz | Exploratory Landscape Analysis of Continuous Space Optimization Problems Using Information Content[END_REF] and Nearest-Better Clustering [START_REF] Kerschke | Detecting Funnel Structures by Means of Exploratory Landscape Analysis[END_REF] feature sets.

Performance Regression Quality of Different Models

Using the key elements described in Section 2.1, we establish two separate regression (true and log-) approaches for each of the 30 regression models, as suggested in [START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF]. The intuition behind doing so lies in the fact that we want to make use of the information captured in the exponent of the actual performance value, which can be interpreted as a distance level to the optimum (for two algorithms with actual target precisions of 10 -2 and 10 -8 , we can say that the latter is 6 distance levels closer to optimum than the former). In the remainder of the paper, we call these 2 approaches the unscaled and the log-approach, and we note that the unscaled one performs better in predicting target precisions "far away from" the optimum, while the log-one is more appropriate when targeting fine-grained cases which are already very close to the optimum. Following common ML practices, we perform a 5-fold stratified leave-one-instance-out cross-validation when training each model to reduce variability and obtain a higher model accuracy, thus carrying out the training on 4 out of 5 instances per each function, testing on the remaining one and combining the results over all folds.

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2
In order to assess and compare the accuracy of different predictions made by the regression models, we compute the Root Mean Square Error (RMSE) and its counterpart log-RMSE values per prediction, and aggregate them across problems and algorithms. The RMSE is a frequently used metric in the ML community and represents the standard deviation of prediction errors; put differently, it measures how spread out those errors are. In our fixed-budget regression, the prediction errors are the distance of the prediction to the true target value. Within this context, the RMSE is computed based on the actual, 'un-logged' data (both for the unscaled and log-approach), whereas the log-RMSE makes use of the 'logged' data.

Table 2 conveniently shows the best achieved prediction accuracies (RMSE and log-RMSE) of the log-approach for different algorithms in the portfolio, for the 2000-sample feature size due to space constraints, for all different budgets, aggregated across problems. Different 'Model' columns correspond to the regression model with the best quality in a specific scenario. Note that the RMlabeled models 1-6 refer to the Decision Tree regressors, 7-18 to the Random Forest regressors, and 19-30 to the Bagging DT regressors. This allows us to draw some first conclusions related to selecting the most efficient regressor, with the Decision Tree family being predominantly chosen in the low-budget setting when optimizing the RMSE, while the presence of the Bagging DT family is stronger when optimizing the log-RMSE in the same setting. Random Forest regressors, however, seem well-performing across the board, which is supported by the fact they are typically an all-round model of choice in related lines of research. Most importantly, we remark that, even when aggregated across problems, the choice of the best regression model is highly dependent on the setting we work in, and varies between algorithms. As illustrated in the example in Figure 2, which is the use-case of CMA-ES-CSA algorithm for budget of 1000 evaluations and 2000-sample feature size, the overall regression quality both in terms of RMSE (on 𝑥-axis) and log-RMSE (on 𝑦-axis), regarded as a Pareto front, i.e. a two-objective min-min problem, is higher for the log-based approach than for the unscaled one. However, it does not always have to be the case; depending on the algorithm, we can also observe situations in which the unscaled approach yields better results. Nevertheless, a remarkable diversity of regression model accuracies on different problems is easily noticed, and supports the idea of possibly having to resort to multiple regressors (each excelling in prediction of one sub-class of problems, for example) to achieve best results, which further supports the claim that ensemble-based models could be significantly more accurate than the standalone ones.

ELA-BASED ALGORITHM SELECTION

After examining the regression accuracy, we proceed to evaluate the performance of two simple algorithm selectors, based on the predictions of the unscaled and the logarithmic approach, respectively, for each of the 30 regression models. The unscaled selector recommends the algorithm for which the unscaled-based model predicted the best performance, and, similarly, the log-selector bases its decision on the best prediction of the log-model. To quantify how well selectors perform per problem instance, we compare the precision of the algorithm chosen by the selector for the instance at hand to the precision of the actual best algorithm for that instance.

We then indicate the overall quality of this selector by computing the RMSE and log-RMSE values (aggregated across all problem instances).

Both in unscaled as well as in the log-approach, some algorithms will be selected more often than others per different problem instance, as seen in Figure 3. For the budget of 1000 function evaluations and 2000-sample feature size, we observe that HCMA is chosen most consistently across different problem instances, while the choice of BIPOP-CMA-ES is very rare. We also notice that on specific instances, there could be one or two particularly frequently selected (thus good) algorithms which are not selected for other problems in the benchmark set (e.g., CMA-ES-CSA for the function 16 in the unscaled approach, or fmincon for the function 21 in the log-approach).

Combined Algorithm Selector. Following common practices in algorithm selection [START_REF] Bischl | ASlib: A benchmark library for algorithm selection[END_REF], we compare the qualities of our two selectors (unscaled and log-based) with two different baselinesone of them being the virtual best solver (VBS, also called the oracle), which gives a lower bound as it always chooses the best-performing algorithm per each problem instance without any additional cost, so it reflects the best performance that could be achieved theoretically. The other baseline is the single best solver (SBS), which stands for the overall (aggregated) performance of the best-performing algorithm from the portfolio. The SBS baselines vary depending on the budget and feature size, and we can also distinguish between the unscaled-approach SBS and the log-approach SBS.

We plot in Figure 4 the quality of the selectors over all problem instances on RMSE and log-RMSE axes, again treating them as a Pareto front with the objective of minimizing both errors, and we see that different unscaled-and log-based selectors already outperform the majority of single algorithms from the portfolio on both RMSE and log-RMSE. We also want to incorporate the observation reported in Section 2.2 that the unscaled approach is better in predicting higher target precision, while the log-approach has better accuracy when targeting smaller performance values.

We hence establish a combined regression approach for all the models in order to benefit from their two complementing strengths. Here again we can define a combined VBS, which is the one that, for each model and problem instance, chooses the better of the two recommended algorithms by the unscaled-and the logarithmic approach, respectively, using the following rule: if the target precision of an algorithm, as predicted by the log-model, is smaller than a certain threshold, we use the log-approach, whereas we use the recommendation of the unscaled approach otherwise. The threshold value chosen here is 0.9 in order to ensure selecting the log-approach recommendation for fine-grained precisions and vice versa. Note that that a complete sensitivity analysis with respect to the threshold value was described in [START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF], where ad-hoc threshold optimization was performed for each regression model in order to minimize RMSE and log-RMSE. Due to space constraints, the sensitivity analysis was out of scope of this work. We apply this strategy to all 30 regression models, and the results for the budget of 1000 evaluations and the 2000-sample feature size are seen in Figure 4. The combined approach clearly outperforms any of the simple regression approaches, the single algorithms and even the combined VBS selectors. This finding highlights the potential of the combined selector, which boosts the quality of its standalone components even in the case when they might not be the optimal regression models for a specific algorithm portfolio and s problem set.

SENSITIVITY ANALYSIS

Lastly, we highlight the differences in regression quality obtained by using different budgets and feature sizes. We have purposefully randomly selected one regression model (RandomForest crit.mse minsplit.4 nest.9) and one algorithm (HCMA) to point out the diversity of regression on different problem instances. The data in Table 3 corresponds to the log-approach of said selector, using 2000-sample feature size, in order to be consistent in terms of the showcased use-case throughout the paper. We immediately observe extremely large RMSE values for the budget of 250 function evaluations, which drastically decrease for the budgets of 500 and 1000. One possible interpretation of this finding could be that in raw performance data, whereas the budget of 250 evaluations was not big enough to allow for getting closer to the optimum in case of certain functions, increasing the budget seem to facilitate getting a better estimate of the optimal solution, thus making a more straightforward way for the regression to perform better. Finally, this again stresses the possible importance of personalizing the regression models to the actual optimization problems, as the results can vary drastically between problem instances.

DISCUSSION, CONCLUSIONS AND FUTURE WORK

While most landscape-aware algorithm selection, design, and configuration studies in the context of numerical black-box optimization do not pay great attention to the configuration of the ML model, we have demonstrated in this work that both the choice of regression technique and its parametrization can have significant impact on the performance of the trained models. Using a classical experimental design from the context of automated algorithm selection [START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF][START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning[END_REF], we have analyzed 30 different regression models, applied them to both the log-and to the unscaled performance data, trained an automated algorithm selector, and analyzed its performance through stratified 5-fold cross validation. We have seen that the regression quality of the different models can vary by several orders of magnitude. This reinforces and justifies the need for meticulously choosing the ML model and its hyper-parameter configuration, as we have seen that picking any single model and tuning it might not at all provide good results (i.e., the discarded ML models from the preliminary step of this work). Differences in the regression models' quality also lead to very diverse performance portfolios in the algorithm selection task, although the impact there is somewhat less severe, since wrong predictions can still result in a lucky choice of algorithms. Additionally, when considering ensembles, hyper-parameter tuning can be expected to further improve upon performance gains, which will depend on the used data set and learning scenario.

Our study suggests that the selection of the ML techniques should be performed with care. It also suggests that the quality of different regression techniques can vary between different types of problems, so that we cannot give a "one size fits all" recommendation for which regression models or which parametrization to favor. As a rule of thumb, different BaggingDT and RandomForest instances provide better results in terms of log-RMSE, for both feature extraction sample sizes and all three budgets of function evaluations. For the RMSE performance criterion, in contrast, DecisionTrees provides best results for some of the algorithms. The important question of choosing the absolute best performing ML model for a certain problem set and algorithm portfolio remains open, but this preliminary work stresses the significance of the efforts that should go towards developing more advanced mechanisms to select the most appropriate one and its hyper-parameter configuration.

We note that the computational costs of training different regression models is rather negligible for the data sizes commonly studied in the landscape-aware black-box algorithm selection and configuration context, which means a validation step can be added before deciding which model to choose and apply to the real usecases (i.e., in the test phase). In general, we believe that the current common practice of studying stratified 5-fold cross validation on the BBOB functions is too limited to give an accurate impression about the potential of carefully choosing the ML model for the automated algorithm selection and configuration in practice. We therefore plan to massively extend our study by adding to our data sets performance data from the Nevergrad platform [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF], which offers benchmark data for very broad ranges of optimization problems on its frequently updated dashboard.

One problem to overcome in cross-validation across different benchmarking platforms is the fact that one needs to ensure that the data corresponds to the same instances of the algorithms -a "CMA-ES" in one platform may be much different than a CMA-ES implementation in another. We therefore believe that a common algorithm repository, interfaced with the various benchmarking suites, would be a useful step towards a better re-usability of results and better training sets for the automated design, selection, and configuration of black-box optimization techniques.

Figure 1 :

 1 Figure1: Single-run target precision of the 12 algorithms on the first five instances of 24 BBOB functions. They represent our portfolio from which we want to select the best-performing algorithm for an unseen problem instance.

Figure 2 :

 2 Figure2: Overall prediction quality of different regression models (RMSE vs. log-RMSE), from both the unscaled (in blue) and the log-approach (in red) on the CMA-ES-CSA algorithm, with the budget of 1000 function evaluations and 2000-sample feature size. Minimizing both error values as a Pareto front, we see that log-based regression models perform significantly better than the unscaled ones.

FIDFigure 3 :

 3 Figure3: Heatmap of the selection frequency of each algorithm from the portfolio per problem instance, both for the unscaled (left) and log-approach (right), for the budget of 1000 function evaluations and the feature size of 250 samples, showing selected functions only for reasons of space.

Figure 4 :

 4 Figure4: Quality of the algorithm selectors (RMSE vs. log-RMSE) for different regression approaches, including the combined selectors, the virtual best combined selectors (combinedVBS), and the selectors based on consistently using a certain algorithm across the board ('Algo' selectors in the legend, among which we can identify the SBS), for the budget of 1000 function evaluations and the 2000-sample feature size. Note that some of the selectors, notably the 'Algo' ones, have been excluded from the figure as otherwise it disrupted the visibility of the best ones.

Table 2 :

 2

							Feature size 2000
			Budget 250			Budget 500			Budget 1000
	Algorithm	RMSE	Model logRMSE Model	RMSE	Model logRMSE Model	RMSE	Model logRMSE Model
	BIPOP-CMA-ES 10368.95 RM27	1.65	RM13 1489.86	RM9	1.83	RM20	73.18	RM7	1.94	RM12
	BrentSTEPqi	58096.65 RM8	2.22	RM30 57828.22 RM9	2.76	RM12 57040.98 RM9	2.84	RM11
	BrentSTEPrr	59475.56 RM3	2.15	RM30 58039.56 RM16	2.70	RM30 57993.93 RM21	2.90	RM12
	CMA-ES-CSA	10216.53 RM7	1.62	RM13	280.26	RM13	1.95	RM30	6.78	RM21	2.38	RM12
	fmincon	17.73	RM5	1.64	RM18	10.36	RM3	1.96	RM18	6.23	RM20	1.64	RM23
	fminunc	254.93	RM21	2.20	RM11	7.05	RM7	2.34	RM29	6.83	RM2	2.35	RM29
	HCMA	1651.12 RM13	1.70	RM23	4.29	RM23	2.10	RM18	2.82	RM14	2.77	RM12
	HMLSL	16.83	RM2	1.62	RM11	10.45	RM6	1.95	RM23	4.01	RM24	1.87	RM12
	IPOP400D	3120.75 RM14	1.72	RM18 1204.75 RM19	1.71	RM11	33.66	RM7	1.99	RM11
	MCS	2514.61	RM2	2.40	RM24 2233.27	RM7	2.74	RM11 1992.51 RM24	2.87	RM11
	MLSL	19.54	RM7	1.77	RM17	11.45	RM6	2.06	RM23	5.39	RM6	2.07	RM22
	OQNLP	26.36	RM13	2.40	RM30	11.34	RM20	2.17	RM17	9.10	RM20	2.29	RM17

Best quality of the log-predictions (RMSE vs. log-RMSE) for each algorithm from the portfolio, for feature size of 2000 samples. Note that the labels in columns named 'Model' correspond to the regression model achieving the said best prediction quality. Different models are the best-performing ones across the portfolio for different budgets of function evaluations.

Table 3 :

 3 Sensitivity analysis of regression quality for the log-approach of the 𝑅𝑎𝑛𝑑𝑜𝑚𝐹𝑜𝑟𝑒𝑠𝑡 𝑐 𝑟𝑖𝑡 .𝑚𝑠𝑒 𝑚 𝑖𝑛𝑠𝑝𝑙𝑖𝑡 .4 𝑛 𝑒𝑠𝑡 .9 regression model for the HCMA algorithm for the feature size of 2000 samples, with respect to the different algorithm budgets. Note that, for reasons of space, we showcase only values of the first instance of the 24 BBOB functions.

		FID	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24
		Budget 250	RMSE logRMSE 0.03 15.35 0.01 0.00 14.80 0.12 0.00 4.01 0.66 1.00 229.24 39.73 44127.00 0.01 21357821.80 3243.65 20853295.16 53.95 1.12 656.09 341.45 7.21 38.76 0.09 6.81 0.49 0.72 3.89 247.95 0.40 0.21 1.12 0.00 1.94 0.04 0.65 0.11 1.19 0.19 0.58 0.40 0.18 0.12 0.15 0.03 0.06 0.09 0.06
	Feature size 2000	Budget 500	RMSE logRMSE 0.01 23.83 7.65 0.67 21.68 0.06 0.00 0.16 1.77 1.00	4.83 1.98	0.52 4.35	10.35 0.28	0.13 0.10	0.00 1.09	0.00 1.50	0.00 0.02	0.00 0.67 13.13 0.01 251.70 72.77 0.11 4.18 0.03 0.84 0.70 0.26 0.55 6.62 0.39 1.09 2.49 0.06 0.09 0.04 0.02 0.02	5.75 0.01
		Budget 1000	RMSE logRMSE 0.00 0.02 80.68 2.80 11.15 0.01 0.00 2.58 0.66 1.00	0.00 1.28	0.41 16.98	0.00 0.01	0.00 3.85	0.00 0.04	0.00 0.53	0.00 0.26	0.00 0.15	0.00 1.01	0.99 33.19	0.10 0.02	0.00 0.18 0.14 2.86 0.10 3.72 0.23 2.96 2.86 0.15 1.48 0.02 3.48 0.01	1.65 0.00

Acknowledgments. This work has been supported by the Paris Ile-de-France region, and the Slovenian Research Agency (research core funding No. P2-0098, project No. Z2-1867, and grant number PR-10465).


Tome Eftimov
email: tome.eftimov@ijs.si

Anja Jankovic
email: anja.jankovic@lip6.fr

Gorjan Popovski
email: gorjan.popovski@ijs.si

Carola Doerr
email: carola.doerr@lip6.fr

Peter Korošec
email: peter.korosec@ijs.si

Personalizing Performance Regression Models to Black-Box Optimization Problems

Keywords: Computing methodologies → Continuous space search, Randomized search, • Theory

Accurately predicting the performance of different optimization algorithms for previously unseen problem instances is crucial for high-performing algorithm selection and configuration techniques. In the context of numerical optimization, supervised regression approaches built on top of exploratory landscape analysis are becoming very popular. From the point of view of Machine Learning (ML), however, the approaches are often rather naïve, using default regression or classification techniques without proper investigation of the suitability of the ML tools. With this work, we bring to the attention of our community the possibility to personalize regression models to specific types of optimization problems. Instead of aiming for a single model that works well across a whole set of possibly diverse problems, our personalized regression approach acknowledges that different models may suite different types of problems. Going one step further, we also investigate the impact of selecting not a single regression model per problem, but personalized ensembles. We test our approach on predicting the performance of numerical optimization heuristics on the BBOB benchmark collection.

CCS CONCEPTS

•

recommendations to tackle these problems, users can rely today on the support of powerful AutoML techniques, which take as input some descriptors of the problem (and possibly also the specific instance) at hand, and which then recommend one or several algorithm instances that the users can apply to their problems. At the heart of many of these AutoML techniques are regression or classification algorithms, which -in one way or the other -predict which algorithm (configuration) could be most suitable for the given task.

In evolutionary computation, a particularly active research question concerns the development of landscape-aware AutoML methods [START_REF] Belkhir | Per instance algorithm configuration of CMA-ES with limited budget[END_REF][START_REF] Derbel | New features for continuous exploratory landscape analysis based on the SOO tree[END_REF][START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF][START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning[END_REF][START_REF] Mersmann | Exploratory landscape analysis[END_REF][START_REF] Muñoz | Algorithm selection for black-box continuous optimization problems: A survey on methods and challenges[END_REF]. Classifying as supervised learning approaches in the broader ML context, the main idea of these methods is to extract useful properties of the problem (instance) at hand, and to use this information to predict which algorithm (configuration) will perform particularly well on it. Landscape-aware AS and AC are trained on performance data, which has been collected through previous optimization tasks or via systematic benchmarking.

Since the focus of most of the studies in evolutionary computation is on the development or assessment of different features, the Machine Learning (ML) model to derive the recommendation is often neglected, and authors satisfy themselves by applying off-theshelf techniques such as default implementations of random forests (RF), support vector machines (SVM), or decision trees (DT), as, for example, available in the scikit toolbox [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF]. It is well known, however, that different predictive models can show quite different performances on different ML tasks. A more systematic approach towards the model selection seems therefore in order.

Our Contribution: The main goal of our work is to analyze to what extent state-of-the-art landscape-aware AS and AC models could benefit from a more careful choice of the ML tools, and, more concretely, how their complementarity can be leveraged to obtain good predictions for broad sets of optimization problems. We evaluate a way of extending the current practice of deploying a single predictive model by combining ensemble regression and personalized regression. While ensemble learning is well-known and the de-facto standard in several ML-applications [START_REF] Zhang | Ensemble machine learning: methods and applications[END_REF], the idea to personalize the regressions is an original research contribution that we propose in this work. In a nutshell, the key idea is that different regression models work best for different types of problems, so that we can improve regression quality by automatically selecting the one(s) that showed best performance for similar problems. We test the impact of each of the suggested extensions on a portfolio of 12 algorithms from the BBOB workshop series [START_REF] Hansen | Data from the BBOB workshops[END_REF], for which task ourselves with predicting the solution quality after a fixed budget of function evaluations, a setting previously suggested in [START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF].

Results: We find that the regression quality improves for 58%-70% of the tested problems, depending on the comparison scenario, which nicely demonstrates that the current practice in performance regression used within the evolutionary computation community has quite some untapped potential.

The computational overhead for training and applying the personalized models is also negligible for the tasks performed in this study. This happens because we are working only with 120 instances from 24 problems. When moving to larger data sets of several GB or even TB in size (as often considered in ML applications), however, we need to consider the impact of data size and data quality on the learning algorithms performances. With larger amounts of data, the so-called offline learning (as used in this paper) can become computationally inefficient. In such cases, one option is to use random or stratified sampling if possible, which can reduce the data size and still preserve the relevant information found in the original data set.

Broader Impact of our Approach: The idea to personalize the regression models is not restricted to performance regression, and not even to optimization. Based on our findings presented in this work, we consider further applications, for example in personalized medicine, as an exciting avenue for future work, since predictive models that are specifically developed for different genotypes and/or phenotype may allow better recommendations that one-size-fits all predictive models, which -unfortunately -are known to come with biases that can cause severe harm.

Note that personalized approach is different from training individual models separately, in that the classification which model to use is done in a data-driven way, and not by an external entity. While we use in this work classification to assign problem instances to problem classes, our approach can easily be extended to allow for interpolation between personalized models. How much such an additional layer would contribute, however, remains to be evaluated in future work. We compare our results to the approach in which the classification step can be omitted and the ground truth problem class is known (this is the "Ensemble-ground" method in Sec. 4.1.) Based on the results, the misclassification of the problem class that leads to selecting a personalized model does not have a significant loss. Even more it can improve the end predictions, however this happens in cases where landscape representations of the problems instances are quite similar.

Structure of the Paper: We summarize selected related works in Sec. 2. The pipeline to create the personalized regression models is presented in Sec. 3, and we discuss a specific use-case in Sec. 4. A critical discussion of our approach and avenues for future work are presented in Sec. 5.

Availability of Data and Code: All project data and code is available at [START_REF] Eftimov | GitHub repository containing all source code and data of the study presented in this paper[END_REF]. Note that this repository does not only include data for the use-cases presented in Sec. 4, but also for all twelve algorithms from the selected portfolio.

RELATED WORK

Our work integrates into ongoing research on landscape-aware algorithm selection (AS) and configuration (AC) [START_REF] Blot | Automatic Configuration of Multi-Objective Local Search Algorithms for Permutation Problems[END_REF][START_REF]Automated Machine Learning -Methods, Systems, Challenges[END_REF][START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF][START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning[END_REF][START_REF] Liefooghe | Landscape-Aware Performance Prediction for Evolutionary Multiobjective Optimization[END_REF], and more specifically to the task of per-instance algorithm selection (PIAS [START_REF] Kerschke | Automated Algorithm Selection: Survey and Perspectives[END_REF]) and configuration (PIAC [START_REF] Belkhir | Per instance algorithm configuration of CMA-ES with limited budget[END_REF][START_REF] Hutter | Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms[END_REF]), which aims at recommending a best suited algorithm (configuration) based on the characteristics of the specific problem instance at hand.

In landscape-aware AS/AC, recommendations are based on features of the problem instances, which are estimated from a finite set {(𝑥, 𝑓 (𝑥))} of evaluated samples via so-called exploratory landscape analysis [START_REF] Mersmann | Exploratory landscape analysis[END_REF]. Most research in this area focuses on the definition or the analysis of features that describe certain characteristics of the optimization problem, and their suitability for automated algorithm selection, configuration, and design. See [START_REF] Lunacek | The dispersion metric and the CMA evolution strategy[END_REF][START_REF] Mersmann | Exploratory landscape analysis[END_REF][START_REF] Muñoz | Exploratory Landscape Analysis of Continuous Space Optimization Problems Using Information Content[END_REF] for examples and further references.

Where dynamic ("online") algorithm configuration is considered, recommendations can also be based on the search behavior or algorithms' state parameters, see [START_REF] Bajer | Gaussian Process Surrogate Models for the CMA Evolution Strategy[END_REF][START_REF] Derbel | New features for continuous exploratory landscape analysis based on the SOO tree[END_REF] for examples and further references.

Concerning the ML techniques, we already mentioned that ensemble learning is quite standard nowadays, as it can result into much better predictions that using only one predictive model [START_REF] Mendes-Moreira | Ensemble approaches for regression: A survey[END_REF][START_REF] Qiu | Ensemble deep learning for regression and time series forecasting[END_REF]. Our idea to build personalized regression models is based on a recent study in personalized nutrition [START_REF] Ispirova | P-NUT: Predicting NUTrient Content from Short Text Descriptions[END_REF], which showed that for different clusters of recipes, different regression models provided the best prediction of the macronutrient values. None of the single predictive model could achieve similar performance in that task. This inspired us to automate the model selection in the context of landscape-aware AS/AC.

PERSONALIZED ML MODELS

To introduce our personalized performance regression pipeline, let us assume that we face a fixed-budget performance regression task, i.e., we assume to be aiming at predicting algorithms' solution quality after a fixed budget of function evaluations has been exhausted. The pipeline presented below can be used for other ML tasks, but the restriction to a specific use case eases the presentation considerably.

The high-level approach is depicted in Fig. 1. The four main steps to obtain a performance prediction 𝑦 for a given problem instance 𝑖 are as follows:

(1) We first apply a feature extraction method to obtain a description of this instance 𝑖. In our context, the features are computed with exploratory landscape analysis (see Sec. 4 for details). (2) We use the instance description to assign instance 𝑖 to a class 𝐶 (𝑖) (i.e., we perform multi-class classification) to obtain a set {𝑦 1 , . . . , 𝑦 𝑚 } of different performance predictions, one per regression model (RM). ( 3) Combine these values to the final prediction 𝑦 = 𝑚 𝑗=1 𝑤 𝑗 (𝑖)𝑦 𝑗 , using the weighting scheme 𝑤 1 (𝑖), . . . , 𝑤 𝑚 (𝑖) of class 𝐶 (𝑖).

The association of the RMs to the different classes as well as the computation of the class-specific weights is handled in a prior ("offline") training phase. Its most relevant steps are illustrated in Fig. 2.

Assuming that we have a set of training instances which are grouped into 𝑛 classes 𝐶 1 , . . . , 𝐶 𝑛 (in our case, these are the problem instances), a set of potential RMs, which are grouped in to 𝑚 classes 𝐴 1 , . . . , 𝐴 𝑚 ,1 , and fixed-budget performance data for an algorithm A, the training phase comprises the following steps:

(1) Compute for each training instance a representation, ideally using the same feature extraction technique that will be used in the applications (i.e., in the "test phase" in proper ML terminology). ( 2) Each RM instance uses the problem representation and the algorithm performance data to train a predictive model. (3) Each RM is evaluated according to its regression performance on the train instances within each optimization problem. (4) For each problem class 𝐶 (𝑖), we select from each RM class 𝐴 𝑗 the configuration 𝑎 𝑗 (𝑖) which achieved the best performance. [START_REF] Blot | Automatic Configuration of Multi-Objective Local Search Algorithms for Permutation Problems[END_REF] We then calculate the importance of each RM 𝑎 𝑗 (𝑖) via a min-max normalization. That is, if we denote by 𝑞(𝑖) = (𝑞 1 (𝑖), . . . , 𝑞 𝑚 (𝑖)) the vector of performance measures for each of the 𝑚 selected configurations for class 𝐶 (𝑖), the importance of 𝑎 𝑗 (𝑖) is computed as

𝑞 𝑗,norm. (𝑖) = max(𝑞(𝑖)) -𝑞 𝑗 (𝑖) max(𝑞(𝑖)) -min(𝑞(𝑖)) , (1)
where we assume a performance measure for which lower values are better (typically, deviation from the ground truth is measured in one way or the other). We then compute the vector 𝑤 (𝑖) = (𝑤 1 (𝑖), . . . , 𝑤 𝑚 (𝑖)) of weights 𝑤 𝑗 = 𝑞 𝑗,norm. (𝑖)

𝑚 𝑗 =1 𝑞 𝑗,norm. (𝑖) , which are used in the fourth step of the application phase described above.

USE-CASE: ELA-BASED FIXED-BUDGET PERFORMANCE REGRESSION

We evaluate our personalized ML pipeline on a standard regression task which aims at predicting the final solution quality of a black-box optimization algorithm after a fixed number of function evaluations. The experimental setup is described in Sec. 4.1. In total, we apply our approach to twelve different optimization algorithms. As a consequence of space limitations, however, we present here only some selected results (Sec. 4.2). A few sensitivity analyses, to test the robustness of our approach, are performed in Sec. 4.3.

Experimental Setup

The experiments were performed on 13-inch MacBook Pro with 2.8 GHz Quad-Core Intel Core i7 processor and 16 GB of RAM. The raw regression data has been collected using Python implementation, while the personalized approach and all the evaluations have been performed using R. Our fixed-budget regression is inspired by [START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF], but applied here to the more diverse set of algorithms suggested in [START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning[END_REF]. Concretely, we aim at predicting the performance of the following 12 algorithms: BrentSTEPqi [START_REF] Posík | Dimension Selection in Axis-Parallel Brent-STEP Method for Black-Box Optimization of Separable Continuous Functions[END_REF], BrentSTEPrr [START_REF] Posík | Dimension Selection in Axis-Parallel Brent-STEP Method for Black-Box Optimization of Separable Continuous Functions[END_REF], CMA-ES-CSA [START_REF] Atamna | Benchmarking ipop-cma-es-tpa and ipop-cma-es-msr on the bbob noiseless testbed[END_REF], HCMA [START_REF] Loshchilov | Bi-population CMA-ES agorithms with surrogate models and line searches[END_REF], HMLSL [START_REF] Pál | Benchmarking a hybrid multi level single linkagealgorithm on the bbob noiseless testbed[END_REF], IPOP400D [START_REF] Auger | Benchmarking the local metamodel CMA-ES on the noiseless BBOB'2013 test bed[END_REF], MCS [START_REF] Huyer | Benchmarking of MCS on the noiseless function testbed[END_REF], MLSL [START_REF] Pál | Benchmarking a hybrid multi level single linkagealgorithm on the bbob noiseless testbed[END_REF], OQNLP [START_REF] Pál | Comparison of multistart global optimization algorithms on the BBOB noiseless testbed[END_REF], fmincon [START_REF] Pál | Comparison of multistart global optimization algorithms on the BBOB noiseless testbed[END_REF], fminunc [START_REF] Pál | Comparison of multistart global optimization algorithms on the BBOB noiseless testbed[END_REF], and BIPOP-CMA-ES [START_REF] Hansen | Benchmarking a BI-population CMA-ES on the BBOB-2009 function testbed[END_REF]. Note here that the latter does not appear in the portfolio analyzed in [START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning[END_REF], but it was added since for one algorithm from the original study the raw performance data was missing.

As performance measure of these algorithms we use their single run fixed-budget target precision after 250, 500, and 1 000 fitness evaluations, respectively, and this for the first five instances of each of the 24 BBOB functions [START_REF] Hansen | Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions[END_REF] provided by the BBOB platform [START_REF] Hansen | COCO: A platform for comparing continuous optimizers in a black-box setting[END_REF]. This performance data is available at [START_REF] Hansen | Data from the BBOB workshops[END_REF], but for our work we used the post-processed and more conveniently queryable repository available at [START_REF] Wang | IOHanalyzer: Performance Analysis for Iterative Optimization Heuristic[END_REF]. The representations of the 120 problem instances are based on exploratory landscape analysis [START_REF] Mersmann | Exploratory landscape analysis[END_REF]. The features were computed by the R-package flacco [START_REF] Kerschke | The R-Package FLACCO for exploratory landscape analysis with applications to multi-objective optimization problems[END_REF], using the uniform sampling procedure with a budget of 400𝑑 (a sensitivity analysis for a 50𝑑 sampling budget will be presented in Sec. 4.3). Following [START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF] we used 56 feature values per instance, which are grouped into five features groups: disp, ela_level, ela_meta, ic, and nbc (see [START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF] for full names and references). To stick to common practice in the evolutionary computation community, we take the raw feature values, i.e., we do not normalize these values nor do we perform any representation learning prior to feeding the values to our ML approaches. To evaluate the personalized ensembles, we used stratified 5cross fold validation, where each fold consists of the first, second, third, fourth, and fifth instances for each problem, respectively. That is, we repeat the whole training and testing process described in Sec. 3 five times, each time leaving out one fold for the testing phase and using the other four for the training. Note that the personalized ensembles for the same problem can be different across the five different runs, since different training data is used. An example will be presented in Table 3. We have taken the stratified 5-cross fold validation approach, since it is the predominantly used one in evolutionary computation [START_REF] Derbel | New features for continuous exploratory landscape analysis based on the SOO tree[END_REF][START_REF] Jankovic | Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants[END_REF].

In a first evaluation of our personalization approach, we used seven regression techniques: Lasso [START_REF] Tibshirani | Regression Shrinkage and Selection via the Lasso[END_REF], ElasticNet [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF], Kernel-Ridge [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF], PassiveAggressive [START_REF] Crammer | Online passive aggressive algorithms[END_REF], DecisionTree [START_REF] Breiman | Classification and regression trees[END_REF], RandomForest [START_REF] Breiman | Random forests[END_REF], and BaggingDT [START_REF] Breiman | Bagging predictors[END_REF]. We applied iterative grid search to each of them to test different hyper-parameters. Evaluating the seven regression techniques using the mean absolute errors (MAE) of the test folds from the stratified 5-cross fold validation, only three regression techniques were selected for further investigation, DecisionTree, RandomForest, and BaggingDT. The tested hyperparameter combinations for each of these techniques are summarized in Table 1. Since all selected techniques are based on trees, the crit parameter can be "mse" -mean squared error, "mae" -mean absolute error and "friedman_mse" -Friedman mean squared error. Regarding the minsplit hyper-parameter, it is the minimum number of data instances a node contains in order to be slit. The nest hyper-parameter defines how many Decision Trees will be built in the RandomForest/BaggingDT RM. These range of the hyperparameters have been selected concerning the data set size and the guidelines available in ML to avoid overfitting. In total, we ended up with 430 different RMs, 30 configurations of Decision-Tree, 200 configurations for RandomForest, and 200 configurations of BaggingDT.

To select the best RM within the three selected regression techniques and to learn their weights for each problem class separately, the mean absolute error from the results obtained on the training fold has been used. This has been done since the data set we used is relatively small, and we cannot split it into train, validation, and test set. In the future, when working with much bigger data sets, validation sets should be used for the weight calculation. Currently, this can lead further to overfitting on the training set, however it can also provide preliminary information about how the methodology fits for new test problem instances.

To associate an instance 𝑖 to a problem class 𝐶 (𝑖), an ensemble with majority vote of three multi-class classification algorithms

✓ ✓ - - Ensemble-class ✓ ✓ ✓ - ✓ Best-train - - ✓ - - Best-train-instance ✓ - ✓ - - Best-test - - - ✓ -
(BaggingDT_crit-entropy_minsplit-2_nest-9, RandomForest_entro-py_nest-9_min-2, and RandomForest_gini_nest-9_min-2) was trained on the same folds used for building the personalized regression models. The hyper-parameters used for training the classifiers are the same as the regression models, with the difference that Gini impurity (gini) or the Information Gain (entropy) [START_REF] Friedman | The elements of statistical learning[END_REF] were used for splitting the nodes in the individual tree. They both measure the impurity of a node.

The comparison is done in the following scenarios, which are summarized in Table 2:

• Ensemble-ground: personalized ensembles for each problem; true problem class, 𝐶 (𝑖), to which the test problem instance 𝑖 belongs, is known as a priori information. That is, we assume in this model that we know which problem class the instance belongs to and our key objective is to evaluate the appropriateness of the class-specific ensemble. • Ensemble-class: this is the approach described in Sec. 3, i.e., we have personalized ensembles for each problem, and the problem class 𝐶 (𝑖) has to be guessed from the instance representation by the classifier. When the classifier correctly predicts the truth problem class, the prediction will be identical to that of the Ensemble-ground model. When the instance is mis-classified to a different problem class, the noise presented in the classifier will affect the selection of the relevant RMs (i.e., RMs for different problem will be selected), which influences the end prediction (which can go in both ways, as we shall see below). • Best-train: the best RM from the three regression techniques is selected based on the MAE obtained across all problems from the training folds (i.e., one RM for all problems). • Best-train-instance: the best RM is selected in the same scenario as the Best-train, but for each problem separately (i.e., not across all problems). Selecting the Best-train-instance model for each problem is a special case of personalized models (i.e., each problem has its own best RM, but we do not combine the predictions of several model for the final output). • Best-test: the best RM from the three regression techniques is selected based on the MAE obtained across all problems from the test folds (i.e., one RM for all problems).

Note that the first four above-mentioned models are learned without evaluating problem instances from the test folds, whereas testing is needed to select the Best-test RM.

BIPOP-CMA-ES Performance Prediction

To evaluate the proposed methodology, the scenario of BIPOP-CMA-ES performance prediction is explored. The experiment is performed for a fixed budget of 1000 function/solution evaluations, and an ELA feature portfolio calculated using a 400𝑑 sample size. The personalized ensembles were trained in two scenarios: once to predict the original target precision achieved, and once to predict the natural logarithm value of the target precision. Regardless of how the target is represented, the benefits of using the proposed methodology is the same. Since this paper is required to adhere to a page limit, we present the models with the natural logarithm value of the target precision. Figure 3 shows the distribution of the absolute error obtained from each test fold for each problem separately in five different scenarios: i) Best-test, ii) Best-train, iii) Best-train-instance iv) Ensembleclass, and v) Ensemble-ground. The Best-test model is RandomFore-st_crit-mse_minsplit-6_nest-20, while the Best-train model is Deci-sionTree_crit-mae_minsplit-4. Table 3 presents the models used in the personalized ensembles for the sixth problem (i.e., Attractive Sector Function) for each fold.

Comparing a single RM vs. personalized ensembles: This comparison involves comparing the results obtained by using a single RM that works well across all problems (i.e., Best-train or Best-test) with the results obtained using the personalized ensembles (i.e., Ensemble-class and Ensemble-ground). By comparing the Best-train, Ensemble-class, and Ensemble-ground models, looking in the medians from the boxplots, it is obvious that the personalized ensembles (i.e., Ensemble-class, and Ensemble-ground) are better than the Best-train for 14 out of 24 optimization problems (i.e., 6, 7, 9, 10, 13, 14, 15, 16, 17, 19, 21, 22, 23, and 24). This comparison actually involves models that have never seen the test instances. More promising results are actually obtained when comparing the Ensemble-class and Ensemble-ground models with the Best-test model. In this case, the personalized ensembles are better in 15 out of 24 problems (i.e., 1, 2, 4, 5, 6, 7, 8, 9, 12, 13, 14, 15, 16, 21, and 24). In addition, we should point out that the selection of the Best-test is done using the information from the test instances, that have never been seen by the personalized models. Table 4 presents the median absolute error obtained from each test fold for each problem separately by the four different regression models. Comparing the median absolute values we can see that the gain by using the personalized ensembles varies between the problems, but this also results from the different target precision range between the problems. Comparing the ground personalized ensembles with the personalized ensembles combined with classification: By comparing the median values between the Ensemble-class and Ensembleground models, we can actually see the influence of the classifier on the end result. When there is a difference between the end prediction results obtained by both models, it means that the classifier predicted the wrong problem class. This happens for four problems (i.e., 3, 4, 10, and 15). In the case of the fourth and the fifteenth problem, the misclassification actually improves the end target prediction. To see which RMs are selected and combined to generate the personalized ensemble, the confusion matrix from the classification is further explored. For the fourth problem, the misclassification happens in the third test fold, where the instance from the fourth problem class (i.e., Büche-Rastrigin function, which is a separable function), is assigned to the third problem (i.e., Rastrigin function, which is also a separable function). For the fifteenth problem (i.e., Rastrigin Function, which is a multi-modal function with adequate global structure), the misclassification happens in the first test fold, where the classifier classifies it into the third problem (i.e., Rastrigin Function, which is a separable function).

These results open new directions for future work; instead of training personalized ensembles on the problem level it will shift to learning them for a whole group of instances which belong to the same cluster. This cluster can be obtained by clustering the ELA representation of the problem instances.

Comparing a single personalized RM with personalized ensembles: To delve deeper, the Ensemble-class and Ensembleground personalized models are compared to the Best-train-instance model. In this scenario, we have limited it to only the best RM for each problem separately, learned using the performance obtained from the training folds, excluding the information from the test instances in the selection. Looking at the median absolute error across the test folds, the personalized ensembles are better than the Best-train-instance models for 13 out of 24 problems (i.e. the problems: 1, 7, 8, 9, 10, 15, 17, 18, 19, 21, 22, 23, and 24). Since the personalized ensembles are based on combining different RMs, the Best-train-instance model for each problem is actually one of the three RMs that are being combined. In most cases, combining the best RM (i.e., Best-train-instance) with the RMs from the other two regression techniques improves the final prediction. However, there are also cases when the end prediction is not better, which further opens the question of how to select the regression techniques that should be included in learning the ensembles.

Mean absolute error vs. median absolute error comparison: In the case when the mean absolute error is used for comparison, the personalized ensembles (i.e., Ensemble-class, and Ensembleground) are better than the Best-train in 17 out of 24 optimization problems (i.e., 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 16, 17, 19, 20, 21, 22, 23, and 24). While comparing them with the Best-test model, they are better in 11 out of 24 problems (i.e., 3, 5, 6, 9, 10, 12, 13, 14, 15, 21, and 24). When the comparison is done with the Best-train-instance, the personalized ensembles are better in 17 out of 24 problems (i.e., 2, 5, 7, 8, 9, 10, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, and 24).

Sensitivity Analysis

To investigate the impact of the different steps used by the methodology on the final predictions, we compare the Best-test, Best-train, and Ensemble-class models in three different scenarios:

(1) We investigate how different sample sizes required to calculate the ELA features influence the prediction of the reached target precision; (2) Results for different budgets for one optimization algorithm and fixed ELA portfolio are presented; (3) The results for different optimization algorithms are presented, for fixed ELA features portfolio and fixed budget. Fixed budget, one optimization algorithm, different sample sizes for ELA calculation: To investigate the impact of different sample sizes required to calculate the ELA features on the final predictions, the 56 selected ELA features were calculated using 50𝑑 and 400𝑑 sample sizes. This resulted in two ELA features portfolios, which were further used as input data to learn personalized ensembles for the BIPOP-CMA-ES in the fixed budget scenario, where the budget was set at 1000 evaluations. Figure 4 presents the relative advantage of the the Ensemble-class-400d model vs. the regression models (Best-train-50d, Best-test-50d, Best-train-400d, Best-test-400d, Ensemble-class-50d) for each problem separately. The suffix 50𝑑 or 400𝑑 in the name of each model presents which ELA feature portfolio is used for learning it. To estimate the advantage, the difference between the median absolute errors has been calculated. Positive values indicate where the Ensemble-class-400d model is better than the other models, while negative values indicate vice-versa. From the figure, it follows that the Ensembleclass-400d model is better than the other models in most of the problems. Comparing the Ensemble-class-50d and the Ensembleclass-400d (i.e., the blue line), it follows that using the ELA feature portfolio calculated with 400𝑑 sample size provides much better results in most of the problems, however there are few problems (i.e., [START_REF] Hutter | Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms[END_REF][START_REF] Huyer | Benchmarking of MCS on the noiseless function testbed[END_REF][START_REF] Kerschke | Automated Algorithm Selection: Survey and Perspectives[END_REF] for which the opposite is true with really small differences in the median absolute errors. The results show us that there is a benefit of using the personalized ensembles. However the selection of the sample size required to calculate the ELA features can influence the final prediction. This also points to the robustness of the ELA features for different samples sizes, which has been already discussed and investigated from another perspective in a recently published paper [START_REF] Renau | Exploratory landscape analysis is strongly sensitive to the sampling strategy[END_REF].

Fixed ELA portfolio, one optimization algorithm, different budgets: Figure 5 presents the relative advantage of Ensembleclass model vs. Best-train model for each of the 3 budgets (250, 500, 1000), in the case when BIPOP-CMA-ES (i.e., the natural logarithm of the target precision) is targeted. Here, the ELA features portfolio is fixed and calculated using a 400𝑑 sample size. Positive values indicate where the Ensemble-class model is better than the Best-train model. From the figure, it follows that personalized ensembles work also for small budgets. Looking at problem 7, it seems that, based on its ELA representation, we can have a good prediction of the target precision reached after 250 and 1000 evaluations, but the Best-train model is better when the budget is 500. The limitation here is that the ELA-problem instance representation is static and the same for all budgets, and only the reached target is different. This means that the ELA representation does not cover information about the algorithm's behavior (i.e., which parts from the search space are visited until some budget). To improve this, further information about the state of the algorithm should be considered and used as input data to train the personalized ensembles for predicting performance in different budgets.

Fixed budget, fixed ELA portfolio, different optimization algorithms: To show the transferability of the proposed methodology to other algorithms than BIPOP-CMA-ES, personalized ensembles were learned to predict the performance of CMA-ES-CSA and IPOP400D, with a fixed budget of 1000 evaluations, and with a fixed ELA feature portfolio calculated using 400𝑑 sample size (Figure 6). Without going in detail into the results, it is obvious that using personalized ensembles improves the final prediction for most of the problems for both algorithms.

DISCUSSION AND FUTURE WORKS

The paper presents the idea of predicting the performance of optimization algorithms, with the aim of selecting a regression model (or an ensemble) for a problem type. Our results demonstrate that there is quite some potential in moving from "generalist" regression models that work well across broad ranges of optimization problems to more problem-specific, personalized regression models. The sensitivity analyses confirm the robustness of our approach.

We note that our study should be seen as a first prototype only. Several extensions are possible and needed. For example, we need to evaluate our methodology on much bigger data sets, to allow for a split into train, validation, and test sets. With such a setting, the train instances are used to train the RMs, the validation instances are then used to select and to evaluate the RMs to be included in the ensembles (this comprises the association of the importance weights that are used to calibrate the predictions of the different models). The test instances are then used to assess the performance of the overall pipeline.

As far as the combination of the output of the different regression models into one prediction is concerned, we plan on evaluating different approaches to derive the weighting schemes. In particular, we believe that a multi-criteria approach to combine different regression performance measures (such as mean root square error, correlation coefficients, etc.) could be promising, to balance the complementary information obtained through each of these statistics.

We used in this work multi-class classification to assign problem instances to problem classes. In practice, instances may stem from problem classes not used in the training phase, so that the classifier cannot assign it to one of the present classes. In such cases, the classification step can be changed with clustering that will return its 𝑘 closest problem instances. Then, the personalized ensembles for the selected problem instances will be used to calculate the performance prediction, which will be further merged with some heuristic to generate the end prediction. Figure 6: Median absolute error between the truth and predicted target precision (i.e., natural logarithmic of the target precision) for each BBOB benchmark problem, for fixed budget 1000, fixed ELA features portfolio calculated using 400d sample size, and two optimization algorithms.

Last but not least, we will evaluate the personalized ensembles trained on one benchmark suite (the BBOB functions in our case) on other benchmark suites (e.g., Nevergrad [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF]), in order to investigate the transferability of the models between the different benchmark collections.

Figure 1 :

 1 Figure 1: Application of the personalized ML models. RM=regression models.

Figure 2 :

 2 Figure

Figure 3 :

 3 Figure3: Evaluation results for BIPOP-CMA-ES performance prediction. The 𝑦-axis corresponds to the absolute error between the truth and predicted target precision (i.e., natural logarithmic of the target precision), while the 𝑥-axis corresponds to each BBOB benchmark problem. The boxplots present the distribution of the absolute error obtained from each test fold for each problem separately in five different scenarios: i) Best-test, ii) Best-train, iii) Best-train-instance iv) Ensemble-class, and v) Ensemble-ground.

4 5 6 7 8 9 Figure 4 :

 94 Figure 4: Relative advantage of the Ensemble-class-400d model vs. the regression models (Best-train-50d, Best-test-50d, Best-train-400d, Best-test-400d, Ensemble-class-50d). The y-axis presents the difference between the median absolute errors. Positive values indicate where the Ensembleclass-400d model is better than the other models, while negative values indicate vice-versa.

4 5 6 7 8 9 1000 Figure 5 :

 910005 Figure 5: Relative advantage of the Ensemble-class vs. Besttrain for each of the 3 budgets (250, 500, 1000). The y-axis presents the difference between the median absolute errors. Positive values indicate where the Ensemble-class model is better than the Best-train model.

4 5 6 7 8 9

 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Best-test Best-train Ensemble-class (a) CMA-ES-CSA.

4 5 6 7 8 9

 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Best-test Best-train Ensemble-class (b) IPOP400D.

Table 1 :

 1 Hyper-parameter values for each regression model class.

	Algorithm	Hyperparameters
	DecisionTree • 𝑐𝑟𝑖𝑡 ∈ {"𝑚𝑠𝑒", "𝑚𝑎𝑒", "𝑓 𝑟𝑖𝑒𝑑𝑚𝑎𝑛_𝑚𝑠𝑒"}
	(30 configs.)	• 𝑚𝑖𝑛𝑠𝑝𝑙𝑖𝑡 ∈ {2, 4, 6, 8, 10, 12, 14, 16, 18, 20}
	RandomForest • 𝑐𝑟𝑖𝑡 ∈ {"𝑚𝑠𝑒", "𝑚𝑎𝑒"}
	(200 configs.)	• 𝑚𝑖𝑛𝑠𝑝𝑙𝑖𝑡 ∈ {2, 4, 6, 8, 10, 12, 14, 16, 18, 20}
		• 𝑛𝑒𝑠𝑡 ∈ {10, 20, 30, 40, 50, 60, 70, 80, 90, 100}
	BaggingDT	• 𝑐𝑟𝑖𝑡 ∈ {"𝑚𝑠𝑒", "𝑚𝑎𝑒"}
	(200 configs.)	• 𝑚𝑖𝑛𝑠𝑝𝑙𝑖𝑡 ∈ {2, 4, 6, 8, 10, 12, 14, 16, 18, 20}
		• 𝑛𝑒𝑠𝑡 ∈ {10, 20, 30, 40, 50, 60, 70, 80, 90, 100}

Table 2 :

 2 Regression models used for evaluation purposes.

	Regression model	Selection Personalized Ensemble MAE on train MAE on test Classification
	Ensemble-ground	✓

Table 3 :

 3 Regression models used in the personalized ensembles for the sixth problem in each fold.

	Fold Models
	1	DecisionTree_crit.mse_minsplit.4
		RandomForest_crit.mae_minsplit.2_nest.90
		BaggingDT_crit.mae_minsplit.2_nest.10
	2	DecisionTree_crit.mae_minsplit.4
		RandomForest_crit.mse_minsplit.6_nest.90
		BaggingDT_crit.mae_minsplit.6_nest.10
	3	DecisionTree_crit.mse_minsplit.4
		RandomForest_crit.mse_minsplit.2_nest.20
		BaggingDT_crit.mae_minsplit.10_nest.10
	4	DecisionTree_crit.mae_minsplit.4
		RandomForest_crit.mae_minsplit.4_nest.30
		BaggingDT_crit.mae_minsplit.2_nest.10
	5	DecisionTree_crit.mae_minsplit.4
		RandomForest_crit.mse_minsplit.2_nest.70
		BaggingDT_crit.mse_ minsplit.10_nest.10

Table 4 :

 4 Median absolute error obtained from each test fold for each problem separately in four different scenarios. Better than both, Best-train and Best-test.

	Problem Best-test Best-train Ensemble-ground Ensemble-class
	1	0.6337	0.2170	0.4718 ★	0.4718 ★
	2	1.0507	0.7152	0.7478 ★	0.7478 ★
	3	0.9530	0.9963	1.0053	1.2729
	4	1.0627	0.7711	1.3661	0.8353 ★
	5	9.0958	0.0000	1.9736 ★	1.9736 ★
	6	3.5115	1.6669	1.5341 △	1.5341 △
	7	2.3472	3.0982	2.0179 △	2.0179 △
	8	2.0920	0.5789	0.6052 ★	0.6052 ★
	9	0.8675	1.0208	0.7480 △	0.7480 △
	10	1.2298	1.8954	1.3431 ⋄	1.2865 ⋄
	11	0.7064	0.6347	0.9910	0.9910
	12	2.3399	1.9997	2.1457 ★	2.1457 ★
	13	1.1155	1.0443	1.0073 △	1.0073 △
	14	3.9245	1.8126	1.3522 △	1.3522 △
	15	0.8055	0.7086	0.3540 △	0.2452 △
	16	0.2879	0.4122	0.2404 △	0.2404 △
	17	3.7476	4.4404	3.7838 ⋄	3.7838 ⋄
	18	2.4728	1.8581	2.6707	2.6707
	19	0.7567	2.1484	1.9176 ⋄	1.9176 ⋄
	20	0.3334	0.4013	0.4151	0.4151
	21	0.4166	0.3046	0.2726 △	0.2726 △
	22	0.9757	1.6932	1.6386 ⋄	1.6386 ⋄
	23	0.2916	0.4672	0.3494 ⋄	0.3494 ⋄
	24	0.3209	0.3103	0.2335 △	0.2335 △
	⋄ Better than Best-train.

★

Better than Best-test. △

In this work, we group in one class all RMs that differ only in the hyper-parameters but use the same basic regression technique.

Acknowledgments. We thank Diederick Vermetten, Leiden University, for providing us the BBOB workshop data [14] in a format that was conveniently post-processed by IOHanalyzer [41]. We also thank Pascal Kerschke, University of Dresden, for providing us with the flacco tool [23] used to compute the feature values for the 24 BBOB functions.

Our work was supported by projects from the Slovenian Research Agency (research core funding No. P2-0098, project No. Z2-1867, and grant number PR-10465), by the Paris Ile-de-France region, and by COST Action CA15140 "Improving Applicability of Nature-Inspired optimization by Joining Theory and Practice (ImAppNIO)".