Juliana Alves Pereira
email: jpereira@inf.puc-rio.br

Mathieu Acher

Hugo Martin

Jean-Marc Jézéquel

Goetz Botterweck

Anthony Ventresque

Learning Software Configuration Spaces: A Systematic Literature Review

Keywords: Systematic Literature Review, Software Product Lines, Machine Learning, Configurable Systems Automated Measurements Configuration Sample (training) sampling prediction Unmeasured Configurations

Most modern software systems (operating systems like Linux or Android, Web browsers like Firefox or Chrome, video encoders like ffmpeg, x264 or VLC, mobile and cloud applications, etc.) are highly configurable. Hundreds of configuration options, features, or plugins can be combined, each potentially with distinct functionality and effects on execution time, security, energy consumption, etc. Due to the combinatorial explosion and the cost of executing software, it is quickly impossible to exhaustively explore the whole configuration space. Hence, numerous works have investigated the idea of learning it from a small sample of configurations' measurements. The pattern "sampling, measuring, learning" has emerged in the literature, with several practical interests for both software developers and end-users of configurable systems. In this systematic literature review, we report on the different application objectives (e.g., performance prediction, configuration optimization, constraint mining), use-cases, targeted software systems, and application domains. We review the various strategies employed to gather a representative and cost-effective sample. We describe automated software techniques used to measure functional and non-functional properties of configurations. We classify machine learning algorithms and how they relate to the pursued application. Finally, we also describe how researchers evaluate the quality of the learning process. The findings from this systematic review show that the potential application objective is important; there are a vast number of case studies reported in the literature related to particular domains or software systems. Yet, the huge variant space of configurable systems is still challenging and calls to further investigate the synergies between artificial intelligence and software engineering.

Introduction

End-users, system administrators, software engineers, and scientists have at their disposal thousands of options (a.k.a. features or parameters) to configure various kinds of software systems in order to fit their functional and non-functional needs (execution time, output quality, security, energy consumption, etc). It is now ubiquitous that software comes in many variants and is highly configurable through conditional compilations, command-line options, runtime parameters, configuration files, or plugins. Software product lines (SPLs), software generators, dynamic, self-adaptive systems, variability-intensive systems are well studied in the literature and enter in this class of configurable software systems [START_REF] Svahnberg | A taxonomy of variability realization techniques: Research articles[END_REF][START_REF] Pohl | Software product line engineering: foundations, principles and techniques[END_REF][START_REF] Apel | Feature-Oriented Software Product Lines: Concepts and Implementation[END_REF][START_REF] Sayagh | Software configuration engineering in practice: Interviews, survey, and systematic literature review[END_REF][START_REF] Benavides | Automated analysis of feature models 20 years later: a literature review[END_REF][START_REF] Cashman | Navigating the maze: the impact of configurability in bioinformatics software[END_REF][START_REF] Hallsteinsen | Dynamic software product lines[END_REF][START_REF] Morin | Models@ run.time to support dynamic adaptation[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF].

From an abstract point of view, a software configuration is simply a combination of options' values. Though customization is highly desirable, it introduces an enormous complexity due to the combinatorial explosion of possible variants. For example, the Linux kernel has 15,000+ options and most of them can have 3 values: "yes", "no", or "module". Without considering the presence of constraints to avoid some combinations of options, there may be 3 15,000 possible variants of Linux -the estimated number of atoms in the universe is 10 80 and is already reached with 300 Boolean options. Though Linux is an extreme case, many software systems or projects exhibit a very large configuration space; this might bring several challenges.

On the one hand, developers struggle to maintain, understand, and test configuration spaces since they can hardly analyze or execute all possible variants. According to several studies [START_REF] Halin | Test them all, is it worth it? assessing configuration sampling on the jhipster web development stack[END_REF][START_REF] Sayagh | Software configuration engineering in practice: Interviews, survey, and systematic literature review[END_REF], the flexibility brought by variability is expensive as configuration failures represent one of the most common types of software failures. Configuration failures is an "undesired effect observed in the system's delivered service" that may occur due to a specific combination of options' values (configurations) [START_REF] Mathur | Foundations of software testing[END_REF]. On the other hand, end-users fear software variability and stick to default configurations [START_REF] Xu | Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software[END_REF][START_REF] Zheng | Automatic configuration of internet services[END_REF] that may be sub-optimal (e.g., the software system will run very slowly) or simply inadequate (e.g., the quality of the output will be unsuitable).

Since it is hardly possible to fully explore all software configurations, the use of machine learning techniques is a quite natural and appealing approach. The basic idea is to learn out of a sample of configurations' observations and hopefully generalize to the whole configuration space. There are several applications ranging from performance prediction, configuration optimization, software understanding to constraint mining (i.e., extraction of variability rules) -we will give a more exhaustive list in this literature review. For instance, end-users of x264 (a configurable video encoder) can estimate in advance the execution time of the command-line at the center of Figure 1, since a machine learning model has been crafted to predict the performance of configurations. End-users may want to use the fastest configuration or know all configurations that meet an objective (e.g., encoding time should be less than 10 seconds). Developers of x264 can be interested in understanding the effects of some options and For all these use-cases, a pattern has emerged in the scientific literature: "sampling, measuring, learning". The basic principle is that a procedure is able to learn out of a sample of configurations' measurements (see Figure 1). Specifically, many software configuration problems can actually be framed as statistical machine learning problems under the condition a sample of configurations' observations is available. For example, the prediction of the performance of individual configurations can be formulated as a regression problem; appropriate learning algorithms (e.g., CART) can then be used to predict the performance of untested, new configurations. In this respect, it is worth noticing the dual use of the term feature in the software or machine learning fields: features either refer to software features (a.k.a. configuration options) or to variables a regressor aims to relate. A way to reconcile and visualize both is to consider a configuration matrix as depicted in Figure 1. In a configuration matrix, each row describes a configuration together with observations/values of each feature and performance property. In the example of Figure 1, the first configuration has the feature no cabac set to False value and the feature ref set to 9 value while the encoding time performance value is 3.1876 seconds. We can use a sample of configurations (i.e., a set of measured configurations) to train a machine learning model (a regressor) with predictive variables being command-line parameters of x264. Unmeasured configurations could then be predicted. Even for large-scale systems like the Linux Kernel, the same process of "sampling, measuring, learning" can be followed (see, e.g., [START_REF] Acher | Learning from thousands of build failures of linux kernel configurations[END_REF][START_REF] Acher | Learning very large configuration spaces: What matters for linux kernel sizes[END_REF]). Some additional steps are worth exploring (like feature engineering prior to learning) while techniques for sampling and learning should be adapted to scale at its complexity, but the general process remains applicable.

Learning software configuration spaces is, however, not a pure machine learning problem and there are a number of specific challenges to address at the intersection of software engineering and artificial intelligence. For instance, the sampling phase involves a number of difficult activities: (1) picking configurations that are valid and conform to constraints among options -one needs to resolve a satisfiability problem;

(2) instrumenting the executions and observations of software for a variety of configurations -it can have an important computational cost and is hard to engineer especially when measuring non-functional aspects of software; (3) meanwhile, we expect that the sample is representative to the whole population of valid configurations otherwise the learning algorithm may hardly generalize to the whole configuration space. The general problem is to find the right strategy to decrease the cost of labelling software configurations while minimizing prediction errors. From an empirical perspective, one can also wonder to what extent learning approaches are effective for real-world software systems present in numerous domains.

While several studies have covered different aspects of configurable systems over the last years, there has been no secondary study (such as systematic literature reviews) that identifies and catalogs individual contributions for machine learning configuration spaces. Thus, there is no clear consensus on what techniques are used to support the process, including which quantitative and qualitative properties are considered and how they can be measured and evaluated, as well as how to select a significant sample of configurations and what is an ideal sample size. This stresses the need for a secondary study to build knowledge from combining findings from different approaches and present a complete overview of the progress made in this field. To achieve this aim, we conduct a Systematic Literature Review (SLR) [START_REF] Kitchenham | Guidelines for performing systematic literature reviews in software engineering[END_REF] to identify, analyze and interpret all available important research in this domain. We systematically review research papers in which the process of sampling, measuring, and learning configuration spaces occurs -more details about our research methodology are given in Section 2. Specifically, we aim of synthesizing evidence to answer the following five research questions:

• RQ1. What are the concrete applications of learning software configuration spaces?

• RQ2. Which sampling methods and learning techniques are adopted when learning software configuration spaces?

• RQ3. Which techniques are used to gather measurements of functional and nonfunctional properties of configurations?

• RQ4. How are learning-based techniques validated?

To address RQ1, we analyze the application objective of the study (i.e., why they apply learning-based techniques). It would allow us to assess whether the proposed approaches are applicable. With respect to RQ2, we systematically investigate which sampling methods and learning techniques are used in the literature for exploring the SPL configuration space. With respect to RQ3, we give an in-depth view of how each study measures a sample of configurations. In addition, RQ4 follows identifying which sampling design and evaluation metrics are used for evaluation.

By answering these questions, we make the following five contributions:

1. We identified six main different application areas: pure prediction, interpretability, optimization, dynamic configuration, evolution, and mining constraints. 2. We provide a framework classification of four main stages used for learning:

Sampling, Measuring, Learning, and Validation. 3. We describe 23 high-level sampling methods, 5 measurement strategies, 51 learning techniques, and 50 evaluation metrics used in the literature. As case studies, we identify 71 real-world configurable systems targeting several domains, and functional and non-functional properties. We relate and discuss the learning and validation techniques with regard to their application objective. 4. We identify a set of open challenges faced by the current approaches, in order to guide researchers and practitioners to use and build appropriate solutions. 5. We build a Web repository to make our SLR results publicly available for the purpose of reproducibility and extension.

Overall, the findings of this SLR reveal that there is a significant body of work specialized in learning software configurable systems with an important application in terms of software technologies, application domains, or goals. There is a wide variety in the considered sampling or learning algorithms as well as in the evaluation process, mainly due to the considered subject systems and application objectives. Practitioners and researchers can benefit from the findings reported in this SLR as a reference when they select a learning technique for their own settings. To this end, this review provides a classification and catalog of specialized techniques in this field.

The rest of the paper is structured as follows. In Section 2, we describe the research protocol used to conduct the SLR. In Section 3, we categorize a sequence of key learning stages used by the ML state-of-the-art literature to explore highly configurable systems. In Section 4, we discuss the research questions. In Section 5, we discuss the current research themes in this field and present the open challenges that need attention in the future. In Section 6, we discuss the threats to the validity of our SLR. In Section 7, we describe similar secondary studies and indicate how our literature review differs from them. Finally, in Section 8, we present the conclusions of our work.

The Review Methodology

We followed the SLR guidelines by Kitchenham and Charters [START_REF] Kitchenham | Guidelines for performing systematic literature reviews in software engineering[END_REF] to systematically investigate the use of learning techniques for exploring the SPL configuration space. In this section, we present the SLR methodology that covers two main phases: planning the review and conducting the review. The paper selection process is shown in Figure 2. Next, We report the details about each phase so that readers can assess their rigor and completeness, and reproduce our findings.

Planning the Review

For identifying the candidate primary studies of Figure 2, we first defined our SLR scope (i.e., identification of the need for a review and specification of the research questions). Then, we developed a review protocol. The need for a systematic review. The main goal of this SLR is to systematically investigate and summarize the state-of-the-art of the research concerning learning techniques in the context of software configurable systems. The purpose of conducting this SLR has partially been addressed in the introduction and was motivated by the lack of a systematic study carried on this topic. According to [START_REF] Kitchenham | Guidelines for performing systematic literature reviews in software engineering[END_REF], an SLR should provide a valuable overview of the status of the field to the community through the summarization of existing empirical evidence supported by current scientific studies.

The outcomes of such an overview can identify whether, or under what conditions, the proposed learning approaches can support various use-cases around configurable systems and be practically adopted (e.g., for which context a specific learning technique is much suitable). By mean of this outcome, we can detect the limitations in current approaches to properly suggest areas for further investigation.

The research questions. The goal of this SLR is to answer the following main research question: What studies have been reported in the literature on learning software configuration spaces since the introduction of Software Product Lines in the early 1990s [START_REF] Kang | Feature-Oriented Domain Analysis (FODA)[END_REF] to date (2019)? However, this question is too broad, so we derived the four sub-questions defined in Section 1. RQ1 classifies the papers with regards to their application objective, i.e., for which particular task the approach is suited and useful. We can group studies into similar categories and then compare them. It is also of interest to identify the practical motivations behind learning approaches. We verified whether the authors indicated a specific application for their approach; otherwise, we classified the approach as pure prediction. RQ2-RQ4 seek to understand key steps of the learning process. RQ2 reviews the set of sampling methods and learning-based techniques used in the literature. RQ3 describes which subject software systems, application domains, and functional and non-functional properties of configurations are measured and how the measurement process is conducted. RQ4 aims to characterize the evaluation process used by the researchers, including the sample design and supported evaluation metric(s). Finally, addressing these questions will allow us to identify trends and challenges in the current state-of-the-art approaches, as well as analysis their maturity to summarize our findings and propose future works.

The review protocol. We searched for all relevant papers published up to May 31st, 2019. The search process involved the use of 5 scientific digital libraries1 : IEEE Xplore Digital Library2 , ACM Digital Library3 , Science Direct4 , Springer-link 5 , and Scopus6 (see Figure 2). These search engines were selected because they are known as the top five preferred on-line databases in the software engineering literature [START_REF] Hoda | Systematic literature reviews in agile software development: A tertiary study[END_REF]. We restricted the search to publication titles and abstracts by applying the selection criteria specified in Section 2.2. However, the library Springer-link only enables a fulltext search. Therefore, we first used the full-text option to generate an initial set of papers (the results were stored in a .bib file). Then, we created a script to perform an expert search in the title and abstract over these results. Each author of this paper had specific roles when performing this SLR. The first author applied the search string to the scientific databases and exported the results (i.e., detailed information about the candidate papers) into a spreadsheet. After this stage, papers were selected based on a careful reading of the titles and abstracts (and if necessary checking the introduction and conclusion). Each identified candidate paper in accordance with the selection criteria defined in Section 2.2 were identified as potentially relevant. When Pereira decided that a paper was not relevant, she provided a short rationale why the paper should not be included in the study. In addition, another researcher checked each excluded paper at this stage. To minimize potential biases introduced into the selection process, any disagreement between researchers were put up for discussion between all authors until a consensus agreement was obtained. This step was done in order to check that all relevant papers were selected.

The search in such databases is known to be challenging due to different search limitations, e.g. different ways of constructing the search string. Thus, apart from the automatic search, we also consider the use of snowballing [START_REF] Wohlin | Guidelines for snowballing in systematic literature studies and a replication in software engineering[END_REF] as a complementary approach. Through snowballing, we searched for additional relevant primary studies by following the references from all preliminary selected studies (plus excluded secondary studies). As we published some works related to the topic of this literature review, we used our knowledge and expertise to complement the pool of relevant papers.

During the data extraction stage, each paper was assigned to one researcher. Pereira coordinated the allocation of researchers to tasks based on the availability of each researcher. The researcher responsible for extracting the data of a specific selected paper applied the snowballing technique to the corresponding paper. Pereira applied the snowballing technique for excluded secondary studies. Each primary study was then assigned to another researcher for review. Once all the primary studies were reviewed, the extracted data was compared. Whenever there were any discrepancies either about the data reported or about the list of additional selected papers derived from the snowballing process, we again resolved the problem through discussions among all authors.

Conducting the Review

The steps involved in conducting the review are: definition of the search string, specification of the selection criteria, and specification of the data extraction process.

As a first step, we used our expertise in the field to obtain an initial pool of relevant studies. Based on this effort, we defined the search string.

The search string. According to Kitchenham et al. [START_REF] Kitchenham | Guidelines for performing systematic literature reviews in software engineering[END_REF] there is no silver bullet for identifying good search strings since, very often, the terminology used in the field is not standardized. When using broad search terms, a large number of irrelevant papers may be found in the search which makes the screening process very challenging. The search string used in this review was first initiated by selecting an initial list of relevant articles by using our expertise in the field.

We identified in the title and abstract the major terms to be used for systematic searching the primary studies. Then, we searched for synonyms related to each major term. Next, we performed several test searches with alternative links between keywords through the different digital libraries. The results from the test searches were continuously discussed among the authors to refine the search string until we were fully satisfied with the capability of the string to detect as much of the initial set of relevant publications as possible. Following this iterative strategy and after a series of test executions and reviews, we obtained a set of search terms and keywords (see Table 1).

Specifically, Table 1 shows the term we are looking for and related synonyms that we considered as keywords in our search. Keywords associated with Product Line allow us to include studies that focus on configurable systems. By combining keywords associated to Learning and Performance Prediction, we can find studies that focus on the use of learning-based techniques for exploring the variability space. In addition, keywords associated with Predict (most specific term) allow us to focus on the application objective of such works. We decided to include the keywords associated with Predict so as to identify the context of the study and have a more restricted number of primary studies. Otherwise, the keywords (Product Line AND (Learning OR Performance Prediction)) return a broad number of studies, e.g. studies addressing the use of learning techniques for product line testing or configuration guidance. In addition, we used keywords from Medicine to exclude studies in this field from our search. We decided to filter these studies once we have tried out the search string without these keywords and we have got many papers in the field. The final result is the following search string:

(Product Line AND (Learning OR Performance Prediction) AND Predict) AND NOT Medicine

The terms Product Line, Learning, Performance Prediction, and Predict are represented as a disjunction of the keywords in Table 1. The search string format we used was slightly modified to meet the search engine requirements of the different scientific databases. For example, the scientific library Science Direct limits the size of the search string. Thus, when searching in this library, we had to split the search string to generate an initial set of papers. Then, we created a script to perform an expert search over this initial set of papers and filter just the relevant ones from these subsets, i.e. we made every effort to ensure that the search strings used were logically and semantically equivalent to the original string in Table 1. The detailed search strings used in each digital search engine and additional scripts are provided in the Web supplementary material [START_REF]Learning software configuration spaces: A systematic literature review[END_REF].

As a second step, we applied the search string to the scientific digital libraries. Figure 2 shows the number of papers obtained from each library. At the end of step 2, the initial search from all sources resulted in a total of 1,627 candidate papers, which includes the 21 papers from our initial pool of relevant studies. As a third step, after removing all duplicated papers (285), we carried out the selection process at the content level.

The selection criteria. We selected the studies using a set of selection criteria for retrieving a relevant subset of publications. First, we only selected papers published up to May 31st, 2019 that satisfied all of the following three Inclusion Criteria (IC):

IC1
The paper is available on-line and in English;

IC2 The paper should be about configurable software systems.

IC3

The paper deals with techniques to statistically learn data from a sample of configurations (see Section 4.1). When different extensions of a paper were observed, e.g., an algorithm is improved by parameter tuning, we intentionally classified and evaluated them as separate primary studies for a more rigorous analysis.

Moreover, we excluded papers that satisfied at least one of the following four Exclusion Criteria (EC):

EC1 Introductions to special issues, workshops, tutorials, conferences, conference tracks, panels, poster sessions, as well as editorials and books.

EC2 Short papers (less than or equal to 4 pages) and work-in-progress.

EC3 Pure artificial intelligence papers.

EC4 Secondary studies, such as literature reviews, articles presenting lessons learned, position or philosophical papers, with no technical contribution. However, the references of these studies were read in order to identify other relevant primary studies for inclusion through the snowballing technique (see Section 2.1). Moreover, we consider secondary studies in the related work section.

We find it useful to give some examples of approaches that were not included:

• the use of sampling techniques without learning (e.g., the main application is testing or model-checking a software product line). That is, the sample is not used to train a machine learning model but rather for reducing the cost of verifying a family of products. For a review on sampling for product line testing, we refer to [START_REF] Medeiros | A comparison of 10 sampling algorithms for configurable systems[END_REF][START_REF] Lopez-Herrejon | A first systematic mapping study on combinatorial interaction testing for software product lines[END_REF][START_REF] Do Carmo | On strategies for testing software product lines: A systematic literature review[END_REF][START_REF] Lee | A survey on software product line testing[END_REF][START_REF] Thüm | A classification and survey of analysis strategies for software product lines[END_REF]. We also discuss the complementary between the two lines of work in Section 5.4;

• the use of state-of-the-art recommendations and visualization techniques for configuration guidance (e.g., [START_REF] Pereira | Personalized recommender systems for product-line configuration processes[END_REF] and [START_REF] Murashkin | Visualization and exploration of optimal variants in product line engineering[END_REF]) and optimization methods based on evolutionary algorithms (e.g., [START_REF] Guo | A genetic algorithm for optimized feature selection with resource constraints in software product lines[END_REF] and [START_REF] Sayyad | On the value of user preferences in searchbased software engineering: a case study in software product lines[END_REF]) since a sample of configurations' measurements is not considered.

• the use of learning techniques to predict the existence of a software defect or vulnerability based on source code analysis (e.g., in [START_REF] Putri | Combining integreted sampling technique with feature selection for software defect prediction[END_REF] and [START_REF] Stuckman | The effect of dimensionality reduction on software vulnerability prediction models[END_REF], features do not refer to configurable systems, instead features refer to properties or metrics of the source code).

During this step, 1,301 papers were excluded, yielding a total of 41 selected papers for inclusion in the review process (see Figure 2). A fourth step of the filtering was performed to select additional relevant papers through the snowballing process. This step considered all included papers, as well as removed secondary studies, which resulted in the inclusion of 28 additional papers. This resulted in the selection of 69 primary papers for data extraction.

The data extraction. The data extraction process was conducted using a structured extraction form in Google Sheets7 to synthesize all data required for further analysis in such a way that the research questions can be answered. In addition, Google Sheets allow future contributions to be online updated by shareholders. First, all candidate papers were analyzed regarding the selection criteria. The following data were extracted from each retrieved study:

• Date of search, scientific database, and search string.

• Database, authors, title, venue, publication type (i.e., journal, conference, symposium, workshop, or report), publisher, pages, and publication year.

• Inclusion criteria IC1, IC2, and IC3 (yes or no)?

• Exclusion criteria EC1, EC2, and EC3 (yes or no)?

• Selected (yes or no)? If not selected, justification regarding exclusion.

Once the list of primary studies was decided, each selected publication was then read very carefully and the content data for each selected paper was captured and extracted in a second form. The data extraction aimed to summarize the data from the selected primary studies for further analysis of the research questions and for increasing confidence regarding their relevance. All available documentation from studies served as data sources, such as thesis, websites, tool support, as well as the communication with authors (e.g., emails exchanged). The following data were extracted from each selected paper:

• RQ1 : Scope of the approach. We classified the approach according to the following six categories: pure prediction, interpretability of configurable systems, optimization, dynamic configuration, mining constraints, and evolution.

• RQ2 : Sampling technique(s).

• RQ3 : Information about subject systems (i.e., reference, name, domain, number of features, and valid configurations) and the measurement procedure. We collected data about the measured (non-)functional properties and the adopted strategies of measurement.

• RQ4 : Short summary of the adopted learning techniques.

• RQ5 : Evaluation metrics and sample designs used by approaches for the purpose of training and validating machine learning models.

The Web supplementary material [START_REF]Learning software configuration spaces: A systematic literature review[END_REF] provides the results of the search procedure from each of these steps. 3. Literature Review Pattern: Sampling, Measuring, Learning Understanding how the system behavior varies across a large number of variants of a configurable system is essential for supporting end-users to choose a desirable product [START_REF] Apel | Feature-Oriented Software Product Lines: Concepts and Implementation[END_REF]. It is also useful for developers in charge of maintaining such software systems. In this context, machine learning-based techniques have been widely considered to predict configurations' behavior and assist stakeholders in making informed decisions. Throughout our review effort, we have observed that such approaches follow a 4-stage process: (1) sampling; (2) measuring; (3) learning; and (4) validation. The aim of this section is to introduce the background behind each of these stages in order to deeply answer the research questions in Section 4.

The stages are sketched in Figure 3. The dashed-line boxes denote the inputs and outputs of each stage. The process starts by building and measuring an initial sample of valid configurations. The set of valid configurations in an SPL is predefined at design time through variability models usually expressed as a feature model [START_REF] Pohl | Software product line engineering: foundations, principles and techniques[END_REF]. Then, these measurements are used to learn a prediction model. Prediction models help stakeholders to better understand the characteristics of complex configurable software systems. They try to describe the behavior of all valid configurations. Finally, the validation step computes the accuracy of the prediction model. In addition, some works use active learning [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF][START_REF] Zuluaga | ε-pal: an active learning approach to the multi-objective optimization problem[END_REF][START_REF] Xi | A smart hill-climbing algorithm for application server configuration[END_REF][START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF] to improve the sample in each interaction based on previous accuracy results until it reaches a configuration that has a satisfactory accuracy. Next, we describe in detail each step.

Sampling. Decision-makers may decide to select or deselect features to customize a system. Each feature can have an effect on a system's non-functional properties. The quantification of the non-functional properties of each individual feature is not enough in most cases, as unknown feature interactions among configuration options may cause unpredictable measurements. Interactions occur when combinations among features share a common component or require additional component(s). Thus, understanding the correlation between feature selections and system non-functional properties is important for stakeholders to be able to find an appropriate system variant that meets their requirements. In Figure 3, let C = {C 1 , C 2 , ..., C n } be the set of n valid configurations, and C i = {f 1 , f 2 , ..., f m } with f j ∈ {0, 1} a combination of m selected (i.e., 1) and deselected (i.e., 0) features. A straightforward way to determine whether a specific variant meets the requirements is to measure its target non-functional property P and repeat the process for all C variants of a system, and then e.g. search for the cheapest configuration C i with C i ∈ C. However, it is usually unfeasible to benchmark all possible variants, due to the exponentially growing configuration space. ML techniques address this issue making use of a small measured sample S C = {s 1 , ..., s k } of configurations, where S C ⊆ C, and the number of samples k and the prediction error are minimal. With the promise to balance measurement effort and prediction accuracy, several sample strategies have been reported in the literature (see Table A.6). For example, Siegmund et al. [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Predicting performance via automated featureinteraction detection[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF] explore several ways of sampling configurations, in order to find the most accurate prediction model. Moreover, several authors [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF][START_REF] Zuluaga | ε-pal: an active learning approach to the multi-objective optimization problem[END_REF][START_REF] Xi | A smart hill-climbing algorithm for application server configuration[END_REF][START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF] have tried to improve the prediction power of the model by updating an initial sample based on information gained from the previous set of samples through active learning. The sample might be partitioned into training, testing and validation sets which are used to train and validate the prediction model (see Section 4.4).

Measuring. This stage measures the set of non-functional properties {p 1 , ..., p l } of a configuration sample S C = {s 1 , ..., s k }, where p 1 = {p 1 (s 1 ), ..., p 1 (s k )}. Non-functional properties are measured either by execution, simulation, static analysis, user feedback or synthetic measurements.

Execution consists of executing the configuration samples and monitoring the measurements of non-functional properties at runtime. Although execution is much more precise, it may incur unacceptable measurement costs since it is often not possible to create suddenly potentially important scenarios in the real environment. To overcome this issue, some approaches have adopted measurement by simulation. Instead of measuring out of real executions of a system which may result in high costs or risks of failure, simulation learns the model using offline environmental conditions that approximate the behavior of the real system faster and cheaper. The use of simulators allows stakeholders to understand the system behavior during early development stages and identify alternative solutions in critical cases. Moreover, simulators can be programmed offline which eliminates any downtime in online environments. In addition to execution and simulation, static analysis infers measurements only by examining the code, model, or documentation. For example, the non-functional property cost can be measured as the required effort to add a feature to a system under construction by analyzing the system cycle evolution, such as the number of lines of code, the development time, or other functional size metrics. Although static analysis may not be always accurate, it is much faster than collecting data dynamically by execution and simulation. Moreover, partial configurations can be also measured. Finally, instead of measuring configurations statically or dynamically, some authors also make use of either user feedback (UF) or synthetic measurements (SM). In contrast to static and dynamic measurements, both approaches do not rely on systems artifacts. User feedback relies only on domain expert knowledge to label configurations (e.g., whether the configuration is acceptable or not). Synthetic measurements are based on the use of learning techniques to generate artificial (non-)functional values to configurable systems [START_REF] Siegmund | Attributed variability models: outside the comfort zone[END_REF]. Researchers can use the THOR generator [START_REF] Siegmund | Attributed variability models: outside the comfort zone[END_REF] to mimic and experiment with properties of real-world configurable systems (e.g., performance distributions, feature interactions).

Learning. The aim of this stage is to learn a prediction model based on a given sample of measured configurations P (S C ) to infer the behavior of non-measured configurations P (C -S C ). The sampling set S C is divided into a training set S T and a validation set S V , where S C = S T + S V . The training set is used as input to learn a prediction model, i.e. describe how configuration options and their interactions influence the behavior of a system. For parameter tuning, interactive sampling, and active learning, the training set is also partitioned into training and testing sets.

Some authors [START_REF] Van Aken | Automatic database management system tuning through large-scale machine learning[END_REF][START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF][START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF] applied transfer learning techniques to accelerate the learning process. Instead of building the prediction model from scratch, transfer learning reuses the knowledge gathered from samples of other relevant related sources to a target source. It uses a regression model that automatically captures the correlation between target and related systems. Correlation means the common knowledge that is shared implicitly between the systems. This correlation is an indicator that there is a potential to learn across the systems. If the correlation is strong, the transfer learning method can lead to a much more accurate and reliable prediction model more quickly by reusing measurements from other sources.

Validating. The validation stage quantitatively analysis the quality of the sample S T for prediction using an evaluation metric on the validation set S V . To be practically useful, an ideal sample S T should result in a (i) low prediction error; (ii) small model size; (iii) reasonable measurement effort. The aim is to find as few samples as possible to yield an understandable and accurate ML model in short computation time. In Section 4.4, we detail the evaluation metrics used by the selected primary studies to compute accuracy.

Overall, exploring the configuration space based on a small sample of configurations is a critical step since in practice, the sample may not contain important feature interactions nor reflect the system real behavior accurately. To overcome this issue, numerous learning approaches have been proposed in the last years. Next, we analyze the existing literature by investigating the more fine-grained characteristics of these four learning stages.

Results and Discussion of the Research Questions

In this section, we discuss the answers to our research questions defined in Section 1. In Section 4.1, we identify the main goal of the learning process. Next, in Section 4.2 and 4.3 we analyze in detail how each study address each learning stage defined in Section 3 to accomplish the goals described in Section 4.1.

RQ1: What are the concrete applications of learning software configuration spaces?

In this section, we analyze the application objective of the selected studies since learning may have different practical interests and motivations. Learning techniques have been used in the literature to target six different scenarios (see Figure 4 and Table 2). In Figure 4, we represent configurations with small circles and variables of a math formula with geometric figures. Next, we describe each individual application and give prominent examples of papers entering in this category. A1 Pure Prediction. The aim is to accurately predict labels p 1 of unmeasured configurations {C 1 , C 2 , ..., C 6 }. Labels can be qualitative (e.g., whether the software configuration has a defect) or quantitative (e.g., the execution time in seconds of a software configuration). The outcome is to associate through prediction some properties to all configurations of the space (see Figure 4a). Guo et al. [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF] is a seminal paper with respect to the use of statistical learning for predicting performances. In this scenario, other factors such as model interpretability and computation cost are less important. In some engineering contexts, the sole prediction of a property of a configuration has limited practical interest per se and is sometimes used as a basis for targeting other applications (e.g., configuration optimization).

! " ($ % ) ! " ($ " ) ! " ($ ' ) ! " ($ ( ) ! " ($ ) ) ! " ($ * ) (a) A1 : Pure Prediction ! " * * ! $ +(! ' * * ! ( ) -(! + * ) (b) A2 : Interpretability Σmax(& ' ) (
A2 Interpretability of configurable systems. Understanding the correlation between configuration options and system quality is important for a wide variety of tasks, such as optimization, program comprehension and debugging. To this end, these studies aim at learning an accurate model that is fast to compute and simple to interpret (see the math formula in Figure 4b). For example, Kolesnikov et al. [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF] explore how so-called performance-influence models quantify options' influences and can be used to explain the performance behavior of a configurable system as a whole.

A3 Optimization. Instead of labeling all configurations, optimization approaches aim at finding a (near-)optimal valid configuration to best satisfy requirements, e.g., max(P 1 ) (see Figure 4c). According to Ochoa et al. [START_REF] Ochoa | A systematic literature review on the semi-automatic configuration of extended product lines[END_REF], there are three types of stakeholders' requirements: resource constraints (threshold such as response time < 1 hour), stakeholders' preferences (e.g., security is extremely more preferable and relevant than response time) and optimization objectives (e.g., minimization of response time). Although the specification of requirements may reduce the configuration space [START_REF] Acher | Familiar: A domain-specific language for large scale management of feature models[END_REF][START_REF] Bak | Clafer: unifying class and feature modeling[END_REF][START_REF] Ochoa | Using decision rules for solving conflicts in extended feature models[END_REF][START_REF] Eichelberger | Using ivml to model the topology of big data processing pipelines[END_REF][START_REF] Roos-Frantz | Qualityaware analysis in product line engineering with the orthogonal variability model[END_REF], searching for the most appropriate configuration is still an overwhelming task due to the combinatorial explosion. Learning approaches exposed in e.g. [START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF] propose a recursive search method to interactively add new samples to train the prediction model until it reaches a sample with an acceptable accuracy.

A4 Dynamic Configuration. There are many dynamic systems (e.g., robotic systems) that aim to manage run-time adaptations of software to react to (uncertain) environmental changes. Without self-adaptation, requirements would be violated.

There are several works that explicitly define a set of adaptation rules in the variability model during design-time. Adaptation rules explicitly define under which circumstances a reconfiguration should take place [START_REF] Pereira | N-dimensional tensor factorization for self-configuration of software product lines at runtime[END_REF]. However, anticipating all contextual changes and defining appropriate adaptation rules earlier is often hard for domain engineers due to the huge variability space and the uncertainty of how the context may change at run-time. Therefore, approaches classified in this group use learning techniques to constantly monitor the environment to detect contextual changes that require the system to adapt at runtime. It learns the influences of contexts online in a feedback loop under time and resource constraints through at least one of the application objectives: (A1) Pure Prediction, (A2) Interpretability, (A3) Optimization, (A5) Mining Constraints, (A6) Evolution (see Figure 4). For example, learning techniques are used in the dynamic scenario to support the synthesis of contextual-variability models, including logical constraints [START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Krismayer | Mining constraints for event-based monitoring in systems of systems[END_REF]. Other approaches [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF][START_REF] Samreen | Daleel: Simplifying cloud instance selection using machine learning[END_REF][START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF][START_REF] Jamshidi | Machine learning meets quantitative planning: Enabling self-adaptation in autonomous robots[END_REF] use learning techniques with the goal of finding a configuration that is optimal and consistent with the current, dynamic context (e.g., given a specific budget).

A5 Mining Constraints. In a configurable system, not all combinations of options' values are possible (e.g., some options are mutually exclusive). Variability models are used to precisely define the space of valid configurations, typically through the specification of logical constraints among options. However, the identifica-tion of constraints is a difficult task and it is easy to forget a constraint leading to configurations that do not compile, crash at run-time, or do not meet a particular resource constraint or optimization goal [START_REF] Temple | Towards adversarial configurations for software product lines[END_REF]. To overcome this issue, learning techniques can be used to discover additional constraints that would exclude unacceptable configurations. These studies seek an accurate and complete set of constraints to restrict the space of possible configurations (see the restricted configurations into the small circle in Figure 4d). Finally, it creates a new variability model by adding the identified constraints to the original model. Therefore, the aim is to accurately remove invalid configurations that were never derived and tested before. Mining constraints approaches work mainly with qualitative properties, such as video quality [START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Towards adversarial configurations for software product lines[END_REF] and defects [START_REF] Yilmaz | Covering arrays for efficient fault characterization in complex configuration spaces[END_REF][START_REF] Krismayer | Mining constraints for event-based monitoring in systems of systems[END_REF][START_REF] Gargantini | Combinatorial interaction testing for automated constraint repair[END_REF][START_REF] Amand | Towards learning-aided configuration in 3d printing: Feasibility study and application to defect prediction[END_REF].

A6 Evolution. In the evolution scenario, a configurable system will inevitably need to adapt to satisfy real-world changes in external conditions, such as changes in requirements, design and performance improvements, and changes in the source code. Thus, new configuration options become available and valid, while existing configurations may become obsolete and invalid (see an example of obsolete configurations in Figure 4e represented by unfilled circles). Consequently, the new variability model structure may influence certain non-functional properties.

In this scenario, it is important to make sure that the learning stage is informed by evolution about changes and the set of sample configurations is readjusted accordingly with the new variability model by excluding invalid configurations (C -C ) and considering the parts of the configuration space not yet covered (C -C). In this case, the measurements for each configuration that includes a new feature or a new interaction is updated.

The use of machine learning techniques to learn software configuration spaces has a wide application objective and can be used for supporting developers or endusers in six main different tasks: Pure Prediction, Interpretability of Configurable Systems, Optimization, Dynamic Configuration, Mining Constraints, and SPL Evolution. It is also possible to combine different tasks (e.g., mining constraints for supporting dynamic configuration [START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Krismayer | Mining constraints for event-based monitoring in systems of systems[END_REF]). There is still room to target other applications (e.g., learning of multiple and composed configurable systems).

RQ2:

Which sampling methods and learning techniques are adopted when learning software configuration spaces?

In this section, we analyze the set of sampling methods used by learning-based techniques in the literature. Also, we aim at understanding which learning techniques were applied and in which context. The complete list of sampling (resp. learning) references can be found in Section Appendix A (resp. Section Appendix B). The interested reader can also visualize the relationship between sampling and learning in the companion Website -we provide an example in Appendix, Figure ??. [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Valov | Empirical comparison of regression methods for variability-aware performance prediction[END_REF][START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF] use a random sample to train, but also to cross-validate their machine learning model. Several works [START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF][START_REF] Zhang | Performance prediction of configurable software systems by fourier learning (t)[END_REF] seek to determine the number of samples in an adaptive, progressive sampling manner and a random strategy is usually employed. Nair et al. [START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF] and Jehooh et al. [START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF] aim at optimizing a configuration. At each iteration, they randomly add an arbitrary number of configurations to learn a prediction model until they reach a model that exhibits a desired satisfactory accuracy. They consider several sizes of samples from tens to thousands of configurations. To focus on a reduced part of the configuration space, Nair et al. [START_REF] Nair | Finding faster configurations using flash[END_REF] and Jehooh et al. [START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF] determine statistically significant parts of the configuration space that contribute to good performance through active learning. In order to have a more representative sample, Valov et al. [START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF] adopted stratified random sampling. This sampling strategy exhaustively divides a sampled population into mutually exclusive subsets of observations before performing actual sampling.

Prior works [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF][START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF] relied on the random selection of features to create a configuration, followed by a filter to eliminate invalid configurations (a.k.a, pseudorandom sampling). Walker's alias sampling [START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF] is an example of pseudo-random sampling. Quasi-random sampling (e.g., sobol sampling) is similar to pseudo-random sampling, however they are specifically designed to cover a sampled population more uniformly [START_REF] Alipourfard | Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics[END_REF][START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF]. However, pseudo-random sampling may result in too many invalid configurations, which makes this strategy inefficient. To overcome this issue, several works [START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF][START_REF] Acher | Varylatex: Learning paper variants that meet constraints[END_REF][START_REF] Zhang | Performance prediction of configurable software systems by fourier learning (t)[END_REF][START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Temple | Towards adversarial configurations for software product lines[END_REF][START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF][START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Jamshidi | Machine learning meets quantitative planning: Enabling self-adaptation in autonomous robots[END_REF] use solver-based sampling techniques (a.k.a., true random sampling).

Sampling and heuristics. Instead of randomly choosing configurations as part of the sample, several heuristics have been developed. The general motivation is to better cover features and features' interactions as part of the sample. The hope is to better capture the essence of the configuration space with a lower sampling size. We describe some heuristics hereafter.

Knowledge-wise heuristic. This heuristic selects a sample of configurations based on its influence on the target non-functional properties. The sampling method described by Siegmund et al. [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF] measures each feature in the feature model plus all known feature interactions defined by a domain expert. Experts detect feature interactions by analyzing the specification of features, implementation assets, and source code, which require substantial domain knowledge and exhaustive analysis. SPLCoqueror8 provides to stakeholders an environment in which they can document and incorporate known feature interactions. For each defined feature interaction, a single configuration is added to the set of samples for measurement. THOR [START_REF] Siegmund | Attributed variability models: outside the comfort zone[END_REF] is a generator for synthesizing synthetic yet realistic variability models where users (researchers) can specify the number of interactions and the degree of interactions. Still, these works call to investigate further questions, such as how to better synthesize knowledge actionable by software developers.

Feature-coverage heuristic. To automatically detect all first-order feature interactions, Siegmund et al. [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Predicting performance via automated featureinteraction detection[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF] use a pair-wise measurement heuristic. This heuristic assumes the existence of a feature interaction between each pair of features in an SPL. It includes a minimal valid configuration for each pair of features being selected. Pairwise requires a number of measurements that is quadratic in the number of optional features. Some authors [START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF][START_REF] Lillack | Improved prediction of non-functional properties in software product lines with domain context[END_REF]] also use a 3-wise feature coverage heuristic to discover interactions among 3 features. Siegmund et al. [START_REF] Siegmund | Predicting performance via automated featureinteraction detection[END_REF] propose a 3rd-order coverage heuristic that considers each minimal valid configuration where three features interact pair-wise among them (adopted by [START_REF] Lillack | Improved prediction of non-functional properties in software product lines with domain context[END_REF]). They also propose the idea that there are hot-spot features that represent a performance-critical functionality within a system. These hot-spot features are identified by counting the number of interactions per features from the feature-coverage and higher-order interaction heuristics. Yilmaz et al. [START_REF] Yilmaz | Reducing masking effects in combinatorialinteraction testing: A feedback drivenadaptive approach[END_REF] adopted even a 4-wise feature coverage heuristic, and Yilmaz et al. [START_REF] Yilmaz | Covering arrays for efficient fault characterization in complex configuration spaces[END_REF] a 5 and 6-wise heuristic. As there are n-th order feature coverage heuristics, the sample set might be likely unnecessarily large which increases measurement effort substantially. However, not every generated sample contains features that interact with each other. Thus, the main problem of this strategy is that it requires prior knowledge to select a proper coverage criterion. To overcome this issue, state-of-the-art approaches might use the interaction-wise heuristic to fix the size of the initial sample to the number of features or potential feature interactions of a system [START_REF] Siegmund | Attributed variability models: outside the comfort zone[END_REF].

Feature-frequency heuristic. The feature-frequency heuristic considers a set of valid configurations in which each feature is selected and deselected, at least, once. Sarkar et al. [START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF] propose a heuristic that counts the number of times a feature has been selected and deselected. Sampling stops when the count of features that were selected or deselected reaches a predefined threshold. Nair et al. [START_REF] Nair | Using bad learners to find good configurations[END_REF] analyse the number of samples required by using the previous heuristic [START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF] against a rank-based random heuristic. Siegmund et al. [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Predicting performance via automated featureinteraction detection[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF] quantify the influence of an individual feature by computing the delta of two minimal configurations with and without the feature. They then relate to each feature a minimum valid configuration that contains the current feature, which requires the measurement of a configuration per feature. Hence, each feature can exploit the previously defined configuration to compute its delta over a performance value of interest. In addition, to maximize the number of possible interactions, Siegmund et al. [START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF] also relate to each feature a maximal valid configuration that contains the current feature.

There are several others sampling heuristics, such as Plackett-Burman design [START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF][START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF] for reasoning with numerical options; Breakdown [START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF] (random breakdown, adaptive random breakdown, adaptive equidistant breakdown) for breaking down (in different sectors) the parameter space; Constrained-driven sampling [START_REF] Gargantini | Combinatorial interaction testing for automated constraint repair[END_REF] (constrained CIT, CIT of constraint validity, constraints violating CIT, combinatorial union, unconstrained CIT) to verify the validity of combinatorial interaction testing (CIT) models; and many others (see Table A .6).

Sampling and transfer learning. Jamshidi et al. [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF] aim at applying transfer learning techniques to learn a prediction model. Jamshidi et al. [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF] consider a combination of random samples from target and source systems for training: {0%, 10%, ..., 100%} from the total number of valid configurations of a source system, and {1%, 2%,..., 10%} from the total number of valid configurations of a target system. In a similar scenario, Jamshidi et al. [START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF] randomly select an arbitrary number of valid configurations from a system before and after environmental changes (e.g., using different hardware, different workloads, and different versions of the system). In another scenario, Jamshidi et al. [START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF] use transfer learning to sample. Their sampling strategy, called L2S, exploits common similarities between source and target systems. L2S progressively learns the interesting regions of the target configuration space, based on transferable knowledge from the source.

Arbitrarily chosen sampling. Chen et al. [START_REF] Chen | Performance prediction of component-based applications[END_REF] and Murwantara et al. [START_REF] Murwantara | Measuring energy consumption for web service product configuration[END_REF] have arbitrarily chosen a set of configurations as their sample is based on their current available resources. Sincero et al. [START_REF] Sincero | Approaching non-functional properties of software product lines: Learning from products[END_REF] use a subset of valid configurations from a preliminary set of (de)selected features. This sample is arbitrarily chosen by domain experts based on the use of features which will probably have a high influence on the properties of interest. In the context of investigating temporal variations, Samreen et al. [START_REF] Samreen | Daleel: Simplifying cloud instance selection using machine learning[END_REF] consider on-demand instances at different times of the day over a period of seven days with a delay of ten minutes between each pair of runs. In a similar context, Duarte et al. [START_REF] Duarte | Learning nondeterministic impact models for adaptation[END_REF] and Chen et al. [START_REF] Chen | Boosting the performance of computing systems through adaptive configuration tuning[END_REF] also sample configurations under different workloads (e.g., active servers and requests per second) at different times of the day. An important insight is that there are engineering contexts in which the sampling strategy is imposed and can hardly be controlled.

All configurations (no sampling). Sampling is not applicable for four of the selected primary studies [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF][START_REF] Kolesnikov | On the relation of external and internal feature interactions: A case study[END_REF][START_REF] Couto | Products go green: Worst-case energy consumption in software product lines[END_REF][START_REF] Zheng | Automatic configuration of internet services[END_REF], mainly for experimental reasons. For example, Kolesnikov et al. [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF][START_REF] Kolesnikov | On the relation of external and internal feature interactions: A case study[END_REF] consider all valid configurations in their experiments and use an established learning technique to study and analyze the trade-offs among prediction error, model size, and computation time of performance-prediction models. For the purpose of their study, they were specifically interested to explore the evolution of the model properties to see the maximum possible extent of the corresponding trade-offs after each iteration of the learning algorithm. So, they performed a whole-population exploration of the largest possible learning set (i.e., all valid configurations). In a similar scenario, Kolesnikov et al. [START_REF] Kolesnikov | On the relation of external and internal feature interactions: A case study[END_REF] explored the use of control-flow feature interactions to identify potentially interacting features based on detected interactions from performance prediction techniques using performance measurements). Therefore, they also performed a whole-population exploration of all valid configurations.

Reasoning about configuration validity. Sampling is realized either out of an enumerated set of configurations (e.g., the whole ground truth) or a variability model (e.g., a feature model). The former usually assumes that configurations of the set are logically valid. The latter is more challenging, since picking a configuration boils down to resolve a satisfiability or constraint problem.

Acher et al. [START_REF] Acher | Varylatex: Learning paper variants that meet constraints[END_REF] and Temple et al. [START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Temple | Towards adversarial configurations for software product lines[END_REF] encoded variability models as Constraint Programming (CSP) by using the Choco solver, while Weckesser et al. [START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF] and Siegmund et al. [START_REF] Siegmund | Attributed variability models: outside the comfort zone[END_REF] employed SAT solvers. Constraint solver may produce clustered configurations with similar features due to the way solvers enumerate solutions (i.e., often the sample set consists of the closest k valid configurations). Therefore, these strategies do not guarantee true randomness as in pseudo-random sampling. Moreover, using CSP and SAT solvers to enumerate all valid configurations are often impractical [START_REF] Mendonca | Efficient compilation techniques for large scale feature models[END_REF][START_REF] Pohl | A performance comparison of contemporary algorithmic approaches for automated analysis operations on feature models[END_REF]. Thus, Jehooh et al. [START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF] encoded variability models as Binary Decision Diagrams (BDDs) [START_REF] Akers | Binary decision diagrams[END_REF], for which counting the number of valid configurations is straightforward. Given the number of valid configurations n, they randomly and uniformly select the k th configuration, where k ∈ {1...n}, a.k.a. randomized truerandom sampling. Kaltenecker et al. [START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF] perform a comparison among pseudo-random sampling, true-random sampling, and randomized true-random sampling.

Reasoning with numerical options. Ghamizi et al. [START_REF] Ghamizi | Automated search for configurations of deep neural network architectures[END_REF] transform numeric and enumerated attributes into alternative Boolean features to be handled as binary options. Temple et al. [START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Temple | Towards adversarial configurations for software product lines[END_REF] adopted random sampling of numeric options, i.e. real and integer values. First, their approach randomly selects a value for each feature within the boundaries of its domain. Then, it propagates the values to other features with a solver to avoid invalid configurations. In a similar scenario, Siegmund et al. [START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF] and Grebhahn et al. [START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF] adopted pseudo-random sampling of numeric options. Siegmund et al. [START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF] claim that it is very unusual that numeric options have value ranges with undefined or invalid holes and that constraints among numeric options appear rarely in configurable systems. Grebhahn et al. [START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF] adopted different reasoning techniques over binary and numeric options, then they compute the Cartesian product of the two sets to create single configurations used as input for learning. In the scenario of reasoning with numerical options, Amand et al. [START_REF] Amand | Towards learning-aided configuration in 3d printing: Feasibility study and application to defect prediction[END_REF] arbitrarily select equidistant parameter values.

Numeric sampling have substantially different value ranges in comparison with binary sampling. The number of options' values to select can be huge while constraints should still be respected. Sampling with numeric options is still an open issue -not a pure SAT problem.

Though random sampling is a widely used baseline, numerous other sampling algorithms and heuristics have been devised and described. There are different trade-offs to find when sampling configurations (1) minimization of invalid configurations due to constraints' violations among options; (2) minimization of the cost (e.g., size) of the sample; (3) generalization of the sample to the whole configuration space. The question of a one-size-fits-all sampling strategy remains open and several factors are to be considered (targeted application, subject systems, functional and non-functional properties, presence of domain knowledge, etc.).

Learning Techniques. Supervised learning problems can be grouped into regression and classification problems. In both cases, the goal is to construct a machine learning model that can predict the value of the measurement from the features. The difference between the two problems is the fact that the value to predict is numerical for regression and categorical for classification. In this literature review, we found a similar dichotomy, depending on the targeted use-case and the NFP of interest.

A regression problem is when the output is a real or continuous value, such as time or CPU power consumption. Most of the learning techniques tackle a supervised regression problem.

CART for regression. Several authors [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF][START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Murwantara | Measuring energy consumption for web service product configuration[END_REF][START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF] use the Classification And Regression Trees (CART) technique, to model the correlation between feature selections and performance. The sample is used to build the prediction model. CART recursively partitions the sample into smaller clusters until the performance of the configurations in the clusters is similar. These recursive partitions are represented as a binary decision tree. For each cluster, these approaches use the sample mean of the performance measurements (or even the majority vote) as the local prediction model of the cluster. So, when they need to predict the performance of a new configuration not measured so far, they use the decision tree to find the cluster which is most similar to the new configuration. Each split of the set of configurations is driven by the (de)selection of a feature that would minimize a prediction error.

CART uses two parameters to automatically control the recursive partitioning process: minbucket and minsplit. Minbucket is the minimum sample size for any leaf of the tree structure; and minsplit is the minimum sample size of a cluster before it is considered for partitioning. Guo et al. [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF] |; the minimum of minbucket is 2; and the minimum of minsplit is 4. It should be noted that CART can also be used for classification problems (see hereafter).

Instead of using a set of empirically-determined rules, Guo et al. [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF] combine the previous CART approach [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF] with automated resampling and parameter tuning, which they call a data-efficient learning approach (DECART). Using resampling, DECART learns a prediction model by using different sample designs (see Section 3). Using parameter tuning, DECART ensures that the prediction model has been learned using optimal parameter settings of CART based on the currently available sample. They compare three parameter-tuning techniques: random search, grid search, and Bayesian optimization. Westermann et al. [START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF] also used grid search for tuning CART parameters.

Approaches suggested by Nair et al. [START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF]] build a prediction model in a progressive way by using CART. They start with a small training sample and subsequently add samples to improve performance predictions based on the model accuracy (see Section 4.4). In each step, while training the prediction model, Nair et al. [START_REF] Nair | Using bad learners to find good configurations[END_REF] compare the current accuracy of the model with the previous accuracy from the prior iteration (before adding the new set of configurations to the training set). If the current accuracy (with more data) does not improve the previous accuracy (with lesser data), then the learning reaches a termination criterion (i.e., adding more samples will not result in significant accuracy improvements).

Performance-influence models. Siegmund et al. [START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF] combine machine learning and sampling heuristics to build so-called performance-influence models. A step-wise linear regression algorithm is used to select relevant features as relevant terms of a linear function and learn their coefficient to explain the observations. In each iterative step, the algorithm selects the sample configuration with the strongest influence regarding prediction accuracy (i.e., yields the model's lowest prediction error) until improvements of model accuracy become marginal or a threshold for expected accuracy is reached (below 19%). The algorithm concludes with a backward learning step, in which every relevant feature is tested for whether its removal would decrease model accuracy. This can happen if initially a single feature is selected because it better explains the measurements, but it becomes obsolete by other features (e.g., because of feature interactions) later in the learning process. Linear regression allows them to learn a formula that can be understood by humans. It also makes it easy to incorporate domain knowledge about an option's influence on the formula. However, the complete learning of a model using this technique required from 1 to 5 hours, depending on the size of the learning set and the size of the models. Kolesnikov et al. [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF] investigate how significant are the trade-offs among prediction error, model size, and computation time.

Other learning algorithms for regression. Westermann et al. [START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF] used Multivariate Adaptive Regression Splines (MARS), Genetic Programming (GP), and Kriging. Zhang et al. [START_REF] Zhang | Performance prediction of configurable software systems by fourier learning (t)[END_REF] used Fourier learning algorithm. In all these works, for a set of samples, it verifies if the resulting accuracy is acceptable for stakeholders (e.g., prediction error rate below 10%). While the accuracy is not satisfactory, the process continues by obtaining an additional sample of measured configurations and iterates again to produce an improved prediction model. Sarkar et al. [START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF] propose a sampling cost metric as a stopping criterion, where the objective is to ensure the most optimal trade-off between measurement effort and prediction accuracy. Sampling stops when the count of features selected and deselected passes a predefined threshold.

Chen et al. [START_REF] Chen | Performance prediction of component-based applications[END_REF] propose a linear regression approach to describe the generic performance behavior of application server components running on component-based middleware technologies. The model focuses on two performance factors: workload and degree of concurrency. Sincero et al. [START_REF] Sincero | Approaching non-functional properties of software product lines: Learning from products[END_REF] employ analysis of covariance for identifying factors with significant effects on the response or interactions among features. They aim at proposing a configuration process where the user is informed about the impact of their feature selection on the NFPs of interest.

Murwantara et al. [START_REF] Murwantara | Measuring energy consumption for web service product configuration[END_REF] use a set of five ML techniques (i.e., Linear regression, CART, Multilayer Perceptrons (MLPs), Bagging Ensembles of CART, and Bagging Ensembles of MLPs) to learn how to predict the energy consumption of web service systems. They use WEKA's implementation of these techniques with its default parameters [START_REF] Hall | The weka data mining software: an update[END_REF].

Learning to rank. Instead of predicting the raw performance value, it can be of interest to predict the rank of a configuration (typically to identify optimal configurations, see RQ1).

Jehooh et al. [START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF] adopt statistical learning techniques to progressively shrink a configuration space and search for near-optimal configurations. First, the approach samples and measures a set of configurations. For each pair of sampled configurations, this approach identifies features that are common (de)selected. Then, it computes the performance influence of each common decision to find the best regions for future sampling. The performance influence measures the average performance over the samples that have the feature selected against the samples that have the feature deselected, i.e., the sample is partitioned by whether a configuration includes a particular feature or not. In addition, Welch's t-test evaluates whether the performance mean of one sample group is higher than the other group with 95% confidence. The most influential decisions are added to the sample. This process continues recursively until they identify all decisions that are statistically certain to improve program performance. They call this a Statistical Recursive Searching technique.

Nair et al. [START_REF] Nair | Using bad learners to find good configurations[END_REF] compute accuracy by using the mean rank difference measurement (the predicted rank order is compared to the optimal rank order). They demonstrate that their approach can find optimal configurations of a software system using fewer measurements than the approach proposed by [START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF]. One drawback of this approach is that it requires a holdout set, against which the current model (built interactively) is compared. To overcome this issue, instead of making comparisons, Nair et al. [START_REF] Nair | Finding faster configurations using flash[END_REF] consider a predefined stopping criterion (budget associated with the optimization process). While the criterion is not met, the approach finds the configuration with the best accuracy and add the configuration to the training set. The model adds the next most promising configuration to evaluate. Consequently, in each step, the approach discards less satisfactory configurations which have a high probability of being dominated, a.k.a. active learning. This process terminates when the predefined stopping condition is reached. Nair et al. [START_REF] Nair | Finding faster configurations using flash[END_REF] demonstrate that their approach is much more effective for multi-objective configuration optimization than state-of-the-art approaches [START_REF] Zuluaga | ε-pal: an active learning approach to the multi-objective optimization problem[END_REF][START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF].

Martinez et al. [START_REF] Martinez | Towards estimating and predicting user perception on software product variants[END_REF] propose the use of data mining interpolation techniques (i.e., similarity distance, similarity radius, weighted mean) for ranking configurations through user feedback on a configuration sample. They estimate the user perceptions of each feature by computing the value of the chi-squared statistic with respect to the correlation score given by users on configurations.

Transfer learning. The previous approaches assume a static environment (e.g., hardware, workload) and NFP such that learning has to be repeated once the environment and NFP changes. In this scenario, some approaches adopt transfer learning techniques to reuse (already available) knowledge from other relevant sources to learn a performance for a target system instead of relearning a model from scratch [START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF][START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF].

Valov et al. [START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF] investigate the use of transfer learning across different hardware platforms. They used 25 different hardware platforms to understand the similarity of performance prediction. They created a prediction model using CART. With CART, the resulting prediction models can be easily understood by end users. To transfer knowledge from a related source to a target source, they used a simple linear regression model.

Jamshidi et al. [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF] use Gaussian Process Models (GPM) to model the correlation between source and target sources using the measure of similarity. GPM offers a framework in which predictions can be done using mean estimates with a confidence interval for each estimation. Moreover, GPM computations are based on linear algebra which is cheap to compute. This is especially useful in the domain of dynamic SPL configuration where learning a prediction model at runtime in a feedback loop under time and resource constraints is typically time-constrained.

Jamshidi et al. [START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF] combine many statistical and ML techniques (i.e., Pearson linear correlation, Kullback-Leibler divergence, Spearman correlation coefficient, paired t-test, CART, step-wise linear regression, and multinomial logistic regression) to identify when transfer learning can be applied. They use CART for estimating the relative importance of configuration options by examining how the prediction error will change for the trained trees on the source and target. To investigate whether interactions across environments will be preserved, they use step-wise linear regression models (a.k.a., performance-influence models). This model learns all pairwise interactions, then it compares the coefficients of the pairwise interaction terms independently in the source and target environments. To avoid exploration of invalid configurations and reduce measurement effort, they use a multinomial logistic regression model to predict the probability of a configuration being invalid, then they compute the correlation between the probabilities from both environments.

Jamshidi et al. [START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF] propose a sampling strategy, called L2S, that exploits common similarities across environments from [START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF]. L2S extracts transferable knowledge from the source to drive the selection of more informative samples in the target environment. Based on identifying interesting regions from the performance model of the source environment, it generates and selects configurations in the target environment iteratively.

Classification problem. Temple et al. [START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF] and Acher et al. [START_REF] Acher | Varylatex: Learning paper variants that meet constraints[END_REF] use CART to infer constraints to avoid the derivation of invalid (non-acceptable) configurations. CART considers a path in a tree as a set of decisions, where each decision corresponds to the value of a single feature. The approach creates new constraints in the variability model by building the negation of the conjunction of a path to reach a faulty leaf. They learn constraints among Boolean and numerical options. As an academic example, Acher et al. [START_REF] Acher | Varylatex: Learning paper variants that meet constraints[END_REF] introduce VaryL A T E X to guide researchers to meet paper constraints by using annotated L A T E X sources. To improve the learning process, Temple et al. [START_REF] Temple | Towards adversarial configurations for software product lines[END_REF] specifically target low confidence areas for sampling. The authors apply the idea of using an adversarial learning technique, called evasion attack, after a classifier is trained with a Support Vector Machine (SVM). In addition, Temple et al. [START_REF] Temple | Learning contextual-variability models[END_REF] support the specialization of configurable systems for a deployment at runtime. In a similar context, Safdar et al. [START_REF] Safdar | Mining cross product line rules with multiobjective search and machine learning[END_REF] infer constraints over multi-SPLs (i.e., they take into account cross-SPLs rules).

Input sensitivity. Several works consider the influence of the input data on the resulting prediction model [START_REF] Lillack | Improved prediction of non-functional properties in software product lines with domain context[END_REF][START_REF] Ding | Autotuning algorithmic choice for input sensitivity[END_REF][START_REF] Thornton | Auto-weka: Combined selection and hyperparameter optimization of classification algorithms[END_REF][START_REF] Xu | Satzilla: portfolio-based algorithm selection for sat[END_REF][START_REF] Hutter | Sequential model-based optimization for general algorithm configuration[END_REF][START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF][START_REF] Afia | Performance prediction using support vector machine for the configuration of optimization algorithms[END_REF][START_REF] Ghamizi | Automated search for configurations of deep neural network architectures[END_REF]. Many authors [START_REF] Ding | Autotuning algorithmic choice for input sensitivity[END_REF][START_REF] Thornton | Auto-weka: Combined selection and hyperparameter optimization of classification algorithms[END_REF][START_REF] Xu | Satzilla: portfolio-based algorithm selection for sat[END_REF][START_REF] Hutter | Sequential model-based optimization for general algorithm configuration[END_REF][START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF][START_REF] Afia | Performance prediction using support vector machine for the configuration of optimization algorithms[END_REF][START_REF] Ghamizi | Automated search for configurations of deep neural network architectures[END_REF] address input sensitivity in algorithm auto-tuning. They use learning techniques to search for the best algorithmic variants and parameter settings to achieve optimal performance for a given input instance. It is well known that the performance of SAT solver and learning methods strongly depends on making the right algorithmic and parameter choices, therefore SATzilla [START_REF] Xu | Satzilla: portfolio-based algorithm selection for sat[END_REF] and Auto-WEKA [START_REF] Thornton | Auto-weka: Combined selection and hyperparameter optimization of classification algorithms[END_REF] search for the best SAT solver and learning technique for a given input instance. Lillacka et al. [START_REF] Lillack | Improved prediction of non-functional properties in software product lines with domain context[END_REF] treat the variability caused by the input data as the variability of the SPL. As it is not feasible to model every possible input data, they cluster the data based on its relevant properties (e.g., 10kB and 20MB in an or-group for file inputs of a compression library).

Reinforcement learning. Sharifloo et al. [START_REF] Sharifloo | Learning and evolution in dynamic software product lines[END_REF] use a reinforcement learning technique to automatically reconfigure dynamic SPLs to deal with context changes. In their approach, learning continuously observes measured configurations and evaluates their ability to meet the contextual requirements. Unmet requirements are addressed by learning new adaptation rules dynamically, or by modifying and improving the set of existing adaptation rules. This stage takes software evolution into account to address new contexts faced at run-time.

Numerous statistical learning algorithms are used in the literature to learn software configuration spaces. The most used are standard machine learning techniques, such as polynomial linear regressions, decision trees, or Gaussian process models. The targeted use-case and the engineering context explain the diversity of solutions: either a supervised classification or regression problem is tackled; the requirements in terms of interpretability and accuracy may differ; there is some innovation in the sampling phase to progressively improve the learning. Still the use of others (more powerful) ML techniques such as deep learning, adversarial learning, and even the idea of learning different models (e.g., one for each application objective) that could co-evolve can be further explored in future works. Also, unsupervised learning is another potential candidate to support the exploration of configurable systems. For instance, clustering techniques can be used to group together configurations and reduce the costly measurements of each individual configuration within a cluster. Still, analyzing its feasibility remains unexplored.

RQ3: Which techniques are used to gather measurements of functional and nonfunctional properties of configurations?

The measuring step takes as input a sample of configurations and measures, for each configuration, their functional properties or non-functional properties (NFPs). In this section, we investigate how the measurement procedures are technically realized. Most proposals consider NFPs, such as elapsed time in seconds. In essence, NFPs consist of a name, a domain, a value, and a unit [START_REF] Benavides | Automated reasoning on feature models[END_REF]. The domain type of an NFP can be either quantitative (e.g., real and integer) or qualitative (e.g., string and Boolean). Quantitative (QT) properties are typically represented as a numeric value, thus they can be measured on a metric scale, e.g., the configuration is executed in 13.63 seconds. Qualitative (QL) properties are represented using an ordinal scale, such as low (-) and high (+); e.g., the configuration produces a high video quality.

As described in Section 3, measurements can be obtained through five different strategies: execution (EX), static analysis (SA), simulation (SI), user feedback (UF), and synthetic measurements (SM). Fig. 5 shows that automated execution is by far the most used technique to measure configuration properties.

There are 71 real-world SPLs documented in the literature (see Table C.8 in appendix). They are of different sizes, complexities, implemented in different programming languages (C, C++, and Java), varying implementation techniques (conditional compilation and feature-oriented programming), and from several different application domains (e.g., operating systems, video encoder, database system) and developers (both academic and industrial). Therefore, they cover a broad spectrum of scenarios. It is important to mention that the same subject systems may differ in the number of features, feature interactions, and in the number of valid configurations -the experimental setup is simply different.

Next, we detail the particularities of NFPs. Specifically, we describe how the measurement is performed, what process and strategies are adopted to avoid biases in the results, and also discuss the cost of measuring.

Time. The time spent by a software configuration to realize a task is an important concern and has been intensively considered under different flavors and terminologies (see Fig. 6). Siegmund et al. [START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF] invested more than two months (24/7) for measuring the response time of all configurations of different subject systems (Dune MGS, *Each application domain or even each system has its own specific definition of performance. These works did not specify the NFP(s) that refer to performance.

HIPAcc, HSMGP, JavaGC, SaC, x264). For each system, they use a different configuration of hardware. The configurations' measurements are reused in many papers, mainly for evaluating the proposed learning process [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF][START_REF] Temple | Learning-based performance specialization of configurable systems[END_REF]. Chen et al. [START_REF] Chen | Performance prediction of component-based applications[END_REF] used a simulator to measure the response time of a sequence of 10 service requests by a client to the server. They use two implementation technologies, CORBA and EJB.

Time is a general notion and can be refined according to the application domain. For instance, Kolesnikov et al. [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF] considered compression time of a file on lrzip (Long Range ZIP). The measurements were conducted on a dedicated server and repeated multiple times to control measurements noise. Kaltenecker et al. [START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF] considered 7-ZIP, a file archiver written in C++, and measured the compression time of the Canterbury corpus 9 .

In another engineering context, several works [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF][START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Zhang | Performance prediction of configurable software systems by fourier learning (t)[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF][START_REF] Valov | Empirical comparison of regression methods for variability-aware performance prediction[END_REF][START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF] measured the encoding time of an input video over 1,152 valid configurations of x264. x264 is a configurable system for encoding video streams into the H.264/MPEG-4 AVC format. As benchmark, the Sintel trailer (735 MB) is used and an encoding from AVI is considered. Kaltenecker et al. [START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF] measured the encoding time of a short piece of the Big Buck Bunny trailer over 216,000 valid configurations of VPXENC (VP9), a video encoder that uses the VP9 format.

Latency has caught the attention of several researchers. Jamshidi et al. [START_REF] Jamshidi | An uncertainty-aware approach to optimal configuration of stream processing systems[END_REF] and [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF] measure the average latency (i.e., how fast it can respond to a request) of three stream processing applications on Apache Storm (WordCount, RollingSort, SOL) over a window of 8 minutes and 2 hours, respectively. Jamshidi et al. [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF] also measure the average latency of the NoSQL database system on Apache Cassandra over a window of 10 minutes. Jamshidi et al. [START_REF] Jamshidi | An uncertainty-aware approach to optimal configuration of stream processing systems[END_REF] performed the measurements on a multi-node cluster on the EC2 cloud. Jamshidi et al. [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF] performed the same measurement procedure reported for CPU usage. The measurements used by Nair et al. [START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF] were derived from Jamshidi et al. [START_REF] Jamshidi | An uncertainty-aware approach to optimal configuration of stream processing systems[END_REF]. Weckesser et al. [START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF] measure the latency of transferred messages over an adaptive Wireless Sensor Networks (WSNs) via simulation. 100 fullycharged nodes were distributed randomly onto a square region for each simulation run. Aken et al. [START_REF] Van Aken | Automatic database management system tuning through large-scale machine learning[END_REF] measure the latency for two OLTP DBMSs (MySQL v5.6, Postgres v9.3) over three workloads (The Yahoo Cloud Serving Benchmark (YCSB), TPC-C, and Wikipedia) during five minutes observation periods. The OLTP DBMS are deployed on m4.large instances with 4 vCPUs and 16 GB RAM on Amazon EC2. They also consider the total execution time of the OLAP DBMS (Actian Vector v4.2) over the TPC-H workload on m3.xlarge instances with 4 vCPUs and 15 GB RAM on Amazon EC2. All of the training data was collected using the DBMSs' default isolation level.

Sharifloo et al. [START_REF] Sharifloo | Learning and evolution in dynamic software product lines[END_REF] measure time throughout the evolution of a configurable system (an SPL). In an SPL evolution scenario, the set of measured configurations may include a removed feature or violate changed constraints. In this case, configurations are removed from the sample. Moreover, a modified feature implies to recompute the configuration measurements.

Other NFPs (CPU power consumption, CPU usage, size, etc.) Beyond time, other quantitative properties have been considered.

Nair et al. [START_REF] Nair | Using bad learners to find good configurations[END_REF] measured the compressed size of a specific input over 432 valid configurations of Lrzip, a compression program optimized for large files. Siegmund et al. [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF] measured the memory footprint of nine configurable systems: LinkedList, Prevayler, ZipMe, PKJab, SensorNetwork, Violet, Berkeley DB, SQLite, and Linux kernel. Nair et al. [START_REF] Nair | Finding faster configurations using flash[END_REF] and Zuluaga et al. [START_REF] Zuluaga | ε-pal: an active learning approach to the multi-objective optimization problem[END_REF] used LLVM, a configurable modular compiler infrastructure. The footprint is measured as the binary size of a compiled configuration.

Several works [START_REF] Murwantara | Measuring energy consumption for web service product configuration[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Couto | Products go green: Worst-case energy consumption in software product lines[END_REF][START_REF] Zuluaga | ε-pal: an active learning approach to the multi-objective optimization problem[END_REF][START_REF] Jamshidi | Machine learning meets quantitative planning: Enabling self-adaptation in autonomous robots[END_REF] measured CPU power consumption. Murwantara et al. [START_REF] Murwantara | Measuring energy consumption for web service product configuration[END_REF] used a simulator to compute CPU power consumption over a web service under different loads. They used a kernel-based virtual machine to conduct the measurements based on several combinations of HTTP servers and variant PHP technologies connected to a MySQL database system. They divided the experiment into blocks of 10 seconds for 100 increasing stages. In the first 10 seconds, they produce loads of one user per second, and in the next 10 seconds, they produce the load of two users per second and so on. This results in 1-100 users per period of 10 seconds. Couto et al. [START_REF] Couto | Products go green: Worst-case energy consumption in software product lines[END_REF] propose a static learning approach of CPU power consumption. Their approach assures that the source code of every feature from a configuration is analyzed only once. To prove its accuracy, they measured at runtime the worst-case CPU power consumption to execute a given instruction on 7 valid configurations of the disparity SPL [START_REF] Venkata | Sd-vbs: The san diego vision benchmark suite[END_REF]. They repeated the measurements 200 times for each configuration using the same input. Since their goal was to determine the worst-case CPU power consumption, they removed the outliers (5 highest and lowest values) and retrieved the highest value from every configuration for validation proposes. Jamshidi et al. [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF] use simulation measurements. Jamshidi et al. [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF] have repeatedly executed a specific robot mission to navigate along a corridor offline in a simulator and measured performance in terms of CPU usage on the CoBot system. To understand the power of transfer learning techniques, they consider several simple hardware changes (e.g., processor capacity) as well as severe changes (e.g., local desktop computer to virtual machines in the cloud) 10 . Each measurement took about 30 seconds. They measured each configuration of each system and environment 3 times.

To overcome the cost of measuring realistic (non-)functional properties, Siegmund et al. [START_REF] Siegmund | Attributed variability models: outside the comfort zone[END_REF] proposed a tool, called Thor, to generate artificial and realistic synthetic measurements, based on different distribution patterns of property values for features, interactions, and configurations from a real-world system. Jamshidi et al. [START_REF] Jamshidi | Machine learning meets quantitative planning: Enabling self-adaptation in autonomous robots[END_REF] adopted Thor to synthetically measure the property energy consumption for a robot to complete a mission consisting of randomly chosen tasks within a map.

Qualitative properties. Instead of measuring a numerical value, several papers assign a qualitative value to configurations. In Acher et al. [START_REF] Acher | Varylatex: Learning paper variants that meet constraints[END_REF], a variant is considered acceptable or not thanks to an automated procedure. In Temple et al. [START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Towards adversarial configurations for software product lines[END_REF], a quantitative value is originally measured and then transformed into a qualitative one through the definition of a threshold. In [START_REF] Queiroz | Towards predicting feature defects in software product lines[END_REF], a variant is considered as acceptable or not based on the static evolution historical analysis of commit messages (i.e., they identify a defect by searching for the following keywords: bug, fix, error, and fail ). The learning process then aims to predict the class of the configuration -whether configurations are acceptable or not, as defined by the threshold.

Software defects are considered in [START_REF] Yilmaz | Covering arrays for efficient fault characterization in complex configuration spaces[END_REF][START_REF] Krismayer | Mining constraints for event-based monitoring in systems of systems[END_REF][START_REF] Gargantini | Combinatorial interaction testing for automated constraint repair[END_REF][START_REF] Yilmaz | Reducing masking effects in combinatorialinteraction testing: A feedback drivenadaptive approach[END_REF][START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF][START_REF] Amand | Towards learning-aided configuration in 3d printing: Feasibility study and application to defect prediction[END_REF]. Yilmaz et al. [START_REF] Yilmaz | Covering arrays for efficient fault characterization in complex configuration spaces[END_REF] and Porter et al. [START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF] characterize defects of the ACE+TAO system. Yilmaz et al. [START_REF] Yilmaz | Covering arrays for efficient fault characterization in complex configuration spaces[END_REF] test each supposed valid configuration on the Red Hat Linux 2.4.9-3 platform and on Windows XP Professional using 96 developer-supplied regression tests. In [START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF], developers currently run the tests continuously on more than 100 largely uncoordinated workstations and servers at a dozen sites around the world. The platforms vary in versions of UNIX, Windows, Mac OS, as well as to real-time operating systems. To examine the impact of masking effects on Apache v2.3.11-beta and MySQL v5.1, Yilmaz et al. [START_REF] Yilmaz | Reducing masking effects in combinatorialinteraction testing: A feedback drivenadaptive approach[END_REF] grouped the test outcomes into three classes: passed, failed, and skipped. They call this approach a ternary-class fault characterization. Gargantini et al. [START_REF] Gargantini | Combinatorial interaction testing for automated constraint repair[END_REF] focus on comparing the defect detection capability on different sample heuristics (see Section 4.2), while Amand et al. [START_REF] Amand | Towards learning-aided configuration in 3d printing: Feasibility study and application to defect prediction[END_REF] deal with comparing the accuracy of several learning algorithms to predict whether a configuration will lead to a defect. In the dynamic configuration scenario, Krismayer et al. [START_REF] Krismayer | Mining constraints for event-based monitoring in systems of systems[END_REF] use event logs (via simulation) from a real-world automation SoS to mine different types of constraints according to real-time defects.

Finally, Martinez et al. [START_REF] Martinez | Towards estimating and predicting user perception on software product variants[END_REF] consider a 5-point user likeability scale with values ranging from 1 (strong dislike) to 5 (strong like). In this work, humans have reviewed and labeled configurations.

Accuracy of measurements. In general, measuring NFPs (e.g., time) is a difficult process since several confounding factors should be controlled. The need to gather measures over numerous configurations exacerbates the problem.

Weckesser et al. [START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF] mitigated the construct threat of the inherent randomness and repeated all runs five times with different random seeds. Measurements started after a warm-up time of 5 minutes. Kaltenecker et al. [START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF] measured each configuration between 5 to 10 times until reaching a standard deviation of less than 10%. Zhang et al. [START_REF] Zhang | Performance prediction of configurable software systems by fourier learning (t)[END_REF] repeated the measurements 10 times.

To investigate the influence of measurement errors on the resulting model, Kolesnikov et al. [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF] and Duarte et al. [START_REF] Duarte | Learning nondeterministic impact models for adaptation[END_REF] conducted a separate experiment. They injected measurement errors to the original measurements and repeated the learning process with polluted datasets. Then, they compared the prediction error of the noisy models to the prediction error of the original models to see the potential influence of measurement errors. For each subject system, Kolesnikov et al. [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF] repeated the learning process five times for different increasing measurement errors.

Dynamic properties are susceptible to measurement errors (due to non-controlled external influences) which may bias the results of the measuring process. To account for measurement noise and be subject to external influences, these properties need to be measured multiple times on dedicated systems. Thus, the total measurement cost to obtain the whole data used in the experiments is overly expensive and time-consuming (e.g., Kaltenecker et al. [START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF] spent multiple years of CPU time). According to Lillacka et al. [START_REF] Lillack | Improved prediction of non-functional properties in software product lines with domain context[END_REF], there is a warm-up phase followed by multiple times runs; and the memory must be set up large enough to prevent disturbing effects from the Garbage Collector, as well as all operations, must be executed in memory so that disk or network I/O will also produce no disturbing effects. Most of the works only consider the variability of the subject system, while they use static inputs and hardware/software environments. Therefore, the resulting model may not properly characterize the performance of a different input or environment, since most of the properties (e.g., CPU power consumption and compression time) are dependent of the input task and the used hardware/software. Consequently, hardware/software must also be taken into account as dependent variables as considered by Jamshidi et al. [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF].

There are some properties that are much accurate, because e.g. they are not influenced by the used hardware, such as footprint. Siegmund et al. [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF] parallelized the measurements of the footprint on three systems and used the same compiler. Moreover, the footprint can be measured quickly only once, without measurement bias.

Cost of measurements. The cost of observing and measuring software can be important, especially when multiple configurations should be considered. The cost can be related to computational resources needed (in time and space). It can also be related to human resources involved in labelling some configurations [START_REF] Martinez | Towards estimating and predicting user perception on software product variants[END_REF].

Zuluaga et al. [START_REF] Zuluaga | ε-pal: an active learning approach to the multi-objective optimization problem[END_REF] and Nair et al. [START_REF] Nair | Finding faster configurations using flash[END_REF] measure the quantitative NFP area of a field-programmable gate array (FPGA) platform consisting of 206 different hardware implementations of a sorting network for 256 inputs. They report that the measurement of each configuration is very costly and can take up to many hours. Porter et al. [START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF] characterization of defects for each configuration ranges from 3 hours on quad-CPU machines to 12-18 hours on less powerful machines. Yilmaz et al. [START_REF] Yilmaz | Covering arrays for efficient fault characterization in complex configuration spaces[END_REF] took over two machine years to run the total of 18,792 valid configurations. In Siegmund et al. [START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF], a single measurement of memory footprint took approximately 5 minutes. Temple et al. [START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF] report that the execution of a configuration to measure video quality took 30 minutes on average. They used grid computing to distribute the computation and scale the measurement for handling 4,000+ configurations. The average time reported by Murwantara et al. [START_REF] Murwantara | Measuring energy consumption for web service product configuration[END_REF] to measure the CPU power consumption of all sampled configurations was 1,000 seconds. The authors set up a predefined threshold to speed up the process.

There are fifteen main NFPs supported in the literature. Some of them are less costly, such as code complexity, which can be measured statically by analyzing the number of code lines [START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF]. As a result, the measurements can be parallelized and quickly done only once. Otherwise, dynamic properties, such as CPU power consumption and response time are directly related to hardware and external influences. While CPU power consumption might be measured under different loads over a predefined threshold time, the property response time is much costly as a threshold can not be defined and the user does not know how long it will take. To support the labeling of features, some works use synthetic NFPs and statistical analysis strategies. As the same NFP may be measured in different ways (e.g., video quality can be measured either by user feedback or execution of a program to automatically attribute labels to features). Practitioners need to find a sweet spot to have accurate measurements with a small sample.

Depending on the subject system and application domain (see the dataset of [START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF]), there are more favorable cases with only a few seconds per configuration. However, even in this case, the overall cost can quickly become prohibitive when the number of configurations to measure is too high. In [START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF], all training simulation runs took approximately 412 hours of CPU time.

Numerous qualitative and quantitative properties of configurations are measured mainly through the use of automated software procedures. For a given subject system and its application domain, there may be more than one measure (e.g., CPU power consumption and video quality for x264). Time is the most considered performance measure and is obtained in the literature through either execution, simulation, static analysis, or synthetic measurement. The general problem is to find a good tradeoff between the cost and the accuracy of measuring numerous configurations (e.g., simulation can speed up the measurement process at the price of approximating the real observations).

RQ4: How are learning-based techniques validated?

In this section, we aim at understanding how the validation process is conducted in the literature.

There are five design strategies documented in the literature to explore the sample data for learning and validation (see Table 3). Merge. Training, testing, and validation sets are merged. Kolesnikov et al. [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF] studied and analyzed the trade-offs among prediction error, model size, and computation time of performance-prediction models. For the purpose of their study, they were specifically interested to explore the evolution of the model properties to see the maximum possible extent of the corresponding trade-offs after each iteration of the learning algorithm. So, they performed a whole-population exploration of the largest possible learning set (i.e., all valid configurations). Therefore, in their validation process, training, testing and validation sets are merged. In a similar scenario, other authors [START_REF] Chen | Performance prediction of component-based applications[END_REF][START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF]] also used the whole sample for both, learning and validation, even they considered a small set of valid configurations as the sample. Some studies justify the use of a merge pool with the need to compare different sampling methods. However, training the algorithm on the validation pool may introduce bias to the results of the accurateness of the approach. To overcome this issue, studies have used other well-established ML design strategies.

In particular, Guo et al. [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF] study the trade-offs among hold-out, cross-validation, and bootstrapping. For most of the case studies, 10-fold cross-validation outperformed hold-out and bootstrapping. In terms of the running time, although these three strategies usually take seconds to run, Guo et al. [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF] show that hold-out is the fastest one, and 10-fold cross-validation tends to be faster than bootstrapping.

Hold-Out (HO). Most of the works [START_REF] Jamshidi | An uncertainty-aware approach to optimal configuration of stream processing systems[END_REF][START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF][START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Valov | Empirical comparison of regression methods for variability-aware performance prediction[END_REF][START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF][START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF][START_REF] Acher | Varylatex: Learning paper variants that meet constraints[END_REF][START_REF] Temple | Towards adversarial configurations for software product lines[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF][START_REF] Yilmaz | Covering arrays for efficient fault characterization in complex configuration spaces[END_REF][START_REF] Krismayer | Mining constraints for event-based monitoring in systems of systems[END_REF][START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF][START_REF] Zuluaga | ε-pal: an active learning approach to the multi-objective optimization problem[END_REF][START_REF] Amand | Towards learning-aided configuration in 3d printing: Feasibility study and application to defect prediction[END_REF][START_REF] Alipourfard | Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF] used a hold-out design. Hold-out splits an input sample S C into two disjoined sets S t and S v , one for training and the other for validation, i.e. S C = S t ∪ S v and S t ∩ S v = ∅. To avoid bias in the splitting procedure, some works repeated it multiple times. For example, in Valov et al. [START_REF] Valov | Empirical comparison of regression methods for variability-aware performance prediction[END_REF], the training set for each sample size was selected randomly 10 times. For transfer-learning applications [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF][START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF], the training set comes from samples of the target and related sources, while the validation set comes from samples only from the target source.

Cross-Validation (CV).

Cross-validation splits an input sample S C into k disjoined subsets of the same size, i.e., S = S 1 ∪ ... ∪ S k , where S i ∩ S j = ∅ (i = j); each subset S i is selected as the validation set S v , and all the remaining k -1 subsets are selected as the training set S t . Yilmaz et al. [START_REF] Yilmaz | Reducing masking effects in combinatorialinteraction testing: A feedback drivenadaptive approach[END_REF] used a 5-fold cross-validation to create multiple models from different subsets of the input data. In a 5-fold cross-validation, the sample S is partitioned into 5 subsets (i.e., S = S 1 ∪ S 2 ∪ S 3 ∪ S 4 ∪ S 5 ) of the same size. Each subset is selected as the validation set and all of the remaining subsets form the training set. Most authors [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Samreen | Daleel: Simplifying cloud instance selection using machine learning[END_REF][START_REF] Martinez | Towards estimating and predicting user perception on software product variants[END_REF][START_REF] Murwantara | Measuring energy consumption for web service product configuration[END_REF][START_REF] Safdar | Mining cross product line rules with multiobjective search and machine learning[END_REF] relayed on a 10-fold crossvalidation. 10-fold cross-validation follows the same idea of a 5-fold cross-validation, producing k = 10 groups of training and validation sets. Guo et al. [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF] show that a 10-fold cross-validation design does not work well for very small samples, e.g., it did not work for an Apache system when the sample size was 9 configurations. Bootstrapping (BT). Four studies [START_REF] Van Aken | Automatic database management system tuning through large-scale machine learning[END_REF][START_REF] Jamshidi | An uncertainty-aware approach to optimal configuration of stream processing systems[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF] relayed on the bootstrapping design. Bootstrapping relies on random sampling with replacement. Given an input sample S C with k configurations, bootstrapping randomly selects a configuration C b , with 1 ≤ b ≤ k and copies it to the training set S t . However, it keeps C b in S C for the next selection. This process is repeated k times. Given the training sample, bootstrapping uses S C \S t as the validation set S v . Nair et al. [START_REF] Nair | Using bad learners to find good configurations[END_REF] used a non-parametric bootstrap test with 95% confidence for evaluating the statistical significance of their approach. In non-parametric bootstrap, the input sample S C is drawn from a discrete set, i.e., 95% of the resulting sample S t should fall within the 95% confidence limits about S C . Dynamic Sector Validation. Westermann et al. [START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF] relayed on two types of Dynamic Sector Validation designs: with local prediction error scope (DSL) and with global prediction error scope (DSG). Dynamic Sector Validation decides if a sample configuration C i ∈ S C is part either of the training or testing set based on the sector's prediction error of the adopted learning techniques. In DSL, all sectors have a prediction error that is less than the predefined threshold, while in DSG the average prediction error of all sectors is less than the predefined threshold.

There are studies where a sample design is not applicable [START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Gargantini | Combinatorial interaction testing for automated constraint repair[END_REF][START_REF] Kolesnikov | On the relation of external and internal feature interactions: A case study[END_REF][START_REF] Couto | Products go green: Worst-case energy consumption in software product lines[END_REF][START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF][START_REF] Siegmund | Attributed variability models: outside the comfort zone[END_REF][START_REF] Xi | A smart hill-climbing algorithm for application server configuration[END_REF][START_REF] Zheng | Automatic configuration of internet services[END_REF][START_REF] Jamshidi | Machine learning meets quantitative planning: Enabling self-adaptation in autonomous robots[END_REF][START_REF] Siegmund | Measuring non-functional properties in software product line for product derivation[END_REF][START_REF] Etxeberria | Performance-based selection of software and hardware features under parameter uncertainty[END_REF], as well as studies [START_REF] Sharifloo | Learning and evolution in dynamic software product lines[END_REF][START_REF] Zhang | Performance prediction of configurable software systems by fourier learning (t)[END_REF][START_REF] Siegmund | Family-based performance measurement[END_REF][START_REF] Hutter | Sequential model-based optimization for general algorithm configuration[END_REF][START_REF] Osogami | Optimizing system configurations quickly by guessing at the performance[END_REF] without further details about the sample design. Given the sampling design, evaluation metrics are used to (learn and) validate the resulting prediction model. Table 4 sketches which evaluation metrics are used in the literature. The first column identifies the study reference(s). The second and third columns identify the name of the evaluation metric and its application objective, respectively. Similar to Table B.7, here the application objective is related to the scenarios in which each evaluation metric has been already used in the SPL literature. Therefore, in future researches, some metrics may be explored in other scenarios as well. Notice that some studies [START_REF] Sharifloo | Learning and evolution in dynamic software product lines[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Sincero | Approaching non-functional properties of software product lines: Learning from products[END_REF] did not report any detail about the validation process. [START_REF] Švogor | An extensible framework for software configuration optimization on heterogeneous computing systems: Time and energy case study[END_REF][START_REF] Saleem | Personalized decision-strategy based web service selection using a learning-to-rank algorithm[END_REF] Rank Accuracy (RA) A3 [START_REF] Van Aken | Automatic database management system tuning through large-scale machine learning[END_REF][START_REF] Ding | Autotuning algorithmic choice for input sensitivity[END_REF][START_REF] Xu | Satzilla: portfolio-based algorithm selection for sat[END_REF][START_REF] Osogami | Optimizing system configurations quickly by guessing at the performance[END_REF] Tuning Time A3 [START_REF] Murwantara | Measuring energy consumption for web service product configuration[END_REF][START_REF] Afia | Performance prediction using support vector machine for the configuration of optimization algorithms[END_REF][START_REF] Samreen | Daleel: Simplifying cloud instance selection using machine learning[END_REF] Mean Square Error (MSE) A3, A4 [START_REF] Samreen | Daleel: Simplifying cloud instance selection using machine learning[END_REF] p-value, R2, Residual Standard Error (RSE) A4 [START_REF] Chen | Boosting the performance of computing systems through adaptive configuration tuning[END_REF] Reward A4 [START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF] Statistical Significance A4 [START_REF] Temple | Towards adversarial configurations for software product lines[END_REF][START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF] Qualitative Analysis A4, A5 [START_REF] Krismayer | Mining constraints for event-based monitoring in systems of systems[END_REF] Ranking Constraints A4, A5 [START_REF] Safdar | Mining cross product line rules with multiobjective search and machine learning[END_REF] Delaney's Statistics A5 [START_REF] Safdar | Mining cross product line rules with multiobjective search and machine learning[END_REF] Distance Function, Hyper-volume (HV) A5 [START_REF] Gargantini | Combinatorial interaction testing for automated constraint repair[END_REF] Equivalence among Combinatorial Models A5 [START_REF] Gargantini | Combinatorial interaction testing for automated constraint repair[END_REF] Failure Index Delta (FID), Totally Repaired Models (TRM)

A5

Evaluation metrics. State-of-the-art techniques rely on 50 evaluation metrics from which it is possible to evaluate the accuracy of the resulting models in different application scenarios. There are metrics dedicated to supervised classification problems (e.g., precision, recall, F-measure). In such settings, the goal is to quantify the ratio of correct classifications to the total number of input samples. For instance, Temple et al. [START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF] used precision and recall to control whether acceptable and non-acceptable configurations are correctly classified according to the ground truth. Others [START_REF] Temple | Towards adversarial configurations for software product lines[END_REF][START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF] use qualitative analysis to identify features with significant effects on defects, and understand feature interactions and decide whether further investigation of features is justified.

There are also well-known metrics for regression problems, such as, Mean Relative Error (MRE -Equation 1) and Mean Absolute Error (MAE -Equation 2). These met-rics aim at estimating the accuracy between the exact measurements and the predicted one.

M RE = 1 |S v | C∈Sv |C (pi) -C (p i ) | C (pi) (1)
M AE = |C (p i ) -C (pi) | C (pi) (2)
Where S v is the validation set, and C (pi) and C (p i ) indicate the exact and predicted value of p i for C ∈ S v , respectively.

Contributions addressing learning-to-rank problems develop specific metrics capable of assessing the ability of a learning method to correctly rank configurations. For example, Nair et al. [START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF] use the error difference between the ranks of the predicted configurations and the true measured configurations. To get insights about when stop sampling, they also discuss the trade-off between the size of the training set and rank difference.

Interpretability metrics. Some metrics are also used to better understand configuration spaces and their distributions, for example, when to use a transfer learning technique. In this context, Jamshidi et al. [START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF] use a set of similarity metrics (Kullback-Leibler, Pearson Linear Correlation, Spearman Correlation Coefficient, and Rank Correlation) to investigate the relatedness of the source and target environments. These metrics compute the correlation between predicted and exact values from source and target systems. Moreover, Kolesnikov et al. [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF] and Valov et al. [START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF] use size metrics as insights of interpretability. Valov et al. [START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF] compare the structure (size) of prediction models built for the same configurable system and trained on different hardware platforms. They measured the size of a performance model by counting the features used in the nodes of a regression tree, while Kolesnikov et al. [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF] define the model size metric as the number of configuration options in every term of the linear regression model. Temple et al. [START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF] reported that constraints extracted from decision trees were consistent with the domain knowledge of a video generator and could help developers preventing non-acceptable configurations. In their context, interpretability is important both for validating the insights of the learning and for encoding the knowledge into a variability model.

Sampling cost.

Several works [START_REF] Alipourfard | Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics[END_REF][START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF] use a sampling cost measurement to evaluate their prediction approach. In order to reduce measurement effort, these approaches aim at sampling configurations in a smart way by using as few configurations as possible. In [START_REF] Alipourfard | Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics[END_REF][START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF], sampling cost is considered as the total number of measurements required to train the model. These authors show the trade-off between the size of the training set and the accuracy of the model. Nair et al. [START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF] compare different learning algorithms, along with different sampling techniques to determine exactly those configurations for measurement that reveal key performance characteristics and consequently reach the minimum possible sampling cost. Sarkar et al. [START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF] introduce a cost model, which they consider the measurement cost involved in building the training and testing sets, as well as the cost of building the prediction model and computing the prediction error. Sampling cost can be seen as an additional objective of the problem of learning configuration spaces, i.e. not only the usual learning accuracy should be considered.

There is a wide range of validation metrics reported in the literature, and some are reused amongst papers (e.g., MRE). However, several metrics can be used for the same task and there is not necessarily a consensus. Finally, most of the studies use a unique metric for validation, which may provide incomplete quantification of the accuracy.

The evaluation of the learning process requires to confront a trained model with new, unmeasured configurations. Hold-out is the most considered strategy for splitting configuration data into a training set and a testing set. Depending on the targeted task (regression, ranking, classification, understanding), several metrics have been reused or defined. In addition to learning accuracy, sampling cost and interpretability of the learning model can be considered as part of the problem and thus assessed.

Emerging Research Themes and Open Challenges

In the previous sections, we have given an overview of the state-of-the-art in sampling, measuring, and learning software configuration spaces 11 . Here, we will now discuss open challenges and their implications for research and practice.

There are a few reports of real-world adoption [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF][START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Jamshidi | Machine learning meets quantitative planning: Enabling self-adaptation in autonomous robots[END_REF][START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF], but overall it seems that despite wide applicability in terms of application domains, subject systems, or measurement properties, there is little concrete evidence that the proposed learning techniques are actually adopted in widespread everyday practice.

For instance, the x264 video encoder is widely considered in the academic literature, but we are not aware of reports that learning-based solutions, as discussed here, have an impact on the way developers maintain it or on the way users configure it.

Hence, in this section, we discuss some scientific and technical obstacles that contribute to this gap between what is available in academic literature and what is actually adopted in practice. In doing so, we aim to identify [START_REF] Svahnberg | A taxonomy of variability realization techniques: Research articles[END_REF] points that should be considered when choosing and applying particular techniques in practice and (2) opportunities for further research. We summarise our findings in Table 5 at the end of the section.

Generalization and transfer learning

A first and important risk when adopting a learning-based approach is that the model may not generalize to new, previously unseen configuration data. Some works have already observed that many factors can influence the performance distribution of a configurable system: the version of the software, the hardware on which the software is executed, the workload, etc. The consequence is that, e.g., a prediction model of a configurable system may reach high accuracy in a given situation (close to the one used for training) and then the performance drops when some of these factors change. The reuse of learning-based models is thus an important concern; the worst-case scenario is to learn a new model for every usage of a configurable system. We notice that several recent works aim to transfer the learning-based knowledge of a configuration space to another setting (e.g., a different hardware setup) [START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF][START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF]. Another research direction is to characterize which and to what extent factors influence the performance distribution of a configurable system. The hope is to propose learning-based solutions that are effective independent of the context in which the configurable software is deployed.

Interpretability

In many engineering scenarios, developers or users of configurable systems want to understand learning-based models and not treat them as back-boxes. This observation partly explains why decision trees (CART) and step-wise linear regressions are the most used learning algorithms (see Table B.7): the resulting models provide interpretable information, through rules or coefficients. The challenge is to find a good-enough compromise between accuracy and interpretability. Interpretability can be the major objective to fulfill (application A2) or an important concern, e.g., mining constraints (application A5) requires the use of rule-based learning. For other tasks such as optimization (A3), accuracy seems more important and hence the choice of a well-performing learning algorithm might be more important than interpretability.

We notice that there are few studies reported in the literature whose evaluation objective is the comprehension of configurable systems, for instance, to understand which options are influential. However, we are not aware of empirical user studies to validate the learning approach with regards to the comprehension by domain experts. The existing works typically rely on a model size metric to report the level of comprehension [START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF][START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF]. How to evaluate interpretability is an open issue in machine learning [START_REF] Molnar | Interpretable Machine Learning[END_REF] that also applies to the learning of software configuration space.

Trade-offs when choosing learning algorithms

Another concern to consider is the computation time needed to apply a statistical learning method. Many factors should be considered: the algorithm itself, the size of the training set, the number of features, the number of trees (for random forest), the hyperparameters, etc. Looking at experiments and subject systems considered in the literature, the size of the training set as well as the number of features remain affordable so far: only a few seconds and minutes are needed to train. However, there are some exceptions with studies involving large systems (e.g., Linux see [START_REF] Acher | Learning from thousands of build failures of linux kernel configurations[END_REF][START_REF] Acher | Learning very large configuration spaces: What matters for linux kernel sizes[END_REF]) or learning procedures that can take hours.

Overall, the choice of a learning algorithm is a tradeoff between different concerns (interpretability, accuracy, training time). Which one practitioners should use and under which conditions? There is still a lack of actionable frameworks that would automatically choose or recommend a suited learning technique based on an engineering context (e.g., targeted NFPs, need of interpretability). So far, tree-based methods constitute a good compromise and can be applied for many applications. More (empirical) research is needed to help practitioners systematically choosing an adequate solution.

Sampling: the quest of the right strategy

Currently, there is no dominant sampling technique that can be used without further consideration in any circumstance by practitioners. We reported 23 highlevel sampling techniques used by state-of-the-art approaches. Data of our systematic review shows that there is no clear relation between the choice of a sampling strategy and a specific application objective (A1, ..., A6). Hence, more research is needed to evaluate the effectiveness of particular sampling techniques.

A first specific challenge when sampling over a configurable system is to generate configurations conforming to constraints. This boils down to solving a satisfiability problem (SAT) or constraint satisfaction problem (CSP) when numerical values should be handled [START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Temple | Towards adversarial configurations for software product lines[END_REF][START_REF] Amand | Towards learning-aided configuration in 3d printing: Feasibility study and application to defect prediction[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF]. Recent results show that uniform, random sampling is still challenging for configurable systems [START_REF] Plazar | Uniform sampling of sat solutions for configurable systems: Are we there yet?[END_REF]. Hence true random sampling is sometimes hard to achieve and some approximate alternatives, such as distance-based sampling, have been proposed [START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF].

In the context of testing software product lines, there are numerous available heuristics, algorithms, and empirical studies [START_REF] Medeiros | A comparison of 10 sampling algorithms for configurable systems[END_REF][START_REF] Lopez-Herrejon | A first systematic mapping study on combinatorial interaction testing for software product lines[END_REF][START_REF] Do Carmo | On strategies for testing software product lines: A systematic literature review[END_REF][START_REF] Lee | A survey on software product line testing[END_REF][START_REF] Thüm | A classification and survey of analysis strategies for software product lines[END_REF]. Though the sample has not the same purpose (efficient identification of bugs), we believe a learning-based process could benefit from works done in the software testing domain. An important question is whether heuristics or sampling criterion devised originally for testing have the same efficiency when applied to learning configuration spaces.

A striking challenge is to investigate whether there exists a one-size-fits-all strategy for efficiently sampling a configuration space. The effectiveness of a sampling strategy may depend on the choice of a learning algorithm [START_REF] Grebhahn | Predicting performance of software configurations: There is no silver bullet[END_REF], which is hard to anticipate. Besides, recent results suggest there is no single dominant sampling even for the same configurable system. Instead, different strategies perform best on different workloads and non-functional properties [START_REF] Pereira | Sampling effect on performance prediction of configurable systems: A case study[END_REF][START_REF] Grebhahn | Predicting performance of software configurations: There is no silver bullet[END_REF]. An open issue is to automatically and a priori select the most effective sampling in a given situation.

Finally, we observe that most of the reported techniques only use as input a model of the configuration space (e.g., a feature model). The incorporation of knowledge and other types of input data in the sampling phase needs more investigation in future research (e.g., system documentation, implementation artifacts, code smells, and system evolution).

Cost-effectiveness of learning-based solutions

A threat to practical adoption is the cost of learning-based approaches. In addition to the cost of training a machine learning model (see the earlier discussion on tradeoffs), the cost of gathering data and measuring configurations can be significant (several days). Most of the NFPs reported in the literature (see Figure 6 and Table C.8) are quantitative and related to performance. An organization may not have the resources to instrument a large and accurate campaign of performance measurements.

A general problem is to reduce the cost of measuring many configurations. For example, the use of a distributed infrastructure (e.g., cloud) has the merit of offering numerous computing resources at the expense of networking issues and heterogenity [START_REF] Leitner | Patterns in the chaos -A study of performance variation and predictability in public iaas clouds[END_REF][START_REF] Bao | Autoconfig: automatic configuration tuning for distributed message systems[END_REF]. Though some countermeasures can be considered, it remains an important practical problem. Hence, there is a tension between the accuracy of measurements and the need to scale up the process to thousands of configurations. At the opposite end of the spectrum, simulation and static analysis are less costly but may approximate the real measures: few studies explore them which demand further investigation. Another issue is how performance measurements transfer to a computational environment (e.g., hardware). The recent rise of transfer learning techniques is an interesting direction, but much more research is needed.

Besides, only a few works deal with uncertainty when learning configuration spaces (e.g., [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF]). There are questions we could ask, such as how many times do we need to repeat measurements? To what degree can we trust the results? The measurement of performance is subject to intensive research: there can be bias, noise or uncertainty [START_REF] Aleti | An efficient method for uncertainty propagation in robust software performance estimation[END_REF][START_REF] Colmant | The next 700 CPU power models[END_REF][START_REF] Trubiani | Plus: Performance learning for uncertainty of software[END_REF] that can have in turn an impact on the effectiveness of the whole learning process.

Ideally, the investments required to sample, measure, and learn configuration spaces should be outweighed by the benefits of effectively performing a task (e.g., finding an optimal configuration). The reduction of the cost is a natural research direction. Another is to find a reasonable tradeoff.

Realistic evaluation

When transferring techniques from early research into practice, we have to be careful with respect to things that can go wrong and effort that is required for the different activities, e.g., data acquisition and preparation, feature engineering, selecting/adapting algorithms and models, evaluation in a controlled experimental setting, deployment for use in practice, and evolution/maintenance. Looking at many papers in ML and related fields, one might get the impression that they focus on activities "in the middle", e.g., the development and evaluation of algorithms, since they can be performed in vitro (in a controlled lab environment), whereas activities at the beginning or the end need to be done in vivo (in "dirty" real-life). Also, see the discussion by Bosch et al. [START_REF] Bosch | Engineering ai systems: A research agenda[END_REF].

This preliminary hypothesis needs further investigation, but it seems that research should move out of the comfort zone in the lab and face the "dirty" reality. Some examples of works that go into this direction are given in the following.

A first step from theoretical research towards applicability is to go beyond reporting bare evaluation metrics (e.g., accuracy) and at least discuss the applicability of the presented approach. As an example, consider Weckesser et al. [START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF] who discuss the applicability and effectiveness of their approach for dynamic reconfiguration with a real-world scenario.

This can be taken further by performing the evaluation (or parts of it) in on a real system. For instance, instead of evaluating a control algorithm with theoretical considerations one can deploy that algorithm onto real physical systems which then have to face the challenges and imperfections of reality. As an example, see Jamshidi et al. [START_REF] Jamshidi | Machine learning meets quantitative planning: Enabling self-adaptation in autonomous robots[END_REF] who deal with the self-adaptation of autonomous robots. They first evaluate their approach in a theoretical setting, which is validated on real systems (theoretical evaluation grounded in practice), and second in a robotics scenario deployed onto robots (evaluation in practice).

Another example is work by Temple et al. [START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF] where the use of machine learning was driven by an open practical problem (how to specify constraints of a product line, how to explore a large configuration space with automated support) and obtained results (rules/constraints) where validated with developers, i.e., through collaboration with domain experts.

A related ingredient for applicable research is to identify and describe problems with a practical perspective and with input from domain experts, ideally with input from actual application scenarios. As an example consider the explicit problem description (of modeling performance as influenced by configuration options) by Kolesnikov et al. [START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF] which is based on a discussion with domain experts (for high-performance computing). The authors complete the paper with an appendix where they discuss the most influential configuration options and link that back to the domain knowledge.

As outlined in the beginning of this section, there is still a gap between research and practice. Some papers in our literature review show it is possible to apply different research methods in order to assess the actual potential of learning-based solutions for engineering real-world systems.

Integrated tools

We observe that several studies provide either open source or publicly available implementations, mainly on top of data science libraries. Tools could assist end-users during, for instance, the configuration process and the choice of options. Variability management tools could be extended to encompass the learning stages reported in this SLR. It could also benefit to software developers in charge of maintaining configurable systems. However, none of the reported studies provide tool support that is fully integrated into existing SPL platforms or mainstream software tools. An exception is SPLConqueror [START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF] that supports an interactive configuration process. We believe the integration of tooling support is key for practical adoption and the current lack may partly explain the gap between practice and research.

Final remarks and summary

The problem of learning configuration spaces is at the intersection of artificial intelligence (mainly statistical machine learning and constraint reasoning) and software engineering. There are many existing works that do not specifically address software systems and are yet related. Hence, there is considerable potential to "connect the dots" and reuse state-of-the-art methods for learning highly-dimensional spaces like software configuration spaces.

For instance, an active research area that is related to choosing the best algorithm is the automated algorithm selection problem [START_REF] Kerschke | Automated algorithm selection: Survey and perspectives[END_REF] where given a computational problem, a set of algorithms, and a specific problem instance to be solved, the problem is to determine which of the algorithms can be selected to perform best on that instance. In our case, the set comprises all (valid) configurations of a single, parameterized algorithm (whereas the set of algorithms come from different software implementation and systems for the problem of algorithm selection). We also believe the line of research work presented in this literature review can be related to specific problems like compiler autotuning (see e.g., [START_REF] Ashouri | A survey on compiler autotuning using machine learning[END_REF]) or database management system tuning (see e.g., [START_REF] Van Aken | Automatic database management system tuning through large-scale machine learning[END_REF]).

Finally, Table 5 summarizes the challenges discussed in this section with (1) points to consider in practical applications and (2) opportunities for further research.

Threats to Validity

This section discusses potential threats to validity that might have affected the results of the SLR. We faced similar threats to validity as any other SLR. The findings of this SLR may have been affected by bias in the selection of the primary studies, inaccuracy in the data extraction and in the classification of the primary studies, and incompleteness in defining the open challenges. Next, we summarize the main threats to the validity of our work and the strategies we have followed to minimize their impact. We discussed the SLR validity with respect to the two groups of common threats to validity: internal and external validity [START_REF] Wohlin | Experimentation in software engineering: an introduction[END_REF].

Internal validity. An internal validity threat concerns the reliability of the selection and data extraction process. To further increase the internal validity of the review results, the search for relevant studies was conducted in several relevant scientific databases, and it was focused not only on journals but also on conferences, symposiums, and workshops. Moreover, we conducted the inclusion and exclusion processes in parallel by involving three researchers and we cross-checked the outcome after each phase. In the case of disagreements, we discussed until a final decision was achieved. Furthermore, we documented potentially relevant studies that were excluded. Therefore, we believe that we have not omitted any relevant study. However, the quality of the search engines could have influenced the completeness of the identified primary studies (i.e., our search may have missed those studies whose authors did not use the terms we used in our search string to specify keywords).

For the selected papers, a potential threat to validity is the reliability and accuracy of the data extraction process, since not all information was obvious to extract (e.g., many papers lacked details about the measurement procedure and the validation design of the reported study). Consequently, some data had to be interpreted which involved subjective decisions by the researchers. Therefore, to ensure the validity, multiple sources of data were analyzed, i.e., papers, websites, technical reports, manuals, and executable. Moreover, whenever there was a doubt about some extracted data in a particular paper, we discussed the reported data from different perspectives in order to resolve all discrepancies. However, we are aware that the data extraction process is a subjective activity and likely to yield different results when executed by different researchers.

External validity. A major threat to external validity is related to the identification of primary studies. Key terms are directly related to the scope of the paper and they can suffer a high variation. We limited the search for studies mainly targeting software systems (e.g., software product lines) and thus mainly focus on software engineering conferences. This may affect the completeness of our search results since we are aware of some studies outside the software engineering community that also address the learning of software configurable systems. To minimize this limitation and avoid missing relevant papers, we also analyzed the references of the primary studies to identify other relevant studies. In addition, this SLR was based on a strict protocol described in Section 2 which was discussed before the start of the review to increase the reliability of the selection and data extraction processes of the primary studies and allow other researchers to replicate this review.

Another external validity concerns the description of open challenges. We are aware that the completeness of open challenges is another limitation that should be considered while interpreting the results of this review. It is also important to explore general contributions from other fields outside the software domain to fully gather the spread knowledge, which may extend the list of findings in this field. Therefore, in future work, the list of findings highlighted in Section 5 may be extended by conducting an additional review, making use of other keywords to be able to find additional relevant studies outside the software community.

Related Work

After the introduction of SLR in software engineering in 2004, the number of published reviews in this field has grown significantly [START_REF] Kitchenham | Systematic literature reviews in software engineering-a systematic literature review[END_REF]. A broad SLR has been conducted by Heradio et al. [START_REF] Heradio | A bibliometric analysis of 20 years of research on software product lines[END_REF] to identify the most influential researched topics in SPL, and how the interest in those topics has evolved over the years. Although these reviews are not directly related to ours, the high level of detail of their research methodology supported to structure and define our own methodology.

Benavides et al. [START_REF] Benavides | Automated analysis of feature models 20 years later: a literature review[END_REF] presented the results of a literature review to identify a set of operations and techniques that provide support to the automatic analysis of variability models. In a similar scenario, Lisboa et al. [START_REF] Lisboa | A systematic review of domain analysis tools[END_REF] and Pereira et al. [START_REF] Pereira | A systematic literature review of software product line management tools[END_REF] conducted an SLR on variability management of SPLs. They reported several dozens of approaches and tools to support stakeholders in an interactive and automatic configuration process. Strict to the application engineering phase, Ochoa et al. [START_REF] Ochoa | A systematic literature review on the semi-automatic configuration of extended product lines[END_REF] and Harman et al. [START_REF] Harman | Search based software engineering for software product line engineering: a survey and directions for future work[END_REF] conducted a literature review on the use of search-based software engineering techniques for optimization of SPL configurations. In contrast to these previous reviews, our SLR provides further details on the use of automatic learning techniques to explore large configuration spaces. We contribute with a catalogue of sampling approaches, measurement procedures, learning techniques, and validation steps that serves as a summarization of the results in this specific field.

In [START_REF] Sayagh | Software configuration engineering in practice: Interviews, survey, and systematic literature review[END_REF], 14 software engineering experts and 229 Java software engineers were interviewed to identify major activities and challenges related to configuration engineering. In complement, a SLR was conducted. In order to select papers, authors focused on those in which the title and abstract contain the keyword "config*", "misconfig*", or "mis-config*". On the one hand, the scope is much broader: It spans all engineering activities of configurable systems whereas our literature review specifically targets learning-based approaches and applications. On the other hand, we learn that configuration spaces are studied in many different engineering contexts and we take care of considering a broader terminology (see Table 1, page 9). Despite different scope and reviewing methodology, the two surveys are complementary. A research direction is to further explore how learning-based approaches classified in this literature review could support configuration engineering activities identified in [START_REF] Sayagh | Software configuration engineering in practice: Interviews, survey, and systematic literature review[END_REF].

In the context of software fault prediction, Malhotra [START_REF] Malhotra | A systematic review of machine learning techniques for software fault prediction[END_REF] conducts an SLR to summarize the learning techniques used in this field and the performance accuracy of these techniques. They classify machine learning techniques into seven categories: Decision Tree, Bayesian Learning, Ensemble Learning, Rule Based Learning, Evolutionary Algorithms, Neural Networks and Miscellaneous. Although our SLR is much restricted by considering only primary studies on the field of software product lines, we also consider fault as a performance metric and thus we also report on most of these techniques. Also, our results depict that the evaluation metrics "precision, recall, and F-Measure" are the most commonly used performance measures to compute the accuracy of reported classification techniques. In addition, our SLR is much broad once it also considers other properties besides fault, therefore we report on both classification and regression techniques and evaluation metrics.

On the context of testing and verifying a software product lines, several works [START_REF] Varshosaz | A classification of product sampling for software product lines[END_REF][START_REF] Medeiros | A comparison of 10 sampling algorithms for configurable systems[END_REF][START_REF] Lopez-Herrejon | A first systematic mapping study on combinatorial interaction testing for software product lines[END_REF][START_REF] Do Carmo | On strategies for testing software product lines: A systematic literature review[END_REF][START_REF] Lee | A survey on software product line testing[END_REF][START_REF] Thüm | A classification and survey of analysis strategies for software product lines[END_REF] discussed product sampling techniques. They classify the proposed techniques into different categories and discuss the required input and criteria used to evaluate these techniques, such as the sample size, the rate of fault detection, and the tool support. Our SLR could benefit from their results through the use of sampling techniques still not explored by learning techniques (e.g. code and requirements coverage), as well as the SPL testing community could benefit from the sampling techniques reported in this paper still not used for testing. In [START_REF] Khalil Abbasi | The anatomy of a sales configurator: An empirical study of 111 cases[END_REF], 111 real-world Web configurators are analyzed but do not consider learning mechanisms.

None of surveys mentioned above directly address the use of learning techniques to explore the behaviour of large configuration spaces through performance prediction. The main topics of previous SLRs include variability model analysis, variability management, configuration engineering, fault prediction and SPL testing. There are several reviews investigating the use of learning-based techniques in many other scenarios, such as spam filtering [START_REF] Guzella | A review of machine learning approaches to spam filtering[END_REF][START_REF] Crawford | Survey of review spam detection using machine learning techniques[END_REF], text-documents classification [START_REF] Khan | A review of machine learning algorithms for text-documents classification[END_REF], automated planning [START_REF] Jiménez | A review of machine learning for automated planning[END_REF], genomic medicine [START_REF] Leung | Machine learning in genomic medicine: a review of computational problems and data sets[END_REF], electricity load forecasting [START_REF] Yildiz | A review and analysis of regression and machine learning models on commercial building electricity load forecasting[END_REF], and others. Overall, these works provide us with a large dataset of sampling, learning, and validation techniques that can be further explored/adapted in the field of software product lines.

Conclusion

We presented a systematic literature review related to the use of learning techniques to analyze large configuration software spaces. We analysed the literature in terms of a four-stage process: sampling, measuring, learning, and validation (see Section 3). Our contributions are fourfold. First, we identified the application of each approach which can guide researchers and industrial practitioners when searching for an appropriate technique that fits their current needs. Second, we classified the literature with respect to each learning stage. Mainly, we give an in-depth view of (i) sampling techniques and employed design; (ii) measurement properties and effort for measurement; (iii) employed learning techniques; and (iv) how these techniques are empirically validated. Third, we provide a repository of all available subject systems used in the literature together with their application domains and qualitative or quantitative properties of interest. We welcome any contribution by the community: The list of selected studies and their classification can be found in our Web supplementary material [START_REF]Learning software configuration spaces: A systematic literature review[END_REF]. Fourth, we identify the main shortcomings of existing approaches and non-addressed research areas to be explored by future work.

Our results reveal that the research in this field is application-dependent: Though the overall scheme remains the same ("sampling, measuring, learning"), the concrete choice of techniques should trade various criterion like safety, cost, accuracy, and interpretability. The proposed techniques have typically been validated with respect to different metrics depending on their tasks (e.g., performance prediction). Although the results are quite accurate, there is still need to decrease learning errors or to generalize predictions to multiple computing environments. Given the increasing interest and importance of this field, there are many exciting opportunities of research at the interplay of artificial intelligence and software engineering.

Figure 1 :

 1 Figure 1: Features, configurations, sample, measurements, and learning.

3 Figure 2 :

 32 Figure 2: Flow of the paper selection process: papers retrieved from each digital library and details of the selection phases.

Figure 3 :

 3 Figure 3: Employed ML stages to explore SPL configuration spaces.

Figure 4 :

 4 Figure 4: Study application objective.

  compute minbucket and minsplit based on the size of the input sample, i.e., if |S C | ≤ 100, then minbucket = | |S C | 10 + 1 2 | and minsplit = 2× minbucket; if |S C | > 100, then minsplit = | |S C | 10 + 1 2 | and minbucket = | minsplit 2

Figure 5 :

 5 Figure 5: Strategies applied to measure the sample of configurations.

  time, encoding time, compression time, I/O time, runtime, speed, timeliness, latency) response rate (throughput, compression ratio) workload memory (memory footprint, memory consumption, CPU usage) defects CPU power consumption performance* video quality (Peak Signal to Noise Ratio) accuracy area size (compressed size, video size) code complexity (iterations) number of pages, user likeability, cost, maintainability, availability, successability, reliability, compliance, best practice, documentation

Figure 6 :

 6 Figure 6: Non-functional properties measured in the literature.*Each application domain or even each system has its own specific definition of performance. These works did not specify the NFP(s) that refer to performance.

Table 1 :

 1 Keywords used to build the search strings.

	Term	Keywords
	Product	product line, configurable (system, software), software configurations,
	Line	configuration of a software, feature (selection, configuration)
	Learning	learning techniques, (machine, model, statistical) learning
	Performance	performance (prediction, model, goal), (software, program, system) per-
	Prediction	formance, (prediction of, measure) non-functional properties
	Predict	predict, measure, transfer learning, optimal (configuration, variant), adap-
		tation rules, constraints, context
	Medicine	gene, medicine, disease, patient, biology, diagnosis, molecular, health,
		brain, biomedical

Table 2 :

 2 Applicability of selected primary studies. A1 : Pure Prediction; A2 : Interpretability of Configurable Systems; A3 : Optimization; A4 : Dynamic Configuration; A5 : Mining Constraints; A6 : SPL Evolution. Random sampling. Several studies have used random sampling [49, 32, 45, 46, 33, 34, 65, 9, 71, 60, 72, 104, 63, 64, 42, 48, 57, 88, 76] with different notions of randomness. Guo et al. [49] consider four sizes of random samples for training: N , 2N , 3N , and M , where N is the number of features of a system, and M is the number of minimal valid configurations covering each pair of features. They choose size N , 2N , and 3N , because measuring a sample whose size is linear in the number of features is likely feasible and reasonable in practice, given the high cost of measurements by execution (see Section 4.3). Valov et al. and Guo et al.

	App. Reference
	A1	[69, 49, 32, 70, 39, 40, 41, 42, 71, 72, 73, 74, 75, 76, 47, 43, 77, 78, 79, 80, 81]
	A2	[46, 50, 82, 48, 43, 83, 84]
	A3	[33, 85, 57, 34, 86, 35, 87, 88, 44, 89, 36, 37, 90, 13, 43, 91, 92, 38, 93, 94, 95, 96, 97,
		98, 99, 100, 101]
	A4	[45, 60, 61, 62, 63, 9, 59, 102, 43, 83, 103]
	A5	[65, 104, 64, 66, 67, 68, 43, 9, 59, 105]
	A6	[102, 13, 43]

Table 3 :

 3 Sample designs reported in the literature.

	Sample Design	Reference
	Merge	[69, 33, 50, 34, 39, 40, 86, 41, 42, 82, 76]
	Hold-Out	[89, 49, 45, 70, 65, 9, 71, 48, 60, 104, 64, 88, 66, 59,
		47, 36, 68, 90, 57, 32, 35, 79, 80, 92, 77, 38, 93, 94,
		95, 96, 99, 83, 103, 81]
	Cross-Validation	[85, 87, 61, 73, 32, 105, 97, 83]
	Bootstrapping	[44, 89, 57, 32, 98]
	Dynamic Sector Validation	[35]

Table 4 :

 4 Validation procedure reported in the literature. A1 : Pure Prediction; A2 : Interpretability of Configurable Systems; A3 : Optimization; A4 : Dynamic Configuration; A5 : Mining Constraints; A6 : SPL Evolution.

	Reference	Evaluation Metric	Applicability
	[71]	Closeness Range, Winner Probability	A1
	[80, 81]	Coverage	A1
	[74]	Jaccard Similarity	A1
	[80]	Performance-Relevant Feature Interactions De-	A1
		tection Accuracy

Table 4 :

 4 Validation procedure reported in the literature. A1 : Pure Prediction; A2 : Interpretability of Configurable Systems; A3 : Optimization; A4 : Dynamic Configuration; A5 : Mining Constraints; A6 : SPL Evolution.

	Reference	Evaluation Metric	Applicability
	[73]	t-masked metric	A1
	[79]	True Positive (TP) Rate, False Positive (FP)	A1
		Rate, Receiver Operating Characteristic (ROC)
	[69, 49, 32, 50, 39, 40, 41, 42, 71,	Mean Relative Error (MRE)	A1, A2, A3, A4
	48, 35, 72, 88, 75, 76, 63, 47, 38,
	98, 83]
	[90, 70, 60, 34, 88, 38, 83, 81]	Sampling Cost	A1, A2, A3, A4
	[35, 72]	Highest Error (HE)	A1, A3
	[45, 33, 85, 87, 47, 89, 78, 92, 77,	Mean Absolute Error (MAE)	A1, A3, A4
	96, 94]
	[76, 105, 100]	Mann-Whitney U-test	A1, A3, A5
	[61, 76]	F-test	A1, A4
	[65, 9, 104, 74, 68]	Precision, Recall	A1, A4, A5
	[66, 73, 68, 105, 79]	Precision, Recall, F-measure	A1, A5
	[84]	GAP	A2
	[46]	Kullback-Leibler (KL), Pearson Linear Correla-	A2
		tion, Spearman Correlation Coefficient
	[48, 50]	Structure of Prediction Models	A2
	[46, 63]	Rank Correlation	A2, A4
	[44, 38]	Domain Experts Feedback	A3
	[101]	Error Probability	A3
	[87]	Global Confidence, Neighbors Density Confi-	A3
		dence, Neighbors Similarity Confidence
	[35]	LT15, LT30	A3
	[57, 34]	Mean Rank Difference	A3
	[85]	Median Magnitude of the Relative Error	A3
		(MdMRE)
	[36]	Pareto Prediction Error	A3
	[94,

Table 5 :

 5 Summary of open challenges and (i) resulting considerations for practical applications and (ii) necessary research.

	Open Challenge	Considerations in Practical	Research Needed
		Applications
	Generalisation and	Given a new "context" (e.g., hardware	Transfer learning from a known
	transfer learning	change), can a learning model be	configuration space to a new
	(Section 5.1)	reused?	configuration one; identification of
			factors that influence performance
			distribution
	Interpretability	Consider techniques with interpretable	Methods for assessing interpretability;
	(Section 5.2)	representation (e.g., CART, step-wise	techniques that generate an
		linear regression); application	explanation in hindsight
		objectives might determine relative
		importance of interpretability
	Learning trade-offs	Beyond raw ML performance (e.g.,	Lack of actionable frameworks to
	(Section 5.3)	accuracy) consider bigger picture (e.g.,	choose techniques based on
		interpretability, integration in	consideration of multiple goals and
		surrounding software system,	trade-offs between them
		robustness against changes)
	Sampling (Section 5.4) No clear relation between application	More work needed on effectiveness and
		objective and chosen sampling	strengths of sampling techniques in
		techniques; different strategies perform	different contexts
		best on different workloads and NFPs
		(see e.g., [116, 115])
	Cost-effectiveness	The cost for executing and measuring	Simulations; distributed
	(Section 5.5)	configurations can be a barrier	infrastructures; handling of
			uncertainty; transfer learning to reuse
			knowledge
	Evaluation/application	Expect that techniques described in	Existing works often evaluated in
	in realistic settings	academic literature do not "just work"	controlled environments, more work
	(Section 5.6)	out of the box	needed where techniques are
			evaluated/applied in realistic settings
	Integrated tools	Consider tools that are open for	Lack of integrated tools for supporting
	(Section 5.7)	integration in real-world projects	developers and users

We decided not to use Google Scholar due to search engine limitations, such as the very strict size of the search string.

http://ieee.org/ieeexplore

http://dl.acm.org

http://www.sciencedirect.com

http://link.springer.com

http://www.scopus.com

https://www.google.com/sheets/about/

http://fosd.de/SPLConqueror.

https://corpus.canterbury.ac.nz/

For a complete list of hardware/software variability, see the repository at https://github.com/pooyanjamshidi/transferlearning

We consolidate the results from all research questions through a set of bubble charts in our supplementary material at https://github.com/VaryVary/ML-configurable-SLR.

Acknowledgments. This research was partially funded by the ANR-17-CE25-0010-01 VaryVary project and by Science Foundation Ireland grant 13/RC/2094. We would like to thank Paul Temple for his early comments on a draft of this article.

Sample Method

Reference Random [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF][START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Valov | Empirical comparison of regression methods for variability-aware performance prediction[END_REF][START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF][START_REF] Zhang | Performance prediction of configurable software systems by fourier learning (t)[END_REF][START_REF] Acher | Varylatex: Learning paper variants that meet constraints[END_REF][START_REF] Jamshidi | Machine learning meets quantitative planning: Enabling self-adaptation in autonomous robots[END_REF][START_REF] Temple | Towards adversarial configurations for software product lines[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF][START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF][START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF][START_REF] Siegmund | Attributed variability models: outside the comfort zone[END_REF][START_REF] Alipourfard | Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics[END_REF][START_REF] Siegmund | Measuring non-functional properties in software product line for product derivation[END_REF][START_REF] Zhang | A mathematical model of performance-relevant feature interactions[END_REF][START_REF] Thornton | Auto-weka: Combined selection and hyperparameter optimization of classification algorithms[END_REF][START_REF] Švogor | An extensible framework for software configuration optimization on heterogeneous computing systems: Time and energy case study[END_REF][START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF][START_REF] Chen | Boosting the performance of computing systems through adaptive configuration tuning[END_REF], 101] Knowledge-wise heuristic [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF][START_REF] Siegmund | Attributed variability models: outside the comfort zone[END_REF][START_REF] Siegmund | Measuring non-functional properties in software product line for product derivation[END_REF], 99] Feature-coverage heuristic [START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF][START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Siegmund | Predicting performance via automated featureinteraction detection[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF][START_REF] Yilmaz | Reducing masking effects in combinatorialinteraction testing: A feedback drivenadaptive approach[END_REF][START_REF] Yilmaz | Covering arrays for efficient fault characterization in complex configuration spaces[END_REF][START_REF] Lillack | Improved prediction of non-functional properties in software product lines with domain context[END_REF][START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF]] Feature-frequency heuristic [START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Siegmund | Predicting performance via automated featureinteraction detection[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Lillack | Improved prediction of non-functional properties in software product lines with domain context[END_REF][START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF][START_REF] Etxeberria | Performance-based selection of software and hardware features under parameter uncertainty[END_REF] Family-based simulation [START_REF] Siegmund | Family-based performance measurement[END_REF] Multi-start local search [START_REF] Hutter | Sequential model-based optimization for general algorithm configuration[END_REF] Plackett-Burman design [START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF][START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF] central composite design [START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF] D-optimal design [START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF] Breakdown [START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF] Sequence type trees [START_REF] Krismayer | Mining constraints for event-based monitoring in systems of systems[END_REF] East-west sampling [START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF] Exemplar sampling [START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF] Constrained-driven sampling [START_REF] Gargantini | Combinatorial interaction testing for automated constraint repair[END_REF] Diameter uncertainty strategy [START_REF] Zuluaga | ε-pal: an active learning approach to the multi-objective optimization problem[END_REF] Historical configurations [START_REF] Van Aken | Automatic database management system tuning through large-scale machine learning[END_REF][START_REF] Saleem | Personalized decision-strategy based web service selection using a learning-to-rank algorithm[END_REF] Latin hypercube sampling [START_REF] Xi | A smart hill-climbing algorithm for application server configuration[END_REF][START_REF] Jamshidi | An uncertainty-aware approach to optimal configuration of stream processing systems[END_REF][START_REF] Bao | Autoconfig: automatic configuration tuning for distributed message systems[END_REF] Neighborhood sampling [START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF][START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF] Input-based clustering [START_REF] Ding | Autotuning algorithmic choice for input sensitivity[END_REF] Distance-based sampling [START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF][START_REF] Ghamizi | Automated search for configurations of deep neural network architectures[END_REF] Genetic sampling [START_REF] Martinez | Towards estimating and predicting user perception on software product variants[END_REF][START_REF] Jamshidi | An uncertainty-aware approach to optimal configuration of stream processing systems[END_REF][START_REF] Safdar | Mining cross product line rules with multiobjective search and machine learning[END_REF]] Interaction tree discovery [START_REF] Song | itree: efficiently discovering high-coverage configurations using interaction trees[END_REF] Arbitrarily chosen [START_REF] Chen | Performance prediction of component-based applications[END_REF][START_REF] Murwantara | Measuring energy consumption for web service product configuration[END_REF][START_REF] Sincero | Approaching non-functional properties of software product lines: Learning from products[END_REF][START_REF] Samreen | Daleel: Simplifying cloud instance selection using machine learning[END_REF][START_REF] Amand | Towards learning-aided configuration in 3d printing: Feasibility study and application to defect prediction[END_REF][START_REF] Queiroz | Towards predicting feature defects in software product lines[END_REF][START_REF] Duarte | Learning nondeterministic impact models for adaptation[END_REF][START_REF] Chen | Boosting the performance of computing systems through adaptive configuration tuning[END_REF] Appendix A. Sampling Methods Table A. [START_REF] Cashman | Navigating the maze: the impact of configurability in bioinformatics software[END_REF] shows the sample methods adopted by each study. The first column is about the method and the second column identifies the study reference(s). There are 23 high-level sample methods documented in the literature. Next, we describe the particularities of the most used sample methods.

Appendix B. Learning Techniques

Table B.7 sketches which learning techniques are supported in the literature and what application objective they address. The first column identifies the study reference(s). The second and third columns identify the name of the learning technique and its application objective, respectively. (Notice that the application objective is related to the scenarios in which each learning technique has been already used in the literature; it means some learning techniques could well be applied for other scenarios in the future). Fourier Learning of Boolean Functions A1 [START_REF] Kolesnikov | On the relation of external and internal feature interactions: A case study[END_REF] Frequent Item Set Mining A1 [START_REF] Siegmund | Family-based performance measurement[END_REF] Graph Family-Based Variant Simulator A1 [START_REF] Couto | Products go green: Worst-case energy consumption in software product lines[END_REF] Implicit Path Enumeration Technique (IPET)

A1

[79] Naive Bayes A1 [START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF][START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF][START_REF] Kolesnikov | On the relation of external and internal feature interactions: A case study[END_REF][START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF][START_REF] Jamshidi | Machine learning meets quantitative planning: Enabling self-adaptation in autonomous robots[END_REF][START_REF] Siegmund | Measuring non-functional properties in software product line for product derivation[END_REF] Step-Wise Multiple Linear Regression A1, A2, A3, A4 [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF][START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Murwantara | Measuring energy consumption for web service product configuration[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Valov | Empirical comparison of regression methods for variability-aware performance prediction[END_REF][START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF][START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF][START_REF] Acher | Varylatex: Learning paper variants that meet constraints[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF][START_REF] Yilmaz | Covering arrays for efficient fault characterization in complex configuration spaces[END_REF][START_REF] Krismayer | Mining constraints for event-based monitoring in systems of systems[END_REF][START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF][START_REF] Zheng | Automatic configuration of internet services[END_REF][START_REF] Zhang | A mathematical model of performance-relevant feature interactions[END_REF][START_REF] Song | itree: efficiently discovering high-coverage configurations using interaction trees[END_REF] Classification and Regression Trees (CART) A1, A2, A3, A4, A5

[43] Kernel Density Estimation and NSGA-II A1, A2, A3, A4, A5, A6 [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Predicting performance via automated featureinteraction detection[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF] Feature's Influence Delta A1, A3 [START_REF] Alipourfard | Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics[END_REF][START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF][START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Van Aken | Automatic database management system tuning through large-scale machine learning[END_REF][START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF][START_REF] Jamshidi | An uncertainty-aware approach to optimal configuration of stream processing systems[END_REF][START_REF] Zuluaga | ε-pal: an active learning approach to the multi-objective optimization problem[END_REF] Gaussian Process Model A1, A3, A4 [START_REF] Murwantara | Measuring energy consumption for web service product configuration[END_REF][START_REF] Samreen | Daleel: Simplifying cloud instance selection using machine learning[END_REF][START_REF] Chen | Performance prediction of component-based applications[END_REF][START_REF] Lillack | Improved prediction of non-functional properties in software product lines with domain context[END_REF] Linear Regression A1, A3, A4 [START_REF] Valov | Empirical comparison of regression methods for variability-aware performance prediction[END_REF][START_REF] Amand | Towards learning-aided configuration in 3d printing: Feasibility study and application to defect prediction[END_REF][START_REF] Queiroz | Towards predicting feature defects in software product lines[END_REF][START_REF] Bao | Autoconfig: automatic configuration tuning for distributed message systems[END_REF][START_REF] Thornton | Auto-weka: Combined selection and hyperparameter optimization of classification algorithms[END_REF] Random Forest A1, A3, A5 [START_REF] Amand | Towards learning-aided configuration in 3d printing: Feasibility study and application to defect prediction[END_REF][START_REF] Temple | Towards adversarial configurations for software product lines[END_REF][START_REF] Valov | Empirical comparison of regression methods for variability-aware performance prediction[END_REF][START_REF] Afia | Performance prediction using support vector machine for the configuration of optimization algorithms[END_REF] Support Vector Machine A1, A3, A5 [START_REF] Amand | Towards learning-aided configuration in 3d printing: Feasibility study and application to defect prediction[END_REF][START_REF] Queiroz | Towards predicting feature defects in software product lines[END_REF][START_REF] Song | itree: efficiently discovering high-coverage configurations using interaction trees[END_REF] C4.5 (J48) A1, A5 [START_REF] Sincero | Approaching non-functional properties of software product lines: Learning from products[END_REF][START_REF] Etxeberria | Performance-based selection of software and hardware features under parameter uncertainty[END_REF] Covariance Analysis A2 [START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF] Multinomial Logistic Regression A2

Bagging Ensembles of CART, Bagging Ensembles of MLPs Thingiverse's 3D printer defects [START_REF] Xi | A smart hill-climbing algorithm for application server configuration[END_REF] IBM WebSphere Application server throughput [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF] Clasp ASP solver response time [START_REF] Zuluaga | ε-pal: an active learning approach to the multi-objective optimization problem[END_REF] SNW Asset management area and throughput [START_REF] Ding | Autotuning algorithmic choice for input sensitivity[END_REF] Binpacking Binpacking algorithm execution time and accuracy [START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF] XGBoost Boosting algorithms training time [START_REF] Sharifloo | Learning and evolution in dynamic software product lines[END_REF] SaaS system Cloud computing response time [START_REF] Ding | Autotuning algorithmic choice for input sensitivity[END_REF] Clustering Clustering algorithm execution time and accuracy [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF] Actian Vector Database system runtime [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF] Apache Cassandra Database system latency [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF][START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF][START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Siegmund | Predicting performance via automated featureinteraction detection[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF][START_REF] Zhang | Performance prediction of configurable software systems by fourier learning (t)[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF][START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF][START_REF] Siegmund | Measuring non-functional properties in software product line for product derivation[END_REF][START_REF] Zhang | A mathematical model of performance-relevant feature interactions[END_REF] Berkeley DB Database system I/O time, memory footprint, performance, response time, code complexity, maintainability, binary size [START_REF] Siegmund | Measuring non-functional properties in software product line for product derivation[END_REF] FAME-DBMS Database system maintainability, binary size, performance [START_REF] Yilmaz | Reducing masking effects in combinatorialinteraction testing: A feedback drivenadaptive approach[END_REF][START_REF] Van Aken | Automatic database management system tuning through large-scale machine learning[END_REF][START_REF] Zheng | Automatic configuration of internet services[END_REF][START_REF] Song | itree: efficiently discovering high-coverage configurations using interaction trees[END_REF] MySQL Database system defects, throughput, latency [START_REF] Van Aken | Automatic database management system tuning through large-scale machine learning[END_REF] Postgres Database system throughput, latency [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF] Prevayler Database system memory footprint, performance [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF][START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Siegmund | Predicting performance via automated featureinteraction detection[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF][START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF][START_REF] Kolesnikov | On the relation of external and internal feature interactions: A case study[END_REF]

SQLite

Database system memory footprint, performance, response time, code complexity, runtime [START_REF] Chen | Performance prediction of component-based applications[END_REF] StockOnline Database system response time [START_REF] Bao | Autoconfig: automatic configuration tuning for distributed message systems[END_REF] Kafka Distributed systems throughput [START_REF] Ghamizi | Automated search for configurations of deep neural network architectures[END_REF] DNN DNNs algorithms accuracy of predictions [START_REF] Acher | Varylatex: Learning paper variants that meet constraints[END_REF] Curriculum vitae Document number of pages [START_REF] Acher | Varylatex: Learning paper variants that meet constraints[END_REF] Paper Document number of pages [START_REF] Duarte | Learning nondeterministic impact models for adaptation[END_REF] RUBiS E-commerce response time [START_REF] Siegmund | Family-based performance measurement[END_REF] EMAIL E-mail client time [START_REF] Kolesnikov | On the relation of external and internal feature interactions: A case study[END_REF] MBED TLS Encryption library response time [START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF] SAP ERP Enterprise Application response time [START_REF] Nair | Finding faster configurations using flash[END_REF] noc-CM-log FPGA CPU power consumption, runtime [START_REF] Nair | Finding faster configurations using flash[END_REF] sort-256 FPGA area, throughput [START_REF] Etxeberria | Performance-based selection of software and hardware features under parameter uncertainty[END_REF] E-Health System Health response time [START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF] HIP A cc Image processing response time [START_REF] Couto | Products go green: Worst-case energy consumption in software product lines[END_REF] Disparity SPL Image processing energy consumption [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF] PKJab Instant messenger memory footprint, performance [START_REF] Hutter | Sequential model-based optimization for general algorithm configuration[END_REF] IBM ILOG Integer solver runtime [START_REF] Westermann | Automated inference of goaloriented performance prediction functions[END_REF] SPECjjbb2005 Java Server response time, throughput [START_REF] Thornton | Auto-weka: Combined selection and hyperparameter optimization of classification algorithms[END_REF] WEKA Learning algorithm accuracy of predictions [START_REF] Ding | Autotuning algorithmic choice for input sensitivity[END_REF] SVD Linear algebra execution time and accuracy [START_REF] Nair | Finding faster configurations using flash[END_REF] Trimesh Mesh solver iterations, response time [START_REF] Siegmund | Family-based performance measurement[END_REF] MBENCH Micro benchmark time [START_REF] Yilmaz | Covering arrays for efficient fault characterization in complex configuration spaces[END_REF][START_REF] Porter | Skoll: A process and infrastructure for distributed continuous quality assurance[END_REF] ACE+TAO system Middleware software defects [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF] SensorNetwork Network simulator memory footprint, performance [START_REF] Weckesser | Optimal reconfiguration of dynamic software product lines based on performanceinfluence models[END_REF] Simonstrator Network simulator latency [START_REF] Zuluaga | ε-pal: an active learning approach to the multi-objective optimization problem[END_REF] NoC Network-based system energy and runtime [START_REF] Ding | Autotuning algorithmic choice for input sensitivity[END_REF] Helmholtz 3D Numerical analysis execution time and accuracy [START_REF] Ding | Autotuning algorithmic choice for input sensitivity[END_REF] Poisson 2D Numerical analysis execution time and accuracy [START_REF] Sincero | Approaching non-functional properties of software product lines: Learning from products[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF] Linux kernel Operating system memory footprint, performance [START_REF] Jamshidi | Learning to sample: exploiting similarities across environments to learn performance models for configurable systems[END_REF] DNN Optimization algorithm response time [START_REF] Martinez | Towards estimating and predicting user perception on software product variants[END_REF] Art system Paint user likeability [START_REF] Grebhahn | Performanceinfluence models of multigrid methods: A case study on triangular grids[END_REF] Multigrid system Equations solving time of each interaction [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF] CoBot System Robotic system CPU usage [START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF] JavaGC Runtime environment response time [START_REF] Švogor | An extensible framework for software configuration optimization on heterogeneous computing systems: Time and energy case study[END_REF] Robot Runtime environment energy consumption and execution time [START_REF] Afia | Performance prediction using support vector machine for the configuration of optimization algorithms[END_REF][START_REF] Xu | Satzilla: portfolio-based algorithm selection for sat[END_REF] Hand SAT solver runtime [START_REF] Afia | Performance prediction using support vector machine for the configuration of optimization algorithms[END_REF][START_REF] Xu | Satzilla: portfolio-based algorithm selection for sat[END_REF] Indu SAT solver runtime [START_REF] Afia | Performance prediction using support vector machine for the configuration of optimization algorithms[END_REF][START_REF] Xu | Satzilla: portfolio-based algorithm selection for sat[END_REF] Rand SAT solver runtime [ VARD on EC2 Text manager response time [START_REF] Gargantini | Combinatorial interaction testing for automated constraint repair[END_REF] Aircraft Toy example defects [START_REF] Siegmund | Family-based performance measurement[END_REF] ELEVATOR Toy example time [START_REF] Gargantini | Combinatorial interaction testing for automated constraint repair[END_REF] WashingMachine Toy example defects [START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Spl conqueror: Toward optimization of non-functional properties in software product lines[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF] Violet UML editor memory footprint, performance [START_REF] Temple | Using Machine Learning to Infer Constraints for Product Lines[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Temple | Towards adversarial configurations for software product lines[END_REF] MOTIV Video encoder video quality [START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF] VP9 Video encoder encoding time [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF][START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF][START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Siegmund | Predicting performance via automated featureinteraction detection[END_REF][START_REF] Siegmund | Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption[END_REF][START_REF] Nair | Finding faster configurations using flash[END_REF][START_REF] Jamshidi | Transfer learning for performance modeling of configurable systems: an exploratory analysis[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF][START_REF] Zhang | Performance prediction of configurable software systems by fourier learning (t)[END_REF][START_REF] Valov | Transferring performance prediction models across different hardware platforms[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF][START_REF] Kaltenecker | Distance-based sampling of software configuration spaces[END_REF][START_REF] Zhang | A mathematical model of performance-relevant feature interactions[END_REF] x264

Video encoder CPU power consumption, encoding time, Peak Signal to Noise Ratio, response time, code complexity, video quality, performance [START_REF] Temple | Learning contextual-variability models[END_REF] OpenCV Video tracking performance [START_REF] Safdar | Mining cross product line rules with multiobjective search and machine learning[END_REF] C60 and MX300 Virtual environment defects [START_REF] Alipourfard | Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics[END_REF] Amazon EC2 Web coud service performance [START_REF] Guo | Variability-aware performance prediction: A statistical learning approach[END_REF][START_REF] Guo | Data-efficient performance learning for configurable systems[END_REF][START_REF] Oh | Finding near-optimal configurations in product lines by random sampling[END_REF][START_REF] Kolesnikov | Tradeoffs in modeling performance of highly configurable software systems[END_REF][START_REF] Nair | Using bad learners to find good configurations[END_REF][START_REF] Sarkar | Cost-efficient sampling for performance prediction of configurable systems (t)[END_REF][START_REF] Siegmund | Predicting performance via automated featureinteraction detection[END_REF][START_REF] Temple | Learning contextual-variability models[END_REF][START_REF] Siegmund | Performance-influence models for highly configurable systems[END_REF][START_REF] Nair | Faster discovery of faster system configurations with spectral learning[END_REF][START_REF] Yilmaz | Reducing masking effects in combinatorialinteraction testing: A feedback drivenadaptive approach[END_REF][START_REF] Zheng | Automatic configuration of internet services[END_REF]

Apache

Web server response rate, response time, workload, defects, throughput [START_REF] Osogami | Optimizing system configurations quickly by guessing at the performance[END_REF] Stock Brokerage Web system throughput, response time [START_REF] Song | itree: efficiently discovering high-coverage configurations using interaction trees[END_REF] vsftpd FTP daemon defects [START_REF] Song | itree: efficiently discovering high-coverage configurations using interaction trees[END_REF] ngIRCd IRC daemon defects