Maria Laura

Brzezinski Meyer

Hélène Waeselynck
email: helene.waeselynck@laas.fr

Fernand Cuesta
email: fernand.cuesta@renault.com

A Case Study on the "Jungle" Search for Industry-Relevant Regression Testing

Keywords: Regression testing, taxonomy, industrial case study

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Regression testing (RT) verifies that the changes in the code did not introduce new issues, that is, the software did not regress. Since re-runing all tests is usually not practical, there is a need for some test optimization. This is called the RT problem and has been an active field of research for decades. The methods that have emerged can be divided into three categories [START_REF] Yoo | Regression testing minimization, selection and prioritization: A survey[END_REF]: Test Suite Minimization (TSM), Test Case Selection (TCS) and Regression Test Prioritization (RTP). TSM aims to reduce the number of tests by removing redundant ones. TCS addresses the dual problem of choosing which tests to run. The selection is often related to the code changes: tests that cover the modified parts of the code are selected. RTP neither removes nor selects tests, but assigns them an order of execution seeking to maximize early fault exposure. An RT strategy may hybridize TSM, TCS and RTP. For example, a minimization technique can be applied after the selection to further reduce the number of tests. Or all tests are prioritized but only the highest-priority ones are selected. Conceptually, the three categories of methods are closely related. All of them involve some comparative characterization of the tests, based on which an optimal decision has to be taken (which tests to remove, retain or execute first). In the article titled "On the search for industry-relevant regression testing research" [START_REF] Ali | On the search for industry-relevant regression testing research[END_REF], Ali et al. noted that, de-spite the large body of work on the RT problem, there is limited dissemination in industry. They identified two main difficulties faced by practitioners. The first one comes from the discrepancies in terminology between academia and industry: it makes it harder for practitioners to search for methods in the literature. The second one is the difficulty to assess the relevance of the methods. The research is typically done in a controlled environment, which differs from the complexity of an industrial scenario. In order to help practitioners to interpret, compare and contrast the methods in the literature, Ali et al. [START_REF] Ali | On the search for industry-relevant regression testing research[END_REF] created a taxonomy. When both the RT problem and existing solutions are mapped onto the taxonomy, practitioners may more easily determine which solutions are best aligned with their problem. The taxonomy has three parts to capture the practitioners' concerns: the industrial context, the desired effects of RT methods and the input information they need (e.g., code changes, ...). This paper presents an application and an evaluation of the taxonomy to assist in a real RT problem. The context is the development of R&D projects at Renault for connected vehicles. We used the taxonomy to extract a set of 8 promising methods out of 52 papers. The methods are currently being experimented at Renault. We report on the beneficial aspects but also the difficulties of using the taxonomy to identify the applicable and relevant RT research. The paper is structured as follows. Section 2 introduces the background of this work. Section 3 describes the methodology we used to apply and assess the taxonomy. The industrial RT problem is first characterized by its context and desired effects (Section 4). Then, Section 5 analyzes the feasibility of the methods from the literature, based on the information items they require. Section 6 analyzes how the desired effects can be measured in the industrial context. Section 7 discusses the threats to validity and Section 8 concludes the paper.

BACKGROUND AND RELATED WORK

Since the early papers in the 90s, regression testing has been widely studied. Many surveys and systematic reviews have been published to keep pace with the evolving research in the area. In [START_REF] Ali | On the search for industry-relevant regression testing research[END_REF], the authors identified eleven reviews of software RT literature between 2010 and 2017. New ones have been published since that time, e.g., [START_REF] Khatibsyarbini | Test case prioritization approaches in regression testing: A systematic literature review[END_REF] [4] [START_REF] Pan | Test case selection and prioritization using machine learning: a systematic literature review[END_REF]. The existing RT methods are diverse. They differ in the (combination of) criteria to compare the tests: code coverage, code change coverage, criticality of tested requirements, similarity of tests, cost, historical effectiveness, ... The core RT algorithms are also diverse, including predefined strategies, strategies optimized by a metaheuristic search, as well as strategies based on machine learning. With so many methods, it is difficult to know which ones to select and apply in a specific context. This is why Ali et al. [START_REF] Ali | On the search for industry-relevant regression testing research[END_REF] introduced a taxonomy, allowing an alignment between methods proposed in the literature and real industrial contexts. The taxonomy focuses on industrial relevance and applicability. It is divided into three parts: context, effect and information. The purpose of the context part is to identify the characteristics of an industrial environment that make regression testing challenging. Three context factors are identified: system-related (size, complexity and type), processrelated (testing process and test techniques) and people-related (organizational factors). The second part of the taxonomy is the desired effect of a method, that is, the measurable improvement reported with its implementation. The effects are divided into three categories: test coverage related, test efficiency and effectiveness, and awareness. Together, the context and desired effects characterize an RT problem. Then, the applicability of candidate solutions is determined by the third part of the taxonomy. The solutions are characterized by the information they utilize, like: development artifacts (from requirements to binary code), information about test cases, test execution attributes, test reports and issues revealed by tests. If some information item is not available in the industrial context, then the method is not feasible in this context. The information part of the taxonomy is the most detailed one, representing 50 out of a total of 68 attributes for all parts (see these 50 attributes in Table 5). The building of the taxonomy followed a principled approach, starting from a literature review and involving practitioners at several steps. The authors also demonstrated the mapping of 38 RT papers to the taxonomy. However, they did not demonstrate the application of their taxonomy to address RT problems in real-world scenarios. This paper provides such a real-world scenario to study the use of the taxonomy.

METHODOLOGY

The study takes the perspective of an industrial company wishing to improve its RT process. From a set of research papers, the company would like to know which methods to select for an experimentation in its context. The general research question is whether the taxonomy is useful to aid in this selection and in the preparation of experiments. The study considers a set of 52 candidate papers. We first took the 38 RT papers already mapped to the taxonomy. They are displayed in Table 1, where we retain the same ID as in [START_REF] Ali | On the search for industry-relevant regression testing research[END_REF] (from S1 to S38). These papers describe methods that were identified as industry-relevant by the authors of the taxonomy and their industrial partners. But none of these papers was published after 2016, and their set does not reflect the growing interest on machine learning (ML)-based methods. So we felt the need to include additional papers. We considered a recent literature review on ML-based RT research [START_REF] Pan | Test case selection and prioritization using machine learning: a systematic literature review[END_REF]. The review classifies 29 papers depending on whether they use supervised learning (SL), unsupervised learning (UL), reinforcement learning (RL), natural language processing (NLP) or a mix of them. We extracted a subset of 12 papers such that each learning technique is covered, as well as each mix of techniques. We also included two extra papers [6] [7], not cited in the review, but which we had previously identified as ML-based work with an industrial focus. Table 2 lists the resulting set of added papers (L1, ... L14). We had to map them to the taxonomy, and did so with a double check by different authors.

The real-world scenario is a Renault project under development. We had full access to its data. We also interacted with the engineers working on the project all along the study. The search for relevant methods followed three steps, shown in Figure 1. Each step leverages the taxonomy for a different purpose, in accordance with the role of the context, effect and information described in [START_REF] Ali | On the search for industry-relevant regression testing research[END_REF].

Step 1 characterizes the industrial RT problem in terms of a context and desired effects.

Step 2 identifies the feasible methods from the set of papers: the utilized information must correspond to information items available in the project.

Step 3 keeps the feasible methods that address the desired effects, and determines how to measure these effects in the industrial context. Each step provides feedback on the taxonomy. Hence, the general question about the utility of the taxonomy is actually refined into three research questions, one for each step.

• RQ1: Does the taxonomy help in describing a specific industrial RT problem?

• RQ2: Does the taxonomy help in identifying the feasible solutions in a specific industrial context?

• RQ3: Does the taxonomy help in finding ways to assess the solutions in a specific industrial context?

RT PROBLEM CHARACTERIZATION

Step 1 of the study characterizes the RT problem by an industrial context and a set of desired improvements. This section presents them. We then provide feedback about the use of the taxonomy for exposing the important characteristics of the problem.

Description of the context

The case study involves an R&D automotive project called Project A in the rest of the paper. Figure 2 provides a highlevel view of its architecture. It is multi-system and composed of several Electronic Control Units (ECUs) connected by a CAN bus (represented by purple lines in Figure 2). The two main components are the infotainment system and the communication one. The infotainment constitutes the main point of contact with the driver and passenger thanks to a screen and connected equipment. This system is the vehicle interface for navigation, sound, image, communication and user settings of the car. The other system is in charge of the communication between on-board systems and external services. A cloud platform is used to store vehicle information and to communicate with other external systems. An example of external system considered in Project A is a smartphone application to lock/unlock the doors of the car. The case study focuses on the integration tests of the on-board software. Table 3 summarizes the context factors based on the taxonomy. Those factors are high-level. The user is expected to put a short note why a given factor is challenging (e.g., "multi-language" put for the complexity factor). We provide below a more extensive description. Each system is multi-language. The infotainment is mainly in C, C++ and Java, while the communication is mainly in C. Their development involves different companies. There is one supplier for infotainment, one for communication and Renault integrates both parts. We consider the tests done at Renault. They run on a test bench that reproduces the on-board architecture and manages the access to the external services.

The binary code of the software under test (SUT) is flashed on the bench to update the ECUs. The size of the flashed code is in the scale of 6 GB. The integration team first tests the infotainment and communication separately, and then their integration. A new binary code version is received weekly from the suppliers. There are two types of tests: automatic and manual. Automatic tests run during the night and manual ones during the working hours. All tests have a priority (P 1, P 2, or P 3) determined by the importance of the functional feature they check. When a Fail is found, the tester conducts an analysis to determine the origin of the problem: infotainment, communication or test automation issue. Accordingly, the issue report is passed to product or test automation managers, who assign a criticality and priority to the issue. The criticality is defined by the impact that the bug has on the client, with K1 corresponding to the strongest impact, and K4 to the weakest one. The priority determines the order in which the suppliers or test automation team should fix the issues, using four levels of priority.

Test automation issues represent about 30% of the Fails of automatic tests. This is not an unusual situation, and reflects the complexity of the test environment for the embedded software. Other authors have studied automotive projects using hardware-in-the-loop test benches [START_REF] Jordan | Unreliable test infrastructures in automotive testing setups[END_REF]. They reported from 74% up to 91% of Fails due to automation issues (unreliable test infrastructures, incorrect tests). In Project A, the test automation calls for a dedicated development and evolution process. All automatic tests are implemented using Robot Framework 1 , following a keyword-driven approach. The main script of a test is concise, based on abstract keywords, but relies on many lower-level resources (keyword libraries, code libraries, configuration files). All in all, the automation code of a test typically represent tens of thousands lines of code (> 10 KLOC), making it quite complex. A test is made available to the integration team only when it has demonstrated a stable correct behavior. If the testing of the SUT reveals an unstable behavior or any other automation issue, then the test becomes unavailable until it is repaired by the test automation team.

To ensure a stable behavior, the test design puts emphasis on making each test a self-contained and standalone unit. Each test is responsible for its complete set-up and clean-up, including hardware reboots, the establishment of a connection to the cloud, and so on. In this way, a test starts and ends in a known state, and has no side effect on other tests.

Manual tests are also defined as standalone units. The tester is free to perform them in any order that matches the priority (P 1 to P 3, as previously explained). In practice, the tester may happen to merge two tests of the same priority that require a similar test bench configuration, in order to save the set-up time. However, this is not always a good practice. A test may need a known initial state like a "fresh boot". If the test is appended to another one without the reboot, its behavior is less predictable. Spurious failures may occur. It is safer to systematically do the set-up specified for the test.

The set-up and cleaning actions are important to achieve stable tests, but are time-consuming. Due to them, the duration of each test -automatic or manual -is typically in tens of minutes (from 20 to 50 minutes). Running all tests would take 7 nights and 9 working days, which would not fit into a week. This time does not account for the analysis of the tests, which must also be done during the week. The practice is to have 2 nights and a half for running the automatic tests, and about 2 days and a half for the manual ones. In the latter case, this is an estimate, because the execution time of manual tests cannot be precisely measured, nor clearly separated from the analysis time. The manual tester immediately performs the analysis when observing a Fail, before going to the next test.

The number of executed tests thus depends on the Fails of the week. The engineers estimate that, on average, 50% of the working time is spent on running the manual tests and 50% on analyzing them. At the beginning of each week, the integration teams decides which tests will be executed. For manual tests, it is not uncommon that the low-priority ones end up not being executed, due to lack of time. Moreover, some tests tend to be rarely selected. As a result, some tests are not executed for months.

Desired Effects

The taxonomy proposes three broad categories of improvements: in test coverage, in efficiency and effectiveness, and in 1 https://robotframework.org/ increasing awareness. For the engineers of Project A, the focus is on efficiency and effectiveness. For this, the taxonomy has several refined objectives shown in Table 4. We discuss below the ones that are the most desirable for Project A.

Following discussions with engineers, the two main objectives are (i) to decrease the time to find faults and (ii) to avoid tests that are never or rarely executed. The sooner the faults are found, the earlier the analysis of these bugs and the faster feedback to the system suppliers and test automation team. Moreover, tests should not be left behind for a long period of time. Otherwise, regression problems may remain unnoticed until late in the project. All in all, the test selection and prioritization could be more efficient than it is currently: it would ideally execute the failing tests first, at the very beginning of the week, and also ensure a circulation of the tests across weeks, none of them being forgotten. In terms of the taxonomy, the decreased time for fault detection clearly corresponds to the first objective, while the circulation of the tests is not explicitly mentioned.

Some supplementary objectives are desired but assigned less importance in the short term. For both manual and automatic tests, a reduced testing time would make the bench available for other projects. Ideally, the usage time of the bench could go from two and a half days to just one day for manual tests, and from two and a half nights to one night for automatic tests. However, the mere reduction in time would not be enough. It would be imperative to maintain the fault detection capability.

If such is not the case, the reduced testing time should at least maintain, if not improve, the severe fault detection capability.

As mentioned in the description of the context, each issue has a criticality. So, it may be acceptable to find less faults if the most critical ones are found.

To sum up, the short-term objectives revolve around a more efficient use of the currently available testing time, while longer-term objectives would consider reducing this time. Note that the solutions might be different for manual and automatic tests. For example, the decreased time for fault detection does not have the same meaning if the analysis is done on the fly (manual tests) or after the nightly execution (automatic tests).

For automatic tests, a fine-grained ordering may not be needed, provided that the failing tests occur in the first night.

Feedback on the taxonomy (RQ1)

The most helpful part of the taxonomy concerned the desired effects, which allowed the identification and prioritization of objectives. The context factors were deemed too high level to fully characterize an industrial problem. However, they were found relevant to guide the discussions with the engineers. The discussion of desired effects revealed a missing one: the circulation of the tests. The taxonomy does not include it. Still, at least one paper analyzed by the authors of the taxonomy proposes a method that ensures the circulation of the tests (S26).

One of the ML-based papers we analyzed (L9) also do. But none of them lets this appear as a measured effect. According to the methodology used to build the taxonomy, the circulation of the tests would then not qualify as a desired effect because "an effect has to be demonstrated as a measurement". We could find another paper in the literature that mentions the circulation of tests, and that explicitly demonstrates this effect (along with other ones) [START_REF] Strandberg | Automated system-level regression test prioritization in a nutshell[END_REF]. We believe that forgotten tests are a recurring problem in many industrial settings, and propose to add their avoidance in the taxonomy. Section 6 will further discuss the evaluation of this effect.

RQ1 (aid in describing the problem):

The context part is too high-level, but the effect part proved very helpful. A missing effect is the circulation of the tests.

ANALYSIS OF RT METHODS FEASIBILITY

Step 2 of the study explores the feasibility of the methods proposed in the set of 52 papers. A method is considered as technically feasible if it is based solely on information items that are available in the industrial context. First, a taxonomybased analysis is done, which uses the items mentioned in the taxonomy. Both the papers and Project A are mapped to the taxonomy, allowing a comparison of the required and available items. It yields a classification into feasible and infeasible methods. Then, we assess the outcomes of the previous analysis by browsing through the papers, and by manually determining whether each method is really (in)feasible in the context. The section ends by feedback on the use of the taxonomy to analyze feasibility.

Taxonomy-based analysis of feasibility

Table 5 shows the mapping of Project A and papers to the taxonomy. The first two columns display the information items of the taxonomy. The third one indicates which items are available for Project A. It can be compared with the fourth column that lists the papers requiring an item. For example, looking at the first item: the changes in requirements are not tracked in Project A, so that the method described in S21 is not feasible. For Project A, a star is marked (X*) if the information is indirectly available: it may be derived from other available data, or replaced by an estimate. For papers Si, the table simply reproduces the mapping done by the taxonomy authors. For Project A and the papers Li, the mapping is ours. Obviously, it depends on our understanding of the taxonomy. This is worth mentioning since we experienced difficulties in interpreting some of the items.

1) Test execution time appearing at different places. In [START_REF] Ali | On the search for industry-relevant regression testing research[END_REF], the authors provided the following explanation: as an attribute of a test case it is an estimation; as an attribute of a test execution, it is measured at runtime; and as an attribute of the test reports, it is further recorded and maintained. However, it was unclear to us why and how to distinguish these cases.

We had a look at papers exemplifying them. For instance, the mapping of S29 has the execution time for both test cases and test executions. But the paper merely mentions an option to take time into consideration. There is no detail on time estimation, measurement or recording. As another example, we could not understand why time in S32 (test executions) does not have the same mapping as in S3 and S15 (test reports).

We finally took the following mapping decisions. For project A, we considered all three execution time items as available.

Strictly speaking, the measured execution time (in test executions or reports) is available for automatic tests only. We marked the information as indirectly available due to manual tests, for which we only have an estimate. For the mapping of papers Li, we decided not to delve into considerations of estimated, measured or recorded values. If a paper used the duration of tests, we mapped it to test case/execution time, and considered the information as available for Project A.

2) Items referring to faults, failures or issues. We feel that the terminology would have deserved an explanation. Is a failure different from a Fail verdict? Is so, there is no added value of test reports/link to failures compared to test reports/verdicts. For instance, we could not explain the different mappings of papers S4 (verdict only) and S18 (verdict + link to failures): both compute failure rates in a test time window. Paper S4 is additionally mapped to test cases/link to faults, but seems more concerned with failures (their frequency, their severity) than faults (i.e., the investigated causes of failures). We observed that, in many papers, "faults" and "failures" are used interchangeably. In Project A, we have data for both. The investigation of faults is tracked by an issue management system. Not all Fails yield the opening of an issue, and there may be a single issue opened for multiple failing tests. We did the mapping as follows. We ignored the item test reports/link to failures. Rather, we used test reports/verdicts to indicate that a method needs the recording of the fails. Some methods do not consider the raw fails, and need a traceability between the tests and the revealed faults. For them, we decided not to distinguish whether the recording is in the direction from tests to faults (test cases/link to faults), or from faults to tests (issues/link to test case). Indeed, one item can be derived from the other by reversing the links. For papers Li, we chose issues/link to test case to represent any relation between tests and faults. The direction of the link is the most frequent one when using an issue management system. For Project A, we explicitly marked both directions as available. We also marked test cases/fault detection probability as indirectly available: estimates can be derived from historical data. 3) Test session (in test reports). It seems obvious that any RT process involves test sessions. However, only a few papers Si are mapped to this item, and we could not understand what makes them different. We decided to ignore this item.

Subject to our interpretation, the taxonomy-based analysis retains 8 out of the 52 papers. They are shown in the first row of Table 6. They use items like the execution time of tests, their cost, their historical effectiveness (fail verdicts, severity of the fails, issues found by the tests), the tested requirements and their priority. All these items are available in Project A. Most of the eliminated paper require design or code artefacts that Project A does not have, as well as coverage information. A few papers are eliminated due to other causes: they consider variability data in multiple variants and code branches (S26, S32), or need to monitor the number of changes in requirements (S21).

Manual feasibility analysis by reading the papers

To assess the aid provided by the taxonomy, we need some ground truth. For this, we performed a manual labeling. We went all papers, and labeled them as feasible or not in the context of Project A. As shown in Table 6, four papers were misclassified by the taxonomy. There are two false positives (S33, L12) and two false negatives (S3, S26). The classification errors are due to two causes: some papers are not correctly mapped to the taxonomy, and the taxonomy misses important items. While reading the papers, we had several disagreements with the mapping proposed for the papers Si in [START_REF] Ali | On the search for industry-relevant regression testing research[END_REF]. The disagreements are reported in Table 7. They concern as much as 50% of papers Si. Most of the time, the disagreement did not affect the final feasibility label. But it did in three cases: S3, S26 and S33. Paper S3 was considered as the same approach as S4. Both were eliminated due to the use of functionality coverage information. But only S4, which extends S3, uses this information: S3 is actually feasible in the context of Project A. Paper S26 was also wrongly eliminated.

It is supposed to use test cases/target variant. While variability is mentioned in the paper, the authors explain that they could not retrieve the information and present a method that does not use it. The type of tests is also not used. For S33, the error is in the opposite direction. The paper was retained but uses data items that were forgotten in the mapping (functionality coverage and configuration variants), making it unfeasible. The other cause of misclassification is when the taxonomy misses an item that is important for feasibility. Since the taxonomy was built from a literature review, it reflects the set of reviewed papers. As mentioned, this set had few methods based on machine learning. By adding a bunch of papers to represent these methods, we could get new information items. Such was the case for L12. It applies NLP techniques and require a natural language description of test cases. There is no item for this in the taxonomy. The paper was labeled as feasible, but actually Project A lacks the required information.

A similar missing element is the natural language description of issues, required by L14. Project A has this information, however L14 is unfeasible due to another reason (type of test). Although it does not cause any problem for our classification, it may yield a misclassification in other contexts where the information is not available.

Feedback on the taxonomy (RQ2)

The availability of information was found an effective criterion to separate which methods are applicable in a given context. For Project A, only 8 out of the 52 papers fulfill this criterion (ground truth). Having read the papers, we are quite confident that the methods are technically feasible in our context. However, their identification via the taxonomy was hindered by understanding issues and paper classification errors. We had a hard time trying to understand the meaning of some information items. What is the difference between execution time for test executions and for reports? What do the many links between tests and failures (or faults, issues) mean? What is a test session? We ended up reading numerous papers just to determine how to interpret the taxonomy. We missed a user guide explaining the rationale and meaning of the items. While browsing through the papers already aligned to the taxonomy, we had disagreements with as much as 50% of the mappings. They concern grouping papers from the same authors while there are differences in the approaches, forgetting items, or marking items that the authors mention but actually do not use. For Project A, the inaccurate mappings caused three classification errors (S3, S26 misclassified as unfeasible, and S33 misclassified as feasible). The inaccuracies reported in Table 7 could be fixed, but there is no maintained repository to update the mappings for future usage.

Other classification errors were due to missing elements in the taxonomy. They concern natural language artefacts used by some machine learning papers. The incompleteness of the taxonomy is unavoidable, as new approaches are continuously added by researchers. Keeping the taxonomy in line with the state of the art would again require a maintenance effort.

RQ2 (aid in selecting the feasible methods): The information items are difficult to interpret, and several papers are inaccurately mapped to the taxonomy. A documentation and maintenance effort would be needed to make the taxonomy more usable.

EVALUATION OF RT METHODS

Step 3 of the study focuses on the evaluation of the candidate methods. Table 8 shows the alignment between the desired effects (from Step 1) and the ones addressed by the feasible methods. The most desired effects are in bold. The circulation of the tests is not in the taxonomy but added for completeness. All the methods have at least one effect in common with Project A, and are thus potentially relevant. However, their relevance cannot be decided based on published results. They must be experimented in the industrial context. To prepare the implementation of experiments, we must gain closer insights into the measurement of effects. For this, we had a look at all papers (feasible or not) mapped to each desired effect.

Decreased time for fault detection

The decreased time for fault detection is measured by 19 papers. Among the used metrics, one stands out and appears in 12 papers: APFD (Average Percentage of Faults Detected) and its variant NAPFD (Normalized APFD). APFD was first introduced in [START_REF] Rothermel | Test case prioritization: an empirical study[END_REF]. It measures how early an ordered test suite reveals the faults. It requires that the faults are known, the tests that reveal each fault are known, and the full suite is executed.

If the faults and their revealing tests are unknown, the metric may use the raw information of the failing tests. The variant NAPFD [START_REF] Qu | Combinatorial interaction regression testing: A study of test case generation and prioritization[END_REF] addresses cases where not all tests are run and thus only a certain percentage of faults (or fails) are detected compared to the full suite. Note that neither APFD nor NAPFD is well-suited when there are few faults. As an alternative, we found that some authors focus on the first detection only. The metric can be relative to the number of tests (as in L1) or to test execution time (as in L9).

$3)'

2SWLPXP +LVWRULFDO

Figure 3: APFD for optimal and historical orders of manual tests.

In project A, the number of revealed faults is quite different for automatic and manual tests. The nightly automatic tests check mature functionalities that rarely fail. Most the weeks, all executed tests pass. When there are Fails, they are linked to few issues (max 3 in a week).

Hence, an APFD-like metric is not adequate. Moreover, since the test results are analyzed during working hours, the precise ordering of the tests during the night is not so useful. From historical data, the first fault (if any) most often surfaces in the first night. In the cases with 2 or 3 faults, there are detections after the first night. Accordingly, appropriate metrics could be the times to first and last detection with the granularity of nights. A candidate method should not degrade the time to first detection, and should make the last detection occur earlier.

In contrast, the manual tests reveal more faults and a finegrained prioritization is useful. A classical APFD metric is possible. Figure 3 shows its application to historical data. Although a subset of tests were executed each week, NAPFD would not be appropriate to analyse the history of Project A. Indeed, we do not know the percentage of faults compared to the full test suite. Rather, we have to consider that the full test suite of a week is exactly the set of tests that ran during this week, and the faults are only the ones revealed by these tests. Using this interpretation, the historical APFD measures the adequacy of the execution order chosen by the tester, given the selection of the week. In Figure 3, we also give the APFD for an (hypothetical) optimal order of tests, where the revealing ones are always put first. The comparison between the historical and optimal order shows that there is room for improvement. We thus retain APFD for manual tests, and the time to first and last detection for automatic tests (measured with the granularity of a night). The metrics will allow us to evaluate the 7 feasible methods that address the effect (see Table 8).

Circulation of the tests

For the circulation of the tests, the taxonomy does not help. No paper from the set measures this effect, although some address it (S26, L9). Outside the set of papers, we identified relevant work in [START_REF] Strandberg | Automated system-level regression test prioritization in a nutshell[END_REF]: the authors check that no test is forgotten in three months of execution. This suggests a measurement of the worst case time between two executions of a test. We applied this metric to historical data over a period of one year. The worst case is more than 3 months for manual tests, and 6 months for automatic ones. If one considers the time to run the complete suite (respectively 9 working days and 7 nights), there is room for improvement. Each week uses 2.5 days and nights for running the tests, so a complete cycle would last less that one month. This shows that it is feasible to decrease the worst case. The candidate methods are obviously S26 and L9. We also consider augmenting the other methods with a circulation effect. A method will then fill only x% of the test time budget, the remaining (100 -x)% being filled by tests according to their seniority of execution. To allow an evaluation on historical data, we will consider that the tests ran during the first x% must be a subset of the tests that really ran during the project, possibly ordered differently. This will allow us to explore tradeoffs between the circulation of the tests and the decreased time for fault detection supplied by the original methods.

Long-term objectives

The long-term objectives in Project A are to reduce the testing time while maintaining the (severe) fault detection capability.

The testing time per week would ideally go down to just one night for automatic tests and one working day for manual tests. We analyzed the historical data to know whether this time would be sufficient to run the tests that reveal the faults.

For automatic tests, it is possible to catch all faults in one night. However, for manual tests, it is not always possible to reveal all faults in just one day, as there are more faults and thus more tests to run. If just the severe faults are considered --the ones having the impact K1 or K2 (cf. Section 4.1) -, the execution time will be less then one day, so they all could be revealed in the first day. Hence, reaching the objectives may be hard but not impossible. The objective of reducing testing time is mentioned in numerous papers: S5 -S18, S23 -S25, S27, S28, S30 -S32, S35 -S37, L2, L4, L8 and L9. In most of the cases there is a comparison between the execution time with and without the selection, either in absolute numbers or in percentages. For Project A, we do to measure this effect, as the desired time reduction is fixed. The key is rather the measurement of the fault revealing power in the reduced time.

Fault detection capability is measured in S7, S8, S13 -S21, S24 -S26, S28, S29, S34, L2, L6, L11 and L14. The metrics used in these papers can be divided into four types: the absolute number of detected faults, the detection rate per test case, the percentage of detected faults -which is the recall metric -and the combination of recall and precision (i.e. F-Measure). For manual tests, the recall is the most adequate metric, since we would like not to miss faults. For automatic tests, as there are few faults, the recall is less appropriate. We will rather use the absolute number of detected faults. We also checked the papers for severe fault detection capability. In S3 and S4, the APFDc metric is used. Introduced in [START_REF] Elbaum | Incorporating varying test costs and fault severities into test case prioritization[END_REF], this metric considers that test cases do not have the same cost in terms of running time or fault severity. Like APFD, it requires the full suite to be executed, and is thus not relevant to methods aiming at test time reduction. In S21, a new metric is introduced: it computes an average severity of faults detected (ASFD) for each requirement. This is not our objective, so it is not relevant. We eventually decided to use NAPFD and recall for manual tests, restricting the measurements to the set of severe faults. In this way, we evaluate whether the reduced suite (one day) misses any severe fault of the full suite (2.5 days), and whether the most critical revealing tests occur early.

For automatic tests, we will simply use the absolute number of severe faults detected in the first night.

6.4. Feedback on the taxonomy (RQ3)

The taxonomy aided in finding metrics to assess the desired effects. The mapping relevantly pointed to papers that measure these effects. We could gain an overview of alternative metrics, choose the ones we will use and evaluate the margins for improvement in the industrial context. Still, there may be some reservation with respect to the granularity of the taxonomy. The effect part is much less detailed than the information one, which had a fine-grained list of items utilized by the methods.

Similarly, the effects could be explicitly refined into metrics. This would have saved time in the identification of metrics, allowing us to concentrate on the preparation of experiments.

RQ3 (aid in the evaluation of solutions): The taxonomy was helpful to decide how to measure the desired improvements. A refined list of metrics would make the taxonomy even more helpful.

THREATS TO VALIDITY

Concerning the threats to internal validity, we can mention the subjective interpretation of the taxonomy elements. We mapped 14 new papers to the taxonomy. To mitigate the introduction of a personal bias in the process, we doublechecked the mapping. Moreover, we explicitly reported the difficulties we encountered and explained the related mapping decisions. We also double checked the disagreements we had on the mapping of the original set of papers. Another threat is the choice of the articles to represent ML-based methods. We made sure that diverse categories of methods were included (supervised learning, unsupervised learning, reinforcement learning and natural language processing). But, like in the original work to build the taxonomy, there is no claim for completeness. In particular, the information part of the taxonomy may have more missing elements than the ones we identified. A last internal threat concerns the understanding of the industrial RT problem. It was addressed by several meetings as well as direct discussions with engineers. Besides, we could consolidate our understanding by consulting the data of Project A, which was made fully available to us. This case study applies the taxonomy to a specific industrial context, which may represent a threat to external validity. Each practitioner has a very unique RT process. However, the 3-step methodology we adopted is generic and intended to serve in any context. The difficulties we reported also do not seem specific to Project A. Hence, we believe that our experience may benefit other researchers and practitioners.

CONCLUSION

This study contributes to a line of research on how to asses industrial relevance and applicability of RT methods. It presents a real-world scenario in which practitioners search for solutions to their specific problem. The taxonomy proposed in [START_REF] Ali | On the search for industry-relevant regression testing research[END_REF] serves as an alignment between the numerous published methods and the industrial problem.

The taxonomy introduces a decomposition into context, effects and information factors. They capture different aspects to consider, forming a principled base from which we derived a three-step approach to tackle the real world scenario. The first step characterizes the problem independently from any solution. The second one studies the feasibility of solutions based on the alignment between the required and available information items. The third step studies the alignment between the desired effects of solutions and the effects addressed by the also prepares the experimental measurement of these effects in the industrial context. Applied to the Renault scenario, this approach successfully identified a set of eight promising RT methods, together with a set of metrics to evaluate them. Using the metrics, we also studied the margins for improvement in the current RT process. While the principle of considering context, effects and information factors proved successful, the lower-level content of the taxonomy was not as useful as it could have been. The use of the taxonomy was hindered by hard-to-interpret elements, missing elements or insufficiently detailed ones, and errors in the classification of candidate methods. Our experience thus shows that the taxonomy needs improvement. A means to manage the improvement would be to deploy and maintain a public repository. Users would find updated versions of the taxonomy, a documentation, a database of papers mapped to the taxonomy, and could contribute by feedback. The version published in [START_REF] Ali | On the search for industry-relevant regression testing research[END_REF] may not be considered as a finished product.

Regarding the real-world scenario, the study continues with the experimentation of the methods. Four methods have already been implemented. They are being applied to the historical project data, confirming the relevance of some.

Figure 1 :

 1 Figure 1: Research structure.

Figure 2 :

 2 Figure 2: Global view of Project A.

Table 1 .

 1 Selected papers on regression testing from[START_REF] Ali | On the search for industry-relevant regression testing research[END_REF]

	ID	Ref	ID	Ref	ID	Ref	ID	Ref
	S1	[8]	S11	[9]	S21 [10]	S31 [11]
	S2	[12]	S12 [13]	S22 [14]	S32 [15]
	S3	[16]	S13 [17]	S23 [18]	S33 [19]
	S4	[20]	S14 [21]	S24 [22]	S34 [23]
	S5	[24]	S15 [25]	S25 [26]	S35 [27]
	S6	[28]	S16 [29]	S26 [30]	S36 [31]
	S7	[32]	S17 [33]	S27 [34]	S37 [35]
	S8	[36]	S18 [37]	S28 [38]	S38 [39]
	S9	[40]	S19 [41]	S29 [42]
	S10 [43]	S20 [44]	S30 [45]

Table 2 .

 2 Added papers on ML-based methods

	ID	Ref Method	ID	Ref	Method
	L1 [46]	SL	L8	[47]	RL
	L2	[6]	SL	L9	[48]	RL
	L3 [49]	SL	L10 [50]	SL + RL
	L4	[7]	SL	L11 [51] SL + NLP
	L5 [52]	UL	L12 [53] SL + NLP
	L6 [54]	UL	L13 [55] SL + NLP
	L7 [56]	RL	L14 [57] UL + NLP

Table 3 .

 3 Context of Project A using taxonomy of[START_REF] Ali | On the search for industry-relevant regression testing research[END_REF]

		Context factor	Project A
		Size	Large-scale
	System-	Complexity	Multi-language
	related	Type of the system	Automotive embedded system
	Process-	Testing process	Weekly integration and issues allocation
	related	Test technique	Automatic and manual
	People-	Cognitive factors	Not relevant
	related	Organizational factors	Multiple companies

Table 4 .

 4 Desired effects for Project A using the taxonomy

		Desired effects
	Test coverage	Feature coverage Input (Pairwise)
		Reduction of test suite
		Reduced testing time	+
		Improved precision
	Efficiency &	Decreased time for fault detection ++
	effectiveness	Reduced need for ressources
		Fault detection capability	+
		Severe fault detection	+
		Reduced cost of failure
	Awareness	Transparency of testing decisions

Table 5 .

 5 Information for Project A using the taxonomy of[START_REF] Ali | On the search for industry-relevant regression testing research[END_REF] and mapping of the set of papers

		Information	Project A	State of the Art Papers
		No of changes in a requirement		S21
		Fault impact	X	S21
	Requeriments	Subjective implementation complexity Perceived completeness	X X	S21 S21
		Perceived treaceability	X	S21
		Customer assigned priority	X	S21, S33, L14
	Design	System models		S13-S18, S27, S35
	artifacts Target variant		S26
		Type of test		S26, L14
		Model coverage		S13-S20
		Functionality coverage		S3, S4
		Static priority	X	S26
		Age	X	S26, L11
	Test	Fault detection probability (estimated)	X*	S22, S29, S33, L13
	cases	Execution time (estimated)	X	L9, L10, L12, L13 S22, S29, L4, L7, L8,
		Cost (estimated)	X*	S22, S33, L12
		Link to requirements	X	S21, S22
		Link to faults	X	S4, S21
		Link to source code		S6-S8, L2
		Execution time	X*	S29, S32
		Database-states		S36
		Invocation counts		S28
		Invocation chains		S28, S31
	Test executions	Runtime component coverage Method coverage Code coverage		S31 S28, L1, L6 L2, L3, L5, L11, L13 S5, S23, S29, S37, S38,
		Browser states		S36
		Class coverage		S6, L10
		Execution time	X*	S3, S4, S13-S18, S28
		Verdicts	X	S1-S4, S13-S18, S26, S32, S34, L1, L2, L4, L6, L7, L8, L9, L10, L11
		Severity	X	S28, S33
	Test reports	Link to packages and their revisions Link to branch Build type		S1 S32 S32
		Link to failure		S13-S18
		Test session		S13-S18, S26
		Variant under test		S32
		Link to fixed file / link to source code		S24, S25, L2, L3
		Fix-time	X*	S32
	Issues	Link to test case	X	S24, S25, S37, L3, L12, L13, L14
		Failure severity	X	S3, S4, L12, L14

Table 6 .

 6 Technically feasible methods according to the taxonomy or a manual labeling (differences in bold)

	Labeling method	Selected papers
	Taxonomy-based labeling	S22, S33, S34, L4, L7, L8, L9, L12
	Manual labeling (ground truth)	S3, S22, S26, S34, L4, L7, L8, L9

Table 7 .

 7 Disagreements with the mapping of papers in[START_REF] Ali | On the search for industry-relevant regression testing research[END_REF]

	ID	Disagreements
	S3	Same mapping as S4 (same authors), but actually uses less information
	S9-S12	Needs the source code of user functions, as well as links from test cases to the code
		Focuses on product lines, but no
	S13-S18	variability-related item is marked. All papers have the same mapping
		while they consider different data subsets
	S19-S20	Wrong mapping to design artefacts/code dependencies (uses a black-box model)
	S26	Wrong mapping to test cases/target variant (does not use variability data)
	S27	Needs the model coverage of test cases
		Wrong mapping to test cases/fault
	S29	detection probability (the paper merely says
		that fault detection could be added)
	S32	Needs the cost of test cases
	S33	Needs the functionality coverage and target variant of test cases
	S35	Needs the model coverage and target variant of test cases

Table 8 .

 8 Addressed Effects

	Addressed effects	Method
	Reduced testing time	S3, L4, L7, L8, L9
	Decreased time for fault detection	S3, S22, S26, L4, L7, L8, L9
	Fault detection capability	S26, S34
	Severe fault detection	S3
	Circulation of the tests	S26, L9

ACKNOWLEDGMENT

The authors would like to thank the teams at Renault Software Factory that work on Project A for their help.

Code dependencies S19, S20, S37, L10 Code change / Revision history S1, S2, S5, S7, S8, S24, S25, S38, L1, L2, L3, L10, L13 Source file S2, S7, S8, S30, S37, L11 Source code No of Contributors S32 Class dependencies S6 Intermediate code Code changes (method or class) S2, S6, S28, L1, L10, L11 Revision history X S6, S29 Component changes S9-S12, S31 Binary code Binary files X S6, S9-S12, S23, S29